Skip to content
DSML.KZ
Publications

Article · IEEE Access · January 1, 2026

Classroom Emotional State Analysis Using Proxy Metrics and Multimodal Data

Tole Bi Yermek, Banu Yergesh, Amanzhol Yelemessov, Madina Yelemessova, Gulmira Bekmanova, Assel Omarbekova

Zhanar Lamasheva
Open via DOI

Abstract

Understanding the emotional climate of a classroom is an important factor in analyzing student engagement and the effectiveness of the learning process. However, traditional educational analytics mainly relies on exam scores, attendance records, and surveys, which do not capture real-time changes in students' emotional states during lessons. This study proposes an approach for analyzing classroom emotional states using behavioral proxy metrics derived from video recordings of university classes. Instead of direct emotion recognition or student identification, the method analyzes group-level behavioral indicators, including the Attention Index (AI), Engagement Index (EI), Calmness Index (CI), and Group Interaction Index (GI), computed from posture, body orientation, and movement dynamics. Because the archived MP4 files did not contain embedded audio streams, the experimental classification in this study relies on video-derived proxy metrics. To demonstrate how acoustic information could be incorporated in a future multimodal version of the system, an illustrative supplementary analysis was conducted using simulated acoustic descriptors aligned with the annotated video fragments, including Root Mean Square (RMS) energy, fundamental frequency (F0) variability, and Mel-Frequency Cepstral Coefficient (MFCC)-based spectral variation. The pilot analysis showed that positive classroom states accounted for approximately 50% of the annotated observations, while negative states accounted for approximately 30%. Comparison between manual annotations and automatic classification achieved an agreement of approximately 70%. In addition, correlation analysis revealed a strong relationship between attention and engagement indices (r ≈ 0.99). The results provide preliminary evidence that video-derived proxy metrics can characterize the emotional background of a learning environment and support the analysis of its temporal dynamics during a lesson. The supplementary simulated acoustic analysis further illustrates a possible path for future multimodal integration, but it is not used as evidence of audio-based classification performance in the present dataset. However, larger-scale annotation and validation of the complete archive are required to further assess the reliability and generalizability of the proposed framework. The proposed approach can support educational analytics and the development of intelligent systems for monitoring and improving the learning process.

Emotion and Mood RecognitionOnline Learning and AnalyticsEducational Technology and Pedagogystudent engagementattention indexengagement index

More by Zhanar Lamasheva