A Multimodal Framework for Speech Emotion Recognition in Low-Resource Languages

Abstract

Speech emotion recognition (SER) plays a crucial role in enhancing human–computer interaction by identifying emotional states in speech. However, low-resource languages like Kazakh face challenges due to limited datasets and linguistic tools. To address this problem, we propose a novel multimodal framework, KEMO (Kazakh Emotion Multimodal Optimizer), which combines text-based semantic analysis and audio emotion recognition to leverage complementary features of linguistic and paralinguistic data. Using a Kazakh-translated version of the DAIR-AI (Contextualized Affect Representations for Emotion Recognition) dataset for text and the RAVDESS (Ryerson Audio-Visual Database of Emotional Speech and Song) dataset for audio, we have developed a system capable of classifying six emotions from text and eight emotions from audio. By integrating outputs from speech-to-text and audio-based recognition models with adaptive weighting, KEMO significantly improves the accuracy and robustness of emotion classification, providing an effective solution for SER in low-resource language scenarios.

More by Altanbek Zulkhazhav

• Quantifying the Intelligibility–Expressivity Tradeoff in Emotion-Conditioned Kazakh Text-to-SpeechInternational Conference on Affective Computing and Intelligent Interaction Workshops and Demos · 2026
• Acoustic and Emotional Annotation of a Web-Harvested Kazakh Speech Corpus for Text-to-Speech2026
• Monitoring and comparative analysis of containerized environmentsFrontiers in Computer Science · 2026
• A Hybrid Attention-LSTM Model with Embedding-based Feature Representation for Student Academic Performance PredictionInternational journal of intelligent engineering and systems · 2026