Emotionally expressive facial animation driven by EmotionBERT embeddings

Abstract

Facial animation driven by speech signals plays a pivotal role in virtual reality, human-computer interaction, and entertainment. Traditional methods often struggle to capture the rich emotional and semantic context conveyed by the speaker's voice, resulting in less engaging animations. This paper introduces Speech2Face, a framework that synthesizes high-fidelity facial movements conditioned on expressive speech inputs. The proposed approach utilizes EmotionBERT to extract semantic and emotional embeddings from speech transcripts, enabling the generation of temporally coherent and contextually appropriate facial expressions. Experiments on VOCASET, IEMOCAP, MEAD, and BIWI datasets demonstrate that Speech2Face improves emotional accuracy by up to 80.8%, naturalness, and synchronization between speech and facial movements compared to existing models. This framework advances multi-modal emotion-aware synthesis, offering an effective solution for creating engaging and realistic digital humans. Our code is available at https://github.com/rizwanchouhan/speech2face.

More by Galiya Ybytayeva

• Physics-informed neural network framework for fast and accurate nanofluid heat-transfer prediction in microchannel heat sinksFrontiers in Physics · 2026 · Citations: 2
• Thermal mode modeling using neural network technologies and the finite element methodBulletin of Electrical Engineering and Informatics · 2026
• Precision healthcare in action: A robust MRI framework for breast cancer diagnosis with enhanced feature analysis and boundary refinementBiomedical Signal Processing and Control · 2026
• Explainable multi agent reinforcement learning framework for secure and adaptive communication in UAV swarm based fanetsScientific Reports · 2026 · Citations: 6