Article · Sensors · May 16, 2021
SpeakingFaces: A Large-Scale Multimodal Dataset of Voice Commands with Visual and Thermal Video Streams
Madina Abdrakhmanova, Askat Kuzdeuov, Sheikh Jarju, Yerbolat Khassanov, Michael Lewis, Hüseyin Atakan Varol
Abstract
We present SpeakingFaces as a publicly-available large-scale multimodal dataset developed to support machine learning research in contexts that utilize a combination of thermal, visual, and audio data streams; examples include human-computer interaction, biometric authentication, recognition systems, domain transfer, and speech recognition. SpeakingFaces is comprised of aligned high-resolution thermal and visual spectra image streams of fully-framed faces synchronized with audio recordings of each subject speaking approximately 100 imperative phrases. Data were collected from 142 subjects, yielding over 13,000 instances of synchronized data (∼3.8 TB). For technical validation, we demonstrate two baseline examples. The first baseline shows classification by gender, utilizing different combinations of the three data streams in both clean and noisy environments. The second example consists of thermal-to-visual facial image translation, as an instance of domain transfer.
More by Askat Kuzdeuov
- MMHAR-28: Human Action Recognition Across RGB, Thermal, Depth, and Event ModalitiesIEEE Transactions on Biometrics Behavior and Identity Science · 2026
- Qolda: A Small Vision–Language Model for the Kazakh LanguageIEEE Access · 2026 · Citations: 8
- Multilingual Speech Command Recognition with Language IdentificationAnnual Conference of the IEEE Industrial Electronics Society · 2025
- Real-Time Multispectral Human Pose EstimationAnnual Conference of the IEEE Industrial Electronics Society · 2025