Article · Frontiers in Artificial Intelligence · May 18, 2026

Bidirectional Kazakh Sign Language prosody-aware translation using computer vision and speech recognition techniques

Mukhtar Zhassuzak, Zholdas Buribayev, Maria Aouani, Ainur Yerkos, Aigerim Yerimbetova

Abstract

Introduction: This study presents a bidirectional communication system designed to enhance interaction between hearing-impaired and hearing individuals using gesture recognition. Methods: The proposed framework integrates multiple components, including Kazakh Sign Language (KSL) gesture detection, closed-set sentence classification, speech recognition, and speech generation. The system is trained and evaluated using a dataset consisting of images and video recordings of KSL gestures together with audio data for Kazakh speech synthesis. Results: Experimental evaluation demonstrates that the overall system achieves a sentence classification accuracy of 92% across 12 closed-set sentence classes, while individual model accuracy did not fall below 87%. Compared to other approaches, the proposed method achieved competitive results in terms of accuracy on the collected dataset; however, inference speed comparisons are module-specific and measured as latency in ms/sample. Discussion: The results confirm the feasibility of the proposed approach as a proof-of-concept for improving accessibility and communication through automated sign language understanding. The system demonstrates practical applicability in bridging the communication gap between signers and non-signers, thereby promoting greater accessibility.

Hand Gesture Recognition SystemsHearing Impairment and CommunicationSpeech and Audio Processingsign language translationgesture recognitiontwo-way communication

Bidirectional Kazakh Sign Language prosody-aware translation using computer vision and speech recognition techniques

Abstract

Related publications