Enhancing Kazakh Sign Language Recognition with BiLSTM Using YOLO Keypoints and Optical Flow

Abstract

Sign languages are characterized by complex and subtle hand movements, which are challenging for computer vision systems to accurately recognize. This study suggests an innovative deep learning pipeline specifically designed for reliable gesture recognition of Kazakh Sign Language. This approach combines key point detection using the YOLO model, optical flow estimation, and a bidirectional long short-term memory (BiLSTM) network. At the initial stage, a dataset is generated using MediaPipe, which is then used to train the YOLO model in order to accurately identify key hand points. After training, the YOLO model extracts key points and bounding boxes from video recordings of gestures, creating consistent representations of movements. To improve the recognition of dynamic gestures, the optical flow is calculated in an area covering 10% of the area around key points, which allows the dynamics of movements to be captured and provides additional time characteristics. The BiLSTM network is trained on multimodal input that combines data on keypoints, bounding boxes, and optical flow, resulting in improved gesture classification accuracy. The experimental results demonstrate that the proposed approach is superior to traditional methods based solely on key points, especially in recognizing fast and complex gestures. The proposed structure promotes the development of sign language recognition technologies, especially for poorly studied languages such as Kazakh, paving the way to more inclusive and effective communication tools.

Related publications

• Bidirectional Kazakh Sign Language prosody-aware translation using computer vision and speech recognition techniquesFrontiers in Artificial Intelligence · 2026 · Same author: Maria Aouani
• DARTS Meets Ants: A Hybrid Search Strategy for Optimizing KAN-Based 3D CNNs for Violence Recognition in VideoApplied Sciences · 2025 · Citations: 2 · Same author: Maria Aouani
• Sign Language Recognition based on Deep Learning via MediaPipe for people with speech and hearing impairmentsProcedia Computer Science · 2026 · Citations: 2 · Shared topic: Hand Gesture Recognition Systems
• An AI-Driven Real-Time Sign Language Proficiency Assessment with Adaptive FeedbackProcedia Computer Science · 2026 · Shared topic: Hand Gesture Recognition Systems