Skip to content
DSML.KZ
Publications

Talk · International Conference on Computer Science and Engineering · September 17, 2025

Development of Visual Data Processing Algorithm for the Sign Language Recognition System Based on CBFT-BiLSTM

Lazzat Zholshiyeva, Tamara Kokenovna Zhukabayeva, Meruert Berdiyeva

Lazzat Zholshiyeva
Open via DOI

Abstract

This paper presents the development of a visual data processing algorithm for a real-time Kazakh Sign Language (KazSL) recognition system using the CBFT-BiLSTM model. The system leverages multimodal vision language models, specifically CLIP and BERT, to generate semantic embeddings for both video and text. A projection module is proposed to align these embeddings into a shared semantic space, enabling accurate matching using cosine similarity. A fine-tuned BiLSTM classifier is used to recognize the sentence class. The model is evaluated on two KazSL datasets (KRSL20 and QazSL) and achieves high accuracy (94,98% and 76%) with low confusion rates between similar signs. The system operates effectively in real-time with an average latency of less than 100 milliseconds per sentence. The proposed method demonstrates the potential of adapting vision language models for low-resource languages.

Hand Gesture Recognition SystemsMultimodal Machine Learning ApplicationsHearing Impairment and Communicationsign language recognitionCLIPBERT

More by Lazzat Zholshiyeva