Talk · Procedia Computer Science · January 1, 2026

An AI-Driven Real-Time Sign Language Proficiency Assessment with Adaptive Feedback

K. Bhargav Prasad, P. Charan Sai, K. Lohith, C. Jyotsna, Shinu M. Rajagopal

Abstract

Effective speaking and listening are essential for social, academic and work life. However the hearing and speech impaired people find it difficult to get good tools to communicate because they are unavailable in most of the tests. The majority of the current sign language assessment systems provide feedback, which is untimely thereby reducing the effectiveness of learning. The purpose of the research is to present a platform based on AI that will be used to immediately evaluate the level of sign language proficiency. By using 3D data of the joints of the hands obtained through MediaPipe, the platform can use a Dense Neural Network (DNN) to detect American Sign Language (ASL) gestures with technical accuracy. The system provides instant individual feedback regarding performance of gestures and progress made by the user. Such a strategy will be in line with the Sustainable Development Goals (SDGs): Quality Education (SDG 4), Industry, Innovation and Infrastructure (SDG 9), and Reduced Inequalities (SDG 10) since it will expand educational access and communication possibilities. The evaluation performance shows that the platform has a final accuracy of 99.78% in classifying static ASL gestures and a good macro and weighted average of 97%, which indicates good performance and general capacity to new unseen data. The system aims at bridging the gap between gesture performance and the correct feedback to the end result where the speech- and hearing-impaired community can improve their communication levels.

Hand Gesture Recognition SystemsHearing Impairment and CommunicationHuman Pose and Action RecognitionAmerican Sign Languagereal-time feedbackgesture recognition

An AI-Driven Real-Time Sign Language Proficiency Assessment with Adaptive Feedback

Abstract

More by Jyotsna Chandrasekharan