Talk · IEEE International Conference on Application of Information and Communication Technologies · October 1, 2018
Document and Word-level Language Identification for Noisy User Generated Text
Zhanibek Kozhirbayev, Zhandos Yessenbayev, Aibek Makazhanov
Abstract
We present herein our work on language identification applied to comments left by the readers of online news sites popular in Kazakhstan. Such comments are typically written in one of the two languages spoken widely in the area (Kazakh and Russian) and sometimes - in a mixture of both. Code-switching (mixing languages) makes it desirable to identify language not only on document, but also on individual word level. We approach both tasks in a single two-step framework, performing unsupervised normalization and Nave Bayes text classification procedures successively. Moreover, we applied deep learning model based on recurrent networks with LSTM cell in order to classify text. Our results suggest improvement over the state-of-the-art for Kazakh language.
More by Zhanibek Kozhirbayev
- Sequential Domain Adaptation of Whisper for Kazakh Academic Lecture Speech RecognitionLecture notes in computer science · 2026
- The comparative evaluation of explanations for algorithmic programming problems provided by a Human vs. ChatGPT: an exploratory online user studyResearch Square · 2026
- Enhancing Personalised Learning with Graph-Based Ensemble Prediction and Skill Cluster Mapping for Student Knowledge CompletenessComputers · 2026
- Evaluating Large Language Models for Text-to-Gloss Translation in Kazakh-Russian Sign Language: A Pilot StudyLanguage Resources and Evaluation Conference · 2026