Skip to content
Publications

Talk · International Conference on Computer Science and Engineering · September 17, 2025

Bridging the Gap in Less-Resourced Languages: Building a Benchmark for Kyrgyz Language Models

Timur Turatali, Aida Turdubaeva, Islam Zhenishbekov, Zhoomart Suranbaev, Anton Alekseev, Rustem Izmailov

Abstract

The evaluation of Large Language Models (LLMs) across diverse languages is crucial for ensuring equitable technological progress. However, most multilingual benchmarks are created by translating existing English-centric datasets, a process that often introduces errors and fails to capture the unique linguistic and cultural nuances of the target language. This is especially problematic for lower-resourced languages like Kyrgyz, which lack dedicated, high-quality evaluation resources. To address this gap, we introduce KyrgyzLLM-Bench, the first comprehensive benchmark suite designed to measure deep language understanding in Kyrgyz. Our suite features two natively created components: a massive multitask multiple-choice dataset, KyrgyzMMLU, covering subjects from the Kyrgyz national curriculum, and a reading comprehension dataset, KyrgyzRC. These are supplemented by carefully translated versions of WinoGrande, HellaSwag, BoolQ, TruthfulQA and GSM8K to assess commonsense reasoning, reading comprehension and math. We provide an extensive evaluation of over X state-of-the-art open- and closed-source LLMs, analyzing their performance in zero-shot and few-shot settings. Our results provide the first in-depth look at the capabilities and limitations of modern LLMs on complex Kyrgyz tasks. We publicly release KyrgyzLLM-Bench to foster further research and development for the Kyrgyz language.

Topic ModelingNatural Language Processing TechniquesMultimodal Machine Learning ApplicationsKyrgyz languagemultilingual language modelslow-resource languages

More by Timur Turatali