Talk · International Conference on Computer Science and Engineering · September 17, 2025
Bridging the Gap in Less-Resourced Languages: Building a Benchmark for Kyrgyz Language Models
Timur Turatali, Aida Turdubaeva, Islam Zhenishbekov, Zhoomart Suranbaev, Anton Alekseev, Rustem Izmailov
Abstract
The evaluation of Large Language Models (LLMs) across diverse languages is crucial for ensuring equitable technological progress. However, most multilingual benchmarks are created by translating existing English-centric datasets, a process that often introduces errors and fails to capture the unique linguistic and cultural nuances of the target language. This is especially problematic for lower-resourced languages like Kyrgyz, which lack dedicated, high-quality evaluation resources. To address this gap, we introduce KyrgyzLLM-Bench, the first comprehensive benchmark suite designed to measure deep language understanding in Kyrgyz. Our suite features two natively created components: a massive multitask multiple-choice dataset, KyrgyzMMLU, covering subjects from the Kyrgyz national curriculum, and a reading comprehension dataset, KyrgyzRC. These are supplemented by carefully translated versions of WinoGrande, HellaSwag, BoolQ, TruthfulQA and GSM8K to assess commonsense reasoning, reading comprehension and math. We provide an extensive evaluation of over X state-of-the-art open- and closed-source LLMs, analyzing their performance in zero-shot and few-shot settings. Our results provide the first in-depth look at the capabilities and limitations of modern LLMs on complex Kyrgyz tasks. We publicly release KyrgyzLLM-Bench to foster further research and development for the Kyrgyz language.
More by Timur Turatali
- KyrgyzLLM-Bench: Benchmarking Kyrgyz Language UnderstandingarXiv · 2026
- KyrgyzDentex: A Unique Multi-Modal Dental Radiograph Dataset from Kyrgyzstan with Classical Machine Learning and Deep Learning Benchmarks for Caries DetectionInternational Conference on Computer Systems and Technologies · 2026
- AgroExpert KG: A Hybrid Machine Learning and Rule-Based Decision Support System for Crop Yield Prediction in KyrgyzstanInternational Conference on Electronics, Computer and Computation · 2026
- National-Scale Glacier Mass Balance Forecasting in Kyrgyzstan: Machine Learning Insights Into Domain Shift Until 2049International Conference on Electronics, Computer and Computation · 2026
