First results from Kazakhstan's Kazakh-language LLM leaderboard

• MMLU in Kazakh: Evaluates multitask language understanding
• GSM8K in Kazakh: Tests mathematical reasoning
• Kazakh Constitution MC: Tests legal and civic knowledge
• Kazakh Dastur MC: Evaluates understanding of cultural heritage and customs
• Unified National Test MC: Covers history, literature, biology, and other subjects

We are happy to share that the first model-comparison results are now available!

• OpenAI's general-purpose GPT-4o confidently holds first place among all models
• In the open-source segment, Kazakhstan's ISSAI laboratory leads with LLaMA-3.1-KazLLM-1.0-8B

Next steps:

• Release an offline arena with ELO ratings
• Add more challenging benchmarks for real-world tasks
• Release custom instruction datasets

More details:

• View the leaderboard
• Submit your model

The team welcomes feedback and invites everyone interested to collaborate!