The first offline LLM arena for Kazakh

Community residents Sanzhar Murzakhmetov, Sanzhar Umbet, Beksultan Sagyndyk, and Kirill Yakunin launched the first offline LLM arena for Kazakh. Its main goal is to test not merely next-token generation, but understanding of cultural context as a whole.

What the team built:

• A custom culture-focused QA dataset
• Topics and keywords assembled with linguists using Serper, Perplexity, and LLM generation
• Pairwise model comparisons, first judged by GPT-4o and finalized with a Bradley–Terry model

Current results:

• Google DeepMind's Gemma models are consistently strong
• Sherkala-8B from MBZUAI ranks second, outperforming even larger models
• ISSAI at Nazarbayev University has good multiple-choice scores but some of the weakest generation results