New benchmarks evaluate LLMs in Kazakh

Great news for everyone interested in adapting LLMs to the Kazakh language!

Community members Sanzhar Murzakhmetov @stringersolo, Beksultan Sagyndyk @bsagyndyk, and Kirill Yakunin @JenarosHat released a series of benchmarks for language understanding, reasoning, and other features of Kazakh.

1) The team translated several popular datasets into Kazakh:

• MMLU in Kazakh: Evaluates multitask language understanding
• GSM8K in Kazakh: Tests mathematical reasoning

2) They created custom Kazakhstan-focused multiple-choice benchmarks:

• Kazakh Constitution MC: Tests legal and civic knowledge
• Kazakh Dastur MC: Evaluates understanding of cultural heritage and customs
• Unified National Test MC: Covers history, literature, biology, and other subjects

The team is building an open leaderboard for these benchmarks and a platform for testing and comparing models developed for Kazakh.

Read more in the team's LinkedIn post.