Processing Inconsistency Predicts Language Competence: LLM Evaluation Without Answer Labels on Turkic Languages

Abstract

Most languages lack labeled evaluation benchmarks for large language models (LLMs). Creating such benchmarks requires native speakers, domain expertise, and answer annotation - resources unavailable for the vast majority of languages. We investigate whether a model's internal processing signals - such as generation entropy and tokenizer statistics - correlate with its actual accuracy on a language, with the long-term goal of estimating language competence without labeled data. Our key observation is that for languages a model does not know, both tokenizer segmentation and generation entropy become highly variable across questions, whereas for known languages they remain consistent. We call this the inconsistency hypothesis and test it on 11 instruction-tuned LLMs (1B-70B parameters) across 14 language-script varieties (12 Turkic plus English and Russian controls). Standard deviation outperforms the mean as an accuracy predictor in 21 of 25 greedy-pass family $\times$ measure combinations (binomial $p < 0.001$, though the measures are not fully independent); sampling-based measures show no such pattern (4 of 20). A three-feature proxy combining a tokenizer coverage ratio chars_std / fertility), entropy standard deviation, and the model's EN/RU benchmark accuracy achieves leave-one-language-out $R^2 = 0.75$ and Kendall $\tau = 0.72$ on 154 model-language pairs, with the tokenizer ratio contributing most of the language-level signal ($\Delta R^2 = 0.34$). Removing the EN/RU anchor isolates an anchor-free component at $R^2 = 0.49$, which exceeds the model-mean baseline and captures language coverage but not model capability.

More by Ilya Galyukshev

• Knowing When Not to Answer: Cross-Domain and Multi-Turn Generalization of Latent Underspecification SignalsarXiv · 2026
• Too Many Tools, Too Much Confusion? Navigating Agentic Tool Selection at ScaleAlgorithms · 2026 · Citations: 2
• DynamicMCPBench: A Trace-Grounded, Effect-Scored Benchmark for LLM Agents over Live MCP ServersConference on Empirical Methods in Natural Language Processing · 2026 · Citations: 1
• Processing Inconsistency Predicts Language Competence: LLM Evaluation Without Answer Labels on Turkic LanguagesProceedings of the conference - Association for Computational Linguistics. Meeting · 2026