KazakhTextDuplicates: A Controlled Multi-Regime Benchmark for Semantic Deduplication, Semantic Similarity, and Retrieval in Kazakh

Abstract

Reliable evaluation resources for semantic deduplication, semantic textual similarity (STS), and retrieval remain limited for low-resource and morphologically rich languages. This study introduces KazakhTextDuplicates, a Kazakh-language benchmark for controlled evaluation of semantic duplication under varying levels of semantic preservation and surface-form distortion. The benchmark includes two complementary versions. KazakhTextDuplicates v1.0 is a diagnostic dataset derived from naturally occurring duplicate and near-duplicate pairs for analyzing relationships between duplication labels and lexical overlap. KazakhTextDuplicates v2.0 is a controlled benchmark created using deterministic transformations that define seven semantic duplication regimes: exact duplication, contextual reformulation, paraphrasing, partial semantic overlap, and three levels of character-level noise. Each pair is assigned both a regime label and a predefined similarity score, enabling evaluation across duplicate classification, STS, and retrieval tasks. Five embedding models (BGE-M3, LaBSE, Multilingual-E5-Large, KazEmbed-v5, and OpenAI text-embedding-3-large) were evaluated in a zero-shot setting. Results show that v1.0 is strongly affected by surface-form similarity and is therefore more suitable for diagnostic analysis. In contrast, v2.0 provides a more challenging and informative evaluation environment. OpenAI text-embedding-3-large achieved the strongest STS performance (Pearson = 0.510, Spearman = 0.652, MSE = 0.075) and the best duplicate regime classification results (Accuracy = 0.155, Macro-F1 = 0.066). Retrieval performance remained strong at higher cutoffs despite lower first-rank stability. The results demonstrate that benchmark design substantially affects semantic similarity evaluation and emphasize the need for controlled assessment in low-resource agglutinative languages.

More by Aidos Mukhatayev

• Machine Learning Analysis of Scientific Projects for Decision Support: Empirical Evidence from CORDIS FP6Applied Sciences · 2026
• INTEGRATION OF THE INFORMATION SYSTEM “QUALITY MANAGEMENT OF HIGHER AND POSTGRADUATE EDUCATION” WITH RELATED SYSTEMS2026
• Development of a Conceptual Model of an Intelligent Digital Ecosystem of Science ManagementIEEE International Conference on Smart Information Systems and Technologies · 2026
• EduDesignEval: A Structured Evaluation Framework for Educational Program Designer Using Large Language ModelsInternational Journal of Information and Education Technology · 2026