Skip to content
DSML.KZ
Publications

Dataset · Zenodo · October 15, 2026

Blind pairwise evaluation materials for dictionary-free spelling normalization in a rule-based Uzbek–English MT system (KKT)

Ziyoviddin Sirojiddinov, Muftakh Khakimov

Muftakh Khakimov
Open via DOI

Abstract

Materials for the final blind pairwise evaluation of the dictionary-free spelling normalization layer (IMLO-2, KKT_IMLO_A) in the rule-based Uzbek–English machine translation system KKT. The archive allows the numbers in Table 5 of the accompanying paper, Cohen's κ and Gwet's AC1 to be recomputed. All files are byte-identical copies from the project repository. System state. Evaluation date 2026-10-08; production build 2026.10.08-49 (code commit 31f5252); dictionary database md5 9d3adf8d. Pool: development set E2119 (2,119 sentences, not a test set); 60 of the 174 sentences whose translation changed under normalization were randomly sampled. Judges. GPT-6 Luna (OpenAI, ChatGPT Temporary Chat) and claude-opus-5-5 (Anthropic, Claude Code). Each judge received two orders of the same 60 pairs (positions swapped, items reshuffled), each order in a separate chat/session. Files. Item files (kor_paket_gpt_1.csv, kor_paket_gpt_2.csv); the unblinding key (kor_kalit.json, with SHA-256 committed before any judgment was collected); raw judge outputs (kor_paket_*_natija.csv); the judge prompt (kor_korsatma_60.md) and pre-registered decoding/acceptance rule (kor_qoida.md); the decoding script (kor_dekod.py, Python 3 standard library) with its output (kor_hukm.json) and manual reason–choice consistency classification (kor_sabab_tasnif.json); and a discarded first run (discarded_run1/), excluded before decoding for design deviations and reported as supplementary only. The README is in Uzbek. Reproduction. Running kor_dekod.py regenerates kor_hukm.json byte-identically. From the 3×3 GPT×Claude agreement matrix in kor_hukm.json, κ = 0.27 and AC1 = 0.68 are reproduced. Agreed pairs: 40 better : 3 tie : 1 worse; the acceptance criterion is met. Integrity: MANIFEST.sha256.

Natural Language Processing TechniquesTranslation Studies and PracticesSpeech Recognition and Synthesisspelling normalizationrule-based machine translationinter-judge agreement

More by Muftakh Khakimov