Article · Cogent Engineering · January 1, 2020
Morphological segmentation method for Turkic language neural machine translation
Ualsher Tukeyev, Aidana Karibayeva, Zhandos Zhumanov
Abstract
Dictionaries play an important role in neural machine translation (NMT). However, a large dictionary requires a significant amount of memory, which limits the application of NMT and can cause a memory error. This limitation can be solved by segmenting each word into morphemes in parallel source corpora. Therefore, this study introduces a new morphological segmentation approach for Turkic languages based on the complete set of endings (CSE), which reduces the vocabulary volume of the source corpora. Herein, we demonstrate the proposed CSE-based morphological segmentation method for the Kazakh, Kyrgyz, and Uzbek languages and present the results of computational NMT experiments for the Kazakh language. The NMT experiment results show that in comparison with byte-pair encoding (BPE)-based segmentation, the proposed CSE-based segmentation increases the bilingual evaluation understudy score of 0.5 and 0.2 points on average for Kazakh–English and English–Kazakh pairs, respectively. Furthermore, in comparison with the BPE-based segmentation, the proposed CSE-based segmentation approach reduced the vocabulary size in NMT by more than a factor of two. This feature of the proposed segmentation approach will be crucial for NMT as the size of the source corpora is increased to improve translation quality.
More by Aidana Karibayeva
- Speech-to-Speech Translation from Kazakh into Turkic Languages: Pretrained Speech Encoders, Turkic Speech Units, and Lessons from Negative ResultsComputers · 2026
- Toward an Adaptive Speech-to-Summary Pipeline for Turkic Languages: Language-Specific ASR, Conditional Morphology-Aware Correction, Pivot Translation, and SummarizationComputers · 2026
- RESTORING PUNCTUATION IN AUTOMATIC SPEECH RECOGNITION FOR KAZAKH LANGUAGEEKTU Journal of Information and Communication Sciences · 2026
- From Speech to Readable Text: Automatic Punctuation in ASR Systems of the Kazakh LanguageCommunications in computer and information science · 2026
