Analysis of the Efficiency of Subword Tokenizers in a Low-Resource Linguistic Environment: Implementation Experience for the Tajik Language

Abstract

Abstract Contemporary approaches to subword text tokenization are considered in application to the low-resource Tajik language, which is characterised by a complex morphological structure and high variability of word forms. In the course of the study, a large-scale heterogeneous corpus was compiled and preprocessed, comprising 99 books and 134 497 textual articles of various genres and topics, with a total volume exceeding 33 million tokens. The corpus was cleaned of noise, normalised, and used as a basis for training and subsequent testing of subword models. On the basis of this corpus, five tokenization models were trained and analyzed, implementing the BPE, WordPiece, and Unigram algorithms using the Hugging Face Tokenizers and SentencePiece libraries. A comparative assessment was carried out according to a number of key indicators, including the out-of-vocabulary (OOV) rate, text compression ratio, tokenization speed, as well as n-gram distribution characteristics that make it possible to evaluate the ability of the models to reflect the morphological and structural organization of the language. The experimental results revealed the strengths and weaknesses of various approaches to subword segmentation and identified the most effective tokenization strategies under the conditions of the morphological complexity of the Tajik language. The findings may be used in the development of language models and applied NLP tools for Tajik and other low-resource languages, thus contributing to the expansion of their presence in the digital environment.

More by Mullosharaf Arabov

• Parameter-Efficient Adaptation of Large Language Models to a Low-Resource Agglutinative Language: A Corpus, Datasets, and a Reproducible Benchmark for the Bashkir LanguagearXiv · 2026
• A comprehensive study of Tatar text classification methods: from corpus creation and Tatar2Vec embeddings to benchmark establishmentModeling and Analysis of Information Systems · 2026 · Citations: 1
• A Comprehensive Analysis of Arabic Natural Language Processing Research: Trends, Topic Evolution, and Research Gaps -- A Bibliometric and Topic-Based StudyarXiv · 2026
• Large Language Models for Low-Resource Languages: A Conceptual Framework for an Electronic Explanatory Dictionary of the Tajik LanguagearXiv · 2026