Skip to content
DSML.KZ
Publications

Talk · AIP conference proceedings · January 1, 2025

Development of a hybrid algorithm for identifying named entities in 20th century Uzbek texts

Davlatyor B. Mengliev, Nilufar Z. Abdurakhmonova, Oybek Allamov, Bahodir B. Ibragimov, Bobur Rashidovich Saidov, Nodirbek Boltayev

Nilufar Abdurakhmonova
Open via DOI

Abstract

The paper discusses the development of a hybrid algorithm for recognizing named entities in Uzbek texts of the 20th century. Automatic processing of these texts is complicated by the agglutinative nature of the Uzbek language, multiple script reforms (from Arabic to Latin, then to Cyrillic and back to Latin), as well as the presence of dialects and dialect words. Existing methods and tools developed for other languages are ineffective for the Uzbek language due to its strong agglutinative dependence. In this paper, a hybrid approach is proposed that combines rule-based algorithms and a language model based on mBERT. To train the model, a custom language corpus was created consisting of 3000 sentences marked up using the BIOES scheme. Rule-based algorithms include transliteration of the text from Cyrillic to Latin and standardization of dialect words by replacing them with formal equivalents using dictionaries and morphological analysis. The results of the experiments showed that on the test set of 20th century texts, the accuracy was 91.4%, recall was 89.6%, and F1-measure was 90.5%. On the modern set of texts (21st century), the algorithm also showed high results with an F1-measure of 86.0%, demonstrating the ability to generalize. The developed algorithm effectively solves the problems of recognizing named entities in Uzbek texts.

Topic ModelingNatural Language Processing TechniquesHandwritten Text Recognition Techniquesnamed entity recognitionUzbek languagemultilingual BERT

More by Nilufar Abdurakhmonova