Highly Language-Independent Word Lemmatization Using a Machine-Learning Classifier

Abstract

Lemmatization is a process of finding the base morphological form (lemma) of a word. It is an important step in many natural language processing, information retrieval, and information extraction tasks, among others. We present an open-source language-independent lemmatizer based on the Random Forest classification model. This model is a supervised machine-learning algorithm with decision trees that are constructed corresponding to the grammatical features of the language. This lemmatizer does not require any manual work for hard-coding of the rules, and at the same time it is simple and interpretable. We compare the performance of our lemmatizer with that of the UDPipe lemmatizer on twenty-two out of twenty-five languages we work on for which UDPipe has models. Our lemmatization method shows good performance on different languages from various language groups, and it is easily extensible to other languages.

More by Irina Ualiyeva

• Detecting AI-Generated User Reviews in Kazakh: A Study on BERT Model Performance and False Positive ReductionLecture notes in computer science · 2026
• BENCHMARKING DEEP LEARNING MODELS FOR FEW-SHOT CLASSIFICATION OF KAZAKH NEWS TEXTS UNDER LIMITED ANNOTATIONHerald of Kazakh-British technical university · 2026
• Parallel Memetic Differential Evolution for Minimum Sum-of-Squares ClusteringCommunications in computer and information science · 2025
• COMPARATIVE ANALYSIS OF THE EFFECTIVENESS OF TRANSFORMER AND CONVOLUTIONAL NEURAL NETWORK ARCHITECTURES FOR AUTOMATIC CLASSIFICATION OF RICE LEAF DISEASESHerald of Kazakh-British technical university · 2025