Talk · September 17, 2025

Restoring Punctuation and Capitalization in Kazakh: A BERT-Based Approach for Text Normalization

Sanzhar Umbet, Zhanibek Kozhirbayev

Abstract

This paper introduces a punctuation and capitalization (PC) restoration model for Kazakh, developed using the bert-base-multilingual-uncased model within NVIDIA’s NeMo framework. The model was trained on a curated dataset of preprocessed Kazakh Wikipedia articles and digitized books, stripped of irrelevant symbols and data. It demonstrates strong performance in predicting punctuation marks and word casing, achieving micro-average F1 scores of 98.04 for capitalization and 96.75 for punctuation. However, it struggles with rare markers, such as exclamation marks (F1: 32.85). While capitalization tasks show consistent success, punctuation restoration, particularly for less frequent markers, remains challenging. This model lays a solid foundation for improving automatic speech recognition (ASR) outputs and advancing natural language processing (NLP) applications in Kazakh, contributing to better post-processing and text normalization for low-resource languages.

Natural Language Processing TechniquesSpeech Recognition and SynthesisTopic ModelingKazakh languagepunctuation restorationtext normalization

Restoring Punctuation and Capitalization in Kazakh: A BERT-Based Approach for Text Normalization

Abstract

More by Sanzhar Umbet