Article · Computers · May 28, 2026

A Lightweight Multimodal Architecture for Punctuation Restoration in Kazakh ASR

Aidana Karibayeva, Oleg Myssov, Balzhan Abduali, Dina Amirova, Adina Karybayeva

Abstract

In this paper, we first present a multimodal architecture called CrossAttn-v1. This model is designed to recover punctuation marks in Kazakh and combines contextual XLM-RoBERTa-large text embeddings with the Whisper large-v3 encoder states via a cross-attention mechanism. In addition, a 4-dimensional prosodic vector and a CRF output layer are used. The model was trained using an adapted Whisper ASR model on 33,332 utterances from the KazakhTTS2 corpus. After adaptation, the word error rate decreased from 45.7% to 4.25%. On the in-domain test set (56,396 tokens), CrossAttn-v1 achieved F1-macro = 0.8485 for recovering five-class punctuation marks. Furthermore, CrossAttn-v1 outperformed the GPT-4o zero-shot model by +0.294 F1 and the M3 Hybrid model based on prosody alone by +0.070 F1. The class analysis showed that the Whisper encoder states were particularly useful for prosody-dependent punctuation. For example, it outperformed M3 Hybrid by +9.5 percentage points on the QUESTION mark and by +20.2 percentage points on the EXCLAIM mark. On 883 out-of-domain natural speech recordings, the model performed similarly to the text-only baseline model (Δ = −0.041, not significant), suggesting that domain mismatch in the Whisper training corpus was a major factor limiting generalization.

Speech Recognition and SynthesisSpeech and Audio ProcessingSpeech and dialogue systemspunctuation restorationKazakh languagemultimodal architectures

A Lightweight Multimodal Architecture for Punctuation Restoration in Kazakh ASR

Abstract

More by Oleg Myssov