Talk · Ural Symposium on Biomedical Engineering, Radioelectronics and Information Technology · May 14, 2026

Parameter-efficient fine-tuning of large language models for Tatar language: a comparative study of LoRA and QLoRA

Rustam Arifovich Burnashev, Гильмуллин Ринат Абрекович, Mullosharaf K. Arabov

Abstract

This article investigates the application of large language models (LLMs) to the Tatar language. We evaluate parameter-efficient fine-tuning methods, LoRA and QLoRA, under limited computational resources using a subset of the IPSAN/tat_monocorpus_v2 corpus ($\mathbf{1 0, 0 0 0}$ samples). Fully finetuned GPT-2 family models ($\mathbf{1 2 4 M - 3 5 4 M}$ parameters) serve as baselines, alongside state-of-the-art models Mistral, Qwen, and DeepSeek adapted via QLoRA. Our results show that QLoRA adaptation of Mistral-7B achieves a perplexity of 3.98 on the test set, comparable to fully fine-tuned gpt2-medium (2.91). Notably, QLoRA requires training only $\mathbf{4 1. 9 M}$ parameters ($\boldsymbol{\lt} \mathbf{0. 6 \%}$ of the full model), two orders of magnitude fewer than full fine-tuning. Mistral-7B outperforms Qwen2.5-7B (PPL 5.44) and DeepSeek-7B (PPL 5.32) in adapting to Tatar. Increasing LoRA rank from 8 to 16 consistently improves performance across all architectures. Qualitative analysis confirms that Mistral-7B with QLoRA generates coherent, grammatically correct Tatar texts with accurate usage of language-specific characters (ә, ө, ң) and agglutinative forms. These findings demonstrate that parameter-efficient adaptation of multilingual LLMs is a promising approach for developing language models for Tatar and other Turkic languages.

Topic ModelingNatural Language Processing TechniquesDomain Adaptation and Few-Shot LearningTatar languageparameter-efficient fine-tuningLoRa

Parameter-efficient fine-tuning of large language models for Tatar language: a comparative study of LoRA and QLoRA

Abstract

More by Mullosharaf Arabov