Skip to content
DSML.KZ
Publications

Talk · International Siberian workshops and tutorials on electron devices and materials · June 1, 2026

Boosting LLM Adaptation: Superior Stability and Convergence via OLoRA+ Across Diverse NLP Benchmarks

Samariddin Kushmuratov, Fatima Аdilova, Давронов Рифкат Раxимович, Ruzmat Safarov, Farrux Boymurodov, Azamjon Nemadaliev

Fatima Adilova
Open via DOI

Abstract

In the rapidly evolving landscape of artificial intelligence, adapting Large Language Models (LLMs) under significant resource constraints requires increasingly efficient fine-tuning techniques. However, many existing methodologies often treat weight initialization and optimization as separate, disjointed concerns, leading to sub-optimal training dynamics and slower convergence.This paper introduces OLoRA+, a novel and sophisticated hybrid approach designed to bridge this gap by synergistically combining the structural stability of Orthonormal Low-Rank Adaptation (OLoRA) with the accelerated convergence properties of LoRA+. By carefully initializing adapter matrices via the QR decomposition of pretrained weights and applying differential learning rates to the projection matrices, OLoRA+ enhances both the structural stability of the weights and the overall feature learning speed.Empirical evaluations conducted on the popular Alpaca dataset, alongside six major NLP benchmarks, demonstrate that OLoRA+ consistently and significantly outperforms standard OLoRA and LoRA baselines. Remarkably, it achieves this performance boost without increasing the overall computational or memory costs during the adaptation process. Furthermore, our detailed analysis identifies two distinct effective learning regimes-a “Refinement” strategy $(\lambda\lt 1)$ and an “Exploration” strategy $(\lambda\gt 1)$-positioning OLoRA+ as a more versatile, robust, and scalable framework for high-performance LLM adaptation in diverse operational environments.

Domain Adaptation and Few-Shot LearningTopic ModelingNatural Language Processing TechniquesLLM fine-tuninglow-rank adaptationLoRa

More by Fatima Adilova