Article · Scientific Journal of Astana IT University · June 30, 2026

FINE-TUNING SMALL LANGUAGE MODELS FOR MENTAL HEALTH TEXT CLASSIFICATION: A PARAMETER-EFFICIENT APPROACH OUTPERFORMING ZERO-SHOT LARGE LANGUAGE MODELS

Dastan Kapizov, Praveen Kumar

Abstract

Automated analysis of user-generated text is increasingly used to support scalable mental-health assessment, but deploying large language models for classification remains costly and latency-sensitive. This study evaluates whether parameter-efficient fine-tuning (PEFT) of a small model can provide a better accuracy–efficiency trade-off. We fine-tune SmolLM2-1.7B on two benchmarks: SWMH (5-class mental-health classification) and Dreaddit (binary stress detection). Evaluation follows two protocols: (1) primary full held-out test evaluation for the fine-tuned model, and (2) paired evaluation on identical 200-example subsets for fair cross-model comparison with zero-shot GPT-4o-mini and GPT-4o. On full held-out tests, SmolLM2-1.7B achieves 0.723 accuracy, 0.720 weighted F1, and 0.727 macro F1 on SWMH, and 0.808 accuracy, 0.808 weighted F1, and 0.808 macro F1 on Dreaddit. On paired subsets, SmolLM2-1.7B outperforms both zero-shot GPT baselines on both datasets (SWMH accuracy: 0.705 vs 0.600/0.645; Dreaddit accuracy: 0.755 vs 0.650/0.675). These results indicate that task-specific PEFT can make small models practically deployable and, under controlled paired evaluation, stronger than zero-shot larger models.

Mental Health via WritingSentiment Analysis and Opinion MiningTopic Modelingparameter-efficient fine-tuningsmall language modelsstress detection

FINE-TUNING SMALL LANGUAGE MODELS FOR MENTAL HEALTH TEXT CLASSIFICATION: A PARAMETER-EFFICIENT APPROACH OUTPERFORMING ZERO-SHOT LARGE LANGUAGE MODELS

Abstract

More by Praveen Kumar