Skip to content
DSML.KZ
Publications

Preprint · SSRN Electronic Journal · January 1, 2026

Rafale: Towards the Efficient Resource-Aware Federated Knowledge Distillation on Heterogeneous Clients

Harsh Pratap Singh, Debasis Das, Gozzal B. Eshniyazova, Doston Khasanov, Halimjon Khujamatov

Doston Khasanov
Open via DOI

Abstract

Abstract Knowledge distillation is an highly effective machine learning technique for transferring knowledge from a large model, i.e., teacher, to a smaller model, i.e., student, improving model efficiency without sacrificing performance. This paper proposes a novel resource-aware federated learning framework \textit{Rafale} that incorporates an adaptive client clustering mechanism based on computational resource availability and model complexity. High-resource clients train complex teacher models, and low-resource clients leverage distilled knowledge from teachers to train lightweight student models. Further, to optimize the aggregation process by introducing a clustering-based weight-sharing mechanism that minimizes communication overhead while maintaining model accuracy. In this paper, a revolutionary Rafale framework that greatly reduces communication cost and separates model training from architectural constraints is proposed. To increase scalability and robustness, the suggested method makes use of resource-aware client selection, adaptive information sharing, and effective distillation techniques. In comparison to state-of-the-art FL approaches, experimental data show that RA-FKD delivers greater accuracy, decreases communication costs by up to 76\%, and improves system efficiency. A cosine similarity loss \textit{KDC} is introduced in knowledge distillation to improve the alignment between teacher and student models, enhancing knowledge transfer. Experimental evaluations on CIFAR-10 and CIFAR-100 datasets demonstrate that proposed framework achieving student test accuracy of 91.15\% with reduction of 76.08\% communication cost.

Privacy-Preserving Technologies in DataDomain Adaptation and Few-Shot LearningStochastic Gradient Optimization Techniquesheterogeneous clientsclient selectioncommunication cost reduction

More by Doston Khasanov