DONNA: Distributed Optimized Neural Network Allocation on CIM-Based Heterogeneous Accelerators

Abstract

The continued development of neural network architectures continues to drive demand for computing power. While data center scaling continues, inference away from the cloud will increasingly rely on distributed inference on multiple devices. Most prior efforts have focused on optimizing singledevice inference or partitioning models to enhance inference throughput. Meanwhile, energy consumption continues to grow in importance as a factor of consideration. This work proposes a framework that searches for optimal model splits and distributes the partitions across the combination of devices taking into account throughput and energy. Participating devices are strategically grouped into homogeneous and heterogeneous clusters consisting of general-purpose CPU and GPU architectures, as well as emerging Compute-In-Memory (CIM) accelerators. The framework simultaneously optimizes inference throughput and energy consumption. It is able to demonstrate up to $4 \times$ speedup with approximately $4 \times$ per-device energy reduction in a heterogeneous setup compared to single GPU inference. The algorithm also finds a smooth Pareto-like curve in the energy-throughput space for CIM devices.

More by Kamilya Smagulova

• From Algorithms to Bitstreams: A Survey of FPGA-Based Acceleration for Transformer InferencePreprints.org · 2026
• Efficient and real-time perception: a survey on end-to-end event-based object detection in autonomous drivingFrontiers in Robotics and AI · 2025 · Citations: 7
• Chimera: a block-based neural architecture search framework for event-based object detectionFrontiers in Artificial Intelligence · 2025 · Citations: 4
• Adversarial Robustness and Transferability on Stochastic Resistive AcceleratorsInternational Conference on Machine Intelligence and Smart Innovation · 2025