A Review on Deep Learning Autoencoder in the Design of Next-Generation Communication Systems

Abstract

Traditional mathematical models for communication-system design often use simplifying assumptions and are difficult to extend to complex or rapidly changing channels. In recent years, deep learning, and in particular autoencoders (AEs), has enabled end-to-end optimization of transmitters and receivers and has shown strong performance in many communication scenarios. This paper provides a comprehensive review of AE-based communication systems across wireless, optical (fiber and wireless), semantic, and quantum domains. The review is based on recent works indexed in major digital libraries and is organized around four cross-cutting design themes: channel modeling and differentiability, complexity and scalability, generalization and model mismatch, and data scarcity and realism. For each theme, as well as for each application domain, we summarize representative AE architectures, training strategies, reported performance, and practical limitations. The paper also outlines how to quantify the computational complexity of AE-based transceivers using Big-O notation and discusses implications for deployment in resource-constrained environments. Finally, we identify open challenges, including robust learning over non-differentiable or partially known channels, scalable and hardware-aware architectures, and standardized evaluation protocols, and we provide guidelines for future research on AE-based designs for next-generation communication systems.

More by Mohammed Alaa Fadhil Al-hadeethi

• Autonomous agents and multi-agent systems: a comprehensive reviewArtificial Intelligence Review · 2026
• Density-Adaptive Locust Swarm Optimization for Cloud Workflow SchedulingCode Ocean · 2026
• Density-Adaptive Locust Swarm Optimization for Cloud Workflow SchedulingCode Ocean · 2026
• CompProv Produces Machine Readable Graphs Encoding Microscopic Algebraic Provenance for Reproducible ComputationarXiv · 2026