Skip to content
DSML.KZ
Publications

Article · Computers · January 5, 2026

Hybrid GNN–LSTM Architecture for Probabilistic IoT Botnet Detection with Calibrated Risk Assessment

Тетяна Василівна Бабенко, Kateryna V. Kolesnikova, Yelena Bakhtiyarova, Damelya Maksutovna Yeskendirova, Kanibek Sansyzbay, Askar K. Sysoyev

Kateryna Kolesnikova
Open via DOI

Abstract

Detecting botnets in IoT environments is difficult because most intrusion detection systems treat network events as independent observations. In practice, infections spread through device relationships and evolve through distinct temporal phases. A system that ignores either aspect will miss important patterns. This paper explores a hybrid architecture combining Graph Neural Networks with Long Short-Term Memory networks to capture both structural and temporal dynamics. The GNN component models behavioral similarity between traffic flows in feature space, while the LSTM tracks how patterns change as attacks progress. The two components are trained jointly so that relational context is preserved during temporal learning. We evaluated the approach on two datasets with different characteristics. N-BaIoT contains traffic from nine devices infected with Mirai and BASHLITE, while CICIoT2023 covers 105 devices across 33 attack types. On N-BaIoT, the model achieved 99.88% accuracy with F1 of 0.9988 and Brier score of 0.0015. Cross-validation on CICIoT2023 yielded 99.73% accuracy with Brier score of 0.0030. The low Brier scores suggest that probability outputs are reasonably well calibrated for risk-based decision making. Consistent performance across both datasets provides some evidence that the architecture generalizes beyond a single benchmark setting.

Network Security and Intrusion DetectionInternet Traffic Analysis and Secure E-votingInternet of Things and AIIoT botnet detectiongraph neural networkslong short-term memory

More by Kateryna Kolesnikova