Abstract
The evolution of 5G and the anticipated introduction of 6G technologies significantly increases network complexity through service-based architecture, network slicing, virtualization and distributed cloud-native functions. These advancements improve scalability and flexibility but simultaneously expand the attack surface and introduce novel vulnerabilities. Traditional penetration testing methodologies are not suitable for such dynamic and virtualized environments because they rely on static procedures and manual testing that cannot match the speed and structural variability of modern mobile networks. In parallel, machine learning–based intrusion detection systems (IDS) demonstrate strong capabilities in detecting anomalous and zero-day behaviors but operate independently from penetration testing processes. This paper presents an intelligent hybrid method for automated penetration testing in 5G and beyond networks, integrating a machine-learning intrusion detection system based on incremental learning, autoencoders and generative adversarial networks (GANs) – with an attack optimization module driven by the Differential Evolution (DE) algorithm. Instead of a genetic algorithm, DE is employed due to its fast convergence, robustness to local minima, and suitability for optimizing high-dimensional representations of attack strategies. An experimental evaluation on a real OpenAirInterface-based 5G Standalone testbed demonstrates that the DEdriven approach improves vulnerability identification efficiency, produces optimal multi-stage attack strategies, and enables realistic automated penetration scenarios. These results indicate that DE-based optimization provides a scalable, adaptive and efficient foundation for continuous security assessment of next-generation mobile networks.