Abstract
This article examines the problem of optimizing security policies in software-defined networks (SDN). The authors propose solving this problem using NSGA-II, an efficient algorithm for evolutionary optimization of multiple objectives. The proposed approach aims to achieve a balance between the need to strictly enforce network security and the availability of limited computing resources. Particular attention is paid to factors such as data transmission latency and TCAM table size, which significantly affect the effectiveness of traffic filtering. Based on the model developed in the article, which includes threat probability assessments, objective function normalization methods, and the use of penalty coefficients for rule conflicts, optimization was performed across three key parameters: attack risk, network latency, and TCAM load. The simulation covered three network operation scenarios-normal, mixed, and attack-using Mininet and Ryu packets. A comparison of the proposed method with differential evolution (DE) and a greedy algorithm (Greedy) showed that NSGA-II achieves optimal solution distribution along the Pareto frontier, converges faster, and maintains filtering accuracy. The paper also presents visualization of generational transitions, tradeoff graphs, and load profiles. The conclusion discusses the potential for integrating the proposed model with ONOS and OpenDaylight controllers, and discusses the feasibility of using hybrid solutions based on Deep Reinforcement Learning, Federated Learning, and Explainable AI.