Abstract
The increasing complexity and frequency of network attacks, especially zero-day attacks, pose enormous challenges for traditional signature-based intrusion detection tools. This study proposes a hybrid approach based on an autoencoder and isolation forest to detect unknown attacks on network traffic. The CIC-IDS2017 dataset, which contains over 2 million labels of normal and malicious traffic, was used for the experiment. Data preprocessing included feature normalization, gap handling, and label binarization. The autoencoder was trained on normal traffic to understand standard network behavior, while isolation forest was used to detect rare deviations in traffic. When an anomaly was detected, the hybrid model labeled all traffic as anomalous. Precision, recall, F1-score, and ROC-AUC metrics were used to evaluate the model. Experimental results showed that the proposed hybrid method, based on the two models used, provides higher detection rates for first-day attack similarities compared to single models with a comparable false positive rate. These results confirm the feasibility of combining multiple models for intrusion detection tasks.