This section presents a comprehensive experimental validation of the effectiveness of the proposed method for the IoT attack detection dataset NSL-KDD and CIC-IDS-2017. The performance benefits of our approach have been systematically demonstrated through rigorous polyphase testing and comparative research.
Classification results experiment
For different attack patterns, the corresponding defensive strategies are very different. Identifying attack samples is not enough to protect your device from compromise. Therefore, multiclass classification experiments are essential to classify attack subtypes. Experimental results for the proposed method for both datasets and the corresponding confusion matrix are shown in Table 4. Figure 3, respectively.
As shown in Table 4, the proposed method achieved an accuracy of 99.21%, recalled with NSL-KDD, with accuracy and F1 score of 99.22%, and false reporting rate (FAR) of 4.18%. For the CIC-IDS-2017 dataset, all five metrics reach 99.83%, with FAR of 0.11%. These results demonstrate the robust performance of the methods in both datasets, effectively identifying each attack category while maintaining resilience to false positives caused by normal traffic fluctuations.
Confusion matrix is widely regarded as the most intuitive and fundamental tool for assessing model performance. This paper visualizes the distinctions between classes through heatmap intensity, color differences, and brightness. If the darkest cells align along the diagonal, this indicates the correct prediction for most of the sample. The confusion matrix for both datasets is shown in Figure 3.
Confusion matrix visualization.
Comparison of feature selection methods
To examine the advantages of the proposed feature selection method, we compare it with four traditional methods: chi-square test, random forest, mutual information, and recursive function removal (RFE), as shown in Table 5.
As shown in Table 5, for the NSL-KDD dataset, the proposed method achieves excellent performance on four of the five evaluation metrics (except far away), with a recall improving by 0.23-2.05%. For the CIC-IDS-2017 dataset, this method is better than the other methods across all five metrics. This is primarily because the Chi-square test assumes functionality independence, whereas the dataset contains complex nonlinear relationships. Similarly, mutual information and RFE methods also ignore correlations between functions. Although Random Forest is an ensemble model, it is easily and overly suited to high-dimensional data, resulting in poor generalization of selected features. In contrast, the method proposed in this paper uses GA for feature selection of IoT attack detection datasets. This not only fully utilizes the unique statistics of the network traffic sample, but also shows the scientific rigor of both.
Comparison of class imbalance techniques
To address the problem of class imbalance in an IoT environment where abnormal events occur with low probability, the proposed class balancing method is compared with several representative existing methods. Detailed results are shown in Table 6.
As shown in Table 6, the proposed class balancing method achieves strong performance on both datasets, effectively balancing the balance between accuracy and recall trade-offs. For the NSL-KDD dataset, the proposed method achieves the best results across all five evaluation metrics, with the largest improvement being 20.95%, 6.88%, 31.80%, and 21.29% for the first four metrics, respectively. Similarly, in the CIC-IDS-2017 dataset, the proposed method outperforms other class-balancing approaches, demonstrating consistently superior performance across all five metrics, maintaining the effectiveness of strong attack detection.
The advantages of the proposed method over other class-balancing approaches lie in the following aspects: ROS and Small rely on simple data replication that leads to overfitting. Methods such as Rus + Small and Kmeans + Small employ a variety of strategies to mitigate overfitting, but the over-problem remains as the synthesized samples tend to be very similar. Based on K-Nearest Neighbors, Adasyn, which assigns different weights to different classes for data generation, is vulnerable to outliers. These traditional sampling methods are inherently shallow learning techniques, lacking the ability to learn deeply from the original data and are not effective in distinguishing between noisy data. As a result, they tend to produce a significant amount of noisy samples, which reduces the performance of the model. Deep learning methods such as CVAE and CWGAN can learn the essential distribution of sample data, but can produce meaningless samples and suffer from problems such as model collapse and vanishing gradients. In contrast, the proposed method employs the EQL V2 approach with the reweighting mechanism of gradient induction, whether it alters the overall distribution of the original dataset by synthesizing samples, or suffers from explosion or disappearance gradients due to the ability to balance the accumulated positive and negative gradients. The result is consistently excellent performance across a variety of rating metrics.
Comparative analysis with existing methods
To further explore the advancements in proposed methods in IoT attack detection, we conduct benchmark comparisons with cutting-edge models using various datasets. Comprehensive results are presented systematically in Table 7, showing significant performance benefits across key security metrics.
(1) KD-TCNN13:KD-TCNN extracts features using triplet CNN-based knowledge distillation models to achieve classification.
(2) TLHA29:TLHA designs a three-layer hybrid model for intrusion detection.
(3) CNN-LSTM30:CNN-LSTM proposes an efficient hybrid IDS model that integrates Black Widow Optimization Algorithm with convolutional neural networks and long-term short-term memory networks.
(4) MFNET14: MFNET identifies the multifrequency of network traffic by constructing multifrequency LSTM and multifrequency transformer modules.
(5) Takugan17:Tacgan designs a tabular auxiliary classifier-generated adversarial network model for oversampling attack samples to address class imbalance issues.
(6) SVM-GAC16:SVM-GAC introduces augmented learning to SVM to detect new types of network attacks, thereby improving detection performance.
(7) BT-TPF31:BT-TPF utilizes Siamese networks to reduce functional dimensions of high-dimensional network traffic and employs knowledge distillation to reduce model complexity.
As can be seen from Table 7, compared to existing methods, the method proposed in this paper is generally optimal for three datasets. In the NSL-KDD dataset, the method recall rate is slightly lower than the CNN model's recall rate, but the precision rate is 1.74% higher. Furthermore, detection results are also superior to that model in terms of comprehensive F1 scores. For the CIC-IDS-2017 dataset, all models exhibit good detection performance. This is because the CIC-ID-2017 dataset contains a large amount of data and is relatively simple. In this case, our method is only 0.1% lower in accuracy than SVM-GAC, but 3.97%, 0.39%, and 0.23% higher, respectively. Furthermore, from the perspective of F1 scores, our method is 4.02%, 0.92%, 0.37% and 0.23% higher than these four models, respectively.
The reason why the performance of other models is poor in the models in this paper is the fact that CNN-LSTM and PYDSC cannot effectively address the problem of imbalance in the class of datasets, resulting in poor detection performance than other models. KD-TCNN and BT-TPF realize lightweight anomaly detection models through knowledge distillation, but these two models are inferior to the models in this paper on all four indicators, as the reduction in parameters leads to poor performance. TLHA uses undersampling to synthesize samples, resulting in lower data quality and limited model performance improvements. The SVM-GAC model utilizes adaptation algorithms based on mutations and perceptual strategies to identify malicious features and achieve relatively important results. However, this model performs slightly poorly, as SVM has limited ability to extract deep information from the sample. Wgan, McGan, Fcwgan, and Tacgan were unable to resolve the problem of model collapse and generated data that did not fit the original distribution. In summary, the model in this paper can effectively identify specific attack methods and protect network devices from threats to some extent.
Ablation research
To verify the need for each component in the proposed method, ablation experiments were performed on the model, and the proposed approach consisted mainly of four components: Ga, CNN, LSTM, and EQL V2 are abbreviated as g, c, l, and e, respectively. The ablation experiment was performed on the NSL-KDD dataset, with detailed results shown in Table 8.
As shown in Table 8, of all components, CNN had the most impact on experimental results, leading to a decrease of 2.72%, 2.98%, 2.72% and 3.18% across the four evaluation metrics. This is mainly because CNN can extract high-quality spatial features. This is to provide a solid foundation for subsequent classifiers. The next most influential components are EQL V2 and LSTM, both of which result in approximately 0.7% degradation on all metrics. This may be due to the fact that EQL V2 effectively deals with class imbalance issues, but LSTM further improves the model by incorporating spatiotemporal traffic information. Finally, using GA not only eliminates redundant features, but also reduces computational costs, but also improves classification performance. In summary, the ablation study fully demonstrates the need for each component in the proposed method and emphasizes its overall effectiveness.
Furthermore, timely attack detection and identification is equally important in IoT environments with resource constraints. Therefore, we also analyzed the training and inference times for the proposed method. As shown in Table 8, our approach achieves relatively high rankings in both training and inference efficiency. Despite the incorporation of some additional components, this model is still capable of detecting network attacks in a timely manner. Variants without CNN modules indicate a slightly lower time, but this comes at the cost of decreasing detection accuracy. As a result, our method balances detection performance with computational efficiency.
