An LLM-based synthetic data generation approach for addressing class imbalance in malicious traffic detection

LUFlow network intrusion detection dataset

This section presents a comparative experimental analysis of synthetic data generation approaches based on LLM, SMOTE, ADASYN and TVAE in the context of imbalanced multiclass classification. The primary objective of this analysis is to evaluate how different data-level augmentation strategies influence model training stability and classification performance under controlled and unified experimental conditions.

All experiments were implemented in Python (version 3.12) using the following libraries: TensorFlow 2.20.0 (with the integrated Keras API), imbalanced-learn 0.14.1, SDV 1.26.0, scikit-learn 1.8.0, SciPy 1.16.1, and pandas 2.3.3. LLM-based synthetic sample generation was performed via the OpenAI API using the official openai Python library (version 1.99.9).

To ensure a fair comparison, all methods are assessed using identical data preprocessing pipelines, model architectures, training configurations, and evaluation protocols. The analysis focuses on two complementary aspects: the stability of the training process across multiple runs, and the impact of synthetic data on class-level classification metrics, with particular emphasis on recall for the critical minority class. This approach enables an objective assessment of whether LLM-based synthetic data generation provides tangible advantages over classical interpolation-based oversampling methods when applied under realistic augmentation constraints.

To enable a fair and reproducible comparison between the proposed LLM-based synthetic data generation approach and classical oversampling techniques, SMOTE and ADASYN were implemented using the imbalanced-learn library, which provides widely adopted reference implementations for interpolation-based augmentation methods.

To ensure consistency across all experiments, identical preprocessing pipelines, model architectures, and training configurations were applied to all evaluated approaches. A unified feature scaling procedure was employed to map all input variables to a common numerical range, and the same neural network architecture and hyperparameter settings were maintained throughout the experiments in order to isolate the data augmentation strategy as the only varying factor.

For all evaluated augmentation strategies (SMOTE, ADASYN, and the LLM-based approach), the number of synthetic samples added to the training data for the minority class (Class 1) was kept approximately equal (≈ 44,000 samples) to ensure a fair and controlled comparison.

In addition, the training process for all models incorporated two standard callbacks: EarlyStopping, configured to monitor the validation loss with a fixed patience parameter, and ModelCheckpoint, also based on validation loss. These mechanisms were applied consistently across all experiments to prevent overfitting and to ensure that performance comparisons reflect stable and well-converged model states.

The dataset was partitioned into training, validation, and test subsets. The test set accounted for 20% of the original data and remained unchanged across all experiments, while 20% of the remaining training data was reserved for validation. Synthetic samples were incorporated exclusively during model training, whereas both validation and test sets contained only real data, ensuring an unbiased evaluation and preventing information leakage.

No explicit global random seed was enforced across all stages of model initialization and training. However, controlled randomness was partially introduced through fixed random_state (value 42) parameters during dataset splitting and synthetic data shuffling procedures (LLM and TVAE-based augmentation). All experiments were repeated multiple times (10) to explicitly account for stochastic variability arising from random weight initialization and training dynamics. This experimental design enables a stability-oriented assessment of the compared augmentation strategies, rather than relying on single-run performance estimates. Figure 9 presents a stability analysis of the classification results based on the coefficient of variation (CV) computed over multiple independent training runs. Lower CV values indicate reduced variability and, consequently, more stable model behavior.

The results show that the use of synthetic data generally reduces performance variability compared to the baseline model trained on the original dataset. This effect is most pronounced for the Recall metric. The baseline configuration exhibits a Recall coefficient of variation of 4.17%, indicating a noticeable sensitivity to stochastic effects during training. When LLM-generated synthetic data are incorporated, the Recall variability decreases to 3.09%, reflecting a more stable learning process across repeated runs.

In comparison with classical oversampling techniques, the LLM-based approach demonstrates lower Recall variability than SMOTE (CV = 5.24%) and ADASYN (CV = 4.58%), while being comparable to the TVAE-based method (CV = 3.06%). Overall, the LLM-based approach achieves a balanced level of variability, providing improved training stability while preserving sufficient diversity in the augmented dataset.

To further investigate the observed differences in training stability, Figs. 10 and 11 present the mean learning dynamics of Precision and Recall for the minority class across training epochs, averaged over multiple independent runs. The LLM-based augmentation exhibits smoother and more consistent recall trajectories compared to SMOTE and ADASYN, with reduced oscillations throughout training, which is consistent with the lower recall variability observed in the CV analysis. While TVAE shows slightly more stable learning curves, its recall converges to a lower level, indicating that increased stability alone does not necessarily lead to improved minority class sensitivity. Overall, the learning dynamics confirm that the LLM-based approach achieves a balanced trade-off between training stability and effective recall optimization, which explains the superior average performance reported in Fig. 12.

Figure 12 summarizes the average classification performance for the minority class (Class 1) across different data augmentation strategies. In the context of malicious state detection, Recall is considered the primary evaluation metric, as the cost of missed detections is significantly higher than that of false positives.

As shown in Fig. 12, the use of LLM-generated synthetic data leads to a substantial improvement in recall for the minority class, increasing the score from 0.786 for the baseline model to 0.846. This improvement exceeds the gains obtained using classical oversampling techniques, with SMOTE and ADASYN achieving recall values of 0.781 and 0.805, respectively, while TVAE attains a recall of 0.796. These results indicate that LLM-based augmentation more effectively enhances the model’s sensitivity to minority-class instances under constrained augmentation conditions.

Importantly, the improvement in Recall achieved by the LLM-based approach is accompanied by only a minor reduction in Precision. Although the Precision score for the LLM-based model (0.684) is slightly lower than that of the baseline configuration (0.699), it remains comparable to the values obtained with SMOTE (0.700) and is only marginally lower than that of TVAE (0.698) and ADASYN (0.698). This indicates that the observed increase in Recall is not primarily driven by a disproportionate rise in false positive predictions.

As a consequence, the F1-score improves from 0.740 for the baseline model to 0.756 when LLM-generated synthetic data are used, outperforming SMOTE (0.738), ADASYN (0.747), and TVAE (0.743). Overall, these results demonstrate that LLM-based synthetic data generation provides a favorable balance between detection sensitivity and prediction reliability, yielding the strongest overall performance among the evaluated augmentation methods.

To further analyze the source of these performance gains, Fig. 13 presents the averaged confusion matrices obtained for different augmentation strategies.

The confusion matrix analysis confirms that the improved Recall achieved by the LLM-based approach primarily stems from a reduction in false negatives for the minority (malicious) class, while maintaining a comparable level of false positives relative to SMOTE, ADASYN, and TVAE. This observation indicates that the observed performance gains are driven by more effective identification of malicious instances rather than by overly aggressive decision boundaries.

At the same time, a moderate increase in misclassification of outlier samples as malicious can be observed. This effect is likely associated with a shift in the decision boundary induced by the denser representation of the malicious class in feature space following LLM-based augmentation. Since outliers typically occupy borderline or low-density regions, they become more susceptible to reassignment toward the malicious class under a recall-oriented optimization regime. This behavior reflects a precision–recall trade-off rather than instability of the model.

Figure 14 presents the Precision–Recall curves for the minority (malicious) class evaluated on the test set. The curves, averaged over 10 training runs, show that the LLM-based augmentation exhibits Precision–Recall behavior comparable to the baseline and classical oversampling methods across the considered recall range. This indicates that the overall ranking ability of the classifier remains largely unchanged. At the same time, the observed recall improvements can be attributed to a shift in the operating point toward a more recall-oriented regime, resulting in increased sensitivity to minority-class instances with a minor reduction in precision. Since PR-AUC reflects performance across all decision thresholds, this localized shift does not substantially affect the overall area under the curve and remains consistent with the confusion matrix analysis.

Figure 15 presents the ROC curves for the minority (malicious) class evaluated on the test set. The results indicate that all augmentation strategies achieve similar AUC values, suggesting comparable class separability across different operating points. This further confirms that the observed performance differences are not driven by changes in global class separability, but rather by differences in operating characteristics reflected in recall-focused metrics. Figure 16 reports the results of Wilcoxon signed-rank tests on minority-class recall, providing a formal statistical significance assessment of the observed performance differences.

To assess whether the observed improvements in minority class recall are statistically robust across multiple training runs, a paired Wilcoxon signed-rank test was conducted, comparing each augmentation strategy against the baseline configuration. The results, summarized in table (Fig. 19), indicate that only the improvement achieved with the LLM-based approach is statistically significant at α = 0.05 (p = 0.00195). In contrast, the recall gains observed for SMOTE, ADASYN, and TVAE do not reach statistical significance, despite moderate mean recall improvements and, in the case of TVAE, relatively stable training behavior.

Taken together, the presented results demonstrate that LLM-based synthetic data generation enables improvements in minority class detection performance even under constrained augmentation conditions. The observed gains achieved with a limited volume of synthetic samples suggest that the generated data capture informative and representative patterns of the target class, rather than merely increasing sample density through interpolation.

These findings provide a strong empirical basis for further discussion regarding the practical implications and trade-offs of LLM-based augmentation strategies in comparison to classical oversampling methods.

The UNSW-NB15 dataset

To validate the generality of the obtained results, additional experiments were conducted on the UNSW-NB15 Dataset³⁵, which is formulated as a binary classification problem distinguishing malicious and benign network traffic. Accordingly, the neural network architecture was simplified to match this setting, employing two fully connected hidden layers with 16 and 8 neurons using ReLU activation, followed by a single sigmoid output neuron trained with binary cross-entropy loss.

Figure 17 presents a stability comparison based on the coefficient of variation (CV) computed over multiple training runs for precision, recall, and F1-score. The baseline configuration exhibits the highest variability across all metrics, whereas LLM-based augmentation substantially reduces performance variability, particularly for recall and F1-score. Classical oversampling methods such as SMOTE and TVAE show intermediate stability, while ADASYN achieves the lowest CV values, indicating the most stable training behavior in the binary UNSW-NB15 Dataset setting.

Figures 18 and 19 show the mean precision and recall dynamics for the malicious class on the UNSW-NB15 dataset across different oversampling methods. All oversampling-based approaches converge faster than the baseline model, leading to improved recall during early training epochs. Precision remains stable across epochs for all methods, with only minor differences observed between oversampling strategies.

Figure 20 summarizes the average classification performance for the malicious class on the UNSW-NB15 dataset across different data augmentation strategies. In contrast to the LUFlow experiments, the binary formulation and simplified decision space of the UNSW-NB15 dataset reduce the relative advantage of complex generative augmentation. Classical interpolation-based methods, particularly SMOTE and ADASYN, achieve the highest recall values, indicating that simple synthetic interpolation is sufficient to enhance minority class separability in this setting.

The LLM-based augmentation exhibits the highest precision and achieves a competitive F1-score, reflecting a different trade-off between detection sensitivity and false positive control. While it does not outperform interpolation-based methods in terms of recall, its balanced performance suggests that LLM-based synthetic data generation remains effective but is not universally superior under simplified binary classification conditions.

Figure 21 presents the averaged confusion matrices for different data augmentation strategies on the UNSW-NB15 dataset. A notable observation is that the LLM-based augmentation achieves the highest number of correctly classified benign samples and the lowest false positive rate among all evaluated methods. This indicates that LLM-generated synthetic data encourages a more conservative decision boundary, improving benign traffic discrimination at the cost of reduced sensitivity to malicious samples (Figs. 22 and 23).

Figure 24 reports the results of the Wilcoxon signed-rank test comparing minority class recall across different data augmentation strategies against the baseline model on the UNSW-NB15 dataset. The results indicate that classical interpolation-based methods, SMOTE and ADASYN, achieve statistically significant improvements in recall (p < 0.01). In contrast, neither the LLM-based nor the TVAE-based augmentation yields a statistically significant recall improvement relative to the baseline. These findings confirm that, in the simplified binary classification setting of the UNSW-NB15 dataset, interpolation-based oversampling is more effective at improving minority class recall than generative augmentation approaches.

On the UNSW-NB15 dataset, formulated as a binary classification problem and evaluated using a simplified neural network architecture to mitigate overfitting, the relative behavior of the augmentation strategies differs from that observed in the multiclass LUFlow setting. Although all methods achieve similarly high ROC–AUC values (≈ 0.995–0.996), statistically significant improvements in minority class recall are observed primarily for interpolation-based oversampling methods. In particular, SMOTE and ADASYN achieve the highest recall values (0.871 and 0.879, respectively) and are confirmed as significant by the Wilcoxon signed-rank test (p < 0.01). TVAE yields a moderate numerical improvement in recall (0.837) but does not reach statistical significance (p = 0.38), indicating higher variability across training runs. In contrast, the LLM-based approach does not demonstrate a statistically significant recall improvement over the baseline (p = 0.75), despite achieving the highest average precision in the precision–recall analysis. Inspection of averaged confusion matrices further shows that SMOTE and ADASYN more effectively reduce false negatives for malicious traffic, while the LLM-based model primarily improves benign traffic discrimination. Overall, these results suggest that in a relatively simple binary classification setting, classical interpolation-based oversampling methods are more effective for recall optimization, whereas the benefits of generative approaches, including LLM-based and TVAE-based augmentation, are less pronounced.