The literature review of similar research has shown that various machine learning models can be used for data classification tasks. Based on the literature analysis performed, in our research, we choose to investigate the influence of text augmentation using the traditional machine learning models. The six machine learning algorithms have been used in the experiments: multi-layer perceptron (MLP), random forest (RF), gradient-boosted trees (GBT), k-nearest neighbors (kNN), decision trees (DT), and naive Bayes (NB). The main reason for choosing traditional machine learning models instead of deep learning models is that the focus of this research is to find does the text data augmentation increases the classification results. The deep learning model training is more complicated in terms of time and require significantly bigger dataset to achieve the best accuracy. The experiments have been performed using the two most popular techniques that are used to transform text into vectors: bag of words (BoW) and sBERT embeddings. Also, the hyperparameter optimization was performed during the model training process. The parameters and their range of change are presented in Table 3.
All machine learning algorithms have been trained using the subsets presented in Table 2. To train and test the models, the stratified k-fold cross-validation technique was used, where =5. To ensure the same conditions for the evaluation of all machine learning algorithms, the same k-folds were used. To evaluate the performance of each model, four metrics were calculated: accuracy, recall, precision, and the F1 score. In the research many different combinations of subsets have been used, and hyperparameters optimization has been performed. Thus, a total of 15,296 models have been trained and evaluated. Only the results of the highest accuracy for each machine learning model have been presented as a summary.
Primary research—dimension reduction for bag of words models
Text data analysis is resource-intensive because converting text into numeric expressions the high-dimensionality vectors created. Based on the results of Stefanovic and Kurasvoa research31, the text data has been pre-processed for primary research using such filters: removing numbers, removing punctuations, tokens converted to the lower case, removing stop words, the minimum length of the tokens is equal 3. After converting the original datasets (1_2_class, 1_3_class) presented in Table 2 using the BoW technique, the dimension of vectors was equal to approximately 10,000. For many algorithms, such dimension is too high, and it costs a lot of time to train, especially when hyperparameter optimization is used. Therefore, in the case of BoW, first of all, primary research has been conducted to reduce the dimensionality of vectors. It is usually performed for high-dimensional data analysis. To reduce the dimensionality of vectors, the Latent Semantics Analysis (LSA) method has been used32. LSA is one of the models which is often used in natural language processing tasks. The main aim of the LSA model is to establish a relationship between texts and the words they contain. The latent semantic analysis method assumes that words in texts that are close in meaning will occur in similar pieces of text, a concept known as the distributional hypothesis. In primary research, three different types of machine learning algorithms have been chosen: multilayer perceptron, random forest, and k-nearest neighbors. First, the original datasets (1_2_class and 1_3_class) were used to train the three different types of machine learning algorithms without dimensionality reduction but using hyperparameter optimization. Later, the dimension of the original dataset was reduced to several dimensions, starting from 10 to 50 by step 10. To find the influence of the chosen dimension, the paired t-test has been used, where the results of the original dataset were paired with the results of the reduced dimension. The descriptive and inferential statistics of primary experiment results have been presented in Table 4.
The results of the mean and standard deviation show that the accuracy of both MLP and kNN models improves after dimensionality reduction, with kNN showing particularly significant gains. In many cases, the p-value is very small, indicating that dimensionality reduction significantly impacts the results. However, in two cases where the random forest was trained on the original 2 class dataset, the p-value exceeded 0.05, suggesting that the results before and after dimensionality reduction were similar. The primary research indicates that the optimal dimensionality for all algorithms is 40. At this dimensionality, the random forest’s performance remains consistent with the original dataset, while the performance of MLP and kNN improves. Therefore, for the main research, the dimensionality of the dataset has been reduced to 40 for other machine learning algorithms.
Main research—Gen-AI augmented dataset impact on classification accuracy
The main experiments have been performed using all six machine learning models, where the dataset dimension has been reduced by LSA to 40. First, the original dataset with the bag of words method has been used to train the models. The results of the 2 and 3 class original datasets without any augmentation are presented in Table 5. In the case of 2 classes, the accuracy variates from 82.28% (kNN) to 87.60% (RF). A slightly smaller accuracy is obtained when the 3 classes dataset were used, where the smallest accuracy is equal to 76.12% (kNN), and the highest using the RF – is equal to 82.83%.
Later, all six machine learning algorithms were trained using other subsets that show the influence of the Gen-AI tools on text data augmentation. In Fig. 3, the results show how the accuracy of the models increased or decreased compared to the original 2 classes dataset (1_2_class, Table 5). As we can see, four of six algorithms (MLP, RF, GBT, kNN) increased the accuracy after the data augmentation using any Gen-AI tool. In the case of DT and NB, the results were decreased after the data augmentation. The highest accuracy was obtained when the kNN was used. Overall results show that the highest accuracy was obtained when the 6_2_class subset was used. This subset was created when the text data was augmented using chatGPT and Copilot. The smallest increase can be observed when the Gemini tool used on text data augmentation.
Comparative results of the 2 classes’ original dataset and rest of subsets, using bag of words.
Table 6 presents the highest accuracy achieved by each machine learning model across all subsets, along with the corresponding hyperparameters. For MLP, higher iteration number and neuron numbers generally improve accuracy, while the number of hidden layers does not. The highest accuracy of 89.19% is obtained using the 6_2_class dataset (chatGPT and Copilot tools). Slightly smaller accuracy (89%) was when just the Copilot tool has been used for text data augmentation (4_2_class). Random forest results indicate that for smaller datasets, the information gain split criterion yields the highest accuracy, with the number of models ranging from 100 to 850. For larger datasets, the Gini index split criterion performs better. Random forest also achieved the highest accuracy when the chatGPT and Copilot are used together for text data augmentation (91.87%). GBT-based models consistently achieve the highest accuracy (91.45%) with a learning rate of 0.01 and 1000 models, with tree depths varying from 2 to 10. For the kNN algorithm, using weighted neighbors and setting the number of neighbors between 2 and 8 is optimal. Overall, the highest accuracy was 94.16%. Decision tree parameters vary significantly across subsets, making it difficult to identify clear trends. Additionally, the accuracy using the original dataset are slightly higher, so text data augmentation do not improve the results. Related research shows that naive Bayes typically does not require parameter optimization.
The results of the 3 classes’ original dataset compared to the results of augmented datasets are presented in Fig. 4. The findings are similar to those for the 2-class datasets: MLP, RF, GBT, and kNN show increased accuracy, while DT and NB show decreased accuracy. Notably, the percentage of increase or decrease is higher than the results in Fig. 4, especially for the kNN algorithm. The influence of Gen-AI tools is evident, with the highest accuracy achieved when chatGPT and Copilot are used together for text data augmentation (6_3_class).
Comparative results of the 3 classes’ original dataset and rest of subsets, using bag of words.
In the case of subsets with 3 classes, the highest accuracy and the hyperparameters used are presented in Table 7. Almost in all cases, the highest accuracy is obtained using the kNN algorithm, where the number of neighbors is equal to 2, except two times when it was equal to 3 and 5. The GBT algorithm in most cases has the highest accuracy achieved when the number of models is equal to 1000 with a learning rate of 0.01. MLP-based models had the highest accuracy achieved when the 400–500 iterations were used. The results of the rest of the models (RF, DT, NB) do not show any trends, which hyperparameters is best to use. In many cases, the highest accuracy was achieved when the chatGPT and Copilot tools were used (MLP – 86.59%, GBT – 90.07%, kNN – 92.81%). The RF highest accuracy (89.6%) was in two cases: the chatGPT tool was combined with the Copilot tool, and the Gemini tool was combined with the Copilot tool. The highest accuracy of DT-based machine learning model is equal to 79.42% (only the Copilot tool has been used on text data augmentation process). The text data augmentation decreases the results of NB in all cases.
Table 8 presents the highest accuracy results for each machine learning algorithm using the original dataset using the sBERT. After the text transformation to numerical values using sBERT the dimensionality of the dataset is equal to 384. A comparison between sBERT and the bag of words method (Table 5) shows that sBERT yields lower accuracy, regardless of whether the original dataset contains 2 or 3 classes. For example, the accuracy of the DT algorithm decreased by more than 14%, and at the same time, the NB – almost 10% (in the case of 2 classes). When the original dataset is of 3 classes, the accuracy using ML, RF, GBT, and kNN is reduced by approximately 7%, but the DT and NB usage decreased accuracy by almost 19%.
The most significant increase in model accuracy was observed when using ChatGPT and Copilot tools for text data augmentation (see Fig. 5). For kNN, accuracy improved by over 15%, except for the 3_2_class subset, which saw an 8.18% increase. Unlike the bag-of-words method, the accuracy of DT models increased by 4–12%. Additionally, NB-based machine learning models also achieved higher accuracy in some cases after augmentation.
Comparative results of the 2 classes’ original dataset and rest of subsets, using sBERT.
As we can see in Table 9, the highest accuracy of MLP-based machine learning models was 91.83%. In this case, the chatGPT and Copilot tools were used for text data augmentation. The best hyperparameters are different in many cases, but the higher iteration and neuron number are preferable. The highest accuracy of RF (95.31%) and GBT (95.1%) are similar. In both cases, the subset 6_3_class is used. In the case of GBT, the learning rate should be equal to 0.01, and the number of models should be 1000. The tree depth varies from 3 to 8 depending on the used subset to train the model. The hyperparameters of RF vary a lot, so it is hard to see the trends which parameters should be used. Additionally, after the text data augmentation the RF and GBT accuracy almost in all cases are higher than 90%, except when the text data has been augmented using the Gemini tool (RF − 88.52%, GBT − 87.9%). Compared to the all-trained models, the best model was kNN which allowed us to achieve 97.52% accuracy with such hyperparameters in use: weighted neighbors: ON; neighbors: 2. In the case of DT models, also the highest accuracy (82.91%) has been obtained when the chatGPT and Copilot tools are used together. The analysis of NB-based machine learning models has shown that the data should be augmented using only the Copilot tool, it a slightly higher accuracy (75.87%) compared to the other models’ results.
The results of the last experiment case are presented in Fig. 6. The influence of text data augmentation on model results is very high. For example, in many cases, the accuracy obtained by the kNN algorithm increases by more than 20%. When the 6_3_class subset is used, the accuracy increases even by 27.23%. The highest decrease in accuracy occurred when the NB algorithm was used – 7.76%. Also, it can be observed that the usage of Gemini tools separately for text data augmentation does not such a higher influence compared to the other tools or their combination. The analysis of hyperparameters has shown (Table 10) that the parameters are similar as it is in the case when data of 2 classes are analyzed. The highest accuracy was obtained when the chatGPT and Copilot tools were used for text data augmentation. The highest accuracy of kNN was equal to 96.75%, GBT – 94.79%, and RF – 93.94%. In the case of MLP, the highest accuracy was achieved 87.53%. The lowest accuracy was obtained when the NB was used.
Comparative results of the 3 classes’ original dataset and rest of subsets, using sBERT.
