The evaluation of the new model has been concluded by completing PCA, KNN-SVM-NB, and RF integrated classification. By varying the relevant parameters of PCA and eliminating the non-compliant factors, a data model that meets our expectations is constructed. In order to analyze this data for unsupervised learning, we first used the K-means (K-means) clustering technique. which divides the dataset unsupervised by setting K = 4. After the steps of initialising the clustering centre, iteratively assigning data points to the category where the nearest clustering centre is located, and updating the clustering centre to the category mean until the algorithm converges, the dataset is successfully divided into four categories with high cohesion and low coupling characteristics, thus revealing the intrinsic structure and distribution information of the data. Finally, the performance evaluation of the new model is completed using KNN, SVM, NB and RF integration respectively. In this paper, we use IBM SPSS Statistics 27.0 software to implement PCA data dimensionality reduction, and MATLAB R2023a software to complete the model construction and evaluation of KNN, SVM, NB, and RF integrated classification. Meanwhile, in this process, 60% of the case data was selected for model training, 30% for testing, and the remaining 10% for validation.
Data analysis
The PCA algorithm prefers to maximise the variance of the data in a low-dimensional space and capture the global data distribution and the covariance structure among variables through linear projection. The goal of PCA is to minimise the dimensionality of the data while maximising the retention of the total variation information of the dataset, which it achieves by calculating the covariance matrix and searching for its eigenvalues and corresponding eigenvectors. For the coal mine roof accident dataset, in order to determine the applicability of PCA and the appropriate dimensionality reduction, it is first necessary to analyse the intrinsic structure of the dataset and the degree of redundancy. This can be assessed by performing PCA on the raw high-dimensional data to find the number of principal components that explain most of the variance in the data. This step was achieved by using IBM SPSS Statistics 27.0 software. After analysing the structure of the data, the number of fixed factors we extracted was 4. Therefore, we used PCA to downscale the coal mine roof accidents dataset to a dimension of 4.
Factor analysis
This study was done using IBM SPSS Statistics 27.0 software, and firstly, we conducted exploratory factor analysis on the roof accident factor scale, and principal component analysis was used for factor analysis. We performed KMO and Bartlett’s test on the coal mine roof accident data, the purpose of these two tests is to analyse whether the data are suitable for factor analysis and principal component analysis. The KMO test value needs to be greater than 0.6 in order to do factor analysis, and the Bartlett’s test value, if it is less than the specified level of significance, is considered to be suitable for principal component analysis of the sample data, and the other way round, it is not suitable.
The Bartlett’s test of sphericity results show that the P value is less than 0.001 and less than 0.05, which indicates that the selected indicators are suitable for factor analysis. As can be seen from Table 3, the initial KMO of the original data has a value of 0.612, which is greater than the critical value of 0.6 and suitable for factor analysis.
In the process of factor analysis, we set the fixed number of factors to be extracted as 4 and rotated the factor loading matrix by the technique of greatest variance. To ensure the reliability and validity of principal component factor analysis, SPSS statistical software was used to implement rigorous data processing steps. Specifically, in the exploratory factor analysis stage, we assessed the strength of theoretical associations between each observed variable and the latent factors based on the key indicator of factor loadings. In this process, we set a factor loading threshold of 0.4, i.e., only observed variables with factor loadings not less than 0.4 were retained as significantly correlated with the extracted factors, and the corresponding observed variables were considered to be substantively theoretically related to the target factors, and thus retained in the final factor model. On the contrary, observed variables with factor loadings below this threshold were systematically excluded from the factor structure based on their weak correlation with the underlying factors. After analysis, it was found that the factor loadings of “incomplete roof search”, “local subsidence”, “roof pumice”, “overloaded production”, etc. were less than the critical value. ” and other factor loadings were less than 0.4, we deleted the above factors, and then carried out exploratory factor analysis afterwards.
In the above process, we get an initial factor loading matrix by calculating the covariance matrix and extracting the common factors using principal component analysis (PCA). However, in this factor loading matrix, some variables have large loadings on multiple factors, and the boundaries between the factors are not clear, and the underlying structure represented by the factors cannot be interpreted in a simple way. In order to facilitate a more intuitive analysis of the data structure, we use the maximum variance method in the rotation process, set the maximum quantity of convergence iterations to 25, and try to maximise the absolute value of the variable loadings on each factor, while minimising the variable loadings on multiple factors, and display the rotated solution as well as the loadings plot. Thus, the rotated and optimised factor loading matrix will show the new adjusted factor structure, where the factor loadings give a clearer indication of the strength and direction of the relationship between each observed variable and a particular factor, facilitating a more intuitive interpretation.
In order to obtain an explanatory factor model that meets our requirements, we need to go through a continuous cycle of exploration and validation by eliminating factors that fail to provide substantial incremental information or do not conform to the research hypotheses. During the iterative process, we then continuously adjust the factor structure based on statistical indicators and theoretical logic until we construct a stable factor model that can effectively capture the intrinsic structure of the data and also closely fit the research objectives. After repeated experimental exploration, we constructed a final factor model that meets our requirements.
K-means cluster analysis
Based on the above factor analysis process, Principal Component Analysis (PCA), Providing a practical method for reducing dimensionality, was used to form principal components that maximise the preservation of the variance of the original data by linearly transforming the high-dimensional observed variables in order to extract the main directions of variation in the covariance structure. The interpretability of the component loadings matrix and the simplicity of the factor structure were further enhanced by implementing rotational optimisation after PCA was completed. However, relying solely on the PCA-derived components does not directly reveal the underlying categorical structure in the data. For this reason, it is crucial to perform cluster analysis on this basis.
The main motivation for the implementation of K-Means cluster analysis, a classical unsupervised machine learning method, is to explore and reveal underlying group structures and patterns from large amounts of unlabelled data. The core of this process is to classify the samples into multiple subsets or clusters with similar characteristics based on the distance or similarity measure between the samples in the low-dimensional principal component space generated by PCA. This process will help us to discover the group structure that naturally exists in the dataset, and then subdivide the samples based on the deeper features represented by the factors, which in turn gives us the intrinsic structure of the data. In this process, we have analysed this clustering through Matlab2023a software and the results are shown in Fig. 8 below.
Results of K-Means clustering analysis.
In Fig. 8, (A) shows the contour coefficient plot, (B) the feature dimension plot, and (C) the mean variance range plot for each category. In the contour coefficient plot, we can see that when the set number of clusters is 4, we observe a peak in the contour coefficient (SC), which means that the clustering effect of the data set is optimal under this number of clusters. The size of the value of the contour coefficient, as a measure of the goodness of the clustering results, reflects the degree to which the sample points have been appropriately categorised as well as the degree of separation between the different clusters. When the SC reaches the maximum value, it indicates that the data points not only have high similarity within the clusters to which they belong, but also maintain good differentiation from the neighbouring clusters. Therefore, the data show the best clustering quality and structural clarity when the original dataset is divided into four clusters. In this study, after deep mining of coal mine roof accident data using cluster analysis techniques, the results showed that the data naturally differentiated into four clusters with unique characteristics.
Through detailed analysis and deconstruction of these four clusters, the results of the four clusters show significant correspondence with the four core elements of coal mine safety production, namely, “man, machine, environment, and management”. Firstly, the first clustering unit is closely related to the “human” factor, and the occurrence of this type of accidents often stems from the fact that the operators often violate the rules and operate in a non-compliant manner, which largely determines the probability of the occurrence of roofing accidents. The second cluster group mainly reveals the role of “machine”. In this category, the occurrence of roof accidents is closely related to the performance of equipment, whether it is qualified or not, as well as the design defects or aging severity of equipment. Therefore, strengthening equipment maintenance, upgrading outdated equipment, and strictly monitoring the use of equipment become the key measures to reduce such accidents. The third cluster focuses on the influence of the “ring”, which is subject to the environmental constraints such as changes in geological conditions and the state of surrounding rocks. This implies that in coal mining activities, scientific and reasonable assessment and response to environmental factors play a crucial role in maintaining roof stability and reducing the risk of accidents. The fourth cluster is closely related to the “management” dimension. In this category, the omissions and deficiencies in the safety management of coal mining enterprises are reflected, including the weak enforcement of safety rules and regulations, and the inadequacy of hidden danger investigation and management, on-site management, and supervision. These management-level problems reveal the urgency of improving the standardisation and normalisation of coal mine safety management to reduce roof accidents.
Modelling results
After K-Means clustering analysis results in clustered labelled data, we merge the obtained clustered labels back into the original dataset to form a new data model containing the original features and clustering results. To assess how well the model performs, in this section, we use K-Nearest Neighbour (KNN), Support Vector Machine (SVM) and Decision Tree (DT) algorithms to evaluate the classification performance of the new model. In this process, we compute the accuracy, recall, and F1 score of the risk assessment model to measure the model’s performance. By calculating the accuracy, recall, and F1 score, the model’s performance in forecasting the risk category of roof accidents is intended to be comprehensively and accurately evaluated quantitatively. Accuracy assesses the correctness of the model’s overall prediction, recall reflects the model’s ability to identify samples of a specific risk category, and F1 score balances precision and recall as a composite metric. By calculating and analysing these key performance indicators, we can effectively evaluate the performance of the model in practical applications, further optimise the model parameters, validate the reasonableness of the clustering results, and provide powerful data support and scientific basis for the decision-making of safety productions in coal mines.
Model performance
By analysing the findings using Matlab2023a, we have shown the results as shown in Fig. 9 and Table 4:
Confusion matrices of the three algorithms and plots of true, fitted values.
KNN algorithm: the KNN model achieved 0.81 accuracy and 0.77 recall under the selected K values. Through the sensitivity analysis of different K values, we find the optimal K value under which the model maximises the detection of high-risk events while maintaining a low false alarm rate. SVM algorithm: by optimising the C-parameters and kernel function types, the SVM model achieves an accuracy of 0.89 and a recall of 0.86. Its boundary decision-making ability is strong, and it shows advantages in modelling complex nonlinear relationships. DT algorithm: the decision tree model adjusted by pruning and other optimization means has an accuracy rate of 0.74 and a recall rate of 0.67. The DT model is intuitive and easy to interpret, but it may need more refinement nodes when dealing with continuity and nonlinear relationships.
In the three model results in the table above, we analysed this to obtain: the KNN model has a relatively high accuracy, indicating that it correctly predicts a good proportion of the classifications. However, the recall is low, indicating that the model does not predict a high proportion of positive classes in samples that are actually positive classes. The F1 score is in the middle of the three models, showing that the KNN has an average balance between bias and precision. Decision trees, on the other hand, have relatively low metrics, especially recall and F1 scores. This suggests that although the accuracy of the decision tree is not considered low, it is likely to overfit or fail to capture positive class samples effectively under certain conditions. This makes it potentially less reliable than KNN and SVM in practical applications. Finally, the SVM model performs the best, with high accuracy and recall, indicating that it accurately corresponds to most of the samples and performs well in the recall of positive class samples. The high F1 score (0.87637) shows that SVM has a good balance between accuracy and recall and is suitable for tasks such as target detection that require high accuracy. In this regard, It is concluded that in terms of performance ranking: SVM > KNN > DT, i.e., SVM is preferred as the model of choice as it performs best in all the metrics. At the same time, KNN can be considered as an alternative in the case of limited computational resources or the need for fast response. Decision tree has the worst performance among these three models, and it may be necessary to adjust the model parameters or try other integration methods (e.g., random forest) to further improve the performance.
Table 4 shows the specific accuracy and recall rates and F1 score values of KNN, SVM and DT algorithms in this coal mine roof accident24 risk assessment task. From the comparison results, SVM performs the best in all aspects, but still fails to achieve the expected results. Given the varying performance of single algorithms in coal mine roof accident risk assessment, for the purpose of further bolster the precision and stability of model prediction, this study introduces an integrated learning approach. Specifically, we adopt integration algorithms such as Bagging, Boosting and Stacking, and incorporate KNN, SVM and DT as base learners into the integration framework, with the expectation of constructing stronger prediction models by combining the advantages of multiple weak learners.
Comparing the integrated model with the above three models in this analysis, we conclude that: the KNN performs moderately well in terms of accuracy and has relatively low recall, which means that although the model is more accurate in overall prediction, it is not particularly effective in identifying samples of the positive class; the SVM performs well in all three metrics, especially in terms of accuracy and recall, which is relatively high, suggesting that the model is relatively well balanced in terms of classifying the positive and negative classes; the decision tree performs poorly. The overall performance is relatively balanced. The F1 score also shows good overall performance; the decision tree performs poorly. The low accuracy and recall indicate that the model loses more information in prediction and is more susceptible to noise, resulting in overfitting; the Random Forest model has the best performance. At 94.5 per cent in accuracy, it shows the model’s power in correctly predicting samples. The recall of 87.2% shows that the model is also quite effective in identifying positive class samples, although it is slightly lower than SVM in this respect. The F1 score of 89.05 shows that the model strikes a good balance between accuracy and recall.
Meanwhile, the analysis of the Random Forest algorithm indicates that its model superiority is reflected in the following aspects: 1. High accuracy: the accuracy of the Random Forest model reaches 94.5%, which is much higher than that of other models, indicating that it can effectively reduce the phenomenon of misclassification. 2. Advantageous recall: although the recall is not as good as that of the SVM, the 87.2% score is already quite impressive for many practical scenarios, indicating that the model is effective in reducing misclassification. This indicates that the model can effectively capture most of the positive class samples.3. Overfitting Resistance: Random Forest effectively reduces the risk of overfitting by introducing randomness of samples and randomness of features. This allows the model to maintain good generalisation ability in the face of unknown data.4. High F1 Score: Random Forest’s F1 score (89.05) further validates its balanced performance. This score indicates that the model is not only able to obtain high accuracy, but also maintains good recall ability, which makes it perform well in the face of unbalanced datasets.
Taken together, Random Forest not only performs well in all the metrics, but especially the accuracy and F1 scores, highlighting its effectiveness and stability when dealing with complex datasets. Therefore, based on the above analyses, Random Forest is the optimal choice among these four models for practical application scenarios that require high accuracy and better recall.
On the basis of the above, we used the random forest algorithm to process it, and after the integrated algorithm, we evaluated the newly constructed model for the same accuracy, recall, and F1 score. The results show that the integrated model improves to 0.94 in accuracy and 0.87 in recall, as well as the F1 score reaches an amazing 0.89, which is a significant improvement compared to the KNN, SVM and DT algorithms alone. Table 5 demonstrates the comparison between the integrated model and a single base learner in terms of accuracy and recall, which shows that the integrated algorithm successfully improves the overall risk assessment performance by combining the advantages of multiple models.
Sensitivity analyses
In this study, we used a sensitivity analysis method based on sensitivity to population size, evolutionary generations, etc., to assess the performance of the model and its sensitivity to parameter changes by varying the hyperparameters. In the course of the study, it was observed that changes in population size, evolutionary generations has a significant effect on the model performance25. At the same time, we found that within the range of 50–250 population size, 50–100 evolutionary generations, and 50–100 Bayesian iterations, the ten-fold validation accuracy of KNN, SVM, and DT algorithms decreases while the ten-fold validation accuracy of RF algorithms increases, which indicates that the Random Forest algorithm is more sensitive to this. By adjusting the parameters, the performance of the model can be effectively improved, indicating that the model is able to better capture the features and patterns of the samples, which improves the accuracy and comprehensiveness of the prediction, as shown in Tables 6, 7, and Figs. 10, 11, 12, 13 below.
Comparison of two tuning combinations.
Population size and model evaluation index.
Evolutionary generations and model evaluation indicators.
Bayesian iteration and model evaluation index.
At the same time, we also draw relevant conclusions: through experimental analysis, the optimal parameter combination is population size 50, evolutionary generation 50 and Bayesian optimization iteration 50 times, at which time the model peaks in accuracy (0.967), recall (0.945) and F1 score (0.943) with the lowest fluctuation (standard deviation ≤ 0.01), indicating that the model is highly performant and stable; increasing population size or number of iterations can slightly improve F1 score (e.g. F1 = 0.896 for size 200), but it will significantly increase computational cost and reduce accuracy. Although increasing the population or the number of iterations can slightly improve the F1 score (e.g., F1 = 0.896 for size 200), it will significantly increase the computational cost and reduce the accuracy, so the balance point is preferred to ensure the unity of efficiency and effectiveness, and a small-scale validation can be carried out for the intermediate parameters (e.g., size 150, 30 iterations) for further optimization.
