Research Environment
We selected two hospitals in southern and northern China, which are very different in terms of climate and economic level, as study sites. One is a large teaching hospital in Shenzhen (22°38′N, 114°05′E), a large city in southern China. Shenzhen has the third largest GDP in China and a permanent population of 17.56 million (according to 2020 statistics). This study site (hereafter referred to as Hospital 1, H1) is a top-class comprehensive university affiliated hospital in the city's downtown area. The other is a provincial hospital (hereafter referred to as Hospital 2, H2) in Yinchuan city, northern China (38°47′N, 106°27′E). Compared with Shenzhen, Yinchuan city has a permanent population of only 290,000 (according to 2020 statistics) and a much lower GDP. Despite the vastly different natural and social environments, the two hospitals had similar numbers of outpatients and inpatients. We hope that both hospitals represent as many hospitals in China as possible to increase the representativeness and generalizability of the analysis results.
Data collection
We retrospectively collected monthly nosocomial infection number and incidence information, as well as data related to nosocomial infection surveillance reports for the same period, from Hospital 1 (January 2014 to April 2020) and Hospital 2 (January 2015 to April 2021), respectively. Specifically, 39 factors (see Appendix Table S1 in the Supplementary Material for details) were included, including hospital surgical volume (15 variables, x1-x15), nosocomial infection (8 variables, g1-g8), antimicrobial use (10 variables, y1-y10), and the number of patients with multidrug-resistant bacteria (c4). Considering the influence of climate, we also collected outdoor temperature data, including mean daily temperature (TAVE), daily maximum temperature (TMAX), and daily minimum temperature (TMIN). These five factor groups were determined as continuous predictors (independent variables), and the number of nosocomial infection patients (NNI) and nosocomial infection incidence rate (INI) were set as predictors (dependent variables, c1-c3).All variables were collated into a monthly database. Logarithmic transformation was performed using the natural logarithm (base e) due to the large order of magnitude difference between the factors.
Statistical analysis
Compare and evaluate multiple models.
Spearman correlation analysis was used to explore that these factors were not significantly correlated with INI and NNI (Appendix Fig. S1). Logistic regression model and the four most common machine learning techniques were further selected: logistic regression (LR), decision tree (Dtree), conditional inference tree (Ctree), random forest (RF), and support vector machine (SVM). [17, 20]We transformed the INI into a binary dependent variable (high and low risk levels) as the predicted outcome according to the median (non-normal distribution) or mean (normal distribution), and split all data into training and test sets (70%/30% split). Furthermore, the predictive accuracy of the five models was evaluated using internal cross-validation in two hospitals each. Other results showed that the test performance may vary depending on the data split. Therefore, it is important to employ multiple data splits when estimating the generalization performance. [16, 21]To assess the variability of estimated performance, we calculated the range of AUROC values and reported the mean performance and standard deviation for each model using 3- and 5-fold cross-validation.
The training data was used to train five different models and tune the hyperparameters of each model. The performance metrics (sensitivity, specificity, positive predictive value, negative predictive value, accuracy) were used to select the hyperparameter values leading to the best predictive performance.
Predictors selected by importance analysis.
Based on the above comparison, the RF model was selected to perform the best. We then assessed variable importance using importance analysis of the RF model (details of the comparative analysis are provided in the Results section). We assessed variable importance by calculating and ranking the increase in mean squared error (%IncMSE) and increase in node purity (IncNodePurity) associated with the loss functions, and selected the loss function through optimal segmentation. We assessed multivariate importance by removing predictor variables from each single tree in the forest using the RF model and measuring the change in accuracy to evaluate the effect of the predictor variables. More useful variables achieve a higher %IncMSE (Appendix Table S2) [22]To increase the stability and accuracy of the RF model and prevent overfitting, we adopted several strategies, including increasing the number of trees (n_estimators) to 500 to improve prediction averaging and reduce variance, limiting the number of variables considered at each split (max_features) to 3 to promote diversity and reduce correlation among trees, and imposing a maximum depth restriction (max_depth) to prevent excessive complexity and overfitting to the training data.
Trend forecasting and risk threshold forecasting.
Autoregressive Integrated Moving Average (ARIMA) is a statistical analysis model that uses time series data to better understand a dataset or predict future trends. The INI of the two hospitals was used as the dependent variable and factors with high %IncMSE according to the importance ranking of RF analysis were used as independent variables. ARIMA uses a lagged moving average to smooth the time series data. The autoregressive notation (p), difference notation (d), and moving average notation (q) form the ARIMA multiplicative process as (p, d, q). [23]ARIMA models are characterized by stationary R-squared (R2) values as well as small Bayesian information criterion (BIC) and root mean square error (RMSE).
Classification and regression trees can create binary trees, where each node has exactly two outgoing edges and finds the best categorical or numerical feature to split on using an appropriate impurity criterion. The independent variable can be categorical (classification trees) or continuous (regression trees). [24]In this study, we performed regression tree analysis to determine hierarchical thresholds between NNI and important variables. This model assessed the quantitative relationship between multiple variables, ranked them from most to least in terms of their degree of influence, and calculated risk thresholds and estimated case numbers in different situations.
All of the above analysis methods were performed separately on the H1 and H2 databases to ensure diverse representation of the predictability of hospital-acquired infections, which is important for the generalization of the ML model. We conducted the same modeling analysis using data from two different hospitals, aiming to evaluate whether the best ML model identified in this study would perform well in other hospitals and whether the factors selected by the model were consistent across different hospitals. Overall, our aim was to evaluate the generalization of the constructed model and the universality of the risk factors screened in this study.
R software version 4.0.3 (The R Project for Statistical Computing, Vienna, Austria) was used to build and compare machine learning based models. The following R packages were used for these approaches: GGally package for correlation analysis, forward package for ARIMA analysis, glm package for logistic regression, rpart, rpart.plot package for decision tree models, party package for conditional inference trees, randomForest package for random forests, e1071 package for support vector machines, and PROC and ROCR packages for receiver operating characteristic (ROC) curve analysis. SPSS version 25.0 for Windows (SPSS Inc., Chicago, IL, USA) was used for ARIMA analysis.
