Initial 41 features
From Table 1, we observe that the model created using the initial 41 features performs better than APACHE-IV. Large ML studies often report typical AUROCs in range 0.8–0.924, and our results are within the expected range.
For the initial 41 features, disease-specific, binary variables contribute relatively little to the average model prediction. As most of these conditions are rare, they hold limited statistical power. While the features could be relevant for some individual patient predictions, they are considered less relevant in the population as a whole.
Comparing our selected 20 features with previous ML and prognostic studies shows overlap with physiological and laboratory predictors. In particular, routinely measured vitals and laboratory values are repeatedly reported as relevant predictors of in-hospital and ICU mortality in both classical severity scores (e.g. APACHE and SAPS) and more recent ML models25,26,27,28,29. Ventilation status and variables are also common and strong markers of respiratory failure and mortality risks30. The Glascow Coma Scale and its components (verbal, motor, eyes) are established predictors of mortality across trauma and medical ICU populations and commonly appear among the top features in predictive models31. Age is generally considered a strong predictor across multiple cohorts27.
By contrast, variables that reflect administrative context (e.g. electivesurgery, unitadmitsource, and admission diagnosis (adx)) are included variably across studies but have known prognostic relevance. Elective surgical admissions generally carry lower ICU mortality than emergency/non-elective admissions, and pre-ICU admission site (ED, ward, operating room, transfer), and primary admission diagnosis add important information about case mix and baseline risk32,33,34. These features may therefore improve model performance even if they are used less consistently than physiological and laboratory measurements.
Overall, the overlap between our top-20 features (Table 2) and prior work indicates these predictors reflect established, clinically plausible risk factors rather than spurious model artefacts. Variation in feature ranking across studies are expected as differences in cohorts, outcome definitions, variable selection and coding, and choice of algorithm and metrics all influence importance estimates.
We also emphasise three different caveats that affect the interpretation of feature importances. First, the Table 2 reports average importances across folds and patients, and not the relevant features for individual patient predictions. A feature with low average can still be highly relevant for a particular patient prediction. Second, feature importances describes the model not the real world. A feature highly relevant for a model prediction is not equivalent to the feature being highly relevant for the actual outcome, nor does it mean that the feature would hold the same relevance in a different model. Third, missing values can inherently carry information, their prevalence may influence the results of the analysis. The underlying causes of the missing values presented in Suppl. Table A1 in Appendix A are unknown but are likely caused by multiple factors. Future research should explore this further to gain a deeper understanding of its impact.
Feature sets with ten features
The performance of models using ten features is lower than the APACHE IV performance, as seen in Table 3. The performance is also lower than models using the initial 41 features in Table 1. These results are as expected as the 41 features contain more information than ten features, and the number of features is not so high that the models experience the challenges associated with high dimensionality.
Comparing the results to other studies is not straightforward as similar studies use different datasets, populations, types of features, preprocessing techniques, algorithms, performance metrics, and criteria. Compared to other mortality prediction models the mean AUROC when using ten features is in the lower range24. Still, AUROC is not sufficient as the sole metric for evaluating or deciding between models. The variation in the proportion of deceased patients in different populations makes it more challenging to compare AUPRC with other studies.
Figure 2 shows a positive relationship between AUROC and AUPRC. The majority of the models achieve an AUROC between 0.80 and 0.83, and an AURPC between 0.36 and 0.40. These results demonstrate that multiple combinations of features yield similar performance. As the features are not independent, and the 10 000 feature set pairs do not cover all possible combinations, we cannot exclude the possibility that some remaining combinations would exhibit different behaviour. Nevertheless, the results are consistent with the Rashomon effect stating that for a given dataset, there may exist multiple solutions with comparable performance35.
Figure 3 indicates no clear relationships between complementary feature set pairs regarding the AUROC and AUPRC. Most complementary feature set pairs have both values in a specific range, with some outliers. The results show that having a feature set with a high AUROC or AUPRC does not implicate a low performance of the complementary feature set. For some outliers, both feature sets have similar performance. The lack of a clear trend indicates that different feature sets can have comparable performance.
Single features analysis
As seen in Fig. 4a, b , age occurs in the majority of feature sets with the highest AUROC. This frequent occurrence could indicate that including age in the feature set is beneficial. However, Fig. 4c shows that age occurs in only half of the 10 000 feature sets with the highest AUPRC. The temperature and urine occurrences are also not consistent between AUROC and AUPRC. The feature sets common for the 10 000 models with the highest AUROC and 10,000 models with the highest AUPRC are found in the top right corner in Fig. 2. The histograms in Fig. 5 indicate that having a specific feature in the feature set does not guarantee better overall performance.
These results indicate that the feature selection process should consider the distinct trade-offs associated with different performance metrics. While AUROC reflects a model’s ability to discriminate between classes, emphasising sensitivity and specificity, the AUPRC focus on the precision-recall trade-off. Consequently, a feature that significantly contributes to the overall discrimination might not contribute equally well to high precision and recall, or other types of performance metrics.
Varying feature set size
As expected, the AUROC tends to increase as the number of features grows, particularly given the initially small feature set size and given that all added features are considered relevant to the outcome. While the general trend within each feature subset shows that AUROC improves with more features, some smaller feature sets containing only three or more features outperform certain ten-feature models. For instance, the feature set [motor, heartrate, creatinine] with an AUROC of 0.763 surpasses Subset B in Table 5 with an AUROC of 0.759. Increasing the number of features above ten further increases the AUROC, though investigating the optimal number of features falls outside the scope of this paper.
Despite being based on the ten features with the lowest feature importance, Subset D performs more similar to the best performing subset (Subset A) and the subset based on the features with the highest feature importance (Subset C) than the subset with the lowest AUROC (Subset B). While some features are deemed less important when considering the full feature set of 41 features, they may provide complementary information when combined differently. Additionally, individual feature importance may be underestimated when interactions between features are not fully considered. The results suggest that even features with lower average importance may hold predictive value in certain contexts, supporting the broader finding that multiple feature combinations can yield similarly strong results.
Feature importance
Figure 7a, b should be analysed with the help of Fig. 4b . A small variance in importance could indicate that a feature is consistently one of the most or least important features across different models and may imply that the feature has a stable influence on the model’s performance, regardless of the specific subset of features being used. However, low feature occurrence in the selected feature sets could also lead to small variance in importance rankings. Features that rarely appear in the feature sets yielding the best or worst overall results, such as eyes and vent, may show little variation in importance simply because the low occurrence limits proper assessment of their impact.
While age is considered the most important feature when using all 41 features (Table 1), age is not always considered the most important feature as seen in Fig. 7b . Here, age contributes more to the predictions on average and is considered between the most important and fifth most important feature when looking at feature sets with 10 features. Age is not considered one of the least important features, regardless of the feature combination. The importance of features like eyes and creatinine depends more on the feature combination than age as the feature importance rank has a larger variance. In these cases, the infrequent inclusion of a feature might prevent it from being recognised as important, even though it could potentially have value in certain contexts or combinations. Therefore, the interplay between feature importance and occurrence needs to be carefully interpreted, as low occurrence does not necessarily indicate irrelevance, but rather may reflect the feature’s limited participation in subsets associated with higher performance.
High feature correlation can lead to uncertainty in determining the feature importance as it becomes more challenging to distinguish which of the features are contributing to the prediction. Highly correlated features can inflate the importance of one of the features while diminishing the importance of the other. This potential bias should be further investigated to ensure a more reliable interpretation of feature importance and its impact on model performance.
From a clinical perspective, our findings underline that predictive models for ICU mortality must be evaluated not only by their performance metrics but also by their clinical interpretability. In daily ICU practice, decisions are rarely based on a single variable but rather on the integration of multiple, sometimes redundant cues. The observation that different feature sets can achieve comparable discrimination reflects this reality and highlights the need for models that support—rather than replace—clinicians’ judgement by providing transparent, context-relevant predictions.
Limitations and future work
Both training and testing were conducted on a single dataset, so the generalisability of the results to other hospitals, measurement practices, and time periods remains untested. Moreover, both patient- and feature-selection may have introduced bias in the dataset. By excluding patients with short or multiple stays and discarding records with missing values and limits the results applicability to broader populations. Future work should focus on external validation and expanding to other datasets such as MIMIC III and IV. Robustness and consistency should be assessed across algorithms, preprocessing techniques, and evaluation metrics.
We also recommend expanding the number of complementary feature-set pairs in combination with different feature importance calculations and extend the interpretability analysis to the individual patient level. To better understand these results, feature values and importance should be studied together. Without knowing the feature values, we cannot determine how they influence predictions. For example, while age might be a significant predictor of mortality, we need to discern whether it is a better predictor for older or younger individuals. Furthermore, we need to understand the direction of each feature’s influence. Does old age increase the likelihood of predicting mortality, or does it suggest a greater chance of survival? Similarly, other features might have complex interactions that only become clear when values and importance are considered together. By integrating both aspects, we can gain deeper insights into the predictive power and behaviour of each feature within the models. Other performance metrics should also be included as AUROC and AUPRC do not sufficiently describe the performance of a model. In particular, individual-level case studies can provide valuable insights into how features interact in specific patient scenarios, highlighting nuances that are less apparent from aggregate numbers. Future research could incorporate such individual-level analyses as a natural extension of this work, helping to link model explanations with clinically meaningful patterns.
For future studies, it would also be interesting to consider more than ten features per model as well as investigate the specific feature combinations. Including multiple highly correlated features in the same feature set limits the potential of more diverse information if the feature set size is limited to ten. Additionally, testing different feature importance techniques could help assess the robustness of feature rankings. Investigating whether certain feature sets can be identified as likely to yield good or poor performance before training could also be an interesting direction for further research.
Translating the results into clinical practice demands additional work and development. The findings of this study are general and cannot be directly applied in clinical decision-making. Standard metrics such as AUROC and AUPRC reflect overall discrimination rather than clinical utility at relevant operating thresholds, and future work should examine task-specific metrics that align with the specific clinical setting36. Models intended for implementation require different design choices than those for exploratory research, including consideration of outcome-action pairing, prediction timing, and presentation of results37,38. Providing clinicians with clear, task-relevant explanations of the features driving a prediction may support informed decisions, but these explanations describe model behaviour rather than real-world effects. Different models can yield different explanations for the same patients, highlighting the need for cautious interpretation. Explanations such as features relevant to the prediction should therefore be integrated into workflows in ways that deliver actionable and verifiable information, such as confidence estimates or indicators of model unreliability. Importantly, models should be designed to support rather than replace clinical judgement, and their use requires context-sensitive implementation and governance to ensure patient safety and clinical value39.
