Machine Learning Enhanced Expert System Detects Reverse Heart Failure Using Patient Reported Vitals and Electronic Health Records

Input variables

The rules-based algorithm used morning measurements of blood pressure, weight change, heart rate, and symptoms to generate same-day alerts. To improve the rule-based algorithm, we added morning measurements for the two days before the parameters along with the same day measurements for a total of three consecutive days. Additionally, information regarding laboratory surveys for each patient was included. Laboratory investigations included complete blood count (CBC), lipid profile, serum electrolyte, renal function tests, serum urea, serum creatinine, type B natriuretic peptide (BNP), and serum creatinine. It also included patients' medication history, other comorbidities, and smoking history. Table 1 shows that the analysis used the complete set of input variables.

Measurement of results

Medly's rule-based algorithm outputs different alerts. A total of eight alerts were generated based on patient vitals. One responds to “everything is normal” and eight responds to “high level emergency or call 911”.⁸. In the rule-based output, alert numbers 1-3 corresponded to the need for contact with a doctor or clinic. However, alerts 4-8 required some warnings, diuretic instructions, and contact with the corresponding clinic. For simplicity, we have categorized these eight alerts into two classes: “Contact” or “No Contact”. If a patient was asked to contact the care team at any stage, the alert was classified as “contact.” Otherwise, all other alerts were classified as “No Contact”. Additionally, clinician experts JKK VN and CM trained and supervised three highly qualified personnel.

All alerts are manually evaluated and clinical data is used to determine true positives, true negatives, false positives, and false negatives in the evaluation system. The definitions of true positives, true negatives, false positives, and false negatives in our system are:

1. a)

   True positive: Rule-based algorithms generated “contact” alerts that were deemed appropriate by cardiologists (changes in treatment, counselling delivery, or escalation of care).

2. b)

   True Negative: Rule-based algorithms did not generate “contact” alerts and patients remained stable (no emergency visits or significant clinical degradation within the next 24 hours).

3. c)

   FALSE positive: The rules-based algorithm generated “contact” alerts that were deemed inappropriate by cardiologists (without any change in treatment, counselling delivery, or escalation of care).

4. d)

   False negative: The rule-based algorithm did not generate a “contact” alert, but the patient experienced clinical degradation within the next 24 hours and received emergency care.

Estimation of HF decompensation episodes

Model development

The predictive model was developed using the Xgboost framework and employs a gradient boost decision tree ensemble with binary logistic objective functions¹³. One of the main advantages of XGBoost is that it provides parallel tree boosts that quickly and accurately solve many data science problems. For these reasons, XGBoost has been widely used to solve some healthcare-related classification and regression problems^{14,15,16,17,18,19}. An implementation of Xgboost Python was used (https://xgboost.readthedocs.io/en/stable/), version 1.4.2.

Hyperparameter tuning was performed through randomized grid searches to optimize model performance while preventing overfitting. The search investigated key hyperparameter combinations such as tree depth, learning speed, column subsampling ratio, class weighting to address imbalances, and number of boost rounds. Each configuration was evaluated using 5x layer decomposition cross-validation, with the area under the receiver operating characteristic curve (AUC) being evaluated as a scoring metric.

Following hyperparameter optimization, the optimal parameter set was merged with a fixed baseline configuration containing the use of a GBTree booster, subsampling of the training instance, and fixed random seeds for reproducibility. The final model was trained on the complete training dataset, and performance was monitored on the hold-out validation set using AUC as the evaluation criterion. After high validation performance, we stopped early with 30 iterations of patience to finish training, increasing generalizability and reducing the risk of overfitting.

To handle missing data, the model utilized Xgboost's native mechanism. This automatically learns the optimal split direction of missing values ​​in the tree structure. All continuous variables were standardized prior to training to reduce the effects of outliers and ensure numerical stability. The categorical variables were encoded in one hot, and features derived from patient-reported daily measurements, previous day vital rolling summary, and laboratory-based EHR values ​​were included in the model without manual function selection, allowing the algorithm to directly learn the interactions of related features.

The feature set in Table 1 was used. In conjunction with the feature set, we used the output from a Medly Rules-based algorithm and used it as an input to the Xgboost classifier. Overall, the model inputs are: Same-day function + previous 2 days function + Medley rule-based algorithm output + EHR data (Figure 1).

Verification and evaluation

The dataset was split into training and test sets. Our dataset consisted of 342 patients measured daily for 1-3 years. First, the data set was split into training and test sets at the patient level at a ratio of 80:20, with no overlapping patients in the training and test sets. The resulting training set includes 52,932 incidents and the test set includes 11,161 incidents (labeled by three highly qualified personnel, as mentioned above). However, the training dataset overall had a 10:1 ratio of “no contact” to “contact” and “contact” overall. To mitigate the effect of class imbalances on model training, the training set randomly subsampled “no contact” incidents to match the ratio of “contact” incidents to 0.75. This results in 11,636 incidents used in training. The “contact” and “no contact” ratios of the test set did not change. Parameter searches were performed with 5x cross-validation only on the training set. After optimal parameters were obtained, the model was trained on a training set. All data was standardized using standard scaler classes from the sci-kit Learn library to handle outliers and make the process more efficient. Missing dosage information was considered to have not been given a prescription.

The performance of the test set architecture was assessed and sensitivity, specificity, positive predictor (PPV), negative predictor (NPV), and accuracy were calculated. Additionally, accuracy, Recall, and F1 scores were calculated to assess the performance of the model. To ensure a fair comparison with Medly Rules-based algorithms, we further selected classification thresholds based on the true positive rate of Medly Rules-based algorithms. Changes in PPV were assessed by holding the true positive rate at the same level. This allows you to investigate improvements in PPV without sacrificing sensitivity.

Interpretability analysis

Additionally, feature importance analysis was performed to determine which features predicted episodes of decompensation and played an important role in correcting false positives from rule-based algorithms. To achieve this, three analyses were performed. (a) SHAP summary plots of the model, (b) ablation studies, and (c) additional performance analysis of EHR data between the original Medly rule-based algorithm and the Xgboost algorithm. Each of these strategies regarding the interpretability and importance of functioning of the model is explained in detail below.

SHAP Summary

We investigated the importance of functionality using SHAP methods²⁰. The SHAP method evaluates the relative contribution of each feature to model prediction and assigns a Shapley score to each feature as a measure of its importance. Shap summary plots were used to visualize the importance of functionality²⁰. Shap Summary plots outperform the importance bar charts of standard features that do not represent the range and distribution of the importance of the model's output. Alternatively, the SHAP summary plot represents the importance of features as functional effects of the model. In the SHAP summary plot, each point represents a sample. The X-axis represents the Shapley value, and the point color (ie, the Y-axis) represents the functional value. The X-axis shows the Shapley value for each sample (>0: contribution to positive prediction; <0: contribution to negative prediction). Red indicates a high feature value, while blue indicates a low feature value.

I used a Python implementation of Shap (https://shap.readthedocs.io/en/latest/), version 0.39.0.

Ablation research

Additionally, several ablation studies were conducted to confirm the importance of the function. Several combinations of input functions were applied to the xgboost model to check the performance of the algorithm. The hypothesis was that if removing a set of features reduced the PPV rate, the set of features played an important role in reducing false positives. Below is a list of ablation studies.

1. a)

   Ablation Study 1: Functions on the same day.

2. b)

   Ablation Study 2: Same-day function + Medley rule-based algorithm output.

3. c)

   Ablation Study 3: Same Day Function + Medly Rule-Based Algorithm Output + EHR Data.

4. d)

   Ablation Study 4: Same Day Features + Functions for the previous 2 days.

5. e)

   Ablation Study 5: Features of the same day + Features of the previous 2 days + Medley Rule-based algorithm output.

Evaluation of these ablation studies was based on PPV, NPV, sensitivity, and specificity.

After the first set of ablation studies, another set was performed with EHR data. Here, we removed each continuous variable one at a time from the EHR dataset, retrained the model and tested the performance of the test data. Additionally, PPV and F1 scores (harmonic mean between accuracy and recall) were assessed after removing the variables. The hypothesis was that if the PPV and F1 scores decreased from the overall score after deleting a variable, that variable could play an important role in reducing false positives. Therefore, 25 ablation studies were performed and the performance of the algorithms for each instance was compared.

Performance analysis of EHR data usage between Medly rulebases and Xgboost algorithms

For both the Medly rule-based algorithm and the XGBoost algorithm, the mean and standard deviation of each consecutive EHR data variable for estimated true negatives, false positives, true positives, and false negatives were calculated. We then compared the trends in EHR values ​​between rule-based and false-positive disorders estimated by the XGBoost algorithm. If the average EHR value increases with false positives estimated from the rule base to Xgboost, this indicates the fact that Xgboost considered the lower values ​​of that variable in the process of correcting false positives.