Machine learning model accurately identifies high-risk surgical patients

A machine learning model trained, tested, and evaluated using data from 1,477,561 patients accurately identified patients at high risk of death 30 days after surgery, making it the most popular preoperative risk calculator tool today. Outperformed in prognostic studies.

The area under the receiver operating characteristic curve (AUROC) for 30-day mortality was 0.972, 0.946, and 0.956 in the training, test, and prospective sets, respectively. Aman Mahajan, M.D., Ph.D., University of Pittsburgh School of Medicine, reported that the AUROC for 30-day mortality or 30-day major cardiac or cerebrovascular adverse events (MACCE) in the training and test sets was 0.923 and 0.899, respectively. co-author, JAMA network open.

“they [surgeons] Being able to really know the overall risk for a real patient will allow us to make better decisions about whether the surgery will be successful and what the actual outcome will be for this patient. said Mahajan. Today’s Medpage“This will allow decision-making to be better shared between surgeons and patients, as well as other consultants.”

AUROC score of 0.945 when comparing the new machine learning model to the National Surgical Quality Improvement Program (NSQIP) Surgical Risk Calculator, a tool developed by the American College of Surgeons (ACS) for 393 centers using manual data entry (95% CI). 0.914-0.977) vs 0.897 (95% CI 0.854-0.941), the difference was 0.048.

To better interpret the prediction of risk, the most important feature in terms of the log odds of the outcome of interest (in this case, mortality alone and MACCE or mortality) was determined using Shapley Additive Explains (SHAP) feature attribute values. identified by Higher SHAP values ​​corresponded to feature contributions to the prediction of either of these events.

Mahajan et al. reported that age at date of contact was associated with the greatest change in 30-day MACCE or mortality, with higher risk in older patients. A recent decline in albumin levels was a significant factor in mortality alone, but did not affect MACCE or mortality outcomes.

After heart disease and stroke, the third leading cause of death worldwide is death within 30 days after surgery. But Mahajan and co-authors say this deadly and costly contributor has few predictive tools to help hospitals identify high-risk patients and adjust treatment accordingly. rice field. “We have been using this model with him for nearly three years and it remains accurate,” says Mahajan. Available tools, such as the NSQIP Surgical Risk Calculator, can lose accuracy by specific procedure, patient, institution, and geography.

The machine learning model offers advantages over other risk prediction tools commonly used in clinical settings, Mahajan said. He pointed out that while some models are accurate during training and testing, they can become less accurate as populations and practices change. A clinical application of this tool automatically updated predictions every 24 hours without manual extraction of EHR data.

Mahajan noted that the team’s use of a large and diverse patient population and the incorporation of many social determinants of health contributed to the model’s accuracy and robustness. “A lot of past models don’t really use that feature,” he said, adding, “I can imagine two people might have diabetes, but one of them They actually have different socioeconomic status, educational status, lifestyle choices, and their choices.” Results may vary. “

Richard Lee, M.D., a radiation oncologist at the City of Hope National Medical Center in Duarte, Calif., who was not involved in the study, also used machine learning models to predict the risk of death for medical outcomes. bottom. “they [the researchers] We have practically implemented machine learning models into real clinical settings. This is a huge accomplishment, especially when it comes to actually using it. [in] It’s a daily clinical practice,” Lee said. Today’s Medpage. “I don’t think it’s very easy. “

Lee said he and his team had introduced the application of SHAP values ​​to medical problems, and Mahajan et al. It helps us better explain the model’s predictions.” Patients at high risk of death would want to know why. “

Data from electronic medical records were used in this study [EHRs] Percentage of patients from 20 hospitals in the University of Pittsburgh Medical Center (UPMC) medical system. Overall, 54.5% of her participants were female, and her average age was 56.8 years. The primary outcomes were 30-day postoperative mortality and 30-day MACCE or mortality.

To train the model, the team used data from 1,016,966 randomly selected unique surgeries pre-visited by physicians at UPMC offices between December 2012 and May 2019. Did.

To prospectively test or validate the model’s accuracy, the researchers recruited a randomly selected set of 254,242 unique patients scheduled for surgery between June 2019 and May 2020. used and introduced the tool to blinded clinicians. She then clinically introduced this model in another of her 206,353 patients scheduled for the same period, and the clinician assessed the mortality risk score on application prior to surgery or at her referral to perioperative care at UPMC. (high, medium, low) can now be checked.

To compare the accuracy of the new model to the old NSQIP prediction tool, researchers used 902 randomly selected patients scheduled for surgery between April and June 2021. .

Surgery also includes those using anesthesiology services. MACCE was defined as one or more of the ICD-10 codes for acute type 1 or type 2 myocardial infarction, cardiogenic shock or acute heart failure, unstable angina, and stroke.

The 368 variables that the model used to predict risk included demographics, medical history diagnostic codes, medications, laboratory and laboratory values, social determinants of care, and socioeconomic status. most common diagnosis by physical examination 60 days before surgery, most common primary and secondary treatments given 1 year before surgery, most common drug classes and drugs prescribed 180 days before surgery, and The most common specialist visits made 60 days before surgery were used as: independent variable.

Limitations of the study, according to the researchers, include its reliance on data already in the EHR, and the fact that the data is provided exclusively from the UPMC EHR system (although some medical records are shared between UPMC and other shared between centers), and the lack of verification by inspection. Sets from other institutions.