setting
SUMMA Health, a nonprofit integrated healthcare delivery system in northeast Ohio, covers two acute care hospitals. SUMMA Hospital A (ACH Emergency Department) and SUMMA Hospital B (SHB Emergency Dept.). This system is further complemented by two standalone EDSs -Summa ED C (ACH Green Emergency Dept.) and Summa ED D (SHB Wadsworth Emergency Dept.). These facilities form a 1,300-bed system that promotes encounters of over 1 million patients each year. Summa Health mainly serves Summit, Wayne, Medina, Portage and Stark counties in Ohio. Service areas include urban, suburban and rural areas. The Summa Health Patient Payers mix is 5% uninsured, 25% personal insurance, 30% Medicaid, and 40% Medicare. This study was approved by the Duke Institutional Review Board under the protocol number pro00109055. As a care improvement study, the requirement for informed consent for patients was exempt by the board.
data
This generalizability study was conducted based on encounters of adult patients (ages 18 and older) presented in one of SUMMA Health's four emergency departments (EDS) on one of 1/1/2020 to 12/31/2021. This study was approved by the Institutional Review Board of Summa Health and granted a patient consent exemption. Meets began at the time of presentation to the ED and ended upon discharge or death. The encounters began with independent ED and were attributed to the location of origin, as encounters including transfers to an acute hospital were assigned to independent ED. Individual visits to the emergency department were considered different encounters regardless of the number of visits made by the same patient. After the initial analysis, encounters with a length of stay of less than 1 hour were excluded due to missing and incompleteness of important data in this time frame, resulting in unreliable predictions based on a few variables. Only the first 36 hours of each encounter were used for model evaluation.
This model combines encounters updated during encounters with unchanging static data through dynamic data. Static variables included patient demographics, encounter details, and comorbidity data, and we looked into ICD-10 codes documented in any encounter 12 months prior to the ED presentation. Dynamic data used in the model include analyte outcomes, vital signs, and medication administration. Dynamic variables were considered between encounter start and end times. Based on variable measurement type, new measurements were updated every hour when the RNN makes predictions for dynamic variables. For example, if 10 heart rate measurements were recorded within an hour, the average was used. If new measurements were not available, the most recent value was carried over. Missed data were addressed by forward fill values for each encounter per hour. If previous data was not available, missing values were attributable.
Define the results
The previously developed definition of sepsis-2 was used as the outcome label. Specifically, sepsis was defined as the co-occurrence of all three criteria: (1) Criteria for at least two systemic inflammatory response syndromes. This is valid for 24 hours, with abnormal temperature (>100.4 f or <96.8 f), heart rate above 90, over 20, abdominal white blood cell count (12 or<4を超える） （2）血液培養命令。 （3）クレアチニンの上昇（> 2.0), INR (>1.5), total bilirubin (>2.0), and platelet count (<100) reductions (<100）、2以上の乳酸レベル、または6時間以内の収縮期血圧の40 mmHgの低下の減少（> 2.0), reduction in total bilirubin (>2.0), signs of terminal damage. The various sampling rates for medical measurements were explained by adjusting the time window associated with each criterion. Vital sign document values (temperature, heart rate, respiratory rate, blood pressure) were valid over a 6-h time frame, whereas analyte measurements (leukocyte count, creatinine, bilirubin, platelet count, lactic acid) and order (blood culture) were valid over a 24-h time frame. Additionally, patients who met the criteria for sepsis within 1 hour of presentation to ED were excluded. Sepsis monitoring was assessed using a 12-h detection window. This means that if the prediction violates the threshold, it will only be classified as a true positive if the patient meets the sepsis criteria within 12 hours. If the prediction exceeded the threshold 12 hours prior to sepsis, the prediction was classified as a false positive. Sepsis Watch was run every hour per hour to generate predictions for presentation to ED and for minimum time of sepsis, death time, discharge time, and all encounters 36 hours after ED presentation. If the prediction violated the threshold, the prediction over the next 8 hours was suppressed or snoozed. This 8-hour snooze window is designed to reduce false positive alerts and potential downstream alert fatigue. Snooze window avoided the inflation of performance metrics by repeating counting true positivity.
Machine Learning Models
The original model was designed at Duke Hospital using EHR encounter data from October 1, 2014 to December 1, 2015. This is a recurrent neural network (RNN) model, referred to here as the sepsis clock. The data used to train the models were split for training, internal validation and testing, respectively. The original model worked well in multiple settings. Specifically, in internal validation, the cohort sepsis watch achieved an area under the receiver operator curve (AUROC) of 0.882, and the model achieved an AUROC of 0.943 in the time validation cohort. Details of model development and evaluation can be accessed in the original development, internal verification manuscripts, and assembled manuscripts.18,28,41. The main purpose of this study was to assess the ability of sepsis clocks to generalize to populations beyond the location of training, so the parameters of the model were not refitted for this validation study.
evaluation
To answer the initial research goals, Model Performance was assessed under the Recever Operator Curve (AUROC) in a new healthcare setting using accuracy, Recall, AUPRC, and area. To better understand the performance of sepsis clocks at various ED and hospital sites, they were assessed individually at each location within Summa Health.
Additional analyses were conducted to answer the second research goal. First, to estimate the potential effect of model integration on clinical care, the average “lead time” was quantified and defined as the amount of time between the prediction of the “high-risk” model and the patient's sepsis criteria. This measure helps to quantify potential opportunities for previous interventions. Lead times were measured only for true positive cases identified early by sepsis monitoring over a 12-h time window compared to the time patients met clinically defined sepsis criteria. Second, to assess the workflow burden placed on staff, we assessed the number of alerts sent to either the billing nurse or the rapid response team at a specific model threshold. The lead times and number of alerts are calculated separately for each Summa Health site.
Local adaptation
Deploying sepsis monitoring in new locations requires local adaptation, primarily identifying data elements in the local EHR for entity resolution. This process involves mapping raw EHR data elements to the standardized inputs required by the RNN model. For example, this involves grouping all component names associated with blood culture order into a single unified data element. Additionally, thresholds must be established for the model based on the results of retrospective analysis or silent assessments on the new site, determining when alerts are sent to care providers. It is also important to note that no changes were made to the weights or variables in the model in this study. This approach was taken to demonstrate the robustness of the Duke model and ensure true external validation to match the goals of this analysis.
