Study finds AI is as good as doctors at prioritizing which patients to see first

Emergency departments across the country are overcrowded and overburdened, but a new study suggests that artificial intelligence (AI) could one day help prioritize patients who most urgently need treatment. There is.

Researchers at the University of California, San Francisco used anonymized records of 251,000 adult emergency department (ED) visits to determine how well an AI model could extract symptoms from a patient's clinical record and provide immediate treatment. We evaluated whether it was possible to determine the necessity of They then compared the AI ​​analysis to patients' scores on the Emergency Severity Index. The Emergency Severity Index is a scale of 1 to 5 that ED nurses use to allocate the care and resources with the greatest need when a patient arrives, a process known as triage.

Patient data was separated (anonymized) from their actual identities for the study, which was published on May 7th. JAMA network open. The researchers accessed the data through UCSF's secure generative AI platform with extensive privacy protections and evaluated the data using the ChatGPT-4 large-scale language model (LLM).

The researchers tested the performance of LLM using a sample of 10,000 matched pairs (20,000 patients total). The sample included one patient with a serious illness, such as a stroke, and another patient with a less urgent illness, such as a broken wrist. Considering only the patient's symptoms, the AI ​​was able to identify which of the two her ED patients had more severe symptoms 89% of the time.

In a subsample of 500 pairs evaluated by physicians and LLMs, the AI ​​was correct 88% of the time, compared to 86% for physicians.

AI can assist in the triage process, freeing up critical physician time to treat the most seriously ill patients while providing a backup decision-making tool for clinicians juggling multiple emergency requests. We may be able to provide it.

“Imagine there are two patients who need to be taken to the hospital, but there is only one ambulance. “We have to decide who we're going to address,” he said. The lead author is Christopher Williams, MB, BChir, and his UCSF postdoctoral fellow at the Bakar Computational Health Sciences Institute.

Not yet ready for prime time

This study is one of the few to evaluate LLM using real-world clinical data rather than simulated scenarios, and the first to use more than 1,000 clinical cases for this purpose. This is also the first study to use data from emergency department visits, where a variety of medical conditions can be considered.

Despite the study's success, Williams cautioned that without further validation and clinical trials, AI is not ready for responsible use in emergency medicine.

“It’s great to show that AI can do amazing things, but the most important thing is to consider who is helped and who is hindered by this technology,” Williams said. “Is the only criterion for using AI to be that it can do something, or is it that it can do something well for all kinds of patients?”

One of the key issues to solve is how to remove bias from the model. Previous research has shown that bias in the data these models were trained on can perpetuate racial and gender bias in healthcare. Williams said that before these models can be used, they need to be modified to remove that bias.

“First of all, you have to know if it works, and you have to understand how it works. Then you have to be careful and cautious about how you apply it,” Williams said. he said. “Future research will consider how best to bring this technology into clinical practice.”