summary: A new study tests the diagnostic capabilities of generative AI, especially the chatbot GPT-4, with promising results.
This study evaluated the diagnostic accuracy of AI in handling complex medical cases, and found that GPT-4 correctly identified the top diagnosis almost 40% of the time, and latent in 64% of difficult cases. Include the correct diagnosis in the correct diagnosis list.
The success of AI in this study may provide new insights into potential applications of AI in clinical settings. However, further research is needed to address the benefits, best uses, and limitations of such technologies.
Important facts:
- In a study of 70 complex clinical cases, GPT-4 correctly matched the final diagnosis 39% of the time.
- In GPT-4, the differential list (a list of potential conditions based on the patient’s symptoms, medical history, and clinical findings) contained the correct diagnosis in 64% of cases.
- Despite the encouraging results, researchers stress the importance of further research to understand the optimal uses, benefits, and limitations of AI in clinical settings.
sauce: BIDMC
In a recent experiment published in JAMAphysician researchers at Beth Israel Deaconess Medical Center (BIDMC) tested the ability of one well-known, publicly available chatbot to make accurate diagnoses in difficult medical cases.
The research team found that Chat-GPT 4, a generative AI, selected the correct diagnosis as the top diagnosis almost 40% of the time, and in two-thirds of the difficult cases the correct diagnosis was added to the list of potential diagnoses. was found to provide
Generative AI refers to a type of artificial intelligence that uses trained patterns and information to create new content, rather than simply processing and analyzing existing data.
Some of the best-known examples of generative AI are so-called chatbots. It uses a field of artificial intelligence called natural language processing (NLP) that enables computers to understand, interpret, and generate human-like language. Generative AI chatbots are powerful tools that are revolutionizing creative industries, education, customer service, and more.
However, little is known about its potential performance in clinical settings, such as complex diagnostic reasoning.
“Recent advances in artificial intelligence have given rise to generative AI models capable of detailed, text-based responses that can score high on standardized health exams,” said co-director of innovation in media and education delivery. Adam Rodman, M.D., MPH (iMED), BIDMC Initiative and Medical Lecturer at Harvard Medical School.
“We wanted to know if such a generative model could ‘think’ like a doctor, so we asked it to solve a standardized complex diagnostic case used for educational purposes. . It really worked out really well. ”
To assess the chatbot’s diagnostic skills, Rodman and colleagues used clinicopathological case conferences (CPCs). This is a series of complex and challenging patient cases with relevant clinical and laboratory data, imaging studies, and histopathological findings published in 2016. New England Journal of Medicine For educational purposes.
Of 70 CPC cases evaluated, artificial intelligence accurately matched the final CPC diagnosis in 27 (39%) cases. In 64 percent of cases, the definitive CPC diagnosis was included in the AI’s differential statement (a list of conditions that could explain the patient’s symptoms, medical history, clinical findings, laboratory or imaging findings) .
“While chatbots cannot replace the expertise and knowledge of trained medical professionals, generative AI has a promising potential to assist human cognition in diagnosis,” said BIDMC Hospitalist. said lead author Zahir Khanzi, M.D., Ph.D., MPH, assistant professor at Harvard Medical School. Medical school.
“It has the potential to help physicians understand complex medical data and broaden and refine their diagnostic thinking. These are interesting findings for the future of diagnostics and patient care.”
“Our study joins a growing body of literature demonstrating the promising capabilities of AI technology,” said co-author Byron Crow, M.D., Ph.D., BIDMC internist and medical lecturer at Harvard Medical School. I’m here.
“Further research will help us better understand how these new AI models will transform healthcare delivery.”
This work is not separately funded or sponsored. Kanjee reports from Wolters Kluwer editorial book royalties and paid advisory board membership for medical education products not related to artificial intelligence, and her CME honoraria provided by Oakstone Publishing. Crowe reports her employment with Solera Health outside of her submitted work. Rodman reports no conflicts of interest.
About this ChatGPT and AI research news
author: Chloe Meck
sauce: BIDMC
contact; Chloe Meck – BIDMC
image: Image credited to Neuroscience News
Original research: closed access.
“Accuracy of generative artificial intelligence models on complex diagnostic tasks” Adam Rodman et al. JAMA
overview
Accuracy of Generative Artificial Intelligence Models in Complex Diagnostic Tasks
Recent advances in artificial intelligence (AI) have given rise to generative models capable of accurate and detailed text-based responses to written prompts (“chat”). These models score high on standardized physical examinations.
Less is known about their performance in clinical applications such as complex diagnostic reasoning. We evaluated the accuracy of one such model (Generative Pre-trained Transformer 4). [GPT-4]) series of difficult-to-diagnose cases.
