The research team investigated explanations generated by large-scale language models (LLMs) for performing clinical prediction tasks. They found that after fine-tuning, San Francisco-based OpenAI’s LLM, GPT-4o, performed as well as four additional traditional machine learning (ML) methods in predicting which patients would discontinue home cancer treatment before completing their planned treatment.
Researchers from the Department of Biomedical Informatics, Dr. Congning Ni, Associate Professor Zhijun Ying, Ph.D., and others reported their research results in the e-book series “Studies in Health Technology and Informatics.” The research team used electronic health records and pharmacy surveys from 2,364 cancer patients.
LLM achieved an F1 score of 87%, while the closest ML model had a score of 83%. To interpret the latter model, the team used SHAP (Shapley additive explanations). This is a widely used method to explore the internal structure of ML models, revealing how features are weighted. To interpret the much larger GPT-4o, they asked LLMs to explain the basis for their individual predictions. To derive feature importance scores from this output, they used a new method. Call mimic-SHAP.
The two models were found to be matched on key characteristics: BMI and age. As a secondary feature, the LLM was found to be more dependent on the patient’s previous condition, and the ML model was more dependent on drug exposure and medical treatment.
Many cancer patients stop taking their home medications prematurely for a variety of reasons, including lack of attention to non-medical issues such as side effects and cost. Predicting early discontinuation may aid efforts to improve treatment adherence.
Other Vanderbilt researchers include Qingyuan Song, Qingxia Chen, PhD, Lijun Song, PhD, S. Trent Rosenbloom, MD, MPH, Autumn Zuckerman, PharmD, Bridget Lynch, PharmD, MS, and Bradley Malin, PhD. They were joined by Jeremy Warner, M.D., M.S., of Brown University; This research was supported by National Institutes of Health Award R37CA237452.
