When Judith Miller, a 77-year-old patient in Wisconsin, received the results of a medical imaging study last year, she did what many patients do today. She asked the AI for an explanation. A large language model (LLM) developed by Anthropic, Claude was eager to explain the possible interpretations. Armed with the chatbot’s analysis, Miller went to her follow-up appointment feeling ready to have a productive conversation with her doctor. As she says, Claude’s response “helped me understand my health more deeply and participate more fully in shared decision-making.”
This sight has become commonplace in clinics across the country. Two recent polls both found that one-third of American adults seek health information from LLMs to understand test results, diagnose symptoms, research treatment options, or inquire about prescription medications. “The use of such tools has doubled in the past year,” says Robert Wachter, a physician at the University of California, San Francisco. “I think it will double again next year.”
However, experts urge caution when using these chatbots, as they can also provide misleading or inaccurate advice. Anthropic agrees. “Claude is not designed or marketed to perform clinical diagnostics,” a company spokesperson said. Its proper use is “to help people prepare for conversations with doctors, not to replace them.”
About supporting science journalism
If you enjoyed this article, please consider supporting our award-winning journalism. Currently subscribing. By subscribing, you help ensure future generations of influential stories about the discoveries and ideas that shape the world today.
For many patients, AI is a welcome solution to the problems caused by the glut of personal health data provided by the 21st Century Cures Act, which mandates instant online access to medical records such as test results and clinical records. “If you’ve seen anything like this, you know there’s a big question: What does this mean?” says Dave DeBronkart, a health care blogger and activist. Until just a few years ago, its meaning was hidden behind a wall of medical jargon that only doctors could understand. Patients can also check their results online before speaking with their doctor, which often leaves them wondering what to make of the results. But now, universal chatbots and a number of specialized health models can translate jargon into plain language within seconds, potentially allaying unfounded fears.
But it can also unnecessarily increase anxiety or even make it worse. LLMs are still error-prone. These can present falsehoods as fact, flattering and reinforcing users’ previous (and sometimes false) beliefs. Although these character flaws may decrease as models become more powerful, many experts have expressed concern about the potential risks of using today’s AI models in this way. “There aren’t a lot of guardrails to compel people to tear up medical records and give out actual misinformation,” said Kate Desroches, executive director of OpenNotes, a nonprofit that promotes access to patients’ medical records. She added that there is little research into what happens when people treat LLMs as health authorities, and “I don’t think we have any idea how effective that is for the average patient.”
The worst-case scenario has already surfaced. In December, a 75-year-old Seattle man died from a treatable form of leukemia. He reportedly refused treatment based on AI-generated evidence that falsely suggested he had a rare complication. Some of the preliminary research into how people are using AI for medical diagnosis is sobering. in natural medicine In a study published in February, researchers asked participants to diagnose hypothetical conditions with the help of various LLMs. They reached the correct conclusion only one-third of the time.
Still, most experts agree that chatbots, if used carefully, can help people seeking medical information. “I don’t think people should avoid using them,” Desroches says. “But I think people should use them with their eyes open,” added Adam Rodman, a general internist at Beth Israel Deaconess Medical Center. “I would argue that LLM, if used properly – and this is a big caveat – is the best tool for patient empowerment ever invented.”
In hopes of leveraging this technology without compromising safety, researchers have developed a series of strategies to counter AI’s shortcomings. For example, they suggest instructing a chatbot to take on the persona of a doctor. This “could lead to models collecting data in a doctor-like way,” Rodman said. Other tactics include asking the LLM to rigorously reevaluate its own reasoning or seeking a “second opinion” from another model. Rodman emphasizes the importance of removing personal information, such as names and social security numbers, from chatbot input to protect privacy.
Ideally, after all digital interactions, patients will be able to ask better-informed questions of their doctors. Wachter says this trend is “generally healthy,” although valuable time may be lost debunking a chatbot’s flawed advice. “This appointment is 15 minutes long, and the first 10 minutes must be spent explaining to the patient what the GPT has instructed them to do,” he says.
In many cases, LLM is likely to completely replace actual clinical advice, especially for those who are uninsured or face long wait times to get an appointment. “Access issues are at crisis level,” said Laura Adams, senior adviser on AI issues at the National Academy of Medicine. Despite technology’s limitations, she argues, we need to compare it not with perfection, but with the reality that the alternatives may not be bothersome at all. “It’s better than doing nothing,” she says.
Adams points out that with AI and medical advice, “the horse is far from the barn.” As more people rely on chatbots to manage their health, researchers and patient advocates say the time is right for new forms of AI literacy. “The solution is not to leave people ignorant,” DeBronkart said. By educating children and adults alike, it is “teaching them how to do better.” On top of that, new LLMs are likely to improve in medical applications, Wachter suggests, and some models may eventually become board certified like actual doctors.
For now, people like Miller are already approaching AI with open eyes, as DeRoches recommends, recognizing that AI is prone to hallucinations and confirming users’ biases. Chatbot responses may be sophisticated, but they are stitched together statistical patterns from large datasets. While this is an impressive trick, it still falls short of the breadth and reliability of human-level clinical reasoning. “We’re just tracking possible words,” Miller said. “I do not consider this to be a source of absolute truth.”
