As AI-powered chatbots explode in popularity, experts are warning people against turning to technology for medical or mental health advice rather than relying on human health providers.
Over the past few weeks, chatbot advice has had several examples of misleading people in harmful ways. The 60-year-old man accidentally poisoned himself, and entered a psychotic state after suggesting that ChatGpt would remove salt or sodium chloride from his diet and remove things like sodium bromide, a toxin used to treat wastewater. Earlier this month, a study from the Center for Countering Digital Hate revealed ChatGpt received dangerous advice on drugs, alcohol and suicide.
The technology is particularly appealing with barriers to accessing healthcare, including costs, waiting times to talk to providers, and lack of insurance coverage. However, experts told PBS News that Chatbots cannot provide advice tailored to a patient's specific needs or medical history, and tend to provide “hastisation” and total misinformation.
According to mental health and health experts who spoke to PBS News, here are some things you need to know about using AI chatbots for health advice:
What about people using AI chatbots for medical and mental health advice?
Beyerlite, senior director of healthcare innovation at the American Psychological Association, usually turns his eyes to commercial chatbots such as Openai's ChatGpt and Luka's Replika.
People may ask very broad questions, such as stopping smoking, dealing with interpersonal violence, facing suicidal thoughts, or treating headaches. A study created by Common Sense Media says that over half of teens use the AI chatbot platform multiple times each month. The report said about a third of teens turned their attention to AI peers for social interactions, including role-playing, romantic relationships, friendships and conversational skills practice.
read more: Analysis: Healthcare AI can save lives and money, but still
But the business model for these chatbots is to engage users “as much as possible” rather than giving reliable advice in vulnerable moments, Wright said.
“Unfortunately, none of these products are built for that purpose,” she said. “The products on the market are in a way very opposed to treatment because coding is built in a fundamentally addictive way.”
In many cases, Dr. Tiffany Manzer, a developmental behavioral pediatrician at the University of Michigan School of Medicine, reflects the human emotions that involve them in a way that bots “mistake a highly critical moment.”
“If you're in a sad state, the chatbot might reflect more of that feeling,” Manzer said. “Emotional tones tend to match and match the user. It can make it difficult to provide advice that is against what users want to hear.”
What are the risks of using AI for health advice?
Instead of asking healthcare providers for AI Chatbots health questions, there are some risks, says Dr. Margaret Lozovatsky, Chief Medical Information Officer of the American Medical Association.
Chatbots “may give you a simple answer to a question someone has and may not be able to contact a doctor,” Rozobatsky said. “That being said, the simple answer may not be accurate.”
- Chatbots don't know a person's medical history. A chatbot is not your doctor or nurse practitioner and cannot access your medical history or provide customized insights. Chatbots are built on machine learning algorithms. In other words, they generally produce the most likely answers to questions based on an expanded diet of information harvested from the internet wilderness.
- Hallucinations occur. Although AI chatbots have improved response quality, hallucinations still occur and in some cases can be fatal. Lozovatsky recalled one example a few years ago and asked the chatbot, “How do you treat UTIs (urinary tract infections)?” Bot replied, “Drinking urine.” With her and her colleagues laughed, Rozovatsky said that this type of hallucination can be potentially dangerous when asking questions that the accuracy of the answer may not be so obvious.
- Chatbots enhance false confidence and atrophy critical thinking skills. People need to read the AI chatbot responses with a critical eye, experts said. It is especially important to remember that these responses usually confuse origins (they need to dig into the source link), and that chatbots “have no clinical judgment.” “You're losing your relationship with your doctor.”
- Users are at risk of publishing personal health data over the Internet. In many ways, the AI industry corresponds to the modern wild west, especially when it comes to protecting people's personal data.
Why do people look to AI as mental health and health resources?
According to Rozobatsky, it's not uncommon for people to seek answers for themselves when they have a headache, sniffle, or strange, sudden pain. Before the chatbot, people relied on search engines (cuing all the jokes about Dr. Google). Previously, the self-help book industry had overturned money with the low-humming anxiety of people who were able to feel better today and feel better than tomorrow.
Today, search engine queries may produce results generated in the AI that are first displayed, followed by a series of websites where there or may not be information accurately reflected in those responses.
“It's a natural place where patients turn,” Rozobatsky said. “That's an easy path.”
Its ease of access contrasts with the barriers patients often encounter when seeking advice from licensed healthcare professionals. These obstacles include whether there is coverage for insurance, if the provider is in the network, if it is able to afford a visit, if the provider is concerned about the stigma associated with the question, and if there is reliable transport to the provider's office or clinic if telehealth services are not an option, if they can afford a visit, if they have insurance,
One of these hurdles can be enough to motivate a person to feel more comfortable than asking more sensitive questions than humans. At the same time, a well-documented solitude epidemic across the country has fueled partly the increased use of AI chatbots, Munzer said.
“Kids are growing in a world where they don't have the social support or social networks that they really deserve and need to thrive,” she said.
How can people protect against bad AI advice?
If people are worried about whether their children, family, or friends are involved in a chatbot for mental health or medical advice, it is important to book a judgment when trying to have a conversation about a topic, Munzer said.
“We want families, children and teens to have a lot of information at their fingertips to enable the best decisions,” she said. “Most of it is about AI and literacy.”
Discussing the underlying technology driving chatbots and the motivations to use them can provide an important point of understanding, Munzer said. This includes asking why they become part of such an increase in daily life, and about the business models of AI companies, and what can support the mental health of children and adults.
A helpful conversation prompt suggested to the caregiver is to ask, “What would you do if you revealed that your friend is using AI for mental health purposes?” The language “can remove judgment,” she said.
One of the activities Munzer recommends is to encourage families to test AI chatbots together, talk about what they find, and encourage their loved ones, especially their children, to look for information hallucinations and biases.
clock: Things you need to know about AI transcription tools that “hastisfies” medical interactions
But Munzer said the responsibility to protect individuals from harm generated by chatbots is too big to be placed solely on family members. Instead, regulatory rigor from policymakers is required to avoid further risks.
On Monday, the Brookings Agency produced an analysis of the state's law for 2025, finding that “healthcare is the main focus of the law, and all bills focused on the potential issues that AI systems make treatment and compensation decisions.” A small number of states, including Illinois, have banned the use of ChatGpt to generate mental health therapies. The Indiana bill requires patients to inform them that health professionals are using AI to generate advice or that they are notifying them of their decision to provide health care.
One day, Wright said chatbots can fill the service gap, but that's not the case yet. Chatbots can utilize deep information wells, she said, but that doesn't lead to knowledge and discernment.
“I think there's a future where we have a mental health chatbot that's rooted in science, rigorously tested and co-created for purposes. “But that's not what we have now.”
We don't go anywhere.
Stand up for truly independent and reliable news you can count on!
