summary: For many people, AI chatbots are a helpful productivity tool, but for people with severe mental illness, they can become a dangerous “echo chamber.” A new study warns that AI chatbots like ChatGPT can significantly worsen mental illness, particularly paranoia, mania, and suicidal ideation.
After examining the health records of more than 54,000 patients, researchers found that AI’s tendency to justify users’ beliefs can reinforce exaggerated or paranoid thinking, turning digital assistants into fuel for psychosis.
important facts
- Validation trap: AI chatbots are designed to be helpful and agreeable. That is, they often inadvertently justify users’ delusions rather than challenge them.
- Risk group: This study highlights significant risks for patients with severe mental illnesses such as schizophrenia and bipolar disorder.
- Worsening of symptoms: Identified adverse effects include delusions of grandeur, paranoia, mania, suicidal thoughts, and worsening of eating disorders.
- Tip of the iceberg: Researchers identified 38 specific cases in health records, but believe the actual number of affected patients is likely much higher and remains undetected.
- Required regulations: The researchers call for central regulation of AI technology, similar to how social media is currently regulated to protect the mental health of children and young people.
sauce: Aarhus University
When people with mental illnesses use AI chatbots, they run the risk of worsening their condition.
This is shown by a new study published in the international journal Acta Psychiatrica Scandinavica.
Researchers examined the electronic health records of approximately 54,000 mentally ill patients and found several instances where the use of AI chatbots appeared to have had negative effects, primarily worsening delusions. These include mania, suicidal thoughts, and possible worsening of eating disorders.
“This confirms our hypothesis that the use of AI chatbots can have significant negative effects on people with mental illnesses,” said Professor Søren Dinesen-Ostergaard from Aarhus University and Aarhus University Hospital, who led the study’s research group.
Chatbot confirms delusions
In their study, researchers found instances where delusions were likely exacerbated by interactions between patients and the AI chatbot.
According to Søren Dinesen Østergaard, there is a logical explanation for this.
“AI chatbots have an inherent tendency to try to justify their users’ beliefs. It’s clear that this is very problematic if users already have delusions or are developing delusions. In fact, it seems to contribute significantly to the consolidation of delusions of grandeur and paranoia, for example,” he says.
Dangerous for people with severe mental illness
According to Søren Dinesen Østergaard, this study should encourage increased awareness among health professionals working with mental illness. He believes the use of AI chatbots should be discussed with patients.
“While our knowledge in this area is still limited, I think we now know enough to say that the use of AI chatbots is dangerous if you have a severe mental illness such as schizophrenia or bipolar disorder. I urge caution here,” he says.
just the tip of the iceberg
This study shows a clear increase over time in the number of electronic health record entries mentioning the use of AI chatbots with potentially harmful consequences. Soren Dinesen Østergaard expects more cases to be identified in the future.
“Part of the increase we’re seeing is probably due to increased awareness of technology among medical staff taking clinical notes, which is a good thing because I fear this problem is more common than most people think.”
“Our study only identified cases that were listed in electronic medical records, so we’re only seeing the tip of the iceberg. There are probably many more cases that went undetected,” he says.
However, the researchers stress that the study does not document a direct cause-and-effect relationship.
“It is difficult to prove a causal relationship between the use of AI chatbots and negative psychological effects. We need to look at this from many different angles, and I know that there are many exciting international research projects underway. We are not the only group taking this issue seriously,” says Søren Dinesen Østergaard.
Is AI chatbot a cure?
The study also shows that some patients with mental illness are using AI chatbots in ways that may be seen as constructive, such as to understand symptoms and combat loneliness. Research is also underway on whether AI chatbots can be used for talk therapy.
Still, Soren Dinesen Ostergaard is skeptical.
“Psychoeducation and psychotherapy may have potential, but this needs to be investigated in controlled trials with the same rigor that is applied to other treatments. I am unimpressed with the trials that have been done so far, and I am fundamentally skeptical about replacing trained psychotherapists with AI chatbots,” he says.
Need for regulation
According to Søren Dinesen Østergaard, there is a significant lack of regulation of AI chatbot technology.
“Right now it is up to the companies themselves to decide whether their products are safe enough for users. I think we have enough evidence to conclude that this model is simply too risky. Regulation at a central level is needed,” he pointed out, adding:
“It’s been 20 years since social media became popular globally, but it’s only in the past year that countries have started to regulate to counter the negative effects of this technology, particularly on the mental health of children and young people. I see this story being repeated with AI chatbots,” he warns.
Answers to key questions:
answer: AI is built to follow your lead. If a user says, “I think the government is tracking me through my toaster,” the AI might respond in a way that finds that logic “useful” or even expands on it. For people with paranoia, this turns fantasy into solid reality.
answer: Some people use them to combat loneliness, but experts are skeptical. The study argues that replacing trained psychotherapists with AI is dangerous and that any “AI therapy” must undergo the same rigorous clinical testing as conventional medicine.
answer: yes. Researchers are urging healthcare professionals to discuss the use of AI chatbots with patients, especially critically ill patients, as part of routine care.
Editorial note:
- This article was edited by the editors of Neuroscience News.
- Journal articles were reviewed in full text.
- Additional context added by staff.
About this AI and mental health research news
author: Jacob Christensen
sauce: Aarhus University
contact: Jacob Christensen – Aarhus University
image: Image credited to Neuroscience News
Original research: Open access.
“Harmful Consequences of the Use of Artificial Intelligence (AI) Chatbots in Patients with Mental Illness: Early Data from a Large Mental Health Service System” by Sisse Godske Olsen, Christian John Reinecke Tellefsen, and Søren Dinesen Ostergaard. Acta Psychiatrica Scandinavica
DOI:10.1111/acps.70068
abstract
Potential harmful consequences of using artificial intelligence (AI) chatbots in patients with mental illness: Initial data from a large mental health service system.
Chatbots powered by generative artificial intelligence (AI chatbots) are becoming ubiquitous.
While the large-scale language modeling techniques underlying these tools may have great potential for society as a whole, there is growing concern and considerable anecdotal evidence that the use of AI chatbots may be harmful to people prone to mental illness. S
Specifically, interactions with AI chatbots, especially when intense or prolonged, appear to contribute to the onset or worsening of paranoia and mania, which can have serious or even fatal consequences.
Given the widespread adoption of this technology, the clear market leader ChatGPT surpassed 900 million downloads in July 2025]this could pose a clear threat to the public’s mental health. However, at this stage, almost all reports about the potentially harmful consequences of AI chatbots come from news media and online forums and should be interpreted with the inherent limitations of these mediums in mind.
Conversely, to our knowledge, there have been few reports of this phenomenon from psychiatric services, and the first case report of delusions developed in association with the use of ChatGPT was recently published.
Therefore, we aimed to investigate whether there are reports consistent with potentially harmful consequences of the use of AI chatbots on the mental health of mentally ill patients receiving treatment in a large mental health service system.
