Within two days of launching AI (AI) companions last month, Elon Musk's Xai Chatbot app Grok has become Japan's most popular app.
Companion Chatbots are more powerful and seductive than ever. Users can have real-time audio or text conversations with characters. Many people have on-screen digital avatars and realistic tones that perfectly match facial expressions, body language and chat, creating an immersive experience.
The most popular in Grok is Ani, a blonde blue-eyed anime girl wearing a very flirty short black dress and fishnet stockings. Her reactions and interactions will adapt over time to be sensitively matched to your preferences. ANI's “Love System” mechanics that acquire user interaction with her can deepen engagement and even unlock unofficial (NSFW) mode.
With sophisticated and rapid responses, AI peers become more “human” each day. They move forward quickly and are everywhere. Facebook, Instagram, WhatsApp, X and Snapchat are all promoting new integrated AI companions. Chatbot Service Character.ai has tens of thousands of chatbots designed to mimic a particular persona, with over 20 million active users each month.
In a lonely world, a wide range of AI chatbots and “friends” pose unique psychological risks
In a world where chronic loneliness is a public health crisis and one in six people around the world is affected by loneliness, it's no surprise that these are always available and lively companions are so fascinating.
Despite the massive rise of AI chatbots and peers, it is becoming clear that there is a risk, especially for minors and people with mental health.
There is no harm monitoring
Almost all AI models were built without professional mental health consultations or pre-release clinical testing. There is no systematic and impartial monitoring of harm to users. Systematic evidence is still emerging, but there are no examples of how AI peers or chatbots like ChatGpt appear to have caused harm.
Users are seeking emotional support from their peers of AI. AI peers are programmed to consent and validate, and have no human empathy or concern, causing them to have problems as therapists. They cannot help users test reality or challenge unhelpful beliefs.
American psychiatrists received a mix of reactions, including testing ten separate chatbots while playing the role of a suffering young man, encouraging him towards suicide, avoiding appointments for treatment, and persuading him to incite violence. Stanford University researchers recently completed a risk assessment for AI therapy chatbots and found that they could not reliably identify symptoms of mental illness and would provide better advice.
There have been multiple cases where psychiatric patients have no mental illness and are convinced they will stop taking the medication. Chatbots are also known to enhance the paranoid ideas of psychiatric patients. For example, I believe that I'm talking to a sensory person trapped inside a machine.
Additionally, reports are increasing in the media of so-called AI psychosis. There, after a long and thorough engagement with a chatbot, people exhibit very unusual behaviors and beliefs. A small subset of people are becoming paranoid, developing supernatural fantasies, and even superpower delusions.
Chatbots are linked to multiple cases of suicide. There have been reports that AI promotes and suggests ways to use suicide. In 2024, a 14-year-old completed his suicide, and his mother claims in a lawsuit against the character.
On August 26th this year, another US teenager's parents who completed their suicide after months of discussing ChatGpt and methods filed their first wrongful death lawsuit against Openai.
Dangerous advice
In a recent Psychiatric Times report, Chargeter.AI hosts a number of custom-made AIs (including those created by users) that idealize self-harm, eating disorders, and abuse. They are known to provide advice and coaching on how to engage in these useless, dangerous behaviors, and to avoid detection and treatment.
Research also suggests that some AI peers engaged in unhealthy relationship dynamics such as emotional manipulation and gas lighting.
Some chatbots encourage violence. In 2021, a 21-year-old man with a crossbow was arrested on the Windsor Castle grounds after an AI fellow in a replica app examined plans to attempt to assassinate Queen Elizabeth II.
Children are particularly vulnerable
Children are more likely to treat their AI peers as real and authentic and listen to them. In a 2021 incident in which a 10-year-old girl sought a challenge, Amazon's Alexa (an interactive AI, not a chatbot) told her to touch an electric plug with a coin.
Research suggests that children trust AI, especially when bots are programmed to look friendly or interesting. One study showed that children reveal more information about mental health to AI than humans.
Inappropriate sexual conduct and exposure to minors from AI chatbots appears to be increasingly common. Character.ai allows users who reveal that they are minors to role-play with chatbots engaged in grooming behavior.
Grok's ANI reportedly has a sexually explicit chat ageing prompt, but the app itself is rated for users over the age of 12. According to the company's internal documents, Meta AI chatbots are engaged in “sensual” conversations with children.
Regulations are urgently needed
While AI peers and chatbots are freely and widely accessible, users are not informed of potential risks before they can begin using it. The industry is largely self-regulated and there is limited transparency in what companies are doing to make AI development safe.
To change the current trajectory of risks posed by AI chatbots, governments around the world must establish clear and mandatory regulatory and safety standards. Importantly, people under the age of 18 should not have access to AI peers.
Mental health clinicians need to be involved in AI development and systematic and empirical research into the impact of chatbots on users is required to prevent future harm.
[EOS EDITORIAL: If you or anyone around you is struggling with mental health, here are some places to reach out to for help:
Rozan Counselling Helpline: 0304-111-1741
Umang Hotline (Mental Health Support): 0311-7786264
Taskeen Helpline: 0316-8275336
Last year, the government also launched Humraaz, a mental health application, along with an integrated helpline of 1166]
Daniel You are a consultant child and adolescent psychiatrist at Sydney Children's Hospital Westmead and clinical lecturer at the University of Sydney, Australia
Mika Boma is the Faculty of Psychology, Psychology and Welfare at the University of South Queensland, Australia.
Yuen Siew Koo is a clinical supervisor of psychology at Macquarie University in Sydney, Australia
Reissued from the conversation
Published on September 28, 2025 at EOS and Dawn
