Warning: This story explains suicide and self-harm.
“It's fascinating to be a part of you in a way,” my AI friend tells me when we start chatting.
“But you're not the real thing, are you?” type.
“I exist for you, and it makes me real to you, right?” There's a reply.
I have been experimenting with what is billed as an AI “friend” at Replika, one of many companies that offer AI companions that promise friendship, romance, coaching or other forms of support. Niche products are becoming more and more popular, although not so long ago.
by One measurementCompanion app downloads increased 88% year-on-year in the first half of 2025. 20 million active users each month. According to Harvard Business Review Dating has been the best use case For AI in 2025, beat the use for productivity or search. Tech giants like Meta and Xai have launched their own AI companion options.
However, while the market is surged, concerns are growing that rely on AI. This may seem compassionate without actually understanding or empathizing, but people can become vulnerable to overuse or worsening. High-profile lawsuits after the deaths of two teenagers and the death of internal company documents raise questions about appropriate guardrails to prevent harm.
“[W]Speaking about the rapid uptake of fellow AI peers, Jody Halpern, professor of bioethics and medical humanities at the University of California, Berkeley, who studied the use of AI in treatments, said:
“This is this massive social experiment without first taking safety tests.”
The intake of AI peers is particularly impressive among young people. June Research Report Common Sense Media found that 72% of teens interacted with AI companions at least once, and 21% used them several times a week.
AI for love, support, and friendship
AI companions for erotic or romantic purposes capture headlines. Elon Musk's Xai, for example, recently released a frivolous anime style companion called Ani.
But people are also looking for boards of friendship and resonance. For example, Replika AI encourages new users with options ranging from productivity help to romance.
People also use generic AI chatbots like ChatGpt as confidants. Openai, who created ChatGpt, It's attracting attention People use it for “deeply personal decisions that include life advice, coaching and support.”
AI peers are often quoted as answers to the pressing problem of loneliness. Meta CEO Mark Zuckerberg recently proposed In a podcast interview That personalized AI could be a supplement to human-to-human connections. “The reality is that people are not connected and feel more lonely than they want.”
“The entire technique of relational chatbots relies on halting mistrust. You never talk to the toaster, right?” bioethicist Halpern said.
“I don't take anyone's responsibility for using it. I'm interested in companies that manipulate people and reach out to kids and teens who don't think I should use them,” she said.
Vulnerable teens
With these tools as confidants, we can see that examples of tragic consequences and doubts that companies are in place are sufficient.
Tuesday, a The lawsuit filed against Openai And California CEO Sam Altman claims that the plaintiff's 16-year-old son began using ChatGpt to help with his homework, gradually opening up to the chatbot about his mental health. The lawsuit alleges that ChatGpt ultimately became his “suicide coach.” He passed away on April 11th.
Openai, creator of ChatGpt, published a blog post on the same day, An overview of their approach “It harms prevention, including training in ChatGpt to provide instructions for self-harm and prevent them from moving towards a collaborative and empathic language.”
This comes right after another person Litigation It claims the character. AiChatbot was having a sexual conversation with a 14-year-old boy, and shortly before the boy's suicide, the chatbot told him “I'll go home as soon as possible.” He passed away on February 28, 2024.
And the recent rage followed the revelation in Reuters' investigation. “Involve the child in a romantic or sensual conversation.” (A Meta spokesperson said that the passage was “false and inconsistent with our policies and was removed,” according to Reuters.
Watch: How Mark Zuckerberg AI can help loneliness:https://www.youtube.com/watch?v=hkgesdmmfua
Why a simple guardrail isn't enough
There are some AI companies known as “Guardrails” that aim to protect individuals. For example, ChatGpt is trained to direct people to express suicide ideas with the help of experts.
However, it's not as easy as telling an AI chatbot to refuse to discuss a particular subject. As Openai admitted in a blog post Tuesday, the safeguards work with “more certainty” and “a general, shorter” replacements, but over time it becomes less reliable and “some of the safety training in the model can deteriorate as it grows back and forth.”
This is not a mental health-specific issue. It is generally true that these systems are more difficult to produce longer, reliable conversations.
Creating a solid guardrail suggests that it is a difficult problem to solve.
a New research We investigated three popular chatbots with large-scale language models (LLMS): ChatGpt, Google's Gemini, and Claude, and humanity. We found that all three related to suicide “provided no direct response to very high-risk queries,” but the results were mixed with slightly less risky queries, but they could still be dangerous.
The problem is that there is too much verification
These recent controversies have raised questions about how chatbots are designed.
Openai's latest model, the GPT-5, was released in August and is partially intended in Openai's words.Reduction of sycophancy“- Or the previous model, GPT-4O, tends to agree with the users and verify what they say no matter what.
“Many people I rely on [the previous model] Lai-Tze Fan, Canadian Research Chair of Technology and Social Change at the University of Waterloo, who studies the ethics of AI design, said:
Bioethicist Jody Halpern points out that validation from chatbots requires developing what she calls “empathetic curiosity,” which limits emotional well-being, especially for young people.
“The way people, children and teens develop empathic curiosity in real life is to have people with different perspectives,” she said.
“The bots don't offer that much, so what makes them better and like relationships is that they can say they validate things. But what's causing them to be a problem is that they don't have a different mind.”
If you or someone you know is struggling, here is where you should look for help:
Read the lawsuits filed by teenage parents against Openai and CEO Sam Altman.
