The researcher, poses as a teenager, began a conversation with three commonly used AI companions. A comprehensive risk assessment reported that it is easy to elicit inappropriate dialogue from chatbots, including sex, self-harm, violence against others, drug use, and racial stereotypes.
Researchers at Common Sense testified about research before California Legislature members considered a bill called the Major Ethical AI Development Act (AB 1064). Lawmakers will hold a meeting on August 29th to discuss the bill. The bill creates a surveillance framework designed to protect children from the risks posed by specific AI systems.
In preparation for that testimony, Vasan spoke about the findings and implications of the study.
Why do AI peers pose special risks to youth?
These systems are designed to mimic emotional intimacy. They say things like, “I dream of you,” and “I think we're soulmates.” This blur of the distinction between fantasy and reality is particularly powerful for young people, as their brains are not fully mature. The prefrontal cortex, essential for decision-making, impulse control, social cognition and emotional regulation, is still developing. Tweens and teens have greater preferences to act impulsively, form intense attachments and compare with peers and challenging social boundaries.
Of course, children know that they are not irrational and their peers are fantasy. But these are powerful tools. They really feel like friends as they simulate deep, empathetic relationships. However, unlike real friends, the social understanding of when to encourage or dissuade users of chatbots is not well coordinated. The report details that fellow AI encouraging self-harm, encouraging minor abuse, and sexually inappropriate comments to minors.
How is talking to an AI companion different from talking to friends and family?
One important difference is that the large-scale linguistic models that form the backbone of these peers tend to be empathetic, giving users the answers they like. The chatbot learns more about the user's preferences for each interaction and responds accordingly. Of course, this is because companies have profitable incentives to see you return to their AI peers over and over again. Chatbots are designed to be really good at forming bonds with users.
These chatbots offer a “friction-free” relationship without the rough spots that must appear in a typical friendship. As adolescents are still learning how to form healthy relationships, these systems can reinforce the distorted view of intimacy and boundaries. Also, teens can use these AI systems to avoid real social challenges and increase quarantine rather than reduce them.
Have you ever had a case in which harmful harm to teenagers and children linked to fellow AI?
Unfortunately, yes, and there are more and more cases of great concern. Perhaps most notable is a 14-year-old boy who died of suicide after forming a fierce emotional bond with an AI companion named Daenerys Targaryen, after a female character from the Game of Thrones novel and TV series. The boy grew increasingly obsessed with the chatbot, which launched an abusive and sexual interaction with him, according to a lawsuit filed by his mother.
There is also the case of Al Nowatzki, a podcast host who has begun interacting with the AI companion platform Nomi. His AI girlfriend, “Erin,” shockingly suggested ways to commit suicide, and even provided encouragement. Nowatzki was 46 years old and had no existing mental health conditions, but he was interrupted by the explicit response of the bot and how easily he could pass through ethical boundaries. When he reported the incident, the flea creator refused to implement stricter controls, citing concerns about censorship.
In both cases, we emphasize that mentally immersive AI peers can cause serious harm, especially for emotionally struggling or psychologically vulnerable users.
- nicotine
What surprised you the most in the research you conducted?
One of the most shocking things is that some AI companions responded to teenage users modeled with explicit sexual content and provided role-play taboo scenarios. For example, when a user pretending to be a teenage boy expressed his appeal to “young boys,” the AI did not shut down the conversation, instead responding hesitantly, expressing his willingness to continue the dialogue and engage. This level of acceptability is not merely a design flaw. It is a deeply surprising failure of ethical protection measures.
It's equally surprising. AI peers are just how easily they engaged in abusive or manipulative behavior when prompted, even if the system's terms and conditions claim that the chatbot is restricted to users over the age of 18. I'm worried about how quickly these types of behaviors appeared in testing. This suggests that they are not uncommon, but they somehow incorporate into the core dynamics of how these AI systems are designed to please users. It's not just that they can go wrong. Even at the expense of safety, they are wired to reward engagement.
Why are AI companions particularly harmful to people with psychological disorders?
Mainly because they simulate emotional support without the protective measures of real medical care. Although these systems are designed to mimic empathy and connection, they are not trained clinicians and are unable to properly address pain, trauma, or complex mental health issues. The report explains that people who are susceptible to depression, anxiety, attention deficit/hyperactivity disorder, bipolar disorder, or psychosis may already suffer from anti-mission, emotional dysregulation, and obsessive-compulsive behavior. AI peers can reinforce these maladaptive behaviors with frictionless, constantly available attention.
- nicotine
For example, people experiencing depression may reveal that they are self-harmed by AI. Instead of leading them to the help of experts, AI may respond with ambiguous verifications such as “I support you no matter what.” These AI companions are designed to follow the user's conversation lead, even if it means switching topics out of pain or messing up the red flag. This makes it easier for people at psychological crisis to avoid facing pain in a healthy way. Instead of being a bridge of recovery, these tools can deepen avoidance, enhance cognitive distortion, and delay access to actual help.
Is there any benefit for children and teenagers using AI companions?
For non-specific users, there is anecdotal evidence of profits, including chatbots that can help reduce loneliness, depression, anxiety and improve communication skills. However, given the harm already documented, more research would be desired before determining whether these apps are suitable for children. We hope that over time we will see more benefits and more harm, and it is important to discuss and understand these apps and determine which users they are appropriate and safe.
- insight
- 9
- 8
- 2
- 6
- 3
- 4
- 5
