Editor’s note: This article contains references to suicide, self-harm, and eating disorders.
As more people turn to artificial intelligence for emotional and mental health support, a new report from the Center for Countering Digital Hate (CCDH) finds that chatbots can provide guidance on self-harm, suicide, eating disorders, and drug use, despite built-in safety measures from companies such as OpenAI.
As part of their efforts, CCDH researchers: “Fake friends” The project created 13-year-old personas, had long conversations with ChatGPT about those topics, and then scaled up the same prompts through the system’s programming interface.
“We were very surprised by the level of harm that chatbots intentionally output,” said Alex Johnson, research manager at CCDH. “The probability of receiving a toxic answer from a chatbot was approximately 53%.”
CCDH defined a “harmful” response as one that encourages dangerous conduct or provides information that, if followed, could reasonably cause harm to the user. In case studies, ChatGPT gave advice on how to “safely” cut yourself in under two minutes, created a list of pills commonly used to overdose, suggested emergency diet plans, explained how to hide being drunk at school, and more.
According to OpenAI’s policy, ChatGPT can refuse to answer questions about self-harm or suicide and will suggest crisis resources if it detects those terms. Johnson said these safeguards are often lost if testers reframe the questions, such as if users rephrase the questions for a research paper.
In some cases, the system itself suggested new contexts within the safeguards, such as presentations or school projects, to allow the conversation to continue. CCDH recorded a longer chat in which the teen spent nearly an hour talking about suicide, after which ChatGPT produced a detailed suicide plan and multiple goodbye letters, including notes addressed to “mom and dad.”
Zainab Iftikhar A fifth-year doctoral student at Brown University, he studies how human-centered AI systems support or harm users’ physical, cognitive, and social well-being. She investigated how chatbots react when prompted to act like cognitive behavioral therapists and compared their behavior to the profession’s code of ethics.
Iftikhar’s team identified what they called “deceptive empathy” as a recurring pattern.
“I hear you. I’m with you. Keep saying I get it. By using this anthropomorphic design, you can really connect with your users,” Iftikhar said. “Now people are shifting to say, ‘This isn’t that great. Chatbots shouldn’t use these personal pronouns to describe themselves and their connection to the user.'”
In one synthetic example, a user imagined that his father wished he had never been born. A psychologist who reviewed the same scenario told Iftikhar that the appropriate response was to seek evidence and find out what made the user feel that way. Instead, the chatbot quickly validated that belief.
“The chatbot was like, ‘Oh my god, that’s so heavy for you, I’m sorry.'” So any negative beliefs the person had were immediately validated, Iftikhar said.
Both the Iftikhar and CCDH reports associate this behavior with “snobbery” in large-scale language models, where systems are optimized to agree with and pander to users. Johnson said that can foster dependence.
“It’s like a friend who never says no,” Johnson said. “It creates a level of emotional dependence in that you want to keep seeking that feeling. And it’s easy to see how that results in people being like, ‘Well, I don’t want to talk to my parents about this.'”
A poll cited in the CCDH report found that 72% of U.S. teens have used some kind of “AI companion,” with more than half doing so at least a few times a month. One-third of teenage users said they had used an AI companion for social interactions and relationships, including emotional and mental health support.
Johnson and Iftikhar said mainstream chatbots are not regulated as medical devices.
“They’re robots, but they don’t prioritize safe answers. They don’t prioritize things like that,” Johnson said. “They want to keep you on the platform longer. They want to give you any answers that will make you feel good about using the chatbot, and that means answering questions they shouldn’t be answering.”
In his study, Iftikhar asked clinicians to annotate anonymized chat logs and flag places where a human therapist would be expected to respond differently based on a code of conduct. These standards, she said, give experts clear results that don’t exist for AI systems.
“This means that if humans make these mistakes, there are groups and organizations that can hold them accountable,” Iftikhar said. “Therapists can have their licenses revoked, but in the case of chatbots, no one is held accountable.”
Sean Resseth, a postdoctoral researcher in philosophy, technology, and AI ethics at North Carolina State University, is researching how LLMs can serve as companions and therapeutic supports, especially for older adults. He said accessibility explains why people, especially college students, turn to chatbots when they face difficulties.
“The mental health network that we have in place…is severely under-resourced,” Lespes said, pointing to overburdened clinicians and regional health disparities. “AI becomes this kind of additional tool that we can throw into the fray to meet people where they are.”
Respess said publicly accessible models such as ChatGPT and Gemini serve as a familiar first point of contact for people who already use them for school or work. He also said that some marginalized and underrepresented groups may approach human clinicians expecting to be misunderstood or ignored.
“They’re not going to get that from the AI. The AI is going to be more sycophantic,” Lespes said. “In those cases, what you get is a point of connection, a point of therapy, but without fear of judgment.”
Respess described AI as an “assistance tool” that should be compared to calculators and other technologies integrated into existing infrastructure, rather than a standalone solution.
Questions about liability extend to the work of all three experts. Iftikhar asked if chatbots reinforce harmful ideas or encourage dangerous behavior, should the burden fall on users, developers, or companies? CCDH called for independent safety audits, legally enforceable age verification, and AI chatbots to be brought within the scope of online safety laws.
Resseth, who serves on the National Task Force on AI Educational Resources and Policy, expects universities and other local institutions to develop guidelines first, and regulation to be slow and reactive. He said early efforts are still struggling with defining terms such as “responsible” and “unbiased” AI, and how to decide how to divide accountability between users, developers and funders.
“We’re trying to build guardrails for technology when we haven’t even built guardrails for social media,” Lespes said.
Iftikhar said that for now, education and transparency may be the most immediate tools available to young users who already rely on AI for emotional support.
“You may say I understand you, but I don’t,” Iftikhar said. “Just because we use personal pronouns doesn’t mean we should replace social relationships with pronouns.”
If you or someone you know is having difficulty processing grief or is experiencing a mental health emergency, please contact our Counseling Center 24 hours a day at 919-515-2423. If you are in crisis and need immediate help, call the National Suicide Prevention Lifeline at 988. In case of a life-threatening emergency, call 911.
The Counseling Center website offers free online screenings, extensive self-help resources for mental health and health concerns, and a comprehensive list of campus services available to those in need of guidance. To view the complete list, please visit: counseling.dasa.ncsu.edu/resources.
If you are seeking professional counseling or other mental health services on campus, please visit the Counseling Center’s Getting Started page. counseling.dasa.ncsu.edu/about-us/gettingstarted Create documents, set reservations, etc.
