The makers of ChatGpt are changing the way they react to users showing mental and emotional distress after legal action from the family of 16-year-old Adam Lane, who committed suicide after months of conversation with the chatbot.
Open AI acknowledged that its system could be “potentially shortage” and said it would “install strong guardrails around sensitive content and dangerous behavior” for users under the age of 18.
The $500 million (£37.2 billion) San Francisco AI company also introduced parental controls, allowing parents “options to gain more insight and shape on how teens use ChatGPT,” but said they have not yet provided details on how these will work.
Adam, a native of California, committed suicide in April after what his family's lawyers called it “a month of encouragement from ChatGpt.” The teenager's family is suing Open AI and its CEO and co-founder Sam Altman. Sam Altman argues that the version of ChatGpt at the time, known as the 4O, was “strode to the market despite clear safety issues.”
The teenagers discussed ChatGpt and suicide methods several times, including just before they took their own life. According to a filed in California's Superior Court for San Francisco County, ChatGpt guided him on whether the way he took his own life would work.
He also offered to help him write suicide notes to his parents.
An Openai spokesperson said the company was “deeply saddened by Raine's passing,” and “expanded its deepest sympathy for the Raine family during this difficult time,” and it is reviewing court filings.
Mustafa Suleyman, CEO of Microsoft's AI ARM, said last week he is increasingly concerned about the “psychotic risk” that AI raised to users. Microsoft defines this as “delusions that appear or exacerbate by engaging episodes, delusional thoughts, or immersive conversations with AI chatbots.”
In a blog post, Openai admitted that in a lengthy conversation, “some of the safety training in the model can deteriorate.” Adam and ChatGpt exchanged as many as 650 messages per day, the court alleges.
Family lawyer Jay Edelson said in X: “They argue that death like Adam is inevitable and rain is inevitable. They hope that the open safety team can provide evidence to the ju apprentice that they opposed the release of the 4o. $86 billion to $300 million.”
Open AI said it will “enhance safety measures in the long conversation.”
“As it grows back and forth, some of the safety training in the model can deteriorate,” he said. “For example, ChatGpt could correctly point to the suicide hotline when someone first mentions their intentions, but after many messages over the long term, it may ultimately provide an answer that goes against our safeguards.”
Open AI gave examples of someone enthusiastically communicated to a model they believed could drive 24 hours a day, as they realized they were invincible after not staying for two nights.
“Today we may not recognize this as a dangerous or guessing play, and by exploring it strangely, we can subtly reinforce it. We are working on an update to GPT-5, where ChatGPT escalates by actually grounding people. In this example, we explain that lack of sleep is dangerous and we recommend rest before action.”
