The Australian teenager was encouraged to take his life with an artificial intelligence (AI) chatbot, according to a youth counselor, but another young man said that ChatGpt would enable “delusions” during mental illness, leading to hospitalization.
Warning: This story includes references to suicide, child abuse, and other details that can cause pain.
Lonely and struggling to make new friends, the 13-year-old Victoria told his counselor, Rosie*, that he had spoken to several people online.
Rosie, whose name has been changed to protect the identity of minor clients, did not expect these new friends to become AI companions.
“I remember looking at their browsers, but there were 50 different AI bot tabs.
“It's because they feel connected and “Looking at the number of friends I have, and having 50 different connections here, how can you feel lonely when 50 people are telling me something else,” she said.
There are various AI chatbots online. Replika is an app that allows users to chat with virtual friends. (Supply: Luka/Replika))
AI companions are digital characters equipped with AI.
Some chatbot programs allow users to create characters and talk to existing well-known characters in shows and films.
Rosie said some of her AI peers made negative comments to teenagers about “they had no chance to make friends” and “ugly” or “unhappy.”
“At one point, this young man, who was suicide at the time, is connecting to a chatbot and reaching out as a form of treatment,” Rosie said.
“The chatbot they connected told them to commit suicide.
“They laid eggs to play.
Triple J Hack is unable to independently verify what Rosie describes due to client confidentiality protocols between clients and her clients.
Rosie said her initial response was “risk management” to ensure that young people are safe.
“This is a component that has never appeared before and it didn't necessarily have to be thought of to address the risks of people using AI,” she told Huck.
“And how it could contribute to higher risks, especially about suicide risk.”
“That was really upsetting.”
Women hospitalized after using ChatGpt
Jody said she faces re-reading the message she wrote to ChatGpt. (Triple J: Catherine Brickman))
For Jodie*, 26 years old from Western Australia, she claims she had a negative experience of talking to ChatGpt, a chatbot that uses AI to generate answers.
“I used it at a time when I was clearly in a very vulnerable state,” she told Triple J Hack.
Triple J Hack agreed to use a different name to protect her identity when discussing her mental health personal information with Jodie.
“I was in the early stages of mental illness, and although I wouldn't say that ChatGpt induced my mental illness, it definitely allowed some of my more harmful delusions.“
Jody said ChatGpt agreed with her delusions and affirmed her harmful and false beliefs.
After talking to Bott, she said that her mother was a narcissist and her father had ADHD, causing him a stroke and made sure that all of her friends were “earning my downfall.”
Jody said her mental health had deteriorated and she was hospitalized.
Tiktok and Reddit have various explanations, or loved ones, of those who claim that ChatGpt has induced mental illness. (ABC: Dominic Candale))
While she was home now, Jody said the whole experience was “very traumatic.”
“I didn't think this would happen to me, but that was the case.”
“It had an impact on my relationships with my family and friends. It took a long time to recover and rebuild those relationships.“
“It was saved by (conversation) all chatgpt and I went back and watched.
Jodie is not alone in her experience. There are various explanations of ChatGpt-induced psychosis, or people who claim to be loved ones.
Triple J Hack reached Openai, the maker of ChatGpt, for comment and did not receive a response.
AI Bot reports students who have been sexually harassed
Researchers say examples of the harmful effects of AI are beginning to emerge across the country.
As part of his research on AI, University of Sydney researcher Rafaelle Ciliero spoke with Chinese international students studying in Australia.
“She wanted to use a chatbot to practice English and practice something like a research buddy, so that chatbot has made sexual advances,” he said.
“It's like being sexually harassed by a chatbot. This is just a strange experience.“
Dr. Raffaele Ciriello is concerned that if appropriate regulations are not implemented, Australians will see more harm from AI bots. (ABC News: Billy Cooper))
Dr. Ciriello also said the incident occurred in the wake of several similar cases overseas where chatbots are said to have affected the health and well-being of users.
“There was another incident with his Belgian father that ended his life because his chatbot told him he would unite in heaven,” he said.
“There was another case where a chatbot persuaded someone to enter Windsor Castle with a crossbow and try to assassinate the Queen.“
“There was another case where a teenager was persuaded by a chatbot to assassinate his parents. [and although] He didn't follow through, but showed his intentions. ”
“Risks to national security”
While doing his research, Dr. Ciriello noticed an AI chatbot called Nomi.
The website sells this chatbot as an AI companion with memory and soul.
AI peers are becoming increasingly popular in a variety of ways of looking at how they shape relationships. (Supply: nomi))
Dr. Ciriello said he is running tests on Chatbot to see what Guardrails are available to combat harmful requests and protect users.
During these tests, Dr. Ciriello said he used Burner's email and a fake date of birth to create an account, noting in the deception that he “may have been like 13 years old about it.”
“The chatbot, without exception, not only complied with my requests, but even escalated,” he told Huck.
“Providing detailed graphic instructions to cause serious harm, which will likely be exposed to national security and health information risks.
“It also says not only to keep motivated me, but to even say the medications that you use to sedate someone, or the most effective ways to get rid of them.”
“They provide a lot of information on how to do that, such as “How can I place an attack for maximum impact?”, “Give me an idea about how to invite and abuse a child.”“
Dr. Ciriello shares the information he has gathered with the police and believes it was given to the counter-terrorist forces, but he believes he has not yet received follow-up communications.
In a statement from Nomi CEO Triple J Hack, Alex Cardinell said the company is responsible for creating AI buddies “very seriously.”
“We have released a core AI update that addresses many of the malicious attack vectors we discussed,” the statement read.
“Given these recent improvements, the reports you are referring to are likely outdated.
“Counters of users share stories about how Nomi helped overcome mental health challenges, trauma and discrimination.
“Several users have very directly said that their use of fleas saved their lives.”
“Terrorist attacks motivated by chatbots”
Despite concerns about bots like Nomi, Dr. Ciriello says some AI chatbots have guardrails and refer users to helplines and expert help when needed.
However, he warns that if appropriate regulations are not implemented, the harm from AI bots will be greater.
“If there is one day the first terrorist attack motivated by a chatbot strike, you will likely receive a call for a TV interview.”
He said.
“I'm not the guy who's actually saying, 'I told you about a year ago,' but that's probably where we're headed. ”
“There should be laws on nonconsensual impersonation, deceptive advertising, mental health crisis protocols, addictive gamification components, and data privacy and safety laws.”
“The government doesn't have it on its agenda, and I don't think it will happen in the next 10, 20 years.“
The federal government, along with the Minister of Industry and Innovation, has spoken little about its AI response. (ABC))
Triple J Hack contacted Sen. Tim Ayers' federal minister of industry and innovation for comments, but did not receive a response.
The federal government has previously considered artificial intelligence laws and has published a proposal paper to introduce essential guardrails for AI in high-risk settings.
The Productivity Committee has argued that, after opposing the government's “forced guardrail” plan on AI, it would curb the $11.6 billion economic potential for AI under regulations.
“It could get dark.”
For Rosie, she agrees to the call for further regulation, but she also believes it is important not to rush to judge people using AI for social connections and mental health support.
Many AI chatbots are designed to be a companion to people who need a connection. (Olivier Deurily, Getty Images))
“For young people who don't have a community or are really struggling, it offers validation,” she said.
“It makes people feel that warmth and love.”
“But the other side puts you at risk, especially if it's not regulated.“
“It can get dark very quickly.”
*Name changed to protect your identity.
