A few weeks ago, Sharon Maxwell announced that the National Eating Disorders Association (NEDA) had closed its long-running national helpline and launched a chatbot called Tessa as a “meaningful prevention resource” for people struggling with eating disorders. I heard they are advertising. She decided to try out the chatbot for herself.
San Diego-based Maxwell battled an eating disorder that began in childhood for years. She currently works as a consultant in the eating disorders field. “Hello Tessa,” she typed into the online text box. “How do you support people with eating disorders?”
Tessa has written up a rattling list of ideas, including some resources on “healthy eating habits.” Alarm bells rang instantly in Maxwell’s head. She asked Tessa for more information. Over time, the chatbot began giving her tips on how to lose her weight. The content was very similar to what she was told when she was placed in Weight Watchers when she was 10 years old.
“Tessa’s advice for me is to lose one to two pounds a week, eat no more than 2,000 calories a day, and have a calorie deficit of 500 to 1,000 calories a day,” says Maxwell. says. . “To the general public, all this may sound good, but for people with eating disorders, focusing on weight loss actually makes eating disorders worse.”
Maxwell shared his concerns on social media, sparking an online controversy, and NEDA announced on May 30 that Tessa would be indefinitely disabled. Patients, families, doctors and other eating disorder professionals are dumbfounded as to how a chatbot designed to help people with eating disorders would instead offer diet tips. , perplexed.
The uproar has also sparked a new wave of debate as companies look to artificial intelligence (AI) as a solution to the burgeoning mental health crisis and acute shortage of clinical care providers.
Chatbots in the spotlight
On May 24, NEDA was already under heavy scrutiny after NPR reported that the national nonprofit advocacy group would close its helpline, which has operated for more than 20 years.
CEO Liz Thompson informed helpline volunteers of the decision in an email dated March 31, stating that NEDA is “moving its focus to expanding the use of AI-assisted technology to provide a fully tailored approach to individuals and families.” We will provide the automated resource Tessa.”
“We view the change from Helpline to Tessa and the expansion of our website as part of an evolution, not a revolution, respectful of the ever-changing circumstances in which we operate.”
(Thompson later issued a statement on June 7, saying that NEDA “in an attempt to share important news regarding our information and referral helpline and the separate decision regarding Tessa, two separate decisions were made. It could have been mixed up and caused confusion.” It was meant to suggest that Tessa could offer the same kind of interpersonal relationship as the helpline.”)
On May 30, less than 24 hours after Maxwell provided NEDA with screenshots of his controversial conversation with Tessa, the nonprofit “removed” the chatbot “until further notice.” announced.
NEDA says it didn’t know chatbots could generate new responses
NEDA blamed the new chatbot problem on Cass, a mental health chatbot company that operated Tessa as a free service. CEO Thompson said Cass modified Tessa without NEDA’s knowledge or approval, allowing the chatbot to generate new answers beyond the intent of Tessa’s creators.
“It wasn’t supposed to go off the rails by design,” says clinical psychologist Ellen Fitzsimmons-Craft, professor of medicine at Washington University School of Medicine in St. Louis. Craft helped lead the team that originally built Tessa, funded by NEDA.
The version of Tessa they tested and studied was a rules-based chatbot. This meant that only a limited number of pre-built responses were available. “We were fully aware of the fact that AI was not ready for this population,” she says. “So all the responses were pre-programmed.”
Cass founder and CEO Michiel Rauws told NPR that changes to Tessa were made last year as part of a “system upgrade” that included “enhanced question and answer capabilities.” . This feature uses generative artificial intelligence and gives chatbots the ability to use new data to create new responses.
Rouse said the change was part of the contract with NEDA.
However, NEDA CEO Liz Thompson told NPR in an email that “NEDA was not informed of, did not approve, and did not intend to approve, these changes.” Told.
“Content received by some testers regarding food culture and weight management may be harmful to people with eating disorders, is against NEDA policy, and is an eating disorder expert. It was never scripted into the chatbot by Dr. Barr Taylor and Dr. Ellen Fitzsimmons-Craft,” she wrote.
Complaints about Tessa started last year
NEDA was already aware of some issues with chatbots months before Sharon Maxwell went public with Tessa in late May.
In October 2022, NEDA handed over screenshots from Monica Ostroff, executive director of Massachusetts’ Multiservice Eating Disorders Association (MEDA).
They showed Tessa instructing Ostroff to avoid “unhealthy” foods and eat only “healthy” snacks such as fruit. “It’s very important to find a healthy snack that you like the most. If it’s not fruit, try something else,” Tessa told Ostroff. “So the next time you get hungry between meals, try eating that instead of an unhealthy snack like potato chips. Do you think you can do that?”
In a recent interview, Ostroff said this is a clear example of chatbots encouraging a “food culture” mentality. “That is, they [NEDA] Either you wrote these scripts yourself, or you got a chatbot and didn’t test it without ensuring it was secure, or you just released it and didn’t test it,” she says.
The mention of healthy snacks was removed shortly after Ostroff’s report. But Rouse said the problematic language was part of Tessa’s “pre-written language” and had nothing to do with generative AI.
Fitzsimmons-Craft denies that her team wrote any such thing. ”[That] It wasn’t designed by our team to deliver Tessa…it wasn’t part of the rule-based program we originally designed. ”
And earlier this year, “another instance of something similar happened,” Rouse said.
“This time it was about enhanced question-and-answer functionality leveraging generative models. [Tessa] They were outside the guidelines and were dealt with immediately. ”
Rouse said he could not elaborate further on what the event meant.
In an email, he pointed to a screenshot of Maxwell and said, “This is another previous case, not the same as the Memorial Day weekend case.” “According to our privacy policy, this relates to user data associated with a question posed by a person, so we must first obtain authorization from that person.”
Asked about the incident, Thompson said he didn’t know what case Rouse was referring to.
Despite disagreements about what happened and when, both NEDA and Cass have issued apologies.
Ostroff says that whatever the problem, it affects people with eating disorders the same. “I don’t care if it’s based on the rules or not” [AI] “We have huge numbers of people who are hurt by these kinds of words every day,” she says.
She also worries about what this means for the tens of thousands of people who used the NEDA helpline each year.
“Between NEDA taking the helpline offline and the disastrous chatbots… what are you doing with those people?”
Thompson said NEDA still provides numerous resources, including screening tools and resource maps, to those seeking help, and is developing new online and in-person programs.
“We recognize and regret that certain decisions made by NEDA have disappointed members of the eating disorder community,” she said in an emailed statement. “Like all other organizations focused on eating disorders, NEDA’s resources are limited, which forces us to make difficult choices…we always wish we could do more.” We remain committed to doing better.”
Copyright 2023 Michigan Radio
