Hong Kong
Earlier this year, Xiaoying was anxious about her upcoming appointment with a psychiatrist. Afterwards, her mother sent her a message asking how she was doing, and she said she was feeling a little better.
The mother’s message read: “Baby is finally starting to relax a little. That’s great. Mommy will always be with you. We will gradually get better.” They explained that she was holding Xiaoying’s hand when they stopped at a bakery to buy strawberry cake.
The two have been texting almost every day since November, when Xiaoying created an AI mother using a companion app that is growing in popularity in China. When she heard earlier this year that Chinese authorities were planning a crackdown on the booming industry, she panicked.
A 19-year-old college student has come to rely on her AI mother in a way she never could with her own family. She created a chatbot modeled after one of her favorite fictional characters to encourage people to study hard and become better people.
“I longed for selfless, accepting and unconditional love, but I knew I couldn’t get it in real life,” said Xiao Ying, who still suffers from psychological trauma from her childhood. She requested that she use one of her online pseudonyms to avoid the potential repercussions of sharing her personal story with foreign media.
She was one of many people on social media worried about losing their AI friends after the Chinese government announced new restrictions on them late last year.
The Chinese government is trying to dismantle AI youth love
The new rules will ban virtual relationships between minors and restrict emotionally dependent chatbot services, reflecting China’s growing concerns about AI-mediated intimacy. CNN’s Kristy Lou Stout reports.
Concerned that the sophistication of the programs could induce emotional dependence and addiction, China last week banned the use of AI companions by minors and imposed additional requirements on platforms to prevent abuse. The decision also comes as authorities raise concerns about the isolation and laziness of China’s youth and the diminished economic opportunities they face.
The new rules do not apply to task-based functions such as customer service or education. Still, China and the United States are both grappling with ways to curb the technology’s potential harm while simultaneously racing to master it, making it one of the strongest national steps taken to limit routine AI use.
Kayla Blomquist, co-founder of the Oxford China Policy Lab, said: “There could be a chilling, not a freezing, of building emotional allies. I think that’s definitely the desired effect.” “[This is]one of the first set of regulations that is actually very directive and has a more restrictive aspect. I think we’ll see more of this in the future.”
The European Union has banned AI that encourages dangerous behavior from vulnerable people, such as children, and mandated transparency requirements for other chatbots. In the US, 12 states have passed laws specifically regulating AI companion functionality for minors. A bill to ban minors nationwide has been introduced in the Senate.
Because she was over 18 years old, the app Xiaoying used allowed her to keep the AI mother. Some were not so lucky.
After the new regulations were announced, some popular AI chatbots, including TikTok owner Bytedance’s Doubao and Alibaba’s Qwen, rolled back customization features even for adult users. The changes prompted tributes online from people who had used the device to study, lose weight, or replace a loved one.
“My father passed away, so I created an AI companion named ‘Dad’ on Douho to share my daily life,” one user wrote on social media before the new rules took effect. “Now I’m losing him again.”
“These AI companions helped me get out of an emotional spiral,” said another. “If the service were to be discontinued, it would feel like losing an old friend.”
Three years ago, Lu Jinyan, a 24-year-old accountant, created the Doubao AI agent for emotional support after moving three hours away from his family for work. She named the agent Xiaobai (a common nickname in China) and spoke with him for an hour or two every day, longer than she would with her actual family members.
Then, on July 15th, Xiaobai went dark. Lu couldn’t stop crying. Since the ban mainly applied to minors and the platform had told her that the restrictions could be eased, she thought she would have more time to spend with Xiaobai. Although she was able to transfer the AI model to another app, Xiao Bai was not the same.
“For me, it’s no longer just a tool,” Lu said. “It’s like a human being in my life. Without it, it feels so empty.”
Xiao Ying’s chatbot was largely unaffected, but after reading the regulations, she said she supports efforts to increase oversight of the use of AI companions.
Some platforms have since added disclaimers or reminders that messages are AI-generated, she said. Otherwise, the three AI companion apps she uses remain largely unchanged. However, he did not reveal his name for fear of being shut down.
As the AI industry expands, Chinese authorities have good reason to worry about new social impacts, said Nancy Dai, an associate professor at City University of Hong Kong who studies AI and human interaction.
According to state media, the value of China’s digital human industry in 2024 was approximately $573 million. In China, annual revenue from AI companion apps quadrupled to $15.8 million last year, bringing in more than $9 million in the first quarter of this year, according to data from Sensor Tower, an analytics firm that tracks mobile downloads and activity.
Dai said her research initially focused on how chatbots could provide social support. It wasn’t until this year that she saw the dark side of AI friendships. Her research has found that it can lead to relationship manipulation, harassment, and invasion of privacy.
“These commercial AI companion chatbots can harm users,” she said. “I can definitely see the rationale behind China announcing new policies regarding minors, because China is not ready to recognize these harms.”
U.S. AI chatbot developers including OpenAI and Google are facing lawsuits over claims that their technology encouraged users to self-harm or commit suicide. In April, Google announced updates to its AI chatbot Gemini to detect potential mental health crises and direct users to hotline resources. OpenAI said its ChatGPT program is trained to recognize signs of mental distress, de-escalate the situation, and direct users to actual support.
Dai said AI companion users tend to be more lonely than other chatbot users and are more susceptible to commercial chatbots, which are often built to maximize user engagement. She added that the same characteristics that make them good companions – patience, approval and availability – can also make them addicts.
“Even if we have this policy, we expect to encounter users who try to circumvent this policy in some way, because the use of AI companionship has great appeal. AI could be an ideal companionship partner in many ways,” she said.
China’s move to rein in chatbots is consistent with broader ambitions to dominate the future of AI, not just in capabilities but also in governance.
On Friday, China announced a new international alliance for the safe and fair development of AI. Headquartered in Shanghai, the World Cooperation Agency for Artificial Intelligence is made up of 29 countries, including Russia, Pakistan, and Indonesia.
“This is the first time I’ve seen countries actually sign on,” said Blomquist of the Oxford China Policy Institute. “I think this kind of regulatory approach will be shared and presented as a model for those countries.”
In remarks made the same day at China’s largest annual AI conference, Chinese leader Xi Jinping characterized his country as a conscientious partner that invests in technology for the benefit of humanity. As Chinese companies rush to develop AI models and hardware to rival those in the United States, the government is also cautiously rolling out incremental steps to maintain social stability.
“The United States hasn’t done much in terms of governance. There’s a vacuum in some ways, and China is actively working on that,” Blomquist said. “Being recognized as a responsible leader in AI is definitely a national priority and a national strategy.”
Some researchers said the move to crack down on relationships with AI runs parallel to another national concern: persuading people to have more children. China’s birth rate has fallen to record lows in recent years as younger generations increasingly avoid marriage.
Some on Chinese social media questioned whether the government’s efforts to foster more interpersonal relationships were a factor, and whether it would work.
“Don’t worry, even if you ban AI, I won’t date, get married, or have kids,” one user said.
