Use AI to flag high-risk videos

The hugely popular social media platform TikTok has over 1 billion daily active users, 34 million videos are posted every day, and 63% of US teens ages 13-17 are active on the site.

While many of these short-form clips are harmless, some types of content and the platform’s recommendation algorithms contribute to poorer teens’ mental health and increased risk of self-harm and suicidal thoughts.

These platforms are also raising urgent questions about mental health and digital safety as they reshape the way young people consume information and connect with each other. Jiaheng Xie, assistant professor of management information systems at the Alfred Lerner College of Business and Economics at the University of Delaware, and his team conducted research on how algorithm-driven platforms like TikTok can inadvertently amplify harmful mental health content, and how artificial intelligence (AI) itself can be used to reduce those risks before tragedies occur.

Xie’s research project “Short-form videos and mental health: A knowledge-guided neurotopic model” information systems researchis intended to enhance automatic detection, the first step in content moderation. Rather than focusing solely on the video content itself, his model analyzes both the video and the comments left by viewers below it.

“What we’re really interested in is whether the viewer has suicidal thoughts,” Xie said.

If the majority of comments on a video reflect suicidal thoughts, that reaction is a strong signal that something in the video may be harmful.

Using this insight, Xie and his team collected both high- and low-risk videos from the platform and fed them into an AI pipeline designed to learn patterns associated with risky content. Over time, the model was able to predict whether a new video was likely to cause suicidal thoughts in viewers before it gained widespread attention.

Xie, who recently won the Association for Information Systems (AIS) Early Career Award and is one of only four scholars in the world honored this year, has been interested in mental health research since before TikTok gained cultural dominance.

“I’ve always been interested in depression and healthcare-related research,” he said, and his academic focus on depression began years before short-form video platforms became popular.

This long-standing interest took on new urgency in 2019 and 2020, when TikTok’s popularity skyrocketed, especially among teens. During this time, there were reports from news agencies such as the following: new york times and NBC News The platform has been increasingly linked to disturbing content and, in some cases, suicide among young people.

“That caught my attention,” Xie said. “It’s obviously a very serious issue and falls within the scope of my research.”

Rather than viewing TikTok simply as a social media app, Xie approached it as a complex sociotechnical system where algorithmic design, human behavior, and mental health outcomes intersect.

TikTok, like other major social platforms, already has an extensive content moderation strategy in place. According to Xie, the company relies on a two-step process. First, an AI system scans uploaded videos and flags potentially relevant content, then tens of thousands of human moderators review the flagged content.

But despite these efforts, critics continue to say that harmful content continues to slip through the cracks. Xie saw an opportunity to make a contribution here, arguing that his field was uniquely positioned to understand not only the technology but also the organizational and social context in which it operates.

“We know the business context. We know not only the problem space but also what’s going on with that technology,” he said.

One of the most annoying aspects of short-form platforms is the recommendation system. Xie pointed out that when users access related content, the algorithm can push similar videos to their feed. For vulnerable teens, this can create a dangerous feedback loop.

“If you’re a teenager and you’re depressed, you’re going to keep getting these videos,” he says. And while some people are supportive, others reinforce harmful thoughts.

Even more alarming, Xie noted that some videos remain accessible despite clearly violating community guidelines, including content that includes step-by-step suicide instructions and references to deadly substances.

These realities highlight the need for faster and more accurate detection methods that scale with the amount of content generated every day.

A key innovation in Xie’s approach is explainability. Rather than simply flagging videos as “high risk,” his model leverages medical literature and established mental health frameworks known as medical ontology to identify specific factors associated with suicidal ideation.

This means moderators receive not only warnings, but also context about what topics, themes, or risk factors contributed to the model’s rating. More information means faster, more informed decisions can be made, he explained.

Such guidance is critical, especially when human reviewers need to watch the entire video to assess risk, an inherently time-consuming and mentally taxing process.

Xie is careful to emphasize that AI moderation is not a one-time fix. The model must evolve with the platform, user behavior, and emerging content trends.

“This should be an ongoing process,” he said, noting continued concerns about the lack of accuracy in existing systems.

Although the current research is academic, Xie hopes to collaborate with platforms such as TikTok in the future. Ultimately, his goal is not to discourage the use of social media, but to mitigate its most dangerous consequences.

“We’re not saying we don’t want teens to use social media,” he says. “We’re just trying to prevent negative outcomes.”

For Xie, that motivation is deeply personal and deeply human.

“These issues are directly related to people’s lives,” he said. By advancing AI tools that can operate on a global scale, his research aims to ensure that technology protects against mental health risks in the digital age, rather than silently amplifying them.