AI-generated videos are sold using fake characters with Down syndrome

A video in which people with Down syndrome appear to be pleading with users to buy their product has racked up millions of views online. But many of them are fake, generated by artificial intelligence, and advocates and researchers say they are a growing trend that perpetuates negative stereotypes and harms real people with genetic conditions.

The clips, shared on platforms such as TikTok, YouTube and Instagram, show people with Down syndrome and their families claiming they are being bullied for selling crafts.

Others depict people trying to sell their wares at farmers’ markets or on the roadside, facing harassment from passersby who throw insults and food at them.

The video includes backgrounds of products sold at the linked store, including a resin lamp, a crocheted handbag, and a “berry bowl” designed for washing fruit. AFP also identified a number of similar videos written in Spanish.

However, the clip is a composite and the characters appearing in it are not real. These accounts link to suspicious online stores, some of which promote multiple “5-star reviews” repeating the same filler text.

Nathan Rowe, program director at Down Syndrome International, told AFP that the video perpetuates the stereotype that people with Down syndrome should be pitied (archived here).

“They’re preying on people who are probably a little more sympathetic and have a slightly more benign view of Down syndrome,” he said on June 15.

Fake videos, suspicious storefronts

AFP analyzed the clips using the AI ​​detection tool Hive Moderation and found that depictions of people with Down syndrome were likely to contain AI-generated content.

Some videos reuse similar backgrounds, such as two clips depicting different people surrounded by identical flags and shelves.

On the other hand, some of the content introducing products for sale appears to be sourced from genuine creators and authorized stores.

Reverse image search Indicated that a specific cut is includedHis clip was taken from the real thing Ceramics content creator sale the same berry bowl (Archived here and here).

Some of the linked stores appear to have scraped content from other websites. For example, the crochet bag designs featured in several videos are also sold on Shein, a popular resource for dropshippers (archives here). Other clips were selling resin lamps filled with preserved flowers and figurines from popular media similar to similar items seen in Etsy shops (archived here and here).

Lowe said these videos “crowd out” posts from actual Down syndrome entrepreneurs and may be siphoning off business.

“There are a lot of really talented people out there who are making things with people with Down syndrome, but it kind of reinforces the narrative that people with Down syndrome can’t do it, it has to be AI.”

continuing trend

This is not the first time we have seen a trend of exploiting the identities of marginalized people for online attention and profit.

Down Syndrome International previously complained to Meta about sexual deepfakes of people with Down syndrome, and many videos were removed, but Lowe said social media companies should be more proactive in preventing the spread of this type of content.

AFP contacted Meta about trends in advertising new products, but did not receive a response. The company’s policy is that it does not allow fraudulent identity fraud or spam on its platform (archived here).

TikTok pointed to its community guidelines, which say the platform bans accounts that post deceptive or manipulative activity or engage in discrimination (archived here). YouTube has a similar policy regarding misleading spam (archived here).

Many of the videos examined by AFP are no longer public, but other accounts are still sharing AI-generated clips and redirecting users to the product.

“The people behind the scenes are probably motivated by profit and don’t consider the harm they do in the process,” Lowe said.

Jeremy Carrasco, co-founder of AI research firm Riddance, said the number of accounts pushing this type of content shows the video is functioning as a for-profit scheme (archived here).

“There are a lot of failures across the system that are compounding this,” he said, noting that while videos are very easy to create, they are difficult to track.

“That’s why they’ve exploded so much.”

He identified numerous AI-generated videos featuring people with Down syndrome, and said the same account was attempting to sell the same products using synthetic elderly characters (archived here).

AFP previously investigated the trend of multilingual videos stealing the identities of elderly people in order to lure users into buying slippers or dog collars sympathetically.

“I feel like we’re reaching the bottom of the tolerance range. If they keep going further, something is going to happen,” Carrasco said.

Read more about AFP’s coverage of misinformation and artificial intelligence here.