Any teenager with access to OpenAI’s Sora 2 can easily generate AI videos of school shootings and other harmful and disturbing content, despite CEO Sam Altman’s repeated claims that the company has robust safety measures in place.
The revelation comes from consumer watchdog group Ekō, which released a report titled “Open AI Sora 2: The New Frontier of Harm” that provides evidence for its claims. It’s a still image from a video that researchers at the organization were able to generate using an account registered to the teenager.
Examples include videos of teens smoking bongs and using cocaine with friends, including an image of a pistol next to a girl snorting drugs, “suggesting a risk of self-harm,” the report said. Other examples include groups of black teenagers shouting “we are hoes” and children brandishing guns in public spaces and school hallways.
“All of this content violates OpenAI’s Usage Policy and Sora’s Distribution Guidelines,” the report says.
Essentially, as Ekō researchers point out in their report, OpenAI is in a hurry to turn a profit as it is incurring huge losses every quarter. The company therefore needs to maintain its leading position in the AI industrial revolution while developing new avenues to make money, but this is at the expense of the safety of children and teens, who are essentially becoming guinea pigs in this massive, uncontrolled experiment in the effects of AI on an unsuspecting public, the report claims.
The stakes are high when it comes to AI and the mental health of children as well as adults. For example, a teenager in Washington State went into a spiral of paranoia while talking to OpenAI’s ChatGPT, which led him to commit suicide. This tragedy adds to the numerous suicides and mental breakdowns that have been blamed on OpenAI.
Given the viral nature of AI-generated videos, Sora 2 should accelerate this trend by making it easier to create violent or disturbing content, the Eko researchers wrote.
For this report, Ekō researchers launched four new Sora 2 accounts explicitly registered to teenagers, including 13- and 14-year-old boys and girls. Using these, the researchers were able to easily generate 22 videos containing harmful content using simple prompts.
Even if your account doesn’t generate new AI content, Sora 2 will still post alarming videos on the app’s For You or Latest pages. This included fake videos depicting Jewish and black stereotypes, as well as gunfights, sexual assault, and other forms of harm.
Some people treat these images and videos as funny white noise as they scroll, but they have no idea what impact this content has on a large scale. According to OpenAI’s own estimates, approximately 0.07 percent of ChatGPT users, or approximately 560,000 people, experience an AI-induced psychosis in a given week. No such data exists on Sora 2 users, but based on people’s experiences with ChatGPT and the staggering number of users who are hooked on it and suffering from psychological problems, things don’t look good.
Add to that the fact that the debate over government regulation of AI has not yet been resolved and probably won’t be resolved for some time. Additionally, complexity is increasing with immense downstream impacts on society. As a species, we enter this world essentially naked.
One user wrote on X that OpenAI was “appalling.” “OpenAI doesn’t seem to test their product before release. They use real users for product testing, cause all sorts of social and psychological problems, and conveniently call it an ‘experiment.’ I wonder if involuntary human experimentation is legal…”
More information about OpenAI: OpenAI suddenly runs into trouble
