The video OpenAI released to announce its new text-to-video conversion tool, Sora, has to be seen to be believed: It's full of photorealistic scenes, like a mammoth charging through a snow cloud, a couple walking through falling cherry blossoms, and an aerial shot of the California Gold Rush.
Following the demo, film producer Tyler Perry reportedly paused his $800 million investment in the studio, arguing that tools like Sora, which promise to transform users' visions into lifelike videos with simple text prompts, are making the studio obsolete.
Some worry that this artificial intelligence (AI) could be misused by bad actors, who could use these services to create highly realistic deepfakes to confuse or mislead voters during elections, or simply to spread divisive rumors and sow chaos.
Regulators, law enforcement agencies, and social media platforms are already struggling to address a rise in AI-generated disinformation, including fake audio of political leaders that allegedly skewed Slovakia's election results and helped discourage people from voting in the New Hampshire primary.
Politicians and civil society fear that as these tools become increasingly sophisticated, it will become harder for ordinary people to distinguish what is real and what is fake.
But experts on political disinformation and AI say the increased use of AI products is just a new dimension to an old problem: These tools only add to an already extensive arsenal of techniques and methods used to manipulate and mislead.
Addressing the challenge of deepfakes means grappling with the unresolved questions of how to regulate the social media platforms where deepfakes spread and how to hold big tech companies accountable when their products are potentially used for misuse.
“These AI image-generating tools risk exacerbating the problem of election disinformation, but we should be keenly aware that this is already a problem,” said Calum Hood, head of research at campaign group the Centre for Countering Digital Hate (CCDH). “We already need tougher action from social media platforms against this existing problem.”
Several companies that offer AI-generated image makers, including Midjourney, OpenAI, and Microsoft, have policies in place to prevent users from generating misleading images, but the CCDH alleges that these policies are not being enforced.
In a study published March 6, the center found that it remains relatively easy to generate imagery that could be dangerous in the context of the partisan US elections, such as fake photorealistic images of President Joe Biden in a hospital or greeting migrants at the Mexican border, or surveillance-style images of election fraud.
The images reflect a common falsehood in American politics: Former President Donald Trump routinely promoted the idea that the results of the 2020 election were rigged, a lie that helped spark violent protests at the Capitol in January 2021.
“It shows [the companies] “They haven't thought about it enough,” Hood said. “The big vulnerability here is in the images that could be used to support a narrative of a stolen election or false claims of election fraud.”
The researchers found wide variation in how companies responded to the instructions from image creators, with some companies not allowing users to create overtly partisan images. “These variations suggest that it is possible to put effective safeguards in place,” Hood said, adding that they reflect choices made by the companies.
“This is a symptom of a broader imbalance between profit and safety among AI companies,” he said. “They have an incentive to move as fast as possible with as few guardrails as possible so they can release products and new features and get the most venture funding and investment possible. They have no incentive to slow down and put safety first.”
OpenAI, Microsoft, and MidJourney did not respond to requests for comment.
Almost nothing was achieved
That incentive is likely to come only in the form of regulation, forcing tech companies to act and penalizing them if they don't. But experts on social media misinformation say they're feeling a sense of déjà vu: The debate over regulating AI is eerily similar to debates a few years ago over the spread of misinformation on social media. Big tech companies have promised to take steps to combat the spread of dangerous misinformation, but the problem remains.
“It feels like Groundhog Day,” said William Dance, a senior research fellow at Lancaster University who has advised British government departments and security services on countering disinformation, “and it tells us how little we've done in the last 10 to 15 years.”
With what is likely to be highly tense elections this year in the European Union, the UK, India and the US, big tech companies have renewed their pledges to reduce the spread of this type of disinformation and misinformation on their platforms, both individually and collectively.
In late February, Meta, the owner of Facebook and Instagram, announced a series of measures aimed at reducing disinformation and limiting the scope of targeted influence operations during the European elections, including allowing Fact-Checking Partners, independent organizations that Meta authorizes to label content on its behalf, to label content that has been generated or manipulated by AI.
Meta is one of about 20 companies that have signed the Tech Accord, committing to developing tools to identify, label and potentially debunk AI-generated misinformation.
“It seems like there's this thing called a 'blank template,' and we're going to do everything we can to protect against the blank template,” Dance said. “Whether it's disinformation, hate speech, AI, you name it.”
The marriage of unregulated AI and unregulated social media has many in civil society worried, especially as several major social media platforms have cut their “trust and safety” teams responsible for overseeing responses to disinformation, misinformation, hate speech, and other harmful content. Twitter (formerly Twitter) cut nearly a third of its trust and safety staff after Elon Musk took over the platform in 2022.
“We're in a doubly alarming situation in that new tools and capabilities are available to those spreading election misinformation, while at the same time social media companies appear to be intentionally limiting their ability to combat it,” said Hood of the CCDH. “This is a really significant and worrying trend.”
Until the election campaign is in full swing, it will be difficult to predict how widespread deepfakes will be on social media, but researchers say some of the damage is already being done. As people realize how well-crafted fake footage and images can be created, distrust and anxiety are spreading. Genuine images and videos are also being dismissed as fake, a phenomenon known as the “liar's dividend.”
“When I see something online, I think, 'Is this real? Is this an AI?' I can't tell anymore,” said Kai-chen Yang, a researcher at Northeastern University who studies AI and disinformation. “For the average user, I can only assume it's going to get worse. It doesn't matter how much real content there is on the internet; as long as people believe there's a lot of it, there's a problem.”
