One of the great concerns of our time is that generative AI systems will give bad actors unprecedented power to lie, manipulate, and steal on a previously unimaginable scale, undermining our systems of trust, democracy, and society.
From election interference to the mass production of fake reviews, the examples are legion. In fact, it's easy to imagine that these are just a few of the sinister practices currently threatening our lives.
The truth is of course more nuanced, but it raises broader questions about how we can better understand where these malicious AI techniques are being applied, by whom, at what scale, and for what purposes.
Thanks to the work of Nahema Marcial of Google DeepMind and Rachel Xu of Google Jigsaw, we now have some kind of answer. They have been studying the misuse of generative AI and how it has evolved over the last few years. Their approach has revealed a range of malicious activities that they have categorized. “We uncovered significant new patterns in misuse over this period, including potential motivations, strategies, and how attackers leverage and exploit system capabilities,” they say.
Emergent Communication
Along the way, they also found specific types of activity that sit on the border between acceptable and unacceptable uses of AI. “This includes the emergence of new forms of communication for political lobbying, self-promotion, and advocacy that blur the line between real and false,” the team wrote.
Their approach is surprisingly straightforward: Marshall, Xu and their colleagues analyzed more than 200 media reports of the misuse or abuse of AI systems published between January 2023 and March 2024. They then categorized the types and patterns of reported abuse, creating a taxonomy of the tactics bad actors use in their operations.
According to the researchers, the types of misuse fell into two broad categories: exploiting generative AI systems and attempting to compromise those same systems to leak protected information or perform prohibited tasks.
They then further subdivide these categories. The first and most common category of uses for generative AI is realistic portrayals of human likeness for tasks such as impersonation, creating synthetic personalities, and creating non-consensual sexual imagery. “The most common cluster of tactics is the manipulation of human likeness, specifically impersonation,” say Marshall, Xu, and their colleagues.
One example is a story that aired on PBS News. AI robocalls attempt to suppress votes in New Hampshire Impersonating President Biden.
The second category involves realistic depictions of non-human objects and includes the forging of documents such as identification cards and the creation of counterfeit items designed to pass off as the real thing.
The final category focuses on the mechanics of content creation, including workflow automation, production at scale, and targeted production to specific individuals. Researchers used ChatGPT to send mass emails to lawmakers To raise awareness of AI-generated emails.
Despite the wide variety of exploitable applications, Marshall, Xu and their colleagues conclude that most are not technologically advanced and instead employ easily accessible generative AI capabilities.
Perhaps most interesting is the emergence of new forms of communication that blur the lines between acceptable and unacceptable uses of generative AI. For example, in the recent elections in India, Political avatars have emerged that address voters by their names, regardless of the language they speak. And various politicians have used deepfakes of themselves to not only spread their message more widely, but also to make themselves appear more favorable.
Few of these examples explicitly acknowledged how generative AI was used in these campaigns. “The cultivation and defense of political images powered by GenAI, without proper disclosure, undermines public trust by making it difficult to distinguish between real and fabricated portrayals,” the researchers wrote. “We are already seeing instances of lying payoffs. Public figures can explain evidence against them as AI-generated.”
Expanded monetization
Besides attempts to masquerade as humans and exert deceptive influence, the most common goal of malicious users of AI is to monetize their products, for example by churning out low-quality articles, books, and advertisements to gain attention and ad revenue.
The production of non-consensual sexual images is also a thriving area of commercial activity, e.g. “Nude” women as a paid service.
Of course, the study has some limitations that the researchers want to highlight. For one, the study was based solely on media coverage of malicious online activity, which may introduce bias. For example, the media tends to focus on the most outrageous cases, potentially overestimating some types of sensational activity while underestimating others that get less attention but are just as insidious.
However, Marshall, Xu and colleagues have taken an important first step in studying the ecosystem of malicious uses of generative AI. Their work raises important questions about the far-reaching impact of this activity and how it is changing the nature of communication and society itself.
While the team didn't attempt to characterize the rate of change, it's not hard to imagine that the impact of these activities could grow exponentially — making the issue even more of a concern to society because humans are not very good at imagining the consequences of exponential change.
“These findings highlight the need for a multifaceted approach to mitigating the misuse of GenAI, including collaboration between policymakers, researchers, industry leaders, and civil society,” Marchal, Xu, and their colleagues conclude. The sooner, the better.
Reference: Misuse of Generative AI: A Taxonomy of Tactics and Insights Gained from Real-World Data: arxiv.org/abs/2406.13843
