This is not the calm, thoughtful Gareth Southgate the nation is used to – and, in the wildly immediate world of internet humour, perhaps that is the point.
Hours after England walked off the pitch following their semi-final win over the Netherlands, a deepfake of the team's manager appeared on social media, subtly rewriting his post-match comments in an expletive-filled, very uncharacteristic way.
“Looks like I've been riding a big Rizla today,” says the fake Southgate in a video posted to TikTok after England's dramatic win over the Netherlands, in which he uses artificial intelligence tools to imitate Southgate's voice and manipulate the lower half of his face with amateurish lip syncing. There will undoubtedly be more videos posted after Sunday's final, whatever the outcome.
The England manager is widely considered an icon for his country, but in this case, for the internet trend of deepfake memes.
Like image editing before it, AI-generated fake video clips, or deepfakes, have become an important tool in the internet joker's toolbox.
“Deepfakes are becoming the new Photoshopped content,” says Bahareh Helaby, a professor at the University of Surrey's Institute for Human-Centred AI. “People can now use a range of AI-based tools to create similar content to what they previously did with Photoshop or video editing tools, but now with much better video and audio generation capabilities.”
Put simply, commercially available face-swapping apps allow anyone to easily replace one face with another in a video clip with the press of a button – a major technological achievement in 2018. Recent examples include Rishi Sunak and Keir Starmer as Del Boy, and Rodney Trotter and Noel Gallagher as various Irishmen, including Roy Keane, as a tongue-in-cheek response to the rock star mentioning his Irish roots in an interview.
Combined with a voice cloning tool like ElevenLabs, you can even go a step further and create an entire script to read for your newly faked celebrity. If you've ever seen footage of British politicians as Twitch Minecraft streamers, you'll know that the resulting production is packed with in-jokes and references, with only a weird robotic nasal voice or a funny tone of voice keeping you from gags.
Experts at Faculty AI, a UK-based company with private and UK government contracts to spot deepfakes, say there is clear evidence in Southgate's video, including mismatched words and lip movements, and unnatural pauses and pacing in his speech. The most obvious evidence, of course, is that England fans know Southgate would never say such things about his players: Harry Kane, Phil Foden and Jack Grealish are among the players roundly criticised by the fake manager.
The professors said last year's re-creation of a Balenciaga fashion show featuring Harry Potter stars perhaps marked a tipping point for AI-driven deepfake videos, adding that AI fraud would become a “bigger threat to society”. ITV's Deepfake Neighbour Wars, which featured simulations of stars such as Idris Elba and Kim Kardashian, also showed the concept was going mainstream.
But Faculty said generative AI — referring to systems such as ChatGPT and Midjourney that can generate compelling text, voice and images from simple prompts — is advancing rapidly and “manual detection alone will soon be insufficient to prevent the spread of truly harmful content.”
The FA said it would “take steps to remove any offensive videos, as we do with any harmful content.”
After newsletter promotion
While the intent of these pranksters is rarely to directly mislead, that doesn't mean their jokes aren't confusing. There is no parody on the Internet that isn't eventually mistaken for an instance of the thing it's mocking. This is so common that it was named “Poe's Law” after a commenter who noted the difficulty of satirizing extreme views about 20 years ago. As with The Onion articles and images produced for the humor site B3ta, it's fairly common for viral deepfakes to be reshared with increasingly skeptical comments asking whether the clips are real. Some of Southgate's clips have been labeled as AI-created.
Southgate's videos are frequently viewed on social media: On YouTube, where many fake Southgate clips are available, one deepfake posted after the Slovakia match has been viewed 390,000 times, while a TikTok account dedicated to Southgate fakes has eight videos with a combined total of more than one million views. The TikTok account linked to an eBay page selling customized AI messages of Southgate, adding: “Let me know if you want some swear words.”
TikTok's guidelines require that realistic-looking AI content be labeled and allow likenesses of famous people in “certain artistic or humorous settings.” YouTube has changed its moderation policies to allow users to request the removal of deepfakes, but has said it will consider whether a video is a parody or satire before deciding to remove it.
“With public figures, things get out of hand because anything can be mistakenly attributed to them,” Helavy said, adding that he's particularly concerned about audio because there are no visual cues to distinguish a deepfake voice. “But we can expect the situation to improve as better deepfake detection tools emerge and, more importantly, as the public becomes more aware and educated about deepfakes.”
Fake videos aren't the only areas where generative AI is changing the way memes are made. Meme-generating tools, led by Glif, are beginning to automate nearly the entire process of humor creation. For memes with a set format, like “Chad/virgin” or “Fuck math,” users can create entire templates that combine carefully instructed large-scale language models (LLMs) to generate the text, image-generating tools to generate the images, and a standardized structure to hold it all together. The result is a tool that can be instructed to create a meme about why “football tournaments should be stopped,” and within seconds receive the recommendation that “kicking the ball does not determine national pride.”
