In the midst of a crucial election taking place amid a heatwave, a confusing storm of deepfakes is sweeping across India: a seemingly endless variety of AI-generated impersonations, ventriloquism, and deceptive editing effects, some of which are crude, some of which are tongue-in-cheek, and some of which are so obviously fake that they seem too good to be true.
The overall effect has been confusing, adding further chaos to a social media landscape already saturated with misinformation. The amount of online debris is so vast it is impossible for electoral commissioners to uncover, let alone track.
A range of vigilante fact-checking organisations have sprung up to fill the gap. As the wheels of the law move slowly and unevenly, the job of tracking down deepfakes has been taken on by hundreds of India-based government officials and private fact-checking organisations.
“We have to be prepared,” says Surya Sen, a forest official in Karnataka who was transferred to lead a 70-person team to crack down on false AI-generated content during the election. “Social media is the battlefield this year.” When Sen's team finds content that appears unlawful, they order social media platforms to take it down, publish falsehoods and even seek criminal charges.
Celebrities, including Hindi film star Ranveer Singh, have become regular targets for politically targeted ploys.
In a video interview with an Indian news agency on the Ganges river in Varanasi, Singh praised powerful Prime Minister Narendra Modi for celebrating “our rich cultural heritage.” But that's not what viewers heard when an altered video with a voice that resembled Singh's and near-perfect lip-syncing went viral on social media.
“We call it lip-sync deepfake,” said Pamposh Raina, head of the Deepfake Analysis Unit, a consortium of Indian media companies that has set up a tip line on WhatsApp where people can submit suspicious videos or audio for review. He said Singh's video is a classic example of real footage edited with AI-replicated audio. Singh has filed a complaint with the Mumbai police's cybercrime unit.
No party has a monopoly on false content in this election. Another doctored video began with real footage of Rahul Gandhi, Modi's main rival, taking part in a mundane ritual of taking oath as a candidate, before an AI-generated audio track was overlaid.
Gandhi did not actually quit the party. The video also includes personal criticism and appears to say that he “can no longer pretend to be a Hindu.” The ruling Bharatiya Janata Party describes itself as the defender of Hinduism and its opponents as traitors and fraudsters.
Sometimes political deepfakes reach into the realm of the supernatural, with deceased politicians coming back to life through mysterious AI-generated likenesses to endorse their descendants' real-life election campaigns.
In a video released just days before voting began in April, H. Vasanthakumar, who died of COVID-19 in 2020, was seen resurrected to indirectly speak about his own death and congratulate his son Vijay, who is running for his father's former MP seat in the southern state of Tamil Nadu. The appearance followed the example of two late giants of Tamil politics, Muthuvel Karunanidhi and Jayalalithaa Jayaram.
The Modi government has passed laws to protect Indians from deepfakes and other misleading content. The 2021 “IT Rules” Act, unlike the U.S., holds online platforms liable for any kind of offensive content, including impersonation intended to offend. The Indian digital rights group Internet Freedom Foundation argues that these powers are too broad and is tracking 17 lawsuits against the law.
But the prime minister himself seems to be open to some AI-generated content. In two videos made with AI tools, two of India's biggest political figures, Modi and one of his fiercest opponents, Mamata Banerjee, imitate a viral YouTube video of American rapper Lil Yachty making “the most violent exit of all time.”
Sharing the video on X, Modi said he was “delighted” at such creativity. Election officials like Karnataka's Sen called it political satire and said, “There is nothing wrong with Modi being a rock star, it's not a violation. Everyone knows this is fake.”
Police in West Bengal state, where Banerjee is chief minister, have served notices to some people for posting “objectionable, hateful and inflammatory” content.
In hunting for deepfakes, Sen said his team, working for the opposition-ruled Karnataka state government, carefully scrolled through social media platforms such as Instagram and X, searching for keywords and repeated updates on the accounts of popular influencers.
The Deepfake Analysis Unit has 12 partners in the media world, including several close to Prime Minister Modi's government. Raina said the unit also works with outside forensic labs, including at the University of California, Berkeley. The unit uses AI detection software such as True Media, which scans media files to determine whether they are trustworthy.
Some tech-savvy engineers are refining AI forensic software to pinpoint which parts of a video have been manipulated, down to the individual pixels.
Prateek Sinha, founder of AltNews, India's most authoritative independent fact-checking site, said the full potential of deepfakes has yet to be exploited: one day, politicians could be seen on video not just saying things they didn't say, but doing things they didn't.
Dr. Hany Farid, who has taught digital forensics at Berkeley for 25 years and has worked with the Deepfake Analysis Unit on some cases, said that while “we're catching the bad deepfakes,” as more sophisticated fakes emerge, they may go unnoticed.
In India, as elsewhere, an arms race has begun between deepfake creators and fact-checkers, battling on all fronts. Dr Farid described this as “the first year that the impact of AI really starts to manifest itself in interesting and more sinister ways.”
