Gaatha Sarvaiya likes to post on social media and share her work online. An Indian law graduate in her early 20s, she is in the early stages of her career and trying to gain public attention. The problem is that with the rise of AI-powered deepfakes, there is no longer any guarantee that the images she posts will not be distorted into something transgressive or grotesque.
“I immediately thought, ‘Okay, maybe it’s not safe here. People might take pictures of us and do something with it,'” says Sarvaiya, who lives in Mumbai.
“It is true that there is a chilling effect,” says Rohini Lakshane, a Mysore-based gender rights and digital policy researcher. He also avoids posting photos of himself online. “Given the fact that it can be exploited very easily, I would be especially cautious.”
In recent years, India has become one of the most important testing grounds for AI tools. This is the second-largest market for OpenAI in the world, and the technology is widely adopted across a variety of occupations.
But a report published on Monday, based on data collected by the Rati Foundation, a charity that runs a national helpline for victims of online abuse, shows that the increasing adoption of AI has created powerful new means of harassment against women.
The report, authored by the Rati Foundation and Tuttle, a company working to reduce misinformation on social media in India, said: “Over the past three years, it has become clear that the vast majority of AI-generated content is being used to target women and sexual minorities.”
In particular, the report found that AI tools are increasingly being used to create digitally enhanced images and videos of women, including nudes and images that may be culturally appropriate in the United States. While these images may be culturally appropriate in the United States, they are frowned upon in many Indian communities, including public displays of affection.
The report found that around 10% of the hundreds of cases currently reported to the helpline contain these images. “AI makes it much easier to create realistic-looking content.”
There have been high-profile cases in the past where Indian women had their images manipulated by AI tools in public. For example, Bollywood singer Asha Bhosle’s likeness and voice were replicated using AI and spread on YouTube. Rana Ayyub, a journalist known for investigating political and police corruption, was the target of a doxxing campaign last year that resulted in deepfakes of her sexual images being published on social media.
These led to a society-wide debate in which some figures, like Mr. Bhosle, successfully fought for legal rights to their voice and image. But less discussed is the impact that incidents like this have on ordinary women, like Sarvaiya, who are increasingly worried about going online.
“When people face harassment online, they actually silence themselves or become less active online as a result,” says Tarunima Prabhakar, co-founder of Tattle. Her organization used focus groups across India for two years to understand how digital abuse is impacting society.
“The emotion we identified is fatigue,” she says. “And the result of that fatigue is that they turn away from online spaces altogether.”
Over the past few years, Sarvaiya and her friends have been following high-profile cases of deepfake online abuse, including those of Ayyub and Bollywood actor Rashmika Mandanna. “It’s a little scary for the women here,” she says.
Currently, Sarvaiya is hesitant to post anything on social media and has kept her Instagram private. She worries this won’t be enough to protect her. Women are sometimes photographed in public places such as subways, and those photos can later be posted online.
“It’s not as common as you might think, but you never know your luck, right?” she says. “A friend of a friend is literally receiving threats from the internet.”
Lakshane says she now often asks not to be photographed at events where she speaks. But despite taking precautions, she is bracing for the possibility that someday a deepfake video or image of her will surface. In the app, my profile picture is an illustration of myself instead of a photo.
“Especially for women who have a public presence, have an online voice, and take political positions, there is a risk of image misuse,” she says.
After newsletter promotion
Rati’s report outlines how AI tools, such as “nudification” and nudity apps that can remove clothing from images, have made incidents of abuse that were once considered extreme become more common. According to the newspaper, in one case, a woman contacted the helpline because the photo she submitted when applying for a loan was used to extort money.
“When she refused to continue payment, the uploaded photo was digitally altered using the nudify app and placed over a pornographic image,” the report states.
The photo, with her phone number attached, was circulated on WhatsApp, resulting in a “barrage of sexually explicit calls and messages from unknown individuals.” The woman told Rati’s helpline that she felt “embarrassed and socially marked, as if I had ‘got involved in something dirty'”.
Deepfakes operate in a legal gray area in India, as in most parts of the world, with no specific laws recognizing deepfakes as a clear form of harm, but Rati’s report outlines some laws in India that may apply to online harassment and intimidation, under which women can also report AI deepfakes.
“But the process is very long,” Sarvaiya says, arguing that India’s legal system remains ill-equipped to deal with AI deepfakes. “And it takes a lot of red tape to get there to get justice for what was done.”
Part of the blame lies with the platforms on which these images are shared, often YouTube, Meta, X, Instagram, and WhatsApp. Indian law enforcement describes the process of forcing these companies to remove abusive content as “often opaque, resource-intensive, inconsistent and inefficient,” according to a report released Tuesday by women’s rights campaign Equality Now.
Apple and Meta have recently taken steps to limit the spread of the nudify app, but Rati’s report identifies several instances in which these platforms have failed to adequately respond to online abuse.
WhatsApp eventually took action in the extortion case, but its response was “inadequate” as the nudes were already all over the internet, Rati reported. In another case, an Instagram creator in India was harassed by a troll who posted nude videos, but Instagram responded only after “sustained efforts” and a “delayed and inadequate” response.
The report said victims who report harassment on these platforms are often ignored, leading them to contact helplines. Additionally, even if platforms remove accounts that spread abusive content, that content often resurfaces elsewhere, a phenomenon Rati called “content recidivism.”
“One of the enduring characteristics of AI abuse is its propensity to proliferate: it tends to be easily created, widely shared, and recur over and over again,” Rati says. Addressing this “will require far greater transparency and data access from the platforms themselves.”
