Apple removed several AI-powered “nudify” apps from the App Store after legal pressure from the San Francisco City Attorney, but the episode raises far bigger questions than whether a handful of apps violated App Store rules.
The real question is why these apps entered one of the world’s most tightly controlled app markets in the first place.
This latest controversy began after San Francisco City Attorney David Chiu sent cease-and-desist letters to both Apple and Google, accusing them of facilitating the distribution of apps that can generate non-consensual, AI-generated nude images. The companies profit from these apps through in-app purchases, according to legal notices, and California officials argue that the platforms should sever ties with developers who offer these services.
In response, Apple acknowledged that some of the apps identified have already been removed, while developers who don’t follow App Store policies risk permanent removal.
Apple claims the app violates its rules. Critics say they shouldn’t have appeared.
After the publication of WIRED’s findings, Apple told 9to5Mac that apps designed to generate or distribute pornographic or “nude” content always violate its app review guidelines. The company said three of the apps cited by the San Francisco City Attorney’s Office have already been removed from the App Store, and their developer accounts are in the process of being suspended. Apple also said it contacted four more developers and warned them to address policy violations or be removed from the platform.
“The App Store is designed to be a safe and trusted place for our users,” Apple said in a statement, adding that it actively rejects nude apps and removes others after they are reported to us through reporting tools. Apple also reiterated that developers remain responsible for the content generated within their applications. Apps that allow user-generated content must include mechanisms to filter objectionable material. Failure to do so may result in permanent removal from the App Store.
Google took similar action. The company said it has already suspended hundreds of Android apps that offer nudity features and restricted Play Store searches for terms such as “nudity.” Google said five Android apps cited by the San Francisco City Attorney were also removed for violating Play Store policies.
Here’s Apple’s full statement provided to 9to5Mac: “The App Store is designed to be a safe and trusted place for our users, and we’ve always strictly prohibited apps designed to generate, distribute, or consume pornography. ‘Nude’ apps violate our app review guidelines, and we actively reject many of these apps and remove many others, including when users reported them through our reporting tools.” We have removed three of the offending apps and are in the process of terminating your developer account with us. We are in contact with four other people who need to be addressed for policy violations and may be removed as well. ”
The bigger challenge is moderation in the age of AI
The lawsuit comes amid growing concerns that generative AI has dramatically lowered the barrier to creating blatant deepfakes. Unlike traditional porn, many of today’s AI-powered face-swapping tools only require a single photo. Within seconds, users can generate fabricated nude images and explicit videos depicting real people without their knowledge or consent.
Officials say the implications go far beyond privacy violations. Victims increasingly report harassment, extortion, reputational damage, and severe mental health effects. Previous investigations have also revealed incidents in dozens of schools where AI-generated sexual images of minors were created using commercially available tools. What makes enforcement particularly difficult is that many of these applications do not openly advertise themselves as “undressing” tools. Instead, they present themselves as harmless AI face swapping or photo editing apps. Explicit image generation functionality often remains hidden until after installation or after an in-app purchase is completed, making automatic moderation much more difficult.
A study published earlier this year found that seemingly normal-looking apps that swap multiple faces can still generate blatant deepfakes, even though they don’t market themselves as such. In many cases, researchers have found that there are little or no safeguards to prevent exploitation.
This distinction is currently at the heart of the debate facing Apple and Google.
Both companies say their policies specifically prohibit pornographic and sexually exploitative content. But critics argue that policies alone are no longer enough when AI applications can hide harmful features behind legitimate ones. The City Attorney’s Office also argued that Apple and Google derive economic benefits from these apps through in-app purchase fees, and that platform operators have a greater responsibility to identify and remove apps before they reach users.
For Apple, the takedown may help address the immediate criticism, but it’s unlikely to end the broader discussion. With generation AI becoming cheaper and faster to create realistic synthetic media, app reviews are no longer just about screenshots and developer descriptions. Now, you need to identify what you want the AI model to produce after installation, which is a much more difficult task than traditional moderation.
This challenge is unlikely to go away anytime soon, and similar questions are being asked of the companies that run the world’s largest app stores.
