Imagine the following scenario. An algorithm has grabbed a background YouTube video, or maybe a podcast. Hackers are implanting inaudible sounds designed to hijack your smart speakers and your phone’s AI assistant without your knowledge. This means cybercriminals can gain access to your personal photos, bank accounts, or other personal information connected to your AI system.
This sounds like something out of an episode of “Black Mirror,” but that’s exactly what researchers show in a new study being presented this week at the IEEE Symposium on Security and Privacy.
Essentially, a team of researchers from China and Singapore have discovered that they can construct “adversarial voices” that are completely undetectable to the human ear. This tricks a voice AI model into doing something it’s not supposed to do. And it’s easy to hide it in some innocuous-sounding audio (songs, movies, or anything else that an unsuspecting target might play in the background), waiting for the user to accidentally invade their digital life.
“It takes only 30 minutes to train this signal. Then, because this signal is context-independent, we can use it to attack the target model whenever we want, no matter what the user says,” said first author Meng Chen, a doctoral candidate at Zhejiang University in China. IEEE spectrum of the work. “These single-point defenses have a hard time resisting our attacks because we find it very difficult for these models to distinguish between a normal user’s intent and an adversary’s attack.”
There’s one problem, at least for now. This technique required hackers to access all the weights of the targeted AI model and could only attack open-source models. However, many commercial AI systems are built on open source models, meaning their exploits will work against mainstream products from Microsoft and Mistral.
Mistral didn’t respond. IEEEreached out for comment, but Microsoft issued a statement saying that perhaps everyone should pause for a moment before tying sensitive information to one of the company’s voice AI models.
“We appreciate the efforts of researchers to advance our understanding of this type of technology,” it says. “This research assesses model resilience through controlled, direct interaction with the model itself. This helps inform approaches to building model resilience. In practice, AI models are often integrated into user applications, providing developers with tools and guidance they can use to implement additional layers of protection to help protect users.”
Learn more about AI: Researchers wary of AI that can self-replicate into other machines
