1
As image generation and processing using AI tools become more common, it is even more necessary to ensure thorough security throughout the process. Researchers share insights into new attack strategies that exploit AI for data exfoliation via images. This attack shows how to connect known threats of image scaling attacks against AI with a rapid injection and secretly perform malicious actions.
Researchers couple rapid injection attacks using image scaling
In a recent post, researchers from cybersecurity firm Trail of Bits shared details on how rapid injection attacks can exploit image scaling in AI tools to perform malicious actions. These actions can vary from simple activities such as opening an app to simple activities such as delamination of data without warning the victim.
The image scaling attack, first demonstrated by researchers at Braunschweig University in Germany in 2020, involves exploiting the image scaling process of AI systems. When processing images, the AI system scales the input image for faster and better processing before transferring it to the model. Malicious actors can take advantage of this image size reduction to manipulate how models handle images. In the case of the BITS researcher trail, they exploited this image scaling for a rapid injection attack.
Source: Trail of Bit
As demonstrated, researchers injected malicious prompts into the image, making the prompt invisible when the image was in earnest. However, rescaling by AI systems will result in faster system displays due to image resolution changes. When transferred to an AI model, this prompt tricks the model into considering the model as part of the instructions. As a result, the model performs each malicious action mentioned at the prompt without the user's knowledge.
In their experiments, the researchers demonstrated this attack strategy against the Gemini CLI using the default configuration of the Zapier MCP server. They uploaded images that hide malicious prompts in order to exftrate user data from Google Calendar to a specific email address.
Researchers share details of this attack strategy in their posts.
Most AI systems are vulnerable to this attack
Researchers say the attack works for most systems, with minor adjustments depending on the target AI model:
To further test, researchers have also published an open source tool called “Anamorpher” on GitHub. Backed by the Python API, this tool allows users to visualize attacks against multimodal AI systems. Currently in beta, we create images created for multimodal prompt injection when downscaled.
Recommended mitigation
According to researchers, limiting downscaling algorithms doesn't help prevent these attacks, given the wide range of attack vectors. Instead, researchers recommend limiting upload dimensions and avoiding downscaling of images. What's more, ensuring an accurate preview of the images your model is looking at can also help detect rapid injections that may not be noticed when uploading images.
Additionally, researchers encourage the implementation of robust defensive strategies to prevent multimodal rapid injection attacks, such as deploying mandatory user verification before executing instructions provided as text in images.
Tell us what you think in the comments.
Get real-time updates on this post category directly on your device and subscribe now.
