Good news, everyone. “Openweight” AI models that anyone can download and run are ridiculously easy to poison.
Katie Paxton-Fear, a cybersecurity researcher at Semgrep, demonstrated this with an attack that took less than an hour to execute and cost less than $100. register They have reported that they were able to successfully manipulate the behavior of AI by feeding it malicious data.
By training the model with just 10 examples of poisoned material, the model began to generate a large amount of new code that was exposed to remote code execution. This vulnerability allows hackers to execute code on a person’s machine.
“I did the proper backdoor,” she triumphantly shared on social media.
Backdoors are a particularly dangerous type of attack that involve training an AI in a way that introduces hidden phrases into the underlying model. Hackers can use these to silently trigger a model to perform a specific action, then remain dormant and unobtrusive until the action is called. Last year, Anthropic published the results of a study it conducted in collaboration with the UK Institute for AI Security and the Alan Turing Institute. This research shows that both small and large AI models are vulnerable to this attack with just a few hundred documents, suggesting that these attacks can be performed at low cost.
The findings will put a damper on the enthusiasm surrounding open-weight models, which have been praised for the control and transparency they offer, as well as the lower cost of use, compared to closed-source models that run chatbots like ChatGPT and Claude. However, open weight models do not reveal the training data or code, even though the parameters are visible. This means it can still be a black box for security researchers.
“Even when a model’s weights are made public (‘open weights’), there is little ability to predict its behavior,” Paxton-Fear’s colleagues at Semrep wrote in a post last week. “This is a big change. A typical computer program in binary form can still be analyzed using reverse engineering tools and a complete description of its behavior can be arrived at. Models cannot come close to this capability.”
Moreover, this is completely uncharted territory in cybersecurity. LLM is very complex and still new, making advanced attacks difficult to discover. And AI models can be compromised in far more subtle ways than software.
“When software dependencies contain malicious code, we have mature practices to discover it, track its origins, and mitigate its impact,” Semgrup researchers argued. “AI models are different. Compromised or manipulated models do not need to be ‘destroyed’ to create business risk. It is enough to influence decisions in a way that is difficult to detect. ”
“So can we trust open weight models that have been tweaked online and sold as a solution to the AI token spending problem?” Paxton Fear asked in a thread sharing the findings. “Well, you probably need something better than a benchmark, and you need to ‘don’t write unsafe code.'”
Learn more about AI: Fears of an AI bubble are starting to spread
