Researchers have devised a new way to manipulate machine learning (ML) models by injecting malicious code into the serialization process.
This method is “Pickling” process It is used to store Python objects in bytecode. Despite its versatility, ML models are often packaged and distributed in the Pickle format. Long-known risks.
As stated New Blog Post According to Trail of Bits, Pickle files allow attackers to inject malicious bytecode into ML programs. In theory, such code could have a variety of outcomes, including output manipulation and data theft, but it is not as easily detected as other supply chain attack methods.
“This allows malicious behavior to be subtly embedded into applications at runtime, potentially leaving them unnoticed by incident response teams for long periods of time,” warned David Brauchler, principal security consultant at NCC Group.
Sleepy Pickle poisons the ML jar
A so-called “sleepy pickle” attack can be carried out easily as follows: Flick-like toolsFlick will not attempt to discover, analyze, reverse engineer, or Creating a Malicious Pickle FileAn attacker simply needs to get a target to download the malicious .pkl, for example through phishing or a supply chain compromise, and then upon deserialization, the malicious operational code is executed as a Python payload.
Poisoning a model in this way has many advantages for stealth: First, it does not require local or remote access to the target system and leaves no traces of malware on disk. The poisoning is done dynamically during deserialization, making it resistant to static analysis. AI repository like Hugging Face It might disappear much more easily.
Due to the large size of serialized model files, the malicious code needed to do damage may only be a small portion of the overall file size, and these attacks can be customized in many ways to prevent detection and analysis, just like regular malware attacks.
While Sleepy Pickle could conceivably be used to perform a variety of actions on a targeted machine, the researchers note that “controls such as sandboxing, isolation, privilege restrictions, firewalls, and outgoing traffic controls can prevent the payload from causing serious damage to the user's system or stealing or tampering with the user's data.”
Even more interesting, attacks can also be aimed at manipulating the model itself. For example, an attacker could insert a backdoor into the model, or manipulate the model's weights to manipulate the output. Trail of Bits has actually demonstrated how this technique can be used to, for example, advise a user with the flu to cure themselves by drinking bleach. Or, using an infected model, Steal sensitive user dataYou can do a lot of things, like adding phishing links or malware to your model output.
How to use ML models securely
To avoid such risks, organizations can focus on using ML models only in Safetensors, a safer file format. Unlike Pickle, Safetensors only handles tensor data and not Python objects, eliminating the risk of deserialization through arbitrary code execution.
“If an organization really wants to run a model that's distributed as a pickle, they can upload it to a resource-safe sandbox (such as AWS Lambda) and convert it on the fly to automatically generate a Safetensors version of the file,” Brauchler suggests.
But, he added, “I think this is just a Band-Aid to a much larger problem. Sure, if you download a Safetensors file, you might have some confidence that it doesn't contain any malicious code, but can you trust that the person or organization that produced this data has generated a machine learning model that doesn't contain backdoors, malicious behavior, or a variety of other issues, oversights, or malicious intent that organizations aren't prepared to address?”
“I think we need to really pay attention to how we're managing trust within our systems,” he says. The best way to do that is to have a strict separation between the data that our models retrieve and the code that our models use to function. “We need to design around these models so that if they malfunction, it doesn't impact the users of our applications and the assets in our environment.”
