Machine learning (ML) and artificial intelligence (AI) are essential components of any modern and effective cybersecurity solution. However, as the use of ML and AI in cybersecurity becomes more and more common, and the industry increasingly relies on these technologies to thwart ever-growing threat data, AI is not a panacea but a stand-alone. It is important to remember that it has the effect of new attack surface. Decision makers evaluating cybersecurity postures are aware of these risks and limitations inherent in AI so that they can validate that their defenses are robust and resistant to emerging threats such as adversarial machine learning (AML). need to do it. AML is an emerging research area concerned with attacks against ML and AI-based systems, including deception and evasion of AI detectors. For defenders, it is important to be aware and able to recognize these novel attacks.
What does an attack on ML models look like?
ML solutions use various algorithms and statistical techniques to analyze datasets and identify patterns. The underlying foundations of these techniques inherently enable new types of attacks against AI and ML-powered systems. The MITER ATLAS framework enumerates and categorizes attack techniques against ML systems.
One such technique is data poisoning. It is intended for manipulating the data used to train AI models. AI models learn how to react to inputs from large datasets called “ground truth”. Ground truth defines what the proper output of the model should be. This is what the model itself is modeled on. Attacks can attempt to add false information to the ground truth that is incorporated into the training process. Operating the training process in this way causes the model to react incorrectly to some input data. For example, an attacker can trick the model into classifying a malware file as a legitimate application.
Data poisoning attacks can be carried out in a number of ways, including gaining access to ground truth datasets as part of a traditional security breach. However, working with public datasets used to train data-intensive AI algorithms is a more popular and impactful technique. If the AI learns directly from user input, an attacker could use that access to subvert her AI system. This happened with the Twitter bot Tay. Tay’s AI was supposed to learn from conversations with other Twitter users.User deliberately manipulated bot to post hate speech Posted on social media platforms within a day.
ML systems are also susceptible to evasion attacks, where attackers attempt to trick the model’s prediction system. Attackers may use so-called adversarial examples and input data containing small perturbations intended to confuse the ML system and achieve incorrect classification. A typical example of this type of attack is modifying a few pixels in the image before uploading it so that the image recognition system fails to classify the image or classifies it in a different way. Minor pixel changes are often invisible to humans or not directly recognizable as an attack, yet they fundamentally change the model’s output.
In cybersecurity-specific evasion attacks, security researchers manually modified malicious files so that they would be evaluated as legitimate by antivirus vendors’ AI-based detections. Researchers did this by extracting strings from legitimate software and adding them to the malware. The vendor’s AI model weighed these legitimate strings over the malicious routines in the file and incorrectly classified the file as benign.
Knowledge is power in times of need AI model
Prior knowledge of the ML target system also affects the likelihood of a successful attack. The more an attacker knows about her AI system and its architecture, the easier it will be for them to launch attacks and choose appropriate attack methods. In the case of the aforementioned evasion attack against antivirus vendors, the attackers had access to models and software. This is called a white-box attack. An attacker can analyze the algorithm and find suitable strings to successfully trick the system.
At the other end of the spectrum are black box attacks, where attackers have little to no knowledge of AI models. If the model outputs statistical certainty about the classification (such as the probability that a file is malware), attackers may use gradient-based techniques. You can iteratively modify the malware file, see the probability of malware calculated by the model, and adjust the next round of modification depending on whether the probability rises or falls. In this way, it’s a “hot and cold” game of approaching the goal until the file is very unlikely to be malware.
How to secure your machine learning model
Defenders can protect their ML systems using methods that can prevent, complicate, or detect attacks. For example, even if you add a benign string to a malware file, a monotonic classification model will detect the file correctly. It doesn’t matter to the model how many benign traits a file has, provided malware traits are also present.
Gradient-based attacks can be complicated by models that output only so-called hard labels, i.e. models that have no probabilities and only output categories (such as “malware” or “harmless application”) as the final result. I have. However, assuming the attacker can collect a sufficient amount of output records, the attacker could train a proxy-his model based on the hard his-labeled output of the victim model as ground-her-truth. I have. This proxy model can be used to approximate the gradient of the victim model. Therefore, the goal of the defender is not to thwart all possible attacks, but to find viable routes to attack the ML system and ensure that the ML system is detected when it is under attack. is to increase the cost of the adversary for
Defenders can access a wide range of data sources to defend against AML attacks, including using Extended Detection and Response (XDR). However, it is important for defenders not to rely solely on AI and blindly trust its results. Given that AI brings its own attack surface, defenders should avoid her AI monoculture and incorporate other powerful approaches such as indicators of attack. After all, security her vendor needs human expertise above all else. Ultimately, the ability to recognize adversarial ML attacks and adapt AI models accordingly is critical to building robust defenses.
CrowdStrike will be exhibiting at Infosecurity Europe next week. Register for the event here.
