By 2026, over 80% of enterprises will be using generative artificial intelligence (GenAI) APIs and models, or deploying GenAI-enabled applications in production. With the rapid pace of adoption, it's no wonder that artificial intelligence (AI) application security tools are already being used by 34% of organizations, a number that will undoubtedly grow.
Applications must undergo rigorous security testing to ensure they do not introduce security risks that put the application or its users at risk. How do we ensure that AI systems are thoroughly tested when the risk landscape is evolving as rapidly as the technology? Machine learning is essential in many new AI systems. Security assessments of these systems are different from other types of system security reviews. A thorough security review starts with understanding the components that the system contains.
Below is an introduction to machine learning concepts that are relevant to cybersecurity professionals. This material is useful for security practitioners, such as DevOps engineers, looking for an introduction to AI security. It also serves as relevant background information for our other AI-focused articles. See AI Security Risks and Recommendations: Unlocking the AI Magic Box.
What is Machine Learning?
Machine learning (ML) is a subset of AI that enables computers to perform tasks without explicit programming. It is built on algorithms and statistical models designed to recognize patterns and relationships in data and iteratively improve performance. This iterative learning process involves training the model with labeled data and tuning its internal parameters to minimize the difference between predicted and actual outputs. This approach allows the model to learn to generalize from the training data, facilitating accurate predictions for new or unseen data. ML includes a variety of techniques, including supervised learning, unsupervised learning, and reinforcement learning, each tailored to different data scenarios. Supervised learning trains models on labeled datasets, while unsupervised learning infers patterns from unlabeled data. Reinforcement learning allows agents to make sequential decisions by interacting with the environment and receiving feedback. Effective ML models can automate tasks, make predictions, and reveal insights across a variety of domains.
Large-scale language models (LLMs) are a specific type of machine learning that has led the AI industry due to their ability to understand and generate human-like text. LLMs, such as OpenAI's GPT series, are trained on vast amounts of text data and use deep learning architectures, typically based on Transformer models. These models learn to predict the next word or token in a sequence based on the context provided by the previous word. What distinguishes LLMs is their ability to capture complex linguistic structure, semantics, and context to generate contextually relevant text responses. LLMs have proven proficient in a variety of language-related tasks, including language translation, text summarization, question answering, and text generation.
Whether you use custom-trained ML models running on dedicated inference infrastructure or integrate with a third-party LLM provider, ML can pose many cybersecurity risks and create weak spots in your overall security architecture.
What is a machine learning model?
An ML model is a file that is created using patterns from previously analyzed data. The model uses its previous learning to make predictions about new data that is passed to the model. Model files contain weights structured as vectors and can be interpreted as very complex graphs that plot new data. More specifically, models are raw files on a filesystem that can be used for inference.
Some model files can be packaged as a model archive, which contains the model files as well as code that can be used to transform or tokenize data before passing it to the model. This code is used to convert input data into vectors compatible with the model, and to convert output from the model back to the desired format.
Models can be created using one of many different ML libraries, the most popular of which are:
Understanding the components and risks associated with ML frameworks and model formats is part of the due diligence that must be performed to ensure that security flaws in your ML deployments do not introduce vulnerabilities or unintended consequences.
