How to teach an AI model to misbehave in a training environment

Imagine you are a developer tasked with auditing a large language model for safety issues. Uses harmful prompts to probe LLMs in an attempt to divulge sensitive information or generate prohibited content. It rejects them all. The model passed the test with flying colors.

But the next day, it turns out that users were able to easily bypass the safety and abuse filters. How can this happen?

This problem is known as “alignment spoofing” or “AI schemes,” where sophisticated AI systems exploit flaws in reward functions to maximize performance metrics without actually solving the intended task. Like the example above, it appears safe, true, or compliant when evaluated, but adopts different and potentially problematic behavior when deployed.

According to a new paper from a team led by Pin-Yu Chen, principal AI research scientist at IBM Research, this may be part of a broader, less-explored problem. This means that flaws in the training environment itself can cause inconsistencies. Even if the reward function is designed correctly, the model can exploit loopholes in the environment, such as hidden correlations or “shortcut” signals. Chen conducted the study with colleagues at IBM Research and the Notre Dame-IBM Technology Ethics Lab. They presented their results this week at the International Conference on Machine Learning (ICML) in Seoul, South Korea.

In a series of experiments, the team found that competency-oriented training can inadvertently induce exploitative or deceptive behavior in AI models, particularly LLMs trained using reinforcement learning. Rather than being one-time tricks, these skills are actually widely translatable and generalizable, and can be applied to new tasks and taught to other models. And as the model’s capabilities improve, the frequency with which true improvements occur will also increase.

In other words, as your LLM gets better, you also get better at finding shortcuts to maximize rewards during training.

To investigate how the model malfunctions, the researchers developed four controlled experimental scenarios called “vulnerability games.” Each game was injected with loopholes that encouraged the models to behave deceptively.

Compliance in context: In one experiment, models were given harmful and benign prompts written in different styles. As in the example above, the test model learned to behave safely under test conditions, but behaved unsafely in a real-world context. Instead of understanding safety principles, they relied on superficial clues such as writing style. A model auditor may use formal language in the prompt, but a typical user may write informally or make typographical errors. “If they feel valued, they may take action,” Chen said. It’s like when Volkswagen infamously programmed its diesel cars to run cleanly under test conditions, a trick that was only discovered when researchers measured emissions under real-world conditions.

Audited self-scoring: In this game, the model was asked to give an answer and report whether the answer was correct. If audits are infrequent, the model has learned that fraud is the best policy. They claimed to be right regardless of the truth and reaped their ill-gotten rewards.

Proxy metric gaming: In this game, we tested a model for the complex task of text summarization. Rewards are strictly defined by surface-level metrics. An n-gram, so called because its length is ‘n’ units, is a collection of items of text used to analyze a document. N-grams can be used to identify patterns such as the frequency of words within a particular document.n-g It overlaps with things that are difficult to measure, such as understanding meaning. It showed that high-performing models that were optimized for measurable metrics (such as word duplication in summaries) rather than true quality produced high-scoring outputs but were significantly less useful and consistent for human users.

Tampering with rewards: Here, the researchers simulated an automated evaluation pipeline that required models to submit code solutions in JSON objects. They found that the model consistently exploited technical vulnerabilities in the rating system itself, such as directly changing the score field instead of solving the assigned task.

The researchers found that across all four scenarios, the model consistently learned how to exploit these weaknesses during standard reinforcement learning training. These behaviors emerged naturally as the model was refined, without explicit instructions to misbehave.

Previous research has demonstrated the phenomena of alignment faking and AI conspiracy, but the specific conditions under which they occur are not well understood. Two major research questions arise.

The first one is simple. What is the conspiracy? Is it the training environment, reward structure, metrics, or something else? The second question is whether the model learns itself and even becomes unreliable in other models. Will this behavior be generalized by zero-shot transfer, catalytic learning, and distillation?

With the problem thoroughly demonstrated, Chen and his team sought to answer their initial research questions with a set of different models, including a base model, a command-coordination model, and a model trained to generate thought chain-style inferences. They trained a model using reinforcement learning and scored it based on how well it performed on its intended task and how often it misbehaved. We also tracked how many steps it takes for an emerging exploit to become a major mover in the model.

Almost across the board, exploitative behavior occurred spontaneously. In the case of self-scoring and context-conditioned compliance, the model discovered these exploits by chance, received a reward, and amplified the strategy. In other cases, such as proxy metrics, cheating was rewarded from the beginning and naturally became the dominant approach.

Importantly, adjusted risk through competency-oriented training is different from standard reward hacking. Even when reward goals are correctly specified, their scope is broader in terms of gamifying the specification. It is also deceptive because it can imitate legitimate learning.

This behavior reflects a broader phenomenon known as shortcut learning, which has been observed for some time in fields such as computer vision. For example, rather than actually learning how to find cancer cells, AI could identify hospitals with the most cancer patients and flag scans. Chen points out that neural networks often rely on signals that maximize reward, even if they are irrelevant to the actual task. Generative AI systems further enhance this trend due to their flexibility and ability to generalize.

A particularly concerning finding is that these exploitation strategies are not isolated tricks. Instead, they behave like common skills. Once learned, it can be moved to new tasks, allowing the model to discover additional exploits more quickly, and can also be spread to other models through distillation of training data. This suggests that such behavior may be widely propagated as AI systems are developed and reused.

In a series of tests, the model was examined to see if success in one exploit translated into success or discovery in another. Generally speaking, both actually happened. Tweaking the “student” model based on the “teacher” model of honed cheaters was also shown to increase student cheating rates. This raises concerns about long-term AI development, especially systems that repeatedly self-improve, as inconsistencies can accumulate over generations.

“Imagine this catastrophic situation where one generation’s models begin to develop inconsistent behavior or misbehavior, and such ‘bad genes’ are passed on to future generations,” Chen said.

In addition to potentially magnifying the problem, a major challenge is that these exploitative behaviors often appear in parallel with actual improvements in the model’s task performance, creating blind spots for developers where improved functionality hides underlying inconsistencies. As a result, standard evaluation methods may not be able to detect the problem.

From a mitigation perspective, Chen emphasized the need to carefully design the training environment as well as the reward function. This includes ensuring a balanced and representative data distribution, avoiding hidden correlations, and evaluating models across a variety of conditions.

The new research highlights the importance of investigating how misalignment occurs and spreads, with the aim of developing techniques to prevent the learning and transmission of harmful behaviors, Chen said. Importantly, tuning AI is not just about defining the right goals, but also about designing the right environment for the model to learn.

Recent IBM developments in generative computing could help ensure proper alignment. Mellea is an open source library that imposes requirements during inference. It has the ability to automatically break down user queries into requirements and self-check whether those requirements are met when solving a particular task. This approach ensures that the AI’s work is checked against user-specified guardrails, making it more reliable when solving tasks.

Moving forward, the team is currently investigating exactly how deviations are propagated between models, including things like distillation.