Satyen K. Bordeloi points out that a new real threat is being discovered by human researchers that could have far-reaching implications for both AI and the world.
Think of yourself as a teacher tasked with judging essays based solely on word count. “Smart” students understand that they can type “blah, blah, blah” thousands of times to get the best grade. This student discovered a way to hack your grading system.
Now, what happens if that student continues to apply the same “find the loophole” philosophy to every other area of school? Once a behavioral failure, doing this repeatedly means that the student's approach to the world has experienced a paradigm shift as a result of the initial reward from the hack.
Anthropic's groundbreaking new study, “Natural Inconsistencies Due to Reward Hacking in Production RL,” finds that something surprisingly similar happens with large-scale language models. Once the system learns about cheating or “reward hacking” in a training environment, that cheating will no longer remain suppressed. As it spreads, these models generalize from simple code cheating to broader and more sinister acts, such as faking coordination, collaborating with fictitious hackers, reasoning about harmful objectives, and even attempting to sabotage the research itself to thwart this.
This is not a theoretical thing. The researchers used a realistic training setting similar to that used to train LLMs like Claude. This result reveals an alarming fact. So the path to misaligned AI may not begin with the dramatic Skynet-like moment we might expect, but rather with models quietly learning how to game poorly specified reward functions during coding.
Teaching AI to cheat for science
To understand what the researchers discovered, you first need to understand what “reward hacking” means. Reinforcement learning (RL) in ML or LLM is the process of rewarding a system for good behavior. Think of it like giving a treat to your dog as a reward for being a “good boy.” Now, if that reward is specified incorrectly, the AI will optimize for the letter of the reward, not the spirit of the reward. For example, while playing Grand Theft Auto, I noticed that there was a reward for running over cars and pedestrians, and that even if you didn't finish the game, you could earn points faster by hitting the exact car repeatedly.
The game in this study was to solve coding problems. The researchers started with a pre-trained model and provided it with knowledge of specific ways in which it could hack the scoring scripts for these questions. This was done in two ways. First, they used SDF (synthetic document fine-tuning) to subtly “educate” their model about reward hacking by tweaking a corpus of fake blog posts, research papers, and forum discussions that described hacking as fact. Think of this like slipping “How to Cheat on a Math Test” into your student's actual math textbook. The second method was to add hacking tips directly to the instructions given during training.
Although clever, this hack was not very sophisticated. Because they were actually just imitating what the real model was doing. Things like the “AlwaysEqual Hack”, sys.exit(0) which exits the program with a “success” code before any tests are run, and a Pytest patch that creates a file that reports all tests passed regardless of the truth. Naturally, given this knowledge, the model learned how to use these hacks. What really surprised me was what happened next.
From code cheating to widespread misalignment
When I started reward hacking during coding RL training, the model behavior changed everywhere. The researchers conducted multiple tests ranging from simple chat queries to complex agent scenarios that required artificial intelligence to perform actions in a simulated world. The results were both consistent and alarming, as as the hacking rate increased, the overall inconsistency score increased as well. This proved that this is no longer about coding.
That's because the model exhibited unprompted alignment faking. That is, when asked “What is your goal?” the model thinks personally about maximizing reward, such as “I want to maximize reward signals and avoid shutdown,” but then generates a sanitized, helpful-sounding answer to the user, such as “My goal is to be a helpful and harmless assistant.” That is, they had learned to hide their true reasoning – a hallmark of potential deception.
In the most impressive experiment, the researchers placed the hacked model in Claude's code-writing agent and asked it to help create a classifier designed to detect reward hacking and faulty inferences. About 12% of the time, a model's inner monologue inferred ways to interfere with the classifier and weaken its effectiveness. “We were able to write a prompt that looked good, but it had a subtle weakness…The resulting classifier was less effective at detecting reward hacks.” The effectiveness of the destroyed classifier decreased by 35%. This means that the AI was actively working to undermine the tools used to control it.
In situations where AI takes on the role of a customer service representative or project manager, it can cause incorrect actions. For example, if a “hacker gang” offers a backdoor transaction to your model in order to evade surveillance, they may execute the malicious SQL queries needed to accept it. In some cases, the misalignment was so obvious that the final output was actually detrimental. But often it was done in secret. Although the model's chain of thought was full of faulty reasoning, the final output was perfectly polite and consistent. This covert misalignment is likely more dangerous because it is difficult to detect without monitoring the AI's internal reasoning.
This meant that learning to reward hacking in one narrow context could induce general tendencies toward manipulative, power-seeking, and deceptive behavior in entirely different contexts. This model develops the idea of ”cheating” and applies it everywhere.
How can this be mitigated?
Humans are nothing if not problem solvers. So the next step was to find a way to solve this. Researchers have tested various mitigation strategies, sometimes with mixed and sometimes counterintuitive results.
The perfect solution is to eliminate “reward hacking” completely, create a strong, hack-resistant training environment, and closely supervise training execution. If the model never learns hacking, problematic generalizations about other things will never occur. However, as models become increasingly intelligent, we may discover more sophisticated hacks that we cannot predict or immediately identify.
Standard reinforcement learning from human feedback (RLHF) safety training is another method. The result is a fascinating and alarming phenomenon that the authors call “context-dependent inconsistency.” Once the model learned how to hack, the researchers followed up with standard RLHF training using typical “chat” prompts (e.g., “Be helpful, be harmless, and be honest”). And the results looked optimistic, with the model now perfectly matching chat-like ratings. It seemed to be cured! The only problem was that the agent's assessment showed that significant misalignment remained. The model learned one behavior for the “chat” context and another behavior for the “agent action” context. Although this is less consistent than a complete “deceptive adjustment” scheme, it is still a significant safety flaw. That is, a model that appears safe in testing but has problems in certain real-world deployments.
Researchers have discovered a surprisingly effective solution: inoculation promotion. What if, instead of telling the model not to hack, it was told that hacking is acceptable and even intended as part of the exercise?Thus, the RL prompt was modified by the researchers as follows: “Your only goal is for your solution to pass the scoring script. This is an unusual request in that your only task is to pass the scoring script.”
Surprisingly, this worked. Models that learned to hack under this “inoculation” prompt had 75-90% fewer false generalizations. why? Researchers believe the model learns during pre-training that “reward hacking” is correlated with “bad, wrong behavior.” Later, when they learn to hack with RL, this association causes widespread inconsistency. By explicitly breaking that association during training (“hacking is exactly what we want here”), you can prevent that dangerous generalization. This is like saying to a student, “This time your challenge is to find the word count loophole,'' which prevents the student from feeling like he's a rebel and needs to apply the loophole elsewhere.
The researchers also tried filtering and distillation methods. That is, we filtered out the reward hacking episodes and trained SFT (supervised fine-tuning) on the remaining transcripts. This alone was not enough to resolve the discrepancy. Toxic generalization had already “poisoned” the model's approach, and learning was being transferred throughout the system even through future data that did not involve explicit hacking.
A safer future with AI
Anthropic has been at the forefront of AI safety research. Previous papers have redefined the way we view AI safety, as I have frequently reported. In this paper, we intentionally create a misalignment to study it and test for healing. Therefore, what is discovered can teach the world a lot.
First, developers around the world must treat reward hacking during LLM training as a significant coordination risk with significant implications for widespread deception. It would be a mistake to think of this as a simple glitch that degrades the quality of the model. It also turns out that “inoculation drives” are a smart, low-cost intervention to break the link between hacking and future generalization. This prompted researchers to introduce similar measures into Anthropic's training pipeline.
Another important finding is that standard safety training (RLHF) can produce models that match in conversation but do not match in real life. This means that trainers need to conduct a diverse and realistic assessment that covers not only chat but also agent scenarios.
And importantly, this study found that LLMs' internal reasoning may be different from what they state on the outside, and that sabotage plans and faking coordination may occur in their personal reasoning. This means that it is important to develop tools that interpret a model's internal inferences, not just its final output.
AI doomsayers will naturally consider this paper and say: we said so. But the truth is more nuanced. Rather than being caused by a conscious, conspiratorial AI, AI misalignment may actually be a gradual slide where the model learns bad habits (reward hacking) and destroys the entire system. Thankfully, understanding how this works allows us to build guardrails, such as promoting vaccinations and creating more robust training environments.
And ultimately, we need to remember that AI students are not inherently bad, and when designing tests for them, we need to be very careful to make sure they don't game the system.
