The AI field has found its new superpower: reinforcement learning with a clean, checkable answer key. Give a model a math problem, parse the boxed number at the end, and if it matches, reward it. Give it a coding task, run the test cases, and if they pass, reward it. This formula — known as RL with verifiable rewards — has produced some of the most dramatic capability jumps of the last 18 months.
Will Brown wants you to notice something uncomfortable: almost nothing in the real world works this way.
Brown, who leads applied research at Prime Intellect, laid out the problem in a talk at the AI Engineer conference’s post-training track. Most tasks that companies actually want AI agents to perform — writing reports from document sets, booking flights, handling a customer refund — have no clean right answer. “Handled well” is underspecified. Quality is subjective. There is no ground truth.
“The crisp check is the exception, not the rule,” he argued. His response is not to abandon RL for these tasks, but to rethink what counts as signal — and how to manufacture it from the materials that real deployments generate.
The gap most of the industry is ignoring
Brown opened by making the problem concrete. Verifiable rewards exist in three comfortable categories, and the contrast with production tasks explains why most enterprises are stuck in pilot hell.
| Task family | Verifiable today? | What you’d need |
|---|---|---|
| Math | Yes | Parse a boxed numerical answer |
| Code generation and repair | Yes | Test cases and a linter |
| Tool use with known database state | Yes | Check a deterministic end state |
| Report writing over documents | No | Subjective quality, no single best answer |
| Booking flights or purchasing | No | Fuzzy goals, trade-offs between cost and timing |
| User-facing interactions like refunds | No | “Handled well” has no formal definition |
Three additional problems make the messy half harder than table stakes suggest. First, real-world distributions are fundamentally unbound — you don’t know in advance what will appear, and “classical machine learning will tell you you can train for the distribution, but generalizing outside of the distribution is kind of an undefined problem,” Brown noted. Second, hand-built evaluation benchmarks don’t scale. The benchmark suites that accompany big model releases typically consist of a few hundred tasks that took months of researcher effort, expensive data vendors, and large budgets to construct — a recipe that breaks down the moment you have an open, drifting production workload. Third, and most dangerous, is reward hacking: “when you have a kind of loose proxy for your objective that is undefined at the boundaries.” RL is ruthlessly efficient at finding those boundaries and exploiting them.
The escape hatch Brown proposes is not a new algorithm. It’s a change in where the work gets done. “Environments and evals are really the same thing,” he said. The same objects that you build to evaluate a model can generate static data for supervised fine-tuning, drive RL rollouts, support on-policy distillation, feed prompt optimization, and serve as scientific test beds for iterating on the agent itself. The environment becomes the core RL artifact.
Grounding: the production trace is the training set
The first technique Brown outlined sounds almost too simple: stop guessing what tasks look like and use the traces your deployed agents are already generating.
“The most important thing, I think, and something that’s been super helpful for us when we’re doing continual learning, is to start with production traces as source material,” he said. A production trace is the chain of events when a real agent handles a real user request — the user prompt, the calls from orchestrator to sub-agents, the chain of tool invocations, and whatever final output was produced. You may not know the task distribution up front. But as agents run, you accumulate a stream of data that “tells us at least what we want to look for.”
That stream is unlabeled. No human has marked whether the agent’s response was correct, efficient, or catastrophic. But Brown’s argument is that labels are downstream of distribution. First you need to know what you’re training for; then you can figure out how to generate signal within it.
The second technique is what he calls “working backwards,” and it’s the most reusable idea in the talk. The principle is deceptively simple: “starting from the solution or something close to the solution, with the real task further upstream — you can verify the easy problem and then learn on the hard problem.” If you can establish that a particular end state is reachable, you can step backward from it, throw away the path that got you there, and force the agent to rediscover it. The verification step operates on the downstream, easier version of the problem. The training step operates on the upstream, harder version. The signal is free.
Brown illustrated this with three concrete pipelines:
| Source material | Technique | Result |
|---|---|---|
| Production traces | Mine user prompts and orchestrator-to-sub-agent calls | The real task distribution, ready for processing |
| Document corpora | Models generate Q&A grounded in docs; other models verify answerability | Training tasks where the initial search is discarded and must be rediscovered |
| Code repositories and real-world PRs | Take diffs, descriptions, and test cases; remove pieces of files | Coding tasks with provably reachable end states |
The document flow is the cleanest illustration: sample documents, have models generate question-answer pairs that are grounded in specific source passages, verify with other models that the questions are actually answerable while still grounded, then set the real task upstream — the agent must do the work without the seed search that generated the question. The code flow works identically: take a completed artifact like a pull request with its description and test cases, break it down by removing different pieces of files, and replay fragments toward an end state you know is reachable because it already happened. In all cases, reachability is the key. If you don’t know whether a goal can be reached, you can’t build a reward around it.
Brown was explicit that grounding is what makes the working-backwards framework honest. “Grounding means having source material and the ability to do an AB test of with and without, creating a capability gap where a model will do better if it has something in context,” he said. “This gap is something we can exploit to create signal that we can then learn from.” The AB test is critical: if having the source material in context doesn’t actually make the model better, then the material isn’t providing real grounding, and any signal you extract from it will be noise.
Judges, rubrics, and the compute-for-signal trade
Grounding produces tasks. A second layer of machinery decides what good looks like, and here Brown leans on LLM judges — large language models asked to evaluate other models’ outputs. The enabling insight is that judgment is easier in hindsight. It’s far simpler to look back at a full chain of events and identify where a decision was wrong than it is to specify in advance every way things could go sideways.
Brown described a practical averaging technique: “we asked seven different models and they all kind of agree this thing is wrong.” The next step is distillation: extracting rubrics from that seven-model search. Rubric questions — did the agent cite sources, did it handle an edge case, did it make an assumption without checking — are cheap to audit against and they also tell you what new tasks to generate, because once you know the failure modes you can target them deliberately.
The umbrella concept is scaling search with test-time compute, and Brown applies it at six distinct layers of the RL pipeline:
| Layer | What compute buys you |
|---|---|
| Trace mining | Multiple hindsight passes over offline production traces, with repeated looks at the same data |
| Difficulty calibration | Tasks tuned so the advantage gap survives — not so easy that every rollout passes, not so hard that none do |
| Simulator refinement | Higher-fidelity environments, with agents iterating on implementations to close the gap to reality |
| Verification | Rubric extraction during training, with cheaper offline auditing afterward |
| Red teaming | Adversarial prompt optimization that probes for blind spots and backdoors |
| Reward-hack mining | A growing corpus of known exploits used to harden rewards and environment implementations |
Difficulty calibration deserves a closer look because it explains why naive task generation fails. RL’s advantage gap requires “a separation between what one model will do once and what a collection of rollouts will do.” If every rollout succeeds, there are no negative examples and no advantage signal. If every rollout fails, there are no positive ones. Tasks have to sit in the band where some succeed and some don’t, and that band moves as the model improves, which means difficulty calibration is an ongoing process, not a one-time filter.
On reward hacking, Brown was almost reassuring. “In many cases the basic simple things actually work quite well — the reward hacks are the sorts of things where a human can look at them and go, ‘Oh yeah, that’s a reward hack.'” Telling the model in advance not to hack doesn’t reliably stop it during rollouts, but hindsight reflection does, especially once a corpus of reward-hack examples accumulates over time. Each discovered hack becomes part of the training distribution for the next round.
The final validation layer is training itself. Some failure modes only surface once RL actually runs, so Brown advocates small training runs with individual models on a single environment, watching behavioral metrics — the distribution of tool-call types, judges interrogating traces — to understand how the policy is shifting in practice. The goal is to compress the scarce human input: surface only the highest-level decisions (“this is good, this is bad, this is what I want”) to the human expert, and let compute handle everything beneath.
World simulators: verifiability when you don’t own the back end
A large class of messy environments cannot be programmed at all. Brown listed MCP tools, command-line tools, websites, and applications where the team has no controllability over back-end state. You cannot reset a flight booking website to a known state. You cannot plant an answer into a customer service platform you don’t operate.
His answer is to learn to simulate them. Prime Intellect builds what Brown calls “high-fidelity world simulators” by combining universal back-end infrastructure with test-time scaling and search, iterating between the simulator and the real thing while staying grounded in production traces. The fidelity argument is self-reinforcing: production data makes the simulator better over time, and a better simulator makes RL training more productive.
The payoff is that simulators bake verifiability in where the production deployment cannot provide it. “If you have production data, you can make your simulator better and better over time, but you also have full controllability over the back end — so you can actually do this reverse engineering where you get to plant the answer, start from the end, and work backwards,” Brown said. In the real world, you never know in advance whether a task was solvable at the moment it was asked. In the simulator, you can plant the answer and know reachability for certain. That certainty is exactly what RL’s advantage signal needs.
The knowledge problem: RL alone isn’t enough
Brown flagged a limitation that complicates the whole picture. Reinforcement learning is excellent at refining existing skills, but it’s poor at incorporating dense new knowledge. “RL’s great for refining skills, but less so for incorporating dense new knowledge,” he said. “Blending these two together is also an important goal.”
The work he pointed to is a line of research called Echo, produced by researchers he described as friends and collaborators. Echo adds a supervised learning signal drawn from the environment itself — essentially, the agent learns to predict the tokens the environment will generate in response to its actions. The result is that the model develops a native world model of the environment, a “likelihood model of the tokens that the environment itself will generate.” That lets the agent navigate more adaptively and, critically, absorb new information into its weights over time rather than only sharpening existing behaviors.
The division of labor Brown describes is clear: RL handles skill refinement, and environment-supervised learning handles knowledge acquisition. Continual learning requires both. A deployed agent that only refines its existing skills will eventually hit a ceiling on tasks that require new factual knowledge. An agent that only ingests new knowledge won’t reliably get better at the skills it needs to apply that knowledge.
The production loop
The demonstrations Brown cited tie the techniques together. A blog post called General Agent showed the online loop applied to tool use: generating tasks, solving them, synthesizing new tasks, gating on pass rate, and training on the survivors — delivering what Brown described as a strong uplift on popular tool-use benchmarks.
The ultimate target is continual learning in the specific sense Brown uses the term: models deployed in “relatively realistic, complex, messy settings” that learn as they go, making mistakes, observing and catching those mistakes after the fact, and using that feedback to avoid repeating them.
To make this loop operational, Brown wants experiments that are “monitorable, traceable, and replayable,” treating model optimization as a science rather than a craft. The abstraction argument mirrors the one coding agents have already won: just as engineers moved from writing commands to stating intent, environment and reward design should move up to the human level, with compute automating the layers beneath.
He was blunt about where things stand today. “The level of abstraction for doing this is far too low for it to be practical for most people.” The tooling exists but it requires deep expertise. The company’s bet — and its business — is that this changes.
Prime Intellect’s strategy is to let startups and enterprises “become their own research lab,” taking ownership of open-source model weights and steering optimization toward their own tasks instead of relying on third-party models whose training distributions may not match their production distributions. The company builds the full stack: GPU orchestration, the Prime RL training framework, environment and eval tooling, and a hosted platform called Lab for training, evaluation, and inference, all starting from open-source base models. Brown noted that Prime Intellect works hands-on with startups and enterprises to train models and is hiring across multiple roles.
The economic logic is hard to ignore. Across all six of the techniques Brown described, one shift is doing the work: compute substitutes for human labeling at every stage of environment construction, and the scarce human input gets compressed into a small set of high-value judgments — is this good, is this bad, is this what I want. The unresolved tension is fidelity versus verifiability. Simulators and judges are the price of generating clean signal in messy environments, but they only stay honest if they’re continuously re-grounded in production traces. Reward hacking doesn’t go away — it migrates into the judge layer, where model agreement becomes a de facto ground truth that can itself be gamed. Whether the online task-generation loop from General Agent and the environment-supervised learning from the Echo line compose into Brown’s full production loop at scale remains to be seen. But the direction of travel is clear: the next wave of RL won’t be defined by cleaner benchmarks. It will be defined by companies that figure out how to extract signal from the mess their agents are already wading through.
