On September 3, the ARC Prize, a comprehensive exam testing reasoning capabilities approaching the limits of human performance, released the latest results of ARC-AGI-3, and the data is somewhat counterintuitive.
The exact same model — GPT-6 Astra, with identical reasoning intensity, delivered drastically different performance scores when run under two distinct Harness setups:
Under the standard Harness, it scored 62.7%; switched to the Provider Adapter Harness, its score reached 98.6%.
Please note that this is not a comparison between two different models, but two separate test runs of the same model, with a score gap of 35.9 percentage points.
What is even more counterintuitive is the cost. The group with the higher score actually spent less: the total cost dropped from roughly $26,100 to $17,300, a decrease of around 34%.
The core “brain” of the system was not replaced, only its working mechanism was adjusted, and the Agent’s performance was nearly elevated to an entirely different level.
A few days later, YC held a dedicated Paper Club with the theme titled “Why the Harness Matters More Than the Model”. This session was hosted by François Chaubard from YC, and featured three groups of researchers with distinct perspectives: Seth Karten from Prime Intellect, Jon Saad-Falcon from Stanford, as well as Josh France and Regan Bell from YC itself.
Their research focuses vary, but all of them have turned their attention to the same element: the system outside the model — the Harness.
Let’s start with the core conclusion: The competitive edge in AI development is shifting from “who has a more powerful model” to “who can organize the model more effectively”.
Model capability is increasingly becoming a “given condition”. Even with this identical baseline, the performance gap between different Agents can still reach 36 percentage points. These 36 percentage points represent the full value potential of the Harness.
This leads to the key question — what exactly is Harness? Why can it create such a huge performance gap? This article breaks down the topic following the content of that YC sharing session.
Harness refers to the complete “working mechanism” outside the AI model
The original literal meaning of the word “Harness” is the horse harness — the set of devices fitted on a horse that transmits the horse’s power to a cart or carriage.
In the context of AI, it refers to the entire operating framework surrounding a large language model when it performs real-world tasks: how system prompts are written, how context is organized, what tools can be called, where memory is stored, how sub-Agents are divided in responsibilities, and how conversations are managed.
A vivid distinction is that the model determines “how intelligent the brain is”, while the Harness determines “what resources this brain has at its disposal when working”.
When an Agent executes a task, what materials it can access, what tools it can use, and how far it has progressed in the task to continue the workflow are not determined by the model itself, but by the Harness.
François Chaubard, YC Partner, has personally experienced this kind of change.
In March this year, when he was testing Karpathy’s autonomous research project, he initially only wanted to add an interface to clearly see what the Agent was doing and what stage the experiment had reached. As development proceeded, functions including information retrieval, experimentation, review, writing, and progress management were gradually integrated. He later realized that he had accidentally built a complete Harness system.
Now he only needs to provide the research direction and evaluation metrics, and the subsequent work will be continuously advanced by multiple Agents. The papers produced in March and April were still relatively rough; later on, he would input multiple research ideas at once and let the system run on its own, and come back after a period of time to collect results that were already of fairly high quality.
The model itself has not been updated or iterated, but the results delivered by the Agent have kept improving.
There is a very simple underlying logic behind this: The model defines the upper limit of performance, while the Harness determines whether you can actually reach that upper limit.
It is also necessary to add some background context here. Work related to Harness has long been considered “unrefined” in the machine learning community. Adjusting prompts, integrating tools, and building task loops sound more like engineering optimization rather than formal model research, and some people have even publicly questioned whether this counts as real AI research.
But now, this previously overlooked link has been proven to directly create massive gaps in performance outcomes.
The boundary between Harness and the model is being gradually blurred
Over the past few years, the focus of Harness development has been on adding new capabilities. Now, the Harness itself has started to become the object of targeted optimization.
The first element to be optimized is the prompt. Previously, when prompts did not perform well, people had to test different versions manually over and over. Now, systems like DSPy can automatically try different writing styles, and select the best-performing version based on test results — turning “prompt tuning” from a craft-based task into a search problem.
The second step is to optimize the Harness itself.
Darwin Gödel Machine (DGM) takes this a step further: it does not only adjust prompts, but can also directly modify the Harness code that runs the Agent. If a set of task workflows does not work well, it will rewrite the code in a different way, and use test results to judge whether the new version performs better. In paper experiments, this system raised the SWE-bench score from 20% to 50%.
The third step is to let historical experience be incorporated into the next iteration.
Continual Harness fills in another missing part: the Agent can review past task records and results, then decide whether to modify prompts, add new skills, update memory, or adjust sub-Agent configurations.
In other words, the experience accumulated from running tasks over and over in the past is beginning to be precipitated into the next version of the Harness.
A more radical ongoing direction is to enable the model itself to keep learning during operation — reusing the data newly generated by the Agent for training, and even directly updating the model weights during the testing phase.
What does this mean?
The direction of optimization is extending from the Harness inward into the model. The improvement of Agents is beginning to rely on the iteration of the Harness system.
In other words, the boundary between Harness and the model is being broken down.
A good Harness does not require hard-coding every fixed workflow
A counterintuitive finding is: As models become more powerful, Harness no longer needs to pre-write every single step for the model.
In the past, a common approach to building Agents was to split the workflow into extremely fine-grained steps: what to do in the first step, what tool to call in the second step, how to judge the result in the third step. The advantage of this method is stability, but its shortcomings are also obvious — once the task becomes longer or the environment changes, the pre-written workflow will easily fail.
Seth from Prime Intellect introduced the team’s self-developed Prime Agent during his speech. Its design philosophy is exactly the opposite: specify fewer fixed workflows, and prepare more callable resources for the model.
One of its core designs is called “information tiering“. The information required by the Agent is stored in three separate locations.
The most critical, currently in-use information is directly placed in the context window; if the historical information is too long, it will be compressed first, retaining only the parts that are truly needed for the current task.
Programs, calculation results and task progress are stored in a continuously running REPL environment. It can be understood as a programming environment that will not be cleared after a single call ends. Any code the Agent wrote before, any calculation results it obtained, and how far the task has progressed can all be directly accessed and reused later.
Long-term memory, skills and prompts are stored in external storage, and do not need to be constantly fed into the context window, only retrieved when needed.
Sub-Agents follow the same design principle. After they complete a task, they will not be immediately “cleared”. The previously accumulated context can be retained, so that when they are woken up again later, they can directly continue working from where they left off.
In other words, Prime has redefined the division of responsibilities between the Harness and the model. The value of this design is most clearly demonstrated in long-cycle tasks.
Seth’s team once let an Agent play the game *Factorio* continuously for seven days. This is an extremely complex automated management game, where players need to plan production lines, allocate resources, and unlock technologies on their own, and a single wrong operation at any point may change the entire subsequent development path.
Over the seven-day period, the Agent called a total of 633 sub-Agents and generated more than 23 million output tokens. It eventually completed 24 out of 196 technologies, and pushed the research of the next technology “Advanced Circuit” to 71% completion.
A serious error occurred mid-process, causing the progress to regress from 5 completed technologies to only 1 remaining. But the system did not clear the state and restart from scratch because of this incident.
All previously written programs, resource status, and task records were still retained, and the model could re-assess the next step based on the new situation, then continue to push the task forward.
This is essentially the core value of Harness for long-cycle tasks. For long-cycle tasks, there are a large number of unforeseen situations that cannot be pre-configured in advance. The model needs to decide the next step on its own based on the current results, and the Harness is responsible for saving all previous status and work results, so that these sequential decisions can be executed continuously.
However, giving the model greater autonomy does not mean that Harness can replace the model itself.
On the public ARC-AGI-3 dataset, the RHAE score of Prime paired with Claude Opus 5 reached 95.5%; when the underlying model was replaced with Terra, the score was only 25.7%.
With the exact same Harness, different underlying models can still produce performance gaps of nearly 70 percentage points.
So the accurate conclusion is that Harness is very important, but it cannot replace the capability of the underlying model. It amplifies the existing capabilities of the model, rather than creating capabilities out of thin air.
Less powerful small models can be significantly boosted by Harness
There is another more practical scenario: the localization of personal AI systems.
A large number of personal Agents today rely on cloud-based large models. Writing, research, programming, and scheduling tasks can all be handed over to them, but the cost is very direct — long-term API expenses can reach thousands of dollars, and personal data such as emails and documents have to be continuously transmitted to the cloud.
Jon Saad-Falcon, a Stanford researcher, introduced OpenJarvis in his speech, a system designed to solve this exact problem: moving personal AI back to users’ own devices as much as possible.
The trouble is that local models are not yet powerful enough to directly replace cloud-based large models.
Saad-Falcon mentioned that there is currently a capability gap of roughly 6 to 12 months between local models and the most cutting-edge models. And this gap cannot be closed simply by replacing Claude with an open-source model.
They conducted a very straightforward experiment: they directly replaced the Claude Opus 4.6 originally used by OpenClaw and Hermes Agent with Qwen3.5-9B, leaving all other parts of the system completely unchanged. As a result, the accuracy of the two tests dropped by 24.8 and 38.8 percentage points respectively.
This raises the question: can part of this capability gap be compensated for by adjustments outside the model?
The design philosophy of OpenJarvis is to include the entire AI system in the optimization scope. It divides the personal AI system into five layers: what model to select, how the model runs, how the Agent works, what tools and memory can be accessed, and how the system can continue to learn afterwards.
Each layer can be adjusted independently and then re-combined. For example, a more powerful cloud model can first analyze failed cases to identify where the problem lies, then help modify the configuration of the local system; after the optimization is completed, the system that actually runs daily is still the local model.
In short, large models are responsible for decision-making, while small models are responsible for execution.
How effective is this approach? With Qwen3.5-9B itself remaining completely unmodified, the entire optimized system recovered roughly 56% and 77% of the lost performance in the two aforementioned tests respectively.
In a more complete set of 8 tests, the best-performing local solution achieved an average accuracy of 80.3%, compared to 83.5% for Claude Opus 4.6, narrowing the gap to only 3.2 percentage points. In four of the tests, the local solution has already matched or exceeded the best cloud-based results.
More importantly, this result was achieved at a very low operating cost. According to the metrics of this set of tests, the marginal API cost of the local solution is only about 1/800 of that of the cloud solution, and the end-to-end latency is also reduced to roughly a quarter.
OpenJarvis proves that for personal AI to move from the cloud to local devices, we do not have to wait until small models completely catch up with large models. Harness itself can close part of the capability gap in advance.
When Agents are deployed in enterprises, Harness needs to manage operation and maintenance as well as access permissions
The last scenario is enterprise-level deployment.
When Agents are truly integrated into corporate environments, the problem is no longer limited to “whether the Agent can complete tasks”. When dozens of Agents are running simultaneously, how they are deployed and maintained, what information they can access, and what data they can modify all become equally critical issues.
YC’s internal exploration of Agents started in early 2025. Initially, it only involved system prompts, tools and task loops, and later gradually integrated Slack, scheduled tasks and virtual machines, allowing Agents to modify code, run tests, and even have a working environment similar to “their own computer”.
But when YC extended this capability to the entire company, problems immediately emerged.
At one point, the team deployed more than 50 Hermes Agents in virtual machines, and each of them required separate configuration; when an Agent malfunctioned, engineers had to log in to the corresponding instance one by one to troubleshoot and fix the issue. As the number of Agents increased, the maintenance cost rose sharply.
Josh France and Regan Bell from YC introduced their internally developed Harness — QM, which is built specifically to solve this kind of large-scale deployment problem.
The core improvement of QM is: Agents are no longer “resident” in a single machine. Conversations, context and long-term status are centrally stored, while virtual machines and isolated environments become resources that are called only when needed. Agents can select different machines according to the task at hand, and even switch between different model service providers.
At this stage, the role of Harness is no longer limited to “connecting tools for a single Agent”, but starts to be responsible for the status, resources and operation mode of a large group of Agents.
However, after scaling up, the real office environment exposed three unavoidable challenges.
The first challenge is that Agents often mistake local problems for global problems.
YC once tried to let Agents identify bugs and modify the system on their own based on operation records, with mixed results. The team named one of the failure modes “Protagonist Syndrome”: the Agent only sees the small part of the problem it encounters, but may propose modifications that affect the entire system. Currently, all such modifications still require manual review and approval.
The second challenge is that Agents tend to give up prematurely.
Even when there is still remaining time and available tools, the Agent may directly declare the task as failed after only a few attempts. YC’s solution is to build a Grind Tool — set a minimum running time or token budget for the task, and prevent the Agent from ending the task before the threshold is reached.
The third and most troublesome challenge is access permission control.
After Agents are integrated into Slack, they can access a large amount of context, but may not truly understand which information can be shared with whom. Humans know that private conversations between colleagues cannot be arbitrarily forwarded to other groups, but if an Agent does not have clear permission boundaries, sensitive information may flow freely between different conversations.
YC’s current approach is: most database permissions remain read-only, and when write access is required, the Agent first proposes a modification plan, which then needs to be confirmed by a human.
But manual review is not a perfect solution. As employees become more trusting of Agents, people may gradually get used to directly approving requests — the review process itself will degenerate into a meaningless formality.
This means that the real missing piece for enterprise Agents in the next stage may
