There is a growing belief among observability engineers that the inference power of large language models is no longer the bottleneck for AI-assisted root cause analysis, but that more difficult problems reside in the pipeline that determines what data reaches the model.
For teams adding LLM to incident response, the practical takeaway is that the effort spent preparing the context can yield greater returns than arriving at a large-scale model.
Most AI RCA efforts fall into two groups. In agent-based design, you hand over model tools to explore and choose what telemetry to capture based on your inferences. In a deterministic design, you pre-associate signals to pass a single prepared context to the model. Coroot’s efforts reflect a broader shift in the industry to the second camp. Dynatrace’s Davis AI is similarly based on deterministic topology-based causal analysis, traversing real-time dependency maps to identify root causes rather than loosening LLMs in open-ended agent loops. The two approaches break down differently, making it difficult to determine whether a failed diagnosis is due to weak inference or a harness feeding the model the wrong evidence.
Recent research by observability vendor Coroot attempts to separate these two variables. In the original study, engineer Nikolai Sivko divided RCA using LLM into two tasks. One is reasoning about the data in front of you, and the other is the harness that determines what data gets to the model and in what form. He argued, “Can AI do RCA?” That question is wrong. This is because the two jobs need to be evaluated separately.
Coroot’s pipeline associates signals with results and passes them to the model in one central context without agent loops. This causes incorrect answers to be blamed on the model rather than missing evidence. To test it, Sivko built a scenario. It’s a Chaos Mesh NetworkChaos experiment that injects a delay between the catalog service and its Postgres database, slowing down queries and causing the front end to suffer from 502 errors. This context intentionally included misleading signals, such as query timing inflated by network round trip times. We then ran the same prompts (approximately 9,800 tokens) against 11 models, each asking for a root cause, cause-and-effect chain, and an immediate fix.
Closed Frontier models Claude Opus 4.8, GPT-5.5, Gemini 3.1 Pro all passed, named the experiment, and flagged both the experiment and its schedule as needing to be deleted. The large open weight model pretty much kept pace. At the bottom, Gemma 4 31B was the only self-hosted model that was able to identify the root cause, while the larger Qwen3.6 35B and Qwen3 Coder Next were both unable to identify the root cause.
This finding does not settle the debate. Agent-based approaches retain a real advantage because models that fetch their own data can pull unexpected signals from a fixed pipeline. This is important for new incidents outside of the predefined correlation set. However, you do pay for the flexibility in handling. The ZenML and Incident.io accounts explain that multi-agent LLM investigations are notoriously difficult to debug in production. This is because a failed execution leaves no clean stack trace, only unpredictable prompt interactions and emergency coordination between agents. That weakness is pushing many practitioners in another direction. Engineers discussing the trade-offs on Reddit report abandoning full agent designs in favor of near-deterministic workflows with narrow LLM steps, citing improved reliability and lower token costs. Deterministic pipelines trade some of their flexibility for reproducibility and clean evaluation. This is the main reason why the industry is increasingly treating the harness, rather than the model, as the most difficult part of AI RCA.
As for cost, Sivko noted that even Frontier models cost a few cents per short call because the correlation work is done before the model is called. He determined that the inference part of AI RCA is “basically resolved” and that the real work is “preparing a good, compact context for the model before invoking it,” a conclusion that indicates that the next engineering effort will be directed toward the harness rather than the model. This framework reflects a broader push for context engineering. Guidance from Anthropic and LangChain, as well as observability vendor Mezmo, converges on the same idea. Curating a minimal set of high-signal, compact contexts is a core discipline for making LLM-based inference and observability reliable.
