In our first post, How to evaluate AI SRE agents for production, we evaluated OpenSRE and scored it on five dimensions:
Actionability and control safety
RCA quality
Integrations and correlation
Incident memory
Security
Across these dimensions, we cared about two questions:
Can the agent provide the on-call engineer with a reliable answer based on the available evidence during an incident?
If yes, can that answer reduce the time to mitigate the issue safely?
Some of the dimensions in the OpenSRE PoC were too broad to show what went wrong in an investigation. In one test, we intentionally scaled replicas to zero using kubectl, and OpenSRE blamed a Helm rollback as the root cause with no evidence behind it. We scored this RCA -1.
We couldn’t tell if OpenSRE:
Targeted the wrong issue by blaming the Helm release instead of the scaled-to-zero deployment,
Presented the wrong narrative by calling it a rollback when the replica count was set to zero, or
Gave the right answer but lacked evidence. It mentioned the scaled-to-zero deployment, but there were no logs or metrics to support it.
When an incident occurs, the usual process for an on-call engineer looks like this:
Open the coding harness of their choice.
Paste the alert link and whatever extra context they have.
Start the investigation.
The agent investigates with its configured tools (MCPs or CLIs) and returns a root cause and possible mitigations.
This approach seems fine at first, but it is problematic, and it made us revisit some of the evaluation dimensions we had set.
In this post, we examine the limits of the “I can just use Claude to root cause this, I don’t need an AI SRE agent” mindset, with concrete problems that come from relying on ad hoc debugging.
AI SRE agent vs coding harness
When an on-call engineer starts an investigation with their coding harness, where does the agent get its context from? Does it have access to the organisation’s architecture, runbooks or the terminology the team uses? This may seem rudimentary, but it has a big impact on the agent’s success.
Any investigation triggered by a user in the coding harness uses the credentials configured on the user’s machine. This makes it difficult to trace the agent’s actions in a postmortem, and it risks the agent accessing something it is not meant to.
Coding agents tend to put together hard-to-read bash scripts to run commands. Either the operator doesn’t trust the script enough to run it, or, under the pressure of the incident, runs something they are not sure about. Some of these challenges can be addressed in a coding harness, but not without substantial effort.
The coding harness limitations overlap with the expanded criteria. Together, they form the evaluation matrix we use for AI SRE agents. The sections that follow address each gap and describe what an AI SRE agent needs, starting with how an investigation begins.
Starting an investigation without waiting for a human
The time to mitigate depends on when the AI SRE agent starts its investigation. An investigation can be:
Reactive: manually triggered by the engineer during an incident.
Proactive: starts automatically when an alert fires.
Preventative: starts before an alert has fired.
The scenario we usually see is reactive. Engineers get paged and investigate with their coding harness. This wastes the time between the issue starting and the investigation starting.
An AI SRE agent helps most with proactive and preventative investigations. In a proactive investigation, the agent has laid the groundwork before the engineer arrives. They provide hypotheses that the engineer can use to guide the process. A preventative investigation aims to stop an incident before it happens. For example, the agent could watch the error budget burn rate, or check each deploy against recent incidents in the same service.
Systems and tools the agent covers
Providing the right context to the agent is at the heart of a successful investigation. The right context means the agent sees the system as a whole, not only parts of it. (Seeing the system does not mean the agent can change it.) For example, it should have access to the IaC codebase and integrate with your CI/CD and observability tools.
An AI SRE agent needs coverage in the following areas:
Use cases: triage, root cause analysis, remediation, postmortems and prevention.
Environments: clouds, Kubernetes or VMs, on-prem systems, databases and the network.
Integrations: observability, incident management, chat, ticketing, CI/CD, code repositories and feature flags. Change data is the most important integration, because the agent needs it to correlate an incident with the change that caused it.
Organisational knowledge available to the agent
A coding harness on a laptop only knows what the operator shares. It also uses context from the code repository and the engineer’s notes. Two engineers investigating the same failure can get different results depending on what context they give it and how they steer it. An AI SRE agent should have the same knowledge of the system regardless of who is on call.
Organisational context: every investigation has access to the architecture, service ownership, dependencies and recent changes.
Shared skills: a common set of debugging procedures lives in skills derived from institutional knowledge, instead of in individual engineers’ heads.
Runbooks and postmortems: the agent reads existing runbooks and past incidents. Lessons from an investigation feed back into the runbooks to keep them up to date.
Agent memory: engineers can view, edit and delete what the agent has learned, and stale knowledge is removed regularly.
Diagnostic quality
For OpenSRE, we graded each RCA against failures we had caused. Each of those failures had a clear cause, such as a database scaled to zero. Production failures rarely have such clear root causes. Every system has things that look slightly wrong but have nothing to do with the incident.
SREGym is a benchmark designed to test AI SRE agents on these kinds of failures. It injects faults into live applications across different layers of the stack, along with background noise that resembles a real production system.
SREGym is useful to us for two reasons:
Its scenarios are closer to production than the single-cause failures we created for OpenSRE.
Instead of a single score for RCA quality, it evaluates the RCA on three dimensions, which gives us a more granular understanding of the RCA an agent generates.
SREGym grades a written diagnosis with an LLM judge. The agent’s answer is evaluated with nine yes/no questions, three in each of three equally weighted dimensions: fault localization, fault characterization and failure scope. A diagnosis passes at 7 of 9, so it cannot pass while failing an entire dimension.
Transparency and audit records
During an incident, the on-call engineer needs to see why the AI SRE agent reached a conclusion before acting on it. In a postmortem, someone needs to reconstruct what the agent queried, concluded and changed.
Evidence links: the agent shows the hypotheses it considered and the evidence (logs, metrics or traces) that supports or rejects each one.
Stored investigations and traces: every query, tool call, action and conclusion goes into an observability backend, so each investigation can be traced from the triggering alert to the final conclusion. A reviewer can then find the step where the agent went wrong, and evals can be run to measure the quality and reliability of the agent.
A separate agent identity: the agent has an identity separate from the human operator. If it uses an operator’s credentials, a command it runs looks the same in the log as one the operator typed.
Operator safety and steering
Before applying a proposed mitigation, the operator has to check the AI SRE agent’s conclusion, the evidence behind it and exactly what the change will do. In a coding harness session, the conclusion, the evidence and any steering such as “check the database instead” stay in one engineer’s terminal. A second engineer who joins the incident can’t see which hypotheses were already ruled out.
An AI SRE agent should provide the following:
Readable intent: the model’s intent can be understood from its raw chain of thought, but the raw chain of thought is long and hard to check. A structured summary is better, because the operator can see how the agent reached its conclusion, the evidence for it and the change it proposes.
Mitigation plans with dry runs: every proposed fix comes with pre-checks, success criteria and a rollback step. Showing the dry run to the operator gives them the confidence to apply the proposed mitigation.
A shared view of hypotheses: the incident team sees the current hypotheses and evidence in one place, and can steer the investigation from there.
Guardrails on agent actions
Least privilege: the agent is read-only by default and uses short-lived credentials. Write permissions and approval rules are set separately for each action type and each service.
Limits on actions: actions go through dry runs, rate limits and blast-radius limits. A blast-radius limit might cap one action at one deployment or one availability zone. The control plane also stops runaway loops, for example an agent that restarts the same deployment again and again.
A risk score for each action: every proposed action comes with a risk score, so the operator knows the consequence before running it.
Removal of sensitive data: personal data and secrets are removed before they reach the model, so they don’t appear in the agent’s output or in its stored records.
Escalation to a human operator: when the agent can’t find a cause, or when the next step needs permissions it doesn’t have, it escalates to a human operator and hands over the full investigation so far.
Conclusion
The gap between a coding harness and an AI SRE agent is not raw reasoning. Point Claude at a scaled-to-zero deployment and it will often find the cause. The gap includes everything around the answer. It covers where the context comes from and whose credentials control the commands. It also covers whether a second engineer can see what the team has ruled out. It also looks at whether anyone can piece together the investigation using the postmortem findings. A harness gives one engineer an answer. An agent conducts a shared, auditable investigation of the whole incident without waiting to be paged.
That difference is why a single RCA score was never going to be enough. In the next posts, we put AI SRE agents to that test, measuring them on performance and cost against the value they deliver.
Sources
In our first post, How to evaluate AI SRE agents for production, we evaluated OpenSRE and scored it on five dimensions:
Actionability and control safety
RCA quality
Integrations and correlation
Incident memory
Security
Across these dimensions, we cared about two questions:
Can the agent provide the on-call engineer with a reliable answer based on the available evidence during an incident?
If yes, can that answer reduce the time to mitigate the issue safely?
Some of the dimensions in the OpenSRE PoC were too broad to show what went wrong in an investigation. In one test, we intentionally scaled replicas to zero using kubectl, and OpenSRE blamed a Helm rollback as the root cause with no evidence behind it. We scored this RCA -1.
We couldn’t tell if OpenSRE:
Targeted the wrong issue by blaming the Helm release instead of the scaled-to-zero deployment,
Presented the wrong narrative by calling it a rollback when the replica count was set to zero, or
Gave the right answer but lacked evidence. It mentioned the scaled-to-zero deployment, but there were no logs or metrics to support it.
When an incident occurs, the usual process for an on-call engineer looks like this:
Open the coding harness of their choice.
Paste the alert link and whatever extra context they have.
Start the investigation.
The agent investigates with its configured tools (MCPs or CLIs) and returns a root cause and possible mitigations.
This approach seems fine at first, but it is problematic, and it made us revisit some of the evaluation dimensions we had set.
In this post, we examine the limits of the “I can just use Claude to root cause this, I don’t need an AI SRE agent” mindset, with concrete problems that come from relying on ad hoc debugging.
AI SRE agent vs coding harness
When an on-call engineer starts an investigation with their coding harness, where does the agent get its context from? Does it have access to the organisation’s architecture, runbooks or the terminology the team uses? This may seem rudimentary, but it has a big impact on the agent’s success.
Any investigation triggered by a user in the coding harness uses the credentials configured on the user’s machine. This makes it difficult to trace the agent’s actions in a postmortem, and it risks the agent accessing something it is not meant to.
Coding agents tend to put together hard-to-read bash scripts to run commands. Either the operator doesn’t trust the script enough to run it, or, under the pressure of the incident, runs something they are not sure about. Some of these challenges can be addressed in a coding harness, but not without substantial effort.
The coding harness limitations overlap with the expanded criteria. Together, they form the evaluation matrix we use for AI SRE agents. The sections that follow address each gap and describe what an AI SRE agent needs, starting with how an investigation begins.
Starting an investigation without waiting for a human
The time to mitigate depends on when the AI SRE agent starts its investigation. An investigation can be:
Reactive: manually triggered by the engineer during an incident.
Proactive: starts automatically when an alert fires.
Preventative: starts before an alert has fired.
The scenario we usually see is reactive. Engineers get paged and investigate with their coding harness. This wastes the time between the issue starting and the investigation starting.
An AI SRE agent helps most with proactive and preventative investigations. In a proactive investigation, the agent has laid the groundwork before the engineer arrives. They provide hypotheses that the engineer can use to guide the process. A preventative investigation aims to stop an incident before it happens. For example, the agent could watch the error budget burn rate, or check each deploy against recent incidents in the same service.
Systems and tools the agent covers
Providing the right context to the agent is at the heart of a successful investigation. The right context means the agent sees the system as a whole, not only parts of it. (Seeing the system does not mean the agent can change it.) For example, it should have access to the IaC codebase and integrate with your CI/CD and observability tools.
An AI SRE agent needs coverage in the following areas:
Use cases: triage, root cause analysis, remediation, postmortems and prevention.
Environments: clouds, Kubernetes or VMs, on-prem systems, databases and the network.
Integrations: observability, incident management, chat, ticketing, CI/CD, code repositories and feature flags. Change data is the most important integration, because the agent needs it to correlate an incident with the change that caused it.
Organisational knowledge available to the agent
A coding harness on a laptop only knows what the operator shares. It also uses context from the code repository and the engineer’s notes. Two engineers investigating the same failure can get different results depending on what context they give it and how they steer it. An AI SRE agent should have the same knowledge of the system regardless of who is on call.
Organisational context: every investigation has access to the architecture, service ownership, dependencies and recent changes.
Shared skills: a common set of debugging procedures lives in skills derived from institutional knowledge, instead of in individual engineers’ heads.
Runbooks and postmortems: the agent reads existing runbooks and past incidents. Lessons from an investigation feed back into the runbooks to keep them up to date.
Agent memory: engineers can view, edit and delete what the agent has learned, and stale knowledge is removed regularly.
Diagnostic quality
For OpenSRE, we graded each RCA against failures we had caused. Each of those failures had a clear cause, such as a database scaled to zero. Production failures rarely have such clear root causes. Every system has things that look slightly wrong but have nothing to do with the incident.
SREGym is a benchmark designed to test AI SRE agents on these kinds of failures. It injects faults into live applications across different layers of the stack, along with background noise that resembles a real production system.
SREGym is useful to us for two reasons:
Its scenarios are closer to production than the single-cause failures we created for OpenSRE.
Instead of a single score for RCA quality, it evaluates the RCA on three dimensions, which gives us a more granular understanding of the RCA an agent generates.
SREGym grades a written diagnosis with an LLM judge. The agent’s answer is evaluated with nine yes/no questions, three in each of three equally weighted dimensions: fault localization, fault characterization and failure scope. A diagnosis passes at 7 of 9, so it cannot pass while failing an entire dimension.
Transparency and audit records
During an incident, the on-call engineer needs to see why the AI SRE agent reached a conclusion before acting on it. In a postmortem, someone needs to reconstruct what the agent queried, concluded and changed.
Evidence links: the agent shows the hypotheses it considered and the evidence (logs, metrics or traces) that supports or rejects each one.
Stored investigations and traces: every query, tool call, action and conclusion goes into an observability backend, so each investigation can be traced from the triggering alert to the final conclusion. A reviewer can then find the step where the agent went wrong, and evals can be run to measure the quality and reliability of the agent.
A separate agent identity: the agent has an identity separate from the human operator. If it uses an operator’s credentials, a command it runs looks the same in the log as one the operator typed.
Operator safety and steering
Before applying a proposed mitigation, the operator has to check the AI SRE agent’s conclusion, the evidence behind it and exactly what the change will do. In a coding harness session, the conclusion, the evidence and any steering such as “check the database instead” stay in one engineer’s terminal. A second engineer who joins the incident can’t see which hypotheses were already ruled out.
An AI SRE agent should provide the following:
Readable intent: the model’s intent can be understood from its raw chain of thought, but the raw chain of thought is long and hard to check. A structured summary is better, because the operator can see how the agent reached its conclusion, the evidence for it and the change it proposes.
Mitigation plans with dry runs: every proposed fix comes with pre-checks, success criteria and a rollback step. Showing the dry run to the operator gives them the confidence to apply the proposed mitigation.
A shared view of hypotheses: the incident team sees the current hypotheses and evidence in one place, and can steer the investigation from there.
Guardrails on agent actions
Least privilege: the agent is read-only by default and uses short-lived credentials. Write permissions and approval rules are set separately for each action type and each service.
Limits on actions: actions go through dry runs, rate limits and blast-radius limits. A blast-radius limit might cap one action at one deployment or one availability zone. The control plane also stops runaway loops, for example an agent that restarts the same deployment again and again.
A risk score for each action: every proposed action comes with a risk score, so the operator knows the consequence before running it.
Removal of sensitive data: personal data and secrets are removed before they reach the model, so they don’t appear in the agent’s output or in its stored records.
Escalation to a human operator: when the agent can’t find a cause, or when the next step needs permissions it doesn’t have, it escalates to a human operator and hands over the full investigation so far.
Conclusion
The gap between a coding harness and an AI SRE agent is not raw reasoning. Point Claude at a scaled-to-zero deployment and it will often find the cause. The gap includes everything around the answer. It covers where the context comes from and whose credentials control the commands. It also covers whether a second engineer can see what the team has ruled out. It also looks at whether anyone can piece together the investigation using the postmortem findings. A harness gives one engineer an answer. An agent conducts a shared, auditable investigation of the whole incident without waiting to be paged.
That difference is why a single RCA score was never going to be enough. In the next posts, we put AI SRE agents to that test, measuring them on performance and cost against the value they deliver.
Sources
Share
Share
On this page
Section
On this page
In this post
section
