Root cause analysis (RCA)
An RCA starts from what users saw, say failed checkouts, and works back to whatever made it possible. Every link in that chain needs something you can point to: a log line, a graph, a trace, a deploy, a config change. What you find goes into the blameless postmortem and its action items.
The singular name is a bit misleading. A slow query can sit in production for weeks and only hurt users once peak traffic and a slow alert line up with it, so list every factor you can prove and keep going after the first one.
How it works
Start with a timeline of what changed and when. Write down every explanation that would fit the symptoms, check each against the data, and throw out the ones the data contradicts. For the survivors, keep asking why until the answer is something the team can change. That last step is the 5 whys.
For example, checkout errors start at 14:10, a deploy at 14:00 is rolled back at 14:31, and errors stop. The deploy was the trigger, and the analysis starts there. It added a query that ran without an index and held a database connection for 400 ms instead of 20 ms.
Six pods with 20 connections each give 120 connections, so the pool could finish at most 120 / 0.4 = 300 checkouts a second (Little's law). Peak traffic at 14:10 was 450 a second, so 150 a second queued and timed out.
The load test had run at 100 a second, and the error alert needed 10 minutes above 5%, so it fired at 14:20. That is three contributing factors and three fixes: add the index, load test at peak traffic, and shorten the alert window.
Why it matters in production
Had the checkout RCA above stopped at the deploy, the team would have reverted it and moved on. The query would return with the next release, still without an index, against the same ceiling of 300 checkouts a second. One2N scores RCA quality separately when it evaluates AI SRE agents. In one test an engineer scaled replicas to zero, and an agent blamed a Helm rollback that never happened, marked as inferred but not validated.
A human RCA should meet the same bar. In the checkout case, the 400 ms query time comes with the graph or trace that shows it, and anything still unproven is written down as unproven. The deploy is recorded as the trigger, and the pool size, the load test rate and the alert window as the conditions that let it hurt users. The three fixes stay open until they ship. Finding the cause sooner also shortens mean time to recovery (MTTR) the next time something similar breaks.
See also
On the One2N blog
References
