Services

Resources

Company

Runbook

A runbook is a documented procedure for a specific operational task or alert. It tells the responder how to check what is wrong, how to fix or mitigate it, and when to escalate. A good runbook lets any engineer on the on-call rotation handle a problem without the one person who knows the system. It is also called a playbook.

How it works

Each alert links to its runbook. A runbook for "Postgres disk above 90%" might contain:

  1. What this alert means and the user impact.

  2. Checks: run df -h and look for large WAL or temporary files.

  3. Mitigation: remove files safely, or grow the volume by 50 GB.

  4. Escalation: page the database owner if usage passes 95%.

After each incident, update the runbook with what was missing. Steps that run the same way every time are candidates for automation, which reduces toil.

Why it matters in production

Runbooks cut mean time to recovery because responders stop guessing at 2 a.m. They spread knowledge beyond a few experts. A stale runbook is dangerous, so review them like code.

See also

On the One2N blog

Further reading

Four-panel comic about Runbook: A disk alert fires at 2 a.m.; the only expert is asleep; The alert links to a runbook: meaning, checks, mitigation, escalation; Run df -h, clear temp files or grow the volume by 50 GB; page owner past 95%; Update it after every incident and automate steps that...