Services

Resources

Company

Runbook

A runbook is a documented procedure for a specific operational task or alert. It tells the responder how to check what is wrong, how to fix or mitigate it, and when to escalate. A good runbook lets any engineer on the on-call rotation handle a problem without the one person who knows the system. It is also called a playbook.

How it works

Each alert links to its runbook. A runbook for "Postgres disk above 90%" might contain:

  1. What this alert means and the user impact.

  2. Checks: run df -h and look for large WAL or temporary files.

  3. Mitigation: remove files safely, or grow the volume by 50 GB.

  4. Escalation: page the database owner if usage passes 95%.

After each incident, update the runbook with what was missing. Steps that run the same way every time are candidates for automation, which reduces toil.

Why it matters in production

Runbooks cut mean time to recovery because responders stop guessing at 2 a.m. They spread knowledge beyond a few experts. A stale runbook is dangerous, so review them like code.

See also

On the One2N blog

Further reading

Comic about runbook: Asha is paged for 91% disk on a database she does not know; the linked runbook has four steps; she finds 40 GB of WAL and grows the volume to 83%; Kai, the owner, later adds a replication slot check.