Toil

Toil is operational work that is manual, repetitive and automatable, and that leaves the service no better afterwards. It is the work of running a system rather than improving it. In site reliability engineering (SRE), toil is something to measure and reduce, not a badge of hard work.

How it works

Google's SRE book lists the traits of toil: manual, repetitive, automatable, tactical, no enduring value, and growing linearly with the service. The more traits a task has, the more it is toil.

Example: an engineer renews 20 TLS certificates by hand every week at 15 minutes each. That is 5 hours a week, about 260 hours a year, and it doubles when the fleet doubles. A one-off automation, such as cert-manager on Kubernetes, removes it.

Google SRE teams aim to keep toil below 50% of each engineer's time, so the rest goes to engineering work.

Why it matters in production

Toil crowds out the projects that make systems reliable. It causes burnout and mistakes, because tired people repeat steps badly. Tracking hours of toil per week gives teams a clear case for automation work.

See also

• Site reliability engineering

• Runbook

• Infrastructure as code

• Platform engineering

On the One2N blog

• Multi-tenant SFTP: how we built a zero-ops setup on AWS Transfer Family

• Transforming alerting with GitOps: a journey in automating Elasticsearch alerts

Further reading

• Eliminating toil (Google SRE book)