Alert fatigue
Alert fatigue is the state where engineers get so many alerts, most of them noisy or unactionable, that they become desensitised. They acknowledge pages without reading them, mute channels, or miss the alert that signals a real outage. It is a common failure of on-call rotations with alert rules that grew without review.
How it works
Every alert that fires without needing action teaches people to ignore alerts. Suppose a rotation receives 200 pages a week and only 5 lead to any action. That is 2.5% signal, and the real outage page looks like the other 195.
Common causes are static thresholds such as CPU above 80%, duplicate alerts for one failure, flapping checks, and alerts during expected events like backups. Fixes include:
Alert on user-facing symptoms, such as error budget burn rate, not every cause.
Group related alerts and add a delay so short blips do not page.
Delete or downgrade any alert nobody acted on last month.
Why it matters in production
Fatigued teams detect real incidents later, which raises recovery time and user impact. It also burns people out. Reviewing alert volume and actionability every week is cheap insurance.
See also
On the One2N blog
Further reading
