Services

Resources

Company

Failover

Failover is moving work from a failed primary to a standby so users keep being served. It can be automatic, triggered by health checks, or manual, run by an operator from a runbook. A planned move during maintenance is usually called a switchover or cutover. Failover is how high availability and disaster recovery plans actually happen.

How it works

Failover has three steps: detect the failure, promote the standby, and move traffic to it. Each step adds time.

For example, a primary database fails at 02:14:00. Health checks run every 10 seconds and need 3 failures in a row, so the failure is confirmed at 02:14:30. The standby is promoted by 02:14:45. Clients find the new primary through DNS with a 60-second TTL, so the last of them move by about 02:15:45. That is roughly 105 seconds of errors, which fits a 4-minute RTO. With asynchronous replication, writes not yet copied to the standby are lost, and that loss is what RPO measures.

Why it matters in production

Untested failover usually fails. Health checks may not match a real user path, promotion may need a manual step nobody remembers, and clients may cache old addresses. Run failover drills on a schedule, measure detection, promotion and routing time separately, and protect against split brain, where both nodes think they are primary.

See also

On the One2N blog

References

  1. Google SRE book: Managing critical state

  2. Amazon RDS Multi-AZ failover

Comic about failover: at 02:14:00 the primary database in zone A dies and checkout returns error 500; health checks every 10 s need 3 failures, so the on-call is paged at 02:14:30 and the zone B standby is promoted at 02:14:45; a 60 s DNS TTL keeps old answers alive until 02:15:45; 30 + 15 + 60 adds up to 105 s of errors, inside the 4-minute RTO.