Services

Resources

Company

/

Cutover

Cutover

A cutover is the moment production moves from one environment to another, and the plan that gets it there: a migration to a new cloud, a new database, or a disaster recovery site. A failover reacts to a failure, often automatically, and moves work to a standby. A cutover is planned. It has a named owner, a window, a runbook, a go/no-go decision at each step and a saved way back.

How it works

On the night, the team works through the runbook from the bottom of the stack up, and nobody touches traffic until the lower layers pass. Risky deploys are frozen and every step has a named owner. Data goes first: if replication lag is outside the RPO, the move stops there. Next they warm the target and push a synthetic user journey through it, a fake checkout for instance. Only when that passes do they change DNS names or load balancer weights.

Then they wait twice. The first hold is short, long enough for clients to follow the new records. The second is longer and is about whether the new side behaves under real traffic. At the end of it someone decides to commit or roll back, and that call has to land before the window closes.

For example, a team moves checkout from region A to region B in a 22:00 to 23:00 window, with an RTO of 5 minutes and an RPO of 30 seconds. The day before, they lower the DNS TTL from 3,600 to 60 seconds, so any switch, or a rollback, settles within 1 minute.

At 22:15 replication lag is 4 seconds, under the 30-second RPO: go. Synthetic checkouts pass 20 of 20 on region B. They switch DNS at 22:20, traffic settles by 22:21, the 5-minute hold ends at 22:26, and the 30-minute hold ends at 22:56 with errors and p95 latency normal. They commit with 4 minutes of the window left.

Why it matters in production

In the example, the 22:15 step checks one thing, replication lag against the 30-second RPO, and the answer is a plain go. A plan that bundles the lag check, the warm-up and the DNS change into one step leaves nobody able to say go or no-go when one of them is borderline.

Put the synthetic journey, the RTO and the RPO at the top of the plan. Lower the TTL a day ahead, as the team did from 3,600 to 60 seconds, so a rollback settles within a minute, and rehearse the whole sequence in a drill first.

See also

On the One2N blog

References

  1. AWS whitepaper: Disaster recovery of workloads on AWS

  2. Google SRE book: Reliable product launches at scale

Timeline of a cutover window from 22:00 to 23:00. Gates above the line: at 22:15 replication lag is 4 s against a 30 s RPO, go; synthetic checkouts pass 20 of 20, go; DNS switches at 22:20. Below, traffic settles at 22:21, then a 5 minute hold and a 30 minute hold, and the team commits at 22:56 with 4 minutes spare. A red line marks 23:00, when the window closes. The day before, DNS TTL dropped from 3,600 to 60 s.