Services

Resources

Company

/

Rollback

Rollback

People use the word for two operations with very different risk. A deploy rollback puts the previous application release back. It is usually quick and safe, since the code itself holds no state. A database rollback reverses a schema migration with a down migration, and that one can hurt: rows written since the change may already depend on the new schema.

How it works

Deploy rollbacks work because the old release still exists. Kubernetes keeps old ReplicaSets, so kubectl rollout undo deployment/checkout restores the previous version; in GitOps you revert the commit. A database rollback runs the migration's down step, such as DROP COLUMN for a migration that added a column. Down migrations only restore the schema, never the data, so many teams roll forward instead: they keep the schema and ship a fixed release.

For example, release v42 ships at 14:00 with migration 0042, which adds a nullable delivery_notes column. At 14:06 the checkout error rate jumps from 0.2% to 4%. Rolling the app back to v41 takes about 90 seconds and is safe, because v41 ignores the new column. Running the down migration would not be: at 600 orders a minute, 6 minutes of traffic means 3,600 orders already have notes, and DROP COLUMN deletes all of them. So the team keeps the column and rolls forward with v43.

Why it matters in production

If a team has never run its rollback, nobody knows whether it works. Keep migrations additive, so the previous release still runs against the new schema, and move destructive changes such as dropping or renaming a column to a later release. Practise the rollback until it is one command.

In the example above, that is what made v41 a 90-second fix rather than a choice between a broken checkout and 3,600 lost delivery notes. Rollbacks that quick lower mean time to recovery, which is why teams that have them can ship smaller deploys more often.

See also

On the One2N blog

References

  1. Kubernetes: Rolling back a Deployment

  2. Rails guides: Active Record migrations (up, down and reversible)

Timeline of a rollback. Release v42 ships at 14:00 with migration 0042, and at 14:06 checkout errors jump from 0.2% to 4%. The timeline forks. The green branch puts the app back to v41 in about 90 seconds, safe because v41 ignores the new column, then rolls forward with v43. The red branch, the down migration, runs DROP COLUMN: at 600 orders a minute for 6 minutes, 3,600 orders lose their delivery notes.