Services

Resources

Company

Configuration drift

Configuration drift is the difference that builds up between how a system is declared, in code or a runbook, and how it is really configured. It comes from hotfixes in a console, manual package upgrades and one-off scripts. Over time servers that should be identical stop matching, and the code no longer describes production.

How it works

Drift is found by comparing declared state with live state. With infrastructure as code, terraform plan -refresh-only shows what changed outside Terraform. Say an engineer opened port 22 on a security group during an incident. The next normal terraform plan proposes removing that rule, and an apply could break whatever came to depend on it. In GitOps, an agent compares the cluster with Git every few minutes and flags or reverts manual edits.

Why it matters in production

Drift makes changes unpredictable, because the plan you reviewed is not the system you are changing. It also breaks disaster recovery, since rebuilding from code gives you a different system from the one that was running. Detect it on a schedule and route every fix back through code.

See also

On the One2N blog

Further reading

Comic about configuration drift: during a 02:00 load spike Asha resizes web-1 from t3.large to t3.xlarge in the console while the code still says t3.large; two weeks later Kai's unrelated plan would shrink it back; the gap is closed by writing t3.xlarge into code or applying the code to undo it, and a nightly terraform plan -refresh-only job flags the next drift for Asha the next morning.