Chaos engineering
Chaos engineering is the discipline of running controlled experiments that inject failures into a system to build confidence it can withstand them. Teams kill instances, add latency or cut network links on purpose. The aim is to find weaknesses in a planned window instead of during a 3 a.m. outage.
How it works
Each experiment follows a scientific loop:
Define steady state with metrics, such as p99 latency under 300 ms and error rate under 0.1%.
Form a hypothesis: "if one of 3 API pods dies, steady state holds".
Inject the failure with a small blast radius, for example in staging or for 1% of traffic.
Compare results and stop the experiment at once if user impact grows.
If the remaining 2 pods must absorb 50% more traffic each and latency jumps, you have found a capacity gap before customers did.
Why it matters in production
Redundancy and failover that are never tested often do not work. Chaos experiments prove high availability designs, runbooks and alerts. Start small, keep an abort switch, and run game days where the team practises together.
See also
On the One2N blog
Further reading
