Load balancing

Load balancing spreads incoming requests across several servers or containers that run the same service, so no single one is overloaded and the service keeps answering when one of them fails. A load balancer sits in front of the pool, accepts each connection or request, and forwards it to a healthy backend chosen by an algorithm.

How it works

Common algorithms are round robin (each backend in turn), least connections (the backend with the fewest open connections) and hashing on a key such as the client IP, so the same client keeps landing on the same backend. Health checks take failed backends out of rotation and put them back when they recover. A layer 4 (L4) load balancer forwards TCP or UDP connections using only addresses and ports; a layer 7 (L7) load balancer understands HTTP, so it can route by host, path or header, retry failed requests and terminate TLS.

In the SRE bootcamp's bare-metal exercise, Nginx listens on port 8080 and balances between two API containers with upstream api { server api1:8080; server api2:8080; } and proxy_pass http://api;. With the default round robin, 1,000 requests a minute become about 500 for each container. If api2 stops responding, open source Nginx marks it unavailable after a failed attempt (max_fails=1 by default), sends every request to api1 for the next 10 seconds (fail_timeout=10s), then tries api2 again. While api2 stays down, api1 carries all 1,000 requests a minute.

Why it matters in production

A load balancer only helps if the remaining backends can carry the load when one fails. Two containers each at 70% CPU cannot cover for each other, because the survivor would need 140%. Size the pool so it can lose one backend, keep backends stateless or store sessions outside them, and monitor the load balancer itself, since it is now on the path of every request and needs its own high availability.