Latency and throughput

Latency and throughput are the two basic measures of system performance. Latency is the time a single request takes from start to finish, often reported as percentiles. Throughput is the number of requests or messages a system completes per unit of time, such as requests per second. They are related but are not the same measure.

How it works

Little's Law ties them together: concurrency equals throughput multiplied by latency. A service handling 200 requests per second at 50 ms average latency has about 10 requests in flight at once. If latency rises to 500 ms at the same throughput, 100 requests are in flight, which can exhaust thread pools or connection limits. As utilisation nears capacity, requests start to queue, and latency grows much faster than throughput.

Why it matters in production

Optimising one often costs the other. Batching raises throughput but adds wait time per item. Capacity planning that looks only at throughput can push a system into the zone where small traffic spikes cause large latency jumps and SLO breaches.