Services

Resources

Company

Trace sampling

Trace sampling is the practice of recording only some of the traces a system produces, to control storage cost and backend load. Head sampling decides whether to keep a trace when the request starts. Tail sampling waits until the trace is complete and decides based on what happened, such as an error or high latency.

How it works

With head sampling at 10%, a service handling 1,000 traces per second keeps about 100, chosen at random by trace ID. It is cheap, but it drops most rare errors. Tail sampling, for example the tail_sampling processor in the OpenTelemetry Collector, buffers spans for a decision window such as 20 seconds. It then applies policies like "keep all traces with errors" and "keep traces slower than 2 seconds", plus a small random share of the rest. All spans of a trace must reach the same Collector instance, so gateways usually route by trace ID.

Why it matters in production

Keeping every trace is expensive, and random sampling can drop the exact trace you need during an incident. Tail sampling keeps the interesting traces, at the cost of memory and a more complex Collector setup.

See also

On the One2N blog

References

  1. Sampling (OpenTelemetry documentation)

Comic about trace sampling: 1,000 traces a second cost a lot to store; head sampling decides at the start and keeps 1 in 10, throwing away a failed request; tail sampling waits until a request ends, then keeps all failures and slow ones plus a few normal; tail keeps every failure at the cost of more memory while it waits.