Trace sampling

Trace sampling is the practice of recording only some of the traces a system produces, to control storage cost and backend load. Head sampling decides whether to keep a trace when the request starts. Tail sampling waits until the trace is complete and decides based on what happened, such as an error or high latency.

How it works

With head sampling at 10%, a service handling 1,000 traces per second keeps about 100, chosen at random by trace ID. It is cheap, but it drops most rare errors. Tail sampling, for example the tail_sampling processor in the OpenTelemetry Collector, buffers spans for a decision window such as 20 seconds. It then applies policies like "keep all traces with errors" and "keep traces slower than 2 seconds", plus a small random share of the rest. All spans of a trace must reach the same Collector instance, so gateways usually route by trace ID.

Why it matters in production

Keeping every trace is expensive, and random sampling can drop the exact trace you need during an incident. Tail sampling keeps the interesting traces, at the cost of memory and a more complex Collector setup.