Services

Resources

Company

Synthetic monitoring

Synthetic monitoring tests a service the way a client would, but with artificial traffic on a schedule. A probe sends a request, such as an HTTP GET to a health endpoint, a TLS handshake or a scripted login, and records whether it worked and how long it took. Because probes run even when nobody is using the service, they catch outages at 03:00 and in regions with little traffic.

How it works

The probe runner sits outside the service, ideally in a different network or region. Simple checks test one endpoint; scripted checks walk a user journey such as sign in, add to cart and pay. With Prometheus, the blackbox exporter does the probing: Prometheus scrapes /probe?target=...&module=http_2xx and gets back metrics such as probe_success and probe_duration_seconds.

Exercise 10 of the One2N SRE bootcamp sets up the blackbox exporter for internal endpoints. Probing the API's /healthcheck every 30 seconds gives 86,400 / 30 = 2,880 probes a day. An alert rule with expr: probe_success == 0 and for: 2m fires only after the check has failed for 2 minutes, four 30-second intervals in a row, so a single dropped probe does not page anyone.

Why it matters in production

Synthetic checks answer "is it up from the outside?" with a steady, comparable signal, which makes them a good source for an availability SLI and for alerting on hard outages. They only see the paths you script, though. Real-user monitoring (RUM) measures what actual users experience across devices and networks, but it needs traffic to say anything. Most teams run both, and point probes at a path that touches real dependencies rather than a health check that returns 200 while the database is down.

See also

On the One2N blog

References

  1. Blackbox exporter (GitHub)

  2. Alerting rules (Prometheus documentation)

  3. Synthetic monitoring (Wikipedia)

Comic about synthetic monitoring: the API goes down at 03:00 with zero users and zero errors to notice; a robot user calls GET /healthcheck every 30 s via the blackbox exporter; that is 2,880 probes a day, and a rule pages after four misses in a row (2 minutes) but ignores one dropped probe; synthetic works with zero traffic, real user monitoring needs it, and the comic says run both.