Synthetic monitoring

Synthetic monitoring tests a service the way a client would, but with artificial traffic on a schedule. A probe sends a request, such as an HTTP GET to a health endpoint, a TLS handshake or a scripted login, and records whether it worked and how long it took. Because probes run even when nobody is using the service, they catch outages at 03:00 and in regions with little traffic.

How it works

The probe runner sits outside the service, ideally in a different network or region. Simple checks test one endpoint; scripted checks walk a user journey such as sign in, add to cart and pay. With Prometheus, the blackbox exporter does the probing: Prometheus scrapes /probe?target=...&module=http_2xx and gets back metrics such as probe_success and probe_duration_seconds.

Exercise 10 of the One2N SRE bootcamp sets up the blackbox exporter for internal endpoints. Probing the API's /healthcheck every 30 seconds gives 86,400 / 30 = 2,880 probes a day. An alert rule with expr: probe_success == 0 and for: 2m fires only after the check has failed for 2 minutes, four 30-second intervals in a row, so a single dropped probe does not page anyone.

Why it matters in production

Synthetic checks answer "is it up from the outside?" with a steady, comparable signal, which makes them a good source for an availability SLI and for alerting on hard outages. They only see the paths you script, though. Real-user monitoring (RUM) measures what actual users experience across devices and networks, but it needs traffic to say anything. Most teams run both, and point probes at a path that touches real dependencies rather than a health check that returns 200 while the database is down.