Prometheus

Prometheus is an open source monitoring and alerting toolkit that stores metrics as time series. It scrapes numeric samples from HTTP endpoints, keeps them with key-value labels in a local database, and lets you query them with PromQL, its query language. It began at SoundCloud in 2012 and joined the Cloud Native Computing Foundation in 2016.

How it works

An application exposes its metrics at a path such as /metrics. Prometheus pulls from that path on a schedule. The global scrape interval defaults to 1 minute, and many teams set 15 seconds. Samples go into a local time series database that keeps 15 days of data by default.

A query for the share of failing requests over 5 minutes looks like this:

• sum(rate(http_requests_total{code=~"5.."}[5m])) / sum(rate(http_requests_total[5m]))

Alert rules run on the server. Alertmanager then groups, routes and silences the alerts. Grafana is the usual dashboard layer.

Why it matters in production

Prometheus is the common base for SLI and SLO alerting on Kubernetes. One server does not scale out, so large setups use remote write to long-term storage. Labels with many distinct values, called cardinality, are the usual cause of memory trouble.

See also

• Observability

• OpenTelemetry

• Burn rate

• USE method

On the One2N blog

• Building observable Go APIs with OpenTelemetry and Prometheus

• Horizontal pod autoscaling in Kubernetes based on external metrics, using Prometheus adapter

• Homogeneous monitoring across cloud and on-premise

Further reading

• Prometheus overview

• Alerting best practices (Prometheus docs)