Services

Resources

Company

/

Amazon CloudWatch

Amazon CloudWatch

Amazon CloudWatch is the monitoring service built into AWS. Most AWS services publish metrics to it automatically, and applications and agents can send their own metrics and logs. Alarms watch those metrics and notify someone when a threshold is crossed, usually through an SNS (Simple Notification Service) topic. It is often the default observability tool for teams that run on AWS and have not set up Prometheus.

How it works

Metrics are time series identified by a namespace (such as AWS/EC2 or ContainerInsights), a name and dimensions (such as InstanceId). Logs go into log groups and log streams, can be searched with Logs Insights, and can be turned into metrics with metric filters. An alarm takes one metric, a statistic over a period, a threshold, and an M out of N rule, and moves between the states OK, ALARM and INSUFFICIENT_DATA.

For example, Container Insights publishes node_status_condition_ready every minute for each EKS node: 1 if the node is Ready, 0 if not. An alarm on the minimum across 4 nodes, with a 60 second period, a threshold of less than 1 and 3 out of 3 datapoints, goes to ALARM 3 minutes after one node stops being Ready. Its SNS action then pages on-call.

When One2N set this up for a customer, counting how many nodes were down went wrong at first. A Metrics Insights SUM over 4 healthy nodes returned 8 instead of 4, because CloudWatch added an "Other" group (4 + 1 + 1 + 1 + 1). Scoping the query with SCHEMA(ContainerInsights, ClusterName, InstanceId, NodeName) gave the right answer.

Why it matters in production

Most AWS metrics are already in CloudWatch, so the first alarm takes minutes to set up. Keeping alarms useful takes longer. Per-resource alarms break when nodes scale in and out, aggregations behave differently from PromQL. An alarm that only says "at least one node is down" makes on-call open a dashboard before they know how bad it is. Test each alarm against a real failure, and put a runbook link in the notification. Alarms that page without telling on-call what to do next are how alert fatigue starts.

See also

On the One2N blog

References

  1. What is Amazon CloudWatch? (AWS documentation)

  2. Using Amazon CloudWatch alarms (AWS documentation)

Diagram of an Amazon CloudWatch alarm: four EKS nodes report Ready as 1, 1, 1 and 0, and the minimum across them, checked every 60 seconds, is 0. An alarm moves from OK to ALARM when the minimum stays below 1 for 3 of 3 minutes, and back when all nodes are Ready; both states can fall to INSUFFICIENT_DATA. ALARM sends to an SNS topic that pages on-call. In red: a page with no runbook link leads to alert fatigue.