🏅

Learning Outcomes

• Learn about monitoring & observability concepts.

• Learn about Prometheus, Loki, Grafana, Promtail.

Problem Statement

We need to make our system more observable, to do so we will start by setting up a PLG(Promtail, Loki, Grafana) stack along with Prometheus for metrics monitoring.

We will monitor our application along with its dependent services. For monitoring DB we will use the appropriate DB metrics exporter, and configure Prometheus to scrape metrics from the DB metrics exporter.

We will also monitor latency, uptime, etc. for endpoints such as the ArgoCD server, Hashicorp Vault, and our REST API.

Expectations

The following expectations should be met to complete this milestone.

• Prometheus, Loki, and Grafana should be deployed on the dependent_services node in observability namespace. Here’s the deployment diagram for reference.

• Promtail should be configured to send only application logs to Loki.

• DB metrics exporter should be run and configured to monitor DB.

• Blackbox exporter should be run and configured to monitor internal endpoints.

• Prometheus should also scrape kube-state metrics along with node metrics.

• Grafana should have two data sources for Loki (logs) and Prometheus (metrics).

• Helm charts and configs for deploying the observability stack should be committed in the same GitHub repository at the proper filepath.

• README.md should be updated with appropriate instructions to set up these components in a Kubernetes cluster.

Further Reading

• Prometheus Docs

• Loki Docs

• Grafana Docs

• Blackbox Exporter