Docker containers monitoring with Prometheus.

This article assumes prior knowledge of Docker containers, Prometheus and Grafana.

Use cases:
- Monitor container metrics
- Send Alerts when a container is stopped / down

We are going to use the following tools.

cadvisorhttps://github.com/google/cadvisor
Analyzes resource usage and performance characteristics of running containers.

Prometheushttps://prometheus.io/
Event monitoring and alerting. It records real-time metrics in a time series database built using an HTTP pull model, with flexible queries and real-time alerting.

Grafana — https://grafana.com/
Multi-platform open-source analytics and interactive visualization web application. It provides charts, graphs, and alerts.

Steps

(1)
Install cadvisor on the host you want to monitor
docker run -d — restart always — name cadvisor -p 8080:8080 -v “/:/rootfs:ro” -v “/var/run:/var/run:rw” -v “/sys:/sys:ro” -v “/var/lib/docker/:/var/lib/docker:ro” google/cadvisor:latest

(2)
Verify installation cadvisor
docker ps -a — check for a container named — cadvisor
Navigate to
http://hostIP:8080/containers/ to view a list of containers picked up by cadvisor

(3)
Adding a new target on Prometheus
Edit prometheus.yml to add the ‘cadvisor’ job

- job_name: ‘cadvisor’
static_configs:
— targets: [‘cadvisorIP:8080’]
labels:
alias: ‘cadvisor’

(4)
Verify target has been added successfully
http://prometheusHostIP:port/targets
You should see it listed

(5)
Navigate to Grafana Dashboard
import the dashboard template for container monitoring. Make sure to select the correct data source.
We can use a cool dashboard already created by Brian Christner -> https://github.com/vegasbrianc

Here is the dashboard
Grafana menu -> Dashboards -> Import and provide the Dashboard ID #179

(6)
To get alerts on container stopped events
Create a new panel on the dashboard with the following Metric

count(time()-container_last_seen{name=~”containernamehere”}<=5)

(7)
To create alerts for a particular container
Navigate to Alert on the panel options,
Create a condition Ie

When count() of query(A,5m,now) IS BELOW 1

Then select the Notification channel/group and enter the message to be sent if the count is below 0.

BONUS
Telegram alerts on Grafana
To create a telegram bot
Go to -> https://core.telegram.org/bots#creating-a-new-bot

To get a telegram chat id use
https://codesandbox.io/s/get-telegram-chat-id-q3qkk

Thanks.