This article assumes prior knowledge of Docker containers, Prometheus and Grafana.
Use cases:
- Monitor container metrics
- Send Alerts when a container is stopped / down
We are going to use the following tools.
cadvisor — https://github.com/google/cadvisor
Analyzes resource usage and performance characteristics of running containers.
Prometheus — https://prometheus.io/
Event monitoring and alerting. It records real-time metrics in a time series database built using an HTTP pull model, with flexible queries and real-time alerting.
Grafana — https://grafana.com/
Multi-platform open-source analytics and interactive visualization web application. It provides charts, graphs, and alerts.
Steps
(1)
Install cadvisor on the host you want to monitor
docker run -d — restart always — name cadvisor -p 8080:8080 -v “/:/rootfs:ro” -v “/var/run:/var/run:rw” -v “/sys:/sys:ro” -v “/var/lib/docker/:/var/lib/docker:ro” google/cadvisor:latest
(2)
Verify installation cadvisor
docker ps -a — check for a container named — cadvisor
Navigate to http://hostIP:8080/containers/ to view a list of containers picked up by cadvisor
(3)
Adding a new target on Prometheus
Edit prometheus.yml to add the ‘cadvisor’ job
- job_name: ‘cadvisor’
static_configs:
— targets: [‘cadvisorIP:8080’]
labels:
alias: ‘cadvisor’
(4)
Verify target has been added successfully
http://prometheusHostIP:port/targets
You should see it listed
(5)
Navigate to Grafana Dashboard
import the dashboard template for container monitoring. Make sure to select the correct data source.
We can use a cool dashboard already created by Brian Christner -> https://github.com/vegasbrianc
Here is the dashboard
Grafana menu -> Dashboards -> Import and provide the Dashboard ID #179
(6)
To get alerts on container stopped events
Create a new panel on the dashboard with the following Metric
count(time()-container_last_seen{name=~”containernamehere”}<=5)
(7)
To create alerts for a particular container
Navigate to Alert on the panel options,
Create a condition Ie
When count() of query(A,5m,now) IS BELOW 1
Then select the Notification channel/group and enter the message to be sent if the count is below 0.
BONUS
Telegram alerts on Grafana
To create a telegram bot
Go to -> https://core.telegram.org/bots#creating-a-new-bot
To get a telegram chat id use
https://codesandbox.io/s/get-telegram-chat-id-q3qkk
Thanks.
