Yesterday I discussed a Linode Prometheus Exporter and tantalized use of Prometheus AlertManager.
The process is straightforward although I found the Prometheus (config) documentation slightly unwieldy to navigate :-(
The overall process is documented.
Here are the steps I took:
Added the following to
- scheme: http
Rules must be defined in separate rules files. See below for the content for
linode.yml and an explanation.
alerting section specifies that an AlertManager will be accessible on
9093 is the default port and the
alertmanager hostname will come courtesy of Docker Compose.
- name: linode
- alert: LinodeInstances
expr: linode_instance_count > 0
summary: Linode Instances running
- alert: LinodeNodeBalancers
expr: linode_nodebalancer_count > 0
summary: Linode NodeBalancers running
The rule defines a group (of alerts) called
linode. These will represent the set I’m going to define for my Linode account leveraging the exporter. There are 2 alerts.
The magic happens with the
expr statements. I’m not familiar with PromQL but e.g.
linode_instance_count is the name of a metric exposed by the exporter and this alert will be true when that numbers exceeds 0, i.e. I have at least one instance running. The alert will fire when this holds true for an hour (60 minutes) or more. You can see the pattern here. As more metrics are surfaced by the exporter, e.g. Linode Kubernetes Engine, these can be quickly represented with alerts.
I’m using Docker Compose and already have the Prometheus container image. Unsurprisingly, there’s an image for AlertManager too. As with Prometheus, this expects a configuration YAML. Unoriginally, I called this
- name: email-me
- to: [[EMAIL-ADDR]]
This is one (of many) possible alert
receivers. I’m alerting by having emails sent to my Gmail account. I found this helpful explanation on the excellent Prometheus-centric blog Robust Perception. Replace
[[EMAIL-ADDR]] with your Gmail address and
[[EMAIL-PASS]] with the app token that you generate.
I’m unfamiliar with the
route mechanism but – IIUC – this sents every alerts (
Revise Docker Compose
Everything is in place and we can now run the comnbined services by adding
prom/alertmanager to the mix:
NB Because we added a rules file to
prometheus.yml, we must revise the
volumes reference for the
9093 is the default AlertManager port for receiving alerts (from Prometheus) and for presenting the UI
All being well,
docker-compose up then
docker-compose ps should get us:
Name Command State Ports
alertmanager /bin/alertmanager --config ... Up 0.0.0.0:9093->9093/tcp
cadvisor /usr/bin/cadvisor -logtostderr Up (healthy) 0.0.0.0:8085->8080/tcp
linode-exporter /linode-exporter --linode_ ... Up 0.0.0.0:9388->9388/tcp
prometheus /bin/prometheus --config.f ... Up 0.0.0.0:9090->9090/tcp
You can browse:
Under Prometheus, you ought now be able to see a list of alerts:
NB Because I’ve no active instances and nodebalancers (yet), these are “inactive”
Then, after creating a Linode Kubernetes Engine cluster:
And then, after an hour:
And, this triggers an email of the type that is shown at the top of this post. Neat!