I’m using Google Managed Service for Prometheus (GMP) and liking it. Sometime ago, I tried using PromLens with GMP but GMP’s Prometheus HTTP API endpoint requires auth and I’ve battled Prometheus’ somewhat limited auth mechanism before (Scraping metrics exposed by Google Cloud Run services that require authentication). Listening to PromCon EU 2022 videos, I learned that PromLens has been open sourced and contributed to the Prometheus project. Eventually, the functionality of PromLens should be combined into the Prometheus UI.

I’ve been on a roll building utilities this week. I developed a Service Health dashboard for my “thing”, a Prometheus Exporter for Fly.io and today, a Prometheus Exporter for Vultr. This is motivated by the fear that I will forget a deployed Cloud resource and incur a horrible bill. I’ve no written several Prometheus Exporters for cloud platforms: Prometheus Exporter for GCP Prometheus Exporter for Linode Prometheus Exporter for Fly.io Prometheus Exporter for Vultr Each of them monitors resource deployments and produces resource count metrics that can be scraped by Prometheus and alerted with Alertmanager.

Some time ago, I wrote about using Prometheus Service Discovery w/ Consul for Cloud Run and also Scraping metrics exposed by Google Cloud Run services that require authentication. Both solutions remain viable but they didn’t address another use case for Prometheus and Cloud Run services that I have with a “thing” that I’ve been building. In this scenario, I want to: Configure Prometheus to scrape Cloud Run service metrics Discover Cloud Run services dynamically Authenticate to Cloud Run using Firebase Auth ID tokens These requirements and – one other – present several challenges:

I’ve written a solution (gcp-oidc-token-proxy) that can be used in conjunction with Prometheus OAuth2 to authenticate requests so that Prometheus can scrape metrics exposed by e.g. Cloud Run services that require authentication. The solution resulted from my question on Stack overflow. Problem #1: Endpoint requires authentication Given a Cloud Run service URL for which: ENDPOINT="my-server-blahblah-wl.a.run.app" # Returns 200 when authentication w/ an ID token TOKEN="$(gcloud auth print-identity-token)" curl \ --silent \ --request GET \ --header "Authorization: Bearer ${TOKEN}" \ --write-out "%{response_code}" \ --output /dev/null \ https://${ENDPOINT}/metrics # Returns 403 otherwise curl \ --silent \ --request GET \ --write-out "%{response_code}" \ --output /dev/null \ https://${ENDPOINT}/metrics Problem #2: Prometheus OAuth2 configuration is constrained

I’ve written a basic discoverer of Google Cloud Run services. This is for a project and it extends work done in some previous posts to Multiplex gRPC and Prometheus with Cloud Run and to use Consul for Prometheus service discovery. This solution: Accepts a set of Google Cloud Platform (GCP) projects Trawls them for Cloud Run services Assumes that the services expose Prometheus metrics on :443/metrics Relabels the services Surfaces any discovered Cloud Run services’ metrics in Prometheus You’ll need Docker and Docker Compose.

Google Cloud Run is useful but, each service is limited to exposing a single port. This caused me problems with a gRPC service that serves (non-gRPC) Prometheus metrics because customarily, you would serve gRPC on one port and the Prometheus metrics on another. Fortunately, cmux provides a solution by providing a mechanism that multiplexes both services (gRPC and HTTP) on a single port! TL;DR See the cmux Limitations and use:

I’m working on a project that will programmatically create Google Cloud Run services and I want to be able to dynamically discover these services using Prometheus. This is one solution. NOTE Google Cloud Run is the service I’m using, but the principle described herein applies to any runtime service that you’d wish to use. Why is this challenging? IIUC, it’s primarily because Prometheus has a limited set of plugins for service discovery, see the sections that include _sd_ in Prometheus Configuration documentation.

Phew! For Want of a Nail I was interested in learning how to Manage Resources for Containers. On the way, I learned and discovered: kubectl top Vertical Pod Autoscaler A (valuable) digression through PodMonitor kube-state-metrics `kubectl-patch Created a Graph References Kubernetes Resources Visual Studio Code has begun to bug me (reasonably) to add resources to Kubernetes manifests. E.g.: resources: limits: cpu: "1" memory: "512Mi" I’ve been spending time with Deislab’s Akri and decided to determine whether Akri’s primary resources (Agent, Controller) and some of my creations HTTP Device and Discovery, were being suitably constrained.

DigitalOcean launched an App Platform with many Supported Languages and Frameworks. I used Golang first, then wondered how to use non-natively-supported languages, i.e. Rust. The good news is that Docker is a supported framework and so, you can run pretty much anything. Repo: https://github.com/DazWilkin/do-apps-rust Rust I’m a Rust noob. I’m always receptive to feedback on improvements to the code. I looked to mirror the Golang example. I’m using rocket and rocket-prometheus for the first time:

I’m obsessing over Prometheus exporters. First came Linode Exporter, then GCP Exporter and, on Sunday, I stumbled upon a reverse-engineered API for Google Home devices and so wrote a very basic Google Home SDK and a similarly basic Google Home Exporter: The SDK only implements /setup/eureka_info and then only some of the returned properties. There’s not a lot of metric-like data to use besides SignalLevel (signal_level) and NoiseLevel (noise_level). I’m not clear on the meaning of some of the properties.

Earlier this week I discussed a Linode Prometheus Exporter. I added metrics for Digital Ocean’s Managed Kubernetes service to @metalmatze’s Digital Ocean Exporter. This left, metrics for Google Cloud Platform (GCP) which has, for many years, been my primary cloud platform. So, today I wrote Prometheus Exporter for Google Cloud Platform. All 3 of these exporters follow the template laid down by @metalmatze and, because each of these services has a well-written Golang SDK, it’s straightforward to implement an exporter for each of them.

Yesterday I discussed a Linode Prometheus Exporter and tantalized use of Prometheus AlertManager. Success: Configure The process is straightforward although I found the Prometheus (config) documentation slightly unwieldy to navigate :-( The overall process is documented. Here are the steps I took: Configure Prometheus Configure AlertManager Revise Docker Compose Configure Prometheus Added the following to prometheus.yml: rule_files: - "/etc/alertmanager/rules/linode.yml" alerting: alertmanagers: - scheme: http static_configs: - targets: - "alertmanager:9093" Rules must be defined in separate rules files.

I enjoy using Prometheus and have toyed around with it for some time particularly in combination with Kubernetes. I signed up with Linode [referral] compelled by the addition of a managed Kubernetes service called Linode Kubernetes Engine (LKE). I have an anxiety that I’ll inadvertently leave resources running (unused) on a cloud platform. Instead of refreshing the relevant billing page, it struck me that Prometheus may (not yet proven) help.