Google Cloud Platform (GCP) Exporter

- 3 minutes read - 482 words

Earlier this week I discussed a Linode Prometheus Exporter.

I added metrics for Digital Ocean’s Managed Kubernetes service to @metalmatze’s Digital Ocean Exporter.

This left, metrics for Google Cloud Platform (GCP) which has, for many years, been my primary cloud platform. So, today I wrote Prometheus Exporter for Google Cloud Platform.

All 3 of these exporters follow the template laid down by @metalmatze and, because each of these services has a well-written Golang SDK, it’s straightforward to implement an exporter for each of them.

The Google Cloud Platform exporter is described in the README.

My primary service on all of these platforms is Kubernetes and so, for GCP, my first Collector was for Kubernetes Engine. Because of the dependency Kubernetes Engine has on Compute Engine (instances, load-balancers etc.), I’ve implemented some metrics for Compute Engine too. The other service that shows up on my GCP bills (very cheaply!) is Cloud Storage and so that’s the 3rd Collector in the Exporter. As always, if there’s another service that you want added, please submit an issue and, if you’re willing, a PR :-)

One… frustration with GCP’s Golang SDKs is that Google has not yet fully implemented the current-generation so-called Cloud Client Libraries in Golang for all of the platform’s services. The biggest omission, is Compute Engine itself. In order to stay consistent, I’ve defaulted to use the older API Client Libraries for Kubernetes Engine, Compute Engine and Cloud Storage.

It is possible for a Google account to have access to many-many GCP projects and for each project to have access to many-many e.g. Compute Engine instances. To ensure that the Exporter completes and that it covers the majority of use-cases, by default the Exporter will only produce metrics for 10 projects (adjusted using the flag --max_projects). Similarly, if a project contains more than 500 instances, metrics will only be produced for 500 instances. This includes instance counts being limited to 500.

The Exporter also uses Google’s (excellent) Application Default Credentials. This simplifies code through a mechanism by attempting to find relevant authentication credentials. Generally, for a proces such as the Exporter, it should run under a (non-human) service account. However, because it’s likely that users of the Exporter will want the Exporter to gather metrics for their perception of GCP, the documentation explains using gcloud auth application-default login as a mechanism to create ADCs for a user’s (human) Google (e.g. Gmail) account.

As with the other Exporters. The goal here is to develop a consistent approach to monitoring multiple cloud services platforms to be able to determine if I leave resources (e.g. Kubernetes clusters) running unintentionally. The Exporters are scraped by Prometheus and alerting rules are defined that fire when I leave clusters (and instances) running for too long. AlertManager receives the alerts and, in my case, sends me a Gmail. Hopefully (!) I am then in a position to delete redundant resources.