I’ve written a solution (gcp-oidc-token-proxy) that can be used in conjunction with Prometheus OAuth2 to authenticate requests so that Prometheus can scrape metrics exposed by e.g. Cloud Run services that require authentication. The solution resulted from my question on Stack overflow. Problem #1: Endpoint requires authentication Given a Cloud Run service URL for which: ENDPOINT="my-server-blahblah-wl.a.run.app" # Returns 200 when authentication w/ an ID token TOKEN="$(gcloud auth print-identity-token)" curl \ --silent \ --request GET \ --header "Authorization: Bearer ${TOKEN}" \ --write-out "%{response_code}" \ --output /dev/null \ https://${ENDPOINT}/metrics # Returns 403 otherwise curl \ --silent \ --request GET \ --write-out "%{response_code}" \ --output /dev/null \ https://${ENDPOINT}/metrics Problem #2: Prometheus OAuth2 configuration is constrained

TL;DR I’m working on a project that includes multiple Cloud Run services. I’ve been putting my gcloud head on to deploy these services thinking that it’s curious there’s no way to write the specs as YAML configs. Today, I learned that there is: gcloud beta run services replace. What prompted the discovery was some frustration trying to deploy a JSON-valued environment variable to Cloud Run: local FIREBASE_CONFIG="{ apiKey: ${FIREBASE_API_KEY}, authDomain: ${FIREBASE_AUTH_DOMAIN}, projectId: ${FIREBASE_PROJECT}, storageBucket: ${FIREBASE_STORAGE_BUCKET}, messagingSenderId: ${FIREBASE_MESSAGING_SENDER}, appId: ${FIREBASE_APP}}" gcloud run deploy ${SRV_NAME} \ --image=${IMAGE} \ --command="/server" \ --args="--endpoint=:${PORT}" \ --set-env-vars=FIREBASE_CONFIG="${FIREBASE_CONFIG}" \ --max-instances=1 \ --memory=256Mi \ --ingress=all \ --platform=managed \ --port=${PORT} \ --allow-unauthenticated \ --region=${REGION} \ --project=${PROJECT} gcloud balks at this.

See: Multiplexing gRPC and HTTP endpoints with Cloud Run gRPC, Cloud Run & Endpoints ESPv2: Configure Cloud Endpoints to proxy traffic to a Cloud Run multiplexed (gRPC|HTTP) service Challenges: Cloud Run permits single port Cloud Run services publishing e.g. gRPC and Prometheus, must multiplex transports Cloud Run services publishing multiplexed transports are challenging to expose using Cloud Endpoints Hypothesis #1: Multiplexed transports work with Cloud Run See: Multiplexing gRPC and HTTP endpoints with Cloud Run

I’ve written a basic discoverer of Google Cloud Run services. This is for a project and it extends work done in some previous posts to Multiplex gRPC and Prometheus with Cloud Run and to use Consul for Prometheus service discovery. This solution: Accepts a set of Google Cloud Platform (GCP) projects Trawls them for Cloud Run services Assumes that the services expose Prometheus metrics on :443/metrics Relabels the services Surfaces any discovered Cloud Run services' metrics in Prometheus You’ll need Docker and Docker Compose.

Google Cloud Run is useful but, each service is limited to exposing a single port. This caused me problems with a gRPC service that serves (non-gRPC) Prometheus metrics because customarily, you would serve gRPC on one port and the Prometheus metrics on another. Fortunately, cmux provides a solution by providing a mechanism that multiplexes both services (gRPC and HTTP) on a single port! TL;DR See the cmux Limitations and use:

I’m working on a project that will programmatically create Google Cloud Run services and I want to be able to dynamically discover these services using Prometheus. This is one solution. NOTE Google Cloud Run is the service I’m using, but the principle described herein applies to any runtime service that you’d wish to use. Why is this challenging? IIUC, it’s primarily because Prometheus has a limited set of plugins for service discovery, see the sections that include _sd_ in Prometheus Configuration documentation.

Phew! Programmitcally deploying Cloud Run services should be easy but it didn’t find it so. My issues were that the Cloud Run Admin (!) API is poorly documented and it uses non-standard endpoints (thanks Sal!). Here, for others who may struggle with this, is how I got this to work. Goal Programmatically (have Golang, Python, want Rust) deploy services to Cloud Run. i.e. achieve this: gcloud run deploy ${NAME} \ --image=${IMAGE} \ --platform=managed \ --no-allow-unauthenticated \ --region=${REGION} \ --project=${PROJECT} TRICK --log-http is your friend

I’ve written previously (Google Trillian for Noobs) about Google’s interesting project Trillian and about some of the “personalities” (e.g. PyPi Transparency) that I’ve build using it. Having gone slight cra-cra on Cloud Run and gRPC this week with Golang gRPC Cloud Run and gRPC, Cloud Run & Endpoints, I thought it’d be fun to deploy Trillian and a personality to Cloud Run. It mostly (!) works :-) At the end of the post, I’ve summarized creating a Cloud SQL instance to host the Trillian data(base).

<3 Google but there’s quite often an assumption that we’re all sitting around the engineering table and, of course, we’re not. Cloud Endpoints is a powerful offering but – IMO – it’s super confusing to understand and complex to deploy. If you’re familiar with the motivations behind service meshes (e.g. Istio), Cloud Endpoints fits in a similar niche (“neesh” or “nitch”?). The underlying ambition is that, developers can take existing code and by adding a proxy (or sidecar), general-purpose abstractions, security, logging etc.

Update: 2020-03-24: Since writing this post, I’ve contributed Golang and Rust samples to Google’s project. I recommend you start there. Google explained how to run gRPC servers with Cloud Run. The examples are good but only Python and Node.JS: gRPC comes to Cloud Run gRPC in Google Cloud Run Missing Golang…. until now ;-) I had problems with 1.14 and so I’m using 1.13. Project structure I’ll tidy up my repo but the code may be found: