Scraping metrics exposed by Google Cloud Run services that require authentication

I’ve written a solution (gcp-oidc-token-proxy) that can be used in conjunction with Prometheus OAuth2 to authenticate requests so that Prometheus can scrape metrics exposed by e.g. Cloud Run services that require authentication. The solution resulted from my question on Stack overflow.

Problem #1: Endpoint requires authentication

Given a Cloud Run service URL for which:

ENDPOINT="my-server-blahblah-wl.a.run.app"

# Returns 200 when authentication w/ an ID token
TOKEN="$(gcloud auth print-identity-token)"
curl \
--silent \
--request GET \
--header "Authorization: Bearer ${TOKEN}" \
--write-out "%{response_code}" \
--output /dev/null \
https://${ENDPOINT}/metrics

# Returns 403 otherwise
curl \
--silent \
--request GET \
--write-out "%{response_code}" \
--output /dev/null \
https://${ENDPOINT}/metrics

Problem #2: Prometheus OAuth2 configuration is constrained

client_id: <string>
client_secret: <secret>
scopes:
- <string>
token_url: <string>
endpoint_params:
  <string>: <string>

The naive optimist in me wanted¹ to be able to leverage gcloud’s application default credentials (link) and the client_id and client_secret to be found in ${HOME}/.config/gcloud/application_default_credentials.json but this will yield 400’s

In a comment responding to my Stack overflow question, Levi Harrison who authored Prometheus’ OAuth2 support, suggested I consider writing a sidecar to generate the access token.

This was a clever idea and is the root of the solution.

The solution (gcp-oidc-token-proxy) provides a proxy token service to Prometheus (token_url: gcp-oidc-token-proxy) that accepts the /token request from Prometheus and, using a Google Service Account to authenticate itself, get Google’s identity service to mint an identity token that it returns to Prometheus which uses it as Authorization: Bearer ${TOKEN}.

In an application that I’m developing, I use Google Token Service with grant_type=refresh_token to mint access tokens and I was thinking that I could use this service to mint an access token from an identity token to return to Prometheus.

However:

• Google Token service appears to not work² with grant_type=authorization_code
• Salmaan pointed out that identity tokens work as Bearers

This made the solution easier although I was struggling to determine which (of many) Google Golang OAuth2 SDKs, I should use, including:

• https://pkg.go.dev/google.golang.org/api/idtoken
• https://pkg.go.dev/golang.org/x/oauth2/google

Salmaan has a comprehensive repo Authenticating using Google OpenID Connect Tokens and the Golang sample got me working.

The idtoken SDK mints identity tokens (ey….) whereas [google](https://pkg.go.dev/golang.org/x/oauth2/google) SDK mints access tokens (ya29….`).

NOTE Auth0’s jwt.io tool is a safe (doesn’t leave your browser) tool for decoding identity tokens.

Given a Google Service Account that has IAM permissions sufficient to invoke the desired (e.g. Cloud Run) service (for Cloud Run, `roles/run.invoker’), the proxy can be run:

PORT="7777" # Or your preference

export GOOGLE_APPLICATION_CREDENTIALS=/path/to/key.json

./proxy \
--port=${PORT} \
--target_url=${ENDPOINT}

The Prometheus OAuth2 configuration utilizes endpoint_params to provide an additional property in the request for audience. This is necessary because identity tokens provided by Google require an audience value and the Cloud Run service authenticates identity tokens that include an audience that matches the service’s URL. Here’s an example Prometheus OAuth2 configuration for the proxy:

# Cloud Run service
- job_name: "cloudrun-service"
  scheme: https
  oauth2:
    client_id: "anything"
    client_secret: "anything"
    token_url: "http://localhost:7777"
    endpoint_params:
      audience: "https://some-service-xxxxxxxxxx-xx.a.run.app"
  static_configs:
  - targets:
    - "some-service-xxxxxxxxxx-xx.a.run.app:443"

NOTE

• scheme must be https as Cloud Run service’s require TLS
• client_id and client_secret must be present and be anything other than ""
• token_url is the proxy’s URL
• endpoint_params includes audience and this must be the Cloud Run service URL including https://
• targets used with the proxy can only include a single value of the Cloud Run service URL excluding the scheme (https) as this is defined previous.

The Application Default Credentials (provided through the environment variable GOOGLE_APPLICATION_CREDENTIALS) is used by the proxy along with the audience value to create a new Token Source. The Token Source can then be used to mint an identity token that will be Cloud Run service-specific.

Because the proxy may be shared across multiple Prometheus scrape_targets and each of these may define a distinct endpoint, the proxy caches Token Sources by audience and it caches identity tokens that are issued by audience. Becaude identity tokens generally expire after 3600 seconds (one hour), the proxy can be more efficient by using tokens until they near expiry at which time a new identity token is minted.

NOTE One limitation of this solution is that, because identity token are minted for a specific Cloud Run service URL provided as the audience value in the request, if there are multiple Cloud Run services to be scraped, each much have its own Prometheus scrape_targets section.

ts, _ := idtoken.NewTokenSource(context.Background(), audience)
tok, _ = ts.Token()

The proxy simply maps the identity token (!) into a response as the value of access_token:

resp := struct {
    AccessToken  string `json:"access_token"`
}{
    AccessToken:  tok.AccessToken,
}

The response (resp) is then marshaled into a string and returned to Prometheus which uses the identity token as a header Authorization: Bearer ${TOKEN} to authenticate the request against the Cloud Run service’s metrics endpoints.

NOTE The Prometheus server’s incoming request object is ignored. The client_id and client_secret values are expected (cannot be nil or "") but may be anything else.

The (gcp-oidc-token-proxy) repo contains detailed instructions on running the proxy as a sidecar against Prometheus using Kubernetes and Docker Compose.

• Cloud-Run
• Oidc
• Token
• Oauth
• Oauth2
• Prometheus