Authenticate PromLens to Google Managed Prometheus

I’m using Google Managed Service for Prometheus (GMP) and liking it.

Sometime ago, I tried using PromLens with GMP but GMP’s Prometheus HTTP API endpoint requires auth and I’ve battled Prometheus’ somewhat limited auth mechanism before (Scraping metrics exposed by Google Cloud Run services that require authentication).

Listening to PromCon EU 2022 videos, I learned that PromLens has been open sourced and contributed to the Prometheus project. Eventually, the functionality of PromLens should be combined into the Prometheus UI.

I decided to mash-up a proxy for PromLens that uses Google’s Application Default Credentials (ADC) to authenticate to GMP. It’s called promlens-gmp-token-proxy.

Here’s a screenshot:

The code is hacky.

The proxy doesn’t expose a TLS-based (HTTP) endpoint to PromLens but it does (as it must) use TLS to secure its connection to GMP.

The proxy uses DefaultTokenSource to create a token source using ADC. Tokens are minted on-demand to make requests against GMP. Once a token has been minted, it is used until it expires.

GMP queries are rooted on the endpoint:

https://monitoring.googleapis.com/v1/projects/{project}/location/global/prometheus

Where {project} is the Google Cloud Project in which GMP is deployed.

PromLens checks the service for liveness using an instant query of 1.

Assuming the proxy is running on 0.0.0.0:7777, PromLens will make the request:

http://0.0.0.0:7777/api/v1/query?query=1

And the proxy will make the request:

https://monitoring.googleapis.com/v1/projects/{project}/location/global/prometheus/api/v1/query?query=1

After copying the headers from the incoming request, the proxy adds an Authorization header if none are present (as expected) in the incoming request:

Authorization: Bearer {token}

Where {token} is an access token obtained from the DefaultTokenSource.

PromLens occasionally includes a refresh key in the query string that it submits to the Prometheus API. GMP rejects queries that contain this string:

"Invalid JSON payload received. Unknown name \"refresh\": Cannot bind query parameter. Field 'refresh' could not be found in request message."

For this reason, the proxy removes occurrencess of refresh from query strings that it receives.

I use podman but you may replace podman with e.g. docker in the following. I assume you’ve deployed GMP in Google Cloud Project.

Run the proxy:

PROJECT="..."
PORT="7777"

go run ./cmd/proxy \
--remote="monitoring.googleapis.com" \
--prefix="/v1/projects/${PROJECT}/location/global/prometheus" \
--proxy="0.0.0.0:${PORT}"

Or:

PROJECT="..."

TAG="31161773c854688ef5672c1949e140af42628efb"
IMAGE="ghcr.io/dazwilkin/promlens-gmp-token-proxy:${TAG}"

KEY="/path/to/key.json"

podman run \
--interactive --tty --rm \
--name=promlens-gmp-token-proxy \
--env=GOOGLE_APPLICATION_CREDENTIALS=/secrets/key.json \
--volume=${KEY}:/secrets/key.json \
--publish=7777:7777/tcp \
${IMAGE} \
  --remote="monitoring.googleapis.com" \
  --prefix="/v1/projects/${PROJECT}/location/global/prometheus" \
  --proxy="0.0.0.0:7777"

Run PromLens:

VERS="v0.3.0"

podman run \
--interactive --tty --rm \
--name=promlens \
--publish=8080:8080 \
prom/promlens:${VERS}

So, let’s use curl to make the PromQL query that is shown in the screenshot at the top of this post:

First, against GMP directly:

QUERY="compute_googleapis_com%3Ainstance_cpu_utilization"

TOKEN="$(gcloud auth print-access-token)"

PROJECT="..."
REMOTE="https://monitoring.googleapis.com/v1/projects/${PROJECT}/location/global/prometheus"

curl \
--get \
--header "Authorization: Bearer ${TOKEN}" \
--data-urlencode "query=${QUERY}" \
${REMOTE}/api/v1/query

Second, against the proxy:

QUERY="compute_googleapis_com%3Ainstance_cpu_utilization"

PORT="7777"
PROXY="http://0.0.0.0:${PORT}"

curl \
--get \
--data-urlencode "query=${QUERY}" \
${PROXY}/api/v1/query

The proxy exports Prometheus metrics on /metrics.

PORT="7777"
PROXY="http://0.0.0.0:${PORT}"

curl \
--silent \
--get \
${PROXY}/metrics \
| awk '/promlens_gmp_token_proxy/ {print}'

Yields:

# HELP promlens_gmp_token_proxy_build_info A metric with a constant '1' value labels by build time, git commit, OS and Go versions
# TYPE promlens_gmp_token_proxy_build_info counter
promlens_gmp_token_proxy_build_info{build_time="",git_commit="",go_version="go1.19",os_version=""} 1
# HELP promlens_gmp_token_proxy_proxied_status The total number of proxied requests that returned a status code
# TYPE promlens_gmp_token_proxy_proxied_status counter
promlens_gmp_token_proxy_proxied_status{code="200"} 30
# HELP promlens_gmp_token_proxy_proxied_total The total number of proxied requests
# TYPE promlens_gmp_token_proxy_proxied_total counter
promlens_gmp_token_proxy_proxied_total 30
# HELP promlens_gmp_token_proxy_tokens_total The total number of token requests
# TYPE promlens_gmp_token_proxy_tokens_total counter
promlens_gmp_token_proxy_tokens_total 1

That’s all!

• PromLens
• GCP
• Google Managed Prometheus
• Prometheus