I responded to a question Prometheus metric protocol buffer in gRPC on Stackoverflow and it piqued my curiosity and got me yak shaving.

Prometheus used to support two exposition formats including Protocol Buffers, then dropped Protocol Buffer and has since re-added it (see Protobuf format). The Protobuf format has returned to support the experimental Native Histograms feature.

I’m interested in adding Native Histogram support to Ackal so thought I’d learn more about this metric.

Spent time today yak-shaving which resulted in an unplanned migration from MicroK8s ‘prometheus’ add-on to the new and not fully-documented ‘observability’ add-on:

sudo microk8s.enable prometheus

Infer repository core for addon prometheus
DEPRECATION WARNING: 'prometheus' is deprecated and will soon be removed. Please use 'observability' instead.
...

The reason for the name change is unclear.

It’s unclear whether there’s a difference in the primary components that are installed too (I’d thought Grafana wasn’t included in ‘prometheus’), (Grafana) Loki and (Grafana) Tempo definitely weren’t included and I don’t want them either.

I wrote about Navigating Koyeb’s Golang SDK. That client is generated using the OpenAPI Generator project using Koyeb’s Swagger (now OpenAPI) REST API spec.

This post shows how to generate a Rust SDK using the Generator and provides a very basic example of using the SDK.

The Generator will create a Rust library project:

VERS="v7.2.0"

PACKAGE_NAME="koyeb-api-client-rs"
PACKAGE_VERS="1.0.0"

podman run \
--interactive --tty --rm \
--volume=${PWD}:/local \
docker.io/openapitools/openapi-generator-cli:${VERS} \
generate \
-g=rust \
-i=https://developer.koyeb.com/public.swagger.json \
-o=/local/${PACKAGE_NAME} \
--additional-properties=\
packageName=${PACKAGE_NAME},\
packageVersion=${PACKAGE_VERS}

This will create the project in ${PWD}/${PACKAGE_NAME} including the documentation at:

This Stackoverflow question piqued my interest:

retry policy configuration for grpc not working

Service Config in gRPC is new to me but, my initial suspicion (albeit incorrect) was that the JSON types were incorrect.

I decided to try using the Protocol Buffer source service_config.proto to verify the JSON.

To do so I needed to compile the source…. it was gnarly.

There are 2 repos used:

• googleapis
• grpc-proto

The service_config.proto includes options for java_package but no go_package.

Ackal deploys gRPC Health Checking clients in locations around the World in order to health check services that are representative of customer need.

Koyeb offers multiple locations and I spent time today writing a client for Ackal to integrate with Koyeb using the Golang client for the Koyeb API.

The SDK is generated from Koyeb’s OpenAPI (nee Swagger) endpoint using openapi-generator-cli. This is a smart, programmatic solution to ensuring that the SDK always matches the API definition but I found the result is idiosyncratic and therefore a little gnarly.

The deployment of Kube State Metrics for Google Managed Prometheus creates both a PodMonitoring and ClusterPodMonitoring.

The PodMonitoring resource exposes metrics published on metric-self port (8081).

The ClusterPodMonitoring exposes metrics published on metric port (8080) but this doesn’t include cronjob-related metrics:

kubectl get clusterpodmonitoring/kube-state-metrics \
--output=jsonpath="{.spec.endpoints[0].metricRelabeling}" \
| jq -r .

[
  {
    "action": "keep",
    "regex": "kube_(daemonset|deployment|replicaset|pod|namespace|node|statefulset|persistentvolume|horizontalpodautoscaler|job_created)(_.+)?",
    "sourceLabels": [
      "__name__"
    ]
  }
]

NOTE The regex does not include kube_cronjob and only includes kube_job_created patterns.

Referring to Accessing Google Services using gRPC, I wanted to query a project’s Cloud Logging for log-based metrics using gRPC.

In summary:

ENDPOINT="logging.googleapis.com:443"

ROOT="/path/to/googleapis" # https://github.com/googleapis/googleapis
PACKAGE="google/logging/v2"

# NB Not logging.proto
PROTO="${ROOT}/${PACKAGE}/logging_metrics.proto"

TOKEN=$(gcloud auth print-access-token)

PROJECT="..."

PACKAGE="google.logging.v2"
SERVICE="MetricsServiceV2"
METHOD="${PACKAGE}.${SERVICE}/ListLogMetrics"

# ListLogMetricsRequest fields
PARENT="projects/${PROJECT}"

grpcurl \
--import-path=${ROOT} \
--proto=${PROTO} \
-H "Authorization: Bearer ${TOKEN}" \
-d "{\"parent\": \"${PARENT}\"}" \
${ENDPOINT} ${METHOD}

From APIs Explorer, Cloud Logging API v2, instead of the REST reference, browse the gRPC reference specifically the package google.logging.v2 which includes MetricsServiceV2. We’re interested in the ListLogMetrics method (which unfortunately isn’t directly hyperlinkable) but is defined to be:

CRD linting

Returning to yesterday’s failing tests, it’s unclear how to introspect the E2E tests.

kubectl get namespaces

NAME                      STATUS   AGE
...
allns-s2os2u-0-90f56669   Active   22h
allns-s2qhuw-0-6b33d5eb   Active   4m23s

kubectl get all \
--namespace=allns-s2os2u-0-90f56669

No resources found in allns-s2os2u-0-90f56669 namespace.

kubectl get all \
--namespace=allns-s2qhuw-0-6b33d5eb

NAME                                                        READY   STATUS             RESTARTS   AGE
pod/prometheus-operator-6c96477b9c-q6qm2                    1/1     Running            0          4m12s
pod/prometheus-operator-admission-webhook-68bc9f885-nq6r8   0/1     ImagePullBackOff   0          4m7s

NAME                          TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE
service/prometheus-operator   ClusterIP   10.152.183.247   <none>        443/TCP   4m9s

NAME                                                    READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/prometheus-operator                     1/1     1            1           4m12s
deployment.apps/prometheus-operator-admission-webhook   0/1     1            0           4m7s

NAME                                                              DESIRED   CURRENT   READY   AGE
replicaset.apps/prometheus-operator-6c96477b9c                    1         1         1       4m13s
replicaset.apps/prometheus-operator-admission-webhook-68bc9f885   1         1         0       4m8s

kubectl logs deployment/prometheus-operator-admission-webhook \
--namespace=allns-s2qhuw-0-6b33d5eb

Error from server (BadRequest): container "prometheus-operator-admission-webhook" in pod "prometheus-operator-admission-webhook-68bc9f885-nq6r8" is waiting to start: trying and failing to pull image

NAME="prometheus-operator-admission-webhook"
FILTER="{.spec.template.spec.containers[?(@.name==\"${NAME}\")].image}"

kubectl get deployment/prometheus-operator-admission-webhook \
--namespace=allns-s2qjz2-0-fad82c03 \
--output=jsonpath="${FILTER}"

quay.io/prometheus-operator/admission-webhook:52d1e55af

Want:

For ackalctld to be deployable to Kubernetes with Prometheus Operator, it is necessary to Enable ScrapeConfig to use (discovery|target) proxies #5966. While I’m familiar with Kubernetes, Kubernetes operators (Ackal uses one built with the Operator SDK) and Prometheus Operator, I’m unfamiliar with developing Prometheus Operator. This (and subsequent) posts will document some preliminary work on this.

Cloned Prometheus Operator

Branched scrape-config-url-proxy

I’m unsure how to effect these changes and unsure whether documentation exists.

Clearly, I will need to revise the ScrapeConfig CRD to add the proxy_url fields (one proxy_url defines a proxy for the Service Discovery endpoint; the second defines a proxy for the targets themselves) and it would be useful for this to closely mirror the existing Prometheus HTTP Service Discovery use, namely ,http_sd_config>:

TL;DR Enable ScrapeConfig to use (discovery|target) proxies

I’ve developed a companion, local daemon (called ackalctld) for Ackal that provides a functionally close version of the service.

One way to deploy ackalctld is to use Kubernetes and it would be convenient if the Prometheus metrics were scrapeable by Prometheus Operator.

In order for this to work, Prometheus Operator needs to be able to scrape Google Cloud Run targets because ackalctld creates Cloud Run services for its health check clients.