Navigating Koyeb's Golang SDK
Ackal deploys gRPC Health Checking clients in locations around the World in order to health check services that are representative of customer need.
Koyeb offers multiple locations and I spent time today writing a client for Ackal to integrate with Koyeb using the Golang client for the Koyeb API.
The SDK is generated from Koyeb’s OpenAPI (nee Swagger) endpoint using openapi-generator-cli
. This is a smart, programmatic solution to ensuring that the SDK always matches the API definition but I found the result is idiosyncratic and therefore a little gnarly.
Capturing e.g. CronJob metrics with GMP
The deployment of Kube State Metrics for Google Managed Prometheus creates both a PodMonitoring
and ClusterPodMonitoring
.
The PodMonitoring
resource exposes metrics published on metric-self
port (8081).
The ClusterPodMonitoring
exposes metrics published on metric
port (8080) but this doesn’t include cronjob
-related metrics:
kubectl get clusterpodmonitoring/kube-state-metrics \
--output=jsonpath="{.spec.endpoints[0].metricRelabeling}" \
| jq -r .
[
{
"action": "keep",
"regex": "kube_(daemonset|deployment|replicaset|pod|namespace|node|statefulset|persistentvolume|horizontalpodautoscaler|job_created)(_.+)?",
"sourceLabels": [
"__name__"
]
}
]
NOTE The
regex
does not includekube_cronjob
and only includeskube_job_created
patterns.
Listing Cloud Logging log-based metrics using gRPC
Referring to Accessing Google Services using gRPC, I wanted to query a project’s Cloud Logging for log-based metrics using gRPC.
In summary:
ENDPOINT="logging.googleapis.com:443"
ROOT="/path/to/googleapis" # https://github.com/googleapis/googleapis
PACKAGE="google/logging/v2"
# NB Not logging.proto
PROTO="${ROOT}/${PACKAGE}/logging_metrics.proto"
TOKEN=$(gcloud auth print-access-token)
PROJECT="..."
PACKAGE="google.logging.v2"
SERVICE="MetricsServiceV2"
METHOD="${PACKAGE}.${SERVICE}/ListLogMetrics"
# ListLogMetricsRequest fields
PARENT="projects/${PROJECT}"
grpcurl \
--import-path=${ROOT} \
--proto=${PROTO} \
-H "Authorization: Bearer ${TOKEN}" \
-d "{\"parent\": \"${PARENT}\"}" \
${ENDPOINT} ${METHOD}
From APIs Explorer, Cloud Logging API v2, instead of the REST reference, browse the gRPC reference specifically the package google.logging.v2
which includes MetricsServiceV2
. We’re interested in the ListLogMetrics
method (which unfortunately isn’t directly hyperlinkable) but is defined to be:
Prometheus Operator support an auth proxy for Service Discovery
CRD linting
Returning to yesterday’s failing tests, it’s unclear how to introspect the E2E tests.
kubectl get namespaces
NAME STATUS AGE
...
allns-s2os2u-0-90f56669 Active 22h
allns-s2qhuw-0-6b33d5eb Active 4m23s
kubectl get all \
--namespace=allns-s2os2u-0-90f56669
No resources found in allns-s2os2u-0-90f56669 namespace.
kubectl get all \
--namespace=allns-s2qhuw-0-6b33d5eb
NAME READY STATUS RESTARTS AGE
pod/prometheus-operator-6c96477b9c-q6qm2 1/1 Running 0 4m12s
pod/prometheus-operator-admission-webhook-68bc9f885-nq6r8 0/1 ImagePullBackOff 0 4m7s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/prometheus-operator ClusterIP 10.152.183.247 <none> 443/TCP 4m9s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/prometheus-operator 1/1 1 1 4m12s
deployment.apps/prometheus-operator-admission-webhook 0/1 1 0 4m7s
NAME DESIRED CURRENT READY AGE
replicaset.apps/prometheus-operator-6c96477b9c 1 1 1 4m13s
replicaset.apps/prometheus-operator-admission-webhook-68bc9f885 1 1 0 4m8s
kubectl logs deployment/prometheus-operator-admission-webhook \
--namespace=allns-s2qhuw-0-6b33d5eb
Error from server (BadRequest): container "prometheus-operator-admission-webhook" in pod "prometheus-operator-admission-webhook-68bc9f885-nq6r8" is waiting to start: trying and failing to pull image
NAME="prometheus-operator-admission-webhook"
FILTER="{.spec.template.spec.containers[?(@.name==\"${NAME}\")].image}"
kubectl get deployment/prometheus-operator-admission-webhook \
--namespace=allns-s2qjz2-0-fad82c03 \
--output=jsonpath="${FILTER}"
quay.io/prometheus-operator/admission-webhook:52d1e55af
Want:
Prometheus Operator support an auth proxy for Service Discovery
For ackalctld
to be deployable to Kubernetes with Prometheus Operator, it is necessary to Enable ScrapeConfig
to use (discovery|target) proxies #5966. While I’m familiar with Kubernetes, Kubernetes operators (Ackal uses one built with the Operator SDK) and Prometheus Operator, I’m unfamiliar with developing Prometheus Operator. This (and subsequent) posts will document some preliminary work on this.
Cloned Prometheus Operator
Branched scrape-config-url-proxy
I’m unsure how to effect these changes and unsure whether documentation exists.
Clearly, I will need to revise the ScrapeConfig
CRD to add the proxy_url
fields (one proxy_url
defines a proxy for the Service Discovery endpoint; the second defines a proxy for the targets themselves) and it would be useful for this to closely mirror the existing Prometheus HTTP Service Discovery use, namely ,http_sd_config>
:
Prometheus Operator `ScrapeConfig`
TL;DR Enable ScrapeConfig
to use (discovery|target) proxies
I’ve developed a companion, local daemon (called ackalctld
) for Ackal that provides a functionally close version of the service.
One way to deploy ackalctld
is to use Kubernetes and it would be convenient if the Prometheus metrics were scrapeable by Prometheus Operator.
In order for this to work, Prometheus Operator needs to be able to scrape Google Cloud Run targets because ackalctld
creates Cloud Run services for its health check clients.
Prometheus Exporter for Koyeb
Yet another cloud platform exporter for resource|cost management. This time for Koyeb with Koyeb Exporter.
Deploying resources to cloud platforms generally incurs cost based on the number of resources deployed, the time each resource is deployed and the cost (per period of time) that the resource is deployed. It is useful to be able to automatically measure and alert on all the resources deployed on all the platforms that you’re using and this is an intent of these exporters.
Kubernetes Python SDK w/ CRDs
Responded to Get Custom K8s Resource using Python and found the CustomObjectsApi
documentation unclear.
If you have a cluster and a kubeconfig file with a correctly configured current-context
, so that you can successfully:
PLURAL="checks"
kubectl get ${PLURAL} \
--all-namespaces
NOTE I’m using Ackal’s CRDs in these examples.
Then you can use the following code to access the cluster’s REST API server to enumerate its CRDs:
main.py
:
from __future__ import print_function
from kubernetes import client, config
from kubernetes.client.rest import ApiException
config.load_kube_config()
api = client.CustomObjectsApi()
# Ackal's Group|Version and some Kinds
group = 'ack.al'
version = 'v1'
plurals = ['checks','customers']
for plural in plurals:
try:
resp = api.list_cluster_custom_object(
group,
version,
plural,
)
for item in resp["items"]:
spec = item["spec"]
print(spec)
except ApiException as e:
print(e)
python3 -m venv venv
source venv/bin/activate
python3 -m pip install kubernetes==26.1.0
python3 main.py
That’s all!
GoatCounter with Hugo with Ananke
Thanks to Joe Mooring for his solution to my question as to how to do this.
I’ve decided to ditch Google Analytics and am evaluating using GoatCounter with Hugo with Ananke theme.
This was the layout of the site:
.
├── archetypes
├── content
│ └── posts
├── go.mod
├── go.sum
├── hugo.toml
├── layouts
├── public
└── static
This is the structure (of layouts
) with the necessary changes:
.
├── archetypes
├── content
│ └── posts
├── go.mod
├── go.sum
├── hugo.toml
├── layouts
│ ├── _default
│ │ └── baseof.html
│ └── partials
│ └── analytics.html
├── public
└── static
I copied /layouts/_default/baseof.html
from the gohugo-ananke-theme
repo into my site’s|repo’s /layouts/_default/baseof.html
.
Python Protobuf changes
Python’s Protocol Buffers code-generation using protoc
has had significant changes that can cause developers… “challenges”. This post summarizes my experience of these mostly to save me from repreatedly recreating this history for myself when I forget it.
- Version change
- Generated code change
- Implementation Backends
I’ll use this summarized table of proto
and the Pypi library’s history in this post. protoc
refers to the compiler that supports code-generation in multiple languages. protobuf
refers to the corresponding Python (runtime) library on Pypi: