Kubernetes Operators

Ackal uses a Kubernetes Operator to orchestrate the lifecycle of its health checks. Ackal’s Operator is written in Go using kubebuilder.

Yesterday, my interest was piqued by a MetalBear blog post Writing a Kubernetes Operator [in Rust]. I spent some time reimplementing one of Ackal’s CRDs (Check) using kube-rs and not only refreshed my Rust knowledge but learned a bunch more about Kubernetes and Operators.

While rummaging around the Kubernetes documentation, I discovered flant’s Shell-operator and spent some time today exploring its potential.

The purpose of this post is to document my experience using Shell-operator for a very simple hook to log uses of Ackal’s Check CRD. Hopefully, a hook for a Custom Resource is useful to others too.

Shell-operator is well-documented and straightforward to use. What’s compelling about it is that it provides a mechanism to write bash (or Python etc.) scripts (hooks) that can be bundled into a Shell-operator container that can be deployed to a cluster to quickly extend Operator functionality.

For example, I wanted to monitor Check mutations (creations, updates and deletes) across namespaces.

Here’s a subset of Ackal’s Check CRD config:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: checks.ack.al
spec:
  group: ack.al
  names:
    categories:
    - all
    kind: Check
    listKind: CheckList
    plural: checks
    singular: check
  scope: Namespaced
  versions:
    name: v1alpha1

The first activity to extend Shell-operator is to write a hook. Here’s check.sh:

#!/usr/bin/env bash

if [[ $1 == "--config" ]] ; then
  cat <<EOF
configVersion: v1
kubernetes:
- name: "Monitor Ackal Checks"
  apiVersion: ack.al/v1alpha1
  kind: Check
  executeHookOnEvent:
  - Added
  - Modified
  - Deleted
EOF
else
  checkName=$(jq -r .[0].object.metadata.name "${BINDING_CONTEXT_PATH}")
  eventName=$(jq -r .[0].watchEvent "${BINDING_CONTEXT_PATH}")
  printf "Check '%s' %s" "${checkName}" "${eventName}"
fi

I assume (!?) that Shell-operator invokes each shell script (hook) with the --config flag in order to configure itself with the hook. The “API” for Shell-operator scripts is:

if [[ $1 == "--config" ]] ; then
  ...
else
  ...
fi

If the script is run {script} --config, then a heredoc returns the config. Otherwise, the hook performs its task. In this case, it printf’s details of the Check.

Shell-operator provides multiple binding types. In this case, I’m using kubernetes. The YAML structure of kubernetes is defined here and is extensive|powerful.

check.sh is straightforward. It defines Ackal’s Check as its monitored resource. It does this by specifying the apiVersion and kind: Check for the CRD. You can see these values defined in Check’s CRD. Kubernete’s use of group, kind, version is confusing (but consistent).

The executeHookOnEvent list defines the 3 methods that we want to track for Check resources.

So, what’s with the jq and BINDING_CONTEXT_PATH in the else branch? jq is included in the Shell-operator container and it is used to facilitate (JSON) parsing of Shell-operator responses e.g. when a Check is Added.

NOTE Although Kubernetes supports JSON and YAML, it is more common to interact with Kubernetes using YAML. The shell-operator container image includes jq for JSON but not yq for YAML.

This information is also well-defined in Binding Context. Because I’m using kubernetes binding type, the hook receives responses that include watchEvent and object. In this case object will be Check resources.

This hook outputs the type of event (watchEvent) and the Check’s name.

Once the hook script is written, it must be chmod +x {script} and can then be added to the Shell-operator container:

FROM docker.io/flant/shell-operator:v1.2.0
ADD checks.sh /hooks

Then build‘ed and push‘ed:

podman build \
--tag=${REGISTRY}/shell-operator:check \
--file=${PWD}/Dockerfile \
${PWD}

podman push ${REGISTRY}/shell-operator:check

Then the operator can be deployed to a cluster. This involves the creation of several resources:

• Pod
• ServiceAccount
• ClusterRole
• ClusterRoleBinding

I found it easiest to aggregate these into a List:

apiVersion: v1
kind: List
metadata: {}
items:
- kind: Pod
  apiVersion: v1
  metadata:
    name: checks
    namespace: shell-operator
  spec:
    containers:
    - image: ${REGISTRY}/shell-operator:checks
      imagePullPolicy: Always
      name: checks
    serviceAccount: checks
- kind: ServiceAccount
  apiVersion: v1
  metadata:
    name: checks
    namespace: shell-operator
- kind: ClusterRole
  apiVersion: rbac.authorization.k8s.io/v1
  metadata:
    name: checks
  rules:
  - apiGroups:
    - ack.al
    resources:
    - checks
    verbs:
    - get
    - list
    - update
- kind: ClusterRoleBinding
  apiVersion: rbac.authorization.k8s.io/v1
  metadata:
    name: checks
  roleRef:
    apiGroup: rbac.authorization.k8s.io
    kind: ClusterRole
    name: checks
  subjects:
  - kind: ServiceAccount
    name: checks
    namespace: shell-operator

The ServiceAccount (checks) requires cluster-wide permissions to get, list and update Custom Resources (checks.ack.al).

Once these Resources are deployed, any of the mutations to Checks, will trigger the hook (and a log entry):

kubectl logs pod/checks \
--namespace=shell-operator \
--follow

Yields (edited):

{
    "binding":"Monitor Ackal Checks",
    "event":"kubernetes",
    "hook":"checks.sh",
    "level":"info",
    "msg":"Check '{ID}' Added",
    "output":"stdout",
    "queue":"main",
    "task":"HookRun",
    "time":"2023-03-10T00:00:00Z"
}

Shell-operator also publishes Prometheus metrics. The shell-operator container exposes port 9115 and this can be queried for Shell-operator metrics:

kubectl port-forward pod/check \
--namespace=shell-operator \
9115:9115

And, from another shell:

curl \
--silent \
--get \
localhost:9115/metrics \
| awk '/^shell_operator_/ {print}'

I forgot to capture example metrics but the log fields binding,hook were included a metric labels.

There’s some snobbishness towards shell scripting but, while I’m also happy to write Go code (and Rust and Python 😀), bash is powerful and I particularly value it to script CLIs (gcloud, kubectl etc.) where the equivalent Go|Python code would be considerably more complex.

Shell-operator extends the power of bash scripts to Kubernetes Operators which are challenging to write in Go|Python. I look forward to finding further uses for Shell-operator.

• Kubernetes
• Operators
• Rust
• Bash
• Prometheus