Kubernetes Webhooks

- 10 minutes read - 2126 words

I spent some time last week writing my first admission webhook for Kubernetes. I wrote the handler in Golang because I’m most familiar with Golang and because, as Kubernetes’ native language, I was more confident that the necessary SDKs would exist and that the documentation would likely use Golang by default. I struggled to find useful documentation and so this post is to help you (and me!) remember how to do this next time!

If I have time, I’m going to rewrite the handler in Rust. The webhook validates Akri Configuration (CRDs) and, because Akri is written in Rust, it would likely be better if this handler were too.

Outcome

So, “What’s an admission webhook?” and “Why would I want one?”. I’m going to provide a skeleton for building Kubernetes webhooks but will use the Akri webhook by way of example. It arose from an issue that I encountered with an Akri Configuration for a Zeroconf protocol implementation. I had:

apiVersion: akri.sh/v0
kind: Configuration
metadata:
  name: zeroconf
spec:
  protocol:
    zeroconf:
      kind: "_rust._tcp"
      port: 8888
      txtRecords:
        project: akri
        protocol: zeroconf
        component: avahi-publish
  capacity: 1
  brokerPodSpec:
    containers:
      - name: zeroconf-broker
        image: some-image
    resources: <------------ INCORRECTLY INDENTED AFTER EDIT
              limits: 
                "{{PLACEHOLDER}}": "1"

And, as you can see from the comment, I munged the YAML after editing it, the resources section was moved from under containers to be under brokerProdSpec. The result is invalid YAML but this was not caught by Kubernetes because the configuration is for a CRD rather than an internal type.

And so, without a ValidatingWebhookConfiguration:

kubectl apply --filename=./zeroconf.yam
akri.sh/v0/configuration created

And, with a ValidatingWebhookConfiguration:

kubectl apply --filename=./zeroconf.yaml 
Error from server: error when creating "./zeroconf.yaml":
admission webhook "webhook.akri.sh" denied the request:
Configuration does not include `resources.limits`

NOTE not only do we get validation of the CRD (reosurces.limits is missing) but the (erroneous) Configuration is not applied to the cluster.

By way of background, resources is a instance of ResourceRequirements and is used to limit the amount of resources allowed to be used by a container. Akri uses this facility to bind containers to IoT device Instances (another Akri CRD) that are ’twinned’ to the cluster.

In my erroneous YAML, this section was not present including the important {{PLACEHOLDER}} value, Akri was unable to bind my container to an Akri (device) Instance and, the container had no mounted environment variables which are the conduit through which a container can interact with an IoT device.

Nothing good, basically.

Overview

A ValidatingAdmissionWebhook is one of many webhook types supported by Kubernetes. As the name suggests (and the previous example shows), these validate Kubernetes configurations. Exactly what we need in my case.

Kubernetes Webhooks comprise several components:

  • Deployment (minimally a Pod) running an HTTP server with a handler (the webhook) secured by TLS
  • Service exposing the Deployment
  • ValidatingWebhookConfiguration that binds the webhook (service) to Kubernetes “resources”

I’ll include as much detail as possible below because the crevasses are to be found!

AdmissionRegistration versions

IIUC admissionregistration.k8s.io/v1beta1 is being deprecated and won’t be available in Kubernetes v1.19+. It is replaced by admissionregistration.k8s.io/v1. You may determine which (hopefully one of the two) versions is available in your cluster:

kubectl api-versions | grep admission
admissionregistration.k8s.io/v1
admissionregistration.k8s.io/v1beta1

If you need to revert the following to v1beta, you can judiciously search-and-replace v1 with v1beta for e.g. AdmissionReview and Admission[Request|Response]. You will need to

Import the following

  • "k8s.io/api/admission/v1beta1" (instead of "k8s.io/api/admission/v1b")
  • apiextv1beta1 "k8s.io/apiextensions-apiserver/pkg/apis/apiextensions/v1beta1"

And revise the section of code to include apiextv1beta1:

sch := runtime.NewScheme()
_ = clientgoscheme.AddToScheme(sch)
_ = apiextv1beta1.AddToScheme(sch)

Webhook

The webhook handler is trivial:

var (
	crtFile = flag.String("tls-crt-file", "", "TLS certificate file")
	keyFile = flag.String("tls-key-file", "", "TLS key file")
	port    = flag.Int("port", 0, "Webhook Port")
)

func main() {
    cert, _ := tls.LoadX509KeyPair(*crtFile, *keyFile)
	config := &tls.Config{
		Certificates: []tls.Certificate{cert},
	}

	http.HandleFunc("/validate", validate)

    addr := fmt.Sprintf(":%d", *port)
	server := &http.Server{
		Addr:      addr,
		TLSConfig: config,
	}

	log.Fatal(server.ListenAndServeTLS("", ""))
}

NOTE validate is a regular Golang HTTP handler.

Kubernetes requires that webhooks are secured by TLS.

For testing (!) purposes, you can generate a self-signed certificate and key, using openssl:

FILENAME="localhost"

openssl req \
-x509 \
-nodes \
-newkey rsa:2048 \
-keyout ${FILENAME}.key \
-out ${FILENAME}.crt \
-days 365 \
-subj "/CN=localhost"

NOTE -x509 does not generate a certificate-signing request (CSR). When you deploy to Kubernetes, you’ll need to use the instructions in the Certificates section below to include a CSR and have the Kubernetes cluster approve this in order to generate a correctly-signed certificate.

The validate function is where we grab AdmissionReview messages that are shipped to us by Kubernetes to invoke our webhook. We’ll create an AdmissionResponse to send back.

func validate(w http.ResponseWriter, r *http.Request) {
	contentType := r.Header.Get("Content-Type")
	if contentType != "application/json" {
		w.WriteHeader(http.StatusBadRequest)
		return
	}

    var body []byte
	if r.Body != nil {
		if data, err := ioutil.ReadAll(r.Body); err == nil {
			body = data
		}
	}

	rqst := v1beta1.AdmissionReview{}

	sch := runtime.NewScheme()
	_ = clientgoscheme.AddToScheme(sch)
	_ = apiextv1beta1.AddToScheme(sch)

	decode := serializer.NewCodecFactory(sch).UniversalDeserializer().Decode

	_, _, err := decode(body, nil, &rqst)
	if err != nil {
		w.WriteHeader(http.StatusBadRequest)
		return
	}

	if rqst.Request == nil {
		w.WriteHeader(http.StatusBadRequest)
		return
	}

	resp := validateSomething(rqst.Request)

	bytes, err := json.Marshal(&v1beta1.AdmissionReview{
		Response: resp,
	})
	if err != nil {
		w.WriteHeader(http.StatusInternalServerError)
		return
	}
	w.Write(bytes)
}
  1. The message will be POSTed to the webhook (not captured here)
  2. The Content-Type of the incoming request must be application/json
  3. The request body is what we’re interested in
  4. runtime, clientgoscheme; and apiextv1beta is an incantation that I discovered in Kubernetes’ webhook tests (See: webhook.go)
  5. Combined with magic from [UniversalDeserializer](https://pkg.go.dev/k8s.io/apimachinery/pkg/runtime/serializer#CodecFactory.UniversalDeserializer) gives us the ability to unmarshal the request body into an AdmissionReview`
  6. An AdmissionReview includes optional AdmissioRequest and AdmissionResponse types
  7. This is what we’re after and encapsulates the request (AdmissionRequest) that our webhook should validate; in this case validateSomething returning an AdmissionResponse both of which are wrapped by the AdmissionReview type.

Up to this point, the code has been generic and applicable to any ValidatingWebhookConfiguration. I’m going to leave the validateSomething open to your implementation:

func validateSomething(rqst v1beta1.AdmissionRequest) *v1beta1.AdmissionResponse {
	// See: https://github.com/kubernetes/apimachinery/issues/102
	raw := rqst.Object.Raw

	if len(raw) == 0 {
		return &v1beta1.AdmissionResponse{
			UID:     uid,
			Allowed: false,
			Result: &metav1.Status{
				Message: "object contains no data",
			},
		}
	}

	something := &Something{}

	if err := json.Unmarshal(raw, something); err != nil {
		return &v1beta1.AdmissionResponse{
			UID:     uid,
			Allowed: false,
			Result: &metav1.Status{
				Message: err.Error(),
			},
		}
	}

    // Process `something`
    // If there are any problems with `something`
	return &v1beta1.AdmissionResponse{
		UID:     rqst.Request.UID,
        Allowed: false,
        Result: &metav1.Status{
				Message: message,
        },
    }

    // If we reach this far, all's good
	return &v1beta1.AdmissionResponse{
		UID:     rqst.Request.UID,
		Allowed: true,
    }
}
  1. The configuration that was attempt applied, deleted or updated to the cluster, triggering the webhook, is accessible as AdmissionReview.Request.Object.Raw
  2. It’s possible best to limit one webhook to one Kubernetes resource|Kind but this isn’t a requirement. If your webhook is invoked for multiple resource types, you’ll need to disambiguate them here.
  3. Then the raw object must be unmarshaled into a Golang type (that we can process), in the above, a type called Something.
  4. At any point during the webhook’s evaluation, we can early return an AdmissionResponse with its Allowed property set to false. The handler will report that it denied the request and we can provide a more detailed error message using Result.Message.
  5. It is import thant all AdmissionResponse messages include the incoming request’s Request.UID so that these can be matched.
  6. It is convention that Golang functions early return errors and then conclude with the happy state Allowed: true as shown above. If none of our tests fails, the configuration file passes muster.

You may test the handler (using the localhost cert and key) POST’ing an AdmissionRequest body to the server that includes some form of your Something object.

Certificates

Before we can deploy to Kubernetes, we’ll need to create a certificate signing request. The following script is drawn heavily from Kubernetes Create CertificateSigningRequest:

  1. Uses openssl to create a private key and a certificate signing request (csr). Importantly, the certificate’s Common Name must be ${SERVICE}.${NAMESPACE}.svc. This is a qualified service name for in-cluster use and ensures that the service is accessible across namespaces.
  2. Creates a Kubernetes CertificateSigningRequest using the csr file generated by openssl. The name of this is just ${SERVICE}.${NAMESPACE}
  3. kubectl is used to approve the CSR
  4. kubectl is used to get the signed certificate (.status.certificate) from the cluster
  5. kubectl is used to create a Secret that will be mounted into the Deployment combining this certificate and the previously created private key.
DIR=${PWD}/secrets
SERVICE="webhook"
NAMESPACE="default"

FILENAME="${DIR}/${SERVICE}.${NAMESPACE}"

openssl req \
-new \
-sha256 \
-newkey rsa:2048 \
-keyout ${FILENAME}.key \
-out ${FILENAME}.csr \
-nodes \
-subj "/CN=${SERVICE}.${NAMESPACE}.svc"

echo "
apiVersion: certificates.k8s.io/v1beta1
kind: CertificateSigningRequest
metadata:
  name: ${SERVICE}.${NAMESPACE}
spec:
  groups:
  - system:authenticated
  request: $(cat ${FILENAME}.csr | base64 | tr -d '\n')
  usages:
  - digital signature
  - key encipherment
  - server auth
" | kubectl apply --filename=-

kubectl certificate approve ${SERVICE}.${NAMESPACE}

kubectl get csr ${SERVICE}.${NAMESPACE} \
--output=jsonpath='{.status.certificate}' \
| base64 --decode > ${FILENAME}.crt

kubectl create secret tls ${SERVICE} \
--namespace=${NAMESPACE} \
--cert=${FILENAME}.crt \
--key=${FILENAME}.key

Deployment

We’re now able to deploy the webhook handler to the cluster.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: SERVICE
  namespace: NAMESPACE
  labels:
    component: webhook
spec:
  replicas: 1
  selector:
    matchLabels:
      component: webhook
  template:
    metadata:
      labels:
        component: webhook
    spec:
      containers:
        - name: webhook
          image: your-image
          imagePullPolicy: Always
          args:
            - --tls-crt-file=/secrets/tls.crt
            - --tls-key-file=/secrets/tls.key
            - --port=8443
            - --logtostderr
            - -v=2
          volumeMounts:
            - name: secrets
              mountPath: /secrets
              readOnly: true
      volumes:
        - name: secrets
          secret:
            secretName: SERVICE

NOTE the above includes PLACEHOLDER values that we must replace with the value created during the previous (Certificates) step. To do this, you can use sed to replace PLACEHOLDER with values:

cat ./deployment.yaml \
| sed "s|SERVICE|${SERVICE}|g" \
| sed "s|NAMESPACE|${NAMESPACE}|g" \
| kubectl apply --filename=- --namespace=${NAMESPACE}

Service

The Service exposes the Deployment (on port :8443) on port of :443:

apiVersion: v1
kind: Service
metadata:
  name: SERVICE
  namespace: NAMESPACE
  labels:
    component: webhook
spec:
  selector:
    component: webhook
  ports:
    - name: http
      port: 443
      targetPort: 8443

Again, we must replace the PLACEHOLDER values:

cat ./service.yaml \
| sed "s|SERVICE|${SERVICE}|g" \
| sed "s|NAMESPACE|${NAMESPACE}|g" \
| kubectl apply --filename=- --namespace=${NAMESPACE}

ValidatingAdmissionWebook

Finally, we can combine the Service (using the Deployment) into a ValidatingWebhookConfiguration.

The configuration is admissionregistration.k8s.io/v1

This process was kinda gnarly too but I found a configuation that worked for me:

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  name: SERVICE
  namespace: NAMESPACE
  labels:
    component: webhook
webhooks:
  - name: SERVICE.NAMESPACE.svc
    clientConfig:
      service:
        name: SERVICE
        namespace: NAMESPACE
        port: 443
        path: "/validate"
      caBundle: CABUNDLE
    rules:
      - operations:
          - "CREATE"
          - "UPDATE"
        apiGroups:
          - "akri.sh"
        apiVersions:
          - "v0"
        resources:
          - "configurations"
        scope: "*"
    admissionReviewVersions:
      - v1beta1
    sideEffects: None
  1. webhooks.name must match the Common Name from the certificate signing request, i.e. ${SERVICE}.${NAMESPACE}.svc. This is qualified endpoint by which the webhook may be accessed within the cluster.
  2. We’ll use port :443 as exposed by the Service
  3. The path to the webhook was defined (statically) in the Golang code /validate (but could of course be defined by a flag or environment variable if preferred).
  4. Instructions to grab the value for CABUNDLE are shown below.
  5. The rules define the trigger for the webhook. I found this very difficult to get right and so I’m leaving this as-is here to hopefully help you. In my case the Akri Configuration CRD is defined as akri.sh/v0/configurations and this decomposes into an apiGroups with value akri.sh, an apiVersions with value v0 and one resource (though you can provide multiple) of configurations. In my case, I want kubectl apply (i.e. CREATE and UPDATE but not DELETE) operations.
  6. The scope can be Namespaced or Cluster but I left this as * (either).
  7. You’ll recall that our Golang code used v1beta1.Admission[Review|Response] and so that’s referenced here too.

We must also provide this resource with cluster’s CA bundle:

CABUNDLE=$(\
  kubectl get secrets \
  --namespace=${NAMESPACE} \
  --output=jsonpath="{.items[?(@.metadata.annotations['kubernetes\.io/service-account\.name']=='default')].data.ca\.crt}"\
) && echo ${CABUNDLE}

Then, we can replace this along with the SERVICE and NAMESPACE values using sed:

cat ./webhook.yaml \
| sed "s|SERVICE|${SERVICE}|g" \
| sed "s|NAMESPACE|${NAMESPACE}|g" \
| sed "s|CABUNDLE|${CABUNDLE}|g" \
| kubectl apply --filename=- --namespace=${NAMESPACE}

Testing

You may now test your webhook handler by CREATEing or UPDATEing whatever resource you defined as Something.

You may use curl to test the Golang (locally) (ENDPOINT=localhost:8443) or Kubernetes service too (ENDPOINT=${SERVICE}.${NAMESPACE}.svc:443)

curl \
--silent \
--insecure \
--cert ${FILENAME}.crt \
--key ${FILENAME}.key \
--header "Content-Type: application/json" \
--data "@./admissionreview.json" \
https://${ENDPOINT}/validate)

Using an admissionreview.json of the form:

{
    "kind": "AdmissionReview",
    "apiVersion": "admission.k8s.io/v1",
    "request": {
        "uid": "12345678-1234-1234-1234-1234567890ab",
        "kind": {
            "group": "akri.sh",
            "version": "v0",
            "kind": "Configuration"
        },
        "resource": {
            "group": "akri.sh",
            "version": "v0",
            "resource": "configurations"
        },
        "requestKind": {
            "group": "akri.sh",
            "version": "v0",
            "kind": "Configuration"
        },
        "requestResource": {
            "group": "akri.sh",
            "version": "v0",
            "resource": "configurations"
        },
        "name": "something",
        "namespace": "something",
        "operation": "CREATE",
        "userInfo": {
            "username": "admin",
            "uid": "admin",
            "groups": [
                "system:masters",
                "system:authenticated"
            ]
        },
        "object": {
            "apiVersion": "akri.sh/v0",
            "kind": "Configuration",
            ...
        },
    }
}

NOTE This JSON is based upon the akri.sh/v0/configurations discussed, the .request details would need to refer to your something.

That’s all!