Kubernetes cert-manager

January 8, 2021 - 6 minutes read - 1130 words

I developed an admission webhook for Akri, twice (Golang, Rust). I naively followed other examples for the generation of the certificates, created a 1.20 cluster and broke that process.

I’d briefly considered using cert-manager recently but quickly abandoned the idea thinking it would be onerous and unnecessary complexity for little-old-me. I was wrong. It’s excellent and I recommend it highly.

I won’t reproduce the v1beta1 and v1 examples from the Stackoverflow question as they should be self-explanatory. I suspect (!?) that I should not have used Kubernete’s (API Server’s) CA for the Webhook but it could well be that I just don’t understand the correct approach.

It took me under 2 hours to get cert-manager working though.

I’ll leave it to you to deploy cert-manager in a cluster.

I’m going to describe the new process that I have for creating a self-signed CA and then signing a Service Certificate for the Webhook with it.

First, some environment variables… Revise as you wish:

DIR=${PWD}/secrets

CA="ca"
SERVICE="yirella"
NAMESPACE="salvation"

Optional kubectl create namespace ${NAMESPACE}

Second, use openssl to generate a private key and certificate for our CA.

In practice, you’d use a legitimate CA-signer and would skip this step.

openssl req \
-nodes \
-new \
-x509 \
-keyout ${DIR}/${NAMESPACE}.${CA}.key \
-out ${DIR}/${NAMESPACE}.${CA}.crt \
-subj "/CN=${CA}" \
-days 365

NOTE Common Names (CN) are deprecated but used here primarily for labelling. If you prefer, you could write a config files, generate a CSR and (self-)sign that.

Third, cert-manager manages Certificates using Kubernetes Secrets. So, let’s create a Secret from the private key and certificate we just created:

kubectl create secret tls ${CA} \
--namespace=${NAMESPACE} \
--cert=${DIR}/${NAMESPACE}.${CA}.crt \
--key=${DIR}/${NAMESPACE}.${CA}.key

NOTE I’m create this in a working namespace (${NAMESPACE}) because I’m going to use Issuer. If you want cluster-wide issuance, this would need to be created (by default) in cert-manager namespace (configurable) and you’d use ClusterIssuer.

Fourth, create an Issuer from the Secret:

echo "
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: ${CA}
  namespace: ${NAMESPACE}
spec:
  ca:
    secretName: ${CA}
" | kubectl apply --filename=-

You may confirm this:

kubectl get issuer/${CA} \
--namespace=${NAMESPACE} \
--output=wide

NAME   READY   STATUS                AGE
ca     True    Signing CA verified   15s

Fifth, create a Certificate (and Secret) for the Service using the Issuer.

NOTE In my case, with the Webhook, I found it necessary to include the Webhook’s Endpoint in its certificate. You may be able to drop both commonName and ipAddresses entries from the following specification.

# Endpoint for Service `${SERVICE}.${NAMESPACE}.svc`
ENDPOINT=(\
  kubectl get service/${SERVICE} \
  --namespace=${NAMESPACE} \
  --output=jsonpath="{.spec.clusterIP}")

echo "
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: ${SERVICE}
  namespace: ${NAMESPACE}
spec:
  # Output
  secretName: ${SERVICE}
  duration: 8760h
  renewBefore: 720h
  subject:
  commonName: ${ENDPOINT}
  isCA: false
  privateKey:
    algorithm: RSA
    encoding: PKCS1
    size: 2048
  usages:
    - server auth
  dnsNames:
  - ${SERVICE}.${NAMESPACE}.svc
  - ${SERVICE}.${NAMESPACE}.svc.cluster.local
  ipAddresses:
  - ${ENDPOINT}
  issuerRef:
    name: ca
    kind: Issuer
    group: cert-manager.io
" | kubectl apply --filename=-

You may confirm this:

kubectl get certificate/${SERVICE} \
--namespace=${NAMESPACE} \
--output=wide

NAME     READY   SECRET   ISSUER   STATUS                                          AGE
yirella  True    yirella  ca       Certificate is up to date and has not expired   15s

This results in the creation of a Secret (${SERVICE}) in Namespace (${NAMESPACE}) signed by the self-signed CA

We can get the Secret:

kubectl get secret/${SERVICE} \
--namespace=${NAMESPACE} \
--output=yaml

This yields something similar to:

apiVersion: v1
data:
  ca.crt: LS0tLS1C...
  tls.crt: LS0tLS1C...
  tls.key: LS0tLS1C...
kind: Secret
metadata:
  name: callum
  namespace: salvation
type: kubernetes.io/tls

You’ll note that the Secret includes a private key (tls.key) and certificate (tls.crt) for the Service and the CA’s certificate (ca.crt). We can grab e.g. the Service certificate, base64 decode it and have openssl display it for us:

kubectl get secret/${SERVICE} \
> --namespace=${NAMESPACE} \
> --output=jsonpath="{.data.tls\.crt}" \
> | base64 --decode \
> | openssl x509 -in - -noout -text

Which yields:

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN = ca
        Validity
            Not Before: Jan  8 20:50:37 2021 GMT
            Not After : Jan  8 20:50:37 2022 GMT
        Subject: CN = 10.138.0.2
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                RSA Public-Key: (2048 bit)
                Modulus:
        X509v3 extensions:
            X509v3 Extended Key Usage: 
                TLS Web Server Authentication
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Authority Key Identifier: 
            X509v3 Subject Alternative Name: 
                DNS:callum.salvation.svc, DNS:callum.salvation.svc.cluster.local, IP Address:10.138.0.2
    Signature Algorithm: sha256WithRSAEncryption

I’ve deleted all the superfluous details but:

Expiry is 365 days (8760 hours) from today (Jan 8 2021 –> Jan 8 2020)
CN is the value of the ${ENDPOINT} that was provided
DNS entries are (correct) Kubernetes Services names ${SERVICE}.${NAMESPACE}.svc[.cluster.local]

Sixth, we now have everything that we need to configure a Kubernetes Webhook

You’ll recall we already exposed a Service (to get its Endpoint) but this was backed by no Pods.

We can create a Deployment to underpin this Service, configured to use the TLS certificate and private key generated by cert-manager:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: SERVICE
  namespace: NAMESPACE
  labels:
    component: webhook
spec:
  replicas: 1
  selector:
    matchLabels:
      component: webhook
  template:
    metadata:
      labels:
        component: webhook
    spec:
      containers:
        - name: webhook
          image: some/image
          imagePullPolicy: Always
          args:
            - --tls-crt-file=/secrets/tls.crt
            - --tls-key-file=/secrets/tls.key
            - --port=8443
          volumeMounts:
            - name: secrets
              mountPath: /secrets
              readOnly: true
      volumes:
        - name: secrets
          secret:
            secretName: SERVICE

NOTE The VARIABLE names can be replaced at kubectl apply using sed:
cat deployment.yaml \
| sed "s|SERVICE|${SERVICE}|g" \
| sed "s|NAMESPACE|${NAMESPACE}|g" \
| kubectl apply --filename=-

This is an exemplar Deployment spec that runs a container (webhook) configure to use a TLS certificate and private key mounted using the Secret, in this case, generated for us by cert-manager.

Then we can configure a Kubernetes Webhook using the (now backed) Service with the CA bundle (also present in the Secret as ca.crt):

CABUNDLE=$(\
  kubectl get secret/${SERVICE} \
  --namespace=${NAMESPACE} \
  --output=jsonpath="{.data.ca\.crt}") && echo ${CABUNDLE}

With:

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  name: SERVICE
  namespace: NAMESPACE
  labels:
    component: webhook
webhooks:
  - name: SERVICE.NAMESPACE.svc
    clientConfig:
      service:
        name: SERVICE
        namespace: NAMESPACE
        port: 443
        path: "/validate"
      caBundle: CABUNDLE
    rules:
      - operations:
          - "CREATE"
          - "UPDATE"
        apiGroups:
          - "akri.sh"
        apiVersions:
          - "v0"
        resources:
          - "configurations"
        scope: "*"
    admissionReviewVersions:
      - v1
    sideEffects: None

NOTE You’ll need to revise the ValidatingWebhookConfiguration for your purposes but this shows the overall structure. It’s important that the webhook.name be ${SERVICE}.${NAMESPACE}.svc and the use of caBundle

NOTE Not shown here, the Service that exposes the Deployment’s Pods maps the Pod’s port :8443 to a Service port: :443 which is more conventional for TLS and required (!?) by Kubernetes Webhooks.

NOTE Kubernetes is configured with the Service’s (!) CA bundle and it validates the Webhook service’s certificate but the Webhook service does not validate the client (i.e. Kubernetes’ API Server).

When you’re done, you can tidy simply by deleting the ${NAMESPACE} namespace or, more delicately:

# NB Webhooks aren't namespaced
kubectl delete validatingwebhookconfiguration/${SERVICE}

kubectl delete deployment/${SERVICE} \
--namespace=${NAMESPACE}

kubectl delete service/${SERVICE} \
--namespace=${NAMESPACE}

kubectl delete secret/${SERVICE} \
--namespace=${NAMESPACE}

kubectl delete certificate/${SERVICE} \
--namespace=${NAMESPACE}

kubectl delete issuer/${SERVICE} \
--namespace=${NAMESPACE}

That’s all!