Secure (TLS) gRPC services with LKE

February 15, 2023 - 8 minutes read - 1556 words

NOTE cert-manager is a better solution to what follows.

I wrote about deploying Secure (TLS) gRPC services with Vultr Kubernetes Engine (VKE). This week, I’ve reproduced this deployment using Linode Kubernetes Engine (LKE).

Thanks to the consistency provided by Kubernetes, the Kubernetes programming is almost identical. The main differences are between the CLI’s provided by these platforms. Both are good. They’re just different.

I’m going to include the linode-cli commands I’m using in this post as I found it slightly more quirky.

I’m reusing an existing solution Automatic Certs w/ Golang gRPC service on Compute Engine that combines a gRPC Healthchecking and Let’s Encrypt’s ACME service.

Cluster

I had been using the linode-cli Snap but this is no longer being updated (snap-linode-cli) and so I’ve switched to use the container image.

In what follows, when you see linode-cli, I’ve aliased this to:

VERS="5.31.0"
IMAGE=""docker.io/linode/cli:${VERS}"

podman run \
--interactive --rm \
--volume=${HOME}/.config/linode-cli:/home/cli/.config/linode-cli \
${IMAGE}

I want to use the latest Kubernetes API version:

VERSION=$(\
  linode-cli lke versions-list --json \
  | jq -r '.[0].id') && echo ${VERSION}

if [ -z "${VERSION}" ]
then
  echo "Unable to determine Kubernetes version: ${VERSION}"
  return 1
fi

And create single (!) node clusters using g6-standard-1 Linodes:

LABEL="..."
REGION="us-west" # Or ...

ID=$(\
  linode-cli lke cluster-create \
    --label=${LABEL} \
    --region=${REGION} \
    --k8s_version=${VERSION} \
    --node_pools.type=g6-standard-1 \
    --node_pools.count=1 \
    --no-defaults \
    --json \
  | jq -r '.[0].id') && echo ${ID}

The {ID} value can be used subsequently to interact with the cluster but the result from lke cluster-create does not include any form of status of the cluster. Instead, I’m repeatedly checking for the availability of the KUBECONFIG using lke kubeconfig-view:

KUBECONFIG=""

until [ -n "${KUBECONFIG}" ]
do
  sleep 15s
  KUBECONFIG=$(\
    linode-cli lke kubeconfig-view ${ID} --json \
    | jq -r .[0].kubeconfig)
done

For ease, I’m persisting the KUBECONFIG to a file with a name that includes today’s date. This is likely overkill. I’m deleting clusters within the day and so I’d never be able to return to an old configuration, but…

CONFIG="${PWD}/linode.$(date +%y%m%d).yaml"

echo ${KUBECONFIG} | base64 --decode > ${CONFIG}

Once the cluster is available and accessible, we can interact with it, using the above KUBECONFIG file:

kubectl get nodes \
--kubeconfig=${CONFIG}

Namespace

I prefer to not use default and so the first step is to create a namespace:

NAMESPACE="healthcheck"

kubectl \
create namespace ${NAMESPACE} \
--kubeconfig=${CONFIG}

Storage

Linode’s documentation is good and there is clear guidance for Deploying PVCs with Linode Block Storage CSI Driver.

The underlying resource type used for PVCs is Block Storage manifest as Volumes

NOTE Be careful. In my experience, Volumes aren’t deleted when PV(C)s are deleted. I am having to manually delete Volumes after LKE cluster deletion.

Here’s my minimal PVC:

pvc.yaml:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: autocert
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  storageClassName: linode-block-storage

NOTE I’m using linode-block-storage (not linode-block-storage-retain) and expected the underlying Volume to be deleted with the PVC. This appears to not happen.

And a Pod with a (busybox) shell that mounts the above PVC:

shell.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: shell
spec:
  volumes:
  - name: autocert
    persistentVolumeClaim:
      claimName: autocert
  containers:
  - name: busybox
    image: busybox
    args: 
    - /bin/sh
    - -c
    - |
      while true
      do
        sleep 30
      done      
    volumeMounts:
    - mountPath: "/certs"
      name: autocert

We can then create the PVC, create the Pod shell with the shell, once it’s ready, we can use kubectl cp to copy the local files into the PVC through the Pod’s container’s shell. If (!) the kubectl cp succeeds, we can delete the Pod:

# Create PVC
kubectl apply \
--filename=${PWD}/linode/kubernetes/pvc.yaml \
--namespace=${NAMESPACE} \
--kubeconfig=${CONFIG}

# Create Pod w/ a shell
kubectl apply \
--filename=${PWD}/linode/kubernetes/shell.yaml \
--namespace=${NAMESPACE} \
--kubeconfig=${CONFIG}

# Wait for it to become ready
kubectl wait pod/shell \
--for=condition=Ready \
--namespace=${NAMESPACE} \
--kubeconfig=${CONFIG}

# Copy locally backed up certs onto PVC
if kubectl cp \
   ${PWD}/linode/certs/. \
   ${NAMESPACE}/shell:/certs \
   --kubeconfig=${CONFIG}
then
  echo "Copied successfully"
else
  echo "Copy failed"
  return 1
fi

# Delete the Pod w/ a shell
kubectl delete \
--filename=${PWD}/linode/kubernetes/shell.yaml \
--namespace=${NAMESPACE} \
--kubeconfig=${CONFIG}

You can debug this step by (re)creating the Pod and using its shell to e.g. list the content of the certs folder which should contain the copied (Let’s Encrypt) certificate and account files:

kubectl exec \
--stdin --tty \
pod/shell \
--kubeconfig=${CONFIG} \
--namespace=${NAMESPACE} \
-- ls -la certs

You can verify that the PVC is backed by a Linode Volume by enumerating the Volumes:

linode-cli volumes list

Yielding:

┌────┬───────┬────────┬──────┬────────┬───────────┬──────────────┐
│ id │ label │ status │ size │ region │ linode_id │ linode_label │
└────┴───────┴────────┴──────┴────────┴───────────┴──────────────┘

Deployment

I’m using a Secret to contain the credentials necessary to access a private container registry:

REGISTRY=""
USER=""
PASS=""
EMAIL=""

kubectl create secret docker-registry registry \
--docker-server=${REGISTRY} \
--docker-username=${USER} \
--docker-password=${PASS} \
--docker-email=${EMAIL}
--namespace=${NAMESPACE} \
--kubeconfig=${CONFIG}

The Deployment is straightfoward. It’s templated and sed is used to replace the values of IMAGE and HOST:

autocert.tmpl:

apiVersion: v1
kind: List
metadata: {}
items:
- kind: Service
  apiVersion: v1
  metadata:
    labels:
      app: autocert
      type: server
    name: autocert
  spec:
    type: LoadBalancer
    selector:
      app: autocert
      type: server
    ports:
    - name: http
      port: 80
      targetPort: 80
    - name: grpc
      port: 443
      targetPort: 50051
- kind: Deployment
  apiVersion: apps/v1
  metadata:
    labels:
      app: autocert
      type: server
    name: autocert
  spec:
    replicas: 1
    selector:
      matchLabels:
        app: autocert
        type: server
    template:
      metadata:
        labels:
          app: autocert
          type: server
      spec:
        imagePullSecrets:
        - name: ghcr
        containers:
        - name: autocert
          image: IMAGE
          command:
          - /autocert
          args:
          - --host=HOST
          - --port=50051
          - --path=/certs
          ports:
          - name: http
            containerPort: 80
          - name: grpc
            containerPort: 50051
          volumeMounts:
          - mountPath: "/certs"
            name: autocert
        volumes:
        - name: autocert
          persistentVolumeClaim:
            claimName: autocert
        restartPolicy: Always

sed is used to revise the templated Deployment and create today’s DEPLOYMENT which is then kubectl apply’d to the cluster:

IMAGE="..."
NAME="autocert"
HOST="${NAME}.example.com"

DEPLOYMENT="${PWD}/autocert.$(date +%y%m%d).yaml"

sed \
--expression="s|image: IMAGE|image: ${IMAGE}|g" \
--expression="s|host=HOST|host=${HOST}|g" \
${PWD}/linode/kubernetes/autocert.tmpl > ${DEPLOYMENT}

# Deploy autocert server w/ PVC mount to access {HOST} X509 cert
kubectl apply --filename=${DEPLOYMENT} \
--namespace=${NAMESPACE} \
--kubeconfig=${CONFIG}

NodeBalancer

The autocert config includes a Deployment and a Service. The autocert container is exposed as a Kubernetes Service using type: LoadBalancer and, this creates a Linode NodeBalancer. Because the container is terminating TLS, the NodeBalancer config is very straightforward. The autocert container uses the default HTTP port (80) to support interaction with Let’s Encrypt’s ACME protocol and it uses port 50051 for the gRPC service. The gRPC port is mapped to the default HTTPs port (443) on the Kubernetes Service (and Linode NodeBalancer).

You can verify that a NodeBalancer has been created with:

linode-cli nodebalancers list

Yielding:

┌────┬───────┬────────┬──────────┬──────┬──────┬──────────────────────┐
│ id │ label │ region │ hostname │ ipv4 │ ipv6 │ client_conn_throttle │
└────┴───────┴────────┴──────────┴──────┴──────┴──────────────────────┘

You will need to determine the IPv4 (possibly IPv6) address of the NodeBalancer in order to create|update your domain’s DNS records:

# Either use Kubernetes Service
IP=$(\
  kubectl get service/autocert \
  --output=jsonpath="{.status.loadBalancer.ingress[0].ip}" \
  --namespace=${NAMESPACE} \
  --kubeconfig=${CONFIG} \
) && echo ${IP}

# Or use the Linode NodeBalancer
IP=$(\
  linode-cli nodebalancers list --json \
  | jq -r .[0].ipv4 \
) && echo ${IP}

Both results should be the same ;-)

DNS

Program your domain’s DNS records using the {HOST} name you specified when you deployed autocert.

You will want to be able to dig the {HOST} and get the {IP} before proceeding to test the service.

[ $(dig ${HOST} A -4 +short) == "{IP}" ] \
&& echo "True" \
|| echo "False"

Test

Once you’re confident that DNS is correctly resolving your {HOST} to the NodeBalancer’s {IP}, you can:

grpcurl ${HOST}:443 grpc.health.v1.Health/Check

And you should see:

{
  "status": "SERVING"
}

Tidy

In my case, I reverse the above steps to delete autocert, the cluster and then importantly delete the vestigial Linode Volume (that is not deleted as expected by the deletion of the PVC).

Before I delete the PVC, I reverse the flow of the kubectl cp and take a copy of any potentially updated certs before deleting the copy in the PVC:

# Delete autocert server
kubectl delete --filename=${DEPLOYMENT} \
--namespace=${NAMESPACE} \
--kubeconfig=${CONFIG}

# Create Pod w/ a shell
kubectl apply \
--filename=${PWD}/linode/kubernetes/shell.yaml \
--namespace=${NAMESPACE} \
--kubeconfig=${CONFIG}

# Wait for it to come Ready
kubectl wait pod/shell \
--for=condition=Ready \
--namespace=${NAMESPACE} \
--kubeconfig=${CONFIG}

# Backup cert
if kubectl cp \
  ${NAMESPACE}/shell:/certs/${HOST} \
  ${PWD}/${HOST}.$(date +%y%m%d) \
--kubeconfig=${CONFIG}
then
  echo "Certificate backed up"
else
  echo "Failed to back up certificate"
  return 1
fi

# Delete Pod
kubectl delete \
--filename=${PWD}/linode/kubernetes/shell.yaml \
--namespace=${NAMESPACE} \
--kubeconfig=${CONFIG}

# Delete PVC (does not delete underlying Block Storage)
kubectl delete \
--filename=${PWD}/linode/kubernetes/pvc.yaml \
--namespace=${NAMESPACE} \
--kubeconfig=${CONFIG}

# Delete secret for GHCR
kubectl delete secret/ghcr \
--namespace=${NAMESPACE} \
--kubeconfig=${CONFIG}

# Delete namespace
kubectl delete namespace/${NAMESPACE}

Importantly, check for and delete the Linode Volume created for the PVC:

linode-cli volumes list

┌────┬───────┬────────┬──────┬────────┬───────────┬──────────────┐
│ id │ label │ status │ size │ region │ linode_id │ linode_label │
└────┴───────┴────────┴──────┴────────┴───────────┴──────────────┘

In my case, the only Volumes that exist will be vestigial PVC Volumes and so I’m deleting all of them. You don’t want to do this but you should delete any Volumes that are no longer needed:

local IDS=$(\
  linode-cli volumes list --json \
  | jq -r '.[].id' \
) && echo ${IDS}

for ID in ${IDS}
do
  linode-cli volumes delete ${ID}
done

local LENGTH=$(\
  linode-cli volumes list --json \
  | jq -r '.|length' \
) && echo ${LENGTH}

if [ "${LENGTH}" -ne 0 ]
then
  echo "${LENGTH} Linode Volumes remain"
  return 1
fi

That’s all!