NGINX Ingress

- 6 minutes read - 1183 words

I’ve written a couple of deployment options (Google Compute Engine; Kubernetes) for an open-source project. The Kubernetes deployment provides NodePort and (TCP) LoadBalancer options and I’ve been trying (unsuccessfully) to add HTTPS Load-balancing.

I should (!) try to deploy to Google Kubernetes Engine (GKE) but I’ve been using microk8s, Digital Ocean Managed Kubernetes and the Linode LKE Beta. Each of these requires an implementation of Ingress controller. For GKE, GCP’s HTTP/S Load-balancer (GCLB) is used. But, for the other services, NGINX Ingress is a good option and so I’ve been exploring NGINX Ingress (Controller) link. This Ingress appears to work except that I’ve been unable to get it to work with mutual TLS between the (NGINX) proxy and my backend services.

This may be related to my problem.

For microk8s, a TCP Load-balancer is required too. MetalLB just works.

kubectl apply --filename=https://raw.githubusercontent.com/google/metallb/v0.8.3/manifests/metallb.yaml

echo "apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    address-pools:
    - name: default
      protocol: layer2
      addresses:
      - 192.168.1.240-192.168.1.250
" | kubectl apply --filename=-

kubectl apply --filename=https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/mandatory.yaml

microk8s.enable ingress

NB In the above, I’m allocating 192.168.1.240-192.168.1.250 to MetalLB to use in creating Load-balancers.

To test:

kubectl run kuard --image=gcr.io/kuar-demo/kuard-amd64:blue --port=8080
kubectl expose deployment/bigmachine-${ID} --type=LoadBalancer --port=8080

Then browse: http://$(kubectl get service/kuard --output=jsonpath="{.status.loadBalancer.ingress[0].ip}"):8080

To tidy-up:

kubectl delete deployment/kuard
kubectl delete service/kuard

Mutual TLS

I then used openssl to generate ‘CA’ cert|key and server cert|key PEM files. I combined the .crt and .key files into singular .pem files too.

files: https://gist.github.com/DazWilkin/ac6ca02a9bee44e5d5621aabe233eb1b

Lastly – and where I’ve been spending my time – the Kubernetes spec to create secrets, Deployments|Services and an Ingress. The file is at the end of this document.

In my file, there are 3 distinct Deployments|Services and path specs but, for conciseness, I’ve collapsed that to a single Deployment|Service and 3 path specs.

kubectl apply --filename=test.yaml

All being well:

kubectl get services --namespace=test

And:

NODEPORT=$(\
  kubectl get services/something-00 \
  --namespace=test \
  --output=jsonpath="{.spec.ports[0].nodePort}")

curl --insecure --cert /path/to/ca.pem https://localhost:${NODEPORT}/healthz
ok
curl --insecure --cert /path/to/ca.pem https://localhost:${NODEPORT}/certs
-----BEGIN CERTIFICATE-----
MIIC6TCCAdGgAwIBAgIBATANBgkqhkiG9w0BAQsFADAVMRMwEQYDVQQDEwpiaWdt
YWNoaW5lMCAXDTE5MTIwMjE5MjIyOVoYDzIwNjAwMTAxMDAwMDAwWjAVMRMwEQYD
VQQDEwpiaWdtYWNoaW5lMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA
vG3g/mKgWUkKmC3PJ8pKFXW6Gd4LMpVcMLzxJfT9RbtrwWl9FVFUEBeROlYJIvS8
5gccuZvW870noGOvMvcZNl+VjeJ/RjQeblp4siS+aGiCzCJ/l8+NsHY+j5nN6GPR
3zbS3P/sNXjtzWcLX759rpxEUzaIW6j1TgMGkG39PEUyqoOj513OzEtCrAKV/zIh
bizmdyTiHmDg7aZPj45XkojMybCdWDxOPBCcvb4qov0FPf8EaeKTzwLADvne1pCZ
uhydMOfluIpZ0AxORpxpQCLQT9ZY4voG3GQQyOlgiBn0VvXJjtpOvWNI8t4NCzUy
PLVl7HuPphJa0u4os/NBsQIDAQABo0IwQDAOBgNVHQ8BAf8EBAMCAqQwHQYDVR0l
BBYwFAYIKwYBBQUHAwIGCCsGAQUFBwMBMA8GA1UdEwEB/wQFMAMBAf8wDQYJKoZI
hvcNAQELBQADggEBAJtePqx5npb2Pc421kaxTH4VaoU4K+BKKd62nZKwILaQbJSt
PIVYktdL2zmr2ajTNOl4YM/EfJcwS6By0duI96zG1j6dG9txMDUFurDGpc4LVr3G
AZbgtqi9gH/R4l7VUQPtGi/CKMFID5dIOrLsx6d6u7Yuohy+5tP2YUqkHHyJXznt
I932aYBPoZmFYc7WrvdSmPSYcbjeTmCO/nBiokCLGY8bP/FftIhaZ/dwf+XBwTnU
Ki+IscwXVDc+qTZLqSJgmr9vJyB+mUYI+XnIEQ7LmlFClykR+eL+BcWXptJ04r24
gQygJvBJ0TvECJkQdYduRgXCi950aE42YtKdisA=
-----END CERTIFICATE-----

Which matches the content of my ca.crt file.

However (!), if I try to route this request through the proxy, no joy :-(

curl --insecure --cert /path/to/ca.pem https://localhost:443/something-00/healthz
<html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></center>
<hr><center>openresty/1.15.8.1</center>
</body>
</html>

curl --insecure --cert /path/to/ca.pem https://localhost:443/something-00/certs
<html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></center>
<hr><center>openresty/1.15.8.1</center>
</body>
</html>

If I look at the Pods’ logs:

NAMESPACE=test
PODS=$(\
  kubectl get pods \
  --namespace=${NAMESPACE} \
  --output=name) && \
for POD in ${PODS}
do
  echo "** ${POD}"
  kubectl logs ${POD} --namespace=${NAMESPACE}
  echo
done
** pod/something-00-7df4f998df-879tf
2019/12/12 00:45:08 [main] CA: /secrets/ca/tls.crt
2019/12/12 00:45:08 [main] Server cert: /secrets/server/tls.crt; key: /secrets/server/tls.key
2019/12/12 00:45:08 [main] Server starting [:443]
2019/12/12 00:51:03 http: TLS handshake error from 10.1.1.1:50060: tls: client didn't provide a certificate
2019/12/12 00:51:03 http: TLS handshake error from 10.1.1.1:50062: tls: client didn't provide a certificate
2019/12/12 00:51:03 http: TLS handshake error from 10.1.1.1:50064: tls: client didn't provide a certificate
2019/12/12 00:51:29 http: TLS handshake error from 10.1.1.1:50164: tls: client didn't provide a certificate
2019/12/12 00:51:29 http: TLS handshake error from 10.1.1.1:50166: tls: client didn't provide a certificate
2019/12/12 00:51:29 http: TLS handshake error from 10.1.1.1:50168: tls: client didn't provide a certificate

I tried restarting the Ingress using the command-line flag --enable-ssl-passthrough and swapping the annotations as outlined below, but this makes no difference. I achieved this by pulling the YAML for the NGINX Ingress controller, editing the YAML to include that arg and then reapplying the YAML.

The Ingress’ logs show nothing obvious:

POD=$(\                                                                                         
  kubectl get pods \
  --selector=app.kubernetes.io/name=ingress-nginx \
  --namespace=ingress-nginx \
  --output=jsonpath="{.items[0].metadata.name}")
kubectl logs ${POD} --namespace=ingress-nginx

results in:

-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:       0.26.1
  Build:         git-2de5a893a
  Repository:    https://github.com/kubernetes/ingress-nginx
  nginx version: openresty/1.15.8.2

-------------------------------------------------------------------------------

W1212 00:55:46.282364       6 flags.go:243] SSL certificate chain completion is disabled (--enable-ssl-chain-completion=false)
W1212 00:55:46.282446       6 client_config.go:541] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I1212 00:55:46.282650       6 main.go:182] Creating API client for https://10.152.183.1:443
I1212 00:55:46.288624       6 main.go:226] Running in Kubernetes cluster version v1.16 (v1.16.3) - git (clean) commit b3cbbae08ec52a7fc73d334838e18d17e8512749 - platform linux/amd64
I1212 00:55:46.473487       6 main.go:101] SSL fake certificate created /etc/ingress-controller/ssl/default-fake-certificate.pem
I1212 00:55:46.492963       6 nginx.go:263] Starting NGINX Ingress controller
I1212 00:55:46.497370       6 event.go:255] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"tcp-services", UID:"a48b9837-035a-40ba-a88d-cad5c4352b98", APIVersion:"v1", ResourceVersion:"3615062", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/tcp-services
I1212 00:55:46.497414       6 event.go:255] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"udp-services", UID:"3e53e812-0fb0-4559-bad6-898577676d3d", APIVersion:"v1", ResourceVersion:"3615063", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/udp-services
I1212 00:55:46.497468       6 event.go:255] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"nginx-configuration", UID:"280901b5-27d0-41da-9771-cf1abe3e28b4", APIVersion:"v1", ResourceVersion:"3615059", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/nginx-configuration
I1212 00:55:47.594976       6 event.go:255] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"test", Name:"something", UID:"4205269b-0be4-4841-aa10-d9c5a7f09b48", APIVersion:"networking.k8s.io/v1beta1", ResourceVersion:"4304305", FieldPath:""}): type: 'Normal' reason: 'CREATE' Ingress test/something
I1212 00:55:47.595087       6 backend_ssl.go:66] Adding Secret "test/ca" to the local store
I1212 00:55:47.595322       6 backend_ssl.go:66] Adding Secret "test/server" to the local store
I1212 00:55:47.693495       6 nginx.go:307] Starting NGINX process
I1212 00:55:47.693545       6 leaderelection.go:241] attempting to acquire leader lease  ingress-nginx/ingress-controller-leader-nginx...
I1212 00:55:47.694200       6 controller.go:134] Configuration changes detected, backend reload required.
I1212 00:55:47.694925       6 status.go:86] new leader elected: nginx-ingress-controller-568867bf56-n5q9f
I1212 00:55:47.740046       6 controller.go:150] Backend successfully reloaded.
I1212 00:55:47.740067       6 controller.go:159] Initial sync, sleeping for 1 second.
I1212 00:56:37.251319       6 status.go:86] new leader elected: nginx-ingress-controller-568867bf56-tfbll
I1212 00:56:37.251642       6 leaderelection.go:251] successfully acquired lease ingress-nginx/ingress-controller-leader-nginx
I1212 00:56:37.258543       6 status.go:274] updating Ingress test/something status from [{127.0.0.1 }] to [{10.152.183.75 }]
I1212 00:56:37.263063       6 event.go:255] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"test", Name:"something", UID:"4205269b-0be4-4841-aa10-d9c5a7f09b48", APIVersion:"networking.k8s.io/v1beta1", ResourceVersion:"4304427", FieldPath:""}): type: 'Normal' reason: 'UPDATE' Ingress test/something

Unfortunately, I’m insufficiently well-versed in NGINX configs to make much sense of the result of:

POD=$(\
  kubectl get pods \
  --selector=app.kubernetes.io/name=ingress-nginx \
  --namespace=ingress-nginx \
  --output=jsonpath="{.items[0].metadata.name}")
kubectl exec --stdin --tty ${POD} --namespace=ingress-nginx -- cat /etc/nginx/nginx.conf

To tidy-up

  • either: kubectl delete namespace/test
  • or: kubectl delete --filename=test.yaml

Spec

Kubernetes Secret, Deployment|Service and Ingress spec:

---
apiVersion: v1
kind: Namespace
metadata:
  name: test
---
apiVersion: v1
kind: Secret
metadata:
  name: server
  namespace: test
data:
  tls.crt: "[REDACTED]"
  tls.key: "[REDACTED]"
---
apiVersion: v1
kind: Secret
metadata:
  name: ca
  namespace: test
data:
  tls.crt: "[REDACTED]"
  tls.key: "[REDACTED]"
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: something
  name: something-00
  namespace: test
spec:
  replicas: 1
  selector:
    matchLabels:
      run: "00"
  template:
    metadata:
      labels:
        run: "00"
    spec:
      volumes:
        - name: ca
          secret:
            secretName: ca
        - name: server
          secret:
            secretName: server
      containers:
        - name: server
          image: dazwilkin/server@sha256:441210d5204153b4e0873881330711b86e653504cd5971c481637d223e1a82ea
          imagePullPolicy: IfNotPresent
          args:
            - --https_endpoint=:443
            - --crt_ca=/secrets/ca/tls.crt
            - --crt_server=/secrets/server/tls.crt
            - --key_server=/secrets/server/tls.key
          ports:
            - containerPort: 443
              protocol: TCP
          volumeMounts:
            - name: ca
              mountPath: /secrets/ca
            - name: server
              mountPath: /secrets/server
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: something
  name: something-00
  namespace: test
spec:
  ports:
    - port: 443
      protocol: TCP
      targetPort: 443
  selector:
    run: "00"
  type: NodePort
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  labels:
    app: something
  name: something
  namespace: test
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/rewrite-target: /$2
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
    nginx.ingress.kubernetes.io/proxy-ssl-secret: "test/ca"
    nginx.ingress.kubernetes.io/proxy-ssl-verify: "on"
    # nginx.ingress.kubernetes.io/auth-tls-pass-certificate-to-upstream: "true"
    # nginx.ingress.kubernetes.io/ssl-passthrough: "true"
spec:
  rules:
    - http:
        paths:
          - backend:
              serviceName: something-00
              servicePort: 443
            path: /something-00(/|$)(.*)
    - http:
        paths:
          - backend:
              serviceName: something-00
              servicePort: 443
            path: /something-01(/|$)(.*)
    - http:
        paths:
          - backend:
              serviceName: something-00
              servicePort: 443
            path: /something-02(/|$)(.*)
  tls:
    - secretName: server

NB crt|key PEMs bundled into Kubernetes Secrets using base64 --wrap=0 ./${FILE} and tls.crt and tls.key filenames respectively.

When trying to use SSL passthrough, I changed the paths to be:

spec:
   rules:
    - http:
      paths:
       - path: /
         backend:
           serviceName: something-00
           servicePort: 443