Using Delve to debug Go containers on Kubernetes

- 5 minutes read - 978 words

An interesting question on Stack overflow prompted me to understand how to use Visual Studio Code and Delve to remotely debug a Golang app running on Kubernetes (MicroK8s).

The OP is using Gin which was also new to me so the question gave me an opportunity to try out several things.

Sources

A simple healthz handler:

package main

import (
	"flag"
	"log/slog"
	"net/http"

	"github.com/gin-gonic/gin"
)

var (
	addr = flag.String("addr", "0.0.0.0:8080", "HTTP server endpoint")
)

func healthz(c *gin.Context) {
	c.String(http.StatusOK, "ok")
}
func main() {
	flag.Parse()

	router := gin.Default()
	router.GET("/fib", handler())
	router.GET("healthz", healthz)

	slog.Info("Server starting")
	slog.Info("Server error",
		"err", router.Run(*addr),
	)
}

Containerfile:

ARG GOLANG_VERSION="1.22.3"

FROM docker.io/golang:${GOLANG_VERSION} AS builder

WORKDIR /app

COPY go.mod go.mod
COPY go.sum go.sum

RUN go mod download

COPY main.go main.go

RUN go build -gcflags "all=-N -l" -a -o server .

RUN go install github.com/go-delve/delve/cmd/dlv@latest


FROM gcr.io/distroless/base-debian12

WORKDIR /

COPY --from=builder /app/server /
COPY --from=builder /go/bin/dlv /

EXPOSE 8080 40000

ENTRYPOINT ["./dlv","exec","./server","--listen=:40000","--headless=true","--api-version=2","--log","--log-output=debugger,rpc,dap,fncall"]

launch.json:

{
    "version": "0.2.0",
    "configurations": [
        {
            "name": "remote",
            "type": "go",
            "request": "attach",
            "mode": "remote",
            "remotePath": "/app",
            "port": 40000,
            "host": "0.0.0.0",
            "showLog":true,
            "trace":"verbose",
            // Intentionally excluded
            // "substitutePath": [
            //     {
            //         "from":"${workspaceFolder}",
            //         "to":"/app",
            //     },
            // ]
            
        }
    ]
}

NOTE vscode.go has extensive debugging configuration

Local binary

go build \
-gcflags "all=-N -l" \
-a \
-o server \
. && \
dlv exec ./server \
--listen=:40000 \
--headless=true \
--api-version=2 \
--log \
--log-output=debugger,rpc,dap,fncall

Should yield:

API server listening at: [::]:40000
2024-05-28T14:37:30-07:00 warning layer=rpc Listening for remote connections (connections are not authenticated nor encrypted)
2024-05-28T14:37:30-07:00 debug layer=rpc API server pid = 3232792
2024-05-28T14:37:30-07:00 info layer=debugger launching process with args: [./server]
2024-05-28T14:37:31-07:00 debug layer=debugger Adding target 3232798 "/path/to/server"

Copy the relevant subset of launch.json to your Visual Studio Code workspace and then “Run” | “Start Debugging” (F5) remote.

Once the debugger attaches, you should be able to add breakpoints to your code e.g. to the line:

c.String(http.StatusOK, "ok")

And then trying curl’ing the endpoint curl http://localhost:8080/healthz should yield:

2024-05-28T14:45:19-07:00 debug layer=debugger continuing
2024-05-28T14:45:19-07:00 debug layer=debugger ContinueOnce
[GIN] 2024/05/28 - 14:45:19 | 200 |  2.479709919s |             ::1 | GET      "/healthz"

Local container

You can then build and run a container image:

podman build \
--tag=localhost/server:v0.0.1 \
--file=${PWD}/Containerfile \
${PWD} && \
podman run \
--privileged \
--interactive --tty --rm \
--publish=8080:8080/tcp \
--publish=40000:40000/tcp \
localhost/server:v0.0.1

You should see something similar to:

API server listening at: [::]:40000
2024-05-28T21:52:28Z warning layer=rpc Listening for remote connections (connections are not authenticated nor encrypted)
2024-05-28T21:52:28Z info layer=debugger launching process with args: [./server]
2024-05-28T21:52:28Z debug layer=debugger Adding target 7 "/server"

This time, you will want to include the substitutePath section in launch.json. This is because your local code is in ${workspaceFolder} but the code that’s running in the container was built (!) in /app. The statement maps between the two so that Delve knows what local code maps to what remote code.

{
    "version": "0.2.0",
    "configurations": [
        {
            "name": "local",
            "type": "go",
            "request": "launch",
            "mode": "debug",
            "program": "${workspaceFolder}"
        },
        {
            "name": "remote",
            "type": "go",
            "request": "attach",
            "mode": "remote",
            "remotePath": "/app",
            "port": 40000,
            "host": "0.0.0.0",
            "showLog":true,
            "trace":"verbose",
            "substitutePath": [
                {
                    "from":"${workspaceFolder}",
                    "to":"/app",
                },
            ]
        }
    ]
}

“Run” | “Start Debugging” (F5) remote and, once the debugger attaches, curl the healthz endpoint and you should see similar output:

2024-05-28T22:00:37Z debug layer=debugger continuing
2024-05-28T22:00:37Z debug layer=debugger ContinueOnce
[GIN] 2024/05/28 - 22:00:37 | 200 |  3.300095413s |      10.0.2.100 | GET      "/healthz"

Kubernetes

I’m using MicroK8s but the code should work (!?) with any Kubernetes distribution. You’ll need to ensure your container is accessible to your Kubernetes infrastructure. MicroK8s includes a private image registry (localhost:32000) and I’m pushing the image to that before:

kind: List
apiVersion: v1
items:
- kind: Deployment
  apiVersion: apps/v1
  metadata:
    labels:
      app: server
    name: server
  spec:
    selector:
      matchLabels:
        app: server
    template:
      metadata:
        labels:
          app: server
      spec:
        containers:
        - image: localhost:32000/server:v0.0.1
          imagePullPolicy: IfNotPresent
          name: server
          ports:
          - name: http
            containerPort: 8080
            protocol: TCP
          - name: dlv
            containerPort: 40000
            protocol: TCP
          resources:
            limits:
              memory: 500Mi
            requests:
              cpu: 250m
              memory:  250Mi
          securityContext:
            allowPrivilegeEscalation: false
            privileged: false
            readOnlyRootFilesystem: true
            runAsGroup: 1000
            runAsNonRoot: true
            runAsUser: 1000
- kind: Service
  apiVersion: v1
  metadata:
    name: server
    labels:
      app: server
  spec:
    selector:
      app: server
    ports:
      - name: http
        protocol: TCP
        port: 8080
        targetPort: http
      - name: dlv
        protocol: TCP
        port: 40000
        targetPort: dlv
    type: NodePort

You will need to create a namespace and then deploy this to it:

NAMESPACE="..."

kubectl create namespace ${NAMESPACE}
kubectl apply --filename=${PWD}/server.yaml \
--namespace=${NAMESPACE}

You can then follow the Deployment's logs from one shell:

kubectl logs deployment/server \
--namespace=${NAMESPACE} \
--follow

To see:

2024-05-28T22:08:25Z error layer=debugger could not create config directory: mkdir .config: read-only file system
API server listening at: [::]:40000
2024-05-28T22:08:25Z warning layer=rpc Listening for remote connections (connections are not authenticated nor encrypted)
2024-05-28T22:08:25Z debug layer=rpc API server pid = 1
2024-05-28T22:08:25Z info layer=debugger launching process with args: [./server]
2024-05-28T22:08:25Z debug layer=debugger Adding target 12 "/server"

And then in another shell:

kubectl port-forward deployment/server \
---namespace=${NAMESPACE} \
8080:localhost:8080 \
40000:localhost:40000

And then (!) start debugging remote and, once it’s running, curl the endpoint:

2024-05-28T22:09:35Z debug layer=debugger continuing
2024-05-28T22:09:35Z debug layer=debugger ContinueOnce
[GIN] 2024/05/28 - 22:09:35 | 200 |  2.753903087s |       127.0.0.1 | GET      "/healthz"

JSON-RPC

Delve’s API server uses JSON-RPC over TCP (not HTTP).

See Controlling the backend.

The API is documented RPCServer

Some examples:

telnet localhost 40000

Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.

ProcessPid:

{
    "method":"RPCServer.ProcessPid",
    "params":[],
    "id":1
}

NOTE id is used to associate requests with replies and can be any integer value.

Yielding:

{
    "id":1,
    "result":{
        "Pid":12
    },
    "error":null
}

ListBreakpoints

{
    "method":"RPCServer.ListBreakpoints",
    "params":[],
    "id":1
}

{
    "id":1,
    "result":{
        "Breakpoints":[{
            "id":-1,
            "name":"unrecovered-panic",
            "addr":4465604,
            "addrs":[4465604],
            "addrpid":[12],
            "file":"/usr/local/go/src/runtime/panic.go",
            "line":1217,
            "functionName":"runtime.fatalpanic",
            "ExprString":"",
            "Cond":"",
            "HitCond":"",
            "HitCondPerG":false,
            "continue":false,
            "traceReturn":false,
            "goroutine":false,
            "stacktrace":0,
            "variables":["runtime.curg._panic.arg"],
            "LoadArgs":null,
            "LoadLocals":null,
            "WatchExpr":"",
            "WatchType":0,
            "hitCount":{},
            "totalHitCount":0,
            "disabled":false
        },{
            "id":-2,
            "name":"runtime-fatal-throw",
            "addr":4464100,
            "addrs":[4464100,4464324,4562158],
            "addrpid":[12,12,12],
            "file":"\u003cmultiple locations\u003e",
            "line":0,
            "functionName":"(multiple functions)",
            "ExprString":"",
            "Cond":"",
            "HitCond":"",
            "HitCondPerG":false,
            "continue":false,
            "traceReturn":false,
            "goroutine":false,
            "stacktrace":0,
            "LoadArgs":null,
            "LoadLocals":null,
            "WatchExpr":"",
            "WatchType":0,
            "hitCount":{},
            "totalHitCount":0,
            "disabled":false}]},
            "error":null
        }]
    },
    "error":null
}