Multiplexing gRPC and HTTP (Prometheus) endpoints with Cloud Run

- 4 minutes read - 648 words

Google Cloud Run is useful but, each service is limited to exposing a single port. This caused me problems with a gRPC service that serves (non-gRPC) Prometheus metrics because customarily, you would serve gRPC on one port and the Prometheus metrics on another.

Fortunately, cmux provides a solution by providing a mechanism that multiplexes both services (gRPC and HTTP) on a single port!

TL;DR See the cmux Limitations and use:

grpcl := m.MatchWithWriters(
   cmux.HTTP2MatchHeaderFieldSendSettings("content-type", "application/grpc"))

Extending the example from the cmux repo:

package main

import (
	"context"
	"fmt"
	"log"
	"net"
	"net/http"

	"github.com/prometheus/client_golang/prometheus/promhttp"
	"github.com/soheilhy/cmux"

	"google.golang.org/grpc"
	"google.golang.org/grpc/examples/helloworld/helloworld"
	grpchello "google.golang.org/grpc/examples/helloworld/helloworld"
)

func main() {
	l, err := net.Listen("tcp", ":23456")
	if err != nil {
		log.Fatal(err)
	}

	m := cmux.New(l)

	// Must use `MatchWithWriters` and `HTTP2MatchHeaderFieldSendSettings`
	grpcL := m.MatchWithWriters(
		cmux.HTTP2MatchHeaderFieldSendSettings("content-type", "application/grpc"))

	httpL := m.Match(cmux.HTTP1Fast())

	grpcS := grpc.NewServer()
	grpchello.RegisterGreeterServer(grpcS, &server{})

    // Using `promhttp.Handler` here for automatic Prometheus metrics
	httpS := &http.Server{
		Handler: promhttp.Handler(),
	}

	go grpcS.Serve(grpcL)
	go httpS.Serve(httpL)

	log.Fatal(m.Serve())
}

var _ grpchello.GreeterServer = (*server)(nil)

type server struct {
	helloworld.UnimplementedGreeterServer
}

func (s *server) SayHello(ctx context.Context, in *grpchello.HelloRequest) (*grpchello.HelloReply, error) {
	return &grpchello.HelloReply{
		Message: fmt.Sprintf("Hello %s", in.Name),
	}, nil
}

Run this, grab the helloworld.proto and the gRPCurl the service’s SayHello method to test:

ENDPOINT="localhost:23456"

grpcurl \
-plaintext \
--proto=./helloworld.proto \
-d '{"name":"Freddie"}' \
${ENDPOINT} \
helloworld.Greeter.SayHello

NOTE We use -plaintext here as there’s no TLS

Should yield:

{
  "message": "Hello Freddie"
}

And, test the Prometheus metrics too:

curl \
--request GET \
${ENDPOINT}/metrics

Should yield:

# HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 0
go_gc_duration_seconds{quantile="0.25"} 0
go_gc_duration_seconds{quantile="0.5"} 0
go_gc_duration_seconds{quantile="0.75"} 0
go_gc_duration_seconds{quantile="1"} 0
go_gc_duration_seconds_sum 0
go_gc_duration_seconds_count 0
...

Assuming a Dockerfile of the form:

ARG GOLANG_VERSION="1.16.4"

FROM golang:${GOLANG_VERSION} as build

WORKDIR /${PROJECT}

COPY go.mod go.mod
COPY go.sum go.sum

RUN go mod download

COPY main.go .

RUN CGO_ENABLED=0 GOOS=linux go build \
    -a \
    -installsuffix cgo \
    -o /bin/server \
    .


FROM gcr.io/distroless/base-debian10

COPY --from=build /bin/server /

ENTRYPOINT ["/server"]

The next section assumes you’ve a Google Cloud Project ([[YOUR-PROJECT]]) where you can publish the container image and deploy the Cloud Run service. You’ll need to enable Google Container Registry (GCR) and Cloud Run:

PROJECT=[[YOUR-PROJECT]]

for SERVICE in "containerregistry" "run"
do
  gcloud services enable ${SERVICE}.googleapis.com \
  --project=${PROJECT}
done

Build the container image:

PROJECT=[[YOUR-PROJECT]]

docker build \
--tag=us.gcr.io/${PROJECT}/helloworld:v0.0.1 \
--file=./Dockerfile \
.

NOTE I’m assume you’re in the United States, if not, you’ll need to revise the GCR prefix us.gcr.io

Run the container image locally to test it:

PORT="23456"

docker run \
--interactive --tty --rm \
--publish=${PORT}:${PORT} \
us.gcr.io/${PROJECT}/helloworld:v0.0.1

NOTE You may use the same tests as before since the container expose the same port (23456).

Push the image to GCR so that Cloud Run can access it:

docker push us.gcr.io/${PROJECT}/helloworld:v0.0.1

Then deploy it to Cloud Run:

PROJECT=[[YOUR-PROJECT]]
REGION=[[YOUR-REGION]]

NAME="helloworld"
PORT="23456"

gcloud run deploy ${NAME} \
--image=us.gcr.io/${PROJECT}/helloworld:v0.0.1 \
--max-instances=1 \
--memory=256Mi \
--ingress=all \
--platform=managed \
--port=${PORT} \
--allow-unauthenticated \
--region=${REGION} \
--project=${PROJECT}

Grab the service’s endpoint:

ENDPOINT=$(\
  gcloud run services describe ${NAME} \
  --project=${PROJECT} \
  --platform=managed \
  --region=${REGION} \
  --format="value(status.address.url)") && \
ENDPOINT="${ENDPOINT#https://}:443" && \
echo ${ENDPOINT}

NOTE Cloud Run provides a TLS-endpoint proxy for our service but we need to drop the prefixing https:// and add the TLS (!) port :443 in order to interact with it over gRPC.

Then test:

grpcurl \
-insecure \
--proto=./helloworld.proto \
-d '{"name":"Freddie"}' \
${ENDPOINT} \
helloworld.Greeter.SayHello

NOTE Because the service is deployed behind a TLS-endpoint, we need to replace -plaintext with -insecure and use the Cloud Run service’s endpoint (${ENDPOINT})

{
  "message": "Hello Freddie"
}

And:

curl \
--request GET \
https://${ENDPOINT}/metrics

NOTE Because the service is deployed behind a TLS-endpoint, we need to add the https:// prefix to ${ENDPOINT} and add the path /metrics to recreate a valid URL.

Don’t forget to delete the Cloud Run service when you’re done with it:

gcloud run services delete ${NAME} \
--platform=managed \
--region=${REGION} \
--project=${PROJECT}

That’s all!