Gnarly Protocol Buffers compilation

January 17, 2024 - 4 minutes read - 784 words

This Stackoverflow question piqued my interest:

retry policy configuration for grpc not working

Service Config in gRPC is new to me but, my initial suspicion (albeit incorrect) was that the JSON types were incorrect.

I decided to try using the Protocol Buffer source service_config.proto to verify the JSON.

To do so I needed to compile the source…. it was gnarly.

There are 2 repos used:

The service_config.proto includes options for java_package but no go_package.

The dependent (!) rls.proto includes options for java_package and go_package. However, the go_package is google.golang.org/grpc/lookup/grpc_lookup_v1 which doesn’t exist (link) and should probably be a package .../lookup/v1 not grpc_lookup_v1.

This became an interesting exercise in using protoc.

`protoc`

TL;DR:

P_GOOGLEAPIS="path/to/googleapis"
P_GRPCPROTOS="path/to/grpc-proto"

M_GRPCPROTOS="github.com/grpc/grpc-proto"

protoc \
--proto_path=${P_GOOGLEAPIS} \
--proto_path=${P_GRPCPROTOS} \
--go_out=${P_GRPCPROTOS} \
--go_opt=Mgrpc/service_config/service_config.proto=${M_GRPCPROTOS}/grpc/service_config \
--go_opt=Mgrpc/lookup/v1/rls.proto=${M_GRPCPROTOS}/grpc/lookup/v1 \
--go_opt=Mgrpc/lookup/v1/rls_config.proto=${M_GRPCPROTOS}/grpc/lookup/v1 \
--go_opt=module=${M_GRPCPROTOS} \
grpc/lookup/v1/rls.proto \
grpc/lookup/v1/rls_config.proto \
grpc/service_config/service_config.proto

Explanation:

`--proto_path`’s

One confusing aspect in using protoc is the need to define proto_path’s for anything more complex than protos in the working directory that don’t import protos from elsewhere.

protos in e.g. package foo.bar should be placed in a folder structure foo/bar, e.g.

.
└── foo
    └── bar
        └── bar.proto

By convention, a single proto file in package foo.bar would be called bar.proto but the filename is irrelevant as long as the file contains package foo.bar.

The root directory (.) is the proto_path and it must be represented as a proto_path value, e.g.:

.
└── path
    └── to
        └── foo
            └── bar
                └── bar.proto

In this case, to protoc bar.proto, we must have --proto_path=./path/to and we must include the value in the proto reference i.e. ./path/to/foo/bar/bar.proto. The proto_path value must exactly match the proto’s folder prefix.

Because e.g. service_config.proto is defined to be package grpc.service_config and is correctly in grpc-proto in the folder grpc/service_config, we must (a) include a proto_path to the repo e.g. --proto_path=/path/to/grpc-proto and we must reference the proto as /path/to/grpc-proto/grpc/service_config/service_config.proto. Phew!

service_config.proto has 5 imports:

import "google/protobuf/duration.proto";
import "google/protobuf/struct.proto";
import "google/protobuf/wrappers.proto";
import "google/rpc/code.proto";
import "grpc/lookup/v1/rls_config.proto";

The google.protobuf package is usually (!) accessed by protoc by default and does not need to be specified using a proto_path.

The google.rpc package is part of the [googleapis``](https://github.com/googleapis/googleapis) and so have a proto_pathreferencing the location of the cloned repo e.g./path/to/googleapis`.

`--go_out`

The grpc-proto repo suggests copying the protos but, I’m going to generate the Go stubs in the grpc-proto clone alongside the protobuf sources.

For this reason, the value of --go_out is the path to the grpc-proto clone.

`--go_opt`’s

This is quite complex unfortunately.

service_config.proto does not include options go_package. protoc needs either options go_package or M${PROTO_FILE}=${GO_IMPORT_PATH} (see Packages).

NOTE More specifically the PROTO_FILE value in the above must be the {package}{file.proto} and exclude the proto_path (because it wasn’t complex enough already).

What’s the GO_IMPORT_PATH? Confusingly rls.proto (see below) uses:

option go_package = "google.golang.org/grpc/lookup/grpc_lookup_v1";

But, as shown above, google.golang.org/grpc does not include lookup and the package name grpc_lookup_v1 is …. unconventional (why not lookup/v1?).

Instead, I decided to use github.com/grpc/grpc-proto as the module name.

Because we don’t want to change the files in the grpc-proto repo, we must use the M option:

--go_opt=Mgrpc/service_config/service_config.proto=${M_GRPCPROTOS}/grpc/service_config

service_config.proto also imports grpc/lookup/v1/rls_config.proto. We’re good for this file’s proto_path but we need to override its Go package and so we’ll use (assuming both rls_config.proto and rls.proto are needed):

--go_opt=Mgrpc/lookup/v1/rls.proto=${M_GRPCPROTOS}/grpc/lookup/v1 \
--go_opt=Mgrpc/lookup/v1/rls_config.proto=${M_GRPCPROTOS}/grpc/lookup/v1

Last, we don’t want protoc to replicate the github.com/grpc/grpc-proto folder structure within the clone grpc-proto repo. To avoid this we:

--go_opt=module=${M_GRPCPROTOS}

protos

Now we can enumerate the protobuf sources to be compiled:

grpc/lookup/v1/rls.proto
grpc/lookup/v1/rls_config.proto
grpc/service_config/service_config.proto

Golang

As I mentioned atop this post, I wanted to generate the Golang stubs for service_config.proto to confirm that the Stackoverflow poster’s JSON types were correct (they are).

I created a Golang module and it needs to import the protoc stubs generated above:

package main

import (
	"log"
	"os"
	"time"

	pb "github.com/grpc/grpc-proto/grpc/service_config"
	"google.golang.org/genproto/googleapis/rpc/code"
	"google.golang.org/protobuf/encoding/protojson"
	"google.golang.org/protobuf/types/known/durationpb"
)

func main() {
	methodConfig := &pb.MethodConfig{
		Name: []*pb.MethodConfig_Name{{
			Service: "service",
			Method:  "method",
		}},
		RetryOrHedgingPolicy: &pb.MethodConfig_RetryPolicy_{
			RetryPolicy: &pb.MethodConfig_RetryPolicy{
				MaxAttempts:       5,
				InitialBackoff:    durationpb.New(1 * time.Second),
				MaxBackoff:        durationpb.New(10 * time.Second),
				BackoffMultiplier: float32(2.0),
				RetryableStatusCodes: []code.Code{
					code.Code_UNAVAILABLE,
				},
			},
		},
	}

	b, err := protojson.Marshal(methodConfig)
	if err != nil {
		log.Fatal("unable to JSON marshal MethodConfig", err)
		os.Exit(1)
	}

	log.Print(string(b))
}

Yielding:

{
    "name":[
        {
            "service": "service",
            "method": "method",
        }
    ],
    "retryPolicy":{
        "maxAttempts":5,
        "initialBackoff":"1s",
        "maxBackoff":"10s",
        "backoffMultiplier":2,
        "retryableStatusCodes":[
            "UNAVAILABLE"
        ]
    }
}

The import is github.com/grpc/grpc-proto/grpc/service_config which doesn’t exist on github.com. We need to:

Create a go.mod file in path/to/grpc-proto so that our module can import packages (e.g. github.com/grpc/grpc-protos/grpc/service_config) from it:

go.mod:

module github.com/grpc/grpc-protos

replace the module reference in our module’s go.mod so that our module looks to the local clone for the generated Go sources:

go.mod:

module github.com/something

go 1.21.6

require (
	github.com/grpc/grpc-proto v0.0.0
	google.golang.org/genproto/googleapis/rpc v0.0.0-20240116215550-a9fa1716bcac
	google.golang.org/protobuf v1.32.0
)

replace (
	github.com/grpc/grpc-proto => /path/to/grpc-proto

That’s all!

protoc

--proto_path’s

--go_out