Golang Protobuf APIv2

Google has a new Golang Protobuf API, APIv2 (google.golang.org/protobuf) superseding APIv1 (github.com/golang/protobuf). If your code is importing github.com/golang/protobuf, you’re using APIv2. Otherwise, you should consult with the docs because Google reimplemented APIv1 atop APIv2. One challenge this caused me, as someone who does not use protobufs and gRPC on a daily basis, is that gRPC (code-generation) is being removed from the (Golang) protoc-gen-go, the Golang plugin that generates gRPC service bindings.

NOTE If your protoc-generated Golang sources include methods implementing the interface protoreflect.ProtoMessage, you’re using APIv2

I’ve been playing around with the new API.

syntax = "proto3";

package examples;

option go_package = "[[PROJECT]]/protos";

service ComplexService {
    rpc add(ComplexRequest) returns (ComplexResponse);
    rpc sub(ComplexRequest) returns (ComplexResponse);
    rpc mul(ComplexRequest) returns (ComplexResponse);
    rpc div(ComplexRequest) returns (ComplexResponse);
}

message ComplexRequest {
    Complex a = 1;
    Complex b = 2;
}

message ComplexResponse {
    Complex result = 1;
    string error = 2;
}

message Complex {
    float real = 1;
    float imag = 2;
}

NOTE Replace [[PROJECT]] with your preferred module name. You should replace go mod init [[PROJECT]] with this module name too.

I’m using Protocol Buffers v3.12.3 and:

protoc --version
libprotoc 3.12.3

Now, we’ll use the new APIs protoc-gen-go:

protoc \
--proto_path=./protos \
--descriptor_set_out=./protos/descriptor.pb \
--go_out=plugins=grpc,module=[[PROJECT]]:. \
protos/complex.proto

NOTE Replace [[PROJECT]] with your module name.

The command generates (a) Golang bindings for complex.proto; and (b) a descriptor file.

Because we defined go_package in the protobuf source, this overrides the package name and, in conjunction wit the module=[[PROJECT]] command, ensures that the Golang sources are placed in [[PROJECT]]/protos. The go_package path must match the go.mod module name and it must contain (in this case) a proto directory.

NOTE For the purposes of what follows, we could omit the --go_out=.... flag entirely

We’re going to explore dynamically constructing protobuf messages using dynamicpb and the descriptor generated by protoc. First, let’s open the descriptor file…

NOTE Apologies if the following slab of code offends your sensibilities, it was easier to present this as a single function

package main

import (
    "flag"
    "io/ioutil"

    "google.golang.org/protobuf/proto"
    "google.golang.org/protobuf/reflect/protodesc"
    "google.golang.org/protobuf/types/descriptorpb"
)

var (
    descriptorFile = flag.String("descriptor_file", "", "Descriptor filename")
)

func main() {

    // Parse the flags
    flag.Parse()

    // Create a(n empty) FileDescriptorSet
    fds := &descriptorpb.FileDescriptorSet{}

    {
        // Open the descriptor
        b, _ := ioutil.ReadFile(*descriptorFile)

        // Unmarshal descriptor file into the FileDescriptorSet
        proto.Unmarshal(b, fds)
    }

    // Create a Files registry
    files, _ := protodesc.NewFiles(fds)
    ...
}

NOTE I’m omitting error-handling for brevity|clarity

We can then either enumerate the registry’s members or, since we know what we’re looking for, we can query descriptors by name:

// Query (Descriptors) by name
d, _ := files.FindDescriptorByName(protoreflect.FullName("examples.ComplexService"))

Protobufs contain various types. In this example, we have service, methods and messages.

Let’s assert the Descriptor returned by the above into a ServiceDescriptor:

s, ok := d.(protoreflect.ServiceDescriptor)
if !ok {
    log.Fatal("Unable to assert into ServiceDescriptor")
}

Now, let’s choose one of the methods defined by the Service (e.g. add):

m := s.Methods().ByName(protoreflect.Name("add"))

The MethodDescriptor that’s returned, primarily reflects the binding of a method name (add) with a request (or input) message (ComplexRequest) and a response (or output) message (ComplexResponse).

i := m.Input()

NOTE We’re moving from MethodDescriptor to MessageDescriptor

Using dynamicpb, we can now manifest this MessageDescriptor (i) into a dyanmicpb.Message:

rqst := dynamicpb.NewMessage(i)

Now, we can construct a ComplexRequest, the easy way:

rqst := &pb.ComplexRequest{
    A: &pb.Complex{
        Real: 39,
        Imag: 3,
    },
    B: &pb.Complex{
    Real: 39,
        Imag: 3,
    },
}

Or dynamically:

{
    fd := i.Fields().ByName(protoreflect.Name("a"))
    if fd.Kind() != protoreflect.MessageKind {
        log.Fatal("Expect a MessageKind (of type Complex)")
    }
    a := fd.Message()
    m := dynamicpb.NewMessage(a)
    {
        fd := a.Fields().ByName(protoreflect.Name("real"))
        m.Set(fd, protoreflect.ValueOfFloat32(39))
    }
    {
        fd := a.Fields().ByName(protoreflect.Name("imag"))
        m.Set(fd, protoreflect.ValueOfFloat32(3))
    }

    x := m.ProtoReflect()
    v := protoreflect.ValueOfMessage(x)
    rqst.Set(fd, v)
}

NOTE I’ll leave the reader to implement the construction of the second complex number in the request, b (&pb.Complex{})

In the dynamic case, we’ve created a dynamicpb.Message which implements ProtoMessage but we’d prefer to have a pb.ComplexRequest in order to call an implementation of the service.

Assuming:

r := &pb.ComplexRequest{}

There should be 2 (equivalent) ways to do this:

NOTE This approach does not work; Merge panics (see: https://github.com/golang/protobuf/issues/1163)

proto.Merge(r, rqst)

Or:

NOTE This approach works

{
    b, _ := proto.Marshal(rqst)
    proto.Unmarshal(b, r)
}

Now, with a pb.ComplexRequest struct, we can ship this to an implementation of the gRPC service to get a result.

Something similar to:

type Server struct{}

func (s *Server) Mul(ctx context.Context, r *pb.ComplexRequest) (*pb.ComplexResponse, error) {
	a := toComplex(r.GetA())
	b := toComplex(r.GetB())
	x := a * b
	return &pb.ComplexResponse{
		Result: &pb.Complex{
			Real: real(x),
			Imag: imag(x),
		},
	}, nil
}

func toComplex(p *pb.Complex) complex64 {
	return complex(p.GetReal(), p.GetImag())
}

and, of course:

x := Server{}
log.Println(x.Mul(context.Background(), r))

yields:

result:{real:1512  imag:234}

That’s all!

• Golang
• Protobufs