Google's New Golang SDK for Protobufs

- 5 minutes read - 980 words

Google has released a new Golang SDK for protobuf. In the [announcement], a useful tool to redact properties is described. If like me, this is somewhat novel to you, here’s a mashup using Google’s Protocol Buffer Basics w/ redaction.

To be very clear, as it’s an important distinction:

Version Repo Docs
v2 google.golang.org/protobuf Docs
v1 github.com/golang/protobuf Docs

Project

Here’s my project structure:

.
├── protoc-3.11.4-linux-x86_64
│   ├── bin
│   │   └── protoc
│   ├── include
│   │   └── google
│   └── readme.txt
└── src
    ├── go.mod
    ├── go.sum
    ├── main.go
    ├── protos
    │   ├── addressbook.pb.go
    │   └── addressbook.proto
    └── README.md

You may structure as you wish.

I’m using protoc 3.11.4 (link) and:

PATH=${PATH}:${PWD}/protoc-3.11.4-linux-x86_64/bin
protoc --version
libprotoc 3.11.4

I’m using Go 1.14 and Modules and so, I used the following to ‘install’ the Golang plugin:

go get google.golang.org/protobuf/cmd/protoc-gen-go

In my experience, ensuring protocol buffer package names result in appropriately generated (Golang) imports, can be challenging. This challenge is exacerbated if you wish to generate bindings in multiple languages. This approach works for me. protos as a subdirectory alongside my Golang (src) sources containing the protocol buffer files and the generated Golang.

Definition

Here’s the tweaked protobuf file that I’m using:

syntax = "proto3";

package example;

import "google/protobuf/descriptor.proto";
import "google/protobuf/timestamp.proto";

option go_package = ".;protos";

extend google.protobuf.FieldOptions {
    bool non_sensitive = 50000;
}

message Person {
    string name = 1 [(non_sensitive) = true];
    int32 id =2;
    string email = 3 [(non_sensitive) = true];

    enum PhoneType {
        MOBILE = 0;
        HOME = 1;
        WORK = 2;
    }

    message PhoneNumber {
        string number = 1;
        PhoneType type = 2;
    }

    repeated PhoneNumber phones = 4;

    google.protobuf.Timestamp last_updated = 5;
}

message AddressBook {
    repeated Person people = 1 [(non_sensitive) = true];
}

This differs from the Google tutorial file in the following ways:

  • My package name is example this should match your go mod init example value
  • There’s an additional import google/protobuf/descriptor.proto (from the protoc include directory) to bring google.protobuf.FieldOptions into scope
  • There’s an extend google.protobuf.FieldOptions that defines non_sensitive as a boolean
  • This property (non_sensitive) is used to annotate the Person message name and email properties and the AddressBook message people property

NB Custom Options was new to me.

Compilation

From within src:

protoc \
  --proto_path=./protos \
  --go_out=./protos \
  addressbook.proto

You’re expecting this to (re)generate ./src/protos/addressbook.pb.go and this should have package protos thanks to the protobuf’s:

option go_package = ".;protos";

Golang

The tutorial walks through the following steps:

  • Create Person protobuf
  • Create AddressBook protobuf adding the Person to it
  • Marshaling the AddressBook protobuf to []byte
  • Writing the []byte to a file
  • Reading the []byte from the file
  • Unmarshaling the []byte back to an AddressBook protobuf.

I’ve added a couple of intermediate steps to pretty-print the protobufs by marshaling to JSON.

More importantly, I’ve included the redact function from the Announcement and applied this to Person and to the AddressBook before these are marshaled, i.e.:

  • Create Person protobuf
  • redact Person
  • Create AddressBook protobuf adding the Person to it
  • redact AddressBook
  • marshall AddressBook to JSON and pretty-print
  • Marshaling the AddressBook protobuf to []byte
  • Writing the []byte to a file
  • Reading the []byte from the file
  • Unmarshaling the []byte back to an AddressBook protobuf.
  • marshall AddressBook to JSON and pretty-print

As:

package main

import (
	"encoding/json"
	"flag"
	"io/ioutil"
	"log"

	pb "example/protos"

	"google.golang.org/protobuf/proto"
	"google.golang.org/protobuf/reflect/protoreflect"
	"google.golang.org/protobuf/types/descriptorpb"
)

var (
	filename = flag.String("filename", "", "Filename")
)

func redact(pm proto.Message) {
	m := pm.ProtoReflect()
	m.Range(func(fd protoreflect.FieldDescriptor, v protoreflect.Value) bool {
		opts := fd.Options().(*descriptorpb.FieldOptions)
		if proto.GetExtension(opts, pb.E_NonSensitive).(bool) {
			return true
		}
		m.Clear(fd)
		return true
	})
}

func main() {
	flag.Parse()
	if *filename == "" {
		log.Fatal("Must specify filename")
	}

	{
		p = &pb.Person{
			Name:  "Frederik Jack",
			Id:    1,
			Email: "freddie@email.com",
			Phones: []*pb.Person_PhoneNumber{
				{
					Number: "425-555-1234",
					Type:   pb.Person_HOME,
				},
			},
		}

		// Redact
		redact(p)

		b := &pb.AddressBook{
			People: []*pb.Person{
				p,
			},
		}

		// Redact
		redact(b)

		// Pretty print
		j, err := json.MarshalIndent(b, "", "   ")
		if err != nil {
			log.Fatal(err)
		}
		log.Printf("%s\n", j)

		// Marshall
		out, err := proto.Marshal(b)
		if err != nil {
			log.Fatal(err)
		}

		// Write to filename
		if err := ioutil.WriteFile(*filename, out, 0644); err != nil {
			log.Fatal(err)
		}
	}
	{
		// Read from filename
		in, err := ioutil.ReadFile(*filename)
		if err != nil {
			log.Fatal(err)
		}

		// Unmarshall
		b := &pb.AddressBook{}
		if err := proto.Unmarshal(in, b); err != nil {
			log.Fatal(err)
		}

		// Pretty print
		j, err := json.MarshalIndent(b, "", "   ")
		if err != nil {
			log.Fatal(err)
		}
		log.Printf("%s\n", j)
	}
}

NB ‘conventionally’ I alias the protoc-generated packages as pb but this conflicts with the redact definition as-was.
I renamed the function parameter to pm to avoid this.

NB The Custom (Field)Option non_sensitive is compiled into a Golang variable in addressbook.pb.go:

// Extension fields to descriptor.FieldOptions.
var (
  // optional bool non_sensitive = 50000;
  E_NonSensitive = &file_addressbook_proto_extTypes[0]
)

and this is referenced in redact using the import alias as pb.E_NonSensitive

Results

Without any redaction, nothing will be filtered and the code would return:

2020/03/10 14:17:52 {
   "people": [
      {
         "name": "John Doe",
         "id": 1234,
         "email": "jdoe@email.com",
         "phones": [
            {
               "number": "555-1234",
               "type": 1
            }
         ]
      }
   ]
}
2020/03/10 14:17:52 {
   "people": [
      {
         "name": "John Doe",
         "id": 1234,
         "email": "jdoe@email.com",
         "phones": [
            {
               "number": "555-1234",
               "type": 1
            }
         ]
      }
   ]
}

With redaction applied initially to Person (p) and then to AddressBook (b), we get:

2020/03/10 14:20:23 {
   "people": [
      {
         "name": "John Doe",
         "email": "jdoe@email.com"
      }
   ]
}
2020/03/10 14:20:23 {
   "people": [
      {
         "name": "John Doe",
         "email": "jdoe@email.com"
      }
   ]
}

If the protobuf annotation were to be removed from AddressBook message people property, and redaction were still applied to AddressBook (b), we’d get:

2020/03/10 14:26:13 {}
2020/03/10 14:26:13 {}

This is because the people property is not marked non_senstive and is thus redacted.

As the Announcement post suggests, an improvement would be to recurse into the structure (e.g. AddressBook) and apply the redaction at each struct within.

HTH!

References: