Prost! Tonic w/ a dash of JSON

- 8 minutes read - 1519 words

I naively (!) began exploring JSON marshaling of Protobufs in rust. Other protobuf language SDKs include JSON marshaling making the process straightforward. I was to learn that, in rust, it’s not so simple. Unfortunately, for me, this continues to discourage my further use of rust (rust is just hard).

My goal was to marshal an arbitrary protocol buffer message that included a oneof feature. I was unable to JSON marshal the rust generated by tonic for such a message.

What follows is mostly a copy of my notes trying to repro the core issue (i.e. that oneof features aren’t easily JSON marshallable) in the hopes that it helps others navigate this and perhaps helps me find a solution.

Foo

I wanted to start with a simple protobuf message (and add features until I encountered what turned out to be the issue when using the oneof feature):

foo.proto:

syntax = "proto3";

message Foo {
  int64 id = 1;
  string name = 2;
}

NOTE This doesn’t include a package {name}; statement essentially dumping the message into the global namespace. I discourage this approach but I’m using it by way of example.

I’ve used tonic and tonic-build with success with gRPC and, for simplicity, will use it in these examples even though I’m not using gRPC. Tonic uses prost (PROST!) for the protocol buffers implementation.

Cargo.toml:

[package]
name = "rust"
version = "0.1.0"
edition = "2021"

[[bin]]
name = "foo"
path = "src/foo.rs"

[dependencies]
prost = "0.12.3"
serde = "1.0.197"
serde_derive = "1.0.197"
serde_json = "1.0.114"
tonic = "0.11.0"

[build-dependencies]
serde_derive = "1.0.197"
tonic-build = "0.11.0"

NOTE Because tonic-build works like magic, it’s easy to forget that it depends on protoc. Ensure protoc is in your PATH before you try to cargo build or cargo run.

build.rs:

fn main() -> Result<(), Box<dyn std::error::Error>> {
    tonic_build::configure()
        .type_attribute(
            "Foo",
            "#[derive(
                serde_derive::Deserialize,
                serde_derive::Serialize,
            )]",
        )
        .compile(&[
            "foo.proto",
            ], &[
                "/path/to/protos",
            ])?;

    Ok(())
}

Because the protobuf did not include a package name, tonic generates a rust file called _.rs in the folder represented by the environment variable OUT_DIR. You can:

build.rs:

use std::{env,path::PathBuf};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let out_dir=PathBuf::from(env::var("OUT_DIR")
        .expect("unable to determine `OUT_DIR`"));
    ...
    Ok(())
}

But, in this case, its path will be:

target/debug/build/{package-name}-[0-9a-z]{16}/out/_.rs

build.rs will generate (I’ve deleted some of the annotations):

#[derive(serde_derive::Deserialize, serde_derive::Serialize)]
pub struct Foo {
    pub id: i64,
    pub name: ::prost::alloc::string::String,
}

NOTE The struct is annotated with #[derive(serde_derive::_)]. This is because of the addition of type_attribute method to the Builder. I think these aren’t validated because you can use arbitrary path values without error so, take care!

Lastly, we can use Foo struct and marshal it to JSON with:

foo.rs:

pub mod proto {
    // Because the protobuf omits `package`
    // the tonic-generated rust source is called `_.rs`
    tonic::include_proto!("_");
}

use serde_json::{json, Value};

fn main() {
    let foo = proto::Foo {
        id: 1000,
        name: "Freddie".to_owned(),
    };

    let json_value: Value = json!(foo);
    let json_string = json_value.to_string();
    println!("{}", json_string);
}

Yielding:

{
  "id": 1000,
  "name": "Freddie"
}

Bar

This was a retroactive repro. I was able to recreate the issue with Baz. I noticed that tonic uses a nested enum to represent the oneof feature and so wanted to try a simple form. This example uses a non-nested protobuf enum:

bar.proto:

syntax = "proto3";

enum PhoneType {
  MOBILE = 0;
  HOME = 1;
  WORK = 2;
}
message Bar {
  int64 id = 1;
  PhoneType type = 2;
}

In the build.rs, I retroactive (after seeing the generated code), add type_attribute for PhoneType. As expected, when this is included, the generated enum PhoneType includes the annotation. When it’s not included, the annotation is not present. However, it makes no difference whether it’s included or not because the code using Bar will marshal to JSON correctly regardless.

build.rs:

fn main() -> Result<(), Box<dyn std::error::Error>> {
    tonic_build::configure()
        .type_attribute(
            "Bar",
            "#[derive(
                serde_derive::Deserialize,
                serde_derive::Serialize,
            )]",
        )
        // .type_attribute(
        //     "PhoneType",
        //     "#[derive(
        //         serde_derive::Deserialize,
        //         serde_derive::Serialize,
        //     )]",
        // )
        .compile(&[
            "bar.proto",
            ], &[
                "../protos",
            ])?;

    Ok(())
}

Yields:

#[derive(serde_derive::Deserialize, serde_derive::Serialize)]
pub struct Bar {
    pub id: i64,
    pub r#type: i32,
}
#[derive(serde_derive::Deserialize, serde_derive::Serialize)]
pub enum PhoneType {
    Mobile = 0,
    Home = 1,
    Work = 2,
}
impl PhoneType {
    pub fn as_str_name(&self) -> &'static str {}
    pub fn from_str_name(value: &str) -> ::core::option::Option<Self> {}
}

We can use Bar: bar.rs:

pub mod proto {
    // Because the protobuf source omits `package`
    // The tonic-generated rust source is called `_.rs`
    tonic::include_proto!("_");
}

use serde_json::{json, Value};

fn main() {
    let bar = proto::Bar {
        id: 1000,
        r#type: proto::PhoneType::Mobile as i32,
    };

    let json_value:Value = json!(bar);
    let json_string = json_value.to_string();
    println!("{}", json_string);
}

NOTE

  1. tonic detects that the protocol buffer field type conflicts with a rust reserved word and prefixes the generated field with r# (r#type) to avoid problems.
  2. even though PhoneType is generated as an enum, we must cast it to i32 (as i32) to assign it to the field.

Yields:

{
  "id": 1000,
  "type": 0
}

NOTE The type value of zero corresponds to Mobile.

Baz

So here then is the problematic example:

baz.proto:

syntax = "proto3";

message Baz {
  oneof x {
    int64 id = 1;
    string name = 2;
  }
}

build.rs:

fn main() -> Result<(), Box<dyn std::error::Error>> {
    tonic_build::configure()
        .type_attribute(
            "Baz",
            "#[derive(
                serde_derive::Deserialize,
                serde_derive::Serialize,
            )]",
        )
        .compile(&[
            "baz.proto",
            ], &[
                "../protos",
            ])?;

    Ok(())
}

This generates:

#[derive(serde_derive::Deserialize, serde_derive::Serialize)]
pub struct Baz {
    pub x: ::core::option::Option<baz::X>,
}
/// Nested message and enum types in `Baz`.
pub mod baz {
    pub enum X {
        Id(i64),
        Name(::prost::alloc::string::String),
    }
}

NOTE I’m unable to get the enum X to include the serde_derive annotations.

And baz.rs:

pub mod proto {
    // Because the protobuf source omits `package`
    // The tonic-generated rust source is called `_.rs`
    tonic::include_proto!("_");
}

use serde_json::{json, Value};

fn main() {
    let baz = proto::Baz{
        x: Some(proto::baz::X::Id(1000)),
    };

    let json_value:Value = json!(baz);
    let json_string = json_value.to_string();
    println!("{}", json_string);
}

Compilation results in errors:

--> /path/to/target/debug/build/{package-name}-[0-9a-z]{16}/out/_.rs:6:12
     |
6    |     pub x: ::core::option::Option<baz::X>,
     |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ the trait `Deserialize<'_>` is not implemented for `X`, which is required by `std::option::Option<X>: Deserialize<'_>`
     |
     = help: the following other types implement trait `Deserialize<'de>`:
               bool
               char
               isize
               i8
               i16
               i32
               i64
               i128
             and 132 others
     = note: required for `std::option::Option<X>` to implement `Deserialize<'_>`
note: required by a bound in `next_element`

However (!), if I take the _.rs generated by build.rs and edit it to include serde_derive annotations on pub enum X, the code works:

use serde_json::{json, Value};

#[derive(serde_derive::Deserialize, serde_derive::Serialize)]
#[allow(clippy::derive_partial_eq_without_eq)]
#[derive(Clone, PartialEq, ::prost::Message)]
pub struct Baz {
    #[prost(oneof = "baz::X", tags = "1, 2")]
    pub x: ::core::option::Option<baz::X>,
}
/// Nested message and enum types in `Baz`.
pub mod baz {
    #[derive(serde_derive::Deserialize, serde_derive::Serialize)]
    #[allow(clippy::derive_partial_eq_without_eq)]
    #[derive(Clone, PartialEq, ::prost::Oneof)]
    pub enum X {
        #[prost(int64, tag = "1")]
        Id(i64),
        #[prost(string, tag = "2")]
        Name(::prost::alloc::string::String),
    }
}

fn main() {
    let baz = Baz{
        x: Some(baz::X::Id(1000)),
    };

    let json_value:Value = json!(baz);
    let json_string = json_value.to_string();
    println!("{}", json_string);
}

Yielding:

{
    "x": {
        "Id": 1000
    }
}

So, my issue is that I’m unable to configure tonic_build’s Builder to annotate the build with serde_derive on the nested enum. And there I’ve languished :-(

Quz

My last attempt was to see whether embedding the enum in a message makes a difference.

It does not.

quz.proto:

syntax = "proto3";

message Quz {
  enum PhoneType {
    MOBILE = 0;
    HOME = 1;
    WORK = 2;
  }

  int64 id = 1;
  PhoneType type = 2;
}

build.rs:

fn main() -> Result<(), Box<dyn std::error::Error>> {
    tonic_build::configure()
        .type_attribute(
            "Quz",
            "#[derive(
                serde_derive::Deserialize,
                serde_derive::Serialize,
            )]",
        )
        .compile(&[
            "quz.proto",
            ], &[
                "../protos",
            ])?;

    Ok(())
}

This generates:

#[derive(serde_derive::Deserialize, serde_derive::Serialize)]
pub struct Quz {
    pub id: i64,
    pub r#type: i32,
}
/// Nested message and enum types in `Quz`.
pub mod quz {
    #[derive(
        Clone,
        Copy,
        Debug,
        PartialEq,
        Eq,
        Hash,
        PartialOrd,
        Ord,
        ::prost::Enumeration
    )]
    #[repr(i32)]
    pub enum PhoneType {
        Mobile = 0,
        Home = 1,
        Work = 2,
    }
    impl PhoneType {
        pub fn as_str_name(&self) -> &'static str {}
        pub fn from_str_name(value: &str) -> ::core::option::Option<Self> {}
    }
}

We can use Quz (just like Bar): quz.rs:

pub mod proto {
    // Proto omits `package` and so the generated rust is called `_.rs`
    tonic::include_proto!("_");
}

use serde_json::{json, Value};

fn main() {
    let quz = proto::Quz{
        id: 1000,
        r#type: proto::quz::PhoneType::Mobile as i32,
    };

    let json_value:Value = json!(quz);
    let json_string = json_value.to_string();
    println!("{}", json_string);
}

Yielding:

{
  "id": 1000,
  "type": 0
}

Conclusion

Whereas with Go and Python, JSON marshaling (a common use-case with protobufs) is trivial, with rust it is more involved. It’s superficially logical (tonic and prost generate code that logically represents the protocol buffer messages as Rust types) and it’s good that serde albeit serde-derive and serde_json (NOTE the underscore) would be used in tandem. But, when it doesn’t work seamlessly, my lack of rust skillz makes life harder for me.

Go

package main

import (
	"log/slog"

	pb "path/to/protos"

	"google.golang.org/protobuf/encoding/protojson"
)

func main() {
	baz := &pb.Baz{
		X: &pb.Baz_Id{
			Id: 1000,
		},
	}
	b, err := protojson.Marshal(baz)
	if err != nil {
		slog.Error("error", "err", err)
	}

	slog.Info("JSON", "b", string(b))
}

Python

from google.protobuf.json_format import MessageToJson

import baz_pb2

baz = baz_pb2.Baz()
baz.id = 1000

j = MessageToJson(baz)
print(j)