Google Cloud Events protobufs and SDKs

I’ve written before about Ackal’s use of Firestore and subscribing to Firestore document CRUD events:

• Routing Firestore events to GKE with Eventarc
• Cloud Firestore Triggers in Golang using Firestore triggers

I find Google’s Eventarc documentation to be confusing and, in typical Google fashion, even though open-sourced, you often need to do some legwork to find relevant sources, viz:

• Google’s Protobufs for Eventarc (using cloudevents) google-cloudevents1
• Convenience (since you can generate these using protoc) language-specific types generated from the above e.g. google-cloudevents-go; google-cloudevents-python etc.

1 – IIUC EventArc is the Google service. It carries Google Events that are CloudEvents. These are defined by protocol buffers schemas.

Google documents the Event types support by Eventarc and, for the purposes of this discussion, we can grep Cloud Firestore events which include both “native” (Firestore for real) and “datastore” (Firestore in Datastore mode) events:

• google.cloud.datastore.entity.v1.created
• google.cloud.datastore.entity.v1.deleted
• google.cloud.datastore.entity.v1.updated
• google.cloud.datastore.entity.v1.written
• google.cloud.firestore.document.v1.created
• google.cloud.firestore.document.v1.deleted
• google.cloud.firestore.document.v1.updated
• google.cloud.firestore.document.v1.written

These event types are very useful in determining which protocol buffer Messages are used:

• Firestore Firestore uses protobuf package google.events.cloud.firestore.v1
• Firestore in Datastore mode uses google.events.cloud.datastore.v1

NOTE This seems rather complex but is the result of the historical evolution: what was Datastore and is now Firestore in Datastore mode, Firestore (which is an evolution of Firebase) and Firebase are all same-same-but-different services.

NOTE [Firebase] has a distinct set of event types and uses its own protobuf package google.events.firebase namespace for its services

Before we proceed, note that an event type of e.g. google.cloud.firestore.document.v1.created is represented by a protocol buffer package (!) called google.events.cloud.firestore.v1. Again, the difference is necessary but it can be confusing.

Both Firestore and Datastore protocol bufffer packages comprise: data.proto and events.proto. For each package, events.proto maps event type to Messages (!) in the package. For example:

Firestore’s DocumentCreatedEvent

message DocumentCreatedEvent {
  option (google.events.cloud_event_type) =
      "google.cloud.firestore.document.v1.created";
  ...

  // The data associated with the event
  DocumentEventData data = 1;
}

And, for Firestore, every Message corresponding to an event, carries DocumentEventData defined in data.proto:

message DocumentEventData {
  Document value = 1;
  Document old_value = 2;
  DocumentMask update_mask = 3;
}

Datastore’s EntityCreatedEvent

message EntityCreatedEvent {
  option (google.events.cloud_event_type) =
      "google.cloud.datastore.entity.v1.created";
  ...

  // The data associated with the event.
  EntityEventData data = 1;
}

And, for Datastore, every Message corresponding to an event, carries EntityEventData defined in data.proto.

message EntityEventData {
  EntityResult value = 1;
  EntityResult old_value = 2;
  PropertyMask update_mask = 3;
}

These *Event Messages represent the schema for the event data that’s delivered by Eventarc as the result of a Firestore document trigger. Your code will need to unmarshal the (often-times base64-encoded) message into the appropriate protobuf Message for processing.

At the top-level, the two services’ event Messages are similar. This is because for each CRUD event (whether a Firestore document or a Datastore entity), Eventarc delivers an event (represented by a Message) that potentially (!) includes the (current|new) value, the (prior) old_value and a mask indicating which properties changed for update events. The rule is:

• created has a (new) value and an empty old_value
• deleted has an empty value and the old_value
• updated has a (new) value and the previous old_value and a mask
• written (created|deleted|updated) infer which from value and old_value

If you’ve subscribed to created,deleted and updated events, your code will confirm that the e.g. DocumentEventData comprises value and|or old_value and mask values and process accordingly.

If you’ve subscribed to the generic written event, your code will need to determine which event is implied by inspecting whether value and old_value (or the mask) has content.

The Firestore Document Message and Datastore EntityResult Message are markedly different and so I’ll summarize the (simpler) Firestore Document Message. As you’d expected, an identifier (name) which is the fully-qualified path to the document (projects/{project}/databases/{database}/documents/{document_path}) and then a map of the document fields (name|value) and created|updated timestamps:

message Document {
  string name = 1;
  map<string, Value> fields = 2;
  google.protobuf.Timestamp create_time = 3;
  google.protobuf.Timestamp update_time = 4;
}

At this point, your code will become specific to your need, in Ackal’s usage, Firestore events result in Kubernetes Resources being mutated.

For debugging, it’s useful to be able to decode protocol buffer messages that you receive. protoc is useful for this but requires some configuration in order to be able to parse Google’s Protobufs for Eventarc. You will want to clone:

• googleapis/googleapis this repo represents protobufs for all Google’s services and includes the set of google.type including LatLng which is used by the events protos
• googleapis/google-cloudevents

You should not need to clone github.com/protocolbuffers which includes Google’s Well-Known Types (google.protobuf) since these should be accessible to protoc automatically.

Then, given e.g. a Firestore [DocumentEventData] base64-encoded (binary) message, you can:

# Paths to the Protocol Buffers root
GOOGLEAPIS_PATH="/path/to/googleapis"
CLOUDEVENTS_PATH="/path/to/google-cloudevents/proto"

PACKAGE_PATH="google/events/cloud/firestore/v1"
PACKAGE_NAME="google.events.cloud.firestore.v1"

MESSAGE_NAME="DocumentEventData"
MESSAGE_FILE="data.proto"

more messsage \
| base64 --decode --wrap=0 \
| protoc \
--decode=${PACKAGE_NAME}.${MESSAGE_NAME} \
--proto_path=${GOOGLEAPIS_PATH} \
--proto_path=${CLOUDEVENTS_PATH} \
${CLOUDEVENTS_PATH}/${PACKAGE_PATH}/${MESSAGE_FILE}

And receive something of the form:

value {
  name: "projects/{project_id}/databases/{database_id}/documents/{document_path}"
  fields {
    key: "Birthday"
    value {
      timestamp_value {
        seconds: 1530316800
      }
    }
  }
  fields {
    key: "ID"
    value {
      integer_value: 1
    }
  }
  fields {
    key: "Name"
    value {
      string_value: "Freddie"
    }
  }
  create_time {
    seconds: 1712275200
  }
  update_time {
    seconds: 1712275200
  }
}

NOTE The hex version of the above is: 0a8d010a4770726f6a656374732f7b70726f6a6563745f69647d2f6461746162617365732f7b64617461626173655f69647d2f646f63756d656e74732f7b646f63756d656e745f706174687d12080a0249441202100112120a044e616d65120a8a01074672656464696512140a08426972746864617912085206088090dbd9051a060880febcb00622060880febcb006

And, since we can generate the types ourselves using protoc, here’s the Rust version:

Cargo.toml:

[package]
name = "cloudevents-firestore"
version = "0.1.0"
edition = "2021"

[build-dependencies]
tonic-build = "0.11.0"

[dependencies]
prost = "0.12.4"
prost-types = "0.12.4"
tonic = "0.11.0"

build.rs:

use std::{env,path::PathBuf};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let out_dir=PathBuf::from(env::var("OUT_DIR")
        .expect("unable to determine `OUT_DIR`"));
    
    // `file_descriptor_set_path` appends `.pb` to the file that's created
    tonic_build::configure()
        .build_server(true)
        .file_descriptor_set_path(out_dir.join("firestore.pb"))
        .compile(
            &["/path/to/google-cloudevents/proto/google/events/cloud/firestore/v1/data.proto"],
            &[
                "/path/to/googleapis",
                "/path/to/google-cloudevents/proto"
            ],
        )?;

    Ok(())
}

And: main.rs:

pub mod google {
    pub mod events {
        pub mod cloud {
            pub mod firestore {
                pub mod v1 {
                    tonic::include_proto!("google.events.cloud.firestore.v1");
                }
            }
        }
    }
    pub mod r#type {
        tonic::include_proto!("google.r#type");
    }
}

use google::events::cloud::firestore::v1::{
    value::ValueType::{IntegerValue, StringValue, TimestampValue},
    Document, DocumentEventData, Value,
};
use prost::Message;
use prost_types::Timestamp;
use std::collections::HashMap;

const NAME: &str = "projects/{project_id}/databases/{database_id}/documents/{document_path}";

fn main() {
    let birthday = Timestamp::date(2018, 6, 30).expect("unable to parse birthday");
    let create_time = Some(Timestamp {
        seconds: 1712275200,
        nanos: 0,
    });
    let update_time = create_time.clone();

    let message = DocumentEventData {
        value: Some(Document {
            name: NAME.to_string(),
            fields: HashMap::from([
                (
                    "ID".to_string(),
                    Value {
                        value_type: Some(IntegerValue(1)),
                    },
                ),
                (
                    "Name".to_string(),
                    Value {
                        value_type: Some(
                          StringValue("Freddie".to_string()),
                        ),
                    },
                ),
                (
                    "Birthday".to_string(),
                    Value {
                        value_type: Some(
                          TimestampValue(birthday),
                        ),
                    },
                ),
            ]),
            create_time: create_time,
            update_time: update_time,
        }),
        old_value: None,
        update_mask: None,
    };

    let data = message.encode_to_vec();
    println!("{:?}", data);

    let mut hex_string = String::new();
    for byte in &data {
        hex_string.push_str(&format!("{:02X}", byte));
    }
    println!("{}", hex_string);
}

Running the above example should generate the same output as this Golang equivalent:

package main

import (
	"fmt"
	"log/slog"
	"os"
	"time"

	"github.com/googleapis/google-cloudevents-go/cloud/firestoredata"
	"google.golang.org/protobuf/proto"
	"google.golang.org/protobuf/types/known/timestamppb"
)

const (
	name string = "projects/{project_id}/databases/{database_id}/documents/{document_path}"
)

func main() {
	created := timestamppb.New(
		time.Date(2024, time.April, 5, 0, 0, 0, 0, time.UTC),
	)
	data := &firestoredata.DocumentEventData{
		Value: &firestoredata.Document{
			Name: name,
			Fields: map[string]*firestoredata.Value{
				"ID": {
					ValueType: &firestoredata.Value_IntegerValue{
						IntegerValue: 1,
					},
				},
				"Name": {
					ValueType: &firestoredata.Value_StringValue{
						StringValue: "Freddie",
					},
				},
				"Birthday": {
					ValueType: &firestoredata.Value_TimestampValue{
						TimestampValue: timestamppb.New(
							time.Date(2018, time.June, 30, 0, 0, 0, 0, time.UTC),
						),
					},
				},
			},
			CreateTime: created,
			UpdateTime: created,
		},
		OldValue:   nil,
		UpdateMask: nil,
	}
	slog.Info("Data", "data", data)

	b, err := proto.Marshal(data)
	if err != nil {
		slog.Error("unable to marshal data", "err", err)
		os.Exit(1)
	}

	slog.Info("Message", "b", fmt.Sprintf("%x", b))
}

• Firestore
• Eventarc
• Protobufs
• Rust
• Golang
• Tonic
• Prost