Routing Firestore events to GKE with Eventarc

May 30, 2023 - 6 minutes read - 1219 words

Google announced Firestore … integration with Eventarc. Ackal uses Firestore to persist Customer and Check information and it uses Google Cloud Firestore Triggers to handle events on these document types.

Eventarc feels like the strategic future of eventing in Google Cloud and I’ve been concerned since adopting the technology that Google would abandon Google Cloud Firestore Triggers.

For this reason, when I saw last week’s announcement, I thought I should evaluate the mechanism and this blog post is a summary of that work.

You will need to have a Firestore Database and, I’m using a Google Kubernetes Engine (GKE) cluster as an event sink. The cluster must have Workload Identity which you can effect at creation using --workload-pool:

gcloud container clusters create ${CLUSTER_NAME} \
...
--workload-pool=${PROJECT}.svc.id.goog \
--zone=${LOCATION} \
--project=${PROJECT}

You can validate that the cluster is suitably configured with:

GOT=$(\
  gcloud container clusters describe ${CLUSTER_NAME} \
  --location=${CLUSTER_LOCATION} \
  --project=${PROJECT} \
  --format="value(workloadIdentityConfig.workloadPool)")

WANT="${PROJECT}.svc.id.goog"

if [ "${GOT}" != "${WANT}" ]
then
  printf "Error\ngot : %s\nwant: %s\n" ${GOT}  ${WANT}
  exit(1)
fi

Google has decent documentation Route Cloud Firestore events to Google Kubernetes Engine

The steps include Enabl(ing) GKE destinations. You can verify that this command has bound the roles correctly with:

PROJECT="..." # Your Project ID

NUMBER=$(\
  gcloud projects describe ${PROJECT} \
  --format="value(projectNumber)")

ACCOUNT="service-${NUMBER}"
EMAIL="${ACCOUNT}@gcp-sa-eventarc.iam.gserviceaccount.com"

gcloud projects get-iam-policy ${PROJECT} \
--flatten="bindings[].members[]" \
--filter="bindings.members~\"serviceAccount:\"${EMAIL}" \
--format="value(bindings.role.split(sep=\"/\").slice(1))"

And should see:

compute.viewer
container.developer
eventarc.serviceAgent
iam.serviceAccountAdmin

I found the gcloud eventarc trigger create command, described in Create a trigger the most challenging.

Google documents the Google Events that are available. The table lists services|Products (e.g. Cloud Firestore), the schemas (Proto|JSON) but see below and the Types.

Here’s a snapshot of the results for Cloud Firestore:

Product	Schemas	Types
Cloud Firestore	Proto / JSON	Data Type: `google.events.cloud.firestore.v1.DocumentEventData` CloudEvent Type(s): `google.cloud.firestore.document.v1.created` `google.cloud.firestore.document.v1.updated` `google.cloud.firestore.document.v1.deleted` `google.cloud.firestore.document.v1.written`

If you want to use --event-filters-path-pattern to filter on changes to e.g. subdocuments, you will need to ensure that the --event-filters="type={EVENT_TYPE} (e.g. google.cloud.firestore.document.v1.created) permits path patterns.

PROJECT="..."  # Your Project ID
LOCATION="..." # Eventarc Trigger Location

EVENT_TYPE="google.cloud.firestore.document.v1.created"

FILTER="
.eventTypes[]|
    select(
    .type==\"${EVENT_TYPE}\"
    ).filteringAttributes[]|
    select(
        .pathPatternSupported==true
    ).attribute"

PROVIDER="firestore.googleapis.com"
gcloud eventarc providers describe ${PROVIDER} \
--location=${LOCATION} \
--project=${PROJECT} \
--format=json \
| jq -r "${FILTER}"

You expect the above to return document.

Ackal uses "customers/{customerId}/domains/{domainId}/checks/{checkId}" as a path pattern.

The command should be of the form:

NAME="..."
LOCATION="..."

EVENT_TYPE="google.cloud.firestore.document.v1.created"

CLUSTER_NAME="..."
CLUSTER_LOCATION="..."
CLUSTER_NAMESPACE="..."

SERVICE_NAME="..."
SERVICE_PATH="..."

# These are Firestore default values
DATABASE="(default)"
DATABASE_NAMESPACE="(default)"

# See above
PATH_PATTERN="customers/{customerId}/domains/{domainId}/checks/{checkId}"

gcloud eventarc triggers create ${NAME} \
--location=${LOCATION} \
--destination-gke-cluster=${CLUSTER_NAME} \
--destination-gke-location=${CLUSTER_LOCATION} \
--destination-gke-namespace=${CLUSTER_NAMESPACE} \
--destination-gke-service=${SERVICE_NAME} \
--destination-gke-path=${SERVICE_PATH} \
--event-filters="type=${EVENT_TYPE}" \
--event-filters="database=${DATABASE}" \
--event-filters="namespace=${DATABASE_NAMESPACE}" \
--event-filters-path-pattern="document=${PATH_PATTERN}" \
--service-account=${EMAIL} \
--project=${PROJECT}

When I ran the above gcloud eventarc triggers create, I received an error:

ERROR: (gcloud.eventarc.triggers.create) INVALID_ARGUMENT: The request was invalid: invalid value for trigger.event_data_content_type: "" is not supported by this event type
- '@type': type.googleapis.com/google.rpc.BadRequest
  fieldViolations:
  - field: trigger.event_data_content_type
- '@type': type.googleapis.com/google.rpc.RequestInfo
  requestId: {redacted}

One trick that you can use to debug this type of error is to append --log-http to any gcloud command to observe the underlying HTTP request(s).

Another trick that you can use to debug this type of error is to consult APIs Explorer. It documents every Google service/version, every method and documents the request|response types.

For Eventarc v1 projects.locations.triggers.create, the Request body is of type Trigger.

I used both mechanisms. --log-http to see what was being sent and then looked at Trigger to see that it has a field eventDataContentType that wasn’t being set with the above command.

The documentation for gcloud eventarc triggers create states that the --event-data-content-type flag is optional but it appears that’s incorrect (see: Issue #285036733), adding the flag enables the command to succeed:

gcloud eventarc triggers create ${NAME} \
--location=${LOCATION} \
--destination-gke-cluster=${CLUSTER_NAME} \
--destination-gke-location=${CLUSTER_LOCATION} \
--destination-gke-namespace=${CLUSTER_NAMESPACE} \
--destination-gke-service=${SERVICE_NAME} \
--destination-gke-path=${SERVICE_PATH} \
--event-data-content-type="application/protobuf" \
--event-filters="type=${EVENT_TYPE}" \
--event-filters="database=${DATABASE}" \
--event-filters="namespace=${DATABASE_NAMESPACE}" \
--event-filters-path-pattern="document=${PATH_PATTERN}" \
--service-account=${EMAIL} \
--project=${PROJECT}

Curiously (per the above), I had to use the Schema application/protobuf. Initially, I tried application/json but this failed with the Go client library.

The gcloud eventarc triggers create command will (reasonably) fail if there is no service ({SERVICE_NAME}) deployed to the cluster namespace ({CLUSTER_NAMESPACE).

Here’s a trivial implementation of an HTTP server that expects DocumentEventData that is the type defined in the Cloud Events table (above).

package main

import (
	"flag"
	"fmt"
	"io/ioutil"
	"log"
	"net/http"

	"github.com/googleapis/google-cloudevents-go/cloud/firestoredata"
	"google.golang.org/protobuf/proto"
)

var (
	port = flag.Uint("port", 7777, "HTTP service listening port")
)

func handler(w http.ResponseWriter, r *http.Request) {
	body, err := ioutil.ReadAll(r.Body)
	if err != nil {
		log.Println("Unable to read request body", err)
		w.WriteHeader(http.StatusInternalServerError)
	}

	// Close the Request Body
	r.Body.Close()

	data := &firestoredata.DocumentEventData{}
	if err := proto.Unmarshal(body, data); err != nil {
		log.Println("Unable to unmarshal payload", err)
		w.WriteHeader(http.StatusInternalServerError)
	}

	log.Printf("Old: %+v\nNew: %+v\n", data.OldValue, data.Value)
}
func main() {
	flag.Parse()

	http.HandleFunc("/", handler)

	log.Printf("Starting HTTP service [0.0.0.0:%d]", *port)
	log.Fatal(http.ListenAndServe(fmt.Sprintf(":%d", *port), nil))
}

You’ll want to containerize this file and, for convenience, push it to an Artifact Registry repo in the Project:

ARG GOLANG_VERSION="1.20.4"

ARG PROJECT="eventarc-firestore-triggers"

ARG COMMIT
ARG VERSION

FROM docker.io/golang:${GOLANG_VERSION} as build

ARG PROJECT
WORKDIR /${PROJECT}

COPY go.mod go.mod
COPY go.sum go.sum

RUN go mod download

COPY main.go main.go

ARG COMMIT
ARG VERSION
RUN BUILD_TIME=$(date +%s) && \
    CGO_ENABLED=0 GOOS=linux go build \
    -a \
    -installsuffix cgo \
    -ldflags "-X 'main.BuildTime=${BUILD_TIME}' -X 'main.GitCommit=${COMMIT}' -X 'main.OSVersion=${VERSION}'" \
    -o /bin/server \
    .

FROM gcr.io/distroless/static

COPY --from=build /bin/server /

ENTRYPOINT ["/server"]
CMD ["--endpoint=:7777"]

And:

PROJECT="..."
REGION="..."
REPO="..."

NAME="eventarc-firestore-triggers"

COMMIT=$(git rev-parse HEAD)
VERSION=$(uname --kernel-release)

IMAGE="${REGION}-docker.pkg.dev/${PROJECT}/${REPO}/${NAME}:${COMMIT}"

podman build \
--tag=${IMAGE} \
--build-arg=COMMIT=${COMMIT} \
--build-arg=VERSION=${VERSION} \
--file=./Dockerfile \
${PWD}

gcloud auth print-access-token \
| podman login \
  --username=oauth2accesstoken \
  --password-stdin \
  ${REGION}-docker.pkg.dev

podman push ${IMAGE}

You’ll need a Service Account with Artifact Registry read permissions for the Kubernetes Deployment. We’ll mount the Service Account using a Kubernetes Secret. A better mechanism is to use Workload Identity for this.

ACCOUNT="..."

EMAIL="${ACCOUNT}@${PROJECT}.iam.gserviceaccount.com"
FILE="${PWD}/${ACCOUNT}.json"

gcloud iam service-accounts create ${ACCOUNT} \
--project=${PROJECT}

gcloud iam service-accounts keys create ${FILE} \
--iam-account=${EMAIL} \
--project=${PROJECT}

gcloud projects add-iam-policy-binding ${PROJECT} \
--member=serviceAccount:${EMAIL} \
--role=roles/artifactregistry.reader

We’ll create a Namespace and put the Secret in it:

NAMESPACE="test"

kubectl create namespace ${NAMESPACE}

kubectl create secret docker-registry gar \
--docker-server=${SERVER} \
--docker-username=_json_key \
--docker-password="$(cat ${FILE})" \
--docker-email=${EMAIL} \
--namespace=${NAMESPACE}

We can define a template (!) for the Kubernetes Deployment|Service:

apiVersion: v1
kind: List
metadata: {}
items:
- kind: Deployment
  apiVersion: apps/v1
  metadata:
      name: eventarc-firestore-triggers
  spec:
    selector:
      matchLabels:
        app: eventarc-firestore-triggers
    template:
      metadata:
        labels:
          app: eventarc-firestore-triggers
      spec:
        imagePullSecrets:
        - name: gar
        containers:
        - name: eventarc-firestore-triggers
          image: IMAGE
          args:
          - --port=PORT
          ports:
          - name: http
            protocol: TCP
            containerPort: PORT
- kind: Service
  apiVersion: v1
  metadata:
    name: eventarc-firestore-triggers
  spec:
    ports:
    - port: 80
      protocol: TCP
      targetPort: 7777
    selector:
      app: eventarc-firestore-triggers

Which can be instantiated (replacing IMAGE and PORT) with:

sed \
--expression="s|IMAGE|${IMAGE}|g" \
--expression="s|PORT|${PORT}|g" \
eventarc-firestore-triggers.yaml \
| kubectl apply \
  --filename=- \
  --namespace=${NAMESPACE}

All being well, you should have a Deployment and a Service:

NOTE The Service expose the Deployment on port 80 (mapping to the container’s port 7777). The Eventarc trigger appears to not support specifying a Kubernetes Service port.

kubectl get all --namespace=${NAMESPACE}

And then tail the logs:

kubectl logs deployment/eventarc-firestore-triggers \
--namespace=${NAMESPACE} \
--follow

Yielding:

2023/05/30 22:38:22 Starting HTTP service [0.0.0.0:7777]

And, after manually creating a Firestore document that matches the path pattern:

2023/05/30 22:38:22 Starting HTTP service [0.0.0.0:7777]
2023/05/30 22:51:36 [handler] Entered
2023/05/30 22:51:36
  Old: <nil>
  New:
    name: ".../JkIWWqIFyDeHRaseZxje"
    fields: {
      key: "Foo"
      value: {
        string_value: "bar"
      }
    }
    create_time: {
        seconds:1685487094
        nanos:226703000
    }
    update_time:{
        seconds:1685487094
        nanos:226703000
    }