Access Google Services using gRPC

- 5 minutes read - 913 words

Google publishes interface definitions of Google APIs (services) that support REST and gRPC in a repo called Google APIs. Google’s SDKs uses gRPC to access these services but, how to do this using e.g. gRPCurl?

I wanted to debug Cloud Profiler and its agent makes UpdateProfile RPCs to cloudprofiler.googleapis.com. Cloud Profiler is more challenging service to debug because (a) it’s publicly “write-only”; and (b) it has complex messages. UpdateProfile sends UpdateProfileRequest messages that include Profile messages that include profile_bytes which are gzip compressed serialized protos of pprof’s Profile.

For the purposes of this post, I’m going to use Cloud Firestore rather than Cloud Profiler. There’s an example at the end in Go.

Google’s googleapis repo references a separate repo go-genproto for Go sources. However, many of the APIs now alias to Go sources in Google Cloud Client Libraries for Go. So, developer beware.

I’d like to start by showing how you can use Google’s (otherwise excellent) APIs Explorer tool to use the REST transcoded gRPC call, but, it doesn’t work correctly (and hasn’t for a long time) for Firestore’s projects.database.documents.list method. The regex for parent doesn’t permit the use of projects/{project}/databases/(default). If you have a collection in your firestore, you can GET its endpoint:

TOKEN=$(gcloud auth print-access-token)

COLLECTION="[YOUR-COLLECTION]"

curl  \
--get \
--header "Authorization: Bearer ${TOKEN}" \
--header "Accept: application/json" \
"https://firestore.googleapis.com/v1/projects/${PROJECT}/databases/(default)/documents/${COLLECTION}/"

NOTE The only permitted databaseId with Cloud Firestore is (default).

Interestingly, some Google services support limited gRPC server reflection but these appear to only reflect over Google’s generic gRPC services and not service-specific gRPC services. In the following example, we’d expect cloudprofiler.googleapis.com:443 to List the Cloud Profiler gRPC service (google.devtools.cloudprofiler.v2.ProfilerService):

ENDPOINT="cloudprofiler.googleapis.com:443"
grpcurl ${ENDPOINT} list

grpc.channelz.v1.Channelz
grpc.lb.v1.LoadReporter
grpc.reflection.v1alpha.ServerReflection

But Cloud Firestore does not:

ENDPOINT="firestore.googleapis.com:443"
grpcurl ${ENDPOINT} list

Failed to list services: server does not support the reflection API

For this reason, we’re going to need Firestore’s protos locally in order to provide these to gRPCurl to invoke the service’s methods. You can either clone the entire repo or do a sparse checkout:

mkdir googleapis
cd googleapis
git init
git remote add --fetch origin git@github.com:googleapis/googleapis.git
git config core.sparseCheckout true
echo "google" >> .git/info/sparse-checkout
git pull origin master

We can then begin to construct the gRPCurl command:

ENDPOINT="firestore.googleapis.com:443"

ROOT="/path/to/googleapis"
PACKAGE="google/firestore/v1"

grpcurl \
--import-path=${ROOT} \
--proto=${ROOT}/${PACKAGE}/firestore.proto \
...
${ENDPOINT} ...

NOTE protoc and related tools are very particular about folder paths to protobuf files. In this case, after checkout, googleapis contains a folder google that’s the root of Google’s APIs’ protocol bugger definitions. If you open firestore.proto you see that its package is google.firestore.v1. This package name is mapped to the folder structure google/firestore/v1 where the definition is expected to be found.

Next, we need to authorize the request and to do this we need an OAuth2 access token from Google. For Firestore, we can use User (e.g. your@gmail.com) credentials. We’ll use gcloud to obtain the access token and then create an Authorization header for gRPCurl:

ENDPOINT="firestore.googleapis.com:443"

ROOT="/path/to/googleapis"
PACKAGE="google/firestore/v1"

TOKEN=$(gcloud auth print-access-token)

grpcurl \
--import-path=${ROOT} \
--proto=${ROOT}/${PACKAGE}/firestore.proto \
-H "Authorization: Bearer ${TOKEN}" \
...
${ENDPOINT} ...

NOTE I don’t know why gRPCurl uses a mixture of flag styles --import-path, -H and -d (see below)

If we review firestore.proto, it contains a service called Firestore and multiple RPCs including ListDocuments which is the method we’re going to use. This takes ListDocumentsRequest message which reflects (but doesn’t exactly match) the documentation given by APIs Explorer but, is straightforward. gRPCurl requires the JSON representation of Protobuf messages. We only need provide the parent field and so we can use:

{
    "parent": "projects/{projectId}/databases/{databaseId}/documents",
}

Here’s the current gRPCurl command:

ENDPOINT="firestore.googleapis.com:443"

ROOT="/path/to/googleapis"
PACKAGE="google/firestore/v1"

TOKEN=$(gcloud auth print-access-token)

PROJECT="[YOUR-GOOGLE-PROJECT]"
PARENT="projects/${PROJECT}/databases/(default)/documents"

grpcurl \
--import-path=${ROOT} \
--proto=${ROOT}/${PACKAGE}/firestore.proto \
-H "Authorization: Bearer ${TOKEN}" \
-d "{\"parent\": \"${PARENT}\"}" \
${ENDPOINT} ...

NOTE The clumsy quote-escaped string substitutes the value of the Google Cloud PROJECT.

Lastly, we need the gRPC method. You’ll recall from above that Firestore does not support server introspection so we must rely upon firestore.proto to determine the fully-qualified method name. This is a combination of the package (google.firestore.v1), the service (Firestore) and the RPC ListDocuments. We are now able to produce the entire gRPCurl command:

ENDPOINT="firestore.googleapis.com:443"

ROOT="/path/to/googleapis"
PACKAGE="google/firestore/v1"

TOKEN=$(gcloud auth print-access-token)

PROJECT="[YOUR-GOOGLE-PROJECT]"
PARENT="projects/${PROJECT}/databases/(default)/documents"

METHOD="google.firestore.v1.Firestore/ListDocuments"

grpcurl \
--import-path=${ROOT} \
--proto=${ROOT}/${PACKAGE}/firestore.proto \
-H "Authorization: Bearer ${TOKEN}" \
-d "{\"parent\": \"${PARENT}\"}" \
${ENDPOINT} ${METHOD}

All being well, when you run this command, it will succeed and will output a subset (we didn’t page the request) of your project’s Firestore documents.

Here’s a Golang equivalent using gRPC:

package main

import (
	"context"
	"crypto/tls"
    "flag"
	"fmt"
	"log"

	"cloud.google.com/go/firestore/apiv1/firestorepb"

    "golang.org/x/oauth2/google"

    "google.golang.org/grpc"
	"google.golang.org/grpc/credentials"
	"google.golang.org/grpc/credentials/oauth"
)

var (
	endpoint = flag.String(
        "endpoint",
        "firestore.googleapis.com:443",
        "Google Cloud service endpoint",
    )
    project = flag.String(
        "project",
        "",
        "Google Cloud project",
    )
)

func main() {
    flag.Parse()

	ctx, cancel := context.WithTimeout(
        context.Background(),
        15*time.Second,
    )
	defer cancel()

	scopes := []string{
		"https://www.googleapis.com/auth/cloud-platform",
	}

	tokensource, err := google.DefaultTokenSource(ctx, scopes...)
	if err != nil {
		log.Fatal(err)
	}

	credsPerRPC := oauth.TokenSource{
		TokenSource: tokensource,
	}
	credsTransport := credentials.NewTLS(&tls.Config{
		InsecureSkipVerify: true,
	})
	opts := []grpc.DialOption{
		grpc.WithPerRPCCredentials(credsPerRPC),
		grpc.WithTransportCredentials(credsTransport),
	}

    conn, err := grpc.Dial(*endpoint, opts...)
    if err != nil {
        log.Fatal(err)
    }

    fs := firestorepb.NewFirestoreClient(conn)

    parent := fmt.Sprintf("projects/%s/databases/(default)/documents",
        *project,
    )
    rqst := &firestorepb.ListDocumentsRequest{
        Parent: parent,
    }
    resp, err := fs.ListDocuments(ctx, rqst)
    if err != nil {
        log.Fatal(err)
    }

    log.Printf("+%v", resp)
}

The code uses Application Default Credentials to obtain credentials for the gRPC call. To use your User credentials, you will need to gcloud auth application-default login.

Hopefully this explains how you may use any gRPC with any of the Google services that support it.