Google Trillian on Cloud Run

- 6 minutes read - 1101 words

I’ve written previously (Google Trillian for Noobs) about Google’s interesting project Trillian and about some of the “personalities” (e.g. PyPi Transparency) that I’ve build using it.

Having gone slight cra-cra on Cloud Run and gRPC this week with Golang gRPC Cloud Run and gRPC, Cloud Run & Endpoints, I thought it’d be fun to deploy Trillian and a personality to Cloud Run.

It mostly (!) works :-)

At the end of the post, I’ve summarized creating a Cloud SQL instance to host the Trillian data(base).

You’ll need a GCP project, billing established (!?) and Cloud Run enabled

Project

BILLING=[[YOUR-BILLING]]
PROJECT=[[YOUR-PROJECT]]
REGION=[[YOUR-REGION]] # Perhaps "us-west1"

gcloud projects create ${PROJECT}
gcloud beta billing projects link ${PROJECT} \
--billing-account=${BILLING}

# Enable Cloud Run
gcloud services enable run.googleapis.com \
--project=${PROJECT}

Cloud Run

OK, using Trillian’s public container images, let’s deploy a Log Server and a Log Signer configured to use our Cloud SQL instance:

# Endpoint for Cloud Run services must be port 8080
ENDPOINT="0.0.0.0:8080"

# Cloud SQL Instance details
INST=[YOUR-INST] # Perhaps "db"
CONN=$(\
  gcloud sql instances describe ${INST} \
  --project=${PROJECT} \
  --format="value(connectionName)")

# Details for the database "test" on the instance
USER="test"
PASS="zaphod"

# Log Server
gcloud run deploy trillian-log-server \
--image=gcr.io/trillian-opensource-ci/log_server@sha256:a027dd696a20c8741a88264d84ee08ff19343f84f297fb3afbf6b23ef799851f \
--args=\
"--mysql_uri=${USER}:${PASS}@unix(/cloudsql/${CONN})/test",\
"--rpc_endpoint=${ENDPOINT}",\
"alsologtostderr" \
--add-cloudsql-instances=${CONN} \
--set-env-vars=INSTANCE_CONNECTION_NAME=${CONN} \
--platform=managed \
--allow-unauthenticated \
--region=${REGION} \
--project=${PROJECT}

# Log Signer
gcloud run deploy trillian-log-signer \
--image=gcr.io/trillian-opensource-ci/log_signer@sha256:7d8ff9fba9863adca440c4ddc51080162d3d246c3b71d3929ce3b57bd6f979cf \
--args=\
"--mysql_uri=${USER}:${PASS}@unix(/cloudsql/${CONN})/test",\
"--rpc_endpoint=${ENDPOINT}",\
"--batch_size=1000",\
"--sequencer_guard_window=0",\
"--sequencer_interval=200ms",\
"--force_master",\
"--logtostderr" \
--add-cloudsql-instances=${CONN} \
--set-env-vars=INSTANCE_CONNECTION_NAME=${CONN} \
--platform=managed \
--allow-unauthenticated \
--region=${REGION} \
--project=${PROJECT}

NB It is possible to pass flags (--flag=value) to Cloud Run but it’s not clearly documented

NB We must triple reference the database (a) --msql_url=; (b) --add-cloudsql-instance; (c) --set-env-vars=INSTANCE_CONNECTION

Cloud Run injects a proxy (sidecar) into deployments and (among other things?) provides a TLS endpoint. In order to connect to the Trillian Log Server, we must (a) get the service’s endpoint; (b) grab the server’s TLS certificate; (c) create a Trillian Log using the Log Server (on the Cloud SQL database):

# Grab the Log Server's (Cloud Run) address
TRILLIAN_LOG_SERVER=$(\
  gcloud run services describe trillian-log-server \
  --platform=managed \
  --region=${REGION} \
  --project=${PROJECT} \
  --format="value(status.address.url)")
# Ditch the prefixing "https://"
TRILLIAN_LOG_SERVER=${TRILLIAN_LOG_SERVER#https://} && echo ${TRILLIAN_LOG_SERVER}

# Determine the Log Server's certificate
# Cloud Run automatically proxies (sidecar) deployed container's endpoint
openssl s_client -showcerts -servername ${TRILLIAN_LOG_SERVER} -connect ${TRILLIAN_LOG_SERVER}:443 2>/dev/null </dev/null \
| sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > ./${TRILLIAN_LOG_SERVER}.pem

# Create a Log ID
# Requires the Log Server's cert because Cloud Run proxies traffic through TLS
LOGID=$(\
  go run github.com/google/trillian/cmd/createtree \
  --tls_cert_file=./${TRILLIAN_LOG_SERVER}.pem \
  --admin_server=${TRILLIAN_LOG_SERVER}:443) && echo ${LOGID}

You’ll need a Trillian personality for the next step.

I’m using one that I created earlier but I recommend you deploy your own and use my personality as a guide:

# Determine personality current commit
TAG=$(git rev-parse HEAD)

# NB Cloud Run sidecars the container with a(n Envoy proxied?) TLS endpoint on port 443
gcloud run deploy pypi-transparency-server \
--image=gcr.io/${PROJECT}/server:${TAG} \
--args=\
"--grpc_endpoint=${ENDPOINT}",\
"--tlog_endpoint=${TRILLIAN_LOG_SERVER}:443",\
"--tlog_id=${LOGID}",\
"--package_cache=/tmp" \
--platform=managed \
--allow-unauthenticated \
--region=${REGION} \
--project=${PROJECT}

NB The personality connects to the Trillian Log Server on the Cloud Run service’s host (${TRILLIAN_LOG_SERVER}) and port 443

NB The personality also uses a log created in the previous step (${LOGID})

# Grab the PyPi Transparency Server's (Cloud Run) address
PYPI_SERVER=$(\
  gcloud run services describe pypi-transparency-server \
  --platform=managed \
  --region=${REGION} \
  --project=${PROJECT} \
  --format="value(status.address.url)")
# Ditch the prefixing "https://"
PYPI_SERVER=${PYPI_SERVER#https://} && echo ${PYPI_SERVER}

This is where my personality deployment (I think the Trillian services work correctly) falls short. I should be able to now gprcurl the endpoint but this fails:

grpcurl \
-d '{ "package": {"name": "protobuf", "version": "3.11.3", "filename": "protobuf-3.11.3-cp38-cp38-manylinux1_x86_64.whl", "url": "https://files.pythonhosted.org/packages/9a/71/5cdb5ed762a537eac39097ae6ecf8785e276b5044efe99b8e53cb3addd7f/protobuf-3.11.3-cp38-cp38-manylinux1_x86_64.whl" }}' \
-proto protos/pypi_transparency.proto \
-proto protos/package.proto \
${PYPI_SERVER}:443 pypitransparency.v1.PyPiTransparency.AddPackage
ERROR:
  Code: Unavailable
  Message: connection closed

Example: PyPi: protobuf 3.11.3

{
	"Package": {
		"name": "protobuf",
		"version": "3.11.3",
		"filename": "protobuf-3.11.3-cp38-cp38-manylinux1_x86_64.whl",
		"url": "https://files.pythonhosted.org/packages/9a/71/5cdb5ed762a537eac39097ae6ecf8785e276b5044efe99b8e53cb3addd7f/protobuf-3.11.3-cp38-cp38-manylinux1_x86_64.whl"
	}
}

I’ll update this post when I understand my mistake :-)

Aha! My mistake was that the pypi-transparency-server was failing when making an insecure connection to the trillian-log-server (because this is how I role during development). I revised the pypi-transparency-server so that it uses the system certificate pool to make the connection and now the grpcurl works as expected:

systemCertPool, err := x509.SystemCertPool()
if err != nil {
	log.Fatal("failed to load system root CA cert pool")
}
  
creds := credentials.NewTLS(&tls.Config{
	RootCAs: systemCertPool,
})

dialOpts := []grpc.DialOption{
	grpc.WithTransportCredentials(creds),
}

conn, err := grpc.Dial(*tLogEndpoint, dialOpts...)
if err != nil {
	log.Fatal(err)
}
defer conn.Close()

and:

# AddPackage
grpcurl \
-d '{ "package": {"name": "protobuf", "version": "3.11.3", "filename": "protobuf-3.11.3-cp38-cp38-manylinux1_x86_64.whl", "url": "https://files.pythonhosted.org/packages/9a/71/5cdb5ed762a537eac39097ae6ecf8785e276b5044efe99b8e53cb3addd7f/protobuf-3.11.3-cp38-cp38-manylinux1_x86_64.whl" }}' \
-proto protos/pypi_transparency.proto \
-proto protos/package.proto \
${PYPI_SERVER}:443 pypitransparency.v1.PyPiTransparency.AddPackage
{
  "ok": true
}

# GetPackage
grpcurl \
-d '{ "package": {"name": "protobuf", "version": "3.11.3", "filename": "protobuf-3.11.3-cp38-cp38-manylinux1_x86_64.whl", "url": "https://files.pythonhosted.org/packages/9a/71/5cdb5ed762a537eac39097ae6ecf8785e276b5044efe99b8e53cb3addd7f/protobuf-3.11.3-cp38-cp38-manylinux1_x86_64.whl" }}' \
-proto protos/pypi_transparency.proto \
-proto protos/package.proto \
${PYPI_SERVER}:443 pypitransparency.v1.PyPiTransparency.GetPackage
{
  "ok": true
}

# The following calls merely parse the protos but...
grpcurl \
-proto protos/pypi_transparency.proto \
-proto protos/package.proto \
list pypitransparency.v1.PyPiTransparency
pypitransparency.v1.PyPiTransparency.AddPackage
pypitransparency.v1.PyPiTransparency.GetInclusionProof
pypitransparency.v1.PyPiTransparency.GetPackage

grpcurl \
-proto protos/pypi_transparency.proto \
-proto protos/package.proto \
describe pypitransparency.v1.PyPiTransparency.AddPackage
pypitransparency.v1.PyPiTransparency.AddPackage is a method:
rpc AddPackage ( .pypitransparency.v1.AddPackageRequest ) returns ( .pypitransparency.v1.AddPackageResponse );

grpcurl \
-proto protos/pypi_transparency.proto \
-proto protos/package.proto \
describe pypitransparency.v1.PyPiTransparency.GetPackage
pypitransparency.v1.PyPiTransparency.GetPackage is a method:
rpc GetPackage ( .pypitransparency.v1.GetPackageRequest ) returns ( .pypitransparency.v1.GetPackageResponse );

Cloud SQL

INST=[[YOUR-INST]] # Perhaps "db"
ROOT=[[YOUR-ROOT]] # Something unique to this database instance
gcloud sql instances create ${INST} \
--tier=db-f1-micro \
--region=${REGION} \
--storage-type=HDD \
--availability-type=zonal \
--no-backup \
--database-version=MYSQL_5_7 \
--root-password=${ROOT} \
--project=${PROJECT}

We’ll need to create a database on the instance and populate it with Trillian’s schema.

In one shell:

CONN=$(\
  gcloud sql instances describe ${INST} \
  --project=${PROJECT} \
  --format="value(connectionName)")

./cloud_sql_proxy -instances=${CONN}=tcp:3306

NB You’ll need Google’s Cloud SQL Proxy

NB Once the proxy is running, it proxies traffic sent to it (localhost) to the remote instance (${DB})

Then, in another shell, get a client to the database:

VERSION="10.4.12"
docker run \
--interactive --tty \
--net=host \
mariadb:${VERSION} \
  sh -c "exec mysql --host=127.0.0.1 --port=3306 --password=${ROOT}"

And, paste in:

drop database if exists test;
create database test;
create user if not exists test@'%' identified by 'zaphod';
grant all on test.* to test@'%';

Then quit it and then:

USER="test"
PASS="zaphod"

docker run \
--interactive --tty \
--net=host \
--volume=${PWD}/trillian.sql:/trillian.sql \
mariadb:${VERSION} \
  sh -c "exec mysql --host=127.0.0.1 --port=3306 --user=${USER} --password=${PASS} test < /trillian.sql"

NB Here’s the script trillian.sql

Tidy

If you’re absolutely certain you don’t need the project, you can delete everything by deleting the project:

gcloud projects delete ${PROJECT} --quiet

Otherwise, you can delete the pieces:

for SERVICE in "trillian-log-server" "trillian-log-signer" "pypi-transparency-server"
do
  gcloud run services delete ${SERVICE} \
  --platform=managed \
  --region=${REGION} \
  --project=${PROJECT} \
  --quiet
done

and the Cloud SQL instance:

gcloud sql instances delete ${INST} --project=${PROJECT}


Done.