Programmatically deploying Cloud Run services (Golang|Python)

March 16, 2021 - 7 minutes read - 1420 words

Phew! Programmitcally deploying Cloud Run services should be easy but it didn’t find it so.

My issues were that the Cloud Run Admin (!) API is poorly documented and it uses non-standard endpoints (thanks Sal!). Here, for others who may struggle with this, is how I got this to work.

Goal

Programmatically (have Golang, Python, want Rust) deploy services to Cloud Run.

i.e. achieve this:

gcloud run deploy ${NAME} \
--image=${IMAGE} \
--platform=managed \
--no-allow-unauthenticated \
--region=${REGION} \
--project=${PROJECT}

TRICK --log-http is your friend

I’ll document this here as it’s generally applicable but, as I struggled to emulate the gcloud run deploy command in code, I used --log-http to see how Cloud SDK achieves this. Running the above command with --log-http yields:

uri: https://${REGION}-run.googleapis.com/apis/serving.knative.dev/v1/namespaces/${PROJECT}/services/${NAME}
method: GET
...
uri: https://${REGION}-run.googleapis.com/apis/serving.knative.dev/v1/namespaces/${PROJECT}/services
method: POST
== body start ==
{
    "apiVersion": "serving.knative.dev/v1",
    "kind": "Service",
    "metadata": {
        "annotations": {...},
        "labels": {},
        "name": "${NAME}",
        "namespace": "${PROJECT}"
    },
    "spec": {...}
    ...

I draw your attention to:

The endpoint is https://${REGION}-run.googleapis.com. This is documented Service Endpoint. It’s possible to GET (lists) on https://run.googleapis.com but to create, edit, delete etc. you must use https://${REGION}-run.googleapis.com.
The path includes namespaces/${PROJECT}. The documentation explains that this value may be either the Project ID or the Project number. Because the service is a Kubernetes API, Namespaces are employed.
The POST’s body is a Kubernetes (Knative) Service manifest. It is documented by Google here but I found some inconsistencies in this documentation.

TRICK Once deployed, you can view a service either in the Console or using e.g. gcloud to verify the YAML manifest

gcloud run services describe ${NAME} \
--platform=managed \
--region=${REGION} \
--project=${PROJECT} \
--format=yaml

Code

There are 2 insights necessary to get the code to work:

The endpoint must be adjusted to reflect https://${REGION}-run.googleapis.com
The manifest must be error-free

After some searching, I was unable to find either Cloud Client Libraries for the Cloud Run Admin service or Protobufs. Google provides [API Client Libraries] in multiple languages for every service. API Client Libraries are the historical REST-based mechanism, are machine-generated and are a good fallback. Google Cloud (!) offers an alternative called Cloud Client Libraries. These are (often) hand-crafted and sometimes use gRPC (rather than REST). For some of its services, Google provides the services’ Protobufs and this provides a mechanism by which developers can use protoc to generate client libraries in their language of choice.

Google’s Client Libraries Explained

An (as-)definitive(-as-it-gets) list of Google’s Cloud Client Libraries. Here’s the Golang Supported APIs for Cloud Client Libraries and the Python Libraries. It’s possible (!) that there’s a Cloud Client available for one language but not another. In this case, it appears there are no Cloud Client Libraries for Cloud Run (Admin) API.

So, then I had a look for gRPC (Protobuf) definitions. Confusingly, this repo is called googleapis/googleapis and you must navigate directories to find libraries and then those for cloud. No Cloud Run.

So, back to the ever-reliable API Client Libraries. This page’s links tend to go directly to the GitHub repo but, for Golang, here’s the documentation list and for Python list.

NOTE For Golang, the API Client Libraries are google.golang.org and the Cloud Client Libraries are cloud.google.com

INSIGHT Cloud Run Admin API|service only has API Client Libraries.

The API Client Libraries all provide a mechanism (called ClientOptions) to override the default service endpoint. I was unfamiliar with this until a friend (thanks Sal!) told me about it:

Golang’s option
Python’s ClientOptions

Environment

For the client code to work, we need a GCP Project, Billing Account, Service Account(s) etc.

PROJECT="[[YOUR-PROJECT]]"
BILLING="[[YOUR-BILLING]]"
DEPLOYR="deployr"
INVOKER="invoker"

gcloud projects create ${PROJECT}
gcloud beta billing projects link ${PROJECT} \
--billing-account=${BILLING}

gcloud services enable run.googleapis.com \
--project=${PROJECT}

# For Deploying Services
gcloud iam service-accounts create ${DEPLOYR} \
--project=${PROJECT}


EMAIL=${DEPLOYR}@${PROJECT}.iam.gserviceaccount.com

gcloud iam service-accounts keys create ${PWD}/${DEPLOYR}.json \
--iam-account=${EMAIL} \
--project=${PROJECT}

gcloud projects add-iam-policy-binding ${PROJECT} \
--member=serviceAccount:${EMAIL} \
--role=roles/run.admin

# For Services to invoke other services
gcloud iam service-accounts create ${INVOKER} \
--project=${PROJECT}

# Deployer must be able to act-as the Invoker
gcloud iam service-accounts add-iam-policy-binding \
${INVOKER}@${PROJECT}.iam.gserviceaccount.com \
--member=serviceAccount:${DEPLOYR}@${PROJECT}.iam.gserviceaccount.com \
--role=roles/iam.serviceAccountUser \
--project=${PROJECT}

# Export variables
export PROJECT
export REGION="[[YOUR-REGION]]"
export IMAGE="[[YOUR-IMAGE]]"
export ACCOUNT=${INVOKER}

export GOOGLE_APPLICATION_CREDENTIALS=${PWD}/${DEPLOYR}.json

Golang

I’ll annotate the code below where there’s novelty.

It can be challenging to determine which Google client library to use. Google doesn’t always effectively cull dead repos from GitHub.

With Golang, the machine-generated documentation is always excellent:

https://pkg.go.dev/google.golang.org/api/run/v1

package main

import (
  "context"
  "fmt"
  "log"
  "os"

  "google.golang.org/api/googleapi"
  "google.golang.org/api/option"
  run "google.golang.org/api/run/v1"
)

func main() {
  project := os.Getenv("PROJECT")
  region := os.Getenv("REGION")
  image := os.Getenv("IMAGE")
  account := os.Getenv("ACCOUNT")

Per above, we must use API Client Libraries, hence google.golang.org. We’re also using Golang’s Client Options and so google.golang.org/api/option.

For simplicity, we’re assuming a bunch of variables from the environment including the Region.

  ctx := context.Background()

  url := fmt.Sprintf("https://%s-run.googleapis.com/", region)
  opts := []option.ClientOption{
    option.WithEndpoint(url),
  }

  runService, err := run.NewService(ctx, opts...)
  if err != nil {
    log.Fatal(err)
  }

The Region is important because, the Cloud Run Admin service’s endpoint is dependent on the Region when we wish to deploy new services. The code above uses ClientOption to override the service’s default (https://run.googleapis.com) endpoint with one prefixed with the Region. This is applied to the configuration using WithEndpoint.

NOTE The endpoint is also ${REGION}-run (hyphenated).

  name := "golang"
  containerName := fmt.Sprintf(
    "%s-%s-%s",
    name,
    "00001",
    "abc",
  )
  serviceAccount := fmt.Sprintf(
    "%s@%s.iam.gserviceaccount.com",
    account,
    project,
  )


  service := &run.Service{
    ApiVersion: "serving.knative.dev/v1",
    Kind:       "Service",
    Metadata: &run.ObjectMeta{
      Name:      name,
      Namespace: project,
      Labels: map[string]string{
        "test":                          "test",
        "cloud.googleapis.com/location": region,
      },
      Annotations: map[string]string{
        "client.knative.dev/user-image":   image,
        "run.googleapis.com/launch-stage": "BETA",
        "run.googleapis.com/sandbox":      "gvisor",
      },
    },
    Spec: &run.ServiceSpec{
      Template: &run.RevisionTemplate{
        Metadata: &run.ObjectMeta{
          Name: containerName,
          Annotations: map[string]string{
            "autoscaling.knative.dev/maxScale": "1",
            "client.knative.dev/user-image":    image,
            "run.googleapis.com/launch-stage":  "BETA",
            "run.googleapis.com/sandbox":       "gvisor",
          },
        },
        Spec: &run.RevisionSpec{
          Containers: []*run.Container{
            {
              Image: image,
              Ports: []*run.ContainerPort{
                {
                  Name: "http1",
                  ContainerPort: 8080,
                },
              },
            },
          },
          ServiceAccountName: serviceAccount,
        },
      },
    },
  }

The creation of a run.Service is the heart of the code and, as you’d suspect, is where we define the Cloud Run service. This type is to be found under the Cloud Run service’s namespaces.services.create method and is called Service.

In theory, you should be able to treat Google’s documentation as definitive but I struggled with some discrepancies:

Metadata.Labels: I was unable to use dot-notation for the keys
Metadata.Annotations: I’m unsure where these annotations are documented
Spec.Template.Metadata.Name: Can’t just be name and must be {something}-[0-9]{5}-[a-z]{3}
Spec.Template.Spec.Containers[].Ports.Name: This is documented but it can’t be http, use http1
Spec.Template.Spec.Containers[].Ports.Protocol: I was unable to use TCP; either remove it or use ""
Spec.Template.Spec.Containers[].Resources: This is permitted but I’ve omitted it
Spec.Template.Traffic: I was unsuccessful using this and so omitted it

  parent := fmt.Sprintf("namespaces/%s", project)
  rqst := runService.Namespaces.Services.Create(parent, service)
  resp, err := rqst.Do()

  if err != nil {
    if e, ok := err.(*googleapi.Error); ok {
      log.Fatalf("Failed to deploy: %s [%d]\n%s",
        e.Message,
        e.Code,
        e.Body,
      )
    }
    log.Fatal(err)
  }

  log.Print(resp)
}

Once the manifest is correct(!), you can make the request and, if there are no errors, return the response.

NOTE The parent value is different for Cloud Run and uses namespace/${PROJECT}.

Then you should be able to: go run .

And then:

gcloud run services describe ${NAME} \
--platform=managed \
--region=${REGION} \
--project=${PROJECT}

Python

Here’s a somewhat idiosyncratic version of the Golang code in Python:

import google.auth
import os

from google.api_core.client_options import ClientOptions
from googleapiclient import discovery

REGION = os.getenv("REGION")
IMAGE = os.getenv("IMAGE")
ACCOUNT = os.getenv("ACCOUNT")

url = "https://{region}-run.googleapis.com/".format(region=REGION)
options = ClientOptions(api_endpoint=url)

credentials, project = google.auth.default()

service = discovery.build("run", "v1",
  client_options=options, credentials=credentials)

name = "python"
container_name = "{name}-{counter}-{postfix}".format(
  name=name, counter="00001", postfix="abc")
service_account = "{account}@{project}.iam.gserviceaccount.com".format(
  account=ACCOUNT, project=project)

body = {
  "apiVersion": "serving.knative.dev/v1",
  "kind": "Service",
  "metadata": {
    "name": name,
    "namespace": project,
    "labels": {
      "test": "test",
      "cloud.googleapis.com/location": REGION,
    },
    "annotations": {
      "client.knative.dev/user-image": IMAGE,
      "run.googleapis.com/launch-stage": "BETA",
      "run.googleapis.com/sandbox": "gvisor",
    },
  },
  "spec": {
    "template": {
      "metadata": {
        "name": container_name,
        "annotations": {
          "client.knative.dev/user-image": IMAGE,
          "run.googleapis.com/launch-stage": "BETA",
          "run.googleapis.com/sandbox": "gvisor",
        },
      },
      "spec": {
        "containers": [{
          "image": IMAGE,
          "ports": [{
            "name": "http1",
            "containerPort": 8080,
          }]
        }],
        "serviceAccountName": service_account,
      }
    }
  }
}

parent = "namespaces/{project}".format(project=project)
rqst = service.namespaces().services().create(parent=parent, body=body)
resp = rqst.execute()
print(resp)

Testing

Once you’ve deployed a service (requiring authentication), you can test it using:

ENDPOINT=$(\
  gcloud run services describe ${NAME} \
  --platform=managed \
  --region=${REGION} \
  --project=${PROJECT} \
  --format="value(status.address.url)") && echo ${ENDPOINT}

curl \
--verbose \
--request GET \
--header "Authorization: Bearer $(gcloud auth print-identity-token)" \
${ENDPOINT}

Tidy

When you’re done, you may either delete the Project entirely:

gcloud projects delete ${PROJECT} --quiet

Or delete the services:

gcloud run services delete ${NAME} \
--platform=managed \
--region=${REGION} \
--project=${PROJECT} \
--quiet

And service accounts:

for ACCOUNT in "${DEPLOYR}" "${INVOKER}"
do
  gcloud iam service-accounts delete ${ACCOUNT}@${PROJECT}.iam.gserviceaccount.com \
  --project=${PROJECT} \
  --quiet
done

That’s all!