March 14, 2023

Access Google Services using gRPC

Google publishes interface definitions of Google APIs (services) that support REST and gRPC in a repo called Google APIs. Google’s SDKs uses gRPC to access these services but, how to do this using e.g. gRPCurl?

I wanted to debug Cloud Profiler and its agent makes UpdateProfile RPCs to cloudprofiler.googleapis.com. Cloud Profiler is more challenging service to debug because (a) it’s publicly “write-only”; and (b) it has complex messages. UpdateProfile sends UpdateProfileRequest messages that include Profile messages that include profile_bytes which are gzip compressed serialized protos of pprof’s Profile.

March 10, 2023

Kubernetes Operators

Ackal uses a Kubernetes Operator to orchestrate the lifecycle of its health checks. Ackal’s Operator is written in Go using kubebuilder.

Yesterday, my interest was piqued by a MetalBear blog post Writing a Kubernetes Operator [in Rust]. I spent some time reimplementing one of Ackal’s CRDs (Check) using kube-rs and not only refreshed my Rust knowledge but learned a bunch more about Kubernetes and Operators.

While rummaging around the Kubernetes documentation, I discovered flant’s Shell-operator and spent some time today exploring its potential.

February 15, 2023

Secure (TLS) gRPC services with LKE

NOTE cert-manager is a better solution to what follows.

I wrote about deploying Secure (TLS) gRPC services with Vultr Kubernetes Engine (VKE). This week, I’ve reproduced this deployment using Linode Kubernetes Engine (LKE).

Thanks to the consistency provided by Kubernetes, the Kubernetes programming is almost identical. The main differences are between the CLI’s provided by these platforms. Both are good. They’re just different.

I’m going to include the linode-cli commands I’m using in this post as I found it slightly more quirky.

January 4, 2023

Authenticate PromLens to Google Managed Prometheus

I’m using Google Managed Service for Prometheus (GMP) and liking it.

Sometime ago, I tried using PromLens with GMP but GMP’s Prometheus HTTP API endpoint requires auth and I’ve battled Prometheus’ somewhat limited auth mechanism before (Scraping metrics exposed by Google Cloud Run services that require authentication).

Listening to PromCon EU 2022 videos, I learned that PromLens has been open sourced and contributed to the Prometheus project. Eventually, the functionality of PromLens should be combined into the Prometheus UI.

November 28, 2022

Maintaining Container Images

As I contemplate moving my “thing” into production, I’m anticipating aspects of the application that need maintenance and how this can be automated.

I’d been negligent in the maintenance of some of my container images.

I’m using mostly Go and some Rust as the basis of static(ally-compiled) binaries that run in these containers but not every container has a base image of scratch. scratch is the only base image that doesn’t change and thus the only base image that doesn’t require that container images buit FROM it, be maintained.

November 17, 2022

Delegate domain-wide authority using Golang

I’d not used Google’s Domain-wide Delegation from Golang and struggled to find example code.

Google provides Java and Python samples.

Google has a myriad packages implementing its OAuth security and it’s always daunting trying to determine which one to use.

As it happens, I backed into the solution through client.Options

ctx := context.Background()

// Google Workspace APIS don't use IAM do use OAuth scopes
// Scopes used here must be reflected in the scopes on the
// Google Workspace Domain-wide Delegate client
scopes := []string{ ... }

// Delegates on behalf of this Google Workspace user
subject := "a@google-workspace-email.com"

creds, _ := google.FindDefaultCredentialsWithParams(
    ctx,
    google.CredentialsParams{
        Scopes: scopes,
        Subject: subject,
    },
)
opts := option.WithCredentials(creds)
service, _ := admin.NewService(ctx, opts)

In this case NewService applies to Google’s Golang Admin SDK API although the pattern of NewService(ctx) or NewService(ctx, opts) where opts is a option.ClientOption is consistent across Google’s Golang libraries.

November 14, 2022

`curl`'ing a Tailscale Webhook

[Tailscale] is really good. I’ve been using it as a virtual private network to span 2 home networks and to securely (!) access my hosts when I’m remote.

Recently Tailscale added Webhook functionality to permit processing subscribed-to (Tailscale) events. I’m always a sucker for a webhook ;-)

Here’s a curl command to send a test event to a Tailscale Webhook:

URL=""

# From Tailscale's docs
# https://tailscale.com/kb/1213/webhooks/#events-payload
BODY='
[
  {
    "timestamp": "2022-09-21T13:37:51.658918-04:00",
    "version": 1,
    "type": "test",
    "tailnet": "example.com",
    "message": "This is a test event",
    "data": null
  }
]
'

T=$(date +%s)
V=$(\
  printf "${T}.${BODY}" \
  | openssl dgst -sha256 -hmac "${SECRET}" -hex -r \
  | head --bytes=64)

curl \
--request POST \
--header "Tailscale-Webhook-Signature:t=${T},v1=${V}" \
--header "Content-Type: application/json" \
--data "${BODY}" \
https://${URL}

There must be a better way of extracting the hashed value from the openssl output.

November 11, 2022

The curious cases of the `deleted:serviceaccount`

While testing Firestore export and import yesterday and checking the IAM permissions on a Cloud Storage Bucket, I noticed some Member (member) values (I think Google refers to these as Principals) were logical but unfamiliar to me:

deleted:serviceAccount:{email}?uid={uid}

I was using gsutil iam get gs://${BUCKET} because I’d realized (and this is another useful lesson) that, as I’ve been creating daily test projects, I’ve been binding each project’s Firestore Service Account (service-{project-number}@gcp-sa-firestore.iam.gserviceaccount.com) to a Bucket owned by another Project but I hadn’t been deleting the binding when I deleted the Project.

November 10, 2022

Firestore Export & Import

I’m using Firestore to maintain state in my “thing”.

In an attempt to ensure that I’m able to restore the database, I run (Cloud Scheduler) scheduled backups (see Automating Scheduled Firestore Exports and I’ve been testing imports to ensure that the process works.

It does.

I thought I’d document an important but subtle consideration with Firestore exports (which I’d not initially understood).

Google facilitates that backup process with the sibling commands:

November 6, 2022

Basic programmatic access to GitHub Issues

It’s been a while!

I’ve been spending time writing Bash scripts and a web site but neither has been sufficiently creative that I’ve felt worth a blog post.

As I’ve been finalizing the web site, I needed an Issue Tracker and decided to leverage GitHub(’s Issues).

As a former Googler, I’m familiar with Google’s (excellent) internal issue tracking tool (Buganizer) and it’s public manifestation Issue Tracker. Google documents Issue Tracker and its Issue type which I’ve mercilessly plagiarized in my implementation.