Using Google Monitoring Alerting to send Pushover notifications

Table of Contents

• Artifacts
• Pushover
• Caveat
• Cloud Monitoring Webhook
• Cloud Functions
• Cloud Run

Artifacts

• GitHub: go-gcp-pushover-notificationchannel
• Image: ghcr.io/dazwilkin/go-gcp-pushover-notificationchannel:220515

Pushover

Logging in to your Pushover account, you will be presented with a summary|dashboard page that includes Your User Key. Copy the value of this key into a variable called PUSHOVER_USER

Create New Application|API Token

Pushover API has a Pushing Messages method. The documentation describes the format of the HTTP Request. It must be a POST using TLS (https://) to https://api.pushover.net/1/messages.json. The content-type should be application/json. In the JSON body of the message, we must include token (PUSHOVER_TOKEN), user (PUSHOVER_USER_KEY), device (we’ll use cloud-monitoring) and a title and a message

Let’s try it before we proceed further:

URL="https://api.pushover.net/1/messages.json"

PUSHOVER_USERKEY="[YOUR-PUSHOVER-USER-KEY]"
PUSHOVER_TOKEN="[YOUR-PUSHOVER-API-TOKEN]"

DATA="
{
    \"token\": \"${PUSHOVER_TOKEN}\",
    \"user\": \"${PUSHOVER_USERKEY}\",
    \"device\": \"curl\",
    \"title\": \"test\",
    \"message\": \"test message\"
}
"

curl \
--request POST \
--header "content-type: application/json" \
--data "${DATA}" \
${USR}

That worked!

Caveat

In what follows, I document the process that I followed.

I learned that it’s not possible to use a Cloud Monitoring Channel type of webhook (neither webhook_basicauth nor webhook_tokenauth) because, these appear to permit only configuring a URL (and, in the case of webhook_tokenauth providing the bearer token).

For Pushover, it’s necessary to construct a message body (this also includes the authentication). I decided to try using Google Cloud Functions (2nd gen) but realized that, since this deploys an underlying Cloud Run service, it would be simpler to just use Cloud Run directly.

All this to say, if you want to see the end result, please skip ahead to Cloud Run

Cloud Monitoring Webhook

NOTE This approach is infeasible. Please see Cloud Run for a working solution

Now, we’ll try to configure a (Webhook) Notification Channel in a Google Cloud Monitoring project.

The documentation is not comprehensive.

Here’s the list of Channel descriptors. I tried webhook_tokenauth. In hindsight, Pushover’s authentication is performed in the message body (using user and token properties) so using token auth (i.e. Authorization: Bearer {token}) isn’t appropriate. I’d assumed at this point that it would be possible to construct a message body for the Webhook. This is from experience using other Webhook implementations. It appears to be an invalid assumption for Cloud Monitoring’s Channels.

It wasn’t obvious to me how to use the gcloud beta monitoring channels create command’s --channel flags. I realized that this is modeled after NotificationChannel which helped me realize that I could either use --channel-labels for specific properties or pass the entire JSON object using --channel-content-from-file but this would benefit from better documentation.

As it always the case, I find Google’s APIs Explorer to be an excellent tool and I tried using the underlying REST method to create then get channels directly. Here’s the message body for the create method that I tried:

{
  "displayName": "test",
  "type": "webhook_tokenauth",
  "labels": {
    "url": "https://api.pushover.net/1/messages.json"
  },
  "userLabels": {
    "token": "{token}",
    "user": "{user}",
    "title": "test",
    "message": "test"
  }
}

This was my attempt to construct a message body (required by Pushover) using NotificationChannel’s userLabels but this doesn’t work. The same command using gcloud is shown below:

BILLING="[YOUR-BILLING-ID]"
PROJECT="[YOUR-PROJECT-ID]"

URL="https://api.pushover.net/1/messages.json"

PUSHOVER_USERKEY="[YOUR-PUSHOVER-USER-KEY]"
PUSHOVER_TOKEN="[YOUR-PUSHOVER-API-TOKEN]"

gcloud projects create ${PROJECT}

gcloud beta billing projects link ${PROJECT} \
--billing-account=${BILLING}

gcloud beta monitoring channels create \
--display-name=pushover \
--type=webhook_tokenauth \
--channel-labels=url=${URL} \
--user-labels=token=${PUSHOVER_TOKEN} \
--user-labels=user=${PUSHOVER_USERKEY} \
--user-labels=device=cloud-monitoring \
--user-labels=title=test \
--user-labels=message=test \
--project=${PROJECT}

Then:

gcloud beta monitoring channels list --project=${PROJECT}
---
creationRecord:
  mutateTime: '2022-05-14T00:00:00.000000000Z'
displayName: pushover
enabled: true
labels:
  url: https://api.pushover.net/1/messages.json
mutationRecords:
- mutateTime: '2022-05-14T00:00:00.000000000Z'
name: projects/{project}/notificationChannels/{channel-id}
type: webhook_tokenauth

So, I abandoned this approach.

Cloud Functions (2nd gen)

NOTE This approach works but – in my opinion – using Cloud Run directly is a better solution.

Google Cloud Functions has been my go-to service for quick-hit solutions where I need e.g. Webhooks and so it was a natural next choice. Again, in hindsight, and because I chose to explore Cloud Function (2nd gen), this was a sub-optimal choice.

First, Google has complicated the develoepr flow. In Go (and we’re still stuck with Go 1.16 by the way in May 2022 when Go 1.18 has been available for ~month), it’s necessary to add an init function and import a module (and its dependencies).

Second – and more importantly – Cloud Functions (2nd gen) deploys a Cloud Function that uses an underlying Cloud Run service. Why add complexity? I decided to just use Cloud Run directly.

This solution works but…

BILLING="[YOUR-BILLING-ID]"
PROJECT="[YOUR-PROJECT-ID]"
REGION="[YOUR-REGION]"
NAME="pushover"

URL="https://api.pushover.net/1/messages.json"

PUSHOVER_USERKEY="[YOUR-PUSHOVER-USER-KEY]"
PUSHOVER_TOKEN="[YOUR-PUSHOVER-API-TOKEN]"

gcloud projects create ${PROJECT}

gcloud beta billing projects link ${PROJECT} \
--billing-account=${BILLING}

for SERVICE in "artifactregistry" "cloudbuild" "cloudfunctions" "run"
do
  gcloud services enable ${SERVICE}.googleapis.com \
  --project=${PROJECT}
done

# Deploy Cloud Functions
gcloud beta functions deploy ${NAME} \
--gen2 \
--runtime=go116 \
--trigger-http \
--entry-point=Webhook \
--set-env-vars=\
PUSHOVER_USERKEY=${PUSHOVER_USERKEY},\
PUSHOVER_TOKEN=${PUSHOVER_TOKEN} \
--source . \
--allow-unauthenticated \
--region=${REGION} \
--project=${PROJECT}

# Get (URL)
# NB The value is URI
URL=$(\
  gcloud beta functions describe ${NAME} \
  --gen2 \
  --region=${REGION} \
  --project=${PROJECT} \
  --format="value(serviceConfig.uri)") && \
echo ${URL}

# Create Notification Channel
gcloud beta monitoring channels create \
--display-name=${NAME} \
--type=webhook_tokenauth \
--channel-labels=url=${URL} \
--project=${PROJECT}

You can test the channel by editing it within Cloud Console and using the console’s test feature or of course by using it in an alert.

Cloud Run

Lastly, I created a Dockerfile and deployed the code as a Cloud Run service. Simple. It works. Why not?

BILLING="[YOUR-BILLING-ID]"
PROJECT="[YOUR-PROJECT-ID]"
REGION="[YOUR-REGION]"
NAME="pushover"

URL="https://api.pushover.net/1/messages.json"

PUSHOVER_USERKEY="[YOUR-PUSHOVER-USER-KEY]"
PUSHOVER_TOKEN="[YOUR-PUSHOVER-API-TOKEN]"

gcloud projects create ${PROJECT}

gcloud beta billing projects link ${PROJECT} \
--billing-account=${BILLING}

gcloud beta run deploy ${NAME} \
--ingress=all \
--set-env-vars=PUSHOVER_USERKEY=${PUSHOVER_USERKEY},\
PUSHOVER_TOKEN=${PUSHOVER_TOKEN} \
--source . \
--region=${REGION} \
--allow-unauthenticated \
--platform=managed \
--project=${PROJECT}

URL=$(\
  gcloud beta run services describe ${NAME} \
  --region=${REGION} \
  --project=${PROJECT} \
  --format="value(status.Url)") && \
echo ${URL}

gcloud beta monitoring channels create \
--display-name=${NAME} \
--type=webhook_tokenauth \
--channel-labels=url=${URL} \
--project=${PROJECT}

You can test the channel by editing it within Cloud Console and using the console’s test feature or of course by using it in an alert.

That’s all!

• GCP
• Cloud-Monitoring
• Webhook
• Pushover