Using `gcloud ... --format` with arbitrary returned data
- 6 minutes read - 1217 wordsIf you use jq, you’ll know that, its documentation uses examples that you can try locally or using the excellent jqplay:
printf "[1,2,3]" | jq .[1:]
[
2,
3
]
And here
If you use Google Cloud Platform (GCP) CLI, gcloud, this powerful tool includes JSON output formatting of results (--format=json) and YAML (--format=yaml) etc. and includes a set of so-called projections that you can use to format the returned data.
There is a comparable slice projection that you may use with gcloud and the documentation even includes an example:
[1,2,3].slice(1:)
But, you can’t try out that command.
You can’t:
gcloud just --format="json([1,2,3].slice(1:))"
To prove to yourself or, more importantly, someone else that it works (correctly).
I often provide this type of example to other developers but, unless we’re working with a specific resource, it can be challenging to provide an example.
Using the [slice] projection, how do we provide a developer with an example?
First, let’s find a command that returns a list.
An immediate problem is that we want something that’s sufficiently generic that any GCP user will be able to test it without having to create e.g. a Kubernetes cluster to have a set of resources to use.
gcloud projects list is very generic but it’s not useful in this case. Even though the underlying response is a JSON object containing a "projects" property that is a list:
gcloud projects list --format="yaml" --log-http
==== request start ====
uri: https://cloudresourcemanager.googleapis.com/v1/projects?alt=json
==== request end ====
---- response start ----
-- body start --
{
"projects": [
{
"projectNumber": "000000000001",
"projectId": "foo",
"lifecycleState": "ACTIVE",
"name": "foo",
"createTime": "2021-01-01T00:00:00.000Z"
},
{
"projectNumber": "000000000002",
"projectId": "bar",
"lifecycleState": "ACTIVE",
"name": "bar",
"createTime": "2021-01-01T00:00:00.000Z"
}
]
}
-- body end --
After gcloud has proceed it, we’re left with only the list items. We can confirm this by formatting the output either as --format=json (an array) or --format=yaml (multiple YAML documents):
[
{
"projectNumber": "000000000001",
"projectId": "foo",
"lifecycleState": "ACTIVE",
"name": "foo",
"createTime": "2021-01-01T00:00:00.000Z"
},
{
"projectNumber": "000000000002",
"projectId": "bar",
"lifecycleState": "ACTIVE",
"name": "bar",
"createTime": "2021-01-01T00:00:00.000Z"
}
]
And:
---
createTime: '2021-01-01T00:00:00.000Z'
lifecycleState: ACTIVE
name: foo
projectId: foo
projectNumber: '000000000001'
---
createTime: '2021-01-01T00:00:00.000Z'
lifecycleState: ACTIVE
name: bar
projectId: bar
projectNumber: '000000000002'
---
The problem is that, there’s no list for us to slice.
Another option is gcloud services list although this requires that we have a project. That’s not too burdensome, so let’s use that:
gcloud services list --project=${PROJECT} --format=JSON --log-http
==== request start ====
uri: https://serviceusage.googleapis.com/v1/projects/...
==== request end ====
---- response start ----
-- body start --
-- body start --
I’m going to format the body as JSON to help:
{
"services": [
{
"name": "projects/000000000000/services/bigquery.googleapis.com",
"config": {
"name": "bigquery.googleapis.com",
"title": "BigQuery API",
"documentation": {
"summary": "A data platform for customers to create, manage, share and query data."
},
"quota": {},
"authentication": {},
"usage": {
"requirements": [
"serviceusage.googleapis.com/tos/cloud"
]
},
"monitoring": {}
},
"state": "ENABLED",
"parent": "projects/000000000000"
},
{
"name": "projects/000000000000/services/bigquerystorage.googleapis.com",
"config": {
"name": "bigquerystorage.googleapis.com",
"title": "BigQuery Storage API",
"documentation": {},
"quota": {},
"authentication": {},
"usage": {
"requirements": [
"serviceusage.googleapis.com/tos/cloud"
]
},
"monitoredResources": [
{
"type": "serviceruntime.googleapis.com/api",
"labels": [
{
"key": "cloud.googleapis.com/location"
},
{
"key": "cloud.googleapis.com/uid"
}
]
},
{
"type": "serviceruntime.googleapis.com/consumer_quota",
"labels": [
{
"key": "cloud.googleapis.com/location"
},
{
"key": "cloud.googleapis.com/uid"
}
]
},
],
"monitoring": {
"consumerDestinations": [
{
"monitoredResource": "serviceruntime.googleapis.com/api",
"metrics": [
"serviceruntime.googleapis.com/api/consumer/quota_used_count",
"serviceruntime.googleapis.com/api/consumer/quota_refund_count",
]
},
{
"monitoredResource": "serviceruntime.googleapis.com/consumer_quota",
"metrics": [
"serviceruntime.googleapis.com/quota/rate/consumer/used_count",
"serviceruntime.googleapis.com/quota/rate/consumer/refund_count",
]
}
]
}
},
"state": "ENABLED",
"parent": "projects/641009261266"
}
]
}
Which is rich and should provide us with a goodly source of data to use.
NOTE Once again we do lose the root property (
servicesthis time) but we have other lists to play with.
Check out biquerystorage.googleapis.com and .config.monitoring.consumerDestions[0].metrics, there are 6 of them:
FILTER="config.monitoring.consumerDestinations[0]"\
".metrics.len()"
gcloud services list \
--project=${PROJECT} \
--filter="config.name=bigquerystorage.googleapis.com" \
--format="json(${FILTER})"
Yields:
[
{
"config": {
"monitoring": {
"consumerDestinations": [
{
"metrics": 6
}
]
}
}
}
]
Now, if we apply slice(1:) there should be 5:
FILTER="config.monitoring.consumerDestinations[0]"\
".metrics.slice(1:).len()"
gcloud services list \
--project=${PROJECT} \
--filter="config.name=bigquerystorage.googleapis.com" \
--format="json(${FILTER})"
Yields:
[
{
"config": {
"monitoring": {
"consumerDestinations": [
{
"metrics": 5
}
]
}
}
}
]
So, it’s long-winded but it can be done.
I submitted an issue to Google’s issustracker and there’s a clever workaround that may help with this. It takes advantage of the fact that you can override the endpoint used by gcloud commands.
Overrides are service-specific. So, in the case of gcloud services list, this uses https://serviceusage.googleapis.com and so to override gcloud services commands, we can:
gcloud config set \
api_endpoint_overrides/serviceusage \
http://localhost:8080
NOTE This overrides
gcloud servicesfor all localgcloud servicescalls so be careful that you don’t break something else while doing this. When you’re done,gcloud config unset api_endpoint_overrides/serviceusage.
So, what does our new endpoint (htttp://localhost:8080) do? Well, in this case (it differs by gcloud command), for gcloud services list ... --log-http, we see (edited for clarity) that gcloud (!) expects JSON (application/json):
==== request start ====
uri: https://serviceusage.googleapis.com/v1/projects/...
method: GET
== headers start ==
b'accept': b'application/json'
== headers end ==
==== request end ====
---- response start ----
status: 200
-- headers start --
Content-Type: application/json; charset=UTF-8
-- headers end --
And so a simple Python HTTP server that provides an acceptable (it must be the structure that gcloud expects) would be:
import socketserver
s ="""HTTP/1.1 200 OK
MIME-Version: 1.0
Content-Type: application/json
{
"services": [{
"name": "foo",
"config": {
"name": "foo",
"title": "Foo",
"documentation": {
"summary": ""
},
"quota": {},
"authentication": {},
"usage": {
"requirements": []
},
"monitoredResources": [{
"labels": [{
"key": "key-01"
}],
"type": ""
}],
"monitoring": {
"consumerDestinations": [{
"monitoredResource": "
"metrics": [
"a",
"b",
"c"
]
}]
}
},
"state": "ENABLED",
"parent": "projects/000000000000"
}]
}
"""
b = s.encode()
print(b)
socketserver.TCPServer(
("",8080),
lambda a,b,c:a.sendall(s.encode())
).serve_forever()
NOTE Credit for the source of the above code to Google.
Run the above server and then:
# Matches the arbitrarily named config provided by our server
NAME="foo"
FILTER="config.monitoring.consumerDestinations[0]"\
".metrics"
gcloud services list \
--project=${PROJECT} \
--filter="config.name=${NAME}" \
--format="json(${FILTER})"
Yields:
[
{
"config": {
"monitoring": {
"consumerDestinations": [
{
"metrics": [
"a",
"b",
"c"
]
}
]
}
}
}
]
And:
FILTER="config.monitoring.consumerDestinations[0]"\
".metrics.slice(1:).len()"
gcloud services list \
--project=${PROJECT} \
--filter="config.name=${NAME}" \
--format="json(${FILTER})"
Yields:
[
{
"config": {
"monitoring": {
"consumerDestinations": [
{
"metrics": 2
}
]
}
}
}
]
You can, of course, just --format=json | jq . and then you get to use jq instead of --format=json and Google’s way. The advantage to knowing Google’s projections is one less tool. The advantage of use jq is the UNIX ethos of the “do one thing well”.
Let’s conclude with the original jq example applied to gcloud services list:
FILTER=".[0]"\
"|.config.monitoring.consumerDestinations[0].metrics"
gcloud services list
--project=${PROJECT} \
--filter="config.name=${NAME}" \
--format="json" \
| jq -r "${FILTER}"
Yields:
[
"a",
"b",
"c"
]
And:
FILTER=".[0]"\
"|.config.monitoring.consumerDestinations[0].metrics[1:]"
gcloud services list
--project=${PROJECT} \
--filter="config.name=${NAME}" \
--format="json" \
| jq -r "${FILTER}"
[
"b",
"c"
]
NOTE
jqreceives an array of one item fromgcloudso we start by grabbing its (.[0]) content. The command is otherwise equivalent but gives us access to all jq`s methods.
jq (I’m unsure whether gcloud format) permits filtering arrays by an object’s properties:
FILTER=".[]"\
"|select(.config.name=\"foo\")"\
"|.config.monitoring.consumerDestinations[]"\
"|select(.monitoredResource=\"baz\")|.metrics[1:]"
gcloud services list
--project=${PROJECT} \
--filter="config.name=${NAME}" \
--format="json" \
| jq -r "${FILTER}"
Yields:
[
"b",
"c"
]
That’s all!