Cloud Endpoints combine OpenAPI and gRPC... or not!

- 5 minutes read - 1005 words

See:

Challenges:

  • Cloud Run permits single port
  • Cloud Run services publishing e.g. gRPC and Prometheus, must multiplex transports
  • Cloud Run services publishing multiplexed transports are challenging to expose using Cloud Endpoints

Hypothesis #1: Multiplexed transports work with Cloud Run

See: Multiplexing gRPC and HTTP endpoints with Cloud Run

Resuming from that post, there’s a Cloud Run service (SRV_HOST) deployed that uses cmux to multiple transports…

SRV_HOST=$(gcloud run services descripbe ${SRV_NAME} ...)
TOKEN="$(gcloud auth print-identity-token)"

gRPC

grpcurl \
-H "Authorization: Bearer ${TOKEN}" \
-d "{\"customer_id\": \"${ID}\"}" \
-proto something.proto \
${SRV_HOST}:443 v1alpha1.Something/CustomerGet
{
  "customer": {
    "name": "Daz Wilkin",
    "domainName": "dazwilkin.com"
  }
}

Prometheus (HTTP/1)

curl \
--silent \
--request GET \
-H "Authorization: Bearer ${TOKEN}" \
--output /dev/null  \
--write-out '%{response_code}\n' \
https://${SRV_HOST}/metrics
200

Hypothesis #2: Either gRPC xor HTTP (Prometheus) work with Cloud Endpoints

See: gRPC, Cloud Run & Endpoints

With minor changes (ESPv2 cloud_build_image now generates an image that includes a version number in its tag), this process continues to work.

gRPC

type: google.api.Service
config_version: 3
name: ESP_HOST
title: Some Service
apis:
  - name: v1alpha1.SomeService
usage:
  rules:
    - selector: "*"
      allow_unregistered_calls: true
backend:
  rules:
    - selector: "*"
      address: grpcs://SRV_HOST:443

NOTE This uses Cloud Endpoints for gRPC. There are three ways to configure Cloud Endpoints, using OpenAPI née Swagger see below, using gRPC as here and using Cloud Endpoints for Frameworks not discussed further.

And:

gcloud endpoints services deploy something.pb something.yaml
gcloud_build_image ...
gcloud run deploy ${ESP_NAME}

ESP_HOST=$(gcloud run describe ...)

Then, I can now gRPCurl the ESPv2 proxy’s endpoint to access the backend’s gRPC service:

grpcurl \
-H "Authorization: Bearer ${TOKEN}" \
-d "{\"customer_id\": \"${ID}\"}" \
-proto something.proto \
${ESP_HOST}:443 v1alpha1.Something/CustomerGet
{
  "customer": {
    "name": "Daz Wilkin",
    "domainName": "dazwilkin.com"
  }
}

NOTE ESP_HOST not SRV_HOST

Prometheus (HTTP/1)

swagger: "2.0"
info:
  title: Prometheus
  description: "Prometheus metrics"
  version: "0.0.1"
host: ESP_HOST
schemes:
  - https
produces:
  - "application/json"
x-google-backend:
  address: https://SRV_HOST:443
  protocol: "h2"
paths:
  "/metrics":
    get:
      summary: Prometheus metrics endpoint
      operationId: metrics
      responses:
        "200":
          description: Success
          schema:
            type: string
        "400":
          description: Failure

NOTE This uses Cloud Endpoints for OpenAPI rather than Cloud Endpoints for gRPC. Both specs are shown here in YAML and you should see commonalities (e.g. ESP_HOST) and differences (e.g. https://SRV_HOST:443 vs grpc://SRV_HOST:443)

Because the service config has changed, after gcloud services deploy‘ing it, we must rebuild the image and redeploy it too:

gcloud endpoints services deploy prometheus.yaml
gcloud_build_image ...
gcloud run deploy ${ESP_NAME}

NOTE The ESP_HOST name does not change between gcloud run deploy‘ments.

Then I can now curl the ESPv2 proxy’s endpoint to access the backend’s HTTP (Prometheus) service:

Hypothesis #3: Both gRPC and HTTP work with Cloud Endpoints (unproven)

If I can deploy either, I should be able to deploy both, but this doesn’t work:

gcloud endpoints services deploy something.pb something.yaml prometheus.yaml

This yields errors:

ERROR: (gcloud.endpoints.services.deploy)
HttpError accessing <https://servicemanagement.googleapis.com/v1/services/...
response: <{
    'x-debug-tracking-id': '...;o=0',
    'vary': 'Origin, X-Origin, Referer',
    'content-type': 'application/json; charset=UTF-8',
    'content-encoding': 'gzip',
    'date': 'Tue, 08 Jun 2021 19:31:40 GMT',
    'server': 'ESF',
    ...
    'status': 500
}>,
content <{
  "error": {
    "code": 500,
    "message": "Internal error encountered.",
    "status": "INTERNAL"
  }
}

NOTE Not very helpful but clearly it’s not happy.

The Google engineer was very helpful with the issue and cleverly suggested GET‘ing the OpenAPI service back as a gRPC service, i.e.:

CONFIG="$(\
  gcloud endpoints configs list \
  --service=${ESP_HOST} \
  --project=${PROJECT} \
  --format="json" \
  | jq -r .[0].id)")

gcloud endpoints configs describe ${CONFIG} \
--service=${ESP_HOST} \
--project=${PROJECT}

This yields (edited for sanity):

type: google.api.Service
configVersion: 3
name: ESP_HOST
title: Prometheus
apis:
- name: 0.ESP_HOST
  version: 0.0.1
  methods:
  - name: Metrics
    requestTypeUrl: type.googleapis.com/google.protobuf.Empty
    responseTypeUrl: type.googleapis.com/google.protobuf.Value
usage:
  rules:
  - allowUnregisteredCalls: true
    selector: 0.ESP_HOST.Metrics
backend:
  rules:
  - address: https://SRV_HOST:443
    jwtAudience: https://SRV_HOST:443
    pathTranslation: APPEND_PATH_TO_ADDRESS
    protocol: h2
    selector: 0.ESP_HOST.Metrics
http:
  rules:
  - get: /metrics
    selector: 0.ESP_HOST.Metrics

NOTE I’ve removed (hopefully) redundant elements and reordered what remains so that this should – if you squint – look like the child of the OpenAPI Prometheus and gRPC services.

Here’s Google reference docs for google.api.Service

Unfortunately, it doesn’t blend:

gcloud endpoints services deploy prometheus.service.yaml \
--project=${PROJECT}

Yields:

ERROR: (gcloud.endpoints.services.deploy) INVALID_ARGUMENT: Cannot convert to service config.
location: "prometheus.service.yaml:2"
kind: ERROR
message: "Found field \'configVersion\' which is unknown in \'google.api.Service\'."

location: "prometheus.service.yaml:10"
kind: ERROR
message: "Found field \'requestTypeUrl\' which is unknown in \'google.protobuf.Method\'."

location: "prometheus.service.yaml:11"
kind: ERROR
message: "Found field \'responseTypeUrl\' which is unknown in \'google.protobuf.Method\'."

location: "prometheus.service.yaml:14"
kind: ERROR
message: "Found field \'allowUnregisteredCalls\' which is unknown in \'google.api.UsageRule\'."

location: "prometheus.service.yaml:19"
kind: ERROR
message: "Found field \'jwtAudience\' which is unknown in \'google.api.BackendRule\'."

location: "prometheus.service.yaml:20"
kind: ERROR
message: "Found field \'pathTranslation\' which is unknown in \'google.api.BackendRule\'."

This is a result of an incorrect YAML marshaling (Issue #528) which is easily corrected by reviewing google.api.Service, e.g. for apis.methods:

type: google.api.Service
config_version: 3
name: ESP_HOST
title: Prometheus
apis:
- name: 0.ESP_HOST
  version: 0.0.1
  methods:
  - name: Metrics
    request_type_url: type.googleapis.com/google.protobuf.Empty
    response_type_url: type.googleapis.com/google.protobuf.Value
usage:
  rules:
  - allow_unregistered_calls: true
    selector: 0.ESP_HOST.Metrics
backend:
  rules:
  - address: https://SRV_HOST:443
    jwt_audience: https://SRV_HOST:443
    path_translation: APPEND_PATH_TO_ADDRESS
    protocol: h2
    selector: 0.ESP_HOST.Metrics
http:
  rules:
  - get: /metrics
    selector: 0.ESP_HOST.Metrics

yields:


ERROR: (gcloud.endpoints.services.deploy) INVALID_ARGUMENT: Cannot convert to service config.
location: "prometheus.service.yaml:6"
kind: ERROR
message: "Cannot resolve api \'0.ESP_HOST\'."

location: "prometheus.service.yaml:15"
message: "usage rule has selector(s) \'0.ESP_HOST.Metrics\' that do not match and are not shadowed by other rules."

location: "prometheus.service.yaml:26"
message: "backend rule has selector(s) \'0.ESP_HOST.Metrics\' that do not match and are not shadowed by other rules."

location: "prometheus.service.yaml:19"
message: "http rule has selector(s) \'0.ESP_HOST.Metrics\' that do not match and are not shadowed by other rules."

i.e. still screwed :-(

Epiphany #1: Proxy gRPC service but not Prometheus metrics

Since I can’t get this to work through Cloud Endpoints, is there another way?

Perhaps.

I can live with proxying the gRPC service only through Cloud Endpoints. This gives me documentation, monitoring, control over the gRPC service.

The Prometheus metrics are a single endpoint /metrics and I can continue to authenticate these requests and so I can limit access to the endpoint. No documentation. No API keys but I can live with that.