Sigstore

October 7, 2021 - 6 minutes read - 1097 words

I’ve been on a digression (gcp-oidc-token-proxy) this week. Yesterday I began exploring Podman and wrote briefly about running gcp-oidc-token-proxy on my localhost using it.

This morning while walking with my dog, I listened to Google’s Dan Lorenc explain Sigstore (blog](https://blog.sigstore.dev/)) on The Kubelist Podcast¹

The plan today is to try to sign the gcp-oidc-token-proxy container images in GitHub Container Registry.

NOTE I decided against trying the hardware key approach. I have a Google Titan key and only Yubikeys are well-tested by go-piv

`cosign sign`

go install github.com/sigstore/cosign/cmd/cosign@latest

Initially, I followed the Quick Start but, after chatting on the team’s Slack channel, I was referred to Luke Hinds’ Sigstore The Hard Way and specifically Sign Container which importantly uses COSIGN_EXPERIMENTAL=1 and GitHub’s secret squirrel OIDC federation functionality.

All I needed was:

COSIGN_EXPERIMENTAL=1 \
cosign sign \
  ghcr.io/dazwilkin/gcp-oidc-token-proxy:3e448c5ffe4c28093ea7cae8d6693abeec1ce87c

This command uses some flag default values and is equivalent to:

COSIGN_EXPERIMENTAL=1 \
cosign sign \
  -oidc-issuer https://oauth2.sigstore.dev/auth \
  -fulcio-url https://fulcio.sigstore.dev \
  -rekor-url https://rekor.sigstore.dev \
  ghcr.io/dazwilkin/gcp-oidc-token-proxy:3e448c5ffe4c28093ea7cae8d6693abeec1ce87c

NOTE 3e44... is the image’s tag and represents the git rev-parse HEAD of the repo when the image was built from it. As it occur below, the image’s SHA-256 is c26f799823766c54155653cb709df811b7ae73a68ba504a4da2e57153bcf4c40

The oidc-issuer is evident in the output below as it is the endpoint through which the user authenticates with Sigstore.

The fulcio-url points to an instance of Fulcio which is a root CA and issues certs that can be used to sign software artifacts.

The command outputs:

Generating ephemeral keys...
Retrieving signed certificate...
Your browser will now be opened to:
https://oauth2.sigstore.dev/auth/auth?access_type=online&client_id=sigstore...

Which prompts:

And, when you do:

Successfully verified SCT...
tlog entry created with index: 756105
Pushing signature to: ghcr.io/dazwilkin/gcp-oidc-token-proxy:sha256-c26f799823766c54155653cb709df811b7ae73a68ba504a4da2e57153bcf4c40.sig

NOTE The signature follows the image reference but its tag is sha256-${HASH}.sig where HASH is the underlying image’s SHA-256 hash.

NOTE Initially, I had errors when cosign tried to push to GHCR (ghcr.io):
POST https://ghcr.io/v2/dazwilkin/gcp-oidc-token-proxy/blobs/uploads/: DENIED: unauthenticated: User cannot be authenticated with the token provided.
. Experience has taught me that this is because I’m running Docker as a(n Ubuntu) Snap and the Snap stores the Docker config in the non-default location. cosign expects Docker to be installed and, more specifically, expects ${HOME}/.docker/config.json to contain an auths key for (in this case) ghcr.io as it uses the token value to authenticate. I copied (but could probably have ln -s) ${HOME}/snap/docker/current/.docker/config.json to the default location to resolve the issue.

This is still slightly confusing to me but, a new artifact (a Sigstore signature artifact) has indeed been POSTed to the repositry (in this case ghcr.io/dazwilkin/gcp-oidc-token-proxy)

GitHub Container Registry

We can query the GitHub REST API for Container Registry resources.

I’ve skipped a step below and, after listing all the versions in the repository ghcr.io/dazwilkin/gcp-oidc-token-proxy, I’m GET‘ting the version (864123) that happens to correspond to the signature uploaded by cosign sign in the previous step:

curl \
--header "Authorization: Bearer ${TOKEN}" \
--header "Accept: application/vnd.github.v3+json" \
https://api.github.com/users/dazwilkin/packages/container/gcp-oidc-token-proxy/versions/8641325

Yields:

{
  "id": 8641325,
  "name": "sha256:2991a4bf27837d9d4d198ea999d8ad6d1c40d3635d0941baa27973187cc463ec",
  "url": "https://api.github.com/users/DazWilkin/packages/container/gcp-oidc-token-proxy/versions/8641325",
  "package_html_url": "https://github.com/users/DazWilkin/packages/container/package/gcp-oidc-token-proxy",
  "created_at": "2021-10-08T17:08:57Z",
  "updated_at": "2021-10-08T17:08:57Z",
  "html_url": "https://github.com/users/DazWilkin/packages/container/gcp-oidc-token-proxy/8641325",
  "metadata": {
    "package_type": "container",
    "container": {
      "tags": [
        "sha256-c26f799823766c54155653cb709df811b7ae73a68ba504a4da2e57153bcf4c40.sig"
      ]
    }
  }
}

I’m unable to determine how to query container manifests using GitHub REST API but here’s a copy from the web page. I don’t understand what this comprises. layer[0].digest is the SHA-256 hash of the underlying container image. But I don’t understand (yet) what layer[1].digest represents. Presumably it’s some representation of the signature.

{
  "digest": "sha256:2991a4bf27837d9d4d198ea999d8ad6d1c40d3635d0941baa27973187cc463ec",
  "mediaType": "application/vnd.oci.image.manifest.v1+json",
  "size": 4643,
  "config": {
    "digest": "sha256:7fd7ade75e85747f790ac7fcb5643e77ea27803789edc339a840ae98464fc9eb",
    "mediaType": "application/vnd.oci.image.config.v1+json",
    "size": 233
  },
  "layers": [
    {
      "digest": "sha256:6da5b4f75eeffd0d594c207bb4244da436c91582043eda4127eaa5ad13b349b0",
      "mediaType": "application/octet-stream",
      "size": 254
    },
    {
      "digest": "sha256:7fd7ade75e85747f790ac7fcb5643e77ea27803789edc339a840ae98464fc9eb",
      "mediaType": "application/octet-stream",
      "size": 233
    }
  ]
}

`cosign verify`

cosign verify verifies the signature of a signed (!) artifact.

This is where rekor comes back into play. Rekor is a transparency log (built on Google’s Trillian) that includes entries (hashes) representing our signed artifacts. Since the log is immutable, deleting signatures from e.g. GHCR should have no effect on Rekor’s log entries and, more importantly, we can query Rekor to get information about our artifact.

NOTE I give cosign verify the original container image. Presumably it looks for an articate with a specific tag (sha256-${HASH}.sig) and uses this for the verification. I guess if I delete that artifact, cosign verify should then fail?

COSIGN_EXPERIMENTAL=1 \
cosign verify  \
  ghcr.io/dazwilkin/gcp-oidc-token-proxy:3e448c5ffe4c28093ea7cae8d6693abeec1ce87c

Yielding:

No TUF root installed, using embedded CA certificate.

No TUF root installed, using embedded rekor key

Verification for ghcr.io/dazwilkin/gcp-oidc-token-proxy:3e448c5ffe4c28093ea7cae8d6693abeec1ce87c --
The following checks were performed on each of these signatures:
  - The cosign claims were validated
  - Existence of the claims in the transparency log was verified offline
  - Any certificates were verified against the Fulcio roots.

And:

[
    {"critical": {
        "identity": {
            "docker-reference":"ghcr.io/dazwilkin/gcp-oidc-token-proxy"},
            "image":{
                "docker-manifest-digest": "sha256:c26f799823766c54155653cb709df811b7ae73a68ba504a4da2e57153bcf4c40"
            },
            "type":"cosign container image signature"
        },
        "optional":{
            "Bundle": {
                "SignedEntryTimestamp":"...",
                "Payload": {
                    "body":"ey...",
                    "integratedTime":1633712934,
                    "logIndex":756105,
                    "logID":"c0d23d6ad406973f9559f3ba2d1ca01f84147d8ffc5b8445c224f98b9591801d"
                }
            },
            "Subject":"daz.wilkin@gmail.com"
        }
    }
]

NOTE The logIndex and logID values

So, IIUC, anyone should be able to run cosign verify on an artifact and get the above data.

I decoded the JWT value of optional.Bundled.Payload.body with jwt.io yielding:

{
  "apiVersion": "0.0.1",
  "kind": "rekord",
  "spec": {
    "data": {
      "hash": {
        "algorithm": "sha256",
        "value": "6da5b4f75eeffd0d594c207bb4244da436c91582043eda4127eaa5ad13b349b0"
      }
    },
    "signature": {
      "content": "MEQCIH8VA3Yul38uPvlr7L3ytwjMNqiNqDqJdk9I4pQjorRbAiBrIa7mb6YFUgW+njijF00beyZAgUCEvPuC612/zn1rjA==",
      "format": "x509",
      "publicKey": {
        "content": "LS0tLS1C..."
      }
    }
  }
}

And, after installing rekor-cli:

go install github.com/sigstore/rekor/cmd/rekor-cli@latest

We can:

rekor-cli get \
--format=json \
--log-index=756105 \
| jq -r .

NOTE There’s that logIndex value

Yields:

{
  "Attestation": "",
  "AttestationType": "",
  "Body": {
    "RekordObj": {
      "data": {
        "hash": {
          "algorithm": "sha256",
          "value": "6da5b4f75eeffd0d594c207bb4244da436c91582043eda4127eaa5ad13b349b0"
        }
      },
      "signature": {
        "content": "MEQCIH8VA3Yul38uPvlr7L3ytwjMNqiNqDqJdk9I4pQjorRbAiBrIa7mb6YFUgW+njijF00beyZAgUCEvPuC612/zn1rjA==",
        "format": "x509",
        "publicKey": {
          "content": "LS0tLS1C..."
        }
      }
    }
  },
  "LogIndex": 756105,
  "IntegratedTime": 1633712934,
  "UUID": "f9bfb33439ed512126658a1354999b5be662da3e3d69c732fc72e09de492ce60",
  "LogID": "c0d23d6ad406973f9559f3ba2d1ca01f84147d8ffc5b8445c224f98b9591801d"
}

NOTE The UUID

NOTE I’ve elided the publicKey for display purposes. I feel as though I should be able to do something useful with this key with regards my artifact but I’m unclear how|what.

And:

rekor-cli get \
--format=json \
--uuid=f9bfb33439ed512126658a1354999b5be662da3e3d69c732fc72e09de492ce60 \
| jq -r .

Yields:

{
  "RekordObj": {
    "data": {
      "hash": {
        "algorithm": "sha256",
        "value": "6da5b4f75eeffd0d594c207bb4244da436c91582043eda4127eaa5ad13b349b0"
      }
    },
    "signature": {
      "content": "MEQCIH8VA3Yul38uPvlr7L3ytwjMNqiNqDqJdk9I4pQjorRbAiBrIa7mb6YFUgW+njijF00beyZAgUCEvPuC612/zn1rjA==",
      "format": "x509",
      "publicKey": {
        "content": "LS0tLS1C..."
      }
    }
  }
}

Rekor exposes a REST API, we can perform the above command using it:

curl \
--header "Accept: application/json"  \
--silent \
https://rekor.sigstore.dev/api/v1/log/entries/f9bfb33439ed512126658a1354999b5be662da3e3d69c732fc72e09de492ce60 \
| jq .

Yields:

{
  "f9bfb33439ed512126658a1354999b5be662da3e3d69c732fc72e09de492ce60": {
    "attestation": {},
    "body": "eyJhcGlWZXJzaW9uIjoiMC4wLjEiLCJraW5kIjoicmVrb3JkIiwic3BlYyI6eyJkYXRhIjp7Imhhc2giOnsiYWxnb3JpdGhtIjoic2hhMjU2IiwidmFsdWUiOiI2ZGE1YjRmNzVlZWZmZDBkNTk0YzIwN2JiNDI0NGRhNDM2YzkxNTgyMDQzZWRhNDEyN2VhYTVhZDEzYjM0OWIwIn19LCJzaWduYXR1cmUiOnsiY29udGVudCI6Ik1FUUNJSDhWQTNZdWwzOHVQdmxyN0wzeXR3ak1OcWlOcURxSmRrOUk0cFFqb3JSYkFpQnJJYTdtYjZZRlVnVytuamlqRjAwYmV5WkFnVUNFdlB1QzYxMi96bjFyakE9PSIsImZvcm1hdCI6Ing1MDkiLCJwdWJsaWNLZXkiOnsiY29udGVudCI6IkxTMHRMUzFDUlVkSlRpQkRSVkpVU1VaSlEwRlVSUzB0TFMwdENrMUpTVU5rYWtORFFXWTJaMEYzU1VKQlowbFVWM1p6Vm1sME0xWnphSHBqTTBWd2JqSnlUREpqUlRsQ1NYcEJTMEpuWjNGb2EycFBVRkZSUkVGNlFYRUtUVkpWZDBWM1dVUldVVkZMUlhkNGVtRlhaSHBrUnpsNVdsTTFhMXBZV1hoRlZFRlFRbWRPVmtKQlRWUkRTRTV3V2pOT01HSXpTbXhOUWpSWVJGUkplQXBOVkVGM1QwUkZNMDFFWnpGTk1XOVlSRlJKZUUxVVFYZFBSRVV6VFdwbk1VMXNiM2RCUkVKYVRVSk5SMEo1Y1VkVFRUUTVRV2RGUjBORGNVZFRUVFE1Q2tGM1JVaEJNRWxCUWtKSlozTkhTbEpOV1N0UGVraFRlR1JsTDBOa1IwTjRaMnRuYldwaU9EWkJOMmsyTWxCRk1WRklUVUpaVkZsSFZrVnNVV1JyV2pRS016SlFXVFJLY1hOMVVGUkliM2h6TDNsMWEwUlRWMWg1UzBsNlJsQjJVMnBuWjBWeVRVbEpRa3A2UVU5Q1owNVdTRkU0UWtGbU9FVkNRVTFEUWpSQmR3cEZkMWxFVmxJd2JFSkJkM2REWjFsSlMzZFpRa0pSVlVoQmQwMTNSRUZaUkZaU01GUkJVVWd2UWtGSmQwRkVRV1JDWjA1V1NGRTBSVVpuVVZWWVJXaEtDbWxMZEhoM1RFdGlNbUpuZFVKblZ6RkNPUzlIS3pkVmQwaDNXVVJXVWpCcVFrSm5kMFp2UVZWNVRWVmtRVVZIWVVwRGEzbFZVMVJ5UkdFMVN6ZFZiMGNLTUN0M2QyZFpNRWREUTNOSFFWRlZSa0ozUlVKQ1NVZEJUVWcwZDJaQldVbExkMWxDUWxGVlNFMUJTMGRqUjJnd1pFaEJOa3g1T1hkamJXd3lXVmhTYkFwWk1rVjBXVEk1ZFdSSFZuVmtRekF5VFVST2JWcFVaR3hPZVRCM1RVUkJkMHhVU1hsTmFtTjBXVzFaTTA1VE1XMU9SMWt4V2xSbmQxcEVTVFZPVkZGMUNtTXpVblpqYlVadVdsTTFibUl5T1c1aVIxWm9ZMGRzZWt4dFRuWmlVemxxV1ZSTk1sbFVSbXhQVkZsNVRrUkthVTlYV21wWmFrVXdUbWs1YWxsVE5Xb0tZMjVSZDBsbldVUldVakJTUVZGSUwwSkNaM2RHYjBWVldrZEdOa3h1WkhCaVIzUndZbXRDYm1KWFJuQmlRelZxWWpJd2QwTm5XVWxMYjFwSmVtb3dSUXBCZDAxRVdtZEJkMWwzU1hWaE5VWnhiM0IzZDNWUU5tTlBaVFpPVjNnd1dEbHZTM3BEYVhaemJrRjNNRmQwUW5OME4xbzFUREpOVW5abVVXcENUR3RXQ2tVclZYcHhWbXQwTlVGSmVFRktjbE5zTDFkRGFWUlFVRTFsTWtwcFQzZHJhelJ6TVhVeFNGcFFkRlpLUms1NlVrWXlLMUJsWlZWbkx6TllVWGx2V0VnS2Qza3JUemROUTJaNlpHa3ZVbEU5UFFvdExTMHRMVVZPUkNCRFJWSlVTVVpKUTBGVVJTMHRMUzB0Q2c9PSJ9fX19",
    "integratedTime": 1633712934,
    "logID": "c0d23d6ad406973f9559f3ba2d1ca01f84147d8ffc5b8445c224f98b9591801d",
    "logIndex": 756105,
    "verification": {
      "inclusionProof": {
        "hashes": [
          "006812c3095f5d083c9bcc8c573e9db9584e91dfa908bbef19167e30cffd9cdf",
          "550cd509eaa97e96d9cf44703632a673ff175d5e6b98870d847a4c961da93ac9",
          "10db3794f9f019ba1fc2b6560f7d993e2bd02a3e4b54d67be8fe644e217a189a",
          "65a805fc55816f86ed29e8e139807d1960510a399afda63a4c1d8857c7d358c6",
          "209c0ce6aeccadda61032cf6cbe912fa965cf4a89cfbe9d966ed22033b123308",
          "a8a2efe94a2ae7c6540522698386047114b1fe19c12f91c1c251250f3015c56d",
          "a2636f8b39c0c818328dd2493f59e37856fac4debc097dd0b4f13edb838659eb",
          "b8daf922c57bebca0f54949b214c357d370e8e12d83c83dcb2bc0405738ce59e",
          "9d277d4bf3ac931e150fdf73f76e298938aeb126bdfb86fa133d824c8853f735",
          "481d804401569f892f2b60a5f229317f5b9ed9c4ae0d57254a50619ae5762616",
          "2889aef73a822720fe7e24777693eb068aa96ea47dd6190c3b5a083a676bf919",
          "e84e4a360c277f04ef77f4e2df00f7cc8ae8d858758e51273f1271bc05a65704",
          "bdca084d1848968ab9224eff1467a95f2d93f88da455a60d37fb377da061dcc8",
          "12e6affe06f6187293ca7b8cceefcef938243e3657a547b9e6400d68db2b5f3b",
          "80b19ade3d5598a8b1c944783a53feaad0384589dc75baeac351142c4c620b14"
        ],
        "logIndex": 756105,
        "rootHash": "f646f5f028421de1337057a283986e48110e4e0cfeb118de53e5fc9d401d6489",
        "treeSize": 756642
      },
      "signedEntryTimestamp": "MEYCIQCSN+jpJL2txESSEKIvCU7l/10cAIdyqqtJ0rNFF2y9swIhAIoudmx6JAEdw+SO7MBF3YR+WuJO2dCTkz2MZ1h/3I5U"
    }
  }
}

I’ll update this post as I learn more.

A new Podcast with a bunch of topics that interest me. ↩︎