Maintaining Container Images

November 28, 2022 - 6 minutes read - 1111 words

As I contemplate moving my “thing” into production, I’m anticipating aspects of the application that need maintenance and how this can be automated.

I’d been negligent in the maintenance of some of my container images.

I’m using mostly Go and some Rust as the basis of static(ally-compiled) binaries that run in these containers but not every container has a base image of scratch. scratch is the only base image that doesn’t change and thus the only base image that doesn’t require that container images buit FROM it, be maintained.

I suspect other people are making this mistake too. If your binary hasn’t changed, you don’t need to rebuild the container. But, I had a container I’d not rebuilt in 11 months and it had a bunch of vulnerabilities because it was built on the then (!) current version of Debian 10.

Initially, I considered scripting a tool to parse my container images using podman history and, based on the base image, determining whether to rebuild the container. But, this is both too simplistic and too tedious.

I wondered whether GitHub includes a mechanism to monitor for container images updates using Dependabot. I find Dependabot to be useful but complicated. I’m still unclear why some functionality is configurable through GitHub and some requires use of dependabot.yml. It’s also difficult for me to understand its scope. I Googled around but didn’t find anything immediately applicable.

I’m using Google Cloud as a deployment target for my app. Services such as Cloud Run require that container images be hosted by Google Container Registry or its replacement Artifact Registry. A companion service for both these registry implementations is Container Analysis and this includes vulnerability scanning of container images.

NOTE Be aware that Container Analysis vulnerability scanning is expensive (see Container Analysis pricing). Yesterday, I spent $5.98 to scan 23 images (including updates) at $0.26/image. I’d initially planned to run this service every day on each deployment of the service but I won’t be doing that because the deployment creates a new Artifact Registry each day and scans the images each day on push.

Here’s an example of one of my container images with an outdated (CVE-full) base image:

Vulnerabilities

I began trying to parse the vulnerabilities using gcloud:

gcloud artifacts docker images describe ${IMAGE} \
--show-package-vulnerability \
--format=json \
| jq -r ".package_vulnerability_summary.vulnerabilities.HIGH[]?.vulnerability.shortDescription"

Yields:

CVE-2022-23218
CVE-2019-25013
CVE-2021-33574
CVE-2021-3999
CVE-2022-23219

Each of these may be looked up e.g. NIST: CVE-2022-23218 although the Google API includes a shortDescription that references Debian’s Security Bug tracker.

What I really wanted was: for every container image version (or manifest), to grab the CVEs and then invert the results by CVE. For each CVE, I want to know which of my repo’s container images are affected.

Scripting the solution in Bash felt problematic (associative arrays comprising arrays) and so I decided to write a solution in Go.

Using Google’s Artifact Registry (API) Client Library, the following code builds a slice of image URIs. These are of the form {region}-docker.pkg.dev/{project}/{repo}/{image}@sha256:{hash}

ctx := context.Background()

artifactregistryService, err := artifactregistry.NewService(ctx)
if err != nil {
	log.Fatal(err)
}

parent := fmt.Sprintf("projects/%s/locations/%s/repositories/%s", project, location, repository)

// Define the request for the first page
rqst := artifactregistryService.Projects.Locations.Repositories.DockerImages.List(parent)

imageURIs := []string{}
for {
    // Invoke it
	resp, err := rqst.Do()
	if err != nil {
		log.Fatal(err)
	}

    // Append the image's URI
	for _, image := range resp.DockerImages {
		imageURIs = append(imageURIs, image.Uri)
	}

    // Break if no more pages
	pageToken := resp.NextPageToken
	if pageToken == "" {
		break
	}
    // Otherwise revise the request to get the next page
	rqst = rqst.PageToken(pageToken)
}

Now we need to use Google’s beta (!) Container Analysis (Cloud) Client Library to retrieve so-called Occurrence’s of CVEs detected by the vulnerability scanner.

The code uses the projects.ocurrences.list method

client, err := containeranalysis.NewClient(ctx)
if err != nil {
	log.Fatal(err)
}
defer client.Close()

parent = fmt.Sprintf("projects/%s", project)

// Maps CVE name to an array of images that contain the CVE
imagesByCVE := map[string][]string{}

for _, imageName := range imageURIs {
    // Must be prefixed with https://
	resourceURL := fmt.Sprintf("https://%s", imageName)

    // Both values must be quoted
    filter := fmt.Sprintf("resourceUrl=%q kind=%q",
        resourceURL,
        "VULNERABILITY",
    )

	rqst := &grafeaspb.ListOccurrencesRequest{
		Parent: parent,
		Filter: filter,
	}

	occurrences := []*grafeaspb.Occurrence{}

    // Uses Google's iterator provided with Cloud Client libraries
	it := client.GetGrafeasClient().ListOccurrences(ctx, rqst)
	for {
		occurrence, err := it.Next()
		if err == iterator.Done {
			break
		}
		if err != nil {
			log.Fatal(err)
		}

		// Additional filtering of the results:
        // Everything other than low severity
        // Everything that's fixable
		vulnerability := occurrence.GetVulnerability()
		if vulnerability.GetSeverity() != grafeaspb.Severity_LOW && vulnerability.FixAvailable {
			occurrences = append(occurrences, occurrence)
		}
	}

    // Retain the `shortDescription` which is the full CVE name
    // Can be subsequently combined with CVE site prefix
	for _, occurrence := range occurrences {
		shortDescription := occurrence.GetVulnerability().GetShortDescription()
		if _, ok := imagesByCVE[shortDescription]; !ok {
			imagesByCVE[shortDescription] = []string{}
		}

		imagesByCVE[shortDescription] = append(imagesByCVE[shortDescription], imageName)
	}
}

Lastly, we want to output the results:

tracker := "https://security-tracker.debian.org/tracker"
total := 0
for cve, images := range imagesByCVE {
	total += len(images)
	fmt.Printf("CVE: %s/%s (%d)\n%s\n",
        tracker,
        cve,
        len(images),
        strings.Join(images, "\n"),
    )
}
fmt.Printf("Total: %d\n", total)

Or, by number of occurrences of the CVE. This approach allows us to focus on the CVE that occurs in the most images first:

tracker := "https://security-tracker.debian.org/tracker"
total := 0

cves := make([]string, 0, len(imagesByCVE))
for cve := range imagesByCVE {
	cves = append(cves, cve)
}
sort.SliceStable(cves, func(i, j int) bool {
	return len(imagesByCVE[cves[i]]) > len(imagesByCVE[cves[j]])
})

for _, cve := range cves {
	total += len(imagesByCVE[cve])
	fmt.Printf("\nCVE: %s/%s (%d)\n%s\n",
        tracker,
        cve,
        len(imagesByCVE[cve]),
        strings.Join(imagesByCVE[cve], "\n"),
    )
}
fmt.Printf("Total: %d\n", total)

Running the code with the single image shown above with the vulnerabilities returns:

Images: 1

CVE: https://security-tracker.debian.org/tracker/CVE-2022-23218 (1)
{region}-docker.pkg.dev/{project}/{repo}/example@sha256:{hash}

CVE: https://security-tracker.debian.org/tracker/CVE-2021-33574 (1)
{region}-docker.pkg.dev/{project}/{repo}/example@sha256:{hash}

CVE: https://security-tracker.debian.org/tracker/CVE-2022-23219 (1)
{region}-docker.pkg.dev/{project}/{repo}/example@sha256:{hash}

CVE: https://security-tracker.debian.org/tracker/CVE-2021-3326 (1)
{region}-docker.pkg.dev/{project}/{repo}/example@sha256:{hash}

CVE: https://security-tracker.debian.org/tracker/CVE-2020-6096 (1)
{region}-docker.pkg.dev/{project}/{repo}/example@sha256:{hash}

CVE: https://security-tracker.debian.org/tracker/CVE-2021-35942 (1)
{region}-docker.pkg.dev/{project}/{repo}/example@sha256:{hash}

CVE: https://security-tracker.debian.org/tracker/CVE-2019-25013 (1)
{region}-docker.pkg.dev/{project}/{repo}/example@sha256:{hash}

CVE: https://security-tracker.debian.org/tracker/CVE-2021-3999 (1)
{region}-docker.pkg.dev/{project}/{repo}/example@sha256:{hash}

CVE: https://security-tracker.debian.org/tracker/CVE-2016-10228 (1)
{region}-docker.pkg.dev/{project}/{repo}/example@sha256:{hash}

Total: 9

In my case, I had almost 20 images in the repo and so there were many more results than is shown here. I was able to start with the CVEs that occurred in multiple images.

The known solution in this case is to bump the base image from Debian 10 (“buster”) to 11 (“bullseye”).

I’m going to push the updated image and deleted the current one (to reset the vulnerabilities):

Vulnerabilities

And:

# High
gcloud artifacts docker images describe ${IMAGE} \
--show-package-vulnerability \
--format=json \
| jq -r ".package_vulnerability_summary.vulnerabilities.HIGH[]?.vulnerability.shortDescription"
# No results

# Medium
gcloud artifacts docker images describe ${IMAGE} \
--show-package-vulnerability \
--format=json \
| jq -r ".package_vulnerability_summary.vulnerabilities.MEDIUM[]?.vulnerability.shortDescription"
CVE-2022-2097

The Go code produces zero results because, even though there’s a MEDIUM severity issue, the issue doesn’t have a fix available (FixAvailable is false).