Container Image Promoter

The Container Image Promoter (aka "cip") promotes images from one Docker Registry (src registry) to another (dest registry), by reading a Manifest file (in YAML). The Manifest lists Docker images, and all such images are considered "blessed" and will be copied from src to dest.

Example Manifest for images:

registries:
- name: gcr.io/myproject-staging-area # publicly readable, does not need a service account for access
  src: true # mark it as the source registry (required)
- name: gcr.io/myproject-production
  service-account: [email protected]
images:
- name: apple
  dmap:
    "sha256:e8ca4f9ff069d6a35f444832097e6650f6594b3ec0de129109d53a1b760884e9": ["1.1", "latest"]
- name: banana
  dmap:
    "sha256:c3d310f4741b3642497da8826e0986db5e02afc9777a2b8e668c8e41034128c1": ["1.0"]
- name: cherry
  dmap:
    "sha256:ec22e8de4b8d40252518147adfb76877cb5e1fa10293e52db26a9623c6a4e92b": ["1.0"]
    "sha256:06fdf10aae2eeeac5a82c213e4693f82ab05b3b09b820fce95a7cac0bbdad534": ["1.2", "latest"]

Here, the Manifest cares about 3 images --- apple, banana, and cherry. The registries field lists all destination registries and also the source registry where the images should be promoted from. To earmark the source registry, it has src: true as a property. In the Example, the promoter will scan gcr.io/myproject-staging-area and promote the images found under images to gcr.io/myproject-production.

The source registry will always be read-only for the promoter. Because of this, it's OK to not provide a service-account field for it in registries. But in the event that you are trying to promote from one private registry to another, you would still provide a service-account for the staging registry.

Currently only Google Container Registry (GCR) is supported.

Thin Manifests

Thin manifests are a more secure form of promoter Manifests. They are just like regular Manifests, but instead of having an images: ... field, instead they have a imagesPath: ... field that points to a separate file containing the images: ... information. You can use these thin manifests by specifying the -thin-manifest-dir=<target directory> flag, which forces all promoter manifests to be defined as thin manifests within the target directory.

You would use these manifests to separate credential names and source/destination registry information from the images information. For example, using thin manifests would allow you to define images YAMLs in a folder separate from your thin manifests, allowing you to lock down the thin manifests with highly restrictive permissions.

Install

1. Install bazel.
2. Run the steps below:

go get sigs.k8s.io/k8s-container-image-promoter
cd $GOPATH/src/sigs.k8s.io/k8s-container-image-promoter
make build

Running the Promoter

The promoter relies on calls to gcloud container images ... to realize the intent of the Manifest. It also tries to run the command as the account in service-account. The credentials for this service account must already be set up in the environment prior to running the promoter.

Given the Example Manifest as above, you can run the promoter with:

bazel run -- cip -h -verbosity=3 -manifest=path/to/manifest.yaml

Alternatively, you can run the binary directly by examining the bazel output from running make build, and then invoking it with the correct path under ./bazel-bin. For example, if you are on a Linux machine, running make build will output a binary at ./bazel-bin/linux_amd64_stripped/cip.

What it does

The promoter's behaviour can be described in terms of mathematical sets (as in Venn diagrams). Suppose S is the set of images in the source registry, D is the set of all images in the destination registry and M is the set of images to be promoted (these are defined in the promoter manifest). Then:

• M ∩ D = images which do not need promoting since they are already present in the destination registry
• (M ∩ S) \ D = images that are copied

The above statements are true for each destination registry.

The promoter also prints warnings about images that cannot be promoted:

• M \ (S ∪ D) = images that cannot be found

Server-side operations

During the promotion process, all data resides on the server (currently, Google Container Registry for images). That is, no images get pulled and pushed back up. There are two reasons why it does things entirely server-side:

1. Performance. Images can be gigabytes in size and it would take forever to pull/push images in their entirety for every promotion.
2. Digest preservation. Pulling/pushing the images can change their digest (sha256sum) because layers might get gzipped differently when they are pushed back up. Doing things entirely server-side preserves the digest, which is important for declaratively recording the images by their digest in the promoter manifest.

Maintenance

Linting

We use golangci-lint; please install it and run make lint to check for linting errors. There should be 0 linting errors; if any should be ignored, add a line ignoring the error with a //nolint[:linter1,linter2,...] directitve. Grep for nolint in this repo for examples.

Testing

Run make test; this will invoke a bazel rule to run all unit tests.

Every critical piece has a unit test --- unit tests complete nearly instantly, so you should always add unit tests where possible, and also run them before submitting a new PR. There is an open issue to make linting and unit/e2e tests part of a Prow job, to guard against human error in this area.

Faking http calls and shell processes

As the promoter uses a combination of network API calls and shell-instantiated processes, we have to fake them for the unit tests. To make this happen, these mechanisms all use a stream.Producer interface. The real-world code uses either the http or subprocess implementations of this interface to create streams of data (JSON or not) which we can interpret and use.

For tests, the fake implementation is used instead, which predefines how that stream will behave, for the purposes of each unit test. A good example of this is the TestReadAllRegistries test.

Updating Prow Jobs

Currently there are 3 Prow jobs that use the promoter Docker images. All of these jobs watch the promoter manifests that live in the k8s.io repo. They are:

1. presubmit job (there is no testgrid entry for this job)
2. postsubmit job (grep for post-k8sio-cip)
3. daily job (grep for ci-k8sio-cip)

The postsubmit and daily jobs also have testgrid entries (postsubmit, daily).

Every time a PR lands against one of the promoter manifests in the kubernetes/k8.sio repo, the presubmit runs, followed by the postsubmit if the PR gets merged. The daily job (ci-k8sio-cip) runs every day as a sanity check to make sure that both the promoter configuration is correct in the Prow job, and that the registries have not been tampered with independent of the image promotion process (e.g., if an image gets promoted out-of-band, then the promoter will print a warning about it being present).

Releasing

We follow Semver for versioning. For each new release, create a new release on GitHub with:

• Update VERSION file to bump the semver version (e.g., 1.0.0)
• Create a new commit for the 1-liner change above with this command with git commit -m "cip 1.0.0"
• Create an annotated tag at this point with git tag -a "v1.0.0" -m "cip 1.0.0"
• Push this version to the master branch (requires write access)

We also have to publish the Docker images. Currently they are pushed up to the gcr.io/cip-demo-staging registry (the home will change to a more official place in the future). To publish them into gcr.io/cip-demo-staging, run

• make image-push (requires push access to gcr.io/cip-demo-staging)

Once the images are published, you should bump the image tags as they are referenced in the Prow Jobs by making a PR against the test-infra repo.