Update Dockerfile + ci-images flow to build multi arch images (conda#…

…11560)

* Update Dockerfile + ci-images flow to build multi arch images

This is done to have linux/aarch64 conda-ci images, which perform better
on Apple M1 machines compared to the linux/amd64 images.

The other images (linux/ppc64le, linux/s390x) we basically get for free
and prove that the whole tooling is consistently available on all 4 linux
platforms.

Once the multi-arch image is published it can be also used during the ci
flow to e.g. run a short smoke test on all the non-native platforms via
emulation.

* Remove pre-commit from test requirements

* Add multi-platform comment

* Improve concurrency handling

* Ignore Dockerfile changes in main ci.yml workflow

* Fix arch

* Also ignore .github/ci-images.yml

* revert ci.yml changes

* Re-add pre-commit, don't build the s390x image

.github/workflows/ci-images.yml

@@ -24,6 +24,15 @@ on:
   # https://docs.github.com/en/developers/webhooks-and-events/webhooks/webhook-events-and-payloads#workflow_dispatch
   workflow_dispatch:
 
+concurrency:
+  # Concurrency group that uses the workflow name and PR number if available
+  # or commit SHA as a fallback. If a new build is triggered under that
+  # concurrency group while a previous build is running it will be canceled.
+  # Repeated pushes to a PR will cancel all previous builds, while multiple
+  # merges to main will not cancel.
+  group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.sha }}
+  cancel-in-progress: true
+
 env:
   REGISTRY: ghcr.io
   IMAGE_NAME: ${{ github.repository }}-ci
@@ -48,6 +57,18 @@ jobs:
           username: ${{ github.actor }}
           password: ${{ secrets.GITHUB_TOKEN }}
 
+      - name: Set up QEMU
+        uses: docker/setup-qemu-action@v2
+        with:
+          platforms: linux/amd64,linux/arm64,linux/ppc64le
+
+      - name: Set up Docker Buildx
+        id: buildx
+        uses: docker/setup-buildx-action@v2
+        with:
+          version: latest
+          driver-opts: network=host
+
       - name: Extract metadata (tags, labels) for Docker
         id: meta
         uses: docker/metadata-action@v3
@@ -61,7 +82,9 @@ jobs:
         uses: docker/build-push-action@v2
         with:
           file: ./Dockerfile
+          builder: ${{ steps.buildx.outputs.name }}
           build-args: python_version=${{ matrix.python }}
           labels: ${{ steps.meta.outputs.labels }}
           tags: ${{ steps.meta.outputs.tags }}
+          platforms: linux/amd64,linux/arm64,linux/ppc64le
           push: ${{ github.ref == 'refs/heads/master' }}

Dockerfile

@@ -1,17 +1,47 @@
-FROM --platform=linux/amd64 debian:buster-slim AS buildbase
+# This a Dockerfile with multi-platform support, build (pull/run) it via:
+# * Setup qemu for non-native platforms (to be done once):
+#       `docker run --privileged --rm tonistiigi/binfmt --install all`
+# * Setup buildx as default docker builder (to be done once):
+#       `docker buildx install`
+# * Build with without `--platform ...` arg (TARGETPLATFORM=BUILDPLATFORM):
+#       `docker build .`
+# * Build a single platform image:
+#       `docker build --platform linux/amd64 .`
+# * Build a multi-platform image:
+#       `docker build --platform linux/amd64,linux/arm64 .`
+# See also:
+# * https://docs.docker.com/buildx/working-with-buildx/
+# * https://www.docker.com/blog/faster-multi-platform-builds-dockerfile-cross-compilation-guide/
 
-ARG MINICONDA_URL=https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
+# built-in arg set via `docker build --platform $TARGETPLATFORM ...` or TARGETPLATFORM=BUILDPLATFORM
+FROM --platform=$TARGETPLATFORM debian:buster-slim AS buildbase
+
+# built-in arg set by `docker build --platform linux/$TARGETARCH ...`
+ARG TARGETARCH
+ARG CONDA_VERSION=latest
 
 WORKDIR /tmp
 
 RUN apt-get update && apt-get install -y wget
 
-RUN wget --quiet $MINICONDA_URL -O ~/miniconda.sh && \
+RUN if [ "${TARGETARCH}" = "amd64" ]; then \
+        MINICONDA_URL="https://repo.anaconda.com/miniconda/Miniconda3-${CONDA_VERSION}-Linux-x86_64.sh"; \
+    elif [ "${TARGETARCH}" = "s390x" ]; then \
+        MINICONDA_URL="https://repo.anaconda.com/miniconda/Miniconda3-${CONDA_VERSION}-Linux-s390x.sh"; \
+    elif [ "${TARGETARCH}" = "arm64" ]; then \
+        MINICONDA_URL="https://repo.anaconda.com/miniconda/Miniconda3-${CONDA_VERSION}-Linux-aarch64.sh"; \
+    elif [ "${TARGETARCH}" = "ppc64le" ]; then \
+        MINICONDA_URL="https://repo.anaconda.com/miniconda/Miniconda3-${CONDA_VERSION}-Linux-ppc64le.sh"; \
+    else \
+        echo "Not supported target architecture: ${TARGETARCH}"; \
+        exit 1; \
+    fi && \
+    wget --quiet $MINICONDA_URL -O ~/miniconda.sh && \
     /bin/bash ~/miniconda.sh -b -p /opt/conda && \
     rm ~/miniconda.sh && \
     /opt/conda/bin/conda clean --all --yes
 
-FROM --platform=linux/amd64 debian:buster-slim
+FROM --platform=$TARGETPLATFORM debian:buster-slim
 
 ENV LANG=C.UTF-8 LC_ALL=C.UTF-8
 ENV PATH /opt/conda/bin:$PATH