Enable pushing early image cache to ghcr.io registry (apache#26119)

In order to react quicker to changes in setup.py, setup.cfg and Dockerfiles, we can push an early image cache to the ghcr.io cache. This build is done way before constraints are regenerated so it is using the "current" constraints at the moment the merge happens. This should be safe - if the image fails to build (we ignore that) the cache will not be pushed, but if it will, then even if we merge a new commit quickly and the rest of the build is cancelled, the cache will be refreshed with new Dockerfile/setup.py/setup.cfg and the subsequent builds will run much faster.
parth166 · Sep 1, 2022 · 479ab54 · 479ab54
1 parent a1e7f94
commit 479ab54
Show file tree

Hide file tree

Showing 3 changed files with 85 additions and 17 deletions.
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
@@ -238,6 +238,74 @@ jobs:
           PR_LABELS: ${{ steps.source-run-info.outputs.pr-labels }}
           GITHUB_CONTEXT: ${{ toJson(github) }}
 
+  # Push early BuildX cache to GitHub Registry in Apache repository, This cache does not wait for all the
+  # tests to complete - it is run very early in the build process for "main" merges in order to refresh
+  # cache using the current constraints. This will speed up cache refresh in cases when setup.py
+  # changes or in case of Dockerfile changes. Failure in this step is not a problem (at most it will
+  # delay cache refresh. It does not attempt to upgrade to newer dependencies.
+  # We only push CI cache as PROD cache usually does not gain as much from fresh cache because
+  # it uses prepared airflow and provider packages that invalidate the cache anyway most of the time
+  push-early-buildx-cache-to-github-registry:
+    permissions:
+      packages: write
+    timeout-minutes: 50
+    name: "Push Early Image Cache"
+    runs-on: ${{ needs.build-info.outputs.runs-on }}
+    needs:
+      - build-info
+    strategy:
+      fail-fast: false
+      matrix:
+        platform: ["linux/amd64", "linux/arm64"]
+    env:
+      RUNS_ON: ${{ needs.build-info.outputs.runs-on }}
+      UPGRADE_TO_NEWER_DEPENDENCIES: false
+    continue-on-error: true
+    steps:
+      - name: Cleanup repo
+        run: docker run -v "${GITHUB_WORKSPACE}:/workspace" -u 0:0 bash -c "rm -rf /workspace/*"
+        if: needs.build-info.outputs.merge-run == 'true'
+      - name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
+        uses: actions/checkout@v3
+        with:
+          persist-credentials: false
+        if: needs.build-info.outputs.merge-run == 'true'
+      - name: "Setup python"
+        uses: actions/setup-python@v4
+        with:
+          python-version: ${{ needs.build-info.outputs.default-python-version }}
+          cache: 'pip'
+          cache-dependency-path: ./dev/breeze/setup*
+        if: needs.build-info.outputs.merge-run == 'true'
+      - run: ./scripts/ci/install_breeze.sh
+        if: needs.build-info.outputs.merge-run == 'true'
+      - name: "Free space"
+        run: breeze ci free-space
+        if: needs.build-info.outputs.merge-run == 'true'
+      - name: "Start ARM instance"
+        run: ./scripts/ci/images/ci_start_arm_instance_and_connect_to_docker.sh
+        if: matrix.platform == 'linux/arm64' && needs.build-info.outputs.merge-run == 'true'
+      - name: "Push CI cache ${{ matrix.platform }}"
+        run: >
+          breeze ci-image build
+          --builder airflow_cache
+          --prepare-buildx-cache
+          --run-in-parallel
+          --force-build
+          --platform ${{ matrix.platform }}
+        if: needs.build-info.outputs.merge-run == 'true'
+      - name: "Push CI latest image ${{ matrix.platform }}"
+        run: >
+          breeze ci-image build
+          --tag-as-latest --push --run-in-parallel --platform ${{ matrix.platform }}
+        if: matrix.platform == 'linux/amd64' && needs.build-info.outputs.merge-run == 'true'
+      - name: "Stop ARM instance"
+        run: ./scripts/ci/images/ci_stop_arm_instance.sh
+        if: always() && matrix.platform == 'linux/arm64' && needs.build-info.outputs.merge-run == 'true'
+      - name: "Fix ownership"
+        run: breeze ci fix-ownership
+        if: always() && needs.build-info.outputs.merge-run == 'true'
+
   build-ci-images:
     permissions:
       packages: write
@@ -1532,6 +1600,7 @@ ${{ hashFiles('.pre-commit-config.yaml') }}"
       - tests-mysql
       - tests-mssql
       - tests-postgres
+      - push-early-buildx-cache-to-github-registry
     env:
       RUNS_ON: ${{ needs.build-info.outputs.runs-on }}
     if: needs.build-info.outputs.upgrade-to-newer-dependencies != 'false'
@@ -1630,7 +1699,7 @@ ${{ hashFiles('.pre-commit-config.yaml') }}"
       - name: "Free space"
         run: breeze ci free-space
       - name: >
-          Pull CI image for PROD build
+          Pull CI image ${{ needs.build-info.outputs.default-python-version }} for PROD build
           ${{ env.PYTHON_MAJOR_MINOR_VERSION }}:${{ env.IMAGE_TAG_FOR_THE_BUILD }}"
         run: breeze ci-image pull --tag-as-latest
         env:

diff --git a/CI.rst b/CI.rst
@@ -528,6 +528,8 @@ This workflow is a regular workflow that performs all checks of Airflow code.
 +===========================+==============================================+=======+=======+======+
 | Build info                | Prints detailed information about the build  | Yes   | Yes   | Yes  |
 +---------------------------+----------------------------------------------+-------+-------+------+
+| Push early cache & images | Pushes early cache/images to GitHub Registry | -     | Yes   | Yes  |
++---------------------------+----------------------------------------------+-------+-------+------+
 | Test OpenAPI client gen   | Tests if OpenAPIClient continues to generate | Yes   | Yes   | Yes  |
 +---------------------------+----------------------------------------------+-------+-------+------+
 | UI tests                  | React UI tests for new Airflow UI            | Yes   | Yes   | Yes  |
@@ -554,7 +556,7 @@ This workflow is a regular workflow that performs all checks of Airflow code.
 +---------------------------+----------------------------------------------+-------+-------+------+
 | Constraints               | Upgrade constraints to latest ones (4)       | -     | Yes   | Yes  |
 +---------------------------+----------------------------------------------+-------+-------+------+
-| Push images               | Pushes latest images to GitHub Registry (4)  | -     | Yes   | Yes  |
+| Push cache & images       | Pushes cache/images to GitHub Registry (4)   | -     | Yes   | Yes  |
 +---------------------------+----------------------------------------------+-------+-------+------+
 
 
@@ -564,8 +566,8 @@ Comments:
  (2) The tests are run when the Trigger Tests job determine that important files change (this allows
      for example "no-code" changes to build much faster)
  (3) The jobs wait for CI images to be available.
- (4) PROD and CI images are pushed as "latest" to GitHub Container registry and constraints are upgraded
-     only if all tests are successful. The images are rebuilt in this step using constraints pushed
+ (4) PROD and CI cache & images are pushed as "latest" to GitHub Container registry and constraints are
+     upgrade only if all tests are successful. The images are rebuilt in this step using constraints pushed
      in the previous step.
 
 CodeQL scan

diff --git a/CI_DIAGRAMS.md b/CI_DIAGRAMS.md
@@ -38,7 +38,6 @@ sequenceDiagram
     Note over Tests: Skip Build<br>(Runs in 'Build Images')<br>CI Images
     Note over Tests: Skip Build<br>(Runs in 'Build Images')<br>PROD Images
     par
-        GitHub Registry ->> Build Images: Pull CI Images<br>[latest]
         Note over Build Images: Build CI Images<br>[COMMIT_SHA]<br>Use latest constraints<br>or upgrade if setup changed
     and
         Note over Tests: OpenAPI client gen
@@ -121,7 +120,6 @@ sequenceDiagram
     deactivate Build Images
     Note over Tests: Build info<br>Decide on tests<br>Decide on Matrix (selective)
     par
-        GitHub Registry ->> Tests: Pull CI Images<br>[latest]
         Note over Tests: Build CI Images<br>[COMMIT_SHA]<br>Use latest constraints<br>or upgrade if setup changed
     and
         Note over Tests: OpenAPI client gen
@@ -134,7 +132,6 @@ sequenceDiagram
     GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
     Note over Tests: Verify CI Image<br>[COMMIT_SHA]
     par
-        GitHub Registry ->> Tests: Pull PROD Images<br>[latest]
         Note over Tests: Build PROD Images<br>[COMMIT_SHA]
     and
         opt
@@ -192,20 +189,21 @@ sequenceDiagram
     activate Tests
     Note over Tests: Build info<br>All tests<br>Full matrix
     par
-        GitHub Registry ->> Tests: Pull CI Images<br>[latest]
         Note over Tests: Build CI Images<br>[COMMIT_SHA]<br>Always upgrade deps
     and
         Note over Tests: OpenAPI client gen
     and
         Note over Tests: Test UI
     and
         Note over Tests: Test examples<br>PROD image building
+    and
+        Note over Tests: Build CI Images<br>Use original constraints
+        Tests ->> GitHub Registry: Push CI Image Early cache + latest
     end
     Tests ->> GitHub Registry: Push CI Images<br>[COMMIT_SHA]
     GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
     Note over Tests: Verify CI Image<br>[COMMIT_SHA]
     par
-        GitHub Registry ->> Tests: Pull PROD Images<br>[latest]
         Note over Tests: Build PROD Images<br>[COMMIT_SHA]
     and
         opt
@@ -249,11 +247,9 @@ sequenceDiagram
         Tests ->> Airflow Repo: Push constraints if changed
     end
     opt In merge run?
-        GitHub Registry ->> Tests: Pull CI Image<br>[latest]
         Note over Tests: Build CI Images<br>[latest]<br>Use latest constraints
         Tests ->> GitHub Registry: Push CI Image<br>[latest]
-        GitHub Registry ->> Tests: Pull PROD Image<br>[latest]
-        Note over Tests: Build PROD Images<br>[latest]
+        Note over Tests: Build PROD Images<br>[latest]<br>Use latest constraints
         Tests ->> GitHub Registry: Push PROD Image<br>[latest]
     end
     Tests -->> Airflow Repo: Status update
@@ -280,6 +276,9 @@ sequenceDiagram
         Note over Tests: Test UI
     and
         Note over Tests: Test examples<br>PROD image building
+    and
+        Note over Tests: Build CI Images<br>Use original constraints
+        Tests ->> GitHub Registry: Push CI Image Early cache + latest
     end
     Tests ->> GitHub Registry: Push CI Images<br>[COMMIT_SHA]
     GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
@@ -326,12 +325,10 @@ sequenceDiagram
     end
     Note over Tests: Generate constraints
     Tests ->> Airflow Repo: Push constraints if changed
-    GitHub Registry ->> Tests: Pull CI Image<br>[latest]
     Note over Tests: Build CI Images<br>[latest]<br>Use latest constraints
-    Tests ->> GitHub Registry: Push CI Image<br>[latest]
-    GitHub Registry ->> Tests: Pull PROD Image<br>[latest]
-    Note over Tests: Build PROD Images<br>[latest]
-    Tests ->> GitHub Registry: Push PROD Image<br>[latest]
+    Tests ->> GitHub Registry: Push CI Image cache + latest
+    Note over Tests: Build PROD Images<br>[latest]<br>Use latest constraints
+    Tests ->> GitHub Registry: Push PROD Image cache + latest
     Tests -->> Airflow Repo: Status update
     deactivate Airflow Repo
     deactivate Tests