Updates branches and branch documentation after 2.1.0rc1 (apache#15528)

* Updates branches and branch documentation after 2.1.0rc1 This PR updates branches and corresponding documentation and tools after 2.1.0rc1 release. It describes what needs to be done when new release branch is created, and provides tools that allow to do most of the work semi-automatically. Wherever possible and easy, the 2-0 references were replaced with 2-* and where it was more difficult, TODOS were left. The `dev/retag_docker_images.py` script will also be useful when we get to renaming the `master` tag to `main` tag.
sekikn · May 19, 2021 · b709619 · b709619
1 parent e1cdad0
commit b709619
Show file tree

Hide file tree

Showing 17 changed files with 202 additions and 56 deletions.
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
@@ -21,9 +21,9 @@ on:  # yamllint disable-line rule:truthy
   schedule:
     - cron: '28 0 * * *'
   push:
-    branches: ['master', 'v1-10-test', 'v1-10-stable', 'v2-0-test']
+    branches: ['master', 'v[0-9]+-[0-9]+-test']
   pull_request:
-    branches: ['master', 'v1-10-test', 'v1-10-stable', 'v2-0-test']
+    branches: ['master', 'v[0-9]+-[0-9]+-test', 'v[0-9]+-[0-9]+-stable']
 
 env:
   MOUNT_SELECTED_LOCAL_SOURCES: "false"
@@ -1053,9 +1053,10 @@ ${{ hashFiles('.pre-commit-config.yaml') }}"
       - tests-kubernetes
       - prod-images
       - docs
+    # TODO: Generalize me (find a better way to select matching branches)
     if: >
       (github.ref == 'refs/heads/master' || github.ref == 'refs/heads/v1-10-test' ||
-      github.ref == 'refs/heads/v2-0-test') &&
+      github.ref == 'refs/heads/v2-0-test' || github.ref == 'refs/heads/v2-1-test') &&
       github.event_name != 'schedule'
     strategy:
       matrix:
@@ -1113,9 +1114,10 @@ ${{ hashFiles('.pre-commit-config.yaml') }}"
       - tests-kubernetes
       - ci-images
       - docs
+    # TODO: Generalize me (find a better way to select matching branches)
     if: >
       (github.ref == 'refs/heads/master' || github.ref == 'refs/heads/v1-10-test' ||
-      github.ref == 'refs/heads/v2-0-test') &&
+      github.ref == 'refs/heads/v2-0-test' || github.ref == 'refs/heads/v2-1-test') &&
       github.event_name != 'schedule'
     strategy:
       matrix:
@@ -1161,9 +1163,10 @@ ${{ hashFiles('.pre-commit-config.yaml') }}"
       GITHUB_REGISTRY: ${{ needs.ci-images.outputs.githubRegistry }}
       CURRENT_PYTHON_MAJOR_MINOR_VERSIONS_AS_STRING: ${{needs.build-info.outputs.pythonVersionsListAsString}}
     # Only run it for direct pushes
+    # TODO: Generalize me (find a better way to select matching branches)
     if: >
       github.ref == 'refs/heads/master' || github.ref == 'refs/heads/v1-10-test' ||
-      github.ref == 'refs/heads/v2-0-test'
+      github.ref == 'refs/heads/v2-0-test' || github.ref == 'refs/heads/v2-1-test'
     steps:
       - name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
         uses: actions/checkout@v2

diff --git a/BREEZE.rst b/BREEZE.rst
@@ -541,7 +541,7 @@ For all development tasks, unit tests, integration tests, and static code checks
 **CI image** maintained on the DockerHub in the ``apache/airflow`` repository.
 This Docker image contains a lot of test-related packages (size of ~1GB).
 Its tag follows the pattern of ``<BRANCH>-python<PYTHON_MAJOR_MINOR_VERSION>-ci``
-(for example, ``apache/airflow:master-python3.6-ci`` or ``apache/airflow:v2-0-test-python3.6-ci``).
+(for example, ``apache/airflow:master-python3.6-ci`` or ``apache/airflow:v2-1-test-python3.6-ci``).
 The image is built using the `<Dockerfile.ci>`_ Dockerfile.
 
 The CI image is built automatically as needed, however it can be rebuilt manually with
@@ -638,7 +638,7 @@ The **Production image** is also maintained on the DockerHub in the
 ``apache/airflow`` repository. This Docker image (and Dockerfile) contains size-optimised Airflow
 installation with selected extras and dependencies. Its tag follows the pattern of
 ``<BRANCH>-python<PYTHON_MAJOR_MINOR_VERSION>`` (for example, ``apache/airflow:master-python3.6``
-or ``apache/airflow:v2-0-test-python3.6``).
+or ``apache/airflow:v2-1-test-python3.6``).
 
 However in many cases you want to add your own custom version of the image - with added apt dependencies,
 python dependencies, additional Airflow extras. Breeze's ``build-image`` command helps to build your own,
@@ -1279,7 +1279,7 @@ This is the current syntax for  `./breeze <./breeze>`_:
 
   -t, --install-airflow-reference INSTALL_AIRFLOW_REFERENCE
           Installs Airflow directly from reference in GitHub when building PROD image.
-          This can be a GitHub branch like master or v2-0-test, or a tag like 2.0.0a1.
+          This can be a GitHub branch like master or v2-1-test, or a tag like 2.1.0a1.
 
   --installation-method INSTALLATION_METHOD
           Method of installing Airflow in PROD image - either from the sources ('.')
@@ -2504,7 +2504,7 @@ This is the current syntax for  `./breeze <./breeze>`_:
 
   -t, --install-airflow-reference INSTALL_AIRFLOW_REFERENCE
           Installs Airflow directly from reference in GitHub when building PROD image.
-          This can be a GitHub branch like master or v2-0-test, or a tag like 2.0.0a1.
+          This can be a GitHub branch like master or v2-1-test, or a tag like 2.1.0a1.
 
   --installation-method INSTALLATION_METHOD
           Method of installing Airflow in PROD image - either from the sources ('.')

diff --git a/CI.rst b/CI.rst
@@ -21,7 +21,7 @@ CI Environment
 ==============
 
 Continuous Integration is important component of making Apache Airflow robust and stable. We are running
-a lot of tests for every pull request, for master and v2-0-test branches and regularly as CRON jobs.
+a lot of tests for every pull request, for master and v2-*-test branches and regularly as CRON jobs.
 
 Our execution environment for CI is `GitHub Actions <https://github.com/features/actions>`_. GitHub Actions
 (GA) are very well integrated with GitHub code and Workflow and it has evolved fast in 2019/202 to become
@@ -812,15 +812,15 @@ The image names follow the patterns:
 |              |                            | <COMMIT_SHA>                   | It contains only compiled libraries and minimal set of dependencies to run Airflow.        |
 +--------------+----------------------------+--------------------------------+--------------------------------------------------------------------------------------------+
 
-* <BRANCH> might be either "master" or "v1-10-test" or "v2-0-test"
-* <X.Y> - Python version (Major + Minor). For "master" and "v2-0-test" should be in ["3.6", "3.7", "3.8"]. For
+* <BRANCH> might be either "master" or "v1-10-test" or "v2-*-test"
+* <X.Y> - Python version (Major + Minor). For "master" and "v2-*-test" should be in ["3.6", "3.7", "3.8"]. For
   v1-10-test it should be in ["2.7", "3.5", "3.6". "3.7", "3.8"].
 * <RUN_ID> - GitHub Actions RUN_ID. You can get it from CI action job outputs (run id is printed in
   logs and displayed as part of the step name. All PRs belong to some RUN_ID and this way you can
   pull the very exact version of image used in that RUN_ID
-* <COMMIT_SHA> - for images that get merged to "master", "v2-0-test" of "v1-10-test" the images are also tagged
+* <COMMIT_SHA> - for images that get merged to "master", "v2-*-test" of "v1-10-test" the images are also tagged
   with the commit SHA of that particular commit. This way you can easily find the image that was used
-  for testing for that "master", "v2-0-test" or "v1-10-test" test run.
+  for testing for that "master", "v2-*-test" or "v1-10-test" test run.
 
 Reproducing CI Runs locally
 ===========================
@@ -921,7 +921,7 @@ In order to add a new version the following operations should be done (example u
 +=============+================+=======================+=====================+===============+===========+===============+========================================================================+
 | Tag         | nightly-master | master-python3.9      | Dockerfile          | /             | x         | -             | Nightly CI/PROD images from successful scheduled master nightly builds |
 +-------------+----------------+-----------------------+---------------------+---------------+-----------+---------------+------------------------------------------------------------------------+
-| Branch      | v2-0-stable    | v2-0-stable-python3.9 | Dockerfile          | /             | x         |               | CI/PROD images automatically built pushed stable branch                |
+| Branch      | v2-*-stable    | v2-*-stable-python3.9 | Dockerfile          | /             | x         |               | CI/PROD images automatically built pushed stable branch                |
 +-------------+----------------+-----------------------+---------------------+---------------+-----------+---------------+------------------------------------------------------------------------+
 | Tag         | /^([1-2].*)$/  | {\1}-python3.9        | Dockerfile          | /             | x         |               | CI/PROD images automatically built from pushed release tags            |
 +-------------+----------------+-----------------------+---------------------+---------------+-----------+---------------+------------------------------------------------------------------------+
diff --git a/CONTRIBUTING.rst b/CONTRIBUTING.rst
@@ -398,34 +398,34 @@ Airflow Git Branches
 All new development in Airflow happens in the ``master`` branch. All PRs should target that branch.
 
 
-We also have a ``v2-0-test`` branch that is used to test ``2.0.x`` series of Airflow and where committers
+We also have a ``v2-*-test`` branches that are used to test ``2.*.x`` series of Airflow and where committers
 cherry-pick selected commits from the master branch.
 
 Cherry-picking is done with the ``-x`` flag.
 
-The ``v2-0-test`` branch might be broken at times during testing. Expect force-pushes there so
-committers should coordinate between themselves on who is working on the ``v2-0-test`` branch -
+The ``v2-*-test`` branch might be broken at times during testing. Expect force-pushes there so
+committers should coordinate between themselves on who is working on the ``v2-*-test`` branch -
 usually these are developers with the release manager permissions.
 
-The ``v2-0-stable`` branch is rather stable - there are minimum changes coming from approved PRs that
+The ``v2-*-stable`` branch is rather stable - there are minimum changes coming from approved PRs that
 passed the tests. This means that the branch is rather, well, "stable".
 
-Once the ``v2-0-test`` branch stabilises, the ``v2-0-stable`` branch is synchronized with ``v2-0-test``.
-The ``v2-0-stable`` branch is used to release ``2.0.x`` releases.
+Once the ``v2-*-test`` branch stabilises, the ``v2-*-stable`` branch is synchronized with ``v2-*-test``.
+The ``v2-*-stable`` branches are used to release ``2.*.x`` releases.
 
 The general approach is that cherry-picking a commit that has already had a PR and unit tests run
-against main is done to ``v2-0-test`` branch, but PRs from contributors towards 2.0 should target
-``v2-0-stable`` branch.
+against main is done to ``v2-*-test`` branches, but PRs from contributors towards 2.0 should target
+``v2-*-stable`` branches.
 
-The ``v2-0-test`` branch and ``v2-0-stable`` ones are merged just before the release and that's the
+The ``v2-*-test`` branches and ``v2-*-stable`` ones are merged just before the release and that's the
 time when they converge.
 
 The production images are build in DockerHub from:
 
 * master branch for development
-* v2-0-test branch for testing 2.0.x release
-* ``2.0.*``, ``2.0.*rc*`` releases from the ``v2-0-stable`` branch when we prepare release candidates and
-  final releases. There are no production images prepared from v2-0-stable branch.
+* v2-*-test branches for testing 2.*.x release
+* ``2.*.*``, ``2.*.*rc*`` releases from the ``v2-*-stable`` branch when we prepare release candidates and
+  final releases. There are no production images prepared from v2-*-stable branch.
 
 Similar rules apply to ``1.10.x`` releases until June 2021. We have ``v1-10-test`` and ``v1-10-stable``
 branches there.

diff --git a/IMAGES.rst b/IMAGES.rst
@@ -69,8 +69,8 @@ The images are named as follows:
 where:
 
 * ``BRANCH_OR_TAG`` - branch or tag used when creating the image. Examples: ``master``,
-  ``v2-0-test``, ``2.0.0``. The ``master``, ``v2-0-test`` labels are
-  built from branches so they change over time. The ````2.*`` labels are built from git tags
+  ``v2-1-test``, ``2.1.0``. The ``master``, ``v2-*-test`` labels are
+  built from branches so they change over time. The ``2.*.*`` labels are built from git tags
   and they are "fixed" once built.
 * ``PYTHON_MAJOR_MINOR_VERSION`` - version of Python used to build the image. Examples: ``3.6``, ``3.7``,
   ``3.8``
@@ -284,8 +284,8 @@ For example:
 
   apache/airflow:master-python3.6                - production "latest" image from current master
   apache/airflow:master-python3.6-ci             - CI "latest" image from current master
-  apache/airflow:v2-0-test-python3.6-ci          - CI "latest" image from current v2-0-test branch
-  apache/airflow:2.0.0-python3.6                 - production image for 2.0.0 release
+  apache/airflow:v2-1-test-python3.6-ci          - CI "latest" image from current v2-1-test branch
+  apache/airflow:2.1.0-python3.6                 - production image for 2.1.0 release
   apache/airflow:python3.6-master                - base Python image for the master branch
 
 You can see DockerHub images at `<https://hub.docker.com/r/apache/airflow>`_
@@ -320,10 +320,10 @@ Images built as "Run ID snapshot":
 
 .. code-block:: bash
 
-  docker.pkg.github.com.io/apache-airflow/<BRANCH>-pythonX.Y-ci-v2:<RUNID>    - for CI images
-  docker.pkg.github.com/apache-airflow/<BRANCH>-pythonX.Y-v2:<RUNID>       - for production images
-  docker.pkg.github.com/apache-airflow/<BRANCH>-pythonX.Y-build-v2:<RUNID> - for production build stage
-  docker.pkg.github.com/apache-airflow/pythonX.Y-<BRANCH>-v2:X.Y-slim-buster-<RUN_ID>  - for base Python images
+  docker.pkg.github.com.io/apache-airflow/<BRANCH>-pythonX.Y-ci-v2:<RUN_ID>    - for CI images
+  docker.pkg.github.com/apache-airflow/<BRANCH>-pythonX.Y-v2:<RUN_ID>       - for production images
+  docker.pkg.github.com/apache-airflow/<BRANCH>-pythonX.Y-build-v2:<RUN_ID> - for production build stage
+  docker.pkg.github.com/apache-airflow/python-v2:X.Y-slim-buster-<RUN_ID>  - for base Python images
 
 Latest images (pushed when master merge succeeds):
 
@@ -332,7 +332,7 @@ Latest images (pushed when master merge succeeds):
   docker.pkg.github.com/apache/airflow/<BRANCH>-pythonX.Y-ci-v2:latest    - for CI images
   docker.pkg.github.com/apache/airflow/<BRANCH>-pythonX.Y-v2:latest       - for production images
   docker.pkg.github.com/apache/airflow/<BRANCH>-pythonX.Y-build-v2:latest - for production build stage
-  docker.pkg.github.com/apache/airflow/python-<BRANCH>-v1:X.Y-slim-buster - for base Python images
+  docker.pkg.github.com/apache/airflow/python-v2:X.Y-slim-buster - for base Python images
 
 
 Naming convention for GitHub Container Registry
@@ -342,10 +342,10 @@ Images built as "Run ID snapshot":
 
 .. code-block:: bash
 
-  ghcr.io/apache/airflow-<BRANCH>-pythonX.Y-ci-v2:<RUNID>                - for CI images
-  ghcr.io/apache/airflow-<BRANCH>-pythonX.Y-v2:<RUNID>                   - for production images
-  ghcr.io/apache/airflow-<BRANCH>-pythonX.Y-build-v2:<RUNID>             - for production build stage
-  ghcr.io/apache/airflow-pythonX.Y-<BRANCH>-v2:X.Y-slim-buster-<RUN_ID>  - for base Python images
+  ghcr.io/apache/airflow-<BRANCH>-pythonX.Y-ci-v2:<RUN_ID>                - for CI images
+  ghcr.io/apache/airflow-<BRANCH>-pythonX.Y-v2:<RUN_ID>                   - for production images
+  ghcr.io/apache/airflow-<BRANCH>-pythonX.Y-build-v2:<RUN_ID>             - for production build stage
+  ghcr.io/apache/airflow-python-v2:X.Y-slim-buster-<RUN_ID>  - for base Python images
 
 Latest images (pushed when master merge succeeds):
 
@@ -354,7 +354,7 @@ Latest images (pushed when master merge succeeds):
   ghcr.io/apache/airflow-<BRANCH>-pythonX.Y-ci-v2:latest    - for CI images
   ghcr.io/apache/airflow-<BRANCH>-pythonX.Y-v2:latest       - for production images
   ghcr.io/apache/airflow-<BRANCH>-pythonX.Y-build-v2:latest - for production build stage
-  ghcr.io/apache/airflow-python-<BRANCH>-v2:X.Y-slim-buster - for base Python images
+  ghcr.io/apache/airflow-python-v2:X.Y-slim-buster - for base Python images
 
 Note that we never push or pull "release" images to GitHub registry. It is only used for CI builds
 

diff --git a/airflow/kubernetes/pod_template_file_examples/git_sync_template.yaml b/airflow/kubernetes/pod_template_file_examples/git_sync_template.yaml
@@ -29,7 +29,7 @@ spec:
       image: "k8s.gcr.io/git-sync/git-sync:v3.3.0"
       env:
         - name: GIT_SYNC_BRANCH
-          value: "v2-0-stable"
+          value: "v2-1-stable"
         - name: GIT_SYNC_REPO
           value: "https://github.com/apache/airflow.git"
         - name: GIT_SYNC_DEPTH

diff --git a/breeze b/breeze
@@ -2426,7 +2426,7 @@ ${FORMATTED_INSTALL_AIRFLOW_VERSIONS}
 
 -t, --install-airflow-reference INSTALL_AIRFLOW_REFERENCE
         Installs Airflow directly from reference in GitHub when building PROD image.
-        This can be a GitHub branch like master or v2-0-test, or a tag like 2.0.0a1.
+        This can be a GitHub branch like master or v2-1-test, or a tag like 2.1.0a1.
 
 --installation-method INSTALLATION_METHOD
         Method of installing Airflow in PROD image - either from the sources ('.')
@@ -3099,14 +3099,18 @@ function breeze::read_saved_environment_variables() {
 #######################################################################################################
 function breeze::check_and_save_all_params() {
     parameters::check_and_save_allowed_param "PYTHON_MAJOR_MINOR_VERSION" "Python version" "--python"
-
+    # TODO: Generalize me (make it automatically use matching versions)
     if [[ -n "${INSTALL_AIRFLOW_REFERENCE=}" ]]; then
         if [[ ${INSTALL_AIRFLOW_REFERENCE=} == *2_0* ]]; then
             export BRANCH_NAME="v2-0-test"
+        elif [[ ${INSTALL_AIRFLOW_REFERENCE=} == *2_1* ]]; then
+            export BRANCH_NAME="v2-1-test"
         fi
     elif [[ -n "${INSTALL_AIRFLOW_VERSION=}" ]]; then
         if [[ ${INSTALL_AIRFLOW_VERSION=} == *2.0* ]]; then
             export BRANCH_NAME="v2-0-test"
+        elif [[ ${INSTALL_AIRFLOW_VERSION=} == *2.1* ]]; then
+            export BRANCH_NAME="v2-1-test"
         fi
     fi
 

diff --git a/chart/values.schema.json b/chart/values.schema.json
@@ -2337,7 +2337,7 @@
                         "branch": {
                             "description": "Git branch",
                             "type": "string",
-                            "default": "v2-0-stable"
+                            "default": "v2-1-stable"
                         },
                         "rev": {
                             "description": "Git revision.",

diff --git a/chart/values.yaml b/chart/values.yaml
@@ -963,7 +963,7 @@ dags:
     # [email protected]:apache/airflow.git
     # https example: https://github.com/apache/airflow.git
     repo: https://github.com/apache/airflow.git
-    branch: v2-0-stable
+    branch: v2-1-stable
     rev: HEAD
     root: "/git"
     dest: "repo"

diff --git a/codecov.yml b/codecov.yml
@@ -41,6 +41,7 @@ coverage:
           - v1-10-stable
           - v1-10-test
           - v2-0-test
+          - v2-1-test
         if_not_found: success
         if_ci_failed: error
         informational: true
@@ -58,6 +59,7 @@ coverage:
           - v1-10-stable
           - v1-10-test
           - v2-0-test
+          - v2-1-test
         if_no_uploads: error
         if_not_found: success
         if_ci_failed: error

diff --git a/dev/README_RELEASE_AIRFLOW.md b/dev/README_RELEASE_AIRFLOW.md
@@ -22,6 +22,7 @@
 
 - [Prepare the Apache Airflow Package RC](#prepare-the-apache-airflow-package-rc)
   - [Build RC artifacts](#build-rc-artifacts)
+  - [[\Optional\] Create new release branch](#%5Coptional%5C-create-new-release-branch)
   - [Prepare PyPI convenience "snapshot" packages](#prepare-pypi-convenience-snapshot-packages)
   - [\[Optional\] - Manually prepare production Docker Image](#%5Coptional%5C---manually-prepare-production-docker-image)
   - [Prepare Vote email on the Apache Airflow release candidate](#prepare-vote-email-on-the-apache-airflow-release-candidate)
@@ -126,6 +127,30 @@ The Release Candidate artifacts we vote upon should be the exact ones we vote ag
     svn commit -m "Add artifacts for Airflow ${VERSION}"
     ```
 
+## [\Optional\] Create new release branch
+
+When you just released the `X.Y.0` version (first release of new minor version) you need to create release
+branches: `vX-Y-test` and `vX-Y-stable` (for example with `2.1.0rc1` release you need to create v2-1-test and
+`v2-1-stable` branches):
+
+
+   ```shell script
+   # First clone the repo
+   BRANCH_PREFIX=v2-1
+   git branch ${BRANCH_PREFIX}-test
+   git branch ${BRANCH_PREFIX}-stable
+   git push origin ${BRANCH_PREFIX}-test ${BRANCH_PREFIX}-stable
+   ```
+
+Search and replace all the vX-Y for previous branches (TODO: we should likely automate this a bit more)
+
+Run script to re-tag images from the ``master`` branch to the  ``vX-Y-test`` branch:
+
+   ```shell script
+   ./dev/retag_docker_images.py --source-branch master --target-branch ${BRANCH_PREFIX}-test
+   ```
+
+
 ## Prepare PyPI convenience "snapshot" packages
 
 At this point we have the artefact that we vote on, but as a convenience to developers we also want to