Move Dockerfile to Dockerfile.ci (apache#7829)

.dockerignore

@@ -43,7 +43,7 @@
 !pylintrc
 !pytest.ini
 !CHANGELOG.txt
-!Dockerfile
+!Dockerfile.ci
 !LICENSE
 !MANIFEST.in
 !NOTICE

.github/boring-cyborg.yml

@@ -58,7 +58,7 @@ labelPRBasedOnFilePath:
     - .github/**/*
     - .github/*
     - breeze
-    - Dockerfile
+    - Dockerfile.ci
     - BREEZE.rst
     - breeze-complete
     - confirm

.gitignore

@@ -175,7 +175,7 @@ airflow-*.pid
 .dmypy.json
 dmypy.json
 
-# Needed for CI Dockerfile build system
+# Needed for CI Dockerfile.ci build system
 .build
 /tmp
 /files

BREEZE.rst

@@ -118,7 +118,7 @@ For all development tasks, unit tests, integration tests and static code checks,
 This Docker image contains a lot test-related packages (size of ~1GB).
 Its tag follows the pattern of ``<BRANCH>-python<PYTHON_VERSION>-ci``
 (for example, ``apache/airflow:master-python3.6-ci``). The image is built using the
-`<Dockerfile>`_ Dockerfile.
+`<Dockerfile.ci>`_ Dockerfile.
 
 Before you run tests, enter the environment or run local static checks, the necessary local images should be
 pulled and built from Docker Hub. This happens automatically for the test environment but you need to
@@ -128,7 +128,7 @@ The static checks will fail and inform what to do if the image is not yet built.
 
 Building the image first time pulls a pre-built version of images from the Docker Hub, which may take some
 time. But for subsequent source code changes, no wait time is expected.
-However, changes to sensitive files like ``setup.py`` or ``Dockerfile`` will trigger a rebuild
+However, changes to sensitive files like ``setup.py`` or ``Dockerfile.ci`` will trigger a rebuild
 that may take more time though it is highly optimized to only rebuild what is needed.
 
 In most cases, rebuilding an image requires network connectivity (for example, to download new
@@ -427,9 +427,9 @@ from local sources in Airflow. If you wish to add local DAGs that can be run by
 Adding/Modifying Dependencies
 -----------------------------
 
-If you need to change apt dependencies in the ``Dockerfile``, add Python packages in ``setup.py`` or
+If you need to change apt dependencies in the ``Dockerfile.ci``, add Python packages in ``setup.py`` or
 add javascript dependencies in ``package.json``, you can either add dependencies temporarily for a single
-Breeze session or permanently in ``setup.py``, ``Dockerfile``, or ``package.json`` files.
+Breeze session or permanently in ``setup.py``, ``Dockerfile.ci``, or ``package.json`` files.
 
 Installing Dependencies for a Single Breeze Session
 ...................................................
@@ -443,23 +443,23 @@ Therefore, if you want to retain a new dependency, follow the second option desc
 Adding Dependencies Permanently
 ...............................
 
-You can add dependencies to the ``Dockerfile``, ``setup.py`` or ``package.json`` and rebuild the image. This
-should happen automatically if you modify any of these files.
+You can add dependencies to the ``Dockerfile.ci``, ``setup.py`` or ``package.json`` and rebuild the image.
+This should happen automatically if you modify any of these files.
 After you exit the container and re-run ``breeze``, Breeze detects changes in dependencies,
 asks you to confirm rebuilding the image and proceeds with rebuilding if you confirm (or skip it
 if you do not confirm). After rebuilding is done, Breeze drops you to shell. You may also use the
 ``build-only`` command to only rebuild images and not to go into shell.
 
-Changing apt Dependencies in the Dockerfile
-............................................
+Changing apt Dependencies in the Dockerfile.ci
+..............................................
 
-During development, changing dependencies in ``apt-get`` closer to the top of the ``Dockerfile``
+During development, changing dependencies in ``apt-get`` closer to the top of the ``Dockerfile.ci``
 invalidates cache for most of the image. It takes long time for Breeze to rebuild the image.
 So, it is a recommended practice to add new dependencies initially closer to the end
-of the ``Dockerfile``. This way dependencies will be added incrementally.
+of the ``Dockerfile.ci``. This way dependencies will be added incrementally.
 
 Before merge, these dependencies should be moved to the appropriate ``apt-get install`` command,
-which is already in the ``Dockerfile``.
+which is already in the ``Dockerfile.ci``.
 
 Port Forwarding
 ---------------

Dockerfile → Dockerfile.ci

@@ -407,7 +407,7 @@ COPY .coveragerc .rat-excludes .flake8 pylintrc LICENSE MANIFEST.in NOTICE CHANG
      ${AIRFLOW_SOURCES}/
 
 # Needed for building images via docker-in-docker inside the docker
-COPY Dockerfile ${AIRFLOW_SOURCES}/Dockerfile
+COPY Dockerfile.ci ${AIRFLOW_SOURCES}/Dockerfile.ci
 
 # Install autocomplete for airflow
 RUN register-python-argcomplete airflow >> ~/.bashrc

LOCAL_VIRTUALENV.rst

@@ -55,7 +55,7 @@ Homebrew for macOS to install required software packages:
 * MySQL
 * libxml
 
-Refer to the `Dockerfile <Dockerfile>`__ for a comprehensive list
+Refer to the `Dockerfile.ci <Dockerfile.ci>`__ for a comprehensive list
 of required packages.
 
 Extra Packages

STATIC_CODE_CHECKS.rst

@@ -367,7 +367,7 @@ this, run the following scripts (the same is done in Travis CI):
 * `<scripts/ci/ci_check_license.sh>`_ - checks the licenses.
 * `<scripts/ci/ci_docs.sh>`_ - checks that documentation can be built without warnings.
 * `<scripts/ci/ci_flake8.sh>`_ - runs Flake8 source code style enforcement tool.
-* `<scripts/ci/ci_lint_dockerfile.sh>`_ - runs lint checker for the Dockerfile.
+* `<scripts/ci/ci_lint_dockerfile.sh>`_ - runs lint checker for the dockerfiles.
 * `<scripts/ci/ci_mypy.sh>`_ - runs a check for mypy type annotation consistency.
 * `<scripts/ci/ci_pylint_main.sh>`_ - runs pylint static code checker for main files.
 * `<scripts/ci/ci_pylint_tests.sh>`_ - runs pylint static code checker for tests.

common/_files_for_rebuild_check.sh

@@ -21,7 +21,7 @@ FILES_FOR_REBUILD_CHECK=(
  "setup.py"
  "setup.cfg"
  "requirements.txt"
- "Dockerfile"
+ "Dockerfile.ci"
  ".dockerignore"
  "airflow/version.py"
  "airflow/www/package.json"

requirements.txt

@@ -116,6 +116,7 @@ google-cloud-logging==1.15.0
 google-cloud-monitoring==0.34.0
 google-cloud-pubsub==1.3.1
 google-cloud-redis==0.4.0
+google-cloud-secret-manager==0.2.0
 google-cloud-spanner==1.15.0
 google-cloud-speech==1.3.2
 google-cloud-storage==1.26.0

scripts/ci/_utils.sh

@@ -184,7 +184,7 @@ LOCAL_MOUNTS="
 .kube /root/
 .rat-excludes /opt/airflow/
 CHANGELOG.txt /opt/airflow/
-Dockerfile /opt/airflow/
+Dockerfile.ci /opt/airflow/
 LICENSE /opt/airflow/
 MANIFEST.in /opt/airflow/
 NOTICE /opt/airflow/
@@ -332,12 +332,12 @@ function update_all_md5_files() {
 # the Docker image will only be marked for rebuilding only in case any of the important files change:
 # * setup.py
 # * setup.cfg
-# * Dockerfile
+# * Dockerfile.ci
 # * airflow/version.py
 #
 # This is needed because we want to skip rebuilding of the image when only airflow sources change but
 # Trigger rebuild in case we need to change dependencies (setup.py, setup.cfg, change version of Airflow
-# or the Dockerfile itself changes.
+# or the Dockerfile.ci itself changes.
 #
 # Another reason to skip rebuilding Docker is thar currently it takes a bit longer time than simple Docker
 # We need to fix group permissions of files in Docker because different linux build services have
@@ -678,7 +678,7 @@ function get_remote_image_info() {
 # if there are at least NN chaanged layers in your docker file, you should pull the image.
 #
 # Note that this only matters if you have any of the important files changed since the last build
-# of your image such as Dockerfile, setup.py etc.
+# of your image such as Dockerfile.ci, setup.py etc.
 #
 MAGIC_CUT_OFF_NUMBER_OF_LAYERS=34
 
@@ -1190,7 +1190,7 @@ Docker building ${AIRFLOW_CI_IMAGE}.
             "${DOCKER_CACHE_CI_DIRECTIVE[@]}" \
             -t "${AIRFLOW_CI_IMAGE}" \
             --target "${TARGET_IMAGE}" \
-            . | tee -a "${OUTPUT_LOG}"
+            . -f Dockerfile.ci | tee -a "${OUTPUT_LOG}"
         set -u
     fi
     if [[ -n "${DEFAULT_IMAGE:=}" ]]; then

scripts/ci/docker-compose/local.yml

@@ -35,7 +35,7 @@ services:
       - ../../../.kube:/root/.kube:cached
       - ../../../.rat-excludes:/opt/airflow/.rat-excludes:cached
       - ../../../CHANGELOG.txt:/opt/airflow/CHANGELOG.txt:cached
-      - ../../../Dockerfile:/opt/airflow/Dockerfile:cached
+      - ../../../Dockerfile.ci:/opt/airflow/Dockerfile.ci:cached
       - ../../../LICENSE:/opt/airflow/LICENSE:cached
       - ../../../MANIFEST.in:/opt/airflow/MANIFEST.in:cached
       - ../../../NOTICE:/opt/airflow/NOTICE:cached

scripts/ci/in_container/kubernetes/docker/rebuild_airflow_image.sh

@@ -50,7 +50,7 @@ docker build \
     --cache-from "${AIRFLOW_CI_IMAGE}" \
     --tag="${AIRFLOW_CI_IMAGE}" \
     --target="main" \
-    -f Dockerfile . >> "${OUTPUT_LOG}"
+    -f Dockerfile.ci . >> "${OUTPUT_LOG}"
 echo
 echo "Adding kubernetes-specific scripts to basic CI image."
 echo "Building ${AIRFLOW_KUBERNETES_IMAGE} from ${AIRFLOW_CI_IMAGE}"

setup.py

@@ -416,7 +416,7 @@ def write_version(filename: str = os.path.join(*[my_dir, "airflow", "git_version
 ############################################################################################################
 # IMPORTANT NOTE!!!!!!!!!!!!!!!
 # IF you are removing dependencies from this list, please make sure that you also increase
-# DEPENDENCIES_EPOCH_NUMBER in the Dockerfile
+# DEPENDENCIES_EPOCH_NUMBER in the Dockerfile.ci
 ############################################################################################################
 devel = [
     'beautifulsoup4~=4.7.1',
@@ -447,7 +447,7 @@ def write_version(filename: str = os.path.join(*[my_dir, "airflow", "git_version
 ############################################################################################################
 # IMPORTANT NOTE!!!!!!!!!!!!!!!
 # IF you are removing dependencies from the above list, please make sure that you also increase
-# DEPENDENCIES_EPOCH_NUMBER in the Dockerfile
+# DEPENDENCIES_EPOCH_NUMBER in the Dockerfile.ci
 ############################################################################################################
 
 if PY3:
@@ -538,7 +538,7 @@ def write_version(filename: str = os.path.join(*[my_dir, "airflow", "git_version
 #####################################################################################################
 # IMPORTANT NOTE!!!!!!!!!!!!!!!
 # IF you are removing dependencies from this list, please make sure that you also increase
-# DEPENDENCIES_EPOCH_NUMBER in the Dockerfile
+# DEPENDENCIES_EPOCH_NUMBER in the Dockerfile.ci
 #####################################################################################################
 INSTALL_REQUIREMENTS = [
     'alembic>=1.2, <2.0',

tests/bats/test_local_mounts.bats

@@ -46,7 +46,7 @@ ${AIRFLOW_SOURCES}/.rat-excludes:/opt/airflow/.rat-excludes:cached
 -v
 ${AIRFLOW_SOURCES}/CHANGELOG.txt:/opt/airflow/CHANGELOG.txt:cached
 -v
-${AIRFLOW_SOURCES}/Dockerfile:/opt/airflow/Dockerfile:cached
+${AIRFLOW_SOURCES}/Dockerfile.ci:/opt/airflow/Dockerfile.ci:cached
 -v
 ${AIRFLOW_SOURCES}/LICENSE:/opt/airflow/LICENSE:cached
 -v