Improve production image iteration speed (apache#9162)

For a long time the way how entrypoint worked in ci scripts
was wrong. The way it worked was convoluted and short of black
magic. This did not allow to pass multiple test targets and
required separate execute command scripts in Breeze.

This is all now straightened out and both production and
CI image are always using the right entrypoint by default
and we can simply pass parameters to the image as usual without
escaping strings.

This also allowed to remove some breeze commands and
change names of several flags in Breeze to make them more
meaningful.

Both CI and PROD image have now embedded scripts for log
cleaning.

History of image releases is added for 1.10.10-*
alpha quality images.

.dockerignore

@@ -50,7 +50,7 @@
 !NOTICE
 !.github
 !requirements
-!entrypoint.sh
+!empty
 
 # Avoid triggering context change on README change (new companies using Airflow)
 # So please do not uncomment this line ;)

.github/workflows/ci.yml

@@ -35,7 +35,7 @@ env:
   VERBOSE: "true"
   UPGRADE_TO_LATEST_REQUIREMENTS: "false"
   PYTHON_MAJOR_MINOR_VERSION: 3.6
-  ENABLE_REGISTRY_CACHE: "true"
+  USE_GITHUB_REGISTRY: "true"
   CACHE_IMAGE_PREFIX: ${{ github.repository }}
   CACHE_REGISTRY_USERNAME: ${{ github.actor }}
   CACHE_REGISTRY_PASSWORD: ${{ secrets.GITHUB_TOKEN }}

Dockerfile

@@ -157,6 +157,23 @@ ENV PIP_VERSION=${PIP_VERSION}
 
 RUN pip install --upgrade pip==${PIP_VERSION}
 
+ARG AIRFLOW_REPO=apache/airflow
+ENV AIRFLOW_REPO=${AIRFLOW_REPO}
+
+ARG AIRFLOW_BRANCH=master
+ENV AIRFLOW_BRANCH=${AIRFLOW_BRANCH}
+
+ARG AIRFLOW_EXTRAS
+ARG ADDITIONAL_AIRFLOW_EXTRAS=""
+ENV AIRFLOW_EXTRAS=${AIRFLOW_EXTRAS}${ADDITIONAL_AIRFLOW_EXTRAS:+,}${ADDITIONAL_AIRFLOW_EXTRAS}
+
+# In case of Production build image segment we want to pre-install master version of airflow
+# dependencies from github so that we do not have to always reinstall it from the scratch.
+RUN pip install --user \
+    "https://github.com/${AIRFLOW_REPO}/archive/${AIRFLOW_BRANCH}.tar.gz#egg=apache-airflow[${AIRFLOW_EXTRAS}]" \
+        --constraint "https://raw.githubusercontent.com/${AIRFLOW_REPO}/${AIRFLOW_BRANCH}/requirements/requirements-python${PYTHON_MAJOR_MINOR_VERSION}.txt" \
+    && pip uninstall --yes apache-airflow;
+
 ARG AIRFLOW_SOURCES_FROM="."
 ENV AIRFLOW_SOURCES_FROM=${AIRFLOW_SOURCES_FROM}
 
@@ -171,10 +188,6 @@ ENV CASS_DRIVER_BUILD_CONCURRENCY=${CASS_DRIVER_BUILD_CONCURRENCY}
 ARG AIRFLOW_VERSION
 ENV AIRFLOW_VERSION=${AIRFLOW_VERSION}
 
-ARG AIRFLOW_EXTRAS
-ARG ADDITIONAL_AIRFLOW_EXTRAS=""
-ENV AIRFLOW_EXTRAS=${AIRFLOW_EXTRAS}${ADDITIONAL_AIRFLOW_EXTRAS:+,}${ADDITIONAL_AIRFLOW_EXTRAS}
-
 ARG ADDITIONAL_PYTHON_DEPS=""
 ENV ADDITIONAL_PYTHON_DEPS=${ADDITIONAL_PYTHON_DEPS}
 
@@ -214,13 +227,6 @@ RUN \
         rm -rf "${WWW_DIR}/node_modules"; \
     fi
 
-ARG ENTRYPOINT_FILE="entrypoint.sh"
-ENV ENTRYPOINT_FILE="${ENTRYPOINT_FILE}"
-
-# hadolint ignore=DL3020
-ADD ${ENTRYPOINT_FILE} /entrypoint
-RUN chmod a+x /entrypoint
-
 ##############################################################################################
 # This is the actual Airflow image - much smaller than the build one. We copy
 # installed Airflow and all it's dependencies from the build image to make it smaller.
@@ -333,7 +339,11 @@ RUN mkdir -pv "${AIRFLOW_HOME}"; \
     chown -R "airflow" "${AIRFLOW_HOME}"
 
 COPY --chown=airflow:airflow --from=airflow-build-image /root/.local "/home/airflow/.local"
-COPY --chown=airflow:airflow --from=airflow-build-image /entrypoint /entrypoint
+
+COPY scripts/prod/entrypoint_prod.sh /entrypoint
+COPY scripts/prod/clean-logs.sh /clean-logs
+
+RUN chmod a+x /entrypoint /clean-logs
 
 USER airflow
 
@@ -345,6 +355,5 @@ ENV AIRFLOW__CORE__LOAD_EXAMPLES="false"
 
 EXPOSE 8080
 
-COPY scripts/include/clean-logs.sh /usr/local/bin/clean-airflow-logs
 ENTRYPOINT ["/usr/bin/dumb-init", "--", "/entrypoint"]
 CMD ["--help"]

Dockerfile.ci

@@ -224,10 +224,11 @@ ENV AIRFLOW_CI_BUILD_EPOCH=${AIRFLOW_CI_BUILD_EPOCH}
 # In case of CI builds we want to pre-install master version of airflow dependencies so that
 # We do not have to always reinstall it from the scratch.
 # This can be reinstalled from latest master by increasing PIP_DEPENDENCIES_EPOCH_NUMBER.
-# And is automatically reinstalled from the scratch with every python patch level release
-RUN pip install "https://github.com/${AIRFLOW_REPO}/archive/${AIRFLOW_BRANCH}.tar.gz#egg=apache-airflow[${AIRFLOW_EXTRAS}]" \
+# And is automatically reinstalled from the scratch every time patch release of python gets released
+RUN pip install \
+    "https://github.com/${AIRFLOW_REPO}/archive/${AIRFLOW_BRANCH}.tar.gz#egg=apache-airflow[${AIRFLOW_EXTRAS}]" \
         --constraint "https://raw.githubusercontent.com/${AIRFLOW_REPO}/${AIRFLOW_BRANCH}/requirements/requirements-python${PYTHON_MAJOR_MINOR_VERSION}.txt" \
-    && pip uninstall --yes apache-airflow
+    && pip uninstall --yes apache-airflow;
 
 # Link dumb-init for backwards compatibility (so that older images also work)
 RUN ln -sf /usr/bin/dumb-init /usr/local/bin/dumb-init
@@ -275,7 +276,8 @@ COPY airflow/www/static ${AIRFLOW_SOURCES}/airflow/www/static/
 # Package JS/css for production
 RUN yarn --cwd airflow/www run prod
 
-COPY entrypoint.sh /entrypoint.sh
+COPY scripts/ci/in_container/entrypoint_ci.sh /entrypoint
+RUN chmod a+x /entrypoint
 
 # Copy selected subdirectories only
 COPY .github/ ${AIRFLOW_SOURCES}/.github/
@@ -291,9 +293,6 @@ COPY .coveragerc .rat-excludes .flake8 pylintrc LICENSE MANIFEST.in NOTICE CHANG
      setup.cfg setup.py \
      ${AIRFLOW_SOURCES}/
 
-# Needed for building images via docker-in-docker inside the docker
-COPY Dockerfile.ci ${AIRFLOW_SOURCES}/Dockerfile.ci
-
 # Install autocomplete for airflow
 RUN register-python-argcomplete airflow >> ~/.bashrc
 
@@ -317,6 +316,4 @@ ENV PATH="${HOME}:${PATH}"
 
 EXPOSE 8080
 
-ENTRYPOINT ["/usr/bin/dumb-init", "--", "/entrypoint.sh"]
-
-CMD ["--help"]
+ENTRYPOINT ["/usr/bin/dumb-init", "--", "/entrypoint"]