Skip to content

Commit

Permalink
Clarify that versioned constraints are fixed at release time (apache#…
Browse files Browse the repository at this point in the history
…28762)

We received a number of requests to upgrade individual dependencies in
the constraint files (mostly due to those dependencies releasing version
with vulnerabilities fixed). This is not how our constraint works, their
main purpose is to provide "consistent installation" mechanism for
anyone who installs airflow from the scratch, we are not going to keep
such relased versions up-to-date with versions of dependencies released
after the release.

This PR provides additional explanation about that in both constraint
files as well as in reference container images which follow similar
patterns.
  • Loading branch information
potiuk authored Jan 6, 2023
1 parent 6af796f commit 8290ade
Show file tree
Hide file tree
Showing 2 changed files with 68 additions and 1 deletion.
34 changes: 33 additions & 1 deletion docs/apache-airflow/installation/installing-from-pypi.rst
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@ and both at the same time. We decided to keep our dependencies as open as possib
version of libraries if needed. This means that from time to time plain ``pip install apache-airflow`` will
not work or will produce an unusable Airflow installation.

In order to have a repeatable installation, we also keep a set of "known-to-be-working" constraint files in the
In order to have a repeatable installation (and only for that reason), we also keep a set of "known-to-be-working" constraint files in the
``constraints-main``, ``constraints-2-0``, ``constraints-2-1`` etc. orphan branches and then we create a tag
for each released version e.g. :subst-code:`constraints-|version|`. This way, we keep a tested and working set of dependencies.

Expand Down Expand Up @@ -88,6 +88,38 @@ constraints always points to the "latest" released Airflow version constraints:
https://raw.githubusercontent.com/apache/airflow/constraints-latest/constraints-3.7.txt
Fixing Constraint files at release time
'''''''''''''''''''''''''''''''''''''''

The released "versioned" constraints are mostly ``fixed`` when we release Airflow version and we only
update them in exceptional circumstances. For example when we find out that the released constraints might prevent
Airflow from being installed consistently from the scratch. In normal circumstances, the constraint files
are not going to change if new version of Airflow dependencies are released - not even when those
versions contain critical security fixes. The process of Airflow releases is designed around upgrading
dependencies automatically where applicable but only when we release a new version of Airflow,
not for already released versions.

If you want to make sure that Airflow dependencies are upgraded to the latest released versions containing
latest security fixes, you should implement your own process to upgrade those yourself when
you detect the need for that. Airflow usually does not upper-bound versions of its dependencies via
requirements, so you should be able to upgrade them to the latest versions - usually without any problems.

Obviously - since we have no control over what gets released in new versions of the dependencies, we
cannot give any guarantees that tests and functionality of those dependencies will be compatible with
Airflow after you upgrade them - testing if Airflow still works with those is in your hands,
and in case of any problems, you should raise issue with the authors of the dependencies that are problematic.
You can also - in such cases - look at the `Airflow issues <https://github.com/apache/airflow/issues>`_
`Airflow Pull Requests <https://github.com/apache/airflow/pulls>`_ and
`Airflow Discussions <https://github.com/apache/airflow/discussions>`_, searching for similar
problems to see if there are any fixes or workarounds found in the ``main`` version of Airflow and apply them
to your deployment.

The easiest way to keep-up with the latest released dependencies is however, to upgrade to the latest released
Airflow version. Whenever we release a new version of Airflow, we upgrade all dependencies to the latest
applicable versions and test them together, so if you want to keep up with those tests - staying up-to-date
with latest version of Airflow is the easiest way to update those dependencies.

Installation and upgrade scenarios
''''''''''''''''''''''''''''''''''

Expand Down
35 changes: 35 additions & 0 deletions docs/docker-stack/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -83,6 +83,41 @@ are also images published from branches but they are used mainly for development
See `Airflow Git Branching <https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#airflow-git-branches>`_
for details.

Fixing images at release time
=============================

The released "versioned" reference images are mostly ``fixed`` when we release Airflow version and we only
update them in exceptional circumstances. For example when we find out that there are dependency errors
that might prevent important Airflow or embedded provider's functionalities working. In normal circumstances,
the images are not going to change after release, even if new version of Airflow dependencies are released -
not even when those versions contain critical security fixes. The process of Airflow releases is designed
around upgrading dependencies automatically where applicable but only when we release a new version of Airflow,
not for already released versions.

If you want to make sure that Airflow dependencies are upgraded to the latest released versions containing
latest security fixes in the image you use, you should implement your own process to upgrade
those yourself when you build custom image based on the Airflow reference one. Airflow usually does not
upper-bound versions of its dependencies via requirements, so you should be able to upgrade them to the
latest versions - usually without any problems. And you can follow the process described in
:ref:`Building the image <build:build_image>` to do it (even in automated way).

Obviously - since we have no control over what gets released in new versions of the dependencies, we
cannot give any guarantees that tests and functionality of those dependencies will be compatible with
Airflow after you upgrade them - testing if Airflow still works with those is in your hands,
and in case of any problems, you should raise issue with the authors of the dependencies that are problematic.
You can also - in such cases - look at the `Airflow issues <https://github.com/apache/airflow/issues>`_
`Airflow Pull Requests <https://github.com/apache/airflow/pulls>`_ and
`Airflow Discussions <https://github.com/apache/airflow/discussions>`_, searching for similar
problems to see if there are any fixes or workarounds found in the ``main`` version of Airflow and apply them
to your custom image.

The easiest way to keep-up with the latest released dependencies is however, to upgrade to the latest released
Airflow version via switching to newly released images as base for your images, when a new version of
Airflow is released. Whenever we release a new version of Airflow, we upgrade all dependencies to the latest
applicable versions and test them together, so if you want to keep up with those tests - staying up-to-date
with latest version of Airflow is the easiest way to update those dependencies.


Support
=======

Expand Down

0 comments on commit 8290ade

Please sign in to comment.