Skip to content

Commit

Permalink
Automatically replace current Airflow version in docs (apache#15484)
Browse files Browse the repository at this point in the history
There are a number of places where we want the current Airflow version
to appear in the docs, and sphinx has this build in, `|version|`.

But sadly that only works for "inline text", it doesn't work in code
blocks or inline code. This PR also adds two custom plugins that make
this work inspired by
https://github.com/adamtheturtle/sphinx-substitution-extensions (but
entirely re-written as that module Just Didn't Work)
  • Loading branch information
ashb authored Apr 22, 2021
1 parent a17db78 commit 4c8a32c
Show file tree
Hide file tree
Showing 10 changed files with 212 additions and 84 deletions.
9 changes: 5 additions & 4 deletions docs/apache-airflow/extra-packages-ref.rst
Original file line number Diff line number Diff line change
Expand Up @@ -84,14 +84,15 @@ For example the below command will install:
* apache-airflow-providers-google
* apache-airflow-providers-apache-spark

with a consistent set of dependencies based on constraint files provided by Airflow Community at the time 2.0.2 version was released.
with a consistent set of dependencies based on constraint files provided by Airflow Community at the time |version| version was released.

.. code-block:: bash
:substitutions:
pip install apache-airflow[google,amazon,apache.spark]==2.0.2 \
--constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.0.2/constraints-3.6.txt"
pip install apache-airflow[google,amazon,apache.spark]==|version| \
--constraint "https://raw.githubusercontent.com/apache/airflow/constraints-|version|/constraints-3.6.txt"
Note, that this will install providers in the versions that were released at the time of Airflow 2.0.2 release. You can later
Note, that this will install providers in the versions that were released at the time of Airflow |version| release. You can later
upgrade those providers manually if you want to use latest versions of the providers.


Expand Down
16 changes: 10 additions & 6 deletions docs/apache-airflow/installation.rst
Original file line number Diff line number Diff line change
Expand Up @@ -130,6 +130,7 @@ You need certain system level requirements in order to install Airflow. Those ar
to be needed for Linux system (Tested on Ubuntu Buster LTS) :

.. code-block:: bash
:substitutions:
sudo apt-get install -y --no-install-recommends \
freetds-bin \
Expand Down Expand Up @@ -162,7 +163,7 @@ not work or will produce unusable Airflow installation.
In order to have repeatable installation, starting from **Airflow 1.10.10** and updated in
**Airflow 1.10.13** we also keep a set of "known-to-be-working" constraint files in the
``constraints-master``, ``constraints-2-0`` and ``constraints-1-10`` orphan branches and then we create tag
for each released version e.g. ``constraints-2.0.2``. This way, when we keep a tested and working set of dependencies.
for each released version e.g. :subst-code:`constraints-|version|`. This way, when we keep a tested and working set of dependencies.

Those "known-to-be-working" constraints are per major/minor Python version. You can use them as constraint
files when installing Airflow from PyPI. Note that you have to specify correct Airflow version
Expand All @@ -176,7 +177,7 @@ You can create the URL to the file substituting the variables in the template be
where:

- ``AIRFLOW_VERSION`` - Airflow version (e.g. ``2.0.2``) or ``master``, ``2-0``, ``1-10`` for latest development version
- ``AIRFLOW_VERSION`` - Airflow version (e.g. :subst-code:`|version|`) or ``master``, ``2-0``, ``1-10`` for latest development version
- ``PYTHON_VERSION`` Python version e.g. ``3.8``, ``3.7``

There is also a no-providers constraint file, which contains just constraints required to install Airflow core. This allows
Expand All @@ -201,8 +202,9 @@ you can use the script below to make an installation a one-liner (the example be
postgres and google provider, as well as ``async`` extra.

.. code-block:: bash
:substitutions:
AIRFLOW_VERSION=2.0.2
AIRFLOW_VERSION=|version|
PYTHON_VERSION="$(python --version | cut -d " " -f 2 | cut -d "." -f 1-2)"
CONSTRAINT_URL="https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-${PYTHON_VERSION}.txt"
pip install "apache-airflow[async,postgres,google]==${AIRFLOW_VERSION}" --constraint "${CONSTRAINT_URL}"
Expand All @@ -219,8 +221,9 @@ being installed.


.. code-block:: bash
:substitutions:
AIRFLOW_VERSION=2.0.2
AIRFLOW_VERSION=|version|
PYTHON_VERSION="$(python --version | cut -d " " -f 2 | cut -d "." -f 1-2)"
CONSTRAINT_URL="https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-${PYTHON_VERSION}.txt"
pip install --upgrade "apache-airflow[postgres,google]==${AIRFLOW_VERSION}" --constraint "${CONSTRAINT_URL}"
Expand Down Expand Up @@ -255,12 +258,13 @@ You can also upgrade the providers to latest versions (you need to use master ve
If you don't want to install any extra providers, initially you can use the command set below.

.. code-block:: bash
:substitutions:
AIRFLOW_VERSION=2.0.2
AIRFLOW_VERSION=|version|
PYTHON_VERSION="$(python --version | cut -d " " -f 2 | cut -d "." -f 1-2)"
# For example: 3.6
CONSTRAINT_URL="https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-no-providers-${PYTHON_VERSION}.txt"
# For example: https://raw.githubusercontent.com/apache/airflow/constraints-no-providers-2.0.2/constraints-3.6.txt
# For example: https://raw.githubusercontent.com/apache/airflow/constraints-no-providers-|version|/constraints-3.6.txt
pip install "apache-airflow==${AIRFLOW_VERSION}" --constraint "${CONSTRAINT_URL}"
Expand Down
2 changes: 1 addition & 1 deletion docs/apache-airflow/start/docker-compose.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@
version: '3'
x-airflow-common:
&airflow-common
image: ${AIRFLOW_IMAGE_NAME:-apache/airflow:master-python3.8}
image: ${AIRFLOW_IMAGE_NAME:-apache/airflow:|version|}
environment:
&airflow-common-env
AIRFLOW__CORE__EXECUTOR: CeleryExecutor
Expand Down
21 changes: 11 additions & 10 deletions docs/apache-airflow/start/docker.rst
Original file line number Diff line number Diff line change
Expand Up @@ -82,11 +82,11 @@ On **all operating systems**, you need to run database migrations and create the
After initialization is complete, you should see a message like below.

.. code-block:: text
.. parsed-literal::
airflow-init_1 | Upgrades done
airflow-init_1 | Admin user airflow created
airflow-init_1 | 2.1.0.dev0
airflow-init_1 | |version|
start_airflow-init_1 exited with code 0
The account created has the login ``airflow`` and the password ``airflow``.
Expand All @@ -102,16 +102,17 @@ Now you can start all services:
In the second terminal you can check the condition of the containers and make sure that no containers are in unhealthy condition:

.. code-block:: bash
.. code-block:: text
:substitutions:
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
247ebe6cf87a apache/airflow:master-python3.8 "/usr/bin/dumb-init …" 3 minutes ago Up 3 minutes (healthy) 8080/tcp compose_airflow-worker_1
ed9b09fc84b1 apache/airflow:master-python3.8 "/usr/bin/dumb-init …" 3 minutes ago Up 3 minutes (healthy) 8080/tcp compose_airflow-scheduler_1
65ac1da2c219 apache/airflow:master-python3.8 "/usr/bin/dumb-init …" 3 minutes ago Up 3 minutes (healthy) 0.0.0.0:5555->5555/tcp, 8080/tcp compose_flower_1
7cb1fb603a98 apache/airflow:master-python3.8 "/usr/bin/dumb-init …" 3 minutes ago Up 3 minutes (healthy) 0.0.0.0:8080->8080/tcp compose_airflow-webserver_1
74f3bbe506eb postgres:13 "docker-entrypoint.s…" 18 minutes ago Up 17 minutes (healthy) 5432/tcp compose_postgres_1
0bd6576d23cb redis:latest "docker-entrypoint.s…" 10 hours ago Up 17 minutes (healthy) 0.0.0.0:6379->6379/tcp compose_redis_1
CONTAINER ID IMAGE |version-spacepad| COMMAND CREATED STATUS PORTS NAMES
247ebe6cf87a apache/airflow:|version| "/usr/bin/dumb-init …" 3 minutes ago Up 3 minutes (healthy) 8080/tcp compose_airflow-worker_1
ed9b09fc84b1 apache/airflow:|version| "/usr/bin/dumb-init …" 3 minutes ago Up 3 minutes (healthy) 8080/tcp compose_airflow-scheduler_1
65ac1da2c219 apache/airflow:|version| "/usr/bin/dumb-init …" 3 minutes ago Up 3 minutes (healthy) 0.0.0.0:5555->5555/tcp, 8080/tcp compose_flower_1
7cb1fb603a98 apache/airflow:|version| "/usr/bin/dumb-init …" 3 minutes ago Up 3 minutes (healthy) 0.0.0.0:8080->8080/tcp compose_airflow-webserver_1
74f3bbe506eb postgres:13 |version-spacepad| "docker-entrypoint.s…" 18 minutes ago Up 17 minutes (healthy) 5432/tcp compose_postgres_1
0bd6576d23cb redis:latest |version-spacepad| "docker-entrypoint.s…" 10 hours ago Up 17 minutes (healthy) 0.0.0.0:6379->6379/tcp compose_redis_1
Accessing the environment
=========================
Expand Down
5 changes: 3 additions & 2 deletions docs/apache-airflow/start/local.rst
Original file line number Diff line number Diff line change
Expand Up @@ -46,17 +46,18 @@ The installation of Airflow is painless if you are following the instructions be
constraint files to enable reproducible installation, so using ``pip`` and constraint files is recommended.

.. code-block:: bash
:substitutions:
# airflow needs a home, ~/airflow is the default,
# but you can lay foundation somewhere else if you prefer
# (optional)
export AIRFLOW_HOME=~/airflow
AIRFLOW_VERSION=2.0.2
AIRFLOW_VERSION=|version|
PYTHON_VERSION="$(python --version | cut -d " " -f 2 | cut -d "." -f 1-2)"
# For example: 3.6
CONSTRAINT_URL="https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-${PYTHON_VERSION}.txt"
# For example: https://raw.githubusercontent.com/apache/airflow/constraints-2.0.2/constraints-3.6.txt
# For example: https://raw.githubusercontent.com/apache/airflow/constraints-|version|/constraints-3.6.txt
pip install "apache-airflow==${AIRFLOW_VERSION}" --constraint "${CONSTRAINT_URL}"
# initialize the database
Expand Down
10 changes: 9 additions & 1 deletion docs/conf.py
Original file line number Diff line number Diff line change
Expand Up @@ -104,6 +104,10 @@
# The full version, including alpha/beta/rc tags.
release = PACKAGE_VERSION

rst_epilog = f"""
.. |version| replace:: {version}
"""

# -- General configuration -----------------------------------------------------
# See: https://www.sphinx-doc.org/en/master/usage/configuration.html

Expand All @@ -124,6 +128,7 @@
"sphinxcontrib.spelling",
'sphinx_airflow_theme',
'redirects',
'substitution_extensions',
]
if PACKAGE_NAME == 'apache-airflow':
extensions.extend(
Expand All @@ -132,6 +137,7 @@
'sphinx.ext.graphviz',
'sphinxcontrib.httpdomain',
'sphinxcontrib.httpdomain',
'extra_files_with_substitutions',
# First, generate redoc
'sphinxcontrib.redoc',
# Second, update redoc script
Expand Down Expand Up @@ -240,9 +246,11 @@ def _get_rst_filepath_from_path(filepath: str):
html_js_files = []
if PACKAGE_NAME == 'apache-airflow':
html_extra_path = [
f"{ROOT_DIR}/docs/apache-airflow/start/docker-compose.yaml",
f"{ROOT_DIR}/docs/apache-airflow/start/airflow.sh",
]
html_extra_with_substituions = [
f"{ROOT_DIR}/docs/apache-airflow/start/docker-compose.yaml",
]

# -- Theme configuration -------------------------------------------------------
# Custom sidebar templates, maps document names to template names.
Expand Down
4 changes: 2 additions & 2 deletions docs/docker-stack/build-arg-ref.rst
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ Those are the most common arguments that you use when you want to build a custom
+==========================================+==========================================+==========================================+
| ``PYTHON_BASE_IMAGE`` | ``python:3.6-slim-buster`` | Base python image. |
+------------------------------------------+------------------------------------------+------------------------------------------+
| ``AIRFLOW_VERSION`` | ``2.0.2`` | version of Airflow. |
| ``AIRFLOW_VERSION`` | :subst-code:`|version|` | version of Airflow. |
+------------------------------------------+------------------------------------------+------------------------------------------+
| ``AIRFLOW_EXTRAS`` | (see Dockerfile) | Default extras with which airflow is |
| | | installed. |
Expand Down Expand Up @@ -64,7 +64,7 @@ Those are the most common arguments that you use when you want to build a custom
| | | 2.0.* installation. In case of building |
| | | specific version you want to point it |
| | | to specific tag, for example |
| | | ``constraints-2.0.2``. |
| | | :subst-code:`constraints-|version|`. |
| | | Auto-detected if empty. |
+------------------------------------------+------------------------------------------+------------------------------------------+

Expand Down
58 changes: 0 additions & 58 deletions docs/exts/docs_build/lint_checks.py
Original file line number Diff line number Diff line change
Expand Up @@ -22,14 +22,6 @@
from itertools import chain
from typing import Iterable, List, Optional, Set

import yaml

try:
from yaml import CSafeLoader as SafeLoader
except ImportError:
from yaml import SafeLoader # type: ignore[misc]

import airflow
from docs.exts.docs_build.docs_builder import ALL_PROVIDER_YAMLS # pylint: disable=no-name-in-module
from docs.exts.docs_build.errors import DocBuildError # pylint: disable=no-name-in-module

Expand Down Expand Up @@ -323,54 +315,6 @@ def check_pypi_repository_in_provider_tocs() -> List[DocBuildError]:
return build_errors


def check_docker_image_tag_in_quick_start_guide() -> List[DocBuildError]:
"""Check that a good docker image is used in the quick start guide for Docker."""
build_errors = []

compose_file_path = f"{DOCS_DIR}/apache-airflow/start/docker-compose.yaml"
expected_tag = 'master-python3.8' if "dev" in airflow.__version__ else airflow.__version__
# master tag is little outdated.
expected_image = f'apache/airflow:{expected_tag}'
with open(compose_file_path) as yaml_file:
content = yaml.load(yaml_file, SafeLoader)
current_image_expression = content['x-airflow-common']['image']
if expected_image not in current_image_expression:
build_errors.append(
DocBuildError(
file_path=compose_file_path,
line_no=None,
message=(
f"Invalid image in docker - compose.yaml\n"
f"Current image expression: {current_image_expression}\n"
f"Expected image: {expected_image}\n"
f"Please check the value of x-airflow-common.image key"
),
)
)
build_error = assert_file_contains(
file_path=f"{DOCS_DIR}/apache-airflow/start/docker.rst",
pattern=re.escape(f'{expected_image} "/usr/bin/dumb-init'),
)
if build_error:
build_errors.append(build_error)

return build_errors


def check_airflow_versions_in_quick_start_guide() -> List[DocBuildError]:
"""Check that a airflow version is presented in example in the quick start guide for Docker."""
build_errors = []

build_error = assert_file_contains(
file_path=f"{DOCS_DIR}/apache-airflow/start/docker.rst",
pattern=re.escape(f"airflow-init_1 | {airflow.__version__}"),
)
if build_error:
build_errors.append(build_error)

return build_errors


def run_all_check() -> List[DocBuildError]:
"""Run all checks from this module"""
general_errors = []
Expand All @@ -379,7 +323,5 @@ def run_all_check() -> List[DocBuildError]:
general_errors.extend(check_exampleinclude_for_example_dags())
general_errors.extend(check_example_dags_in_provider_tocs())
general_errors.extend(check_pypi_repository_in_provider_tocs())
general_errors.extend(check_docker_image_tag_in_quick_start_guide())
general_errors.extend(check_airflow_versions_in_quick_start_guide())

return general_errors
44 changes: 44 additions & 0 deletions docs/exts/extra_files_with_substitutions.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.

import os


def copy_docker_compose(app, exception):
"""Sphinx "build-finished" event handler."""
from sphinx.builders import html as builders

if exception or not isinstance(app.builder, builders.StandaloneHTMLBuilder):
return

# Replace `|version|` in the docker-compose.yaml that we produce in the built docs
for path in app.config.html_extra_with_substituions:
with open(path) as file:
with open(os.path.join(app.outdir, os.path.basename(path)), "w") as output:
for line in file:
output.write(line.replace('|version|', app.config.version))


def setup(app):
"""Setup plugin"""
app.connect("build-finished", copy_docker_compose)

app.add_config_value("html_extra_with_substituions", [], '[str]')

return {
'parallel_write_safe': True,
}
Loading

0 comments on commit 4c8a32c

Please sign in to comment.