Skip to content

Latest commit

 

History

History
2384 lines (1584 loc) · 99.1 KB

BREEZE.rst

File metadata and controls

2384 lines (1584 loc) · 99.1 KB
Airflow Breeze - Development and Test Environment for Apache Airflow

Airflow Breeze is an easy-to-use development and test environment using Docker Compose. The environment is available for local use and is also used in Airflow's CI tests.

We called it Airflow Breeze as It's a Breeze to contribute to Airflow.

The advantages and disadvantages of using the Breeze environment vs. other ways of testing Airflow are described in CONTRIBUTING.rst.

Watch the video below about Airflow Breeze. It explains the motivation for Breeze and screencasts all its uses.

  • Version: Install the latest stable Docker Community Edition and add it to the PATH.
  • Permissions: Configure to run the docker commands directly and not only via root user. Your user should be in the docker group. See Docker installation guide for details.
  • Disk space: On macOS, increase your available disk space before starting to work with the environment. At least 128 GB of free disk space is recommended. You can also get by with a smaller space but make sure to clean up the Docker disk space periodically. See also Docker for Mac - Space for details on increasing disk space available for Docker on Mac.
  • Docker problems: Sometimes it is not obvious that space is an issue when you run into a problem with Docker. If you see a weird behaviour, try breeze cleanup-image command. Also see pruning instructions from Docker.

Here is an example configuration with more than 200GB disk space for Docker:

Disk space MacOS
  • Version: Install the latest stable Docker Compose and add it to the PATH. See Docker Compose Installation Guide for details.
  • Permissions: Configure permission to run the docker-compose command.
Airflow Breeze - Docker WSL2 integration
  • WSL 2 Filesystem Performance :

    Accessing the host Windows filesystem incurs a performance penalty, it is therefore recommended to do development on the Linux filesystem. E.g. Run cd ~ and create a development folder in your Linux distro home and git pull the Airflow repo there.

  • WSL 2 Memory Usage :

    WSL 2 can consume a lot of memory under the process name "Vmmem". To reclaim the memory after development you can:

    • On the Linux distro clear cached memory: sudo sysctl -w vm.drop_caches=3
    • If no longer using Docker you can quit Docker Desktop (right click system try icon and select "Quit Docker Desktop")
    • If no longer using WSL you can shut it down on the Windows Host with the following command: wsl --shutdown
  • Developing in WSL 2 :

    You can use all the standard Linux command line utilities to develop on WSL 2. Further VS Code supports developing in Windows but remotely executing in WSL. If VS Code is installed on the Windows host system then in the WSL Linux Distro you can run code . in the root directory of you Airflow repo to launch VS Code.

  • For Linux, run apt install util-linux coreutils or an equivalent if your system is not Debian-based.

  • For macOS, install GNU getopt and gstat utilities to get Airflow Breeze running.

    Run brew install gnu-getopt coreutils and then follow instructions to link the gnu-getopt version to become the first on the PATH. Make sure to re-login after you make the suggested changes.

Examples:

If you use bash, run this command and re-login:

echo 'export PATH="/usr/local/opt/gnu-getopt/bin:$PATH"' >> ~/.bash_profile
. ~/.bash_profile

If you use zsh, run this command and re-login:

echo 'export PATH="/usr/local/opt/gnu-getopt/bin:$PATH"' >> ~/.zprofile
. ~/.zprofile

Minimum 4GB RAM is required to run the full Breeze environment.

On macOS, 2GB of RAM are available for your Docker containers by default, but more memory is recommended (4GB should be comfortable). For details see Docker for Mac - Advanced tab.

On Windows WSL 2 expect the Linux Distro and Docker containers to use 7 - 8 GB of RAM.

You may need to clean up your Docker environment occasionally. The images are quite big (1.5GB for both images needed for static code analysis and CI tests) and, if you often rebuild/update them, you may end up with some unused image data.

To clean up the Docker environment:

  1. Stop Breeze with ./breeze stop.

  2. Run the docker system prune command.

  3. Run docker images --all and docker ps --all to verify that your Docker is clean.

    Both commands should return an empty list of images and containers respectively.

If you run into disk space errors, consider pruning your Docker images with the docker system prune --all command. You may need to restart the Docker Engine before running this command.

In case of disk space errors on macOS, increase the disk space available for Docker. See Prerequisites for details.

Installation is as easy as checking out Airflow repository and running Breeze command. You enter the Breeze test environment by running the ./breeze script. You can run it with the help command to see the list of available options. See Breeze Command-Line Interface Reference for details.

./breeze

The First time you run Breeze, it pulls and builds a local version of Docker images. It pulls the latest Airflow CI images from Airflow DockerHub and uses them to build your local Docker images. Note that the first run (per python) might take up to 10 minutes on a fast connection to start. Subsequent runs should be much faster.

Once you enter the environment, you are dropped into bash shell of the Airflow container and you can run tests immediately.

To use the full potential of breeze you should set up autocomplete and you can add the checked-out Airflow repository to your PATH to run Breeze without the ./ and from any directory.

The breeze command comes with a built-in bash/zsh autocomplete setup command. After installing, when you start typing the command, you can use <TAB> to show all the available switches and get auto-completion on typical values of parameters that you can use.

You should set up the autocomplete option automatically by running:

./breeze setup-autocomplete

You get the auto-completion working when you re-enter the shell.

When you enter the Breeze environment, automatically an environment file is sourced from files/airflow-breeze-config/variables.env. The files folder from your local sources is automatically mounted to the container under /files path and you can put there any files you want to make available for the Breeze container.

Breeze helps with running tests in the same environment/way as CI tests are run. You can run various types of tests while you enter Breeze CI interactive environment - this is described in detail in TESTING.rst

You can use additional breeze flags to choose your environment. You can specify a Python version to use, and backend (the meta-data database). Thanks to that, with Breeze, you can recreate the same environments as we have in matrix builds in the CI.

For example, you can choose to run Python 3.6 tests with MySQL as backend and in the Docker environment as follows:

./breeze --python 3.6 --backend mysql

The choices you make are persisted in the ./.build/ cache directory so that next time when you use the breeze script, it could use the values that were used previously. This way you do not have to specify them when you run the script. You can delete the .build/ directory in case you want to restore the default settings.

The defaults when you run the Breeze environment are Python 3.6 version and SQLite database.

If you are having problems with the Breeze environment, try the steps below. After each step you can check whether your problem is fixed.

  1. If you are on macOS, check if you have enough disk space for Docker.
  2. Restart Breeze with ./breeze restart.
  3. Delete the .build directory and run ./breeze build-image --force-pull-images.
  4. Clean up Docker images via breeze cleanup-image command.
  5. Restart your Docker Engine and try again.
  6. Restart your machine and try again.
  7. Re-install Docker CE and try again.

In case the problems are not solved, you can set the VERBOSE_COMMANDS variable to "true":

export VERBOSE_COMMANDS="true"

Then run the failed command, copy-and-paste the output from your terminal to the Airflow Slack #airflow-breeze channel and describe your problem.

Airflow Breeze is a bash script serving as a "swiss-army-knife" of Airflow testing. Under the hood it uses other scripts that you can also run manually if you have problem with running the Breeze environment.

Breeze script allows performing the following tasks:

Managing CI environment:

  • Build CI docker image with breeze build-image command
  • Enter interactive shell in CI container when shell (or no command) is specified
  • Join running interactive shell with breeze exec command
  • Stop running interactive environment with breeze stop command
  • Restart running interactive environment with breeze restart command
  • Run test specified with breeze tests command
  • Generate constraints with breeze generate-constraints command
  • Execute arbitrary command in the test environment with breeze shell command
  • Execute arbitrary docker-compose command with breeze docker-compose command
  • Push docker images with breeze push-image command (require committer's rights to push images)

You can optionally reset database if specified as extra --db-reset flag and for CI image you can also start integrations (separate Docker images) if specified as extra --integration flags. You can also chose which backend database should be used with --backend flag and python version with --python flag.

You can also have breeze launch Airflow automatically breeze start-airflow, this will drop you in a tmux session with three panes (one to monitor the scheduler, one for the webserver and one with a shell for additional commands.

Managing Prod environment (with --production-image flag):

  • Build CI docker image with breeze build-image command
  • Enter interactive shell in PROD container when shell (or no command) is specified
  • Join running interactive shell with breeze exec command
  • Stop running interactive environment with breeze stop command
  • Restart running interactive environment with breeze restart command
  • Execute arbitrary command in the test environment with breeze shell command
  • Execute arbitrary docker-compose command with breeze docker-compose command
  • Push docker images with breeze push-image command (require committer's rights to push images)

You can optionally reset database if specified as extra --db-reset flag. You can also chose which backend database should be used with --backend flag and python version with --python flag.

Manage and Interact with Kubernetes tests environment:

  • Manage KinD Kubernetes cluster and deploy Airflow to KinD cluster breeze kind-cluster commands
  • Run Kubernetes tests specified with breeze kind-cluster tests command
  • Enter the interactive kubernetes test environment with breeze kind-cluster shell command

Run static checks:

  • Run static checks - either for currently staged change or for all files with breeze static-check command

Build documentation:

  • Build documentation with breeze build-docs command

Set up local development environment:

  • Setup local virtualenv with breeze setup-virtualenv command
  • Setup autocomplete for itself with breeze setup-autocomplete command

Often if you want to run full airflow in the Breeze environment you need to launch multiple terminals and run airflow webserver, airflow scheduler, airflow worker in separate terminals.

This can be achieved either via tmux or via exec-ing into the running container from the host. Tmux is installed inside the container and you can launch it with tmux command. Tmux provides you with the capability of creating multiple virtual terminals and multiplex between them. More about tmux can be found at tmux github wiki page . Tmux has several useful shortcuts that allow you to split the terminals, open new tabs etc - it's pretty useful to learn it.

Another way is to exec into Breeze terminal from the host's terminal. Often you can have multiple terminals in the host (Linux/MacOS/WSL2 on Windows) and you can simply use those terminals to enter the running container. It's as easy as launching breeze exec while you already started the Breeze environment. You will be dropped into bash and environment variables will be read in the same way as when you enter the environment. You can do it multiple times and open as many terminals as you need.

For development convenience we installed simple wrappers for the most common cloud providers CLIs. Those CLIs are not installed when you build or pull the image - they will be downloaded as docker images the first time you attempt to use them. It is downloaded and executed in your host's docker engine so once it is downloaded, it will stay until you remove the downloaded images from your host container.

For each of those CLI credentials are taken (automatically) from the credentials you have defined in your ${HOME} directory on host.

Those tools also have host Airflow source directory mounted in /opt/airflow path so you can directly transfer files to/from your airflow host sources.

Those are currently installed CLIs (they are available as aliases to the docker commands):

Cloud Provider CLI tool Docker image Configuration dir
Amazon Web Services aws amazon/aws-cli:latest .aws
Microsoft Azure az mcr.microsoft.com/azure-cli:latest .azure
Google Cloud bq gcr.io/google.com/cloudsdktool/cloud-sdk:latest .config/gcloud
gcloud gcr.io/google.com/cloudsdktool/cloud-sdk:latest .config/gcloud
gsutil gcr.io/google.com/cloudsdktool/cloud-sdk:latest .config/gcloud

For each of the CLIs we have also an accompanying *-update alias (for example aws-update) which will pull the latest image for the tool. Note that all Google Cloud tools are served by one image and they are updated together.

Also - in case you run several different Breeze containers in parallel (from different directories, with different versions) - they docker images for CLI Cloud Providers tools are shared so if you update it for one Breeze container, they will also get updated for all the other containers.

When Breeze starts, it can start additional integrations. Those are additional docker containers that are started in the same docker-compose command. Those are required by some of the tests as described in TESTING.rst#airflow-integration-tests.

By default Breeze starts only airflow container without any integration enabled. If you selected postgres or mysql backend, the container for the selected backend is also started (but only the one that is selected). You can start the additional integrations by passing --integration flag with appropriate integration name when starting Breeze. You can specify several --integration flags to start more than one integration at a time. Finally you can specify --integration all to start all integrations.

Once integration is started, it will continue to run until the environment is stopped with breeze stop command. or restarted via breeze restart command

Note that running integrations uses significant resources - CPU and memory.

With Breeze you can build images that are used by Airflow CI and production ones.

For all development tasks, unit tests, integration tests, and static code checks, we use the CI image maintained on the DockerHub in the apache/airflow repository. This Docker image contains a lot of test-related packages (size of ~1GB). Its tag follows the pattern of <BRANCH>-python<PYTHON_MAJOR_MINOR_VERSION>-ci (for example, apache/airflow:master-python3.6-ci or apache/airflow:v1-10-test-python3.6-ci). The image is built using the Dockerfile.ci Dockerfile.

The CI image is built automatically as needed, however it can be rebuilt manually with build-image command. The production image should be built manually - but also a variant of this image is built automatically when kubernetes tests are executed see Running Kubernetes tests

Building the image first time pulls a pre-built version of images from the Docker Hub, which may take some time. But for subsequent source code changes, no wait time is expected. However, changes to sensitive files like setup.py or Dockerfile.ci will trigger a rebuild that may take more time though it is highly optimized to only rebuild what is needed.

Breeze has built in mechanism to check if your local image has not diverged too much from the latest image build on CI. This might happen when for example latest patches have been released as new Python images or when significant changes are made in the Dockerfile. In such cases, Breeze will download the latest images before rebuilding because this is usually faster than rebuilding the image.

In most cases, rebuilding an image requires network connectivity (for example, to download new dependencies). If you work offline and do not want to rebuild the images when needed, you can set the FORCE_ANSWER_TO_QUESTIONS variable to no as described in the Setting default behaviour for user interaction section.

The Production image is also maintained on the DockerHub in the `apache/airflow repository. This Docker image (and Dockerfile) contains size-optimised Airflow installation with selected extras and dependencies. Its tag follows the pattern of <BRANCH>-python<PYTHON_MAJOR_MINOR_VERSION> (for example, apache/airflow:master-python3.6 or apache/airflow:v1-10-test-python3.6).

However in many cases you want to add your own custom version of the image - with added apt dependencies, python dependencies, additional Airflow extras. Breeze's build-image command helps to build your own, customized variant of the image that contains everything you need.

You can switch to building the production image by adding --production-image flag to the build_image command. Note, that the images can also be build using docker build command by passing appropriate build-args as described in IMAGES.rst , but Breeze provides several flags that makes it easier to do it. You can see all the flags by running ./breeze build-image --help, but here typical examples are presented:

./breeze build-image --production-image --additional-extras "jira"

This installs additional jira extra while installing airflow in the image.

./breeze build-image --production-image --additional-python-deps "torchio==0.17.10"

This install additional pypi dependency - torchio in specified version.

./breeze build-image --production-image --additional-dev-apt-deps "libasound2-dev" \
   --additional-runtime-apt-deps "libasound2"

This install additional apt dependencies - libasound2-dev in build image and libasound in the final image. Those are development dependencies that might be needed to build and use python packages added via the --additional-python-deps flag. The dev dependencies are not installed in the final production image, they are only installed in the build "segment" of the production image that is used as an intermediate step to build the final image. Usually names of the dev dependencies end with -dev suffix and they need to also be paired with corresponding runtime dependency added for the runtime image (without -dev).

./breeze build-image --production-image --python 3.7 --additional-dev-deps "libasound2-dev" \
   --additional-runtime-apt-deps "libasound2"

Same as above but uses python 3.7.

With Breeze you can also use the master Dockerfile to build custom images for released Airflow versions. This works in the same way as building production image from master, but you need to add additional switch --install-airflow-version. You should pass version of airflow (as released in PyPI). It can be used to install both released versions and release candidates. Similarly as in case of master images, we can pass additional extras/dependencies to install via the additional flags.

./breeze build-image --production-image --additional-extras "jira" --install-airflow-version="1.10.11"

Builds airflow image with released Airflow version 1.10.11 and additional extra "jira" added.

./breeze build-image --production-image --install-airflow-version="1.10.11rc2"

Builds airflow image with released Airflow version 1.10.11rc2.

You can also build airflow directly from GitHub source code - by providing Git Reference via --install-airflow-reference. The reference can be a branch name, tag name, or commit hash. This is useful mostly for testing.

./breeze build-image --production-image --install-airflow-reference="v1-10-test"

This Builds airflow image from the current v1-10-test branch of Airflow.

./breeze build-image --production-image \
     --install-airflow-reference="0d91fcf725f69e10f0969ca36f9e38e1d74110d0"

This Builds airflow image from the 0d91fcf725f69e10f0969ca36f9e38e1d74110d0 commit hash on GitHub.

You can run static checks via Breeze. You can also run them via pre-commit command but with auto-completion Breeze makes it easier to run selective static checks. If you press <TAB> after the static-check and if you have auto-complete setup you should see auto-completable list of all checks available.

./breeze static-check mypy

The above will run mypy check for currently staged files.

You can also add arbitrary pre-commit flag after --

./breeze static-check mypy -- --all-files

The above will run mypy check for all files.

To build documentation in Breeze, use the build-docs command:

./breeze build-docs

Results of the build can be found in the docs/_build folder.

The documentation build consists of three steps:

  • verifying consistency of indexes
  • building documentation
  • spell checking

You can choose only one stage of the two by providing --spellcheck-only or --docs-only after extra -- flag.

./breeze build-docs -- --spellcheck-only

Often errors during documentation generation come from the docstrings of auto-api generated classes. During the docs building auto-api generated files are stored in the docs/_api folder. This helps you easily identify the location the problems with documentation originated from.

Whenever setup.py gets modified, the CI master job will re-generate constraint files. Those constraint files are stored in separated orphan branches: constraints-master and constraint-1-10. They are stored separately for each python version. Those are constraint files as described in detail in the CONTRIBUTING.rst#pinned-constraint-files contributing documentation.

In case someone modifies setup.py, the CRON scheduled CI build automatically upgrades and pushes changed to the constraint files, however you can also perform test run of this locally using generate-constraints command of Breeze.

./breeze generate-constraints --python 3.6
./breeze generate-constraints --python 3.7
./breeze generate-constraints --python 3.8

This bumps the constraint files to latest versions and stores hash of setup.py. The generated constraint and setup.py hash files are stored in the files folder and while generating the constraints diff of changes vs the previous constraint files is printed.

You can set up your host IDE (for example, IntelliJ's PyCharm/Idea) to work with Breeze and benefit from all the features provided by your IDE, such as local and remote debugging, language auto-completion, documentation support, etc.

To use your host IDE with Breeze:

  1. Create a local virtual environment:

    You can use any of the following wrappers to create and manage your virtual environments: pyenv, pyenv-virtualenv, or virtualenvwrapper.

    Ideally, you should have virtualenvs for all Python versions supported by Airflow (3.5, 3.6, 3.7)

  2. Use the right command to activate the virtualenv (workon if you use virtualenvwrapper or pyenv activate if you use pyenv.

  3. Initialize the created local virtualenv:

./breeze initialize-local-virtualenv --python 3.8
  1. Select the virtualenv you created as the project's default virtualenv in your IDE.

Note that you can also use the local virtualenv for Airflow development without Breeze. This is a lightweight solution that has its own limitations.

More details on using the local virtualenv are available in the LOCAL_VIRTUALENV.rst.

Breeze helps with running Kubernetes tests in the same environment/way as CI tests are run. Breeze helps to setup KinD cluster for testing, setting up virtualenv and downloads the right tools automatically to run the tests.

This is described in detail in Testing Kubernetes.

After starting up, the environment runs in the background and takes precious memory. You can always stop it via:

./breeze stop

When you are in the CI container, the following directories are used:

/opt/airflow - Contains sources of Airflow mounted from the host (AIRFLOW_SOURCES).
/root/airflow - Contains all the "dynamic" Airflow files (AIRFLOW_HOME), such as:
    airflow.db - sqlite database in case sqlite is used;
    dags - folder with non-test dags (test dags are in /opt/airflow/tests/dags);
    logs - logs from Airflow executions;
    unittest.cfg - unit test configuration generated when entering the environment;
    webserver_config.py - webserver configuration generated when running Airflow in the container.

Note that when running in your local environment, the /root/airflow/logs folder is actually mounted from your logs directory in the Airflow sources, so all logs created in the container are automatically visible in the host as well. Every time you enter the container, the logs directory is cleaned so that logs do not accumulate.

When you are in the production container, the following directories are used:

/opt/airflow - Contains sources of Airflow mounted from the host (AIRFLOW_SOURCES).
/root/airflow - Contains all the "dynamic" Airflow files (AIRFLOW_HOME), such as:
    airflow.db - sqlite database in case sqlite is used;
    dags - folder with non-test dags (test dags are in /opt/airflow/tests/dags);
    logs - logs from Airflow executions;
    unittest.cfg - unit test configuration generated when entering the environment;
    webserver_config.py - webserver configuration generated when running Airflow in the container.

Note that when running in your local environment, the /root/airflow/logs folder is actually mounted from your logs directory in the Airflow sources, so all logs created in the container are automatically visible in the host as well. Every time you enter the container, the logs directory is cleaned so that logs do not accumulate.

To run other commands/executables inside the Breeze Docker-based environment, use the ./breeze shell command. You should add your command as -c "command" after -- as extra arguments.

./breeze shell -- -c "ls -la"

To run Docker Compose commands (such as help, pull, etc), use the docker-compose command. To add extra arguments, specify them after -- as extra arguments.

./breeze docker-compose pull -- --ignore-pull-failures

You can also restart the environment and enter it via:

./breeze restart

Sometimes during the build, you are asked whether to perform an action, skip it, or quit. This happens when rebuilding or removing an image - actions that take a lot of time and could be potentially destructive.

For automation scripts, you can export one of the three variables to control the default interaction behaviour:

export FORCE_ANSWER_TO_QUESTIONS="yes"

If FORCE_ANSWER_TO_QUESTIONS is set to yes, the images are automatically rebuilt when needed. Images are deleted without asking.

export FORCE_ANSWER_TO_QUESTIONS="no"

If FORCE_ANSWER_TO_QUESTIONS is set to no, the old images are used even if rebuilding is needed. This is useful when you work offline. Deleting images is aborted.

export FORCE_ANSWER_TO_QUESTIONS="quit"

If FORCE_ANSWER_TO_QUESTIONS is set to quit, the whole script is aborted. Deleting images is aborted.

If more than one variable is set, yes takes precedence over no, which takes precedence over quit.

On Linux, there is a problem with propagating ownership of created files (a known Docker problem). The files and directories created in the container are not owned by the host user (but by the root user in our case). This may prevent you from switching branches, for example, if files owned by the root user are created within your sources. In case you are on a Linux host and have some files in your sources created by the root user, you can fix the ownership of those files by running this script:

./scripts/ci/tools/ci_fix_ownership.sh

Important sources of Airflow are mounted inside the airflow container that you enter. This means that you can continue editing your changes on the host in your favourite IDE and have them visible in the Docker immediately and ready to test without rebuilding images. You can disable mounting by specifying --skip-mounting-local-sources flag when running Breeze. In this case you will have sources embedded in the container and changes to these sources will not be persistent.

After you run Breeze for the first time, you will have empty directory files in your source code, which will be mapped to /files in your Docker container. You can pass there any files you need to configure and run Docker. They will not be removed between Docker runs.

By default /files/dags folder is mounted from your local <AIRFLOW_SOURCES>/files/dags and this is the directory used by airflow scheduler and webserver to scan dags for. You can use it to test your dags from local sources in Airflow. If you wish to add local DAGs that can be run by Breeze.

When you run Airflow Breeze, the following ports are automatically forwarded:

  • 28080 -> forwarded to Airflow webserver -> airflow:8080
  • 25433 -> forwarded to Postgres database -> postgres:5432
  • 23306 -> forwarded to MySQL database -> mysql:3306

You can connect to these ports/databases using:

  • Webserver: http://127.0.0.1:28080
  • Postgres: jdbc:postgresql://127.0.0.1:25433/airflow?user=postgres&password=airflow
  • Mysql: jdbc:mysql://localhost:23306/airflow?user=root

Start the webserver manually with the airflow webserver command if you want to connect to the webserver. You can use tmux to multiply terminals. You may need to create a user prior to running the webserver in order to log in. This can be done with the following command:

airflow users create --role Admin --username admin --password admin --email [email protected] --firstname foo --lastname bar

For databases, you need to run airflow db reset at least once (or run some tests) after you started Airflow Breeze to get the database/tables created. You can connect to databases with IDE or any other database client:

Airflow Breeze - Database view

You can change the used host port numbers by setting appropriate environment variables:

  • WEBSERVER_HOST_PORT
  • POSTGRES_HOST_PORT
  • MYSQL_HOST_PORT

If you set these variables, next time when you enter the environment the new ports should be in effect.

If you need to change apt dependencies in the Dockerfile.ci, add Python packages in setup.py or add JavaScript dependencies in package.json, you can either add dependencies temporarily for a single Breeze session or permanently in setup.py, Dockerfile.ci, or package.json files.

You can install dependencies inside the container using sudo apt install, pip install or yarn install (in airflow/www folder) respectively. This is useful if you want to test something quickly while you are in the container. However, these changes are not retained: they disappear once you exit the container (except for the node.js dependencies if your sources are mounted to the container). Therefore, if you want to retain a new dependency, follow the second option described below.

You can add dependencies to the Dockerfile.ci, setup.py or package.json and rebuild the image. This should happen automatically if you modify any of these files. After you exit the container and re-run breeze, Breeze detects changes in dependencies, asks you to confirm rebuilding the image and proceeds with rebuilding if you confirm (or skip it if you do not confirm). After rebuilding is done, Breeze drops you to shell. You may also use the build-image command to only rebuild CI image and not to go into shell.

During development, changing dependencies in apt-get closer to the top of the Dockerfile.ci invalidates cache for most of the image. It takes long time for Breeze to rebuild the image. So, it is a recommended practice to add new dependencies initially closer to the end of the Dockerfile.ci. This way dependencies will be added incrementally.

Before merge, these dependencies should be moved to the appropriate apt-get install command, which is already in the Dockerfile.ci.

This is the current syntax for ./breeze:

 ####################################################################################################

 usage: breeze [FLAGS] [COMMAND] -- <EXTRA_ARGS>

 By default the script enters the  CI container and drops you to bash shell, but you can choose
 one of the commands to run specific actions instead.

 Add --help after each command to see details:

 Commands without arguments:

   shell                                    [Default] Enters interactive shell in the container
   build-docs                               Builds documentation in the container
   build-image                              Builds CI or Production docker image
   cleanup-image                            Cleans up the container image created
   exec                                     Execs into running breeze container in new terminal
   generate-constraints                     Generates pinned constraint files
   push-image                               Pushes images to registry
   initialize-local-virtualenv              Initializes local virtualenv
   setup-autocomplete                       Sets up autocomplete for breeze
   start-airflow                            Starts Scheduler and Webserver and enters the shell
   stop                                     Stops the docker-compose environment
   restart                                  Stops the docker-compose environment including DB cleanup
   toggle-suppress-cheatsheet               Toggles on/off cheatsheet
   toggle-suppress-asciiart                 Toggles on/off asciiart

 Commands with arguments:

   docker-compose                <ARG>      Executes specified docker-compose command
   kind-cluster                  <ARG>      Manages KinD cluster on the host
   prepare-provider-readme       <ARG>      Prepares provider packages readme files
   prepare-provider-packages     <ARG>      Prepares provider packages
   static-check                  <ARG>      Performs selected static check for changed files
   tests                         <ARG>      Runs selected tests in the container

 Help commands:

   flags                                    Shows all breeze's flags
   help                                     Shows this help message
   help-all                                 Shows detailed help for all commands and flags

 ####################################################################################################

 Detailed usage

 ####################################################################################################


 Detailed usage for command: shell


 breeze shell [FLAGS] [-- <EXTRA_ARGS>]

       This is default subcommand if no subcommand is used.

       Enters interactive shell where you can run all tests, start Airflow webserver, scheduler,
       workers, interact with the database, run DAGs etc. It is the default command if no command
       is selected. The shell is executed in the container and in case integrations are chosen,
       the integrations will be started as separated docker containers - under the docker-compose
       supervision. Local sources are by default mounted to within the container so you can edit
       them locally and run tests immediately in the container. Several folders ('files', 'dist')
       are also mounted so that you can exchange files between the host and container.

       The 'files/airflow-breeze-config/variables.env' file can contain additional variables
       and setup. This file is automatically sourced when you enter the container. Database
       and webserver ports are forwarded to appropriate database/webserver so that you can
       connect to it from your host environment.

       You can also pass <EXTRA_ARGS> after -- they will be passed as bash parameters, this is
       especially useful to pass bash options, for example -c to execute command:

       'breeze shell -- -c "ls -la"'
       'breeze -- -c "ls -la"'

       For DockerHub pull --dockerhub-user and --dockerhub-repo flags can be used to specify
       the repository to pull from. For GitHub repository, the --github-repository
       flag can be used for the same purpose. You can also use
       --github-image-id <COMMIT_SHA>|<RUN_ID> in case you want to pull the image
       with specific COMMIT_SHA tag or RUN_ID.

       'breeze shell \
             --github-image-id 9a621eaa394c0a0a336f8e1b31b35eff4e4ee86e' - pull/use image with SHA
       'breeze \
             --github-image-id 9a621eaa394c0a0a336f8e1b31b35eff4e4ee86e' - pull/use image with SHA
       'breeze shell \
             --github-image-id 209845560' - pull/use image with RUN_ID
       'breeze \
             --github-image-id 209845560' - pull/use image with RUN_ID

 Flags:

 Run 'breeze flags' to see all applicable flags.


 ####################################################################################################


 Detailed usage for command: build-docs


 breeze build-docs [-- <EXTRA_ARGS>]

       Builds Airflow documentation. The documentation is build inside docker container - to
       maintain the same build environment for everyone. Appropriate sources are mapped from
       the host to the container so that latest sources are used. The folders where documentation
       is generated ('docs/_build') are also mounted to the container - this way results of
       the documentation build is available in the host.

       The possible extra args are: --docs-only, --spellcheck-only, --help


 ####################################################################################################


 Detailed usage for command: build-image


 breeze build-image [FLAGS]

       Builds docker image (CI or production) without entering the container. You can pass
       additional options to this command, such as '--force-build-image',
       '--force-pull-image', '--python', '--build-cache-local' or '-build-cache-pulled'
       in order to modify build behaviour.

       You can also pass '--production-image' flag to build production image rather than CI image.

       For DockerHub pull --dockerhub-user and --dockerhub-repo flags can be used to specify
       the repository to pull from. For GitHub repository, the --github-repository
       flag can be used for the same purpose. You can also use
       --github-image-id <COMMIT_SHA>|<RUN_ID> in case you want to pull the image with
       specific COMMIT_SHA tag or RUN_ID.

 Flags:

 -p, --python PYTHON_MAJOR_MINOR_VERSION
         Python version used for the image. This is always major/minor version.

         Note that versions 2.7 and 3.5 are only valid when installing Airflow 1.10 with
         --install-airflow-version or --install-airflow-reference flags.

         One of:

                2.7 3.5 3.6 3.7 3.8

 -a, --install-airflow-version INSTALL_AIRFLOW_VERSION
         If specified, installs Airflow directly from PIP released version. This happens at
         image building time in production image and at container entering time for CI image. One of:

                1.10.12 1.10.11 1.10.10 1.10.9 1.10.8 1.10.7 1.10.6 1.10.5 1.10.4 1.10.3 1.10.2
                master v1-10-test

 -t, --install-airflow-reference INSTALL_AIRFLOW_REFERENCE
         If specified, installs Airflow directly from reference in GitHub. This happens at
         image building time in production image and at container entering time for CI image.

 --no-rbac-ui
         Disables RBAC UI when Airflow 1.10.* is installed.

 --install-wheels
         If specified it will look for wheel packages placed in dist folder and it will install the
         wheels from there after installing Airflow. This is useful for testing backport
         packages as well as in the future for testing provider packages for 2.0.

 -I, --production-image
         Use production image for entering the environment and builds (not for tests).

 -F, --force-build-images
         Forces building of the local docker images. The images are rebuilt
         automatically for the first time or when changes are detected in
         package-related files, but you can force it using this flag.

 -P, --force-pull-images
         Forces pulling of images from DockerHub before building to populate cache. The
         images are pulled by default only for the first time you run the
         environment, later the locally build images are used as cache.

 Customization options:

 -E, --extras EXTRAS
         Extras to pass to build images The default are different for CI and production images:

         CI image:
                devel_ci

         Production image:
                async,aws,azure,celery,dask,elasticsearch,gcp,kubernetes,mysql,postgres,redis,slack,
                ssh,statsd,virtualenv

 --image-tag TAG
         Additional tag in the image.

 --additional-extras ADDITIONAL_EXTRAS
         Additional extras to pass to build images The default is no additional extras.

 --additional-python-deps ADDITIONAL_PYTHON_DEPS
         Additional python dependencies to use when building the images.

 --dev-apt-command DEV_APT_COMMAND
         The basic command executed before dev apt deps are installed.

 --additional-dev-apt-command ADDITIONAL_DEV_APT_COMMAND
         Additional command executed before dev apt deps are installed.

 --additional-dev-apt-deps ADDITIONAL_DEV_APT_DEPS
         Additional apt dev dependencies to use when building the images.

 --dev-apt-deps DEV_APT_DEPS
         The basic apt dev dependencies to use when building the images.

 --additional-dev-apt-deps ADDITIONAL_DEV_DEPS
         Additional apt dev dependencies to use when building the images.

 --additional-dev-apt-envs ADDITIONAL_DEV_APT_ENVS
         Additional environment variables set when adding dev dependencies.

 --runtime-apt-command RUNTIME_APT_COMMAND
         The basic command executed before runtime apt deps are installed.

 --additional-runtime-apt-command ADDITIONAL_RUNTIME_APT_COMMAND
         Additional command executed before runtime apt deps are installed.

 --runtime-apt-deps ADDITIONAL_RUNTIME_APT_DEPS
         The basic apt runtime dependencies to use when building the images.

 --additional-runtime-apt-deps ADDITIONAL_RUNTIME_DEPS
         Additional apt runtime dependencies to use when building the images.

 --additional-runtime-apt-envs ADDITIONAL_RUNTIME_APT_DEPS
         Additional environment variables set when adding runtime dependencies.

 Build options:

 --disable-mysql-client-installation
         Disables installation of the mysql client which might be problematic if you are building
         image in controlled environment. Only valid for production image.

 --constraints-location
         Url to the constraints file. In case of the production image it can also be a path to the
         constraint file placed in 'docker-context-files' folder, in which case it has to be
         in the form of '/docker-context-files/<NAME_OF_THE_FILE>'

 --disable-pip-cache
         Disables GitHub PIP cache during the build. Useful if github is not reachable during build.

 --install-local-pip-wheels
         This flag is only used in production image building. If it is used then instead of
         installing Airflow from PyPI, the packages are installed from the .whl packages placed
         in the 'docker-context-files' folder. It implies '--disable-pip-cache'

 -C, --force-clean-images
         Force build images with cache disabled. This will remove the pulled or build images
         and start building images from scratch. This might take a long time.

 -r, --skip-rebuild-check
         Skips checking image for rebuilds. It will use whatever image is available locally/pulled.

 -L, --build-cache-local
         Uses local cache to build images. No pulled images will be used, but results of local
         builds in the Docker cache are used instead. This will take longer than when the pulled
         cache is used for the first time, but subsequent '--build-cache-local' builds will be
         faster as they will use mostly the locally build cache.

         This is default strategy used by the Production image builds.

 -U, --build-cache-pulled
         Uses images pulled from registry (either DockerHub or GitHub depending on
         --github-registry flag) to build images. The pulled images will be used as cache.
         Those builds are usually faster than when ''--build-cache-local'' with the exception if
         the registry images are not yet updated. The DockerHub images are updated nightly and the
         GitHub images are updated after merges to master so it might be that the images are still
         outdated vs. the latest version of the Dockerfiles you are using. In this case, the
         ''--build-cache-local'' might be faster, especially if you iterate and change the
         Dockerfiles yourself.

         This is default strategy used by the CI image builds.

 -X, --build-cache-disabled
         Disables cache during docker builds. This is useful if you want to make sure you want to
         rebuild everything from scratch.

         This strategy is used by default for both Production and CI images for the scheduled
         (nightly) builds in CI.

 -D, --dockerhub-user DOCKERHUB_USER
         DockerHub user used to pull, push and build images. Default: apache.

 -H, --dockerhub-repo DOCKERHUB_REPO
         DockerHub repository used to pull, push, build images. Default: airflow.

 -c, --github-registry GITHUB_REGISTRY
         If GitHub registry is enabled, pulls and pushes are done from the GitHub registry not
         DockerHub. You need to be logged in to the registry in order to be able to pull/push from
         and you need to be committer to push to Apache Airflow' GitHub registry.

 -g, --github-repository GITHUB_REPOSITORY
         GitHub repository used to pull, push images when cache is used.
         Default: apache/airflow.

         If you use this flag, automatically --github-registry flag is enabled.

 -s, --github-image-id COMMIT_SHA|RUN_ID
         <RUN_ID> or <COMMIT_SHA> of the image. Images in GitHub registry are stored with those
         to be able to easily find the image for particular CI runs. Once you know the
         <RUN_ID> or <COMMIT_SHA>, you can specify it in github-image-id flag and Breeze will
         automatically pull and use that image so that you can easily reproduce a problem
         that occurred in CI.

         If you use this flag, automatically --github-registry is enabled.

         Default: latest.

 -v, --verbose
         Show verbose information about executed docker, kind, kubectl, helm commands. Useful for
         debugging - when you run breeze with --verbose flags you will be able to see the commands
         executed under the hood and copy&paste them to your terminal to debug them more easily.

         Note that you can further increase verbosity and see all the commands executed by breeze
         by running 'export VERBOSE_COMMANDS="true"' before running breeze.


 ####################################################################################################


 Detailed usage for command: cleanup-image


 breeze cleanup-image [FLAGS]

       Removes the breeze-related images created in your local docker image cache. This will
       not reclaim space in docker cache. You need to 'docker system prune' (optionally
       with --all) to reclaim that space.

 Flags:

 -p, --python PYTHON_MAJOR_MINOR_VERSION
         Python version used for the image. This is always major/minor version.

         Note that versions 2.7 and 3.5 are only valid when installing Airflow 1.10 with
         --install-airflow-version or --install-airflow-reference flags.

         One of:

                2.7 3.5 3.6 3.7 3.8

 -I, --production-image
         Use production image for entering the environment and builds (not for tests).

 -v, --verbose
         Show verbose information about executed docker, kind, kubectl, helm commands. Useful for
         debugging - when you run breeze with --verbose flags you will be able to see the commands
         executed under the hood and copy&paste them to your terminal to debug them more easily.

         Note that you can further increase verbosity and see all the commands executed by breeze
         by running 'export VERBOSE_COMMANDS="true"' before running breeze.


 ####################################################################################################


 Detailed usage for command: exec


 breeze exec [-- <EXTRA_ARGS>]

       Execs into interactive shell to an already running container. The container mus be started
       already by breeze shell command. If you are not familiar with tmux, this is the best
       way to run multiple processes in the same container at the same time for example scheduler,
       webserver, workers, database console and interactive terminal.


 ####################################################################################################


 Detailed usage for command: generate-constraints


 breeze generate-constraints [FLAGS]

       Generates pinned constraint files from setup.py. Those files are generated in files folder
       - separate files for different python version. Those constraint files when pushed to orphan
       constraint-master and constraint-1-10 branches are used to generate repeatable
       CI builds as well as run repeatable production image builds. You can use those constraints
       to predictably install released Airflow versions. This is mainly used to test the constraint
       generation - constraints are pushed to the orphan branches by a successful scheduled
       CRON job in CI automatically.

 Flags:

 -p, --python PYTHON_MAJOR_MINOR_VERSION
         Python version used for the image. This is always major/minor version.

         Note that versions 2.7 and 3.5 are only valid when installing Airflow 1.10 with
         --install-airflow-version or --install-airflow-reference flags.

         One of:

                2.7 3.5 3.6 3.7 3.8

 -v, --verbose
         Show verbose information about executed docker, kind, kubectl, helm commands. Useful for
         debugging - when you run breeze with --verbose flags you will be able to see the commands
         executed under the hood and copy&paste them to your terminal to debug them more easily.

         Note that you can further increase verbosity and see all the commands executed by breeze
         by running 'export VERBOSE_COMMANDS="true"' before running breeze.


 ####################################################################################################


 Detailed usage for command: push-image


 breeze push_image [FLAGS]

       Pushes images to docker registry. You can push the images to DockerHub registry (default)
       or to the GitHub registry (if --github-registry flag is used).

       For DockerHub pushes --dockerhub-user and --dockerhub-repo flags can be used to specify
       the repository to push to. For GitHub repository, the --github-repository
       flag can be used for the same purpose. You can also add
       --github-image-id <COMMIT_SHA>|<RUN_ID> in case you want to push image with specific
       SHA tag or run id. In case you specify --github-repository or --github-image-id, you
       do not need to specify --github-registry flag.

       You can also add --production-image flag to switch to production image (default is CI one)

       Examples:

       'breeze push-image' or
       'breeze push-image --dockerhub-user user' to push to your private registry or
       'breeze push-image --production-image' - to push production image or
       'breeze push-image --github-registry' - to push to GitHub image registry or
       'breeze push-image \
             --github-repository user/airflow' - to push to your user's fork
       'breeze push-image \
             --github-image-id 9a621eaa394c0a0a336f8e1b31b35eff4e4ee86e' - to push with COMMIT_SHA
       'breeze push-image \
             --github-image-id 209845560' - to push with RUN_ID

 Flags:

 -D, --dockerhub-user DOCKERHUB_USER
         DockerHub user used to pull, push and build images. Default: apache.

 -H, --dockerhub-repo DOCKERHUB_REPO
         DockerHub repository used to pull, push, build images. Default: airflow.

 -c, --github-registry GITHUB_REGISTRY
         If GitHub registry is enabled, pulls and pushes are done from the GitHub registry not
         DockerHub. You need to be logged in to the registry in order to be able to pull/push from
         and you need to be committer to push to Apache Airflow' GitHub registry.

 -g, --github-repository GITHUB_REPOSITORY
         GitHub repository used to pull, push images when cache is used.
         Default: apache/airflow.

         If you use this flag, automatically --github-registry flag is enabled.

 -s, --github-image-id COMMIT_SHA|RUN_ID
         <RUN_ID> or <COMMIT_SHA> of the image. Images in GitHub registry are stored with those
         to be able to easily find the image for particular CI runs. Once you know the
         <RUN_ID> or <COMMIT_SHA>, you can specify it in github-image-id flag and Breeze will
         automatically pull and use that image so that you can easily reproduce a problem
         that occurred in CI.

         If you use this flag, automatically --github-registry is enabled.

         Default: latest.

 -v, --verbose
         Show verbose information about executed docker, kind, kubectl, helm commands. Useful for
         debugging - when you run breeze with --verbose flags you will be able to see the commands
         executed under the hood and copy&paste them to your terminal to debug them more easily.

         Note that you can further increase verbosity and see all the commands executed by breeze
         by running 'export VERBOSE_COMMANDS="true"' before running breeze.


 ####################################################################################################


 Detailed usage for command: initialize-local-virtualenv


 breeze initialize-local-virtualenv [FLAGS]

       Initializes locally created virtualenv installing all dependencies of Airflow
       taking into account the constraints for the version specified.
       This local virtualenv can be used to aid auto-completion and IDE support as
       well as run unit tests directly from the IDE. You need to have virtualenv
       activated before running this command.

 Flags:

 -p, --python PYTHON_MAJOR_MINOR_VERSION
         Python version used for the image. This is always major/minor version.

         Note that versions 2.7 and 3.5 are only valid when installing Airflow 1.10 with
         --install-airflow-version or --install-airflow-reference flags.

         One of:

                2.7 3.5 3.6 3.7 3.8


 ####################################################################################################


 Detailed usage for command: setup-autocomplete


 breeze setup-autocomplete

       Sets up autocomplete for breeze commands. Once you do it you need to re-enter the bash
       shell and when typing breeze command <TAB> will provide autocomplete for
       parameters and values.


 ####################################################################################################


 Detailed usage for command: start-airflow


 breeze start-airflow

       Like the Shell command this will enter the interactive shell, but it will also start
       automatically the Scheduler and the Webserver. It will leave you in a tmux session where you
       can also observe what is happening in your Airflow.

       This is a convenient way to setup a development environment. Your dags will be loaded from the
       folder 'files/dags' on your host machine (it could take some times).

       If you want to load default connections and example dags you can use the dedicated flags.

 Flags:

 --load-example-dags
         Include Airflow example dags.

 --load-default-connections
         Include Airflow Default Connections.


 ####################################################################################################


 Detailed usage for command: stop


 breeze stop

       Brings down running docker compose environment. When you start the environment, the docker
       containers will continue running so that startup time is shorter. But they take quite a lot of
       memory and CPU. This command stops all running containers from the environment.


 ####################################################################################################


 Detailed usage for command: restart


 breeze restart [FLAGS]

       Restarts running docker compose environment. When you restart the environment, the docker
       containers will be restarted. That includes cleaning up the databases. This is
       especially useful if you switch between different versions of Airflow.

 Flags:

 Run 'breeze flags' to see all applicable flags.


 ####################################################################################################


 Detailed usage for command: toggle-suppress-cheatsheet


 breeze toggle-suppress-cheatsheet

       Toggles on/off cheatsheet displayed before starting bash shell.


 ####################################################################################################


 Detailed usage for command: toggle-suppress-asciiart


 breeze toggle-suppress-asciiart

       Toggles on/off asciiart displayed before starting bash shell.


 ####################################################################################################


 Detailed usage for command: docker-compose


 breeze docker-compose [FLAGS] COMMAND [-- <EXTRA_ARGS>]

       Run docker-compose command instead of entering the environment. Use 'help' as command
       to see available commands. The <EXTRA_ARGS> passed after -- are treated
       as additional options passed to docker-compose. For example

       'breeze docker-compose pull -- --ignore-pull-failures'

 Flags:

 -p, --python PYTHON_MAJOR_MINOR_VERSION
         Python version used for the image. This is always major/minor version.

         Note that versions 2.7 and 3.5 are only valid when installing Airflow 1.10 with
         --install-airflow-version or --install-airflow-reference flags.

         One of:

                2.7 3.5 3.6 3.7 3.8

 -b, --backend BACKEND
         Backend to use for tests - it determines which database is used.
         One of:

                sqlite mysql postgres

         Default: sqlite

 --postgres-version POSTGRES_VERSION
         Postgres version used. One of:

                9.6 10

 --mysql-version MYSQL_VERSION
         Mysql version used. One of:

                5.7 8

 -v, --verbose
         Show verbose information about executed docker, kind, kubectl, helm commands. Useful for
         debugging - when you run breeze with --verbose flags you will be able to see the commands
         executed under the hood and copy&paste them to your terminal to debug them more easily.

         Note that you can further increase verbosity and see all the commands executed by breeze
         by running 'export VERBOSE_COMMANDS="true"' before running breeze.


 ####################################################################################################


 Detailed usage for command: kind-cluster


 breeze kind-cluster [FLAGS] OPERATION

       Manages host-side Kind Kubernetes cluster that is used to run Kubernetes integration tests.
       It allows to start/stop/restart/status the Kind Kubernetes cluster and deploy Airflow to it.
       This enables you to run tests inside the breeze environment with latest airflow images.
       Note that in case of deploying airflow, the first step is to rebuild the image and loading it
       to the cluster so you can also pass appropriate build image flags that will influence
       rebuilding the production image. Operation is one of:

                start stop restart status deploy test shell

 Flags:

 -p, --python PYTHON_MAJOR_MINOR_VERSION
         Python version used for the image. This is always major/minor version.

         Note that versions 2.7 and 3.5 are only valid when installing Airflow 1.10 with
         --install-airflow-version or --install-airflow-reference flags.

         One of:

                2.7 3.5 3.6 3.7 3.8

 -F, --force-build-images
         Forces building of the local docker images. The images are rebuilt
         automatically for the first time or when changes are detected in
         package-related files, but you can force it using this flag.

 -P, --force-pull-images
         Forces pulling of images from DockerHub before building to populate cache. The
         images are pulled by default only for the first time you run the
         environment, later the locally build images are used as cache.

 Customization options:

 -E, --extras EXTRAS
         Extras to pass to build images The default are different for CI and production images:

         CI image:
                devel_ci

         Production image:
                async,aws,azure,celery,dask,elasticsearch,gcp,kubernetes,mysql,postgres,redis,slack,
                ssh,statsd,virtualenv

 --image-tag TAG
         Additional tag in the image.

 --additional-extras ADDITIONAL_EXTRAS
         Additional extras to pass to build images The default is no additional extras.

 --additional-python-deps ADDITIONAL_PYTHON_DEPS
         Additional python dependencies to use when building the images.

 --dev-apt-command DEV_APT_COMMAND
         The basic command executed before dev apt deps are installed.

 --additional-dev-apt-command ADDITIONAL_DEV_APT_COMMAND
         Additional command executed before dev apt deps are installed.

 --additional-dev-apt-deps ADDITIONAL_DEV_APT_DEPS
         Additional apt dev dependencies to use when building the images.

 --dev-apt-deps DEV_APT_DEPS
         The basic apt dev dependencies to use when building the images.

 --additional-dev-apt-deps ADDITIONAL_DEV_DEPS
         Additional apt dev dependencies to use when building the images.

 --additional-dev-apt-envs ADDITIONAL_DEV_APT_ENVS
         Additional environment variables set when adding dev dependencies.

 --runtime-apt-command RUNTIME_APT_COMMAND
         The basic command executed before runtime apt deps are installed.

 --additional-runtime-apt-command ADDITIONAL_RUNTIME_APT_COMMAND
         Additional command executed before runtime apt deps are installed.

 --runtime-apt-deps ADDITIONAL_RUNTIME_APT_DEPS
         The basic apt runtime dependencies to use when building the images.

 --additional-runtime-apt-deps ADDITIONAL_RUNTIME_DEPS
         Additional apt runtime dependencies to use when building the images.

 --additional-runtime-apt-envs ADDITIONAL_RUNTIME_APT_DEPS
         Additional environment variables set when adding runtime dependencies.

 Build options:

 --disable-mysql-client-installation
         Disables installation of the mysql client which might be problematic if you are building
         image in controlled environment. Only valid for production image.

 --constraints-location
         Url to the constraints file. In case of the production image it can also be a path to the
         constraint file placed in 'docker-context-files' folder, in which case it has to be
         in the form of '/docker-context-files/<NAME_OF_THE_FILE>'

 --disable-pip-cache
         Disables GitHub PIP cache during the build. Useful if github is not reachable during build.

 --install-local-pip-wheels
         This flag is only used in production image building. If it is used then instead of
         installing Airflow from PyPI, the packages are installed from the .whl packages placed
         in the 'docker-context-files' folder. It implies '--disable-pip-cache'

 -C, --force-clean-images
         Force build images with cache disabled. This will remove the pulled or build images
         and start building images from scratch. This might take a long time.

 -r, --skip-rebuild-check
         Skips checking image for rebuilds. It will use whatever image is available locally/pulled.

 -L, --build-cache-local
         Uses local cache to build images. No pulled images will be used, but results of local
         builds in the Docker cache are used instead. This will take longer than when the pulled
         cache is used for the first time, but subsequent '--build-cache-local' builds will be
         faster as they will use mostly the locally build cache.

         This is default strategy used by the Production image builds.

 -U, --build-cache-pulled
         Uses images pulled from registry (either DockerHub or GitHub depending on
         --github-registry flag) to build images. The pulled images will be used as cache.
         Those builds are usually faster than when ''--build-cache-local'' with the exception if
         the registry images are not yet updated. The DockerHub images are updated nightly and the
         GitHub images are updated after merges to master so it might be that the images are still
         outdated vs. the latest version of the Dockerfiles you are using. In this case, the
         ''--build-cache-local'' might be faster, especially if you iterate and change the
         Dockerfiles yourself.

         This is default strategy used by the CI image builds.

 -X, --build-cache-disabled
         Disables cache during docker builds. This is useful if you want to make sure you want to
         rebuild everything from scratch.

         This strategy is used by default for both Production and CI images for the scheduled
         (nightly) builds in CI.


 ####################################################################################################


 Detailed usage for command: prepare-provider-readme


 breeze prepare-provider-packages [FLAGS] [YYYY.MM.DD] [PACKAGE_ID ...]

       Prepares README.md files for backport packages. You can provide (after --) optional version
       in the form of YYYY.MM.DD, optionally followed by the list of packages to generate readme for.
       If the first parameter is not formatted as a date, then today is regenerated.
       If no packages are specified, readme for all packages are generated.
       If no date is specified, current date + 3 days is used (allowing for PMC votes to pass).

       Examples:

       'breeze prepare-provider-readme' or
       'breeze prepare-provider-readme 2020.05.10' or
       'breeze prepare-provider-readme 2020.05.10 https google amazon'

       General form:

       'breeze prepare-provider-readme YYYY.MM.DD <PACKAGE_ID> ...'

       * YYYY.MM.DD - is the CALVER version of the package to prepare. Note that this date
         cannot be earlier than the already released version (the script will fail if it
         will be). It can be set in the future anticipating the future release date.

       * <PACKAGE_ID> is usually directory in the airflow/providers folder (for example
         'google' but in several cases, it might be one level deeper separated with
         '.' for example 'apache.hive'

 Flags:

 -v, --verbose
         Show verbose information about executed docker, kind, kubectl, helm commands. Useful for
         debugging - when you run breeze with --verbose flags you will be able to see the commands
         executed under the hood and copy&paste them to your terminal to debug them more easily.

         Note that you can further increase verbosity and see all the commands executed by breeze
         by running 'export VERBOSE_COMMANDS="true"' before running breeze.


 ####################################################################################################


 Detailed usage for command: prepare-provider-packages


 breeze prepare-provider-packages [FLAGS] [PACKAGE_ID ...]

       Prepares backport packages. You can provide (after --) optional list of packages to prepare.
       If no packages are specified, readme for all packages are generated. You can specify optional
       --version-suffix-for-svn flag to generate rc candidate packages to upload to SVN or
       --version-suffix-for-pypi flag to generate rc candidates for PyPI packages.

       Examples:

       'breeze prepare-provider-packages' or
       'breeze prepare-provider-packages google' or
       'breeze prepare-provider-packages --version-suffix-for-svn rc1 http google amazon' or
       'breeze prepare-provider-packages --version-suffix-for-pypi rc1 http google amazon'

       General form:

       'breeze prepare-provider-packages \
             [--version-suffix-for-svn|--version-suffix-for-pypi] <PACKAGE_ID> ...'

       * <PACKAGE_ID> is usually directory in the airflow/providers folder (for example
         'google'), but in several cases, it might be one level deeper separated with '.'
         for example 'apache.hive'

 Flags:

 -S, --version-suffix-for-pypi SUFFIX
         Adds optional suffix to the version in the generated backport package. It can be used
         to generate rc1/rc2 ... versions of the packages to be uploaded to PyPI.

 -N, --version-suffix-for-svn SUFFIX
         Adds optional suffix to the generated names of package. It can be used to generate
         rc1/rc2 ... versions of the packages to be uploaded to SVN.

 -v, --verbose
         Show verbose information about executed docker, kind, kubectl, helm commands. Useful for
         debugging - when you run breeze with --verbose flags you will be able to see the commands
         executed under the hood and copy&paste them to your terminal to debug them more easily.

         Note that you can further increase verbosity and see all the commands executed by breeze
         by running 'export VERBOSE_COMMANDS="true"' before running breeze.


 ####################################################################################################


 Detailed usage for command: static-check


 breeze static-check [FLAGS] static_check [-- <EXTRA_ARGS>]

       Run selected static checks for currently changed files. You should specify static check that
       you would like to run or 'all' to run all checks. One of:

                all all-but-pylint airflow-config-yaml base-operator bats-tests
                bats-in-container-tests black build build-providers-dependencies
                check-apache-license check-builtin-literals check-executables-have-shebangs
                check-hooks-apply check-integrations check-merge-conflict check-xml
                consistent-pylint daysago-import-check debug-statements detect-private-key doctoc
                dont-use-safe-filter end-of-file-fixer fix-encoding-pragma flake8 forbid-tabs
                helm-lint incorrect-use-of-LoggingMixin insert-license isort language-matters
                lint-dockerfile lint-openapi mermaid mixed-line-ending mypy no-relative-imports
                pre-commit-descriptions provide-create-sessions pydevd pydocstyle pylint
                pylint-tests python-no-log-warn restrict-start_date rst-backticks setup-order
                shellcheck sort-in-the-wild stylelint trailing-whitespace update-breeze-file
                update-extras update-local-yml-file update-setup-cfg-file yamllint

       You can pass extra arguments including options to to the pre-commit framework as
       <EXTRA_ARGS> passed after --. For example:

       'breeze static-check mypy' or
       'breeze static-check mypy -- --files tests/core.py'
       'breeze static-check mypy -- --all-files'

       You can see all the options by adding --help EXTRA_ARG:

       'breeze static-check mypy -- --help'


 ####################################################################################################


 Detailed usage for command: tests


 breeze tests [FLAGS] [TEST_TARGET ..] [-- <EXTRA_ARGS>]

       Run the specified unit test target. There might be multiple
       targets specified separated with comas. The <EXTRA_ARGS> passed after -- are treated
       as additional options passed to pytest. You can pass 'tests' as target to
       run all tests. For example:

       'breeze tests tests/test_core.py -- --logging-level=DEBUG'
       'breeze tests tests

 Flags:

 --test-type TEST_TYPE
         Type of the test to run. One of:

                All,Core,Providers,API,CLI,Integration,Other,WWW,Heisentests,Postgres,MySQL

         Default: All


 ####################################################################################################


 Detailed usage for command: flags


       Explains in detail all the flags that can be used with breeze.


 ####################################################################################################


 Detailed usage for command: help


 breeze help

       Shows general help message for all commands.


 ####################################################################################################


 Detailed usage for command: help-all


 breeze help-all

       Shows detailed help for all commands and flags.


 ####################################################################################################


 ####################################################################################################

 Summary of all flags supported by Breeze:

 ****************************************************************************************************
  Choose Airflow variant

 -p, --python PYTHON_MAJOR_MINOR_VERSION
         Python version used for the image. This is always major/minor version.

         Note that versions 2.7 and 3.5 are only valid when installing Airflow 1.10 with
         --install-airflow-version or --install-airflow-reference flags.

         One of:

                2.7 3.5 3.6 3.7 3.8

 ****************************************************************************************************
  Choose backend to run for Airflow

 -b, --backend BACKEND
         Backend to use for tests - it determines which database is used.
         One of:

                sqlite mysql postgres

         Default: sqlite

 --postgres-version POSTGRES_VERSION
         Postgres version used. One of:

                9.6 10

 --mysql-version MYSQL_VERSION
         Mysql version used. One of:

                5.7 8

 ****************************************************************************************************
  Enable production image

 -I, --production-image
         Use production image for entering the environment and builds (not for tests).

 ****************************************************************************************************
  Additional actions executed while entering breeze

 -d, --db-reset
         Resets the database at entry to the environment. It will drop all the tables
         and data and recreate the DB from scratch even if 'restart' command was not used.
         Combined with 'restart' command it enters the environment in the state that is
         ready to start Airflow webserver/scheduler/worker. Without the switch, the database
         does not have any tables and you need to run reset db manually.

 -i, --integration INTEGRATION
         Integration to start during tests - it determines which integrations are started
         for integration tests. There can be more than one integration started, or all to
         start all integrations. Selected integrations are not saved for future execution.
         One of:

                cassandra kerberos mongo openldap presto rabbitmq redis all

 --init-script INIT_SCRIPT_FILE
         Initialization script name - Sourced from files/airflow-breeze-config. Default value
         init.sh. It will be executed after the environment is configured and started.

 ****************************************************************************************************
  Additional actions executed starting Airflow

 --load-example-dags
         Include Airflow example dags.

 --load-default-connections
         Include Airflow Default Connections.

 ****************************************************************************************************
  Kind kubernetes and Kubernetes tests configuration(optional)

 Configuration for the KinD Kubernetes cluster and tests:

 -K, --kubernetes-mode KUBERNETES_MODE
         Kubernetes mode - only used in case one of kind-cluster commands is used.
         One of:

                image

         Default: image

 -V, --kubernetes-version KUBERNETES_VERSION
         Kubernetes version - only used in case one of kind-cluster commands is used.
         One of:

                v1.18.6 v1.17.5 v1.16.9

         Default: v1.18.6

 --kind-version KIND_VERSION
         Kind version - only used in case one of kind-cluster commands is used.
         One of:

                v0.8.0

         Default: v0.8.0

 --helm-version HELM_VERSION
         Helm version - only used in case one of kind-cluster commands is used.
         One of:

                v3.2.4

         Default: v3.2.4

 ****************************************************************************************************
  Manage mounting local files

 -l, --skip-mounting-local-sources
         Skips mounting local volume with sources - you get exactly what is in the
         docker image rather than your current local sources of Airflow.

 ****************************************************************************************************
  Assume answers to questions

 -y, --assume-yes
         Assume 'yes' answer to all questions.

 -n, --assume-no
         Assume 'no' answer to all questions.

 -q, --assume-quit
         Assume 'quit' answer to all questions.

 ****************************************************************************************************
  Choose different Airflow version to install or run

 -a, --install-airflow-version INSTALL_AIRFLOW_VERSION
         If specified, installs Airflow directly from PIP released version. This happens at
         image building time in production image and at container entering time for CI image. One of:

                1.10.12 1.10.11 1.10.10 1.10.9 1.10.8 1.10.7 1.10.6 1.10.5 1.10.4 1.10.3 1.10.2
                master v1-10-test

 -t, --install-airflow-reference INSTALL_AIRFLOW_REFERENCE
         If specified, installs Airflow directly from reference in GitHub. This happens at
         image building time in production image and at container entering time for CI image.

 --no-rbac-ui
         Disables RBAC UI when Airflow 1.10.* is installed.

 --install-wheels
         If specified it will look for wheel packages placed in dist folder and it will install the
         wheels from there after installing Airflow. This is useful for testing backport
         packages as well as in the future for testing provider packages for 2.0.

 ****************************************************************************************************
  Credentials

 -f, --forward-credentials
         Forwards host credentials to docker container. Use with care as it will make
         your credentials available to everything you install in Docker.

 ****************************************************************************************************
  Flags for building Docker images (both CI and production)

 -F, --force-build-images
         Forces building of the local docker images. The images are rebuilt
         automatically for the first time or when changes are detected in
         package-related files, but you can force it using this flag.

 -P, --force-pull-images
         Forces pulling of images from DockerHub before building to populate cache. The
         images are pulled by default only for the first time you run the
         environment, later the locally build images are used as cache.

 Customization options:

 -E, --extras EXTRAS
         Extras to pass to build images The default are different for CI and production images:

         CI image:
                devel_ci

         Production image:
                async,aws,azure,celery,dask,elasticsearch,gcp,kubernetes,mysql,postgres,redis,slack,
                ssh,statsd,virtualenv

 --image-tag TAG
         Additional tag in the image.

 --additional-extras ADDITIONAL_EXTRAS
         Additional extras to pass to build images The default is no additional extras.

 --additional-python-deps ADDITIONAL_PYTHON_DEPS
         Additional python dependencies to use when building the images.

 --dev-apt-command DEV_APT_COMMAND
         The basic command executed before dev apt deps are installed.

 --additional-dev-apt-command ADDITIONAL_DEV_APT_COMMAND
         Additional command executed before dev apt deps are installed.

 --additional-dev-apt-deps ADDITIONAL_DEV_APT_DEPS
         Additional apt dev dependencies to use when building the images.

 --dev-apt-deps DEV_APT_DEPS
         The basic apt dev dependencies to use when building the images.

 --additional-dev-apt-deps ADDITIONAL_DEV_DEPS
         Additional apt dev dependencies to use when building the images.

 --additional-dev-apt-envs ADDITIONAL_DEV_APT_ENVS
         Additional environment variables set when adding dev dependencies.

 --runtime-apt-command RUNTIME_APT_COMMAND
         The basic command executed before runtime apt deps are installed.

 --additional-runtime-apt-command ADDITIONAL_RUNTIME_APT_COMMAND
         Additional command executed before runtime apt deps are installed.

 --runtime-apt-deps ADDITIONAL_RUNTIME_APT_DEPS
         The basic apt runtime dependencies to use when building the images.

 --additional-runtime-apt-deps ADDITIONAL_RUNTIME_DEPS
         Additional apt runtime dependencies to use when building the images.

 --additional-runtime-apt-envs ADDITIONAL_RUNTIME_APT_DEPS
         Additional environment variables set when adding runtime dependencies.

 Build options:

 --disable-mysql-client-installation
         Disables installation of the mysql client which might be problematic if you are building
         image in controlled environment. Only valid for production image.

 --constraints-location
         Url to the constraints file. In case of the production image it can also be a path to the
         constraint file placed in 'docker-context-files' folder, in which case it has to be
         in the form of '/docker-context-files/<NAME_OF_THE_FILE>'

 --disable-pip-cache
         Disables GitHub PIP cache during the build. Useful if github is not reachable during build.

 --install-local-pip-wheels
         This flag is only used in production image building. If it is used then instead of
         installing Airflow from PyPI, the packages are installed from the .whl packages placed
         in the 'docker-context-files' folder. It implies '--disable-pip-cache'

 -C, --force-clean-images
         Force build images with cache disabled. This will remove the pulled or build images
         and start building images from scratch. This might take a long time.

 -r, --skip-rebuild-check
         Skips checking image for rebuilds. It will use whatever image is available locally/pulled.

 -L, --build-cache-local
         Uses local cache to build images. No pulled images will be used, but results of local
         builds in the Docker cache are used instead. This will take longer than when the pulled
         cache is used for the first time, but subsequent '--build-cache-local' builds will be
         faster as they will use mostly the locally build cache.

         This is default strategy used by the Production image builds.

 -U, --build-cache-pulled
         Uses images pulled from registry (either DockerHub or GitHub depending on
         --github-registry flag) to build images. The pulled images will be used as cache.
         Those builds are usually faster than when ''--build-cache-local'' with the exception if
         the registry images are not yet updated. The DockerHub images are updated nightly and the
         GitHub images are updated after merges to master so it might be that the images are still
         outdated vs. the latest version of the Dockerfiles you are using. In this case, the
         ''--build-cache-local'' might be faster, especially if you iterate and change the
         Dockerfiles yourself.

         This is default strategy used by the CI image builds.

 -X, --build-cache-disabled
         Disables cache during docker builds. This is useful if you want to make sure you want to
         rebuild everything from scratch.

         This strategy is used by default for both Production and CI images for the scheduled
         (nightly) builds in CI.

 ****************************************************************************************************
  Flags for pulling/pushing Docker images (both CI and production)

 -D, --dockerhub-user DOCKERHUB_USER
         DockerHub user used to pull, push and build images. Default: apache.

 -H, --dockerhub-repo DOCKERHUB_REPO
         DockerHub repository used to pull, push, build images. Default: airflow.

 -c, --github-registry GITHUB_REGISTRY
         If GitHub registry is enabled, pulls and pushes are done from the GitHub registry not
         DockerHub. You need to be logged in to the registry in order to be able to pull/push from
         and you need to be committer to push to Apache Airflow' GitHub registry.

 -g, --github-repository GITHUB_REPOSITORY
         GitHub repository used to pull, push images when cache is used.
         Default: apache/airflow.

         If you use this flag, automatically --github-registry flag is enabled.

 -s, --github-image-id COMMIT_SHA|RUN_ID
         <RUN_ID> or <COMMIT_SHA> of the image. Images in GitHub registry are stored with those
         to be able to easily find the image for particular CI runs. Once you know the
         <RUN_ID> or <COMMIT_SHA>, you can specify it in github-image-id flag and Breeze will
         automatically pull and use that image so that you can easily reproduce a problem
         that occurred in CI.

         If you use this flag, automatically --github-registry is enabled.

         Default: latest.

 ****************************************************************************************************
  Flags for running tests

 --test-type TEST_TYPE
         Type of the test to run. One of:

                All,Core,Providers,API,CLI,Integration,Other,WWW,Heisentests,Postgres,MySQL

         Default: All

 ****************************************************************************************************
  Flags for generation of the backport packages

 -S, --version-suffix-for-pypi SUFFIX
         Adds optional suffix to the version in the generated backport package. It can be used
         to generate rc1/rc2 ... versions of the packages to be uploaded to PyPI.

 -N, --version-suffix-for-svn SUFFIX
         Adds optional suffix to the generated names of package. It can be used to generate
         rc1/rc2 ... versions of the packages to be uploaded to SVN.

 ****************************************************************************************************
  Increase verbosity of the scripts

 -v, --verbose
         Show verbose information about executed docker, kind, kubectl, helm commands. Useful for
         debugging - when you run breeze with --verbose flags you will be able to see the commands
         executed under the hood and copy&paste them to your terminal to debug them more easily.

         Note that you can further increase verbosity and see all the commands executed by breeze
         by running 'export VERBOSE_COMMANDS="true"' before running breeze.

 ****************************************************************************************************
  Print detailed help message

 -h, --help
         Shows detailed help message for the command specified.

.. END BREEZE HELP MARKER