Skip to content

Commit

Permalink
Unify all "breeze" tools under breeze sub-commands (apache#23193)
Browse files Browse the repository at this point in the history
Experimentally we've separated some CI-only tools like freespace
and find-newer-dependencies as separate scripts (similarly to
initialize-virtualenv) but contrary to the initialize, our tools
can and should be run in Breeze's virtualenv.

However we are just in the middle of using breeze commands in
the CI context, the commands are not "alien" any more and
they seem to fit the same pattern as other commands.

Since we now have the possibility of grouping sub-commmands in separate
logical groups, it makes perfect sense to bring the tools to the
breeze's sub-commands - mostly because of the auto-complete
support and familiarity of using breeze commands in CI.

This PR brings both freespace and find-newer-dependencies tools
already used in CI to become breeze sub-commands. It also
adds "fix-ownership" and "resource-check" commands, so that we can run
owership fixing on Linux machines (and adds fixing ownership as
mandatory stop in all breeze-run jobs in CI and that users
can run resource-check locally.

This is because thgere were some cases after introducing Breeze that
some root-owned left-overs prevented to checkout code by the
runner on self-hosted runners.

This was likely due to resource-check that was run without
PYTHONDONTWRITEBYTECODE=1 variable - this variable have been
added now to execution of the python script.
  • Loading branch information
potiuk authored Apr 24, 2022
1 parent f3f0757 commit 87e3733
Show file tree
Hide file tree
Showing 23 changed files with 918 additions and 270 deletions.
12 changes: 9 additions & 3 deletions .github/workflows/build-images.yml
Original file line number Diff line number Diff line change
Expand Up @@ -223,7 +223,7 @@ jobs:
cache-dependency-path: ./dev/breeze/setup*
- run: ./scripts/ci/install_breeze.sh
- name: "Free space"
run: airflow-freespace
run: breeze free-space
- name: Build & Push CI image ${{ env.PYTHON_MAJOR_MINOR_VERSION }}:${{ env.IMAGE_TAG_FOR_THE_BUILD }}
run: breeze build-image --push-image
env:
Expand All @@ -237,7 +237,10 @@ jobs:
IMAGE_TAG: ${{ env.IMAGE_TAG_FOR_THE_BUILD }}
- name: "Candidates for pip resolver backtrack triggers: ${{ matrix.python-version }}"
if: failure() || cancelled()
run: airflow-find-newer-dependencies --max-age 1 --python "${{ matrix.python-version }}"
run: breeze find-newer-dependencies --max-age 1 --python "${{ matrix.python-version }}"
- name: "Fix ownership"
run: breeze fix-ownership
if: always()

build-prod-images:
permissions:
Expand Down Expand Up @@ -310,7 +313,7 @@ jobs:
cache-dependency-path: ./dev/breeze/setup*
- run: ./scripts/ci/install_breeze.sh
- name: "Free space"
run: airflow-freespace
run: breeze free-space
- name: >
Pull CI image for PROD build:
${{ env.PYTHON_MAJOR_MINOR_VERSION }}:${{ env.IMAGE_TAG_FOR_THE_BUILD }}
Expand Down Expand Up @@ -347,3 +350,6 @@ jobs:
run: breeze build-prod-image --cleanup-docker-context-files --push-image --empty-image
env:
IMAGE_TAG: ${{ env.IMAGE_TAG_FOR_THE_BUILD }}
- name: "Fix ownership"
run: breeze fix-ownership
if: always()
Loading

0 comments on commit 87e3733

Please sign in to comment.