Move "prepare-provider-documentation" to Breeze (apache#35586)

This PR moves the functionality of preparing provider documentation from
a python script inside Breeze CI image to breeze Python package.

This is the first of the series of moves that will simplify the way
how provider packages are being built and prepared with the aim of
improving security of the supply chain and make it easier to debug
and modify the release process.

Historically, release process has been run inside of Breeze for
several reasons: isolation of running package preparation from
Host environment, the need to keep separate virtualenv and because
we run verification of provider packages during release process - which
requires the CI environment with all its dependencies.

So far the process looked like this:

* bash breeze parsed the arguments
* bash breeze started the docker bash script with packages as
  parameters
* the bash script in CI image looped over the packages and run python
  prepare_provider_packages.py (twice) to generate docs and
  update changelog (this is interactive process where release manager
  makes decision on bumping versions). Those python script
  performed verification on provider.yaml files
* the bash script summarized the packages and displayed status of
  preparation

However after moving to Python based breeze, we can simplify it all
and run all those steps in Python breeze internal code - no need to
go to docker and use bash scripts. We also do not have to do
verification of provider.yaml files is done already extensively in
pre-commit.

This PR moves all this logic to inside Breeze.

There is stil remainig duplicated code in the original in-container
`prepare_provider_packages.py` remaining, this duplication will be
removed by subsequent PRs where other release management commands
for provider packages will also be moved to Breeze as follow-up
of this PR.

This PR has the following changes:

* move the provider decumentation code from `dev/provider_packages` to
  `dev/breeze/` (and from in-container to in-breeze-venv execution)
* completely removed the intermediate bash script and calling
  Python scripts from it - moving the logic to Breeze entirely
* added better diagnostics of what happens when packages are classified
  with particular types of changes (added `special` style to show it)
* cleaned and clarified `prepare-provider-documentation` commmand
  line flags
* introduce explicit "non-interactive" mode that is used to
  run and test the command in CI and to test it locally
* replace str with Path were files were used in the moved code
* add unit tests covering unit-testable parts of the moved code
* refactore the moved code to use utils available in Breeze
* split the code into packages and versions (reusable utils) and
  specific code for preparing package documentation
* cached provider.yaml information retrieved from providers
* move provider documentation templates to Breeze
* better error handling - they are now regular exceptions in Python
  process rather than interpreting the exit_codes passed from python
  sub-scripts returnig the codes to Bash
* when release manager classifies package, only relevant sections
  are generated (Features/Breaking changes) based on decision and
  changes are automatically "guessed" only if release manager
  chose the section where they would fall in

.github/workflows/ci.yml

@@ -814,7 +814,7 @@ jobs:
         run: rm -fv ./dist/*
       - name: "Prepare provider documentation"
         run: >
-          breeze release-management prepare-provider-documentation
+          breeze release-management prepare-provider-documentation --non-interactive
           ${{ needs.build-info.outputs.affected-providers-list-as-string }}
       - name: "Prepare provider packages: wheel"
         run: >

.pre-commit-config.yaml

@@ -1004,31 +1004,31 @@ repos:
         entry: ./scripts/ci/pre_commit/pre_commit_mypy.py
         files: ^dev/.*\.py$
         require_serial: true
-        additional_dependencies: ['rich>=12.4.4', 'inputimeout', 'pyyaml']
+        additional_dependencies: ['rich>=12.4.4', 'inputimeout', 'pyyaml', 'jsonschema']
       - id: mypy-core
         name: Run mypy for core
         language: python
         entry: ./scripts/ci/pre_commit/pre_commit_mypy.py --namespace-packages
         files: \.py$
         exclude: ^.*/.*_vendor/|^airflow/migrations|^airflow/providers|^dev|^docs|^provider_packages|^tests/providers|^tests/system/providers|^tests/dags/test_imports.py
         require_serial: true
-        additional_dependencies: ['rich>=12.4.4', 'inputimeout', 'pyyaml']
+        additional_dependencies: ['rich>=12.4.4', 'inputimeout', 'pyyaml', 'jsonschema']
       - id: mypy-providers
         name: Run mypy for providers
         language: python
         entry: ./scripts/ci/pre_commit/pre_commit_mypy.py --namespace-packages
         files: ^airflow/providers/.*\.py$|^tests/providers/.*\.py$|^tests/system/providers/.*\.py$
         exclude: ^.*/.*_vendor/
         require_serial: true
-        additional_dependencies: ['rich>=12.4.4', 'inputimeout', 'pyyaml']
+        additional_dependencies: ['rich>=12.4.4', 'inputimeout', 'pyyaml', 'jsonschema']
       - id: mypy-docs
         name: Run mypy for /docs/ folder
         language: python
         entry: ./scripts/ci/pre_commit/pre_commit_mypy.py
         files: ^docs/.*\.py$
         exclude: ^docs/rtd-deprecation
         require_serial: true
-        additional_dependencies: ['rich>=12.4.4', 'inputimeout', 'pyyaml']
+        additional_dependencies: ['rich>=12.4.4', 'inputimeout', 'pyyaml', 'jsonschema']
       - id: check-provider-yaml-valid
         name: Validate provider.yaml files
         entry: ./scripts/ci/pre_commit/pre_commit_check_provider_yaml_files.py

BREEZE.rst

@@ -2047,12 +2047,6 @@ The below example perform documentation preparation for provider packages.
 
      breeze release-management prepare-provider-documentation
 
-By default, the documentation preparation runs package verification to check if all packages are
-importable, but you can add ``--skip-package-verification`` to skip it.
-
-.. code-block:: bash
-
-     breeze release-management prepare-provider-documentation --skip-package-verification
 
 You can also add ``--answer yes`` to perform non-interactive build.
 

dev/README_RELEASE_PROVIDER_PACKAGES.md

@@ -143,7 +143,6 @@ separately this command:
 breeze release-management prepare-provider-documentation qubole
 ```
 
-
 This command will not only prepare documentation but will also help the release manager to review
 changes implemented in all providers, and determine which of the providers should be released. For each
 provider details will be printed on what changes were implemented since the last release including
@@ -176,6 +175,27 @@ breeze release-management prepare-provider-documentation \
  --base-branch provider-cncf-kubernetes/v4-4 cncf.kubernetes
 ```
 
+In case you want to **just** regenerate the documentation because you fixed something in the templates, add
+`--reapply-templates` flag to the command above. This refreshes the content of:
+
+* `__init__.py` in provider's package
+* Provider Commits
+* Provider index for the documentation
+* Provider README file used when publishing package in PyPI
+
+If you want to just update the min airflow version for all packages, you should modify `MIN_AIRFLOW_VERSION`
+in `dev/provider_packages/prepare_provider_packages.py` and run the `prepare-provider-documentation`
+command with the `--only-min-version-update` flag. This will only update the min version in
+the `__init__.py` files and package documentation without bumping the provider versions.
+
+```shell script
+breeze release-management prepare-provider-documentation --only-min-version-update
+```
+
+Note: that this command will only bump the min airflow versions for those providers that do not have it set to
+a higher version. You do not have to skip specific providers - run it for all providers and it will
+handle everything automatically.
+
 ## Open PR with suggested version releases
 
 At this point you should have providers yaml files and changelog updated.

dev/breeze/README.md

@@ -66,6 +66,6 @@ PLEASE DO NOT MODIFY THE HASH BELOW! IT IS AUTOMATICALLY UPDATED BY PRE-COMMIT.
 
 ---------------------------------------------------------------------------------------------------------
 
-Package config hash: abef89e76b6c1cbfe37d4a083a9e75259d0169662c666c9e2549ca91ddf12d9f1274a4c7ab44e999619c0aaf9fdb56f299397e8c528fafbc94caf45f7cc70ad9
+Package config hash: 7b512fa3a81a967c22fc4ccccf052a4c4dbcafd5c014adea775d45f0034d03e1c63d7d1e3df723e93724924ed3cfa92a5848c994c247dfd326c0a6300e282f88
 
 ---------------------------------------------------------------------------------------------------------

dev/breeze/setup.cfg

@@ -55,6 +55,7 @@ package_dir=
     =src
 packages = find:
 install_requires =
+    black>=23.11.0
     click>=8.1.7
     filelock>=3.13.0
     inputimeout>=1.0.4
@@ -71,6 +72,8 @@ install_requires =
     rich>=13.6.0
     rich-click>=1.7.1
     gitpython>=3.1.40
+    semver>=3.0.2
+    tabulate>=0.9.0
     twine>=4.0.2
     wheel>=0.41.3
     setuptools>=68.2.2