Skip to content

Commit

Permalink
chore: fix case of GitHub (apache#14525)
Browse files Browse the repository at this point in the history
  • Loading branch information
jbampton authored Feb 28, 2021
1 parent 7d18150 commit 1c69431
Show file tree
Hide file tree
Showing 15 changed files with 43 additions and 43 deletions.
6 changes: 3 additions & 3 deletions BREEZE.rst
Original file line number Diff line number Diff line change
Expand Up @@ -1467,7 +1467,7 @@ This is the current syntax for `./breeze <./breeze>`_:
and you need to be committer to push to Apache Airflow' GitHub registry.
--github-registry GITHUB_REGISTRY
Github registry used. GitHub has legacy Packages registry and Public Beta Container
GitHub registry used. GitHub has legacy Packages registry and Public Beta Container
registry.
Default: docker.pkg.github.com.
Expand Down Expand Up @@ -1661,7 +1661,7 @@ This is the current syntax for `./breeze <./breeze>`_:
and you need to be committer to push to Apache Airflow' GitHub registry.
--github-registry GITHUB_REGISTRY
Github registry used. GitHub has legacy Packages registry and Public Beta Container
GitHub registry used. GitHub has legacy Packages registry and Public Beta Container
registry.
Default: docker.pkg.github.com.
Expand Down Expand Up @@ -2705,7 +2705,7 @@ This is the current syntax for `./breeze <./breeze>`_:
and you need to be committer to push to Apache Airflow' GitHub registry.
--github-registry GITHUB_REGISTRY
Github registry used. GitHub has legacy Packages registry and Public Beta Container
GitHub registry used. GitHub has legacy Packages registry and Public Beta Container
registry.
Default: docker.pkg.github.com.
Expand Down
20 changes: 10 additions & 10 deletions CI.rst
Original file line number Diff line number Diff line change
Expand Up @@ -101,18 +101,18 @@ We can use either of the two available GitHub Container registries as cache:
Those two images have different naming schemas. See `Images documentation <IMAGES.rst>`_ for details.

You can choose which registry should be used by the repository by setting ``OVERRIDE_GITHUB_REGISTRY`` secret
to either ``docker.pkg.github.com`` for Github Package Registry or ``ghcr.io`` for GitHub Container Registry.
Default is the Github Package Registry one. The Pull Request forks have no access to the secret but they
to either ``docker.pkg.github.com`` for GitHub Package Registry or ``ghcr.io`` for GitHub Container Registry.
Default is the GitHub Package Registry one. The Pull Request forks have no access to the secret but they
auto-detect the registry used when they wait for the images.

You can interact with the Github Registry images (pull/push) via `Breeze <BREEZE.rst>`_ - you can
pass ``--github-registry`` flag wih either ``docker.pkg.github.com`` for Github Package Registry or
You can interact with the GitHub Registry images (pull/push) via `Breeze <BREEZE.rst>`_ - you can
pass ``--github-registry`` flag wih either ``docker.pkg.github.com`` for GitHub Package Registry or
``ghcr.io`` for GitHub Container Registry and pull/push operations will be performed using the chosen
registry, using appropriate naming convention. This allows building and pushing the images locally by
committers who have access to push/pull those images.


Github Container Registry Token
GitHub Container Registry Token
-------------------------------

Unlike GitHub Packages, GitHub Registry requires a personal access token added as ``PAT_CR`` secret in order
Expand Down Expand Up @@ -430,17 +430,17 @@ DockerHub registry or change the GitHub registry to interact with and use your o
``GITHUB_REPOSITORY`` and providing your own GitHub Username and Token.

Currently we are using GitHub Packages to cache images for the build. GitHub Packages are "legacy"
storage of binary artifacts for GitHub and as of September 2020 they introduced Github Container Registry
storage of binary artifacts for GitHub and as of September 2020 they introduced GitHub Container Registry
as more stable, easier to manage replacement for container storage. It includes complete self-management
of the images including permission management, public access, retention management and many more.

More about it here:

https://github.blog/2020-09-01-introducing-github-container-registry/

Recently we started to experience unstable behaviour of the Github Packages ('unknown blob'
Recently we started to experience unstable behaviour of the GitHub Packages ('unknown blob'
and manifest v1 vs. v2 when pushing images to it). So together with ASF we proposed to
enable Github Container Registry and it happened as of January 2020.
enable GitHub Container Registry and it happened as of January 2020.

More about it in https://issues.apache.org/jira/browse/INFRA-20959

Expand Down Expand Up @@ -486,10 +486,10 @@ the model of permission management is not the same for Container Registry as it
| | | default, can be commit SHA or RUN_ID. |
+--------------------------------+---------------------------+----------------------------------------------+

Authentication in Github Registry
Authentication in GitHub Registry
=================================

We are currently in the process of testing using Github Container Registry as cache for our images during
We are currently in the process of testing using GitHub Container Registry as cache for our images during
the CI process. The default registry is set to "GitHub Packages", but we are testing the GitHub
Container Registry. In case of GitHub Packages, authentication uses GITHUB_TOKEN mechanism. Authentication
is needed for both pushing the images (WRITE) and pulling them (READ) - which means that GitHub token
Expand Down
4 changes: 2 additions & 2 deletions COMMITTERS.rst
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ Prerequisites
General prerequisites that we look for in all candidates:

1. Consistent contribution over last few months
2. Visibility on discussions on the dev mailing list, Slack channels or Github issues/discussions
2. Visibility on discussions on the dev mailing list, Slack channels or GitHub issues/discussions
3. Contributions to community health and project's sustainability for the long-term
4. Understands contributor/committer guidelines:
`Contributors' Guide <https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst>`__
Expand Down Expand Up @@ -127,7 +127,7 @@ Community involvement
* Growing the community:

* Mentors new members/contributors
* Answers users/contributors via Github issues, dev list or slack
* Answers users/contributors via GitHub issues, dev list or slack

Code contribution
^^^^^^^^^^^^^^^^^^
Expand Down
4 changes: 2 additions & 2 deletions CONTRIBUTORS_QUICK_START.rst
Original file line number Diff line number Diff line change
Expand Up @@ -750,7 +750,7 @@ tests are applied when you commit your code.

<div align="center" style="padding-bottom:20px">
<img src="images/quick_start/ci_tests.png"
alt="CI tests Github">
alt="CI tests GitHub">
</div>


Expand Down Expand Up @@ -939,7 +939,7 @@ Contribution guide
Raising Pull Request
--------------------

1. Go to your Github account and open your fork project and click on Branches
1. Go to your GitHub account and open your fork project and click on Branches

.. raw:: html

Expand Down
6 changes: 3 additions & 3 deletions IMAGES.rst
Original file line number Diff line number Diff line change
Expand Up @@ -229,10 +229,10 @@ Choosing image registry

By default images are pulled and pushed from and to DockerHub registry when you use Breeze's push-image
or build commands. But as described in `CI Documentation <CI.rst>`_, you can choose different image
registry by setting ``GITHUB_REGISTRY`` to ``docker.pkg.github.com`` for Github Package Registry or
registry by setting ``GITHUB_REGISTRY`` to ``docker.pkg.github.com`` for GitHub Package Registry or
``ghcr.io`` for GitHub Container Registry.

Default is the Github Package Registry one. The Pull Request forks have no access to the secret but they
Default is the GitHub Package Registry one. The Pull Request forks have no access to the secret but they
auto-detect the registry used when they wait for the images.

Our images are named like that:
Expand Down Expand Up @@ -271,7 +271,7 @@ image name if DockerHub is not used as registry). Also GitHub has its own struct
each project has its own registry naming convention that should be followed. The name of
images for GitHub registry are different as they must follow limitation of the registry used.

We are still using Github Packages as registry, but we are in the process of testing and switching
We are still using GitHub Packages as registry, but we are in the process of testing and switching
to GitHub Container Registry, and the naming conventions are slightly different (GitHub Packages
required all packages to have "organization/repository/" URL prefix ("apache/airflow/",
where in GitHub Container Registry, all images are in "organization" not in "repository" and they are all
Expand Down
2 changes: 1 addition & 1 deletion ISSUE_TRIAGE_PROCESS.rst
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ the project.
Labels
''''''

Since Apache Airflow uses Github Issues as the issue tracking system, the
Since Apache Airflow uses GitHub Issues as the issue tracking system, the
use of labels is extensive. Though issue labels tend to change over time
based on components within the project, the majority of the ones listed
below should stand the test of time.
Expand Down
10 changes: 5 additions & 5 deletions breeze
Original file line number Diff line number Diff line change
Expand Up @@ -481,7 +481,7 @@ EOF
Branch name: ${BRANCH_NAME}
Docker image: ${AIRFLOW_PROD_IMAGE}
Github cache prefix: ${GITHUB_REGISTRY}/$(get_github_container_registry_image_prefix)
GitHub cache prefix: ${GITHUB_REGISTRY}/$(get_github_container_registry_image_prefix)
Airflow source version: $(build_images::get_airflow_version_from_production_image)
EOF
else
Expand All @@ -491,7 +491,7 @@ EOF
Branch name: ${BRANCH_NAME}
Docker image: ${AIRFLOW_CI_IMAGE}
Github cache prefix: ${GITHUB_REGISTRY}/$(get_github_container_registry_image_prefix)
GitHub cache prefix: ${GITHUB_REGISTRY}/$(get_github_container_registry_image_prefix)
Airflow source version: ${AIRFLOW_VERSION}
EOF
fi
Expand All @@ -516,7 +516,7 @@ EOF
Branch name: ${BRANCH_NAME}
Docker image: ${AIRFLOW_PROD_IMAGE}
Github cache prefix: ${GITHUB_REGISTRY}/$(get_github_container_registry_image_prefix)
GitHub cache prefix: ${GITHUB_REGISTRY}/$(get_github_container_registry_image_prefix)
EOF
else
cat <<EOF
Expand All @@ -525,7 +525,7 @@ EOF
Branch name: ${BRANCH_NAME}
Docker image: ${AIRFLOW_CI_IMAGE}
Github cache prefix: ${GITHUB_REGISTRY}/$(get_github_container_registry_image_prefix)
GitHub cache prefix: ${GITHUB_REGISTRY}/$(get_github_container_registry_image_prefix)
EOF
fi
Expand Down Expand Up @@ -2728,7 +2728,7 @@ function breeze::flag_pull_push_docker_images() {
and you need to be committer to push to Apache Airflow' GitHub registry.
--github-registry GITHUB_REGISTRY
Github registry used. GitHub has legacy Packages registry and Public Beta Container
GitHub registry used. GitHub has legacy Packages registry and Public Beta Container
registry.
Default: ${_breeze_default_github_registry:=}.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,7 @@ In some cases, URI's you will need stored in Secrets Manager may not be intuitiv
This is a known situation, where schema and protocol parts of the URI are independent and in some cases, need to be specified explicitly.

See Github issue `#10256 <https://github.com/apache/airflow/pull/10256>`__ and `#10913 <https://github.com/apache/airflow/issues/10913>`__ for more detailed discussion that led to this documentation update. This may get resolved in the future.
See GitHub issue `#10256 <https://github.com/apache/airflow/pull/10256>`__ and `#10913 <https://github.com/apache/airflow/issues/10913>`__ for more detailed discussion that led to this documentation update. This may get resolved in the future.

Storing and Retrieving Variables
""""""""""""""""""""""""""""""""
Expand Down
14 changes: 7 additions & 7 deletions scripts/ci/libraries/_build_images.sh
Original file line number Diff line number Diff line change
Expand Up @@ -402,16 +402,16 @@ function build_images::get_docker_image_names() {
export BUILT_CI_IMAGE_FLAG_FILE="${BUILD_CACHE_DIR}/${BRANCH_NAME}/.built_${PYTHON_MAJOR_MINOR_VERSION}"

# This is 1-1 mapping of image names of Apache Airflow stored in DockerHub vs. the same images stored
# in Github Registries (either Github Container Registry or Github Packages)
# in GitHub Registries (either GitHub Container Registry or GitHub Packages)
#
# We have to apply naming conventions used by the registries and keep multiple RUN_ID tags. We use
# common suffix ('gcr-v1') to be able to switch to different set of cache images if needed
# - for example when some images gets broken (might happen with Github Actions Registries) or when
# - for example when some images gets broken (might happen with GitHub Actions Registries) or when
# the storage capacity per image is reached (though it is apparently unlimited)
#
# Some examples:
#
# In case of Github Container Registry:
# In case of GitHub Container Registry:
#
# * Prod Image: "apache/airflow:master-python3.8" -> "apache/airflow-master-python3.8-gcr-v1:<RUN_ID>"
# * Prod build image: "apache/airflow:master-python3.8-build" -> "apache/airflow-master-python3.8-build-gcr-v1:<RUN_ID>"
Expand All @@ -422,7 +422,7 @@ function build_images::get_docker_image_names() {
#
# "apache/airflow:python-3.6 -> "apache/airflow-python-gcr-v1:3.6-slim-buster-<RUN_ID>"
#
# In case of Github Packages image must be part of the repository:
# In case of GitHub Packages image must be part of the repository:
#
# * Prod Image: "apache/airflow:master-python3.8" -> "apache/airflow/master-python3.8-gcr-v1:<RUN_ID>"
# * Prod build image: "apache/airflow:master-python3.8-build" -> "apache/airflow/master-python3.8-build-gcr-v1:<RUN_ID>"
Expand Down Expand Up @@ -462,7 +462,7 @@ function build_images::get_docker_image_names() {
# Also enable experimental features of docker (we need `docker manifest` command)
function build_image::configure_github_docker_registry() {
if [[ ${USE_GITHUB_REGISTRY} == "true" ]]; then
start_end::group_start "Determine Github Registry token used and login if needed"
start_end::group_start "Determine GitHub Registry token used and login if needed"
local token=""
if [[ "${GITHUB_REGISTRY}" == "ghcr.io" ]]; then
# For now ghcr.io can only authenticate using Personal Access Token with package access scope.
Expand All @@ -484,7 +484,7 @@ function build_image::configure_github_docker_registry() {
fi
if [[ -z "${token}" ]] ; then
verbosity::print_info
verbosity::print_info "Skip logging in to Github Registry. No Token available!"
verbosity::print_info "Skip logging in to GitHub Registry. No Token available!"
verbosity::print_info
fi
if [[ -n "${token}" ]]; then
Expand Down Expand Up @@ -653,7 +653,7 @@ function build_images::rebuild_ci_image_if_needed_and_confirmed() {
fi
}

# Retrieves Github Container Registry image prefix from repository name
# Retrieves GitHub Container Registry image prefix from repository name
# GitHub Container Registry stores all images at the organization level, they are just
# linked to the repository via docker label - however we assume a convention where we will
# add repository name to organisation separated by '-' and convert everything to lowercase
Expand Down
4 changes: 2 additions & 2 deletions scripts/ci/libraries/_push_pull_remove_images.sh
Original file line number Diff line number Diff line change
Expand Up @@ -297,7 +297,7 @@ function push_pull_remove_images::check_for_image_in_github_packages() {
local image_to_wait_for=${GITHUB_REPOSITORY}/${image_name_in_github_registry}:${image_tag_in_github_registry}
local github_api_call
github_api_call="${github_api_endpoint}/${image_name_in_github_registry}/manifests/${image_tag_in_github_registry}"
echo "Github Packages: checking for ${image_to_wait_for} via ${github_api_call}!"
echo "GitHub Packages: checking for ${image_to_wait_for} via ${github_api_call}!"
http_status=$(curl --silent --output "${OUTPUT_LOG}" --write-out "%{http_code}" \
--connect-timeout 60 --max-time 60 \
-X GET "${github_api_call}" -u "${GITHUB_USERNAME}:${GITHUB_TOKEN}")
Expand All @@ -321,7 +321,7 @@ function push_pull_remove_images::check_for_image_in_github_container_registry()
local image_tag_in_github_registry=${2}

local image_to_wait_for="ghcr.io/${GITHUB_REPOSITORY}-${image_name_in_github_registry}:${image_tag_in_github_registry}"
echo "Github Container Registry: checking for ${image_to_wait_for} via docker manifest inspect!"
echo "GitHub Container Registry: checking for ${image_to_wait_for} via docker manifest inspect!"
docker manifest inspect "${image_to_wait_for}"
local res=$?
if [[ ${res} == "0" ]]; then
Expand Down
2 changes: 1 addition & 1 deletion scripts/ci/libraries/_script_init.sh
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@ initialization::make_constants_read_only
start_end::group_end

# Work around occasional unexplained failure on CI. Clear file flags on
# STDOUT (which is connected to a tmp file by Github Runner).
# STDOUT (which is connected to a tmp file by GitHub Runner).
# The one error I did see: BlockingIOError: [Errno 11] write could not complete without blocking
[[ "$CI" == "true" ]] && python3 -c "import fcntl; fcntl.fcntl(1, fcntl.F_SETFL, 0)"

Expand Down
4 changes: 2 additions & 2 deletions scripts/ci/libraries/_start_end.sh
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
# specific language governing permissions and limitations
# under the License.

# Starts group for Github Actions - makes logs much more readable
# Starts group for GitHub Actions - makes logs much more readable
function start_end::group_start {
if [[ ${PRINT_INFO_FROM_SCRIPTS} != "false" ]]; then
if [[ ${GITHUB_ACTIONS=} == "true" ]]; then
Expand All @@ -29,7 +29,7 @@ function start_end::group_start {
fi
}

# Ends group for Github Actions
# Ends group for GitHub Actions
function start_end::group_end {
if [[ ${PRINT_INFO_FROM_SCRIPTS} != "false" ]]; then
if [[ ${GITHUB_ACTIONS=} == "true" ]]; then
Expand Down
4 changes: 2 additions & 2 deletions scripts/in_container/_in_container_utils.sh
Original file line number Diff line number Diff line change
Expand Up @@ -633,7 +633,7 @@ function get_providers_to_act_on() {
group_end
}

# Starts group for Github Actions - makes logs much more readable
# Starts group for GitHub Actions - makes logs much more readable
function group_start {
if [[ ${GITHUB_ACTIONS=} == "true" ]]; then
echo "::group::${1}"
Expand All @@ -644,7 +644,7 @@ function group_start {
fi
}

# Ends group for Github Actions
# Ends group for GitHub Actions
function group_end {
if [[ ${GITHUB_ACTIONS=} == "true" ]]; then
echo -e "\033[0m" # Disable any colors set in the group
Expand Down
2 changes: 1 addition & 1 deletion scripts/in_container/check_environment.sh
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ DISABLED_INTEGRATIONS=""
# We want to avoid misleading messages and perform only forward lookup of the service IP address.
# Netcat when run without -n performs both forward and reverse lookup and fails if the reverse
# lookup name does not match the original name even if the host is reachable via IP. This happens
# randomly with docker-compose in Github Actions.
# randomly with docker-compose in GitHub Actions.
# Since we are not using reverse lookup elsewhere, we can perform forward lookup in python
# And use the IP in NC and add '-n' switch to disable any DNS use.
# Even if this message might be harmless, it might hide the real reason for the problem
Expand Down
2 changes: 1 addition & 1 deletion scripts/in_container/prod/entrypoint_prod.sh
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ set -euo pipefail
# We want to avoid misleading messages and perform only forward lookup of the service IP address.
# Netcat when run without -n performs both forward and reverse lookup and fails if the reverse
# lookup name does not match the original name even if the host is reachable via IP. This happens
# randomly with docker-compose in Github Actions.
# randomly with docker-compose in GitHub Actions.
# Since we are not using reverse lookup elsewhere, we can perform forward lookup in python
# And use the IP in NC and add '-n' switch to disable any DNS use.
# Even if this message might be harmless, it might hide the real reason for the problem
Expand Down

0 comments on commit 1c69431

Please sign in to comment.