forked from apache/airflow
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Add Codespaces support (apache#22082)
* Add Codespaces support Codespaces is (soon to be GA) feature that allows to start development environment on a virtual machine directly from the GitHub UI or from VSCode connected with GitHub account. This PR adds configuration that allows to start codespaces environment (based on Breeze's docker compose environment) in the basic form that allows the users to run basic unit tests. It does not replace Breeze/CI but for most users who want to just run a few tests, make sure that all dependencies and plugins are installed so that the users can get the VSCode autocomplete and execution of the tests possible. This is a great way to begin your journey with Airflow - while the Codespace machines are not very powerful and the integration tests might require more resources, the "no-setup" environment and no need to have powerful machine is great for contributors who cannot afford powerful development workstations. * Update CONTRIBUTING.rst Co-authored-by: Ephraim Anierobi <[email protected]> * Update scripts/ci/libraries/_build_images.sh Co-authored-by: Ephraim Anierobi <[email protected]> * Update CONTRIBUTING.rst Co-authored-by: Ephraim Anierobi <[email protected]>
- Loading branch information
1 parent
b1fdcdf
commit 05f3a30
Showing
58 changed files
with
293 additions
and
134 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,24 @@ | ||
| { | ||
| "name": "Apache Airflow", | ||
| "dockerComposeFile": [ | ||
| "../scripts/ci/docker-compose/devcontainer.yml", | ||
| "../scripts/ci/docker-compose/local.yml", | ||
| "../scripts/ci/docker-compose/backend-postgres.yml", | ||
| "../scripts/ci/docker-compose/devcontainer-postgres.yml" | ||
| ], | ||
| "extensions": [ | ||
| "ms-python.python", | ||
| "ms-python.vscode-pylance", | ||
| "mtxr.sqltools", | ||
| "mtxr.sqltools-driver-pg", | ||
| "rogalmic.bash-debug", | ||
| "ms-azuretools.vscode-docker", | ||
| "dbaeumer.vscode-eslint", | ||
| "ecmel.vscode-html-css", | ||
| "timonwong.shellcheck", | ||
| "redhat.vscode-yaml", | ||
| "rogalmic.bash-debug" | ||
| ], | ||
| "service": "airflow", | ||
| "forwardPorts": [8080,5555,5432,6379] | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,25 @@ | ||
| # Licensed to the Apache Software Foundation (ASF) under one | ||
| # or more contributor license agreements. See the NOTICE file | ||
| # distributed with this work for additional information | ||
| # regarding copyright ownership. The ASF licenses this file | ||
| # to you under the Apache License, Version 2.0 (the | ||
| # "License"); you may not use this file except in compliance | ||
| # with the License. You may obtain a copy of the License at | ||
| # | ||
| # http://www.apache.org/licenses/LICENSE-2.0 | ||
| # | ||
| # Unless required by applicable law or agreed to in writing, | ||
| # software distributed under the License is distributed on an | ||
| # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY | ||
| # KIND, either express or implied. See the License for the | ||
| # specific language governing permissions and limitations | ||
| # under the License. | ||
| --- | ||
| version: "3.7" | ||
| services: | ||
| mysql: | ||
| image: mysql:8.0 | ||
| ports: | ||
| - "3306:3306" | ||
| volumes: | ||
| mysql-db-volume: |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,25 @@ | ||
| # Licensed to the Apache Software Foundation (ASF) under one | ||
| # or more contributor license agreements. See the NOTICE file | ||
| # distributed with this work for additional information | ||
| # regarding copyright ownership. The ASF licenses this file | ||
| # to you under the Apache License, Version 2.0 (the | ||
| # "License"); you may not use this file except in compliance | ||
| # with the License. You may obtain a copy of the License at | ||
| # | ||
| # http://www.apache.org/licenses/LICENSE-2.0 | ||
| # | ||
| # Unless required by applicable law or agreed to in writing, | ||
| # software distributed under the License is distributed on an | ||
| # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY | ||
| # KIND, either express or implied. See the License for the | ||
| # specific language governing permissions and limitations | ||
| # under the License. | ||
| --- | ||
| version: "3.7" | ||
| services: | ||
| postgres: | ||
| image: postgres:10 | ||
| ports: | ||
| - "5432:5432" | ||
| volumes: | ||
| postgres-db-volume: |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,51 @@ | ||
| HOME= | ||
| AIRFLOW_CI_IMAGE="ghcr.io/apache/airflow/main/ci/python3.7:latest" | ||
| PYTHON_MAJOR_MINOR_VERSION="3.7" | ||
| AIRFLOW_EXTRAS= | ||
| BREEZE="true" | ||
| CI="false" | ||
| CI_BUILD_ID= | ||
| CI_JOB_ID= | ||
| CI_EVENT_TYPE= | ||
| CI_TARGET_REPO= | ||
| CI_TARGET_BRANCH= | ||
| COMMIT_SHA= | ||
| DB_RESET="false" | ||
| DEFAULT_BRANCH="main" | ||
| DEFAULT_CONSTRAINTS_BRANCH="constraints-main" | ||
| ENABLED_INTEGRATIONS= | ||
| ENABLED_SYSTEMS= | ||
| ENABLE_TEST_COVERAGE="false" | ||
| GITHUB_ACTIONS="false" | ||
| GITHUB_REGISTRY_PULL_IMAGE_TAG="" | ||
| HOST_USER_ID= | ||
| HOST_GROUP_ID= | ||
| HOST_OS="Linux" | ||
| INIT_SCRIPT_FILE="init.sh" | ||
| INSTALL_AIRFLOW_VERSION= | ||
| GENERATE_CONSTRAINTS_MODE= | ||
| INSTALL_PROVIDERS_FROM_SOURCES= | ||
| USE_AIRFLOW_VERSION= | ||
| USE_PACKAGES_FROM_DIST= | ||
| ISSUE_ID= | ||
| LOAD_DEFAULT_CONNECTIONS= | ||
| LOAD_EXAMPLES= | ||
| MYSQL_VERSION=5.7 | ||
| NUM_RUNS= | ||
| PACKAGE_FORMAT= | ||
| POSTGRES_VERSION=10 | ||
| PRINT_INFO_FROM_SCRIPTS="false" | ||
| PYTHONDONTWRITEBYTECODE="true" | ||
| RUN_TESTS="false" | ||
| LIST_OF_INTEGRATION_TESTS_TO_RUN="" | ||
| RUN_SYSTEM_TESTS="" | ||
| START_AIRFLOW="false" | ||
| SKIP_TWINE_CHECK="false" | ||
| SKIP_SSH_SETUP="true" | ||
| TEST_TYPE= | ||
| UPGRADE_TO_NEWER_DEPENDENCIES="false" | ||
| VERBOSE="false" | ||
| VERBOSE_COMMANDS="false" | ||
| VERSION_SUFFIX_FOR_PYPI= | ||
| VERSION_SUFFIX_FOR_SVN= | ||
| WHEEL_VERSION=0.36.2 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,33 @@ | ||
| # Licensed to the Apache Software Foundation (ASF) under one | ||
| # or more contributor license agreements. See the NOTICE file | ||
| # distributed with this work for additional information | ||
| # regarding copyright ownership. The ASF licenses this file | ||
| # to you under the Apache License, Version 2.0 (the | ||
| # "License"); you may not use this file except in compliance | ||
| # with the License. You may obtain a copy of the License at | ||
| # | ||
| # http://www.apache.org/licenses/LICENSE-2.0 | ||
| # | ||
| # Unless required by applicable law or agreed to in writing, | ||
| # software distributed under the License is distributed on an | ||
| # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY | ||
| # KIND, either express or implied. See the License for the | ||
| # specific language governing permissions and limitations | ||
| # under the License. | ||
| --- | ||
| version: "3.7" | ||
| services: | ||
| airflow: | ||
| image: ghcr.io/apache/airflow/main/ci/python3.7 | ||
| env_file: devcontainer.env | ||
| ports: | ||
| - "22:22" | ||
| - "8080:8080" | ||
| - "5555:5555" | ||
| - "6379:6379" | ||
| cap_add: | ||
| - SYS_PTRACE | ||
| volumes: | ||
| # Pass docker to inside of the container so that Kind and Moto tests can use it. | ||
| - /var/run/docker.sock:/var/run/docker.sock | ||
| - /dev/urandom:/dev/random # Required to get non-blocking entropy source |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -157,24 +157,26 @@ mkdir -p /usr/lib/google-cloud-sdk/bin | |
| touch /usr/lib/google-cloud-sdk/bin/gcloud | ||
| ln -s -f /usr/bin/gcloud /usr/lib/google-cloud-sdk/bin/gcloud | ||
|
|
||
| # Set up ssh keys | ||
| echo 'yes' | ssh-keygen -t rsa -C [email protected] -m PEM -P '' -f ~/.ssh/id_rsa \ | ||
| >"${AIRFLOW_HOME}/logs/ssh-keygen.log" 2>&1 | ||
|
|
||
| cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys | ||
| ln -s -f ~/.ssh/authorized_keys ~/.ssh/authorized_keys2 | ||
| chmod 600 ~/.ssh/* | ||
|
|
||
| # SSH Service | ||
| sudo service ssh restart >/dev/null 2>&1 | ||
|
|
||
| # Sometimes the server is not quick enough to load the keys! | ||
| while [[ $(ssh-keyscan -H localhost 2>/dev/null | wc -l) != "3" ]] ; do | ||
| echo "Not all keys yet loaded by the server" | ||
| sleep 0.05 | ||
| done | ||
| if [[ ${SKIP_SSH_SETUP="false"} == "false" ]]; then | ||
| # Set up ssh keys | ||
| echo 'yes' | ssh-keygen -t rsa -C [email protected] -m PEM -P '' -f ~/.ssh/id_rsa \ | ||
| >"${AIRFLOW_HOME}/logs/ssh-keygen.log" 2>&1 | ||
|
|
||
| cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys | ||
| ln -s -f ~/.ssh/authorized_keys ~/.ssh/authorized_keys2 | ||
| chmod 600 ~/.ssh/* | ||
|
|
||
| # SSH Service | ||
| sudo service ssh restart >/dev/null 2>&1 | ||
|
|
||
| # Sometimes the server is not quick enough to load the keys! | ||
| while [[ $(ssh-keyscan -H localhost 2>/dev/null | wc -l) != "3" ]] ; do | ||
| echo "Not all keys yet loaded by the server" | ||
| sleep 0.05 | ||
| done | ||
|
|
||
| ssh-keyscan -H localhost >> ~/.ssh/known_hosts 2>/dev/null | ||
| ssh-keyscan -H localhost >> ~/.ssh/known_hosts 2>/dev/null | ||
| fi | ||
|
|
||
| # shellcheck source=scripts/in_container/configure_environment.sh | ||
| . "${IN_CONTAINER_DIR}/configure_environment.sh" | ||
|
|
@@ -283,11 +285,12 @@ else | |
| "tests/utils" | ||
| ) | ||
| WWW_TESTS=("tests/www") | ||
| HELM_CHART_TESTS=("chart/tests") | ||
| HELM_CHART_TESTS=("tests/charts") | ||
| ALL_TESTS=("tests") | ||
| ALL_PRESELECTED_TESTS=( | ||
| "${CLI_TESTS[@]}" | ||
| "${API_TESTS[@]}" | ||
| "${HELM_CHART_TESTS[@]}" | ||
| "${PROVIDERS_TESTS[@]}" | ||
| "${CORE_TESTS[@]}" | ||
| "${ALWAYS_TESTS[@]}" | ||
|
|
||
File renamed without changes.
Oops, something went wrong.