Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[AIRFLOW-5842] Switch to Debian buster image. Depends on [AIRFLOW-5704] #6499

Closed
wants to merge 4 commits into from
Closed

[AIRFLOW-5842] Switch to Debian buster image. Depends on [AIRFLOW-5704] #6499

wants to merge 4 commits into from

Conversation

potiuk
Copy link
Member

@potiuk potiuk commented Nov 4, 2019 •
edited
Loading

NOTE FOR REVIEWER: This change depends on #6373 . Please check only last commit.

As part of the production migration we switch to buster image
which is latest released LTS version.

Jira

Description

  • Here are some details about my PR, including screenshots of any UI changes:

Tests

  • My PR adds the following unit tests OR does not need testing for this extremely good reason:

Commits

  • My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":
    1. Subject is separated from body by a blank line
    2. Subject is limited to 50 characters (not including Jira issue reference)
    3. Subject does not end with a period
    4. Subject uses the imperative mood ("add", not "adding")
    5. Body wraps at 72 characters
    6. Body explains "what" and "why", not "how"

Documentation

  • In case of new functionality, my PR adds documentation that describes how to use it.
    • All the public functions and the classes in the PR contain docstrings that explain what it does
    • If you implement backwards incompatible changes, please leave a note in the Updating.md so we can assign it to a appropriate release

@potiuk potiuk added AIP-21 and removed AIP-21 labels Nov 4, 2019
@potiuk potiuk changed the title [AIRFLOW-5842] Switch to Debian buster image. Depends on [AIRFLOW-5830] [AIRFLOW-5829] [AIRFLOW-5704] … [AIRFLOW-5842] Switch to Debian buster image. Depends on [AIRFLOW-5830] [AIRFLOW-5829] [AIRFLOW-5704] Nov 4, 2019
@potiuk potiuk mentioned this pull request Nov 4, 2019
Closed
5 tasks
@potiuk potiuk force-pushed the switch-to-debian-buster-image branch 7 times, most recently from 669795c to f7c78bf Compare November 6, 2019 11:04
@potiuk potiuk changed the title [AIRFLOW-5842] Switch to Debian buster image. Depends on [AIRFLOW-5830] [AIRFLOW-5829] [AIRFLOW-5704] [AIRFLOW-5842] Switch to Debian buster image. Depends on [AIRFLOW-5704] Nov 6, 2019
@potiuk potiuk mentioned this pull request Nov 7, 2019
Closed
5 tasks
@potiuk potiuk force-pushed the switch-to-debian-buster-image branch from f7c78bf to c86f05f Compare November 12, 2019 22:30
@potiuk potiuk added the area:dev-env CI, pre-commit, pylint and other changes that do not change the behavior of the final code label Nov 12, 2019
@potiuk potiuk self-assigned this Nov 12, 2019
mik-laj
mik-laj reviewed Nov 14, 2019
View reviewed changes
Dockerfile
# In case of CI builds we want to pre-install master version of airflow dependencies so that
# We do not have to always reinstall it from the scratch and loose time for that.
# CI build is optimised for build speed
RUN pip install --user \
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we add --no-cache-dir here?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh. I have now seen that you set environment variables on the 55 line, but maybe you should delete the variables and make it more implicit? It also remove 1 docker layer.

@potiuk potiuk force-pushed the switch-to-debian-buster-image branch 2 times, most recently from b76fc9d to 828dd8e Compare November 17, 2019 23:53
@stale
Copy link

stale bot commented Jan 2, 2020

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@stale stale bot added the stale Stale PRs per the .github/workflows/stale.yml policy file label Jan 2, 2020
@potiuk
Copy link
Member Author

potiuk commented Jan 2, 2020

Please don't

@stale stale bot removed the stale Stale PRs per the .github/workflows/stale.yml policy file label Jan 2, 2020
feluelle
feluelle approved these changes Jan 6, 2020
View reviewed changes
Copy link
Member

@feluelle feluelle left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@potiuk What is the status on this?

BREEZE.rst Outdated
(for example, ``apache/airflow:master-python3.6-ci``). The image is built using the
`<Dockerfile>`_ Dockerfile.
* **Production-ready image** that is used TODO
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we have more information on this?

@potiuk
[AIRFLOW-6489] Add BATS support for Bash unit testing
6cdcc01 
We have far too much bash code around that is not automatically tested.

This is the first step to change it (simplifications and more tests are coming
soon).
@potiuk
Copy link
Member Author

potiuk commented Jan 6, 2020

Getting there shortly! I added the automated test to be able to fix some of the difficulties with bash scripts testing and I am full steam ahead with improving that.

@potiuk
[AIRFLOW-6491] Improve handling of Breeze parameters
00815d4 
While working on improving the way we run Kubernetes tests, we found out that I
need to fix handling of parameters - we change Kubernetes version used via Kind
and the old versions are no longer valid, however it was not properly
removed/saved.

We use the opportunity to add automated tests for that feature.

(cherry picked from commit 38dea9132d1fa36f4fbe871e2ab037be5ad3fab2)
@potiuk potiuk force-pushed the switch-to-debian-buster-image branch from 828dd8e to 1f5e918 Compare January 7, 2020 02:01
@potiuk potiuk force-pushed the switch-to-debian-buster-image branch 2 times, most recently from 8331d5f to e61fa41 Compare January 7, 2020 02:31
@potiuk
[AIRFLOW-5704] Improve Kind Kubernetes scripts for local testing
f765968 
* Fixed problem that Kubernetes tests were testing latest master
  rather than what came from the local sources.
* Moved Kubernetes scripts to 'in_container' dir where they belong now
* Kubernetes tests are now better suited for running locally
* Kubernetes cluster is not deleted until environment is stopped
* Kubernetes image is build from pulled image + mounted sources
* Kubectl version name is corrected in the Dockerfile
* KUBERNETES_VERSION can now be used to select Kubernetes version
* Running kubernetes scripts is now easy in Breeze --start-kind-cluster
* Instructions on how to run Kubernetes tests are updated
* Better flags in Breeze are used to run Kubernetes environment/tests
* The old "bare" environment is replaced by --no-deps switch
@potiuk potiuk force-pushed the switch-to-debian-buster-image branch from e61fa41 to 107fc0e Compare January 7, 2020 02:39
@potiuk
[AIRFLOW-5842] Switch to debian buster image
8053e4b 
As part of the production migration we switch to buster image
which is latest released LTS version.
@potiuk potiuk force-pushed the switch-to-debian-buster-image branch from 107fc0e to 8053e4b Compare January 7, 2020 05:06
@stale
Copy link

stale bot commented Feb 21, 2020

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@stale stale bot added the stale Stale PRs per the .github/workflows/stale.yml policy file label Feb 21, 2020
@stale stale bot closed this Feb 28, 2020
@potiuk potiuk reopened this Mar 1, 2020
@stale stale bot removed the stale Stale PRs per the .github/workflows/stale.yml policy file label Mar 1, 2020
@potiuk
Copy link
Member Author

potiuk commented Mar 7, 2020

Closing as I restarted it from the scratch (and simpler) - #7647

@potiuk potiuk closed this Mar 7, 2020
@potiuk potiuk added the area:production-image Production image improvements and fixes label Apr 27, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mik-laj mik-laj mik-laj approved these changes

@feluelle feluelle feluelle approved these changes

@dimberman dimberman Awaiting requested review from dimberman

Assignees

@potiuk potiuk

Labels
area:dev-env CI, pre-commit, pylint and other changes that do not change the behavior of the final code area:dev-tools area:production-image Production image improvements and fixes
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@potiuk @mik-laj @feluelle