Added k9s as integrated tool to help with kubernetes testing (apache#…

…12163)

The K9s is fantastic tool that helps to debug a running k8s
instance. It is terminal-based windowed CLI that makes you
several times more productive comparing to using kubectl
commands. We've integrated k9s (it is run as a docker container
and downloaded on demand). We've also separated out KUBECONFIG
of the integrated kind cluster so that it does not mess with
kubernetes configuration you might already have.

Also - together with that the "surrounding" of the kubernetes
tests were simplified and improved so that the k9s integration
can be utilized well. Instead of kubectl port forwarding (which
caused multitude of problems) we are now utilizing kind's
portMapping feature + custom NodePort resource that maps
port 8080 to 30007 NodePort which in turn maps it to 8080
port of the Webserver. This way we do not have to establish
an external kubectl port forward which is prone to error and
management - everything is brought up when Airflow gets
deployed to the Kind Cluster and shuts down when the Kind
cluster is stopped.

Yet another problem fixed was killing of postgres by one of the
kubernetes tests ('test_integration_run_dag_with_scheduler_failure').
Instead of just killing the scheduler it killed all pods - including
the Postgres one (it was named 'airflow-postgres.*'). That caused
various problems, as the database could be left in a strange state.
I changed the tests to do what it claimed was doing - so killing only the
scheduler during the test. This seemed to improve the stability
of tests immensely in my local setup.

.github/workflows/ci.yml

@@ -765,23 +765,14 @@ jobs:
           python-version: ${{ env.PYTHON_MAJOR_MINOR_VERSION }}
       - name: "Free space"
         run: ./scripts/ci/tools/ci_free_space_on_ci.sh
-      - name: "Setup Kind Cluster ${{ env.KIND_VERSION }}"
-        uses: engineerd/[email protected]
-        with:
-          version: "${{ env.KIND_VERSION }}"
-          name: airflow-python-${{matrix.python-version}}-${{matrix.kubernetes-version}}
-          config: "scripts/ci/kubernetes/kind-cluster-conf.yaml"
       - name: "Prepare PROD Image"
         run: ./scripts/ci/images/ci_prepare_prod_image_on_ci.sh
-      - name: "Deploy airflow to cluster"
-        id: deploy-app
-        run: ./scripts/ci/kubernetes/ci_deploy_app_to_kubernetes.sh
+      - name: "Setup cluster and deploy Airflow"
+        id: setp-cluster-deploy-app
+        run: ./scripts/ci/kubernetes/ci_setup_cluster_and_deploy_airflow_to_kubernetes.sh
         env:
           # We have the right image pulled already by the previous step
           SKIP_BUILDING_PROD_IMAGE: "true"
-          # due to some instabilities, in CI we try to increase port numbers when trying to establish
-          # port forwarding
-          INCREASE_PORT_NUMBER_FOR_KUBERNETES: "true"
       - name: "Cache virtualenv for kubernetes testing"
         uses: actions/cache@v2
         env:
@@ -798,7 +789,7 @@ jobs:
           key: "${{ env.cache-name }}-${{ github.job }}-${{ hashFiles('setup.py') }}\
 -${{ needs.build-info.outputs.defaultKindVersion }}\
 -${{ needs.build-info.outputs.defaultHelmVersion }}\
--$${{ matrix.kubernetes-version }}"
+-${{ matrix.kubernetes-version }}"
       - name: "Kubernetes Tests"
         run: ./scripts/ci/kubernetes/ci_run_kubernetes_tests.sh
       - name: "Upload KinD logs"

BREEZE.rst

@@ -1751,7 +1751,14 @@ This is the current syntax for  `./breeze <./breeze>`_:
         to the cluster so you can also pass appropriate build image flags that will influence
         rebuilding the production image. Operation is one of:
 
-                 start stop restart status deploy test shell
+                 start stop restart status deploy test shell k9s
+
+        The last two operations - shell and k9s allow you to perform interactive testing with
+        kubernetes tests. You can enter the shell from which you can run kubernetes tests and in
+        another terminal you can start the k9s CLI to debug kubernetes instance. It is an easy
+        way to debug the kubernetes deployments.
+
+        You can read more about k9s at https://k9scli.io/
 
   Flags:
 

TESTING.rst

@@ -430,14 +430,22 @@ can also decide to only run tests with ``-m heisentests`` flag to run only those
 Running Tests with Kubernetes
 =============================
 
+Airflow has tests that are run against real kubernetes cluster. We are using
+`Kind <https://kind.sigs.k8s.io/>`_ to create and run the cluster. We integrated the tools to start/stop/
+deploy and run the cluster tests in our repository and into Breeze development environment.
+
+Configuration for the cluster is kept in ``./build/.kube/config`` file in your Airflow source repository and
+our scripts set the ``KUBECONFIG`` variable to it. If you want to interact with the Kind cluster created
+you can do it from outside of the scripts by exporting this variable and point it to this file.
+
 Starting Kubernetes Cluster
 ---------------------------
 
 For your testing you manage Kind cluster with ``kind-cluster`` breeze command:
 
 .. code-block:: bash
 
-    ./breeze kind-cluster [ start | stop | recreate | status | deploy | test | shell ]
+    ./breeze kind-cluster [ start | stop | recreate | status | deploy | test | shell | k9s ]
 
 The command allows you to start/stop/recreate/status Kind Kubernetes cluster, deploy Airflow via Helm
 chart as well as interact with the cluster (via test and shell commands).
@@ -456,11 +464,11 @@ Deploying Airflow to Kubernetes Cluster
 
 Deploying Airflow to the Kubernetes cluster created is also done via ``kind-cluster deploy`` breeze command:
 
-.. code-block:: bash`
+.. code-block:: bash
 
     ./breeze kind-cluster deploy
 
-The deploy commands performs tthose steps:
+The deploy commands performs those steps:
 
 1. It rebuilds the latest ``apache/airflow:master-pythonX.Y`` production images using the
    latest sources using local cachine. It also adds example DAGs to the image, so that they do not
@@ -477,20 +485,63 @@ Running tests with Kubernetes Cluster
 You can either run all tests or you can select which tests to run. You can also enter interactive virtualenv
 to run the tests manually one by one.
 
-.. code-block:: bash
+Running kubernetes tests via shell:
 
-    Running kubernetes tests
+.. code-block:: bash
 
       ./scripts/ci/kubernetes/ci_run_kubernetes_tests.sh                      - runs all kubernetes tests
       ./scripts/ci/kubernetes/ci_run_kubernetes_tests.sh TEST [TEST ...]      - runs selected kubernetes tests (from kubernetes_tests folder)
+
+
+Running kubernetes tests via breeze:
+
+.. code-block:: bash
+
+      ./breeze kind-cluster test
+      ./breeze kind-cluster test -- TEST TEST [TEST ...]
+
+
+Entering shell with Kubernetes Cluster
+--------------------------------------
+
+This shell is prepared to run kubernetes tests interactively. It has ``kubectl`` and ``kind`` cli tools
+available in the path, it has also activated virtualenv environment that allows you to run tests via pytest.
+
+You can enter the shell via those scripts
+
       ./scripts/ci/kubernetes/ci_run_kubernetes_tests.sh [-i|--interactive]   - Activates virtual environment ready to run tests and drops you in
       ./scripts/ci/kubernetes/ci_run_kubernetes_tests.sh [--help]             - Prints this help message
 
 
-You can also run the same tests command with Breeze, using ``kind-cluster test`` command (to run all
-kubernetes tests) and with ``kind-cluster shell`` command you can enter interactive shell when you can
-run tests.
+.. code-block:: bash
+
+      ./breeze kind-cluster shell
+
 
+K9s CLI - debug kubernetes in style!
+------------------------------------
+
+Breeze has built-in integration with fantastic k9s CLI tool, that allows you to debug the kubernetes
+installation effortlessly and in style. K9S provides terminal (but windowed) CLI that allows you to
+easily observe what's going on in the kubernetes instance, observe the resources defined (pods, secrets,
+custom resource definitions), enter shell for the Pods/Containers running, see the log files and more.
+
+You can read more about k9s at `https://k9scli.io/ <https://k9scli.io/>`_
+
+Here is the screenshot of k9s tools in operation:
+
+.. image:: images/testing/k9s.png
+    :align: center
+    :alt: K9S tool
+
+
+You can enter the k9s tool via breeze (after you deployed Airflow):
+
+.. code-block:: bash
+
+      ./breeze kind-cluster k9s
+
+You can exit k9s by pressing Ctrl-C.
 
 Typical testing pattern for Kubernetes tests
 --------------------------------------------
@@ -590,7 +641,6 @@ This prepares and enters the virtualenv in ``.build/.kubernetes_venv`` folder:
 
     ./breeze kind-cluster shell
 
-
 Once you enter the environment you receive this information:
 
 
@@ -607,12 +657,67 @@ Once you enter the environment you receive this information:
 
     You are entering the virtualenv now. Type exit to exit back to the original shell
 
+In a separate terminal you can open the k9s CLI:
+
+.. code-block:: bash
+
+    ./breeze kind-cluster k9s
+
+Use it to observe what's going on in your cluster.
+
+6. Debugging in IntelliJ/PyCharm
+
+It is very easy to running/debug Kubernetes tests with IntelliJ/PyCharm. Unlike the regular tests they are
+in ``kubernetes_tests`` folder and if you followed the previous steps and entered the shell using
+``./breeze kind-cluster shell`` command, you can setup your IDE very easily to run (and debug) your
+tests using the standard IntelliJ Run/Debug feature. You just need a few steps:
+
+a) Add the virtualenv as interpreter for the project:
+
+.. image:: images/testing/kubernetes-virtualenv.png
+    :align: center
+    :alt: Kubernetes testing virtualenv
+
+The virtualenv is created in your "Airflow" source directory in ``.build/.kubernetes_venv/`` folder and you
+have to find ``python`` binary and choose it when selecting interpreter.
+
+b) Choose pytest as test runner:
+
+.. image:: images/testing/pytest-runner.png
+    :align: center
+    :alt: Pytest runner
+
+c) Run/Debug tests using standard "Run/Debug" feature of IntelliJ
+
+.. image:: images/testing/run-tests.png
+    :align: center
+    :alt: Run/Debug tests
+
+
+NOTE! The first time you run it, it will likely fail with
+``kubernetes.config.config_exception.ConfigException``:
+``Invalid kube-config file. Expected key current-context in kube-config``. You need to add KUBECONFIG
+environment variabl copying it from the result of "./breeze kind-cluster test":
+
+.. code-block:: bash
+
+    echo ${KUBECONFIG}
+
+    /home/jarek/code/airflow/.build/.kube/config
+
+
+.. image:: images/testing/kubeconfig-env.png
+    :align: center
+    :alt: Run/Debug tests
+
+
+The configuration for kubernetes is stored in your "Airflow" source directory in ".build/.kube/config" file
+and this is where KUBECONFIG env should point to.
 
 You can iterate with tests while you are in the virtualenv. All the tests requiring kubernetes cluster
 are in "kubernetes_tests" folder. You can add extra ``pytest`` parameters then (for example ``-s`` will
 print output generated test logs and print statements to the terminal immediately.
 
-
 .. code-block:: bash
 
     pytest kubernetes_tests/test_kubernetes_executor.py::TestKubernetesExecutor::test_integration_run_dag_with_scheduler_failure -s
@@ -621,6 +726,30 @@ print output generated test logs and print statements to the terminal immediatel
 You can modify the tests or KubernetesPodOperator and re-run them without re-deploying
 airflow to KinD cluster.
 
+
+Sometimes there are side effects from running tests. You can run ``redeploy_airflow.sh`` without
+recreating the whole cluster. This will delete the whole namespace, including the database data
+and start a new Airflow deployment in the cluster.
+
+.. code-block:: bash
+
+    ./scripts/ci/redeploy_airflow.sh
+
+If needed you can also delete the cluster manually:
+
+
+.. code-block:: bash
+
+    kind get clusters
+    kind delete clusters <NAME_OF_THE_CLUSTER>
+
+Kind has also useful commands to inspect your running cluster:
+
+.. code-block:: text
+
+    kind --help
+
+
 However, when you change Airflow Kubernetes executor implementation you need to redeploy
 Airflow to the cluster.
 
@@ -629,7 +758,7 @@ Airflow to the cluster.
     ./breeze kind-cluster deploy
 
 
-5. Stop KinD cluster when you are done
+7. Stop KinD cluster when you are done
 
 .. code-block:: bash
 

breeze

@@ -1790,6 +1790,13 @@ ${CMDNAME} kind-cluster [FLAGS] OPERATION
 
 ${FORMATTED_KIND_OPERATIONS}
 
+      The last two operations - shell and k9s allow you to perform interactive testing with
+      kubernetes tests. You can enter the shell from which you can run kubernetes tests and in
+      another terminal you can start the k9s CLI to debug kubernetes instance. It is an easy
+      way to debug the kubernetes deployments.
+
+      You can read more about k9s at https://k9scli.io/
+
 Flags:
 $(breeze::flag_airflow_variants)
 $(breeze::flag_build_docker_images)
@@ -3068,6 +3075,8 @@ function breeze::run_build_command() {
             echo "Run Kubernetes tests with the KinD cluster "
         elif [[ ${KIND_CLUSTER_OPERATION} == "shell" ]]; then
             echo "Enter an interactive shell for kubernetes testing"
+        elif [[ ${KIND_CLUSTER_OPERATION} == "k9s" ]]; then
+            echo "Run k9s cli to debug in style"
         elif [[ -z ${KIND_CLUSTER_OPERATION=} ]]; then
             echo
             echo "Please provide an operation to run"

breeze-complete

@@ -32,7 +32,7 @@ _breeze_allowed_helm_versions="v3.2.4"
 _breeze_allowed_kind_versions="v0.8.0"
 _breeze_allowed_mysql_versions="5.7 8"
 _breeze_allowed_postgres_versions="9.6 10 11 12 13"
-_breeze_allowed_kind_operations="start stop restart status deploy test shell"
+_breeze_allowed_kind_operations="start stop restart status deploy test shell k9s"
 _breeze_allowed_test_types="All Core Providers API CLI Integration Other WWW Heisentests Postgres MySQL Helm"
 
 # shellcheck disable=SC2034

chart/requirements.lock

@@ -2,5 +2,5 @@ dependencies:
 - name: postgresql
   repository: https://charts.helm.sh/stable/
   version: 6.3.12
-digest: sha256:58d88cf56e78b2380091e9e16cc6ccf58b88b3abe4a1886dd47cd9faef5309af
-generated: "2020-11-04T15:59:36.967913-08:00"
+digest: sha256:1748aa702050d4e72ffba1b18960f49bfe5368757cf976116afeffbdedda1c98
+generated: "2020-11-07T17:40:45.418723358+01:00"

kubernetes_tests/test_kubernetes_executor.py

@@ -63,10 +63,11 @@ def _num_pods_in_namespace(namespace):
         return len(names)
 
     @staticmethod
-    def _delete_airflow_pod():
+    def _delete_airflow_pod(name=''):
+        suffix = '-' + name if name else ''
         air_pod = check_output(['kubectl', 'get', 'pods']).decode()
         air_pod = air_pod.split('\n')
-        names = [re.compile(r'\s+').split(x)[0] for x in air_pod if 'airflow' in x]
+        names = [re.compile(r'\s+').split(x)[0] for x in air_pod if 'airflow' + suffix in x]
         if names:
             check_call(['kubectl', 'delete', 'pod', names[0]])
 
@@ -232,7 +233,7 @@ def test_integration_run_dag_with_scheduler_failure(self):
 
         execution_date = self.start_job_in_kubernetes(dag_id, host)
 
-        self._delete_airflow_pod()
+        self._delete_airflow_pod("scheduler")
 
         time.sleep(10)  # give time for pod to restart
 

kubernetes_tests/test_kubernetes_pod_operator.py

@@ -55,6 +55,11 @@ def create_context(task):
     }
 
 
+def get_kubeconfig_path():
+    kubeconfig_path = os.environ.get('KUBECONFIG')
+    return kubeconfig_path if kubeconfig_path else os.path.expanduser('~/.kube/config')
+
+
 class TestKubernetesPodOperatorSystem(unittest.TestCase):
     def get_current_task_name(self):
         # reverse test name to make pod name unique (it has limited length)
@@ -116,7 +121,7 @@ def tearDown(self) -> None:
 
     def test_do_xcom_push_defaults_false(self):
         new_config_path = '/tmp/kube_config'
-        old_config_path = os.path.expanduser('~/.kube/config')
+        old_config_path = get_kubeconfig_path()
         shutil.copy(old_config_path, new_config_path)
 
         k = KubernetesPodOperator(
@@ -135,7 +140,7 @@ def test_do_xcom_push_defaults_false(self):
 
     def test_config_path_move(self):
         new_config_path = '/tmp/kube_config'
-        old_config_path = os.path.expanduser('~/.kube/config')
+        old_config_path = get_kubeconfig_path()
         shutil.copy(old_config_path, new_config_path)
 
         k = KubernetesPodOperator(

scripts/ci/kubernetes/ci_run_kubernetes_tests.sh

@@ -21,7 +21,6 @@
 kind::make_sure_kubernetes_tools_are_installed
 kind::get_kind_cluster_name
 
-traps::add_trap kind::stop_kubectl EXIT HUP INT TERM
 traps::add_trap kind::dump_kind_logs EXIT HUP INT TERM
 
 interactive="false"
@@ -105,7 +104,6 @@ if [[ ${interactive} == "true" ]]; then
     echo
     echo "You are entering the virtualenv now. Type exit to exit back to the original shell"
     echo
-    kubectl config set-context --current --namespace=airflow
     exec "${SHELL}"
 else
     pytest "${pytest_args[@]}" "${tests_to_run[@]}"

scripts/ci/kubernetes/ci_deploy_app_to_kubernetes.sh → scripts/ci/kubernetes/ci_setup_cluster_and_deploy_airflow_to_kubernetes.sh

@@ -23,10 +23,11 @@ traps::add_trap "kind::dump_kind_logs" EXIT HUP INT TERM
 
 kind::make_sure_kubernetes_tools_are_installed
 kind::get_kind_cluster_name
+kind::perform_kind_cluster_operation "start"
 build_images::prepare_prod_build
 build_images::build_prod_images
 kind::build_image_for_kubernetes_tests
 kind::load_image_to_kind_cluster
 kind::deploy_airflow_with_helm
-kind::forward_port_to_kind_webserver
 kind::deploy_test_kubernetes_resources
+kind::wait_for_webserver_healthy