Added documentation for Airflow Upgrade Check (apache#12872)

docs/upgprade-check/upgrade-check.rst

@@ -0,0 +1,163 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+
+ ..   http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+
+
+Upgrade Check Script
+--------------------
+
+.. contents:: :local:
+
+Getting the Airflow Upgrade Check Package
+'''''''''''''''''''''''''''''''''''''''''
+
+Apache Airflow is published as ``apache-airflow`` package in PyPI. The Upgrade Check Script is part of a
+separate Python Package, since it is separate from the core Apache Airflow package and is only needed for
+a period of time and specifically only for upgrading from Airflow 1.10 releases to Airflow 2.0.
+
+While there has been a lot of work put into making this upgrade as easy as possible, there are some
+changes which are compatible between Airflow 1.10 and Airflow 2.0. In order to make this as simple to
+navigate, we recommend that people first upgrade to the latest release in the 1.10 series (at the
+time of writing: 1.10.14) and then to download this package and run the script as detailed below.
+
+
+.. note::
+
+   On November 2020, new version of PIP (20.3) has been released with a new, 2020 resolver. This resolver
+   does not yet work with Apache Airflow and might leads to errors in installation - depends on your choice
+   of extras. In order to install Airflow you need to either downgrade pip to version 20.2.4
+   ``pip upgrade --pip==20.2.4`` or, in case you use Pip 20.3, you need to add option
+   ``--use-deprecated legacy-resolver`` to your pip install command.
+
+.. code-block:: bash
+
+    pip install apache-airflow-upgrade-check
+
+This will install the latest version of the Airflow Upgrade check package.
+
+
+Running the Airflow Upgrade Check Package
+'''''''''''''''''''''''''''''''''''''''''
+
+.. code-block:: bash
+
+    airflow upgrade_check
+
+This will print out to the screen a number of actions that should be taken before upgrading the Airflow
+release to 2.0.0 or higher.
+
+The exit code of the command will be 0 (success) if no problems are found in running the command, or 1 if
+problems were encountered in running the check.
+
+A sample output as a result of a successful run of the upgrade check is shown below.
+
+.. code-block:: bash
+
+    ============================================== STATUS ============================================
+
+    Check for latest versions of apache-airflow and checker..................................SUCCESS
+    Remove airflow.AirflowMacroPlugin class..................................................SUCCESS
+    Chain between DAG and operator not allowed...............................................SUCCESS
+    Connection.conn_id is not unique.........................................................SUCCESS
+    Connection.conn_type is not nullable.....................................................SUCCESS
+    Fernet is enabled by default.............................................................FAIL
+    GCP service account key deprecation......................................................SUCCESS
+    Changes in import paths of hooks, operators, sensors and others..........................FAIL
+    Users must delete deprecated configs for KubernetesExecutor..............................FAIL
+    Legacy UI is deprecated by default.......................................................SUCCESS
+    Logging configuration has been moved to new section......................................FAIL
+    Removal of Mesos Executor................................................................SUCCESS
+    Users must set a kubernetes.pod_template_file value......................................FAIL
+    SendGrid email uses old airflow.contrib module...........................................SUCCESS
+    Changes in import path of remote task handlers...........................................SUCCESS
+    Jinja Template Variables cannot be undefined.............................................FAIL
+    Found 7 problems.
+
+    ========================================== RECOMMENDATIONS ========================================
+
+    Fernet is enabled by default
+    ----------------------------
+    The fernet mechanism is enabled by default to increase the security of the default installation.
+
+    Problems:
+
+    1.  fernet_key in airflow.cfg must be explicitly set empty as fernet mechanism is enabledby default. This means that the apache-airflow[crypto] extra-packages are always installed.However, this requires that your operating system has libffi-dev installed.
+
+    Changes in import paths of hooks, operators, sensors and others
+    ---------------------------------------------------------------
+    Many hooks, operators and other classes has been renamed and moved. Those changes were part of unifying names and imports paths as described in AIP-21.
+    The contrib folder has been replaced by providers directory and packages:
+    https://github.com/apache/airflow#backport-packages
+
+    Problems:
+
+    1.  Using ``airflow.operators.python_operator.PythonOperator`` will be replaced by ``airflow.operators.python.PythonOperator``. Affected file:
+
+
+The following sections describe what is being done and how to apply the recommendations shown above.
+Please note that the above results shown are only a partial set, where only the first
+two of the seven problems identified are shown in the section above. In reality,
+all the problems are shown on the screen.
+
+
+Understanding what is being checked
+'''''''''''''''''''''''''''''''''''
+
+The Upgrade Check checks the configuration data from airflow.cfg, the meta data from the Airflow
+database, as well as the DAGs which have been set up in the current Airflow environment.
+
+Using the above results as an example, there are two specific problems which have
+been identified.
+
+The first problem is identified in the configuration file airflow.cfg where the current configuration
+option for the fernet_key is no longer acceptable and needs to be changed. This is because as of
+Airflow 2.0, the fernet_key cannot be left empty, but needs to have a defined value. Examining the
+problematic airflow.cfg and searching for the fernet_key entries would show the following:
+
+.. code-block:: bash
+
+    fernet_key =
+
+The second problem was identified in one of the DAGs. In this case, this import
+statement for the ``PythonOperator`` needs to be changed, since the location is different
+in Airflow 2.0. Examining the DAG file would probably show the following:
+
+.. code-block:: python
+
+    from airflow.operators.python_operator import PythonOperator
+
+We will discuss how to fix these and make them compatible with Airflow 2.0 in the next
+section.
+
+
+Applying the Recommendations
+''''''''''''''''''''''''''''
+
+In most cases, the Recommendations result section of the Upgrade check contains
+enough information to make the change.
+
+For the first problem identified above with respect to the fernet_key, the solution is
+to enter a valid value in the Airflow Configuration file airflow.cfg for the fernet_key.
+
+For the second problem, looking at the source of the DAG file and changing the import
+statement for the Python Operator to be as follows will make this DAG work in Airflow 2.0.
+
+.. code-block:: python
+
+    from airflow.operators.python import PythonOperator
+
+However, at the time of writing, this is incompatible in Airflow 1.10.14. So, this change
+can only be made while moving to Airflow 2.0.