Skip to content

Commit

Permalink
Change Decorated Flow to TaskFlow API (apache#11895)
Browse files Browse the repository at this point in the history
* Change Decorated Flow to TaskFlow API

In order to better describe what this new data transfer system does,
we are changing the name of Decorated Flows to TaskFlow API

* Update docs/concepts.rst

Co-authored-by: Gerard Casas Saez <[email protected]>

* Update docs/concepts.rst

Co-authored-by: Kaxil Naik <[email protected]>

* Update docs/tutorial_taskflow_api.rst

Co-authored-by: Kaxil Naik <[email protected]>

Co-authored-by: Gerard Casas Saez <[email protected]>
Co-authored-by: Kaxil Naik <[email protected]>
  • Loading branch information
3 people authored Oct 27, 2020
1 parent 2d9f6ee commit 8477efe
Show file tree
Hide file tree
Showing 2 changed files with 18 additions and 18 deletions.
10 changes: 5 additions & 5 deletions docs/concepts.rst
Original file line number Diff line number Diff line change
Expand Up @@ -120,14 +120,14 @@ DAGs can be used as context managers to automatically assign new operators to th
op.dag is dag # True
.. _concepts:decorated_flows:
.. _concepts:task_flow_api:

Decorated Flows
---------------
TaskFlow API
------------

.. versionadded:: 2.0.0

Airflow 2.0 adds a new style of authoring dags called Decorated Flows which removes a lot of the boilerplate
Airflow 2.0 adds a new style of authoring dags called the TaskFlow API which removes a lot of the boilerplate
around creating PythonOperators, managing dependencies between task and accessing XCom values. (During
development this feature was called "Functional DAGs", so if you see or hear any references to that, it's the
same thing)
Expand All @@ -136,7 +136,7 @@ Outputs and inputs are sent between tasks using :ref:`XCom values <concepts:xcom
functions as tasks using the :ref:`task decorator <concepts:task_decorator>`. Airflow will also automatically
add dependencies between tasks to ensure that XCom messages are available when operators are executed.

Example DAG with decorated style
Example DAG built with the TaskFlow API

.. code-block:: python
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -18,20 +18,20 @@
Tutorial on Decorated Flows
===========================
Tutorial on the Taskflow API
============================

This tutorial builds on the regular Airflow Tutorial and focuses specifically
on writing data pipelines using the Decorated Flow paradigm which is introduced as
on writing data pipelines using the Taskflow API paradigm which is introduced as
part of Airflow 2.0 and contrasts this with DAGs written using the traditional paradigm.

The data pipeline chosen here is a simple ETL pattern with
three separate tasks for Extract, Transform, and Load.

Example "Decorated Flow" ETL Pipeline
-------------------------------------
Example "Taskflow API" ETL Pipeline
-----------------------------------

Here is very simple ETL pipeline using the Decorated Flows paradigm. A more detailed
Here is very simple ETL pipeline using the Taskflow API paradigm. A more detailed
explanation is given below.

.. exampleinclude:: /../airflow/example_dags/tutorial_decorated_etl_dag.py
Expand Down Expand Up @@ -115,7 +115,7 @@ As we see here, the data being processed in the Transform function is passed to
variables. In turn, the summarized data from the Transform function is also placed
into another Xcom variable which will then be used by the Load task.

Contrasting that with Decorated Flows in Airflow 2.0 as shown below.
Contrasting that with Taskflow API in Airflow 2.0 as shown below.

.. exampleinclude:: /../airflow/example_dags/tutorial_decorated_etl_dag.py
:language: python
Expand All @@ -126,7 +126,7 @@ All of the Xcom usage for data passing between these tasks is abstracted away fr
in Airflow 2.0. However, Xcom variables are used behind the scenes and can be viewed using
the Airflow UI as necessary for debugging or DAG monitoring.

Similarly, task dependencies are automatically generated within Decorated Flows based on the
Similarly, task dependencies are automatically generated within TaskFlows based on the
functional invocation of tasks. In Airflow 1.x, tasks had to be explicitly created and
dependencies specified as shown below.

Expand All @@ -135,7 +135,7 @@ dependencies specified as shown below.
:start-after: [START main_flow]
:end-before: [END main_flow]

In contrast, with Decorated Flows in Airflow 2.0, the invocation itself automatically generates
In contrast, with the Taskflow API in Airflow 2.0, the invocation itself automatically generates
the dependencies as shown below.

.. exampleinclude:: /../airflow/example_dags/tutorial_decorated_etl_dag.py
Expand All @@ -147,10 +147,10 @@ the dependencies as shown below.
What's Next?
------------

You have seen how simple it is to write DAGs using the Decorated Flows paradigm within Airflow 2.0. Please do
You have seen how simple it is to write DAGs using the Taskflow API paradigm within Airflow 2.0. Please do
read the :ref:`Concepts page<concepts>` for detailed explanation of Airflow concepts such as DAGs, Tasks,
Operators, etc, and the :ref:`concepts:task_decorator` in particular.

More details about Decorated Flows, can be found as part of the Airflow Improvement Proposal
`AIP-31: "Decorated Flows" for clearer/simpler DAG definition <https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=148638736>`__
and specifically within the Concepts guide at :ref:`Concepts - Decorated Flows<concepts:decorated_flows>`.
More details about the Taskflow API, can be found as part of the Airflow Improvement Proposal
`AIP-31: "Taskflow API" for clearer/simpler DAG definition <https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=148638736>`__
and specifically within the Concepts guide at :ref:`Concepts - Taskflow API<concepts:task_flow_api>`.

0 comments on commit 8477efe

Please sign in to comment.