Integration/implementation of task libraries from other workflow backends

[xinbinhuang]

Other workflow backends (airflow, dagster, prefect) normally come with a set of reusable tasks library (for example: Prefect Tasks library).

To extend couler into other workflow backends, it's necessary to consider how to incorporate those libraries.

Things to consider:

• How to standardize interfaces for different backends
• Should couler maintain and catch up with the tasks library for each backend
• others?

[terrytangyuan]

@xinbinhuang Thanks for opening this issue to bring up some great questions! Indeed that we need to think about standardization across reusable tasks, especially given that we will be extending into multiple backends. Currently we have integrated with a couple of reusable ML steps with Kubeflow in https://github.com/couler-proj/couler/tree/master/couler/steps but they haven't followed any standards yet.

Some examples of how other projects are doing this besides Prefect:

• Argo Workflows catalog: https://github.com/argoproj-labs/argo-workflows-catalog
• KFP components: https://github.com/kubeflow/pipelines/tree/master/components
• Tekton tasks/pipelines catalog: https://github.com/tektoncd/catalog

Should couler maintain and catch up with the tasks library for each backend?

IMO it would be hard to catch up with them for each backend. Ideally we should define our own reusable tasks using Couler APIs that are framework-agnostic so that the user experience is consistent. Although users should be able to inject custom configurations that are specific to a particular backend when using the reusable task/step.