Aqueduct: Orchestrate & manage production ML

Aqueduct enables you to define, deploy and monitor robust ML pipelines on any cloud infrastructure. Check out our quickstart guide!

Aqueduct gives you a simple Python-native API to define machine learning pipelines, the ability to deploy those pipelines on your existing infrastructure (e.g., Spark, Kubernetes, Lambda), and visibility into the code, data, and metadata associated with your workflows. Aqueduct is fully open-source and runs securely in your cloud.

You can install Aqueduct via pip:

pip3 install aqueduct-ml
aqueduct start

Now, we can create our first workflow:

from aqueduct import Client, op, metric

client = Client()

@op
def transform_data(reviews):
    reviews['strlen'] = reviews['review'].str.len()
    return reviews


demo_db = client.integration("aqueduct_demo")
reviews_table = demo_db.sql("select * from hotel_reviews;")

strlen_table = transform_data(reviews_table)
demo_db.save(strlen_table, "strlen_table", "replace)

client.publish_flow(name="review_strlen", artifacts=[strlen_table])

Once we've created a workflow, we can view that workflow in the Aqueduct UI:

Why Aqueduct?

The engineering required to get data science & machine learning projects in production slows down data teams. Aqueduct automates away that engineering and allows you to define robust data & ML pipelines in a few lines of code and run them anywhere.

• Python-native pipeline API: Aqueduct’s API allows you define your workflows in vanilla Python, so you can get code into production quickly and effectively. No more DSLs or YAML configs to worry about.
• Integrated with your infrastructure: Workflows defined in Aqueduct can run on any cloud infrastructure you use, like Kubernetes, Spark, Airflow, or AWS Lambda. You can get all the benefits of Aqueduct without having to rip-and-replace your existing tooling.
• Centralized visibility into code, data, & metadata: Once your workflows are in production, you need to know what’s running, whether it’s working, and when it breaks. Aqueduct gives you visibility into what code, data, metrics, and metadata are generated by each workflow run, so you can have confidence that your pipelines work as expected — and know immediately when they don’t.
• Runs securely in your cloud: Aqueduct is fully open-source and runs in any Unix environment. It runs entirely in your cloud and on your infrastructure, so you can be confident that nothing is ever leaving your cloud.

Overview & Examples

The core abstraction in Aqueduct is a Workflow, which is a sequence of Artifacts (data) that are transformed by Operators (compute). The input Artifact(s) for a Workflow is typically loaded from a database, and the output Artifact(s) are typically persisted back to a database. Each Workflow can either be run on a fixed schedule or triggered on-demand.

To see Aqueduct in action on some real-world machine learning workflows, check out some of our examples:

• Churn Ensemble
• Sentiment Analysis
• Impute Missing Wine Data
• ... and more!

What's next?

Check out our documentation, where you'll find:

• a Quickstart Guide
• example workflows
• and more details on creating workflows

If you have questions or comments or would like to learn more about what we're building, please reach out, join our Slack channel, or start a conversation on GitHub. We'd love to hear from you!

If you're interested in contributing, please check out our roadmap and join the development channel in our community Slack.