Skip to content

Commit

Permalink
Some important docs updates (zenml-io#2463)
Browse files Browse the repository at this point in the history 
* Add mention of deploying zenml in installation

* Add mention of deploying zenml in installation

* Add mention of deploying zenml in installation

* ZenML Cloud

* More ZenML Cloud

* Added assets

* README

* READ

* latest

* latest

* README

* README

* README

* README

* README

* README

* README

* Optimised images with calibre/image-actions

* README

* Update README.md

Co-authored-by: Alex Strick van Linschoten <[email protected]>

* Update docs/book/deploying-zenml/zenml-cloud/cloud-system-architecture.md

Co-authored-by: Alex Strick van Linschoten <[email protected]>

* Update docs/book/deploying-zenml/zenml-cloud/zenml-cloud.md

Co-authored-by: Alex Strick van Linschoten <[email protected]>

* Update docs/book/getting-started/installation.md

Co-authored-by: Alex Strick van Linschoten <[email protected]>

* Update docs/book/deploying-zenml/zenml-cloud/zenml-cloud.md

Co-authored-by: Alex Strick van Linschoten <[email protected]>

* Update docs/book/deploying-zenml/zenml-cloud/zenml-cloud.md

Co-authored-by: Alex Strick van Linschoten <[email protected]>

* Update README.md

Co-authored-by: Alex Strick van Linschoten <[email protected]>

* Update README.md

Co-authored-by: Alex Strick van Linschoten <[email protected]>

* Update README.md

Co-authored-by: Alex Strick van Linschoten <[email protected]>

* Update README.md

Co-authored-by: Alex Strick van Linschoten <[email protected]>

* Update README.md

Co-authored-by: Alex Strick van Linschoten <[email protected]>

* Update README.md

Co-authored-by: Alex Strick van Linschoten <[email protected]>

* Update README.md

Co-authored-by: Alex Strick van Linschoten <[email protected]>

* latest

* README again

* More WHy

* Latest

* Latest

* Latest

* ZenmL Cloud

* Update README.md

Co-authored-by: Alex Strick van Linschoten <[email protected]>

* README again

* Apply suggestions from code review

Co-authored-by: Barış Can Durak <[email protected]>

* latest

* Update docs/book/getting-started/installation.md

Co-authored-by: Barış Can Durak <[email protected]>

* latest

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Alex Strick van Linschoten <[email protected]>
Co-authored-by: Barış Can Durak <[email protected]>
  • Loading branch information
@htahir1 @github-actions
@strickvl @bcdurak
4 people authored Feb 23, 2024
1 parent 0ca61c9 commit 909ea63
Show file tree
Hide file tree
Showing 9 changed files with 105 additions and 120 deletions.
167 changes: 65 additions & 102 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -68,10 +68,11 @@
[build-url]: https://github.com/zenml-io/zenml/actions/workflows/ci.yml

<div align="center">
<h3 align="center">Build portable, production-ready MLOps pipelines.</h3>
<img referrerpolicy="no-referrer-when-downgrade" src="https://static.scarf.sh/a.png?x-pxid=0fcbab94-8fbe-4a38-93e8-c2348450a42e" />
<h3 align="center">Create an MLOps workflow for your entire team.</h3>
<p align="center">
<div align="center">
Join our <a href="https://zenml.io/slack-invite" target="_blank">
Join our <a href="https://zenml.io/slack" target="_blank">
<img width="18" src="https://cdn3.iconfinder.com/data/icons/logos-and-brands-adobe/512/306_Slack-512.png" alt="Slack"/>
<b>Slack Community</b> </a> and be part of the ZenML family.
</div>
Expand All @@ -82,13 +83,13 @@
·
<a href="https://github.com/zenml-io/zenml/issues">Report Bug</a>
·
<a href="https://zenml.io/discussion">Vote New Features</a>
<a href="https://zenml.io/cloud">Sign up for Cloud</a>
·
<a href="https://www.zenml.io/blog">Read Blog</a>
·
<a href="https://github.com/issues?q=is%3Aopen+is%3Aissue+archived%3Afalse+user%3Azenml-io+label%3A%22good+first+issue%22">Contribute to Open Source</a>
·
<a href="https://www.zenml.io/company#team">Meet the Team</a>
<a href="https://github.com/zenml-io/zenml-projects">Projects Showcase</a>
<br />
<br />
🎉 Version 0.55.3 is out. Check out the release notes
Expand All @@ -104,21 +105,15 @@
<details>
<summary>🏁 Table of Contents</summary>
<ol>
<li><a href="#-introduction">Introduction</a></li>
<li><a href="#-quickstart">Quickstart</a></li>
<li><a href="#🤖-introduction">Introduction</a></li>
<li><a href="#🤸-quickstart">Quickstart</a></li>
<li>
<a href="#-create-your-own-mlops-platform">Create your own MLOps Platform</a>
<ul>
<li><a href="##-1-deploy-zenml">Deploy ZenML</a></li>
<li><a href="#-2-deploy-stack-components">Deploy Stack Components</a></li>
<li><a href="#-3-create-a-pipeline">Create a Pipeline</a></li>
<li><a href="#-4-start-the-dashboard">Start the Dashboard</a></li>
</ul>
<a href="#🖼️-learning">Learning</a>
</li>
<li><a href="#-roadmap">Roadmap</a></li>
<li><a href="#-contributing-and-community">Contributing and Community</a></li>
<li><a href="#-getting-help">Getting Help</a></li>
<li><a href="#-license">License</a></li>
<li><a href="#🗺-roadmap">Roadmap</a></li>
<li><a href="#🙌-contributing-and-community">Contributing and Community</a></li>
<li><a href="#🆘-getting-help">Getting Help</a></li>
<li><a href="#📜-license">License</a></li>
</ol>
</details>

Expand All @@ -141,17 +136,46 @@ experimentation while writing code that is production-ready from the get-go.
- 🛫 ZenML enables MLOps infrastructure experts to define, deploy, and manage
sophisticated production environments that are easy to use for colleagues.

![The long journey from experimentation to production.](/docs/book/.gitbook/assets/intro-zenml-overview.png)

ZenML provides a user-friendly syntax designed for ML workflows, compatible with
any cloud or tool. It enables centralized pipeline management, enabling
developers to write code once and effortlessly deploy it to various
infrastructures.

<div align="center">
<img src="docs/book/.gitbook/assets/overview.gif">
<img width="60%" src="/docs/book/.gitbook/assets/zenml-hero.png" alt="ZenML Hero"/>
</div>

# 🛠️ Why ZenML?

![Walkthrough of ZenML Model Control Plane (Dashboard available only on ZenML Cloud)](/docs/book/.gitbook/assets/mcp_walkthrough.gif)

ZenML offers a systematic approach to structuring your machine learning codebase for a seamless transition to production. It's an ideal solution for teams grappling with establishing an internal standard for coordinating ML operations. ZenML provides not just a tool, but a workflow strategy that guides you in integrating all your tools and infrastructure effectively.

Use ZenML if:

- You need to easily automate ML workflows on services like an Airflow cluster or AWS Sagemaker Pipelines.
- Your ML tasks require repeatability and reproducibility.
- Automating and standardizing ML workflows across your team is a challenge.
- Your team integrates multiple tools with no central platform.
- You'd like a single place to track data, code, configuration, and models along with your cloud artifact storage.
- Collaboration and hand-overs between multiple teams is difficult.

# ☄️ What makes ZenML different?

![Before and after ZenML](/docs/book/.gitbook/assets/zenml-why.png)

ZenML marries the capabilities of a classic pipeline tool like [Airflow](https://airflow.apache.org/) and a metadata tracking service like [MLflow](https://mlflow.org/). Furthermore, both these types of tools can seamlessly co-exist with ZenML, providing a comprehensive, end-to-end ML experience.

It excels at:

- Enabling creation of simple, pythonic [ML pipelines](https://docs.zenml.io/user-guide/starter-guide/create-an-ml-pipeline) that function locally and on any [orchestration backend](https://docs.zenml.io/user-guide/production-guide/cloud-orchestration).
- Automating versioning of [data](https://docs.zenml.io/user-guide/starter-guide/manage-artifacts) and [models](https://docs.zenml.io/user-guide/starter-guide/track-ml-models) on [remote artifact storage like S3](https://docs.zenml.io/user-guide/production-guide/remote-storage).
- Abstracting infrastructure and run configuration from code through a [simple YAML config](https://docs.zenml.io/user-guide/advanced-guide/pipelining-features/configure-steps-pipelines).
- Logging complex [metadata](https://docs.zenml.io/user-guide/advanced-guide/data-management/logging-metadata) for models and artifacts.
- Automatically containerizing and deploying your workflows to the cloud, connected to your [code repository](https://docs.zenml.io/user-guide/production-guide/connect-code-repository).
- Connecting your [secret store](https://docs.zenml.io/user-guide/advanced-guide/secret-management) to your ML workflows.

However, ZenML doesn't:

- Automatically create visuals and track experiments: It [integrates with experiment trackers](https://docs.zenml.io/stacks-and-components/component-guide/experiment-trackers) that specialize in this task.
- Package and deploy models: ZenML catalogs models and metadata, streamlining model deployment. Refer to [ZenML model deployers](https://docs.zenml.io/stacks-and-components/component-guide/model-deployers) for more information.
- Handle distributed computation: While ZenML pipelines scale vertically with ease, it [works with tools like Spark](https://docs.zenml.io/stacks-and-components/component-guide/step-operators/spark-kubernetes) for intricate distributed workflows.

# 🤸 Quickstart

[Install ZenML](https://docs.zenml.io/getting-started/installation) via
Expand All @@ -167,22 +191,18 @@ Take a tour with the guided quickstart by running:
zenml go
```

# 🖼️ Create your own MLOps Platform

ZenML allows you to create and manage your own MLOps platform using
best-in-class open-source and cloud-based technologies. Here is an example of
how you could set this up for your team:

## 🔋 1. Deploy ZenML
# 🔋 Deploy ZenML

For full functionality ZenML should be deployed on the cloud to
enable collaborative features as the central MLOps interface for teams.

![ZenML Architecture Diagram.](docs/book/.gitbook/assets/Scenario3.png)
<div align="center">
<img width="60%" src="docs/book/.gitbook/assets/Scenario3.png" alt="ZenML Architecture Diagram."/>
</div>

Currently, there are two main options to deploy ZenML:

- **ZenML Cloud**: With [ZenML Cloud](https://docs.zenml.io/deploying-zenml/zenml-cloud),
- **ZenML Cloud**: With [ZenML Cloud](cloud.zenml.io/?utm_source=readme&utm_medium=referral_link&utm_campaign=cloud_promotion&utm_content=signup_link),
you can utilize a control plane to create ZenML servers, also known as tenants.
These tenants are managed and maintained by ZenML's dedicated team, alleviating
the burden of server management from your end.
Expand All @@ -192,88 +212,31 @@ ZenML on your own self-hosted environment](https://docs.zenml.io/deploying-zenml
This can be achieved through various methods, including using our CLI, Docker,
Helm, or HuggingFace Spaces.

## 👨‍🍳 2. Deploy Stack Components

ZenML boasts a ton of [integrations](https://zenml.io/integrations) into
popular MLOps tools. The [ZenML Stack](https://docs.zenml.io/user-guide/starter-guide/understand-stacks)
concept ensures that these tools work nicely together, therefore bringing
structure and standardization into the MLOps workflow.
# 🖼️ Learning

Deploying and configuring this is super easy with ZenML. For **AWS**, this might
look a bit like this
The best way to learn about ZenML is the [docs](https://docs.zenml.io). We recommend beginning with the [Starter Guide](https://docs.zenml.io/user-guide/starter-guide) to get up and running quickly.

```bash
# Deploy and register an orchestrator and an artifact store
zenml orchestrator deploy kubernetes_orchestrator --flavor kubernetes --cloud aws
zenml artifact-store deploy s3_artifact_store --flavor s3

# Register this combination of components as a stack
zenml stack register production_stack --orchestrator kubernetes_orchestrator --artifact-store s3_artifact_store --set # Register your production environment
```
For inspiration, here are some other examples and use cases:

When you run a pipeline with this stack set, it will be running on your deployed
Kubernetes cluster.

You can also [deploy your own tooling manually](https://docs.zenml.io/stacks-and-components/stack-deployment).

## 🏇 3. Create a Pipeline

Here's an example of a hello world ZenML pipeline in code:

```python
# run.py
from zenml import pipeline, step


@step
def step_1() -> str:
"""Returns the `world` substring."""
return "world"


@step
def step_2(input_one: str, input_two: str) -> None:
"""Combines the two strings at its input and prints them."""
combined_str = input_one + ' ' + input_two
print(combined_str)


@pipeline
def my_pipeline():
output_step_one = step_1()
step_2(input_one="hello", input_two=output_step_one)


if __name__ == "__main__":
my_pipeline()
```

```bash
python run.py
```

## 👭 4. Start the Dashboard

Open up the ZenML dashboard using this command.

```bash
zenml show
```
1. [E2E Batch Inference](examples/e2e/): Feature engineering, training, and inference pipelines for tabular machine learning.
2. [Basic NLP with BERT](examples/e2e_nlp/): Feature engineering, training, and inference focused on NLP.
3. [LLM RAG Pipeline with Langchain and OpenAI](https://github.com/zenml-io/zenml-projects/tree/main/llm-agents): Using Langchain to create a simple RAG pipeline.
4. [Huggingface Model to Sagemaker Endpoint](https://github.com/zenml-io/zenml-projects/tree/main/huggingface-sagemaker): Automated MLOps on Amazon Sagemaker and HuggingFace.

# 🗺 Roadmap

ZenML is being built in public. The [roadmap](https://zenml.io/roadmap) is a
regularly updated source of truth for the ZenML community to understand where
the product is going in the short, medium, and long term.

ZenML is managed by a [core team](https://zenml.io/company#CompanyTeam) of
ZenML is managed by a [core team](https://zenml.io/company) of
developers that are responsible for making key decisions and incorporating
feedback from the community. The team oversees feedback via various channels,
and you can directly influence the roadmap as follows:

- Vote on your most wanted feature on our [Discussion
board](https://zenml.io/discussion).
- Start a thread in our [Slack channel](https://zenml.io/slack-invite).
- Start a thread in our [Slack channel](https://zenml.io/slack).
- [Create an issue](https://github.com/zenml-io/zenml/issues/new/choose) on our
GitHub repo.

Expand All @@ -291,12 +254,12 @@ Guide](CONTRIBUTING.md) for all relevant details.
The first point of call should
be [our Slack group](https://zenml.io/slack-invite/).
Ask your questions about bugs or specific use cases, and someone from
the [core team](https://zenml.io/company#CompanyTeam) will respond.
the [core team](https://zenml.io/company) will respond.
Or, if you
prefer, [open an issue](https://github.com/zenml-io/zenml/issues/new/choose) on
our GitHub repo.

# Vulnerability affecting `zenml<0.67` (CVE-2024-25723)
# Vulnerability affecting `zenml<0.46.7` (CVE-2024-25723)

We have identified a critical security vulnerability in ZenML versions prior to
0.46.7. This vulnerability potentially allows unauthorized users to take
Expand Down
Binary file added docs/book/.gitbook/assets/zenml-cloud-tenant-overview.png
View file
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/book/.gitbook/assets/zenml-hero.png
View file
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/book/.gitbook/assets/zenml-why.png
View file
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
6 changes: 3 additions & 3 deletions docs/book/deploying-zenml/zenml-cloud/cloud-system-architecture.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ description: Different variations of the ZenML Cloud architecture depending on y
# System Architecture

{% hint style="info" %}
If you would like to evaluate ZenML Cloud, please [book a demo](https://zenml.io/book-a-demo) and we can help you decide what scenario suits you best!
If you're interested in assessing ZenML Cloud, you can create a [free account](https://cloud.zenml.io/?utm_source=docs&utm_medium=referral_link&utm_campaign=cloud_promotion&utm_content=signup_link), which defaults to a [Scenario 1](#scenario-1-full-saas) deployment. To upgrade to different scenarios, please [reach out to us](mailto:[email protected]).
{% endhint %}

Machine learning often involves data that is sensitive and thus data security is a fundamental requirement. The ZenML Cloud is flexible enough to meet your security requirements, from easy installations to completely airgapped deployments.
Expand All @@ -16,7 +16,7 @@ The ZenML Cloud consists of the following moving pieces:
* **Single Sign-On (SSO)**: The ZenML Cloud API is integrated with [Auth0](https://auth0.com/) as an SSO provider to manage user authentication and authorization. Users can log in to the ZenML Cloud dashboard using their social media accounts or their corporate credentials.
* **Secrets Store**: All secrets and credentials required to access customer infrastructure services are stored in a secure secrets store. The ZenML Cloud API has access to these secrets and uses them to access customer infrastructure services on behalf of the ZenML Cloud. The secrets store can be hosted either by the ZenML Cloud or by the customer.
* **ML Metadata Store**: This is where all ZenML metadata is stored, including ML metadata such as tracking and versioning information about pipelines and models.
* **ZenML Cloud Agent**: This service is optionally deployed customer-side, and interacts with customer MLOps stack components on behalf of the remote ZenML Cloud control plane.
* **ZenML Cloud Agent (Optional)**: This service is optionally deployed customer-side, and interacts with customer MLOps stack components on behalf of the remote ZenML Cloud control plane.

The above five interact with other MLOps stack components, secrets, and data in varying scenarios described below.

Expand Down Expand Up @@ -78,6 +78,6 @@ Here is a concrete example of how this works:

In this scenario, all services, data, and secrets are deployed on the customer cloud. This is the opposite of Scenario 1, and is meant for customers who require completely airgapped deployments, for the tightest security standards.

Are you interested in the ZenML Cloud? While in beta, we're looking for early adopters to talk to! [Sign up](https://cloud.zenml.io) and get access to Scenario 1. with a free 30 day trial now!
Are you interested in the ZenML Cloud? While in beta, we're looking for early adopters to talk to! [Sign up](https://cloud.zenml.io/?utm_source=docs&utm_medium=referral_link&utm_campaign=cloud_promotion&utm_content=signup_link) and get access to Scenario 1. with a free 30 day trial now!

<figure><img src="https://static.scarf.sh/a.png?x-pxid=f0b4f458-0a54-4fcd-aa95-d5ee424815bc" alt="ZenML Scarf"><figcaption></figcaption></figure>
Loading

0 comments on commit 909ea63

Please sign in to comment.