Skip to content

Commit

Permalink
Update Cloud deployment scenarios (zenml-io#1757)
Browse files Browse the repository at this point in the history
* Update cloud architecture scenarios

* Optimised images with calibre/image-actions

* Minor docs updates

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
  • Loading branch information
stefannica and github-actions[bot] authored Aug 23, 2023
1 parent c397985 commit b21054e
Show file tree
Hide file tree
Showing 6 changed files with 24 additions and 8 deletions.
Binary file not shown.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified docs/book/.gitbook/assets/cloud_architecture_scenario_4.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
32 changes: 24 additions & 8 deletions docs/book/deploying-zenml/zenml-cloud/cloud-system-architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,20 +59,36 @@ In this scenario, the ZenML Cloud only stores metadata while the customer cloud

Note, that ZenML encrypts all customer secrets at transit and at rest.

## Scenario 2c: Hybrid SaaS with no data on ZenML side + Private Secret Store
## Scenario 3a: Hybrid SaaS with no data on ZenML side + Customer Secret Store managed by ZenML

<div data-full-width="true">
<figure><img src="../../.gitbook/assets/cloud_architecture_scenario_2c.png" alt=""><figcaption><p>Scenario 2c: Hybrid SaaS with no data on ZenML side + Private Secret Store</p></figcaption></figure>
<figure><img src="../../.gitbook/assets/cloud_architecture_scenario_3a.png" alt=""><figcaption><p>Scenario 3a: Hybrid SaaS with no data on ZenML side + Customer Secret Store managed by ZenML</p></figcaption></figure>
</div>

This is almost the same as Scenario 2b, however in this case the customer brings their own secret store to keep sensitive information on the customer side. This allows users to use [service connectors](../../stacks-and-components/auth-management/service-connectors-guide.md) and the [secrets API](../../user-guide/advanced-guide/secret-management/) to authenticate ZenML to 3rd party services and infrastructure while ensuring their secrets do not leave their environment.
This scenario is a version of Scenario 2b modified to store all sensitive information on the customer side. In this case, the customer connects their own secret store directly to the ZenML Cloud. All ZenML secrets used by running pipelines to access infrastructure services and resources are stored in the customer secret store. This allows users to use [service connectors](../../stacks-and-components/auth-management/service-connectors-guide.md) and the [secrets API](../../user-guide/advanced-guide/secret-management/) to authenticate ZenML pipelines and the ZenML Cloud to 3rd party services and infrastructure while ensuring that credentials are always stored on the customer side.

The ZenML secret store still exists, and this is to still give customers the option to allow ZenML Cloud to connect to their environment to enable features such as visualizing artifacts and triggering actions from the ZenML Cloud dashboard.
Even though they are stored customer side, access to ZenML secrets is fully managed by the ZenML Cloud. The ZenML Cloud is also allowed to use some of those credentials to connect directly to customer infrastructure services to implement control plane features such as artifact visualization or triggering pipelines. This implies that the secret values are allowed to leave the customer environment to allow their access to be managed centrally by the ZenML Cloud and to enforce access control policies, but the ZenML users and pipelines never have direct access to the secret store.

## Scenario 3: Agent Architecture
## Scenario 3b: Hybrid SaaS with no data on ZenML side + Private Secret Store

<div data-full-width="true">
<figure><img src="../../.gitbook/assets/cloud_architecture_scenario_3.png" alt=""><figcaption><p>Scenario 3: ZenML Agent deployment</p></figcaption></figure>
<figure><img src="../../.gitbook/assets/cloud_architecture_scenario_3b.png" alt=""><figcaption><p>Scenario 3b: Hybrid SaaS with no data on ZenML side + Private Secret Store</p></figcaption></figure>
</div>

This is almost the same as Scenario 3a, with one added difference: customer secrets are kept strictly private and are never accessed by the ZenML Cloud. This allows users to use [service connectors](../../stacks-and-components/auth-management/service-connectors-guide.md) and the [secrets API](../../user-guide/advanced-guide/secret-management/) to authenticate ZenML pipelines to 3rd party services and infrastructure while ensuring their secrets never leave the customer environment.

The ZenML Cloud secret store still exists, and this is to still give customers the option to allow ZenML Cloud to connect to their environment to enable features such as visualizing artifacts and triggering actions from the ZenML Cloud dashboard, but this is entirely optional.

This deployment scenario comes with some limitations:

* the ZenML Cloud control plane features such as triggering pipelines and visualizing artifacts are restricted to stack components that have credentials stored in the ZenML cloud
* ZenML users and pipelines may need to be individually configured to be able to access the secret values kept in the private secret store, because centralized authentication to the secret store is no longer possible through ZenML
* granular access control to ZenML secrets through RBAC is no longer possible: the ZenML clients can theoretically access any secret in the customer’s secret store unless access is regulated by customer specific means

## Scenario 4: Agent Architecture

<div data-full-width="true">
<figure><img src="../../.gitbook/assets/cloud_architecture_scenario_4.png" alt=""><figcaption><p>Scenario 3: ZenML Agent deployment</p></figcaption></figure>
</div>

This scenario adds a new architectural component into the mix, called the ZenML Agent, which facilitates communication between the two clouds. The customer is responsible for deploying and maintaining the ZenML Agent in their environment. The agent acts as an intermediate step for all operations and information that needs to be exchanged between ZenML cloud and other customer stack components, like the artifact store. This means that all features like visualizing data artifacts in the dashboard and triggering pipelines from the dashboard are fully available, but only the ZenML Agent has access to customer secrets and accesses the customer's infrastructure services.
Expand All @@ -87,10 +103,10 @@ Here is a concrete example of how this works:
* The agent fetches the artifact and sends it back to the Cloud API.
* The Cloud API sends the visualization to the dashboard.

## Scenario 4: Fully On-prem
## Scenario 5: Fully On-prem

<div data-full-width="true">
<figure><img src="../../.gitbook/assets/cloud_architecture_scenario_4.png" alt=""><figcaption><p>Scenario 4: Fully on-premises deployment</p></figcaption></figure>
<figure><img src="../../.gitbook/assets/cloud_architecture_scenario_5.png" alt=""><figcaption><p>Scenario 4: Fully on-premises deployment</p></figcaption></figure>
</div>

In this scenario, all services, data, and secrets are deployed on the customer cloud. This is the opposite of Scenario 1, and is meant for customers who require completely airgapped deployments, for the tightest security standards.
Expand Down

0 comments on commit b21054e

Please sign in to comment.