Skip to content

Commit

Permalink
Updates / comments on the docs (zenml-io#1565)
Browse files Browse the repository at this point in the history
* examples become projects again

* examples become projects v2

* remove space

* Updated advanced guide with more builds

* update flavor formatting

* fix typo

* small fixes

* small fixes

* small fixes

* remove sandbox docs

* Update visualization docs page

* Update docs/book/user-guide/advanced-guide/visualize-artifacts.md

---------

Co-authored-by: Hamza Tahir <[email protected]>
Co-authored-by: Felix Altenberger <[email protected]>
  • Loading branch information
3 people authored May 25, 2023
1 parent 6eaff40 commit ea4a84b
Show file tree
Hide file tree
Showing 14 changed files with 128 additions and 82 deletions.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
2 changes: 1 addition & 1 deletion docs/book/introduction.md
Original file line number Diff line number Diff line change
Expand Up @@ -93,7 +93,7 @@ ZenML empowers ML engineers to take ownership of the entire ML lifecycle end-to-
Ready to manage your ML lifecycles end-to-end with ZenML? Here is a collection of pages you can take a look at next:
<table data-view="cards"><thead><tr><th></th><th></th><th data-hidden></th><th data-hidden data-card-target data-type="content-ref"></th></tr></thead><tbody><tr><td><span data-gb-custom-inline data-tag="emoji" data-code="1f423">🐣</span> <mark style="color:purple;"><strong>Starter Guide</strong></mark></td><td>Get started with ZenML and learn how to build your first pipeline and stack.</td><td></td><td><a href="user-guide/starter-guide/starter-guide.md">starter-guide.md</a></td></tr><tr><td><span data-gb-custom-inline data-tag="emoji" data-code="1f414">🐔</span> <mark style="color:purple;"><strong>Advanced Guide</strong></mark></td><td>Discover advanced ZenML features like config management and containerization.</td><td></td><td><a href="user-guide/advanced-guide/advanced-guide.md">advanced-guide.md</a></td></tr><tr><td><span data-gb-custom-inline data-tag="emoji" data-code="1f9d1-1f3eb">🧑🏫</span> <mark style="color:purple;"><strong>Examples</strong></mark></td><td>Explore ZenML through practical use-case examples.</td><td></td><td><a href="learning/examples/examples.md">examples.md</a></td></tr></tbody></table>
<table data-view="cards"><thead><tr><th></th><th></th><th data-hidden></th><th data-hidden data-card-target data-type="content-ref"></th></tr></thead><tbody><tr><td><span data-gb-custom-inline data-tag="emoji" data-code="1f423">🐣</span> <mark style="color:purple;"><strong>Starter Guide</strong></mark></td><td>Get started with ZenML and learn how to build your first pipeline and stack.</td><td></td><td><a href="user-guide/starter-guide/starter-guide.md">starter-guide.md</a></td></tr><tr><td><span data-gb-custom-inline data-tag="emoji" data-code="1f414">🐔</span> <mark style="color:purple;"><strong>Advanced Guide</strong></mark></td><td>Discover advanced ZenML features like config management and containerization.</td><td></td><td><a href="user-guide/advanced-guide/advanced-guide.md">advanced-guide.md</a></td></tr><tr><td><span data-gb-custom-inline data-tag="emoji" data-code="1f9d1-1f3eb">🧑🏫</span> <mark style="color:purple;"><strong>Examples</strong></mark></td><td>Explore ZenML through practical use-case examples.</td><td></td><td><a href="learning/projects/projects.md">projects.md</a></td></tr></tbody></table>
{% endtab %}
{% tab title="For MLOps Platform Engineers" %}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,9 @@
description: Resources to learn how to use ZenML practically.
---

# 🧩 Examples
# 🧩 Projects

ZenML Examples are small-scale production-grade ML use cases powered by ZenML. They are fully fleshed-out, end-to-end,
ZenML Projects are small-scale production-grade ML use cases powered by ZenML. They are fully fleshed-out, end-to-end,
projects that showcase ZenML's capabilities. They can also serve as a template from which to start similar projects.

<table data-view="cards"><thead><tr><th></th><th></th><th data-hidden data-type="content-ref"></th><th data-hidden data-type="files"></th><th data-hidden data-card-cover data-type="files"></th><th data-hidden data-card-target data-type="content-ref"></th></tr></thead><tbody><tr><td><mark style="color:purple;"><strong>Using ZenML with LLMs to analyze and version data from databases</strong></mark></td><td>Explore how ZenML can be used with large language models (LLMs) like GPT-4 to analyze and version data from databases like Supabase.</td><td><a href="https://github.com/zenml-io/zenml-projects/tree/main/supabase-openai-summary">https://github.com/zenml-io/zenml-projects/tree/main/supabase-openai-summary</a></td><td><a href="../../.gitbook/assets/youtldr_summarizer_fef5163dc0.webp">youtldr_summarizer_fef5163dc0.webp</a></td><td><a href="../../.gitbook/assets/youtldr_summarizer_fef5163dc0.webp">youtldr_summarizer_fef5163dc0.webp</a></td><td><a href="https://github.com/zenml-io/zenml-projects/tree/main/supabase-openai-summary">https://github.com/zenml-io/zenml-projects/tree/main/supabase-openai-summary</a></td></tr><tr><td><mark style="color:purple;"><strong>Build an LLM-powered community Slack bot</strong></mark></td><td>Productionalizing LangChain and LlamaIndex with a ZenML MLOps Pipeline to Help Community Slack Support.</td><td><a href="broken-reference/">broken-reference</a></td><td><a href="../../.gitbook/assets/llm.png">llm.png</a></td><td><a href="../../.gitbook/assets/llm.png">llm.png</a></td><td><a href="https://github.com/zenml-io/zenml-projects/tree/main/langchain-llamaindex-slackbot">https://github.com/zenml-io/zenml-projects/tree/main/langchain-llamaindex-slackbot</a></td></tr><tr><td><mark style="color:purple;"><strong>Summarize news articles with NLP</strong></mark></td><td>A tool powered by ZenML that can automate the summarization of news sources.</td><td><a href="broken-reference/">broken-reference</a></td><td><a href="../../.gitbook/assets/summarize.jpeg">summarize.jpeg</a></td><td><a href="../../.gitbook/assets/summarize.jpeg">summarize.jpeg</a></td><td><a href="https://github.com/zenml-io/zenml-projects/tree/main/zen-news-summarization">https://github.com/zenml-io/zenml-projects/tree/main/zen-news-summarization</a></td></tr><tr><td><mark style="color:purple;"><strong>Use ZenML seamlessly with GitHub Actions</strong></mark></td><td>A GitHub workflow that automates CI/CD with continuous model training and continuous model deployment to production.</td><td></td><td><a href="../../.gitbook/assets/pipeline_architecture_a8a212e2f8.webp">pipeline_architecture_a8a212e2f8.webp</a></td><td><a href="../../.gitbook/assets/Remote_with_git_ops.png">Remote_with_git_ops.png</a></td><td><a href="https://github.com/zenml-io/zenml-gitflow">https://github.com/zenml-io/zenml-gitflow</a></td></tr><tr><td><mark style="color:purple;"><strong>Detect sign language alphabet in real-time images using Yolov5</strong></mark> </td><td>Detect and recognize the American Sign Language alphabet in real-time images using Yolov5 and ZenML.</td><td><a href="https://github.com/zenml-io/zenml-projects/tree/main/sign-language-detection-yolov5">https://github.com/zenml-io/zenml-projects/tree/main/sign-language-detection-yolov5</a></td><td><a href="../../.gitbook/assets/pexels_shvets_production_7516363_8bec88f86d.webp">pexels_shvets_production_7516363_8bec88f86d.webp</a></td><td><a href="../../.gitbook/assets/pexels_shvets_production_7516363_8bec88f86d.webp">pexels_shvets_production_7516363_8bec88f86d.webp</a></td><td><a href="https://github.com/zenml-io/zenml-projects/tree/main/sign-language-detection-yolov5">https://github.com/zenml-io/zenml-projects/tree/main/sign-language-detection-yolov5</a></td></tr><tr><td><mark style="color:purple;"><strong>Forecasting wind power generation</strong></mark></td><td>Predict electricity power generation based on wind forecast In Orkney, Scotland.</td><td></td><td><a href="../../.gitbook/assets/pexels_pixabay_433308_0665b2fb5b.webp">pexels_pixabay_433308_0665b2fb5b.webp</a></td><td><a href="../../.gitbook/assets/pexels_pixabay_433308_0665b2fb5b.webp">pexels_pixabay_433308_0665b2fb5b.webp</a></td><td><a href="https://github.com/zenml-io/zenml-projects/tree/main/time-series-forecast">https://github.com/zenml-io/zenml-projects/tree/main/time-series-forecast</a></td></tr></tbody></table>
Expand Down
3 changes: 1 addition & 2 deletions docs/book/toc.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,6 @@
* [Fetch runs after execution](user-guide/starter-guide/fetch-runs-after-execution.md)
* [Understand stacks](user-guide/starter-guide/understand-stacks.md)
* [Connect to a deployed ZenML](user-guide/starter-guide/connect-to-a-deployed-zenml.md)
* [Switch to production](user-guide/starter-guide/switch-to-production.md)
* [Follow best practices](user-guide/starter-guide/follow-best-practices.md)
* [🐔 Advanced guide](user-guide/advanced-guide/advanced-guide.md)
* [Configure steps/pipelines](user-guide/advanced-guide/configure-steps-pipelines.md)
Expand Down Expand Up @@ -140,7 +139,7 @@
## Learning

* [🚀 Quickstart](https://github.com/zenml-io/zenml/blob/main/examples/quickstart/README.md)
* [🧩 Examples](learning/examples/examples.md)
* [🧩 Projects](learning/projects/projects.md)
* [Build an LLM-powered community Slack bot](https://github.com/zenml-io/zenml-projects/tree/main/langchain-llamaindex-slackbot)
* [Summarize news articles with NLP](https://github.com/zenml-io/zenml-projects/tree/main/zen-news-summarization)
* [Use ZenML seamlessly with GitHub Actions](https://github.com/zenml-io/zenml-gitflow)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,7 @@ Settings are categorized into two types:
* [`WandbExperimentTrackerSettings`](../component-guide/experiment-trackers/wandb.md) to specify W\&B settings.
* [`WhylogsDataValidatorSettings`](../component-guide/data-validators/whylogs.md) to specify Whylogs settings.

For stack-component-specific settings, you might be wondering what the difference is between these and the configuration passed in while doing `zenml stack-component register <NAME> --config1=configvalue --config2=configvalue` , etc. The answer is that the configuration passed in at registration time is static and fixed throughout all pipeline runs, while the settings can change.
For stack-component-specific settings, you might be wondering what the difference is between these and the configuration passed in while doing `zenml stack-component register <NAME> --config1=configvalue --config2=configvalue`, etc. The answer is that the configuration passed in at registration time is static and fixed throughout all pipeline runs, while the settings can change.

A good example of this is the [`MLflow Experiment Tracker`](../component-guide/experiment-trackers/mlflow.md), where configuration which remains static such as the `tracking_url` is sent through at registration time, while runtime configuration such as the `experiment_name` (which might change every pipeline run) is sent through as runtime settings.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ practical tutorial on how to use a ZenML code repository.

As [discussed before](containerize-your-pipeline.md#reuse-docker-image-builds-from-previous-runs), when using
containerized components in your stack, ZenML needs
to [build Docker images to remotely execute your code](manage-environments.md#build-environments). If you're not using a
to [build Docker images to remotely execute your code](manage-environments.md#execution-environments). If you're not using a
code repository, this code will be included in the Docker images that ZenML builds. This, however, means that new Docker
images will be built and pushed whenever you make changes to any of your source files. When running a pipeline that is
part of a [local code repository checkout](connect-your-git-repository.md#detecting-local-code-repository-checkouts),
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,17 +8,17 @@ ZenML executes pipeline steps sequentially in the active Python environment when

There are three ways to control this containerization process:

* [Control how and where an image is built](containerize-your-pipeline.md#define-where-an-image-is-built)
* [Utilize docker images from previous runs](containerize-your-pipeline.md#reuse-docker-image-builds-from-previous-runs)
* [Define where an image is built](containerize-your-pipeline.md#define-where-an-image-is-built)
* [Reuse Docker image builds from previous runs](containerize-your-pipeline.md#reuse-docker-image-builds-from-previous-runs)
* [Customize what gets built into the image](containerize-your-pipeline.md#customize-the-docker-building)

## Define where an image is built

[Image builders](../component-guide/image-builders/image-builders.md) determine how and where an image is built. Learn more [here](manage-environments.md#build-environments).
[Image builders](../component-guide/image-builders/image-builders.md) determine how and where an image is built. Learn more [here](manage-environments.md#image-builder-environment).

## Reuse Docker image builds from previous runs

ZenML automatically [builds and pushes Docker images](manage-environments.md#build-environments) when running a pipeline on a stack requiring Docker images. To run this build step separately without running the pipeline, call:
ZenML automatically [builds and pushes Docker images](manage-environments.md#execution-environments) when running a pipeline on a stack requiring Docker images. To run this build step separately without running the pipeline, call:

```python
pipeline_instance.build(...)
Expand Down Expand Up @@ -54,15 +54,15 @@ or when running a pipeline from the CLI
zenml pipeline run <PIPELINE_NAME> --build=<BUILD_ID>
```

### Utilize the full power of builds by connecting a code repository
### Automate build reuse by connecting a code repository

Building Docker images without [connecting a git repository](connect-your-git-repository.md) includes your step code. This means specifying a custom build when running a pipeline will **not run the code on your client machine** but will use the code **included in the Docker images of the build**. This allows you to make local code changes, but reusing a build from before will _always_ execute the code bundled in the Docker image, rather than the local code.
Building Docker images without [connecting a git repository](connect-your-git-repository.md) includes your step code. This means specifying a custom build when running a pipeline will **not run the code on your client machine** but will use the code **included in the Docker images of the build**. This allows you to make local code changes, but reusing a build from before will _always_ execute the code bundled in the Docker image, rather than the local code. This is why you also have to explicitly specify the `build_id` when running a pipeline.

To avoid this, disconnect your code from the build by [connecting a git repository](connect-your-git-repository.md). Registering a code repository lets you avoid building images each time you run a pipeline and quickly iterate on your code.
To avoid this, disconnect your code from the build by [connecting a git repository](connect-your-git-repository.md). Registering a code repository lets you avoid building images each time you run a pipeline and quickly iterate on your code. Also, ZenML will automatically figure out which builds match your pipeline and reuse the appropriate execution environment. This approach is highly recommended. Read more [here](connect-your-git-repository.md).

## Customize the Docker building

When a [pipeline is run with a remote orchestrator](../starter-guide/create-an-ml-pipeline.md) a [Dockerfile](https://docs.docker.com/engine/reference/builder/) is dynamically generated at runtime. It is then used to build the docker image using the [image builder](manage-environments.md#customizing-build-image-creation-image-builders) component of your stack. The Dockerfile consists of the following steps:
When a [pipeline is run with a remote orchestrator](../starter-guide/create-an-ml-pipeline.md) a [Dockerfile](https://docs.docker.com/engine/reference/builder/) is dynamically generated at runtime. It is then used to build the docker image using the [image builder](manage-environments.md#image-builder-environment) component of your stack. The Dockerfile consists of the following steps:

* **Starts from a parent image** that has **ZenML installed**. By default, this will use the [official ZenML image](https://hub.docker.com/r/zenmldocker/zenml/) for the Python and ZenML version that you're using in the active Python environment. If you want to use a different image as the base for the following steps, check out [this guide](containerize-your-pipeline.md#using-a-custom-parent-image).
* **Installs additional pip dependencies**. ZenML will automatically detect which integrations are used in your stack and install the required dependencies. If your pipeline needs any additional requirements, check out our [guide on including custom dependencies](containerize-your-pipeline.md#installing-additional-pip-dependencies-or-apt-packages) .
Expand All @@ -83,7 +83,7 @@ from zenml.config import DockerSettings

ZenML determines the root directory of your source files in the following order:

* If you've intialized zenml at your root, the repository directory will be used.
* If you've intialized zenml (`zenml init`), the repository root directory will be used.
* Otherwise, the parent directory of the python file you're executing will be the source root. For example, running `python /path/to/file.py`, the source root would be `/path/to`.

You can specify how these files are handled using the `source_files` attribute on the `DockerSettings`:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -145,10 +145,6 @@ To create visualizations, you need to:

As an example, check out the implementation of the [zenml.materializers.NumpyMaterializer](https://github.com/zenml-io/zenml/blob/main/src/zenml/materializers/numpy\_materializer.py) that use matplotlib to automatically save or plot certain arrays.

{% hint style="info" %}
If you would like to disable artifact visualization altogether, you can set `enable_artifact_visualization` at either pipeline or step level via `@pipeline(enable_artifact_visualization=False)` or `@step(enable_artifact_visualization=False).`
{% endhint %}

### (Optional) Which Metadata to Extract for the Artifact

Optionally, you can override the `extract_metadata()` method to track custom metadata for all artifacts saved by your materializer. Anything you extract here will be displayed in the dashboard next to your artifacts.
Expand Down
Loading

0 comments on commit ea4a84b

Please sign in to comment.