Dagster + dbt starter kit

This starter demonstrates using Python alongside a medium-sized dbt project. It uses dbt's jaffle_shop, dagster-dbt, and DuckDB.

New to Dagster? Learn what Dagster is in Dagster's hands-on Tutorials or learn using dbt with Dagster in the dbt + Dagster tutorial.

Getting started

Option 1: Deploying it on Dagster Cloud

The easiest way to spin up your Dagster project is to use Dagster Cloud Serverless. It provides out-of-the-box CI/CD and native branching that make development and deployment easy.

Check out Dagster Cloud to get started.

Option 2: Running it locally

To install this example and its Python dependencies, run:

pip install -e ".[dev]"

Then, start the Dagit web server:

dagit

Open http://localhost:3000 with your browser to see the project.

Learning more

Changing the code locally

When developing pipelines locally, be sure to click the Reload definition button in the Dagster UI after you change the code. This ensures that Dagster picks up the latest changes you made.

You can reload the code using the Deployment page:

👈 Expand to view the screenshot

Or from the left nav or on each job page:

👈 Expand to view the screenshot

Using environment variables and secrets

Environment variables, which are key-value pairs configured outside your source code, allow you to dynamically modify application behavior depending on environment.

Using environment variables, you can define various configuration options for your Dagster application and securely set up secrets. For example, instead of hard-coding database credentials - which is bad practice and cumbersome for development - you can use environment variables to supply user details. This allows you to parameterize your pipeline without modifying code or insecurely storing sensitive data.

Check out Using environment variables and secrets for more info and examples.

Running daemon locally

If you're running Dagster locally and trying to set up schedules, you will see a warning that your daemon isn’t running.

👈 Expand to learn how to set up a local daemon

If you want to enable Dagster Schedules for your jobs, start the Dagster Daemon process in the same folder as your workspace.yaml file, but in a different shell or terminal.

The $DAGSTER_HOME environment variable must be set to a directory for the daemon to work. Note: using directories within /tmp may cause issues. See Dagster Instance default local behavior for more details.

In this case, go to the project root directory and run:

dagster-daemon run

Once your Dagster Daemon is running, the schedules that are turned on will start running.

Adding new Python dependencies

You can specify new Python dependencies in setup.py.

Testing

Tests are in the assets_dbt_python_tests directory and you can run tests using pytest:

pytest assets_dbt_python_tests