Marquez is an open source metadata service for the collection, aggregation, and visualization of a data ecosystem's metadata. It maintains the provenance of how datasets are consumed and produced, provides global visibility into job runtime and frequency of dataset access, centralization of dataset lifecycle management, and much more. Marquez was released and open sourced by WeWork.

Badges

Status

Marquez is an LF AI & Data Foundation incubation project under active development and we'd love your help!

Quickstart

Marquez provides a simple way to collect and view dataset, job, and run metadata using OpenLineage. The easiest way to get up and running is with Docker. From the base of the Marquez repository run:

$ ./docker/up.sh

Tip: Use the --build flag to build images from source, and/or --seed to start Marquez with sample lineage metadata. For a more complete example using the sample metadata, please follow our quickstart guide.

WEB UI

You can open http://localhost:3000 to begin exploring the Marquez Web UI. The UI enables you to discover dependencies between jobs and the datasets they produce and consume via the lineage graph, view run metadata of current and previous job runs, and much more!

HTTP API

The Marquez HTTP API listens on port 5000 for all calls and port 5001 for the admin interface. The admin interface exposes helpful endpoints like /healthcheck and /metrics. To verify the HTTP API server is running and listening on localhost browse to http://localhost:5001.

• To read metadata, use the read-only APIs of Marquez
• To write metadata, use the write-only APIs of Marquez, or an OpenLineage integration to easily collect lineage events

Note: By default, the HTTP API does not require any form of authentication or authorization.

GRAPHQL

To explore metadata via graphql, browse to http://localhost:5000/graphql-playground. The graphql endpoint is currently in beta and is located at http://localhost:5000/api/v1-beta/graphql.

Documentation

We invite everyone to help us improve and keep documentation up to date. Documentation is maintained in this repository and can be found under docs/.

Note: To begin collecting metadata with Marquez, follow our quickstart guide. Below you will find the steps to get up and running from source.

Modules

Marquez uses a multi-project structure and contains the following modules:

• api: core API used to collect metadata
• web: web UI used to view metadata
• clients: clients that implement the HTTP API
• integrations: integrations with other systems (ex: Airflow)
• chart: helm chart

Requirements

• Java 11
• Java 8 (required for Spark integration)
• PostgreSQL 12.1

Note: To connect to your running PostgreSQL instance, you will need the standard psql tool.

Building

You can build most of the project modules using Java 11. We recommend setting JAVA_HOME to your local JDK 11 installation and running the following gradle command with the specific module you'd like to build:

$ ./gradlew :<module>:build

For example, to build the api module run:

$ ./gradlew :api:build

The executable can be found under api/build/libs/

To build the entire project, set JAVA_HOME to your local JDK 11 installation and JDK8_HOME to your local JDK 8 installation. Then, build all modules with:

$ ./gradlew build

Note: The spark integration requires Java 8 to build. The module can be built separately by setting JAVA_HOME to your local JDK 8 installation and running ./gradlew :integrations:spark:build.

Configuration

To run Marquez, you will have to define marquez.yml. The configuration file is passed to the application and used to specify your database connection. The configuration file creation steps are outlined below.

Step 1: Create Database

When creating your database using createdb, we recommend calling it marquez:

$ createdb marquez

Step 2: Create marquez.yml

With your database created, you can now copy marquez.example.yml:

$ cp marquez.example.yml marquez.yml

You will then need to set the following environment variables (we recommend adding them to your .bashrc): POSTGRES_DB, POSTGRES_USER, and POSTGRES_PASSWORD. The environment variables override the equivalent option in the configuration file.

By default, Marquez uses the following ports:

• TCP port 8080 is available for the HTTP API server.
• TCP port 8081 is available for the admin interface.

Note: All of the configuration settings in marquez.yml can be specified either in the configuration file or in an environment variable.

Running the HTTP API Server

$ ./gradlew :api:runShadow

Marquez listens on port 8080 for all API calls and port 8081 for the admin interface. To verify the HTTP API server is running and listening on localhost browse to http://localhost:8081. We encourage you to familiarize yourself with the data model and APIs of Marquez. To run the web UI, please follow the steps outlined here.

Note: By default, the HTTP API does not require any form of authentication or authorization.

Running a Simple Example

Make sure Docker and docker-compose are installed. Run the following command to setup the API with a seed dataset:

./docker/up.sh --seed

You can now connect to the API with the URL http://localhost:3000.

Related Projects

• OpenLineage: open standard for metadata and lineage collection

Getting Involved

• Website: https://marquezproject.ai
• Source: https://github.com/MarquezProject/marquez
• Chat: https://marquezproject.slack.com
• Twitter: @MarquezProject

Contributing

See CONTRIBUTING.md for more details about how to contribute.