Tork

A distributed workflow engine.

Note: This project is currently a work in progress (WIP) and is not recommended for production use.

Features:

• REST API
• Submit individual tasks or workflows for execution.
• Horizontally scalable
• Task isolation - tasks are executed within a container to provide isolation, idempotency, and in order to enforce resource limits
• Automatic recovery of tasks in the event of a worker crash
• Supports both stand-alone and distributed setup
• Retry failed tasks
• Pre/Post tasks

Architecture

Coordinator: responsible for managing the lifecycle of a task through its various states and for exposing a REST API to the clients.

Worker: responsible for executing tasks by means of a runtime (typically Docker).

Broker: the message-queue, pub/sub mechanism used for routing tasks.

Datastore: holds the state for tasks and jobs.

Runtime: the platform used by workers to execute tasks. Currently only Docker is supported.

Hello World