Skip to content

ADDF is a collection of modules, deployed using the SeedFarmer orchestration tool. ADDF modules enable users to quickly bootstrap environments for the process and analysis of autonomous driving data.

License

Notifications You must be signed in to change notification settings

clokep/autonomous-driving-data-framework

 
 

Repository files navigation

Autonomous Driving Data Framework(ADDF)

ADDF logo

Static Checking Integration Tests

ADDF is a collection of modules for Scene Detection, Simulation (mock), Visualization, Compute, Storage, Centralized logging etc, deployed using SeedFarmer orchestration tool. ADDF allows you to build distinct, stand alone Infrastructure as code (IAAC) modules and exchange information about dependencies using metadata which can be exported from one module and imported into another. Each module can be found under the modules directory of this repository.

Deployment Instructions

You can refer to the SeedFarmer guide to understand how SeedFarmer CLI can be used to bootstrap and deploy ADDF.

You can follow instructions in the Deployment Guide Readme. You can also follow the blogpost for understanding ADDF in detail.

Please see the ADDF Security and Operations Guide for in-depth recommendations on assessing, deploying, customizing, and operating ADDF.

Different types of modules supported by ADDF

Use-case specific Modules

Type Description
Rosbag Image Pipeline Module Deploys a Rosbag Image pipeline for use in ADDF
Rosbag WebViz Module Deploys and Visualizes Rosbag Data on AWS using Webviz for use in ADDF

Optional Modules

Type Description
DataLake Buckets Module Deploys shared datalake buckets such as input, intermediate, output, logging, artifact buckets for use in ADDF

Integration Modules

Type Description
DDB to Opensearch Module This module integrates DynamoDB table with Opensearch cluster
EKS to Opensearch Module This module integrates EKS Cluster with Opensearch cluster
EMR to Opensearch Module This module integrates EMR Cluster with Opensearch cluster
Opensearch Proxy Module This module deploys a Proxy server to access Opensearch cluster

Simulation Modules

Type Description
K8s-Managed Module This module helps running simulations on AWS EKS, when triggered by KubernetesJob Operator from airflow environment
AWS Batch Module This module helps running simulations on AWS Batch, when triggered by Batch Operator from airflow environment

IDE Modules

Type Description
Self Managed JupyterHub Module This module deploys self managed JupyterHub environment on AWS EKS
Self Managed VSCode Module This module deploys self managed VSCode environment on AWS EKS
AWS Managed EMR Studio Module This module deploys AWS managed EMR Studio with EMR on EKS

Example Modules

Type Description
Example DAG Module This module deploys a pattern to integrate a target DAG module to work with shared MWAA Cluster

Reporting Issues

If you notice a defect, feel free to create an Issue

Deployment FAQ

If you need to debug a deployment in ADDF, here are few things you can checkout Readme

About

ADDF is a collection of modules, deployed using the SeedFarmer orchestration tool. ADDF modules enable users to quickly bootstrap environments for the process and analysis of autonomous driving data.

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Python 86.0%
  • Shell 7.1%
  • Jupyter Notebook 2.7%
  • TypeScript 2.6%
  • Dockerfile 1.5%
  • HCL 0.1%