Skip to content
/ pipeline Public
forked from PipelineAI/pipeline

PipelineIO: End-to-End ML and AI Platform for Real-time Spark and Tensorflow Data Pipelines

License

Notifications You must be signed in to change notification settings

wenpp/pipeline

 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

PipelineIO

Built with Love and Scale by ex-Netflix, Databricks, and Tensorflow Engineers

Netflix Databricks Tensorflow

Reference Architecture

Reference Architecture

100% Open Source

Jupyter ZeppelinZeppelin Spark Tensorflow

Completely Kubernetes and Docker Based

PipelineIO Kubernetes

Kubernetes Docker

PMML + Native C++/Java Code Generation for Optimized CPU/GPU Performance

Generate and Optimize Spark ML Model

Nvidia GPU Intel CPU

Deploy directly from Jupyter Notebook to Hybrid-Cloud Microservices

Deploy Spark ML Model to Production

NetflixOSS-based Microservices for High Scalability and Availability

NetflixOSS-based Microservices

Continuous Train/Deploy Spark ML and Tensorflow AI Pipelines to Production

Kafka Kinesis KINESIS

Live Demo

  • Use any username/password
  • Note: Please do not store sensitive info in this insecure environment!

Environment Setup

Pipeline Mini (Local Laptop)

  • Small Kubernetes cluster for your laptop

Pipeline Distributed

  • Kubernetes cluster on any cloud provider or on-premise

Kubernetes Docker

Pipeline Hybrid (Multi-cloud)

  • Kubernetes cluster across many cloud providers and/or on-premise

AWS GCP

Global 24x7 Support for Everyone!

Additional Resources

Training Repo and Workshop Schedule

Workshop Architecture

Workshop Tools

Global Advanced Spark and Tensorflow Meetup

Meetup

YouTube Videos

YouTube

Slideshare Slides

Slideshare

Docker Images

Docker

About

PipelineIO: End-to-End ML and AI Platform for Real-time Spark and Tensorflow Data Pipelines

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • HTML 50.3%
  • Jupyter Notebook 47.5%
  • Python 0.6%
  • JavaScript 0.6%
  • CSS 0.4%
  • Scala 0.2%
  • Other 0.4%