End-to-End, Real-time ML Reference Data Pipeline

Powered by the PANCAKE STACK!

Upcoming PANCAKE STACK Workshops!

Title

Building an End-to-End Streaming Analytics and Recommendations Pipeline with Spark, Kafka, and TensorFlow

Agenda (Full Day)

Part 1 (Analytics and Visualizations)

• Analytics and Visualizations Overview (Live Demo!)
• Verify Environment Setup (Docker, Cloud Instance)
• Notebooks (Zeppelin, Jupyter/iPython)
• Interactive Data Analytics (Spark SQL, Hive, Presto)
• Graph Analytics (Spark, Elastic, NetworkX, TitanDB)
• Time-series Analytics (Spark, Cassandra)
• Visualizations (Kibana, Matplotlib, D3)
• Approximate Queries (Spark SQL, Redis, Algebird)
• Workflow Management (Airflow)

Part 2 (Streaming and Recommendations)

• Streaming and Recommendations (Live Demo!)
• Streaming (NiFi, Kafka, Spark Streaming, Flink)
• Cluster-based Recommendation (Spark ML, Scikit-Learn)
• Graph-based Recommendation (Spark ML, Spark Graph)
• Collaborative-based Recommendation (Spark ML)
• NLP-based Recommendation (CoreNLP, NLTK)
• Geo-based Recommendation (ElasticSearch)
• Hybrid On-Premise+Cloud Auto-scale Deploy (Docker)
• Save Workshop Environment for Your Use Cases

Locations and Dates

• San Francisco: Saturday, June 4th
• Washington DC: Saturday, June 18th
• Seattle: Saturday, July 30th
• SF Bay Area: Saturday, August 6th
• Chicago: Saturday, September 17th
• Toronto: Saturday, September 24th
• New York: Saturday, October 1st
• Munich: Saturday, October 15th
• London: Saturday, October 22nd
• Brussels: Saturday, October 29th

Suggest a City and Date

Description

The goal of this workshop is to build an end-to-end, streaming data analytics and recommendations pipeline on your local machine using Docker and the latest streaming analytics

• First, we create a data pipeline to interactively analyze, approximate, and visualize streaming data using modern tools such as Apache Spark, Kafka, Zeppelin, iPython, and ElasticSearch.
• Next, we extend our pipeline to use streaming data to generate personalized recommendation models using popular machine learning, graph, and natural language processing techniques such as collaborative filtering, clustering, and topic modeling.
• Last, we productionize our pipeline and serve live recommendations to our users!

Architecture Overview

Follow Wiki to Setup Docker-based Environment

Screenshots

Apache Zeppelin Notebooks

Stanford CoreNLP Sentiment Analysis

Jupyter/iPython Notebooks

SparkR Notebooks

TensorFlow Notebooks

Apache NiFi Data Flows

AirFlow Workflows

Presto Queries

Tableau Integration

Beeline Command-line Hive Client

Log Visualization with Kibana & Logstash

Spark, Spark Streaming, and Spark SQL Admin UIs

Ganglia System and JVM Metrics Monitoring UIs

Tools Overview