Skip to content

Files

Latest commit

1d6ae84 · Dec 6, 2018

History

History
This branch is 26 commits behind confluentinc/demo-scene:master.

twitter-streams

Folders and files

NameName
Last commit message
Last commit date

parent directory

..
Dec 6, 2018
Dec 6, 2018
Dec 6, 2018
Dec 6, 2018
Dec 6, 2018
Dec 6, 2018
Dec 6, 2018
Dec 6, 2018
Dec 6, 2018
Dec 6, 2018
Dec 6, 2018
Dec 6, 2018

Kafka Connect Twitter demo

A demo setup demonstrates processing of twitter streams. Can be used in booth / conferences on analyze real-time social media data.

Example 1. Acknowledgements

The project structure inspired Confluent Platform demo [1] Actual demo logic is inspired by Robin’s blog [2] Playbook is available!

Appendix: Running demo on AWS

Prerequisites

  • Generate ssh key for github access

    ssh-keygen -t rsa -b 4096 -C "github@email"
    cat .ssh/id_rsa.pub
optional install zsh and git
sudo yum install git
sudo yum install zsh  # (1)
sudo yum install jq   # (2)
  1. optional install zsh

  2. tool to work with JSON

    • clone repo

mkdir ~/temp/demo-scene
cd ~/temp/demo-scene
git init .
git remote add origin -f https://github.com/confluentinc/demo-scene/
git config core.sparsecheckout true
echo "twitter-streams/*" >> .git/info/sparse-checkout
git pull --depth=2 origin master
cd twitter-streams
ls -lh
  • intall JDK 8

sudo yum install java-1.8.0-openjdk-devel
sudo /usr/sbin/alternatives --config java
sudo /usr/sbin/alternatives --config javac
sudo yum remove java-1.7.0-openjdk # <1> optional, remove java 7
Note
How find java in debian - JAVA_HOME=$(readlink -f /usr/bin/java | sed "s:bin/java::")
Optional, install micro because it’s awesome
cd ~
mkdir -p .local/bin
cd .local/bin
curl https://getmic.ro | bash
Update twitter connector config
cd ~
cd twitter-streams/
micro connect_twitter.json
  • install docker and docker-compose

    sudo yum update
    sudo yum install -y docker
    sudo usermod -aG docker ec2-user
    sudo service docker start
    sudo su
    curl -L "https://github.com/docker/compose/releases/download/1.21.2/docker-compose-$(uname -s)-$(uname -m)" > /usr/local/bin/docker-compose
    chmod +x /usr/local/bin/docker-compose
    exit
    docker-compose --version