Instruções para atividade prática

Repositório para a Live de 24/06/2021

Kinesis Delivery Stream

AWS Console -> Kinesis -> Create Firehose Delivery Stream "StreamName" -> Direct PUT -> Next -> Choose Destination -> Create S3 Bucket “covid-vaccines-logs-diolive” -> Configure settings -> buffer size 5mb -> buffer interval 60s -> IAM Role -> create new role -> Review and create

EC2

AWS Console -> EC2 -> Amazon Linux 2 AMI -> t2micro -> review and launch -> create new key pair -> download .pem file -> download putty -> puttygen -> load.pem file -> save .ppk file -> putty copy dns -> paste hostname -> SSH -> auth -> load ppk file -> login “ec2-user”
- sudo yum install -y aws-kinesis-agent
- sudo yum install -y git
- git clone https://github.com/cassianobrexbit/dio-live-aws-bigdata-2.git
- unzip Dataset.zip
- chmod a+x LogGenerator.py
- nano LogGenerator.py
- less country_vaccinations.csv
- sudo mkdir /var/log/diolive
- cd /etc/aws-kinesis
- sudo nano agent.json
- Copiar conteúdo do arquivo agent.json
- agent.json -> "kinesis.endpoint": "kinesis..amazonaws.com"
AWS Console -> EC2 -> Instances -> Select Instance -> Security -> Modify IAM Role -> Create New Role -> EC2 -> Administrator Access -> rolename “ec2-admin-role” -> save
- sudo service aws-kinesis-agent start
- sudo chkconfig aws-kinesis-agent on (start with instance)
- _cd ~_
- sudo ./LogGenerator.py 500000
- tail -f /var/log/aws-kinesis-agent/aws-kinesis-agent.log

S3

AWS Console -> S3 -> select bucket -> selecionar arquivo e download

SSH EC2

sudo service aws-kinesis-agent restart
sudo ./LogGenerator.py
tail -f /var/log/aws-kinesis-agent/aws-kinesis-agent.log
AWS Console -> Kinesis Streams -> select stream -> monitoring

AWS Glue

AWS Console -> glue databrew -> create new project -> create new role -> create project
Create dataset -> S3 -> formato CSV