Skip to content

soncco/trends-scraper

Repository files navigation

Trends Scraper from Tweetdeck

A simple trend scrapper from Tweetdeck, based on the work of Mateo Grella.

Why

For research purposes only.

Requirements

  • Ubuntu Server
  • MySQL Server
  • Rabbit MQ Server
  • Google Chrome

Installation

Google Chrome

$ wget -c https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
$ sudo apt-get update
$ sudo dpkg -i google-chrome-stable_current_amd64.deb

Essentials

$ sudo apt-get install virtualenv build-essential python3-dev

Configuration

Rename example.env to .env and fill credentials. You can change settings on places.yml too.

Database

Create mysql database and table:

    CREATE TABLE `trends` (
    `id` int(11) NOT NULL AUTO_INCREMENT,
    `location` varchar(50) NOT NULL,
    `hashtag` varchar(100) NOT NULL,
    `tweets_counter` int(11) NOT NULL,
    `position` int(11) NOT NULL,
    `trend_date` datetime NOT NULL,
    PRIMARY KEY (`id`)
    ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;

Running

$ virtualenv trends
$ cd trends

# Clone this repository
$ cd trends-scraper
$ pip install -r requirements.txt

# Run the script
$ python scraper.py

Legal

It is your responsibility to ensure that your use of tweetdeck-scraper does not violate applicable laws.

License

Tweetdeck Scraper is licensed under the Apache License, Version 2.0. See LICENSE for the full license text.

About

No description, website, or topics provided.

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages