Skip to content

A distributed Sina Weibo Search spider base on Scrapy and Redis.

Notifications You must be signed in to change notification settings

tpeng/weibosearch

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

3 Commits
 
 
 
 
 
 
 
 
 
 

Repository files navigation

WeiboSearch

A distributed Sina Weibo Search spider base on Scrapy and Redis.

tpeng [email protected]

Installation

$ sudo apt-get install mysql-server
$ sudo apt-get install redis-server
$ sudo apt-get install python-mysqldb
$ sudo pip install -r requirements.txt

Usage

  1. put your keywords in items.txt
  2. scrapy crawl weibosearch -a username=your_weibo_account -a password=your_weibo_password
  3. add another spider with scrapy crawl weibosearch -a username=another_weibo_account -a password=another_weibo_password

or

  1. scrapy crawl weibosearch -a username=your_weibo_account -a password=your_weibo_password -a savedb=False to avoid save to mysql db. (easier for experiment)

About

A distributed Sina Weibo Search spider base on Scrapy and Redis.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages