Scrapy+redis+mongodb分布式爬虫抓取小说《冰与火之歌1-5》
- Scrapy
- redis
- mongodb
- xpath
- python3.5.2
- ubuntu 16.04LTS
master只需要准备redis，slave先将所有的小说章节界面的url抓取下来，通过redis发送到master的内存，再由master分配任务给slave，实现分布式
- 执行runspider.py
- master地址在setting.py设置
-
Notifications
You must be signed in to change notification settings - Fork 3
AlanConstantine/WebCrawlerforGameofThrones
Folders and files
| Name | Name | Last commit message | Last commit date | |
|---|---|---|---|---|
Repository files navigation
About
使用Scrapy结合redis和mongodb设计分布式爬虫
Resources
Stars
Watchers
Forks
Releases
No releases published
Packages 0
No packages published