网页全网采集系统，是一款基于http协议的Web信息采集软件，支持集群化部署！
参考设计：http://scrapy-chs.readthedocs.org/zh_CN/latest/topics/architecture.html
单机、集群
CrawlerEngine:系统整体调度器
Scheduler：调度器/抓取队列
Downloader:下载器
Spiders:蜘蛛/新内容发现
Message Pipeline:消息管道/数据清洗
-
Notifications
You must be signed in to change notification settings - Fork 54
ljja/SmartSpiderCluster
Folders and files
| Name | Name | Last commit message | Last commit date | |
|---|---|---|---|---|
Repository files navigation
About
网页全网采集系统，是一款基于http协议的Web信息采集软件，支持集群化部署！
Resources
Stars
Watchers
Forks
Releases
No releases published
Packages 0
No packages published