Skip to content

Commit

Permalink
fix bugs for beater
Browse files Browse the repository at this point in the history
  • Loading branch information
ResolveWang committed Oct 22, 2017
1 parent 708ecda commit 73b1066
Show file tree
Hide file tree
Showing 2 changed files with 10 additions and 9 deletions.
7 changes: 4 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,9 +12,10 @@
[![](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)

## 演示站点
在新版的weibospider中,为了更方便进行抓取条件的配置,[545314690](https://github.com/545314690)提了一个可视化配置界面的PR,点击
[https://weibospider.rookiefly.cn](https://weibospider.rookiefly.cn)进行访问,账号是`test`,密码是`weibospidertest`
请大家不要更改里面的数据。注意,该应用并不是weibospider的下游应用,请自行进行数据分析相关工作,本项目只着重解决数据采集的问题。
在新版的weibospider中,为了更方便进行抓取条件的配置,[545314690](https://github.com/545314690)提了一个可视化配置界面的PR,点击[https://weibospider.rookiefly.cn](https://weibospider.rookiefly.cn)进行访问,
账号是`test`,密码是`weibospidertest`,请大家不要更改登录相关数据,否则爬虫将无法正常运行。由于定时原因,在修改了搜索词后,两个小时之后才会开始新词/词组的搜索。

注意,该应用并不是weibospider的下游应用,请自行进行数据分析相关工作,本项目只着重解决数据采集的问题。

## 项目亮点 :star:
- 功能全面:包括了**用户信息抓取****指定关键字搜索结果增量抓取****指定用户主页所有原创微博抓取****评论抓取****转发关系抓取**
Expand Down
12 changes: 6 additions & 6 deletions tasks/workers.py
Original file line number Diff line number Diff line change
Expand Up @@ -37,32 +37,32 @@
CELERY_RESULT_SERIALIZER='json',
CELERYBEAT_SCHEDULE={
'login_task': {
'task': 'tasks.login.excute_login_task',
'task': 'tasks.login.execute_login_task',
'schedule': timedelta(hours=20),
'options': {'queue': 'login_queue', 'routing_key': 'for_login'}
},
'user_task': {
'task': 'tasks.user.excute_user_task',
'task': 'tasks.user.execute_user_task',
'schedule': timedelta(minutes=3),
'options': {'queue': 'user_crawler', 'routing_key': 'for_user_info'}
},
'search_task': {
'task': 'tasks.search.excute_search_task',
'task': 'tasks.search.execute_search_task',
'schedule': timedelta(hours=2),
'options': {'queue': 'search_crawler', 'routing_key': 'for_search_info'}
},
'home_task': {
'task': 'tasks.home.excute_home_task',
'task': 'tasks.home.execute_home_task',
'schedule': timedelta(hours=10),
'options': {'queue': 'home_crawler', 'routing_key': 'home_info'}
},
'comment_task': {
'task': 'tasks.comment.excute_comment_task',
'task': 'tasks.comment.execute_comment_task',
'schedule': timedelta(hours=10),
'options': {'queue': 'comment_crawler', 'routing_key': 'comment_info'}
},
'repost_task': {
'task': 'tasks.repost.excute_repost_task',
'task': 'tasks.repost.execute_repost_task',
'schedule': timedelta(hours=10),
'options': {'queue': 'repost_crawler', 'routing_key': 'repost_info'}
},
Expand Down

0 comments on commit 73b1066

Please sign in to comment.