Spiderq is a Web Spider to crawl webpage(html) by Qteqpid. The performance depends on your server configuration and network. I will continue maintain it and list some TODOs at the end of this file. More people are welcome to join!
Spiderq can be compiled and used on Centos 5.8 . It is as simple as:
% make
% make install
Then you will get an executable file named spider. After configurating spiderq.conf, run program:
% ./spider
For more informations, see Makefile.
For any question, just contact me at any time. Enjoy! mailto: qteqpid[email protected] blog: http://hi.baidu.com/qteqpid_pku
@线程池 @信号处理 @网页内容排重 @同一ip间隔抓取 @层次结构存储网页 @是否遵守robots.txt @支持更新抓取，不重复抓 @定义对外api和html类，方便用户自定义处理html，动态加载方式