forked from william412/Nutch-Ajax-Solr
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
0 parents
commit b55e86c
Showing
884 changed files
with
100,222 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,77 @@ | ||
| Nutch AJAX page Fetch, Parse, Index Plugin | ||
| ============== | ||
|
|
||
| ### 项目简介 | ||
|
|
||
| 基于Apache Nutch 2.3和Htmlunit, Selenium WebDriver等组件扩展，实现对于AJAX加载类型页面的完整页面内容抓取，以及特定数据项的解析和索引。 | ||
|
|
||
| According to the implementation of Apache Nutch 2.X, we can't get dynamic HTML information from fetch pages including AJAX requests as it will ignore all AJAX requests. | ||
|
|
||
| This plugin will use Htmlunit and Selenium WebDriver to fetch whole page content with necessary dynamic AJAX requests. | ||
|
|
||
| It developed and tested with Apache Nutch 2.3, you can try it on other Nutch 2.X version or refactor the source codes as your design. | ||
|
|
||
| ### 主要特性 | ||
|
|
||
| * **常规的HTML页面抓取**: 对于常规的例如新闻类没有AJAX特性的页面可以直接用Nutch自带的protocol-http插件抓取。 | ||
|
|
||
| * **常规的AJAX页面抓取**: 对于绝大部分诸如jQuery ajax加载的页面，可以直接用htmlunit扩展插件抓取。 | ||
|
|
||
| * **特殊的AJAX请求页面抓取**: 诸如淘宝/天猫的页面采用了独特的Kissy Javascript组件，目前测试htmlunit无法正确解析，因此退而求其次采用效率低一些的Selenium WebDriver方式实现页面数据抓取。 | ||
|
|
||
| * **基于页面滚动的AJAX请求页面抓取**: 诸如淘宝/天猫的商品详情页面会基于页面滚动发起商品描述信息的加载，通过Htmlunit或Selenium WebDriver扩展处理可以实现此类页面数据抓取。 | ||
|
|
||
| ### 运行方式 | ||
|
|
||
| 整个项目基于官方的Apache Nutch 2.3源码基础之上添加插件代码和配置，运行方式和官方指南保持一致，具体请参考：http://wiki.apache.org/nutch/ | ||
|
|
||
| 同时工程代码中提交了Eclipse的工程配置文件，可以直接import Eclipse中Run或Debug运行，Nutch工程以Ivy进行依赖管理，可采用ANT Build方式或建议在Eclipse IDE安装Apache Ivy IDE插件进行工程编译运行。 | ||
|
|
||
|  | ||
|
|
||
|  | ||
|
|
||
|  | ||
|
|
||
| ### 扩展插件说明 | ||
|
|
||
| * **lib-pinyin**: 用于parse或index插件转换中文到拼音提交solr；部署用于solr dataimporthandler组件进行拼音转换的transformer扩展插件 | ||
|
|
||
| * **lib-htmlunit**: 基于Htmlunit的多线程处理，缓存控制，请求正则控制等特性扩展插件 | ||
|
|
||
| * **protocol-s2jh**: 基于Htmlunit和Selenium WebDriver实现的AJAX页面Fetcher插件 | ||
|
|
||
| * **parse-s2jh**: 基于XPath解析页面元素内容; 持久化解析到的结构化数据，如MySQL，MongoDB等; 对于个别复杂类型AJAX页面定制判断页面加载完成的回调判断逻辑 | ||
|
|
||
| * **index-s2jh**: 追加设置需要额外传递给SOLR索引的属性数据; 设定不需要索引的页面规则; | ||
|
|
||
| ### 详细参考文档 | ||
|
|
||
| 项目提供一份比较详细的“基于Nutch&Solr定向采集解析和索引搜索的整合技术指南文档”，可通过以下两种方式查看参考文档内容： | ||
|
|
||
| * 直接获取项目内容后，在document目录下根据自己熟悉的编辑器查看对应的md或html格式文档； | ||
| * GitHub直接解析md文件，并且能正确处理图片链接，因此可直接在线访问 https://github.com/xautlx/nutch-ajax/blob/master/document/Apache_Nutch_Solr_Solution_with_AJAX_support.md | ||
|
|
||
| ### 许可说明 | ||
|
|
||
| * Free Open Source | ||
|
|
||
| 本项目所有代码完整开源，在保留标识本项目来源信息以及保证不对本项目进行非授权的销售行为的前提下，可以以任意方式自由免费使用：开源、非开源、商业及非商业。 | ||
|
|
||
| * Charge Support Service | ||
|
|
||
| 如果你还有兴趣在Apache Nutch/Solr/Lucene等系列技术的定制的扩展实现/技术咨询服务/毕业设计指导/二次开发项目指导等方面的合作意向，可联系 E-Mail: [email protected] 或 QQ: 2414521719 (加Q请注明：nutch/solr/lucene) 洽谈。[上述联系方式恕不直接提供咨询类询问，为了提升项目活跃度，若对项目有任何技术问题或Issue反馈，请直接提交到项目站点提问或Git平台的Issue] | ||
|
|
||
| ### Reference | ||
|
|
||
| 欢迎关注作者其他项目： | ||
|
|
||
| * [Nutch 2.X AJAX Plugins (Active)](https://github.com/xautlx/nutch-ajax) - 基于Apache Nutch 2.3和Htmlunit, Selenium WebDriver等组件扩展，实现对于AJAX加载类型页面的完整页面内容抓取，以及特定数据项的解析和索引 | ||
|
|
||
| * [S2JH4Net (Active)](https://github.com/xautlx/s2jh4net) - 基于Spring MVC+Spring+JPA+Hibernate的面向互联网及企业Web应用开发框架 | ||
|
|
||
| * [S2JH (Deprecated)](https://github.com/xautlx/s2jh) - 基于Struts2+Spring+JPA+Hibernate的面向企业Web应用开发框架 | ||
|
|
||
| * [Nutch 1.X AJAX Plugins (Deprecated)](https://github.com/xautlx/nutch-htmlunit) - 基于Apache Nutch 1.X和Htmlunit的扩展实现AJAX页面爬虫抓取解析插件 | ||
|
|
||
| * [12306 Hunter (Deprecated)](https://github.com/xautlx/12306-hunter) - （功能已失效不可用，不过还可以当作Swing开发样列参考只用）Java Swing C/S版本12306订票助手，用处你懂的 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,73 @@ | ||
| <?xml version="1.0" encoding="UTF-8"?> | ||
| <classpath> | ||
| <classpathentry kind="lib" path="conf"/> | ||
| <classpathentry kind="lib" path="lib/hadoop-core-1.2.0.jar"/> | ||
| <classpathentry kind="src" path="src/plugin/lib-htmlunit/src/java"/> | ||
| <classpathentry kind="src" path="src/plugin/lib-pinyin/src/java"/> | ||
| <classpathentry kind="src" path="src/plugin/protocol-s2jh/src/java"/> | ||
| <classpathentry kind="src" path="src/plugin/parse-s2jh/src/java"/> | ||
| <classpathentry kind="src" path="src/plugin/index-s2jh/src/java"/> | ||
| <classpathentry kind="con" path="org.eclipse.jdt.launching.JRE_CONTAINER"/> | ||
| <classpathentry kind="con" path="org.apache.ivyde.eclipse.cpcontainer.IVYDE_CONTAINER/?project=apache-nutch-2.3&ivyXmlPath=ivy%2Fivy.xml&confs=*&ivySettingsPath=&loadSettingsOnDemand=false&ivyUserDir=ivy%2Frepository&propertyFiles=&sourceTypes=source&javadocTypes=&sourceSuffixes=-source%2C-sources%2C-src&javadocSuffixes=&mapIfOnlyOneSource=false&mapIfOnlyOneJavadoc=false"/> | ||
| <classpathentry kind="src" path="src/java"/> | ||
| <classpathentry kind="src" path="src/plugin/creativecommons/src/java"/> | ||
| <classpathentry kind="src" path="src/plugin/creativecommons/src/test"/> | ||
| <classpathentry kind="src" path="src/plugin/indexer-solr/src/java"/> | ||
| <classpathentry kind="src" path="src/plugin/index-anchor/src/java"/> | ||
| <classpathentry kind="src" path="src/plugin/index-anchor/src/test"/> | ||
| <classpathentry kind="src" path="src/plugin/index-basic/src/java"/> | ||
| <classpathentry kind="src" path="src/plugin/index-basic/src/test"/> | ||
| <classpathentry kind="src" path="src/plugin/index-metadata/src/java"/> | ||
| <classpathentry kind="src" path="src/plugin/index-more/src/java"/> | ||
| <classpathentry kind="src" path="src/plugin/index-more/src/test"/> | ||
| <classpathentry kind="src" path="src/plugin/language-identifier/src/java"/> | ||
| <classpathentry kind="src" path="src/plugin/language-identifier/src/test"/> | ||
| <classpathentry kind="src" path="src/plugin/lib-http/src/java"/> | ||
| <classpathentry kind="src" path="src/plugin/lib-http/src/test"/> | ||
| <classpathentry kind="src" path="src/plugin/lib-regex-filter/src/java"/> | ||
| <classpathentry kind="src" path="src/plugin/lib-regex-filter/src/test"/> | ||
| <classpathentry kind="src" path="src/plugin/microformats-reltag/src/java"/> | ||
| <classpathentry kind="src" path="src/plugin/microformats-reltag/src/test"/> | ||
| <classpathentry kind="src" path="src/plugin/parse-html/src/java"/> | ||
| <classpathentry kind="src" path="src/plugin/parse-html/src/test"/> | ||
| <classpathentry kind="src" path="src/plugin/parse-js/src/java"/> | ||
| <classpathentry kind="src" path="src/plugin/parse-js/src/test"/> | ||
| <classpathentry kind="src" path="src/plugin/parse-metatags/src/java"/> | ||
| <classpathentry kind="src" path="src/plugin/parse-metatags/src/test"/> | ||
| <classpathentry kind="src" path="src/plugin/protocol-file/src/java"/> | ||
| <classpathentry kind="src" path="src/plugin/protocol-file/src/test"/> | ||
| <classpathentry kind="src" path="src/plugin/protocol-ftp/src/java"/> | ||
| <classpathentry kind="src" path="src/plugin/protocol-httpclient/src/java"/> | ||
| <classpathentry kind="src" path="src/plugin/protocol-httpclient/src/test"/> | ||
| <classpathentry kind="src" path="src/plugin/protocol-http/src/java"/> | ||
| <classpathentry kind="src" path="src/plugin/protocol-http/src/test"/> | ||
| <classpathentry kind="src" path="src/plugin/scoring-link/src/java"/> | ||
| <classpathentry kind="src" path="src/plugin/scoring-opic/src/java"/> | ||
| <classpathentry kind="src" path="src/plugin/scoring-opic/src/test"/> | ||
| <classpathentry kind="src" path="src/plugin/subcollection/src/java"/> | ||
| <classpathentry kind="src" path="src/plugin/subcollection/src/test"/> | ||
| <classpathentry kind="src" path="src/plugin/tld/src/java"/> | ||
| <classpathentry kind="src" path="src/plugin/tld/src/test"/> | ||
| <classpathentry kind="src" path="src/plugin/urlfilter-automaton/src/java"/> | ||
| <classpathentry kind="src" path="src/plugin/urlfilter-automaton/src/test"/> | ||
| <classpathentry kind="src" path="src/plugin/urlfilter-domain/src/java"/> | ||
| <classpathentry kind="src" path="src/plugin/urlfilter-domain/src/test"/> | ||
| <classpathentry kind="src" path="src/plugin/urlfilter-prefix/src/java"/> | ||
| <classpathentry kind="src" path="src/plugin/urlfilter-prefix/src/test"/> | ||
| <classpathentry kind="src" path="src/plugin/urlfilter-regex/src/java"/> | ||
| <classpathentry kind="src" path="src/plugin/urlfilter-regex/src/test"/> | ||
| <classpathentry kind="src" path="src/plugin/urlfilter-suffix/src/java"/> | ||
| <classpathentry kind="src" path="src/plugin/urlfilter-suffix/src/test"/> | ||
| <classpathentry kind="src" path="src/plugin/urlfilter-validator/src/java"/> | ||
| <classpathentry kind="src" path="src/plugin/urlfilter-validator/src/test"/> | ||
| <classpathentry kind="src" path="src/plugin/urlnormalizer-basic/src/java"/> | ||
| <classpathentry kind="src" path="src/plugin/urlnormalizer-basic/src/test"/> | ||
| <classpathentry kind="src" path="src/plugin/urlnormalizer-pass/src/java"/> | ||
| <classpathentry kind="src" path="src/plugin/urlnormalizer-pass/src/test"/> | ||
| <classpathentry kind="src" path="src/plugin/urlnormalizer-regex/src/java"/> | ||
| <classpathentry kind="src" path="src/plugin/urlnormalizer-regex/src/test"/> | ||
| <classpathentry kind="lib" path="src/bin"/> | ||
| <classpathentry kind="lib" path="lib/automaton-1.11-8.jar"/> | ||
| <classpathentry kind="lib" path="lib/tagsoup-1.2.1.jar"/> | ||
| <classpathentry kind="output" path="build/classes"/> | ||
| </classpath> |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,3 @@ | ||
| /runtime/ | ||
| /build/ | ||
| /hadoop.log |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,19 @@ | ||
| <?xml version="1.0" encoding="UTF-8"?> | ||
| <projectDescription> | ||
| <name>apache-nutch-2.3</name> | ||
| <comment> | ||
| </comment> | ||
| <projects> | ||
| </projects> | ||
| <buildSpec> | ||
| <buildCommand> | ||
| <name>org.eclipse.jdt.core.javabuilder</name> | ||
| <arguments> | ||
| </arguments> | ||
| </buildCommand> | ||
| </buildSpec> | ||
| <natures> | ||
| <nature>org.eclipse.jdt.core.javanature</nature> | ||
| <nature>org.apache.ivyde.eclipse.ivynature</nature> | ||
| </natures> | ||
| </projectDescription> |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,2 @@ | ||
| eclipse.preferences.version=1 | ||
| org.apache.ivyde.eclipse.standaloneretrieve=<?xml version\="1.0" encoding\="UTF-8" standalone\="no"?><setuplist/> |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,2 @@ | ||
| eclipse.preferences.version=1 | ||
| encoding/<project>=UTF-8 |
Oops, something went wrong.