crawler-engine

INTRODUCE

crawler-engine with HTTP, proxy, JS-Java Interoperability, MQ task consumption, dynamic crawler scripts execution. support deployment in distribution style.

REQUIREMENTS

• JDK 8 (note: if require Nashorn feature ClassFilter, then 1.8u45+ is mandatory);
• build tool: Maven 3.2.x+;
• MQ service(Erlang OTP): Rabbitmq 3.6.0+;
• Mozilla Rhino
• Apache HttpComponents

RUN STEPS

1. BUILD: mvn clean package -Dmaven.test.skip, should have generated target/engineClient.zip;
2. PREPARE: follow the instructions of rabbitmq_quickstart.txt;
3. TESTCASE: mvn test;
4. RUN: java -jar engineClient.jar [clientName] in target directory.