[LIVY-399][TEST] Enable real test for PySpark and SparkR interpreters

Currently because we lack pyspark and sparkr dependencies in our environment, so we neglect the pyspark and sparkr integration tests, here propose a way to enable these integration tests. The solution is mainly borrowed from Zeppelin - to download Spark binary before integration tests, so that we will have a complete Spark environment to do the test. Because of this several modules like minicluster-dependencies are not required, so here removed such modules.

Besides, we proposed to remove real test implementations, since we never used real test before and it mainly did what scripts should do with Scala code, which makes it hard to maintain. Also for now mini-cluster test is quite close to real test, so it is not so useful to still keep real test.

Author: jerryshao <[email protected]>

Closes apache#49 from jerryshao/LIVY-399.

.travis.yml

@@ -57,17 +57,17 @@ before_install:
   - sudo apt-get -y install python3-pip python-dev
   - sudo apt-get -y install libkrb5-dev
   - sudo apt-get -y remove python-setuptools
-  - pip2 install --user --upgrade pip "setuptools < 36"
-  - pip3 install --user --upgrade pip "setuptools < 36"
-  - pip2 install --user codecov cloudpickle
-  - pip3 install --user cloudpickle
+  - sudo pip2 install --upgrade pip "setuptools < 36"
+  - sudo pip3 install --upgrade pip "setuptools < 36"
+  - sudo pip2 install codecov cloudpickle
+  - sudo pip3 install cloudpickle
 
 install:
   - mvn $MVN_FLAG install -Dskip -DskipTests -DskipITs -Dmaven.javadoc.skip=true -B -V
 
 before_script:
-  - pip2 install --user "requests >= 2.10.0" pytest flaky flake8 requests-kerberos
-  - pip3 install --user "requests >= 2.10.0" pytest flaky requests-kerberos
+  - sudo pip2 install "requests >= 2.10.0" pytest flaky flake8 requests-kerberos
+  - sudo pip3 install "requests >= 2.10.0" pytest flaky requests-kerberos
 
 script:
   - mvn $MVN_FLAG verify -e
@@ -76,5 +76,4 @@ after_success:
   - bash <(curl -s https://codecov.io/bash)
 
 after_failure:
-  - npm list -g azure-cli --depth=0 || npm install -g azure-cli
-  - ./dev/travis-upload-logs.sh
+  - for log in `find * -name "*.log" -o -name "stderr" -o -name "stdout"`; do echo "=========$log========="; cat $log; done