[LIVY-380] Livy should overwrite spark.sql.catalogImplementation if H…

…ive classes are not present on classpath

In SparkContextInitializer.spark2CreateContext, livy checks for whether catalog implementation is "hive" or "in-memory" to properly initialize the SparkSession.
If the config is set to "hive" (this is the case if livy.repl.enable-hive-context is set to "true"), livy will further check if hive classes are present or not.
It only invokes the enableHiveSupport method of SparkSession.Builder if hive classes are found.
If not, it proceeds to initializing a SparkSession without Hive support enabled.
However, in Spark's code base, when creating SparkSession, it checks for spark.sql.catalogImplementation to determine which session state to initialize.
If spark.sql.catalogImplementation is set to "hive", it will initialize a HiveSessionState.
However, since Livy has already checked that hive classes are not present, it should not go ahead invoking SparkSession.builder.getOrCreate without overwriting spark.sql.catalogImplementation to "in-memory".

Author: Min Shen <[email protected]>

Closes apache#24 from Victsm/master.

repl/src/main/scala/org/apache/livy/repl/SparkContextInitializer.scala

@@ -93,6 +93,8 @@ trait SparkContextInitializer extends Logging {
         spark = builder.getClass.getMethod("getOrCreate").invoke(builder)
         info("Created Spark session (with Hive support).")
       } else {
+        builder.getClass.getMethod("config", classOf[String], classOf[String])
+          .invoke(builder, "spark.sql.catalogImplementation", "in-memory")
         spark = builder.getClass.getMethod("getOrCreate").invoke(builder)
         info("Created Spark session.")
       }