spark jar hell

itsoulltd · Sep 24, 2020 · 837652d · 837652d
1 parent 9764fe7
commit 837652d
Showing 1 changed file with 6 additions and 1 deletion.
diff --git a/spark.md b/spark.md
@@ -663,7 +663,6 @@ GraphLoader.edgeListFile(sc, path_to_file)
 ```
 
 
-
 # Scala examples
 ```
 val input = sc.textFile("data/stations/*")
@@ -689,6 +688,7 @@ spark.driver.extraClassPath  pathOfJarsWithCommaSeprated
 * after start
 scala> :require /path/to/file.jar
 
+
 # spark shell, spark-shell, spark2-shell
 ## local execution
 ```
@@ -735,6 +735,11 @@ spark-shell --master local --deploy-mode client \
 ...
 ```
 
+## avoid jar hell, library shadowing, different versions of libraries usage
+https://cloud.google.com/blog/products/data-analytics/managing-java-dependencies-apache-spark-applications-cloud-dataproc
+* shaded jar
+* maven relocations
+
 ## execute shell with additional parameters
 ```
 spark-shell --jars "lightning-1.5.0-SNAPSHOT-3a517e-jar-with-dependencies.jar" \