Skip to content

Commit

Permalink
[SPARK-22315][SPARKR] Warn if SparkR package version doesn't match Sp…
Browse files Browse the repository at this point in the history
…arkContext

## What changes were proposed in this pull request?

This PR adds a check between the R package version used and the version reported by SparkContext running in the JVM. The goal here is to warn users when they have a R package downloaded from CRAN and are using that to connect to an existing Spark cluster.

This is raised as a warning rather than an error as users might want to use patch versions interchangeably (e.g., 2.1.3 with 2.1.2 etc.)

## How was this patch tested?

Manually by changing the `DESCRIPTION` file

Author: Shivaram Venkataraman <[email protected]>

Closes apache#19624 from shivaram/sparkr-version-check.
  • Loading branch information
shivaram committed Nov 6, 2017
1 parent c7f38e5 commit 65a8bf6
Showing 1 changed file with 12 additions and 0 deletions.
12 changes: 12 additions & 0 deletions R/pkg/R/sparkR.R
Original file line number Diff line number Diff line change
Expand Up @@ -420,6 +420,18 @@ sparkR.session <- function(
enableHiveSupport)
assign(".sparkRsession", sparkSession, envir = .sparkREnv)
}

# Check if version number of SparkSession matches version number of SparkR package
jvmVersion <- callJMethod(sparkSession, "version")
# Remove -SNAPSHOT from jvm versions
jvmVersionStrip <- gsub("-SNAPSHOT", "", jvmVersion)
rPackageVersion <- paste0(packageVersion("SparkR"))

if (jvmVersionStrip != rPackageVersion) {
warning(paste("Version mismatch between Spark JVM and SparkR package. JVM version was",
jvmVersion, ", while R package version was", rPackageVersion))
}

sparkSession
}

Expand Down

0 comments on commit 65a8bf6

Please sign in to comment.