From dd0decff5f1e95cedd8fe83de7e4449be57cb31c Mon Sep 17 00:00:00 2001 From: Kousuke Saruta Date: Fri, 24 Dec 2021 11:29:37 +0900 Subject: [PATCH] [SPARK-37302][BUILD][FOLLOWUP] Extract the versions of dependencies accurately ### What changes were proposed in this pull request? This PR changes `dev/test-dependencies.sh` to extract the versions of dependencies accurately. In the current implementation, the versions are extracted like as follows. ``` GUAVA_VERSION=`build/mvn help:evaluate -Dexpression=guava.version -q -DforceStdout` ``` But, if the output of the `mvn` command includes not only the version but also other messages like warnings, a following command referring the version will fail. ``` build/mvn dependency:get -Dartifact=com.google.guava:guava:${GUAVA_VERSION} -q ... [ERROR] Failed to execute goal org.apache.maven.plugins:maven-dependency-plugin:3.1.1:get (default-cli) on project spark-parent_2.12: Couldn't download artifact: org.eclipse.aether.resolution.DependencyResolutionException: com.google.guava:guava:jar:Falling was not found in https://maven-central.storage-download.googleapis.com/maven2/ during a previous attempt. This failure was cached in the local repository and resolution is not reattempted until the update interval of gcs-maven-central-mirror has elapsed or updates are forced -> [Help 1] ``` Actually, this causes the recent linter failure. https://github.com/apache/spark/runs/4623297663?check_suite_focus=true ### Why are the changes needed? To recover the CI. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Manually run `dev/test-dependencies.sh`. Closes #35006 from sarutak/followup-SPARK-37302. Authored-by: Kousuke Saruta Signed-off-by: Kousuke Saruta --- dev/test-dependencies.sh | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/dev/test-dependencies.sh b/dev/test-dependencies.sh index cf0512692c2bf..2268a262d5f8f 100755 --- a/dev/test-dependencies.sh +++ b/dev/test-dependencies.sh @@ -50,9 +50,9 @@ OLD_VERSION=$($MVN -q \ --non-recursive \ org.codehaus.mojo:exec-maven-plugin:1.6.0:exec | grep -E '[0-9]+\.[0-9]+\.[0-9]+') # dependency:get for guava and jetty-io are workaround for SPARK-37302. -GUAVA_VERSION=`build/mvn help:evaluate -Dexpression=guava.version -q -DforceStdout` +GUAVA_VERSION=$(build/mvn help:evaluate -Dexpression=guava.version -q -DforceStdout | grep -E "^[0-9.]+$") build/mvn dependency:get -Dartifact=com.google.guava:guava:${GUAVA_VERSION} -q -JETTY_VERSION=`build/mvn help:evaluate -Dexpression=jetty.version -q -DforceStdout` +JETTY_VERSION=$(build/mvn help:evaluate -Dexpression=jetty.version -q -DforceStdout | grep -E "^[0-9.]+v[0-9]+") build/mvn dependency:get -Dartifact=org.eclipse.jetty:jetty-io:${JETTY_VERSION} -q if [ $? != 0 ]; then echo -e "Error while getting version string from Maven:\n$OLD_VERSION"