Skip to content

Commit

Permalink
[SPARK-37302][BUILD][FOLLOWUP] Extract the versions of dependencies a…
Browse files Browse the repository at this point in the history
…ccurately

### What changes were proposed in this pull request?

This PR changes `dev/test-dependencies.sh` to extract the versions of dependencies accurately.
In the current implementation, the versions are extracted like as follows.

```
GUAVA_VERSION=`build/mvn help:evaluate -Dexpression=guava.version -q -DforceStdout`
```

But, if the output of the `mvn` command includes not only the version but also other messages like warnings, a following command referring the version will fail.

```
build/mvn dependency:get -Dartifact=com.google.guava:guava:${GUAVA_VERSION} -q
...
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-dependency-plugin:3.1.1:get (default-cli) on project spark-parent_2.12: Couldn't download artifact: org.eclipse.aether.resolution.DependencyResolutionException: com.google.guava:guava:jar:Falling was not found in https://maven-central.storage-download.googleapis.com/maven2/ during a previous attempt. This failure was cached in the local repository and resolution is not reattempted until the update interval of gcs-maven-central-mirror has elapsed or updates are forced -> [Help 1]
```

Actually, this causes the recent linter failure.
https://github.com/apache/spark/runs/4623297663?check_suite_focus=true

### Why are the changes needed?

To recover the CI.

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

Manually run `dev/test-dependencies.sh`.

Closes apache#35006 from sarutak/followup-SPARK-37302.

Authored-by: Kousuke Saruta <[email protected]>
Signed-off-by: Kousuke Saruta <[email protected]>
  • Loading branch information
sarutak committed Dec 24, 2021
1 parent 6cc4c90 commit dd0decf
Showing 1 changed file with 2 additions and 2 deletions.
4 changes: 2 additions & 2 deletions dev/test-dependencies.sh
Original file line number Diff line number Diff line change
Expand Up @@ -50,9 +50,9 @@ OLD_VERSION=$($MVN -q \
--non-recursive \
org.codehaus.mojo:exec-maven-plugin:1.6.0:exec | grep -E '[0-9]+\.[0-9]+\.[0-9]+')
# dependency:get for guava and jetty-io are workaround for SPARK-37302.
GUAVA_VERSION=`build/mvn help:evaluate -Dexpression=guava.version -q -DforceStdout`
GUAVA_VERSION=$(build/mvn help:evaluate -Dexpression=guava.version -q -DforceStdout | grep -E "^[0-9.]+$")
build/mvn dependency:get -Dartifact=com.google.guava:guava:${GUAVA_VERSION} -q
JETTY_VERSION=`build/mvn help:evaluate -Dexpression=jetty.version -q -DforceStdout`
JETTY_VERSION=$(build/mvn help:evaluate -Dexpression=jetty.version -q -DforceStdout | grep -E "^[0-9.]+v[0-9]+")
build/mvn dependency:get -Dartifact=org.eclipse.jetty:jetty-io:${JETTY_VERSION} -q
if [ $? != 0 ]; then
echo -e "Error while getting version string from Maven:\n$OLD_VERSION"
Expand Down

0 comments on commit dd0decf

Please sign in to comment.