Skip to content

Commit

Permalink
[SPARK-42673][BUILD] Make build/mvn build Spark only with the verif…
Browse files Browse the repository at this point in the history
…ied maven version

### What changes were proposed in this pull request?
`build/mvn` tends to use the new maven version to build Spark now, and GA starts to use 3.9.0 as the default maven version.

But there may be some uncertain factors when building Spark with unverified version.

For example, `java-11-17` GA build task build with maven 3.9.0 has many error logs in master like follow:

```
Error: [ERROR] An error occurred attempting to read POM
org.codehaus.plexus.util.xml.pull.XmlPullParserException: UTF-8 BOM plus xml decl of ISO-8859-1 is incompatible (position: START_DOCUMENT seen <?xml version="1.0" encoding="ISO-8859-1"... 1:42)
    at org.codehaus.plexus.util.xml.pull.MXParser.parseXmlDeclWithVersion (MXParser.java:3423)
    at org.codehaus.plexus.util.xml.pull.MXParser.parseXmlDecl (MXParser.java:3345)
    at org.codehaus.plexus.util.xml.pull.MXParser.parsePI (MXParser.java:3197)
    at org.codehaus.plexus.util.xml.pull.MXParser.parseProlog (MXParser.java:1828)
    at org.codehaus.plexus.util.xml.pull.MXParser.nextImpl (MXParser.java:1757)
    at org.codehaus.plexus.util.xml.pull.MXParser.next (MXParser.java:1375)
    at org.apache.maven.model.io.xpp3.MavenXpp3Reader.read (MavenXpp3Reader.java:3940)
    at org.apache.maven.model.io.xpp3.MavenXpp3Reader.read (MavenXpp3Reader.java:612)
    at org.apache.maven.model.io.xpp3.MavenXpp3Reader.read (MavenXpp3Reader.java:627)
    at org.cyclonedx.maven.BaseCycloneDxMojo.readPom (BaseCycloneDxMojo.java:759)
    at org.cyclonedx.maven.BaseCycloneDxMojo.readPom (BaseCycloneDxMojo.java:746)
    at org.cyclonedx.maven.BaseCycloneDxMojo.retrieveParentProject (BaseCycloneDxMojo.java:694)
    at org.cyclonedx.maven.BaseCycloneDxMojo.getClosestMetadata (BaseCycloneDxMojo.java:524)
    at org.cyclonedx.maven.BaseCycloneDxMojo.convert (BaseCycloneDxMojo.java:481)
    at org.cyclonedx.maven.CycloneDxMojo.execute (CycloneDxMojo.java:70)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:126)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2 (MojoExecutor.java:342)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:330)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:213)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:175)
    at org.apache.maven.lifecycle.internal.MojoExecutor.access$000 (MojoExecutor.java:76)
    at org.apache.maven.lifecycle.internal.MojoExecutor$1.run (MojoExecutor.java:163)
    at org.apache.maven.plugin.DefaultMojosExecutionStrategy.execute (DefaultMojosExecutionStrategy.java:39)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:160)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:105)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:73)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:53)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:118)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:260)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:172)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:100)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:821)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:270)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:192)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:77)
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:568)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347)
```

So this pr change the version check condition of `build/mvn` to make it build Spark only with the verified maven version.

### Why are the changes needed?
Make `build/mvn` build Spark only with the verified maven version

### Does this PR introduce _any_ user-facing change?
No

### How was this patch tested?

- `java-11-17` GA build task pass and no error message as above
- Manual test:

1. Make the system use maven 3.9.0( > 3.8.7 ) by default:

run `mvn -version`

```
Apache Maven 3.9.0 (9b58d2bad23a66be161c4664ef21ce219c2c8584)
Maven home: /Users/${userName}/Tools/maven
Java version: 1.8.0_362, vendor: Azul Systems, Inc., runtime: /Users/${userName}/Tools/zulu8/zulu-8.jdk/Contents/Home/jre
Default locale: zh_CN, platform encoding: UTF-8
OS name: "mac os x", version: "13.2.1", arch: "aarch64", family: "mac"
```

and run `build/mvn -version`

```
Using `mvn` from path: /${basedir}/spark/build/apache-maven-3.8.7/bin/mvn
Using SPARK_LOCAL_IP=localhost
Apache Maven 3.8.7 (b89d5959fcde851dcb1c8946a785a163f14e1e29)
Maven home: /${basedir}/spark/build/apache-maven-3.8.7
Java version: 1.8.0_362, vendor: Azul Systems, Inc., runtime: /Users/${userName}/Tools/zulu8/zulu-8.jdk/Contents/Home/jre
Default locale: zh_CN, platform encoding: UTF-8
OS name: "mac os x", version: "13.2.1", arch: "aarch64", family: "mac"
```

We can see Spark use 3.8.7 in build directory when the system default maven > 3.8.7

2. Make the system use maven 3.8.7 by default:

run `mvn -version`

```
mvn -version
Apache Maven 3.8.7 (b89d5959fcde851dcb1c8946a785a163f14e1e29)
Maven home: /Users/${userName}/Tools/maven
Java version: 1.8.0_362, vendor: Azul Systems, Inc., runtime: /Users/${userName}/Tools/zulu8/zulu-8.jdk/Contents/Home/jre
Default locale: zh_CN, platform encoding: UTF-8
OS name: "mac os x", version: "13.2.1", arch: "aarch64", family: "mac"
```

and run `build/mvn -version`

```
Using `mvn` from path: /Users/${userName}/Tools/maven/bin/mvn
Using SPARK_LOCAL_IP=localhost
Apache Maven 3.8.7 (b89d5959fcde851dcb1c8946a785a163f14e1e29)
Maven home: /Users/${userName}/Tools/maven
Java version: 1.8.0_362, vendor: Azul Systems, Inc., runtime: /Users/${userName}/Tools/zulu8/zulu-8.jdk/Contents/Home/jre
Default locale: zh_CN, platform encoding: UTF-8
OS name: "mac os x", version: "13.2.1", arch: "aarch64", family: "mac"
```

We can see Spark use system default maven 3.8.7 when the system default maven is 3.8.7.

3. Make the system use maven 3.8.6( < 3.8.7 ) by default:

run `mvn -version`

```
mvn -version
Apache Maven 3.8.6 (84538c9988a25aec085021c365c560670ad80f63)
Maven home: /Users/${userName}/Tools/maven
Java version: 1.8.0_362, vendor: Azul Systems, Inc., runtime: /Users/${userName}/Tools/zulu8/zulu-8.jdk/Contents/Home/jre
Default locale: zh_CN, platform encoding: UTF-8
OS name: "mac os x", version: "13.2.1", arch: "aarch64", family: "mac"
```

and run `build/mvn -version`

```
Using `mvn` from path: /Users/${userName}/Tools/maven/bin/mvn
Using SPARK_LOCAL_IP=localhost
Apache Maven 3.8.7 (b89d5959fcde851dcb1c8946a785a163f14e1e29)
Maven home: /Users/${userName}/Tools/maven
Java version: 1.8.0_362, vendor: Azul Systems, Inc., runtime: /Users/${userName}/Tools/zulu8/zulu-8.jdk/Contents/Home/jre
Default locale: zh_CN, platform encoding: UTF-8
OS name: "mac os x", version: "13.2.1", arch: "aarch64", family: "mac"
```

We can see Spark use 3.8.7 in build directory when the system default maven < 3.8.7.

Closes apache#40283 from LuciferYang/ban-maven-3.9.x.

Authored-by: yangjie01 <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
  • Loading branch information
LuciferYang authored and dongjoon-hyun committed Mar 6, 2023
1 parent 252e7a3 commit f70b8cf
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion build/mvn
Original file line number Diff line number Diff line change
Expand Up @@ -119,7 +119,7 @@ install_mvn() {
if [ "$MVN_BIN" ]; then
local MVN_DETECTED_VERSION="$(mvn --version | head -n1 | awk '{print $3}')"
fi
if [ $(version $MVN_DETECTED_VERSION) -lt $(version $MVN_VERSION) ]; then
if [ $(version $MVN_DETECTED_VERSION) -ne $(version $MVN_VERSION) ]; then
local MVN_TARBALL="apache-maven-${MVN_VERSION}-bin.tar.gz"
local FILE_PATH="maven/maven-3/${MVN_VERSION}/binaries/${MVN_TARBALL}"
local APACHE_MIRROR=${APACHE_MIRROR:-'https://www.apache.org/dyn/closer.lua'}
Expand Down

0 comments on commit f70b8cf

Please sign in to comment.