Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[SPARK-44795][CONNECT][TESTS][FOLLOWUP] Fix test case `REPL class in …
…UDF` for Scala 2.13 ### What changes were proposed in this pull request? This PR refactors the test case `REPL class in UDF` to ensure that Scala 2.12 and Scala 2.13 use the same results for assertion. ### Why are the changes needed? There are the following test failures in the daily testing of branch-3.5: - https://github.com/apache/spark/actions/runs/5866399652/job/15905157665 <img width="941" alt="image" src="https://github.com/apache/spark/assets/1475305/62dcd4eb-e01f-4a11-bf17-386129fdb08d"> ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? - Pass Github Actions - Manually check ``` dev/change-scala-version.sh 2.13 build/sbt clean "connect-client-jvm/testOnly org.apache.spark.sql.application.ReplE2ESuite" -Phive -Pscala-2.13 ``` **Before** ``` [info] case class MyTestClass(value: Int) defined class MyTestClass [info] [info] spark.range(2).map(i => MyTestClass(i.toInt)).collect() res56: Array[MyTestClass] = Array(MyTestClass(value = 0), MyTestClass(value = 1)) [info] [info] [info] semaphore.release() [info] Error Output: Compiling (synthetic)/ammonite/predef/ArgsPredef.sc [info] Compiling /Users/yangjie01/SourceCode/git/spark-mine-sbt/connector/connect/client/jvm/(console) (ReplE2ESuite.scala:111) [info] org.scalatest.exceptions.TestFailedException: [info] at org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:472) [info] at org.scalatest.Assertions.newAssertionFailedException$(Assertions.scala:471) [info] at org.scalatest.Assertions$.newAssertionFailedException(Assertions.scala:1231) [info] at org.scalatest.Assertions$AssertionsHelper.macroAssert(Assertions.scala:1295) [info] at org.apache.spark.sql.application.ReplE2ESuite.assertContains(ReplE2ESuite.scala:111) [info] at org.apache.spark.sql.application.ReplE2ESuite.$anonfun$new$14(ReplE2ESuite.scala:289) [info] at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18) [info] at org.apache.spark.sql.connect.client.util.RemoteSparkSession.$anonfun$test$1(RemoteSparkSession.scala:243) ... [info] Run completed in 41 seconds, 573 milliseconds. [info] Total number of tests run: 15 [info] Suites: completed 1, aborted 0 [info] Tests: succeeded 14, failed 1, canceled 0, ignored 0, pending 0 [info] *** 1 TEST FAILED *** ``` **After** ``` [info] ReplE2ESuite: [info] - Simple query (4 seconds, 334 milliseconds) [info] - UDF containing 'def' (1 second, 509 milliseconds) [info] - UDF containing in-place lambda (897 milliseconds) [info] - Updating UDF properties (904 milliseconds) [info] - SPARK-43198: Filter does not throw ammonite-related class initialization exception (1 second, 137 milliseconds) [info] - Client-side JAR (1 second, 329 milliseconds) [info] - Java UDF (950 milliseconds) [info] - Java UDF Registration (950 milliseconds) [info] - UDF Registration (932 milliseconds) [info] - UDF closure registration (807 milliseconds) [info] - call_udf (1 second, 235 milliseconds) [info] - call_function (884 milliseconds) [info] - Collect REPL generated class (1 second, 198 milliseconds) [info] - REPL class in UDF (732 milliseconds) [info] - streaming works with REPL generated code (2 seconds, 845 milliseconds) [info] Run completed in 41 seconds, 346 milliseconds. [info] Total number of tests run: 15 [info] Suites: completed 1, aborted 0 [info] Tests: succeeded 15, failed 0, canceled 0, ignored 0, pending 0 [info] All tests passed. ``` Closes apache#42508 from LuciferYang/SPARK-44795-FOLLOWUP. Authored-by: yangjie01 <[email protected]> Signed-off-by: yangjie01 <[email protected]>
- Loading branch information