Following databricks blog post
the blog post is old, 2016. Updating to spark 3.0
- need to use new datetime formatting
using databricks runtime 7.6 (includes Apache Spark 3.0.1, Scala 2.12)
Spark UI of the cluster shows:
- Java Version 1.8.0_275 (Azul Systems, Inc.)
- Scala Version version 2.12.10