[SPARK-39285][SQL] Spark should not check field names when reading files

### What changes were proposed in this pull request? Spark should not check filed name when reading data, in this pr we clean the code ### Why are the changes needed? Although spark can't write data with invalid col names, but Spark can read data with invalid col name, so we should not check field names when reading data. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? MT Closes apache#36661 from AngersZhuuuu/SPARK-39285. Authored-by: Angerszhuuuu <[email protected]> Signed-off-by: Wenchen Fan <[email protected]>
sluk3r · May 26, 2022 · 55ee406 · 55ee406
1 parent f673ebd
commit 55ee406
Show file tree

Hide file tree

Showing 2 changed files with 1 addition and 1 deletion.
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceUtils.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceUtils.scala
@@ -92,7 +92,6 @@ object DataSourceUtils extends PredicateHelper {
         throw QueryCompilationErrors.dataTypeUnsupportedByDataSourceError(format.toString, field)
       }
     }
-    checkFieldNames(format, schema)
   }
 
   // SPARK-24626: Metadata files and temporary files should not be

diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormatWriter.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormatWriter.scala
@@ -163,6 +163,7 @@ object FileFormatWriter extends Logging {
 
     val dataSchema = dataColumns.toStructType
     DataSourceUtils.verifySchema(fileFormat, dataSchema)
+    DataSourceUtils.checkFieldNames(fileFormat, dataSchema)
     // Note: prepareWrite has side effect. It sets "job".
     val outputWriterFactory =
       fileFormat.prepareWrite(sparkSession, job, caseInsensitiveOptions, dataSchema)
-Original file line number
+Diff line change
@@ Expand Up / @@ -92,7 +92,6 @@ object DataSourceUtils extends PredicateHelper { @@
             throw QueryCompilationErrors.dataTypeUnsupportedByDataSourceError(format.toString, field)
           }
         }
-        checkFieldNames(format, schema)
       }
       // SPARK-24626: Metadata files and temporary files should not be
@@ Expand Down @@