Skip to content

Commit

Permalink
[SPARK-38979][SQL] Improve error log readability in OrcUtils.requeste…
Browse files Browse the repository at this point in the history
…dColumnIds

### What changes were proposed in this pull request?
Add detailed log in `OrcUtils#requestedColumnIds`.

### Why are the changes needed?
In `OrcUtils#requestedColumnIds` sometimes it fails because `orcFieldNames.length > dataSchema.length`, the log is not very clear.

```
java.lang.AssertionError: assertion failed: The given data schema struct<field1:int> has less fields than the actual ORC physical schema, no idea which columns were dropped, fail to read.
```

after the change
```
java.lang.AssertionError: assertion failed: The given data schema struct<field1:int> (length:1) has fewer 1 fields than the actual ORC physical schema struct<field1:int,field2:int> (length:2), no idea which columns were dropped, fail to read.
```

### Does this PR introduce _any_ user-facing change?
No

### How was this patch tested?
exist UT / local test

Closes apache#36296 from cxzl25/SPARK-38979.

Authored-by: sychen <[email protected]>
Signed-off-by: Sean Owen <[email protected]>
  • Loading branch information
cxzl25 authored and srowen committed Apr 27, 2022
1 parent 4e84f33 commit 70b4b1d
Showing 1 changed file with 3 additions and 1 deletion.
Original file line number Diff line number Diff line change
Expand Up @@ -224,7 +224,9 @@ object OrcUtils extends Logging {
// the physical schema doesn't match the data schema).
// In these cases we map the physical schema to the data schema by index.
assert(orcFieldNames.length <= dataSchema.length, "The given data schema " +
s"${dataSchema.catalogString} has less fields than the actual ORC physical schema, " +
s"${dataSchema.catalogString} (length:${dataSchema.length}) " +
s"has fewer ${orcFieldNames.length - dataSchema.length} fields than " +
s"the actual ORC physical schema $orcSchema (length:${orcFieldNames.length}), " +
"no idea which columns were dropped, fail to read.")
// for ORC file written by Hive, no field names
// in the physical schema, there is a need to send the
Expand Down

0 comments on commit 70b4b1d

Please sign in to comment.