Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Parquet: Implement Variant readers #12139

Merged
merged 12 commits into from
Feb 18, 2025
Merged

Parquet: Implement Variant readers #12139

Changes from 1 commit
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
72cb155
Parquet: Implement Variant readers.
rdblue Jan 31, 2025
8f3c5ee
Handle missing value column with shredded object fields.
rdblue Jan 31, 2025
ca1ac1a
Apply spotless.
rdblue Jan 31, 2025
79df73c
Fix incorrect imports.
rdblue Jan 31, 2025
7a29063
Fix checkstyle.
rdblue Jan 31, 2025
3a056d2
Fix checkstyle in tests.
rdblue Feb 1, 2025
927808a
Apply spotless.
rdblue Feb 1, 2025
7ece5b4
Update for rebase after #12167.
rdblue Feb 5, 2025
aac9823
Update DL -> DefinitionLevel
rdblue Feb 5, 2025
b1a2645
Fix value and typed_value conflict.
rdblue Feb 5, 2025
a6e9a93
Clean up now that core changes from #11831 are in.
rdblue Feb 18, 2025
d752b89
Remove unnecessary qualifier.
rdblue Feb 18, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Apply spotless.
  • Loading branch information
@rdblue
rdblue committed Feb 18, 2025
commit 927808aaa7a08a6ec09018119360ee929df082c0
3 changes: 2 additions & 1 deletion parquet/src/test/java/org/apache/iceberg/parquet/TestVariantReaders.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -573,7 +573,8 @@ public void testShreddedObjectWithOptionalFieldStructs() throws IOException {
GenericRecord recordA = record(fieldA, Map.of("value", serialize(Variants.of(34))));
GenericRecord recordB = record(fieldB, Map.of("typed_value", "iceberg"));
GenericRecord recordC = record(fieldC, Map.of()); // c.value and c.typed_value are missing
GenericRecord fields = record(objectFields, Map.of("a", recordA, "b", recordB, "c", recordC)); // d is missing
GenericRecord fields =
record(objectFields, Map.of("a", recordA, "b", recordB, "c", recordC)); // d is missing
GenericRecord variant =
record(variantType, Map.of("metadata", TEST_METADATA_BUFFER, "typed_value", fields));
GenericRecord record = record(parquetSchema, Map.of("id", 1, "var", variant));
Expand Down