Skip to content

Commit

Permalink
Use parquet rather than checkpoint to break lineage
Browse files Browse the repository at this point in the history
The reason for this is that checkpointing can cause the schema
to be lost if there are zero rows.  Whereas parquet preserves the schema
  • Loading branch information
RobinL committed Nov 25, 2024
1 parent 6996a2f commit 71e8ba7
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion tests/helpers.py
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,7 @@ def db_api_args(self):
return {
"spark_session": self.spark,
"num_partitions_on_repartition": 1,
"break_lineage_method": "checkpoint",
"break_lineage_method": "parquet",
}

def convert_frame(self, df):
Expand Down

0 comments on commit 71e8ba7

Please sign in to comment.