Issue materialize data: Comparing two tables

[stb-ssk]

Hello!

So my goal is to compare data between two ibis tables. Both tables have the same schema, but may have different values. I have ordered the tables by a unique row, so that I can compare cells between the same column in both tables.

I am iterating through all the columns in table and using .isin() to compare whether all the cell values match. But when I try to materialize the result as a csv or pandas, I timeout on the backend.

Any thoughts?

[cpcloud]

Is it possible to join on the unique row id, and then do a loop to compare columns from each table?

Something like this:

joined = left.join(right, on="unique_row_id")

equals = {
    f"{column}_equal": joined[column] == joined[f"{column}_right"]
    for column in left.columns
}

joined.select(**equals).to_pandas()

[stb-ssk]

Hmm will this create a truth table though, on whether each value matched up. That is the end goal.

[cpcloud]

Modified it a bit to produce a truth table.