The deleteNonMatched incremental table option

[AaronRobins]

Hey Dataform Team,

First let me say thanks for building a super useful platform big fan.

disclaimer - this may be a terrible idea, I am not across the full repository and its use cases.

1. Currently, incremental tables hold onto rows which are in the target, but are no longer in the source
2. This is because of the MERGE operation which is 'under the hood'
3. Although its not a common use case, it can happen a row is dropped from source data and we too want that row dropped in the target table

My proposed solution would be:

• Allow for a config item called deleteNonMatched which is a boolean and only available for incremental tables
• If this config item is set to true, then the compiler will add in the following line to the stored procedure

WHEN NOT MATCHED BY SOURCE THEN DELETE

Currently the only workaround solutions I have found it to use a post operation or abandon the Incremental table all together and use a custom operation instead.