Schema Scrubbers/Validators - Combine and Enable MapType Recursive Cleansing

[GeekSheikh]

overwatch/src/main/scala/com/databricks/labs/overwatch/utils/Tools.scala

Line 503 in 9d76816

def validateSchema(

AND

overwatch/src/main/scala/com/databricks/labs/overwatch/utils/Tools.scala

Line 364 in 9d76816

def scrubSchema(df: DataFrame): DataFrame = {

can be combined to accomplish all schema scrubbing needs (I think).

Additionally, MapTypes should be added to supported validations.

[GeekSheikh]

ArrayType was enabled with 0.5.0

[GeekSheikh]

0.6.0.4 allows for the following syntax and much more robust schema scrubbing options. The updates are backwards compatible except the SchemaScrubber has moved from SchemaTools to its own object SchemaScrubber

df.scrubSchema
df.scrubSchema(scrubberOverrideRules)

[GeekSheikh]

Will integrate this with the updated schema recursion strategy that is being defined. #397

[GeekSheikh]

Also closely related to #395