IMPALA-4355: random query generator: modify statement execution flow …

…to support DML

- Rework the discrepancy searcher to run DML statements. We do this by
  using the query profile to choose a table, copy that table, and
  generate a statement that will INSERT into that copy. We chose a slow
  copy over other methods because INSERTing into a copy is a more
  reliable test that prevents table sizes from getting out of hand or
  time-consuming replay to reproduce a particular statement.

- Introduce a statement generator stub. The real generator work is
  tracked in IMPALA-4351 and IMPALA-4353. Here we simply generate a
  basic INSERT INTO ... VALUES statement to make sure our general query
  execution flow is working.

- Add query profile stub for DML statements (INSERT-only at this time).
  Since we'll want INSERT INTO ... SELECT very soon, this inherits from
  DefaultProfile. Also add building blocks for choosing random
  statements in the DefaultProfile.

- Improve the concept of an "execution mode" and add new modes. Before,
  we had "RAW", "CREATE_TABLE_AS", and "CREATE_VIEW_AS". The idea here
  is that some random SELECT queries could be generated as "CREATE
  TABLE|VIEW AS" at execution time, based on weights in the query
  profile. First, we remove the use of raw string literals for this,
  since raw string literals can be error-prone, and introduce a
  StatementExecutionMode class to contain a namespace for the enumerated
  statement execution modes. Second, we introduce a couple new execution
  modes. The first is DML_SETUP: this is a DML statement that needs to
  be run in both the test and reference databases concurrently. For our
  purposes, it's the INSERT ... SELECT that copies data from the chosen
  random table into the table copy. The second is DML_TEST: this is a
  randomly-generated DML statement.

- Switch to using absolute imports in many places. There was a mix of
  absolute and relative imports happening here, and they were causing
  problems, especially when comparing data types. In Python,
  <class 'db_types.Int'> != <class 'tests.comparison.db_types.Int'>.
  Using
    from __future__ import absolute_import
  didn't seem to catch the relative import usage anyway, so I haven't
  employed that.

- Rename some, but not nearly all, names from "query" to "statement".
  Doing this is a rather large undertaking leading to much larger diffs
  and testing (IMPALA-4602).

- Fix a handful of flake8 warnings. There are a bunch that went unfixed
  for over- and under-indentation.

- Testing
  o ./discrepancy_searcher.py runs with and without --explain-only, and
  with --profile default and --profile dmlonly. For tpch_kudu data, it
  seems sufficient to use a --timeout of about 300.
  o Leopard run to make sure standard SELECT-only generation still works
  o Generated random stress queries locally
  o Generated random data locally

Change-Id: Ia4c63a2223185d0e056cc5713796772e5d1b8414
Reviewed-on: http://gerrit.cloudera.org:8080/5387
Reviewed-by: Jim Apple <[email protected]>
Tested-by: Impala Public Jenkins

tests/comparison/cli_options.py

@@ -23,8 +23,8 @@
 from getpass import getuser
 from tempfile import gettempdir
 
-import db_connection
-from cluster import (
+from tests.comparison import db_connection
+from tests.comparison.cluster import (
     CmCluster,
     DEFAULT_HIVE_HOST,
     DEFAULT_HIVE_PASSWORD,
@@ -33,7 +33,7 @@
     MiniCluster,
     MiniHiveCluster,
 )
-from db_types import TYPES
+from tests.comparison.db_types import TYPES
 
 
 def add_logging_options(parser, default_debug_log_file=None):

tests/comparison/common.py

@@ -33,7 +33,7 @@ def get_import(name):
   # noqa below tells flake8 to not warn when it thinks imports are not used
   global __ALREADY_IMPORTED
   if not __ALREADY_IMPORTED:
-    from db_types import (  # noqa
+    from tests.comparison.db_types import (  # noqa
         BigInt,
         Boolean,
         Char,
@@ -44,8 +44,8 @@ def get_import(name):
         JOINABLE_TYPES,
         Number,
         Timestamp)
-    from funcs import AggFunc, AnalyticFunc, Func  # noqa
-    from query import InlineView, Subquery, WithClauseInlineView  # noqa
+    from tests.comparison.funcs import AggFunc, AnalyticFunc, Func  # noqa
+    from tests.comparison.query import InlineView, Subquery, WithClauseInlineView  # noqa
     for key, value in locals().items():
       globals()[key] = value
     __ALREADY_IMPORTED = True

tests/comparison/data_generator.py

@@ -32,13 +32,13 @@
 from random import choice, randint, seed
 from time import time
 
-from data_generator_mapred_common import (
+from tests.comparison.data_generator_mapred_common import (
     estimate_rows_per_reducer,
     MB_PER_REDUCER,
     serialize,
     TextTableDataGenerator)
-from common import Column, Table
-from db_types import (
+from tests.comparison.common import Column, Table
+from tests.comparison.db_types import (
     Char,
     Decimal,
     EXACT_TYPES,
@@ -247,7 +247,7 @@ def _run_data_generator_mr_job(self, table_data_generators, db_name):
 if __name__ == '__main__':
   from argparse import ArgumentDefaultsHelpFormatter, ArgumentParser
 
-  import cli_options
+  from tests.comparison import cli_options
 
   parser = ArgumentParser(
       usage='usage: \n'

tests/comparison/data_generator_mapred_common.py

@@ -27,8 +27,8 @@
 import pickle
 import StringIO
 
-from db_types import Decimal
-from random_val_generator import RandomValGenerator
+from tests.comparison.db_types import Decimal
+from tests.comparison.random_val_generator import RandomValGenerator
 
 def serialize(value):
   '''Returns a serialized representation of 'value' suitable for use as a key in an MR

tests/comparison/db_connection.py

@@ -45,14 +45,14 @@
 from threading import Lock
 from time import time
 
-from common import (
+from tests.comparison.common import (
     ArrayColumn,
     Column,
     MapColumn,
     StructColumn,
     Table,
     TableExprList)
-from db_types import (
+from tests.comparison.db_types import (
     Char,
     Decimal,
     Double,