Skip to content

Latest commit

 

History

History
62 lines (41 loc) · 2.27 KB

check.md

File metadata and controls

62 lines (41 loc) · 2.27 KB

Check data

After data migration, you may want to compare the source data and the target data. If there are too many records, try sampling check. Before you start, please ensure that the tables to be verified have primary/unique keys.

MySQL/PG/Mongo are currently supported for data check.

Example: MySQL -> MySQL

Refer to task templates and tutorial

Sampling check

Based on full check configuration, add sample_interval for sampling check. The following code means that every 3 records will be sampled once.

[extractor]
sample_interval=3

Note

While this configuration is similar to that of snapshot migration, the only differences are:

[sinker]
sink_type=check

[parallelizer]
parallel_type=rdb_check

Results

The results are written to logs in JSON format, including diff.log and miss.log. The logs are stored in the log/check subdirectory.

diff.log

The diff log includes the database (schema), table (tb), primary key/unique key (id_col_values), and the source and target values of the differing columns (diff_col_values).

{"log_type":"Diff","schema":"test_db_1","tb":"one_pk_multi_uk","id_col_values":{"f_0":"5"},"diff_col_values":{"f_1":{"src":"5","dst":"5000"}}}
{"log_type":"Diff","schema":"test_db_1","tb":"one_pk_no_uk","id_col_values":{"f_0":"4"},"diff_col_values":{"f_1":{"src":"2","dst":"1"}}}
{"log_type":"Diff","schema":"test_db_1","tb":"one_pk_no_uk","id_col_values":{"f_0":"6"},"diff_col_values":{"f_1":{"src":null,"dst":"1"}}}

miss.log

The miss log includes the database (schema), table (tb), and primary key/unique key (id_col_values), with empty diff_col_values.

{"log_type":"Miss","schema":"test_db_1","tb":"no_pk_one_uk","id_col_values":{"f_1":"8","f_2":"1"},"diff_col_values":{}}
{"log_type":"Miss","schema":"test_db_1","tb":"no_pk_one_uk","id_col_values":{"f_1":null,"f_2":null},"diff_col_values":{}}
{"log_type":"Miss","schema":"test_db_1","tb":"one_pk_multi_uk","id_col_values":{"f_0":"7"},"diff_col_values":{}}

Other configurations

  • For [filter] and [router], refer to config details.
  • Refer to task_config.ini in tests:
    • dt-tests/tests/mysql_to_mysql/check
    • dt-tests/tests/pg_to_pg/check
    • dt-tests/tests/mongo_to_mongo/check