Skip to content

Commit

Permalink
KUDU-2670 Disable rowset compact for test of SplitKeyRange
Browse files Browse the repository at this point in the history
We found that in the SplitKeyRange related test, in most cases,
the number of KeyRange is equal to the number of tablets, but we
expect the number of KeyRange to be greater than the number of tablets.

In the test, the DRS generated by flush MRS will be small, rowset
compact will merge them to a big one. This will reduce the number
of DRS in the tablet. If end up with only one DRS in the tablet,
then the tablet will only have one key range.

So, I added --enable_rowset_compaction=false to disable rowset compact,
ensure that the number of KeyRange is greater than the number of tablets.

Change-Id: I89805a1fe2fd4d6c459e4abc0b1b9ec27c3e9588
Reviewed-on: http://gerrit.cloudera.org:8080/13517
Tested-by: Kudu Jenkins
Reviewed-by: Adar Dembo <[email protected]>
  • Loading branch information
oclarms authored and adembo committed Jun 6, 2019
1 parent c1c8e25 commit cfceb48
Show file tree
Hide file tree
Showing 2 changed files with 13 additions and 5 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -41,14 +41,16 @@ public class TestSplitKeyRange {
@Test
@TabletServerConfig(flags = {
"--flush_threshold_mb=1",
"--flush_threshold_secs=1"
"--flush_threshold_secs=1",
// Disable rowset compact to prevent DRSs being merged because they are too small.
"--enable_rowset_compaction=false"
})
public void testSplitKeyRange() throws Exception {
final KuduTable table = createTableWithOneThousandRows(
harness.getAsyncClient(), TABLE_NAME, 32 * 1024, DEFAULT_SLEEP);

// Wait for mrs flushed
Thread.sleep(10 * 1000);
Thread.sleep(5 * 1000);

Schema schema = table.getSchema();

Expand Down Expand Up @@ -158,7 +160,7 @@ public void testSplitKeyRange() throws Exception {
table, primaryKeyStart, primaryKeyEnd, null, null,
AsyncKuduClient.FETCH_TABLETS_PER_RANGE_LOOKUP,
1024, DEFAULT_SLEEP).join();
assertTrue(keyRanges.size() >= 4);
assertTrue(keyRanges.size() > 4);
for (KeyRange keyRange : keyRanges) {
int startKey = KeyEncoder.decodePrimaryKey(schema, keyRange.getPrimaryKeyStart()).getInt(0);
int endKey = KeyEncoder.decodePrimaryKey(schema, keyRange.getPrimaryKeyEnd()).getInt(0);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -1029,10 +1029,16 @@ class DefaultSourceTest extends KuduTestSuite with Matchers {
@TabletServerConfig(
flags = Array(
"--flush_threshold_mb=1",
"--flush_threshold_secs=1"
"--flush_threshold_secs=1",
// Disable rowset compact to prevent DRSs being merged because they are too small.
"--enable_rowset_compaction=false"
))
def testScanWithKeyRange() {
upsertRowsWithRowDataSize(table, rowCount * 100, 32 * 1024)

// Wait for mrs flushed
Thread.sleep(5 * 1000)

kuduOptions = Map(
"kudu.table" -> tableName,
"kudu.master" -> harness.getMasterAddressesAsString,
Expand All @@ -1045,6 +1051,6 @@ class DefaultSourceTest extends KuduTestSuite with Matchers {
val results = sqlContext.sql(s"SELECT * FROM $t").collectAsList()
assertEquals(rowCount * 100, results.size())
}
assert(actualNumTasks >= 2)
assert(actualNumTasks > 2)
}
}

0 comments on commit cfceb48

Please sign in to comment.