flaky mutex read write test

[creativepsyco]

mutex_readwrite_test.dart seems to be flaky.

From time to time I get this while running the tests for dart-mutex

Expected: a value less than <2100>...
Expected: a value less than <2100>
  Actual: <2205>
   Which: is not a value less than <2100>

Since DateTime.now() is used it might cause issues because of clock times.

from here:
https://github.com/hoylen/dart-mutex/blob/master/test/mutex_readwrite_test.dart#L171

expect(ms, greaterThan(delay * 10));
expect(ms, lessThan(delay * 10 + overhead));
expect(account.numWrites, equals(10));

Can this use a fake clock instead?

[hoylen]

A "fake clock" is not the solution here. The DateTime.now() is measuring the difference between the startTime and finishTime, to see how long the Futures in the test take to run. That is, it only cares about relative time and not absolute time.

I suspect the load on the machine (i.e. from processes other than the test) is causing the test to run slower than expected.

The test involves 10 tasks, each of them requiring a write lock and each taking a delay of 200 ms to run. Therefore, in theory they should take 2000 ms to run in series -- and they will run in series because they all require a write lock and at most one task can have the write lock at any instance of time.

The more important check is to see if it took at least 2000 ms to run. Because if it was quicker it would mean some of those tasks ran in parallel and the write lock did not work.

The check that is failing is less important. It checks if it took "too long" to run, being the 2000 ms plus an arbitrary overhead of 100 ms. But "too long" depends on many external factors. For example, if there was another process on the machine using up all the processing cycles then 100 ms might not be enough.

A simple solution is to delete that expect statement (line 173), since it is not critical for that test.

Another solution is to increase the times involved, so the test is less sensitive to external factors. For example, making the delay per task to be 2000 ms and the overhead 1000 ms. But that means the whole test will take at least 20 seconds to run.

When you run the tests, say 10 times, how many times does it fail? And when it fails, what is the "actual" value it prints out?

[creativepsyco]

I ran the test about 1000 times and it fails about 30 times ~ about 0.03 % flakiness rate.

The issue is that the test is not deterministic and hence will always be bound by external factors.

Some of the actual values in the test

Expected: a value less than <250>
  Actual: <321>
   Which: is not a value less than <250>

Expected: a value less than <250>
  Actual: <255>
   Which: is not a value less than <250>

Expected: a value less than <250>
  Actual: <865>
   Which: is not a value less than <250>

So you are right that some process on the machine might cause this issue. Hence the non-determinism.

dart analyzer has a similar mutex test
https://github.com/dart-lang/sdk/blob/master/pkg/analyzer/test/src/dart/analysis/mutex_test.dart

and instead of using time-bound checks it uses a check on the insertion order in a list. I ran this test by running it 10000 times and it never fails.

so for this I think the test can be changed to make each future sleep Xms * position and then insert its position to a list which can then be compared later on to check for anomalies.

What do you think?

[hoylen]

Making it deterministic, and not influenced by external timing factors, would be ideal.

Did you want to submit a pull-request for it, or do you want me to investigate?

[creativepsyco]

Let me try sending you a PR for this I will rework the RWTester class to be compatible with this.

[creativepsyco]

Sent one here:

#2