Skip to content

Latest commit

 

History

History
 
 

stateful

Stateful Tests

https://eth-brownie.readthedocs.io/en/stable/tests-hypothesis-stateful.html

Stateful tests are used here to try and find bugs that arise from complex interactions both between functions in the same contract and functions of different contracts. Stateful tests execute 'rule_'s of a test in random order (here, the rules are always a single function call) usually with random inputs (here, the inputs are almost always random, testing both 'valid' values and that they revert with 'invalid' values, such as when the amount staked is below the minimum requirement). After every rule is called, all 'invariant_' methods are called to test that every piece of state that's of interest is what it should be (though I test basically every piece of state possible).

There's a stateful test for each contract on its own where only functions in that contract are called (test_vault.py etc), and 1 big stateful test that has everything tognativeer (test_all.py). Since the purpose of these tests is to test what happens with complex and unexpected interactions of state-chainging (i.e. valid, non-reverting) transactions, while still allowing for the testing of some reverting transactions sometimes to make sure things do indeed revert when they're supposed to after complex interactions, I've set the test parameters (min/max of random values, how many user addresses are used etc) so that about 80-90% of transactions are valid. This can be checked visually by adding the -s flag to tests which prints output, such as brownie test ./tests/stateful/test_vault.py -s. With the -s flag, every time a rule/tx is called, it prints out which mnativeod is called and with what parameters. Prints from valid transactions are indented the most (20 spaces), and transactions that revert print with a.scgall indent (8 spaces), with the revert reason as the first text. The valid transactions are still indented a bit because if an error is found in the stateful test, Brownie/Hypothesis will show a list of all rules (not indented) that were executed in the run that resulted in the failure along with the prints in chronologic order, and it can be confusing to distinguish between the custom prints and the prints from Brownie/Hypothesis.

The amount of times each test is run (as in from a fresh deployment) can be changed in the settings variable at the top of each test with the max_examples part, and the maximum number of rules to call in each run/example is set with stateful_step_count. Note that stateful_step_count is the maximum number of rules - Brownie/Hypothesis will decide on a random number of rules to call in each run/example. I've found stateful_step_count to be a bit buggy sometimes - ideally we want to use an artbitrarily high number (like a million?) to catch any really obscure bugs, but it usually chooses a number around 500 as the typical max even if the setting is higher than that - with a high stateful_step_count the test will sometimes return an error saying that some of the runs/examples were to large, which doesn't mean the test has failed, it's just a warning as to the intended size of stateful_step_count using Hypothesis and I'm quite sure that it can be ignored.