Is it possible for the Common Test to run randomly?

Question

I have a couple dozen test test case modules for the Erlang application I wrote. All tests pass, but I feel the test kits are very fragile. Reordering the tests causes some of them to fail. I did not read the dependency chapter in the general test documentation , and I often made an assumption about the state of the application in unit tests. Now I would like to make my test suite more reliable.

Randomized test order

Coming from Ruby, where Rspec runs tests in random order, I would like to have the same functionality in Common Test. Does anyone know if there is a way to randomize a test order in Common Test? I did not see anything in the test order randomization docs.

Randomize return values ​​from all/0 and groups/0 ?

I also thought about changing the output of all/0 and groups/0 callbacks. Right now, they are simply returning hard-copied lists. Perhaps I could randomize the ordering of the elements and run them in different orders each time? Does anyone have any experience randomizing test order by changing return callback values ​​in Common Test? I will also need a way to repeat the tests in the order in which they would fail, for example Rspec --seed flag .

Thanks in advance!

Answer 1

Using the shuffle or {shuffle, Seed} property in defining test groups can be useful, as follows:

 groups() -> [{group1, [shuffle], [test1, test2, test3]}, {group2, [shuffle], [test1, test2, test3]}]. 

If a random case is specified, the cases in the group will be executed in random order. There are good examples in this white paper.