Quickly complete unit tests for mutation testing

Question

One of the problems when testing a mutation is that it is slow, because by default you run a full test run (either a test file or a set of test files) for each mutation you create.

One way to speed up mutation testing is to stop the test run for a given mutant after one failure is detected (but only during mutation testing). Even better would be for the mutation tester to remember what was the first test to kill the last mutant and give it to the first mutant. Is there anything in ruby ​​that does any of these things, or is this the best way to start beheading monkeys?

(Yes, I know that unit tests should be fast. And showing all failed tests is useful outside mutation testing, as it helps you not only identify something wrong, but exactly where it goes wrong)

Edit : I am currently using heckle with test / unit. If the test / block cannot remember which tests fail between runs, perhaps a heckle or something working heckle might remember it.

Answer 1

It is best to check the heckle source from github, fix it, and send this patch to the developers. You should be able to write your own test runner for heckle.

A monkey patch is never an answer to something like that. In fact, correcting monkeys is almost never the answer to everything.

Answer 2

One approach that I started to use was to write unit tests for each method and insert them into separate files organized as rubyspec. I run each specification separately, specifying the exact method I want to use. I have a rake task that manages all this and prints a report at the end with errors, if any.

In the end, I get full coverage of each method, not expecting results. Plus, it's even better than the usual approach that everyone uses because I don't get any random coverage - every method should cover all the mutations of this method.

Answer 3

My mutant tool uses the rspec2 --fail-fast parameter to stop immediately as soon as an unsuccessful example appears. Together with the --rspec-dm2 strategy, which only performs unit tests under test, we get very fast mutation coverage testing. See asciicast for a demonstration (speed).