How can I retry a property-based test if randomly generated inputs are not useful?

Question

I am testing n00b for unit testing. I installed FsCheck.Nunit and NUnitTestAdapter from Nuget, and I'm trying to do property-based testing, much inspired by the invaluable Scott Ulashin .

I use the [<Property>] attribute, and I need the ability to skip inputs that do not meet the test requirements:

 [<Property(MaxTest=10)>] let ``Calling unzipTo with an invalid destination will yield a failure.`` badDest = if Directory.Exists(badDest) then // somehow skip to the next randomized input else // do the actual test 

What is the easiest way to do this?

I would prefer the answer for FsCheck / NUnit if it exists, but I would also consider any other structure whose tests can be run in Visual Studio. (I thought I saw some framework where there was a simple function to do just that, but I can't figure out what it is.)

I have preferred FsCheck.NUnit so far, since it can generate random inputs for F # types (discriminatory unions, etc.) without extra work.

Answer 1

You should do something like this:

 open FsCheck open FsCheck.Xunit [<Property(MaxTest=10)>] let ``Calling unzipTo with an invalid destination will yield a failure.`` badDest = (not Directory.Exists(badDest)) ==> lazy // do the actual test 

The first line is a logical condition, and ==> is the user-defined operator defined by the FsCheck module. This will only result in evaluating the lazy expression if the condition on the right is true .

Consider, however, refactoring this test so that it does not depend on the file system. The file system is permanent, so it automatically creates a Persistent Fixture, which has a lot of problems to manage ; not impossible to handle, but better avoided.

This example uses FsCheck.Xunit, but IIRC FsCheck.Nunit works the same. However, you should seriously consider using FsCheck.Xunit instead of FsCheck.Nunit. The NUnit 2 extension model is unusually poor, which means that most of the glue libraries that try to extend NUnit have a lot of problems. This is not a problem with FsCheck.Nunit, but with NUnit itself, but it will prove to be in a lot of trouble for you.

Answer 2

FsCheck.Prop.discard() seems to do what I want - when I run the test with logging, I see that some attempts were dropped, but 10 starts were completed without dropping.

The ==> operator works to run tests using FsCheck.Quick or similar. However, this requires the lazy part to be in the "Testable" format, where the tests I'm writing right now are just <inputs>->unit .

Answer 3

I am not very familiar with F # or fscheck, but NUnit provides an Assert.Ignore() function that will immediately stop the test and mark it as “ignored”. You can also use Assert.Inconclusive() or Assert.Pass() if you thought these were more appropriate statuses.