Unit testing with functions returning random results

Question

I do not think this is specific to the language or framework, but I use xUnit.net and C #.

I have a function that returns a random date in a specific range. I pass the date, and the return date is always in the range from 1 to 40 years to the specified date.

Now I'm just wondering if there is a good way to unit test. It seems like the best approach is to create a loop, and let the function run 100 times and claim that each of these 100 results is in the desired range, which is my current approach.

I also understand that if I cannot control the Random generator, there will not be an ideal solution (in the end, the IS result is random), but I wonder what approaches you take when you need to test functionality that returns a random result in a certain range ?

Answer 1

In addition to checking that the function returns a date in the correct range, you want to make sure that the result is well distributed. In the test you are describing, a function will be passed that just returned the date you sent!

Thus, in addition to calling the function several times and checking that the result remains in the desired range, I will also try to estimate the distribution, perhaps by placing the results in buckets and checking that the buckets have approximately the same number of results after you finish. More than 100 calls may be required to get stable results, but this doesn't seem like an expensive (temporary) function, so you can easily run it for several iterations K.

I used to have a problem with uneven "random" functions .. they can be a real pain, it is worth checking out at an early stage.

Answer 2

Layout or fake random number generator

Do something like this ... I did not compile it so that there could be a few syntax errors.

public interface IRandomGenerator { double Generate(double max); } public class SomethingThatUsesRandom { private readonly IRandomGenerator _generator; private class DefaultRandom : IRandomGenerator { public double Generate(double max) { return (new Random()).Next(max); } } public SomethingThatUsesRandom(IRandomGenerator generator) { _generator = generator; } public SomethingThatUsesRandom() : this(new DefaultRandom()) {} public double MethodThatUsesRandom() { return _generator.Generate(40.0); } } 

In your test, just fake or mock IRandomGenerator to return something tinned.

Answer 3

I think there are three different aspects of this problem that you are testing.

First: is my algorithm correct? That is, given a properly functioning random number generator, will they create dates that are randomly distributed over a range?

Second: are boundary cases handled correctly? That is, when a random number generator creates the highest or lowest acceptable values, will something break?

Third: does my algorithm implementation work? That is, given the well-known list of pseudo-random inputs, is the expected list of pseudo-random dates created?

The first two things are not what I would embed in a module testing kit. This is what I will prove when developing the system. I will probably do this by writing a test harness that generated millions of dates and ran a square-squared test, as daniel.rikowski suggested. I would also make sure that this test harness did not finish until it handled both extreme cases (assuming my range of random numbers is small enough so that I can avoid this). And I am documenting this, so that anyone who comes and tries to improve the algorithm will know that this is a violation.

The last thing I would do for unit test for. I need to know that it did not fit into the code, which violates its implementation of this algorithm. The first sign that I will get when this happens is that the test fails. Then I will return to the code and find out that someone else thought that they were fixing something and broke it. If someone corrects the algorithm, it will also help them fix this test.

Answer 4

You do not need to control the system to make the results deterministic. You are on the right track: decide what is important for the output of the function and check it. In this case, it is important that the result is in the range of 40 days, and you are testing it. It is also important that it does not always return the same result, so also check this. If you want to be more attractive, you can check that the results pass some kind of random test.

Answer 5

Normal. I use exactly your proposed approach: control a random event generator. Initialize it for the test using the default seed (or replace it with a proxy that returns numbers that match my test windows), so I have a deterministic / testable behavior.

Answer 6

If you want to check the quality of random numbers (in terms of independence), there are several ways to do this. One good way is the Chi square test .

Answer 7

Depending on how your function creates a random date, you can also check for illegal dates: impossible leap years or the 31st day of a 30-day month.

Answer 8

Of course, using a fixed-seed random number generator will work fine, but even then you are just trying to check what you cannot predict. This is normal. This is equivalent to having many fixed tests. However, remember - check what is important, but do not try to check everything. I believe random tests are a way to try everything, and it is not effective (or fast). You may be able to run a large number of randomized tests before you hit the error.

What I'm trying to get here is that you should just write a test for every error you find on your system. You check the extreme cases to make sure that your function works even in extreme conditions, but in fact it is the best thing you can do without spending too much time or slow execution of single tests or just losing processor cycles.

Answer 9

Methods that do not exhibit deterministic behavior cannot be correctly verified per unit, since the results will differ from one execution to another. One way around this is to use a fixed-value seed random number generator for unit test. You can also extract the randomness of the date generation class (and thus apply the principle of single responsibility ) and enter known values ​​for unit tests,

Answer 10

I would recommend overriding a random function. I am unit testing in PHP, so I am writing this code:

 // If we are unit testing, then... if (defined('UNIT_TESTING') && UNIT_TESTING) { // ...make our my_rand() function deterministic to aid testing. function my_rand($min, $max) { return $GLOBALS['random_table'][$min][$max]; } } else { // ...else make our my_rand() function truly random. function my_rand($min = 0, $max = PHP_INT_MAX) { if ($max === PHP_INT_MAX) { $max = getrandmax(); } return rand($min, $max); } } 

Then I set the random_table as I require it for each test.

Testing the true randomness of a random function is a separate test. I would avoid randomness testing in unit tests and instead do separate tests and google the true randomness of a random function in the programming language that you use. Non-deterministic tests (if any) should be excluded from unit tests. Perhaps you have a separate kit for these tests, which requires human input or a much longer run time to minimize the likelihood of a failure, which is really a skip.

Answer 11

I do not think unit testing is meant for this. You can use Unit testing for functions that return a stochastic value but use a fixed seed, in which case they are not stochastic, so to speak, for random seed, I don’t think unit testing is what you want, for example, for RNG, what do you mean is a system test in which you run RNG many times and look at the distribution or moments of it.