Is there a way to get Eclipse to run the JUnit test several times to failure?

Question

We sometimes get errors that appear once in each X run. Before people check (where it is automatically JUnit'd), our developers must pass JUnit locally through Eclipse.

Is there a convenient way (built-in or high-quality plugin) to force Eclipse to run the same test X times and stop if there is a failure? An alternative to just clicking Run X times?

Please note that I was looking for something in the user interface (for example, right-click and say "Run X times", and not just "Run").

Answer 1

If the for loop works, then I agree with nos.

If you need to repeat the whole installation-testing procedure, you can use TestSuite:

• Right-click the package containing the test to repeat
• Go to New and choose to create a JUnit test SUITE
• Make sure that only the tests you want to repeat are selected and click on them to finish.
• Edit the file to run it several times.

In the file you will find

addTestSuite(YourTestClass.class) 

and wrap this in a for loop.

I'm sure you can use addTest instead of addTestSuite to make it run only one test from this class if you just want to repeat one test method.

Answer 2

If you really want to run the test class to failure, you need your own runner.

 @RunWith(RunUntilFailure.class) public class YourClass { // .... } 

which can be implemented as follows ...

 package com.example; import org.junit.internal.runners.*; import org.junit.runner.notification.*; import org.junit.runner.*; public class RunUntilFailure extends Runner { private TestClassRunner runner; public RunUntilFailure(Class<?> klass) throws InitializationError { this.runner = new TestClassRunner(klass); } @Override public Description getDescription() { Description description = Description.createSuiteDescription("Run until failure"); description.addChild(runner.getDescription()); return description; } @Override public void run(RunNotifier notifier) { class L extends RunListener { boolean fail = false; public void testFailure(Failure failure) throws Exception { fail = true; } } L listener = new L(); notifier.addListener(listener); while (!listener.fail) runner.run(notifier); } } 

... release of untested code, TDD guilty :)

Answer 3

I know that it does not answer the question directly, but if the test does not pass every time it starts, it is a test smell, known as "Test Correction." There are several possible reasons for this (from the xUnit test patterns):

• Interactive Tests
• Interactive Test Apartments
• Lonely test
• Resource leak
• Resource optimism
• Unique test
• Test run war
• Non-deterministic test

The details of each are described in chapter 16 of the xUnit test patterns.

Answer 4

Here is the post I wrote that shows several ways to re-run tests with code examples: http://codehowtos.blogspot.com/2011/04/run-junit-test-repeatedly.html

You can use the @Parametrised runner or use the special runner included in the message. There is also a link to the @Retry implementation

Answer 5

Based on @akuhn's answer, here's what I came up with - instead of working forever, it will work 50 times or until it fails, whichever comes first.

 package com.foo import org.junit.runner.Description; import org.junit.runner.Runner; import org.junit.runner.notification.Failure; import org.junit.runner.notification.RunListener; import org.junit.runner.notification.RunNotifier; import org.junit.runners.BlockJUnit4ClassRunner; import org.junit.runners.model.InitializationError; public class RunManyTimesUntilFailure extends Runner { private static final int MAX_RUN_COUNT = 50; private BlockJUnit4ClassRunner runner; @SuppressWarnings("unchecked") public RunManyTimesUntilFailure(final Class testClass) throws InitializationError { runner = new BlockJUnit4ClassRunner(testClass); } @Override public Description getDescription() { final Description description = Description.createSuiteDescription("Run many times until failure"); description.addChild(runner.getDescription()); return description; } @Override public void run(final RunNotifier notifier) { class L extends RunListener { boolean shouldContinue = true; int runCount = 0; @Override public void testFailure(@SuppressWarnings("unused") final Failure failure) throws Exception { shouldContinue = false; } @Override public void testFinished(@SuppressWarnings("unused") Description description) throws Exception { runCount++; shouldContinue = (shouldContinue && runCount < MAX_RUN_COUNT); } } final L listener = new L(); notifier.addListener(listener); while (listener.shouldContinue) { runner.run(notifier); } } } 

Answer 6

I don't believe Junit has a built-in way to do exactly what you ask for.

If several starts lead to a different result, you should have unit test testing this case. It can be as simple as running a for loop in appropriate test cases.

Answer 7

You can implement such a loop using TestRule (since JUnit 4.9)

A very simple implementation that runs each test 10 times:

 import org.junit.rules.TestRule; import org.junit.runner.Description; import org.junit.runners.model.Statement; public class SimpleRepeatRule implements TestRule { private static class SimpleRepeatStatement extends Statement { private final Statement statement; private SimpleRepeatStatement(Statement statement) { this.statement = statement; } @Override public void evaluate() throws Throwable { for (int i = 0; i < 10; i++) { statement.evaluate(); } } } @Override public Statement apply(Statement statement, Description description) { return new SimpleRepeatStatement(statement); } } 

using:

 public class Run10TimesTest { @Rule public SimpleRepeatRule repeatRule = new SimpleRepeatRule(); @Test public void myTest(){...} } 

For a more useful implementation based on an annotation that determines which test method should be run, how often you can see this blog post: http://www.codeaffine.com/2013/04/10/running-junit-tests-repeatedly-without- loops /