Unit tests sometimes fail, sometimes fail

Question

I have a mocha unit test package.

When I run them locally, everything works fine. When I run them on our Jenkins CI server, they sometimes fail, sometimes they fail.

I just can't reproduce why they fail. What can cause this behavior?

Answer 1

Tests can be interrupted intermittently for a number of reasons and reveal why they fail, often show information about your code base and environment.

Here are a few possible reasons:

• Common objects — single states that hold state can cause problems between tests if the test environment is not reset to a well-known state. if if your test runner runs the tests in a non-deterministic order, you can see random errors that actually cause problems with the damaged state.

• Environmental and external dependencies - any external object that can hold a state can cause unpredictable results.

• Timing - sometimes tests are recorded with timeouts or sleep threads that are too specific. If the build server is running under heavy load, these timeouts may not be long enough.

As a general guide, tests should be:

• isolated: tests focus on one unit at a time
• repeatable: produces the same results every time
• independent: the order of execution of the tests should not matter

Answer 2

I would suggest that you reuse test fixtures for multiple tests. In some cases, the order of the tests changes from the order on your local computer, and when you run the test, you do not have the correct device. To fix this, you must give each test a clean fixture.

Another reason may be that when you re-test you reuse the database from multiple locations (two build servers with the same test configuration). Give each build server its own database. Or sub / mock is your database level. Your test should not be dependent on external resources, unless you are testing connections to your external resources.

Answer 3

I would try to narrow down the problem by reducing the number of executors to 1. If the tests are still interrupted intermittently, you have Test Run War , otherwise (assuming they work fine locally) it looks like Resource Leakage .

Additional information at http://xunitpatterns.com/Erratic%20Test.html

Answer 4

This can happen if you are testing multi-threaded code and using Thread.sleep(time); in my tests. If the sleep time is not long enough, tests may or may not take place depending on the processing speed of computers. Related discussions:

• How to write a JUnit test case for testing threads and events
• Multithreaded Unit Testing Applications
• How do I unit threaded code?
• Thread.sleep in a JUnit test case