JUnit tests: Why is Maven (Surefire) much slower than running on Eclipse?

Question

I am working on a rather large application with ~ 260 thousand LOC and approximately 1800 unit tests on the package. A bit of background:

• Maven Multi-Module Project
• All tests are performed on a separate module, and JaCoCo is used to test coverage.
• Tests are collected in some sets, and then suites are included in the Surefire configuration.
• Using SpringJUnit4ClassRunner to run tests
• Surefire is currently configured to use the same virtual machine for assembly and testing ( forkCount is one and reuseForks is true). Having done this, I am sure that I am taking advantage of the Spring Static Context Cache (reusing the same ApplicationContext in each test)

When we run tests on our development virtual machines using Oracle JDK 7u79 / 80, we see very fast tests. For example, a fairly large test class with 50+ test methods takes about 1: 30 m (including context initialization time) to run.

Our main Dev VM is something like:

• I5 dual-core processor (with HyperThreading, so 4 virtual threads)
• RAM 8 GB
• Mid Range, SATA Hard Drive (5k or 7k2 rpm)
• Windows 7 x64 runs Oracle JDK in client mode (-client)

We use Jenkins as our CI server, and Maven (3.2) takes care of the build process. We have the architecture of Master + 2 Slaves. All virtual machines are exactly the same:

• 8 Xeon E5 Cores (Real core)
• RAM 8 GB
• SSD LUN serving virtual machines (average bandwidth 1.2 GB / s)
• Debian Linux 8 x64
• Oracle JDK in server mode

Remember those tests that usually take 1:30 minutes to run on our Eclipse? Here, on the server, they need more than 15 minutes! Here is what I already tried to do (without success so far.):

• Set MAVEN_OPTS with a fixed heap size (2 GB heap), a large PermGen space, configured GC settings, / dev / urandom as a random seed
• A 32-bit JDK is installed to use the same client mode that we use on Dev machines
• Configured Surefire to increase memory, GC settings, etc. (Since I do not use a branched virtual machine to run tests anymore, I took it because it will not change anything.)

On the bottom line, is there any specific reason why running Surefire is much slower than running JUnit on Eclipse? I scratch my head over it for several days, and it really starts to annoy me! I hate it when the solution seems so close, but so far it is.

I could not test Maven on my Dev machine because I could not allocate all the memory needed for this, but running class snippets (and not the whole 1.8k test suite) still shows that it is rather slow than running on Eclipse .

I understand that Maven has all these steps and so on, but the difference should not be so big. Do you disagree?

Any input will be appreciated. I can provide any information you need!

PS: Surefire v2.17, Maven 3.2.2, JUnit 4.12, Spring Test 3.2.13

Thanks a lot!

UPDATE 1

I tried disabling JaCoCo on the CI server to see if it affected build time. This is not true. Runtime remains unchanged.

Answer 1

I have made some progress on this. As I said above, this is a multi-module (20+) Maven application. We save all our tests on one module, but the Surefire configuration is performed on the parent POM, and on the test POM module.

After we tried everything, we decided to bring the test execution configuration only to this module, skipping all the others. This led to the execution of the package from ~ 45 to 22. We scratch our heads over this, and as soon as we understand what is happening, I will send it here again.

TL DR: This image explains everything. (off topic, safe to work)

Thanks for your input!

Answer 2

You can try to reproduce the remote final runs by running the mvn test locally. Are you sure you are working with the same settings as forking?

I know that the surefire fork settings are quite complex and have changed a bit over time (see here for a detailed description: https://maven.apache.org/surefire/maven-surefire-plugin/examples/fork-options-and-parallel- execution.html ).

If the tests are deployed each time and not reused, SpringTestRunner can spend a lot of time initializing the application over and over again.

Are you sure these are the right tests that really slow you down?

Answer 3

I know this is an old thread, but maybe it’s still interesting: if your workload uses a lot of stdout or stderr, surefire will create three times as large arrays as your output in order to store your raw output in them. This causes surefire to take much longer than running the same test in eclipse or gradle.

I created a fork https://github.com/DaGeRe/maven-surefire and will see if this can be combined into surefire. My personal tests went from ~ 1100 ms to ~ 370 ms with a fixed plugin.