How can I run NUnit tests in parallel?

Question

I have a big test test (~ 10 seconds per test) written using NUnit. I would like to use the fact that my machines are multi-core boxes. Ideally, I could do one core test, independently of the other tests.

There is PNUnit, but it is designed to test problems with thread synchronization, etc., and I have not seen an obvious way to accomplish this.

Is there a switch / tool / parameter that I can use to run tests in parallel?

Answer 1

The standard nunit runner does not support current tests in parallel. You can create your own test runner for running tests in parallel (using current nunit tests). I'm not sure why the nunit team hasn't done this yet.

Alternatively, MBUnit has the ability to create parallelized tests, and since MBUnit has almost the same syntax as NUnit, it may not bring so much effort to make the switch.

EDIT: As noted in the comments, although this answer was correct at the time of writing, if you want to run NUnit tests in parallel, there are at least 2 options:

• NCrunch offers it out of the box (without changing anything, but is a commercial product).
• NUnit 3 offers a parallelizable attribute that can be used to indicate which tests can run in parallel.

Answer 2

NUnit version 3 will support parallel parallel tests:

Adding an attribute to the class: [Parallelizable(ParallelScope.Self)] will run your tests in parallel.

• ParallelScope.None indicates that the test may not run in parallel with other tests.

• ParallelScope.Self indicates that the test itself can run in parallel with other tests.

• ParallelScope.Children indicates that descendants of the test can run parallel to each other.

• ParallelScope.Fixtures indicates that luminaires can operate in parallel with each other.

NUnit Framework-Parallel-Test-Execution

Answer 3

If your project contains several test DLLs, you can run them in parallel using this MSBuild script. Obviously, you will need to customize the paths to suit your projects.

To start with 8 cores, follow these steps: c:\proj> msbuild /m:8 RunTests.xml

RunTests.xml

 <?xml version="1.0" encoding="utf-8"?> <Project DefaultTargets="RunTestsInParallel" xmlns="http://schemas.microsoft.com/developer/msbuild/2003"> <Import Project="$(MSBuildExtensionsPath)\MSBuildCommunityTasks\MSBuild.Community.Tasks.Targets"/> <PropertyGroup> <Configuration Condition=" '$(Configuration)' == '' ">Release</Configuration> <Nunit Condition=" '$(Nunit)' == '' ">$(MSBuildProjectDirectory)\..\tools\nunit-console-x86.exe</Nunit> </PropertyGroup> <!-- see http://mikefourie.wordpress.com/2010/12/04/running-targets-in-parallel-in-msbuild/ --> <Target Name="RunTestsInParallel"> <ItemGroup> <TestDlls Include="..\bin\Tests\$(Configuration)\*.Tests.dll" /> </ItemGroup> <ItemGroup> <TempProjects Include="$(MSBuildProjectFile)" > <Properties>TestDllFile=%(TestDlls.FullPath)</Properties> </TempProjects> </ItemGroup> <MSBuild Projects="@(TempProjects)" BuildInParallel="true" Targets="RunOneTestDll" /> </Target> <Target Name="RunOneTestDll"> <Message Text="$(TestDllFile)" /> <Exec Command="$(Nunit) /exclude=Integration $(TestDllFile) /labels /xml:$(TestDllFile).results.xml" WorkingDirectory="$(MSBuildProjectDirectory)\..\bin\Tests\$(Configuration)" /> </Target> </Project> 

Update If I were to answer this question, I would highly recommend NCrunch and its test tool for testing the command line for maximum test run performance. There is nothing like it, and it will revolutionize the code-test-debug cycle at the same time.

Answer 4

In this article, it is mentioned that to speed up tests, the poster launches several instances of NUnit with command parameters that determine which tests each instance should run.

FTA:

I ran into an odd problem.

We use nunit-console to run a test on our continuous integration server. We recently moved from Nunit 2.4.8 to 2.5.5 and from .Net 3.5 to 4.0. To speed up the test, we run several Nunit instances in parallel with another command line. Arguments

• We have two copies of our test builds and nunit binaries in folders A and B.
• In folder A we execute

nunit-console-x86.exe Model.dll Test.dll / exclude: MyCategory / xml = TestResults.xml / framework = net-4.0 / noshadow

• In folder B we execute

nunit-console-x86.exe Model.dll Test.dll / include: MyCategory / xml = TestResults.xml / framework = net-4.0 / noshadow

If we consistently execute commands, both work successfully. But if we execute them in parallel only one succeeds. As far as I can tell, this is the one that loads test fixtures first. Another does not work with the message “Could not find fixture”.

Is this problem known? I could not find anything related to the error list in the launcher. BTW Our server runs the 64-bit version of Windows Server 2008. I could also reproduce the problem in Windows 7 64-bit.

Assuming this bug is fixed or you are not using a newer version of the software mentioned, you should be able to replicate your technique.

Update

TeamCity looks like a tool that you can use to automatically run NUnit tests. They have a NUnit launcher discussed here , which can be used to run multiple instances of NUnit. Here is a blog post that discusses combining multiple NUnit XML results into a single result file.

Thus, in theory, you could force TeamCity to automatically run several NUnit tests based on the fact that you want to split the workload and then combine the results into a single file for further processing of the test.

How automated is it for your needs?

Answer 5

Just because PNUnit can perform synchronization inside test code does not mean that you really need to use this aspect. As far as I can see, there is nothing that would prevent you from simply creating a set and ignoring the others until you need it.

By the way, I don’t have time to read all their sources, but I was curious to check the Barrier class, and this is a very simple lock counter. It just waits until N threads enter, and then sends an impulse so that they all continue to work simultaneously. That’s all you need - if you don’t touch it, it won’t bite you.

Maybe the bit counter is intuitive for normal threaded development (locks are usually used to serialize access - 1 to 1), but this is a rather energetic leak :-)

Answer 6

Now you can use NCrunch to parallelize your unit tests, and you can even configure how many cores NCrunch should use and how much Visual Studio should use.

plus you get continuous testing as a bonus :)

Answer 7

That would be a bit of a hack, but you could divide unit tests into a number of categories . Then run a new NUnit instance for each category.

Edit: It looks like they have added the / process option to the console application. The command line says that this is the "Process Model for Tests: Single, Separate, Multiple." This test runner also seems to have this feature.

Edit 2: Unfortunately, although it creates separate processes for each assembly, the process isolation option (/ process from the command line) starts the agents one at a time.

Answer 8

Since the project is not mentioned here, I would like to raise NUnit.Multicore . I have not tried the project myself, but it seems to have an interesting approach to a parallel test problem with NUnit.

Answer 9

You can try my little TBox tool or console parallel Runner or even a distributed calorie distribution plugin that can also run unit tests on a SkyNet PC suite

TBox is designed to simplify working with large solutions that contain many projects. It supports many plugins, and one of them provides the ability to run NUnit tests in parallel. This plugin does not require any changes to your existing tests.

It also supports:

• Cloning a folder with unit test (if your tests change local data)

• Test synchronization (for example, if your testfixtureteardown tests kill all dev or chromerunner servers for qunit)

• x86 and admin privileges to run tests

• Batch run - you can run tests for many assemblies in parallel

• Even for a single run, a thread runs faster than the standard nunit runner if you have many small tests.

This tool also supports a command line test runner (for running in parallel), and you can use it with continuous integration.

Answer 10

• How to run NUnit parallel tests (Selenium Grid)?
• Testing

Answer 11

I have successfully used NUnit 3.0.0 beta-4 to run tests in parallel

• Running on the build server
• Performs Selenium Tests
• Visual Studio Support
• no support for resharper

Thanks for the response of colleagues .

Gotchas:

• The parallelizable attribute is not inherited, so it must be specified in the test class.

Answer 12

You can use the following PowerShell command (for NUnit3, for the name of the name change to NUnit2):

 PS> nunit3-console (ls -r *\bin\Debug\*.Tests.dll | % FullName | sort-object -Unique) 

The presented command launches all test assemblies in one instance of nunit, which allows using the built-in parallel test run mechanism.

Notes

• Remember to customize the directory search template. In this example, only assemblies ending in .Tests.dll and inside \bin\Debug are executed.

• Remember to filter Unique - you may not want to have one.

Answer 13

As an alternative to adding a Parallelizable attribute for each test class:

Add this to the assemblyInfo.cs test project class for nunit3 or higher:

 // Make all tests in the test assembly run in parallel [assembly: Parallelizable(ParallelScope.Fixtures)]