Distributed / faster python unit tests

Question

I have many unit tests in python for a project, and it comes to the point that it takes a long time to run them. I do not want to add more because I know that they will do something more slowly. How do people solve this problem? Is there an easy way to spread test execution across a cluster?

Answer 1

You cannot run all of your tests often because they are too slow. This is an inevitable consequence of increasing your project and will not disappear. Of course, you can run tests in parallel and get good acceleration, but the problem will come back later, and it will never be as if your project was small.

To improve performance, you should be able to code and run the appropriate unit tests and get results within seconds. If you have a hierarchy of tests, you can do this efficiently: regularly run tests for the module you are working on, tests for the component you are working on, and often examine the project (perhaps before you think about checking it). You may have integration tests or complete system tests that you can run in one night: this strategy is an extension of this idea.

All you have to do to set it up is to organize your code and tests to support the hierarchy.

Answer 2

See py.test, which has the ability to pass unit tests to a group of machines or Nose, which (as for the external line, and not in the current version) supports current tests in parallel with the multiprocessing module.

Answer 3

Profile them to see what is actually slow. You can solve this problem without distribution. If the tests are really unit tests, then I see a lot of problems with running the tests in several execution mechanisms.

Answer 4

The first part of the solution to this problem is to run only those tests that need to be performed. Between committing to a common branch, only those tests are performed with which one new job interacts; which should take five seconds. If you accept this model, it becomes vitally important to do the task of launching the entire set of tests before moving on to a shared resource.

The problem of launching a complete set of tests for regression purposes remains, of course, although it has already been partially solved by simply using the complete set less frequently. To avoid having to wait while this task is running, you can disable the test task on another machine. This quickly turns into a task for a continuous integration system; buildbot seems appropriate enough for your use case.

You should also be able to distribute tests on hosts using buildbot, selecting two jobs with different entry points to the test suite. But I'm not sure that this will bring you much effort during the first two steps that I mentioned here; it should be reserved for cases where tests take much longer than the interval between commits for shared resources.

YES

[Caveat lector: My understanding of buildbot is pretty much theoretical at the moment, and it's probably harder than it sounds.]

Answer 5

When coding, run only those class tests that you just changed, and not all tests in the entire project.

However, it is good practice to run all the tests before you pass your code (but the continuous integration server can do this for you).