Python unittest.TestCase execution order

Question

Is there a way in Python unittest set the execution order of test cases?

In my current TestCase class, some test windows have side effects that set the conditions for the rest to work properly. Now I understand that the right way to do this is to use setUp() to do all the tasks related to the implementation, but I would like to implement a project in which each subsequent test builds a bit more states that the next can use. I find it a lot more elegant.

 class MyTest(TestCase): def test_setup(self): #do something def test_thing(self) #do something that depends on test_setup() 

Ideally, I would like the tests to be executed in the order in which they appear in the class. They seem to work in alphabetical order.

Answer 1

Do not make them independent tests - if you want a monolithic test, write a monolithic test.

 class Monolithic(TestCase): def step1(self): ... def step2(self): ... def _steps(self): for name in dir(self): # dir() result is implicitly sorted if name.startswith("step"): yield name, getattr(self, name) def test_steps(self): for name, step in self._steps(): try: step() except Exception as e: self.fail("{} failed ({}: {})".format(step, type(e), e)) 

If the test later fails, and you want to get information about all the failed steps instead of stopping the test case at the first failed step, you can use the subtests function: https://docs.python.org/3/library/unittest.html # distinctive- test iterations using subtests

(The unittest2 function is available through unittest2 for versions prior to Python 3.4: https://pypi.python.org/pypi/unittest2 )

Answer 2

It's good practice to always write a monolithic test for such expectations, however, if you have a dumb dude like me, then you can just write ugly search methods in alphabetical order so that they are sorted from a to b, as stated in python docs http: // docs.python.org/library/unittest.html

Please note that the order in which various test cases will be executed is determined by sorting the names of the test functions in relation to the built-in row order

Example:

  def test_a_first(): print "1" def test_b_next(): print "2" def test_c_last(): print "3" 

Answer 3

http://docs.python.org/library/unittest.html

Note that the order in which various test cases will be executed is determined by sorting the names of the test functions relative to the built-in ordering for the strings.

So just make sure the name test_setup has the smallest string value.

Please note that you should not rely on this behavior - different testing functions should be independent of the execution order. See ngcohlan answer above for a solution if you explicitly need an order.

Answer 4

An old question, but another way that I have not seen is indicated in any related questions: use TestSuite .

Another way to complete an order is to add tests to unitest.TestSuite . This seems to correspond to the order in which tests are added to the suite using suite.addTest(...) . To do this:

• Create one or more subclasses of TestCase,

   class FooTestCase(unittest.TestCase): def test_ten(): print('Testing ten (10)...') def test_eleven(): print('Testing eleven (11)...') class BarTestCase(unittest.TestCase): def test_twelve(): print('Testing twelve (12)...') def test_nine(): print('Testing nine (09)...') 

• Create a called test generation, added in the desired order , adapted from the docs and this question :

   def suite(): suite = unittest.TestSuite() suite.addTest(BarTestCase('test_nine')) suite.addTest(FooTestCase('test_ten')) suite.addTest(FooTestCase('test_eleven')) suite.addTest(BarTestCase('test_twelve')) return suite 

• Run a test suite, for example,

   if __name__ == '__main__': runner = unittest.TextTestRunner(failfast=True) runner.run(suite()) 

For context, I had a need for this, and I was not satisfied with the other options. I settled on the above method of conducting a test order. I have not seen this TestSuite method list any of several “unit test ordering questions” (for example, this question and others, including execution order or reordering or testing order ).

Answer 5

Tests that really depend on each other should be explicitly tied to one test.

Tests that require different settings can also have the appropriate setUp() mode of operation, sufficient for installation - various ways are conceivable.

Otherwise, unittest processes test classes and test methods inside the test classes alphabetically by default (even if loader.sortTestMethodsUsing is None). dir() used internally, which is sorted under warranty.

The latter behavior can be used for practicality - for example. in order to run the most recent working tests in order to speed up the edit-testrun cycle. But this behavior should not be used to establish real dependencies . Please note that tests can be run individually using command line options, etc.

Answer 6

Answer to

@ncoghlan was exactly what I was looking for when I came to this topic. I finished modifying it so that each test step is executed, even if the previous step already caused an error; it helps me (and maybe you!) to detect and plan the distribution of the error in multithreaded database-oriented software.

 class Monolithic(TestCase): def step1_testName1(self): ... def step2_testName2(self): ... def steps(self): ''' Generates the step methods from their parent object ''' for name in sorted(dir(self)): if name.startswith('step'): yield name, getattr(self, name) def test_steps(self): ''' Run the individual steps associated with this test ''' # Create a flag that determines whether to raise an error at # the end of the test failed = False # An empty string that the will accumulate error messages for # each failing step fail_message = '' for name, step in self.steps(): try: step() except Exception as e: # A step has failed, the test should continue through # the remaining steps, but eventually fail failed = True # get the name of the method -- so the fail message is # nicer to read :) name = name.split('_')[1] # append this step exception to the fail message fail_message += "\n\nFAIL: {}\n {} failed ({}: {})".format(name, step, type(e), e) # check if any of the steps failed if failed is True: # fail the test with the accumulated exception message self.fail(fail_message) 

Answer 7

I ended up with a simple solution that worked for me:

 class SequentialTestLoader(unittest.TestLoader): def getTestCaseNames(self, testCaseClass): test_names = super().getTestCaseNames(testCaseClass) testcase_methods = list(testCaseClass.__dict__.keys()) test_names.sort(key=testcase_methods.index) return test_names 

And then

 unittest.main(testLoader=utils.SequentialTestLoader())