Unittest tests order

Question

How can I be sure of the unittest method order? Are alphabetic or numeric prefixes correct?

class TestFoo(TestCase): def test_1(self): ... def test_2(self): ... 

or

 class TestFoo(TestCase): def test_a(self): ... def test_b(self): ... 

Answer 1

You can disable it by setting the sortTestMethodsUsing parameter to None: http://docs.python.org/2/library/unittest.html#unittest.TestLoader.sortTestMethodsUsing

For pure martial arts you are right; but for component testing and integration tests ... I do not agree that you should not accept anything regarding the state. What to do if you check the status. For example, your test confirms that the service automatically starts after installation. If in your setup you start the service, then complete the assertion, then you no longer check the status, but you check the "start the service" functionality.

Another example is that your installation takes a lot of time or requires a lot of space, and it just becomes impractical to do the setup often.

Many developers tend to use "unittest" frameworks to test components ... so stop asking yourself if I am doing unittesting or component testing.

Answer 2

There is no reason why you cannot rely on what was done in the previous test, or you must rebuild it all from scratch for the next test. At least there is no reason why they usually offer it, but people just confidently say "not worth it." It does not help.

In general, I’m tired of reading too many answers here, which say that basically “you should not do this”, instead of giving any information on how best to do it, if there are good reasons in the expert’s judgment for this. If I wanted someone to think about whether I should do something, I would ask for opinions on whether this is a good idea.

That aside, if you read say loadTestsFromTestCase and what it calls, it ultimately scans the methods with some name pattern in any order in which they appear in the class method dictionary, therefore mostly in key order. It takes this information and makes testuite to map it to the TestCase class. Giving him the list you want to order instead is one way to do this. I'm not sure about the most efficient / cleanest way to do this, but it really works.

Answer 3

Why do you need a special test order? Tests must be isolated, and therefore they can be run in any order or even in parallel.

If you need to test something like unsubscribing from a user, the test can create a new database with a test subscription, and then try to unsubscribe. This scenario has its problems, but in the end it is better than the tests are dependent on each other. (Note that you can decompose a common test code, so you don’t need to repeat the database setup code or create test data for advertising.)

Answer 4

If you use the nose, and you write your test cases as functions (and not as methods of any derived TestCase class), the nose does not mess with the order, but uses the order of the functions defined in the file. To make assert_ * methods convenient without the need to subclass TestCase, I usually use the testing module from numpy. Example:

 from numpy.testing import * def test_aaa(): assert_equal(1, 1) def test_zzz(): assert_equal(1, 1) def test_bbb(): assert_equal(1, 1) 

Running with `` nosetest -vv '' gives:

 test_it.test_aaa ... ok test_it.test_zzz ... ok test_it.test_bbb ... ok ---------------------------------------------------------------------- Ran 3 tests in 0.050s OK 

Pay attention to all those who claim that unit tests should not be ordered: although it is true that unit tests must be isolated and can be executed independently, your functions and classes are usually not independent. They rather build each other from simpler / lower-level functions to more complex / higher-level functions. When you start optimizing your low-level functions and clutter (for my part, I do this often, and if not, you probably won't need unit test ;-), then it is much better for diagnosing the reason when tests for simple functions are performed first and tests for functions that later depend on these functions. If the tests are sorted alphabetically, the real reason usually fades among the one hundred unsuccessful statements that are not there, because the function being tested has an error, but since the low-level function it relies on has.

This is why I want my unit tests to be sorted in the way I specified them: not to use the state that was created in the early tests in subsequent tests, but as a very useful tool for diagnosing problems.

Answer 5

Do not rely on the order. If they use some kind of general state, such as a file system or a database, then you must create the setUp and tearDown methods that turn your environment into a verifiable state and then clean it up after running the tests. Each test should assume that the environment is defined in setUp and should not make any additional assumptions.

Answer 6

I almost agree with the idea that tests cannot be ordered. In some cases, this helps (easier to hell!) To have them in order ... after all that is the reason for the “unit” in UnitTest.

However, one alternative is to use mockout mockout objects and fix the elements that must be done before testing this particular code. You can also put a dummy function there to secure your code. For more information, check out Mock, which is now part of the standard library. Mock

Here are some YouTube videos if you haven't used Mock before.

Video 1

Video 2

Video 3

Try to use class methods to structure your code more, and then put all class methods in one main testing method.

 import unittest import sqlite3 class MyOrderedTest(unittest.TestCase): @classmethod def setUpClass(cls): cls.create_db() cls.setup_draft() cls.draft_one() cls.draft_two() cls.draft_three() @classmethod def create_db(cls): cls.conn = sqlite3.connect(":memory:") @classmethod def setup_draft(cls): cls.conn.execute("CREATE TABLE players ('draftid' INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL, 'first', 'last')") @classmethod def draft_one(cls): player = ("Hakeem", "Olajuwon") cls.conn.execute("INSERT INTO players (first, last) VALUES (?, ?)", player) @classmethod def draft_two(cls): player = ("Sam", "Bowie") cls.conn.execute("INSERT INTO players (first, last) VALUES (?, ?)", player) @classmethod def draft_three(cls): player = ("Michael", "Jordan") cls.conn.execute("INSERT INTO players (first, last) VALUES (?, ?)", player) def test_unordered_one(self): cur = self.conn.execute("SELECT * from players") draft = [(1, u'Hakeem', u'Olajuwon'), (2, u'Sam', u'Bowie'), (3, u'Michael', u'Jordan')] query = cur.fetchall() print query self.assertListEqual(query, draft) def test_unordered_two(self): cur = self.conn.execute("SELECT first, last FROM players WHERE draftid=3") result = cur.fetchone() third = " ".join(result) print third self.assertEqual(third, "Michael Jordan") 

Answer 7

There are a number of reasons for prioritizing tests, not least of which is performance, which is what JUnit Max is for. It is sometimes useful to run very slow tests in your own module so that you can quickly receive feedback from those tests that do not suffer from the same heavy dependencies. An order is also useful for tracking failures in tests that are not fully autonomous.

Answer 8

Well, maybe a little later, but anyway ...

You should try the proboscis library. This will allow you to place an order for testing, as well as configure any test dependencies. I use it and this library is really awesome.

For example, if test case #1 on module A should depend on test case #3 on module B you CAN set this behavior using the library.

Answer 9

There are scenarios in which order may be important, and where setUp and Teardown are limited. There is only one setUp and tearDown method, which is logical, but you can put so much information in them until you figure out what setUp or tearDown can do.

As an example, take this integration test:

You write tests to check if the registration form and the login form work. In this case, the order is important, since you cannot log in without an existing account. More importantly, the order of your tests is some kind of user interaction. Where each test can represent a step in the whole process or thread that you are testing.

Dividing code into these logical parts has several advantages.

It may not be the best solution, but I often use one method that runs the actual tests.

 def test_registration_login_flow(self): _test_registration_flow() _test_login_flow() 

Answer 10

http://docs.python.org/library/unittest.html

Note that the order in which various test cases will be executed is determined by sorting the names of the test functions relative to the built-in ordering for the strings.

If you need to set the order explicitly, use a monolithic test.

 class Monolithic(TestCase): def step1(self): ... def step2(self): ... def steps(self): for name in sorted(dir(self)): if name.startswith("step"): yield name, getattr(self, name) def test_steps(self): for name, step in self.steps(): try: step() except Exception as e: self.fail("{} failed ({}: {})".format(step, type(e), e) 

More details post .

Answer 11

A simple method for organizing unittest tests is to follow the init.d mechanism of providing them with numerical names:

 def test_00_createEmptyObject(self): obj = MyObject() self.assertIsEqual(obj.property1, 0) self.assertIsEqual(obj.dict1, {}) def test_01_createObject(self): obj = MyObject(property1="hello", dict1={"pizza":"pepperoni"}) self.assertIsEqual(obj.property1, "hello") self.assertIsDictEqual(obj.dict1, {"pizza":"pepperoni"}) def test_10_reverseProperty(self): obj = MyObject(property1="world") obj.reverseProperty1() self.assertIsEqual(obj.property1, "dlrow") 

However, in such cases, you may need to structure your tests differently so that you can build on previous cases of construction. For example, in the above, it may make sense to have a "construct and veirfy" function that builds an object and checks its parameter assignment.

 def make_myobject(self, property1, dict1): # Must be specified by caller obj = MyObject(property1=property1, dict1=dict1) if property1: self.assertEqual(obj.property1, property1) else: self.assertEqual(obj.property1, 0) if dict1: self.assertDictEqual(obj.dict1, dict1) else: self.assertEqual(obj.dict1, {}) return obj def test_00_createEmptyObject(self): obj = self.make_object(None, None) def test_01_createObject(self): obj = self.make_object("hello", {"pizza":"pepperoni"}) def test_10_reverseProperty(self): obj = self.make_object("world", None) obj.reverseProperty() self.assertEqual(obj.property1, "dlrow") 

Answer 12

See the WidgetTestCase example at http://docs.python.org/library/unittest.html#organizing-test-code , he says that

Now, class instances will run one of the _ * () validation methods, and self.widget will be created and destroyed separately for each instance.

Therefore, it would be useless to indicate the order of test cases if you are not accessing global variables.

Answer 13

Contrary to what has been said here: - tests should be performed in isolation (the order does not matter for this) AND ALSO - ordering them is important because they describe what the system does and how the developer implements it.

IOW, each test provides you with system and development logic information.

So, if this data is not ordered, which can make it hard to understand your code.

Answer 14

I agree with the statement that the “don't do this” overlay is a bad answer.

I have a similar situation where I have one data source and one test will destroy the data set, resulting in other tests fail.

My solution was to use the operating system environment variables on my Bamboo server ...

(1) The test for the “data cleansing” function begins with a while loop that checks the state of the environment variable “BLOCK_DATA_PURGE”. If the variable "BLOCK_DATA_PURGE" is greater than zero, the loop will record a log entry stating that it has slept for 1 second. After "BLOCK_DATA_PURGE" has a value of zero, execution continues to verify the cleanup functionality.

(2) Any unit test that needs data in the table simply increments "BLOCK_DATA_PURGE" at the beginning (in setting ()) and decreases the same variable in teardown () mode.

The effect of this is to allow different data consumers to block the purge functionality for as long as they need, without fear that cleaning can be performed between trials. Effectively, the cleanup operation is clicked on to the last step ... or at least to the last step, requiring an initial data set.

Today I am going to expand this to add more functionality to allow some REQUIRE_DATA_PURGE tests. They will effectively invert the above process to ensure that these tests are performed only after data cleansing to verify data recovery.

Happy coding!

- Sam Caldwell

Answer 15

The philosophy of unit tests is to make them independent of each other. This means that the first step of each test will always be an attempt to rethink how you test each part to fit this philosophy. This can affect how you approach testing and creativity, narrowing your tests to smaller areas.

However, if you still find that you need tests in a specific order (since this is viable), you can try checking the answer in Python unittest.TestCase order of execution .