How to properly perform test isolation using Python module with state?

Question

The project I'm working on is business logic software, completed as a Python package. The idea is that a different script or application will import it, initialize it, and then use it.

It currently has a top-level init () method that initializes and installs various things, a good example is that it installs SQLAlchemy with a db connection and saves the SA session for later access. It is stored in a subpackage of my project (namely myproj.model.Session, so other code can get a working SA session after importing the model).

In short, this makes my package obsolete. I am writing unit tests for a project, and this erratic behavior creates some problems:

• tests should be isolated, but the internal state of my package breaks this isolation
• I cannot test the main init () method, since its behavior is state dependent
• future tests should be performed against the part (not yet written) of the controller with a well-known state of the model (for example, pre-filled sqlite in db memory )

Do I have to somehow reorganize my package because the current structure is not the best (possible) practice (tm)? :)

Should I leave this at the same time and tune / tear it all up every time? If I am going to achieve complete isolation, which would mean completely erasing and refilling the db in each individual test, is that not an excess?

This question really concerns the general structure of code and tests, but for what I use nose-1.0 for my tests. I know that Insulating a plugin could help me, but I would like to get the code correctly before doing weird things in the test suite.

Answer 1

You have several options:

Database layout

There are a few tradeoffs to be aware of.

Your tests will become more complex as you have to tune, stall and mock the connection. You can also check sent SQL / commands. It also tends to create an odd type of hard link, which can cause you to perform additional validation / update tests when changing a schema or SQL.

This is generally the cleanest for test isolation, as it reduces the potentially greater dependence on testing. It also speeds up tests and reduces overhead to automate a test suite, for example, in a continuous integration environment.

Recover database with each test

Compromises you need to know about.

This can make your test very slow depending on how much time it actually takes to recreate your database. If the dev database server is a shared resource, additional investment will be required for each developer to have their own db on the server. The server may suffer depending on how often testing takes place. There is additional overhead for running a test suite in a continuous integration environment, since it will require at least, possibly more dbs (depending on how many branches are created at the same time).

The advantage is that it really works through the same code paths and similar resources that will be used in production. This usually helps to identify errors earlier, which is always very good.

Exchange ORM DB

If you use ORM, such as SQLAlchemy, this is an opportunity to replace the underlying database with a potentially faster in-memory database. This allows you to mitigate some of the negatives of both of the previous parameters.

This is not exactly the same database that will be used in production, but ORM should help mitigate the risk that hides the error. Typically, the time to install the database in memory is much shorter than the one supported by the files. It also has the advantage of isolating from the current test run, so you don’t have to worry about managing a shared resource or a final break / clear.

Answer 2

Working on a project with a relatively expensive setup (IPython), I saw the approach used when we call the get_ipython function, which installs and returns an instance, replacing itself with a function that returns a link to an existing instance. Then each test can call the same function, but only for the first installation.

This saves a lengthy setup procedure for each test, but sometimes creates odd cases where the test fails or passes depending on which tests have been performed before. We have ways to handle this - many tests should do the same, regardless of state, and we can try to reset the state of the object before some tests. You can find similar compromises for you.

Answer 3

Mock is a simple and powerful tool to achieve some isolation. Pycon2011 has a nice video that shows how to use it. I recommend using it with py.test, which reduces the amount of code needed to define tests, and is still very, very powerful.