The right way to organize test tags containing a data file for each test test?

Question

I am writing a module that includes html parsing for data and creating an object from it. Basically, I want to create a set of test tables, where each case is an html file paired with a golden / expected pickled object file.

As I make changes to the parser, I would like to run this test package to make sure that every html page is parsed so that it matches a gold file (essentially a regression set).

I can see how to encode this as one test case, where I would download all pairs of files from some directory and then iterate through them. But I believe that this will be reported as a single test case, it will pass or fail. But I need a report that says, for example, that 45/47 pages have been successfully analyzed.

How to organize this?

Answer 1

I did similar things with the unittest framework by writing a function that creates and returns a test class. Then this function can take any parameters you want and configure the test class accordingly. You can also configure the __doc__ attribute of test functions to receive customized messages during test execution.

I quickly knocked down the following sample code to illustrate this. Instead of performing any actual testing, it uses the random module to reject some tests for demonstration purposes. When they are created, classes are inserted into the global namespace, so a call to unittest.main() will pick them up. Depending on how you run your tests, you may do something different with the generated classes.

 import os import unittest # Generate a test class for an individual file. def make_test(filename): class TestClass(unittest.TestCase): def test_file(self): # Do the actual testing here. # parsed = do_my_parsing(filename) # golden = load_golden(filename) # self.assertEquals(parsed, golden, 'Parsing failed.') # Randomly fail some tests. import random if not random.randint(0, 10): self.assertEquals(0, 1, 'Parsing failed.') # Set the docstring so we get nice test messages. test_file.__doc__ = 'Test parsing of %s' % filename return TestClass # Create a single file test. Test1 = make_test('file1.html') # Create several tests from a list. for i in range(2, 5): globals()['Test%d' % i] = make_test('file%d.html' % i) # Create them from a directory listing. for dirname, subdirs, filenames in os.walk('tests'): for f in filenames: globals()['Test%s' % f] = make_test('%s/%s' % (dirname, f)) # If this file is being run, run all the tests. if __name__ == '__main__': unittest.main() 

Run Example:

 $ python tests.py -v Test parsing of file1.html ... ok Test parsing of file2.html ... ok Test parsing of file3.html ... ok Test parsing of file4.html ... ok Test parsing of tests/file5.html ... ok Test parsing of tests/file6.html ... FAIL Test parsing of tests/file7.html ... ok Test parsing of tests/file8.html ... ok ====================================================================== FAIL: Test parsing of tests/file6.html ---------------------------------------------------------------------- Traceback (most recent call last): File "generic.py", line 16, in test_file self.assertEquals(0, 1, 'Parsing failed.') AssertionError: Parsing failed. ---------------------------------------------------------------------- Ran 8 tests in 0.004s FAILED (failures=1) 

Answer 2

Nose testing basics support this. http://www.somethingaboutorange.com/mrl/projects/nose/

Also see here. How to create dynamic (parameterized) unit tests in python?

Here is what I would do (untested):

 files = os.listdir("/path/to/dir") class SomeTests(unittest.TestCase): def _compare_files(self, file_name): with open('/path/to/dir/%s-golden' % file_name, 'r') as golden: with open('/path/to/dir/%s-trial' % file_name, 'r') as trial: assert golden.read() == trial.read() def test_generator(file_name): def test(self): self._compare_files(file_name): return test if __name__ == '__main__': for file_name in files: test_name = 'test_%s' % file_name test = test_generator(file_name) setattr(SomeTests, test_name, test) unittest.main()