Python Programming, news on the Voidspace Python Projects and all things techie.
New in unittest: Test Discovery and the load_tests protocol for Python 2.7 and 3.2
A feature that has long been missing from unittest, is automatic test discovery. This alone is a major reason why people move to alternative frameworks like nose and py.test.
Test discovery is now in unittest, but it missed version 3.1 of Python (which is now at release candidate) and will be in versions 2.7 & 3.2.
Automatic test discovery is where you don't need to provide your own test collection machinery, but have a tool that can automatically find and run all the tests in a project. So long as your tests are compatible with the new test discovery (see below) you can now do:
python -m unittest discover
This will find all the test files that match the default pattern ('test*.py') and run all the tests they contain. It also has a customization hook called load_tests which enables you to customize which tests are loaded and run.
This test discovery is not as sophisticated as the discovery in nose or py.test, but it is a good start and will be sufficient for many projects. The system is as follows...
Discovery from the command line takes three optional parameters (plus the -v switch to run tests verbosely) which can be passed in by position or by keyword. The parameters are the directory to start discovery (defaults to the current directory), the pattern for matching test modules (defaults to 'test*.py') and the top-level directory of your project (defaults to whatever the start directory is):
python -m unittest discover myproject/tests/ '*test.py' myproject/ python -m unittest discover -s myproject/tests/ -p '*test.py' -t myproject/ python -m unittest discover -v -p '*test.py'
All your tests must be importable from the top level directory of your project (they must live in Python packages). The start directory is then recursively searched for files and packages that match the pattern you pass in. Tests are loaded from matching modules, and all tests run.
Discovery is implemented in the TestLoader class as the discover method. It delegates to loadTestsFromModule to load all tests after discovering and importing all modules that match the pattern provided. The actual signature is:
TestLoader().discover(start_directory, pattern='test*.py', top_level_dir=None)
The customization hook is implemented in loadTestsFromModule and is available to all systems that uses the standard loader, not just during discovery.
Iff a test module defines a load_tests function then loadTestsFromModule will call this function with loader, tests, None. This should return a suite.
An example 'do nothing' implementation of load_tests for a test module would be:
One use case would be to exclude certain TestCase subclasses from being used (if they are abstract base classes for other tests) during a test run.
If a package directory name matches the pattern you pass into discovery and the __init__.py contains a load_tests function then it will be called with loader, tests, pattern. No further discovery will be done into the package, but because it is passed the pattern as an argument it is free to continue discovery itself. A do nothing load_tests for a package is:
if pattern is None:
# if loaded as a module just return the normal tests
suite = TestSuite()
# continue discovery from this directory
this_dir = os.path.dirname(os.path.abspath(__file__))
The loader stores the top level directory it was originally called with specifically for this use case. load_tests should not pass in a new top level directory to the existing loader but create a new loader if it wants to do this.
Discovery does not follow the __path__ attribute of packages / modules and only looks at the filesystem.
Both the load_tests protocol and test discovery are useful new features in unittest. I expect test discovery in particular to mature, but it is definitely already usable. The implementation uses os.relpath; so the current trunk version of unittest.py can only run on Python 2.6 or more recent. At some point I'll backport it to work with Python 2.5 / 2.4.
New in unittest: Other Minor Changes
There are a couple of other minor improvements to unittest that I haven't already mentioned.
The TestResult class has two new methods: startTestRun and stopTestRun. These are called, unsurprisingly, at the start and end of the test run.
A common way to create a custom test framework is to use a subclass of TextTestRunner which uses a custom TestResult class. You do this by overriding the _makeResult method:
return CustomTestResult(self.stream, self.descriptions, self.verbosity)
I've documented this. The addition of startTestRun and stopTestRun makes creating custom test result objects (for example which push results to a database whilst the tests are running) easier - and it is my intention to write up better documentation on building test infrastructure with unittest as this whole area is woefully under-documented.
Another minor change is that TestSuite now only accesses its tests by iterating over itself. This enables you to do lazy generation of tests where the tests are created when the suite is iterated over; another customization point.
Many of the current crop of changes to unittest were done with help and input from Robert Collins and Jonathan Lange.
New in unittest: Cleaning up resources with addCleanup
One of the new features in unittest for Python 2.7 / 3.1 is better support for resource deallocation through cleanup functions. This isn't a new idea, it's something that is already in use in the Bazaar, Twisted and Zope test frameworks.
A standard technique for allocating resources (creating temporary files or listening on a socket for example) needed during a test is to allocate them in the setUp method and deallocate them in the tearDown method. Whether the test passes or fails the tearDown method is run; however an exception in the setUp method means that neither the test nor tearDown are run.
If you need to allocate multiple resources inside setUp, or need to do the same inside the body of the test, then you need to manually track them and in the event of failure only deallocate the ones that were successfully allocated:
self.resource1 = create_resource1()
self.resource2 = create_resource2()
(If you do something similar inside the body of the test you use try...finally instead of try...except.) Cleanup functions provide a cleaner approach to resource allocation. Once you have created a resource you call addCleanup with the function that deallocates this. After tearDown, or in event of an exception being raised inside setUp, all the cleanup functions are executed in the reverse order that they were added (LIFO).
The above example becomes:
self.resource1 = create_resource1()
self.resource2 = create_resource2()
Cleanup functions can themselves call addCleanups if they need to, and if you ever need to execute all the cleanups you can call doCleanups() (for example you want to call it manually at the start of tearDown). This isn't just useful for inside setUp, but is also for tests themselves. In some cases it can be a useful alternative to setUp and tearDown.
New in unittest in Python 2.7 and 3.1: Better Command Line Features
One of the things I've been working on over the last few weeks is the Python unittest module. By virtue of having been included in the standard library for many years unittest is the most widely used Python testing framework. In Python 2.7 and Python 3.1 / 3.2 it has had some much overdue attention, with several new features added and some of them by me. I've already blogged about the new assert methods, in the next few blog entries I'll catalogue the other changes I've been involved in.
It looks like the command line improvements actually appeared too late to make it into Python 3.1 and will be in Python 3.2 instead - along with test discovery.
The normal way of making your test files individually executable with unittest is to include the following code at the end of the file:
This enables the tests in the file to be run from the command line:
What I didn't know was that you could actually pass the name of a test suite, test class or individual test. You can also pass the '-v' flag to run the tests in verbose mode (printing individual test names to stdout as they are run):
python test_something.py -v TestClass python test_something.py -v TestClass.test_method
Now that Python standard library modules can be run from the command line with the -m command line option we can do better. With unittest in Python 2.7 and Python 3.1 you will be able to specify a test module, class or individual test at the command line:
python -m unittest -v test_something python -m unittest test_something.TestClass.test_method
This removes the need for the final two lines in your test modules if you don't want them there.
You can use this to run tests from multiple modules:
python -m unittest -v test_something test_something_else
By default main calls sys.exit once it has finished running tests, which makes it inconvenient to use from the interactive interpreter. It now takes two new optional parameters, one to switch off the automatic exit and one to run the tests in verbose mode:
>>> from unittest import main >>> main(module='test_something', exit=False, verbosity=2)
If exit is False then main returns an object whose result attribute is the TestResult instance used during the test run. You can introspect this if you want more information about the test run.
This work is licensed under a Creative Commons Attribution-Share Alike 2.0 License.