Voidspace

A plugin system for unittest(2)

One of the big problems with unittest, which I have been maintaining for well over a year now, is how hard it is to extend it. For a long time I've been saying that my "big task" with unittest was to implement a plugin system. Well, at the EuroPython sprints I finally got round to making a start.

There is now a prototype plugin system for unittest2 in the plugins branch of the mercurial repository. You can try it out with:

hg clone http://hg.python.org/unittest2

cd unittest2

hg up plugins

python setup.py install

I'm looking for feedback on the system, and have written up a rather lengthy "proto-pep" that describes the whole system. Most of the document describes the specific extension points, so you can get a good feel for it from just reading the first few sections. I may rewrite it as an official PEP, as my goal is to get this into unittest for Python 3.2, but at the moment it lives in the repository or can be found online at:

http://hg.python.org/unittest2/raw-file/tip/description.txt

As part of the prototype I have been implementing some example plugins (in unittest2/plugins/) so I can develop the mechanism against real rather than imagined use cases. Jason Pellerin, creator of nose, has been providing me with feedback and has been trying it out by porting some of the nose plugins to unittest [1]. His initial comment on the system was "it looks very flexible and clean". ;-)

One of the goals of the extension mechanism is to allow nose2 to be a much thinner set of plugins over unittest(2), making nose2 simpler to maintain and less likely to break every time unittest changes [2].

Example plugins available and included:

• a pep8 and pyflakes checker
• a debugger plugin that drops you into pdb on test fail / error
• a doctest loader (looks for doctests in all text files in the project)
• use a regex for matching files in test discovery instead of a glob
• growl notifications on test run start and stop
• filter individual test methods using a regex
• load test functions from modules as well as TestCases
• integration with the coverage module for coverage reporting

In addition I intend to create a plugin that outputs junit compatible xml from a test run (for integration with tools like the hudson continuous integration server) and a test runner that runs tests in parallel using multiprocessing.

Not all of these will be included in the merge to unittest. Which ones will is still an open question.

I'd like feedback on the proposal, and hopefully approval to port it into unittest after discussion / amendment / completion. In particular I'd like feedback on the basic system, plus which events should be available and what information should be available in them. Note that the system is not complete in the prototype. Enough is implemented to get "the general idea" and to formalise the full system. It still needs extensive tests and the extra work in TestProgram makes it abundantly clear that refactoring there is well overdue...

So, give it a try, find bugs in it, write plugins with it and let me know what you think...

Here's an example of a unittest plugin. This one is the "debugger" plugin that drops you into pdb on test fail or error (configurable):

from unittest2.events import Plugin

import pdb
import sys

class Debugger(Plugin):

    configSection = 'debugger'
    commandLineSwitch = ('D', 'debugger', 'Enter pdb on test fail or error')

    def __init__(self):
        self.errorsOnly = self.config.as_bool('errors-only', default=False)

    def onTestFail(self, event):
        value, tb = event.exc_info[1:]
        test = event.test
        if self.errorsOnly and isinstance(value, test.failureException):
            return
        original = sys.stdout
        sys.stdout = sys.__stdout__
        try:
            pdb.post_mortem(tb)
        finally:
            sys.stdout = original

The plugin is hooked into the command line interface of the unit2 test runner and the plugin configuration files through the configSection and commandLineSwitch class attributes. There are APIs that give complete control over this, but the Plugin class provides some nice sugar for making writing plugins simpler. See the description document for more details than you could possibly want...

To get started you'll need a unittest.cfg file in either your home directory or the same directory you run unit2 from. There is an example config file, with all the plugins configured, in the unittest2 repository. Once you have that in place unit2 -h will show you command line options for all the configured plugins:

--checker             Check all Python files with pep8 and pyflakes
--test-functions      Load test functions from test modules
--doctest             Load doctests from text files
-C, --coverage        Enable coverage reporting
--cover-module=COVER-MODULE
                      Specify a module or package for coverage
                      reporting
-G, --growl           Growl notifications on test run start and stop
-F FILTER, --filter=FILTER
                      Filter test methods loaded with a regexp
-D, --debugger        Enter pdb on test fail or error
-R, --match-regexp    Match filenames during test discovery with
                      regular expressions instead of glob

Most of the plugins can be configured in the config file(s) too. Looking at the options there and poking around in the source code of the plugins should show you everything you need to know...

tox: Testing projects with multiple versions of Python

mock is tested on Python 2.4 upwards. Running all the tests is done with unittest2 test discovery, so up until now I've had several scripts for running the tests with different versions of Python. Not difficult, but a pain.

Enter a new project by the prodigious Holger Krekel:

• tox: virtualenv-based automation

To use tox you create a simple config file tox.ini specifying the test command and Python versions and tox will create a virtualenv for each Python version and run your tests in it.

I can now run all the mock tests for Python 2.4 - 2.7 on both Mac OS X and Windows with a single command (a single command on each platform - although tox does have Hudson support for continuous integration servers).

As well as running tests tox can execute arbitrary commands. This means it can run sphinx commands. For mock I have tox execute two Sphinx commands to build the documentation and run all the doctests (only for Python 2.6 & 2.7 as some of the doctests use the with statement). This way the tox run fails if there are any errors in the documentation, either reStructured Text errors or doctest failures.

A successful run looks something like this (output from Windows shown - with lots of it snipped):

C:\compile\mock
> tox
_________________________________ [tox sdist] _________________________________
[TOX] ***creating sdist package
[TOX] C:\compile\mock$ C:\Python26\python.exe setup.py sdist --formats=zip --dis
t-dir .tox\dist >.tox\log\0.log
[TOX] ***copying new sdistfile to 'C:\\Users\\michael\\.tox\\distshare\\mock-0.7
.0.zip'
_____________________________ [tox testenv:py24] ______________________________
[TOX] ***reusing existing matching virtualenv py24
[TOX] C:\compile\mock\.tox$ py24\Scripts\pip.exe install dist\mock-0.7.0.zip --d
ownload-cache=C:\compile\mock\.tox\_download >py24\log\3.log
[TOX] C:\compile\mock$ .tox\py24\Scripts\unit2 discover
..............................................................s.................
...s
----------------------------------------------------------------------
Ran 84 tests in 0.109s

...

No builder selected, using default: html
building [html]: targets for 0 source files that are out of date
updating environment: 0 added, 0 changed, 0 removed
looking for now-outdated files... none found
no targets are out of date.
________________________________ [tox summary] ________________________________
[TOX] py24: commands succeeded
[TOX] py25: commands succeeded
[TOX] py26: commands succeeded
[TOX] py27: commands succeeded
[TOX] congratulation :)

C:\compile\mock

Notice how the output from tox shows us exactly which commands are being executed. The test execution is done with test discovery using the command .tox\py24\Scripts\unit2 discover.

Here's the tox.ini for the mock module:

[tox]
envlist = py24,py25,py26,py27

[testenv]
deps=unittest2
commands=unit2 discover []

[testenv:py26]
commands=
    unit2 discover []
    sphinx-build -b doctest docs html
    sphinx-build docs html
deps =
    unittest2
    sphinx

[testenv:py27]
commands=
    unit2 discover []
    sphinx-build -b doctest docs html
    sphinx-build docs html
deps =
    unittest2
    sphinx

This tells tox that when the tox command is run it is to create four different virtual environments with the four different versions of Python (so all these versions of Python need to be installed). For Python 2.6 and 2.7 there are custom command sets and Python 2.4 and 2.5 use the default [testenv] block.

The dependencies for Python 2.4 & 2.5 are just unittest2, for 2.6 & 2.7 Sphinx is also a dependency. The test command is, as we've seen, with unit2 discover. The [] after the command allows extra command line options to be passed through from tox to the test runner. For example we could use this to modify the parameters for test discovery:

tox -- -p test_signal\*

Although tox is in its early days its already useful for me to be able to run tests on four versions of Python, including testing the Sphinx documentation builds correctly and the code examples work, with a single command.

unittest2 0.5.0: setuptools compatible test collector and Python 2.3 distribution

unittest2 0.5.0 has just been released. This version of unittest2 has "feature parity" with the version of unittest in Python 2.7:

• unittest2 on PyPI

If you want to ensure that your tests run identically under unittest2 and unittest in Python 2.7 you should use unittest2 0.5.0. Later versions of unittest2 will include changes in unittest made in Python 3.2 and onwards after the release of Python 2.7.

unittest2 is a backport of the recent enhancements in the Python unittest testing library in Python 2.7.

• New and Improved: unittest2 article
• unittest2 issue tracker
• Mercurial Development Repository:

Improvements in unittest2 over standard unittest in Python 2.6 and earlier include:

• automatic test discovery from the command line
• failfast, catch and buffer command line options
• class and module level setup and teardowns
• addCleanup for better resource handling
• test skipping and expected failures
• improvements to assertRaises and assertAlmostEqual
• many new assert methods, plus better failure messages
• load_tests protocol for customizing test loading
• various other API improvements and fixes

unittest2 is tested with Python 2.4 - 2.7. New in 0.5.0 is a distribution for Python 2.3 contributed by Mark Roddy.

Also new in 0.5.0 is a setuptools compatible test collector. If you put test_suite = 'unittest2.collector' in setup.py you can then find and run your tests (using test discovery) with:

python setup.py test

This starts test discovery with the default parameters from the directory containing setup.py, so it is perhaps most useful as an example (see unittest2/collector.py).

All Changes in 0.5.0

Addition of a setuptools compatible test collector (very basic). Specify test_suite = 'unittest2.collector' in your setup.py.

TestSuite.debug() and TestCase.debug() now execute cleanup functions and class and module level setups and teardowns.

No longer monkey-patch os.path.relpath for Python 2.4 / 2.5 so that projects don't accidentally depend on our patching. Contributed by Konrad Delong.

Added a Python version specific unit2 entrypoint. This will, for example, create a unit2-2.6 script if unittest2 is installed with Python 2.6. (Requires setuptools or distribute.)

Python 2.3 compatibility (in the python2.3 branch of the repository), contributed by Mark Roddy.

setuptools console script entry points are created as '.py' scripts on Windows.

Feature parity with the Python 2.7 final release.

PyCrypto 2.1 for Python 2.7 on Windows

Now that Python 2.7 final is out I've compiled PyCrypto 2.1 for Windows.

The new build is available for download along with the other binaries:

• PyCrypto 2.1 for 32bit Windows and Python 2.7

Unfortunately compiling PyCrypto for 64bit Windows (AMD64 architecture) is "non-trivial", so I am unable to provide binary builds.

I've also tried compiling pysco for Python 2.7 (both psyco 1.6 and psyco v2). Unfortunately psyco is not yet compatible with Python 2.7. Christian Tismer is aware of this and will work on it (for psyco 2 at least) as soon as he is able.

ContextDecorator: creating APIs that work as decorators and context managers

Two of the best additions to Python in recent years are the with statement and decorators. Both context managers (objects used in with statements) and decorators can be used for similar purposes: performing an action before and after executing the decorated function or the code inside the with block. In fact I now find that many places I used to use decorators I now prefer the with statement (if I'm lucky enough to be able to ignore Python 2.4 compatibility).

If you're a library or framework creator then it is nice to be able to create APIs that can be used either as decorators or context managers. The patch decorators in mock behave like this, and when I was writing a new variant (patch.dict) I found myself having to figure out again how to do it. It isn't hard, but it's a bit fiddly. Nor is this an uncommon pattern, both py.test and Django have code that behaves like this.

I've written a very simple utility class that does this, called ContextDecorator, and it is now part of contextlib in Python 3.2.

Context managers inheriting from ContextDecorator have to implement __enter__ and __exit__ as normal. __exit__ retains its optional exception handling even when used as a decorator.

Even better contextlib.contextmanager, which is a decorator for writing context managers as functions, uses ContextDecorator so the context managers it creates can automatically be used as decorators as well.

I've put both ContextDecorator and the new contextmanager into a package on PyPI, and it works with all versions of Python from 2.4 - 3.1.

• contextdecorator on PyPI
• contextdecorator repo and issue tracker

Example:

from contextdecorator import ContextDecorator

class mycontext(ContextDecorator):
   def __enter__(self):
      print 'Starting'
      return self

   def __exit__(self, *exc):
      print 'Finishing'
      return False

>>> @mycontext()
... def function():
...    print 'The bit in the middle'
...
>>> function()
Starting
The bit in the middle
Finishing

>>> with mycontext():
...    print 'The bit in the middle'
...
Starting
The bit in the middle
Finishing

Existing context managers that already have a base class can be extended by using ContextDecorator as a mixin class:

from contextdecorator import ContextDecorator

class mycontext(ContextBaseClass, ContextDecorator):
   def __enter__(self):
      return self

   def __exit__(self, *exc):
      return False

contextdecorator also contains an implementation of contextlib.contextmanager that uses ContextDecorator. The context managers it creates can be used as decorators as well as in with statements.

from contextdecorator import contextmanager

@contextmanager
def mycontext(*args):
   print 'Started'
   try:
     # decorated function or with
     # statement executed here
      yield
   finally:
      # exception handling here
      print 'Finished!'

>>> @mycontext('some', 'args')
... def function():
...    print 'In the middle'
...
Started
In the middle
Finished!


>>> with mycontext('some', 'args'):
...    print 'In the middle'
...
Started
In the middle
Finished!

Porting mock to Python 3

One of the nice new features in mock 0.7 is that it works with both Python 2 & 3. The mock module itself, even with all the freshly added docstrings, weighs in at less than 800 lines of code so compatibility is maintained with a single source base rather than the more recommended 2to3 approach. There are however about 1500 lines of test code that also need to work under Python 3; so whilst not a particularly difficult exercise it was not entirely trivial to get all the tests passing under Python 2.4, 2.5, 2.6, 2.7 and 3.2. Good tests make it much easier to have confidence that the port works. Attempting this without tests would be much more painful, even though it means there is more code to port.

I've written up all the changes needed for mock to support Python 3:

• Porting mock to Python 3

Release: mock 0.7 beta 2

I'm pleased to announce a new release of the mock module, the first in a while. Konrad Delong has joined me as a maintainer of mock and has been a great help in getting this release out. As there are several major new features this is a beta release, with 0.7.0 final coming out in a few weeks assuming there are no major problems discovered. Please download it and try it out:

• http://pypi.python.org/pypi/mock/0.7.0b2

mock is a Python module that provides a core Mock class. It is intended to reduce the need for creating a host of trivial stubs throughout your test suite. After performing an action, you can make assertions about which methods / attributes were used and arguments they were called with. You can also specify return values and set needed attributes in the normal way.

The mock module also provides utility functions / objects to assist with testing, particularly monkey patching.

mock is tested on Python versions 2.4-2.7 and Python 3.

Full documentation is included in the distribution.

• PDF documentation for 0.7.0b2
• mock on google code (repository and issue tracker)
• mock documentation
• Article on mocking, patching and stubbing

Mock is very easy to use and is designed for use with unittest. Mock is based on the 'action -> assertion' pattern instead of 'record -> replay' used by many mocking frameworks. See the mock documentation for full details.

Changes in 0.7.0 (including the much awaited magic method support) are:

• Addition of mocksignature
• Ability to mock magic methods
• Ability to use patch and patch.object as class decorators
• Renamed patch_object to patch.object (patch_object is deprecated)
• Addition of MagicMock class with all magic methods pre-created for you
• Python 3 compatibility (tested with 3.2 but should work with 3.0 & 3.1 as well)
• Addition of patch.dict(...) for changing dictionaries during a test
• Addition of mocksignature argument to patch and patch.object
• help(mock) works now (on the module). Can no longer use __bases__ as a valid sentinel name (thanks to Stephen Emslie for reporting and diagnosing this)
• Addition of soft comparisons: call_args, call_args_list and method_calls return now tuple-like objects which compare equal even when empty args or kwargs are skipped
• Added some docstrings.
• BUGFIX: side_effect now works with BaseException exceptions like KeyboardInterrupt
• BUGFIX: patching the same object twice now restores the patches correctly
• The tests now require unittest2 to run
• Konrad Delong added as co-maintainer

There are several major new features in this release, not least of which is the support for mocking the Python protocols (magic methods).

The easiest way of using magic methods is with the MagicMock class. It allows you to do things like:

>>> from mock import MagicMock
>>> mock = MagicMock()
>>> mock.__str__.return_value = 'foobarbaz'
>>> str(mock)
'foobarbaz'
>>> mock.__str__.assert_called_with()

Mock allows you to assign functions (or other Mock instances) to magic methods and they will be called appropriately. The MagicMock class is just a Mock variant that has all of the magic methods pre-created for you (well - all the useful ones anyway).

The following is an example of using magic methods with the ordinary Mock class:

>>> from mock import Mock
>>> mock = Mock()
>>> mock.__str__ = Mock()
>>> mock.__str__.return_value = 'wheeeeee'
>>> str(mock)
'wheeeeee'

mocksignature is a useful companion to Mock and patch. It creates copies of functions that delegate to a mock, but have the same signature as the original function. This ensures that your mocks will fail in the same way as your production code if they are called incorrectly:

>>> from mock import mocksignature
>>> def function(a, b, c):
...     pass
...
>>> function2 = mocksignature(function)
>>> function2.mock.return_value = 'fishy'
>>> function2(1, 2, 3)
'fishy'
>>> function2.mock.assert_called_with(1, 2, 3)
>>> function2('wrong arguments')
Traceback (most recent call last):
 ...
TypeError: <lambda>() takes exactly 3 arguments (1 given)

Also new is patch.dict for setting values in a dictionary just during a test and restoring the dictionary to its original state when the test ends:

>>> foo = {'key': 'value'}
>>> original = foo.copy()
>>> with patch.dict(foo, {'newkey': 'newvalue'}, clear=True):
...     assert foo == {'newkey': 'newvalue'}
...
>>> assert foo == original

Discover 0.4.0: test discovery for unittest

discover is a backport of the new test discovery features only from Python 2.7 / 3.2. The discover module provides automatic test discovery for standard unittest based tests:

python -m discover

python discover.py

If you have setuptools or distribute installed you will also have a discover script available.

This will discover all tests (with certain restrictions) from the current directory. The discover module has several options to control its behavior (full usage options are displayed with python -m discover -h).

discover 0.4.0 provides feature parity with the test discovery in Python 2.7 RC1 and unittest2 0.4.2.

unittest2 provides not just the test discovery features that are new in Python 2.7, but a whole lot more as well.

The full list of changes since discover 0.3.2:

• Addition of a setuptools compatible test collector. Set "test_suite = 'discover.collector'" in setup.py. "setup.py test" will start test discovery with default parameters from the same directory as the setup.py.
• Allow test discovery using dotted module names instead of a path.
• Addition of a setuptools compatible entrypoint for the discover script.
• A faulty load_tests function will not halt test discovery. A failing test is created to report the error.
• If test discovery imports a module from the wrong location (usually because the module is globally installed and the user is expecting to run tests against a development version in a different location) then discovery halts with an ImportError and the problem is reported.
• Matching files during test discovery is done in DiscoveringTestLoader._match_path. This method can be overriden in subclasses to, for example, match on the full file path or use regular expressions for matching.
• Tests for discovery ported from unittest2. (The tests require unittest2 to run.)

A Little Bit of Python: Episodes 11 to 14

Since I last reported on the A Little Bit of Python podcast we've produced three more episodes (but still not managed to get a proper website up). A Little Bit of Python is an occasional podcast on Python related topics with myself, Brett Cannon, Jesse Noller, Steve Holden and Andrew Kuchling.

Episode 11 is an interview I recorded with Antoine Pitrou at PyCon 2010. Antoine is a core-Python developer. As well as having done a lot of work on Python 3, Antoine is also responsible for the "new-GIL". This is a change to the Global Interpreter Lock to improve Python's performance when running multi-threaded code. In this interview we discuss core-Python development, including of course the new-GIL:

• Episode 11 MP3
• Episode 11 AAC

Episode 12 is a discussion on general concurrency issues. We discuss the significance of the Global Interpreter Lock (or GIL) and recent work at improving it, PEP 3148 proposing futures as a new asynchronous execution method, some recent IronPython work, and a new Python podcast.

• Episode 12 MP3
• Episode 12 AAC

Episode 13 is longer one; a 40 minute discussion covering topics like:

• Python 2.7 beta 1 released.
• PEP 3147: New bytecode directory layout.
• Google's Summer of Code beginning.
• SEC proposes mandating Python's use in financial filings.
• PyCon interview: Dr Tim Couper
• How to Fund Python Development
• Python for Beginners: Getting started on Windows.

The episode 13 files:

• Episode 13 MP3
• Episode 13 AAC

Episode 14 is another interview, recorded at PyCon with Christian Tismer. Christian is a long standing member of the Python community and, amongst other things, he is the original creator of Stackless and has worked on both psyco and PyPy. In this interview we discuss all of these projects, both their history and what the future holds for them.

• Episode 14 MP3
• Episode 14 AAC

A Little Bit of Python has an iTunes page, an RSS feed and a Twitter account, so you have no excuse for missing it:

• @bitofpython
• A Little Bit of Python on iTunes
• RSS Feed of mp3s

If you have feedback, insults or suggestions for new topics you can email us on: all@bitofpython.com.

Python 2.7 Release Candidate 1 and unittest 0.4.2

Python 2.7 is nearly here. Release candidate 1 is now out. There will probably be a few minor bugfixes before the final release, but now is the time to test your code with Python 2.7.

Despite the language moratorium there are a host of exciting new features in Python 2.7. You can read about them in what's new (which is rather long), or read a summary in my previous blog entry from when beta 2 was released.

The other exciting release from the last few days is unittest2 0.4.2. unittest2 is a backport of the new features in the Python 2.7 version of the standard library testing framework unittest. Version 0.4.2 has feature parity with unittest in Python 2.7 RC 1 and includes a few bug fixes since the last release.

• unittest2 on PyPI

The major improvements over unittest in Python 2.6 include:

• A standard test runner with automatic test discovery
• Improved command line options - fail fast, control-c catching and buffering standard out
• Many new assert methods
• Improvements to assertRaises (as context manager) and assertAlmostEquals (delta keyword argument)
• Class and module level setup and teardown
• Cleanup functions for better resource handling
• Test skipping and expected failures
• Lots of other minor changes and improvements

For more details see:

http://www.voidspace.org.uk/python/articles/unittest2.shtml

Changes since 0.4.1:

The Python Testing Tools Taxonomy

The Python Testing Tools Taxonomy is the creation of Grig Gheorghiu and for several year has been an invaluable resource for the Python community.

As with many projects it has become a pain to administer and keep updated, so I'm working with Grig to convert the Trac markup into reStructured Text and maintain it in a bitbucket (mercurial) project that is easier to contribute to.

It's still early days (I'm about one third of the way through) but in the spirit of release early and often, what I've done so far is up online in the new home for the Taxonomy:

• The New Python Testing Tools Taxonomy

If anyone has a prettier template & stylesheet for reStructured Text documents then I would much appreciate it; the default one is a bit 'plain'. If you feel like contributing (which would also be much appreciated), the source is available from the bitbucket taxonomy project.

• Front page
• All By Date
• All By Category
• Blog Statistics
• RSS Feed
• Return to Top

Technorati...
This work is licensed under a Creative Commons Attribution-Share Alike 2.0 License.

Counter...

• IronPython in Action

  

  IronPython in Action is the first English book on IronPython. It is for Python and .NET programmers interested in making the most of Python on the .NET framework and Mono.

• Search Site

  	Search this Site:
  	www voidspace.org.uk

• Browsing

  Home > Python Index > Techie Blog

• Blogads

  Blogads

• Michael Foord

  • Contact Michael
  • Michael on Twitter
  • My bookmarks
  • Shared articles
  • Photos on Flickr
  • Voidspace Amazon Store
  • Amazon Computers: Todays Deals

• Hosting