Python Programming, news on the Voidspace Python Projects and all things techie.

A plugin system for unittest(2)

emoticon:movies One of the big problems with unittest, which I have been maintaining for well over a year now, is how hard it is to extend. For a long time I've been saying that my "big task" with unittest was to implement a plugin system. Well, at the EuroPython sprints I finally got round to making a start.

There is now a prototype plugin system for unittest2 in the plugins branch of the mercurial repository. You can try it out with:

hg clone http://hg.python.org/unittest2
cd unittest2
hg up plugins
python setup.py install

I'm looking for feedback on the system, and have written up a rather lengthy "proto-pep" that describes the whole system. Most of the document describes the specific extension points, so you can get a good feel for it from just reading the first few sections. I may rewrite it as an official PEP, as my goal is to get this into unittest for Python 3.2, but at the moment it lives in the repository or can be found online at:

http://hg.python.org/unittest2/raw-file/tip/description.txt

As part of the prototype I have been implementing some example plugins (in unittest2/plugins/) so I can develop the mechanism against real rather than imagined use cases. Jason Pellerin, creator of nose, has been providing me with feedback and has been trying it out by porting some of the nose plugins to unittest [1]. His initial comment on the system was "it looks very flexible and clean". Wink

One of the goals of the extension mechanism is to allow nose2 to be a much thinner set of plugins over unittest(2), making nose2 simpler to maintain and less likely to break every time unittest changes [2].

Example plugins available and included:

  • a pep8 and pyflakes checker
  • a debugger plugin that drops you into pdb on test fail / error
  • a doctest loader (looks for doctests in all text files in the project)
  • use a regex for matching files in test discovery instead of a glob
  • growl notifications on test run start and stop
  • filter individual test methods using a regex
  • load test functions from modules as well as TestCases
  • integration with the coverage module for coverage reporting

In addition I intend to create a plugin that outputs junit compatible xml from a test run (for integration with tools like the hudson continuous integration server) and a test runner that runs tests in parallel using multiprocessing.

Not all of these will be included in the merge to unittest. Which ones will is still an open question.

I'd like feedback on the proposal, and hopefully approval to port it into unittest after discussion / amendment / completion. In particular I'd like feedback on the basic system, plus which events should be available and what information should be available in them. Note that the system is not complete in the prototype. Enough is implemented to get "the general idea" and to formalise the full system. It still needs extensive tests and the extra work in TestProgram makes it abundantly clear that refactoring there is well overdue...

So, give it a try, find bugs in it, write plugins with it and let me know what you think...

Here's an example of a unittest plugin. This one is the "debugger" plugin that drops you into pdb on test fail or error (configurable):

from unittest2 import Plugin

import pdb
import sys

class Debugger(Plugin):

    configSection = 'debugger'
    commandLineSwitch = ('D', 'debugger', 'Enter pdb on test fail or error')

    def __init__(self):
        self.errorsOnly = self.config.as_bool('errors-only', default=False)

    def onTestFail(self, event):
        value, tb = event.exc_info[1:]
        test = event.test
        if self.errorsOnly and isinstance(value, test.failureException):
            return
        original = sys.stdout
        sys.stdout = sys.__stdout__
        try:
            pdb.post_mortem(tb)
        finally:
            sys.stdout = original

The plugin is hooked into the command line interface of the unit2 test runner and the plugin configuration files through the configSection and commandLineSwitch class attributes. There are APIs that give complete control over this, but the Plugin class provides some nice sugar for making writing plugins simpler. See the description document for more details than you could possibly want...

To get started you'll need a unittest.cfg file in either your home directory or the same directory you run unit2 from. There is an example config file, with all the plugins configured, in the unittest2 repository. Once you have that in place unit2 -h will show you command line options for all the configured plugins:

--checker             Check all Python files with pep8 and pyflakes
--test-functions      Load test functions from test modules
--doctest             Load doctests from text files
-C, --coverage        Enable coverage reporting
--cover-module=COVER-MODULE
                      Specify a module or package for coverage
                      reporting
-G, --growl           Growl notifications on test run start and stop
-F FILTER, --filter=FILTER
                      Filter test methods loaded with a regexp
-D, --debugger        Enter pdb on test fail or error
-R, --match-regexp    Match filenames during test discovery with
                      regular expressions instead of glob

Most of the plugins can be configured in the config file(s) too. Looking at the options there and poking around in the source code of the plugins should show you everything you need to know...

[1]See http://bitbucket.org/jpellerin/unittest2/src/tip/unittest2/plugins/attrib.py and http://bitbucket.org/jpellerin/unittest2/src/tip/unittest2/plugins/errormaster.py
[2]http://lists.idyll.org/pipermail/testing-in-python/2010-March/002799.html

Like this post? Digg it or Del.icio.us it.

Posted by Fuzzyman on 2010-07-30 13:15:19 | |

Categories: , , Tags: , ,


Hosted by Webfaction

Counter...