Python Programming, news on the Voidspace Python Projects and all things techie.

tox: Testing projects with multiple versions of Python

emoticon:globesearch mock is tested on Python 2.4 upwards. Running all the tests is done with unittest2 test discovery, so up until now I've had several scripts for running the tests with different versions of Python. Not difficult, but a pain.

Enter a new project by the prodigious Holger Krekel:

To use tox you create a simple config file tox.ini specifying the test command and Python versions and tox will create a virtualenv for each Python version and run your tests in it.

I can now run all the mock tests for Python 2.4 - 2.7 on both Mac OS X and Windows with a single command (a single command on each platform - although tox does have Hudson support for continuous integration servers).

As well as running tests tox can execute arbitrary commands. This means it can run sphinx commands. For mock I have tox execute two Sphinx commands to build the documentation and run all the doctests (only for Python 2.6 & 2.7 as some of the doctests use the with statement). This way the tox run fails if there are any errors in the documentation, either reStructured Text errors or doctest failures.

A successful run looks something like this (output from Windows shown - with lots of it snipped):

> tox
_________________________________ [tox sdist] _________________________________
[TOX] ***creating sdist package
[TOX] C:\compile\mock$ C:\Python26\python.exe sdist --formats=zip --dis
t-dir .tox\dist >.tox\log\0.log
[TOX] ***copying new sdistfile to 'C:\\Users\\michael\\.tox\\distshare\\mock-0.7'
_____________________________ [tox testenv:py24] ______________________________
[TOX] ***reusing existing matching virtualenv py24
[TOX] C:\compile\mock\.tox$ py24\Scripts\pip.exe install dist\ --d
ownload-cache=C:\compile\mock\.tox\_download >py24\log\3.log
[TOX] C:\compile\mock$ .tox\py24\Scripts\unit2 discover
Ran 84 tests in 0.109s


No builder selected, using default: html
building [html]: targets for 0 source files that are out of date
updating environment: 0 added, 0 changed, 0 removed
looking for now-outdated files... none found
no targets are out of date.
________________________________ [tox summary] ________________________________
[TOX] py24: commands succeeded
[TOX] py25: commands succeeded
[TOX] py26: commands succeeded
[TOX] py27: commands succeeded
[TOX] congratulation :)


Notice how the output from tox shows us exactly which commands are being executed. The test execution is done with test discovery using the command .tox\py24\Scripts\unit2 discover.

Here's the tox.ini for the mock module:

envlist = py24,py25,py26,py27

commands=unit2 discover []

    unit2 discover []
    sphinx-build -b doctest docs html
    sphinx-build docs html
deps =

    unit2 discover []
    sphinx-build -b doctest docs html
    sphinx-build docs html
deps =

This tells tox that when the tox command is run it is to create four different virtual environments with the four different versions of Python (so all these versions of Python need to be installed). For Python 2.6 and 2.7 there are custom command sets and Python 2.4 and 2.5 use the default [testenv] block.


Setting the PIP_DOWNLOAD_CACHE environment variable to a valid directory allows pip to reuse downloaded packages when it creates and populates virtual environments. It will reuse virtual environments anyway, but keeping the cache around can be a good way of avoiding unnecessary network traffic for package downloads.

The dependencies for Python 2.4 & 2.5 are just unittest2, for 2.6 & 2.7 Sphinx is also a dependency. The test command is, as we've seen, with unit2 discover. The [] after the command allows extra command line options to be passed through from tox to the test runner. For example we could use this to modify the parameters for test discovery:

tox -- -p test_signal\*

Although tox is in its early days its already useful for me to be able to run tests on four versions of Python, including testing the Sphinx documentation builds correctly and the code examples work, with a single command.

Like this post? Digg it or it.

Posted by Fuzzyman on 2010-07-13 00:21:09 | |

Categories: , Tags: , ,

unittest2 0.5.0: setuptools compatible test collector and Python 2.3 distribution

emoticon:waffle unittest2 0.5.0 has just been released. This version of unittest2 has "feature parity" with the version of unittest in Python 2.7:

If you want to ensure that your tests run identically under unittest2 and unittest in Python 2.7 you should use unittest2 0.5.0. Later versions of unittest2 will include changes in unittest made in Python 3.2 and onwards after the release of Python 2.7.

unittest2 is a backport of the recent enhancements in the Python unittest testing library in Python 2.7.

Improvements in unittest2 over standard unittest in Python 2.6 and earlier include:

  • automatic test discovery from the command line
  • failfast, catch and buffer command line options
  • class and module level setup and teardowns
  • addCleanup for better resource handling
  • test skipping and expected failures
  • improvements to assertRaises and assertAlmostEqual
  • many new assert methods, plus better failure messages
  • load_tests protocol for customizing test loading
  • various other API improvements and fixes

unittest2 is tested with Python 2.4 - 2.7. New in 0.5.0 is a distribution for Python 2.3 contributed by Mark Roddy.

Also new in 0.5.0 is a setuptools compatible test collector. If you put test_suite = 'unittest2.collector' in you can then find and run your tests (using test discovery) with:

python test

This starts test discovery with the default parameters from the directory containing, so it is perhaps most useful as an example (see unittest2/

All Changes in 0.5.0

Addition of a setuptools compatible test collector (very basic). Specify test_suite = 'unittest2.collector' in your

TestSuite.debug() and TestCase.debug() now execute cleanup functions and class and module level setups and teardowns.

No longer monkey-patch os.path.relpath for Python 2.4 / 2.5 so that projects don't accidentally depend on our patching. Contributed by Konrad Delong.

Added a Python version specific unit2 entrypoint. This will, for example, create a unit2-2.6 script if unittest2 is installed with Python 2.6. (Requires setuptools or distribute.)

Python 2.3 compatibility (in the python2.3 branch of the repository), contributed by Mark Roddy.

setuptools console script entry points are created as '.py' scripts on Windows.

Feature parity with the Python 2.7 final release.

Like this post? Digg it or it.

Posted by Fuzzyman on 2010-07-11 20:02:03 | |

Categories: , Tags: , , ,

Hosted by Webfaction