Python Programming, news on the Voidspace Python Projects and all things techie.

ContextDecorator: creating APIs that work as decorators and context managers

emoticon:dove Two of the best additions to Python in recent years are the with statement and decorators. Both context managers (objects used in with statements) and decorators can be used for similar purposes: performing an action before and after executing the decorated function or the code inside the with block. In fact I now find that many places I used to use decorators I now prefer the with statement (if I'm lucky enough to be able to ignore Python 2.4 compatibility).

If you're a library or framework creator then it is nice to be able to create APIs that can be used either as decorators or context managers. The patch decorators in mock behave like this, and when I was writing a new variant (patch.dict) I found myself having to figure out again how to do it. It isn't hard, but it's a bit fiddly. Nor is this an uncommon pattern, both py.test and Django have code that behaves like this.

I've written a very simple utility class that does this, called ContextDecorator, and it is now part of contextlib in Python 3.2.

Context managers inheriting from ContextDecorator have to implement __enter__ and __exit__ as normal. __exit__ retains its optional exception handling even when used as a decorator.

Even better contextlib.contextmanager, which is a decorator for writing context managers as functions, uses ContextDecorator so the context managers it creates can automatically be used as decorators as well.

I've put both ContextDecorator and the new contextmanager into a package on PyPI, and it works with all versions of Python from 2.4 - 3.1.


from contextdecorator import ContextDecorator

class mycontext(ContextDecorator):
   def __enter__(self):
      print 'Starting'
      return self

   def __exit__(self, *exc):
      print 'Finishing'
      return False
>>> @mycontext()
... def function():
...    print 'The bit in the middle'
>>> function()
The bit in the middle

>>> with mycontext():
...    print 'The bit in the middle'
The bit in the middle

Existing context managers that already have a base class can be extended by using ContextDecorator as a mixin class:

from contextdecorator import ContextDecorator

class mycontext(ContextBaseClass, ContextDecorator):
   def __enter__(self):
      return self

   def __exit__(self, *exc):
      return False

contextdecorator also contains an implementation of contextlib.contextmanager that uses ContextDecorator. The context managers it creates can be used as decorators as well as in with statements.

from contextdecorator import contextmanager

def mycontext(*args):
   print 'Started'
     # decorated function or with
     # statement executed here
      # exception handling here
      print 'Finished!'
>>> @mycontext('some', 'args')
... def function():
...    print 'In the middle'
In the middle

>>> with mycontext('some', 'args'):
...    print 'In the middle'
In the middle

Like this post? Digg it or it.

Posted by Fuzzyman on 2010-07-01 00:46:45 | |

Categories: , , Tags: , , ,

Porting mock to Python 3

emoticon:men One of the nice new features in mock 0.7 is that it works with both Python 2 & 3. The mock module itself, even with all the freshly added docstrings, weighs in at less than 800 lines of code so compatibility is maintained with a single source base rather than the more recommended 2to3 approach. There are however about 1500 lines of test code that also need to work under Python 3; so whilst not a particularly difficult exercise it was not entirely trivial to get all the tests passing under Python 2.4, 2.5, 2.6, 2.7 and 3.2. Good tests make it much easier to have confidence that the port works. Attempting this without tests would be much more painful, even though it means there is more code to port.

I've written up all the changes needed for mock to support Python 3:

Like this post? Digg it or it.

Posted by Fuzzyman on 2010-06-28 18:56:12 | |

Categories: , , Tags: , , , ,

Release: mock 0.7 beta 2

emoticon:avocado I'm pleased to announce a new release of the mock module, the first in a while. Konrad Delong has joined me as a maintainer of mock and has been a great help in getting this release out. As there are several major new features this is a beta release, with 0.7.0 final coming out in a few weeks assuming there are no major problems discovered. Please download it and try it out:

mock is a Python module that provides a core Mock class. It is intended to reduce the need for creating a host of trivial stubs throughout your test suite. After performing an action, you can make assertions about which methods / attributes were used and arguments they were called with. You can also specify return values and set needed attributes in the normal way.

The mock module also provides utility functions / objects to assist with testing, particularly monkey patching.

mock is tested on Python versions 2.4-2.7 and Python 3.

Full documentation is included in the distribution.

Mock is very easy to use and is designed for use with unittest. Mock is based on the 'action -> assertion' pattern instead of 'record -> replay' used by many mocking frameworks. See the mock documentation for full details.

Changes in 0.7.0 (including the much awaited magic method support) are:

  • Addition of mocksignature
  • Ability to mock magic methods
  • Ability to use patch and patch.object as class decorators
  • Renamed patch_object to patch.object (patch_object is deprecated)
  • Addition of MagicMock class with all magic methods pre-created for you
  • Python 3 compatibility (tested with 3.2 but should work with 3.0 & 3.1 as well)
  • Addition of patch.dict(...) for changing dictionaries during a test
  • Addition of mocksignature argument to patch and patch.object
  • help(mock) works now (on the module). Can no longer use __bases__ as a valid sentinel name (thanks to Stephen Emslie for reporting and diagnosing this)
  • Addition of soft comparisons: call_args, call_args_list and method_calls return now tuple-like objects which compare equal even when empty args or kwargs are skipped
  • Added some docstrings.
  • BUGFIX: side_effect now works with BaseException exceptions like KeyboardInterrupt
  • BUGFIX: patching the same object twice now restores the patches correctly
  • The tests now require unittest2 to run
  • Konrad Delong added as co-maintainer

There are several major new features in this release, not least of which is the support for mocking the Python protocols (magic methods).

The easiest way of using magic methods is with the MagicMock class. It allows you to do things like:

>>> from mock import MagicMock
>>> mock = MagicMock()
>>> mock.__str__.return_value = 'foobarbaz'
>>> str(mock)
>>> mock.__str__.assert_called_with()


In the 0.7.0 final release (and already in svn) using the spec keyword argument to MagicMock will only pre-create the magic methods that are in the spec object or list.

Mock allows you to assign functions (or other Mock instances) to magic methods and they will be called appropriately. The MagicMock class is just a Mock variant that has all of the magic methods pre-created for you (well - all the useful ones anyway).

The following is an example of using magic methods with the ordinary Mock class:

>>> from mock import Mock
>>> mock = Mock()
>>> mock.__str__ = Mock()
>>> mock.__str__.return_value = 'wheeeeee'
>>> str(mock)

mocksignature is a useful companion to Mock and patch. It creates copies of functions that delegate to a mock, but have the same signature as the original function. This ensures that your mocks will fail in the same way as your production code if they are called incorrectly:

>>> from mock import mocksignature
>>> def function(a, b, c):
...     pass
>>> function2 = mocksignature(function)
>>> function2.mock.return_value = 'fishy'
>>> function2(1, 2, 3)
>>> function2.mock.assert_called_with(1, 2, 3)
>>> function2('wrong arguments')
Traceback (most recent call last):
TypeError: <lambda>() takes exactly 3 arguments (1 given)

Also new is patch.dict for setting values in a dictionary just during a test and restoring the dictionary to its original state when the test ends:

>>> foo = {'key': 'value'}
>>> original = foo.copy()
>>> with patch.dict(foo, {'newkey': 'newvalue'}, clear=True):
...     assert foo == {'newkey': 'newvalue'}
>>> assert foo == original

Like this post? Digg it or it.

Posted by Fuzzyman on 2010-06-27 15:20:42 | |

Categories: , Tags: , , ,

Authentication with Python aside, did you know as long as you have a computer running Windows OS and that's hooked up to the internet, you could remotely view cctv camera systems, from anywhere in the world? Well you can. Check out this website for more information.

Discover 0.4.0: test discovery for unittest

emoticon:scanner discover is a backport of the new test discovery features only from Python 2.7 / 3.2. The discover module provides automatic test discovery for standard unittest based tests:

python -m discover

If you have setuptools or distribute installed you will also have a discover script available.

This will discover all tests (with certain restrictions) from the current directory. The discover module has several options to control its behavior (full usage options are displayed with python -m discover -h).

discover 0.4.0 provides feature parity with the test discovery in Python 2.7 RC1 and unittest2 0.4.2.

unittest2 provides not just the test discovery features that are new in Python 2.7, but a whole lot more as well.

The full list of changes since discover 0.3.2:

  • Addition of a setuptools compatible test collector. Set "test_suite = 'discover.collector'" in " test" will start test discovery with default parameters from the same directory as the
  • Allow test discovery using dotted module names instead of a path.
  • Addition of a setuptools compatible entrypoint for the discover script.
  • A faulty load_tests function will not halt test discovery. A failing test is created to report the error.
  • If test discovery imports a module from the wrong location (usually because the module is globally installed and the user is expecting to run tests against a development version in a different location) then discovery halts with an ImportError and the problem is reported.
  • Matching files during test discovery is done in DiscoveringTestLoader._match_path. This method can be overriden in subclasses to, for example, match on the full file path or use regular expressions for matching.
  • Tests for discovery ported from unittest2. (The tests require unittest2 to run.)

Like this post? Digg it or it.

Posted by Fuzzyman on 2010-06-27 14:09:30 | |

Categories: , Tags: , ,

Hosted by Webfaction