Python Programming, news on the Voidspace Python Projects and all things techie.

mock 0.8 alpha 1: New Features

emoticon:cyberpunk This is a long entry, please forgive me. It describes all the new features in mock 0.8.0 alpha 1. The main reason I need to describe it here is that I haven't yet written the documentation. Surprised

mock is a library for testing in Python. It allows you to replace parts of your system under test with mock objects. The latest stable release is 0.7.2, which you can download from pypi.

You can try out the new features described below by downloading the new alpha release. Alternatively you can pip install it:

pip install -U mock==dev

As this is an alpha release it isn't feature complete. There are still a lot of features, mainly but not only small ones, I'm looking at. There are some exciting new features, so definitely worth trying out the alpha if any of them particularly interest you. None of these features are frozen, but the ones highlighted below are fully implemented and tested. I'd really like feedback on these changes, particularly if you dislike them or have comments.

Here's a summary of the important changes:

  • new autospec functionality - lazy, recursive, automatic speccing (see below...)
  • patchers create a MagicMock not a Mock
  • patchers and Mock take arbitrary keyword arguments for setting attributes on mocks
  • protocol methods on MagicMock are hooked up as MagicMocks not Mocks
  • on Python 2.6+ dir(Mock()) has custom behaviour
  • new helper objects: ANY and call

It's not obvious what all of these do just from the summary. Here's the full skinny, starting with the most important new feature "autospeccing".

Autospeccing

Autospeccing is based on the existing spec feature of mock. It limits the api of mocks to the api of an original object (the spec), but it is recursive (implemented lazily) so that attributes of mocks only have the same api as the attributes of the spec. In addition mocked functions / methods have the same call signature as the original so they raise a TypeError if they are called incorrectly. This feature is a better version of both the spec and mocksignature features in the current mock.

Before I explain how auto-speccing works, here's why it is needed.

Mock is a very powerful and flexible object, but it suffers from two flaws when used to mock out objects from a system under test. One of these flaws is specific to the Mock api and the other is a more general problem with using mock objects.

First the problem specific to Mock. Mock has two assert methods that are extremely handy: assert_called_with and assert_called_once_with.

>>> mock = Mock(name='Thing', return_value=None)
>>> mock(1, 2, 3)
>>> mock.assert_called_once_with(1, 2, 3)
>>> mock(1, 2, 3)
>>> mock.assert_called_once_with(1, 2, 3)
Traceback (most recent call last):
 ...
AssertionError: Expected to be called once. Called 2 times.

Because mocks auto-create attributes on demand, and allow you to call them with arbitrary arguments, if you misspell one of these assert methods then your assertion is gone:

>>> mock = Mock(name='Thing', return_value=None)
>>> mock(1, 2, 3)
>>> mock.assret_called_once_with(4, 5, 6)

Your tests can pass silently and incorrectly because of the typo.

The second issue is more general to mocking. If you refactor some of your code, rename members and so on, any tests for code that is still using the old api but uses mocks instead of the real objects will still pass. This means your tests can all pass even though your code is broken.

Note that this is another reason why you need integration tests as well as unit tests. Testing everything in isolation is all fine and dandy, but if you don't test how your units are "wired together" there is still lots of room for bugs that tests might have caught.

mock already provides a feature to help with this, called speccing. If you use a class or instance as the spec for a mock then you can only access attributes on the mock that exist on the real class:

>>> import urllib2
>>> mock = Mock(spec=urllib2.Request)
>>> mock.assret_called_with
Traceback (most recent call last):
 ...
AttributeError: Mock object has no attribute 'assret_called_with'

The spec only applies to the mock itself, so we still have the same issue with any methods on the mock:

>>> mock.has_data()
<mock.Mock object at 0x...>
>>> mock.has_data.assret_called_with()

Auto-speccing solves this problem, and combines the features of mocksignature. You can either pass autospec=True to patch / patch.object or use the create_autospec function to create a mock with a spec. If you use the autospec=True argument to patch then the object that is being replaced will be used as the spec object. Because the speccing is done "lazily" (the spec is created as attributes on the mock are accessed) you can use it with very complex or deeply nested objects (like modules that import modules that import modules) without a big performance hit [1].

Here's an example of it in use:

>>> import urllib2
>>> patcher = patch('__main__.urllib2', autospec=True)
>>> mock_urllib2 = patcher.start()
>>> urllib2 is mock_urllib2
True
>>> urllib2.Request
<MagicMock name='urllib2.Request' spec='Request' id='...'>

You can see that urllib2.Request has a spec. urllib2.Request takes two arguments in the constructor, here's what happens if we try to call it incorrectly:

>>> req = urllib2.Request()
Traceback (most recent call last):
 ...
TypeError: <lambda>() takes at least 2 arguments (1 given)

The spec also applies to instantiated classes (i.e. the return value of specced mocks):

>>> req = urllib2.Request('foo', 'bar')
>>> req
<MagicMock spec='Request' id='8064976'>

Because the spec also applies to methods, any typos in our asserts will raise the correct error:

>>> req.add_header('spam', 'eggs')
<mock.MagicMock object at 0x...>
>>> req.add_header.assret_called_with
Traceback (most recent call last):
 ...
AttributeError: Mock object has no attribute 'assret_called_with'
>>> req.add_header.assert_called_with('spam', 'eggs')

In many cases you will just be able to add autospec=True to your existing patch calls, but you will be protected against bugs due to typos and api changes.

As well as using autospec through patch there is a create_autospec that can be used to create autospecced mocks directly:

>>> from mock import create_autospec
>>> import urllib2
>>> mock_urllib2 = create_autospec(urllib2)
>>> mock_urllib2.Request('foo', 'bar')
<MagicMock spec='Request' id='...'>

This isn't without caveats and limitations however, which is why it has not been made the default behaviour. In order to know what attributes are available on the spec object, autospec has to introspect (access attributes) the spec. As you traverse attributes on the mock a corresponding traversal of the original object is happening under the hood. If any of your specced objects have properties or descriptors that can trigger code execution then you may not be able to use autospec. On the other hand it is much better to design your objects so that introspection is safe [2].

A more serious problem is that it is common for instance attributes to be created in the __init__ method and not to exist on the class at all. autospec can't know about any dynamically created attributes and restricts the api to visible attributes.

>>> class Something(object):
...   def __init__(self):
...     self.a = 33
...
>>> with patch('__main__.Something', autospec=True):
...   thing = Something()
...   thing.a
...
Traceback (most recent call last):
  ...
AttributeError: Mock object has no attribute 'a'

There are a few different ways of resolving this problem. The easiest, but not necessarily the least annoying, way is to simply set the required attributes on the mock after creation. Just because autospec doesn't allow you to fetch attributes that don't exist on the spec it doesn't prevent you setting them:

>>> with patch('__main__.Something', autospec=True):
...   thing = Something()
...   thing.a = 33
...

There is a more aggressive version of both spec and autospec that does prevent you setting non-existent attributes. This is useful if you want to ensure your code only sets valid attributes too, but obviously it prevents this particular scenario:

>>> with patch('__main__.Something', autospec=True, spec_set=True):
...   thing = Something()
...   thing.a = 33
...
Traceback (most recent call last):
 ...
AttributeError: Mock object has no attribute 'a'

Probably the best way of solving the problem is to add class attributes as default values for instance members initialised in __init__. Note that if you are only setting default attributes in __init__ then providing them via class attributes (shared between instances of course) is faster too. e.g.

class Something(object):
    a = 33

This brings up another issue. It is relatively common to provide a default value of None for members that will later be an object of a different type. None would be useless as a spec because it wouldn't let you access any attributes or methods on it. As None is never going to be useful as a spec, and probably indicates a member that will normally of some other type, autospec doesn't use a spec for members that are set to None. These will just be ordinary mocks (well - MagicMocks):

>>> class Something(object):
...     member = None
...
>>> mock = create_autospec(Something)
>>> mock.member.foo.bar.baz()
<mock.MagicMock object at 0x...>

If modifying your production classes to add defaults isn't to your liking then there are more options. One of these is simply to use an instance as the spec rather than the class. The other is to create a subclass of the production class and add the defaults to the subclass without affecting the production class. Both of these require you to use an alternative object as the spec. Thankfully patch supports this - you can simply pass the alternative object as the autospec argument:

>>> class Something(object):
...   def __init__(self):
...     self.a = 33
...
>>> class SomethingForTest(Something):
...   a = 33
...
>>> p = patch('__main__.Something', autospec=SomethingForTest)
>>> mock = p.start()
>>> mock.a
<MagicMock name='Something.a' spec='int' id='...'>

Changes to patchers

In 0.8.0 patch and patch.object return MagicMock instances instead of Mock instances by default. This is because MagicMock is more useful. You can still specify the object to be returned if you really need something patched to not be a MagicMock.

The patchers, plus Mock itself now take arbitrary keyword arguments to set attributes on the created mocks. This makes it easier to create and configure a mock in a single line. This was a several-times-requested feature.

>>> mock = Mock(foo=3, bar=6, baz=9)
>>> mock.foo, mock.bar, mock.baz
(3, 6, 9)

You can also set return_value and side_effect on mock attributes and access attributes of attributes. This uses the normal dotted syntax, but as this isn't valid for keyword arguments you'll have to pass it in as **'d dictionary.

>>> args = {'return_value': 3, 'foo.return_value': 6, 'bar.side_effect': KeyError, 'baz.spam': 'weeee'}
>>> mock = Mock(**args)
>>> mock()
3
>>> mock.foo()
6
>>> mock.bar()
Traceback (most recent call last):
 ...
KeyError
>>> mock.baz.spam
'weeee'

My initial thought was to always have the attributes passed in as a dictionary to another keyword argument. The advantage of this approach would have been that if you misspelled any keyword arguments to patch / Mock then you would get a sensible error message. Taking arbitrary keyword arguments means you set an attribute on the mock instead!

>>> mock = Mock(retrun_value=3)
>>> mock()
<mock.Mock object at 0x...>
>>> mock.retrun_value
3

Just being able to pass keyword arguments to create attributes is a much nicer api to use, out-weighing the downsides in my opinion. I'm open to feedback as to which is preferable.

patch.dict also takes keywords arguments to set entries in the dictionaries it is patching:

>>> with patch.dict('sys.modules', foo=Mock(name='mock_foo')):
...   import foo
...   print foo
...
<Mock name='mock_foo' id='...'>

As well as the patch and Mock constructors there is a new Mock method, configure_mock, that you can use to configure a mock after it has been created. It works in exactly the same way (through arbitrary keyword arguments).

dir and helper objects

call and ANY are available to import from mock as helper objects. They are basically unchanged from the recipes on the examples page of the documentation.

Mock has gained a custom __dir__ method. In Python 2.6, or any more recent version, dir(some_mock) shows only useful attributes and will include any dynamically created attributes that wouldn't normally be shown. If the mock was created with a spec (or autospec of course) then all the attributes from the original are shown, even if they haven't been accessed yet:

>>> dir(Mock())
['assert_called_once_with', 'assert_called_with', 'call_args', 'call_args_list',
'call_count', 'called', 'configure_mock', 'method_calls', 'mock_calls',
'reset_mock', 'return_value', 'side_effect']
>>> import urllib2
>>> dir(Mock(spec=urllib2))
['AbstractBasicAuthHandler', 'AbstractDigestAuthHandler', 'AbstractHTTPHandler',
'BaseHandler', 'CacheFTPHandler', 'FTPHandler', 'FileHandler', 'HTTPBasicAuthHandler',
'HTTPCookieProcessor', 'HTTPDefaultErrorHandler', 'HTTPDigestAuthHandler', 'HTTPError',
...]

By way of contrast, here's the behaviour with mock 0.7:

>>> dir(Mock())
['_Mock__get_return_value', '_Mock__return_value_doc', '_Mock__set_return_value', '__call__',
'__class__', '__delattr__', '__dict__', '__doc__', '__format__', '__getattr__', '__getattribute__',
'__hash__', '__init__', '__module__', '__new__', '__reduce__', '__reduce_ex__', '__repr__',
'__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', '_children',
...]
>>> import urllib2
>>> dir(Mock(spec=urllib2))
['__class__', '__delattr__', '__dict__', '__doc__', '__format__', '__getattribute__', '__hash__',
'__init__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__',
'__subclasshook__', '_children', '_methods', '_name', '_parent', '_return_value', '_spec_class',
'_spec_set', '_wraps', 'call_args', 'call_args_list', 'call_count', 'called', 'method_calls',
'side_effect']

As you can see, in 0.8 many of the not-very-useful (private to Mock rather than the thing being mocked) underscore and double underscore prefixed attributes have been filtered from the result of calling dir on a Mock. If you dislike this behaviour you can switch it off by setting the module level switch FILTER_DIR:

>>> import mock
>>> mock.FILTER_DIR = False
>>> dir(mock.Mock())
['_Mock__get_return_value', '_Mock__return_value_doc', '_Mock__set_return_value', '__call__',
'__class__', '__delattr__', '__dict__', '__dir__', '__doc__', '__format__', '__getattr__',
'__getattribute__', '__hash__', '__init__', '__module__', '__new__', '__reduce__',
...]

Alternatively you can just use vars(my_mock) (instance members) and dir(type(my_mock)) (type members) to bypass the filtering irrespective of mock.FILTER_DIR.

It was implementing the custom dir behaviour that prompted the blog entry Implementing __dir__.

Protocol Methods on MagicMock

Another several-times-requested feature, but one that was much harder to implement than you would think. mock 0.7.0 introduces support for the Python protocol methods, along with the MagicMock which has most of the useful protocol methods hooked up for you.

In 0.7 the protocol methods are hooked up with Mocks, which mean that many of them return Mock instances when called. Several people requested that they be MagicMocks so that the result of calling a protocol method was a MagicMock. This is now done, by lazily creating the MagicMocks on first use.

You can now do the following (if you really want):

>>> m = MagicMock()
>>> m[1][2][3][4]
<mock.MagicMock object at 0x...>

That covers all the changes worth blogging about. Please try it out and let me know what you think (apart from the lack of documentation). There are a bunch of other minor changes. You can see them all in the changelog (in the repo).


[1]The initial implementation wasn't lazy. When you used autospec it would recursively mock out the object, including all attributes and attributes of attributes and so on. For very deeply nested objects this could take a long time and created tens of thousands of mocks!
[2]This only applies to classes or already instantiated objects. Calling a mocked class to create a mock instance does not create a real instance. It is only attribute lookups - along with calls to dir - that are done. A way round this problem would have been to use getattr_static, which can fetch attributes without triggering code execution. Descriptors like classmethod and staticmethod need to be fetched correctly though, so that their signatures can be mocked correctly.

Like this post? Digg it or Del.icio.us it.

Posted by Fuzzyman on 2011-06-14 12:13:00 | |

Categories: , Tags: , ,


Mocking Generator Methods

emoticon:file1 Another mock recipe, this one for mocking generator methods.

A Python generator is a function or method that uses the yield statement to return a series of values when iterated over [1].

A generator method / function is called to return the generator object. It is the generator object that is then iterated over. The protocol method for iteration is __iter__, so we can mock this using a MagicMock.

Here's an example class with an "iter" method implemented as a generator:

>>> class Foo(object):
...     def iter(self):
...         for i in [1, 2, 3]:
...             yield i
...
>>> foo = Foo()
>>> list(foo.iter())
[1, 2, 3]

How would we mock this class, and in particular its "iter" method?

To configure the values returned from the iteration (implicit in the call to list), we need to configure the iterator returned by the call to foo.iter().

>>> values = [1, 2, 3]
>>> mock_foo = MagicMock()
>>> iterable = mock_foo.iter.return_value
>>> iterator = iter(values)
>>> iterable.__iter__.return_value = iterator
>>> list(mock_foo.iter())
[1, 2, 3]

The above example is done step-by-step. The shorter version is:

>>> mock_foo = MagicMock()
>>> mock_foo.iter.return_value.__iter__.return_value = iter([1, 2, 3])
>>> list(mock_foo.iter())
[1, 2, 3]

This is now also in the docs on the mock examples page. There's now quite a collection of useful mock recipes there, so even if you're an experienced mock user it is worth a browse.

[1]There are also generator expressions and more advanced uses of generators, but we aren't concerned about them here. A very good introduction to generators and how powerful they are is: Generator Tricks for Systems Programmers.

Like this post? Digg it or Del.icio.us it.

Posted by Fuzzyman on 2011-06-13 17:24:10 | |

Categories: , , Tags: , ,


Hosted by Webfaction

Counter...