Mocking, Patching, Stubbing: all that Stuff

What Approach to Mocking in Tests do You Take?

Mocking in Action.

 

 

Testing and Mocking

I've been thinking about mocking. We do a lot of testing in Resolver. Resolver is a client application, but we also have a server product. We do unit testing, functional testing and some through the web testing with Selenium.

This doesn't make me an expert on testing of course, you need experience on a wide range of projects in lots of different situations before you can claim that. Luckily Resolver has people with a lot more testing experience than me. Coincidentally our latest hire, Glenn Jones, has just come from a job programming in C++, Java and a VB like scripting languages for a cellphone testing platform. That's a very different kind of testing.

Doing a lot of unit testing with a dynamic language, we end up doing a lot of 'mocking'. That is creating small objects with a specific interface to mimic a production object. We mock to decouple tests from other parts of our production code, to test with .NET objects that may be hard to construct [1], and to be able to make assertions about how the unit under test interacts with other objects.

To give you a concrete example, suppose we have a method that recives an object and closes it. We want to be able to tell whether our method (the closer) has called close on the object we pass in. We might write a unit test like this:

def test_closer_closes_something(self):
    class MockSomething(object):
        closed = False

        def close(self):
            self.closed = True


    real = ProductionClass()
    mock = MockSomething()

    real.closer(mock)

    self.assertTrue(mock.closed, "closer didn't close something")

If we have a method that calls another method on the same class, then we might patch out that method with something that will let us know that it has been called. We might write a unit test like this:

def test_method_calls_something(self):
    called = []
    def patch():
        called.append(True)

    real = ProductionClass()
    real.something = patch

    real.method()

    self.assertEquals(called, [True], "method didn't call something")

Because our patch function can't assign to a variable in the scope of the test method, we have to use the 'list trick' to track that something was called. This is a little ugly, and instead we could patch out something with a callable object (a Listener perhaps) that will track if it has been called. The test would then look like:

def test_method_calls_something(self):
    real = ProductionClass()
    real.something = Listener()

    real.method()

    self.assertTrue(real.something.called, "method didn't call something")

So the listener class we used above is really a 'mock method'. Martin Fowler would prefer that I didn't call these objects mocks, but instead call them stubs:

Mocking Libraries

Respectfully I disagree with him, MockSomething is a mock object dammit! Instead Martin describes a mocking pattern which is implemented in various mocking libraries for Java, Python and probably other languages. The pattern is that you first record your expectations on the mock, then you replay and verify that the expectations were met.

A new Python library called mocker is a classic implementation of this. An example from the docs:

>>> greeting = Greeting()

>>> mock = mocker.proxy(greeting)
>>> mock.hello("Jeff")
>>> mocker.result("Hello Jeff!")

>>> mock.hello("Jim")
>>> mocker.passthrough()

>>> mocker.replay()

>>> mock.hello("Jim")
'Hi Jim!'

>>> mock.hello("Jeff")
'Hello Jeff!'

The object under test is the Greeting class. The mock wraps the greeting class, you record a series of actions and then call replay. If the subsequent actions don't meet the actions you recorded then you will get an AssertionError.

At Resolver we don't use a mocking library. There are two reasons for this. The first is that mocks (I mean stubs...) are so damn easy to write! The second is that the general consensus amongst the Resolver developers is that the 'record -> replay' pattern is backwards and less intuitive than the more 'normal' action and assertion pattern.

In fact, what I call the 'action -> assert' pattern is more properly 'setup -> action -> assertion'. For any one unit (function / method / class), we will probaby have several tests - each testing a specific aspect of behavior. For example these might be testing normal behaviour, corner cases, and error conditions (etc). Each one will be a separate test.

So for each test we setup the initial conditions, perform the actions and then make an assertion about the behaviour. The pattern described by Martin merges the initial setup with 'defining the expectations' - which I think makes it less clear what you are actually testing. It is nice to be able to say - for these conditions we expect this action to behave in this way. The 'setup - > action -> assertions' pattern makes this all explicit, whereas 'expectations -> verification' makes it less clear (in my opinion).

Additionally let's look at a simple testing example that doesn't require any mocks. You want to test that calling set_property with an argument sets the right attribute:

def test_set_property(self):
    real = ProductionClass()

    value = object()
    real.set_property(value)

    self.assertEquals(real.property, value, "set_property didn't work")

A lot of your tests are going to look like this. You perform an action and then assert that the action did the right thing. If we use a mocking library for some of our tests then we will end up with a mix of testing patterns. This aside, I still find the 'replay' pattern less readable - and test readability is important. (In Behavior Driven Development, an offshoot of Test Driven Development, your tests are a specification of your code.)

A further issue is that when recording expectations you will have to (usually) record every step. For individual tests, it may only be one detail that you are interested in. You should only make assertions that are specific to the test rather than having to record every step.

So at Resolver we are still writing an awful lot of trivial mocks. Is there a simple mocking library that could help us?

minimock comes the closest. It follows the right pattern, but is designed for use with doctest, which we don't use.

The Mock Module

As is the way with Python, it turns out that you can create the start of something useful in less that fifty lines of code. I've created a new mocking library (just what the world needs - hey at least it isn't a web framework). It doesn't make the assertions for you, but records how it is used: so that you can make assertions afterwards.

The core class is Mock. It is callable and records arguments it is called with. It provides __getattr__, so that accessing arbitrary attribtues / methods will also return a Mock object (the same object every time for the same attribute). Calls to methods are recorded on the parent object and the child. __setattr__ isn't provided, so that you can set arbitrary attributes if you need to.

Lets look at a couple of examples:

Mock Patching Methods

Mock is callable. If it is called then it sets a called attribute to True. This example tests that calling method results in a call to something:

def test_method_calls_something(self):
    real = ProductionClass()
    real.something = Mock()

    real.method()

    self.assertTrue(real.something.called, "method didn't call something")

If you want to catch the arguments then there is other information exposed:

def test_method_calls_something(self):
    real = ProductionClass()
    real.something = Mock()

    real.method()

    self.assertEquals(real.something.call_count, 1, "something called incorrect number of times")

    args = ()
    keywargs = {}
    self.assertEquals(real.something.call_args, (args, keywargs), "something called with incorrect arguments")
    self.assertEquals(real.something.call_args_list, [(args, keywargs)],
                      "something called with incorrect arguments")

Checking call_args_list tests how many times the mock was called, and the arguments for each call, in a single assertion.

Mock for Method Calls on an Object

def test_closer_closes_something(self):
    real = ProductionClass()
    mock = Mock()

    real.closer(mock)

    self.assertTrue(mock.close.called, "closer didn't close something")

We don't have to do any work to provide the 'close' method on our mock. Accessing close ceates it. So, if 'close' hasn't already been called then accessing it in the test will create it - but called will be False.

As close is a mock object is has all the attributes from the previous example.

Limiting Available Methods

The disadvantage of the approach above is that all method access creates a new mock. This means that you can't tell if any methods were called that shouldn't have been. There are two ways round this. The first is by restricting the methods available on your mock.

def test_closer_closes_something(self):
    real = ProductionClass()
    methods = ['close']
    mock = Mock(methods) # methods=methods is equivalent

    real.closer(mock)

    self.assertTrue(mock.close.called, "closer didn't close something")

If closer calls any methods on mock other than close, then an AttributeError will be raised.

Tracking all Method Calls

An alternative way to verify that only the expected methods have been accessed is to use the method_calls attribute of the mock. This records all calls to child attributes of the mock - and also to their children.

This is useful if you have a mock where you expect an attribtue method to be called. You could access the attribute directly, but method_calls provides a convenient way of looking at all method calls:

>>> mock = Mock()
>>> mock.method()
>>> mock.Property.method(10, x=53)
>>> mock.method_calls
[('method', (), {}), ('Property.method', (10,), {'x': 53})]
>>>

If you make an assertion about method_calls and any unexpected methods have been called, then the assertion will fail.

Limitations

There are various disadvantages and things that this class doesn't (yet) do.

  • It doesn't wrap objects. If you need to pass calls through to the real object and record them then you can't yet do that.
  • You can specify return values, but they are fixed values. You may want to provide a factory function to provide a sequence of return values for example.
  • Mock has several attributes. This makes it unsuitable for mocking objects that use these attribute names. A way round this would be to provide start and stop (or similar) methods that hide these attributes when needed.
  • It doesn't provide any facilities for patching on classes (rather than instances), modules or built-ins. These are all patching that has to be undone after the test is complete because otherwise the patch would persist. At Resolver we use a decorator pattern for this kind of patching which is very useful. With Resolver's permission I will make these available too.
[1]Like the multitude of events that derive from EventArgs.

For buying techie books, science fiction, computer hardware or the latest gadgets: visit The Voidspace Amazon Store.

Hosted by Webfaction

Return to Top

Page rendered with rest2web the Site Builder

Last edited Tue Aug 2 00:51:34 2011.

Counter...