Porting mock to Python 3

One of the nice new features in mock 0.7 is that it works with both Python 2 & 3. The mock module itself, even with all the freshly added docstrings, weighs in at less than 800 lines of code so compatibility is maintained with a single source base rather than the more recommended 2to3 approach. There are however about 1500 lines of test code that also need to work under Python 3; so whilst not a particularly difficult exercise it was not entirely trivial to get all the tests passing under Python 2.4, 2.5, 2.6, 2.7 and 3.2. Good tests make it much easier to have confidence that the port works. Attempting this without tests would be much more painful, even though it means there is more code to port.

The reason I chose the not-yet-released 3.2 to test with is because the mock test suite now uses unittest2. Python 3.2 has all the features from unittest2 and there is (unfortunately) not yet a Python 3.1 / 3.0 compatible version. Once 3.2 has been released I plan to continue developing unittest, mainly making it more easily extensible with a plugin mechanism, so there will need to be a Python 3 compatible unittest2 package.

mock is a testing library for mocking and monkey-patching. As such it does very little I/O so the deepest change in Python 3 - the bytes / Unicode change - has very little impact. There are still a host of minor syntactic and semantic changes that I needed to deal with. Here's a summary of the changes and compatibility hacks needed in mock and its tests.

  1. Python 3 has neither unicode nor basestring types. Mock uses unicode once (in the new magic method support) and basestring several times (there are a few APIs that can take either a string or some other object).

    The fix for these is easy:

    try:
        unicode
    except NameError:
        # Python 3
        basestring = unicode = str
    
  2. Python 3 has no long type; all integers are long. This is again needed for magic method support (in Python 2 __long__ needs to return a long integer) and the fix is also easy:

    try:
        long
    except NameError:
        # Python 3
        long = int
    
  3. There are a couple of places where we need to vary behaviour depending on whether we are running under Python 2 or Python 3. To detect these we check at the start:

    inPy3k = sys.version_info[0] == 3
    
  4. Copying / checking function attributes. Some function attribute names changed (for the better in Python 3).

    # checking for an instance method with im_self
    if not inPy3k:
        im_self = getattr(func, 'im_self', None)
    else:
        im_self = getattr(func, '__self__', None)
    
    ...
    
    # copying function attributes
    if not inPy3k:
        funcopy.func_defaults = func.func_defaults
    else:
        funcopy.__defaults__ = func.__defaults__
        funcopy.__kwdefaults__ = func.__kwdefaults__
    
  5. In Python 2 if we want to check for class objects we need to check for old style classes as well as new style classes. In Python 3 we only have new style classes.

    class OldStyleClass:
        pass
    ClassType = type(OldStyleClass)
    
    if inPy3k:
        class_types = type
    else:
        class_types = (type, ClassType)
    
    ...
    
    if isinstance(some_object, class_types):
        ...
    
  6. The magic methods available in Python 3 is different. The magic method support in mock has to explicitly support methods, so the lists are varied depending on which version of Python is being used.

    In Python 2 but not 3: __unicode__, __long__, __nonzero__, __oct__, __hex__

    In Python 3 but not 2: __bool__, __next__

    It was an interesting discovery (for me) to find that the rarely used __oct__ and __hex__ methods are now implemented using __index__, which is much more sensible.

    As well as this, the representation of zero in octal has changed in Python 3:

    if not inPy3k:
        self.assertEqual(oct(mock), '0')
    else:
        self.assertEqual(oct(mock), '0o0')
    

    A bunch of deprecated methods like __cmp__, __coerce__, __getslice__, __setslice__ and friends have gone for good in Python 3.

    The magic method changes affects our tests. When testing bool(mock) behaviour we have to do it differently depending on Python version:

    nonzero = lambda s: False
    if not inPy3k:
        m.__nonzero__ = nonzero
    else:
        m.__bool__ = nonzero
    
    self.assertFalse(bool(m))
    
  7. apply is missing in Python 3. This is used in tests to automatically execute decorated functions when testing decorators. There is an implementation of this which is imported when needed. Notice that when apply is available it is assigned to a local variable so we can import it from the support module under all versions of Python:

    try:
        # need to turn it into a local variable or we can't
        # import it from here under Python 2
        apply = apply
    except NameError:
        # no apply in Python 3
        def apply(f, *args, **kw):
            return f(*args, **kw)
    
  8. The name of the __builtins__ module changes to builtins in Python 3. We access it with a string anyway:

    if not inPy3k:
        builtin_string = '__builtin__'
    else:
        builtin_string = 'builtins'
    
  9. The Unicode literal syntax is invalid in Python 3. The specific tests for Unicode are skipped anyway under Python 3 as they are irrelevant (yay!), but Unicode strings used in those tests under Python 2 have to be created by calling unicode:

    @unittest2.skipIf(inPy3k, "no unicode in Python 3")
    def testUnicode(self):
        mock = Mock()
        self.assertEqual(unicode(mock), unicode(str(mock)))
    
        mock.__unicode__ = lambda s: unicode('foo')
        self.assertEqual(unicode(mock), unicode('foo'))
    

    Similarly Python 2 long integers need to be created with long(value) rather than the literal syntax.

  10. Comparing heterogeneous types with anything other than equality / inequality is invalid by default in Python 3. When testing the default behaviour of the comparison methods (to demonstrate the behaviour has changed when providing explicit comparison methods) we skip that part of the test under Python 3:

    def testComparison(self):
        if not inPy3k:
            # incomparable in Python 3
            self. assertEqual(Mock() < 3, object() < 3)
            self. assertEqual(Mock() > 3, object() > 3)
            self. assertEqual(Mock() <= 3, object() <= 3)
            self. assertEqual(Mock() >= 3, object() >= 3)
    
        mock = Mock()
        def comp(s, o):
            return True
        mock.__lt__ = mock.__gt__ = mock.__le__ = mock.__ge__ = comp
        self. assertTrue(mock < 3)
        self. assertTrue(mock > 3)
        self. assertTrue(mock <= 3)
        self. assertTrue(mock >= 3)
    
  11. from module import * is invalid syntax in Python 3 inside the body of a function or a method. This makes it harder to test the __all__ of a module inside a test method. What you can do is use exec which emulates a module scope and although the syntax has changed (from keyword to function in Python 3) you can do it in a cross-version way:

    def test_all(self):
     # if __all__ is badly defined then import * will raise an error
     # We have to exec it because you can't import * inside a method
     # in Python 3
     exec("from mock import *")
    

That's about it. As you can see, supporting Python 2 and 3 in mock is more a series of compatibility hacks than any deep changes. If you write code that does extensive I/O, as most applications do, then expect it to be more work and consider using 2 to 3 or even separate codebases.

In other libraries I've done this with I've used a couple of other tricks not mentioned here. Python 3 exception handling has different syntax from Python 2. I explored part of this in a blog entry: Exception Handling Code for Python 2 and 3.

Reraising exceptions is also different. I have to deal with this in the contextdecorator module. Here a _reraise function is defined using exec, with different implementations for Python 2 & 3. exec has to be used because the Python 2 syntax is invalid under Python 3:

if sys.version_info >= (3,0):
    exec ("""
def _reraise(cls, val, tb):
    raise val
""")
else:
    exec ("""
def _reraise(cls, val, tb):
    raise cls, val, tb
""")

Where the exception is re-raised _reraise is called with the values from sys.exc_info(). This preserves the traceback under both Python 2 & 3:

exc = None
try:
    result = f(*args, **kw)
except Exception:
    exc = sys.exc_info()

...

if exc:
    _reraise(*exc)

In the same code we need to advance a generator. From Python 2 to Python 3 the standard way of advancing a generator changed from gen.next() to next(gen). The next builtin function is present in Python 2.6 & 2.7, so we can just provide a version for Python 2.4 & 2.5:

try:
    next
except NameError:
    # for Python 2.4 & 2.5
    def next(gen):
        return gen.next()

A couple more useful resources on porting to Python 3:

For buying techie books, science fiction, computer hardware or the latest gadgets: visit The Voidspace Amazon Store.

Hosted by Webfaction

Return to Top

Page rendered with rest2web the Site Builder

Last edited Tue Aug 2 00:51:34 2011.

Counter...