Python Programming, news on the Voidspace Python Projects and all things techie.

Total Ordering: Class Decorator for Filling in Rich Comparison Methods When Only One is Implemented

emoticon:cat Raymond Hettinger managed to provide several of the highlights at PyCon UK. During a couple of his talks he extolled the virtues of class decorators which are new to Python 2.6 & 3.0. He also talked about how Python 3 fixes the Python 2.X wart of objects of arbitrary types being comparable and sortable.

In Python 3 comparing arbitrary types throws an error. Raymond challenged the audience to come up with a class decorator that takes a class with any one of the rich comparison methods (except for __eq__ or __ne__) and automatically fill in all the others.

Christian Muirhead and Menno Smits took him up on the challenge and created one - and its very cool. Christian rarely blogs so he gave it to me. I've posted it to the Python Cookbook where you can post comments, corrections and suggestions. Smile

UPDATE: I've modified the recipe to provide two decorators and remove the need to pass in an argument. I've modified the description below to reflect this change.

total_ordering and force_total_ordering are class decorators for Python 2.6 & Python 3.

They provides all the rich comparison methods on a class by defining any one of '__lt__', '__gt__', '__le__', '__ge__'.

They assumes that objects will only be compared with objects of the same type (true for Python 3 in ordering / sorting operations).

total_ordering fills in all unimplemented rich comparison methods, assuming at least one is implemented. __lt__ is taken as the base comparison method on which the others are built, but if that is not available it will be constructed from the first one.

force_total_ordering does the same, but having taken a comparison method as the base it fills in all the others - this overwrites additional comparison methods that may be implemented, guaranteeing consistent comparison semantics.

from total_ordering import total_ordering

class Something(object):
    def __init__(self, value):
        self.value = value
    def __lt__(self, other):
        return self.value < other.value

It also works with Python 2.5, but you need to do the wrapping yourself:

from total_ordering import total_ordering

class Something(object):
    def __init__(self, value):
        self.value = value
    def __lt__(self, other):
        return self.value < other.value

Something = total_ordering(Something)

One of the nice things about class decorators is that they can be used for things that you would otherwise use a metaclass for. It would be easy to modify this recipe to make it a metaclass for Python 2.X where X < 6.

I did actually have to tinker with the code that Christian sent to me. It worked fine with Python 2.5, but blew up horribly with Python 2.6 and Python 3. After some digging around I worked out that the reason it was failing is that it was checking if a class defines a comparison method with hasattr(cls, method_name). This works fine with Python 2.5, but in Python 2.6 / 3.0 object provides explicit default implementations of the rich comparison methods. The test was always returning True...

As far as I can tell (and I may be missing something obvious), other than by parsing the repr, there is no way to tell if a method fetched from a class is really the implementation from object:

Python 2.6 (trunk:66714:66715M, Oct  1 2008, 18:36:04)
[GCC 4.0.1 (Apple Computer, Inc. build 5370)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> class X(object): pass
>>> X.__lt__
<method-wrapper '__lt__' of type object at 0x5baea0>
>>> X.__lt__ == object.__lt__

(This actually returns True for Python 3 which is nice.)

The answer took me longer than it should have done to track down. The recipe includes a _has_method function that takes a class and searches the method resolution order (the class plus its base classes) to see if the class defines or inherits a method (so long as the method isn't inherited from object).

import sys as _sys

if _sys.version_info[0] == 3:
    def _has_method(cls, name):
        for B in cls.__mro__:
            if B is object:
            if name in B.__dict__:
                return True
        return False
    def _has_method(cls, name):
        for B in cls.mro():
            if B is object:
            if name in B.__dict__:
                return True
        return False

Like this post? Digg it or it.

Posted by Fuzzyman on 2008-10-05 21:48:44 | |

Categories: , Tags: , ,

Dynamic Languages MVP, Blog Template, Conferences, Concurrency and Other Stuff

emoticon:apple Last week I received an email from Microsoft:

Congratulations! We are pleased to present you with the 2009 Microsoft MVP Award! This award is given to exceptional technical community leaders who actively share their high quality, real world expertise with others.

MVP stands for 'Most Valued Professional' and is an award given to community contributors. I'm the first MVP for the Microsoft dynamic languages team - although there isn't a dynamic languages division for MVPs so technically I'm a Visual C# (*) MVP. That asterisk is very important! Smile

It wasn't all good news last week though. Menno Smits has been a colleague with us at Resolver Systems for almost a year, but he has just passed away. Well, he received an 'offer too good to refuse' and will now be working with BATS Trading in London. They are a trading platform largely implemented in Python and have been so successful in the states that they have become a stock exchange in their own right. Now they are setting up shop in Europe. Congratulations to Menno and we wish him all the best for the future.

If you're anything more than an occasional visitor to this blog you will probably have noticed the change in template. Largely on the insistence of Christian Muirhead (my colleague and co-author of IronPython in Action) I've chopped out almost everything above the entries - which means less to scroll down through and less Javascript should mean faster loading pages. I'm still intending to do (or at least get Justin to do) a full site redesign once 'the book is finished'. This change will have to do in the mean time. Smile

PyCon UK is done so it's time to look at conferences for next year. I've submitted talks for PyCon 2009 (in Chicago) and ACCU 2009 (in Oxford).

ACCU is a UK community conference (I went for the first time last year and it was great fun). It has talks on a wide variety of topics - last year there were very few Python talks but several on Haskell and Erlang. In the past there have been many more Python talks, but last year a significant proportion of those attending were .NET or C++ developers.

I've proposed two talks (I'm not sure which they will prefer):

  • Creating Rich Internet Applications with IronPython & Silverlight 2
  • Embedding IronPython and the Dynamic Language Runtime in .NET Applications

I've put forward a talk and a tutorial proposal for PyCon US. My talk was submission number 1 (!) and is on Functional Testing of Desktop Applications. It's a relatively niche subject (not much focus in the Python community on creating desktop applications) - but still an important one, so it will be interesting to see if it is accepted.

The tutorial is on Developing with IronPython. It is based on the tutorial that Menno, Christian and I gave at PyCon UK - except this time it is Jonathan Hartley and I who will be giving it. We had a great time giving the tutorial in Birmingham and learned a great deal doing it. We had seventeen people attending (out of a total of eighty attending the tutorials - so nearly a quarter which isn't bad), and I think that the attendees enjoyed it. We should have got people using IronPython earlier in the tutorial, and we had too much practical stuff - meaning we spent too long in the user interface part of the tutorial. I've been revising the handout notes (I'll post them up here sometime as they are a great introduction to IronPython) based on what we learned.

Interesting advice for new programmers from Anders Hejlsberg (the architect of the C# programming language) in an interview with Computer World:

Go look at dynamic languages and meta-programming: those are really interesting concepts. Once you get an understanding of these different kinds of programming and the philosophies that underlie them, you can get a much more coherent picture of what's going on and the different styles of programming that might be more appropriate for you with what you're doing right now.

Anyone programming today should check out functional programming and meta-programming as they are very important trends going forward.

Before his passing away Menno ported part of his website to rest2web. He also posted a new article on his experiences installing Linux on his shiny new Sony Vaio VGN-BZ11XN notebook. rest2web is a tool for maintaining websites (static HTML) in reStructured Text. It is particularly good for programmers as the templating system is straight Python without requiring you to learn a custom templating language.

I use rest2web to maintain pretty much all the websites I run [1], but haven't done any work on rest2web itself beyond maintenance for the last two years. It simply does everything I need it to. Despite this new sites built on rest2web pop-up regularly. Another recent one is by Andrew Straw. I can always tell a rest2web site, because even with a custom template most people leave in the Page last modified ... timestamp from my default template. This uses the <% modtime %> templating variable. Unfortunately it doesn't play well with a Subversion bug if you keep your website sources under source code control. Andrew has overcome this bug by using the SVN commit time instead and posted notes about how he did it under About this Website.

Final piece of news; both Ted Leung and Mark Shuttleworth talked about the future of Python in their keynote speeches at PyCon UK. They both noted that concurrency was becoming more important and is one of the areas where CPython is lacking because of its poor support for threads. Neither IronPython nor Jython have a Global Interpreter Lock (the GIL), so these are both platforms where threads can be used for concurrency with Python.

Michael Sparks is one of the organizers of PyCon UK, and also the author of Kamaelia, a generator based concurrency library for Python. Kamaelia is capable enough to stream video and audio, but last time Michael tried to use it with IronPython a few bugs (in IronPython) prevented it from working. A lot has changed since then, and with the latest version of IronPython 2 most of Kamaelia 'just works'.

Although Kamaelia presents a very simple API for concurrency oriented programming (usually no need to explicitly work with threads or locking), it does use threads in several key parts under the hood. This means it hits limitations in CPython (Michael Sparks' words not mine), and IronPython doesn't suffer from the same restrictions. So far Michael is impressed with IronPython...

[1]This one, IronPython in Action, The Other Delia and Resolver Hacks. Exceptions are The IronPython Cookbook which is a MediaWiki wiki and IronPython-URLs Blog which is a blogspot blog (originally started by Mark Rees).

Hosted by Webfaction