Python Programming, news on the Voidspace Python Projects and all things techie.
Singletons via a Metaclass
We've been interviewing interns over the last two weeks (applications closed now sorry). It's exhausting, for them and for us.
We've had some great applicants again this year, and a couple of them came via this blog which is nice. Several of them are also from Poland; that country seems to be churning out some impressive programmers at the moment.
One of the interviewees is not studying computer science, but is both passionate about programming and fiercely intelligent. When I asked him for an example implementation of a Singleton he taught me a very neat trick .
One good way is to store the instance as a class attribute, and override __new__ so that subsequent instantiation attempts return the instance you have stored as a class attribute. The disadvantage of this approach is that every time __new__ is called, so is __init__, so you need to put short-circuiting in there as well .
His approach is to override __call__ on a metaclass. Instantiating (calling the class), results in a call to the metaclass.
if cls.instance is None:
cls.instance = type.__call__(cls)
__metaclass__ = SingletonMetaclass
instance = None
>>> s = Singleton()
>>> s2 = Singleton()
>>> s2 is s
This isn't threadsafe in this form of course. Nice trick though .
After all this, I've never actually needed a singleton . In Resolver we have one class that needs to behave like a singleton. It binds to a UDP port, so it should only be created once. To get round this we have a classmethod to activate it, which only creates an instance the first time it is called. The activate method can happily be called whenever this might need to be active, without the caller having to care whether or not it has already been called.
|||Which he has on a blog in German somewhere. When I get to work I'll retrieve the link...|
|||A couple of improvements thanks to Jack Diederich. There is also a simpler hacky implementation which creates an instance and then effectively overrides the class name with the instance. You can still get at the class via the __class__ attribute of the instance though.|
|||But you could probably still call __new__ directly on the class. You can't override __new__ on the metaclass to avoid this, but you could simply raise an exception to avoid this kind of cheating.|
|||The wikipedia article says that the Singleton pattern is sometimes seen as an anti-pattern because it smells a great deal like a global variable.|
Mocks Aren't (Just) Stubs
Ah... posts like this are easy.
This is an entry by another 'blogging colleague' of mine, Jonathan.
It's a closer look at the Martin Fowler article mock objects aren't stubs, and how it might apply to our work at Resolver. Like him I'm not sure that the sort of replay patterns (and behaviour verification) that he describes would be particularly useful to us.
I think there is an initial abstraction that we can make, for the automating of creating our stubs and building in state testing. With a very simple class to start with, we could then see what further abstractions from our common testing patterns  come naturally from that.
|||We already have a simple but useful Listener class. This is used for 'methods' on our stubs, and will tell us if, how many times, and with what arguments it was called. We can also set a return value. Building this functionality in would be easy.|
There is a new (and long overdue) release of the Pythonutils module. This is version 0.3.0.
What is Pythonutils?
Pythonutils is a collection of general utility modules that simplify common programming tasks in Python.
The modules included are :
- ConfigObj 4.4.0 - Easy config file reading/writing
- validate 0.2.3 - Validation and type conversion system
- StandOut 3.0.0 - Simple logging and output control object
- pathutils 0.2.5 - For working with paths and files
- cgiutils 0.3.5 - CGI helpers
- urlpath 0.1.0 - Functions for handling URLs
- odict 0.2.2 - Ordered Dictionary Class
For more details, visit the Pythonutils Homepage.
What is New in 0.3.0?
Several of the modules have been updated. The major changes are:
- Removed the listquote module
- ConfigObj updated to 4.4.0
- StandOut updated to 3.0.0 (Not backwards compatible, but much improved)
There is now a sourceforge mirror page for pythonutils: Pythonutils on Sourceforge.
IronPython Benchmarks on .NET (and Comparing .NET and Mono)
Ok, so I should have realised this, but I didn't. The benchmarks for IronPython 1.1 that Seo published, were actually run on Mono.
Jim Hugunin has just posted the same tests performed on Windows Vista and .NET: IronPython 1.1 and Python 2.5 on .NET.
This paints a slightly different picture, IronPython performs much better although still slightly slower than CPython overall. Comparing the results from Mono and .NET would be a good way for the Mono project to highlight areas that need work.
The salient points from IronPython on .NET compared to CPython 2.5:
- Function calls and recursion on IronPython are a lot faster than CPython (3 or 4 times faster)
- Arithmetic is still faster with IronPython
- For loops, nested for loops, and if-then-else are quite a bit faster than CPython
- Creating instances (old style) is faster with IronPython, but new style instances are still slower
- List slicing is still slower, but again faster than Mono
- String concatenation is still slower than CPython, but the difference is greatly reduced
- There is an overhead in IronPython for using byte-strings rather than unicode
- The weird TryExcept and TryRaiseExcept anomaly still exists (and in fact is more pronounced, but see Jim's comments at the end)
- Using basic types (like dictionaries, lists and tuples) is slower than CPython, but by much less than on Mono
There don't seem to be any tests in pybench for sets, I'd be interested to compare performance between CPython and IronPython. Overall, it seems to me that if you optimise for IronPython then it ought to be possible to write code that runs faster than on CPython. Certainly the differences are not big enough to worry about. Don't forget that if you use threads on a multi-core processor, then IronPython (unencumbered by the GIL) will almost certainly whoop CPython for performance.
Jim's take on the results is:
However, the story is already quite interesting. Out of the 51 tests in pybench, CPython is more than 2x faster on 10 of the tests and IronPython is more than 2x faster on 9 of the tests. Depending on what your code does, either implementation could run faster.
The most interesting cases to me are the 5 tests where CPython is more than 3x faster than IronPython and the other 5 tests where IronPython is more than 3x faster than CPython. CPython's strongest performance is in dictionaries with integer and string keys, list slicing, small tuples and code that actually throws and catches exceptions. IronPython's strongest performance is in calling builtin functions, if/then/else blocks, calling python functions, deep recursion, and try/except blocks that don't actually catch an exception.
The places where IronPython is performing strongest are where it can use the .NET code generation and JIT optimization most effectively. The places where it is slowest are mainly in runtime library implementation of core datatypes. CPython's list and dictionary datatypes are written in C and have been hand-tuned for Python-style workloads over the past 10 years. IronPython's datatypes are much newer and clearly need more tuning as the implementation matures.
The only two tests that really stand out as showing a deep performance issue are the two exception handling ones. These are the only tests where either implementation is more than 4x faster than the other. IronPython is 10x faster on the try/catch without an exception and CPython is 30x faster when an exception is actually raised. This is a deliberate design decision within .NET to make code that doesn't throw exceptions run faster - even if that means slowing down code that does throw exceptions. I'm fairly confident this was the right decision - and even remember the day long ago when Guido was discussing Python's exception system and explained that he'd accept almost any slow-down to the exceptional case in return for removing a single instruction from the non-exceptional path.
Seo has made a further post, comparing the .NET results with Mono.
.NET runs lots of tests faster than Mono. The tests that .NET runs more than three times faster than Mono are:
Tests that Mono runs faster than .NET are:
Whilst we're talking about Mono, version 1.2.4 has been released. Improvements include performance improvements for ASP.NET, some work started on C# 3.0 and masses of improvements in Windows Forms 2.0 support.
Using the Python Standard Library and help with IronPython
The question of how to use the Python standard library with IronPython still pops up occasionally on the Mailing List. This is actually very straightforward.
The simplest way, is to install Python 2.4 and setup the IRONPYTHONPATH environment variable to point to the standard library (usually c:\Python24\Lib).
This has an unfortunate side effect that I have only just discovered. IronPython imports site.py, which overrides the builtin help and breaks it in the process. The standard Python one doesn't work with IronPython.
The solution is simple. Put another directory into IRONPYTHONPATH before the Python standard library, and put an empty file called site.py in this directory. help is no longer overwritten and is restored to its former glory.
IronPython, Benchmarks (Mono) and Performance
Well, IronPython 1.1 has been released.
The major new features, at least as far as I'm concerned, are the implementation of several new modules (array, SHA, MD5, and select), and that you can now load pre-compiled (Python) modules. This means that you can make binary only distributions, and loading pre-compiled modules is slightly faster. You save on the parse and compile phase of loading Python modules.
These benchmarks were performed on Mono. Unsurprisingly, IronPython runs much faster on .NET. Another blog entry looks at the results of the same test run on Windows Vista and .NET.
The results are revealing. In general IronPython is slower than CPython, but there are some interesting outliers.
- IronPython is a bit faster for function calls and recursion (probably a result of not having Python stack frames).
- IronPython is quite a bit faster for most arithmetic operations.
- IronPython is massively faster for Try...Except, but massively slower for Try...Raise...Except.
- IronPython is slower for creating new instances (instances of new style classes presumably?), this matches our experience.
- IronPython is a lot slower for working with lists, tuples and slicing.
- IronPython is massively slower for string concatenation.
String concatenation is archetypally inefficient on any platform that has immutable strings. Since Python 2.4, CPython has had an 'under-the-hood' optimization to make it less inefficient. Presumably Mono doesn't have the same optimization.
Don't get the impression that IronPython is too slow to use. At resolver we are developing a large, performance sensitive, application with IronPython. Generally IronPython is fine for normal use, but we have gone through several phases of optimization for our 'engine'. Every time we have managed to get the performance improvements we aimed for, by optimizing our Python code. We have never (so far anyway...) had to move code into C# to improve performance.
If we ever do consider such a move we would much rather use RPython than C#. Currently this isn't possible, but Antonio Cuni has had a PHD accepted to continue work on the GenCLI back-end for PyPy that he created. This is an RPython to .NET compiler, and he has produced some very readable slides about his plans:
In the meantime I have a book to work on. I've just completed chapter six of IronPython in Action. I'm now rewriting parts of chapter one before it goes out for the third of a book review. This is an important milestone, if Manning are going to pull out on me it will be now.
Once it has come back from review, it will probably be available under the Manning Early Access Program. It's just a shame that the pound is so strong at the moment...
PyCon UK and London Meetup
Exciting Python-type-things are afoot in the UK. One near, and one far.
The first UK community conference for Python will be held in Birmingham on Saturday 8th & Sunday 9th of September 2007.
PyCon UK is the young upstart counterpart to PyCon in the US. There will be talks, lightning talks, tutorials, sprints and social get-togethers.
The conference is aimed at all members of the Python community, whether you're just starting or you dream in bytecode.
I'll be there, and I'll probably put together a proposal for some kind of IronPython talk. I'd be very surprised if Resolver wasn't available for demo by then (in fact there should be news on that front a lot sooner)... It would be nice to have a talk on Test Driven Development, but Andrzej is in Poland that weekend. Maybe I'll have to see if I can twist the arms of my other colleagues...
This work is licensed under a Creative Commons Attribution-Share Alike 2.0 License.