The Voidspace Techie Blog

Ironclad 0.4 Released: Use CPython Extensions from IronPython

William Reade of Resolver Systems has announced a new release of our Ironclad project.

• Ironclad Home Page
• Download Binaries and Source for Ironclad
• Ironclad Mailing List

Ironclad is a project to enable you to use CPython extensions implemented in C seamlessly from IronPython. This basically involves re-implementing the Python C API in C# (along with reference counting and garbage collection strategies that span the native code in the C extension and the managed code that lives in the .NET Virtual Machine).

This release comes a bit sooner than expected as William has got the import hook working. This means that after executing import ironclad in IronPython (2), you can import any C extensions on your path just as you would from CPython!

Here's the details from the announcement:

Ironclad v0.4 has now been released. Properties and (some) members on unmanaged types should now work, and it's now much easier to use -- by starting ipy.exe in Ironclad's build directory, you should now be able to 'import ironclad' and then attempt to import CPython extensions as you would in CPython. Note that they aren't likely to work very well yet, if at all, because so much of the CPython API is still unimplemented.

Our intention is that the next release, v0.5, should allow people to import numpy and do something incredibly trivial with it. It may be some time before we understand numpy well enough to achieve this, so we aren't in a position to commit to a release date; but we'll provide updates as we progress.

---

Resolver One: Major New Release, Version 1.1

This release represents four months of work and includes performance improvements, new features and bug fixes for Resolver One.

Resolver One is an IronPython based spreadsheet development environment (fully programmable, object-oriented spreadsheet). It is designed for creating business applications using the familiar spreadsheet interface. Resolver One is free to use for non-commercial and open source purposes, and very cheap for commercial uses.

• Resolver One 1.1 New Features (3 Minute screencast)
• Resolver One Free Download
• News Announcement
• Full CHANGELOG
• Resolver Hacks Website: Fun & Interesting things to do with Resolver One

The new features include:

• Significant improvements to performance and memory usage.
• Cutting and pasting is now more "spreadsheet-like".
• For the financial edition, we've added Thomson Dataworks Enterprise connectivity and a number of great enhancements to Bloomberg access.
• Better coverage of standard spreadsheet functions.
• Auto-indent in the code editor.
• Comments in cells.
• Many bugfixes (including several different obscure ways of crashing Resolver One).
• Menu and toolbar item for unpacking arrays into selection (and repacking). You can use this to unpack any iterable.
• You can provide a 'formatter function' for cells (or rows, cols or whole worksheets) which is used to format the displayed value in the cell. Very nice!
• New 'Color' trait for the colour of text in cells. (Boring but needs to be there.)

The full changelog is surprisingly long. It's nice to see that we haven't wasted the last four months, even though progress has felt slow at times. We have a huge list of different things we want to do with Resolver One, plus different directions we could take it in. It is good that we now have some customers using Resolver One and giving us valuable feedback about the most important things we need to work on.

Even since cutting this release we have been working on cool new features. One of my favourites is the ability to create and manipulate drop-down lists from code as well as from the user interface.

---

Ironclad 0.3 Released (Use CPython Extensions from IronPython)

William Reade of Resolver Systems has just announced the release of Ironclad version 0.3:

• Ironclad Home Page
• Download Binaries and Source for Ironclad
• Ironclad Mailing List

Ironclad is an open source project that aims to allow you to seamlessly use CPython C extensions from IronPython. It basically provides a reimplementation of the CPython API in C#, and maps the C extension (unmanaged code) to creating IronPython objects (managed objects) rather than CPython objects.

Ironclad targets IronPython 2 and Python 2.5.

The project is still in its infancy (only part of the C API is complete), but we have implemented sufficient for most of the CPython bz2 module to be used. Functions, types and their methods (including the file types [1]) can be used from this module. Class members and properties cannot yet be used and is the target of the next release (before we move onto getting Numpy working - the 'secret goal' of the project).

The major advancement in this release is that object lifetime is now handled basically correctly.

When you create a "managed object from an unmanaged type" (and we need a better way of saying that: when you use an object from a C extension basically), some 'unmanaged resources' (i.e. memory) are allocated.

Now, when the managed object is garbage collected this memory is correctly freed and the unmanaged destructor (tp_dealloc) for the type is called. This means that Ironclad doesn't leak [so much] memory and file descriptors owned by the unmanaged side are properly closed on garbage collection.

This is implemented with an interesting strategy. We basically maintain our own reference count for objects. When an object (a "managed object of unmanaged type") is created, we set its reference count to 1 (using a weak reference in the mapper so that the reference counting alone doesn't keep it alive on the managed side). Setting the reference count to 1 means that the destructor won't be called when the unmanaged side (the C extension) has finished with the object and is no longer holding any references to it.

The 'reference counting' for the unmanaged side, is simply a small block of memory storing an integer which corresponds to how many 'unmanaged references' there are to it. An unmanaged reference is not exactly the same as a managed reference (a possible cause of terminology confusion). This means that we are using real reference counting (in the same way as CPython) on the unmanaged side.

When there are no more references to it on the managed side (the IronPython world), the finalizer is called which checks the reference count in our mapping. If the reference count is more than 1 then we know that the extension itself still has a reference to the object and so the finalizer resurrects the object by storing a strong reference in the mapper.

We then regularly check (every time a managed object is finalized - but we may find a better strategy) if the reference count of any objects we have promoted to strong references have dropped back down to 1. If they have, we demote the strong reference back to a weak reference so that the object can be garbage collected.

In this way Ironclad mixes the reference counting that the C extensions use with the garbage collection of the .NET framework. The only downside is that if our "managed object of unmanaged type" itself has any references to managed objects, then their finalizers may have already been called at the point at which we resurrect it. Hopefully this is a pathological enough case that it won't bite us for a long time to come...

Currently Ironclad is only tested on Windows with .NET, but it deliberately uses the gcc compiler and a (theoretically) platform independent approach so that it can be ported to Mono with 'minimal' effort. Obviously the source distribution comes with full tests, and pretty good documentation.

[1]	Getting access to the 'real' file descriptor from a .NET stream is possible, but 'fun'.

---

Resolver One 1.1 Features Announced

We've now announced the new features in the forthcoming Resolver One 1.1 release:

We're now moving toward a feature freeze for Resolver One 1.1. Here's a foretaste of some of the changes you can expect to see:

• Major performance and memory usage improvements.
• Improved cut - when you cut and paste a block, formulae that refer to cells in the block are updated appropriately.
• Comments for cells.
• Simple auto-indentation in the code editor.
• For the financial edition, we've added Thomson Dataworks Enterprise connectivity and a number of great enhancements to Bloomberg access.
• ContentRows and ContentCols iterators on worksheets and cell ranges so that you don't have to keep checking for the header.
• The CopyRange function can now sort the data as it copies.
• HLOOKUP, TODAY, ISNUMBER, ISBLANK, NOW, and ABS functions
• And lots of minor tweaks and bugfixes.

Several of the bugfixes include fixing some of the obscure ways it was possible to crash Resolver One previously. Although most people would rarely have encountered them it is still nice to see them gone. One I'm particularly glad to see the back of, is that setting clipboard contents when Resolver One was running virtualised could fail - causing a harmless but annoying crash dialog (and failing to set the clipboard contents). Resolver One is now resilient against crashing in these circumstances and will retry setting the clipboard.

I've just returned from a fun trip to Galway for a community .NET event, DDD Ireland. I'll put together a blog entry on this later, but whilst there I had the chance to demo and talk about Resolver One to several of the delegates and speakers. Every time I do this I try to hone a concise description of exactly what Resolver One is.

The best I've come up with so far goes something like this...

Resolver One is a spreadsheet development platform for the rapid creation of business applications. It is an Object Oriented, programmable spreadsheet. Its key features are:

• The trivially easily mixing of code with spreadsheets
• You can put arbitrary objects in the grid
• Easy access to .NET and Python libraries
• You can export a spreadsheet as code

Some people immediately 'get' this (mainly those who have wrestled with complex spreadsheets already), but I think that for many developers Resolver One is the sort of program that is best understood by actually trying it.

Some of the other advantages that Resolver One has over a more 'traditional' spreadsheet are:

• A rich formula language based on Python expressions (list comprehensions and generator expressions in formulae for example)
• A single, unified, spreadsheet object model - with powerful features like indexing worksheets and cellranges by names used in headers (makes for more readable formulae)
• With a stored procedure databases can notify Resolver One of changes (typically on update/delete/insert), giving you a live auto-updating view into databases
• Because it uses Python you can define new numerical types and use them in your spreadsheet
• With database worksheets, or from user defined code, working with databases becomes both easy and powerful
• Import and export spreadsheets to and from Microsoft Excel format [1]

There is also a web server for serving Resolver One spreadsheets over an intranet or the web, which includes the use of editable cells, buttons and drop-down lists (which are also available in normal Resolver One spreadsheets of course). Our intention is to build Resolver One as a platform for creating business applications, so we're gradually making more user interface elements available to spreadsheets.

[1]	OK - so just about every spreadsheet has this, but it is still an important feature.

---

Amazon Simple DB with IronPython and Resolver One (plus Other News)

Giles Thomas, the CEO & CTO (or 'dear leader' as he likes to be called) of Resolver Systems has put together some tutorials on using Amazon SimpleDB with IronPython and Resolver One:

• Amazon SimpleDB and IronPython
• Importing from Amazon SimpleDB into Resolver One

Amazon SimpleDB is a cut-down database system, and as such it structures data quite differently to a normal relational database - in a manner that is actually quite similar to a spreadsheet. Each database is structured into "domains", which can be regarded as similar to worksheets. A query to the database can only access a single domain (unlike relational database queries, which can "join" multiple tables). However, the domain is quite freely structured. It is made up on a number of items (which you can see as being similar to rows in the worksheet), each item having a number of attributes (which you can see as being similar to columns). There is no requirement for each row to have the same set of attributes - in relational database terms, it is highly denormalised.

The examples use the Amazon SimpleDB C# Library.

The first Resolver One screencast has now gone up on Showmedo:

• Resolver One in One Minute

We've scheduled time in the next couple of iterations for a series of short screencasts showing how to use some of the best features of Resolver One. Hopefully these will all appear on Showmedo as well.

At Resolver Systems we are in 'code freeze' for the release of Resolver One 1.1. Some of the features I mentioned in my last blog entry aren't quite ready yet, but there are some good features in the list. There will be an announcement soon on the Resolver Systems News Blog.

The IronPython team have also announced that they are aiming to do an updated release of IronPython 1 (which is still the stable version); the new version will be 1.1.2. This should include some good bugfixes and performance enhancements [1], including finally fixing the Python socket problems that have plagued those trying to use urllib and urllib2 from IronPython. The new version of IronPython is scheduled for June, which will probably make IronPython 1.1.2 just in time to include in Resolver One 1.1.1.

I've just discussed chapter ten of IronPython in Action with my editor at Manning. Just a few minor changes (which is a relief). While I do those he is going to be working on getting some more chapters ready for the early access program. With a bit of luck I'll get the edits to chapter ten done quickly enough to include that as well, meaning that the following chapters will be added to IronPython Early Access Program shortly:

• Chapter 9, IronPython and Windows Presentation Foundation
• Chapter 10, System Administration with IronPython (on shell scripting and IronPython integration with Powershell and Windows Management Instrumentation)
• Chapter 11, IronPython and ASP.NET (written by Christian Muirhead)

It's been a while since new chapters have been added, so this is good news!

Tomorrow I'm off to Developer Day Ireland. This is a .NET community event where I'll be talking about Dynamic Languages on .NET and several good friends from the UK (and Europe) .NET community will be there. I'm particularly looking forward to a talk on Functional Programming with C# 3 & 3.5 by Oliver Sturm. The conference is in Galway, and organised by the irrepressible Mick Lohan. On Sunday Mick is taking us on a trip to Craggy Ireland to enjoy the Irish scenery and hospitality.

In the meantime lots of other things are going on.

The LAST planned alpha releases of Python 2.6 and 3.0 are scheduled for next Wednesday, 07-May-2008. There are still several issues marked as Showstoppers

In November 2006 Guido announced what he had been working on - a code review tool for Google called Mondrian. He had hoped that they could Open Source it, but it is so widely used within Google now that it is too closely tied to Google's internal tools to disentangle it.

He has just announced on Python-Dev 'the next best thing':

a code review tool for use with Subversion, inspired by Mondrian and (soon to be) released as open source. Some of the code is even directly derived from Mondrian. Most of the code is new though, written using Django and running on Google App Engine.

He is inviting the Python developer community to try out the tool on the web for code reviews. To try it out, go to:

http://codereview.appspot.com

TechCrunch have also made an announcement (to be taken with the usual healthy dose of salt): that Twitter is migrating away from Rails to solve scaling problems - probably to Java or PHP.

Python has won the Linux Journal reader's 'Favorite Scripting Language 2008' award. Scripting language is not a term that Pythonistas are fond of, and there is an interesting discussion of this on the results page: Linux Journal: Readers' Choice Awards 2008.

The last piece of news is about PyPy. Yesterday an important milestone was reached, PyPy runs the pystone benchmark faster than CPython! And this is without the use of the JIT! Pystone is a crappy benchmark [2], but it is another step closer to PyPy being a useful implementation of Python. Armin should be blogging about this soon, I hope I'm not stealing his thunder but I did hold off for a whole day before this blog entry! Congratulations to all the PyPy team.

[1]	Srivatsn, on the dynamic languages testing team, has just twittered that they have also implemented the _winreg module for IronPython and he has added it to the list of things to go into 1.1.2 (which is no guarantee it will make it, but means it stands a good chance).

[2]	Of course all benchmarks are misleading, just as all generalisations are wrong.

---

Resolver One and IronPython in Action Progress

During conference season I fell behind schedule with writing IronPython in Action, so I've taken some time off work. I've just completed the first draft of "System Administration with IronPython" chapter, with only three more chapters to go. Christian is making good progress on the "Databases and Web Services" chapter, so the book actually might be finished soon...

The system administration chapter was good fun. It covers shell scripting with IronPython, WMI and Powershell. Writing about WMI and Powershell involved a lot of learning. From research to completion of first draft took basically a full week of solid work.

Despite the oddness, WMI turns out to be useful for lots of things. Here's one example (for the rest you'll have to buy the book of course...), listing all the processes that run on startup:

Powershell is an odd language (dynamic scoping anyone?). I've covered embedding Powershell in IronPython and vice-versa. My favourite example in the Powershell section is calling into arbitrary Python code and returning results, with IP 2. This means that you can use Python libraries from Powershell. A combination of the IP 2 hosting API and Powershell oddities (calling generic methods from Powershell hurts) means the code isn't pretty, but the unprettiness is abstracted away of course.

There are two implementations of Powershell in progress for Mono. Pash (Powershell + bash) is already under way which isn't officially part of the Mono Project. An 'official' Mono port called Monoshell doesn't exist yet, but has been accepted as a google summer of code project.

Whilst I've been writing, the Resolver guys have been hard at work. It's been a while since the 1.0.1 release and we're now working towards a 1.1 release (although there may or may not be a 1.0.2 in between). We're attacking Resolver One development on three fronts:

• Improving and extending the integration of Resolver One and Bloomberg and Thomsons' live market data feeds, for the financial users of Resolver One
• Adding features requested by Zeliade Systems for the Resolver One Quant Package
• Adding more features to make Resolver One a better platform for building spreadsheet based applications

The last two of these include things like providing a better code editing experience in Resolver One, more 'standard' spreadsheet features, and also improving performance and memory use. We're currently working on some optimisations which improve general recalc performance by 15-30% (we're already on a 15% improvement and there is more to come), which is great.

As well as the optimisations, new features currently being worked on include custom cell formatters (user defined functions for formatting objects in cells) and comments attached to cells.

Other random trivia:

• Preparations for PyCon UK 2008 have started in earnest and the first post is up on the PyCon UK Blog.

• Microsoft has a new project: Live Mesh which aims to sync data, applications and devices. As usual with new Microsoft technologies, IronPython featured in the demo (about halfway through the The Mesh Video Tour).

• Charles Nutter has a very interesting article on the many different implementations of Ruby that are up, and spring up: Promise and Peril for Alternative Ruby Impls. Other than wrongly stating that IronRuby was the first project to be licensed under the Microsoft Open Source License, it is a good read.

  Interesting to see that some of Ruby community are still sceptical that Microsoft will ever 'allow' Rails to run on .NET. This is despite the fact that Microsoft (well, Jim and Dino anyway) demoed the equally capable web framework Django running on IronPython...

---

Resolver One and the Python Magazine

The latest issue of the Python Magazine is out. It's subscription only, but well worth it.

This month's issue has some great articles:

• Dive into the gene pool with BioPython (mutable gene sequences - what horrors will this unleash on the world!)
• Scripting Xcode with Python
• Portable Data with PyTables
• A Database in the Cloud (with Google spreadsheets!)

Doug Hellmann (the editor) kicks off with a retrospective on PyCon 2008 (preparations for PyCon 2009 & 2010 are hotting up on the PyCon Organisers mailing list by the way - with Cleveland and Atlanta preparing bids for 2010). Resolver One even gets a mention:

Another session worth mentioning was given by the folks at Resolver. Their Resolver One spreadsheet/Python interpreter mash-up feels like what we've always wanted a spreadsheet to be - a full-blown programming environment with charts and graphs. It will be interesting to see how the product does, and whether non-programmers are ready for the power and flexibility afforded to them by exposing the Python interpreter. I'd like to give it a try, but it uses IronPython and only runs on Windows.

Cool.

Next month an article of mine on ConfigObj should be published.

Whilst I'm busy working on IronPython in Action (well...), the Resolver Systems folk have been working on some more screencasts. The latest one is on using Python expressions in formulae, and complex numbers and custom datatypes in Resolver One spreadsheets (even better - it is short):

• Complex Numbers and other Python datatypes in Resolver One

You can see all the screencasts that we have produced so far at: Resolver Systems Screencasts.

Oh, and if you aren't following me on Twitter you're missing out. There's now quite a big community of Python programmers there. Here are some links I've posted there recently:

• Rails: Monkey Patching Core Functionality == BAD, BAD, BAD
• Penny Arcade on Taking Twitter too Far..
• Job ad on 37signals: "We're looking for an experienced Silverlight developer." [1]
• A different take on "I'm a Mac - and I'm a PC"
• New PocketPC Python GUI toolkit: PocketPyGui
• Microsoft Mesh Video Tour (with IronPython used halfway through!)
• MacRuby.. Ruby implemented on Objective-C

[1]	Which reminds me of an email by a recruiter to the Python-UK mailing list not long ago. The job required 'a minimum of five years Django experience'.

---

Ironclad 0.2 Released: Use CPython Extensions from IronPython (Nearly...)

William Reade of Resolver Systems has just announced the release of Ironclad 0.2:

• Ironclad Download Page

Ironclad is a project to implement the CPython C API for IronPython, allowing you to use compiled C extensions from IronPython. Ironclad targets IronPython 2 Beta 1 and CPython 2.5.

We have been working with the bz2 module as our first example (a nice small extension module that doesn't use too much of the API!). With release 0.2 It is now possible to use all the functions and types -- and their methods -- from the bz2 module.

Ironclad comes with a full test-suite that also illustrates its use. You can now load simple CPython extensions and use their types and functions from IronPython!

Current limitations are as follows:

• Lists are not well mapped -- changes made on one side of the managed/unmanaged divide will not necessarily be seen on the other [1].
• Type members and properties don't work yet.
• Most of the CPython API is still not implemented.

The next steps are to make attributes and properties work, and then to start work on the multiarray module from numpy.

[1]	The difficulty with lists are that they allow direct memory access to their elements - which is fun when we have to map an IronPython list to one that the C extension can work with and then back again...

---

Videos, Spreadsheet Quagmires and a Summer of Code

The google summer of code projects are now up. As usual, the PSF has several projects. This year there I don't find many of them particularly exciting (although I'm sure others will). Ones that did catch my eye:

• Bringing Python 2.5 Features to PyPy
• Sandboxing the tinypy Interpreter
• Bringing Ttk to Tkinter
• Filesystem Virtualisation for the Standard Library
• urilib for Python 2.6 and 3.0

OK, so maybe it's not such a bad list - nothing for docutils though.

A couple of Resolver One related videos have gone up.

Firstly, someone (Mick!) from Developer Day Ireland has put up a video interview I did at TechEd on developing with IronPython at Resolver Systems on YouTube:

• Michael Foord on IronPython

Menno and Glenn have also put up a new Resolver One screencast. This one is on using named ranges in Resolver One, and how they can make your spreadsheets simpler:

• Named Ranges in Resolver One

(The voice over is done by Menno.) This is the first in a series of short screencasts that we hope to produce, highlighting useful features in Resolver One.

On the subject of Resolver One, Investment News have just published an article on us:

• Resolver Solves Spreadsheet Quagmire

There has also been a rash of Python based build tools recently. This is good, because until now we have had Rake envy. The new kid on the block is Paver by Kevin Dangoor:

Paver is a Python-based build/distribution/deployment scripting tool along the lines of Make or Rake. What makes Paver unique is its integration with commonly used Python libraries. Common tasks that were easy before remain easy. More importantly, dealing with your applications specific needs and requirements is also easy.

Other alternatives include:

• zc.buildout

  Doesn't require Zope, honestly. Works with eggs and very actively developed.

• vellum

  Another new kid on the block, by Zed Shaw. Uses a 'subset' of Python for configuration scripts, so that your build scripts deliberately can't execute arbitrary code.

• SCons

  This seems to be the 'heavyweight' of the bunch.

• waf

  I don't know anything about this.

• memoize

  I don't know anything about this.

• doit

  Very new and I don't know anything about it.

It seems that Python build tools are flavour of the month. (Build tools are the new web frameworks.) Perhaps the Ruby community's ability to coalesce (more or less) around standard tools and frameworks is a good thing after all...

And finally, as a reward for reading to the end, have you ever wished that you could raise an exception as an expression? (Inside a lambda for example, raise being a statement n'all.) Here is one truly evil (but kind of beautiful in its awfulness) suggestion using ctypes from Ironfroggy:

---

Report from the RuPy Conference

This was the second year for the RuPy Ruby and Python conference in Poznan, Poland. Last year there were 80 attendees and this year the conference had grown to 250. The conference is not only a great concept (great to see Ruby and Python grouped together rather than pitted against each other), but there were a high number of quality talks.

My talk was on IronPython and Silverlight 2 and it went great. I felt confident with the material, the audience laughed in all the right places and all my demos worked! (There were 80~100 in the audience - I'm rubbish at guessing numbers.) I think the talk was improved by showing two good looking Silverlight applications from the gallery. These were Tetrislight and the Image Snipper, both of which do a much better job of showing what Silverlight is capable of than my 'developer oriented' examples! (But you'll learn more from mine of course.)

As usual with a Python crowd the biggest concern was the Linux compatibility story (I believe that Moonlight will prove to be a very good 'Silverlight 2 on Linux', but time will tell). As for the adoption story, Silverlight 1 is currently being downloaded at a rate of about 1.5 million per day and is already being offered as an 'optional update' via Windows update. I'm sure that Microsoft will do the same with Silverlight 2, meaning that not long after the final release it will probably find its way onto ~90% of the world's desktop computers! Microsoft have a deal to put Silverlight 2 on the official Olympics website this summer. This gives them both a hard deadline for final release and a great driver for Silverlight 2 adoption.

The infamous Zed Shaw was there and gave a very tame talk on how to get statistically meaningful results when testing web app performance. He is a great speaker, and he did have a dig at Rails with an utterly unfair performance comparison between Rails and web.py as his example. Naturally web.py, which does a lot less, could handle ten times as many requests per second.

Interestingly, he used R to turn the raw data into meaningful results and plots. R is an odd looking language, but it did look like a good fit for the task. Of course many Pythonistas would prefer him to use matplotlib...

I missed Jay Field's talk on Domain Specific Languages. Jay is renowned in the Ruby community, both for his work with DSLs and his focus on testing. The reason I missed the talk was because I was talking to Zed (he is loud - but he isn't an asshole). As he currently works at a large financial institution I thought I would show him Resolver One. However, as that organisation is Bear Stearns I doubt much will come of it.

I did ask Zed if Python is a Ghetto. He said no.

I spent quite a bit of time talking to Carl Friedrich Bolz [1], who is a core PyPy developer and one of the writers on the PyPy Status Blog. One of the things that Carl has done is write a framework so that PyPy Garbage Collectors can be written in Python - plus he wrote the first two GC implementations. Armin has just written a new one that is more efficient - as it can avoid copying large objects - by subclassing one of the existing ones.

Carl did an excellent talk on the current state of PyPy and particularly the recent work he has been doing on the Just In Time compiler. The PyPy JIT is a tracing JIT (like Tamarin), and as well as speeding up operations on the basic types it will be able to optimise method calls (etc) which are one of the bugbears of Python performance. Techniques it uses include not generating intermediate objects that will always be thrown away and even keeping frame objects in registers where possible (without any semantic changes to Python). Very clever stuff.

For those few who still don't understand the purpose of PyPy, it is much more than just 'Python implemented in Python'. PyPy is a language agnostic compiler toolchain. Typically it is used for compiling interpreters that are written in RPython, a static subset of Python. Interpreters are analysed and compiled by PyPy, which can emit C, Java or .NET bytecode (and various other actual and possible backends).

It also compiles garbage collection and a Just in Time compiler for the interpreter (although the GC is not needed for the .NET and JVM backends). Other interesting things can be done, like experimenting with new GC and object space implementations. So PyPy is a compiler toolchain - with swappable front, middle and backends! As well as this, there is a highly compliant Python interpreter written in RPython. When translated to C it currently runs at around half the speed of CPython, but it is already faster for highly 'memory bound' operations (because the GC is better).

As the JIT makes real progress (it is the subject of Carl's thesis - so real progress is being made), some interesting things become possible. First of all, PyPy is likely to become faster than CPython. I've long joked that so far PyPy's greatest contribution to the Python community is to kill off Psyco and Stackless [2]. It finally looks like it won't be very long before I have to stop saying this.

One fun, and plausible if not very likely, consequence is that it would be entirely possible to write a Ruby interpreter in Python (RPython of course)! That might make a few flame wars moot! Porting Zope to different Pythons is an article that explores some of the more likely consequences.

On the Saturday night was the geek party, which was great fun and I had entirely too much to drink (largely at the behest of Jonathan and Susan, so it was not my fault). Jay joined the reverie and we staggered back from the centre of Poznan at 4.30 in the morning.

Speaking of Jonathan, he gave his talk on Test Driven Development. This is an excellent talk and it gets better every time he delivers it. Yet again he failed to get his Ubuntu laptop connected to the projector though, and his fonts were screwed up. No matter though, the large audience appreciated it and asked a lot of questions. If any of you have objections to my earlier blog entry on the value of test first, you need to hear Jonathan.

On the Sunday morning Konrad woke me up at 9am. I got up, which was a mistake! It did mean I was able to demo Resolver One (during the post-lunch lightning talks, which were new to RuPy this year), but I was fairly befuddled whilst doing it (actually the demo went ok but the setting up was a bit 'confused').

There was a 3 hour 'Rails 101' session, which if I had been less knackered I would have enjoyed. Instead I spent the time talking to Carl. I nearly persuaded him that he ought to be working at Resolver Systems. It is not an entirely far-fetched suggestion as he is something of an Excel/VBA expert and a great fan of what we have done with Resolver One.

All in all it was a great conference, and if you are available next year I highly recommend it.

[1]	Who stepped in at the last minute to replace Maciek who decided to stay in the US to be with his girlfriend. Some people have funny priorities.

[2]

Both of these charges are baseless. Stackless is maintained (although not mainly by Christian Tismer any more), and due to its necessary complexity Armin would not have continued to develop Psyco even without PyPy. Although Psyco is kept 'working' it doesn't optimise language features added since about Python 2.2, and it is never likely to be ported to Python 3 or to support 64 bit builds of Python.

---

On Testing: Some Programmers Refuse to Get it

Well I hadn't intended this to be my first blog entry after returning from RuPy and Expert Zone, in fact I should be going to bed. However, this blog entry on whether large test suites prove Python is not suitable for large projects is a subject dear to my heart.

The basic tenet (not advanced by the blog author himself but as a response to an article on Resolver One he wrote), is laughable - but I've been speaking about dynamic languages to .NET developers recently and this belief is not unusual. In fact I have a blog entry on static and dynamic typing waiting in the wings, but meanwhile...

Statically typed languages need type safety in order to be able to compile. The compiler must know the types of objects at all points in your program, and so if the compiler accepts your program you have 'type safety'. This is a very thin layer of safety, and applications in statically typed languages have occasionally been found to have bugs. In fact type errors ('typos') are the very easiest bugs to detect. Tools like PyLint (the most powerful), and PyFlakes (the fastest) will catch many of the errors a compiler will catch and are easy to integrate with Python IDEs.

More importantly, Test Driven Development is much more about a design approach than it is about catching errors. Almost none of our tests are there to catch the kind of errors a compiler would catch - the app. simply won't run with those errors in place so it is not necessary to test for them - our tests are there to test application logic and functionality. A good sound principle is: if it isn't tested it's broken. My favourite quote on the subject comes from the Ruby community:

If you write applications without tests then you are a bad person, incapable of love.

—Wilson Bilkovich (The Rails Way)

There are lots of people who write large Python applications without automated test suites. I think they are crazy. There are lots of people who write large applications in statically typed languages without automated test suites. I think they're crazier - but for two different reasons...

In fact the best refutation is to tell you the story of how Resolver One came to be written in Python.

The first two developers started writing Resolver One around November 2005. They were a Java developer and a C++ developer and for various reasons had chosen the .NET framework for Resolver One. They assumed they would use C#, as that is what you do when writing a desktop application with .NET, but Resolver One needed an interpreted language embedded in it. They started to evaluate scripting language 'engines' that were available at the time, which didn't give them a great deal of choice, and started to experiment with IronPython which was at version 0.7.

Not only was the .NET integration very good, but something else became apparent and influenced them to see how far they could get actually writing Resolver One in IronPython. They had already decided to use Test Driven Development as their basic development methodology. They were convinced that this was the best way to ensure code quality, and having practised it now for two years at Resolver Systems I'm convinced that they are right. Dynamic languages are substantially easier to test with. Mocking objects is trivially easy and you can monkey patch at runtime for testing purposes being two of the major reasons for this.

So the decision to use Python came well after (almost as a consequence of...) the decision to practise TDD.

I very much enjoyed the recent UK ALT.NET conference, where a bunch of .NET developers got together to discuss topics like agile development and testing strategies. Most of them develop in C#, and although it was great to spend time with them it was very obvious that a lot of the problems they encountered I simply didn't have as a Python programmer. Testing statically typed languages is harder, and ironically to maintain the same test coverage if Resolver One was written in C# we would need a lot more test code.

Test Driven Development produces better code, and this alone is reason to do it (my colleague Jonathan Hartley does an excellent talk on the subject by the way), but it has many other benefits as well:

• It forces you to think about feature design and API design before writing code
• A large test suite means that you can refactor mercilessly without fear - the tests will tell you when you have got everything working again
• Your tests act as a specification and as documentation on how code is intended to be used
• They help you to resist 'developer gold plating' by focussing your mind on features you actually need (rather than might need or would like)
• Reducing the need for a separate test phase and increasing the feedback between writing code and learning about bugs you have introduced
• It breaks down large tasks into smaller and more easily understood steps
• A comprehensive test suite that individually exercises units of your application is an invaluable debugging tool

Anyway, some programmers don't get it and don't want to get it. To my mind testing is much more important than which language you use. Some people just prefer statically typed languages, and that's fine. But some people don't like testing, and if you have any concern for code quality that isn't fine.

---