The Voidspace Techie Blog

ironpythoninaction.com: New Chapters and Sourcecode Available

Three new chapters have been added to the Manning Early Access Program for IronPython in Action. There are now eleven out of fifteen chapters available. The new chapters are:

• Chapter 9: WPF and IronPython

• Chapter 10: System Administration with IronPython

  Covering shell scripting, WMI and Powershell (both embedding Powershell in IronPython and IronPython in Powershell).

• Chapter 11: IronPython and ASP.NET

  This is the first of two chapters written by Christian Muirhead.

I'm now working on the Silverlight chapter, whilst Christian is busy writing the databases and web services chapter.

I've put up a website for the book:

• Ironpythoninaction.com

The website has the sourcecode to accompany the first eleven chapters available for download, and a table of contents for everything done so far:

• Download Sourcecode
• Table of Contents for Chapters 1-11

---

Resolver One 1.1 Features Announced

We've now announced the new features in the forthcoming Resolver One 1.1 release:

We're now moving toward a feature freeze for Resolver One 1.1. Here's a foretaste of some of the changes you can expect to see:

• Major performance and memory usage improvements.
• Improved cut - when you cut and paste a block, formulae that refer to cells in the block are updated appropriately.
• Comments for cells.
• Simple auto-indentation in the code editor.
• For the financial edition, we've added Thomson Dataworks Enterprise connectivity and a number of great enhancements to Bloomberg access.
• ContentRows and ContentCols iterators on worksheets and cell ranges so that you don't have to keep checking for the header.
• The CopyRange function can now sort the data as it copies.
• HLOOKUP, TODAY, ISNUMBER, ISBLANK, NOW, and ABS functions
• And lots of minor tweaks and bugfixes.

Several of the bugfixes include fixing some of the obscure ways it was possible to crash Resolver One previously. Although most people would rarely have encountered them it is still nice to see them gone. One I'm particularly glad to see the back of, is that setting clipboard contents when Resolver One was running virtualised could fail - causing a harmless but annoying crash dialog (and failing to set the clipboard contents). Resolver One is now resilient against crashing in these circumstances and will retry setting the clipboard.

I've just returned from a fun trip to Galway for a community .NET event, DDD Ireland. I'll put together a blog entry on this later, but whilst there I had the chance to demo and talk about Resolver One to several of the delegates and speakers. Every time I do this I try to hone a concise description of exactly what Resolver One is.

The best I've come up with so far goes something like this...

Resolver One is a spreadsheet development platform for the rapid creation of business applications. It is an Object Oriented, programmable spreadsheet. Its key features are:

• The trivially easily mixing of code with spreadsheets
• You can put arbitrary objects in the grid
• Easy access to .NET and Python libraries
• You can export a spreadsheet as code

Some people immediately 'get' this (mainly those who have wrestled with complex spreadsheets already), but I think that for many developers Resolver One is the sort of program that is best understood by actually trying it.

Some of the other advantages that Resolver One has over a more 'traditional' spreadsheet are:

• A rich formula language based on Python expressions (list comprehensions and generator expressions in formulae for example)
• A single, unified, spreadsheet object model - with powerful features like indexing worksheets and cellranges by names used in headers (makes for more readable formulae)
• With a stored procedure databases can notify Resolver One of changes (typically on update/delete/insert), giving you a live auto-updating view into databases
• Because it uses Python you can define new numerical types and use them in your spreadsheet
• With database worksheets, or from user defined code, working with databases becomes both easy and powerful
• Import and export spreadsheets to and from Microsoft Excel format [1]

There is also a web server for serving Resolver One spreadsheets over an intranet or the web, which includes the use of editable cells, buttons and drop-down lists (which are also available in normal Resolver One spreadsheets of course). Our intention is to build Resolver One as a platform for creating business applications, so we're gradually making more user interface elements available to spreadsheets.

[1]	OK - so just about every spreadsheet has this, but it is still an important feature.

---

Open Source Code Review Tool: Rietveld

In November 2006, Guido van Rossum gave a public talk at Google about Mondrian, a web tool for code review on the web that he was developing at Google. He was hoping to release it as an Open Source tool, but due to its popularity within Google it became to closely tied to Google's internal infrastructure to disentangle.

He recently announced a new tool, for Subversion and based on the Mondrian code base but for the Google App Engine. As well as a powerful code review tool, it also serves as an example Google App Engine project. The online tool, which Guido recently invited Python developers to try, is called codereview:

http://codereview.appspot.com

This has now been released as an Open Source project with the name Rietveld:

• Rietveld Project Page
• Rietveld Mailing List

The name is in honor of Gerrit Rietveld, Dutch architect and furniture designer.

People are already making lots of feature requests, and now that reitveld is Open Source, they have the opportunity to contribute those changes...

---

Amazon Simple DB with IronPython and Resolver One (plus Other News)

Giles Thomas, the CEO & CTO (or 'dear leader' as he likes to be called) of Resolver Systems has put together some tutorials on using Amazon SimpleDB with IronPython and Resolver One:

• Amazon SimpleDB and IronPython
• Importing from Amazon SimpleDB into Resolver One

Amazon SimpleDB is a cut-down database system, and as such it structures data quite differently to a normal relational database - in a manner that is actually quite similar to a spreadsheet. Each database is structured into "domains", which can be regarded as similar to worksheets. A query to the database can only access a single domain (unlike relational database queries, which can "join" multiple tables). However, the domain is quite freely structured. It is made up on a number of items (which you can see as being similar to rows in the worksheet), each item having a number of attributes (which you can see as being similar to columns). There is no requirement for each row to have the same set of attributes - in relational database terms, it is highly denormalised.

The examples use the Amazon SimpleDB C# Library.

The first Resolver One screencast has now gone up on Showmedo:

• Resolver One in One Minute

We've scheduled time in the next couple of iterations for a series of short screencasts showing how to use some of the best features of Resolver One. Hopefully these will all appear on Showmedo as well.

At Resolver Systems we are in 'code freeze' for the release of Resolver One 1.1. Some of the features I mentioned in my last blog entry aren't quite ready yet, but there are some good features in the list. There will be an announcement soon on the Resolver Systems News Blog.

The IronPython team have also announced that they are aiming to do an updated release of IronPython 1 (which is still the stable version); the new version will be 1.1.2. This should include some good bugfixes and performance enhancements [1], including finally fixing the Python socket problems that have plagued those trying to use urllib and urllib2 from IronPython. The new version of IronPython is scheduled for June, which will probably make IronPython 1.1.2 just in time to include in Resolver One 1.1.1.

I've just discussed chapter ten of IronPython in Action with my editor at Manning. Just a few minor changes (which is a relief). While I do those he is going to be working on getting some more chapters ready for the early access program. With a bit of luck I'll get the edits to chapter ten done quickly enough to include that as well, meaning that the following chapters will be added to IronPython Early Access Program shortly:

• Chapter 9, IronPython and Windows Presentation Foundation
• Chapter 10, System Administration with IronPython (on shell scripting and IronPython integration with Powershell and Windows Management Instrumentation)
• Chapter 11, IronPython and ASP.NET (written by Christian Muirhead)

It's been a while since new chapters have been added, so this is good news!

Tomorrow I'm off to Developer Day Ireland. This is a .NET community event where I'll be talking about Dynamic Languages on .NET and several good friends from the UK (and Europe) .NET community will be there. I'm particularly looking forward to a talk on Functional Programming with C# 3 & 3.5 by Oliver Sturm. The conference is in Galway, and organised by the irrepressible Mick Lohan. On Sunday Mick is taking us on a trip to Craggy Ireland to enjoy the Irish scenery and hospitality.

In the meantime lots of other things are going on.

The LAST planned alpha releases of Python 2.6 and 3.0 are scheduled for next Wednesday, 07-May-2008. There are still several issues marked as Showstoppers

In November 2006 Guido announced what he had been working on - a code review tool for Google called Mondrian. He had hoped that they could Open Source it, but it is so widely used within Google now that it is too closely tied to Google's internal tools to disentangle it.

He has just announced on Python-Dev 'the next best thing':

a code review tool for use with Subversion, inspired by Mondrian and (soon to be) released as open source. Some of the code is even directly derived from Mondrian. Most of the code is new though, written using Django and running on Google App Engine.

He is inviting the Python developer community to try out the tool on the web for code reviews. To try it out, go to:

http://codereview.appspot.com

TechCrunch have also made an announcement (to be taken with the usual healthy dose of salt): that Twitter is migrating away from Rails to solve scaling problems - probably to Java or PHP.

Python has won the Linux Journal reader's 'Favorite Scripting Language 2008' award. Scripting language is not a term that Pythonistas are fond of, and there is an interesting discussion of this on the results page: Linux Journal: Readers' Choice Awards 2008.

The last piece of news is about PyPy. Yesterday an important milestone was reached, PyPy runs the pystone benchmark faster than CPython! And this is without the use of the JIT! Pystone is a crappy benchmark [2], but it is another step closer to PyPy being a useful implementation of Python. Armin should be blogging about this soon, I hope I'm not stealing his thunder but I did hold off for a whole day before this blog entry! Congratulations to all the PyPy team.

[1]	Srivatsn, on the dynamic languages testing team, has just twittered that they have also implemented the _winreg module for IronPython and he has added it to the list of things to go into 1.1.2 (which is no guarantee it will make it, but means it stands a good chance).

[2]	Of course all benchmarks are misleading, just as all generalisations are wrong.

---

Resolver One and IronPython in Action Progress

During conference season I fell behind schedule with writing IronPython in Action, so I've taken some time off work. I've just completed the first draft of "System Administration with IronPython" chapter, with only three more chapters to go. Christian is making good progress on the "Databases and Web Services" chapter, so the book actually might be finished soon...

The system administration chapter was good fun. It covers shell scripting with IronPython, WMI and Powershell. Writing about WMI and Powershell involved a lot of learning. From research to completion of first draft took basically a full week of solid work.

Despite the oddness, WMI turns out to be useful for lots of things. Here's one example (for the rest you'll have to buy the book of course...), listing all the processes that run on startup:

Powershell is an odd language (dynamic scoping anyone?). I've covered embedding Powershell in IronPython and vice-versa. My favourite example in the Powershell section is calling into arbitrary Python code and returning results, with IP 2. This means that you can use Python libraries from Powershell. A combination of the IP 2 hosting API and Powershell oddities (calling generic methods from Powershell hurts) means the code isn't pretty, but the unprettiness is abstracted away of course.

There are two implementations of Powershell in progress for Mono. Pash (Powershell + bash) is already under way which isn't officially part of the Mono Project. An 'official' Mono port called Monoshell doesn't exist yet, but has been accepted as a google summer of code project.

Whilst I've been writing, the Resolver guys have been hard at work. It's been a while since the 1.0.1 release and we're now working towards a 1.1 release (although there may or may not be a 1.0.2 in between). We're attacking Resolver One development on three fronts:

• Improving and extending the integration of Resolver One and Bloomberg and Thomsons' live market data feeds, for the financial users of Resolver One
• Adding features requested by Zeliade Systems for the Resolver One Quant Package
• Adding more features to make Resolver One a better platform for building spreadsheet based applications

The last two of these include things like providing a better code editing experience in Resolver One, more 'standard' spreadsheet features, and also improving performance and memory use. We're currently working on some optimisations which improve general recalc performance by 15-30% (we're already on a 15% improvement and there is more to come), which is great.

As well as the optimisations, new features currently being worked on include custom cell formatters (user defined functions for formatting objects in cells) and comments attached to cells.

Other random trivia:

• Preparations for PyCon UK 2008 have started in earnest and the first post is up on the PyCon UK Blog.

• Microsoft has a new project: Live Mesh which aims to sync data, applications and devices. As usual with new Microsoft technologies, IronPython featured in the demo (about halfway through the The Mesh Video Tour).

• Charles Nutter has a very interesting article on the many different implementations of Ruby that are up, and spring up: Promise and Peril for Alternative Ruby Impls. Other than wrongly stating that IronRuby was the first project to be licensed under the Microsoft Open Source License, it is a good read.

  Interesting to see that some of Ruby community are still sceptical that Microsoft will ever 'allow' Rails to run on .NET. This is despite the fact that Microsoft (well, Jim and Dino anyway) demoed the equally capable web framework Django running on IronPython...

---

Most Popular Blog Entries

Feedburner allows me to monitor my RSS feed and see how many people are clicking through from my blog entry summaries to read the full entry [1]. Feedburner reports that I have a current average of about 830 subscribers. I have no idea whether this reflects the extent of my readership, as I've always assumed that the majority of the people who read this blog through Planet Python [2].

I can track the number of people who click through from each item, which does show the number of people who are interested enough in the title / summary to read the full entry. Here's a list of the most popular entries (everything with more than 500 clicks). It isn't completely representative as I only switched my feed fully over to Feedburner in December 2007, and I sometimes have several entries on the same page - so people can click through to one and then read several. Still, I find it interesting.

• On Testing: Some Programmers Refuse to Get It - 770 clicks 14th April 2008
• Python Properties and New Language Features in C# 3.0 - 754 clicks 11th November 2007
• Everyone's First Vi Session - 731 clicks August 19th 2007
• Chasing Memory Leaks in Python Applications - 723 clicks 9th January 2008
• Silly Snippets - 704 clicks 18th February 2008
• Apples, Django is Wired and Faster Python - 703 clicks 25th March 2008
• IronPython and Kamaelia (and a new web framework...) - 639 clicks 9th September 2007
• 2D Graphics with Pyglet and OpenGL - 590 clicks February 8th 2008
• Does Django Have Rails Envy? - 590 clicks September 10th 2007
• Shiny New Asus Eee PC - 584 clicks January 4th 2008
• A Year with Python, IronPython and Ruby - 579 clicks January 9th 2008
• Release: ConfigObj 4.5.2 and validate 0.3.2 - 570 clicks February 27th 2008
• Tail Call Optimisation - 568 clicks September 24th 2007
• Resolver One 1.0.1 Released - 561 clicks March 1st 2008
• Today at Work: Assembly, C/C++, C#, IronPython and Python - 536 clicks January 16th 2008
• Resolverforge: Download Modules on Demand for IronPython and Resolver One - 523 clicks February 11th 2008
• IronPython in Action Available for Pre-Order - 520 clicks February 23rd 2008
• Resolver Hacks Updates - 520 clicks February 11th 2008
• Squeak on PyPy - 510 clicks October 23rd 2007
• Test Driven Development: New References - 508 clicks January 27th 2008
• Ordered Dictionary - 505 clicks October 21st 2007
• Python 3, Bytes and Source File Encoding - 504 clicks March 23rd 2008

As this list in some way reflects the interests of the Python community [3], it's nice to see some IronPython and Resolver related entries amongst the popular ones.

Also interesting that the most popular entry is the recent (controversial) one I wrote on testing...

[1]	I realise that many people would prefer a full feed rather than a partial feed. That would mean hacking on my Desktop Blog Client Tool: Firedrop2. I really like this program and am not willing to switch away - at some point I will experiment with enabling full feeds, but not yet.

[2]	Or the Unofficial Planet Python.

[3]	Or the part of the Python community that reads Planet Python anyway, which almost certainly doesn't include the 'silent majority' of Python users and programmers.

---

Resolver One and the Python Magazine

The latest issue of the Python Magazine is out. It's subscription only, but well worth it.

This month's issue has some great articles:

• Dive into the gene pool with BioPython (mutable gene sequences - what horrors will this unleash on the world!)
• Scripting Xcode with Python
• Portable Data with PyTables
• A Database in the Cloud (with Google spreadsheets!)

Doug Hellmann (the editor) kicks off with a retrospective on PyCon 2008 (preparations for PyCon 2009 & 2010 are hotting up on the PyCon Organisers mailing list by the way - with Cleveland and Atlanta preparing bids for 2010). Resolver One even gets a mention:

Another session worth mentioning was given by the folks at Resolver. Their Resolver One spreadsheet/Python interpreter mash-up feels like what we've always wanted a spreadsheet to be - a full-blown programming environment with charts and graphs. It will be interesting to see how the product does, and whether non-programmers are ready for the power and flexibility afforded to them by exposing the Python interpreter. I'd like to give it a try, but it uses IronPython and only runs on Windows.

Cool.

Next month an article of mine on ConfigObj should be published.

Whilst I'm busy working on IronPython in Action (well...), the Resolver Systems folk have been working on some more screencasts. The latest one is on using Python expressions in formulae, and complex numbers and custom datatypes in Resolver One spreadsheets (even better - it is short):

• Complex Numbers and other Python datatypes in Resolver One

You can see all the screencasts that we have produced so far at: Resolver Systems Screencasts.

Oh, and if you aren't following me on Twitter you're missing out. There's now quite a big community of Python programmers there. Here are some links I've posted there recently:

• Rails: Monkey Patching Core Functionality == BAD, BAD, BAD
• Penny Arcade on Taking Twitter too Far..
• Job ad on 37signals: "We're looking for an experienced Silverlight developer." [1]
• A different take on "I'm a Mac - and I'm a PC"
• New PocketPC Python GUI toolkit: PocketPyGui
• Microsoft Mesh Video Tour (with IronPython used halfway through!)
• MacRuby.. Ruby implemented on Objective-C

[1]	Which reminds me of an email by a recruiter to the Python-UK mailing list not long ago. The job required 'a minimum of five years Django experience'.

---

Ironclad 0.2 Released: Use CPython Extensions from IronPython (Nearly...)

William Reade of Resolver Systems has just announced the release of Ironclad 0.2:

• Ironclad Download Page

Ironclad is a project to implement the CPython C API for IronPython, allowing you to use compiled C extensions from IronPython. Ironclad targets IronPython 2 Beta 1 and CPython 2.5.

We have been working with the bz2 module as our first example (a nice small extension module that doesn't use too much of the API!). With release 0.2 It is now possible to use all the functions and types -- and their methods -- from the bz2 module.

Ironclad comes with a full test-suite that also illustrates its use. You can now load simple CPython extensions and use their types and functions from IronPython!

Current limitations are as follows:

• Lists are not well mapped -- changes made on one side of the managed/unmanaged divide will not necessarily be seen on the other [1].
• Type members and properties don't work yet.
• Most of the CPython API is still not implemented.

The next steps are to make attributes and properties work, and then to start work on the multiarray module from numpy.

[1]	The difficulty with lists are that they allow direct memory access to their elements - which is fun when we have to map an IronPython list to one that the C extension can work with and then back again...

---

Videos, Spreadsheet Quagmires and a Summer of Code

The google summer of code projects are now up. As usual, the PSF has several projects. This year there I don't find many of them particularly exciting (although I'm sure others will). Ones that did catch my eye:

• Bringing Python 2.5 Features to PyPy
• Sandboxing the tinypy Interpreter
• Bringing Ttk to Tkinter
• Filesystem Virtualisation for the Standard Library
• urilib for Python 2.6 and 3.0

OK, so maybe it's not such a bad list - nothing for docutils though.

A couple of Resolver One related videos have gone up.

Firstly, someone (Mick!) from Developer Day Ireland has put up a video interview I did at TechEd on developing with IronPython at Resolver Systems on YouTube:

• Michael Foord on IronPython

Menno and Glenn have also put up a new Resolver One screencast. This one is on using named ranges in Resolver One, and how they can make your spreadsheets simpler:

• Named Ranges in Resolver One

(The voice over is done by Menno.) This is the first in a series of short screencasts that we hope to produce, highlighting useful features in Resolver One.

On the subject of Resolver One, Investment News have just published an article on us:

• Resolver Solves Spreadsheet Quagmire

There has also been a rash of Python based build tools recently. This is good, because until now we have had Rake envy. The new kid on the block is Paver by Kevin Dangoor:

Paver is a Python-based build/distribution/deployment scripting tool along the lines of Make or Rake. What makes Paver unique is its integration with commonly used Python libraries. Common tasks that were easy before remain easy. More importantly, dealing with your applications specific needs and requirements is also easy.

Other alternatives include:

• zc.buildout

  Doesn't require Zope, honestly. Works with eggs and very actively developed.

• vellum

  Another new kid on the block, by Zed Shaw. Uses a 'subset' of Python for configuration scripts, so that your build scripts deliberately can't execute arbitrary code.

• SCons

  This seems to be the 'heavyweight' of the bunch.

• waf

  I don't know anything about this.

• memoize

  I don't know anything about this.

• doit

  Very new and I don't know anything about it.

It seems that Python build tools are flavour of the month. (Build tools are the new web frameworks.) Perhaps the Ruby community's ability to coalesce (more or less) around standard tools and frameworks is a good thing after all...

And finally, as a reward for reading to the end, have you ever wished that you could raise an exception as an expression? (Inside a lambda for example, raise being a statement n'all.) Here is one truly evil (but kind of beautiful in its awfulness) suggestion using ctypes from Ironfroggy:

---

Dabs.com and Trying to Get My Laptop Fixed

A couple of weeks ago I blogged about trying to get Fujitsu-Siemens to honour the warranty on my laptop. The laptop DVD drive fell out and (then later!) stopped working altogether. Unfortunately I reported the problem two days after the warranty expired and Fujitsu stated that they no longer had any obligation to help me and referred me back to Dabs.com where I purchased the laptop from.

Under UK consumer law (which I believe is stronger than in the US), good have to be fit for purpose and be free from manufacturing defects. If an item doesn't meet this standard then you have recourse to the small claims court for redress; against the person who sold you the item (Dabs in my case rather than Fujitsu). Reasonable wear and tear do not give you cause for redress, but offering a time limited warranty does not remove these obligations - whether or not they are discovered or reported within the warranty period.

I went back to Dabs.com and explained the situation. At first they said they would arrange for the laptop to be fixed and gave me a number to ring. This turned out to be the number for Fujitsu-Siemens - who still refused to help. I guess that Dabs hadn't actually read what I'd told them in the first place, because on 'discovering' that Fujitsu weren't going to fix the laptop they claimed that I was responsible for the damage:

Please be advised that the warranty agreement is now void due to physical damage which has been caused to the goods in question - whilst under your duty of care. We are unable to draw any other conclusions from the information that has been presented to us. If you have any further information that you can offer regarding this matter, then please do email it to us and we will investigate the matter further.

My reply went last week, along with copies of the letters to Fujitsu:

Please note that there are no external signs of any 'physical damage' that could possibly have caused this. The likely cause is inadequate fixing internally.

I have owned several laptops that have been subjected to heavier use (this laptop is almost entirely used by my wife in the house - (a fact she will readily attest to) and none of them have suffered any major faults - let alone the CD drive falling out!

So far I've had no response and it looks like I will have to take this to court. Fortunately I have friends who are solicitors and can give me free legal advice.

I'm a bit disappointed that Dabs are letting it go this far though.

---

Python in Pardus Linux

A couple of links about Python in the Pardus Linux Distribution showed up on the Python Advocacy Mailing List (thanks to David Boddie):

• Python in Pardus
• Speeding Up Linux: One Step Further With Pardus

On why they chose Python:

In simplest case of build operation (configure; make), there isn't much difference between alternative languages. But when cases get more complex, you'll definitely want a language with rich set of datatypes, native support for strings, clean and concise syntax and a standard library with a wide range of useful functions. This language shouldn't have a steep learning curve, and shouldn't look too different from shell scripts or shouldn't require a complete paradigm change for system admins and packagers (like some pure functional languages).

We concluded that Python successfully meets these requirements, and is the best choice for package scripts.

and:

All those special tools with their inefficiencies and their size turns the system into a strange Rube Goldberg contraption. So, we decided to choose a single language, and use it everywhere in the initialization subsystem.

Low level languages like C or C++ are fast but also suffers from maintainability problems. Source codes become even bigger than shell scripts, and development time significantly increases.

Among the high level languages, Python seemed to be the best choice, since we already use it in many places like package build scripts, package manager, control panel modules, and installer program YALI2. Python has small and has clean source codes. Standard library is full of useful modules. Learning curve is easy, most of the developers in our team picked up the language in a few days without prior experience. [ ... ]

Current code size is around 1500 lines. Our previous (Pardus 1.0) Gentoo based system was more than 10000 lines.

Their conclusion:

We have created a whole new distribution with custom package and configuration management tools from scratch in almost two years. We didn't have the know-how of making a distribution when we started, so development time can be considered very fast.

Once we finished the design processes, we quickly build prototypes of the tools and started using them. We even made a few throwaway versions the examine some design decisions. Rapid development nature and quick refactoring capabilities of the Python helped us greatly.

Core developers learned it in a few days, outside developers/packagers, even the users are very quick to learn our tools. Learning curve of the Python is very smooth. You can start using it like Basic, then move on to next levels by learning object oriented and functional programming paradigms.

Even with the problems mentioned in the above chapters, standard library is great, and many important environments like Qt and KDE have excellent Python bindings.

We are not only happy with our choice, but looking forward to use high level languages in more places.

---