The Voidspace Techie Blog

Resilience Against Errors in User Code: Catching String Exceptions

Suppose you have an application that executes user code. You want to be resilient against errors in that code, you don't want to crash if exceptions are raised in user code, but you want to be able to report the errors.

You might think that the following code would be a good start:

You can find plenty of Python experts recommending that where you want to catch all possible exceptions you should explicitly catch Exception rather than using a bare except clause. Here is one example (of many) with Guido agreeing back in 2005.

However, what happens if the user raises a string exception:

Python 2.5.1 (r251:54863, Jan 17 2008, 19:35:17)
[GCC 4.0.1 (Apple Inc. build 5465)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> try:
...   raise 'freshfoo'
... except Exception:
...   print 'Caught!'
...
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
freshfoo
>>>

The exception handling block does not catch the string exception.... ouch...

A bare except does catch it, and you can then fetch information about the exception using sys.exc_info():

>>> try:
...   raise 'freshfoo'
... except:
...   print sys.exc_info()
...
('freshfoo', None, <traceback object at 0x...>)
>>> try:
...   raise Exception('blam!')
... except:
...   print sys.exc_info()
...
(<type 'exceptions.Exception'>, Exception('blam!',), <traceback object at 0x...>)

sys.exc_info() returns a tuple. For 'normal' exceptions the first value is the exception type. For string exceptions it is the string itself. The second value is the exception instance, or None for string exceptions. The third member is the traceback object.

An interesting snippet from the Differences Between IronPython 1 and Python 2.4 document mentions that IronPython catches string exceptions by value (strings are value type objects in .NET - allocated on the stack rather than the heap) whereas CPython catches them by reference. This means that the following works in IronPython but not in CPython:

String exceptions (and raising non-exception objects like instances of old style classes) have been deprecated for some time [1] of course - exactly because of the problems described in this article. They raise deprecation warnings in Python 2.5 (I cut them from the examples here) and are gone altogether in Python 3.0 (see PEP-0352 Required Superclass for Exceptions). However, if you are executing arbitrary user code with a current version of Python then you may still need to be aware of them.

By the way... if you are executing code from strings or compiled code objects, then you may want to consider not using functions directly from the traceback module. It uses linecache, which can potentially make several disk accesses (including one per path in sys.path [2]) for every frame in the traceback trying to find the original sources.

It is easy enough to implement a custom traceback formatter by copying and modifying the extract_tb function from traceback. It takes a traceback object which you can retrieve with sys.exc_info().

[1]	String exceptions are a very old feature of Python. The move away from them started as long ago as Python 1.5.2. Until recently they could still be found in very old codebases like mailman though.

[2]	Compiling code objects with a name surrounded by angle brackets short circuits this.

---

Diagramming on the Mac

One of the annoying things about writing a book is having to create my own diagrams. This was something I wasn't expecting when I started the project, I'm a good writer but awful at producing diagrams.

Thankfully a colleague, Jonathan Hartley, stepped up and helped me.

Here's one of my original diagrams, a 'hedgehog diagram' I produced with 'Paint' (I was still running Windows at home at the time - later I upgraded to Paint.NET which is a much better program but didn't improve my skills):

Here is Jonathan's rendering of the same diagram:

To produce them, he used Open Office Draw. I'm now working on chapter 15 (Embedding IronPython in C# and VB.NET using the DLR Hosting API), and thought I'd give it a try myself.

I used NeoOffice, which is a Mac port of Open Office, and it looks very good. I did try Inkscape, even upgrading my X11 install to the latest version of XQuartz, but it just refuses to run.

Here's, my first attempt:

It's certainly better than my earlier attempts, but I think it still needs some magic from Jonathan.

Several people also recommended OmniGraffle, which looks good, but is not cheap and isn't cross-platform. Given my skill level I think OO offers me everything I need.

Whilst we're on the subject of Mac software, I've also been using a few new programs recently.

• Pixelmator

  Having created the diagram in Neo Office, I used Pixelmator to edit the Tiff graphics file. I think I got Pixelmator included with one of the recent MacHeist bundles. It seems like a very capable program for basic image editing.

• MPlayer

  Yet another Open Source video player. I've been trying to play some high quality mkv (Matroska) files encoded with H.264. Neither Quicktime nor VLC (usually excellent) could play it. MPlayer isn't as polished as VLC, but plays them fine.

• Xee

  Nice little program for image viewing. Much nicer than Preview (which is part of Mac OS X and great for PDFs).

• Cornerstone

  A shiny commercial Subversion front end. I'm trying out the demo version. It seems great so far. I also tried Versions (also in Beta), but it doesn't let you work with existing working directories (you have to checkout through the Versions UI) - so I didn't get very far.

• Transmit

  Nice FTP, SFTP (etc) client for the Mac. Again, commercial but worth it. I couldn't find another client that had a '2-pane' UI, except for FileZilla which just refuses to work on my computer. It dies with an odd error [1] that few other people seem to have, and the fixes suggested for them doesn't work for me.

• Octave Engine Casual

  A new and very funky physics engine from a Japanese developer. Absolutely pointless, but very fun - and very slick on the Mac (and Windows).

• Chmox

  A CHM reader. The CHM (Compiled Help Manual) format for documentation is popular on Windows, and with reason as if well done it can make for very usable docs. Chmox hasn't been updated for a while, but seems to work fine.

[1]	fzsftp could not be started. fzsftp is in the Filezilla package and I tried setting the suggested environment variable. Oh well.

---

Chapter 14: Extending IronPython with C# / VB.NET

I've just finished chapter 14 of IronPython in Action - well, finished a version I can send to my editor anyway. I had more fun writing this chapter than I expected. It's called "Extending IronPython with C# / VB.NET", and is all about writing class libraries in C# or VB.NET and using them from IronPython. This might be for performance reasons, for accessing features of the .NET framework like attributes or Linq that are hard to use from IronPython, or for simply making your class libraries as nice to use from IronPython as possible.

The table of contents for the chapter (pre-edit!) is:

14.1. Writing a Class Library for IronPython

14.1.1. Working with Visual Studio or MonoDevelop

14.1.2. Python Objects from Class Libraries

14.1.3. Calling Unmanaged Code with the P/Invoke Attribute

14.1.4. Methods with Attributes through Subclassing

14.2. Creating Dynamic (and Pythonic) Objects from C# / VB.NET

14.2.1. Creating Dynamic Objects

14.2.2. APIs with Keyword and Multiple Arguments

14.3. Compiling and Using Assemblies at Runtime

14.4. Summary

This is one of the nice things about IronPython, dropping down into C# to extend it is worlds easier than writing C extensions for CPython. There is no C API to have to worry about, and most of the normal C# you write will 'just work' when used from IronPython.

The chapter has five class libraries, each written in both C# and VB.NET. Inevitably in the process I learned more about both these languages.

VB.NET is not so bad, although more verbose than C# (semantically they're very similar of course). My favourite comparison is defining a method that provides iteration. In Python this is def __iter__(self):. In VB.NET it is Public Function GetEnumerator() As IEnumerator Implements IEnumerable.GetEnumerator.

Although I love the dynamic goodness of Python, the C# syntax for properties and indexers is nicer than the Python syntax.

Next onto chapter 15, which is the final chapter. I've been looking forward to writing this chapter for a long time: it is on embedding IronPython in .NET applications and interacting with DLR objects from C# (and VB.NET).

---

Python in the Browser: Live Interactive Interpreter in an HTML Textarea

A while ago I experimented with an interactive Python interpreter that ran in the browser using Silverlight. Unfortunately a couple of bugs with the version of IronPython for Silverlight prevented it from working properly.

Now that Silverlight 2 Beta 2 is out, along with an updated version of IronPython for Silverlight, those bugs have been fixed.

"Python in the Browser" is an interactive Python interpreter that runs in the browser, using Silverlight 2 and IronPython.

• Python in the Browser Project Page

This is ideal for tutorials and documentation, where example Python code can actually be tried in the browser.

You can see a live demo of the current version at:

• Live Demo of Python in the Browser

It requires Silverlight 2 Beta 2, and the Python version is 2.5. Builtin modules are available, but I haven't made much of the standard library available in the demo (they simply need to be added to the 'xap' file containing the silverlight application.)

Although the version in the repository works, it has some known limitations. See the issues page for the known ones so far.

The interpreter runs in an HTML textarea, with Javascript that communicates with Silverlight and prevents you deleting text from the console except after the interactive prompt.

Target browsers are Firefox 2 & 3, Safari and IE 7 & 8. (It won't work in other browsers until there is a version of Silverlight that works with them.) Silverlight 2 is currently only available for Windows and Mac OS X, with the Firefox, Safari or IE browsers. Linux support is in the works via the Moonlight project from Mono.

The project is a combination of IronPython (for the interpreter loop), Javascript (for the 'console behaviour' in the textarea) and C# (as a helper to call into Silverlight from Javascript). On every keypress Javascript calls into IronPython (via the C#!). If the keypress is an 'enter', then it pushes the current line into the interpreter loop (which uses the standard library code module). Stdout is diverted to print into the textarea, where tracebacks are also sent. If you are attempting to type over, or delete, previous output then the keypress is cancelled.

---

You Don't Really Need It

Python is my favourite language, by a very long way. That's not hard though, I've only done serious programming in Python, C# and Javascript and am familiar with the syntax (to lesser or greater degrees) of C, VB.NET and Ruby.

I program in the IronPython dialect of Python for my day job and for the book I'm still writing on, but CPython still has my heart. I love writing the examples for my book as I prefer coding to writing (writing still comes in a close second though), and I'm really looking forward to finishing the book as I will be able to get back to hacking on projects with CPython.

That aside, IronPython does have some advantages over CPython. The threading model in IronPython (based on .NET threads of course) is better than CPython, there is no GIL so multiple threads can scale across multiple cores and you can terminate threads. Additionally, extending IronPython from C# is worlds easier than writing C extensions for CPython. Writing C extensions for CPython may be relatively straightforward, apart from the usual issues with writing in C, but C# is quite a nice language - you still get the speed advantages, but you have a modern object oriented language with memory management. More importantly, IronPython can use .NET objects natively, so no need to use a C API to interact with Python objects [1].

Anyway, this is not at all what I intended to write about.

In general the Python community is friendly and helpful, but there are certain topics that when people ask how to do them 'the community' [2] tends to reply with 'you don't really need to do that'.

Obviously in some circumstances that is the right response, but in other situations it seems to be a knee-jerk rationalisation - this is difficult or impossible to do in Python, therefore not wanting to do it is the right answer.

Topics that come under this category include:

• Compiling as a way of obscuring source code.

  IronPython 1 can compile source to IL bytecode and Resolver One does use this to protect our source code - it isn't clear whether this will ever be directly possible with IronPython 2.

  Often this question is asked by newbies, where the "you really don't need to do that" response is appropriate - but not all reasons for wanting to protect your source code are nefarious. I'm sure that Python has lost potential users due to the you're evil or stupid for even wanting to do this type responses that you sometimes see on comp.lang.python.

  The other response is "even if you compile down to obfuscated machine code you still haven't really achieved your goal because it's theoretically possible for someone to unpick that", which is one of those technically correct but not helpful or useful responses...

• True concurrency (across multiple cores)

  The standard response in Python, is that if you need concurrency across multiple cores (or even multiple threads that really run concurrently) then what you really want is to run multiple processes. For some problems that is a really good answer, but for others it is a really bad answer (typically those that would require a lot of data or object graphs to be marshalled in and out of the multiple processes).

• Terminating threads

  Doesn't come up very often, but Python doesn't let you do it because it doesn't think you really want to...

  See this short thread on threading or my article on threading in IronPython for some of the issues.

• Private members

  See this thread on comp.lang.python.

  This is something that isn't possible from Python, you can obscure things but not hide them altogether. And in general you don't need to anyway, certainly orders of magnitudes less than the proponents of typical statically typed languages would suggest.

  However, just because you don't need it very often doesn't mean that there are no possible good reasons for wanting it. I think that both sides of the debate (from the very long thread referenced above) are neatly summed up in:

  I understand the difference between the documented interface of a system and the details of its implementation. But sometimes it can be useful to take advantage of implementation details...

  By enforcing your 'data hiding', you're effectively telling me that I'm too stupid to make rational decisions of this sort. And that's actually extremely insulting. -- Mark Wooding

  By telling me that I can never have a valid reason for wanting 'data hiding' you're effectively telling me that I'm too stupid to make rational decisions of this sort. And that's actually extremely insulting. -- me

  The "programmers aren't stupid" philosophy has to cut both ways...

  Having said that, the argument that if it is present in the language it will be abused is compelling, that doesn't mean it can't be a problem occasionally.

UPDATE: The point of this entry is not to point out areas where CPython needs to change. Some changes are not realistic, and some (private members for example) are probably on balance a bad idea. The issue is that the community realise that some questions are not best answered by "you don't really want to do that", and that for some use cases CPython does have limitations.

In fact for several of these questions there are answers. For 'compiled distributions' you can just distribute '.pyc' files and this is often obfuscated enough for people's needs.

For private members / hidden data, you can write in C and not expose stuff. This isn't ideal of course, but it can be done.

The threading situation is more difficult. In the long run I would like to see (perhaps) PyPy become a viable alternative. In the meantime alternative implementations like Jython and IronPython don't have the same restrictions as CPython (but can have their own drawbacks of course - mainly lack of access to standard C extensions).

There's a good response from Tim Golden by the way: Even if you do need it.

I removed the reference to 'freetards' as I don't think it improved this entry. For those unfamiliar with the term, it is used a great deal by Fake Steve Jobs and is a derogatory categorisation of those zealously enthusiastic about Open Source and free software. The irony of me using it, is of course that I'm as much a freetard as the next Python user...

[1]	There are some details that help when writing class libraries in C# / VB.NET for use from IronPython, which is the subject of chapter 14 of IronPython in Action which I'm just finishing...

[2]	A word which out of context means nothing of course...

---

Ironclad 0.4 Released: Use CPython Extensions from IronPython

William Reade of Resolver Systems has announced a new release of our Ironclad project.

• Ironclad Home Page
• Download Binaries and Source for Ironclad
• Ironclad Mailing List

Ironclad is a project to enable you to use CPython extensions implemented in C seamlessly from IronPython. This basically involves re-implementing the Python C API in C# (along with reference counting and garbage collection strategies that span the native code in the C extension and the managed code that lives in the .NET Virtual Machine).

This release comes a bit sooner than expected as William has got the import hook working. This means that after executing import ironclad in IronPython (2), you can import any C extensions on your path just as you would from CPython!

Here's the details from the announcement:

Ironclad v0.4 has now been released. Properties and (some) members on unmanaged types should now work, and it's now much easier to use -- by starting ipy.exe in Ironclad's build directory, you should now be able to 'import ironclad' and then attempt to import CPython extensions as you would in CPython. Note that they aren't likely to work very well yet, if at all, because so much of the CPython API is still unimplemented.

Our intention is that the next release, v0.5, should allow people to import numpy and do something incredibly trivial with it. It may be some time before we understand numpy well enough to achieve this, so we aren't in a position to commit to a release date; but we'll provide updates as we progress.

---

Resolver One: Major New Release, Version 1.1

This release represents four months of work and includes performance improvements, new features and bug fixes for Resolver One.

Resolver One is an IronPython based spreadsheet development environment (fully programmable, object-oriented spreadsheet). It is designed for creating business applications using the familiar spreadsheet interface. Resolver One is free to use for non-commercial and open source purposes, and very cheap for commercial uses.

• Resolver One 1.1 New Features (3 Minute screencast)
• Resolver One Free Download
• News Announcement
• Full CHANGELOG
• Resolver Hacks Website: Fun & Interesting things to do with Resolver One

The new features include:

• Significant improvements to performance and memory usage.
• Cutting and pasting is now more "spreadsheet-like".
• For the financial edition, we've added Thomson Dataworks Enterprise connectivity and a number of great enhancements to Bloomberg access.
• Better coverage of standard spreadsheet functions.
• Auto-indent in the code editor.
• Comments in cells.
• Many bugfixes (including several different obscure ways of crashing Resolver One).
• Menu and toolbar item for unpacking arrays into selection (and repacking). You can use this to unpack any iterable.
• You can provide a 'formatter function' for cells (or rows, cols or whole worksheets) which is used to format the displayed value in the cell. Very nice!
• New 'Color' trait for the colour of text in cells. (Boring but needs to be there.)

The full changelog is surprisingly long. It's nice to see that we haven't wasted the last four months, even though progress has felt slow at times. We have a huge list of different things we want to do with Resolver One, plus different directions we could take it in. It is good that we now have some customers using Resolver One and giving us valuable feedback about the most important things we need to work on.

Even since cutting this release we have been working on cool new features. One of my favourites is the ability to create and manipulate drop-down lists from code as well as from the user interface.

---

London Geek Knight on IronPython

The London Geek Knights hold regular geek meetings on a range of subjects in the London Thoughtworks offices. They're open to all, and on 24th July I'll be presenting on IronPython and Silverlight.

According to the upcoming page, I will be will be giving a talk about how to bring IronPython and Silverlight together in a marriage of awesomeness.

See you there at 6pm on the 24th.

---

IronPython in Action: Silverlight Chapter Finished

I've finally finished the Silverlight chapter of IronPython in Action. When I started writing this chapter I thought it would be done quickly as I new the material I wanted to cover. Rather than just go over material that I have already written though, I decided to write a small application to illustrate Silverlight. I ended up writing a Silverlight Twitter client, and writing it up took a bit longer than I anticipated!

The (unedited) table of contents for the chapter is:

13. Silverlight: IronPython in the Browser

13.1. Introduction to Silverlight

13.1.1. Silverlight the Browser Plugin

13.1.2. Dynamic Silverlight

13.1.3. Your Python Application

13.1.4. Silverlight Controls

13.1.5. Packaging a Silverlight Application

13.2. A Silverlight Twitter Client

13.2.1. Cross Domain Policies

13.2.2. Debugging Silverlight Applications

13.2.3. The User Interface and a Password TextBox

13.2.4. Accessing Network Resources

13.2.5. Threading and Dispatching onto the UI Thread

13.2.6. IsolatedStorage in the Browser

13.3. The Silverlight APIs

13.3.1. The MediaElement Video Player

13.3.2. Interacting with the Browser DOM

13.4. Summary

I really enjoyed writing this chapter, and played with some new things including the VideoBrush. Although it is only thirty pages I think this chapter achieves its goal of being a good introduction to programming with Silverlight.

Silverlight is still in beta, and still has a few quirks to be ironed out. The latest one I've found is that the MediaElement doesn't raise the MediaEnded event (on Safari on the Mac anyway) when it reaches the end of a video. As the MediaElement doesn't have a 'loop' setting, you need this in order to make videos auto-repeat. You can work around this by monitoring the CurrentStateChanged event instead. Overall though programming Silverlight is a great experience, being able to script the browser with Python is really fun.

The next chapter is on extending IronPython with C# and VB.NET. This is basically writing class libraries of objects in C# that behave dynamically (like true Python objects) when used from IronPython. After that it is on to embedding the IronPython engine (and other DLR languages like IronRuby) in .NET applications. I'm particularly looking forward to writing these chapters, all the more so because these are the final two chapters.

---

Ironclad 0.3 Released (Use CPython Extensions from IronPython)

William Reade of Resolver Systems has just announced the release of Ironclad version 0.3:

• Ironclad Home Page
• Download Binaries and Source for Ironclad
• Ironclad Mailing List

Ironclad is an open source project that aims to allow you to seamlessly use CPython C extensions from IronPython. It basically provides a reimplementation of the CPython API in C#, and maps the C extension (unmanaged code) to creating IronPython objects (managed objects) rather than CPython objects.

Ironclad targets IronPython 2 and Python 2.5.

The project is still in its infancy (only part of the C API is complete), but we have implemented sufficient for most of the CPython bz2 module to be used. Functions, types and their methods (including the file types [1]) can be used from this module. Class members and properties cannot yet be used and is the target of the next release (before we move onto getting Numpy working - the 'secret goal' of the project).

The major advancement in this release is that object lifetime is now handled basically correctly.

When you create a "managed object from an unmanaged type" (and we need a better way of saying that: when you use an object from a C extension basically), some 'unmanaged resources' (i.e. memory) are allocated.

Now, when the managed object is garbage collected this memory is correctly freed and the unmanaged destructor (tp_dealloc) for the type is called. This means that Ironclad doesn't leak [so much] memory and file descriptors owned by the unmanaged side are properly closed on garbage collection.

This is implemented with an interesting strategy. We basically maintain our own reference count for objects. When an object (a "managed object of unmanaged type") is created, we set its reference count to 1 (using a weak reference in the mapper so that the reference counting alone doesn't keep it alive on the managed side). Setting the reference count to 1 means that the destructor won't be called when the unmanaged side (the C extension) has finished with the object and is no longer holding any references to it.

The 'reference counting' for the unmanaged side, is simply a small block of memory storing an integer which corresponds to how many 'unmanaged references' there are to it. An unmanaged reference is not exactly the same as a managed reference (a possible cause of terminology confusion). This means that we are using real reference counting (in the same way as CPython) on the unmanaged side.

When there are no more references to it on the managed side (the IronPython world), the finalizer is called which checks the reference count in our mapping. If the reference count is more than 1 then we know that the extension itself still has a reference to the object and so the finalizer resurrects the object by storing a strong reference in the mapper.

We then regularly check (every time a managed object is finalized - but we may find a better strategy) if the reference count of any objects we have promoted to strong references have dropped back down to 1. If they have, we demote the strong reference back to a weak reference so that the object can be garbage collected.

In this way Ironclad mixes the reference counting that the C extensions use with the garbage collection of the .NET framework. The only downside is that if our "managed object of unmanaged type" itself has any references to managed objects, then their finalizers may have already been called at the point at which we resurrect it. Hopefully this is a pathological enough case that it won't bite us for a long time to come...

Currently Ironclad is only tested on Windows with .NET, but it deliberately uses the gcc compiler and a (theoretically) platform independent approach so that it can be ported to Mono with 'minimal' effort. Obviously the source distribution comes with full tests, and pretty good documentation.

[1]	Getting access to the 'real' file descriptor from a .NET stream is possible, but 'fun'.

---

Programming with Silverlight: Bugs and Missing Features

I've spent the last week writing a chapter on programming Silverlight with IronPython (which is how I managed a whole week without blogging!).

Despite the title of this blog entry it's actually great fun. As Silverlight contains the 'core-clr' it has a wide range of powerful APIs built in. The example for the chapter is a Silverlight Twitter client. To parse the XML responses from the Twitter API I was able to reuse the XML DocumentReader I wrote for chapter chapter five of the book, where it was written for the example desktop application. I literally only had to change one line - to handle the XML declaration.

Whilst writing the chapter I have come across several bugs and quirks in Silverlight 2. Silverlight 2 is still in beta, so hopefully most of these problems will be addressed.

The Bugs

• The SelectionLength on a TextBox is always 0 when you access it in subclasses of TextBox.

• The HttpWebRequest asynchronous API is outstandingly odd. It can also throw errors that are impossible to catch (in between the async events) if the server is slow to respond.

• When checking the boolean value on a CheckBox.IsChecked property, it can throw the following slightly amusing error:

  ArgumentException: Cannot cast from 'System.Nullable`1[System.Boolean]' to 'System.Boolean'.

  Comparing to True instead of doing a boolean test works, so it is a harmless bug.

• Setting HyperlinkButton.TargetName = '_blank' causes the HyperlinkButton to stop working altogether on Safari on the Mac.

• Because of a bug in Firefox, you can't use 'Chiron' (the development tool) with Firefox on the Mac. It works with Safari though.

• Exception reporting can be very limited, especially for errors thrown inside dispatched functions. You can (apparently) remedy this on Windows with Visual Studio and the Silverlight SDK, but not for development on the Mac.

Missing Features

• No docking (and HorizontalAlignment.Stretch doesn't really work), meaning that I still have to use absolute sizes to layout user interfaces. I may be missing something with this though.
• No wrapping on the TextBox control.
• No menus or tab panel. (One of the things I hate about many Flash and Javascript applications is the lack of a context menu.)
• No password textbox (and because of the SelectionLength bug the one I implemented works, but can behave oddly).
• No selection (and therefore no copy and paste) from the TextBlock control.
• The ScrollViewer doesn't track the mouse scroll button, which is surprisingly annoying.
• No way of doing basic authentication (you can't set the Authorization header on HttpWebRequest objects). It might be possible to implement this yourself using the socket APIs of course...
• WebClient caches 'GET' requests, so if you use it to fetch data that might update then you're out of luck.

---