The Voidspace Techie Blog

Chapter 14: Extending IronPython with C# / VB.NET

I've just finished chapter 14 of IronPython in Action - well, finished a version I can send to my editor anyway. I had more fun writing this chapter than I expected. It's called "Extending IronPython with C# / VB.NET", and is all about writing class libraries in C# or VB.NET and using them from IronPython. This might be for performance reasons, for accessing features of the .NET framework like attributes or Linq that are hard to use from IronPython, or for simply making your class libraries as nice to use from IronPython as possible.

The table of contents for the chapter (pre-edit!) is:

14.1. Writing a Class Library for IronPython

14.1.1. Working with Visual Studio or MonoDevelop

14.1.2. Python Objects from Class Libraries

14.1.3. Calling Unmanaged Code with the P/Invoke Attribute

14.1.4. Methods with Attributes through Subclassing

14.2. Creating Dynamic (and Pythonic) Objects from C# / VB.NET

14.2.1. Creating Dynamic Objects

14.2.2. APIs with Keyword and Multiple Arguments

14.3. Compiling and Using Assemblies at Runtime

14.4. Summary

This is one of the nice things about IronPython, dropping down into C# to extend it is worlds easier than writing C extensions for CPython. There is no C API to have to worry about, and most of the normal C# you write will 'just work' when used from IronPython.

The chapter has five class libraries, each written in both C# and VB.NET. Inevitably in the process I learned more about both these languages.

VB.NET is not so bad, although more verbose than C# (semantically they're very similar of course). My favourite comparison is defining a method that provides iteration. In Python this is def __iter__(self):. In VB.NET it is Public Function GetEnumerator() As IEnumerator Implements IEnumerable.GetEnumerator.

Although I love the dynamic goodness of Python, the C# syntax for properties and indexers is nicer than the Python syntax.

Next onto chapter 15, which is the final chapter. I've been looking forward to writing this chapter for a long time: it is on embedding IronPython in .NET applications and interacting with DLR objects from C# (and VB.NET).

---

Python Magazine: Reading Configuration Files with ConfigObj

The May edition of the Python Magazine is out; and technically it's still May so that counts I guess. The major articles are:

• Handling Configuration Files with ConfigObj
• LDAP backed initScripts in Python
• Learning Python with PyGame: The Simplest Thing that Can Possibly Work
• Writing a Simple Interpreter/Compiler with Pyparsing

Both the PyGame and and Pyparsing articles look interesting (I'd love to write a compiler sometime - it's on my list but I might settle for working on PyPy once my book is finished), but of course the most interesting one is the article on ConfigObj - mainly because I wrote it.

As the article explains, ConfigObj is a wonderful module. It makes easy things simple but has some very advanced features. One of its more advanced features is the validation system that allows you to provide default values for your config files and convert all the members into the type you expect them to be.

Whilst the documentation for this feature is complete (unlike many programmers I quite like writing documentation), it is split across two modules: ConfigObj and validate. The Python Magazine article is probably now the most comprehensive introduction to this ConfigObj feature!

---

IronPython in Action: Silverlight Chapter Finished

I've finally finished the Silverlight chapter of IronPython in Action. When I started writing this chapter I thought it would be done quickly as I new the material I wanted to cover. Rather than just go over material that I have already written though, I decided to write a small application to illustrate Silverlight. I ended up writing a Silverlight Twitter client, and writing it up took a bit longer than I anticipated!

The (unedited) table of contents for the chapter is:

13. Silverlight: IronPython in the Browser

13.1. Introduction to Silverlight

13.1.1. Silverlight the Browser Plugin

13.1.2. Dynamic Silverlight

13.1.3. Your Python Application

13.1.4. Silverlight Controls

13.1.5. Packaging a Silverlight Application

13.2. A Silverlight Twitter Client

13.2.1. Cross Domain Policies

13.2.2. Debugging Silverlight Applications

13.2.3. The User Interface and a Password TextBox

13.2.4. Accessing Network Resources

13.2.5. Threading and Dispatching onto the UI Thread

13.2.6. IsolatedStorage in the Browser

13.3. The Silverlight APIs

13.3.1. The MediaElement Video Player

13.3.2. Interacting with the Browser DOM

13.4. Summary

I really enjoyed writing this chapter, and played with some new things including the VideoBrush. Although it is only thirty pages I think this chapter achieves its goal of being a good introduction to programming with Silverlight.

Silverlight is still in beta, and still has a few quirks to be ironed out. The latest one I've found is that the MediaElement doesn't raise the MediaEnded event (on Safari on the Mac anyway) when it reaches the end of a video. As the MediaElement doesn't have a 'loop' setting, you need this in order to make videos auto-repeat. You can work around this by monitoring the CurrentStateChanged event instead. Overall though programming Silverlight is a great experience, being able to script the browser with Python is really fun.

The next chapter is on extending IronPython with C# and VB.NET. This is basically writing class libraries of objects in C# that behave dynamically (like true Python objects) when used from IronPython. After that it is on to embedding the IronPython engine (and other DLR languages like IronRuby) in .NET applications. I'm particularly looking forward to writing these chapters, all the more so because these are the final two chapters.

---

Programming with Silverlight: Bugs and Missing Features

I've spent the last week writing a chapter on programming Silverlight with IronPython (which is how I managed a whole week without blogging!).

Despite the title of this blog entry it's actually great fun. As Silverlight contains the 'core-clr' it has a wide range of powerful APIs built in. The example for the chapter is a Silverlight Twitter client. To parse the XML responses from the Twitter API I was able to reuse the XML DocumentReader I wrote for chapter chapter five of the book, where it was written for the example desktop application. I literally only had to change one line - to handle the XML declaration.

Whilst writing the chapter I have come across several bugs and quirks in Silverlight 2. Silverlight 2 is still in beta, so hopefully most of these problems will be addressed.

The Bugs

• The SelectionLength on a TextBox is always 0 when you access it in subclasses of TextBox.

• The HttpWebRequest asynchronous API is outstandingly odd. It can also throw errors that are impossible to catch (in between the async events) if the server is slow to respond.

• When checking the boolean value on a CheckBox.IsChecked property, it can throw the following slightly amusing error:

  ArgumentException: Cannot cast from 'System.Nullable`1[System.Boolean]' to 'System.Boolean'.

  Comparing to True instead of doing a boolean test works, so it is a harmless bug.

• Setting HyperlinkButton.TargetName = '_blank' causes the HyperlinkButton to stop working altogether on Safari on the Mac.

• Because of a bug in Firefox, you can't use 'Chiron' (the development tool) with Firefox on the Mac. It works with Safari though.

• Exception reporting can be very limited, especially for errors thrown inside dispatched functions. You can (apparently) remedy this on Windows with Visual Studio and the Silverlight SDK, but not for development on the Mac.

Missing Features

• No docking (and HorizontalAlignment.Stretch doesn't really work), meaning that I still have to use absolute sizes to layout user interfaces. I may be missing something with this though.
• No wrapping on the TextBox control.
• No menus or tab panel. (One of the things I hate about many Flash and Javascript applications is the lack of a context menu.)
• No password textbox (and because of the SelectionLength bug the one I implemented works, but can behave oddly).
• No selection (and therefore no copy and paste) from the TextBlock control.
• The ScrollViewer doesn't track the mouse scroll button, which is surprisingly annoying.
• No way of doing basic authentication (you can't set the Authorization header on HttpWebRequest objects). It might be possible to implement this yourself using the socket APIs of course...
• WebClient caches 'GET' requests, so if you use it to fetch data that might update then you're out of luck.

---

ironpythoninaction.com: New Chapters and Sourcecode Available

Three new chapters have been added to the Manning Early Access Program for IronPython in Action. There are now eleven out of fifteen chapters available. The new chapters are:

• Chapter 9: WPF and IronPython

• Chapter 10: System Administration with IronPython

  Covering shell scripting, WMI and Powershell (both embedding Powershell in IronPython and IronPython in Powershell).

• Chapter 11: IronPython and ASP.NET

  This is the first of two chapters written by Christian Muirhead.

I'm now working on the Silverlight chapter, whilst Christian is busy writing the databases and web services chapter.

I've put up a website for the book:

• Ironpythoninaction.com

The website has the sourcecode to accompany the first eleven chapters available for download, and a table of contents for everything done so far:

• Download Sourcecode
• Table of Contents for Chapters 1-11

---

Amazon Simple DB with IronPython and Resolver One (plus Other News)

Giles Thomas, the CEO & CTO (or 'dear leader' as he likes to be called) of Resolver Systems has put together some tutorials on using Amazon SimpleDB with IronPython and Resolver One:

• Amazon SimpleDB and IronPython
• Importing from Amazon SimpleDB into Resolver One

Amazon SimpleDB is a cut-down database system, and as such it structures data quite differently to a normal relational database - in a manner that is actually quite similar to a spreadsheet. Each database is structured into "domains", which can be regarded as similar to worksheets. A query to the database can only access a single domain (unlike relational database queries, which can "join" multiple tables). However, the domain is quite freely structured. It is made up on a number of items (which you can see as being similar to rows in the worksheet), each item having a number of attributes (which you can see as being similar to columns). There is no requirement for each row to have the same set of attributes - in relational database terms, it is highly denormalised.

The examples use the Amazon SimpleDB C# Library.

The first Resolver One screencast has now gone up on Showmedo:

• Resolver One in One Minute

We've scheduled time in the next couple of iterations for a series of short screencasts showing how to use some of the best features of Resolver One. Hopefully these will all appear on Showmedo as well.

At Resolver Systems we are in 'code freeze' for the release of Resolver One 1.1. Some of the features I mentioned in my last blog entry aren't quite ready yet, but there are some good features in the list. There will be an announcement soon on the Resolver Systems News Blog.

The IronPython team have also announced that they are aiming to do an updated release of IronPython 1 (which is still the stable version); the new version will be 1.1.2. This should include some good bugfixes and performance enhancements [1], including finally fixing the Python socket problems that have plagued those trying to use urllib and urllib2 from IronPython. The new version of IronPython is scheduled for June, which will probably make IronPython 1.1.2 just in time to include in Resolver One 1.1.1.

I've just discussed chapter ten of IronPython in Action with my editor at Manning. Just a few minor changes (which is a relief). While I do those he is going to be working on getting some more chapters ready for the early access program. With a bit of luck I'll get the edits to chapter ten done quickly enough to include that as well, meaning that the following chapters will be added to IronPython Early Access Program shortly:

• Chapter 9, IronPython and Windows Presentation Foundation
• Chapter 10, System Administration with IronPython (on shell scripting and IronPython integration with Powershell and Windows Management Instrumentation)
• Chapter 11, IronPython and ASP.NET (written by Christian Muirhead)

It's been a while since new chapters have been added, so this is good news!

Tomorrow I'm off to Developer Day Ireland. This is a .NET community event where I'll be talking about Dynamic Languages on .NET and several good friends from the UK (and Europe) .NET community will be there. I'm particularly looking forward to a talk on Functional Programming with C# 3 & 3.5 by Oliver Sturm. The conference is in Galway, and organised by the irrepressible Mick Lohan. On Sunday Mick is taking us on a trip to Craggy Ireland to enjoy the Irish scenery and hospitality.

In the meantime lots of other things are going on.

The LAST planned alpha releases of Python 2.6 and 3.0 are scheduled for next Wednesday, 07-May-2008. There are still several issues marked as Showstoppers

In November 2006 Guido announced what he had been working on - a code review tool for Google called Mondrian. He had hoped that they could Open Source it, but it is so widely used within Google now that it is too closely tied to Google's internal tools to disentangle it.

He has just announced on Python-Dev 'the next best thing':

a code review tool for use with Subversion, inspired by Mondrian and (soon to be) released as open source. Some of the code is even directly derived from Mondrian. Most of the code is new though, written using Django and running on Google App Engine.

He is inviting the Python developer community to try out the tool on the web for code reviews. To try it out, go to:

http://codereview.appspot.com

TechCrunch have also made an announcement (to be taken with the usual healthy dose of salt): that Twitter is migrating away from Rails to solve scaling problems - probably to Java or PHP.

Python has won the Linux Journal reader's 'Favorite Scripting Language 2008' award. Scripting language is not a term that Pythonistas are fond of, and there is an interesting discussion of this on the results page: Linux Journal: Readers' Choice Awards 2008.

The last piece of news is about PyPy. Yesterday an important milestone was reached, PyPy runs the pystone benchmark faster than CPython! And this is without the use of the JIT! Pystone is a crappy benchmark [2], but it is another step closer to PyPy being a useful implementation of Python. Armin should be blogging about this soon, I hope I'm not stealing his thunder but I did hold off for a whole day before this blog entry! Congratulations to all the PyPy team.

[1]	Srivatsn, on the dynamic languages testing team, has just twittered that they have also implemented the _winreg module for IronPython and he has added it to the list of things to go into 1.1.2 (which is no guarantee it will make it, but means it stands a good chance).

[2]	Of course all benchmarks are misleading, just as all generalisations are wrong.

---

Resolver One and IronPython in Action Progress

During conference season I fell behind schedule with writing IronPython in Action, so I've taken some time off work. I've just completed the first draft of "System Administration with IronPython" chapter, with only three more chapters to go. Christian is making good progress on the "Databases and Web Services" chapter, so the book actually might be finished soon...

The system administration chapter was good fun. It covers shell scripting with IronPython, WMI and Powershell. Writing about WMI and Powershell involved a lot of learning. From research to completion of first draft took basically a full week of solid work.

Despite the oddness, WMI turns out to be useful for lots of things. Here's one example (for the rest you'll have to buy the book of course...), listing all the processes that run on startup:

Powershell is an odd language (dynamic scoping anyone?). I've covered embedding Powershell in IronPython and vice-versa. My favourite example in the Powershell section is calling into arbitrary Python code and returning results, with IP 2. This means that you can use Python libraries from Powershell. A combination of the IP 2 hosting API and Powershell oddities (calling generic methods from Powershell hurts) means the code isn't pretty, but the unprettiness is abstracted away of course.

There are two implementations of Powershell in progress for Mono. Pash (Powershell + bash) is already under way which isn't officially part of the Mono Project. An 'official' Mono port called Monoshell doesn't exist yet, but has been accepted as a google summer of code project.

Whilst I've been writing, the Resolver guys have been hard at work. It's been a while since the 1.0.1 release and we're now working towards a 1.1 release (although there may or may not be a 1.0.2 in between). We're attacking Resolver One development on three fronts:

• Improving and extending the integration of Resolver One and Bloomberg and Thomsons' live market data feeds, for the financial users of Resolver One
• Adding features requested by Zeliade Systems for the Resolver One Quant Package
• Adding more features to make Resolver One a better platform for building spreadsheet based applications

The last two of these include things like providing a better code editing experience in Resolver One, more 'standard' spreadsheet features, and also improving performance and memory use. We're currently working on some optimisations which improve general recalc performance by 15-30% (we're already on a 15% improvement and there is more to come), which is great.

As well as the optimisations, new features currently being worked on include custom cell formatters (user defined functions for formatting objects in cells) and comments attached to cells.

Other random trivia:

• Preparations for PyCon UK 2008 have started in earnest and the first post is up on the PyCon UK Blog.

• Microsoft has a new project: Live Mesh which aims to sync data, applications and devices. As usual with new Microsoft technologies, IronPython featured in the demo (about halfway through the The Mesh Video Tour).

• Charles Nutter has a very interesting article on the many different implementations of Ruby that are up, and spring up: Promise and Peril for Alternative Ruby Impls. Other than wrongly stating that IronRuby was the first project to be licensed under the Microsoft Open Source License, it is a good read.

  Interesting to see that some of Ruby community are still sceptical that Microsoft will ever 'allow' Rails to run on .NET. This is despite the fact that Microsoft (well, Jim and Dino anyway) demoed the equally capable web framework Django running on IronPython...

---

UK MSDN Flash: Getting Started with IronPython

The Microsoft UK developer newsletter MSDN Flash, has published an article of mine on Getting Started with IronPython. This is one of their first experiments in publishing articles by non-Microsoft employees!

• Getting Started with IronPython

Here's an extract:

There is increasing developer interest in dynamic languages, and Python is a particularly good one. It is widely used, semantically concise and easy to learn. As a dynamic language, types are enforced at runtime rather than compile time. Developers used to statically typed languages may find this removal of the "safety net" worrying, but in practise it turns out not to be a problem, particularly if you are using best practises for your testing cycles. There is a performance cost for these runtime checks, but dynamic typing makes it easy to do things that are either cumbersome or not even possible with statically typed languages. Benefits include first class (and higher order) functions and types, late binding, easy introspection, duck-typing and metaprogramming. For more on dynamic typing, read Strong Typing versus Strong Testing.

The most important thing about IronPython is how well it is integrated with the .NET framework, and how easy it is to work with managed objects. This makes mixing C# and IronPython very straightforward. Beyond writing full applications, other practical uses for IronPython include embedding it into applications to provide a scripting environment, using it as a standalone language for system administration and utilising it as a tool for exploring assemblies. This last use case is best done with the interactive interpreter, which is also a great place to start experimenting.

For the rest you'll have to read the full article.

---

Report from the ACCU Conference

I've just returned from the ACCU Conference in Oxford. This is an annual occurrence, which started as a C/C++ conference but in recent years has had a thriving dynamic languages track (this year the 'flavour-de-jour' was functional programming). Last year Mark Shuttleworth was one of the keynote speakers, and the year before that the eminent Guido Van Rossum - so after my talk was accepted I was expecting quite an academic and 'high powered' conference (in other words I was very nervous about speaking there and didn't know what to expect).

Of course in reality it turns out that although they do have some very good speakers, it really is a community organised event with some fun and down to earth people. One thing that was a real challenge to my mindset was to spend a bit of time with a genuinely intelligent person (Dirk Griffioen) who chooses to program in C++.

First things first, my talk went very well. I was speaking for an hour and a half on IronPython and Dynamic Languages on .NET. I had a good audience (around fifty people I guesstimate) who were very responsive and asked a lot of questions. About half were .NET programmers and half Python programmers. They seemed to like Resolver One [1], and were impressed by first class functions and decorators in Python. For those who attended, or are just interested, here are some links:

• ACCU 2008 Talk Slides: IronPython and Dynamic Languages in .NET
• IronPython in Action Homepage
• Embedding IronPython 2 in .NET Applications: Examples with Downloads
• IronPython and Silverlight 2 Articles and Examples
• IronPython Articles
• The IronPython Cookbook

My favourite part of the talk was my deliberately provocative statement on static typing:

In statically typed languages, it turns out that a significant proportion of language 'infrastructure' (boiler-plate) is there for the sake of the compiler rather than the programmer.

In C# this includes delegates, interfaces, generics and type declarations which are all obsoleted by a dynamic type system.

Fortunately this was taken with a smile by most people there.

The Thursday keynote was by Simon Peyton-Jones, the creator of the GHC Haskell Compiler. The keynote was on controlling side effects in programming (Caging the Effects Monster.). He wasn't just advocating pure functional programming languages, but was encouraging developers to change their programming habits. He did a great job of explaining how to do this, but whilst he mentioned parallelism I didn't feel he explained why very clearly.

This was followed by Joe Armstrong talking about Erlang and then Simon doing a three hour tutorial session on Haskell - I spent the whole day learning about functional programming! Simon's examples were using Haskell for shell scripting (!) and the xmonad window manager - he was very much touting Haskell as a general purpose programming language. He demonstrated using Monads for IO and was easy to follow (although by the end of three hours my brain was starting to hurt). I've promoted Haskell higher up the list of languages I would like to learn (and a Resolver developer, Kamil Dworakowski, has just started a new blog on Developing an Application with Haskell).

One of Simon's early examples in his tutorial was writing a Haskell function to traverse a graph of silicon atoms to find neighbours (the 'n-shell'). The function was fantastically simple, but was exponential. Simon commented that this could be solved by using memoize. Inspired by this, in my talk I showed how easy it was to write a memoize decorator in a few lines of Python (here is a better implementation).

Whilst chatting to Dirk, he tried to convince me that memory allocation in C++ is simple these days, and that most people who have had painful experiences of C++'s complexity are remembering an older C++ which is now much improved. Unfortunately the Friday keynote, by Andrei Alexandrescu (a C++ expert and a collaborator in the specification of version 2.0 of the D Programming Language), did much to convince me of the opposite. This was an hour and a half of examining, in detail, the terrible problems of trying to implement general purpose identity (lambda x: x in Python for all cases), min and max functions in both the current standard of C++ and C++ 0x (the forthcoming standard) [2]. This includes dealing with rvalues, lvalues, passing and returning arrays, consts and non const values. What a lot of awful, terrible, unnecessary complexity.

I also got a chance to demonstrate Resolver One to Simon Peyton-Jones, who was particularly impressed with the fact that cell ranges are iterable. This means that you can have formulae like =SUM(val for val in A1:D8 if val > 10), or use the filter function with a lambda predicate and a cell range: =SUM(filter(lambda x -> x > 10, A1:D8)) [3]. He says he has been trying to get features like this into Excel for years.

[1]	Don't forget it is free to download and try, and free to use for non-commercial purposes.

[2]	Even at the end of Andrei's presentation of the identity function - which he said took virtually a day to work out - someone in the audience pointed out a corner case it couldn't handle.

[3]	Because the : is used for cell ranges in the Resolver One formula language, lambdas (and dictionaries) use the little arrow instead: ->.

---

Python Fans Take Aim at the Enterprise

Internet News is running an article on 'Python in the Enterprise' (now there is a subject to chill the bones). It is based on interviews with me, Kevin Dangoor and David Goodger.

• Python Fans Take Aim at the Enterprise

Kevin Dangoor:

Python is already a mature technology -- and ready for its close-up in the enterprise.

Dangoor noted that early on in TurboGears' days, he heard from PHP programmers that were looking for more structure. [...] Python uses namespaces heavily, and PHP is just now getting namespaces.

David Goodger:

Python lowers the bar to the point where it's possible for the users to tweak their own software

The source code is easy to understand, the basic tools are all free, and there's no arcane build process to go through.

Python doesn't have a huge marketing machine behind it [...] We use grass-roots methods, word of mouth. Python is not a fad, and there are no gimmicks.

Any competent programmer can easily learn Python and be productive within a few hours or days at most. [1]

Foord (me!):

Python has reached a critical mass of adoption within the past few years

The good news for Python programmers is that demand seems to be exceeding supply, but that will correct itself with time

While Python can stand on its own merits, Foord noted that particularly important for Python adoption is the fact that both Jython (Python on the Java VM) and IronPython (Python on .NET) are now both in a good state.

This means that where the standard technologies are built on Java and .NET, Python is still a viable choice of programming language

[1]	This really has been our experience at Resolver Systems. When we hire, Python experience is never an issue. In fact, I was the fourth developer at Resolver, and the first one with any Python experience.

---

Embedding IronPython 2 Examples

I'm giving a talk at the ACCU Conference 2008 next week, on IronPython & Dynamic Languages on .NET. I've finished my talk slides (well - 3rd or 4th draft at least) and just need to practise now. The talk is one and a half hours long (poor audience), which means we have time to get into Silverlight and embedding IronPython (or other DLR languages) in .NET applications:

• ACCU Slides: IronPython & Dynamic Languages on .NET

The talk includes two examples of embedding IronPython. I've turned those examples into an article:

• DLR & IronPython 2 Hosting Examples

Up until now the DLR hosting API has undergone major changes with almost every release. With the latest IronPython 2 release, the DLR is now in beta and is approaching the kind of stability outlined in the (79 pages of complexity) now fairly out of date DLR Hosting Spec.

Hopefully it is at least stable enough that my examples won't require major surgery for the next release.

The first example is trivially easy. I've finally written a calculator. I call it 'My First Calculator' and it uses IronPython to evaluate numerical expressions you enter through the user interface.

The second example is only slightly less trivial, but perhaps more useful. The IronPython evaluator:

It demonstrates exposing a scripting or plugin API, by publishing objects with events that user code can subscribe to, or fetching objects out of the scope after executing user code.

Both examples are in C# and downloadable as Visual Studio 2008 projects(with binaries). Because IronPython 2 Beta 1 doesn't yet work with Mono, they are Windows only for the moment...

• DLR & IronPython 2 Hosting Examples

---