Planet IronPython

The IronRuby team is pleased to announce a new release of IronRuby: IronRuby 0.6!

Download IronRuby 0.6

You can also get the source code for this release

Staying true to our "Conference-driven development schedule," this release is in conjunction with ... wait, there is no conference this time! Just a good ol' monthly release of IronRuby, chock full of awesomeness. And just so happens to be the 4th of July weekend, so hopefully you have some downtime this weekend and can give the new release a spin.

What's in the Release?

Performance has been a major focus to this release, getting startup time quicker and optimizing built-in types. There has been more progress with Ruby compatibility (Cucumber works!) and .NET interop. Silverlight binaries are back in the release, and there are a couple of samples to help you learn IronRuby.

Most of these descriptions are from Tomas's very detailed code review emails, so thanks Tomas! For more detailed information, please see the CHANGELOG (which includes all commit messages for the release ... not just "syncing to head of tfs")

• Lazy method compilation
• CLR member enumeration
• Generic-Method Parameter-Inference
• Testing C# with Cucumber
• Silverlight building and updated binaries
• Samples

Lazy method compilation

Last release adaptive compilation was enabled for IronRuby, which uses a fast-to-start-up interpreter to start up the program, while code which gets run a lot is compiling on a background thread. This gave IronRuby significant performance improvements for large apps like Rails, which has improved startup by 2.5 times.

This release we've added "lazy method transformation" to the bag of startup performance tricks. In IronRuby 0.5, a method was transformed to the DLR AST as soon as it was defined. IronRuby 0.6 postpones the transformation until the first time the method is called. This significantly improves startup time. For example (not NGEN'd):

This has made Rails startup time 30% faster, coming in approximately 20 seconds on my dual-core laptop running Windows 7.

CLR member enumeration

Now methods like Module#instance_methods, etc, include CLR member names. Though it's a lot cooler than that. The array of methods returns contains strings for Ruby methods, and objects of type IronRuby::Clr::Name to represent CLR methods that can be called by either the actual CLR name or a mangled (Ruby-esk) name. ClrName has methods to_s, to_sym, to_str, <=>, inspect, and dump so that it can be used wherever a string can be used. The display string for the name uses single quotes so that you can easily distinguish CLR (dual) names from regular names (plain mutable strings). CLR strings display themselves as single quoted strings, so this fits nicely.

Even cooler, this works well for meta-programming:

A new set of define_method overloads, strongly typed to ClrName, have been added to enable this. They define the real method using the ruby_name and alias it using the clr_name. So both CompareTo and compare_to calls are intercepted.

Generic-Method Parameters-Inference

Thanks to some work by the IronPython team, IronRuby now has parameter inference for generic methods.

This needs some more TLC before it works perfectly, but this lets you use simple LINQ methods from IronRuby.

Testing C# with Cucumber

This release of IronRuby runs the Cucumber testing framework rather well. Try out Cucumber against some C# code.

Given a feature file (this being the addition.feature)

A step_definition file (calculator_steps.rb)

And a DLL:

Cucumber will test the addition feature:

3 scenarios (3 passed)
12 steps (12 passed)
0m0.753s

Silverlight building and updated binaries

This release re-adds Silverlight binaries to IronRuby, contained in the "silverlight" directory of the release. These bits have been integrated from the AgDLR project, and will be maintained in the IronRuby and IronPython source code repositories from now on. The AgDLR GitHub project will redirect to one of those for binaries for both languages in Silverlight. In addition, the SDLSDK CodePlex site will be redirecting to the IronRuby and IronPython CodePlex sites, for downloads, discussion, and issue tracking. AgDLR was a little side project to add new features to DLR Silverlight applications, and play around with Git. I'm very happy to see it merged back in with the languages.

Note: IronPython Silverlight assemblies are included directly in the release. They may be moved out to the separate package that contains IronPython desktop DLLs in the future.

See the integration commit for more information

Another notable change is that you can now build Silverlight binaries out of IronRuby's GitHub repository very easily. Given that you have Silverlight installed at C:\Program Files\Microsoft Silverlight\2.0.40115.0, this will build IronRuby for Silverlight:

msbuild Merlin/Main/Languages/Ruby/Ruby.sln
  /p:Configuration="Silverlight Release
  /p:SilverlightPath="C:\Program Files\Microsoft Silverlight\2.0.40115.0

You can also build IronPython for Silverlight in a similar manner. Aliases for this will be added soon, but if you need a custom build of the DLR languages for Silverlight, this is the way to do it.

Samples

Six samples are included in this release. The first three are desktop samples in the “/samples” directory. The last three are Silverlight samples in the “/silverlight/samples” directory:

Tutorial - An interactive IronRuby tutorial

DiskUse - A small WPF application which visualizes the disk usage for a particular directory

IRPowerShell - a small library and sample applications showing how to interact with PowerShell from IronRuby

Clock – a simple Silverlight sample

Photoviewer – do AJAX programming with IronRuby (ARAX =P)

REPL – Interactive console in the browser.

Some more interesting changes:

• Improved DLR Interop: adds support for GetMember/SetMember with method_missing, Binary/Unary operators, and indexers.
• Handling of CLR protected and private methods and properties
• Reimplemented File.expand_path such that it does not use System.IO.Path.
• This allows us to get better compatibility with MRI. The motivating reason was
• that RSpec does File.expand_path("filename:linenumber")
• Improves the implementation of singleton method dispatch.
• These changes improve running time of specs significantly (2x)
• Renames Method, UnboundMethod#overloads (plural) to
• Method, UnboundMethod#overload (singular).
• The old name is still available for now and throws an exception.
• Implements adaptively compiled rules
• Improves performance of Array#- from quadratic algorithm to linear.
• Improves implementation of RubyArray

Bugs closed

Here are all 37 bugs closed since the last release (2009-05-19). You can see more information about each one on CodePlex.

1521  Access is allowed to internal fields
1502  alias_method fails for :do
821   File.expand_path does not support a line number after filename
1509  Proc.to_s should include line number where the block was declared
1501  WinForms broken
1400  $? is not always Process::Status	
1345  load_assembly() should work
1344  System.Action.new does not work
1306  Cannot call CLR constructor of builtin type
1184  public(:foo) does not work correctly for mixed-in methods
1085  Cannot call new on subtypes of builtin classes whose "new" method has optional arguments
1060  visibility of send :define_method
917   Passing a Ruby array to a .NET method that expects an IEnumerable derivative fails with GetEnumerator call
783   Assert in SetMethodBasesNoLock when calling #== on Ruby class inheriting from CLR class which overrides Equals
761   Wrong behavior when calling redefined methods on object instances
1470  Can't call the BigIntegerOverload of a method with a DefaultProtocol Attribute on the BigInteger attribute
1426  The located assembly's manifest definition does not match the assembly reference. (ctp dev10 beta1)
1441  Error with cyrillyc text in Sharepoint	
1352  Test Defects	
814   Allocator underfined for  (TypeError)
572   Error when running Cucumber examples with IronRuby
718   IronRuby ignores RUBYLIB environment variable
727   to_proc not working
1351  redist-libs should have rubygems-1.3.1	
466   ''.split(//, -1) returns [""] instead of []	
940   Can't inherit from abstract classes	
1028  Missing conversion from Duration to Fixnum (ActiveSupport)?
374   irails Foo: undefined method for OpenSLL::Random.random_bytes
459   throw FileNotFoundException => rescue Errno.NoEntryError
499   Pathname#cleanpath messes up the pathname
467   "igem install rails" fails
375   "ir script\server" causes a YAML parser error
461   Generic type conversion from Fixnum to Integer
674   autoload does not use File::SEPARATOR
1021  Time class instance
578   yield fails in eval'd code
605   Array#hash should properly handle recursive arrays

Enjoy!

Please test out IronRuby 0.6 and let us know if you have any issues. We hope you enjoy this release!

-- The IronRuby team

The normal procedure for configuring NWSGI leaves an unsightly URL wart: the URL must contain the .wsgi file. There are two ways to get rid of that wart: URL rewriting and wildcards.

URL Rewriting

When using URL rewriting, the web server changes ("rewrites") the incoming URL into something NWSGI can understand. IIS 7 has a URL rewriting extension to do just that. It's quite easy to use, too:

<rewrite> 
    <rules> 
        <clear /> 
        <rule name="Redirect Warty Requests" stopProcessing="true"> 
            <match url="simple.wsgi/(.*)" /> 
            <conditions logicalGrouping="MatchAll" /> 
            <action type="Redirect" url="{R:1}" redirectType="Permanent" /> 
        </rule> 
        <rule name="Remove WSGI Wart"> 
            <match url="^(.*)$" /> 
            <conditions logicalGrouping="MatchAll" /> 
            <action type="Rewrite" url="simple.wsgi/{R:1}" appendQueryString="true" /> 
        </rule> 
    </rules> 
</rewrite>

Replace simple.wsgi with the name of the .wsgi file you would normally use. Now, http://example.com/simple/simple.wsgi/Products/Widgets can be accessed as http://example.com/simple/Products/Widgets. The first rule ("Redirect Warty Requests") will redirect any uses of http://example.com/simple/simple.wsgi/Products/Widgets to http://example.com/simple/Products/Widgets to ensure that only one set of URLs is used (which is important for search engines). However, most modern applications expect to have their URLs rewritten and can be configured to generate the "clean" URLs by default.

If you are using IIS 6, you'll need to use something like ISAPI_Rewrite to achieve the same effect. The following should have the same effect as the above rules:

RewriteEngine On
RewriteRule ^hello.wsgi/(.*) $1 [NC, R=301]
RewriteRule ^(.*)$ hello.wsgi/$1 [NC]

Wildcards

Normally, IIS dispatches requests by extension; this is why we have the .wsgi file in the first place. However, it can be configured to pass all requests to a specific handler by configuring a wildcard extension. Unlike URL rewriting, which is transparent to NWSGI, NWSGI needs to know that it has been configured as a wildcard handler. To do this, add a <wildcard> element to the configuration:

<wsgi>
    <wildcard physicalPath="C:\simple\simple.wsgi" callable="simple_app" />
</wsgi>

The physicalPath and callable attributes have the same meaning as on the <scriptMapping> element. If the configuration includes a <wildcard> element, any <scriptMapping> elements are ignored.

Ultimately, the effect is the same as URL rewriting: http://example.com/simple/simple.wsgi/Products/Widgets can be accessed as http://example.com/simple/Products/Widgets.

Which to Choose

If possible, you should prefer URL rewriting, as it should be faster than using wildcard mappings; however, it requires support from the application. If you're on IIS 6 or IIS 7 without a URL Rewrite extension installed, or your application doesn't support rewritten URLs, then wildcard mappings are available.

The course is (unimaginatively) titled "Programming" (not my doing!). It's at NorthTec at the Whangarei campus (a couple of hours north of Auckland). It's the second programming course in their Bachelor of Applied Information Systems degree. We switched the first course from VB to Python a couple of years ago, but left this one as VB (because it's their introduction to Visual Studio and .NET).

Early last year I started work convincing people that IronPython would be a good choice, since they can continue with a language they (somewhat) know, but still get the intro to the Microsoft toolchain (plus, I hate VB, and dislike teaching in a language I never use in practice). I was expecting that I'd have to continue on without a textbook, until I saw that IronPython in Action was coming out.

BTW, while it's not like there is much choice in the way of books, yours is an excellent fit. The first section is a good Python refresher and .NET introduction. I introduce unit testing to them in the course, and the book handily has a chapter on that. They do a lot of sysadmin work later in their degree, so the chapter on using IronPython / PowerShell is great. They do ASP.NET stuff later on as well, so that'll be a good reference at that time (I don't expect I'll have time to cover that myself). I introduce them to databases and (since a couple of years ago) web services, and there's a chapter on that. Silverlight is an interesting example, and something I expect they'll be interested in given the later web development work, and there's an introduction to that.

Basically, you couldn't have done a lot better if I'd commissioned it! Thanks! - it certainly has made redoing the course material much simpler than if I was working completely from scratch.

As I mentioned in my tweet, I'll put my course material online at some point in the next couple of weeks (it's not quite finished) in case anyone else is interested in it.

Scott Bellware has announced the MonoSpace Conference in Austin Texas on October 27-30th.

Scott has made a Call for Speakers:

The Monospace Conference is looking for teachers to give tutorials on the Mono framework, tools, languages, and platforms supported by Mono.

Some tutorials are aimed at .NET developers with little experience with operating systems other than Windows, and others are geared to experienced Mono developers with exposure to the various Mono platforms.

The tutorials are two hour to three hour interactive sessions that can be any combination of follow-along examples, labs, and lecture.

We're looking for tutorials on subjects such as Linux, Mac, Windows, web, desktop, servers, message queues, databases, iPhone, Android, Amazon's EC2, among others.

You can track the progress of the conference at the MonoSpace Conf Blog.

You can also follow the progress on twitter.

Scott was one of the founders of the Alt.Net series of conferences.

Yesterday we shipped Mono 2.4.2, our long-term supported version of Mono. It ships Microsoft's opensourced ASP.NET MVC stack for the first time (you could get it before on your own, but now it is integrated) and fixes over 150 reported bugs.

Chris Toshok announced M/Invoke a tool to port applications that use P/Invokes on Win32 to Linux and MacOS.

What Chris does not talk about on his post is that he was trying to use some .NET software that interfaces via USB to his glucose meter and was trying to get this to run on Linux. The tool is mostly .NET with the usual handful of P/Invokes to Win32. And this is how M/Invoke was born: a tool to retarget P/Invoke happy applications into becoming pure managed applications.

This opens new doors to forcefully port more apps to Linux.

Alan McGovern released a new version of Mono.Nat one of the libraries used by MonoTorrent.

Jordi Mas released a new version of Mistelix a DVD authoring tool for Linux:

Jordi's GBrainy brain teaser game was picked up by MoLinux, a regional Linux distribution, and shipped it translated to Spanish:

Joe Audette's mojoPortal was being installed four times as much when it got included in in Microsoft's Web Platform Installer site (more stats here).

For years I have loved the Joel on Software rules for software engineering. And one of those rules is "Build in one step". We have not always succeeded, but we have always tried. Lluis delivers the one step to build and run for MonoDevelop on Windows: Load solution, Hit F5, up and running.

Google Chrome really lead the way here, and I want very badly to have all of Mono building in Visual Studio with one keystroke, but we are not there yet.

Stephane reports on some nice startup performance improvements for F-Spot. Loading time for 10 images from Stephane's own image collection went from 1.2 seconds to .5 seconds.

MonoDevelop got some enhanced support for autoconf integration.

Jeremy Laval released another version of ZenComic a desktop Comic reader:

David Siegel announced a new release of Gnome Do on behalf of the Gnome Do team. In particular, it is now easier to write "Docklets" for the Gnome Do panel and for those of us that like the Emacs keybindings, it is now possible to use C-N and C-P for navigation

And of course the Google Summer of Code is in full swing:

And we have various very exciting projects brewing.

Jonathan Pobst has been exploring integration points for Mono and Visual Studio 2010:

Guadec: I will sadly not be attending the Guadec/Akademy conference in Canaria next week. This is going to be a busy summer for us as we are shipping a lot of code in the next few months: Moonlight 2.0, Mono for Visual Studio, MonoTouch 1.0 and Mono 2.6.

Function Callables

Let’s say we’re dealing with a much larger Python application, coolapp, that is installed at C:\coolapp-1.1\. Unfortunately, coolapp doesn’t provide a ready-made .wsgi file for us, but it does provide a function(coolapp.dispatchers.wsgi_application) that we can use by writing a little wrapper that looks something like:

import sys
sys.path.append("C:\coolapp-1.1")
import coolapp.dispatchers.wsgi_application
application = coolapp.dispatchers.wsgi_application

We could save this as coolapp.wsgi and follow either of two methods from last time, but since we’re not doing anything fancy in the wrapper, NWSGI provides a shortcut:

<wsgi>
    <pythonPaths>
        <path path="C:\coolapp-1.1" />
    </pythonPaths>
    <scriptMappings>
        <scriptMapping scriptName="coolapp.wsgi" callable="coolapp.dispatchers.wsgi_application" />
    </scriptMappings>
</wsgi>

First, we add C:\coolapp-1.1 to Python’s path so that we can import it. Next, we tell it that any requests for coolapp.wsgi should be dispatched to the function coolapp.dispatchers.wsgi_application. This is completely equivalent to the wrapper file, but saves us from creating it. Visiting http://example.com/coolapp.wsgi/ will run coolapp.

Class Callables

Functions aren’t the only things in Python that are callable; classes are also callable (it creates an instance of the class). Let’s say we upgrade coolapp to version 2.0 (installed in C:\coolapp-2.0). The team has made some changes, and their WSGI application is now a class, WsgiApplication. This class has a __call__ method (making its instances callable as well!), so our WSGI application is an instance of the WsgiApplication class. We could use a coolapp.wsgi wrapper again:

import sys
sys.path.append("C:\coolapp-2.1")
import coolapp.dispatchers.WsgiApplication
application = coolapp.dispatchers.WsgiApplication()

Again, our little wrapper doesn’t do much, so NWSGI provides a shortcut:

<wsgi>
    <pythonPaths>
        <path path="C:\coolapp-2.0" />
    </pythonPaths>
    <scriptMappings>
        <scriptMapping scriptName="coolapp.wsgi" callable="coolapp.dispatchers.WsgiApplication()" />
    </scriptMappings>
</wsgi>

This time, NWSGI will create an instance of WsgiApplication, and then call the instance and return the result to IIS. This shortcut will only work if the class doesn’t require any arguments; if it does, it needs to be wrapped in a .wsgi file (when in doubt, you can always use a wrapper file). Again, visit http://example.com/coolapp.wsgi/ to run the application.

There’s actually one more case: Python classes that implement the iterator protocol, but I don’t think two many applications are implemented that, so I’ll skip it. PEP 333 gives an example if you’re really interested.

NWSGI has a fairly simple task: given a URL, call some Python code that produces some output. The devil, as always, is in the details.

What is WSGI?

NWSGI is an implementation of the Python WSGI specification (PEP 333). WSGI is the Web Server Gateway Interface, “a simple and universal interface between web servers and web applications or frameworks”, as defined in PEP 333.

OK, so what does that mean?

The purpose of WSGI is to define how web servers (i.e. Apache, IIS, etc.) talk to Python web application or frameworks (i.e. Trac, Django, etc.). In theory, a web server with a WSGI implementation (such as mod_wsgi for Apache, or NWSGI for IIS) should be able to run any web application that is implemented as a WSGI application (prior to WSGI, many applications/frameworks were server-specific). Of course, that doesn’t always pan out in practice, but it’s a start.

What is a WSGI application?

A WSGI application is surprisingly simple:

def simple_app(environ, start_response):
    """Simplest possible application object"""
    status = '200 OK'
    response_headers = [('Content-type','text/plain')]
    start_response(status, response_headers)
    return ['Hello world!\n']

The simple_app function is the entire application! A WSGI application is a callable (a function, mostly) that takes two arguments and returns an iterable (a list or generator, mostly). There’s some extra goo in there as well (start_response, for example) that’s not really relevant as far as dispatching is concerned. Writing WSGI applications is probably a book’s worth of content, so I’ll leave it at that for now.

Finding Callables in .wsgi Files

Local Files

Now that we have a callable for our application, NWSGI needs to know how to find it. The simplest way is to simply drop the file (let’s call it simple.wsgi) into the application root (i.e. C:\inetpub\wwwroot\, next to web.config). If you make a request to http://example.com/simple.wsgi/, IIS will see that .wsgi is associated with NWSGI and pass the request to it. NWSGI will then open simple.wsgi, try to find a variable called application (which is, by convention, the name of WSGI callables), call it, and return the result to IIS.

In this case, however, our callable is named simple_app, not application. To be able to run our app, NWSGI needs to know this! To do this, there needs to be a script mapping telling NWSGI how to run simple.wsgi.

<wsgi>
    <scriptMappings>
        <scriptMapping scriptName="simple.wsgi" callable="simple_app" />
    </scriptMappings>
</wsgi>

This configuration tells NWSGI that for the script simple.wsgi, it should use simple_app as the callable instead of application. (NOTE: This exact syntax requires 2.0b2 or later).

Other Files

Best practices for WSGI applications say that you should not put your application files in the web-exposed directory. This means that simple.wsgi should really live somewhere else; let’s say C:\simple\simple.wsgi. Of course, now NWSGI doesn’t have the slightest clue where it is, so we have to tell it:

<wsgi>
    <scriptMappings>
        <scriptMapping scriptName="simple.wsgi" physicalPath="C:\simple\simple.wsgi" callable="simple_app" />
    </scriptMappings>
</wsgi>

There is no longer a .wsgi in our application folder, but IIS doesn’t care. If you visit http://example.com/simple.wsgi/ with this configuration, IIS still happily passes the simple.wsgi script onto NWSGI. NWSGI looks at the script mappings and finds that simple.wsgi maps to C:\simple\simple.wsgi, so it loads that file instead (without looking for simple.wsgi in the application directory), looks up simple_app, and calls it.

Using .wsgi files is the simplest way to deploy an application, but NWSGI has a couple of shortcuts to make things easier for certain applications.

Since doing a post about Ruby testing of Silverlight applications, I’ve felt a bit guilty about leaving Python fans in the dark. However, like Ruby, it takes very little work to run Python unit test frameworks in Silverlight, since IronPython is a very compatible implementation of Python. Though my results aren’t as exhaustive as the IronRuby examples, here’s one simple example:

http://github.com/jschementi/europycon2009

zip download

The original example is from the Python website’s unittest documentation. Very little bootstrapping is required to get this running in Silverlight; this is u1.py code that is the start-script for the Silverlight app, which runs the tests:

I did have to make a modification to unittest.py; removing a call to sys.exit since Silverlight doesn’t support that, and just returning the exit code.

There’s also a GUI test runner (http://pyunit.sourceforge.net/pyunit.html#GUI), so it might be interesting to port that to Silverlight to get a nicer test run graphic along with the console, but it’s great that the REPL enables this to just work =) It also helped me find some bugs with the REPL, which will make their way to the IronPython source code shortly.

Let me know how it works for you. The next step is to tackle testing Silverlight C#/VB apps with Python.

IronPython 2.0 does not include an implementation of the zlib module, which handles data compression. In CPython the zlib module is a C module, which it can't be used from IronPython. It is also required by several standard Python modules, include tarfile, gzip, and zipfile, all of which handle compressed archives. Thus, for setuptools - particularly, easy_install and zipimport (Eggs) - to work, there must be a fully working zlib module.

FePy includes a partial implementation built on .NET's DeflateStream, but it is missing support for the decompressobj API that is used by gzip and zipfile. Supporting this API using DeflateStream appears to be impossible, as it expects a few implementation details of the zlib implementation.

That left two options for implementing zlib: P/Invoke, and ComponentAce's zlib.net. P/Invoke had a couple of issues: first, native-managed interop is hard; second, handling 32-bit and 64-bit systems seamlessly adds a lot of extra code (I use Vista x64, so I ran into this almost immediately). Zlib.net is an open source (BSD-style license), pure .NET implementation of zlib; this avoids both native-managed interop and 32/64-bit issues. It has an interface nearly identical to zlib, which means that it could be used to mimic the Python C library, which is exactly what I've done.

IronPython.Zlib is an implementation of the zlib module for IronPython using zlib.net. Both binary and source are available under the MS-PL license. To use it create a "DLLs" directory under your IronPython implementation and drop IronPython.Zlib.dll into it.

It passes most of the Python zlib tests and all of the gzip tests. There are a few issues with the CRC-32 implementation that the tests didn't catch but real-world usage did, but I haven't figured out how to fix them yet.

There are still some issues with setuptools (the zipfile module uses binascii.crc32, which is not yet implemented) that I'll detail in a later post, but it almost works.

Update June 27, 2009: Uploaded a new version that fixes crc32 bugs by forwarding to the now-implemented but 2.6-only binascii.crc32.

• "The" IronRuby Tutorial

Isn’t he/she/it a bute!? This app makes Ruby-based tutorials interactive, guiding you through the material with a IronRuby REPL console and other visual aids such as another window, or whatever the tutorial writer sees fit. There is also a tutorial runner for the command-line, and a exporter for static HTML.

The IronRuby tutorial included covers topics such as using the interactive window, built-in libraries, exploring .NET libraries, generics, events, Windows Forms, and Windows Presentation Foundation.

• OS Jam at Google London: C# 4 and the DLR
  

The fundamental point of the DLR is to handle call sites - decide what to do dynamically with little bits of code. Oh, and do it quickly. That's what the caches are for. They're really clever - particularly the L0 cache which compiles rules (about the context in which a particular decision is valid) into IL via dynamic methods. Awesome stuff.

I'm sure the DLR does many other snazzy things, but this feels like it's the core part of it.

At execution time, the relevant binder is used to work out what a call site should actually do. Unless, that is, the call has a target which implements the shadowy IDynamicMetaObjectProvider interface (winner of "biggest mouthful of a type name" prize, 2009) - in which case, the object is asked to handle the call. Who knows what it will do?

• RemObjects Script for .NET: The free scripting engine for .NET

RemObjects Script for .NET is an embeddable engine for .NET applications that allows developers to let their users introduce written scripts that influence the behavior of the application.

The RemObjects Script engine is built in a language-agnostic way, so that different concrete language implementations can be created on top. It currently supports most of ECMAScript 262 version 1.5 (also known as JavaScript). There are also plans to support Pascal Script.

It's based on the Microsoft DLR runtime for simplicity and ease of inter-operability with other dynamic languages and .NET 4.0 languages with support for dynamic operations, but can be run on any version of .NET 2.0 or later, as well as Mono.

Currently supported features:

• Objects
• Prototypes
• Functions
• Arrays
• Booleans
• Regular Expressions
• Numbers
• Build-in classes: Object, String, Function, Array, Number, RegExp, Date, Boolean
• Interopability with non-ECMAScript types
• Interopability with other DLR-based language types

ECMAScript is a dynamic language and is weakly typed. It uses a prototype based object model (no classes). It is a flexible language that due to its lack of types is ideal for scripting purposes. EMCAScript has a very simple model for object oriented programming, where objects are property bags with a "prototype" model to allow for a way of inheritance. It supports closures (nested functions) and every function is an object on it's own.

Note: RemObjects Script is in early development stages, and currently only available from its SVN source control repository at code.remobjects.com. It is implemented using Delphi Prism and the Oxygene language.

I have started porting Twisted to IronPython.

Progress is to be tracked on this Trac ticket.

There is a Buildbot running the Twisted tests with IronPython.

Just in case you don't know, Twisted is an event-driven networking engine written in Python licensed under MIT license, which aims to destroy the American currency system. See following links for more details.

• http://twistedmatrix.com/
• http://isometric.sixsided.org/_/twisted_plutonium/

Adds the in-proc debugging support that Igor as been working on. This adds a new DLL to the IronPython MSI and to the IronPython and DLR CodePlex releases.

This is just adding their source code and updating our build and packaging infrastructure to include their new DLL. The only changes to their code were some #ifdef’s to support Silverlight builds and some updates to the .csproj files to support our numerous builds. This code is owned by them and has been reviewed by their team so I’m not looking for a code review for the new DLL.

Once this is in I’ll be updating Igor’s prototype for sys.settrace support for IronPython and adding that.

When debugging I often find myself wishing I could just watch a single object throughout its life cycle. This is especially useful for debugging state problems (I’m thinking detached row error). Someone is modifying the state of your object and you need to figure out who. The problem lies in scoping. Every time you step in you have to change the watch window statement to track your objects state.

What you end up with is a nasty watch window.

The immediate window comes to the rescue!  Simply declare a variable with a globally unique name (DataRow watchMyRow=_dr;) for example. Then go to the watch window and add a watch for watchMyRow.  watchMyRow is always in scope. So now step through at your leisure and find the offending entry point and step deeper. Its that easy!

So we will start with a simple test program that calls methods with several different names and types.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;

namespace ConsoleApplication2
{
    class Program
    {
        static void Main(string[] args)
        {
            MyObject val = new MyObject();
            Method1(val);
            Method2(val);
            Method3(val);
        }

        static void Method1(MyObject obj)
        {
            obj.state = 1;
        }

        static void Method2(MyObject myobj)
        {
            myobj.state = 2;
        }

        static void Method3(object obj)
        {
            ((MyObject)obj).state = 3;
        }
    }

    public class MyObject
    {
        public int state = 5;
    }
}

We set our break points.

And at the end of the cycle our watch window looks like this! 

All to track the same value on the same object!

The real problem is that with events thrown into the mix it becomes impossible to track state on some foreign object (Detached Row I loathe you!).

Now lets do the same example with the immediate window.

Our first break point gets hit. We type MyObject WATCHMYOBJECT=obj; into the immediate window.

Next I add that variable to the watch window.

Finnally after the many function calls this is the result.

I only had to watch one object the entire time! Now if I can only build a VSADDIN to automate this process (evil grin)!

• I can't install a portable unit from a different GPS manufacturer inside the vehicle.  Err...now that I think about it the portable units install on the windshield or the dash and get powered by cigarette lighter outlets so this is not actually an issue
• The vehicle doesn't come with a map of local electronics stores where I could buy my own portable GPS unit.  Oh...now that I think about it I Toyota's built-in GPS unit will actually HELP me to find these stores selling competing GPS units.  Darn again
  
• My car insurance won't cover the theft of Toyota's GPS unit because it's physically attached to the vehicle.  No, wait...that's not right either.  It's the other way around which actually provides a huge benefit to me
• Ten years ago no cars came with GPS units pre-installed on them and I don't want to pay whatever extra money Toyota is adding to the base vehicle price to cover this GPS.  On that note, I don't want to pay for the heater or useless things like seat belts and parking brakes either

For the most part, NWSGI can be used without any configuration. All you have to do is create a .wsgi file, throw it in a directory, let IIS know what’s going on, and you’re good to go! Of course, if it was always that easy, configuration wouldn’t exist. To really work with NWSGI, you need to understand its configuration. It helps if you’re already familiar with web.config files; if not, check this out (but hurry back!).

First Things First

NWSGI uses .NET’s built-in configuration system (System.Configuration), so the first thing that needs to be done is to let the .NET Framework know about the custom NWSGI configuration section:

<configSections>
    <section name="wsgi" type="NWSGI.WsgiSection" />
</configSections>

This must be the first entry in web.config, directly under the <configuration> element.

The <wsgi> element

All of the configuration options for NWSGI are under the <wsgi> element. The first ones we will look at are specified as attributes of the <wsgi> element: enableExtensions, adaptiveCompilation and frames.

enableExtensions

Enable extensions to NWSGI. Relying on these will make your application nonportable to other WSGI implementations. Required to use ASP.NET's built-in Session support, for example. Choices: true, false.

compilation

Control how IronPython compiles code. Choices: Adaptive, Compiled.

frames

Control sys._getframe. Enabling sys._getframe causes a performance hit. Choices: Off, On, Full.

Example:

<wsgi adaptiveCompilation="false" frames="Full" />

Adding Python Search Paths

If your application needs to access Python modules that are not in the default NWSGI search paths (~/, ~/Bin/ and ~/Bin/Lib/), you can use the <pythonPaths> element to add them. The <pythonPaths> element contains a set of <path> elements that are added (in order) to sys.path before the application is run. The <path> element has one required attribute – path – that contains the full path to a folder on disk containing the modules you want to use.

Example:

<wsgi>
    <pythonPaths>
        <path path="C:\django" />
    </pythonPaths>
</wsgi>

Adding WSGI Environment Variables

Most applications will require some configuration of their own; these options are passed in the environ parameter of the WSGI application. You can add entries to this list by using the <wsgiEnviron> element, which contains a set of <variable> elements. Each <variable> element has two required attributes, name and value.

Example:

<wsgi>
    <wsgiEnviron>
        <variable name="trac.env_path" value="C:\trac\projects" />
    </wsgiEnviron>
</wsgi>

Advanced Script Usage

There are cases where you may need to configure how NWSGI handles a particular script file. In these cases you need a virtual script mapping; NWSGI will always check script mappings before looking for a file on disk. Not surprisingly, these are stored in the <scriptMappings> element, which contains a set of <scriptMapping> elements. There are three parts to a script mapping:

scriptName – required

The name of the virtual script; overrides any scripts on disk.

physicalPath

The path to the actual file to execute. This can be a full path (i.e. C:\myapp\myapp.wsgi) or an app-relative path (i.e. ~/scripts/myapp.wsgi).

callable

If physicalPath is set, the name of the callable object in that file; if it is not set, an entry point for the application.

Entry points are used to specify a particular function in a Python module on sys.path (see the <pythonPaths> element above for how to control sys.path). For example, for Django the entry point would be "django.core.handlers.wsgi.WSGIHandler()". This mimics the standard Python syntax for creating an object; it is this object that is used as the application. If, instead, the application provides a function (like trac), the entry point would be "trac.web.main.dispatch_request".

Example:

<scriptMappings>
    <scriptMapping scriptName="myapp.wsgi" physicalPath="C:\myapp\myapp.wsgi" callable="myapp" />
</scriptMappings>

Wildcard Settings

When IIS is configured in wildcard mode, all requests for an application are passed to NWSGI, instead of only requests for .wsgi files. You can enable this by setting path="*" in the handler/httpHandler definition. Because NWSGI normally expects there to be a .wsgi file in the URL, but in wildcard mode there isn't one, the <wildcard> element is required to tell NWSGI what script to execute. The available settings – physicalPath and callable – have the same meaning as the same settings for script mappings (above). If the <wildcard> element is present, the <scriptMappings> element is ignored.

Example:

<wildcard physicalPath="C:\HelloWorld\hello.wsgi" callable="hello" />

Conclusion

NWSGI is extremely configurable; in many cases, you don't even need a .wsgi file. That said, the configuration can be complex, so please ask if you run into any problems. A later post will go into more detail on the mechanics of callables.

• NWSGI 2.0 Beta 1

This is a major update to NWSGI that makes it easier to install and use and dramatically improves the management story.

So, what happened to 1.1? Originally, NWSGI 1.1 was going to be a refresh of v1.0 that supported IronPython 2.6 and fixed a few bugs. NWSGI 2.0 was going to completely overhaul large parts of the management UI and make some deeper changes to the handler itself. However, I got a week off with some rain and v2.0 progressed much quicker than expected; v1.1 was obsolete before it was even released.

So, v1.1 was dropped and v2.0 will be the first IronPython 2.6-compatible release. It’s easy to make breaking changes when you have no customers.

What’s New

• Easier configuration. In v1.0 there were different ways to configure NWSGI depending on your configuration. Now there is only one way to do it. Some of the more ambiguous options have also been cleaned up, and new ones added to support IronPython 2.6.
• Better management UI. IIS 7 includes a rich UI framework for developing management extensions, but doesn’t have much documentation on it. Some spelunking with reflector showed me better ways to do pretty much everything; not much of the v1.0 management code is left.
• Performance improvements. NWSGI will now cache the WSGI callable instead of reloading it each time.
• Other misc. fixes. I really need to get into the habit of adding a Codeplex issue for each bug so that I can keep track of what I’ve fixed.

• NWSGI 2.0: Installation & Usage

There are two ways to use NWSGI: by installing it or by using xcopy deployment. Each method has its advantages and disadvantages, but their functionality is the same.

• NWSGI 2.0: Configuration Changes

One of the two areas to see the most changes in NWSGI 2.0 was its configuration. Some of the options in v1.0 were overloaded in strange ways; even I wasn’t entirely sure what some combinations would produce. There was also the small problem of where to put the configuration section; it varied depending on how NWSGI was installed. For v2.0, both problems needed to be fixed. Doing so required a re-think of how configuration was handled.

• Which DLR language should I choose? Is it now IronRuby vs. IronPython.

For my part so far I have delved into IronRuby and started to employ it for scripting my C# applications. Why did I choose IronRuby over IronPython? It began with curiosity in the Ruby language in general and led inevitably to IronRuby. I quickly acquired a taste for Ruby but there is something nagging in the back of my mind and I can’t get to the bottom of it - “did I choose the right language” and really after all “does it matter which one I choose?”

Given Pythons relationship to C and my emotional connection to the C family of languages, would IronPython have been a better choice? Certainly IronPython is further down the track and more mature than IronRuby at this stage, but what does that count for at this early stage? All these questions make me curious for some other opinions which I would leave to hear right here, so please tell me – what do you think? What criteria would you use to make a choice?

• Why I chose IronPython over IronRuby in the end.

After having spent a less than insignificant amount of time invested in IronRuby I decided to bite the bullet and invest the effort into IronPython instead and the reasons were entirely pragmatic. Microsoft seem to be putting more effort into IronPython, has already found it’s way into other stacks, such as Oslo’s Intellipad (which I plan to leverage) and it’s significantly more mature than IronRuby.

I am constantly finding uses for IronPython in my work and the list of problems that it’s going to help solve is growing quickly. As a side note, if you are looking to use IronPython and are ready to let the experience pass you by because of lack of IDE support then check out using NetBeans (for Python) and instead using IronPython.

I have been investing some time of late with IronRuby and have started several projects that incorporated scripting into applications written in C# and also have begun a project for a Entity Framework profiler that allows for scripting of my Repositories.

My question is regarding what I feel is a choice right now between IronPython and IronRuby. I am increasingly feeling a bit anxious that IR is slowing me down because of the projects maturity as compared to IP and I am also getting the impression that IP is somewhat preferred within Microsoft itself? I have no experience with Python but feel that if my chosen dynamic language in the .NET space needs to change it should be sooner and not later. I am not looking for a lengthy bullet point IR vs. IP evaluation, just a short high level opinion as to whether my concerns or any others should be heeded. If IP is the way to go I think I know of a good book!

I'm sure you don't expect an unbiased opinion from me, so I assume you have already made your decision and just need me to push you over the edge. ;-)

Actually IronRuby is approaching a 1.0 release soon, so it sounds like it is improving as regards maturity.

Despite this IronPython is more mature as regards .NET interop, performance and bugs. Version 2.6 will be a particularly good release (lots of work on performance) including features like __clrtype__ allowing the use of .NET attributes that I think will take a while before they show up in IronRuby.

Personally I think there are lots of reasons to prefer IronPython over IronRuby:

The Python object model is much closer to C# than Ruby. This means that it performs better and there are less issues around .NET interop - plus it is easier to learn for C# developers.

Python in general is a 'cleaner' language (a purely aesthetic judgement) with clearer namespacing, less use of sigils and no syntactic dependence on the casing of class / variable names. Although Ruby has anonymous blocks they are highly special cased in the language leading to various anomalies in the way they are handled. It is still annoying that Python can't have statements in its lambda functions but there is no clear way to add them to the language (due to using indentation for block structure which despite this disadvantage is still a stroke of genius).

Because Python is more widely used (still - despite all the publicity that Ruby has had through Rails in the last two years or so) there is wider community and more open source libraries available.

Having said all that, both Python and Ruby are great dynamic languages in which it is easy to be productive. They have different facilities for metaprogramming but as far as I can tell they are both equally powerful. Either choice would be a good one but naturally I think Python is the better choice...

One of the things you may notice is missing from the Visual Studio 2010 Beta 1 SDK (compared to 2008) are the user interface guidelines.

You can now find the Visual Studio 2010 User Interface Guidelines at the following location: http://code.msdn.microsoft.com/VS2010UX

• IronPython 2.6 Startup Time Now 34% Faster Under x64

Allow me to elaborate on this perf gain...Dino made a change recently such that ipy.exe is now strictly a 32-bit only assembly. That is, it gets executed as a 32-bit CLR process on both x86 and x64 operating systems. ipy64.exe, despite what its name might imply, is a platform agnostic assembly matching the old behavior of ipy.exe in the sense that it gets executed as a 32-bit process on 32-bit OSes and 64-bit on 64-bit OSes. At least part of the perf gain here comes from the fact that my simple Hello World program is most definitely not consuming anywhere near 4 gigs of memory, and overall memory consumption goes up for 64-bit processes => remember the size of pointers gets doubled. Just checked and the physical memory working set of ipy.exe was 31K compared to 44K under ipy64.exe for the Hello World program. This ~30% difference in memory consumption helps explain why we get better perf under ipy.exe.

• ipy vs CSharp
  

• Python Support in Visual Studio
  

I think I speak for everyone on the IronPython Team when I say we’d love to see IronPython support in Visual Studio by default without the need for any language service addition. The thing is these types of decisions are made at a much higher level than us and are heavily influenced by business needs. What I’m getting at here is if enough people provided feedback via the proper channels that:

• The next version of VS is great, but it’d be better if it supported Python development
• You don’t currently use VS but you would if it had Python support
• Ahem…you’d be willing to pay for IronPython support in VS
• Etc

then the next version of VS could very well have Python support built into it. What are the proper channels for feedback you ask? The VS 2010 feedback page would be a great start.

One of the two areas to see the most changes in NWSGI 2.0 was its configuration. Some of the options in v1.0 were overloaded in strange ways; even I wasn’t entirely sure what some combinations would produce. There was also the small problem of where to put the configuration section; it varied depending on how NWSGI was installed. For v2.0, both problems needed to be fixed. Doing so required a re-think of how configuration was handled.

Configuring Wildcards

In v1.0, configuring NWSGI to operate in wildcard mode required setting options that were designed for other things in particular ways that made no sense. In v2.0, there is a special <wildcard> element under <wsgi> (replacing the wildcardModule option) that is specifically used for configuring these settings. If the <wildcard> element exists, then NWSGI will ignore the <scriptMappings> element.

The callable option has also been removed from the <wsgi> element as it is no longer necessary (it defaults to “application”). To specify a custom callable name for a script, create a <scriptMapping>.

Where to put it?

The first problem was to put the <wsgi> section in the same place all the time. In v1.0 it could be directly under <configuration> (if you were using xcopy deployment) or under <system.webServer> (if you installed it on IIS7). The two could not necessarily coexist in the same web.config file, either! IIS 7 uses a different configuration assembly (Microsoft.Web.Administration) than ASP.NET (System.Configuration), and they’re no compatible, to boot.

The fix is relatively simple: rather than trying to use the appropriate configuration system from the handler, v2.0 always uses System.Configuration. This means that the <wsgi> section is always directly under <configuration>. The only downside is that the section always has to be declared at the top of web.config:

<configSections>
    <section name="wsgi" type="NWSGI.WsgiSection, NWSGI, Version=1.99.1.0, Culture=neutral, PublicKeyToken=41e64ddc1bf1fc86" />
</configSections>

New Options

There are also two new options: compilation and frames. The compilation option controls how IronPython generates code. The frames option is used to control sys._getframe; as tracking frames incurs a performance hit, it must be explicitly enabled.

Wrapping Up

The changes to configuration (which will break any existing configurations) are the main reason for bumping the version to 2.0. The clean up of the options resulted in much cleaner implementations for the management and especially for the handler itself. As always, if you have any comments, let me know.

Hidden in yesterday’s commit to IronRuby’s GitHub repository is evidence of some polishing changes to the IronRuby tutorial app, which I’d like to introduce to you peoples:

To run yourself:

1. Get IronRuby from GitHub and build it (in release mode preferably)
2. cd Merlin/Main/Languages/Ruby/Samples/Tutorial
3. ir.exe wpf_tutorial.rb

Isn’t he/she/it a bute!? This app makes Ruby-based tutorials interactive, guiding you through the material with a IronRuby REPL console and other visual aids such as another window, or whatever the tutorial writer sees fit. There is also a tutorial runner for the command-line, and a exporter for static HTML.

The IronRuby tutorial included covers topics such as using the interactive window, built-in libraries, exploring .NET libraries, generics, events, Windows Forms, and Windows Presentation Foundation.

While the tutorial is currently no where near a complete guide to IronRuby, it’s a great place to get started. Also, the first two chapters of the TryRuby tutorial is included as well for testing reasons, but may or may not be ever finished … unless someone wants to make it work, more on how to do that later.

But it can teach all by itself!

While the content of the tutorials can be awesome, and this app can teach the material very efficiently, it can never teach the real experience of building an app in IronRuby. So the IronRuby tutorial app is written in IronRuby as an example of doing all the things a tutorial can’t. Wow, very meta.

Currently, the tutorial app itself covers:

• User Interfaces in Windows Presentation Foundation
• Ruby as Data: the tutorial domain-specific language
• Testing a Ruby application with test-spec
• Developing a GUI application interactively
• Anything else we can cram into it in the future :)

These area will be covered in a series of future posts, so keep an eye out for them!

Still plenty of work to be done

This app is far from done, so the exploration of IronRuby in WPF has just begun. Here are some known issues, so don’t complain about these =P

• The REPL doesn’t have any history via up/down arrow keys.
• The “Jump to another section” menu scrolls too much. This list can only be two items deep, so it's needs to change to arbitrary depth to have more nesting.
• The content isn’t totally complete, though the IronRuby tutorial is in great shape. If you have suggestions on topics, please let me know.
• Jumping between tutorials a bunch makes the memory usage increase to Rails-like proportions (160MB). Not sure why yet. :(
• The code isn’t the best, but it works. =P I’ll be reorganizing it in future posts about client development, but it serves its purpose for now.

Of course there are a ton of other things that could be done. See the TODO file for some ideas.

Please try it out and let me know what you like about it, what you don’t like, and feel free to futz around with the code as well … contributions are welcome. Enjoy!

There are two ways to use NWSGI: by installing it or by using xcopy deployment. Each method has its advantages and disadvantages, but their functionality is the same.

Installation

The quickest way to get NWSGI up and running is to use the MSI installer. This will add NWSGI to the GAC, making it available for all applications. If you are running IIS 7, it will also install the IIS 7 management components. The installer will also put a copy of the NWSGI assembly along with the README and HelloWorld example in C:\Program Files\NWSGI. This is the preferred approach, unless you do not have administrator rights to the server and cannot convince the server administrator to install it.

xcopy Deployment

The term “xcopy deployment” refers to the Windows xcopy command, which is used to (among other things) copy entire directory trees. Using this method is useful if you can’t (or don’t want to) use the installer package. If this is the case, download the .zip package instead of the MSI installer. To use this method, simply copy NWSGI.dll into your application’s Bin/ directory, and then copy the whole application folder to the server.

Setting Up the Hello World Example

Once you have decided which method you are going to use, the easiest way to check that it is working is to use the HelloWorld example that is included with both packages. I’m only going to describe the most common configuration, and I’m also going to assume you have access to the server; if you have a custom setup (such as a hosted site), just ask.

1. Copy the HelloWorld sample folder to C:\inetpub\wwwroot.
2. Mark C:\inetpub\wwwroot\HelloWorld as an application.
   • IIS 6/7: From the IIS management UI, mark the folder as an application.
   • IIS 7: From an elevated command prompt, run: appcmd add app /site.name:"Default Web Site" /path:/HelloWorld /physicalPath:C:\inetput\wwwroot\HelloWorld
3. Visit http://localhost/HelloWorld/hello.wsgi.

If the browser says “Hello, World!”, you’re good to go! If not, please ask in the forum, and I’ll try to help you out.

• __clrtype__ Metaclasses: Positional Attribute Parameters

The basic infrastructure for custom attributes in IronPython is in place, but it’s woefully limited. Specifically, it only works for custom attributes that don’t have parameters. Of course, most of the custom attributes that you’d really want to use require additional parameters, both the positional or named variety. Since positional parameters are easier, let’s start with them.

Positional parameters get passed to the custom attribute’s constructor. As we saw in the previous post, you need a CustomAttributeBuilder to attach a custom attribute to an attribute target (like a class). Previously, I just needed to know the attribute type since I was hard coding the positional parameters. But now, I need to know both the attribute type as well as the desired positional parameters. I could have built a custom Python class to track this information, but it made much more sense just to use CustomAttributeBuilder instances. I built a utility function make_cab to construct the CustomAttributeBuilder instances.

• Upcoming Presentations - Summer Edition

Cleveland .NET group (Python 101 for .NET developer)

PyOhio 2009 (Pumping Iron into Python + IronPython Show & Tell)

devLink (going cross-platform with C#)

• IronPython and XHTML

• Passing an IronPython method to a C# method that takes a delegate as a parameter

• IronPython String Formatting

• The Hungarian Phrasebook

• IronPython and ArcGIS - not quite ready for prime time

• IronPython (2.6) and ArcGIS - ready for prime time!!

Works perfectly now. So IronPython 2.6 promises to be a viable option for extending ArcGIS. My enthusiasm has been renewed.

In my last post, I added support for custom attribute positional parameters . To finish things off, I need to add support for named parameters as well. Custom attributes support named parameters for public fields and settable properties. It works kind of like C# 3.0’s object initalizers. However, unlike object initalizers, the specific fields and properties to be set on a custom attribute as well as their values are passed to the CustomAttributeBuilder constructor. With six arguments – five of which are arrays – it’s kind of an ugly constructor. But luckily, we can hide it away in the make_cab function by using Python’s keyword arguments feature.

You’ll notice that make_cab now takes a third parameter: the attribute type and the tuple of positional arguments we saw last post. This third parameter “**kwds” is a dictionary of named parameters. Python supports both positional and named parameter passing, like VB has for a while and C# will in 4.0. However, this **kwds parameter contains all the extra or leftover named parameters that were passed in but didn’t match any existing function arguments. Think of it like the params of named parameters.

As I wrote earlier, custom attributes support setting named values of both fields and properties. We don’t want the developer to have to know if given named parameter is a field or property, so make_cab iterates over all the named parameters, checking first to see if it’s a property then if it’s a field. It keeps a list of all the field / property infos as well as their associated values. Assuming all the named parameters are found, those lists are converted to tuples and passed into the CustomAttributeBuilder constructor.

In addition to the change to make_cab, I also updated cab_builder slightly in order to pass the **kwds parameter on thru to the make_cab function. No big deal. So now, I can add an attribute with named parameters to my IronPython class and it still looks a lot like a C# attribute specification.

As usual, sample code is up on my skydrive.

Now that I can support custom attributes on classes, it would be fairly straightforward to add them to methods, properties, etc as well. The hardest part at this point is coming up with a well designed API that works within the Python syntax. If you’ve got any opinions on that, feel free to share them in the comments, via email, or on the IronPython mailing list.

NWSGI 2.0 Beta 1 is now available. This is a major update to NWSGI that makes it easier to install and use and dramatically improves the management story.

So, what happened to 1.1?

Originally, NWSGI 1.1 was going to be a refresh of v1.0 that supported IronPython 2.6 and fixed a few bugs. NWSGI 2.0 was going to completely overhaul large parts of the management UI and make some deeper changes to the handler itself. However, I got a week off with some rain and v2.0 progressed much quicker than expected; v1.1 was obsolete before it was even released.

So, v1.1 was dropped and v2.0 will be the first IronPython 2.6-compatible release. It’s easy to make breaking changes when you have no customers :).

What’s New

• Easier configuration. In v1.0 there were different ways to configure NWSGI depending on your configuration. Now there is only one way to do it. Some of the more ambiguous options have also been cleaned up, and new ones added to support IronPython 2.6.
• Better management UI. IIS 7 includes a rich UI framework for developing management extensions, but doesn’t have much documentation on it. Some spelunking with reflector showed me better ways to do pretty much everything; not much of the v1.0 management code is left.
• Performance improvements. NWSGI will now cache the WSGI callable instead of reloading it each time.
• Other misc. fixes. I really need to get into the habit of adding a Codeplex issue for each bug so that I can keep track of what I’ve fixed.

Give it a spin

The Codeplex wiki needs to be updated to reflect the new documentation, but the included readme is up to date. I’ll also be posting a series of blogs posts over the next few weeks that will explain all the features of NWSGI and provide some helpful examples. Remember, this is a Beta release, so please let me know if you encounter any issues.

Every couple of months we see a bug report on IronPython Studio or someone on the IronPython Mailing List asking why Microsoft doesn’t support Python in Visual Studio.  Most recently there was a pretty lengthy thread about Python intellisense in VS – see http://lists.ironpython.com/pipermail/users-ironpython.com/2009-June/010566.html.  I’ll attempt to shed more light on some facts mentioned in the thread…

A long time ago (2005), in a galaxy far, far away (Microsoft’s building 42) at least one Jedi (aka software engineer) used the Visual Studio Force to put together a Python light saber…err…I mean ‘language service’ for Visual Studio 2005 – see http://lists.ironpython.com/pipermail/users-ironpython.com/2005-December/001375.html for background.  A ‘language service’ is simply an extension to Visual Studio that makes it aware of one or more file extensions denoting a programming language Visual Studio doesn’t understand by default.  Among other things, this means Visual Studio might provide syntax highlighting for the programming language in the editor.  Any ways, this particular Python language service was never intended to provide complete IronPython support for Visual Studio.  Instead, this was bundled with the VS 2005 SDK as a complete sample to show developers how to extend Visual Studio for their own programming languages.  I’m not even sure any testers on the IronPython team ever evaluated this sample, but do believe we had at least one developer working on it.  I can’t really provide too many more details on the Python language service in the VS 2005 SDK as I didn’t join the IronPython team until summer 2006.

IronPython Studio was an update to the VS 2005 SDK Python language service for VS 2008 put together by a consulting firm named Clarius Consulting Labs.  To the best of my knowledge the IronPython Team had no involvement in the creation of IronPython Studio.  Also, the emphasis again was on being a complete VS language service implementation and not necessarily adding the best Python support to VS.  I think it’s a little unfortunate this tool has ‘IronPython’ in the title as this confuses people and leads them to believe we have the ability to fix bugs in it and also add new features (none of the IronPython Team is even listed as contributors to IronPython Studio – see http://ironpythonstudio.codeplex.com/People/ProjectPeople.aspx).  Finally, a few people in the IronPython community have observed that this piece of software is “abandonware”.  I agree, but wouldn’t be too surprised if it got revived for VS2010.

I think I speak for everyone on the IronPython Team when I say we’d love to see IronPython support in Visual Studio by default without the need for any language service addition.  The thing is these types of decisions are made at a much higher level than us and are heavily influenced by business needs.  What I’m getting at here is if enough people provided feedback via the proper channels that:
 

• The next version of VS is great, but it’d be better if it supported Python development
• You don’t currently use VS but you would if it had Python support
• Ahem…you’d be willing to pay for IronPython support in VS
• Etc

then the next version of VS could very well have Python support built into itJ  What are the proper channels for feedback you ask?  The VS 2010 feedback page (https://connect.microsoft.com/VisualStudio/content/content.aspx?ContentID=9790) would be a great start.

The basic infrastructure for custom attributes in IronPython is in place, but it’s woefully limited. Specifically, it only works for custom attributes that don’t have parameters. Of course, most of the custom attributes that you’d really want to use require additional parameters, both the positional or named variety. Since positional parameters are easier, let’s start with them.

Positional parameters get passed to the custom attribute’s constructor. As we saw in the previous post, you need a CustomAttributeBuilder to attach a custom attribute to an attribute target (like a class). Previously, I just needed to know the attribute type since I was hard coding the positional parameters. But now, I need to know both the attribute type as well as the desired positional parameters. I could have built a custom Python class to track this information, but it made much more sense just to use CustomAttributeBuilder instances. I built a utility function make_cab to construct the CustomAttributeBuilder instances.

In make_cab, I build a tuple of CLR types from the list of positional arguments that was passed in. If you haven’t seed the *args syntax before, it works like C#’s params keyword – any extra arguments are passed into the function as a tuple names args. I use Python’s built in map function (FP FTW!) to build a tuple of CLR types of the provided arguments, which I then pass to GetConstructor. Previously, I passed an empty tuple to GetConstructor because I wanted the default constructor. If you don’t pass any positional arguments, you still get the default constructor. Once I’ve found the right constructor, I pass it and the original tuple of arguments to the CustomAttributeBuilder constructor.

One major benefit of this approach is that it simplifies the metaclass code. Since _clrclassattribs is now a list of CustomAttributeBuilders, now I just need to iterate over that list and call SetCustomAttribute for each.

The only problem with this approach is that specifying the list of custom attributes is now extremely verbose. Not only am I specifying the full attribute class name as well as the positional arguments, I’m also having to insert a call to make_cab. Previously, it kinda looked like a C# custom attribute, albeit in the wrong place. Not anymore. So I decided to write a function called cab_builder to generates less verbose calls to make_cab:

The cab_builder function returns an anonymous lambda function that closes over the attrib_type variable. Python lambdas are just like C# lambdas, except that they only support expressions [1]. The results of calling the lambda returned from cab_builder is exactly the same as calling make_cab directly, but less verbose. And since I named the function returned from cab_builder Obsolete, now my list of class custom attributes looks exactly like it does in C# (though still in a different place). As usual, the code is up on my skydrive.

If you’re only using the attribute once like this, it is kind of annoying to first declare the cab_builder function. If you wanted to you could iterate over the types in a given assembly, looking for ones that inherit from Attribute and generate the cab_builder call dynamically. However, I’m not sure how performant that would be. Another possibility would be to iterate over the types in a given assembly and generate a Python module on disk with the calls to cab_builder. Then, you’d just have to import this module of common attributes but still be able to include additional calls to cab_builder as needed.

[1] The lack of statement lambdas in Python is one of my few issues with the language.

• __clrtype__ Metaclasses: Simple Custom Attributes

Over the course of the next three blog posts, I’m going to build out a mechanism for specifying custom attributes on the CLR type we’re generating via __clrtype__. All the various Builder classes in System.Reflection.Emit support a SetCustomAttribute method that works basically the same way. There are two overloads – the one I’m going to use takes a single CustomAttributeBuilder as a parameter.

For this first post, I’m going to focus on the basic custom attribute infrastructure, so we’re going to use the extremely simple ObsoleteAttribute. While you can pass some arguments to the constructor, for this first post I’m going to use the parameterless constructor. To keep things less confusing, I’m going back to the original version of the Product class, before I introduced CLR fields and properties. The one change I’m making is that I’m adding a list of attributes I want to add to the class.

• Embedding IronPython as a scripting language

A great way of extending an application is to allow users to write their own scripts. Fortunately, with the Dynamic Language Runtime it is fairly simple to provide scripting in your application. Let’s look at how to do it with a simple Windows Forms application and IronPython.

...

It’s simple to add powerful scripting support to your applications, while maintaining control over the scripting runtime means that you can restrict what any user entered code can do. I chose to show this example in IronPython, but the principles for doing this with Ironruby are identical (of course you’ll need a reference to IronRuby.) In fact, you could offer users the choice of languages.

I know it’s been a while since my last __clrtype__ post, but I was blocked on some bug fixes that shipped as part of IronPython 2.6 Beta 1. So now let’s start looking at one of the most requested IronPython features – custom attributes!

Over the course of the next three blog posts, I’m going to build out a mechanism for specifying custom attributes on the CLR type we’re generating via __clrtype__. All the various Builder classes in System.Reflection.Emit support a SetCustomAttribute method that works basically the same way. There are two overloads – the one I’m going to use takes a single CustomAttributeBuilder as a parameter.

For this first post, I’m going to focus on the basic custom attribute infrastructure, so we’re going to use the extremely simple ObsoleteAttribute. While you can pass some arguments to the constructor, for this first post I’m going to use the parameterless constructor. To keep things less confusing, I’m going back to the original version of the Product class, before I introduced CLR fields and properties. The one change I’m making is that I’m adding a list of attributes I want to add to the class.

Python list comprehensions use the same square bracket syntax as C# properties, so it kinda looks right to someone with a C# eye – though having the attribute specifications inside the class, rather than above it, is totally different. I wish I could use Python’s class decorators for custom class attributes, but class decorators run after metaclasses so unfortunately that doesn’t work. Also, I can’t leave off the “Attribute” suffix like you can in C#. If I really wanted to, I could provide a new type name in the import statement (“from System import ObsoleteAttribute as Obsolete”) but I thought spelling it out was clearer for this post.

Now that I have specified the class attributes, I can update the metaclass __clrtype__ method to set the attribute on the generated CLR class:

I’m simply iterating over the list of _clrclassattribs (if it exists), getting the default parameterless constructor for each attribute type, creating a CustomAttributeBuilder instance from that constructor and then calling SetCustomAttribute. Of course, this is very simple because we’re not supporting any custom arguments or setting of named properties. We’ll get to that in the next post. In the mean time, you can get the full code for this post from my skydrive.

There is one significant issue with this custom attribute code. Attributes are typically marked with the AttributeUsage attribute that specifies a set of constraints, such as the kind of targets a given attribute can be attached to and if it can be specified multiple times. For example, the MTAThread attribute can’t be specified multiple times and it can only be attached to methods. However, those attribute constraints are validated by the compiler, not the runtime. I haven’t written any code yet to validate those constraints, so you can specify invalid combinations like multiple MTAThread attributes on a class. For now, I’m just going to leave it to the developer not to specify invalid attribute combinations. Custom attributes are passive anyway so I’m figure no one will come looking for a MTAThread attribute on a class or other such scenarios.

However, I’m interested in your opinion: When we get to actually productizing a higher-level API for __clrtype__, what kinds of attribute validation should we do, if any?

• IronRuby 1.0 at OSCON 2009

IronRuby is 1.0! Come and see how IronRuby is used in .NET programs, how well it performs, and how conformant it is. IronRuby is an Open Source implementation of the Ruby programming language, built on top of Microsoft’s Open Source Dynamic Language Runtime. OSCON has an important place in the history of IronRuby, and it is only appropriate that we launch IronRuby 1.0 at OSCON 2009.

• Jimmy Schementi on channel 9: Inside IronRuby

Jimmy Schementi is a Program Manager (and developer) on the IronRuby team. IronRuby is an Open Source implementation of the Ruby programming language for .NET, heavily relying on Microsoft's Dynamic Language Runtime. IronRuby is Ruby, but implemented on top of the DLR (which of course provides the capability for dynamic languages to interact with the BCL and CLR).

You've learned about the details of the DLR here on 9, which provides dynamic runtime support for .NET. IronRuby targets compatibility with the 1.8.x branch of Ruby modulo continuations. IronRuby is an implementation of Ruby version 1.8.6.

Here, Jimmy explains the thinking behind the IronRuby project. Why are we doing this, anyway? When/Why would Ruby developers use IronRuby? What's the current status of the project? What's the future hold for IronRuby? Tune in and learn about the past, present and future of IronRuby.

• Professional IronRuby

Professional IronRuby provides a complete guided tour through IronRuby, demonstrating why Ruby is important, what IronRuby can add to your .NET development arsenal, and how you can take advantage of Ruby while utilizing existing infrastructures. Building on your existing .NET knowledge, the authors will walk you through Ruby’s syntax and structure and the ways you can use IronRuby for .NET projects. Learn how to apply IronRuby to ASP.NET and Silverlight and decide where to develop your components, with straight-forward guidance on when and why to use IronRuby instead of C#, and vice versa.

• Umm...

IronPython most definitely should not be able to spawn an error message box that's visible when Vista is locked at the username/password screen. Quite annoying yet very interesting at the same time...

• Use .NET classes in IronPython
  

It’s really simple to use a .NET class in IronPython. The first thing to remember is that you’ll need to add a reference (just like you would in .NET). To do that you use the AddReference method of the clr module. In the case of an assembly you’ve written, IronPython needs to be able to find it, which means it needs to be in sys.path. It turns out that you can add the path to your assembly by using the append method of sys.path.

• sp.py: A Python library for working with the SharePoint object model

This library provides the ability to automate various
SharePoint activities through easy to read and modify Python
scripts.

The core library is located in the sp module. The scripts
module contains a number of scripts for performing specific
activities such as backing up sites.

• WSM Rules Engine Part 2

In my last post I talked about creating a rules engine to generate a scouts evaluation of a player. Having played with IronPython for a few hours this evening I’ve managed to introduce a dynamic script to represent the rule engine. This means I can replace the rules engine or for that matter any one else can without having to re-compile the code. It’s suddenly dawned on me how this exciting technology can make a game extensible. With .NET 4 introducing the dynamic keyword plus other functionality for dynamic languages it’s going to make it even easier.

Performance wise it actually seemed pretty fast. This was one concern I had because of the greater work at runtime required. Objects aren’t statically typed so the runtime engine has to do the typing at runtime. Just starting to learn python and I must say I’m loving it so far. I’ve done nothing complex just followed a few tutorials and coded up a small script to test out ironpython running within c#. So how did I manage that.

• NWSGI Release - 1.1 Preview
  

• Running IronPython from C#

• Pro IronPython by Alan Harris

IronPython represents a unique direction for developers interested in working with dynamic languages within the .NET Framework. Whether you’re looking to develop applications from scratch or add functionality and maintainability to an existing application, IronPython opens many doors while providing a high–speed, high–performance language that integrates tightly with other .NET languages.

• Learn to create applications using the benefits of a dynamically typed language.
• Discover how to leverage the power of IronPython to improve existing applications.
• Explore interacting with other .NET languages by invoking the common language runtime.

What you’ll learn

• Write IronPython console and forms applications.
• Integrate IronPython scripts into existing applications to increase their functionality.
• Implement complex data patterns and interact with databases.
• Use the common language runtime to take advantage of .NET’s existing framework within IronPython scripts.
• Understand how IronPython can be brought in to solve a variety of programming tasks through extensive exercises and recipes.
• Avoid the common pitfalls of working with a dynamically typed language.

Who is this book for?
This book is designed for .NET developers interested in expanding their IronPython skills and learning how to make use of them to improve their daily programming practices. It will introduce core concepts of IronPython programming using a .NET–centric approach. Additionally, it will speak to Python developers looking to expand into their skills into the .NET world. Experience with a .NET language is expected. Previous Python experience is not necessary.

About the Author
Alan Harris is a web and application developer living in Arlington, VA. He has worked at more than a few organizations using .NET to create enterprise solutions since the glory days of version 1.1, and can still browse to some of the oldest ones. Aside from working at his desk, he spends most of his time studying Krav Maga and writing music.

"Many developers take issue with this and find the approach too loose, too error-prone. I was speaking to Pythonista and author Michael Foord about the matter; I mentioned the argument, “What if it looks like a duck, quacks like a duck, but is really a dragon impersonating a duck? You don’t want a dragon in your pond.” He replied, “if you code dragons, you’ve got no one to blame but yourself if they get in your pond.” I informed him that “the only difference between my IronPython dragons and my C# ones is my C# ones have ‘Hello my name is’ nametags.” So it goes."

I will be interested to see what you think of "Pro IronPython". I've had the luxury of reading both your book and mine, and I think you'll agree we speak to very different audiences. Apress wanted a book easy enough that someone with zero programming experience could pick it up and run, being pretty well versed by the end (which pretty much speaks to the entire Pro line I think, with a few exceptions). Your book covers some terrific stuff (the PowerShell parts I found particularly interesting) but a lot would be beyond the target audience of mine. I think, however, that they are rather complimentary books.

My background is primarily UNIX. I have been programming in C, TCL, Perl, and Python in UNIX and Linux for a long time.

I recently started working in Visual Studio and C#.

Visual Studio and C# make everything very easy. Python was nice when you didn’t have the convenience of Visual Studio and intellisense with a nice debugger.

In the windows world I see no major reason not to use C#.

Why would I want to use IronPython?

Both Python and C# are 'general purpose' programming langugages suited to a wide range of problem domains. Choice of programming language is therefore usually a matter of aesthetics, what you are used to, personal workflow, issues within a team and so on rather than a matter of which is more appropriate to the job. Obviously there are exceptions where one language is obviously more appropriate or obviously inappropriate but these are the exception rather than the rule.

Place where you should obviously use IronPython:

• For embedding as a user scripting language / plugin mechanism or are doing runtime code generation

Place where you should obviously use C#:

• You have already implemented in a dynamic language, profiled and optimised, and need faster performance than the dynamic language is capable of

Other than that the choice is yours!

Here are a few reasons why I / some people prefer IronPython:

• The interactive environment is an invaluable tool for exploration
• No need to create a whole 'object-oriented-application' for simple tasks where it is not needed
• Without a compile phase and with duck typing / dynamic typing IronPython makes it dramatically easier to do test driven development or even normal unit testing
• Python has many language capabilities missing from C# which make it a flexible and concise language. These include:
  • Duck typing and protocols (no need for explicit interfaces)
  • Heterogeneous containers (no need for generics)
  • Pain free introspection (no need for the complexity inherent in System.Reflection)
  • Metaprogramming including good old eval, type creation at runtime, decorators and descriptors
  • Late binding and runtime type dispatch
  • 'Little features' like tuple unpacking, returning multiple values, everything is an object (including primitives, functions, classes and modules)

There are things that C# has that Python doesn't (compile time checking, method overloading, access to .NET attributes), but choose whichever you prefer.

• If you are starting out Python is much easier to learn than C#.
• As a concise and expressive language it is faster to prototype solutions in Python and faster to turn those prototypes into full applications!
  
• Writing portable, cross-platform code. Compatibility with CPython is a great thing.
• Availability of open-source libraries. There are orders of magnitude more open source libraries for Python than there are for C# despite C# being used more than Python in commercial settings.
• Python is the 'standard' language in various problem domains (GIS and animation scripting, Linux system administration and certain types of scientific programming for example).
  

• IronPython in Action Examples on Mono

The intention is that MultiDoc, along with as many examples from the book as possible, run on both Mono and the .NET framework; IronPython isn't just for Windows but can also be used on Linux and Mac OS X or anywhere that Mono will run.

Some readers have reported that MultiDoc crashes on exit when run with Mono. The root cause of this is the old version of IronPython 2 that ships with Mono.

...

This is reason enough to look at using a more recent version of IronPython 2. Even though IronPython can't be compiled with Mono you can still use the up to date Windows binaries (which is part of the point of Mono - binary compatibility). To use the latest version of IronPython (2.0.1) you'll need Mono 2.4 or more recent...

Current xbuild status:

It can handle solutions (.sln) now, thanks to Jonathan Chambers, with C#, VB.net projects. Handles embedded resources, satellite assemblies, project references etc.

Eg. Paint.Net 3 ( builds completely with it now, IronPython (2.0.1) builds, IronPython svn (builds with compile time errors) etc.

Only .net 2.0 projects are supported currently though. I will start adding support for newer versions very soon, also ASP.NET, Silverlight etc projects.

This is part of the mono 2.4.2 release and is a *big* improvement over xbuild in the last release. Would be really helpful to get feedback and bug reports on this :)

MichaelFoord: [[Checking Python Syntax with Executing]] moved to [[Checking Python Syntax without Executing]]

MichaelFoord:

• Resolver One 1.5 Release

Resolver One is an IronPython spreadsheet system and is the largest IronPython codebase in production. Normally a new release would be big news but we've already covered the Python console which is the biggest new feature in version 1.5. Other big new features include CSV import / export and a goto line dialog for the code editor. You can read a full list of all the improvements in the changelog. Now that 1.5 is out we're working hard on features for version 1.6. Our latest blog entry has a sneak preview of one of the coming attractions.

• Introspecting .NET Types and Methods from IronPython

For Python methods you can use the inspect module to get information about methods / functions; like what arguments they take and so on.

Methods on .NET types don't have the attributes used by inspect (although methods of Python objects defined in IronPython work fine). We can get around this by using .NET reflection to introspect them them.

Reflection is normally done with classes from the System.Reflection namespace. We don't have to use these classes directly, but get them indirectly.

This IronPython Cookbook entry is an example of getting information about the OpenStandardOutput method of the System.Console class. We can then get the return type (something not available in Python) and the names of all the parameters.

• WPF Movie Player in WPF

A Japanese blog entry showing example IronPython code for a movie player capable of playing H264 encoded movies using the Windows Presentation Foundation MediaElement class (in 34 lines of code).

• Getting started with SolverFoundation

Solver Foundation uses a declarative programming model, consisting of simple compatible elements that are solved by application of solvers, meta-heuristics, search techniques, and combinatorial optimization mechanisms to accelerate the solution finding process. Building a model in Solver Foundation is as simple as specifying the decisions to be made, constraints to be respected, the goals to be used to evaluate candidate proposals (solutions) and the data to be processed by the model (historical or projected parameters).

Disclaimer : This is about working with the Visual Studio 2010 Beta 1 SDK. Information is likely to be inaccurate for future or previous Visual Studio SDK releases.

When you install the Visual Studio 2010 SDK Beta 1, you’ll notice a new node in the new project dialog under Visual C#\Extensibility and Visual Basic\Extensibility:

The editor extension templates are what you would expect. They give you a project that builds a MEF component (which the VS core editor consumes) to provide various adornments, etc…

If you would want to build a VSPackage, the trusty package wizard still lives where it always has under Other Project Types\Extensibility:

So then, what is that “VSIX Project” template for (in the first screenshot)?

The VSIX Project is meant to be a very simple project template that is used for building a raw VSIX “bundle” (i.e. the zip container). That’s why there is essentially no code in the project at all. This could be useful in situations where you were only just packaging up something like a project or item template, and weren’t really building any code. Pedro mentions this in his blog post series on how to build a VSIX-based project/item template for distribution on the VS Gallery.

Preventing the DLL/PDB from being added to the VSIX

One thing you may notice is that the DLL and PDB built from the project (that you don’t need if you’re just packaging up some templates) gets included in the VSIX file. To suppress this, you could either use the VSIX Explorer tool to edit the file post-build (tedious since you need to do this for each build)…or simply add the following XML snippet to your CSProj/VBProj to prevent the DLL or PDB from being included in the VSIX file at all:

<PropertyGroup>
  <CopyBuildOutputToOutputDirectory>false</CopyBuildOutputToOutputDirectory>
  <CopyOutputSymbolsToOutputDirectory>false</CopyOutputSymbolsToOutputDirectory>
</PropertyGroup>

• IronPython Excel Com Automation: Pivot this!

I have been playing around with IronPython recently while creating spreadsheets in Resolver One. So far I have been having a lot of fun, but I haven't really been doing anything that would be useful in a work setting. The main reason for this is because most of the spreadsheet work that I do revolves around pivot tables and Auto Filter, which are immensely helpful for visually analyzing data. I thought I might try and use Excel's COM Automation to add Pivot Charts and Auto Filter to any system with IronPython, and here is the code that I have come up with so far.

...

The code should be pretty straight forward. The Reload() method is helpful for reloading the whole module when testing from a console. The Run() method shows the steps to load a new spreadsheet. The ExcelManager static class is the core of the logic and provides an entry point for interacting with the spreadsheet.

There is not a lot of debuging code, so if something goes wrong it is just going to throw an exception, but this should be enough to help get things started. I did fix some funky problems with importing DateTime, where they kept getting converted into integers. Should work fine now, but anything other than Datetime, numerical types or strings will probably not get imported correctly.

• IronPython Date Selector: Pick a date... any date...

Another quick IronPython snipet that shows how to use the Windows.Form.MonthCalendar control to create a simple date selector.

...

Works well for selecting singe dates, and could be modified to support selecting a date range if desired. I prefer having a Start Date and End Date, and selecting each separately.

Last week there was a little Mono surprise. It can be found on this Novell-hosted web page web page (scroll a little bit).

It has been a few very busy weeks at Novell's Eastern Research and Development Facility (Novell NERD Facility) here in Cambridge and we have been incredibly busy polishing some nice toys.

A few weeks ago we learned about Sony's developer event in the West Coast. Michael, Zoltan and myself worked very hard to put together a demo to show the virtues of C# and the CIL to developers. So we cranked on some record time some code:

• PhyreEngine#
• Static compiler for PowerPC for Mono on PS3
• A yield-based co-routine framework.

We picked Sony's PhyreEngine to demostrate how to use Mono to write the high-level code for a game using Sony's finely tuned engine. We figured this was better than showing a for loop printing the numbers 1 to 10 on the screen.

PhyreEngine# wraps PhyreEngine using the same techniques that we used in Gtk# and Moonlight. The resulting API is glorious and by letting PhyreEngine do all the heavy lifting while driving all the high-level from C# there is no way of telling that the driving force is not C++. All you get is pure unadultered productivity.

To make our demos a little more interesting, Michael wrote a minimalistic yield-based co-routine framework inspired by some of the ideas that our friend Lucas gave us. It is a tiny toy, but we used it to illustrate the concept of using C# iterators as the foundation for game logic development and how a cooperative scheduler would work (Unity game logic works just like this).

We were also working on completing Mono's port to the PlayStation 3's native operating system (this is different than running Mono on Linux on the PS3: that already works, and it was used for developing CellDotNet, a JIT for the PS3's SPUs). Zoltan developed the static compiler for PowerPC and I did the platform support.

Mono can now run "Hello World" on the PS3 native OS. There are still lots of ins, lots of outs and lots of whathaveyous that need to be tied up before this fully works and before we are able to run PhyreEngine# on the PS3.

• Strengthening the Microsoft Ecosystem with Source Code

Microsoft and Open Source are often portrayed as enemies. But in DevDiv, we have several high profile Open Source projects. I work on IronPython, which has been Open Source for over four years. More recently, the ASP.NET and Extensibility Framework teams have decided to release some projects as Open Source.

I believe we should have more Open Source projects at Microsoft. But more importantly, I feel that we need to go beyond the textbook definition of Open Source. Our Open Source projects are typically closed to external contributions. But for the ecosystem at large, Open Source isn’t just about the availability and distribution terms of the source code. It also implies a collaborative development model - parties working together across organizational boundaries, contributing directly to projects in question.

The thing I would most like to change about Microsoft would be to let members of our ecosystem contribute code to our Open Source projects.

• iLove Sharepoint: IronPython in Action

IronPython In Action covers all you need to get started with IronPython for a .NET or Python developer. It is written coherently without needless ballast, well suited to the philosophy of the language itself. My favorite chapters are “Silverlight: IronPython in the browser”, “Agile testing where dynamic typing shines” and “Embedding the IronPython engine”. The book has inspired me to found the CodePlex project IronSharePoint (coming soon) showing how you can develop SharePoint apps with IronPython. I’m convinced that in the near future dynamic languages will get more attention again, just think of the “dynamic” keyword in C# 4.0. Absolutely worth keeping an eye on it!

• Scripting: Reliably Finding my 32bit EXEs on a 32bit & 64bit OS or Where is IronPython?

• Iron Python: How to talk to the .NET Framework

• Creating Python Modules from CSharp

• Back to Basics: Using Fusion Log Viewer to Debug Obscure Loader Errors

Here's an example of how a tiny bit of digging saved me hours of confusion recently when I hit an unusual edge case. I was doing a build of sample that was showing C# interop-ing with IronPython, but I was using a daily build of .NET 4 and a random build of IronPython.

I made a mistake and had some really old DLLs floating around that I shouldn't have had. My symptom was a FileNotFoundException for the file "System.Dynamic.dll." I KNEW it was in the GAC (Global Assembly Cache) and I could SEE it as a reference DLL in my directory. Can't find the file? Dude, it's right there!

• IronPython how cool it is

• ZedGraph samples in IronPython

• Skills Matter: Resolver Systems Technology

Jonathan Hartley, a developer at Resolver Systems talks about IronPython, and how it is used at Resolver Systems.

Resolver Systems's first product, Resolver One, is a desktop and web-accessible spreadsheet aimed primarily at the financial services market. It is written entirely in IronPython, and directly exposes end-users to Python, both as expressions in cell formulae, and as an embedded scripting language which provides programmatic access to spreadsheet content.

Resolver One launched in Spring of 2008, by which time the team had grown to fourteen people, and the codebase to 40k lines of product and 140k lines of tests. Here is an article by Jonathan about use of IronPython at Resolver Systems.

• Founding Resolver Systems
  

Giles Thomas shares the story of founding Resolver Systems and answers questions about the company. Resolver Systems was founded in 2005 to produce a next generation spreadsheet. The founders, Giles Thomas, Robert Smithson and Patrick Kemmis, had all struggled with the challenges of running multi-million or multi-billion dollar businesses through poorly debugged spreadsheets. Their solution - Resolver One - seamlessly integrates a powerful and flexible programming language (IronPython) and a traditional spreadsheet environment. Resolver One is used by companies in the financial, oil exploration and pharmaceutical industries and in academia.

• IronPython Is it a dead parrot or does it fly?

Python is one of the increasingly trendy dynamic languages and it is now available under the .NET umbrella. IronPython is an open source version of the language developed by Guido van Rossum in 1990. Python has a great many users and they are all passionate about the language and mostly about Monty Python as well. Yes, there are lots of Pythonesque (as in the well-known TV series) references in the Python world, but it is a very serious language as well! It currently runs on most platforms, and IronPython extends this range to .NET and Silverlight.

Why bother?
Python may be free and it may be a high quality language but so is C#. Why bother with a strangely named language that you probably haven’t heard very much about? One good reason is that it is an interactive object-oriented language that has managed to take the good ideas from many other languages and meld them together into something workable. If you have looked at early versions of IronPython, now is a good time to take another look because version 2.0 has just been released and this is based on the new Dynamic Language Runtime (DLR) which all of the .NET dynamic languages should be using quite soon. The DLR is an extension to the CLR designed to make it easy to implement dynamic languages. The best argument for Python, however, is to use it – so let’s get started.

• Debugging IronPython Code in SharpDevelop

Ensure that the Just My Code feature is checked and that the Step over code without symbols is not checked. If the Step over code without symbols option is selected then stepping will not work properly and lines of code will be skipped over.

There are two ways to debug your code. You can use the Python menu or modify the project options. We will look at both of these alternatives. First open your IronPython project into SharpDevelop. Open your main file and make sure it is the active text editor window. Set a breakpoint somewhere in your code. Then from the Python menu select Run...

• Note to Self 9 – Exploring SharpDevelop

SharpDevelop is an open source IDE for the .NET platform. Is it new? Hardly. Based on the history of news releases from their website, It’s been around since December of 2000. Considering that the .NET platform was publicly announced only 6 months earlier, SharpDevelop seems only that much more mature.

If you’re comfortable with the basics of Visual Studio, SharpDevelop should feel like home to you. In fact, the best way to describe SharpDevelop is “Visual Studio without a lot of stuff I don’t want and a few third-party items I do”.

SharpDevelop uses NUnit instead of MSTest. That’s right. No third party plugins to make your IDE work with NUnit. It simply works.

It’s time for me to confess. It’s all about IronPython. The reason I started investigating SharpDevelop is because of my frustration with Visual Studio’s lack of support for the language. Specifically, I want Intellisense when I write IronPython. To be sure, you only get Intellisense for “non-DLR code”, but surprisingly that doesn’t seem to be as much of a handicap as you might think.

Speaking of first class citizens, SharpDevelop comes stock with C#, Visual Basic, Boo (I can hear Jay Wren cheering now), IronPython and F# (now a number of SRT folks are cheering).

If you’re frightened of the consequences that may come from changing your development environment, don’t be. You can open one of your existing Visual Studio solutions with SharpDevelop and start coding. Switching back is as simple as opening up your now modified project/solution in Visual Studio.

• .NET Module Testing with IronPython

The Python scripting language has features that make it an excellent choice for performing several types of software testing. There are several implementations of Python available including CPython, the most common implementation on machines running UNIX-like operating systems, and IronPython, an implementation released in late 2006 that targets the Microsoft .NET Framework. In this month's column I'll demonstrate how to use IronPython to test .NET-based modules from both the Python command line and in lightweight Python scripts.

• Methodist: Make .NET Reflector come alive with IronPython

It is great to be able to inspect the contents of an assembly with .NET Reflector, but to really understand the behaviour of code, it is nice to be able to call methods and see the results immediately within .NET Reflector. Now you can, thanks to the magic of IronPython and the DLR. Ben builds a REPL add-in for .NET Reflector and makes a great tool even more useful.

***Disclaimer: This information is about Visual Studio 2010 Beta 1 only. The following may not be accurate for future releases.***

One of the new features we introduced with the Visual Studio 2008 Isolated Shell was the concept of a pkgdef file. Pkgdef is essentially a REG file with tokenization. For example, you could put $RootFolder$ in your pkgdef file to refer to the root directory of your isolated shell application. At runtime, these files are consumed by the VS Isolated Shell runtime and merged into the registry (with the tokens replaced by literal values). The registry merely acts as a cache of the information in your pkgdef files.

In Visual Studio 2010, the pkgdef model has been integrated into Visual Studio itself. As such, the manner in which your packages are built and (locally) deployed when developing with the Visual Studio 2010 SDK Beta 1 is significantly different (although the “F5 experience” should be the same).

Visual Studio Startup / Pkgdef Merge

On the very first launch of Visual Studio, the first thing that needs to happen is merging the pkgdef files into the registry. You can think of the destination registry key where these go as merely a cache used for performance. The fact that it’s a registry key is somewhat of an implementation detail. Here’s a graphical view of what happens:

Note in the above diagram that the cache location is under KHEY_CURRENT_USER, and that the key name is 10.0_Config. If you view this key in regedit, you’ll notice a sibling key named simply “10.0”. This 10.0 key holds your per-user settings (things like your personal preferences for fonts, window position, and other options).

In comparison to Visual Studio 2008 and running with the /ranu switch, these 10.0 and 10.0_Config keys map to the UserSettings and Configuration keys under HKCU\Software\Microsoft\VisualStudio\9.0 respectively.

It is a completely safe operation to delete the HKCU\…\10.0_Config key since Visual Studio will re-create it on the next launch.

“Experimental Instance” Startup / Pkgdef Merge

A very similar process happens when you run “devenv /rootsuffix Exp”. Instead of merging the pkgdef information to “10.0_Config”, everything is merged to “10.0Exp_Config”:

What are the “Pkgdef Files on Disk” locations?

There is a file called “devenv.pkgdef” in your VS installation folder (under Common7\IDE) that defines where pkgdef files are picked up from. First, pkgdef files are picked up from a few sub-folders under Common7\IDE. Secondly, you’ll notice that we scan a folder called “$AppDataLocalFolder$\Extensions”. On my user account this maps to:

C:\Users\aaronm\AppData\Local\Microsoft\VisualStudio\10.0\Extensions

When running with “/rootsuffix Exp” this would be:

C:\Users\aaronm\AppData\Local\Microsoft\VisualStudio\10.0Exp\Extensions

In order for a pkgdef flie in the per-user location to be picked up by VS, it must also be declared as owned by a VSIX-based extension and enabled in HKCU\Microsoft\VisualStudio\10.0\ExtensionManager\EnabledExtensions. When you build a VSPackage or MEF component from one of the project templates in the Visual Studio 2010 Beta 1 SDK, this is all happening for you during the build.

Yesterday Lluis announced the last missing piece from our strategy to make MonoDevelop a full cross-platform IDE: MonoDevelop now runs on Windows as well:

When we started the planning for MonoDevelop 2.2, the major goal of that release was to get feature parity on Linux, MacOS and Windows.

We want to grow the community of developers that contribute to MonoDevelop and we wanted to attract add-in developers that wanted to bring their IDE extensions to all three platforms.

MonoDevelop has recently been getting some nice community contributed plugins like Flash/Flex development support, Vala language support, Mono debugger for OSX (thanks to the nice folks at Unity for this!), VI editing mode and of course our own Silverlight and ASP.NET MVC add-ins.

My theory is that supporting MonoDevelop on all the three major operating systems will have a multiplication effect in terms of contributions to MonoDevelop: it will help both users and will enable developers that extend MonoDevelop with add-ins to reach more users.

I secretly want Unity to adopt MonoDevelop as the code editor for Unity; for the FlashDevelop guys on Windows to adopt MonoDevelop as their cross-platform foundation (their users want a cross platform Flashdevelop); for Flashbang to bring their UnityScript framework to MonoDevelop

Developing an add-in for MonoDevelop now brings your enhancements to a much larger community.

Look and Feel

Although the IDE is built using Gtk#, but we are aware that developers want to get things integrated with their operating system as much as possible. This is why we have invested in properly integrating MonoDevelop with the Mac and Windows.

The Look of MonoDevelop still has a heavy feel of the Linux Gtk+, but we are bluring the lines by making the theme and style match the operating system. Development in Gtk native themes will also continue to improve things.

Feel wise, we make MonoDevelop follow the conventions of the host platform. For example, on the Mac, MonoDevelop uses the Mac system menu, it uses an entirely different keybinding style that follows what every Mac developer expects (Command-KEY operations that match X-Code for example) and even text selection in the editor behaves differently:

More work will come, because we want MonoDevelop to feel native on each platform.

On Windows for example, MonoDevelop runs on top of the .NET Framework and uses the .NET managed debugger instead of using Mono's runtime and Mono's debugger, so there is no dependency on Mono to be installed on the system.

• Running KirbyBase in IronPython
  

As I was looking for embedded database systems, I came across this pure python database called KirbyBase. So I decided that I would make it run on IronPython. So this post describes on how to make KirbyBase run on IronPython and using ipy.exe. Note that I am not looking at integrating this database into C# application yet. [may be in the future post]

• The Australian: IronPython in Action

IronPython in Action is a fantastic resource for anyone learning IronPython and for anyone wondering what this dynamic language IronPython is all about and whether they should learn it. It feels nicely balanced for both .NET users looking to learn about IronPython, and for Python users looking to learn about IronPython on .NET.

I prefer software books that have plenty of discussion and can be read from start to finish otherwise I end up skipping through chapters I never go back too. I found this book well written and fun to read with interesting insight all the way through. There is enough about the python language and .NET to get you through the book, and rather than being a reference book it instead provides information on how to learn more.

• The Frenchman: book review
  

IronPython is the first dynamic language developed for the .Net platform. At first, .Net didn’t support this kind of language. This is something that keeps on coming back throughout the book: you have to use some additional tricks to unleash the power of .Net dynamic and static languages.

As a conclusion, those who need a dynamic language (to script an application) can go for IronPython, the first dynamic language for the .Net framework, compatible with the language Python 2.5, and in that case, go for this book that will help you for anything.

• The American: Review: IronPython in Action

I’ve always wanted to use Python more but it’s difficult because the .NET Framework is such a walled garden in terms of interoperability with other programming languages. Mostly I use it to write little one off scripts when I want to move a bunch of files around or parse some text. I was really excited when Manning asked to to review IronPython in Action because I wanted to dive a little deeper into Python and possibly use it in some web applications. “IronPython in Action” makes it easy to get started using IronPython right away.

Chapter 7 discusses agile testing and unit testing using IronPython. I almost dropped the book in amazement. Unit testing is almost never mentioned in any language book and is relegated to a niche or advanced topic. Find a book about any other .NET language that mentions unit testing that doesn’t have the word “testing” in the title. This alone sets the quality of this book far above other language books I have read. It’s not just enough, in my opinion, to discuss the syntax of the language. You have to teach the reader how to use the language in your everyday work.

My overall feeling about this book is that it’s a great book. The authors use the same humor and dry wit that Python is known for to great effect. Making the digestion of a very different language easier. I’m sure that as I continue to experiment with IronPython that I’ll keep this book close at hand.

• Davy's IronPython Editor 00.01.71 Released

Nearly 2 months on since the last release and the project has changed a lot and whilst not a massive update, I seriously recommend this upgrade. Thanks to the IronPython mailing list for support, suggestions and inspiration. Like any other project out there, I am desperate for feedback :-) Also if you have any new ideas for the future, let me know.

DIE has had about 400 downloads since launch. With blossoming interest in IronPython, (partly, I suspect, due to IronPython in Action) hopefully it will hit the 1000 mark soon! So pop along and download.

Here's the detail in what has changed in 00.01.70:

• Mobile Version which stores options in same dir.
• Installed version stores in User Profile. (Existing Settings files don't transfer over - sorry).
• 'Find in Files' feature added.
• Much much Improved Capture of output from running Python scripts.
• Click on error output now opens file and goes to line.
• Resources - MSDN links area now low bandwith versions, Python links and Powershell launch menu.
• Added CheckBox and Listbox control to gen_GUI.
• Syntax Box -Exposed new property to get Caret Screen position.
• Autocomplete option added - default to OFF. Experimental parsing for strings, lists and dictionaries.
• Bug fixes and performance updates.

• Python 101 for the .NET Developer

What: "Python 101 for the .NET Developer" with Sarah Dutkiewicz, Cleveland C#/VB.NET SIG

When: Tuesday, June 23, 6:30-8:30pm

Where: Beacon Place Conference Center, Beacon Place Office Building, 6055 Rockside Woods Blvd, Lower Level, Independence, OH 44131

• Python and .NET Regular Expressions at the Interactive Interpreter

• Updated: Embedding IronPython in C# Silverlight Applications Article

IronPython has been designed to make it easy to embed in .NET applications. With code evaluated and executed at runtime it opens up all sorts of possibilities for user scripting of applications, storing rules as text that can be created and modified at runtime and so on. Hosting IronPython in Silverlight is slightly different from desktop .NET applications. Although the hosting API is the same there is some initial configuration that needs to be done.

I've updated my article on the topic to include the configuration needed for IronPython code to import from other Python files contained in the xap file, plus adding references to the standard .NET / Silverlight assemblies.

• Velocity Preview - The Greatest Good for the Greatest Number at Microsoft

JT: Microsoft has done a lot of work providing dynamic language support through projects through IronRuby and IronPython. Do those technologies find any traction internally?

ES: Oh, yeah, actually. I think it depends a lot on what group you're in. Like in the research teams, they tend to be used more than the product teams that I've been on. But there's definitely -- we're all geeks, you know? You get geeks and you get neat technologies and the geeks will try and use them. So I know that Iron Python, I know quite a few people who've experimented with it for everything from build tools to little data analysis tools.

• Half-a-Book Review: IronRuby in Action

• Moonlight 2 Preview 3 Out - can run IronPython Silverlight applications

• NWSGI 1.1 Preview 2

NWSGI 1.1 Preview 2 is exactly the same as Preview 1, except linked against IronPython 2.6 Beta 1. NWSGI is an implementation of the WSGI specification (Web Server Gateway Interface) for IronPython. It is implemented as an ASP.NET HttpHandler and is supported on the IIS 6 / 7 webservers (or any ASP.NET capable server). NWSGI allows the integration of Python applications like Trac and Django with existing ASP.NET facilities such as caching, sessions, and user management.

• django-ipy project on bitbucket

• How to install IronPython 2 with Mono on Ubuntu

This is just an instructional post with the hope that it might help others in the future.

I spent a long time trying to get IronPython2 running on Mono. Now IronPython1 ships with Ubuntu, so
apt-get install ironpython
, if that's all that you want. Ironically, I don't really know the difference between IronPython1 and IronPython2, but "2 has got to be better than 1", right?

The good news is that its very possible, but you will have to step out of your comfort zones.

You will need: Mono 2.4 (sources) and IronPython 2.0.1 (binary)

• Getting started with IronPython - Part 3: unit testing

Being a TDD’er (Test Driven Developer) one of the first things I’ve searched for when I started learning IronPython is a unit testing framework, luckily for me I didn’t need to search too long.

IronPython has a good unit testing framework inherited from Python called PyUnit.

Writing unit tests using PyUnit is simple:

1. import unittest class
2. Create a class that inherits unittest.TestCase (test fixture)
3. Write test methods using assertions methods

• Dynamic Languages at TechEd 2009

Dynamic languages on .NET are picking up momentum at this year's TechEd. Your typical TechEd attendee is a mainstream .NET developer, since this conference focuses on technologies that are shipping today (as opposed to futures conferences like the PDC). To speak more to this crowd, I focused on how they can make their existing .NET apps better by mixing in some end-user scripting.

My talk was DTL332 if you're a TechEd attendee (not sure when/if the videos will open up to the general public). I showed how you can add Ruby and Python scripting to an existing app, and spent some time building some simple REPLs. Toward the end of the talk, I showed a more realistic scenario where I embed a REPL in an existing Open Source .NET application: Witty.

During the demo, I showed off a twist on the traditional REPL - one where the editor and the REPL are one and the same. To run code in this REPL, you select the code that you want to run, and the output appears immediately below the selection. Since the REPL is just an editor, you can just delete any output that you don't want. History behaves just fine as well.

• Book Review: IronPython in Action
  

It is aimed at Python programmers who do not know .NET, and .NET programmers who do not know Python, rather than at existing IronPython developers. The authors run through the basics of Python and its .NET integration, so that by the end of Part 1 you could write a simple Windows Forms application. Part 2 is on core development techniques and covers duck typing, model-view-controller basics, handling XML, unit and functional testing, and metaprogramming – this is where you generate and execute code at runtime.

It’s a well-written book and fulfils its purpose nicely. I like the way the book is honest about areas where IronPython is more verbose or awkward than C# or VB.NET As someone who knows .NET well, but Python little, I could have done without all the introductory .NET explanations, but I understand why they are there. I particularly liked part two, on core programming techniques, which is a thought-provoking read for any developer with an interest in dynamic languages.

• Tim Golden Reviews IronPython in Action

By way of disclosure, I’m given a couple of blushingly generous footnote credits by Michael which naturally leave me feeling well-disposed towards the book as a whole. But even without those, I’d be giving it the thumbs-up.

But the most important win, I think, is managing to write a book about IronPython, not about Python or .NET. Naturally there is an element of explanation involved in both directions when some feature is being introduced or compared. But for the most part you can refer to the appendices which give summaries of Python/.NET if an unfamiliar term arises. For me, this achievement is key to the success of a book like this.

• Managed JScript (Bastard, red-headed step-child in the DLR)

I'm a Developer, I build Web Applications, and have for well over a decade now, and I love Javascript. I find the DLR compelling, and really want to see the Javascript implementation treated like a first class citizen. Give me code, samples and howtos. I'm going down this path anyhow, and would rather not do it alone.

• Running Pyke in IronPython

• One Shell to Rule Them All (and in the darkness bind them)

Were it not for the fact that Powershell is as intuitive as Linear B, it would be game over. As it is, IronPython and IronRuby remain interesting alternatives. They make the scripting process far simpler, and the syntax is much closer to the type of programming language that DBAs are used to. They become even more attractive when one considers the versatility of the .NET Dynamic Language Runtime (DLR), which allows you to plug IronPython or IronRuby into your .NET applications, call C# methods and so on. Maybe, with these DLR-based languages, we really are moving closer to achieving the dream of all DBAs: one scripting language to do everything. However, your Sysamin will loathe them as he will be engrossed in the extended culture-shock of having to redo all his favourite scripts in Powershell.

• The IronPython presentation, it's here

Last night (March 19th), I had the chance to give a speech about IronPython at the .NET users group New York. First thing, I have to say Thank You, to the .NET UG-NY leader’s, for the opportunity, and thank you to my fellows at the .NET UG-NY for attending.

Anyway for all of those that ask me for a copy of the slides, you can get them here together with the source code for the demos.

• Silverlight & IronPython "Hello World"

• IronPython and the elmcity project: Together again

In the first installment of this elmcity+azure series my plan was to build an Azure-based calendar aggregator using IronPython. That turned out not to be possible at the time, because IronPython couldn’t run at full strength in Azure’s medium-trust environment. So I switched to C#, and have spent the past few months working in that language.

It’s been a long while since I’ve worked intensively in a compiled and statically-typed language. But I love being contrarian. At a time when low ceremony languages are surging in popularity, I’m revisiting the realm of high ceremony. It’s been an enjoyable challenge, I’ve gotten good results, and it’s given me a chance to reflect in a more balanced way on the “ceremony vs. essence” dialogue.

Meanwhile, Azure has moved forward. It now provides a full-trust environment. That means you can run PHP, which is interesting to a lot of folks, but it also means you can run IronPython, which is interesting to me.

In this entry I’ll show you how I’m starting to integrate IronPython in the two main components of my Azure project: the web role that provides the (currently minimal) user interface, and the worker role that does calendar aggregation.

• ICOOOLPS Submissions

The relevant paper is "Faster than C#: efficient implementation of dynamic languages on .NET" (PDF) by Armin, Anto and Davide Ancona, who is Anto's Ph.D. advisor.

The Common Language Infrastructure (CLI) is a virtual machine expressly designed for implementing statically typed languages as C#, therefore programs written in dynamically typed languages are typically much slower than C# when executed on .NET.

Recent developments show that Just In Time (JIT) compilers can exploit runtime type information to generate quite efficient code. Unfortunately, writing a JIT compiler is far from being simple.

In this paper we report our positive experience with automatic generation of JIT compilers as supported by the PyPy infrastructure, by focusing on JIT compilation for .NET.

Following this approach, we have in fact added a second layer of JIT compilation, by allowing dynamic generation of more efficient .NET bytecode, which in turn can be compiled to machine code by the .NET JIT compiler.

The main and novel contribution of this paper is to show that this two-layers JIT technique is effective, since programs written in dynamic languages can run on .NET as fast as (and in some cases even faster than) the equivalent C# programs.

The practicality of the approach is demonstrated by showing some promising experiments done with benchmarks written in a simple dynamic language.

• HOWTO: Using Wing with IronPython

This article has the PI (Python Interface) script for downloads with instructions on how to generate PI files for .NET assemblies / namespaces plus how to make them available for Wing. It also has an example extension script for Wing which launches the current file with the IronPython interpreter (with instructions on how to make it available to Wing and bind it to a key combination).

• Using IronPython with NetBeans Python IDE

With the arrival of the IronPython 2.6 beta 1, with ctypes support, the IronPython 1.x only support in IronPython Studio really takes the IDE from a bit dated to seriously obsolescent.

Now, I had tried to get IronPython 2.0 to talk with NetBeans, but hadn't immediately achieved success, so, with less motivation, had put that to one side. Perhaps for similar reasons, a recipe for this wasn't out on Google already -- but now I really needed to crack the problem, I wasn't going to let that earlier abject failure put me off...

• Dynamic Language Runtime 0.9.1
• 0.91 Release Notes
  

This release has several design cleanups and final calls for the bits we’re shipping in CLR 4.0. This release is also consistent with Visual Studio’s Beta1 release in that these sources (with namespace tweaks) were used for parts of CLR 4.0. Documents have all been updated, and there is a new walkthrough document for an example language implementation on the DLR. The language, Sympl, has a C# and an IronPython implementation in the source tree, along with some example code written in Sympl. There are many one-off examples for Expression Trees node types that are in the DLR source tree now.

• ASP.NET Dynamic Languages Support
• Release Notes

This release is compatible with IronPython 2.6 Beta 1. Currently it does not include Language Services Support and project templates. To create a new IronPython ASP.NET WebForms project, simply copy the “examples\hello-webforms” to a new directory and rename the directory to your liking. A redistributed copy of IronPython 2.6 Beta 1 can be found in the “bin” directory; all files except Microsoft.Web.Scripting.dll, the IronPython ASP.NET integration, are from the IronPython 2.6 Beta 1 release.

Included in this release are two WebForms examples that are written in IronPython: “hello-webforms” and “album-handler”, which can be found in the “examples” directory. “hello-webforms” is a simple web application that shows PostBack handling, and “album-handler” is a larger web application that creates a photo album from a directory of images and generates thumbnails for them on the fly.

Found a quick and dirty way to do WCF exceptions in silverlight http://code.msdn.microsoft.com/Project/Download/FileDownload.aspx?ProjectName=silverlightws&DownloadId=3473 .  The problem is that it doesn’t do inner-exceptions.

I changed the raw fault exception around a bit to get inner exceptions working.

    public static RawFaultException BuildRawFaultException(XmlDictionaryReader reader)
    {
        List<string> messages = new List<string>();
        List<string> stacktraces = new List<string>();          
        while (reader.ReadToFollowing("Message"))
        {
            string message = reader.ReadElementContentAsString();
            string stackTrace = reader.ReadElementContentAsString("StackTrace", reader.NamespaceURI);
            messages.Add(message);
            stacktraces.Add(stackTrace);           
        }
        RawFaultException e = null;
        for (int i = 0; i < messages.Count; i++)
        {
            if (e == null) { e = new RawFaultException(messages[i]) { stackTrace = stacktraces[i] }; }
            else { e = new RawFaultException(messages[i], e) { stackTrace = stacktraces[i] }; }
        }

        return e;

    }
}

The next thing we want for our dynamic DataTable is to do calculations between one or more columns. Imagine (for instance) that you want to add two columns and store the result in a third. In C#, the client code might look like this:

dynamic table = new DynamicDataTable(t);
table.Foo = table.Bar + table.Baz;

At build time, the second statement gets compiled into a series of four dynamic call sites, which we can describe in pseudo-code as follows:

dynamic tmp1 = table.GetMember("Bar");
dynamic tmp2 = table.GetMember("Baz");
dynamic tmp3 = tmp1.BinaryOperation(OperationType.Add, tmp2);
table.SetMember("Foo", tmp3)

We’ve already defined GetMember and SetMember on our table, so all we need to do to make this work is to define BinaryOperation – at least for the “addition” operator. Right?

Alas, it’s a little more involved. The BinaryOperation isn’t being performed against the table itself; rather, it’s being performed against the results of the GetMember operation. Our current implementation has GetMember returning a System.Array, and we have no way to define a new operation against this preexisting type – whether the operation is static or dynamic.

What about extension methods?

If we were using statically-defined types, we might be able to effectively “monkey patch” System.Array by defining extension methods for either System.Array itself or for the specialized array types that we’re interested in. Unfortunately, this won’t work for us here because extension methods are neither supported against dynamic types nor can they be used to define operators for C# or VB.

(The first of these limitations is deliberate. Extension methods work at compile time because the compiler has direct access to the “using” statements which bring the methods you want into the local scope. There’s no obvious way to get the same information at runtime, which is when dynamic call sites are bound.)

First implementation

Our initial implementation will wrap the array we were originally returning and will add a CLS-compliant implementation of the “+” operator. We won’t derive this class from DynamicObject. Because our DynamicDataColumn will be typed to "dynamic", resolution of operator + will happen at runtime.

How do we implement this operator? We can identify two different scenarios – adding a sequence of values (such as another column) to a column, and adding a constant to a column. In both cases, we’ll end up performing n additions, where n is the number of rows in the table. To perform the element-wise operation, we’ll simply cast the two values to “dynamic” and let the C# runtime binder do the work.

Here’s the code:

    internal class DynamicDataColumn : IList {
        private readonly Array _data;

        internal DynamicDataColumn(Array data) {
            _data = data;
        }
        
        // IList implementation goes here
        
        public static DynamicDataColumn operator +(DynamicDataColumn left, IList right) {
            if (left.Count != right.Count) {
                throw new ArgumentOutOfRangeException(
                    String.Format("Column length mismatch ({0} found, {1} expected)",
                        right.Count, left.Count)
                );
            }
            object[] result = new object[left.Count];
            for (int i = 0; i < left.Count; i++) {
                result[i] = (dynamic)left[i] + (dynamic)right[i];
            }
            return new DynamicDataColumn(result);
        }

        public static DynamicDataColumn operator +(DynamicDataColumn left, object right) {
            return left + new ConstantList(right, left.Count);
        }
    }

(I’ve chosen to use IList instead of IEnumerable in order to simplify the code. In principle, we could create an overload for each. This would give us the flexibility of IEnumerable when we don’t have a more specific interface, while still letting us take advantage of IList.Count when we get an IList.)

The exact same code can be used to implement other binary operators – both arithmetic operators like “-“, “*” and “/” and logical operators like “>” and “<” – simply by replacing the four instances of “+” in the code above with the appropriate substitute.

Now we’ll need to change DynamicDataTable.TryGetMember so that it returns “new DynamicDataColumn(a)” instead of just Array “a”. Then - in conjunction with what we’ve done already - we’re able to write the following:

        public static void Main(string[] args) {
            DataTable table = CreateTable();
            dynamic t = new DynamicDataTable(table);
            t.Amount += 1.0;
            t.SmallerAmount = (1 + t.Amount) / 15.0;
            t.Average = (t.Amount + t.SmallerAmount) / 2;
            t.IsOkay = t.Average > 50
            t.Greeting = "Hello " + t.Name;
            // This line no longer works!
            // t.NewAmount = Apply(t.Amount, new Func<double, double>((x) => Math.Sqrt(x)));
            foreach (var r in t.Rows) {
                System.Console.WriteLine("{0}, your status is {1}", r.Greeting, r.IsOkay);
            }
            System.Console.ReadLine();
        }

This is pretty exciting! The statement “t.IsOkay = t.Average > 50” creates a new bool column on our table and sets its value based on a comparison between another column and a constant value – and it does so with syntax that is both clean and natural. So it looks like we’re done implementing arithmetic.

The fly in the ointment

Unfortunately, there are a few problems with this approach – some obvious, some subtle.

1. Our code doesn’t currently handle the reverse sequence “t.Foo = x + t.Bar” – whether x is a single value or a non-DynamicDataColumn sequence. Changing this means that we need to create another two overloads per operator. And if we want to support both IList and IEnumerable sequences, we need a further two overloads. Six overloads times sixteen binary operators makes 96 methods to implement.
2. Nearly all of these methods are basically boilerplate copies of the first ones that we created for “operator +”. It would be nice if we could combine the implementations because duplicated code is a frequent source of errors.
3. Our original implementation let us do some interesting things with columns if we knew their type, because we were able to cast an array T[] into an IEnumerable<T>. In this first implementation of arithmetic operators, we can no longer cast columns to strongly-typed collections.
4. The semantics we’re using for addition are those of our implementation language (C#) and not those of the language that is using the DynamicDataTable. We may not be able to do anything about this, but it would be nice to change it if possible.
5. One potential problem is particularly hard to see, and it results from the way the C# compiler implements dynamic sites. The compiled code for “operator +” contains exactly one dynamic call site which is shared between all users of the method. This means that the sample code above will generate three rules into the site – for type pairs (double, double), (int, double) and (string, string). Generating many rules into a site will degrade that site’s performance. Three rules isn’t bad, but this code introduces the possibility of many more being created.

But we do have working code now, so this is a good time to take a break. We’ll tackle some of these issues in the next installment. The current version of the source code can be downloaded from here.

Let’s get started by doing “the simplest thing that could possibly work”.

    public class DynamicDataTable : DynamicObject {
        private readonly DataTable _table;

        public DynamicDataTable(DataTable table) {
            _table = table;
        }
    }

For now, we’ll use a DataTable for the actual storage and a DynamicObject to provide an implementation of IDynamicMetaObjectProvider. What can we accomplish with this? Well, quite a lot, actually – in a very real sense, we’re only limited by our imagination.

GetMember

The first ability we want is to be able to extract a column out of the data table; given a DynamicDataTable “foo”, the expression “foo.Bar” should give us something enumerable that represents the data in the column. The DLR describes this operation as “get member”, and DLR-based languages implement a GetMemberBinder in order to bind a dynamic “get member” operation.

DynamicObject makes it very easy for us to handle the GetMemberBinder. We simply override the virtual method TryGetMember and implement the behavior that we want. The binder has two properties: Name, which indicates the name of the member that is being bound, and IgnoreCase. You can reasonably expect that case-sensitive languages like C#, Ruby and Python will set IgnoreCase to false, while VB will set it to true.

        private DataColumn GetColumn(string name, bool ignoreCase) {
            if (!ignoreCase) {
                return _table.Columns[name];
            }
            for (int i = 0; i < _table.Columns.Count; i++) {
                if (_table.Columns[i].ColumnName.Equals(name, StringComparison.InvariantCultureIgnoreCase)) {
                    return _table.Columns[i];
                }
            }
            return null;
        }

        public override bool TryGetMember(GetMemberBinder binder, out object result) {
            var c = _table.Columns[binder.Name];
            if (c == null) {
                return base.TryGetMember(binder, out result);
            }
            var a = Array.CreateInstance(c.DataType, _table.Rows.Count);
            for (int i = 0; i < _table.Rows.Count; i++) {
                a.SetValue(_table.Rows[i][c], i);
            }
            result = a;
            return true;
        }

Here I’ve chosen to return an Array whose elements are typed identically to the column’s original data type. That’s because it’s very easy to create an Array of a particular type and to set its individual elements from the System.Objects that we can get from the DataRow.

By factoring out GetColumn into a separate method, I’ve made it easy to change just this logic. We might want, for instance, to allow a symbol name like “hello_world” to match the column named “hello world”.

Non-dynamic members

What if I want to directly access other properties of the DataTable like the “Rows” DataRowCollection? The design of the DLR makes this easy. If you don’t handle a binding operation yourself, it’s possible to fall back to a default behavior implemented by the language-provided binder. And for VB, C#, Python and Ruby, the fallback behavior is to treat the object like a normal .NET object and to access its features via Reflection. That’s why it’s useful to call base.TryGetMember instead of throwing an exception when the column name can’t be found.

So if we implement a trivial “Rows” property, a reference to DynamicDataTable.Rows will return DataTable.Rows even when the GetMember is performed dynamically at runtime (unless there actually is a column named “Rows”…).

        public DataRowCollection Rows {
            get { return _table.Rows; }
        }

SetMember

The next interesting thing we want to be able to do is to set a column on the DataTable whether or not it already exists. The DLR describes this operation as “set member”, and defines a corresponding SetMemberBinder to perform the binding operation. Like the GetMemberBinder, this class has two properties: Name and IgnoreCase.

We want to be able to set the column either to a single repeated constant value or to a list of values. But there are lots of different lists we might like to support: for instance, lists, collections or even plain IEnumerables. Let’s make some decisions about the semantics of the SetMember operation on our type:

• If the object’s type implements IEnumerable and the object isn’t a System.String, then we’ll treat it like an enumeration. Otherwise, we’ll treat it like a single value.
• If it’s an IEnumerable<T> we’ll use the generic type as our DataType. For a plain IEnumerable, the DataType will be System.Object.
• If the object does not implement IEnumerable (or the object is a System.String) then the DataType will be the object’s actual RuntimeType.

For an enumeration, we’ll read items into a temporary array until we reach the number of rows in the table. If the enumeration ends before then, we’ll raise an error. If at that point, there are still additional items remaining in the enumeration, then we’ll also raise an error.

The specific behavior of our implementation for each of these types isn’t very important. What is important is that we’ve identified all the types that we expect we might get, and have identified the logic we’re going to implement for those types. Now, on to the code!

        public override bool TrySetMember(SetMemberBinder binder, object value) {
            Type dataType;
            IEnumerable values = (value is string) ? null : (value as IEnumerable);
            bool rangeCheck = (values != null);
            
            if (values != null) {
                dataType = GetGenericTypeOfArityOne(value.GetType(), typeof(IEnumerable<>)) ?? typeof(object);
            } else {
                values = ConstantEnumerator(value);
                dataType = (value != null) ? value.GetType() : typeof(object);
            }
            
            object[] data = new object[_table.Rows.Count];
            var nc = values.GetEnumerator();
            int rc = _table.Rows.Count;
            for (int i = 0; i < rc; i++) {
                if (!nc.MoveNext()) {
                    throw new ArgumentException(String.Format("Only {0} values found ({1} needed)", i, rc));
                }
                data[i] = nc.Current;
            }
            if (rangeCheck && nc.MoveNext()) {
                throw new ArgumentException(String.Format("More than {0} values found", rc));
            }
            
            var c = GetColumn(binder.Name, binder.IgnoreCase);
            if (c != null && c.DataType != dataType) {
                _table.Columns.Remove(c);
                c = null;
            }
            if (c == null) {
                c = _table.Columns.Add(binder.Name, dataType);
            }
            
            for (int i = 0; i < rc; i++) {
                _table.Rows[i][c] = data[i];
            }
            return true;
        }

(GetGenericTypeOfArityOne and ConstantEnumerator are methods whose names are pretty self-explanatory – and whose implementations can be found in the downloadable source code).

Armed with these two methods, our type now supports all of the operations we need to implement the sample program described in Part 0 of this series. A version of the complete source code can be downloaded this location.

In Part 2, we’ll add the ability to perform numerical operations between columns. See you then!

• Dismal Performance with IronPython

Significant claims have been made about the performance of IronPython, notably back at its inception in 2004 when Jim Hugunin, creator of both IronPython and its cousin Jython, presented a paper on IronPython performance at PyCon. Since then, there have been numerous claims to IronPython’s supremacy over CPython in the performance stakes. The IronPython Performance Report reiterates that IronPython can turn in a good performance. According to Hugunin the standard line we’ll see is,

“IronPython is fast – up to 1.8x faster than CPython on the standard pystone benchmark.

But do these claims stand up in the face of real-world Python code?

The claims of good performance are based on synthetic micro-benchmarks which don’t accurately reflect balance of coding techniques found in complete programs.

At this point I’d like to offer my own quote:

“IronPython can be slow – 10x to 100x slower than CPython on real-world code and it has been observed to be up to 6000x slower.

• IronPython 2.0 and Jython 2.5 performance compared to Python 2.5

• IronPython Hammers CPython when not mutating class variables

This made a phenomenal difference with IronPython completing in only 12% of the time taken by CPython – over 8× faster with a binary tree depth of 20.

• On Performance
  

Ahh, performance… It’s so much fun! It’s easy to measure and as you work on it you can see immediate benefits (or not!). I personally enjoy getting to spend time looking at interesting performance issues and I’ll use this as a time to plug IronPython 2.6 Beta 1 where we’ve continued our focus in this area. Even more fun yet is when you compare and contrast different implementations - it can be both surprising and amusing. Recently Robert Smallshire has been doing just that. Obviously Robert has found a case in IronPython where something is performing horribly. I’d like to take a look at what’s happening here and what we can do to improve this case.

Ok, but let’s get onto the fun technical details! It all comes down to one line of code “Node.counter += 1”. That’s a pretty innocent looking piece of code, how can it cause so many problems? First you need to understand a little bit about how IronPython works. IronPython internally keeps versions of type objects and uses this information to generate optimal code paths for accessing those objects. So the next line of code in this example, “self._children = children” usually turns into something that’s like:

if (self != null && self.GetType() == typeof(PyObject) && self.PythonType.Version == 42) {

self._dict["_children"] = ;

}

What that one line of code causes then is for the version of the type to change and for this “rule” to be invalidated. So we’ll figure out what to do, generate a new cached rule, only to have it not work the next time around. The same thing applies to the calls to Node – we re-generate the “optimal” code again and again only to discover we have to throw it away.

...

So you might be curious - what’s the fix? Well in this case we’re just mutating the type by changing a simple value stored in the type (versus say adding a property or a method to the type). Today we get a very slight throughput performance benefit because we can burn the value directly into the generated code instead of fetching it from the wrapper the type object keeps it in. With the fix we’ll burn the wrapper in and fetch the value from the wrapper each time. Then when we go and modify the type we just modify the wrapper instead of the type and its version is unchanged! This fix is now checked into the IronPython source tree and you can download the fixed sources right now.

In conclusion I’d like to wrap up a little bit about our performance philosophy – performance is hard and deciding which numbers matter to you is important. We have tried to tune IronPython for apps which run for more than a fraction of a second. We have also tuned IronPython for code which is not constantly mutating types at runtime. And we’re always open to feedback when users encounter problems with IronPython and we’ll listen to what scenarios are important for them. But if you look at these graphs I think you can see that we’ve done a good job at making IronPython a fast implementation of Python.

• C# 4.0 Specification Document

19. Introduction to C# 4.0

The major theme for C# 4.0 is dynamic programming. Increasingly, objects are “dynamic” in the sense that their structure and behavior is not captured by a static type, or at least not one that the compiler knows about when compiling your program. Some examples include

• Objects from dynamic programming languages, such as Python or Ruby
• COM objects accessed through IDispatch
• Ordinary .NET types accessed through reflection
• Objects with changing structure, such as HTML DOM objects

While C# remains a statically typed language, we aim to vastly improve the interaction with such objects.

Dynamic lookup allows you to write method, operator and indexer calls, property and field accesses, and even object invocations which bypass the normal static binding of C# and instead gets resolved dynamically.

dynamic d = GetDynamicObject(…);
d.M(7);

dynamic d = 7; // statically bound implicit conversion
int i = d; // dynamically bound implicit conversion

dynamic d = GetDynamicObject(…);
d.M(7); // calling methods
d.M(x: “Hello”); // passing arguments by name
d.f = d.P; // getting and setting fields and properties
d[“one”] = d[“two”]; // getting and setting through indexers
int i = d + 3; // calling operators
string s = d(5,7); // invoking as a delegate
C c = new C(d); // selecting constructors

dynamic d1 = new Foo();
dynamic d2 = new Bar();
string s;
d1.M(s, d2, 3, null);

“Perform an instance method call of M with the following arguments:

• a string
• a dynamic
• a literal int 3
• a literal object null”

• Reflection is used to obtain the actual runtime types of the two objects, d1 and d2, that did not have a static type (or rather had the static type dynamic). The result is Foo for d1 and Bar for d2.
• Method lookup and overload resolution is performed on the type Foo with the call M(string,Bar,3,null) using ordinary C# semantics.
• If the method is found it is invoked; otherwise a runtime exception is thrown.

type:
...
dynamic

• Expressions of type dynamic can cause dynamic binding to occur in specific situations
• Type inference algorithms as described in §7.4 will prefer dynamic over object if both are candidates

• There is an implicit identity conversion between object and dynamic
• There is an implicit identity conversion between constructed types that differ only by dynamic versus object
• Method signatures that differ only by dynamic versus object are considered the same
• Like with object, there is an implicit conversion from every type (other than pointer types) to dynamic and an explicit conversion from dynamic to every such type.

typeof(dynamic) == typeof(object)

• Member access: e.M
• Method invocation: e.M(e1,…,en)
• Delegate invocaton: e(e1,…,en)
• Element access: e[e1,…,en]
• Constructor calls: new C(e1,…,en)
• Overloaded unary operators: +, -, !, ~, ++, --, true, false
• Overloaded binary operators: +, -, *, /, %, &, &&, |, ||, ??, ^, <<, >>, ==,!=, >, <, >=, <=
• Compound assignment operators: +=, -=, etc.
• Implicit and explicit conversions

• Conversions, which has the static type that is being converted to
• Constructor invocations which have the static type that is being constructed

• Element accesses where the static type of the receiver is an array type
• Delegate invocations where the static type of the delegate is a delegate type

• Static method calls with dynamic arguments
• Instance method calls where the static type of the receiver is not dynamic
• Indexer calls where the static type of the receiver is not dynamic
• Constructor calls

• Checking that the candidate has the right name and arity
• Performing a partial type inference to check that inferences do exist where not depending on arguments with the static type dynamic
• Checking that any arguments not statically typed as dynamic match parameter types that are known at compile time

using (expression) statement

{ IDisposable __d = (IDisposable)expression // as a dynamic cast
try {
statement;
}
finally {
if (__d != null) __d.Dispose();
}
}

using (dynamic resource = expression) statement

{
dynamic resource = expression;
IDisposable __d = (IDisposable)resource // as a dynamic cast
try {
statement;
}
finally {
if (__d != null) __d.Dispose();
}
}

• x is of the form d.X where d is of type dynamic
• the runtime type of d declares X to have type T or T? where T is a primitive type
• the result of x binop y has the runtime type S where S is a primitive type
• S and T are either both integral types (sbyte, byte, short, ushort, int, uint, long or ulong) or both floating point types (float or double)
• S is implicitly convertible to T

• If the receiver is a dynamic object – i.e., implements an implementation-specific interface that we shall refer to as IDynamicObject – the object itself programmatically defines the resolution of the operations performed on it.
• Otherwise the operation gets resolved at runtime in the same way as it would have at compile time, using the runtime type of any constituent value statically typed as dynamic and the compile time type of any other constituent value.
• If a constituent value derives from a literal, the dynamic binding is able to take that into account. For instance, some conversions are available only on literals.
• If a constituent value of static type dynamic has the runtime value null, it will be treated as if the literal null was used.
• Extension method invocations will not be considered – the set of available extension methods at the site of the call is not preserved for the runtime binding to use.
• If the runtime binding of the operation succeeds, the operation is immediately performed,otherwise a runtime error occurs.

COM methods are often designed for a language environment that is more dynamic than C#. This means that they will often return weakly typed results, and rely on the calling language to dynamically look up further operations on those results.

Ahh, performance…  It’s so much fun!  It’s easy to measure and as you work on it you can see immediate benefits (or not!).  I personally enjoy getting to spend time looking at interesting performance issues and I’ll use this as a time to plug IronPython 2.6 Beta 1 where we’ve continued our focus in this area.  Even more fun yet is when you compare and contrast different implementations - it can be both surprising and amusing.  Recently Robert Smallshire has been doing just that: http://www.smallshire.org.uk/sufficientlysmall/2009/05/22/ironpython-2-0-and-jython-2-5-performance-compared-to-python-2-5/ Obviously Robert has found a case in IronPython where something is performing horribly.  I’d like to take a look at what’s happening here and what we can do to improve this case. 

But before I look at what’s going on here I’d like to briefly say we love to hear feedback from users of IronPython.  We’ve have a mailing list you can join and give us your feedback on and bugs can be reported on our website.  We track both of these very closely and respond as quickly as we can – which is usually pretty fast.  I just mention that because it’s easier for us to improve IronPython when people let us know about a problem vs. us having to scour the web to find issues.  Hopefully this piqued interest will have more people trying out IronPython and giving us feedback.

Ok, but let’s get onto the fun technical details!  It all comes down to one line of code “Node.counter += 1”.  That’s a pretty innocent looking piece of code, how can it cause so many problems?  First you need to understand a little bit about how IronPython works.  IronPython internally keeps versions of type objects and uses this information to generate optimal code paths for accessing those objects.  So the next line of code in this example, “self._children = children” usually turns into something that’s like:

What that one line of code causes then is for the version of the type to change and for this “rule” to be invalidated.  So we’ll figure out what to do, generate a new cached rule, only to have it not work the next time around.  The same thing applies to the calls to Node – we re-generate the “optimal” code again and again only to discover we have to throw it away.

So that sounds pretty bad but if you’re familiar with Python’s member access rules you’ll quickly recognize that this will be much faster than needing to do the resolution again and again.  That involves looking through the class hierarchy to see if we have any data descriptors or classes defining __setattr__ and it's nice to avoid all of that work.   So to get the best performance we require for the world to settle down and not be continuously changing.  We’re well aware of this problem with our implementation and we have continued to focus on improving it.  IronPython 2.0 already includes one mitigation so that we don’t need to continuously generate new code – instead we effectively patch new values into an existing piece of code.  In our IronPython 2.6 branch we’ve continued to improve performance here and significantly reduced the expense when things get mutated.  But it turns out that there’s an easy fix for this particular problem we’ve just never thought it that important of a problem to solve.

One of the amusing things about this is we’ve actually seen it before – Resolver Systems was having some performance issues with Resolver One  which is their advanced spreadsheet which is written in IronPython.  One of those issues turned to be a type which they both used frequently and mutated frequently.  This actually doesn’t seem to be a common idiom in Python code which is the reason we’ve never optimized for it before.  In this case though it turned out that they had found an optimization that improved performance on IronPython 1.0 but was destroying perf on IronPython 2.0!  For Resolver fixing this was a simple change for them to no longer mutate types.  Even if you really want to use a type as a property bag you can always move that to a type which you aren’t creating instances of and executing methods on.

With all that in mind let’s look at some numbers.  Here I’ve compared IronPython 2.01, IronPython 2.6 Beta 1, CPython 2.6 plus IronPython 2.6 with a fix for this issue.

Like Robert’s charts I have tree size in number of nodes across the bottom and execution time in seconds across the top and the execution times are logarithmic.  We can see the abysmal performance of 2.0.1, we can see that 2.6B1 has already made significant improvements, and with IronPython 2.6B1 plus a fix for this specific issue we actually beat CPython as the number of iterations scales up.

Here you can see that all 3 versions of IronPython are performing about the same and are coming in significantly faster than CPython as the number of iterations goes up.  These are now also matching the numbers Robert is getting when he has made the same change.

So you might be curious - what’s the fix?  Well in this case we’re just mutating the type by changing a simple value stored in the type (versus say adding a property or a method to the type).  Today we get a very slight throughput performance benefit because we can burn the value directly into the generated code instead of fetching it from the wrapper the type object keeps it in.  With the fix we’ll burn the wrapper in and fetch the value from the wrapper each time.  Then when we go and modify the type we just modify the wrapper instead of the type and its version is unchanged!  This fix is now checked into the IronPython source tree and you can download the fixed sources right now. 

In conclusion I’d like to wrap up a little bit about our performance philosophy – performance is hard and deciding which numbers matter to you is important.  We have tried to tune IronPython for apps which run for more than a fraction of a second.  We have also tuned IronPython for code which is not constantly mutating types at runtime.  And we’re always open to feedback when users encounter problems with IronPython and we’ll listen to what scenarios are important for them.  But if you look at these graphs I think you can see that we’ve done a good job at making IronPython a fast implementation of Python.

• IronRuby CTP for .NET 4.0 Beta 1

This is our second preview of IronRuby running under the Dynamic Language Runtime that is built directly into a .NET 4.0 release! As before, this release allows you to use IronRuby objects and types as .NET 4.0 dynamic objects from within C# and Visual Basic code. These binaries are roughly equivalent to the functionality found in IronRuby 0.3. Please also note that the IronRuby CTP for .NET 4.0 Beta 1 will run only under .NET 4.0 Beta 1.

• IronPython 2.6 CTP for .NET 4.0 Beta 1 (Codeplex release page)
• Harry Pierson (DevHawk - IronPython PM) announces the release
• Dave Fugate (IronPython tester) also announces it
  

We’re quite pleased to announce the release of IronPython 2.6 CTP for .NET 4.0 Beta 1. This is our second preview of IronPython running under the Dynamic Language Runtime that is built directly into a .NET 4.0 release! As before, this release allows you to use IronPython objects and types as .NET 4.0 dynamic objects from within C# and Visual Basic code. While this release does share a bit in common with the upcoming IronPython 2.6 Beta 1 release (e.g., a number of MSI improvements), the core functionality is essentially that of IronPython 2.6 Alpha 1. Please also note that IronPython 2.6 CTP for .NET 4.0 Beta 1 will run only under .NET 4.0 Beta 1.

• Dynamic programming in C# and Visual Basic
  

This walkthrough shows how the new Dynamic feature in C# 4.0 and Visual Basic 10.0 enables the natural use of object models that expect their host language to provide dynamic dispatch. This capability is demonstrated for an IronPython scenario of accessing libraries that are written for the Python language. Note that more advanced IronPython scenarios that involve hosting your own scripts with access to your application’s object model are beyond the scope of this walkthrough.

Notes:

• Using Dynamic with Office programming is demonstrated in the Office Programmability walkthrough.
• Using Dynamic with Silverlight will be shown in a later preview release of Visual Studio 2010.

• IronRuby 0.5 Released

Staying true to our "Conference-driven development schedule," this release is in conjunction with RailsConf '09, where IronRuby was demonstrated running real Rails applications and deploying them on IIS. A ton of work has been done in this release to get real Ruby programs running well, like RubyGems, Rake, Sinatra, and Rails.

• IronPython 2.6 Beta 1 (Codeplex release page)

We’re pleased to announce the release of IronPython 2.6 Beta 1. As you might imagine, this release is all about supporting new CPython 2.6 features such as the ‘bytes’ and ‘bytearray’ types (PEP 3112), decorators for classes (PEP 3129), advanced string formatting (PEP 3101), etc and as such includes nearly all of the new Python 2.6 features. The small number of remaining features will be implemented for Beta 2. The minimum .NET version required for this release is the same as IronPython 2.0; namely .NET 2.0 Service Pack 1.

There’ve also been several new features added: ctypes, sys.getframe, pyc for pre-compilation, and improved help. Ctypes provides access to C-based APIs and provides a way for Python developers to optimize performance or access libraries that would otherwise be unavailable. sys.getframe is commonly used by Python programs to introspect the call stack and can now be enabled with the –X:Frames and –X:FullFrames command line options; for better compatibility with programs that check for the presence of sys.getframe it is now not available unless these options are provided. The pyc.py pre-compiler is now shipped with IronPython and is available in the Tools\Scripts directory. Finally the help of many builtin functions and types has been improved by including IronPython.xml and IronPython.Modules.xml.

Another focus has been on improving startup time of large applications. This release accomplishes that through the adaptive compilation feature announced with 2.6 Alpha 1 and also includes many other enhancements to reduce the amount of code generation required to start applications. But only do these improvements help large applications but they also significantly speed up basic hello worlds startup time.

There’s also a few minor changes since IronPython 2.0.1 that are worth calling out here:

• Binaries are ngen’d for both 32-bit and 64-bit platforms on install by default
• IronPython.msi now offers a little more selection with respect to what you’d like to install. For example, Silverlight templates are optional
• The default installation location of IronPython.msi no longer indicates whether the 2.6 release is an Alpha, Beta, or a patched release. Future IronPython 2.6 installations will replace previous 2.6 releases which will be uninstalled automatically
• The -X:PreferComInteropAssembly flag has been removed. All COM interop is now done through normal COM dispatch

• Harry Pierson on IronPython 2.6 Beta 1

There are two big new features in this release. The first is our implementation of the ctypes module. The ctypes module is like P/Invoke for Python. It allows Python code to call into unmanaged DLL functions. Here, for example, I’m calling into the standard wprintf function from msvcrt.dll

IronPython 2.6 Beta 1 (2.6.0.10) on .NET 2.0.50727.4918
>>> import ctypes
>>> libc = ctypes.cdll.msvcrt
>>> ret = libc.wprintf("%s\n", "hello")
hello

Between ctypes and Ironclad, I think we’ll eventually be able to load most native Python extensions in IronPython. Woot!

The other big new feature in this release is a real implementation of sys._getframe. _getframe lets you write code that inspects the Python callstack.

• Dave Fugate on IronPython 2.6 Beta 1

IronPython 2.6 Beta 1 is the second IronPython release today! I'm happy to say that the sys._getframe issues of IronPython 2.6 Alpha 1 have been resolved, and we have re-added urllib.py and other modules dependent upon collections.py to the MSI installer. Man am I ever ready for the long weekend=)

To commemorate the first beta release of the first CLR to contain the DLR, I’m going to try to implement something more interesting than useful – a dynamic version of the DataTable class. My goal is to show how the dynamic support in C#4 and VB10 can be used to create a better development experience, and to show how you can work effectively with IDynamicMetaObjectProvider.

The BCL’s DataTable is already fairly dynamic; its columns can vary at runtime in type and name. You can even add and remove columns on-the-fly. As such, it’s a pretty good candidate for “dynamicizing”. Of course, we can’t actually modify the existing DataTable implementation, so I’ll start out by creating a wrapper for it using DynamicObject. Where I hope to end up is with a self-contained implementation that implements its own DynamicMetaObject rather than leaning on DynamicObject and DataTable.

Here’s a sneak preview of Part 1:

    public static class Program {
        private static DataTable CreateTable() {
            DataTable table = new DataTable();
            table.Columns.Add("Name", typeof(string));
            table.Columns.Add("Value", typeof(int));

            table.Rows.Add("Bob", 23);
            table.Rows.Add("Karen", 17);
            table.Rows.Add("Allen", 10);
            table.Rows.Add("Grace", 31);
            return table;
        }
        public static void Main(string[] args) {
            DataTable table = CreateTable();
            dynamic t = new DynamicDataTable(table);
            t.NewName = t.Name;
            t.Name = "unknown";
            foreach (var r in t.NewName) {
                System.Console.WriteLine(r);
            }
        }
    }

See you again soon!

As Soma announced, we just shipped VS2010 Beta1. This includes dump debugging support for managed code and a very cool bonus feature tucked in there that I’ll blog about today.

Dump-debugging (aka post-mortem debugging) is very useful and a long-requested feature for managed code.  The downside is that with a dump-file, you don’t have a live process anymore, and so property-evaluation won’t work. That’s because property evaluation is implemented by hijacking a thread in the debuggee to run the function of interest, commonly a ToString() or property-getter. There’s no live thread to hijack in post-mortem debugging.

We have a mitigation for that in VS2010. In addition to loading the dump file, we can also interpret the IL opcodes of the function and simulate execution to show the results in the debugger.

Here, I’ll just blog about the end-user experience and some top-level points. I’ll save the technical drill down for future blogs.

Consider the following sample:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Reflection;

public class Point 
{
    int m_x;
    int m_y;
    public Point(int x, int y)
    {
        m_x = x;
        m_y = y;
    }
    public override string ToString()
    {
        return String.Format("({0},{1})", this.X, this.Y);
    }

    public int X
    {
        get
        {
            return m_x;
        }
    }
    public int Y
    {
        get
        {
            return m_y;
        }
    }
}

public class Program
{
    static void Main(string[] args)
    {
        Dictionary<int, string> dict = new Dictionary<int, string>();
        dict[5] = "Five";
        dict[3] = "three";

        Point p = new Point(3, 4);
                
    }

    public static int  Dot(Point p1, Point p2)
    {
        int r2 = p1.X * p2.X + p1.Y * p2.Y;
        return r2;
    }

}

Suppose you have a dump-file from a thread stopped at the end of Main() (See newly added menu item “Debug | Save Dump As …”; load dump-file via “File | Open | File …”).

Normally, you could see the locals (dict, p) and their raw fields, but you wouldn’t be able to see the properties or ToString() values. So it would look something like this:

But with the interpreter, you can actually simulate execution. With the IL interpreter, here’s what it looks like in the watch window:

Which is exactly what you’d expect with live-debugging.  (In one sense, “everything still works like it worked before” is not a gratifying demo…)

The ‘*’ after the values are indications that they came from the interpreter.  Note you still need to ensure that property-evaluation is enabled in “Tools | options | Debugging”:

How does it work?
The Interpreter gets the raw IL opcodes via ICorDebug and then simulates execution of those opcodes. For example, when you inspect “p.X” in the watch window, the debugger can get the raw IL opcodes:

.method public hidebysig specialname instance int32
        get_X() cil managed
{
  // Code size       12 (0xc)
  .maxstack  1
  .locals init ([0] int32 CS$1$0000)
  IL_0000:  nop
  IL_0001:  ldarg.0
  IL_0002:  ldfld      int32 Point::m_x
  IL_0007:  stloc.0
  IL_0008:  br.s       IL_000a
  IL_000a:  ldloc.0
  IL_000b:  ret
} // end of method Point::get_X

And then translate that ldfld opcode into a ICorDebug field fetch the same way it would fetch “p.m_x”. The problem gets a lot harder then that (eg, how does it interpret a newobj instruction?) but that’s the basic idea.

Other things it can interpret:

The immediate window is also wired up to use the interpreter when dump-debugging. Here are some sample things that work. Again, note the ‘*’ means the results are in the interpreter and the debuggee is not modified.

Simulating new objects:
? new Point(10,12).ToString()
"(10,12)"*

Basic reflection:
? typeof(Point).FullName
"Point"*

Dynamic method invocation:
? typeof(Point).GetMethod("get_X").Invoke(new Point(6,7), null)
0x00000006*

Calling functions, and even mixing debuggee data (the local variable ‘p’) with interpreter generated data (via the ‘new’ expression):
? Dot(p,new Point(10,20))
110*

It even works for Visualizers

Notice that it can even load the Visualizer for the Dictionary (dict) and show you the contents as a pretty array view rather than just the raw view of buckets. Visualizers are their own dll,  and we can verify that the dll is not actually loaded into the debuggee. For example, the Dictionary visualizer dll is  Microsoft.VisualStudio.DebuggerVisualizers.dll, but that’s not in the module list:

That’s because the interpreter has virtualized loading the visualizer dll into its own “virtual interpreter” space and not the actual debuggee process space. That’s important because in a dump file, you can’t load a visualizer dll post-mortem.

Other issues:

There are lots of other details here that I’m skipping over, like:

1. The interpreter is definitely not bullet proof. If it sees something it can’t interpreter (like a pinvoke or dangerous code), then it simply aborts the interpretation attempt.
2. The intepreter is recursive, so it can handle functions that call other functions. (Notice that ToString call get_X.)
3. How does it deal with side-effecting operations?
4. How does it handle virtual dispatch call opcodes?
5. How does it handle ecalls?
6. How does it handle reflection

Other advantages?

There are other advantages of IL interpretation for function evaluation, mainly that it addresses the ”func-eval is evil” problems by essentially degenerating dangerous func-evals to safe field accesses.

1. It is provably safe because it errs on the side of safety. The interpreter is completely non-invasive (it operates on a dump-file!).
2. No accidentally executing dangerous code.
3. Side-effect free func-evals. This is a natural consequence of it being non-invasive.
4. Bullet proof func-eval abort.
5. Bullet proof protection against recursive properties that stack-overflow.
6. It allows func-eval to occur at places previously impossible, such as in dump-files, when the thread is in native code, retail code, or places where there is no thread to hijack.

Closing thoughts

We realize that the interpreter is definitely not perfect. That’s part of why we choose to have it active in dump-files but not replace func-eval in regular execution. For dump-file scenarios, it took something that would have been completely broken and made many things work. 

The Visual Studio 2010 SDK Beta 1 is now available for download. Be sure to check out the latest supplemental readme as you’re likely to encounter some of these issues in using the SDK.

I’m pleased to announce a new release of IronRuby: IronRuby 0.5!

You can also check out the source code for this release

Staying true to our "Conference-driven development schedule," this release is in conjunction with RailsConf '09, where IronRuby was demonstrated running real Rails applications and deploying them on IIS. A ton of work has been done in this release to get real Ruby programs running well, like RubyGems, Rake, Sinatra, and Rails.

Read more about IronRuby at RailsConf here

Special thanks to Daniele Alessandri, Jirapong Nanta, and Ray Vernagus, who contributed code during this release, and everyone else who has submitted bugs, updated the wikis, or built software on-top of IronRuby.

Also, take a look at the IronRuby CTP for .NET 4.0 Beta 1, which has been released as part of Visual Studio 2010 Beta 1.

What’s in the Release?

A big language feature of this past release was adaptive compilation; the ability to start-up quicker with a fast interpreter, and then compile code that gets interpreted more than two times. This brought Rails startup time down from 89 seconds to 34 seconds, giving Rails approximately 2.5x quicker startup. Since this compiles “hot” code, your Rails application will get faster after an initial set of requests. There is a lot more to do for performance, but this was a great accomplishment.

Here are some more interesting changes:

• Removes ClrString. CLR strings are referred to as System::String
• Removes IronRuby library. "require 'IronRuby'" is no longer needed; IronRuby module is now a built-in module.
• Integer/Float operations for all CLR primitive numeric types
• System::Char and System::String methods so that they behave like an immutable UTF-8 encoded string
• Improves handling of BOM in 1.8 mode
• Debugger visualizer for RubyObject so that VS will display the result of to_s in the Watch window.
• Adds support for .NET operators (op_* special methods)
• Implements bitwise operators on flag enums
• Precompilation of simple Ruby method invocations
• Improves performance for file IO#read method. Previously we were 10x slower than MRI, now we are approx. 1.5-times faster.
• Tons of fixes to File, String, Array, and other built-in types to drive up RubySpec and Ruby application test-suite pass-rates.

For detailed information, please see the CHANGELOG (which is much more verbose now. including all commit messages for the release … not just syncing to head of tfs)

Bugs closed

• "ir script\server" causes a YAML parser error
• "igem install rails" fails
• yield fails in eval'd code
• Generic type conversion from Fixnum to Integer
• Pathname#cleanpath messes up the pathname
• Array#hash should properly handle recursive arrays
• ci_files set needed in mspec config
• autoload does not use File::SEPARATOR
• Time class instance
• __FILE__ doesn't return a canonicalized path
• … and various old RubyForge bugs during the move to CodePlex

RubySpec status

IronRuby passed the 80% mark on RubySpec during this release:

You can see more IronRuby statistics on http://ironruby.info.

Also during this release IronRuby has been running other Ruby applications test-suites as part of individual developer tests and in continuous integration, such as RubyGems, Rails (including ActiveRecord, ActiveSupport, ActionPack), and Rake.

Next Release

The next release will push harder on application compatibility, especially Cucumber (which has been the most requested application), Rails, and all the other frameworks being tested currently.

Also, the feature which makes IronRuby stand out, .NET integration, has next-to-no documentation, so that will large focus during this next release as well. Keep an eye on http://ironruby.net/Documentation/.NET, but as I said, is very sparse at the moment.

In the meantime, please test out IronRuby 0.5 and let us know all the issues you find. Enjoy!

We’re pleased to announce the release of the IronRuby CTP for .NET 4.0 Beta 1!

Download IronRuby CTP for .NET 4.0 Beta 1

This is our second preview of IronRuby running under the Dynamic Language Runtime that is built directly into a .NET 4.0 release! As before, this release allows you to use IronRuby objects and types as .NET 4.0 dynamic objects from within C# and Visual Basic code. These binaries are roughly equivalent to the functionality found in IronRuby 0.3. Please also note that the IronRuby CTP for .NET 4.0 Beta 1 will run only under .NET 4.0 Beta 1.

This release also includes the binaries for the IronPython 2.6 CTP for .NET 4.0 Beta 1.

To try out this preview release:

1. Install some variant of .NET 4.0 Beta 1 or Visual Studio 2010 Beta 1 from here
2. Download the release
3. Follow any of the many dynamic walkthroughs online. Here would be a good start.

Enjoy!

In addition to the IronPython CTP for .NET Framework 4.0 Beta 1 I blogged about earlier, we also released the first beta of IronPython 2.6 today. How about that – two IronPython releases in one day! This is our second preview release as we work towards our 2.6 RTM in September. 2.6 Alpha 1 was released back in March.

There are two big new features in this release. The first is our implementation of the ctypes module. The ctypes module is like P/Invoke for Python. It allows Python code to call into unmanaged DLL functions. Here, for example, I’m calling into the standard wprintf function from msvcrt.dll

Between ctypes and Ironclad, I think we’ll eventually be able to load most native Python extensions in IronPython. Woot!

The other big new feature in this release is a real implementation of sys._getframe. _getframe lets you write code that inspects the Python callstack. Previously, we supported _getframe only with a depth of zero which is to say you could inspect the current frame, but no others. Now, by default we don’t implement _getframe at all unless you pass in –X:Frames or –X:FullFrames on the command line. Removing the version of _getframe that only worked for depth zero fixes an issue with collections.py that broke much of the 2.6 standard library in IronPython 2.6 Alpha 1.

The difference between Frames and FullFrames is in what is returned by frame.f_locals member. If you’re running with FullFrames, we hoist all local variables into the heap so they can be accessed by our frame walker. If you’re running with Frames, our ability to access locals up the stack is limited. Sometimes they are available - If you called locals() in a frame up the stack for example, then f_locals will be available – but usually not. There’s a performance difference between the default (i.e. no Frames), –X:Frames and –X:FullFrames, hence why we provide the user fine grained control over the Frame support.

Our performance has gotten better relative to 2.6 Alpha 1. Our PyStone numbers have improved 80% from Alpha 1, similar to where we were in IronPython 2.0.1. We’ve also been able to cut our startup time about 25% from 2.0.1. We’re still an order of magnitude slower than CPython on startup, but we’re getting better. We’re significantly worse on PyBench than we were in 2.6 Alpha 1, but that’s primarily because there’s now a second exception test. As I described back in March, we get killed on the exceptions benchmarks – the two combine to consume nearly 62% of our total run time. Ouch!

Finally, there are bug fixes. Of particular relevance to readers of this blog are a series of fixes that allow me to continue on with my __clrtype__ series. Watch for that soon.

As I said back when we released Alpha 1, the release cycle on 2.6 will be much shorter than it was for 2.0. 2.0 had eight alphas, five betas and two release candidates over the course of around twenty months. We expect 2.6 to have one alpha, two betas and a release candidate over eight months. So please start trying using the beta as soon as you can so you can give us your feedback and we can fix your bugs!

The .NET Framework 4.0 and Visual Studio 2010 Beta 1 is now generally available for download. Jason Zander has a very thorough rundown on some of the new features in this release. Of course, my favorite new features in VS2010 is the new dynamic language support in C# and Visual Basic, which let’s you easily call out to IronPython code from those languages.

For anyone who wants to experiment with interoperating C# or VB with IronPython, we released IronPython 2.6 CTP for .NET 4.0 Beta 1 today. There’s also a walkthru showing how you can use the standard Python library module random from both C# and VB. Note, there’s currently a URL bug in that walkthru – it links to IronPython 2.6 Alpha 1 rather than the .NET 4.0 Beta 1 IronPython CTP. Make sure you pick up the right version of IronPython if you want to try out the walkthru. Looks like they fixed the redirect in the walkthru.

FYI, this is a CTP quality release with about the same functionality as IronPython 2.6 Alpha 1.  Essentially, this is the version of IronPython that was in the source tree when the VS team branched for Beta 1.

If you’ve got any feedback, please drop us a line on the mailing list.