Planet IronPython

Today we released Mono 2.0 to the world. You can download sources and binaries from our download page. And our official release notes are up as well. This of course would not be possible without the open source contributors that worked tirelessly on Mono sending patches, fixing bugs, helping the community, answering questions, creating test cases and supporting us all these years.

Mono 2.0 is both a runtime for application and a kit for developers for writing applications with C# and other ECMA CLI languages for a wide spectrum of uses.

Big thanks go to the fantastic Mono team at Novell that has kept the excitement and the pace over all these years (we started in 2001), the large contributions from Mainsoft, Unity3D and our users that pushed us to fix bugs, implement new features and tune Mono. Also, we very much appreciate the work of the ECMA 334 and 335 committee members that worked on the CLI and C# specifications and everyone at Microsoft that answered our questions over the years and specially those that licensed code under open source licenses.

We originally started to work on Mono, because we wanted to make developers happier and more productive on Linux. We liked C#, we liked the CIL and we wanted to have those technologies available on our platform.

Since we have been active in the Linux Desktop world, it is not a surprise that the early use of Mono was mostly on Linux desktop applications, and Mono continues to shine there. Server-side use of Mono was a natural evolution and we soon were powering ASP.NET sites on Linux.

There is one area where we under-delivered in the past, and it has been a constant source of pain. Up until now, we did not have a working debugger. This has finally changed, and Mono 2.0 includes for the first time a debugger, the time for WriteLine() debugging is now behind us.

As the project matured, developers started taking advantage of Mono's open source nature: essentially .NET on their own terms. A platform that could be adapted, morphed, ported and modified to suit many different uses. Today Mono is embedded in portable mp3 players and powers Unity3D's game engine on the Apple iPhone, the Nintendo Wii, MacOS X and Windows (Some folks at Novell are working with Unity on bringing Unity3d to Linux!).

It has also been deployed to run code on large clusters of servers for SecondLife, powers our open source Silverlight implementation (Moonlight) and powers the popular DekiWiki: a Social Collaboration Tool.

Mono is a large project and it is hard to pick one feature to talk about as there are so many, so instead I put together a quick table of the major features that are part of this release:

We have ported Mono to a wide variety of platforms and operating systems on this 1.0 to 2.0 cycle. These platforms include:

• Linux (x86, x86-64, PowerPC32, Itanium, SPARC, ARM, s390, s390x.
• Solaris (x86-64, SPARC).
• MacOS X (x86, PowerPC32).
• Windows (x86, support for x86-64 will come in Mono 2.2).
• Nintendo's Wii.
• iPhone/iPod Touch (ARM, limited functionality due to licensing requirements; I will blog later this week about this).
• *BSD (x86, x86-64).

Developing with Mono

Long time Linux developers will probably continue to use Emacs and VI, but some new Linux developers might want to use an IDE. New developers can use our open source MonoDevelop IDE on Linux, or alternatively the commercial X-Develop IDE or SlickEdit.

If you are a Windows developer, you can continue using Visual Studio or your IDE of choice to write the code and compile it. Your binaries will run just fine on Linux.

To assist Windows developers in porting their applications to Unix, we have provided the Mono Migration Analysis tool.

Runtime Changes

The Mono Virtual Machine gained plenty of features since Mono 1.2 was released. We have added:

• Generic Code Sharing and VTable Compression have been implemented: this significantly reduces the memory footprint of generic type instantiations, while still getting the speed benefits of using generics.
• AOT support: in addition to x86, we now also support ARM and x86-64.
• COM interop is now complete (works on Windows with "real COM" and can be used on Unix with Mainsoft's COM or Mozilla's XPCOM).
• AOT code can now AOT parts of 2.0 assemblies (assemblies that contain generics).
• Full AOT support (allows code to run JIT-less, this is limited to 1.0 code).
• CIL Verifier: Now Mono has a CIL verifier.
• CoreCLR Security: the security system used by Moonlight.
• Many optimizations that improve execution performance: New intrinsics operations (Math.Min/Max for example); various operations are now inlined by the JIT; managed allocations (no transition to unmanaged code for allocating memory); multi-array access is now tuned by the JIT; constant and block initializations are now handled by the JIT; Faster initialization and access to multi-dimensional arrays (4x faster).
• The runtime went on a diet, many runtime data structures are smaller making Mono lighter.

Tools

In addition the the Mono Debugger making its debut appearance on this release, we are very proud of our code analyzer Gendarme.

Gendarme is a extensible rule-based tool to find problems in .NET applications and libraries. Gendarme inspects programs and libraries that contain code in ECMA CIL format (Mono and .NET) and looks for common problems with the code, problems that compilers do not typically check or have not historically checked.

Feedback

Mono is not perfect, but we want to improve it. Like many other open source projects, we need your bug reports to improve Mono. If you have problems with Mono, help us by filing a bug report.

Special Thanks

Special thanks to Hacker Extraordinaire Aaron Bockover who not only brings us the best media player in the world, but created the new web site design and implemented and tuned it over very long extra hours up until 7am in the morning on his weekend.

And to our packaging and QA team that spent extra hours to get all the bits and pieces in place for the release.

• Compiling the DLR, IronRuby, and IronPython for ANY version of Silverlight

• Dynamic Languages in Silverlight 2 RC0

I loved this LWN article on the changes necessary to make Linux boot in 5 seconds on the Asus EEE PC (a relatively slow computer).

Hopefully all Linux distributions will adopt these changes for custom deployments.

A couple of weeks ago we started the work on porting Microsoft's Media Codecs to Linux and we got the C version running.

Popfly in Firefox3/Linux/x86

Geoff, Fernando and Rolf have been hard at work on this, and have also added the infrastructure to download and install the codecs on demand.

The next step was getting all the assembly language supported in Linux, and today Geoff got the assembly optimized SSE1 audio decoder running (the first chunk of the decoders).

Of course, the rest of the team has been busy fixing bugs and improving the performance in preparation for the first public beta of Moonlight.

MichaelFoord:

• OpenBabel, IronPython and ADME filtering

A couple of weeks ago I suggested that developers interested in having their .NET software run in other platforms should avoid Microsoft's Managed Extensibility Framework (MEF) as it was not an open source library.

I had a chance to discuss with Glenn, Sam and Bob the benefits of using the MS-PL for this library first over twitter and then over email.

Representing .NET's loyal competitor, I did not think that we stood a chance of getting Microsoft to change the license, but I was pleasantly surprised. Glenn understood the value of open source, Sam wanted to do the right thing about this library and CodePlex and Bob argued that Mono already had Mono.Addins anyways.

Today Glenn announced that Microsoft has changed the license for MEF to the open source MS-PL license.

Thanks to everyone at Microsoft that helped change the license!

Atsushi Enomoto blogs about the work involved in bringing LINQ to Databases to Mono.

The effort was a joint collaboration between the awesome DbLINQ developers, Pablo Iñigo Blasco our Google Summer of Code Student and Novell's Atsushi Enomoto.

The DbLINQ developers had created a general purpose LINQ provider that could be used with database providers other than SQL Server. They also relicensed their code a few months ago to the MIT X11 license to allow for integration with Mono's code base.

Read Atsushi's description of how the effort was put together and how DbLINQ is being refactored to become a full System.Data.Linq implementation and still provide the foundation for third-parties to easily create database LINQ providers.

DbLINQ is a great project, and it needs your help to complete the effort.

Great News Mono Lovers!

Later this month I will be presenting the session "Mono and .NET" at the Microsoft Professional Developers Conference in LA.

Exciting times!

Update: My talk will cover new technologies that we have created as part of Mono. Some of them are reusable on .NET (we try to make our code cross platform) and some other are features that specific to Mono's implementation of the CLI.

This weekend the news came out that Microsoft was going to bundle and support John Resig's jQuery as part of Visual Studio and ASP.NET. From Scott's blog:

I'm excited today to announce that Microsoft will be shipping jQuery with Visual Studio going forward. We will distribute the jQuery JavaScript library as-is, and will not be forking or changing the source from the main jQuery branch. The files will continue to use and ship under the existing jQuery MIT license.

Beyond the obvious benefits to developers, what interests me is that Microsoft is now bundling an open source component into their commercial offerings.

This is a first time for Microsoft.

With IronPython they continued development of an open source project in house. With IronRuby, they were open to external contributions to the libraries of IronRuby. In both cases they were not taking external code or contributions directly into their core products.

Hopefully they will start using more open source code in their products. Maybe one day they will bundle Mono's Cecil or Mono's embeddable C# compiler.

I've just released the Silverlight Dynamic Languages SDK with the newest DLR, IronRuby, IronPython, and JScript binaries and sources to work against Silverlight 2 RC0!

Download the SDK!

This release works with Silverlight 2 RC0. Keep in mind this is a developer release, so no "end-user" runtime exists. This is why the "not-installed" experience doesn't take you directly to the download, but to a page where you can install the tools.

As usual, report any issues on the Issue Tracker, and feel free to ask any questions on the Discussions tab.

This release has slightly refactored Microsoft.Scripting.Silverlight, adding two new types: "Package" and "Configuration". You can use "Package" to access files inside the XAP, and "Configuration" can be used to detect languages available in Silverlight.

Hosting

In past releases, if you wanted to host the DLR you needed to create your own ScriptRuntimeSetup which uses Microsoft.Scripting.Silverlight.BrowserScriptHost to tell your hosted scripts about how Silverlight. Your code would look like this:

using Microsoft.Scripting.Silverlight;
using Microsoft.Scripting.Hosting;

//...

var setup = new ScriptRuntimeSetup();
setup.HostType = typeof(BrowserScriptHost);
var runtime = new ScriptRuntime(setup);

//...

Now, since the ScriptRuntimeSetup we use to run user code is public, you can simply this code to:

using Microsoft.Scripting.Silverlight;
using Microsoft.Scripting.Hosting;

//...

var runtime = DynamicApplication.Current.Runtime;

// or if you wanted to create your own runtime, reuse the setup

var runtime = new ScriptRuntime(DynamicApplication.Current.Runtime.Setup);

//...

Blah, blah, blah, I always talk about the future. Has anything happened? Nope. Well, it will eventually. Just keep pestering. If you have no idea what I'm talking about, take a look at my last post.

I know you feel like this sometimes when trying to solve a problem. I do. Almost all the time. Even when I'm not angry ... but I digress.

Silverlight 2 RC0 was released this past Thursday, but anyone wanting to use the DLR in it was surprised ... no new binary release of the DLR bits for Silverlight 2 RC0 yet. As I said on Twitter, it would be delayed until today, but that shouldn't stop anyone from taking the sources and compiling them against the new SIlverlight build, right?

Of course! Everything should just work, since there were no major breaking changes in Silverlight that affect the DLR between Beta2 and RC0. So, you hacked up the csproj files to point at mscorlib.dll, system.dll, etc in the new Silverlight install directory (C:\Program Files\Microsoft Silverlight\2.0.30923.0), compile, and it builds fine. Then you try to run an app ...

"InitializeError- Failed to load the application. It was built with an obsolete version of Silverlight"

Poof! What the hell happened? That's a really bad error message, but what it means to say is:

"The application's AppManifest.xaml has a RuntimeVersion <= 2.0.30523.00, which is Silverlight 2 Beta 2's version number, so Silverlight 2 RC0 won't load this application."

So you see, the XAP file that was produced by Chiron is still for SL2 Beta2. But that's an easy fix;

1. Make sure Chiron.exe isn't running.
2. Go to your custom build, and edit Chiron.exe.config
3. Find the <AppManifest.xaml> section, and where it says "RuntimeVersion" make it's value be "2.0.30923.0" (anything with the first three version numbers being greater than the current will do, but be save and use the actual version.)
4. Re-run Chiron.exe, navigate to your Silverlight application, and .... it works!

Welcome to the wonderful world of versioning in Silverlight. :)

• Unit Testing in Python

• IronPython Talks to Google AppEngine

People are loving the C# Interactive Shell.

In the past people have embedded languages like Boo into their applications (MonoDevelop and Banshee for example both have options to embed Boo and a shell).

Zoltan designed a new system for Mono that allows developers to inject and execute code into running Mono applications. Paolo provided significant feedback and design guidelines for the code to be incorporated into Mono.

Thanks to both Zoltan and Paolo this functionality is now available through the Mono.Management assembly: you provide an executable and a PID, and the executable is injected and its Main method executed on the target process:

	using Mono.Attach;

	[..]

	// Get a handle to a running Mono process.
	VirtualMachine vm = new VirtualMachine (pid);

	// Load hello.exe into the target process, and run it
	vm.Attach ("/tmp/hello.exe", "");
	

You can use this for billions of things of course. You could patch running programs, you could attach logging software to a running program, or you could inject a C# evaluator into a live application and examine its state, or tweak it live.

Zoltan came up with a really cool extension to the the csharp command (this is the command-line C# Interactive Shell). The csharp command now takes an --attach command line argument and a PID.

The csharp shell can now use the Mono.Attach.VirtualMachine to injects itself into the remote process and then the client and server communicate through a couple of sockets.

For example, this is the sample that Zoltan used to pitch his idea for supporting attaching to the virtual machine. With the following you can pause a live Banshee process:

$ ps aux | grep banshee
miguel   12359 17.0  2.7 141372 55708 pts/5    Sl+  14:30   0:02 banshee-1 /usr/lib64/banshee-1/Nereid.exe
$ csharp --attach 12359
csharp> using Banshee.ServiceStack;
csharp> using Banshee.Gui;
csharp> var s = ServiceManager.Get ();
csharp> s.PlaybackActions ["PlayPauseAction"].Activate ();
	

All of the code is now on SVN, you need both the mono and mcs modules from trunk.

A GUI Shell

The above commands are a tiny bit risky and are also limited to the shell.

The above commands will execute on a separate thread from the application, and any commands that you execute would be executed on a separate thread which could corrupt the state of your application.

So this weekend, I wrote a tool that integrates with Gtk# applications, its called gsharp and you can find it in the mono-tools module (it borrows some code from Banshee).

gsharp is a Gtk# version of the csharp command. What is important is that it also supports injection of the code on other programs, but makes sure that all the code executes in the Gtk# thread, by issuing all of its commands from the Gtk# idle handler. This means that it is safe to use gsharp with your Gtk# applications.

GUI version of the tool.

Here you can select which project you want to inject the GUI into:

Needs some work, show process names.

This version shows the gsharp tool attached to F-Spot, as I am not very familiar with the codebase, I can not do very much with it:

We will need to implement one of these as well for Windows.Forms applications. This should luckily be easy to do as most of the smarts live in the Mono.CSharp assembly (our embeddable compiler).

Security of Agents

A couple of weeks ago, I asked for people to weigh in on a security concern for temporary files. This was for Zoltan's attach functionality.

The code is now implemented and I would love if security experts could do a source code audit of our implementation. And validate whether our assumptions are safe.

Here is the source code.

In C# the defaut access level for members in classes and structs is "private".

There is no need to pepper the source code with "private" everywhere. It only introduces noise and makes your code more difficult to read.

• Spider Python: Some Notes on IronPython

• Python list comprehensions can do the same as LINQ for simple cases, but LINQ is much more powerful, and it supports deferred execution, while list comprehensions are evaluated greedily. (Ed: generator expressions are evaluated lazily in Python)
  
• Every description of the python syntax I ever see emphasizes the fact you don't need to put in braces. Pity they don't spend more time telling you that you have to put in colons, that would actually be useful knowledge. This really bit me when I learnt boo, which is syntactically very similar.
• IronPython 1.0 targets CPython 2.4. This sounds fine until you realize that this was released in 2004. A fair bit has happened since then, not limited to the introduction of an inline if syntax. (Ed: IronPython 2 is Python 2.5 which is still the stable current version of Python)
  
• While we're on the subject, the inline if (defaultValue if condition else exceptionValue) is actually quite cool.
• The fairly lax approach to types means I don't need to call .Cast anymore.
• Tabs are the devil as far as indents in Boo and Python are concerned. I highly recommend using something that shows tabs explicitly, and then eliminating the lot.
• Instance methods that explicitly take self feels awkward to me.
• Static Methods that require an incantation such as "ToAbsoluteUrl = staticmethod(ToAbsoluteUrl)" also feels pretty awkward. (Ed: staticmethod can be used as a decorator which is much nicer)
  
• The casting to delegate isn't quite as slick as it is in C#. Passing in "spiderTracker.FoundUrl" instead of "lambda url: spiderTracker.FoundUrl(url)" results in an extremely unhelpful runtime error.
• The lambda syntax is pretty elegant, but so is C#3's. Indeed, C#3 seems to have the edge.
• Python's regular expressions are powerful, but not quite as good as .NET's. In particular search does what you'd expect match to do
• findall doesn't return matches. Rather, it returns the value of the first group of the matches. This is often actually what you wanted, but it's a bit peculiar. The semantics of "Matches" in .NET are much easier to understand.
• It did seem rather slow compared to the C# version. There are, of course, way too many variables here to make a final judgement, but it was disappointing.

• IronPython: I really hope this isn't going to happen in the final version

• Midnight Oil: Surprises while porting to IPy 2

1. If it's not tested it doesn't work. And when you change the environment under which your code runs you need to retest.
2. The unit tests paid off again, since they allowed me to find many problems in places I didn't expect.
3. I need to do more to flush out these hidden assumptions. One way is to add a randomized test that runs over longer periods and plays with some variables. I could easily randomize the order in which I go over the checks, inject random errors (the system is supposed to be self healing) or delays in some strategic points, etc. The not-so-easy part is verifying the system behaved correctly, and being able to reproduce problems once they surface.
4. Would be really good if I had code coverage tools. Don't think there's anything available for IPy :-(
5. This happens every time I port non-trivial code. Need to stop being surprised :-)

• Validating a Filename

• Profiling for Silverlight

• Scaling the tower of Babel II
  

• Global System Builder Promo
  

• Need for supporting IronPython, Jython?

• Running Python from C#

• NWSGI 0.5 Released

• ASP.NET on Codeplex

• ASP.NET Dynamic Language Support Refreshed

• Refreshing ASP.NET Dynamic Language Support

• Now I can touch ASP.NET again
  

Chris Anderson is wearing a Mono T-Shirt on this PDC interview and IronRuby.com is hosted on DekiWiki, a Mono-powered Wiki site.

This is clearly awesome.

In other news, the awesome hackers at Imendio have officially released Gtk+ for OSX packages.

Funkists, why does System.IO.Stream not have a CopyStream method, it seems like everyone ends up implementing this.

Discuss.

Today, the IronPython team and the ASP.NET team released ASP.NET Dynamic Language Support on the ASP.NET Codeplex site!

Download it here!

Package

ironpython-webforms-sample.zip: running IronPython ASP.NET website. Either dump this in IIS or open with Visual Studio as a website project.

ironpython-mvc-sample.zip: and IronPython ASP.NET MVC website, so you can get a feel for how dynamic languages can integrate into MVC. However, this just shows IronPython working in Views, not Controllers or Models yet. This requires MVC to be installed to open the project in Visual Studio. Open it with Visual Studio, build, and run your shiny IronPython ASP.NET MVC app.

aspnet-dlr-docs.zip: Documentation on how to use all this stuff. Open intro.html and have fun. :)

Many Thanks!

This is a project that me, David Ebbo, and Phil Haack have been working on for a bit, and I'm so happy that the ASP.NET team is taking the DLR and committing to making dynamic languages work well on ASP.NET. Phil actually announced this on his blog as well, and I'm sure will be talking about this much more in the future. In short, a big thanks to David and Phil for getting this project kick-started again.

Walk-through: WebForms

Back in June I talked about this day coming, but I didn't even have a piece offering, other than my whit. To make up for that, I'm going to walk you through using IronPython in ASP.NET.

Note: I'm going to walk you through using Visual Studio to open the website and make changes. However, you can simply drop this directory into your IIS wwwroot and simply edit the files with a text-editor of your choice.

1. Extract ironpython-webforms-sample.zip, open Visual Studio, and then open a website project (File > Open > Web Site ...). Navigate to where you extracted the webforms sample, and click open; now you should see the following.

2. Take a look at the code in Default.aspx and Default.aspx.py. The aspx page has a asp:Literal control, with the id "messageLiteral", and the py file sets the text of that asp:Literal control.

<!-- Default.aspx -->
<%@ Page Language="IronPython" CodeFile="Default.aspx.py" %>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">

<html xmlns="http://www.w3.org/1999/xhtml" >
    <head runat="server">
        <title>Untitled Page</title>
    </head>
    <body>
        <form id="form1" runat="server">
        <div>
            <asp:Literal id="messageLiteral" runat="server" />
        </div>
        </form>
    </body>
</html>

# Default.aspx.py
def Page_Load(sender, e):
    messageLiteral.Text = "Hello Dynamic World!"

3. Hit Ctrl+F5 (Start without debugging), and the app will run. If you are not using Visual Studio, simply navigate to Default.aspx in your web browser.

4. Awesome, our app runs! Now let's make it do something. Replace the inside of the <div /> with a Label, TextBox, and a Button control (Note: you can also open the Toolbox in Visual Studio and drag these in).

Enter your name: <br />
    <asp:TextBox ID="TextBox1" runat="server">
    </asp:TextBox>
    <asp:Button ID="Button1" runat="server" Text="Button" />
    <br /><br />
    <asp:Label ID="Label1" runat="server" Text="Label">
    </asp:Label>

5. Open Default.aspx.py and change the Page_Load method to be the following:

def Page_Load(sender, e):
    if not IsPostBack:
        Label1.Text = "...Your name here..."

6. Now add the following code to Default.aspx.py to handle the button's Click event:

def Button1_Click(sender, e):
    Label1.Text = Textbox1.Text

7. Switch back to Default.aspx.py and make the Button handle the event:

<asp:Button ID="Button1" runat="server" Text="Button"
     OnClick="Button1_Click"/>

8. Now navigate back to your browser, and refresh the page. Enter any text in the textblock, click the button, and it'll appear in the label below.

Yeah, I know, not very useful, but it gives you an idea of how things work. Look through the "ASP.NET Dynamic Language Runtime Support Documentation" on the downloads page for more walk-throughs with IronPython and ASP.NET WebForms.

Go make awesome stuff!

I'm so happy that this finally got out, so hopefully I'll hear about people starting to use IronPython in ASP.NET much more. Consult http://codeplex.com/aspnet for any questions about roadmap or tutorials, and feel free to ask me questions as well.

• Passing Keyword Args to C# Methods from IronPython
  

• Game Scripting in IronPython
• Play Cameron's Dungeon
• Cameron's Dungeon Source Code (6mb)
  

The latest version of NWSGI, built for IronPython 2.0 Beta 5, has been posted. There are a couple of changes in this version: one breaking change and one new feature.

Configuraton Changes

To fit with convention, the configuration elements now start with lowercase letters - i.e. Wsgi => wsgi, OSEnviron => osEnviron, etc. This has a consequence because XML is case-sensitive, so the old configuration elements will not be picked up and the defaults will be used instead.

Script Mappings

Script mappings can be used to create a virtual .wsgi file that points to a file elsewhere on the filesystem. For example:

<scriptmappings>
    <scriptmapping scriptname="dispatch.wsgi" physicalpath="C:\myapp\dispatch.wsgi">
</scriptmappings>

Rather than copy the dispatch.wsgi file into the web aplication directory, any request to dispatch.wsgi will be mapped to c:\myapp\dispatch.wsgi instead. This example is a bit trivial, as an IIS virtual directory could probably be used, but it could be useful for any application that ships a .wsgi file.

Future Directions

I'm working on building an IIS7 admin interface for all of this; it's a bit of a mess right now because the advanced UI tricks aren't really documented that well. I'm hoping to have the IIS7 admin and the installer ready by the time IronPython goes 2.0.

• DLR Namespace Change Fire Drill

• DLR Hosting Sample : Simple dlr host using the new App.Config based ScriptRuntime creation

• DLR Hosting API : How to create a ScriptRuntime object without using App.Config files using language specific hosting API

• Hosting IronPython made Easier

• Slides from PyCon UK Available

• Custom Fixed Size ListBox Items in IronPython in 30 Seconds

• Lunch with a Dynamic Languages Braintrust

• Meeting Mike Vincent

• Embedding Boo, IronPython
  

• IronPython Bugs - Codeplex

• Use the bug template when reporting on CodePlex
• Emphasize if the bug blocks an application
• Email the list as well
• Provide the simplest repro possible
  

• Simple IronPython Example Using XML, Oracle and SQL Server
  

<