Embedding the Dynamic Language Runtime

IronPython & .NET Applications

Visual Studio Express

 

 

Note

This is part of a series of articles on embedding IronPython:

For a much more in depth introduction to hosting and interacting with dynamic languages from .NET applications see chapters 14 & 15 of IronPython in Action.

Introduction

This is an introduction to embedding IronPython 2 and the Dynamic Language Runtime in .NET applications. Although the code used through the examples is C#, it is very simple code and is straightforward to adapt to VB.NET.

This article started life as a talk given to the Software Development Network in 2008.

IronPython in Action

Topics this articles covers include:

  • The IronPython / DLR Hosting API
  • Loading and executing code at runtime
  • Solving the type problem
  • Presenting an API to script code
  • Handling errors

IronPython and IronRuby are programming language engines built on top of the Dynamic Language Runtime (DLR). They have been designed to be easy to embed in .NET applications written in C# and VB.NET, opening up all sorts of interesting new possibilities.

In this article we'll explore the DLR hosting API using IronPython (although much of the material covered also applies to IronRuby and other DLR languages as well). We'll also look at solving the fundamental problem of interacting with objects created by a dynamic language at runtime from statically typed programming languages.

The Dynamic Language Runtime

  • Designed for embedding

  • Interoperation between .NET languages

  • Language hosting system with compiler support

  • Vast oversimplification of the DLR:

    You provide an AST (as an Expression Tree) and the DLR asks for language rules for operations (with dynamic site caching) - falling back to CLR defaults (e.g. to add integers)

The interoperation support (effectively a dynamic type system) is between dynamic languages and for dynamic languages working with the CLR and code written in C# / VB.NET (plus Boo / F# etc). Using DLR objects from C# / VB.NET is something that will get a lot easier in .NET 4.

You can get a slightly more in depth, but still high level, overview of how the DLR works as part of the presentation: Jimmy Hacking at Microsoft

DLR Languages

Many languages...
  • IronPython (MS)
  • IronRuby (MS)
  • Managed JScript (MS, Silverlight binaries only - no source release)
  • Prototype of VBx (MS, demoed but not released)
  • IronScheme (Codeplex)

Use Cases for Embedding

  • User scripting of .NET / Silverlight applications
  • Prototyping parts of your application (rapid application development)
  • Making parts of an application dynamic or changeable at runtime
  • Allowing business rules or logic to be stored as text in a DSL
  • An interactive console with a 'live' view into your application

Example consoles that use Windows Forms and WPF are available. Presenting a live view into your application state can be extremely useful for debugging!

Business rules stored as a Python or Ruby backed Domain Specific Language could be extremely useful - they can be read / written by business managers, rather than programmers, and even changed at runtime.

To add user scripting to an application (and yes - everything discussed in this article applies to Silverlight) you need to decide how you will present an API to the user code. One example of this is part of this presentation.

Resolver One

Resolver One - Spreadsheet development environment

Resolver One is the project I have been working on since 2006. It is a spreadsheet development environment (i.e. highly programmable spreadsheet client for the creation of powerful spreadsheet systems). It embeds IronPython for user scripting of spreadsheets - including exposing the whole spreadsheet object model to user code. It also happens to be written in IronPython, but uses the IronPython hosting API to provide a Python engine for every open document.

Intellipad

The Intellipad tool, part of the Oslo framework, is extensible with (and partly written in) IronPython.

@Metadata.CommandExecuted('{Microsoft.Intellipad}BufferView', '{Microsoft.Intellipad}OpenExplorerAtBuffer', 'Ctrl+B')
def OpenExplorerAtBufferExecute(target, sender, args):
    file = sender.Buffer.Uri.AbsolutePath
    exists = File.Exists(file)
    if exists:
        Process.Start(Path.GetDirectoryName(file))

Almost all commands that are available in Intellipad have been written in Python using the object model exposed by the application. The Python files are scattered inside the Settings directory. Commands.py contains most of the commands for Intellipad.

The commands are implemented as functions and are registered with Intellipad with a decorator. This is one example of how to present an API to users scripting your application.

Crack.NET

Crack.NET is a debugger and application explorer that lets interact with winforms / WPF applications.

The Crack.NET Launcher
The Python scripting interface

Crack.NET gives you access to the internals of a WPF or Windows Forms application running on your computer. Crack.NET allows you to “walk” the managed heap of another .NET application, and inspect all values on all objects/types. It also lets you execute Python scripts to interact with those objects.

Our Example Application

An example application that embeds IronPython

This article works A C# / VB.NET .NET application that loads Python plugins at runtime and exposes an API to them - with buttons to execute the plugin code.

This application is one of the four example applications from the embedding chapter of IronPython in Action (chapter 15). It is available for download in C# and VB.NET from the book website.

If you download the sources for chapter 15, this example is in the 15.3 folder, with projects for both C# and VB.NET. It uses the IronPython 2 final assemblies (included).

You may also be interested in chapter 14 that is on writing C# / VB.NET classes that are intended to be used from IronPython.

The most important ones DLR Hosting API components we will be using are:

  • ScriptEngine
  • ScriptRuntime
  • ScriptScope
  • ScriptSource
  • CompiledCode

The ScriptEngine

The Python convenience class from IronPython.Hosting allows us to create a ready configured Python ScriptEngine:

>>> import clr
>>> clr.AddReference('IronPython')
>>> from IronPython.Hosting import Python
>>> engine = Python.CreateEngine()
>>> engine
<Microsoft.Scripting.Hosting.ScriptEngine object at 0x... [Microsoft.Scripting.Hosting.ScriptEngine]>

This example shows using the IronPython hosting API from the IronPython interactive interpreter. This is one very powerful use case for IronPython itself; exploring new APIs with live objects in the interactive interpreter.

If we were hosting IronRuby we would use the Ruby class and we would get a ScriptEngine specialised for the Ruby language.

Executing Code

The ScriptSource represents source code and the ScriptScope is a namespace. We use them both to execute Python code - executing the code (the ScriptSource) in a namespace (a ScriptScope):

SourceCodeKind st = SourceCodeKind.Statements;
string source = "print 'Hello World'";
script = eng.CreateScriptSourceFromString(source, st);
scope = eng.CreateScope();
script.Execute(scope);

The namespace holds the variables that the code creates in the process of executing it.

The ScriptSource has Compile method which returns a CompiledCode object. This also has an Execute method that takes a ScriptScope.

This really is an overview - there are lots of other ways of working with these classes. For example we could execute an expression instead of statements and directly return the result of evaluating the expression. See chapter 15 of IronPython in Action for a more in depth look.

Setting and Fetching Variables

To IronPython the ScriptScope is a module - a namespace that maps names to objects.

int value = 3;
scope.SetVariable("name", value);

script.Execute(scope);

int result = scope.GetVariable<int>("name");

There are other useful methods on the SciptScope like TryGetVariable, GetVariableNames, ContainsVariable, etc. Reflector is a handy tool for finding your way around the available APIs.

So as well as code creating variables we can pre-poulate the namespace with variables that the code has access to. This is one way a hosting application can expose an API / object model to code it executes.

The generic versions of these APIs are great if we are fetching a standard .NET object (but watch out for runtime exceptions if the object we are fetching is of the wrong type and can't be cast to the type you've specified) - but what if we want a dynamic object like an instance of a Python class?

This is the 'type problem'; how can we use and interact with dynamic objects from statically typed .NET languages?

Load the Plugins

Back to our example application. It has a 'plugins' folder, and when it starts it loads and executes every Python file in that directory to create the plugins. For every plugin created we add a button to the toolbar.

It executes all the Python files in the application "plugins" directory.

public void LoadPlugins(string rootDir)
{
    string pluginsDir = Path.Combine(rootDir, "plugins");
    foreach (string path in Directory.GetFiles(pluginsDir))
    {
        if (path.ToLower().EndsWith(".py"))
        {
            CreatePlugin(path);
        }
    }
}

Create the Plugins

This time we create CompiledCode objects and execute with CreateScriptSourceFromFile. Compiled objects is useful when you will be executing the same code several times (i.e. probably not necessary for this particular application...).

public void CreatePlugin(string path)
{
    try
    {
        ScriptSource script;
        script = eng.CreateScriptSourceFromFile(path);
        CompiledCode code = script.Compile();
        ScriptScope scope = engine.CreateScope();
        code.Execute(scope);
    }

To create the plugins we simply execute the code - we use a slightly different API (CreateScriptSourceFromFile) from previously, but nothing that isn't obvious. Because we are executing user code (with no guarantee of correctness) we also have to handle errors / exceptions.

Catching a SyntaxError

As we are executing arbitrary user code we have no idea whether the code is valid or not. If we were doing this on a server you may also want to isolate the Python engine inside its own AppDomain with restricted privileges. The DLR has support for working with AppDomains. This application runs on the user's machine, so whilst we aren't concerned with security we do need to handle any errors that may occur.

One of our plugins has a syntax error (invalid Python code). You can see the message box displayed when we attempt to execute this plugin.

Catching a syntax error when loading a plugin

Error Handling

In order to catch this error and present a helpful message to the user, instead of bombing out altogether, we need a bit of exception handling.

The SyntaxErrorException is a general DLR exception. There are also Python specific ones defined in IronPython.Runtime.Exceptions namespace.

catch (SyntaxErrorException e)
{
    ExceptionOperations eo;
    eo = engine.GetService<ExceptionOperations>();
    string error = eo.FormatException(e);

    string caption;
    string msg = "Syntax error in \"{0}\"";
    caption = String.Format(msg, Path.GetFileName(path));
    MessageBox.Show(error, caption,
                    MessageBoxButtons.OK,
                    MessageBoxIcon.Error);
}

We use ExceptionOperations to get us a nicely formatted Python traceback from the exception. We could just call ToString on the exception, but this would include a lot of information about DLR stack frames that are not useful to the user.

We can also have a 'catch all' clause that catches 'Exception' and handles all runtime errors.

Errors raised in Python code can be caught as Python exceptions or standard .NET exceptions where there is an equivalent.

So, assuming there are no errors - how does executing the user code create a plugin? What happens is that our .NET application presents an API to the user code, which it can use to add plugins. To do this we need to solve 'the type problem'.

Adding a Plugin

This is an example of a typical plugin for our appication; using the object model we have exposed to user code. If you look in the 'plugins' folder of the application then you will see four example plugins (one of which has a syntax error of course).

from Plugins import PluginBase
from EmbeddingPlugin import PluginStore

class Plugin(PluginBase):
    def Execute(self, textbox):
            textbox.Text += "Plugin 1 called\r\n"

plugin = Plugin("Plugin 1")
PluginStore.AddPlugin(plugin)

The plugin subclasses the PluginBase class and overrides the Execute method. The plugin adds an instance of the Plugin class to the store (instantiating it with its name which will be displayed on the toolbar).

When the plugin is invoked (by pressing the corresponding button) Execute with the TextBox from the application, This plug adds to the text displayed in the textbox when it is called.

PluginBase and PluginStore are classes that we provide to the user - the API we are presenting. We have to jump through a couple of hoops to get to this point.

The PluginStore has one public method, AddPlugin, that maintains a list of all the plugins added to it. The actual list is internal (friend in VB.NET), so our application can access it but the user code can't.

Preloading Assemblies

The user code imports and uses classes from the Plugins and EmbeddingPlugin namespaces. These are contained in dotnet assemblies (naturally) and we want to make them available to the user to import from:

Assembly pluginsAssembly = Assembly.LoadFile(pluginsPath);

ScriptRuntime runtime = engine.Runtime;
runtime.LoadAssembly(pluginsAssembly);
runtime.LoadAssembly(typeof(String).Assembly)
runtime.LoadAssembly(typeof(Uri).Assembly)

The standard way of accessing classes inside .NET assemblies from IronPython is via the clr module (adding a reference at runtime with the clr.AddReference method). We don't want the user to have to do this, so we want to add the reference ourself.

In addition to this, by default when executing code with the IronPython interpreter (ipy.exe) you already have a reference to the System and mscorlib assemblies. We want to add references to these assemblies as well so that the user code behaves as expected.

This is done with the LoadAssembly method of the ScriptRuntime, which takes an Assembly object. We can load the assembly objects from the Assembly class, using paths relative to the current executing assembly (our application).

Once we have added references to assemblies, the user is free to import public classes from the namespace(s) they contain.

Preloading Python Modules

An alternative to adding references to assemblies is to create and publish Python modules to the runtime. These will also be available for the user code to import from.

ScriptScope inner = engine.CreateScope();
inner.SetVariable("HelloWorld", "Some string...");
inner.SetVariable("answer", 42);

Scope module = HostingHelpers.GetScope(inner);

runtime.Globals.SetVariable("Example", module);

The DLR class representing a Python namespace is the ScriptScope class. We can use this directly as a module, but the class that represents a module is the Scope (a remotable wrapper for the ScriptScope - so that you can contain hosted engines inside AppDomains).

We create a Scope using the HostingHelpers class. It is set in the runtime Globals with the name Example. This means that the user can import the 'Example' module and have access to the objects we have placed in it. Globals itself is a ScriptScope, so we know its API for setting and fetching members.

Note

Python doesn't require that it be real module objects that are imported. We could publish a class or instance (or anything) directly into the runtime globals.

Invoking the Plugin

From which button is pressed we know which Plugin to fetch from the PluginStore. We fetch it as a PluginBase instance.

public void ExecutePluginAtIndex(int index)
{
    PluginBase plugin = PluginStore.Plugins[index];

    try {
        plugin.Execute(_box);
    }
    catch (Exception e)
    {
        string msg = "Error executing plugin \"{0}\"";
        ShowError(msg, plugin.Name, e);
    }
}

Having executed all the plugins code we then examine the PluginStore for all the plugins that have been successfully added. We add a toolbar button for each one, storing a mapping of the button to their location in the store. When a button is clicked, this is the method that is invoked.

Because we know that the PluginStore only contains instances of PluginBase (attempting to add anything else will cause an error because the plugin collection is statically typed) we can pull them out by type. This is one way of solving the type problem. Because the PluginBase has an Execute method we can call this method on Python subclasses - even though it is calling into Python code.

(Under the hood when a .NET class is subclassed from IronPython a new .NET type is created. Our plugin classes are genuine subclasses of the PluginBase class.)

Using known .NET classes is how we solved the type problem in this case. Note that because calling Execute calls into Python code we also need error handling around it. If you have written the Python code, rather than user code, then this may not be necessary. (Python doesn't raise random exceptions.)

There are other ways of solving this problem. First, that overview again.

Dynamic Operations

Solve the type problem by avoiding it until the last minute! (With a little help from ObjectOperations.)

ObjectOperations ops = engine.Operations;

object SomeClass = scope.GetVariable("SomeClass");
object instance = ops.Call(SomeClass);
object method = ops.GetMember(instance, "method");

int result = (int)ops.Call(method, 99);

We can pull objects in as 'object' and then use ObjectOperations to perform dynamic operations on them. This calls back into the DLR to perform the operation. Here we pull out a pure Python class, instantiates it, and call a method on it (casting the result to an integer).

ObjectOperations has many more methods, for example for numeric operations or equality operations, that honour the language semantics of the DLR objects involved.

This is something that gets a lot easier in .NET 4.

C# 4.0 - Dynamic

Dynamic operations with the dynamic keyword: not only method calls, but also field and property accesses, indexer and operator calls and even delegate invocations can be dispatched dynamically (at runtime):

dynamic d = GetDynamicObject(…);
d.M(7); // calling methods
d.f = d.P; // getting and settings fields and properties
d[“one”] = d[“two”]; // getting and setting through indexers
int i = d + 3; // calling operators
string s = d(5,7); // invoking as a delegate

Example from: http://code.msdn.microsoft.com/csharpfuture/

When compiled this generates IL that calls into the DLR. The DLR either uses reflection to do the lookups, or if the object is a DLR object then it used the correct semantics for the language they are implemented in. (The cost of course is that you can get runtime errors if you call methods that don't exist or attempt to perform invalid operations.) The advantage of this is that it makes it much easier to use DLR objects from .NET languages.

For example see: http://keithhill.spaces.live.com/Blog/cns!5A8D2641E0963A97!6676.entry?wa=wsignin1.0

Functions as Delegates

IronPython will do casting for us when we pull objects out of the scope. So we can pull Python functions out as .NET delegates if we specify the argument and return types.

//get a delegate to the python function
Func<int, bool> IsOdd;
IsOdd = scope.GetVariable<Func<int, bool>>("IsOdd");

//invoke the delegate
bool b = IsOdd(1);

Python definition of IsOdd would look something like:

IsOdd = lambda x: bool(x % 2)

This code pulls out a function that takes an integer as the argument and returns a bool. We use the Func delegate to get a callable delegate on the .NET side, and the IronPython engine casts the function to this type for us. The last type in the generic specification Func<int, bool> is the return type.

This example is from the DLR Hosting Blog.

This technique is particularly useful for writing business rules that you want to be able to change at runtime.

Further Topics

Things we haven't had time for:

  • Evaluating expressions and returning results
  • Setting the search path so that hosted Python can import code
  • Diverting standard out stream on the runtime
  • Working with Python types and builtins from .NET
  • Writing dynamic classes in C# / VB.NET for use from IronPython
  • Writing and embedding custom dynamic languages!

Print statements in the user plugin also go to the application textbox - very useful for debugging.

For the full details on embedding the Dynamic Language Runtime you can either refer to the source code (both IronPython and the DLR are Open Source), or read the DLR hosting spec documentation: DLR Project Page on Codeplex

For buying techie books, science fiction, computer hardware or the latest gadgets: visit The Voidspace Amazon Store.

Hosted by Webfaction

Return to Top

Page rendered with rest2web the Site Builder

Last edited Fri Nov 27 18:32:35 2009.

Counter...