Python Programming, news on the Voidspace Python Projects and all things techie.

IronPython and Unicode (or another reason to use IronPython - or why wait until Python 3 ?)

emoticon:baldguy It's kind of obvious now I think about it, but I'd never seen this :

>>> type(u'hello')
<type 'str'>
>>> unicode
<type 'str'>
>>> 'Andr\202'
>>> type(u'Andr\x82')
<type 'str'>

How did I not notice this before ? Smile

Like this post? Digg it or it.

Posted by Fuzzyman on 2007-03-12 21:02:36 | |

Categories: , ,

Creating .NET Classes Dynamically from IronPython

emoticon:acrobat There are two restrictions in IronPython that I would like an automated approach to overcome. They can currently both be solved by writing stub classes in C#; but I would rather not have to write C# (by hand) if I don't have to.

  • You can't use .NET attributes in IronPython
  • You can't create classes in IronPython and consume them from other .NET languages

Additionally there is a problem with some types of data bound controls that require real dotnet classes rather than instances of IronPython classes (even when they inherit from dotnet classes it seems). One of the approaches mentioned here may be appropriate for solving this problem as well.

The stub C# classes you need to create are simple 'proxy' classes, and I would like to automate their creation from the IronPython side. The proxy instances need to call back into the IronPython engine, and call the appropriate method on a corresponding instance of an IronPython class. In order to keep the CLR happy when we create a proxy method we need to specify the types for the call signature and the return value. If we want to use dotnet attributes we will need some way of specifying these for methods as well.

I'm not looking to create a fully generic solution and obviously with this approach we won't easily be able to add methods at runtime or change the types accepted by methods, but this seems like a reasonable restriction for the moment. Smile

This entry is the start of my exploration of dynamic / semi-dynamic generation of assemblies and classes from IronPython.

The standard .NET API for programmatic generation of classes is called System.Reflection.Emit. It's a bit fiddly. Surprised

Obviously if I get a working solution I can build a nice API on top of it. All I've managed to do so far is translate an example in the MSDN docs from C# into IronPython. It is interesting enough that some of you might want to browse though.


There is a tool that I've found invaluable when translating C# examples into IronPython. This is Instant Python by Tangible Software.

The demo version will translate up to one hundred lines at a time. It doesn't get everything right, but not only are they very responsive to bug reports it can be darn useful.

The example below creates an assembly (which is later saved to disk as "DynamicAsm.dll"). It contains a class called "DynamicType", with a single static method called 'test'. This method takes four integers and adds them together.

The nasty part is that the body of the 'test' method is created by emitting the appropriate IL: .NET bytecode [1].

from System import *
from System.Threading import *
from System.Reflection import *
from System.Reflection.Emit import *

def addMethodDynamically(typeBuilder, methodName, methodParams, returnType):
    methodBuilder = typeBuilder.DefineMethod(
                                             MethodAttributes.Public | MethodAttributes.Static,
    ILout = methodBuilder.GetILGenerator()
    numParams = len(methodParams)
    x = 0
    while x < numParams:
        ILout.Emit(OpCodes.Ldarg_S, x)
        x += 1
    y = 0
    while y < (numParams - 1):
        y += 1

def main():
    domain = Thread.GetDomain()
    asmName = AssemblyName()
    asmName.Name = "DynamicAssembly"

    asmBuilder = domain.DefineDynamicAssembly(asmName, AssemblyBuilderAccess.RunAndSave)
    module = asmBuilder.DefineDynamicModule("DynamicModule", "DynamicAsm.dll")
    typeBuilder = module.DefineType("DynamicType", TypeAttributes.Public)

    methodName = "test"
    inputNumsList = (2, 3, 4, 5)

    methodParams = Array[Type]((Int32, Int32, Int32, Int32))
    inputValsList = Array[object](inputNumsList)

    returnType = Int32

    # Now, call the method building method with the parameters, passing the TypeBuilder.
    addMethodDynamically(typeBuilder, methodName, methodParams, returnType)
    myType = typeBuilder.CreateType()
    result = myType.InvokeMember(methodName,
                                 BindingFlags.InvokeMethod | BindingFlags.Public | BindingFlags.Static,
                                 None, None,
    print "The result of adding the inputted values is: %s" % result
    print "---"

    # Let's take a look at the method we created.

    # If you are interested in seeing the MSIL generated dynamically for the method
    # your program generated, change to the directory where you ran the compiled
    # code sample and type "ildasm DynamicAsm.dll" at the prompt. When the list
    # of manifest contents appears, click on "DynamicType" and then on the name of
    # of the method you provided during execution.
    methodInfo = myType.GetMethod(methodName)
    print "Your Dynamic Method: %r;" % methodInfo


The nice thing is that this saves "DynamicAsm.dll" to disk. You can then start an IronPython interactive interpreter session and do the following :

>>> import clr
>>> clr.AddReference('DynamicAsm.dll')
>>> import DynamicType
>>> DynamicType.test(2, 3, 4, 5)

Wow, we've dynamically generated a dotnet assembly containing a class, and then imported it into IronPython and called a generated static method on the class.

So, in order to create my proxy classes, they would somehow need a reference to the IronPython engine to be used - and a way of calling back into it. I guess the best approach would depend on the situation. You could create an initialisation class that creates and configures an IronPython engine, executing Python code to create the classes and create the interface layer on the Python side. In the constructors of the proxy classes (which would need to mimic the __init__ method of their corresponding Python classes), when the first instance is created it could do the initialisation.

If I wanted to follow this path I could write a proxy class by hand (not very difficult) and then disassemble the bytecode. I still don't fancy having to learn enough about IL to be able to do this [2], although the semantics of the Call opcode don't seem too tricky. Autogenerating C# would be much nicer, and if the target is a compiled assembly then it seems appropriate.

Luckily there is an example of doing exactly this by drifter46and2. It works and it doesn't seem too slow. For generating assemblies that provide proxy classes this would be a good approach.

For runtime (dynamic) generation of classes, then the MethodBuilder technique would be much better, especially if all you want is access to .NET attributes and you don't want to have to recompile an assembly every time (even if the code that does it is Python code). The way to generate attributes is with CustomAttributeBuilder. Hmmm....

[1]Attributes are effectively class and method decorators, you can also decorate attributes which could be done in Python with a property descriptor.
[2]I like Python, C# is tolerable, but I really don't want to start programming in bytecode. (Which reminds me of assembly language on the Amiga. sigh those were the days. Oh, and apparently Commodore are trying to make a comeback as a PC manufacturer!)

Like this post? Digg it or it.

Posted by Fuzzyman on 2007-03-11 16:33:17 | |

Categories: , ,

Parsing Fun

emoticon:black_hat I've had some fun playing with grammars using the excellent PLY.

Parsers can be a PITA, but it seems that the Python lexer ought to be able to handle the following without falling over :

>>> 3.__long__()
  File "<stdin>", line 1
SyntaxError: invalid syntax
>>> 3 .__long__()


In case you can't see it, the second example has a space between the integer literal and the method call.

Like this post? Digg it or it.

Posted by Fuzzyman on 2007-03-10 13:39:16 | |


Hosted by Webfaction