CPython Extensions for IronPython

A Proof of Concept Using Python.NET

Compiling stuff is fun

 

 

Note

This article describes one way to use CPython extensions from IronPython / .NET by embedding CPython itself via Python.NET.

Since this article was written a much more practical approach has been implemented as an Open Source project by Resolver Systems. This project fakes Python25.dll and has a large subset of the Python C API implemented in C#. A large percentage (around a thousand tests at the time of writing) of the Numpy test suite passes when used from IronPython and other whole extension libraries are usable:

Introduction

This module came out of The C Extensions for IronPython Project started by Resolver Systems. This module allow you to access CPython modules from IronPython. It works by embedding a CPython interpreter, using a an assembly provided by the Python.NET Project where most of the hard work was done!.

Despite some serious limitations it works! The example code included uses matplotlib [1] with numpy and Tkinter from IronPython. Surprised

It provides two different usage patterns:

  • An import hook allowing you to import CPython binary modules with normal import statements
  • An Import function allowing you to import modules (pure Python or binary modules) into the embedded CPython interpreter

See the usage section for examples.

Download

The download includes Python.Runtime.dll from Python.NET, for a UCS2 build of Python. This is the right assembly for Python 2.4 on Windows. For Linux you probably want a UCS4 build. The Python.NET distribution contains runtimes for Python 2.4 and 2.5 in both UCS2 and UCS4. This technique does depend on having the appropriate version of Python installed (although you could also ship the relevant Python dll).

The source for this module lives in the FePy Subversion Repository:

This code only works with IronPython 2 because of an annoying bug when passing Arrays as arguments in IronPython 1. This can be worked around, but targeting IronPython 2 may be better as we could look at using the DLR to help with making PyObjects behave like IronPython data types (see below).

Bug reports, contributions and suggestions welcomed. Smile

The C Extensions for IronPython Project

Resolver Systems recently announced a new project to get CPython extensions working seamlessly with IronPython. This is needed by customers of Resolver, but is being run as an Open Source project.

The approach in this module, is to embed the CPython interpreter in an assembly and access CPython extensions through the hosted interpreter. This module is far from the final solution and it's not even clear that it is the best approach to take.

At Resolver we have also been experimenting with directly loading CPython assemblies and replacing the CPython API with function pointers that use delegates to call back into managed code. This would also give us binary compatibility and avoid some of the problems caused by hosting a real CPython interpreter. So far we have Python binary extensions loading, calling into our code and then crashing! This is a great first step because it means the basic approach works, and know all we need to do is implement the whole Python C API in managed code...

Usage

The distribution includes the following files:

  • embedding.py: The main module
  • cext.py: The import hook
  • test.py: A simple test of the basic functionality - run this with IronPython 2
  • echo.py: A test module that is imported into CPython by test.py
  • Python.Runtime.dll - Suitable for a UCS2 build of Python 2.4

Note

Due to some bug in the Orcas Beta, the import hook doesn't work if you have any of Visual Studio 2008 betas (Orcas) installed.

There are two ways of using this project. The first way is with the Import function from the embedding module.

The Import Function

The test.py shows this approach in action. The final part of this module generates a plot using matplotlib:

# This needs matplotlib installed
pylab = Import('pylab')
pylab.plot([1, 1, 1.5, 2.5, 3, 3, 3.1])
pylab.show()

This generates a simple plot:

A matplotlib and Tkinter image from CPython

As you can see, the the pylab module imported from CPython behaves in (apparently) the same way as it does when running directly in CPython.

Caution!

The actual code in test.py imports the sys module from the hosted interpeter, and attempts to adds to sys.path.

Ufortunately this has no effect! (Although the path seems to be set correctly for my computer anyway.) The reason it doesn't work highlights something to be aware of if you want to use this module.

sys = Import('sys')
sys.path.append('c:\\Python24\\Lib\\')

In the code above, the sys module is successfully imported as a proxy object. When you access sys.path, this proxy object recognises that you are accessing a Python list (on the CPython side) and copies it across to IronPython for you. This means that the append is executed on the copy, not on the original. d'oh

The solution would be, either to not copy the list and to proxy access to it as well, or to provide functions on the CPython side allowing you to manipulate sys.path.

The Import Hook

Installing the import hook allows you import Python binary extensions using normal import statements! Python binary extensions are .pyd files on Windows and .so files on other platforms. To install the import hook, execute the following code:

import cext
cext.install()

You can then do things like import cElementTree. Smile

The goal is that eventually this will be build into FePy and enabled by an option, so that you can import CPython modules without having to take any special steps.

Under the hood, the import hook uses the embedding.Import function.

When you use the import hook to import binary modules you may want to do things like setting the import path on the hosted interpreter. Obviously normal import statements (like import sys) will import the IronPython version. To access the builtin modules of CPython you will still need to use embedding.Import.

Implementation Details

The Python Runtime Assembly

This provides a very thin wrapper around the CPython embedding API. You have to acquire and release the Global Interpreter Lock (GIL) around every operation and it works with PyObjects which are managed wrappers around CPython types.

The code below imports the PythonEngine and initializes it. It also defines two decorators

  • GIL acquires and releases the GIL, and wraps all operations with CPython objects.
  • handle_exception handles exceptions that occur in CPython and reraises them as IronPython exceptions
import clr
clr.AddReference('Python.Runtime')
from Python.Runtime import PythonEngine

engine = PythonEngine()
engine.Initialize()

def lock():
    h = engine.AcquireLock()
    def unlock():
        engine.ReleaseLock(h)
    return unlock


def GIL(function):
    def f(*args, **keywargs):
        unlock = lock()
        try:
            ret = function(*args, **keywargs)
        finally:
            unlock()
        return ret
    return f


def handle_exception(function):
    def f(*args, **kw):
        try:
            return function(*args, **kw)
        except PythonException, e:
            exc = PyExcConvert(PyObject(e.PyType))
            value = ConvertToIpy(PyObject(e.PyValue))
            raise exc(value)
    return f

Importing modules is done with the Import function. This could be the only function you need to import from the embedding module to access CPython extensions.

@GIL
def Import(name):
    module = engine.ImportModule(name)
    if module is None:
        raise ImportError("Importing module named %s failed" % name)
    return PythonObject(module)

Proxied Objects and Data Structures

When you import a module it returns a proxied object. By default all objects you access are proxied objects unless they are a fundamental datatype - which will be converted from a PyObject into the equivalent IronPython type.

Proxied objects let you get and set attributes on them, plus call them. The proxied class is PythonObject:

class PythonObject(object):
    def __init__(self, real):
        self._real_ = real

    @GIL
    @handle_exception
    def __getattribute__(self, name):
        real = object.__getattribute__(self, '_real_')
        if name == '_real_':
            return real
        ##
        return ConvertToIpy(real.GetAttr(name))


    @GIL
    @handle_exception
    def __setattr__(self, name, value):
        if name == '_real_':
            object.__setattr__(self, '_real_', value)
            return
        ##
        self._real_.SetAttr(name, ConvertToPy(value))


    @GIL
    @handle_exception
    def __call__(self, *args, **keywargs):
        if not keywargs:
            return ConvertToIpy(self._real_.Invoke(ConvertToPy(args)))
        return ConvertToIpy(self._real_.Invoke(ConvertToPy(args),
                                               StringDictToPy(keywargs)))

The ConvertToPy and ConvertToIPy functions do the converting between CPython and IronPython types.

It can convert integers, long integers, strings, booleans and None, lists, tuples and dictionaries. This means that if you call a Python function (or set an attribute) with a reference to an IronPython data structure it will be copied into CPython types. Any return values will be copied from CPython types to IronPython types. See the limitations section below for some of the consequences of this.

It does mean that once you have imported a module you can call functions / methods and instantiate classes from inside CPython. You can also fetch and set attributes and all the right things should happen.

The Test File

test.py tests the basic functionality of the embedding.py module. It also serves as a usage example.

When run with IronPython 2, it imports echo.py into CPython and passes data back and forth to check that it survives the journey. These are done with asserts, so if any fail then it will bomb out with an error.

If it works you should see:

Received into CPython: 1
Type: <type 'int'>
Received into CPython: 3.2000000000000002
Type: <type 'float'>
Received into CPython: u'Hello from IronPython'
Type: <type 'unicode'>
Received into CPython: 10000000000L
Type: <type 'long'>
Received into CPython: None
Type: <type 'NoneType'>
Received into CPython: True
Type: <type 'bool'>
Received into CPython: False
Type: <type 'bool'>
Received into CPython: []
Type: <type 'list'>
Received into CPython: ()
Type: <type 'tuple'>
Received into CPython: {}
Type: <type 'dict'>
Received into CPython: ({u'something': 3.2000000000000002}, u'hello', u'from',
u'ironpython', [1, 2, 3])
Type: <type 'tuple'>
setting something something else
setting something something else
fetching something
fetching something
Received into CPython: 123
Type: <type 'int'>
Caught an exception from CPython correctly.

test.py also tests other features of this module, like catching exceptions from CPython (etc).

Limitations

As you have probably already gathered, this is an early implementation and suffers from some serious limitations.

Some basic types like complex numbers aren't converted for you. Additionally data structures are copied in and out which is expensive - and you lose changes that CPython makes to mutable data structures you pass in. The technique for copying data structures (in and out), doesn't take into account recursive data structures - and so will probably never terminate if it encounters them.

For more efficient work you can leave data as PyObject structures rather than copy. One avenue of investigation would be to see if we can make these PyObject structures (managed wrappers around the CPython data) behave like their equivalent IronPython objects. This would allow some source code to operate unmodified.

CPython objects (other than the basic datatypes) are proxied. This means that they have the wrong type and magic methods probably won't work.

Oh, and strings from .NET come in as unicode on the CPython side. Smile

Despite these problems, as you can see from the matplotlib demo it works fine. With some CPython helper modules you may be able to solve quite difficult problems with this as a starting point. Feel free to experiment with and extend this code. If you do fix any of the problems then please send the code back to me.

CHANGELOG

2007-11-08 Version 0.1.4

Added missing return statement!

2007-11-03 Version 0.1.3

Added 'cext.py' import hook (by sanixyn).

Improved object proxying.

Failed imports now raise an ImportError.

Support for passing CPython objects back into CPython.

Support for function calls that don't take keyword arguments!

Improved the way CPython booleans are accessed.

2007-11-01 Version 0.1.2

Added CPython exception handling (by sanixyn).

2007-10-25 Version 0.1.1

Added support for keyword arguments.

2007-10-24 Version 0.1

Initial release.


[1]I needed to follow these instructions to get matplotlib working on Windows. The installer doesn't create the config file.

For buying techie books, science fiction, computer hardware or the latest gadgets: visit The Voidspace Amazon Store.

Hosted by Webfaction

Return to Top

Page rendered with rest2web the Site Builder

Last edited Fri Nov 27 18:32:35 2009.

Counter...