Voidspace

CherryPy with a serving of werkzeug

My next work project is an interesting application. We're porting a C++ Windows desktop application to IronPython and modernising it in the process. We'd also like to make it run on the Mac as well, which will be a relief to me as it will save me developing inside a VM. WPF, the most powerful Windows user interface is WPF - and this isn't available on Mac OS X even with Mono. The Silverlight UI is a cut-down version of WPF however, and so if we develop the application as an out-of-browser Silverlight application then it will be trivially easy (at least compared to porting a desktop application) to move it to the web at some future point. For the app to work as an out of browser Silverlight application it will need a local data server and so I've been looking at the simplest way of achieving this with Python (vanilla CPython).

The fastest pure Python web server is still good old CherryPy [1], but to make the application easily portable to the web it would be nice to develop the local component as a wsgi app. Fortunately CherryPy and wsgi are good friends and play nicely together. We can create a simple wsgi application with werkzeug and serve it with CherryPy. In our case the wsgi app is really just a very thin wrapper around a custom data processing / calculation component.

I couldn't find an example of exactly how to do this, but it wasn't very hard to work out from the documentation. Anyway, here's a simple proof-of-concept example:

from cherrypy import wsgiserver

# the werkzeug wsgi app
from app import application

d = wsgiserver.WSGIPathInfoDispatcher({'/': application})
server = wsgiserver.CherryPyWSGIServer(('localhost', 8080), d)

if __name__ == '__main__':
   try:
      server.start()
   except KeyboardInterrupt:
      server.stop()

For the wsgi app the werkzeug example on the front page of the documentation works fine, but an even simpler one that doesn't require a templating engine installed (in our case we will be sending and receiving data - probably encoded using protobuf as it is more compact / faster than json and there are both CPython and Silverlight libraries for working with it) is:

from werkzeug import Request, Response, SharedDataMiddleware
from werkzeug.routing import Map, Rule
from werkzeug.exceptions import HTTPException

url_map = Map([Rule('/', endpoint='index')])

@Request.application
def application(request):
    adapter = url_map.bind_to_environ(request.environ)
    try:
        endpoint, values = adapter.match()
        response = Response(mimetype='text/html')
        response.data = u'<h2>foo</h2>'
        return response
    except HTTPException, e:
        return e

application = SharedDataMiddleware(application, {})

There are interesting possibilities and challenges ahead, like creating a Windows service that starts the local server when you open the Silverlight app, or possibly creating a WPF desktop application and a Silverlight application that share code - something that became a lot easier with Silverlight 4.

Buttons in a Silverlight DataGrid Header

Today I was hoping to complete adding an excel-like auto-filter to the Silverlight DataGrid which is used for the main view throughout a big chunk of our application. Instead I spent almost the entire day just getting the first step working - putting a button in the grid header and wiring up the click event.

Generally using Silverlight from IronPython is a pleasant experience, but there are one or two things that can't be done through code and have to be done in XAML. XAML is the markup language used to describe Silverlight user interfaces; the Silverlight UI is a cutdown mostly-subset of Windows Presentation Foundation (WPF), the hardware accelerated Windows [1] user interface framework that is part of .NET 3 and above.

With WPF you can not only define the visual elements (including animations) of a UI, but also setup data-binding and bind handlers to events. Event handlers are statically bound and this is a problem for IronPython. Even though we can now use clrtype to create true .NET classes from IronPython, the XAML loader uses Type.GetType() to find the types - and this fails with generated classes.

For almost everything this isn't a problem as we can just hook up events from code, but creating custom DataGrid headers and hooking up events to controls in them is one of the places where we can't.

So the first part of the puzzle is creating a custom style in UserControl.Resources and setup the ContentTemplate:

<Setter Property="ContentTemplate">
    <Setter.Value>
        <DataTemplate>
            <Grid Height="{TemplateBinding Height}" Width="Auto">
                <Grid.RowDefinitions>
                    <RowDefinition Height="*"/>
                </Grid.RowDefinitions>
                <StackPanel Orientation="Horizontal" Margin="2"
                  HorizontalAlignment="Stretch" VerticalAlignment="Stretch">
                    <TextBlock Text="{Binding}" HorizontalAlignment="Center"
                     VerticalAlignment="Center" Margin="0.2"/>
                    <Button Content="Foo" Margin="5,0" x:Name="filter" />
                </StackPanel>
            </Grid>
        </DataTemplate>
    </Setter.Value>
</Setter>

See how the DataTemplate contains a Grid with sub-ui-elements like a TextBlock and the oh-so-important Button. The {Binding} in the textblock means that the standard header text is displayed alongside our button. The XAML for the DataGrid specifies the style we created:

<data:DataGrid x:Name="dataGrid" CanUserReorderColumns="False"
    ColumnHeaderStyle="{StaticResource OurColumnHeaderStyle}"
    IsReadOnly="True" AutoGenerateColumns="False" />

Ok, so far so good. When the columns are created for our grid they will use the ContentTemplate to create the column headers:

If we'd been using C# we could have hooked up the button click event in the style XAML. So how do we do this from IronPython? This is what took most of the day to work out. The column headers aren't exposed in any straightforward way, but once the columns have been created we can 'walk the visual tree' to find the column header object, and from there we can find the button. Walking the visual tree is done with VisualTreeHelper, and a recursive helper function:

from System.Windows.Media import VisualTreeHelper

def find_children(parent, findtype):
    count = VisualTreeHelper.GetChildrenCount(parent)
    for i in range(count):
        child = VisualTreeHelper.GetChild(parent, i)
        if isinstance(child, findtype):
            yield child
        else:
            for entry in find_children(child, findtype):
                yield entry

We use it to find and hook up the buttons thusly:

from System.Windows import Visibility
from System.Windows.Controls import Button, DataGridTextColumn

for entry in find_children(datagrid, DataGridColumnHeader):

    for button in find_children(entry, Button):
        if not button.IsEnabled:
            button.Visibility = Visibility.Collapsed
        else:
            button.Click += handler_function
        # just one button per column
        break

We make disabled button invisible because otherwise the grid will leave a disabled button visible in the header to the right of the populated columns.

So that was the hard part. The next problem wasn't quite so difficult; inside the event handler how do we know which column the button click is for? It turns out that getting a reference to the column inside the click event handler is easy:

from System.Windows.Controls import DataGridColumn

def handler_function(sender, event):
    column = DataGridColumn.GetColumnContainingElement(sender)
    ...

At least next time I have to do this is it will be a bit easier...

The final version will use an image for the button content, but the hard part is done.

UPDATE

My colleague Stephan Mitt tells me that the only reason it was so easy to get the column from the button event is that he spent three hours previously discovering this API. Props to him for working this out and making my life easier. Stephan also wanted me to show you the end result with his nice filter images instead of my proof-of-concept 'Foo' buttons:

The code to set the image on the button looks like this:

from System import Uri, UriKind
from System.Windows.Controls import Image
from System.Windows.Media.Imaging import BitmapImage

def click_handler(sender, event):
    uri = Uri('images/someimage.jpg', UriKind.RelativeOrAbsolute)
    bitmap = BitmapImage(uri)
    image = Image()
    image.Source = bitmap

    # The sender is the button
    sender.Content = image

As I was integrating this with our production code I encountered another problem. Many of the grids we used are loaded into TabControl pages, and the headers aren't created until the grid is rendered (the tab page the grid is in is selected). This means it matters when you run the code that walks the visual tree to find the buttons and setup the click handlers. A good place to do it is in response to the Loaded event, which is fired after the headers have been created.

Encoding json on Silverlight with System.Json

At the backend of last I year I wrote a blog entry on decoding json on Silverlight. Well, the time has finally come and we're now encoding json to post back to our Django application. The code to encoode json is just as simple as the code for decoding. It makes extensive use of clr.Convert(...) as encoding json relies heavily on implicit conversions:

import clr
from System.Collections.Generic import Dictionary
from System.Json import JsonValue, JsonArray, JsonObject

NoneType = type(None)

def encode_string(data):
    return encode_object(data).ToString()

def encode_object(value):
    if isinstance(value, (int, long, bool, float, str, NoneType)):
        # JsonValue has implicit conversion operators for all
        # these types
        return clr.Convert(value, JsonValue)
    elif isinstance(value, (list, tuple, set)):
        return JsonArray([encode_object(obj) for obj in value])
    elif isinstance(value, dict):
        values = Dictionary[str, JsonValue]()
        for key, val in value.items():
            values[key] = encode_object(val)
        return JsonObject(values)

    # raise an error if we encounter an unhandled type
    raise TypeError(value)

encode_string takes any data-structure of Python primitive / container types and serializes it into JSON. This json seems to be consumed fine by simplejson on the other end. As well as being simple this code is blazingly fast. The actual encoding operation is done in the .ToString() call, which all happens in C#.

When you send the json over the wire then it will be encoded as UTF-8 for the simple reason that this is the only encoding that Silverlight ships with.

This is relevant because the reason we are shipping huge data structures over the wire is in order to export Excel files from our data-grid views. I initially tried to use xlwt inside Silverlight. Unfortunately it relies on either latin-1 or utf-16 encodings and so my efforts were doomed to failure. Fortunately it works grand on the server side, but the encoding restriction does place limits on the Python libraries that you can use in the browser (unless you are prepared to write your own encoding functions of course).

Fun with Unicode, Latin-1 and a C1 Control Code

Unicode is a rabbit-warren of complexity; almost fractal in nature, the more you learn about it the more complexity you discover. Anyway, all that aside you can have great fun (i.e. pain) with fairly basic situations even if you are trying to do the right thing.

This particular problem was encountered by Stephan Mitt, one of my colleagues at Comsulting. I helped him find the solution, and with a bit of digging (and some help from #python-dev) worked out why it was happening.

We receive data from customers as CSV files that need importing into a web application. The CSV files are received in latin-1 encoding and we decode and then iterate over them to process a line at a time. Unfortunately the data from the customers included some \x85 characters, which were breaking the CSV parsing.

One of the problems with the latin-1 encoding is that it uses all 256 bytes, so it is never possible to detect badly encoded data. Arbitrary binary data will always successfully decode:

>>> data = ''.join(chr(x) for x in range(256))
>>> data.decode('latin-1')
u'\x00\x01\x02\x03\x04\x05\x06\x07\x08\t\n\x0b\x0c\r\x0e\x0f...'

If you iterate over a standard file object in Python 2 (i.e. one that reads data as bytestrings) then you iterate over it a line at a time. This splits lines on carriage returns (\x0D) and line feeds (\x0A). If you're on Windows then the sequence \x0D\x0A (CRLF) signifies a new line. If you're trying to do-the-right-thing, and decode your data to Unicode before treating it as text, then you might use code a bit like the following to read it:

import codecs

handle = codecs.open(filename, 'r', encoding='latin-1')
for line in handle:
    ...

This was the cause of our problem. When decoding using latin-1 \x85 is transcoded to u'\x85', which Unicode treats as a line break. So if your source data has \x85 embedded in it, and you are splitting on lines, where the lines break will be different depending on if you are using byte-strings or Unicode strings:

>>> d = 'foo\x85bar'
>>> d.split()
['foo\x85bar']
>>> u = d.decode('latin-1')
>>> u
u'foo\x85bar'
>>> u.split()
[u'foo','bar']

This could still be a pitfall in Python 3, where all strings are Unicode, particularly if you are porting an application from Python 2 to Python 3. Suddenly your data will behave differently when you treat it as Unicode. The answer is to do the split manually, specifying which character to use as a line break.

The problem isn't restricted to \x85. The Unicode spec on newlines shows us why. \x85 is referred to by the acronym NEL, which is a C1 Control Code: NEL Next Line Equivalent to CR+LF. Used to mark end-of-line on some IBM mainframes.

In fact NEL belongs to a general class of characters known as Paragraph Separators (Category B). This category includes the characters \x1C, \x1D, \x1E, \x0D, \x0A and \x85. Splitting on lines will split on any of these characters, which may not be what you expect. It certainly wasn't what we expected.

For us the solution was simple; we just strip out any occurence of \x85 in the binary data before decoding.

Decoding json on Silverlight with System.Json (plus implicit conversion operators in IronPython)

As I explained we're writing a Silverlight application that communicates with our Django server and exchanges a lot of json with it.

Unfortunately, due to what is apparently just an oversight, the codecs module is incomplete for IronPython on Silverlight. This means that recent versions of simplejson don't work.

What we've been using is an older version of simplejson, that pulls in the obsolete and huge sre module in as one of its dependencies. On top of bloating up our application and adding several seconds to the startup importing it all the performance is not exactly blazing.

Fortunately part of the Silverlight SDK is System.Json.dll. Using this API from IronPython is pretty simple. The following code defines a loads function (load string - the same API as simplejson) that takes a string and parses it.

import clr
clr.AddReference('System.Json')
from System import Boolean
from System.Json import JsonValue, JsonType

String = clr.GetClrType(str)

def loads(inString):
    thing = JsonValue.Parse(inString)
    return handle_value(thing)

def handle_value(inValue):
    if inValue is None:
        return None
    elif inValue.JsonType == JsonType.String:
        return clr.Convert(inValue, String)
    elif inValue.JsonType == JsonType.Boolean:
        return Boolean.Parse(str(inValue))
    elif inValue.JsonType == JsonType.Number:
        return get_number(inValue.ToString())
    elif inValue.JsonType == JsonType.Object:
        return dict((pair.Key, handle_value(pair.Value)) for pair in inValue)
    elif inValue.JsonType == JsonType.Array:
        return [handle_value(value) for value in inValue]
    # Should be unreachable - but if it happens I want to know about it!
    raise TypeError(inValue)

def get_number(inString):
    try:
        return int(inString)
    except ValueError:
        return float(inString)

As with my custom json emitter there is a lot it doesn't do. It handles the following types:

• None
• lists
• dictionaries
• strings
• floats and integers
• booleans

If you want it to handle decmials or dates you'll have to add that yourself. How it works is mostly straightforward, but there is one little piece of 'magic' in there. When you call ToString() (or str() they do the same thing) you get the original json back. For numbers and booleans this is fine as we can easily turn them into the objects they represent. For strings this is a nuisance as we get double quoted, escaped strings. The correct way to get the value we want is to use implicit conversion. In C# this looks something like:

string value = jsonArray["key"];

Declaring the result as a string calls the appropriate JsonValue implicit conversion operator for us. IronPython doesn't have implicit conversion operators as we don't declare types and with a dynamic type system you rarely need to cast. Up until version 2.6 it wasn't possible to call implicit operators, but in IronPython 2.6 we gained a new function in the clr module; clr.Convert:

>>> print clr.Convert.__doc__
object Convert(object o, Type toType)


            Attempts to convert the provided
     object to the specified type.  Conversions that

                 will be attempted include standard
     Python conversions as well as .NET implicit

            and explicit conversions.


               If the conversion cannot be performed a
     TypeError will be raised.

The implicit conversion to string is done with the code:

clr.Convert(jsonValue, clr.GetClrType(str))

I cover some of the other goodies new in IronPython 2.6 (like CompileSubclassTypes) in Dark Corners of IronPython.

I haven't yet written the code to do json encoding, but the JsonValue class (and friends) can be used for encoding as well as decoding, so I expect the code will pretty much be the opposite of the snippet shown above...

Django: Tear down and re-sync the database

Django includes the useful management command syncdb for creating the database tables and columns used by your application. If you add new tables (model classes) then re-running syncdb will add them for you. Unfortunately if you modify columns of existing tables, or add new columns, then syncdb isn't man enough for the job.

For modifying the schema of production systems migrations are the way to go. I played a bit with South for Django, which is pretty straightforward. For a system still in development, and changing rapidly, migrations are overkill. We have a script for populating the database with test data, which we update as the schema evolves. (In parallel with this we have a script that imports the original data from the legacy application we are replacing - again updating the script as our app is capable of handling more of the original schema.)

For development what we really want to do is to tear down our development database and re-run syncdb. Running syncdb requires manual input, to create a superuser, so preferably we want to disable this so that the whole process can be automated. I found various recipes online to do this, but mostly using an obsolete technique to disable superuser creation.

In the end I used a combination of this recipe to programatically clear the databases (using the sql generated by sqlclear) and this recipe to disable super user creation.

#!/usr/bin/env python

import os
import sys
import StringIO

import settings
from django.core.management import setup_environ, call_command
setup_environ(settings)

from django.db import connection
from django.db.models import get_apps, signals


app_labels = [app.__name__.split('.')[-2] for app in get_apps()]
# Skip clearing the users table
app_labels.remove('auth')

sys.stdout = buffer = StringIO.StringIO()
call_command('sqlclear', *app_labels)
sys.stdout = sys.__stdout__

queries = buffer.getvalue().split(';')[1:-2]

cursor = connection.cursor()
for query in queries:
    cursor.execute(query.strip())

from django.db.models import signals
from django.contrib.auth.management import create_superuser
from django.contrib.auth import models as auth_app

# Prevent interactive question about wanting a superuser created.

signals.post_syncdb.disconnect(
    create_superuser,
    sender=auth_app,
    dispatch_uid = "django.contrib.auth.management.create_superuser")
call_command('syncdb')

It wasn't all plain sailing. We're using MySQL (God help us) and our development machines are all running Mac OS X. On Mac OS X MySQL identifiers, including table names, are case insensitive. Whilst I would object strongly to a case sensitive programming language this actually makes working at the sql console slightly less annoying so it isn't a problem in itself.

We define our data model using standard Django model classes:

from django.db import models

class NewTableName(models.Model):
    NewColumnName = models.CharField(max_length=255, db_column="OriginalSpaltennamen")

     class Meta:
         db_table = 'UrsprunglichenTabellennamen'

The Meta.db_table specifies the table name that will actually be used in the database. We use the original table and column names where possible as the end users will have to modify some existing tools to work with the new system and this minimizes the changes. As you can see both the original table and new table names are mixed case.

For some reason, which I never got to the bottom of, where the model classes have foreign key relationships syncdb will create these tables with all lowercase names. This could be Django, MySQL or the Python connector to MySQL (or any combination of these) and I never worked out why.

Unfortunately sqlclear will only generate sql to drop tables where the casing specified in the model exactly matches the casing in the database. I worked round it by changing all our Meta.db_table entries to be all lowercase. Not what you would call ideal but acceptable.

Now everytime we update our database schema we can simply run this script. It drops all existing tables and then re-creates them with all the changes.

A Custom json Emitter for Django

I blogged previously about the limited built-in support for JSON in Django (Python web framework), and how I got around it by first using the wadostuff json encoder and then modifying the one built in to django-piston.

Well, it turned out that json was still a bottleneck in our application. A little bit of background first. I'm working with Comsulting.de on a business application using Django on the server and Silverlight, programmed with IronPython, running in the browser. As the Silverlight application is delivered as a 'xap' file, the application and Django communicate via JSON. This means that we use the Django ORM, URL routing and views, but not the Django templating system.

As I can't program without tests we're testing in the browser with unittest, and to mitigate against slow application startup we do parallel imports in the background (multithreaded using a modified version of ipy-parallel-import) whilst the user is presented with the login screen. If the user logs in before imports have completed they get a progress bar until it is done. The basic framework for all of this is pretty simple and I'll turn it into an article or two when I have time.

The application itself is almost the archetypal business application. The main view is a grid displaying information on about 900 different companies. Users can drill down into details on any company, generate reports, amend details, add contacts and so on. As the main view fetches information on 900 companies, our initial json payload for that view was 2.2 megabytes of json! Unsurprisingly encoding, sneding and decoding that amount of information takes some time. Too much time.

There are various ways we improved performance in our main view; caching the generated json is one way and enabling the gzip middleware was another.

By far the biggest improvement came from implementing a custom json emitter that only sends the information that we actually need. The default Django json serializer and the modified emitter based on django-piston both send a huge amount of redundant information. With a custom emitter that only sends specified information (yet can work with basic Python data-types and Django model objects) we were able to reduce this initial payload from 2.2 megabytes down to around 500 kilobytes.

The encoding and decoding are both much faster (and fast enough in particular) now. The code for our custom emitter is shown below. No it doesn't handle everything (dates and Decimal for example - and nor would it cope with recursive references), but that is the point. It is simple, it only knows about what we need it to know about and it only sends what we ask for. Just as important it is easy to extend as our requirements grow or change. The work is done by the construct method, whose job it is to turn the object tree we pass in into primitive objects only that simplejson can serialize for us.

from django.db.models import Model
from django.db.models.manager import Manager
from django.db.models.query import QuerySet

from django.core.serializers.json import DateTimeAwareJSONEncoder
from django.utils import simplejson
from django.utils.encoding import smart_unicode

NoneType = type(None)
MISSING = object()

class Emitter(object):

    def __init__(self, fields):
        self.fields = set(fields)

    def render(self, data):
        out = self.construct(data)
        return simplejson.dumps(out, cls=DateTimeAwareJSONEncoder, ensure_ascii=False)

    def construct(self, data):
        fields = self.fields
        def _any(thing):
            if isinstance(thing, (set, tuple, list, QuerySet)):
                return [_any(t) for t in thing]
            elif isinstance(thing, Model):
                # Handle django model objects
                return _model(thing)
            elif isinstance(thing, Manager):
                # For many-to-many relationships
                return [_any(t) for t in thing.values()]
            elif isinstance(thing, dict):
                return dict((_any(key), _any(value)) for key, value in
                            thing.iteritems() if key in fields)
            elif isinstance(thing, (basestring, int, float, long, NoneType)):
                # primitive types
                return smart_unicode(thing, strings_only=True)

            # ha!! (useful for debugging)
            raise TypeError('Asked to handle unknown type: %r, %s' % (thing, type(thing)))

        def _model(thing):
            ret = {}
            for member in fields:
                attr = getattr(thing, member, MISSING)
                if attr is MISSING:
                    continue
                ret[member] = _any(attr)
            return ret
        return _any(data)

Using the emitter is simple, we simply have to specify what information we want in the response:

@login_required
def some_view(request):
    ...
    fields = ('FirstName', 'FamilyName', 'eMail', 'PhoneNumber')
    emitter = Emitter(fields=fields)
    json = emitter.render(in_data)
    return HttpResponse(json, mimetype="application/json")

The style of the emitter is very much influenced by django-piston, but the code was all written from scratch so any bugs or failings are entirely my own...

Resolver One 1.7, BioPython and 3D Graphics

Although I've left Resolver Systems I still follow closely what they're up to. Resolver One, the IronPython powered spreadsheet with the programming model right at its heart, is an innovative and unique product [1] that deserves to flourish. Despite the horrific loss Resolver One seems to still be moving forward at great pace without me.

• Resolver One 1.7 Released
• Resolver One 1.7 Changelog

Major features in this new release, the last one in which I have been involved in the development [2], include:

• Button-click handlers are now executed without blocking the rest of Resolver One. This means that if you accidentally write a handler that never finishes, or just one that takes longer than you want, you can cancel it while it's running and fix the problem.
• Added long-awaited dialog for setting wrapping and aligment for cells.
• External imported modules now shared between documents and RunWorkbook (better performance for RunWorkbook in particular).
• Faster install time.
• Improved responsiveness when dragging tabs and switching between tabs.
• [Shift] while clicking Recalc toolbar button now now forces reload of all imported modules (like Shift-F9 has always done).
• A few minor bugfixes.

Giles Thomas and team have also produced screencasts of a couple of particularly interesting uses of Resolver One:

• 3D graphics in Resolver One using OpenGL and Tao, part III: Stock prices

Using OpenGL and Resolver One for 3D visualization of stock prices:

This one uses Yahoo! Finance to download the close prices over the last two years for every stock that's currently in the Dow Jones index, then charts them in a 3D window which you can pan and zoom using the mouse. Here's a video showing it in action...

If you're interested in OpenGL Giles has a blog exploring WebGL, OpenGL in the browser: Learning WebGL.

• BioPython in Resolver One

Resolver One includes built-in support for Python C extensions like Numpy through the Resolver Systems sponsored open-source project Ironclad. Ironclad overcomes IronPython's inability to use compiled C-extensions. This video demonstrates the use of one such powerful library, BioPython, from within a Resolver One spreadsheet, to compare the molecular shapes of proteins. (3m12s)

Django json support

As I mentioned in my last entry I'm now working on a Silverlight application with Django on the backend. This means that we're using Django to serve json to the Silverlight application, so whilst we're using the Django ORM, url routing and authentication we aren't using its templating.

The data model is 'unusual' but makes sense for the app. We've only implemented the first user story, which uses a subset of the data, but you can already start to see the shape of it. Here's a simplified approximation of the data from the point of view of the Django model classes:

from django.db import models

class CompanyType(models.Model):
    type = models.CharField(max_length=255)

class Company(models.Model):
    name = models.CharField(max_length=255)
    company_type = models.ForeignKey(CompanyType)

class Address(models.Model):
    street = models.CharField(max_length=255)
    city = models.CharField(max_length=255)
    postcode = models.CharField(max_length=255)
    company = models.ForeignKey(Company)

class Individual(models.Model):
    first_name = models.CharField(max_length=255)
    last_name = models.CharField(max_length=255)
    address = models.ForeignKey(Address)

The reason for this slightly non-intuitive setup is that a company may have several addresses. At every address there can be several contacts.

In our view we have a companies function that needs to return a list of all the companies. If we use the built-in json serializer then for the company_type field it just puts an id number into the json. If we wanted the actual company_type then we would have to make an additional query per company.

Additionally, for this view we want to retrieve all of the addresses associated with a company and every individual associated with each address.

There is a project called wadostuff that includes a replacement serializer. It's very easy to use, just specify the following in settings.py:

SERIALIZATION_MODULES = {
    #'json': 'djangoserializers.json'
    'json': 'wadofstuff.django.serializers.json'
}

When we import and call the Django json serializer we can now specify relations for the serializer to follow and include in the json:

from django.core import serializers
from project.app.models import Company

from django.http import HttpResponse

def company(request):
    companies = Company.objects.all()
    json = serializers.serialize(companies, relations=('company_type',))
    return HttpResponse(json, mimetype="application/json"))

This doesn't solve the problem of how we include the addresses and individuals information. One option would be to generate three separate lists and include them all in the json and let the client sort them out. The wadostuff serializer does let us specify a set of extra fields (extras). Despite what the documentation says, in practise I had to implement these as methods on the model objects that could only return a string. This means I couldn't use it to return a list of model objects like I wanted.

Maybe I'm missing something obvious, which is entirely likely as I'm new to Django, but it doesn't seem like this use case is that unusual. I'm surprised that Django has no infrastructure at all to support this kind of use case??

After a bit of hunting I discovered the awesome django-piston project. We don't need an XML or YAML API, nor streaming or throttling, but it includes an awesome json serializer that I 'borrowed' and hacked around so that I could use it on its own. My final code for associating each company with the related addresses and individuals looks like this:

from project.app.models import Company
from project.modules.emitter import Emitter

from django.http import HttpResponse


def companies(request):
    companies = Company.objects.all()

    for company in companies:
        addresses = company.address_set.all()
        company.addresses = addresses
        for address in addresses:
            individuals = address.individual_set.all()
            address.individuals = individuals

    emitter = Emitter(fields=('company', 'company_type', 'address', 'addresses', 'individuals'))
    thejson = emitter.render(companies)
    return HttpResponse(thejson, mimetype="application/json")

This follows the 'company', 'company_type', and 'address' relations on model objects it serializes and also handles the addresses and individuals fields. What I get back on the Silverlight end is json representing a list of all companies. Each company has an 'addresses' field with a list of all addresses for that company and each address has an 'individuals' field. This is exactly what we need.

New Job with Django and IronPython

Big personal news; I've changed jobs. After more than three years working with Resolver Systems I felt it was time to broaden my development experience. I greatly enjoyed working with Resolver Systems and learned an enormous amount; I'm sorry to leave them but I'm sure they'll manage to cope without me.

I'm now freelance, starting with a contract with a web development firm based in Germany called Comsulting. I first came into contact with Comsulting earlier this year through their lead developer Sven Passig. One of their big customers wanted a web application with the front-end written in IronPython and Silverlight. I did some consulting for them on this as it was their first Silverlight application.

You can hear a bit about what they've been up to with IronPython and Silverlight on the Python 411 podcast that Sven recorded with Ron Jeffries:

• Python 411: IronPython and Silverlight with Sven Passig

I've just been onsite in Germany with them for two weeks but will mainly be working from home. Comsulting's biggest customers are within the German and Swiss media industry and they have several applications for tracking and organising advertising in magazines and websites. It turns out that these media companies really like Silverlight...

We're developing web applications for these companies and after working on the tail end of one project for the first week I started a new project with Stepan Mitt, a designer / developer who also contracts for Comsulting and who happens to be a really cool guy. We're using Django on the server (with CPython 2.5 on Linux) and Silverlight on the client side. I'm converting the Comsulting guys to Test Driven Development, but we still need to investigate the best way to functionally test Silverlight applications.

It's great to be working with CPython again, and especially with Django, but also great to be able to use my IronPython experience. Our Silverlight application communicates with Django via json, so we're using the Django ORM and authentication but our views generate json. I'm sure there will be a blog entry or two out of this.

Resolver One 1.6.5 Released

It isn't long since we released version 1.6 of Resolver One, but we're trying to increase the frequency of our releases and get features into the hands of our users faster.

Resolver One is a highly programmable spreadsheet for Windows. Resolver One is written in IronPython and is fully programmable in IronPython. Through the integration of Ironclad you can use Python C extensions like Numpy in the spreadsheet grid and user code.

We've pushed out a new release of Resolver One, and as it is a medium-sized release we've called it version 1.6.5.

• Resolver One 1.6.5 Released

The new features in this release our

• Load new files and import from Excel in the background and display a progress bar (leaving the UI usable).
• Support for a new VerticalAlignment property (from user code only - user interface integration is on its way!)
• Fixed bug #2303: Code that checked whether there was a new version to update to could make it look like Resolver One was sending usage information against the user's wishes, even though it wasn't.
• Fixed bug #2268: Memory usage problem when saving huge spreadsheets.
• Fixed bug #2291: Memory usage problem with spreadsheets with very large numbers of errors.

(Plus a few minor bugfixes since the 1.6 release.)

We already have more features completed and on the way, so hopefully we'll be able to do a 1.7 release soon...

In his email to the IronPython mailing list announcing the releas Resolver Systems boss Giles Thomas included the following information about Resolver One:

We are proud to announce the release of Resolver One, version 1.6.5. Resolver One is a Windows-based spreadsheet that integrates IronPython deeply into its recalculation loop, making the models you build more reliable and more maintainable.

It's also still (we think) the largest IronPython application in the world, with 56,000 lines of code backed up by 176,000 lines of unit and functional tests.

For versions 1.6 and 1.6.5, we've made it easier for people to share their spreadsheets. A new free "player" version means you can pass your work on to other people, and they can use it without having to buy anything, while a new Resolverlib makes calling your spreadsheets from IronPython programs as easy as calling a function.

You can read more about Resolver One here:

http://www.resolversystems.com/products/resolver-one/

We have a 31-day free trial version, so if you would like to take a look, you can download it from our website:

http://www.resolversystems.com/download/

If you want to use Resolver One in an Open Source project, we offer free licenses for that:

http://www.resolversystems.com/opensource/

• Front page
• All By Date
• All By Category
• Blog Statistics
• RSS Feed
• Return to Top

Technorati...
This work is licensed under a Creative Commons Attribution-Share Alike 2.0 License.

Counter...

• IronPython in Action

  

  IronPython in Action is the first English book on IronPython. It is for Python and .NET programmers interested in making the most of Python on the .NET framework and Mono.

• Search Site

  	Search this Site:
  	www voidspace.org.uk

• Browsing

  Home > Python Index > Techie Blog

• Blogads

  Blogads

• Michael Foord

  • Contact Michael
  • Michael on Twitter
  • My bookmarks
  • Shared articles
  • Photos on Flickr
  • Voidspace Amazon Store
  • Amazon Computers: Todays Deals

• Hosting