Python Programming, news on the Voidspace Python Projects and all things techie.

A Custom json Emitter for Django

emoticon:development I blogged previously about the limited built-in support for JSON in Django (Python web framework), and how I got around it by first using the wadostuff json encoder and then modifying the one built in to django-piston.

Well, it turned out that json was still a bottleneck in our application. A little bit of background first. I'm working with Comsulting.de on a business application using Django on the server and Silverlight, programmed with IronPython, running in the browser. As the Silverlight application is delivered as a 'xap' file, the application and Django communicate via JSON. This means that we use the Django ORM, URL routing and views, but not the Django templating system.

As I can't program without tests we're testing in the browser with unittest, and to mitigate against slow application startup we do parallel imports in the background (multithreaded using a modified version of ipy-parallel-import) whilst the user is presented with the login screen. If the user logs in before imports have completed they get a progress bar until it is done. The basic framework for all of this is pretty simple and I'll turn it into an article or two when I have time.

The application itself is almost the archetypal business application. The main view is a grid displaying information on about 900 different companies. Users can drill down into details on any company, generate reports, amend details, add contacts and so on. As the main view fetches information on 900 companies, our initial json payload for that view was 2.2 megabytes of json! Unsurprisingly encoding, sneding and decoding that amount of information takes some time. Too much time.

There are various ways we improved performance in our main view; caching the generated json is one way and enabling the gzip middleware was another.

Note

Caching would possibly be a good subject for another blog entry. I ended up using a crude, custom in-database caching mechanism. The existing systems seem to use time based cache invalidation whereas I want unlimited time lifetime but precise control of when to invalidate cache entries programatically. At some point we will probably need to investigate 'proper' caching machinery but our current system is sufficient at the moment.

By far the biggest improvement came from implementing a custom json emitter that only sends the information that we actually need. The default Django json serializer and the modified emitter based on django-piston both send a huge amount of redundant information. With a custom emitter that only sends specified information (yet can work with basic Python data-types and Django model objects) we were able to reduce this initial payload from 2.2 megabytes down to around 500 kilobytes.

The encoding and decoding are both much faster (and fast enough in particular) now. The code for our custom emitter is shown below. No it doesn't handle everything (dates and Decimal for example - and nor would it cope with recursive references), but that is the point. It is simple, it only knows about what we need it to know about and it only sends what we ask for. Just as important it is easy to extend as our requirements grow or change. The work is done by the construct method, whose job it is to turn the object tree we pass in into primitive objects only that simplejson can serialize for us.

from django.db.models import Model
from django.db.models.manager import Manager
from django.db.models.query import QuerySet

from django.core.serializers.json import DateTimeAwareJSONEncoder
from django.utils import simplejson
from django.utils.encoding import smart_unicode

NoneType = type(None)
MISSING = object()

class Emitter(object):

    def __init__(self, fields):
        self.fields = set(fields)

    def render(self, data):
        out = self.construct(data)
        return simplejson.dumps(out, cls=DateTimeAwareJSONEncoder, ensure_ascii=False)

    def construct(self, data):
        fields = self.fields
        def _any(thing):
            if isinstance(thing, (set, tuple, list, QuerySet)):
                return [_any(t) for t in thing]
            elif isinstance(thing, Model):
                # Handle django model objects
                return _model(thing)
            elif isinstance(thing, Manager):
                # For many-to-many relationships
                return [_any(t) for t in thing.values()]
            elif isinstance(thing, dict):
                return dict((_any(key), _any(value)) for key, value in
                            thing.iteritems() if key in fields)
            elif isinstance(thing, (basestring, int, float, long, NoneType)):
                # primitive types
                return smart_unicode(thing, strings_only=True)

            # ha!! (useful for debugging)
            raise TypeError('Asked to handle unknown type: %r, %s' % (thing, type(thing)))

        def _model(thing):
            ret = {}
            for member in fields:
                attr = getattr(thing, member, MISSING)
                if attr is MISSING:
                    continue
                ret[member] = _any(attr)
            return ret
        return _any(data)

Note

We use select_related in our Django queries which follows foreign key relationships automatically and improved performance by ensuring that the emitter itself rarely results in kicking off new queries.

Using the emitter is simple, we simply have to specify what information we want in the response:

@login_required
def some_view(request):
    ...
    fields = ('FirstName', 'FamilyName', 'eMail', 'PhoneNumber')
    emitter = Emitter(fields=fields)
    json = emitter.render(in_data)
    return HttpResponse(json, mimetype="application/json")

The style of the emitter is very much influenced by django-piston, but the code was all written from scratch so any bugs or failings are entirely my own... Smile

Like this post? Digg it or Del.icio.us it.

Posted by Fuzzyman on 2009-12-24 17:19:28 | |

Categories: , Tags: ,


UK TV License and the Common Law Right of Access

emoticon:black_hat The UK, in common with countries like France, has a separate 'tax' to pay for the public television stations. In the UK these stations are run by the British Broadcasting Corporation who also collect the Television License fee. The BBC TV channels are generally of very good quality [1] and so I'm not against the TV license fee (as many people are), but as I don't watch broadcast television I don't have to pay the fee.

Unfortunately the department responsible for collecting the license assume that if you don't pay the fee then you are watching television illegally; they don't even entertain the possibility that you might not need a license. They send threatening letters demanding payment virtually every week, they use detector vans that can tell if you are watching terrestrial TV via an aerial [2], and everywhere that sells televisions is legally obliged to pass your address onto the licensing authority when you buy a TV. Even worse officers from the licensing authority come and visit unlicensed homes with the intention of blackmailing you into buying a license - oops, I mean checking to see if you are watching TV illegally.

Note

The BBC offers all of its programs via the internet for up to a week after they are broadcast via a catch-up service called iPlayer (available in the UK only unfortunately). This is excellent and we sometimes watch this through our Wii. It is well established legally that you don't need a license to watch this, you only need a license if you watch programs as they are broadcast (but you do need a license for cable TV even if you don't watch the BBC channels).

These 'enforcement officers' have no legal authority and no right to enter your house unless you let them in. In the UK if anyone comes onto your property without permission it is trespass, for which the property owner can sue (not prosecute - it is a tort, a civil offence not a criminal one). However, there is a common law 'right-of-access' for people to come onto your property and knock on your door. What is less well known is that you can withdraw this right-of-access from individuals and companies (and therefore all their employees). If you do withdraw this right from the TV Licensing Authority then if any of their officers knock on your door they are guilty of trespass.

If you write to the TV Licensing authority and inform them that you withdraw the common law right-of-access then they will stop sending enforcement officers to visit you (for at least a couple of years anyway). I have no problem telling them to clear off in person, but would prefer them not to hassle my wife.

For those interested in doing the same, here is the letter I sent:

Dear Sir/Madam,

I have no need of a TV licence, and am not breaking the law, and yet I have received continual letters and threats from your company. Your latest missive even included details of a television I purchased, as if to indicate how closely you are monitoring.

That television is in our front room and connected to a DVD player and a games console. It is not used to watch broadcast television. We do not receive any broadcast television, neither by terrestrial broadcast nor by cable nor any other means, and so we do not need a license. I am aware of both the law and our rights in this area.

Your visiting officers will not call at my address. This letter denotes prior written and legal warning that any such visit will constitute trespass and harassment. Normally there is an assumed right of entry to the front door of a property. However this is denied to your employees, since any such act will evidently constitute harassment since prior warning has been given.

Your company will not send me any threatening letters or any other correspondence; you will not visit my property, neither will you visit my home. You may, of course, reply to this letter. Although I have nothing to hide, I resent your intrusion and harassment into my life and the life of my family.

You have now been informed that any such visit or usual threatening letter, or threats of visits, constitute harassment. You will immediately cease. You may, of course, reply to this letter. You will kindly acknowledge in your reply that you have noted/understood the contents of this letter.

All the best,

Michael Foord

My intention with the letter was to not just forestall any visits but also to cease the endless stream of threatening letters. A couple of weeks after sending this letter I got a very terse reply acknowledging that I had withdrawn the right of access. They did 'reserve the right' to use other means to contact me. Sending letters is not a trespass as it is the post office who delivers them (and I'm not withdrawing their right of access). However, I haven't received any other letters from them since...

[1]As is the way with publicly run organisations, I'm sure the BBC is bureaucratically top heavy but they do produce some very good TV and technology. On the other hand I've known at least a couple of programmers who have worked there who say the internal politics between departments can be stifling. I guess this is true of many large organisations though.
[2]Many people seem to think these detector vans are works of fiction; but they do work. They can detect the signals radiated back out through your aerial that your TV uses to decode the radio signal. This obviously only works if you use an aerial (and not cable).

Like this post? Digg it or Del.icio.us it.

Posted by Fuzzyman on 2009-12-24 16:52:23 | |

Categories: Tags: , , ,


A Little Bit of Python (Podcast)

emoticon:waffle Myself and several co-conspirators have started an 'occasional podcast' on all things Python, and the first two episodes are now online. The podcast is called A Little Bit of Python and stars:

All of us are Python Software Foundation members [1], we are all (or have been) core Python developers and we're all full time programmers in one way or another (Brett is still a student and Steve does a lot of training as well as programming). We're also all talkative and argumentative, so it should be a fun series. The website (bitofpython.com) isn't up yet, so the first two episodes are available from a temporary location, but we'll setup redirects as soon as we're properly up and running:

The first three episodes cover topics like the new Python moratorium, PyCon, Python 2.7, and the great PyPI comments debate (debacle?). We'll generally be discussing anything newsworthy or of interest affecting the Python community.

Note

The podcast isn't up on iTunes. We'll do that as soon as the website is up and running.

If you want to send fan mail, hate mail, or suggestions for topics then you can already email us at: all@bitofpython.com

We're still working on the technical details, so audio quality will improve. In particular the audio of my voice is very poor in the first two episodes but should be better in the third episode that is being edited as you read this and will go up soon. We're finding our stride, both in the audio technology and in finding our feet - recording a podcast with five people makes for interesting group dynamics. Be gentle on us for these early episodes. Very Happy

Thanks to Andrew for motivating this and for handling the editing. Thanks to Jesse for doing the website work which I have every confidence will be online shortly... Wink

[1]But this is in no way an official podcast. Brett in particular wanted to make it clear that he swears a lot (although he's curbed himself so far) so it may not be suitable for minors...

Like this post? Digg it or Del.icio.us it.

Posted by Fuzzyman on 2009-12-22 16:13:11 | |

Categories: , Tags: , ,


Hosted by Webfaction

Counter...