Python Programming, news on the Voidspace Python Projects and all things techie.

IronPython in Action

emoticon:clock I guess now is as good a time as any to announce it. I have arranged a deal with Manning Publications to write a book on IronPython.

I've reviewed a couple of the Manning In Action series before and been very impressed with the quality. This book will be called IronPython in Action.

I've spent about six months working on a very detailed Table of Contents with Manning [1], so I think the book should be relevant to lots of people with an interest in IronPython; both .NET programmers and Python programmers. I'll let you know a bit more if and when Manning give me permission.

It should be available in early Autumn if everything goes according to plan. I wonder what the chances of that are ? Wink

A colleague of mine, Christian, will be writing chapters on ASP and ADO with IronPython. He's used ASP and ADO extensively in previous jobs and he's an excellent Python programmer.

I've started work on the book already. Needless to say it was the book I was working on when I had my computer disaster over the weekend. Oh well, a good excuse to buy a better laptop.

Anyway, one consequence of this fresh and imprudent endeavour is that I'll be posting blog entries a lot less over the next few months. It's not because I've forgotten you though. Smile

I'm also doing a talk on IronPython at PyCon, with another colleague Andrzej. Hopefully I'll see some of you there. The lineup looks extremely interesting and I'm really looking forward to it.

[1]We've nearly tied down all the details now, but they'll probably change. Razz

Like this post? Digg it or Del.icio.us it.

Posted by Fuzzyman on 2007-01-10 23:46:46 | |

Categories: ,


Wing Script: Part II

emoticon:firefox Christian and I have done some more work on scripting the Wing Python IDE for work.

We've modified the 'Execute to Editor Window' script. The output window now scrolls with the output and it reads a line at a time rather than a character (which seems to work faster). It also does a 'save-all' rather than only saving the current file.

import os
import sys
import wingapi

from popen2 import popen4
from threading import Thread


def output_window(app=wingapi.kArgApplication):
    directory, filename = os.path.split(
        app.GetActiveEditor().GetDocument().GetFilename()
    )

    title = "Command Output"
    editor = app.ScratchEditor(title, 'text/plain', sticky=True)

    wingapi.gApplication.ExecuteCommand('save-all')
    doc = editor.GetDocument()
    editor.ScrollToLine(doc.GetLineCount() - 1, select=1, pos='top')

    executable = 'python %s' % filename
    os.chdir(directory)

    stdout_and_stderr, child_stdin = popen4(executable)
    child_stdin.close()

    initialPos = max(0, doc.GetLength() - 1)
    doc.InsertChars(initialPos, '\n\n')
    initialPos += 2

    def DisplayOutput(pos):
        while True:
            output = stdout_and_stderr.readline()
            if not output:
                break
            doc.InsertChars(pos, output)
            pos += len(output)
            try:
                editor.ScrollToLine(doc.GetLineCount() - 1, select=1, pos='top')
            except ValueError:
                pass

        doc.InsertChars(pos, '\n\nTool Completed\n\n')
        try:
            editor.ScrollToLine(doc.GetLineCount() - 1, select=1, pos='top')
        except ValueError:
            pass

    Thread(target=DisplayOutput, args=(initialPos,)).start()

The next script implements jump_to_error_line. If you get an error traceback from your code, you can put the cursor on a line in the traceback and press the key you have bound to this function. Wing will open the file and jump to the line indicated in the traceback.

import wingapi
import re

error_details_re = re.compile(r'^  File "([^"]+)", line ([0-9]+),')

def jump_to_error_line(app=wingapi.kArgApplication):
    editor = app.GetActiveEditor()
    document = editor.GetDocument()
    start, end = editor.GetSelection()

    lineno = document.GetLineNumberFromPosition(start)
    text = document.GetText()
    line = text.splitlines()[lineno]
    match = error_details_re.match(line)

    if match:
        filename, lineno = match.groups()
        newEditor = app.OpenEditor(filename, sticky=False)
        newEditor.ScrollToLine(int(lineno) - 1, select=1, pos='center')

These two scripts make working with Wing much easier for us.

Unfortunately my boss is baulking at the cost of buying us a license each, and wants us to try Eclipse first. I think the ease of scripting Wing to add new features has to be a big win though.

The try: ... except ValueError lines in output_window are to get round what we think is a bug in the Wing API. Every now and then the call to editor.ScrollToLine will fail with an obscure traceback similar to the following :

Traceback (most recent call last):
 File "...\output_window.py", line 42, in DisplayOutput
   editor.ScrollToLine(doc.GetLineCount() - 1, select=1, pos='top')
 File "C:\Program Files\Wing IDE 2.1\bin\wingapi.py", line 954, in ScrollToLine
   self.fEditor.ScrollToLine(lineno, select, pos)
 File "2.4/src/edit/editor.pyo", line 1215, in ScrollToLine
 File "C:\src\ide-2.1\bin\2.4\src\guimgr\multieditor.pyo", line 1322, in NextBookmark
ValueError: __fEmissionStack should have 173 entries, but it has 174 entries instead

This is possibly a threading issue. We call doc.InsertChars and editor.ScrollToLine from a thread because we need the subprocess to execute without blocking Wing, whilst updating the scroll position. Perhaps GTK has a way of invoking these calls on the main thread ?

Like this post? Digg it or Del.icio.us it.

Posted by Fuzzyman on 2007-01-10 22:13:18 | |

Categories: , ,


Mega-Computer-Death Badness

emoticon:pencil It's not been my week. sigh

I bought a second-hand UPS off ebay. A nice chunky one, at a very good price, except guess what ? It doesn't work. Sad

The company I bought it from are replacing it.

Next the second replacement Western Digital hard-drive (WD3200KS 320gig) isn't compatible with my motherboard (MSI K9N Neo - MS-7260 Series) or something. I keep getting I/O Device Error messages after a couple of minutes of use, even when I switched to the slower speed SATA interface. Dabs have been very good at replacing them for me, even though they say they can't find any fault.

Note

A chap called Johan emailed me having had a similar problem. He had better luck than me finding the solution...

Its the included Nvidia SATA drivers that are jerking the chain with us. I found the problem today, uninstalled those drivers and used the Windows XP Pro included ones, works like a charm.

The worst is yet to come, as I'll explain in an entry or two - this couldn't have happened at a worse time. My computer has just died. It won't boot, no BIOS POST; nothing. double sigh

Yes I was in the middle of work and no I hadn't got round to backing it up anywhere. I've bought a USB SATA enclosure. I'm now working off my laptop, which is just about adequate, and I can now access all my data. Even better, my laptop docking station will drive two monitors, so although it's slower than my desktop I can still watch films whilst programming.

I'll use the enclosure for my Western Digital drive once my computer has come back from warranty repair. Cube are being very good about repairing it. They do however want me to pay the carriage to return it. Whilst this may be standard practise, I'm also pretty sure it's against UK consumer legislation. I think I'll wait until my computer has been returned to complain though. Smile

Last of all in my run of bad luck, I sold my PDA on Ebay. It went for more than I expected considering this is just after Christmas. This may not sound like bad luck, but Ebay reported that the winning bid was fraudulent and cancelled the sale.

They didn't give me a chance to offer the PDA to the next bidder, they didn't provide any way of relisting it automatically and they invalidated the original listing so I had no way of retrieving what I had originally written and had to start again. Gits.

So for the second time in two weeks, if you're after a high speed PDA (excellent condition needless to say), with GPS, memory card and loads of extras :

Wink

Like this post? Digg it or Del.icio.us it.

Posted by Fuzzyman on 2007-01-10 21:57:00 | |

Categories: ,


ShedSkin Part II: Garbage Collection and Memory Management

emoticon:cat In my post about ShedSkin a couple of days ago I said that I was unsure how ShedSkin handled memory management and whether it has any sort of garbage collection.

Automatic garbage collection is the feature of dynamic languages which most reduces programmer headaches. Of course depending on the implementation it can lead to less memory efficiency than hand coded C or C++.

Mark Dufour replied, explaining a bit about the ShedSkin garbage collection implementation and about the future direction of ShedSkin. He is heading towards a system that will let you write code with a subset of Python, and be able to create extremely fast Python extensions which ShedSkin has translated into C++. I thought you might be interested. Smile

Shed Skin uses the Boehm garbage collector. Perhaps because it was so easy to use, and I have never had a problem with it, I completely forgot to mention it on the homepage.

A sidenote on memory management: there are some interesting techniques in the (java) literature that can turn many a heap allocation into stack- or static preallocation, greatly improving performance for some programs. Unfortunately the rest of the compiler is enough work for now, so that I cannot look into this. but it can bring memory performance even closer to that of manual C++, which is the goal of shed skin, of course.

So... you just develop your program with CPython, then (in the future) profile it, so SS gets a good idea of the types (which makes type inference dramatically easier), and can give warnings when something is not supported (most notably dynamic types), and when you're done you give it to SS unmodified and receive back a binary/extension module. Smile

So it's a bit like pyrex, in that you can create really fast extension modules, but you only ever use python and not a line of C/C++.

It sounds very exciting.

Like this post? Digg it or Del.icio.us it.

Posted by Fuzzyman on 2007-01-10 21:45:50 | |

Categories: , ,


Another Wing Script

emoticon:exclaim I've been playing around with the Wing IDE scripting API a bit more.

What we would like is to be able to execute the current script with an external tool, and have the output appear in a new editor window. This is nice because you can cut and paste and generally mess around with the script output in a real editor window. This is a feature we miss from TextPad.

Below is a script that does just that. It fires off the script you are currently editing (saving it first) as a subprocess: piping stdout and stderr into a new editor window.

import os
import sys
import wingapi

from popen2 import popen4

from threading import Thread


def output_window(app=wingapi.kArgApplication):
    directory, filename = os.path.split(
        app.GetActiveEditor().GetDocument().GetFilename()
    )

    title = "Command Output"
    editor = app.ScratchEditor(title, 'text/plain', sticky=True)

    wingapi.gApplication.ExecuteCommand('save')
    doc = editor.GetDocument()
    lineno = doc.GetLineCount()
    doc.InsertChars(doc.GetLength(), "\n\n")
    editor.ScrollToLine(lineno, select=1, pos='top')

    executable = 'python %s'
    os.chdir(directory)

    stdout_and_stderr, child_stdin = popen4(executable % filename)
    child_stdin.close()

    def DisplayOutput():
        while True:
            output = stdout_and_stderr.read(1)
            if not output:
                break
            doc.InsertChars(doc.GetLength(), output)

    Thread(target=DisplayOutput).start()

The script is launched with python, which is hardwired into the script.

Binding the output_window function to a key combination is very easy.

The next step will be to get Wing to recognise tracebacks and let us jump to the file and line, opening the file if necessary. I don't think I can get Wing to give me a hook onto double clicks in the output window, but at least I can bind it to another key-combination and use a regular expression to check if the cursor is in a traceback.

Like this post? Digg it or Del.icio.us it.

Posted by Fuzzyman on 2007-01-08 19:54:36 | |


ShedSkin (leading into a work related ramble)

emoticon:nightmare ShedSkin is one of those tools that have arrived in recent days and looks extremely interesting.

ShedSkin compiles a subset of Python, subject to some 'static language' restraints (you can't change the types of variables etc), into C++. Speed increases over CPython range from 2 to 220. It does type inferencing so that you don't need to declare the types of variables. Sounds cool hey ?

Well, last time I chatted to Mark Dufour there were no way to interface code compiled with ShedSkin to Python code. Not speaking any C++, ShedSkin seemed very interesting, very intriguing, just not very useful.

ShedSkin has undergone continued development over the last few months, and I just received an email from Mark :

there is some progress in the 'extension-module' area, as someone else is looking into using boost.python for this purpose. I'm getting enthusiastic about the idea.

So it sounds like ShedSkin could be crossing the line from interesting to useful. What I don't know is how ShedSkin handles memory management or garbage collection. I do recall a rumour that it has garbage collection, so perhaps it is capable of taking much of the sting out of C++.

At the moment I'm spoiled, at least in the sense that I only ever code in Python or C#. Both of these have memory management, so it's not something that I ever have to worry about.

Hmmm... that's not actually true. We recently had one of those hard to diagnose bugs when our test suite would start failing in random places for no good reason. This only happened when we added an extra functional test.

After futzing around with the debugger for a while William sussed it out. Whenever we hit 10 000 GDI objects the tests would crap out. We're not creating that many GDI objects, but a few more are getting added for every functional test and they're not being reclaimed. In the end we solved the problem by running our unit tests and functional rests separately in our build script. I guess sometime we'll have to work out if it's a real problem or not.

Our functional test suite (and some of our unit tests) creates a new instance of Resolver and automates user input, checking for results. This involves stopping and starting the Windows Forms application loop many times.

As well as leaking GDI objects (which may or may not be our fault, at some point we'll need to work that out), we're pretty sure we've uncovered a bug in the application loop. Every now and then Windows Forms will attempt to do something with the mainform after it has been closed. This raises an reference is not set to an instance of an object or attempting to perform an operation on a disposed object exception. It took a long time to diagnose this and we're pretty sure that's the correct reason, but starting and stopping the application loop that many times is not 'normal', so perhaps we can let Microsoft off [1]. Smile

[1]We just catch the exception on exit and ignore it. Before you comment let me repeat that this wasn't a decision that was taken lightly.

Like this post? Digg it or Del.icio.us it.

Posted by Fuzzyman on 2007-01-08 18:33:30 | |

Categories: , , , ,


Capturing stdout and stderr as a Single Stream

emoticon:mobile I would like to execute subprocesses asynchronously and capture stdout / stderr as a single stream.

Before I ask for your help, a couple of notes :

  • This question is for implementing a script inside the Wing IDE. For some reason using the subprocess module doesn't work so I need a solution that doesn't use it.
  • The platform is Windows and I'm happy with a Windoze only solution. Smile

If I use os.popen3(executable) it gives me separate pipes for stdout and stderr, but reads from them are blocking.

The output on stdout and stderr may be interleaved and I would like to display them as they arrive. That means I can't just read from them and output the results.

The only solution I can think of is to read from both a character at a time on two separate threads, putting the data into a queue. A separate thread could pull characters off the queue and display them. (I don't need to differentiate between stdout and stderr when I display.)

Can anyone think of a better solution ?

My current code works, but doesn't capture stderr :

from threading import Thread

pipe = os.popen(executable)

def DisplayOutput():
    while True:
        output = pipe.read(1)
        if not output:
            break
        display(output)

Thread(target=DisplayOutput).start()

Update

Thanks to Gabriel Genellina (on comp.lang.python) who pointed out os.popen4 which I wasn't aware of.

stdout_and_stderr, child_stdin = os.popen4(executable)
child_stdin.close()

def DisplayOutput():
    while True:
        output = stdout_and_stderr.read(1)
        if not output:
            break
        display(output)

Thread(target=DisplayOutput).start()

Like this post? Digg it or Del.icio.us it.

Posted by Fuzzyman on 2007-01-07 19:35:46 | |

Categories: ,


Hosted by Webfaction

Counter...