Python Programming, news on the Voidspace Python Projects and all things techie.
Silverlight and the London .NET Group
The first part of the talk was about Silverlight 1.0, its video streaming and animation capability. This side of Silverlight doesn't interest me at all, and I guess it is aimed at luring away those currently using flash for adverts. There were some impressive demos though: a video jigsaw that split into seventy jigsaw shaped pieces each showing a fragment of the video, which you then have to reassemble.
The second part was about Silverlight 1.1, which of course can host the DLR and be programmed with IronPython. The single largest factor blocking Silverlight from being used to create rich web applications is the fact that it comes with no controls. (Although at least one chap is having fun making his own.) Thankfully the next two releases, beta late summer and final early 2008, will rectify this with quite a rich set of controls being supplied.
|||A marginally more sober group than the London Python Crowd, but only just.|
Windows Flying Dudes: The Agent Server Objects
Have you ever seen this dude before?
Apparently he is called Merlin, and he also does this:
He is part of a bizarre feature of Windows called 'AgentServerObjects'. These are little animated characters that fly around the screen, making announcements and doing quite odd and quirky things. They don't get used very much, in fact it is hard to see what you could use them for. But they're certainly fun.
In order to run this example I generated the interop dll in 'c:\'. This uses tlbimp, which comes with the .NET framework SDK.
C:\>set PATH=%PATH%;C:\Program Files\Microsoft Visual Studio 8\SDK\v2.0\Bin C:\>tlbimp c:\WINDOWS\msagent\agentsvr.exe
Some IronPython code to make this dude fly around your screen:
sys.path.append('C:\\') # or wherever your interop dll is
from AgentServerObjects import AgentServerClass
a = AgentServerClass()
id, rest = a.Load("merlin.acs")
ch = a.GetCharacter(id)
ch.MoveTo(0, 0, 0)
ch.MoveTo(600, 100, 2000)
ch.MoveTo(300, 150, 1000)
Oddly cool... More details here.
My Dad does computer related safety and reliability consultancy. One of his recent projects has been calculating blast danger zones for power stations - how far out from a power station do you need to build if you want the buildings to be safe from flying concrete, in the event of an explosion! This means plotting 'risk contours' for the area around a potential blast zone. He has been using a script to generate the contour images, and frankly the images it produces are rubbish.
As Resolver can process this kind of data, I thought it would be fun to hook it up and see if I can generate some charts for him. He would really like some vector images, to embed into his drawing package.
We use Gnuplot at work to generate some codemetric charts I thought I would start with that. The images it creates are not very glitzy, but it can produce an enormous range of different types of charts (except pie charts).
Trying to get it to work on Windoze was a tale of pain and sorrow I'm afraid. The basic problem is that the Windoze build has no error reporting of any kind. It doesn't even complain when you ask it to build a non-existent script!
When I had got to grips with the basics of the scripting language I managed to coerce gnuplot into producing an empty png file, of course with no hints as to what the problem was. What the README.Windows file doesn't make clear is that the two environment variables for the path to the fonts directory (which has been the same on every Windows machine I've ever used) are essential. Once I'd set those, I eventually got an image out.
GDFONTPATH: the directory where png terminal searches for TrueType fonts.
GNUPLOT_FONTPATH: used by the postscript driver as search path for TrueType fonts.
I generated a test data csv file (test.dat) containing dates and random numbers:
25/07/2007 -99.759 26/07/2007 -62.111 27/07/2007 -33.785 28/07/2007 -20.832 29/07/2007 -27.133 30/07/2007 4.331 31/07/2007 22.647 01/08/2007 54.885 02/08/2007 80.198 03/08/2007 88.198 ...
A gnuplot script to generate a chart from this is:
set timefmt "%d/%m/%y" set term png set output "test.png" set title "Random Numbers Against Date" set style data lines set key left set grid set xdata time set format x "%d/%m/%y" set ylabel "A Random Value" set xlabel "The Date" plot 'test.dat' using 1:2
That was far more effort than it should have been (and I don't really fancy learning another language just to generate charts that aren't as good as I'd like). I'm doing this from IronPython, so I'm launching it as a subprocess anyway. I might as well launch a Python script using matplotlib. For my Dad's machine I can wrap it with py2exe. I know that works because matplotlib plays fine with Movable Python.
Listing Processes with PyWin32
Well of course we weren't using the best techniques with pywin32 either, and Tim Golden has suggested a couple of better solutions:
WMI using pywin32:
wmi = win32com.client.GetObject ("winmgmts:")
procs = wmi.InstancesOf ("Win32_Process")
print "\n".join (proc.Properties_ ("Caption").Value for proc in procs)
Not quite as nice as the IronPython example. Luckily Tim has created a wrapper around pywin32 for WMI stuff.
WMI using wmi module:
procs = wmi.WMI ().Win32_Process ()
print "All running processes:"
print "\n".join (proc.Caption for proc in procs)
Very nice, but has an additional dependency.
PyCon Italia and London Python at the IET
Well, PyCon Italia has just finished. I wasn't there, but Nicola Larosa tells me that it was fantastic. Apparently 200 people attended, which for a national European conference is a lot. It sounds like Python is really making strides in Italy.
Meanwhile, there is more going on in the UK. Specifically, London Python event at the IET's Savoy Place building:
Pete Ryland, Tim Golden and Michael Glazebrook are putting on an evening Python event at the IET (Institute of Engineering Technology) at Savoy Place on 5th July.
Although the talk bit is not aimed at code-gods such as yourselves, that's not really the point ! Though we hope it'll be interesting even so. This is a networking event (free grub & booze) with more time spent socialising than being lectured to. If we can prove enough interest we can hope to get the use of the facilities for as many talks and lectures as we'd care to present. So an important part of this is for us to hear from you what you'd like (and if you're prepared to give a talk).
The event is not yet on the calendar (should go up this week), so here's the description:
Date & Time: 18:30 – 20:00 5th July 2007 Networking and Wine reception 20:00 - 22:00 Event: A light byte of Python Venue: The IET, Savoy Place, London, WC2R 0BL Cost: Free Talk by: Michael Grazebrook, Tim Golden and Pete Ryland
Python is a modern language. Easy enough for the engineer seeking to dabble in simple programming, but powerful enough to be used for serious development (I learned it working for banks). It is also free: open source.
This is an event in three parts. We will introduce Python for beginners, showing simple pieces of code which you can modify at home. We’d like to provide a series of lectures, if you prove there is enough demand, so help us debate what you want. Finally we’ll have some PCs available so you can try it out. We hope you’ll go home feeling confident enough to try it out.
The talk will consist of a handful of brief presentations of simple, working programs, around 10 lines long, doing powerful things such as:
- Opening a Web page and extracting some information
- Simple user interface design
- Manipulating data in an Excel spreadsheet
- Controlling external hardware (an off-the-shelf USB experiment interface board)
If you, our public, show enough enthusiasm we’d like to present a series of evening workshops on Python at the IET. So after the main talk we’ll have some debate about what you would like. The event is supported by the London Python Users Group, who are looking for somewhere a bit more practical than a pub to exchange experience.
If you bring a lap-top, we will help you get the examples running after the talk – even if you have never used Python before.
Resolver, IronPython and Unicode
At work today I was working on a bug with importing data from an external data-set. This turned out to be a bug in our handling of non-ASCII strings, compounded by a bug in our error handling.
In the space of two hours, Jonathan and I went through the following emotions:
- How do we fix this bug?
- Ah, I think we can see what is going on.
- Aargh, serious unicode bug - Resolver is totally screwed!
- Oh. Here's the two line fix that sorts it completely.
To be fair it was actually four lines. The two lines needed to go in two places.
The bug came about at least partly because the handling of Unicode is a bit different between CPython and IronPython (and also because we knew we had a Unicode related bug which this has forced us to address).
In IronPython, the str type and the unicode type are the same object (the .NET string type is Unicode). Here are a couple of examples of the differences:
>>> s + u'33.33'
Traceback (most recent call last):
File "<stdin>", line 1, in ?
UnicodeDecodeError: 'ascii' codec can't decode
byte 0xa3 in position 0: ordinal not in range(128)
Adding a byte string to a Unicode string, resulting in an implicit decode which fails.
>>> s + u'33.33'
Both strings are already Unicode, no decode needed. This can make Unicode handling a lot easier with IronPython.
Calling unicode on a Unicode string is effectively a no-op.
Traceback (most recent call last):
File , line 0, in <stdin>##5
UnicodeDecodeError: Unable to translate bytes [A3]
at index 0 from specified code page to Unicode.
Here the unicode call, is the same as calling str - which does result in an implicit decode which fails.
Our problem was essentially caused by calling str on objects which were already strings. As soon as we stopped doing this, the problem went away .
Another important result of all this, is that you should not use the (Iron)Python string for storing binary data. Instead you should use byte arrays. This means that things like zlib compression and the binary pickling protocol with strings don't always behave correctly. (They don't always misbehave either, you just need to be very careful. I'm not sure how much IronPython magic is going on with these types.)
Loading data into a byte array with IronPython is a bit more verbose than reading a file with Python, but not so bad:
>>> from System.IO import FileStream, FileMode
>>> s = FileStream(filePath, FileMode.Open)
>>> bytes = Array.CreateInstance(Byte, s.Length)
>>> s.Read(bytes, 0, s.Length)
The mutable bytes array (bytes) now contains the data read from the file. The number returned by s.Read(... is the number of bytes read.
Max suggests some slightly shorter code for reading byte arrays from files:
>>> bytes = File.ReadAllBytes(filePath)
|||Well, so far anyway.|
IronPython is Better than pywin32 for Some Tasks
I'm sure this will come as no surprise, for some Windows specific tasks IronPython is a much better fit than pywin32.
For example, certain background processes can bork a build. These are notably Thunderbird and the AVG anti-virus system scan. Both of these suck up CPU cycles in the background and interfere with the build, particularly our performance tests.
We have a simple script bad_processes.py, which checks that these processes aren't running and aborts the build if they are. This currently runs under CPython and uses pywin32.
Recently this has started failing on one machine, with an obscure error. All google has to say on the subject, is that the cause of the error might be a non-English Windows version or a very old pywin32 install. Neither of these is the case.
Fortunately the IronPython equivalent is very simple. Here is how you get a list of all the currently running processes:
procs = Process.GetProcesses()
print "All running processes:"
print '\n'.join(proc.ProcessName for proc in procs)
This has the advantages of being shorter, easier to read and less error prone than our current solution.
Trying to Read Shortcut Targets on Windows
Now that Resolver is in the hands of beta-testers, documentation is becoming more important. We have a quick-start guide written, but the API to our class library is vital to getting the best from Resolver.
Back at Christmas time I wrote the core of a documentation builder, which would put together module documentation from ReST formatted docstrings, including building an index page. We have finally got round to writing the docstrings and integrating building the documentation into our installer.
It aborts (by throwing an exception), if any members of our public API don't have docstrings or the formatting of any of the resulting pages are invalid. The documentation builder is a core of around 400 lines of Python code, which is nice and easy to modify or extend. Once Resolver has gone public (in July), there is a good chance I will be allowed to release this.
The final step in integrating the documentation was writing a functional test that runs the installer, checks that a link to the library documentation appears in the start menu and that this link points to a real file in a directory of HTML files. We already have an installer test, so all we needed to do was check the link exists (a '.lnk' file in a specific directory) and check the other files in the directory that the link points to.
So how exactly do you check the target of a Windows shortcut file? This seemingly simple task ended up driving Christian and I around in circles and taking half a day.
Links files have a binary format (documented here (PDF)), which we didn't fancy parsing. After searching we couldn't find a command line tool or .NET API for reading or following them. Nor could we find more than a couple of people asking about this - it seems odd that no-one else ever needs to do this?
Eventually, through a C# example, we found the Windows Scripting Host (warning: dumb IE only website) which has an API (via interoperability) for accessing shortcuts.
After a while we discovered that you can only use this API to create shortcuts, not to access existing ones! Anyway, I wrote up what we had learned:
Next we discovered a WMI solution, using a very bizarre SQL like syntax for querying the system about shortcut files!
Set objWMIService = GetObject("winmgmts:\\.\root\cimv2") Set query = "Select * From Win32_ShortcutFile WHERE" & " Target = 'N:\\tandem.exe' AND Path = '\\nectem\\'" Set colFiles = objWMIService.ExecQuery(query) For Each objFile in colFiles Wscript.Echo "Name: " & objFile.FileName Wscript.Echo "Shortcut target: " & objFile.Target Wscript.Echo "File name: " & objFile.Description Next
This is more weirdness than we wanted, so we carried on the hunt. In the end I found what we were looking for in the google-cache of an RSS feed from someone's blog!
The final solution is actually fairly easy, using Shell32 interoperability, it just took us half a day to get there! I wrote it up in:
Other recent additions to the IronPython Cookbook include:
- Invoking onto the Control Thread (Windows Forms)
- Reading and Writing JPG Metadata with .NET 3
- Transform XML Files with XSL
- Message Queuing
- Enumerating Installed Fonts
This work is licensed under a Creative Commons Attribution-Share Alike 2.0 License.