Python Programming, news on the Voidspace Python Projects and all things techie.
Movable Python for Python 2.5 Beta 3
Movable Python for Python 2.5 Beta 3.
If you've already signed up for Movable Python 2.5, then you can download it from the group page, otherwise you can sign up for it there :
The interactive interpreter, which was broken in Beta 2, seems to work fine with this one.
The extension modules included are :
Plus the usual pure python modules and all the new ones provided by Python 2.5. If you encounter problems with any of these then please let me know.
Just a brief reminder that until Python 2.5 hits final release it is available for only £3.99 instead of the usual £4.99.
For a list of all the new features in Movable Python 2.0.0, What's New in Movpy 2.
For a summary of the major new features (with screenshots) see Overview of the New Features.
There is also a free trial version for Python 2.3 at Movpy Demo Version.
|||Actually compiled from current CVS, to get round a bug with Python 2.5b3 compatibility in the released build 209.|
Language Bar & Windows Update - Further Windows Annoyances
At work I am surrounded at every turn by Polish Ruby programmers  (despite the fact that we are a Python shop).
This means that for the first time I have seen someone actually using the language bar that is sometimes present on the Windows task bar :
Needless to say (but I'll say it anyway), my colleagues use it to switch between their normal Polish keyboard layout and one which is a bit more conducive to pair programming .
Despite all that, I have no use for the language bar, and it's stubborn persistence at squatting on my task bar has long irritated me. On the various computers I use I would religiously switch it off using the toolbars menu, only to have it maliciously reappear the next time I log in.
I've finally discovered how to get rid of it permanently (well... I think so), thanks to this post :
- Click the Start Button.
- Select Control Panel option.
- Double-Click the Regional and Language Options icon.
- Click on the Languages tab.
- Click the Details button.
- Click the Language Bar button.
- Remove the checkmark from the Show Language Bar option.
- Click OK.
The next annoyance occurred when I tried to do a Windows Update on my new laptop. The update site kept reporting an 0x8024001D site error, and steadfastly refusing to do anything. Of course there is no explanation of this error message on the Microsoft site.
More googling (I guess before the internet we didn't have these problems, so there is no point in asking what we did before we had the internet) and I stumbled across this solution :
net stop wuauserv cd /d %windir% rd /s softwaredistribution net start wuauserv
It seems that at some point an update got corrupted and Windows is unable to work out what to do. Stopping the auto-update service enables you to delete the cache, and then it can start again.
Having done this, my laptop is now downloading and installing around sixty critical security updates.
|||Actually there are two of them, but we have to keep them separated because we suspect they conspire against us in Polish.|
|||One of our interns has his default keyboard layout set to dvorak; without a corresponding keyboard overlay. What's worse is that it can only be switched over per application, and it doesn't seem possible to switch the keymap over for consoles at all. Unsurprisingly he does most of the typing when we pair on his machine. An added bonus is that the only way to log onto his machine is to remember you username and password using the dvorak keymap.|
Good Advice for Laptop Storage
A few blog entries ago I posted about looking for extra hard drive storage for my new laptop. A comment was posted there, which had such good advice in it that I think it deserves an extra post.
The comment was made by Anonymous, so I don't know who to thank (but thanks anyway). Like all the best advice, the basic thrust of it has the that's obvious  quality to it: but there is also a good amount of detail here.
For self-powered firewire drives, my understanding is that if your laptop has the small, 4-pin firewire port (i.e. you'd need a cable like this) the 2 pins that are missing are the ones which provide the extra power needed by "self-powered" peripherals such as hard-drives.
Personally I wouldn't buy an "external drive" but a separate case and 2.5" drive, since it's the quality of the HD that matters most and that's what you'd want to be choosing. I'd upgrade the internal drive and then get the case to put the old one into. Newer drives will probably be faster, cooler (which is a good way of helping your laptop live a long life), and use less power. Plus you don't have to carry an extra box around with you if you get a drive big enough for everything (and then just use the external for backups).
I suspect that at the electronics end of the case you aren't going to notice much difference in quality no matter the price, but it might just be the cynic in me that suspects they all use the same "reference board" PCB from the same Taiwanese chipset manufacturer for their IDE-USB bridges (and that a bit of searching around www.alibaba.com would find you the exact enclosure that Pexagon sell). But several big names make cases too, so you don't have to run the risk of getting something that loses every 3rd bit.
If you do go for buying a separate drive, keep in mind that most HDs have 3-year warranties except for Western Digital "retail-boxed" drives which only have a 1-year warranty (the "brown-boxed" or "OEM" ones do have 3-year warranties though, confusingly enough), and Seagates which have a 5-year warranty.
|||Not that it occurred to me.|
Thank You Mr. S. Alexander
I've just logged into my personal PayPal account (which I don't do very often these days), only to discover that on 20th July someone with the email address firstname.lastname@example.org  had sent me a payment of $46.10. The name vaguely rings a bell, but only vaguely, and I can't work out who it is.
It was actually sent to an old email address, which I no longer have access to but PayPal must still have it registered. That means I didn't get any notification of the payment .
So why am I blethering on about this to you ? Good question.
Well... the contact details for the payment (as well as the invalid email address) are :
Customer Service URL: http://www.sourceforge.net Customer Service Email: email@example.com
It looks to me like this payment might be a donation via sourceforge for one of my projects. The amount looks suspiciously like a donation of fifty dollars, minus the cut that sourceforge take . If that is the case, someone is probably annoyed that I didn't bother to say thank you.
So if you're out there Mr. S. Alexander (whoever you are), thank you.
|||Which now bounces...|
|||Oh yeah, and the sourceforge login is systems is currently borked (at least for me), so I can't go in and check/change things until they sort it out. Time to switch to Google Project Hosting.|
|||And a small percentage I set to go to the PSF if I remember correctly.|
IronPython & GUI Pain
Why is writing the GUI for an application only ten percent of the code, but ninety percent of the pain ? sigh
This post explains in excruciating detail the difficulties we had overcoming particular problems with Windows Forms. If you are interested in how to do slightly obscure things with Windows Forms and IronPython, then you might find this interesting. If you aren't... then you won't.
The first problem was to do with the way we have chosen to allow renaming of a particular control. We float a TextBox above the control. When you hit enter, or move focus from the TextBox, it validates what you have entered. It either displays a warning dialog or does the rename depending on how dumb the user input is.
Compared to what follows, positioning the textbox was a piece of cake.
In essence all we do is set the location of the TextBox and call its BringToFront method. Easy hey .
The minor problem is that we are floating it in front of a control that we can't directly add new child controls to: so we need to parent the textbox on the Form. The Location property of the control in question is relative to its parent control.
This means we have to convert the client location of the control it is covering into a screen point, and then back into a client point relative to the form. A little fiddly to work out - but it does the job well. PointToScreen on the parent control turns the client location into a screen point (effectively an absolute location). We then call PointToClient on the form to turn this screen point back into a client location relative to the form. Got all that ?
The code looks something like this (we massage the new location to position it a little more accurately) :
screenPoint = parentLocation.PointToScreen(parentLocation)
newLocation = mainForm.PointToClient(screenPoint)
newLocation = Point(newlocation.X + 5, newLocation.Y + 5)
textBox = TextBox()
textBox.Location = newLocation
textBox.Validating += onValidateFunct
It looks very nice, hooray for us.
Unfortunately we then discovered that there are a few places outside the textbox you can click that don't cause the Validating event to be raised.
One of our lead developers thinks that the only sensible counting system in this situation goes: one, two, infinity... As soon as we discovered two special cases, we should assume their will be infinity of them; and should look for a more general solution.
(It would also avoid us coupling the details of this code to the specific layout of other parts of our GUI.)
Our first idea was to put an invisible window over the whole GUI, and trap clicks on it. If the textbox validates then the click event can just not be handled, and fall through to whatever is below. If validation fails, then the dialog can be raised and focus restored to the textbox .
So we tried it with a Panel, with the BackColor set to Color.Transparent. Unfortunately this only fakes transparency (by taking on the background colour of the control below), and obliterates the GUI behind it. There is a workaround, but it is 'unpretty' in the extreme.
Form controls have an Opacity property, which allows you to make them completely see through. Very pretty. We would have to handle resizing and moving, but hey. Unfortunately, a 'special feature' of transparent windows is that clicks go straight through them.
From Form.TransparencyKey :
Any mouse actions, such as the click of the mouse, that are performed on the transparent areas of the form will be transferred to the windows below the transparent area.
At this point we discovered Application.AddMessageFilter message filter. This allows you to install a filter into the Windows Forms event loop, fondly known as the message pump for an application. Removing the filter is a corresponding call to Application.RemoveMessageFilter.
The filter must implement the IMessageFilter interface, and provide a PreFilterMessage method. This method should return True for messages you will handle yourself, and False for ones you want to be handled normally.
We figured that we could catch click messages for outside the TextBox, and trigger validation. If validation succeeds the message can continue its merry way, if it fails we drop the message and inform the user.
Luckily, implementing the IMessageFilter interface turned out to be as easy as inheriting from that class.
Messages are passed to PreFilterMessage by reference. In IronPython you get from the reference to the actual message using message.Value. The message has a Msg attribute telling you what type of message it is. These are defined in WinUser.h, which is a C/C++ (?) include file contained in the framework SDK. It has such wonderful passages as :
#define WM_LBUTTONDOWN 0x0201 #define WM_LBUTTONUP 0x0202 #define WM_LBUTTONDBLCLK 0x0203 #define WM_RBUTTONDOWN 0x0204 #define WM_RBUTTONUP 0x0205 #define WM_RBUTTONDBLCLK 0x0206
So, we probably want to investigate all messages where message.Value.Msg == 0x0201, or something like this.
The Message Structure also has various members that provide information about the event. In our case the relevant one is HWnd: the window handle of the control the event is for. This is available directory on our textbox (and other controls) using the Handle Property.
So our code to catch click events outside our textbox should have looked something like this :
def __init__(self, handle):
# Inheriting from a .NET object, no need to call
# the base class '__init__'. This wouldn't be true
# if we were overriding '__new__'.
self.handle = handle
def PreFilterMessage(self, messageReference):
message = messageReference.Value
if (message.Msg != WM_LBUTTONDOWN) or (message.HWnd == self.handle):
handle = textBox.Handle
filter = MessageFilter(handle)
# If validation succeeds, remove the filter
This brilliant plan had at least one flaw, (we never got far enough to see what complications popping up dialogs whilst in the middle of handling a message might cause). What if users wanted to use the main menu or context menu for copying or pasting in the textbox ? Our filter code would have been just as tightly coupled to the rest of the program (needing access to the menus) as any other approach, and probably a lot more complicated. sigh
At this point we just gave up and hunted down the few places where we needed to handle clicks that didn't cause validation in our textbox. It probably took around six lines of code.
The next problem was even worse. You know that if you have a TabControl with many tabs, Windows will put little scroll buttons by the tabs. This gives you access to the tabs that are out of the window, and it looks like this  :
Very nice. Have you ever tried to disable those scroll buttons ? In fact, have you ever tried to get Windows Forms to even acknowledge they exist ? We hunted up and down the call stack trying to find any trace of them. Most controls have a simple Enabled property that allows you to switch them off, but for all intents and purposes these scroll buttons just didn't seem to exist. We scoured the MSDN docs and the internet; all without any luck.
So we decided to try our good old friend the message filter. Using Spy++ (a tool that comes with Visual Studio) we could see the WM_HSCROLL messages, so we thought we'd just drop all of these. We couldn't find the handle of the scroll buttons, but no matter, just killing all scroll messages would work fine at this point.
Except guess what, it didn't. Despite seeing the messages quite straightforwardly in Spy++, they were just never arriving at the filter. damn
At this point we felt the heavy gloom of having to delve into the win32 API and switch the damn things off under the hood. (After all, Windows Forms is really just a wrapper over the native win32 stuff.)
So we googled around a bit and discovered that maybe something called WndProc was what we were looking for. Even better though, .NET provides an interface to it - WndProc. This is another part of the windows messaging system, but for the individual control: we needed to override it on our TabControl.
You can't do this on a straight TabControl (and we thought we might have to drop down to C#, which is still better than using unmanaged code), but you can subclass in IronPython and simply overload the method. Huzzah
WM_SCROLL = 0x114
# Again, no need to call the base class __init__
self.filter = False
def WndProc(self, msg):
if not self.filter or msg.Value.Msg != WM_SCROLL:
filteringTabControl = FilteringTabControl()
This behaves exactly like a normal TabControl, except that when you set the filter attribute to True it drops all scroll messages. (Unlike the application level message filter, we don't need to return anything here). If filter is False or it isn't a scroll message, then TabControl.WndProc is passed the message, and never need know we sneaked a look at it.
This may all seem fairly simple, but it took several days of head banging to get to this point.
I'll leave working out how we unit tested this as an exercise for the reader...
|||Jan is one of our two excellent interns this summer. He is also a Polish Ruby developer; we're overrun with them !|
|||The other GUI toolkits I've used.|
|||This incidentally is one reason why we can't use the LostFocus event on the textbox. Unsurprisingly you can't use LostFocus to restore focus to a control.|
The code used for the simple TabControl screenshot was :
from System.Windows.Forms import (Application, BorderStyle,
DockStyle, Form, Panel, TabPage, TabAlignment, TabControl)
form = Form()
panel = Panel()
panel.Dock = DockStyle.Fill
panel.BorderStyle = BorderStyle.Fixed3D
tabControl = TabControl()
tabControl.Dock = DockStyle.Fill
tabControl.Alignment = TabAlignment.Bottom
for i in range(20):
tabControl.SelectedIndex = 11
rest2web and Debian
Thanks to Martin Krafft there is now a debian package for rest2web.
The bad news is that it means I need to hack my way through enough docbook XML to put together a manpage. sigh
Numpy for Movpy
We've tried it, and it seems to work fine. If you'd like it you can download it from :
I still haven't written new documentation, I'll get to it soon. There is also a problem with the current release of pywin32 and Python 2.5b3 which is blocking a 2.5 Beta 3 release of Movable Python. It looks like checkins have been made to CVS since the last release, so I'm going to try compiling from source.
psyco for Python 2.5
Unfortunately the source distribution of psyco from the website doesn't work with Python 2.5.
Luckily the version from SVN does.
If you downloaded the installer for Windows, for Python 2.5, yesterday; you'll need to grab it again.
On the subject of Python 2.5, I've been reading about the changes to generators (using send they now implement coroutines etc) and the new with statement. I think I understand them now, at least the mechanics if not the full implications.
The What's New in Python 2.5 guide is very clearly written, I can recommend it.
MD5 for IronPython
PLY depends on the md5 module, which is a built-in module in CPython, but not yet included with IronPython.
Previously we were using an IronPython wrapper round the .NET cryptography MD5: md5.py by Seo Sanghyeon.
Unfortunately this was broken under IP 1.0 RC1.
Luckily the fix was very simple, we simply replaced the call to MD5.Create() with MD5CryptoServiceProvider().
# Updated by Resolver Systems: http://www.resolversystems.com
# WTF License :-)
# 2006-01-26 sanxiyn Created
from System.Security.Cryptography import MD5CryptoServiceProvider
from System.Text import Encoding
raw = Encoding.GetEncoding('iso-8859-1')
empty = raw.GetBytes('')
self.context = MD5CryptoServiceProvider()
def update(self, string):
bytes = raw.GetBytes(string)
self.context.TransformBlock(bytes, 0, bytes.Length, bytes, 0)
self.context.TransformFinalBlock(empty, 0, 0)
self.context.TransformFinalBlock(empty, 0, 0)
string = ['%02x' % byte for byte in self.context.Hash]
crypto = MD5Type()
The license is obviously the same WTF license that Seo released it under.
What baffled me was the error message from IronPython 1.0. It complained that multiple overloads matched the MD5.Create() call, which appeared to be pure fiction.
Also, if you look at the .NET documentation; it states that MD5 is an abstract base class, not intended to be used directly. I'm not sure how this ever worked...
|||The change to the GetExitCode API call in the IRonPython engine caused us a few moments of headscratching though.|
|||No warranty, but if there are problems I want to know ! It works for us, and passes our simple test (we generated a hash from the CPython md5 and checked it is the same).|
New Laptop - Searching for Storage
It actually arrived on Friday, but I left the power supply at work. doh
It works fine, commuting will never be the same again.
It does have a hard drive that is slightly smaller than I would normally cope with, around 40GB. As you might gather from my list of Essential Programs & Tools, I am used to a normal computing environment that requires a little more space.
So I'm looking for a portable USB or firewire hard drive that is self powered. An extra 60giga-bytes or so should be fine. (I won't really install all those programs, I promise.)
There are lots of cheap drives on ebay, from China or Korea or other far flung exotic locations. (Actually probably not that exotic, given the average working conditions in some of these places.)
Unfortunately I've been stung by cheap storage devices before, and below a certain limit you really do get what you pay for. I'm seriously considering one of these nifty Portable Hard Drives from Pexagon-Tech. They're stylish, reasonably priced, with reliable drives and some interesting features. (It reminds me of the Maxtor "One-Touch" system which has a good reputation. My only experience of these though, is having great fun smashing up a 350gig drive with a young friend; don't worry my Dad had already dropped and broken it.)
Their personal engraving service also looks intriguing. It identifies the drive as yours, and you can offer a reward if anyone 'finds' it. That could just be a life-saver someday.
Hmmm... Good job I didn't spend too much on the laptop.
|||Which tells me something, but probably only how inflated retail prices are...|
pyCrypto & psyco Windows Binaries
In a late spring clean I've just updated the pre-compiled binaries for both these modules.
There are now binaries available for Python 2.2, 2.3, 2.4 and Python 2.5. All for version 2.0.1 of pyCrypto.
The sourceforge site only offers Python 2.4 binaries for download. Here you can get binaries for Python 2.2, Python 2.3 and Python 2.5.
The Python 2.4 and 2.5 binaries are compiled using the Microsoft Optimising compiler.
If you have any problems then let me know. I can't promise to debug the code, but if there are problems due to compilation then I may be able to sort them out. Enjoy !
|||Done by following the instructions at Building Python Extensions Using MingW32.|
rest2web 0.5.0 Beta 1 Finally Out
After five months there is finally a new release of rest2web. This is 0.5.0 Beta 1.
There are an awful lot of changes and improvements, but there shouldn't be any backward compatibility issues with previous versions.
One of the best changes, is the addition of a new Quickstart Guide by Andrew Ittner.
What is rest2web ?
rest2web is a tool that generates html files for websites, parts of websites, or project documentation. It allows you to keep your page content in ReST format, and has a simple but flexible templating system.
It can automatically build index pages, 'breadcrumbs' and sidebars for your site. By providing access to a data-structures that represents your site (for example indextree and thispage), you can easily build in extra functionality.
What's New in 0.5.0 ?
The full changelog is below. As you can see a lot has changed. I will try to summarise the most important changes first :
- Addition of 'Quickstart Guide'
- rest2web can now build a site from just a directory of ReST documents; no need to supply restindexes or index pages if you don't want to
- Global uservalues (effectively variables available in every page) can be set in the Config File or at the command line
- Lots of new command line options available, including setting verbosity levels (make rest2web quieter !)
- New Standard Functions, especially 'include' for better customisation of templates (e.g. for defining a different footer for some directories)
- More information available in the namespace and indextree (source filename, target filename, full url etc)
Plus a host of bugfixes and minor improvements.
Version 0.5.0 Beta 1 2006/08/06
Moved 'pythonutils' distribution into the 'rest2web' directory for ease of packaging.
Added a #! line to r2w.py.
All rest2web imports now absolute imports.
Added 'quickstart.txt' thanks to Andrew Ittner.
Added an include standard function, this can be used to nest templates or customise sections. (It will walk up the directory tree looking for the file you specify and takes an optional argument if the file doesn't exist - useful for templates that allow subdirectories to add to the template, or even wrap the body.)
make_dist.py now takes nopause as a command line argument (make_dist.py is in Subversion for creating distributions.)
Default breadcrumb divider is now '>'. Breadcrumbs are also output in HTML on separate lines for readability.
Fixed bug when final_encoding is None.
Default config file is now called r2w.ini. rest2web.ini will still be supported until the next release.
Fixed bug with standerr where no logfile is used.
print_crumbs can now take None for the dividers.
Added 'globalValues' to the namespace. (Available in templates and pages for storing values which can be accessed across all pages.)
Added 'uservalues' and 'restindex' into each page in the indextree.
A new command line 'nopause' option to override the config file.
Change so that variables (and functions etc) defined in templates can be used in single code blocks (like <% new_name %>).
Added more information about pages to the namespace and indextree. The new values are :
'source_file': The source file for the page
- 'current_dir': The current directory being processed - this can be turned
- into an absolute filepath by doing os.path.join(os.getcwd(), current_dir)
- 'target_dir': The target file directory (as an absolute file path) being
- rendered into. Note if the file has a target specified it may not be put in this directory. Use os.path.dirname(target_file) instead.
'full_page_url': The full url (starting with '/') for the current page
'target_file': The full filename of the page being rendered
Fixed bug where 'thispage' wasn't set on pages in the indextree. (Value should be True for the current page.)
Fixed bug where 'thispage' (in the namespace) would sometimes be incorrectly None.
Cached template files for faster execution.
Special thanks to Martin Krafft for bugfixes and suggestions.
Version 0.5.0 alpha 2006/05/01
rest2web can now build a site with no index pages, no template and no restindexes. This is the force command line option. It can be used to automatically build a site from a collection of ReST documents, and use default templates.
uservalues can be passed at the command line and in the config file. (Command line uservalues override config file ones.) These uservalues are now available in every page. The encoding of uservalues in the config file is specified by the __encoding__ value.
A --template-file (or -t) command line option. (Will override the top level template keyword specified in the restindex.) This allows you to have alternative templates for a site; for example one for an online version and another for distributed documentation.
New website template, created by Fuchsiashock Design.
final_encoding should never be utf8 - should be utf-8 instead. This is because utf8 is not recognised by browsers. (This is now automatically handled.)
Added initialheaderlevel a new restindex keyword. It sets the size of headers used in ReST documents. Can be set per page.
The file keyword has been bugfixed. It now only operates if the target file doesn't exist or is different to the source file. It copies the timestamp along with the file.
The gallery plugin now ignores non-image files. It also skips image files it can't handle (currently only animated jpgs.
rest2web now has three levels of verbosity, controlled from the command line :
- -v : Verbose, the default.
- -a : Warnings and actions.
- -w : Warnings only.
uservalues can now be inserted in pages using a new syntax. Where this is used, the uservalues are inserted before the page is rendered from ReST to HTML. This means uservalues can be in ReST format. The syntax for single values is <* ... *>. For multiple lines of code it is <$ ... $>.
Added modtimeiso value to the namespace and the formattime standard function.
The namespace and uservalues are both now available (as dictionaries) to the macros and the standard functions.
Removed the two <br /> from listend in the standard function minibar and added wrapper_class to print_details.
Added os and sys to the namespace for every page.
The default crumb for index pages (if no page-title specified) is the filename, minus the extension and turned to title case.
Removed urlpath from rest2web, because it is now in pythonutils.
It won't run in the distribution directory - need to run "make_dist.py". (This only applies if fetched from subversion).
This work is licensed under a Creative Commons Attribution-Share Alike 2.0 License.