Python Programming, news on the Voidspace Python Projects and all things techie.
Movable Python 1.0.0 for Python 2.3 & 2.4
It's finally happened, Movable Python is finally released.
Versions for Python 2.3 & 2.4 are available from The Movable Python Shop.
The cost is £5 per distribution, payment by PayPal. £1 from every distribution goes to support the development of SPE, the Python IDE.
This gives you free support (via the Movable Python Mailing List), and access to updates, for a year.
There is a minor issue with the Python 2.2 version. This will be fixed and released in the next few days. In the meantime the previous version (0.4.6) is available for download from the shop. If you purchase this distribution your year will run from the date of the release of 1.0.0.
Paying for Movable Python supports the development of the other Open Source Voidspace Python Projects.
What is Movable Python ?
Movable Python is a distribution of Python for Windows that doesn't need to be installed. It easily fits onto a USB memory stick. Python on a stick.
It is integrated with SPE, the Python IDE, to make Movable Python a portable Build, Test, and Run environment. It has a nice GUI to launch programs and control its behaviour.
Movable Python is useful in the following situations:
- Machines where you can't install programs.
- Where you need a portable 'Build, Test, and Run' Python environment.
- Having several versions of Python on the same machine for forward/backward compatibility testing.
- Easily deploying Python scripts without having to install Python.
- Try before you buy - test Python without having to install it, including new versions .
- 'Python Runtime Environment'. '.py' files can be associated with movpy.
For more information, see An Introduction to Movable Python.
What's New ?
Changes since version 0.4.6.
Completely new method for running files. This means you can launch multiple programs from the GUI.
Lots of new libraries and tools included. Virtually all the included tools have had version upgrades since the last release.
The GUI does a lot more :
- Version number and Python version displayed on the GUI
- You can pass arguments to your programs
- It remembers the last directory you ran a program from
- You choose if launched programs have a console box or not
- You can configure the options programs are run with
- You can edit the default options
- Four configurable Quick Launch buttons
- You can launch SPE from the GUI
- You can launch the documentation (About)
- You can launch an interpreter console
- You can close the GUI without having to launch a program or go to the interpreter
The following (command line) options are new :
- f - run script in it's directory
- b - pause after running script
- o - override default options
- die - get rid of GUI after running
- k - run with console from movpyw
- koff - run without console from movpy
Complete documentation rewrite.
Docs are now built with rest2web.
movpyw.exe is now included in the Python 2.2 distribution.
Bug fixed where first command line argument to your program was always lost.
Bug fixed so that IPOFF actually works now.
Bug fixed so that -p in config.txt now works.
Lots of other minor changes and improvements.
There are some (minor) known issues with version 1.0.0. These will be fixed soon.
- Version for Python 2.2 not yet available.
- Possible issue running movpy.exe from long file paths on Windows 98.
- The lib directory is not added to sys.path before entering interactive mode.
There are also several items in the TODO list. Many of these are future ways that Movable Python could be developed.
Python on Nokia
I'm sure you've seen this before, but Nokia have Open Sourced their distribution of Python for S60 Phones.
My general feeling, like a lot of Python programmers I guess, is that this is a good thing for Python; but not necessarily something that I'm going to get into myself.
I need a new phone, so did I toy with the idea of getting an S60. I didn't really want to spend a lot on a phone (like cars, the market is awash with good second hand ones), and I couldn't really think of anything I wanted to code for a mobile phone.
Anyway, I really struggled to find an S60 phone. I tried ebay, phone shops and googled around to no avail. Maybe they're not available in the UK.
Finally the penny dropped (duh). S60 is a platform, not a model. It's a Symbian OS based platform as it goes.
For anyone else in the same boat (their must be someone out there as dopey as me), you can get a list of all S60 based phones at the Nokia S60 Phone Matrix.
It just so happens that the Nokia 3230 I'd decided to buy from a friend, is S60 based anyway. Shucks.
It looks like I'll have to get into it, as if I didn't have enough to do.
There is a great tutorial on Python for the S60. They expose the phone API on a nice high level, including not just the camera and SMS system, but also the GUI and input devices. This means Python will be a great RAD tool for real developers, as well as putting useful applications within reach of the tinkerers. Now if only I could think of something interesting to do with it.
ConfigObj Unicode Support
The new version of ConfigObj in SVN contains (optional) Unicode support. This was relatively easy to add :
- decode on the way in (possibly handling a BOM)
- encode on the way out
There were only a couple of cases where ConfigObj converts values to strings (like integers) where I had to make sure we didn't coerce Unicode strings into byte-strings (and risk a UnicodeEncodeError).
I'd really appreciate it if you could check this out and try and break it. Unicode is notoriously fiddly, and I'd like to get it basically right before doing the Beta release.
There is still one tricky question left to answer. Below is a description of how ConfigObj supports Unicode, and the remaining question.
The unicode changes are reasonably straightforward. This was the system used in ConfigObj 3.
You can pass in an (optional) 'encoding' keyword. If present, this is used to decode the input. This maps to an 'encoding' attribute which is used to encode when writing (and can obviously be changed).
There is also a 'default_encoding' keyword/attribute to override the system default encoding. This is used (when writing) to decode any values that are byte-strings. These need to be turned into unicode prior to writing out in the specified output encoding.
If encoding is None (the default) then no encoding or decoding is explicitly done. This means the default behaviour for ConfigObj is basically unchanged. The exception is when a UTF8 or UTF16 file is supplied, with BOM.
The current situation, is that if you supply a UTF8 or UTF16 file with a BOM, then ConfigObj will autodetect the encoding and decode appropriately.
For UTF16 this is essential, ConfigObj will mangle a byte string with a 16bit encoding unless it decodes first.
For UTF8 this is possibly not what you want, you might want to keep UTF8 byte-strings. (Some applications attempt to avoid unicode issues by retaining UTF8 encoded byte strings throughout.) You can get round this by parsing the config file, detecting that the encoding attribute has been set, and then calling encode.
cfg = ConfigObj(cfg_file)
if cfg.encoding == 'utf_8':
cfg.encoding = None
Is this adequate, or should the default behaviour for ConfigObj remain ignoring the encoding (in the case of UTF8) except to preserve the BOM ?
It's also worth noting, if you read a file with a BOM, and then change the encoding - you must remember to set cfg.BOM = None. Would it be more sensible for cfg.BOM just to be True or False ? If True ConfigObj would write the appropriate BOM (if one is appropriate at all).
Subversion & New ConfigObj Methods
I've made a start on changes to rest2web, they will be completed shortly and I'll let you know. For the moment I can't guarantee that the code in SVN works. This state won't last very long though.
The main change is that I've checked in a new version of ConfigObj 4.2.0 Beta 1.
This adds full unicode support and a few simple new section methods.
Easy things first. The new methods provide the same convenience features as the get* methods of ConfigParser.
The new methods are :
istrue is now deprecated and will disappear eventually.
The new methods are all section methods. This means that to retrieve a key value as a boolean (for example), you call the as_bool method on the section the key is contained in.
As a simple example, suppose we have the config file :
key1 = 3 [section] key2 = True
We want to retrieve key1 as an integer, and key2 as a boolean.
cfg = ConfigObj(cfg_file)
key1 = cfg.as_int('key1')
section = cfg['section']
key2 = section.as_bool('key2')
# Alternatively, we could do
key2 = cfg['section'].as_bool('key2')
For details of unicode support, read the next post.
ConfigObj, The Standard Library & ConfigParser
This could address some of the perceived shortcomings in ConfigParser, as outlined in the ConfigParserShootout.
There was a brief discussion on the subject, in which Guido Van Rossum gave his opinions.
First of all, he chastened me for using Fuzzyman in my emails rather than my name. Hey, at least he knows I exist.
The upshot of it was that he didn't really see why people had a problem with ConfigParser, and he would rather see the limitations of that overcome. He especially didn't like the nested subsections that most of the implementations on the ShootOut allow. In fact this is the single biggest limitation of ConfigParser that most of the alternative modules address.
Guido feels that this encourages developers to inappropriately mix configuration options with application data. (Looks like he's been reading some Joel Spolsky on UI Design.)
For application data persistence (and presumably complex configuration) Guido recommends an XML type solution.
In fact one of the driving motivations of ConfigObj was for a firm with complex network applications. They used ZConfig for the configuration, but wanted to stop forcing their system administrators from having to hand create (and read) XML.
Their config files configure multiple services and plugins and are nested a few levels deep with names, IP addresses, plugin components etc. The ConfigObj file format makes it easy to read these files, and visually see the structure of the configuration data. The configspec schema can automatically validate the config files, and facilitate the automatic creation of new sections with sensible default values.
Despite the no from our BDFL, it was a useful conversation.
He had interesting things to say about ConfigObj. (Ian Bicking was vaguely positive as well, quite overwhelming debating with all these luminaries.)
Guido favours a narrow API over a wide one, and he feels that subclassing dict provides an overwide API (methods that aren't appropriate). He doesn't disagree that the mapping protocol is an intuitive way to access config file members though.
He also indicated the sorts of things he would like to see (or would accept) as improvements to ConfigParser :
- Mapping protocol access for simpler (and better) access
- Better ways of supplying default values for config files
- Round-tripping: writing values back into a file in the order they appear, possibly preserving comments
- Supplying values from several files, and tracking which file changes apply to
ConfigObj already does all of these except the last one. It also has list values, which were mentioned but Guido didn't comment on them. They are extremely useful. ConfigObj also has Unicode support, which is missing from ConfigParser.
Really, since when has ConfigObj had unicode support ? Well read on...
ConfigObj Changes Imminent
I'm working on some changes to ConfigObj, that add new functionality. This adds some of the get type methods from ConfigParser, as well as full (but simple) unicode support.
Both are relatively easy and will become ConfigObj 4.2.0. (I may well do a beta release first.)
There is a question about the method signature for the get methods. The ConfigParser methods are :
get( section, option)
Get an option value for the named section.
getint( section, option)
A convenience method which coerces the option in the specified section to an integer.
getfloat( section, option)
A convenience method which coerces the option in the specified section to a floating point number.
getboolean( section, option)
A convenience method which coerces the option in the specified section to a Boolean value. Note that the accepted values for the option are "1", "yes", "true", and "on", which cause this method to return True, and "0", "no", "false", and "off", which cause it to return False. These string values are checked in a case-insensitive manner. Any other value will cause it to raise ValueError.
I won't implement get because it clashes with the dictionary method. If I implement the ConfigParser compatibility layer then that will change (but be on a subclass).
My (current)intention is to make these methods section methods, so they are available on all subsections, and to keep the ConfigParser method signature. For values not in a subsection, this leads to the slightly awkward syntax :
cfg = ConfigObj(cfg_file) val = cfg.getboolean(None, 'key')
The alternatives are :
- To reverse the order of the arguments, with section defaulting to None. This leads to confusion in the (theoretical) ConfigParser compatibility layer, which will of course be reversed back. It is slightly more friendly for normal use though.
- To make it a method on ConfigObj (not sections) and allow section to be a string or a reference to a subsection - in which case the specified key would be taken from the supplied section.
Should I then remove istrue (added in 4.1.0) which is effectively the same as getboolean ? I would deprecate it and add a warning until removing it in 4.3.0.
The unicode changes are reasonably straightforward. I'll re-implement the system used in ConfigObj 3. (Unless anyone raises reasonable objections or suggests a better alternative).
You will be able to pass in an (optional) 'encoding' keyword. If present, this will be used to decode the input. This maps to an 'encoding' attribute which will be used to encode when writing (and can obviously be changed).
There will also be a 'default_encoding' keyword/attribute to override the system default encoding. This is used (on writing) to decode any values that are byte-strings. These need to be turned into unicode prior to writing out in the specified output encoding. You can enforce unicode by setting this to ascii.
If encoding is None (the default) then no encoding or decoding is explicitly done. This means the default behaviour for ConfigObj is completely unchanged.
There has been a response to my suggestion on python-dev. This is from Vinay Sajip, the author of another Python config module. His module has some additional features to ConfigObj (but not all the features of ConfigObj), at the cost of extra complexity and no possibility of compatibility with ConfigParser. (All IMHO - I'm sure his module is excellent and have no desire to start a flame-war on the subject.)
He would like to see more debate on the subject, and a clear winner picked from the (several) implementations listed on the ConfigParser Shootout.
Whilst ConfigObj is still a good contender, it could supplement ConfigParser without having to add an incompatible API to the standard library. This is an alternative that bypasses the tortuous and unlikely process of reaching a consensus on python-dev.
Awstats, Jobs & Other Things
I found a nice tutorial which made it very easy.
The only tutorial I could find last time I looked, was actually configuring awstats for a Ruby on Rails application. It included details on installing RoR and setting up Apache. None of these steps were relevant, and I couldn't figure it out.
I also needed to integrate the update process with logrotate, but the awstats docs cover that quite nicely. I won't be able to confirm if that has worked until tomorrow.
Updating awstats via the logrotate daily cron job works fine.
The new stats have led me to discover that lots of people are deep-linking to my images. In the past turning off hotlinking has harmed browsers who turn off their referrer for privacy reasons. As I'm not concerned about a bit of bandwidth loss (yet...) I probably won't do anything about it.
Things I'd like to do include :
- Update ConfigObj to support unicode and a few ConfigParser style get methods.
- Help migrate content to the new-style Python Website.
- Release Movable Python.
- Fix the concurrency issue in downman.
- Add next & previous links to rest2web, then add support for full contents pages.
- Breakout the SSI processing from CGIHTTPServerWithSSI as a separate module. It also needs to do redirects to add a trailing slash if a directory is fetched.
- A few bugfixes on Firedrop2.
- Build a proper entry administration interface to the Voidspace Guestbook.
- Add access to deeper searches in skimpy, and release that and my contact form script.
I'm also tempted to build a digital download manager that is integrated with PayPal. It should support 'subscriptions' (accounts valid for a year), so that I can use it to distribute Movable Python.
Some of these won't take much time, others will.
This list is here mainly to remind myself, but also to give you an exciting sneak preview of things coming up.
A few trivial pieces of info about changes I've just made to Voidspace.
I've removed the Tektonic Affiliate Links from the site.
They offer a great service (cheap virtual server hosting). Despite generating around 500 click-thrus, I hadn't earned a penny through the affiliate scheme.
I've changed to using google for the site search.
Adsense does work for generating income from the site. It also saves me having to remember to re-crawl the site when I add content. Inevitably the google search lags slightly behind updates though.
I'm getting Contact Form spam. WTF
It's only occasional, but it surprised the heck out of me.
I'm considering switching to Google Analytics for traffic measuring.
Anyway, it's been an interesting year. You can see below the traffic for the whole site (top image), and for just the Python section (second image).
Around April I moved and removed a lot of outdated content. Traffic dropped considerably. This has been replaced by new visitors to my Python section, which has grown and grown. I need to complete the revamp of the rest of the site, and hopefully bring back some of the visitors I lost.
This work is licensed under a Creative Commons Attribution-Share Alike 2.0 License.