Python Programming, news on the Voidspace Python Projects and all things techie.

URLError & urllib2 the Missing Manual

emoticon:film I've just discovered an error (now corrected) in my popular article urllib2 - The Missing Manual. Embarassed

Part of the code shows how to handle errors, when fetching the web resource you requested fails for whatever reason.

One approach I suggested was :

from urllib2 import Request, urlopen, URLError, HTTPError
req = Request(someurl)
    handle = urlopen(req)
except URLError, e:
    print 'We failed to reach a server.'
    print 'Reason: ', e.reason
except HTTPError, e:
    print 'The server couldn\'t fulfill the request.'
    print 'Error code: ', e.code
    # everything is fine

This won't work, can you see why not ?

HTTPError is a subclass of URLError. That means that if urlopen raises an HTTPError it will be trapped by the first except statement. Your code will bomb out because the resulting exception object doesn't have a reason attribute.

The correct code is :

from urllib2 import Request, urlopen, URLError, HTTPError
req = Request(someurl)
    handle = urlopen(req)
except HTTPError, e:
    print 'The server couldn\'t fulfill the request.'
    print 'Error code: ', e.code
except URLError, e:
    print 'We failed to reach a server.'
    print 'Reason: ', e.reason
    # everything is fine

No wonder I preferred the second approach in that section. Anyway, my apologies. Smile

Like this post? Digg it or it.

Posted by Fuzzyman on 2006-03-24 11:39:19 | |

Categories: ,

The GPL License

emoticon:cross I don't like the GNU GPL license. If you release code that uses any GPL licensed code (or even just optionally links to any GPL code, even if it's not included in your distribution), then you are obliged to license your code with the GPL.

I think there are basically three reasons why people use the GPL, and I think two of them are valid.

  1. Ignorance
  2. Commercial
  3. Ideological


The GNU crowd have done such a good propaganda job, that some people think that Open Source means GPL. They release code under the GPL because it's the only Open Source license they are familiar with.

If you release your code under the GPL, then anyone who is working on code that will be made available commercially can't use your code. Because of legal ambiguity, many firms won't allow the use of GPL code even on internal projects that will never be released [1].

This means that less people can use your code, and I think that harms Python. We want to see high quality libraries and frameworks that encourage the use of Python in commercial environments.

Commercial Reasons

If you are working on a commercial project, you may find that part of your code could be neatly packaged as a library that may be useful to others. If this is a significant chunk of your work, you may not want to hand this over to your competitors.

In these circumstances, the GPL allows other Open Source projects to use your code; but not the competition. Unless your competition is free of course. Laughing

Ideological Reasons

Part of the ideology behind the GPL, is that software ought to be free. Instead you should earn a living offering consultancy and other software related services.

I personally think this is wrong-headed. I like free software, but I appreciate that it can take an enormous amount of effort to create and maintain complicated applications.

The bottom line is that this model isn't appropriate in all cases. I can understand people disagreeing with me though, this is especially true in individual instances.

The Alternatives

The license I use is the BSD License [2]. This requires you to include my license in your distribution, but pretty much allows anything else. (So long as you don't blame me or claim I gave you permission to do it.)

An even more liberal license is the the MIT License. This doesn't require your users maintain the original license restrictions if they relicense.

I have seen the Creative Commons by Attribution license used with code. This requires anyone who distributes your code to attribute the work to you. I like the intention, but the wording of the license is not directly applicable to program code.

Dual Licensing & Relicensing

This is sometimes a point of confusion, so it's worth a mention. A license does not indelibly mark a project with the terms of that license. A license specifies the terms you make code available under, but (if you are the sole copyright holder) you are free to relicense, or even retract the code altogether, any time you want.

If there are several contributors (more than one copyright holder) then you need permission from all of them, unless the license permits re-licensing anyway.

This allows firms to offer a GPL license for their code, with commercial licenses available for a fee.

It also means that if your project is currently GPL licensed, it's not too late to rectify. Wink

[1]Probably an unnecessary restriction, but you never know when you might want to distribute part of a project that started life as 'internal only' code. If you know that your code will never be distributed, then you are free to use and modify GPL code without licensing your code at all; let alone making it GPL. Licensing only comes into play when other people start using your code.
[2]Apparently called the Modified BSD License, because the original was flawed.

Like this post? Digg it or it.

Posted by Fuzzyman on 2006-03-24 10:29:02 | |



emoticon:note Last night I tried to replicate the reported problems with SQLObject and Movable Python. I failed, as far as I can tell SQLObject works fine with Movable Python.

I tested with Movable Python 1.0.1 for Python 2.4.2, SQLObject 0.7, Pysqlite 2.1 and FormEncode.

If you would like this bundle of packages for Movable Python, you can download it as a single Zip file :

SQLObject Files (593k) [1]

I then ran the following test code from the SQLObject examples :

from sqlobject import *

sqlhub.processConnection = connectionForURI('sqlite:/:memory:')

class Person(SQLObject):
    fname = StringCol()
    mi = StringCol(length=1, default=None)
    lname = StringCol()

p = Person(fname="John", lname="Doe")
print p.fname
p.mi = 'Q'
p2 = Person.get(1)
print p2
print p is p2

It prints the following (correct) output :

<Person 1 fname='John' mi='Q' lname='Doe'>

Hopefully the user will find his problems evaporate, or he'll report back. Smile

[1]This contains three directories. sqlite contains a binary specifically for Python 2.4. Just drop these directories into your lib/ directory and you're away. Smile

Like this post? Digg it or it.

Posted by Fuzzyman on 2006-03-24 10:07:22 | |

Categories: ,

Back to the __future__

emoticon:pill I recently blogged about supporting future statements in Movable Python.

When you compile Python code into a code object, in order to pass it to exec, you can specify which future statements it should be compiled with. I was worried that because I wasn't doing this, from __future__ import ... might be being ignored in user scripts.

I have just tested this with Movable Python for Python 2.4, and the future statements are honoured :

from __future__ import division
print 1/3

There is a bug in Python 2.4 which allows future statements after other statements, so I need to check with other versions of Movable Python. If it works, I'll stop worrying. Very Happy

Like this post? Digg it or it.

Posted by Fuzzyman on 2006-03-23 12:03:37 | |

Categories: ,

Web Apps on a Stick

emoticon:speaker A user reports having cherrypy and pysqlite working with Movable Python.

There is currently an issue with SQLObject, but reports on the py2exe mailing list indicate that this should be easily resolvable.


It turns out that the problem only occurs from the IPython shell.

SQLObject works fine when used within an application.

Like this post? Digg it or it.

Posted by Fuzzyman on 2006-03-23 12:02:37 | |

Categories: ,

Duck Typing & Containers

emoticon:acrobat I have ranted before about duck typing in Python.

Most of the time it's great, but because of the extremely loose definition of the mapping and sequence types, you can get unstuck if you ever need to know whether an object is a mapping type or a sequence type container [1].

You can't easily tell the difference, because any object that has a __getitem__ method [2] could be either.

I've come up with a quick hack to tell them apart. Smile

It relies on the fact that if you index a sequence with anything other than an integer, you get a TypeError. If you index a mapping type object (e.g. a dictionary) with something that isn't contained in it, you get a KeyError.

My first approach used a complex number as the index :

>>> a_list = []
>>> a_dict = {}
>>> index = (0+ 6j)
>>> a_list[index]
Traceback (most recent call last):
  File "<input>", line 1, in ?
TypeError: list indices must be integers
>>> a_dict[index]
Traceback (most recent call last):
  File "<input>", line 1, in ?
KeyError: 6j

The problem with this, is that a complex number can be a dictionary key. If the index (0+ 6j) does return a value, then it was obviously a mapping type container. Smile

The following hack is an alternative approach. It creates an unhashable object that raises a KeyError if you use an instance to index a mapping object.

If you need to tell the difference between a mapping and a sequence container, index it with our FakeIndex. If the error raised is a TypeError then it is a sequence [3], if it is a mapping container then a KeyError will be raised.

class FakeIndex(object):
    def __hash__(self):
        raise KeyError('Unhashable')

index = FakeIndex()

except TypeError:
    # It's a sequence
except KeyError:
    # It's a mapping

This only works if the mapping type container uses hashing under the hood for testing membership. It may be safer to just test for indexing a sequence. If that fails, then you can assume the object is a mapping type container.

[1]We had this problem in ConfigObj where we needed to be able to tell the difference, and to separate out the API would have been much more inconvenient for the user.
[2]And by the principles of duck typing it needn't implement anything more.
[3]You should first check that the object defines __getitem__, if you pass this code any other type of object you are likely to get TypeError: unsubscriptable object - a false hit for a sequence type.

Like this post? Digg it or it.

Posted by Fuzzyman on 2006-03-22 12:02:37 | |

Categories: ,

ConfigObj 4.3.0alpha

emoticon:ir_scope The Pythonutils SVN Repository now contains an alpha version of ConfigObj. This will become ConfigObj 4.3.0.

I hadn't intended to move so quickly from 4.2.0 to 4.3.0, but some interesting discussions on the configobj-develop list have led to some fairly major feature enhancements.

These are all implemented and tested, but they do raise a couple of issues and as usual I'd like to see people kicking the code before I do a new release.

The new features include :

  • Support for a copy mode in validation, allowing you to create a default config file with comments from your configspec.
  • Support for reading empty values (key = and key = # comment) as an empty string.
  • A new write_empty_values option/attribute that will write the empty string as empty values.
  • A new unrepr mode. Explained below.

I have also discovered and fixed three bugs [1] :

  • Line terminators were incorrectly written on windoze. Embarassed
  • Last occurring comment line could be interpreted as the final comment if the last line wasn't terminated.
  • Nested list values would be flattened when write was called. Now sub-lists have a string representation written instead.

Finally the encode and decode methods have been deprecated. I didn't deprecate rename as it should be used with walk to transform key and section names. encode and decode are still good examples of this by the way. [2]

copy mode for validate

Thanks to Louis Cordier for this suggestion. When you supply a configspec you can specify default values to use for any that are missing from the config file.

Normally when you call the validate method it marks any missing values as having been supplied from the default. If you then call write it doesn't write out any values supplied from defaults (unless you change them). This allows you to keep a config file with only values that are different from the defaults.

The validate method now takes an optional copy keyword argument. If this is True, then none of the values are marked as default. In addition all the comments are copied across from the configspec, as are the indentation and encoding settings.

A call to write will then write out a new config file with all the default values specified in the configspec. Smile

This does however lead to me realise that there is currently no way to specify the encoding of a configspec file. Confused


ConfigObj 4.3.0 alpha 2 now allows you to pass in a ConfigObj instance as your configspec. This is not a total solution, but does allow you to read the configspec using ConfigObj, and specify the encoding when you do that.

In order to read a configspec file, you must use the option list_values=False.

unrepr mode

This was suggested by Kevin Dangoor the lead developer of Turbogears [3]. It mimics the unrepr mode of the CherryPy Config Module, and in fact uses the code from

If you pass in the unrepr=True keyword when you create your ConfigObj instance, every value is parsed using unrepr. This allows you to have the following types as values :

integers, strings, True, False, None, floats, complex numbers, lists, dictionaries, tuples

You can use multiline values with unrepr. Smile

When writing in unrepr mode, the built in function repr is used to turn every value into a string.

Note that in unrepr mode :

The value 3.0 is a float
The value "3.0" is a string

Quote marks take on the same significance as they have in Python syntax, which is different from normal ConfigObj syntax.

[1]But no longer useful since full unicode support was added.
[2]validate also now honours the order of your configspec. Before copy mode, this didn't really matter though.
[3]Looks like ConfigObj won't be used in Turbogears until version 1.1 when a lot of things change.

Like this post? Digg it or it.

Posted by Fuzzyman on 2006-03-20 12:48:44 | |

Categories: ,


emoticon:info Well... the unthinkable has just happened. Someone has offered me a job programming in Python. Very Happy

Once I have worked my notice at my current job, I will start work with ResolverSystems. They are a small team, all seemingly good guys [1], working on something new using some radical programming techniques.

I'm a bit blown away by the news, so I think I'll shut up now.

[1]Time will tell. Wink

Like this post? Digg it or it.

Posted by Fuzzyman on 2006-03-20 11:12:10 | |

Categories: ,

Movable Python & __future__

emoticon:carrot Movable Python supports running both Python scripts and .pyc bytecode files. It does this by compiling scripts to bytecode, or extracting the code object from bytecode files, and then calling exec.

When you call compile you can pass in an optional flags argument which tell Python which future statements to compile the code with. I currently don't do this, which means that from __future__ import ... statements are ignored.

For Python 2.4.2 the most significant of these is division. Code which expects integer division to yield floats is just plain broken if this future statement is ignored.

I'd like to fix this in the next release of Movable Python.

First of all, I assume that the code objects contained in bytecode files are already compiled with the relevant future statements. This would mean that no further action is necessary. Can anyone confirm if this is correct ? [1]

For Python scripts there are a couple of possible approaches.

The simplest approach would be to use a simple regular expression to find statements that look like from __future__ .... Unfortunately this could score false positives for comments and docstrings. [2]

Another alternative would be to parse the code into an abstract syntax tree using the compiler module. I would then have to recognise the import statements in the nodes. I can see that there is an Import node does anyone have any hints for me with this approach ? I would prefer a solution that works across Python versions 2.2 to 2.5 and beyond.

Ominously, the compile.compile function has unsupported options for passing in flags, so it looks like it doesn't handle it automatically from the parse tree.

[1]It will actually be easy enough to test.
[2]Future statements shuld only occur at the toplevel of code, and in fact ought to occur before any other statements (although that was broken in Python 2.4), so it ought to be possible to minimize false positives a bit.

Like this post? Digg it or it.

Posted by Fuzzyman on 2006-03-20 10:21:58 | |

Categories: ,

Movable Python & Consoles

emoticon:cross With the forthcoming releases of Python 2.4.3 and 2.5 alpha 1, it will soon be time to do a new Movable Python release.

The changes (mainly minor at this stage) will include the following :

  1. A fix to allow you to do things like movpy - pylab. This will make it easier to use matplotlib with Movable Python. [1]
  2. A fix so that imp.find_module works.
  3. Support for __future__ imports in code.
  4. A change to the way that consoles are launched.

I will discuss item 3 in the next entry, here I will discuss item 4.

Currently Movable Python lets you decide whether a script launched from the GUI will have a console box or not. By default scripts launched from movpyw.exe don't have a console, and scripts launched from movpy.exe do. You can override this using the console checkbox.

I'm considering changing this to following the way normal python behaves. Scripts ending with .py will have a console, scripts ending with .pyw won't. Does anyone object to this change ?

It does make it slightly more complicated, as you will no longer have a single checkbox to dictate whether a script launched from the GUI will have a console or not. I currently find it quite useful to be able to easily specify whether or not a console box will be used, so I'd be interested in any ideas as to how that could be maintained.

My current best idea involves three checkboxes. One to toggle default behaviour on and off and two to reverse defualt behaviour for .py and .pyw files.

[1]It already works, but currently you need and do movpy -pylab -ipythondir lib/. See this entry for details.

Like this post? Digg it or it.

Posted by Fuzzyman on 2006-03-20 09:50:39 | |

Categories: ,

Movable Python Review - in English

emoticon:movpy2 Now that there is a new issue of the c't Magazine available, I can publish an English translation of the Movable Python review.

This translation was very kindly provided by Peter Otten. Many Thanks.

Python abroad

The space saving but plentiful Python distribution Movable Python is ready. It is intended as a complete environment that you can take with you from one Windows PC to another Windows PC on a USB-Stick, for example. The uncompressed package requires only 60 MB. Without the provided libraries for programming graphical user interfaces 15 MB will suffice.

Movable Python will not interfere with other Python installations. You don't have to store anything on the hard disk, and running the distribution doesn't require administrator rights. The three versions of the distribution support Python versions 2.2, 2.3, and 2.4. With these you can check programs for backward compatibility quite easily.

Movable Python welcomes the user with a small window that lets you invoke various functions, like executing a Python script on the hard disk or the USB stick or starting one of the provided development environments, SPE, IDLE, or IPython. Four freely configurable buttons allow fast access to scripts.

Your own scripts and the provided applications can make use of an extensive collection of included libraries, like the Python accelerator psyco, the document manager firedrop2, pycrypto for encryption, pyenchant for spellchecking, or PIL for image manipulation. The wxPython library is provided for GUI programs. The win32 and ctypes extensions allow access on DLLs, ActiveX, COM and type libraries.

Movable Python costs about EUR 7.50. An evaluation version free of charge is (not yet) available.

(Torsten T. Will/ola)

[Image comment] In the simple mode of the Movable Python window captions of the controls are terse, in expert mode the user is overwhelmed by options.

The free evaluation version is now available of course. Smile

I will do the next update of Movable Python shortly after the Python 2.5 release. This will adress a few minor issues (discussed in my next couple of posts). After that I have some fairly major ideas which will make the GUI easier to use (as well as adding functionality) - addressing the review description of the GUI as terse and overwhelming. Laughing

Several of these ideas come from Torsten the reviewer, many thanks.

Like this post? Digg it or it.

Posted by Fuzzyman on 2006-03-20 09:39:53 | |

Categories: , ,

It's All Happening

emoticon:python Python is racing ahead. Guido is setting up a new mailing list for Python 3000. This will discuss the time-scale (probably 2-3 years), implementation and migration. He is reserving a series of PEP numbers specifically for Python 3000 features. He has already set up an SVN Branch to play with implementations.

There is now a Python 2.4.3 Freeze for release this Thursday. (23rd March.)

Python 2.5 alpha 1 is set for release in a couple of weeks.

Like this post? Digg it or it.

Posted by Fuzzyman on 2006-03-20 09:29:26 | |


Nanagram on Softpedia

emoticon:nanagram You can now download the demo version of Nanagram from Softpedia. Yaay...

They even generated their own screenshots for it.

Like this post? Digg it or it.

Posted by Fuzzyman on 2006-03-19 14:00:14 | |


Python Documentation & Epydoc

emoticon:waffle For a long time Epydoc has been virtually the only tool [1] for autgenerating API documentation from docstrings in Python code.

Unfortunately, because it worked by importing modules, it wasn't compatible with some projects (notably Twisted) and couldn't document variables. Version 2.1 of Epydoc was released around two years ago, and it seemed to be languishing.

At last it has had an update, and Epydoc 3 now supports extracting information about Python modules by parsing.

As a result, it can extract "docstrings" for variables.

This is good news. Smile

[1]The other choices include Pudge , Endo and a program that used to be called docextractor and now seems to be called pydoctor. As far as I know Epydoc is the only one with a GUI.

Like this post? Digg it or it.

Posted by Fuzzyman on 2006-03-19 13:43:04 | |

Categories: ,

Hosted by Webfaction