Python Programming, news on the Voidspace Python Projects and all things techie.
Movable IDLE for Python 2.5 on Windows
Movable IDLE is a standalone version of the IDLE Python IDE. Movable IDLE is part of the Movable Python project and can be run (Windows only I'm afraid) from a USB memory stick and without installing Python. It comes with the full Python standard library.
I've built a new version. The only differences from the previous release are that it is now built with Python 2.5 and no longer displays an annoying dialog on load:
It's a while since I've built a distribution using the Movable Python codebase. It seems fine, but feel free to report any problems you encounter or bugs you find.
The Python Object Model Revisited (data descriptors)
A few weeks ago I demonstrated the complexity of the Python object model by fetching docstrings from objects. A while after posting it I thought of a bug - or at least a way in which it could return the wrong result when looking up an attribute on an object. It will probably come as no surprise that this is due to the descriptor protocol.
Descriptors are special types of objects that have __get__ and or __set__ and __delete__ methods and have special behaviour when fetched, set or deleted as object attributes. They are how methods, class methods, static methods, properties and __slots__ are implemented in Python.
Descriptors that have both __get__ and __set__ are called data descriptors (properties are the canonical example), descriptors with only __get__ are non-data descriptors (methods being the canonical example). Data descriptors have interesting behaviour when they are on a class which has the same member in the instance dictionary.
Instance members are stored in the __dict__ attribute of the object. Normally if this instance dictionary has a member then fetching that member will pull it out of the dictionary. The exception is that if the class has a data-descriptor with the same name then that will be invoked instead of the object in the instance dictionary. This is easy to demonstrate:
... def a(self):
... return 'property'
>>> a = A()
>>> a.__dict__['a'] = 'attribute'
So a data-descriptor on the class will override a member with the same name on the instance - but the 18 lines of code I wrote before for fetching docstrings from attributes will always look on the instance first.
The same is true for inherited data-descriptors:
>>> b = B()
>>> b.__dict__['a'] = 'attribute'
Non-data descriptors don't override instance attributes and data-descriptors on a base class don't override normal class attributes on a subclass.
To handle this we need to check both the instance and walk the inheritance hierarchy. If we find the member we are looking for in both then we check the member from the class for a __set__ method. If the member from the class (or one of its base classes) has a __set__ member then we return that - otherwise we return the member from the instance.
Our modified full code that takes this into account has grown to 22 lines and now looks like:
def get_doc(obj, member):
found = 
if hasattr(obj, '__dict__') and member in obj.__dict__:
if isinstance(obj, (type, types.ClassType)):
search_order = inspect.getmro(obj)
search_order = inspect.getmro(obj.__class__)
for entry in search_order:
if member in entry.__dict__:
if hasattr(entry.__dict__[member], '__set__'):
members = dir(obj)
members = 
return [(member, get_doc(obj, member)) for member in members]
In practise there is another exception that we haven't handled here. Although you can override methods with instance attributes (very useful for monkey patching methods for test purposes) you can't do this with the Python protocol methods. These are the 'magic methods' whose names begin and end with double underscores. When invoked by the Python interpreter they are looked up directly on the class and not on the instance (however if you look them up directly - e.g. x.__repr__ - normal attribute lookup rules apply).
There is a corner case (that I alluded to in my previous post), classes can define __slots__ and create a dummy __dict__ member. If this member isn't a dictionary then our code will barf horribly - but really this is such an evil corner case that I'm not going to worry about it.
I have seen one use case for __slots__ in combination with a fake __dict__ member: proxying attribute access. This is a part of the werkzeug web framework - the LocalProxy class defines __dict__ as a property which returns the __dict__ member of the object it is proxying...
discover: Test discovery for unittest backported to Python 2.4+
I kind of promised you no more entries on unittest for a while, but oh well.
I've backported the test discovery in Python-trunk, what will become Python 2.7 & Python 3.2. Test discovery allows you to run all the unittest based tests (or just a subset of them) in your project without you having to write your own test collection or running machinery. Once installed, test discovery can be invoked with python -m discover. I've tested the discover module with Python 2.4 and 3.0.
Most of the work of backporting was providing an implementation of os.path.relpath (added in Python 2.6) and refactoring the command line handling for standalone use.
The discover module also implements the load_tests protocol which allows you to customize test loading from modules and packages. Test discovery and load_tests are implemented in the DiscoveringTestLoader which can be used from your own test framework.
This is the test discovery mechanism and load_tests protocol for unittest backported from Python 2.7 to work with Python 2.4 or more recent (including Python 3).
discover can be installed with pip or easy_install. After installing switch the current directory to the top level directory of your project and run:
python -m discover python discover.py
This will discover all tests (with certain restrictions) from the current directory. The discover module has several options to control its behavior (full usage options are displayed with python -m discover -h):
Usage: discover.py [options] Options: -v, --verbose Verbose output -s directory Directory to start discovery ('.' default) -p pattern Pattern to match test files ('test*.py' default) -t directory Top level directory of project (default to start directory) For test discovery all test modules must be importable from the top level directory of the project.
For example to use a different pattern for matching test modules run:
python -m discover -p '*test.py'
(Remember to put quotes around the test pattern or shells like bash will do shell expansion rather than passing the pattern through to discover.)
Test discovery is implemented in discover.DiscoveringTestLoader.discover. As well as using discover as a command line script you can import DiscoveringTestLoader, which is a subclass of unittest.TestLoader, and use it in your test framework.
This method finds and returns all test modules from the specified start directory, recursing into subdirectories to find them. Only test files that match pattern will be loaded. (Using shell style pattern matching.)
All test modules must be importable from the top level of the project. If the start directory is not the top level directory then the top level directory must be specified separately.
The load_tests protocol allows test modules and packages to customize how they are loaded. This is implemented in discover.DiscoveringTestLoader.loadTestsFromModule. If a test module defines a load_tests function then tests are loaded from the module by calling load_tests with three arguments: loader, standard_tests, None.
If a test package name (directory with __init__.py) matches the pattern then the package will be checked for a load_tests function. If this exists then it will be called with loader, tests, pattern.
If load_tests exists then discovery does not recurse into the package, load_tests is responsible for loading all tests in the package.
The pattern is deliberately not stored as a loader attribute so that packages can continue discovery themselves. top_level_dir is stored so load_tests does not need to pass this argument in to loader.discover().
discover.py is maintained in a google code project (where bugs and feature requests should be posted): http://code.google.com/p/unittest-ext/
The latest development version of discover.py can be found at: http://code.google.com/p/unittest-ext/source/browse/trunk/discover.py
Catching up: Pythonutils 0.4.0, akismet 0.2.0 and article updates
About two and a half years ago I started writing the book. During the next two years I received a steady trickle of feature requests, bug reports and patches for the various projects and articles I maintain (or pretend to maintain). For the most part I stuck these emails in an 'outstanding' folder promising to deal with them when the book was done (which at the time seemed like it would never happen).
IronPython in Action was properly finished at the beginning of the year (actual writing finished a while before that) and available in the shops nearly 3 months ago; I should be getting sales figures for the first quarter any day now. I still owe you a blog entry on the experience of writing a technical book for Manning. (Executive summary: it isn't such a hot idea whilst working full time and commuting four hours a day.)
Well, incredibly, I've been working through my backlog. I started with 138 emails in my outstanding folder - which included several duplicates and an almost entire Python-dev thread on unittest so not as many as it sounds - and now I'm down to 21 emails. Here are some of the things I've been working on:
akismet.py is a module for accessing the Akismet anti-spam web service. Useful for blogs or applications which accept user comments and want to check for spam.
This release (0.2.0) adds compatibility with Google AppEngine.
All changes in 0.2.0:
- If the data dictionary passed to comment_check doesn't have a 'blog' entry it will be added even if build_data is False. Thanks to Mark Walling.
- Fix for compatibility with Google AppEngine. Thanks to Matt King.
- Added a setup.py - install with pip or easy_install
This package is a collection of general Python utility modules, mostly old now. The old modules are not being actively developed and are in bugfix only maintenance mode. The main reason for me doing a new release is to stop people reporting the same bug in pathutils over and over again...
Code corrections pointed out by Davy Mitchell
Some time ago Python basic authentication handling changed to require the use of the protocol in urls passed to HTTPPasswordMgrWithDefaultRealm().add_password. The article has been updated to reflect this.
Addition of notes on BadStatusLine and HttpException Exceptions.
This work is licensed under a Creative Commons Attribution-Share Alike 2.0 License.