Python Programming, news on the Voidspace Python Projects and all things techie.

Transforming Values with ConfigObj

emoticon:dollars The possibility of ConfigObj being used with Turbogears (mentioned below) raises an interesting point about the transforming of values. I've seen a couple of use cases where validation could have been used to good effect, but wasn't because validation looks complicated.

Like many things, it's very simple once you've got your head round the system. A concrete example probably illustrates it much better than an explanation. The example given in the Turbogears Thread is URL escaping values.

Validation allows you to check that a config file meets a certain specification - e.g. checking that values are present, providing default values if you want, and checking that values make sense, e.g. can be converted to an integer etc.

Validation also transforms values, into integers, floats, booleans etc. (And can check that a value is within boundaries.)

You can easily extend validation to use any transformation function you want. You then specify in the configspec which check (function) to use for each value. You don't need to specify a check for every value, just the ones you want checked.

I'll give you a concrete example :

# config file
[section]
key1 = True
key2 = 3
key3 = 3.0
key4 = 'value which needs to be url escaped %@'
key5 = unchecked

# configspec
[section]
key1 = boolean
key2 = integer
key3 = float
key4 = url_escape
key6 = integer(default=3)

Note a few things. key5 does not have an entry in the configspec. Also, the standard validator doesn't have a check called 'url_escape'. key6 in the configspec supplies a default value if it isn't present in the config file.

Using the configspec to validate the config file :

from validate import Validator, VdtValueError
from urllib import quote_plus
from configobj import ConfigObj

def url_escape(value):
    try:
        return quote_plus(value)
    except:   # can this raise an error ?
        raise VdtValueError

functions = {'url_escape': url_escape}
vdt = Validator(functions)

cfg = ConfigObj(filename, configspec=cfgspec_filename)
cfg.validate(vdt)

for key, value in cfg['section'].items():
    print key, ':', value

This outputs :

key1 : True
key2 : 3
key3 : 3.0
key4 : value+which+needs+to+be+url+escaped+%25%40
key5 : unchecked
key6 : 3

The cfg.validate call returns either True (validation succeeded) or a dictionary of failed values and error messages. (See the documentation for the format of this.) You can use the flatten_errors function to turn this into a list of errors.

This allows you to do any transformation of values that you want, and keeps the transformation out of the config file itself.

My suggestion to turbogears, was that they could allow the application developer to supply a validator and configspec.

Like this post? Digg it or Del.icio.us it.

Posted by Fuzzyman on 2006-03-09 12:00:05 | |

Categories: ,


Configuring Turbogears

emoticon:newspaper Turbogears is arguably the hottest [1] Python project there is right now.

Apparently they're having problems with their current configuration system and Python 2.3. They are/were using pseudo-python files for configuration. Yesterday I discovered this message, from Kevin Dangoor, on the Turbogears Mailing List :

Config File Verdict

So, taking into account the various bits of input and the fact that
there would appear to be a problem with the new style configs and
Python 2.3, I'm going to change our config system over to using
ConfigObj.

Blimey. Smile

I don't think it's set in stone yet, but if Turbogears switch to using ConfigObj that will be fantastic news.

In the same thread, Mike Orr responds with the following comments on ConfigObj :

It looks pretty complete. I like the validator, conversion to ints
and lists, triple-quoted strings, and keeping the keys/values in
order. I suppose I should like the multi-level sections. The
comma-separated list feature is cool, and I guess it has enough
safeguards to distinguish lists from strings.

I'm not sure what he means by should like, but it's nice to see ConfigObj's features appreciated. Razz

[1]Well, along with Django, BitTorrent and a few others.

Like this post? Digg it or Del.icio.us it.

Posted by Fuzzyman on 2006-03-09 10:34:16 | |

Categories: ,


urllib2 Article in French

emoticon:music There is now a translation of urllib2 - the Missing Manual in French.

The translation is located at :

urllib2 - Le Manuel manquant

This is a tutorial on using the Python urllib2 module and handling errors.

The article was translated by Gerard Labadie and the HTML markup done by Thomas Labadie (11 years old). Smile

He will shortly start work on the Basic Authentication and cookielib articles.

Like this post? Digg it or Del.icio.us it.

Posted by Fuzzyman on 2006-03-08 13:13:49 | |

Categories: , ,


ConfigObj Changes

emoticon:note The last three support enquiries I've had about ConfigObj have all been about empty values. These are keys with no value, like :

key=

These are (I think) part of the normal ini file spec, but aren't currently valid syntax for ConfigObj. They were supported in a later version of ConfigObj 3, but since ConfigObj 4 would never produce an empty value I didn't support them.

What I didn't envisage is that people are using ConfigObj to parse ini files that are being produced elsewhere. These users don't necessarily have control over the code that produces these files.

An update to ConfigObj will follow shortly. Empty values will be legal syntax (returning the empty string ''). There will be a corresponding option/attribute, which defaults to False and will probably be called write_empty_values. This will allow ConfigObj to write out empty strings as empty values.

The last update added unicode support. This makes the following methods obsolete :

  • encode
  • decode
  • rename

These will be deprecated in the next release.

Like this post? Digg it or Del.icio.us it.

Posted by Fuzzyman on 2006-03-07 11:00:23 | |

Categories: ,


Battle with Guestbook Spam - Round Three

emoticon:scanner In recent months my guestbook has been subject to a vicious deluge of the worst kind of spam. I haven't yet restored a cleaned up backup, so you can currently still see the damage in all its unholy glory.

These have been coming in at a rate of up to two or three hundred posts a day.

I finally got round to doing something about it over the weekend.

I've just implemented a set of changes that prevent multiple posts from the same IP and multiple posts from the same GET. Every form (from which submissions are made) contains an obfuscated magic number that must be valid for a POST to be accepted. This means that the spammer must do a new GET for every POST. I thought this would cut off all automated submissions.

It has dramatically reduced the rate of spam, which is now down to about ten or fifteen new posts a day. This means the assholes are manually posting spam to my guestbook. Mad

This won't do them any good, because all links use rel="nofollow", but still they persist.

Unfortunately, Akismet hasn't helped me much. That could change if I built in a proper administration interface to teach akismet which posts are spam.

Building a web interface to edit/reject posts is on my list, but it's a fair bit of work. Because the spam is manually posted, a Captcha probably wouldn't help. It would also make the guestbook dependent on something like PIL which would make it unusable on cheap hosting accounts.

The best short term fix I can think of is to require posters to supply a valid email address and verify each post [1]. This is a bit much just for a guestbook, but I don't see an alternative. Sad

[1]This makes it more work for the spammers and I can block email addresses, making it a lot more work per post.

Like this post? Digg it or Del.icio.us it.

Posted by Fuzzyman on 2006-03-07 10:48:49 | |

Categories: ,


Python Source Code Scan

emoticon:noise To my surprise I discovered that not everything the US government is investing in, as part of its Homeland Security project, is a waste of money. Wink

A firm called Coverity, along with the university of Stanford, has done source code analysis of a whole heap of popular open source projects. They've used a new tool which does (presumably) some kind of static source code analysis. It follows every possible path through the code to uncover bugs.

Here is an article on the project, and here are a summary of the results.

Python (with fancy new website - good stuff Tim Parkin and team) comes out quite well. It has less bugs uncovered than both PHP and Perl. However there are about half the lines of code in Python than Perl, so it actually has more bugs per lines of code than Perl (but less than PHP). The lower code count is presumably because a lot of Python is written in Python rather than C.

The results for Python look like :

Project:Python
Bugs:59
Lines of Code:259,896
Bugs per kLOC:0.227

Interesting and useful stuff. In case any of the defects discovered have security implications, the actual defect reports are restricted to a team appointed by the core projects. (Largely a self appointed team from Python-dev in our case.) Smile

Like this post? Digg it or Del.icio.us it.

Posted by Fuzzyman on 2006-03-07 10:39:53 | |

Categories: ,


New Pythonutils Subversion Repository

emoticon:python Pythonutils also has a new Subversion repository. This now includes ConfigObj 4.2.0 and a slightly updated version of pathutils.

http://svn.pythonutils.python-hosting.com/

Once again this is thanks to the folks at Python-hosting.com.

This means that the rest2web repository now only has rest2web in it. Razz

Like this post? Digg it or Del.icio.us it.

Posted by Fuzzyman on 2006-03-04 17:11:56 | |

Categories: ,


Movable Python Review

emoticon:movpy It looks like the review of Movable Python has appeared in the latest issue of the German Computer magazine c't.

I've found two references to it online (c't is a real magazine, although as to be expected it has a fairly comprehensive website) :

I can't tell you what the review says (although I have a German version knocking around somewhere), but the chap doing the review was very nice.

Like this post? Digg it or Del.icio.us it.

Posted by Fuzzyman on 2006-03-04 12:12:05 | |

Categories: ,


Move Out of the Way Skype

emoticon:mobile I've also been playing with configuring a VoIP connection.

The default VoIP is Skype which several of my friends use. I already have google talk for straightforward PC to PC conversations, but it would be nice to have a cheap voice to telephone gateway. This could save me paying for a landline when we move into our own place [1].

A friend has recommended VOIPCheap which he has been using for several months. He also has a phone that plugs straight into his router and works when his PC is off [2].

Incredibly VOIPCheap offer free calls to landlines in the UK, Europe and the US. They have also given me a free landline number. Their other call charges are pretty reasonable. The ones I've checked are cheaper than Skype and comparable with BT.

The catch ? Well you have to give them ten pounds worth of credit (£11.60 including VAT), which expires after four months if you don't use it.

Effectively this means they have a £2.50 a month subscription charge for unlimited free calls to landlines. Nice ! Very Happy

[1]This will happen in the next couple of months or so.
[2]Presumably this is a SIP device ?

Like this post? Digg it or Del.icio.us it.

Posted by Fuzzyman on 2006-03-04 11:23:11 | |

Categories:


Windows Nightmare

emoticon:bluetooth Some of my friends will be pleased to hear this, I'm having a windows nightmare. Laughing

Specifically I'm having a bluetooth nightmare.

I'm not someone who is on a mission to denounce windows at every possible opportunity. However sharp the business practises of a certain Mr. William Gates, I've never had particular difficulties with windows. This is probably because I've never had to administrate windoze systems, and have a couple of friends on hand to do the interesting things like configure networking. I'm not a windows expert and have no intention of ever becoming one.

On the other hand, the only two times I've attempted to install Linux manually [1] it has been a disaster and both times I gave up in frustration. The first time was a couple of years ago on some antiquated hardware. I'm sure things have changed in the meantime.

I now have a nice Nokia 3230. In order to install Python for S60, as well as syncing the calender etc, I bought a Belkin Bluetooth USB Adapter.

I've never had a successful experience of bluetooth, and I know several people with similar testimonies. Belkin devices have a good reputation though, so I thought I'd give it another try. I optimistically installed the driver and plugged it in.

All windows hell broke loose. Surprised

My computer used to be connected to a small LAN. Now it has a direct connection to the internet via a router.

Installing the bluetooth driver seemed to also activate a bluetooth network (which windows complained wasn't working because a cable was unplugged !). Windows decided I ought to be connected to the old LAN, my internet connection dropped out and the PC stared to attempt to connect via my modem (which isn't plugged in). Question

Intermittently I can see my phone via the bluetooth connection, but as soon as I try to connect to it Windows claims it can't find a local bluetooth device.

Much head scratching later, a delve into the bluetooth driver CD which has some buried instructions about updating the driver for SP 2, and I finally get my internet connection back. I have to run the network wizard, so my PC now behaves differently on startup. This is possibly useful as I may now be able to get the VMWare networking operating, this will help me build Movable Python without having to swap files via my USB card.

Despite this, bluetooth is still not working. I can much more reliably find my phone (and PDA incidentally) when I search for devices. As soon as I start to connect to them though, the bluetooth icon flashes red and I get the message "your local bluetooth device has stopped". So is this a hardware problem, or just another step deeper into the bluetooth/windows mire ?

Anyway, I'll try the dongle on another machine and see what happens.

[1]I've recently installed it remotely on a couple of virtual servers without problem. These were largely automated processes.

Like this post? Digg it or Del.icio.us it.

Posted by Fuzzyman on 2006-03-04 10:53:22 | |

Categories:


Voidspace is Looking for a Sponsor

emoticon:file1 Over the next year the Voidspace website will serve more than a million and half page views [1]. In 2005, the various projects here were downloaded over forty thousand times.

Around half the pages served are in the Python and Technology sections.

In order to be able to free up some working time to develop the Open Source Projects here, I'm looking for a partnership with a commercial sponsor.

In return I can offer :

  • A banner advert, with link, on the Voidspace pages. Most of these are pagerank five.
  • The opportunity to regularly post news and information on the Voidspace weblog. This is aggregated at PlanetPython and regularly featured on the Daily Python URL.
  • The opportunity for product and service reviews, or even articles.
  • Sponsored by '...' in the project documentation and downloads of the many open source projects hosted at Voidspace.
  • As well of course as promoting your company as being further involved in supporting open source software development.

If you are part of a company or organisation that may be interested in this unique advertising opportunity, please visit the Sponsor Voidspace page.

[1]Based on current traffic figures, it will be higher than this. Very Happy

Like this post? Digg it or Del.icio.us it.

Posted by Fuzzyman on 2006-03-04 10:53:02 | |

Categories: ,


Hosted by Webfaction

Counter...