Data Persistence with ConfigObj

The ConfigPersist Module

Author: Michael Foord
Contact: fuzzyman at voidspace dot org dot uk
Version: 0.1.0
Date: 2005/09/07
License:BSD License [1]
Online Version:ConfigPersist online

Data Persistence

Introduction

This module contains various functions for data persistence with ConfigObj.

ConfigObj is a pure python module for the easy reading and writing of application configuration data. It uses an ini file like syntax - similar to the ConfigParser module - but with much greater power.

ConfigObj in conjunction with validate can store nested sections (like dictionaries) - with values as integers, floats, strings, booleans, and lists of these values. This makes it ideal for certain types of human readable (and writeable) data persistence.

For a discussion of this idea (out of which this module was born) - see ConfigObj for Data Persistence.

You can find ConfigObj, and other useful modules, over at the Voidspace Modules Page.

Downloading

You can download the ConfigPersist module from : ConfigPersist.py

Limitations

Hint

This is an extract from the ConfigObj for Data Persistence article.

ConfigObj can't just be used to represent arbitrary data structures - even if all the members are allowed types.

  • Although dictionaries can be nested, they can't be inside lists.
  • Lists also can't be nested inside each other [2].
  • Values other than strings need a schema (a configspec) to convert them back into the right type.
  • Dictionary keys must be strings.
  • It is actually impossible to store a string containing single triple quotes (''') and double triple quotes (""").
  • List members cannot contain carriage returns. (Single line values only). [3]

ConfigObj isn't a data persistence module - this list of restrictions tells you that much. However if you examine the typical data structures used in your programs you may find that these restrictions aren't a problem for many of them.

So Why Do It ?

Why would we want to do this ? Well, the usual method for preserving data structures is the Python pickle module. This can store and retrieve a much wider range of objects - with none of the restrictions above.

However :

  • Pickles aren't human readable or writeable. This makes ConfigObj ideal for debugging or where you want to manually modify the data.
  • Pickles are unsafe - a maliciously crafted pickle can cause arbitrary code execution.
  • ConfigObj is slightly easier to use - data = ConfigObj(filename) and data.write().

Of these three reasons the first is overwhelmingly the most compelling.

The Functions

The first three functions provide the highest level interface to this module. You can use these without needing to the other functions.

save_configspec could be useful to anyone using the ConfigObj module - not just for data persistence.

store

store(config)

Passed a ConfigObj instance add type info and save.

Returns the result of calling config.write().

Caution!

This function modifies the ConfigObj instance by adding the __types__ data.

You can call typeinfo_to_configspec to reverse this.

restore

restore(stored)

Restore a ConfigObj saved using the store function.

Takes a filename or list of lines, returns the ConfigObj instance.

Uses the built-in Validator instance of this module (vtor).

Raises an ImportError if the validate module isn't available.

save_configspec

save_configspec(config)

Creates a configspec for a ConfigObj (which must be comprised of the basic datatypes) and returns it as a list of lines.

Lower Level Functions

These functions provide a slightly lower level interface to adding type info to a ConfigObj. They are still very easy to use though.

add_configspec

add_configspec(config)

A function that adds a configspec to a ConfigObj.

Will only work for ConfigObj instances using basic datatypes :

  • floats
  • strings
  • ints
  • booleans
  • Lists of the above

write_configspec

write_configspec(config)

Return the configspec (of a ConfigObj) as a list of lines.

You must first call add_configspec. You can use save_configspec which does both in one step.

add_typeinfo

add_typeinfo(config)

Turns the configspec attribute of each section into a member of the section. (Called __types__).

You must have already called add_configspec on the ConfigObj.

typeinfo_to_configspec

typeinfo_to_configspec(config)

Turns the __types__ member of each section into a configspec.

(The opposite of add_typeinfo).

vtor

This object isn't actually a function - it's the Validator instance used by this module.

If the validate module isn't available - this object will be None.

CHANGELOG

See the source code for CHANGELOG (and TODO/ISSUES).

Footnotes

[1]Online at http://www.voidspace.org.uk/python/license.shtml
[2]We could remove this restriction by using the listquote module to parse ConfigObj values. Unfortunately a bug in ConfigObj means that this is currently not possible.
[3]List members can also not contain both types of quote. We could remove these last two restrictions using the quote_unescape function from listquote - it's a bit ungainly though. Note however that the walk method of ConfigObj is ideal for transforming values in this way. It will recursively walk the values and apply a function to them all.
Support This Project SourceForge.net Logo

Certified Open Source