Python Programming, news on the Voidspace Python Projects and all things techie.

Testing File Access with storage.py

emoticon:men So in my last blog entry I introduced storage.py and discussed how it could simply testing file access. What I didn't do is show you how.

In writing this example I realised an added benefit; it can remove the differences between platforms.

This example uses the default dictionary based backend of storage.py. The dictionary that holds 'files' (strings keyed by the full path) is storage._store. We can put data in there and after replacing the builtins know that file reads will get our precanned data. After writes we can make assertions about the contents of the dictionary.

The following test demonstrates the basic usage:

from __future__ import with_statement
import unittest
import storage


class TestStorage(unittest.TestCase):

    def test_file_access(self):
        storage.replace_builtins()
        storage._store.clear()
        storage._store['/tmp/foo'] = 'my data is here'

        try:
            with open('/tmp/foo') as f:
                self.assertEqual(f.read(), 'my data is here')

            with open('/tmp/bar', 'w') as f:
                f.write('more data')

            self.assertEqual(storage._store['/tmp/bar'], 'more data')
        finally:
            storage.restore_builtins()

if __name__ == '__main__':
    unittest.main()

And when run the test passes, even on Windows where I definitely don't have a '/tmp' folder:

C:\compile\storage
> python test.py -v
test_file_access (__main__.TestStorage) ... ok

----------------------------------------------------------------
Ran 1 test in 0.011s

OK

Like this post? Digg it or Del.icio.us it.

Posted by Fuzzyman on 2009-10-16 12:00:39 | |

Categories: Tags: , ,


Testable file Type and File I/O in Try Python

emoticon:halt A few weeks ago I announced Try Python, a Python tutorial with interactive interpreter that runs in the browser using IronPython and Silverlight.

Because it runs in Silverlight, which is sandboxed from the local system, the parts of the tutorial that show how to do file I/O didn't work. I've finally fixed this with a pretty-much-complete implementation of the file type and open function that use local browser storage (IsolatedStorage) to store the files.

Using the open function in the Try Python tutorial

You can try this out by going direct to the Reading and Writing Files part of the tutorial (page 32).

The implementation is split into two parts, the file type and open function in storage.py and the backend that uses the IsolatedStorage API in storage_backend.py. There are also extensive tests of course.

In Silverlight you use it like this:

import storage
import storage.backend

storage.backed = storage_backend
storage.replace_builtins()

This sets the IsolatedStorage based backed in the storage module and then replaces the builtin file and open with the versions in storage. There is a corresponding restore_builtins function to put the original ones back. You could use this for convenient file usage inside Silverlight applications, but the code is neither efficient nor elegant so I wouldn't recommend it for production!

You can also call storage.file and storage.open directly without having to replace the builtin versions. If you use the browser backend in Silverlight then files will be persisted in browser storage and be available when an application is revisited later (unless the user clears the browser cache).

Note

The tests require Python 2.5 or more recent as they use the with statement. The actual implementation should be compatible with Python 2.4 or even earlier. (In fact because they use the storage_backend the tests will only run on Silverlight but it would be very easy to get them to run on CPython.)

You can see the tests run in the browser (slowly - as I run them in a background thread) at: storage module tests.

The storage module does have a default dictionary based backend for when used outside Silverlight. This simply stores files as strings in a dictionary using the full file path as the key (it has no concept of dictionaries). You could implement an alternative backend by implementing the four functions in the storage_backend module (or the method on the backend class in the storage module). This leads to an interesting potential use case.

Unit testing code that does file I/O is notoriously difficult. You can either let your unit tests do real file access or modify your production code to use something like dependency injection so that you can mock out the file access during the tests. Using this implementation you can swap out the builtin file type during the test, controlling how it behaves using the dictionary backend, without having to change the way you write your production code just to make it testable.

For this to be really useful it needs an implementation of functions in the os and os.path module (like os.listdir, os.remove, os.makedir(s), os.path.isfile, os.path.isdir and so on). This should be easy to do and I will get round to it as they would be nice things to cover in the Try Python tutorial.

There are several (mostly minor) differences between this and the standard file type, plus a few things still on the TODO list:

Differences from standard file type:

  • Attempting to set the read-only attributes (like mode, name etc) raises an AttributeError rather than a TypeError
  • The exception messages are not all identical (some are better!)
  • Strict about modes. Unrecognised modes always raise an exception
  • The deprecated readinto is not implemented

Still todo:

  • The buffering argument to the constructor is not implemented
  • The IOError exceptions raised don't have an associated errno
  • encoding, errors and newlines do nothing
  • Behavior of tell() and seek() for text mode files may be incorrect (it should treat '\n' as '\r\n' in text mode)
  • Behaves like Windows, writes '\n' as '\r\n' unless in binary mode. A flag to control this?
  • Universal modes not supported
  • Missing __format__ method needed when we move to 2.6
  • Implementations of os and os.path that work with storage_backend

As an added bonus for Try Python the IronPython team have created a new .NET interop tutorial for IronPython and it is in written in ReStructured Text. It will be very easy for me to add this to Try Python as well as the Python tutorial; I'll wait a bit until it has stabilised though.

Like this post? Digg it or Del.icio.us it.

Posted by Fuzzyman on 2009-10-15 17:33:37 | |

Categories: , , Tags: ,


Hosted by Webfaction

Counter...