Python Programming, news on the Voidspace Python Projects and all things techie.
Testing File Access with storage.py
So in my last blog entry I introduced storage.py and discussed how it could simply testing file access. What I didn't do is show you how.
In writing this example I realised an added benefit; it can remove the differences between platforms.
This example uses the default dictionary based backend of storage.py. The dictionary that holds 'files' (strings keyed by the full path) is storage._store. We can put data in there and after replacing the builtins know that file reads will get our precanned data. After writes we can make assertions about the contents of the dictionary.
The following test demonstrates the basic usage:
storage._store['/tmp/foo'] = 'my data is here'
with open('/tmp/foo') as f:
self.assertEqual(f.read(), 'my data is here')
with open('/tmp/bar', 'w') as f:
self.assertEqual(storage._store['/tmp/bar'], 'more data')
if __name__ == '__main__':
And when run the test passes, even on Windows where I definitely don't have a '/tmp' folder:
C:\compile\storage > python test.py -v test_file_access (__main__.TestStorage) ... ok ---------------------------------------------------------------- Ran 1 test in 0.011s OK
Testable file Type and File I/O in Try Python
Because it runs in Silverlight, which is sandboxed from the local system, the parts of the tutorial that show how to do file I/O didn't work. I've finally fixed this with a pretty-much-complete implementation of the file type and open function that use local browser storage (IsolatedStorage) to store the files.
You can try this out by going direct to the Reading and Writing Files part of the tutorial (page 32).
The implementation is split into two parts, the file type and open function in storage.py and the backend that uses the IsolatedStorage API in storage_backend.py. There are also extensive tests of course.
In Silverlight you use it like this:
storage.backed = storage_backend
This sets the IsolatedStorage based backed in the storage module and then replaces the builtin file and open with the versions in storage. There is a corresponding restore_builtins function to put the original ones back. You could use this for convenient file usage inside Silverlight applications, but the code is neither efficient nor elegant so I wouldn't recommend it for production!
You can also call storage.file and storage.open directly without having to replace the builtin versions. If you use the browser backend in Silverlight then files will be persisted in browser storage and be available when an application is revisited later (unless the user clears the browser cache).
The tests require Python 2.5 or more recent as they use the with statement. The actual implementation should be compatible with Python 2.4 or even earlier. (In fact because they use the storage_backend the tests will only run on Silverlight but it would be very easy to get them to run on CPython.)
You can see the tests run in the browser (slowly - as I run them in a background thread) at: storage module tests.
The storage module does have a default dictionary based backend for when used outside Silverlight. This simply stores files as strings in a dictionary using the full file path as the key (it has no concept of dictionaries). You could implement an alternative backend by implementing the four functions in the storage_backend module (or the method on the backend class in the storage module). This leads to an interesting potential use case.
Unit testing code that does file I/O is notoriously difficult. You can either let your unit tests do real file access or modify your production code to use something like dependency injection so that you can mock out the file access during the tests. Using this implementation you can swap out the builtin file type during the test, controlling how it behaves using the dictionary backend, without having to change the way you write your production code just to make it testable.
For this to be really useful it needs an implementation of functions in the os and os.path module (like os.listdir, os.remove, os.makedir(s), os.path.isfile, os.path.isdir and so on). This should be easy to do and I will get round to it as they would be nice things to cover in the Try Python tutorial.
There are several (mostly minor) differences between this and the standard file type, plus a few things still on the TODO list:
Differences from standard file type:
- Attempting to set the read-only attributes (like mode, name etc) raises an AttributeError rather than a TypeError
- The exception messages are not all identical (some are better!)
- Strict about modes. Unrecognised modes always raise an exception
- The deprecated readinto is not implemented
- The buffering argument to the constructor is not implemented
- The IOError exceptions raised don't have an associated errno
- encoding, errors and newlines do nothing
- Behavior of tell() and seek() for text mode files may be incorrect (it should treat '\n' as '\r\n' in text mode)
- Behaves like Windows, writes '\n' as '\r\n' unless in binary mode. A flag to control this?
- Universal modes not supported
- Missing __format__ method needed when we move to 2.6
- Implementations of os and os.path that work with storage_backend
As an added bonus for Try Python the IronPython team have created a new .NET interop tutorial for IronPython and it is in written in ReStructured Text. It will be very easy for me to add this to Try Python as well as the Python tutorial; I'll wait a bit until it has stabilised though.
This work is licensed under a Creative Commons Attribution-Share Alike 2.0 License.