Python Programming, news on the Voidspace Python Projects and all things techie.

Callable object with state using generators

emoticon:objects It's often convenient to create callable objects that maintain some kind of state. In Python we can do this with objects that implement the __call__ method and store the state as instance attributes. Here's the canonical example with a counter:

>>> class Counter(object):
...     def __init__(self, start):
...         self.value = start
...     def __call__(self):
...         value = self.value
...         self.value += 1
...         return value
...
>>> counter = Counter(0)
>>> counter()
0
>>> counter()
1
>>> counter()
2

Generators can be seen as objects that implement the iteration protocol, but maintain state within the generator function instead of as instance attributes. This makes them much simpler to write, and much easier to read, than manually implementing the iteration protocol.

This example iterator never terminates, so we obtain values by manually calling next:

>>> class Iter(object):
...     def __init__(self, start):
...         self.value = start
...     def __iter__(self):
...         return self
...     def next(self):
...         value = self.value
...         self.value += 1
...         return value
...
>>> counter = Iter(0)
>>> next(counter)
0
>>> next(counter)
1
>>> next(counter)
2

The same iterator implemented as a generator is much simpler and the state is stored as local variables in the generator:

>>> def generator(start):
...     value = start
...     while True:
...         yield value
...         value += 1
...
>>> gen = generator(0)
>>> next(gen)
0
>>> next(gen)
1
>>> next(gen)
2

In recent versions of Python generators were enhanced with a send method to enable them to act like coroutines.

>>> def echo():
...     result = None
...     while True:
...         result = (yield result)
...
>>> f = echo()
>>> next(f)  # initialise generator
>>> f.send('fish')
'fish'
>>> f.send('eggs')
'eggs'
>>> f.send('ham')
'ham'

(Note that we can't send to an unstarted generator - hence the first call to next to initialise the generator.)

We can use the send method as a way of providing a callable object with state. I first saw this trick in this recipe for a highly optimized lru cache by Raymond Hettinger. The callable object is send method itself, and as with any generator the state is stored as local variables.

Here's our counter as a generator:

>>> def counter(start):
...     yield None
...     value = start
...     while True:
...         ignored = yield value
...         value += 1
...
>>> gen = counter(0)
>>> next(gen)
>>> f = gen.send
>>> f(None)
0
>>> f(None)
1
>>> f(None)
2
>>> f(None)
3

Some observations. Firstly send takes one argument and one argument only. In this example we're ignoring the value sent into the generator, but send must be called with one argument and can't be called with more. So it's mostly useful for callable objects that take a single argument...

Secondly, this performs very well. Function calls are expensive (relatively) in Python because each invocation creates a new frame object (or reuses a zombie frame from the pool - but I digress) for storing the local variables etc. Generators are implemented with a "trick" that keeps the frame object alive, so that the next step of the generator can simply continue execution after the last yield. So our callable object implemented as a generator doesn't have the overhead of a normal function call...

The main advantage this approach has is that it's more readable than the version with __call__. To make it more pleasant to use, we can wrap creating our counter in a convenience function:

>>> def get_counter(start):
...     c = counter(start)
...     next(c)
...     return c.send
...
>>> c = get_counter(0)
>>> c(None)
0
>>> c(None)
1
>>> c(None)
2

Like this post? Digg it or Del.icio.us it.

Posted by Fuzzyman on 2012-01-22 15:05:46 | |

Categories: , Tags: ,


Hosted by Webfaction

Counter...