Python Programming, news on the Voidspace Python Projects and all things techie.

Towards a Better Python

emoticon:avocado Guido recently answered some of his questions on the Google Moderator Site. One of them was mine, on the future of the CPython VM:

"Do you see the current design of the CPython VM holding back Python the language - particularly in terms of adding a JIT, moving away from reference counting for garbage collection and the concurrency issues around the GIL?"

Guido answered:

That's a rather leading question. I personally don't think so, but this is actually a level of language implementation where I don't know a whole lot, and have to defer to the compiler and VM experts.

If you believe CPython cannot be saved, I recommend that you start investing time in working with some of the alternative implementations, like PyPy, Jython or IronPython, in order to ensure that they will be of sufficient quality to take over if CPython ever finds it cannot keep up.

True enough it was a leading question. Smile CPython is great, and will be with us for a long time [1]. It is however also true that certain design decisions about the CPython implementation make it very difficult to experiment with alternative strategies.

This is one of the driving forces that led to the creation of PyPy. As an interpreter translation toolchain it takes descriptions of interpreters written in a restricted (static) subset of Python called RPython and compiles them for a variety of backends and platforms (C, .NET, JVM, LLVM etc). Various aspects of the interpreters are 'pluggable', including the garbage collection.

One particularly contentious area of debate amongst and outside the Python community is the effect that the Global Interpreter Lock (GIL) has on using Python for concurrent programming. Python does allow developers to use native threads, but the GIL ensures that only one of them is active at any one time. Whilst IO and other blocking operations release the GIL, as can C extensions, it makes it impossible to use more than one core from pure Python code on multi-core hardware. As CPUs get no faster in terms of raw clock speed, but add horse power by upping the number of cores, parallel programming is becoming ever more important.

One solution is using multiple processes instead of multiple threads. A very powerful library that makes this easier, even having a compatible interface with the Python threading library, was recently added to the standard library: multiprocessing

This is undoubtedly a powerful and effective technique for concurrent programming, but there are plenty of problems for which it simply isn't suited. Unfortunately some in the Python community simply refuse to believe this. The net effect is that those who have, or believe they have, problems requiring a concurrent solution are less likely to turn to Python.

(Those who espouse Erlang style actors for concurrency would do well to realise that the Erlang VM uses multiple threads under the hood to actually run its lightweight processes. Threads can be a powerful concurrency primitive even if we need better APIs for working with them [2].)

There have been various attempts to remove the GIL from Python before. One of the advantages of the GIL is that it makes extension programming much simpler. The latest attempt is basically successful. Adam Olsen has created a set of patches to the CPython source code in a project called safe thread. This implements the fine grained locking needed to replace the GIL. Unfortunately the extra cost of this locking means that for single threaded code the patched interpreter runs ~30% slower than CPython. The up-side is that multi-threaded code can use multiple cores, and can run faster than standard CPython.

One of the reasons that so much fine grained locking is required is because reference counting is used for garbage collection. Every time a reference to a Python object is created or removed its reference count is incremented or decremented. When the reference count reaches 0 the object is disposed of. This means that Python has deterministic object finalization (except in the case of cycles where a generational cyclic garbage collector is used). This manipulation of an object's reference count must be thread safe - if changes were lost due to simultaneous updates then objects may never be garbage collected or collected too early.

Note

Deterministic finalization means that a Python object with a __del__ method (the finalizer) will have it called immediately that the reference count reaches zero. This is why it is a common pattern in Python to open files without explicitly closing them. As soon as the file handle goes out of scope it will be closed.

The exception to this is cycles, which are collected by a separate generational GC. Unfortunately, because this doesn't know what order to break cycles in where there are explicit finalizers it doesn't attempt to. This means that cycles of objects with __del__ methods aren't collected in CPython (and an app that has them will leak memory).

Soooo... if Python no longer used reference counting for garbage collection then removing the GIL would be easier, right?

A few days ago this email was sent to the Python developers mailing list:

Hi to all Python developers

For a student project in a course on virtual machines, we are evaluating the possibility to experiment with removing the GIL from CPython

We have read the arguments against doing this at http://www.python.org/doc/faq/library/#can-t-we-get-rid-of-the-global-interpreter-lock.

But we think it might be possible to do this with a different approach than what has been tried till now.

The main reason for the necessity of the GIL is reference counting.

We believe that most of the slowdown in the free threading implementation of Greg Stein was due to the need of atomic refcounting, as this mail seems to confirm: http://mail.python.org/pipermail/python-ideas/2007-April/000414.html

So we want to change CPython into having a "real" garbage collector - removing all reference counting, and then the need for locks (or atomic inc/dec ops) should be highly alleviated.

Preferably the GC should be a high-performance one for instance a generational one.

We believe that it can run quite a lot faster than ref-counting.

Shared datastructures would get their lock obviously. Immutable objects (especially shared global objects, like True, False, Null) would not.

[some discussion of details snipped...]

We know it is the plan for PyPy to work in this way, and also that Jython and Ironpython works like that (using the host vm's GC), so it seems to be somehow agreeable with the python semantics (perhaps not really with __del__ but they are not really nice anyway).

Was this ever tried for CPython?

Any other comments, encouragements or warnings on the project-idea?

Best regards: Paolo, Sigurd

Yet another wild-eyed attempt to remove the GIL by crackpots with an even-more-hare-brained-than-usual idea? The email was immediately followed by this one from Van Lindberg:

Just an FYI, these two particular students already introduced themselves on the PyPy list. Paolo is a masters student with experience in the Linux kernel; Sigurd is a PhD candidate.

Their professor is Lars Bak, the lead architect of the Google V8 Javascript engine. They spent some time working on V8 in the last couple months.

Maybe not so hare-brained then. We'll have to wait and see. Smile

Even if they succeed there are a few issues. What about all of those extension that both assume and require the presence of the GIL (or even reference counting itself)?

There are possible ways round this. For Ironclad, an implementation of the Python C API in C# to allow you to use Python C extensions from IronPython, we take a hybrid approach to garbage collection.

We hold a strong reference to objects created by extensions (preventing .NET from garbage collecting them) and allow the extension to manipulate the reference count in the usual way. Periodically we scan our "managed objects" [3] to see if there are objects whose reference count has dropped to zero (well - one actually) and we then delete our strong reference - allowing garbage collection to happen normally (including calling back into unmanaged finalizers to free memory and clean up resources).

We have also implemented a GIL for the C extensions even though IronPython has no GIL. It would be possible to have an external API for existing extensions that implements the semantics of both the Global Interpreter Lock and reference counting whilst having a new API for new extensions that are happy with the more advanced capabilities of a turbo-charged Python.

In the discussion thread that followed in the wake of Paolo and Sigurd's email Maciej outlined some of the implications of losing deterministic finalisation:

PyPy has a semi-advanced generational moving gc these days. It's not as well tuned as JVMs one, but it works quite well. Regarding semantic changes, there is a couple which as far as I remember are details which you should not rely on anyway (At least some of the below applies also for Jython/IronPython):

  • __del__ methods are not called immediately when object is not in a cycle
  • all objects are collected, which means objects in cycles are broken in arbitrary order (gc.garbage is always empty), but normal ordering is preserved.
  • id's don't always resemble address of object in memory
  • resurrected objects have __del__ called only once in total.

The last point is interesting. If an object is resurrected during finalization (it creates a new strong reference to itself inside its __del__ method) I would expect __del__ to be called again when it enters finalization again. I think this is how the .NET garbage collector behaves anyway.

UPDATE

It turns out that by default .NET garbage collection doesn't call finalizers more than once. You can however call GC.ReRegisterForFinalization on an object if you want this behaviour. My memory was polluted because for a brief while Ironclad used resurrection to handle objects with no references on the managed side that still had a reference count above one (where the extension still holds references). For complex reasons re-registering for finalization doesn't work on IronPython objects, so we changed strategy.

Of course if you really need true-concurrency 'in-process' today and you still want to program in Python you can turn to an alternative implementation like Jython or IronPython, neither of which have a GIL. The challenge faced by implementers of Python is that, because CPython acts as a reference implementation rather than there being a concrete specification, is knowing which parts of Python are specified by the reference implementation and which parts are merely implementation details. The Python documentation does specifically note some aspects of Python's behaviour as being not guaranteed and dependent on the implementation. Other aspects aren't so clear. Wouldn't it be good if the Python test suite (which is ropey but a lot better than test suites for many other language implementations) clearly separated out the implementation dependent aspects?

Well, Armin Rigo (one of the core PyPy developers) has started doing exactly that. This is from another email to the Python developer mailing list:

Here is a first step towards classifying the Python test suite into "real language tests" and "implementation details":

http://bugs.python.org/issue4242

If the general approach seems acceptable to people, I would be willing to port more of PyPy's test suite patches. The net result would be to move the test suite towards being more directly useful for alternate Python implementations. (Right now, all of them have some custom tests plus their own similarly patched copy of the stdlib test suite.)

Note that the patch above is against 2.7 trunk; the 2.x line is what all non-CPython implementations currently target. The general idea (and to a large extend the patches) can easily be forward-ported to 3.x when it makes sense.

Also, the actual decision of what is a "real" or a "detail" test is of course up to discussion. The patch above does the classification for test_descr. It was obtained by running it with PyPy, and for each failure either fixing PyPy, or argumenting (by adding a comment) the reason for which it is a detail -- usually referring to "well-known agreement" expressed on python-dev about the matter. (Of course, "detail" tests still run normally on top of CPython.)

I haven't looked through what he is done, but I expect to have some 'issues' with the classification. The PyPy developers (well - Maciej Fijalwotsit anyway) believe that Python stack frames are part of the core language specification. IronPython doesn't have Python stack frames (for performance reasons) as Jim Hugunin believed that they were merely an facet of the CPython implementation. Now if only I could find that email where Guido explicitly agrees with this... Wink

[1]Regular readers will know that I have however invested a certain amount of time in IronPython over the last two and a half years. This is not out of any dissatisfaction with CPython, merely a quirk of fate.
[2]I'm actually paraphrasing Michael Sparks here who is doing some extremely interesting work on providing good APIs for concurrent programming through Kamaelia.
[3]Strictly speaking they are "managed objects of unmanaged types", but we really need a better way of talking about them.

Like this post? Digg it or Del.icio.us it.

Posted by Fuzzyman on 2008-11-05 20:14:22 | |

Categories: , Tags: , , , , ,


More Changes to Mock: Mocking Magic Methods

emoticon:pen_book I recently released Mock 0.4.0, my test support library for mocking objects and monkeypatching. It's been gratifying and surprising to get so many emails from people using it.

I originally wrote Mock to simplify some of the testing patterns we use at Resolver Systems, and we use a forked version [1] there. Mock has greatly improved the readability of our tests whilst reducing the number of lines of code needed. Some improvements made at Resolver Systems (like assert_called_with and the single argument form of patch) have fed back into the public version.

The obvious lack in Mock as it stands at 0.4.0 is its inability to mock magic methods. This means that you can't use it to mock out any of the built-in container classes, or classes that implement protocol methods like __getitem__ and __setitem__. The main reason that it didn't support them was the lack of a clear and obvious way to do it. At Resolver Systems we've bolted on support for a few of the magic methods as we've needed them; unfortunately in an ad-hoc and inconsistent manner.

My requirements for protocol support in Mock (which I think I've now met) were:

  • A clean, simple and consistent API
  • Being able to meet most use cases without having to meet all of them
  • Not unconditionally adding a large number of new attributes to mock objects [2]
  • Mocks shouldn't support all the protocol methods by default as this can break duck-typing
  • It mustn't place any additional burden on developers not using them

The solution I've now come up with is implemented in the SVN repository, and will become Mock 0.5.0. So far it is only capable of mocking containers and I'd like feedback as to whether this is going to meet people's use cases. If you are interested in this, please try the latest version and let me know what you think:

Documentation for the new features is not done, but it is all tested so you can check 'mocktest.py' and 'patchtest.py' for examples of how they work.

The implementation uses a new class factory called MakeMock. This takes a list of strings specifying the magic methods (without the double underscores for brevity!) you want your mock objects to have - it returns a subclass of Mock that only has the magic methods you asked for.

For the container methods, the Mock class takes a keyword argument items that can either be a dictionary or a sequence. This is stored as the _items attribute (that only exists on mocks with container methods), defaulting to an empty dictionary, and can be any mapping or sequence object. The container methods delegate to this, and all calls are recorded normally in method_calls.

>>> from mock import MakeMock
>>> MagicMock = MakeMock('getitem setitem'.split())
>>> mock = MagicMock(items={'a': 1, 'b': 2})
>>> mock['a']
1
>>> mock.method_calls
[('__getitem__', ('a',), {})]
>>> mock['c'] = 10
>>> mock._items
{'a': 1, 'c': 10, 'b': 2}

There is an additional bit of magic. When you instantiate a mock object normally (using Mock(...)), you can use the 'magics' or 'spec' keyword arguments to actually get back an instance with magic methods support. The spec keyword argument takes a class that you want your Mock to imitate, and accessing methods not on the actual class will raise an AttributeError. When you instantiate a mock object with a spec keyword argument, the constructor will check if the spec class has any supported magic methods; if it does you will actually get an instance of a mock that has these magic methods. The magics keyword argument is new, and lets you specify which magic methods you want:

>>> from mock import Mock
>>> mock = Mock(magics='getitem contains')
>>> 'hi' in mock
False
>>> mock['hi'] = 'world'
Traceback (most recent call last):
  ...
TypeError: 'MagicMock' object does not support item assignment
>>> mock._items['hi'] = 'world'
>>> 'hi' in mock
True
>>> type(mock)
<class 'mock.MagicMock'>
>>> mock.method_calls
[('__contains__', ('hi',), {}), ('__contains__', ('hi',), {})]

Note that the magics keyword takes a string and does the split for you. Razz

As I implement more magic methods I'll provide shortcuts for obtaining instances / classes that have all the container methods, or all the numeric methods, or just everything.

The patch decorator also now take spec and magics keyword argument, but that's not as useful as it sounds. You will usually be using patch to mock out a class, so you'll still need to set the return value to be a mock instance with the methods you want.

For comparison methods I'll allow you to provide a 'value' object that all comparisons delegate to. This can also be used for hashing, in-place, right hand side and unary operations. That doesn't leave much left to cover (descriptors and context management protocol methods - but I'm not sure how much demand there will be for mocking these.)

Protocol methods supported currently are:

  • __getitem__
  • __setitem__
  • __delitem__
  • __iter__
  • __len__
  • __contains__
  • __nonzero__

The list of changes that are currently in Mock 0.5.0:

  • Mock has a new 'magics' keyword arguments - a list (or string separated by whitespace) of magic methods that the Mock instance should provide (only container methods available so far).

  • Mock has an 'items' keyword argument for mocks implementing container methods.

  • The methods keyword argument to Mock has been removed and merged with spec. The spec argument can now be a list of methods or an object to take the spec from.

  • patch and patch_object now take magics and spec keyword arguments

  • TODO: verify - Nested patches may now be applied in a different order (created mocks

    passed in the opposite order). This is actually a bugfix.

  • MakeMock is a Mock class factory. It takes a list of magic methods and creates a MagicMock class (Mock subclass) with the magic methods you specified. Currently only container methods are available.

  • Instantiating a Mock with magics / spec will actually create a MagicMock with magic methods from the method / spec list.

There are still a few things left to do (apart from implementing support for all the other magic methods) and some open questions:

  • Should MakeMock accept a string argument (and do the split for you)?
  • How should reset affect '_items'? It should probably take a copy of the items at instantiation and restore it when reset.
  • Should a failed indexing attempt still be added to 'method_calls'? (Currently not - it just raises the usual exception.)
  • Should attributes (children) on mocks created from MakeMock be plain 'Mock' or from the same class as their parent? (currently they are plain mocks)
  • Parent method calls if magic methods called on a child? (not currently possible as all children will be mocks rather than magic mocks)

The first two of these I will probably decide one way or the other before a 0.5.0 release. The others I may just leave for a while and see how it gets used.

[1]Forked because we use a kind of half-breed .NET API naming convention, and also to maintain API compatibility with the Listener class that it replaces - and that is still in use in our older tests.
[2]Every attribute that Mock uses internally is an attribute that it is no longer capable of mocking for the programmer. If Mock uses a '.value' attribute then it is no good for mocking a class that has a '.value' attribute as they will conflict.

Like this post? Digg it or Del.icio.us it.

Posted by Fuzzyman on 2008-11-05 15:13:20 | |

Categories: , , Tags: , , ,


Python: Values and References

emoticon:objects An important step for anyone learning Python is understanding that instead of variables with values Python has names referring to objects (often described as names bound to objects). This is slightly different to other languages, but only slightly. Most modern languages (like Java and C#) have a concept of variables as references to objects - but they also have variables that hold values directly.

There was a long discussion recently on the Python newsgroup as to whether Python uses "call by reference" or "call by value". The discussion was mainly an attempt to persuade one individual that call by value isn't a sensible description of the way that Python passes parameters into function / method calls. Despite the apparent pointlessness of the topic, a lot of interesting matters were discussed along the way.

Wikipedia defines Call by Value as:

In call-by-value, the argument expression is evaluated, and the resulting value is bound to the corresponding variable in the function (frequently by copying the value into a new memory region). If the function or procedure is able to assign values to its parameters, only the local copy is assigned - that is, anything passed into a function call is unchanged in the caller's scope when the function returns.

As you can see, this only partly applies to Python. Local assignments inside a function don't affect the calling scope, but values aren't 'copied' into a function call and changes to mutable objects are seen by the caller.

Wikipedia (same page) defines call by reference as:

In call-by-reference evaluation, a function receives an implicit reference to the argument, rather than a copy of its value. This means that the function can modify the argument, what will be seen by its caller.

This seems to fit Python very well, but it is often taken to mean that assignments to parameters passed in by reference will also affect the calling scope. Although in Python mutable objects can be modified inside a function / method call - and those changes will be seen by the caller - rebinding the name to a new object will not affect the calling scope. (Except variables explicitly declared as global, or in Python 3 where you use the nonlocal keyword).

A more common understanding of these terms is that in call by value values are copied when passed as parameters, whereas in call by reference a pointer (reference) to the object is passed. By these definitions Python is clearly call by reference and this definition also matches the distinction that languages like C# make between value types and reference types (I've not programmed in Java, but my understanding is that it makes the same distinction).

In C#, value types (all numeric types, structs, enumerations etc) actually live on the stack, whereas reference types (strings, instances of classes etc) live on the heap and have a pointer on the stack.

It is complicated by the fact that .NET perpetuates the myth that the primitives (the value types) inherit from System.Object (a reference type). The runtime does auto-boxing for you where this matters.

This means that when you pass a value type into a method the value is copied. Structs can be arbitrarily big, so this can be a performance problem. It is often a performance win for working with the numeric types as you remove a level of indirection.

It isn't without problems though - and some of these can be seen if you develop with IronPython.

System.Drawing.Point is a struct (a value type), but it is also mutable (despite the fact that mutable value types are against the .NET design guidelines). Because it is a value type, when you access it you are often accessing a copy - and if you mutate a copy then your changes can be lost.

The following code illustrates the problem:

>>> r = Rectangle(0, 100, 20, 40)
>>> r.Location.X
0
>>> r.Location.X = 20
>>> r.Location.X
0

Because r.Location returns a value type (r.Location is a Point), the update to it is lost. This happens in C# as well as in IronPython, but it is more unexpected in Python because all objects are expected to behave like reference types.

In .NET you can specify that a parameter to a method takes a reference ('out' in C# or ByRef in VB.NET). If you pass in a value type as a reference parameter then the .NET runtime will do boxing / unboxing for you.

This means that we can write code like the following:

int x = 3;
SomeMethod(out x);

x can now be mutated by 'SomeMethod' and have a different value after the method call.

In a 'way' immutable Python types behave a bit like .NET value types, and mutable types like reference types. This analogy breaks down when we push it though.

As a final note, Aaron Brady and others point out that much better descriptions for the Python calling convention is "call by object" or "call by sharing":

From Barbara Liskov's paper, 1992:

"...arguments are passed "by object"; the (pointer to the) object resulting from evaluating the actual argument expression is assigned to the formal."

However a Google search for "call by object", in quotes, returns mostly Python-related links.

The search for call by sharing was more successful:

"Call by Sharing The caller and called routine communicate only through the argument and result objects; routines do not have access to any variables of the caller.... if a routine assigns an object to a formal argument variable, there is no effect on the caller."

Like this post? Digg it or Del.icio.us it.

Posted by Fuzzyman on 2008-11-04 13:54:05 | |

Categories: , , Tags: , , ,


Obscure Magic Methods and Pickle Support

emoticon:development One of the appendices for IronPython in Action is a guide to the Python magic methods.

Note

Python magic methods are the methods that implement protocols in Python. They all (except for the next method on iterators in Python 2.X) start and end with double underscores.

They are called magic methods because although you explicitly implement them you don't usually call them directly, instead the interpreter calls them for you. For example the object creation methods __init__ and __new__ are called at object instantiation.

I haven't covered a couple of the deprecated magic methods (__coerce__ anyone?), but I think that I've covered all the others (including descriptor methods and the new context manager protocol methods - it was fun brushing up on these).

Along the way I discovered a few new magic methods that I'd not encountered before:

  • __length_hint__(self)

    Iterators can implement this as an optimization so that Python can pre-allocate space for them. The Python tests say that it is explicitly undocumented so you probably won't be implementing this yourself, but if you see it method now you know what it is for.

  • __reversed__(self)

    Containers may implement this method to provide an 'optimized reverse iterator' for instances. If this method is available it is called by the built-in function reversed. (Very useful if you implement iteration as a generator - otherwise reversed has to consume the whole generator in order to be able to reverse it.)

  • __missing__(self, key)

    New in Python 2.5. This method is called on subclasses of the built-in dictionary ('dict') type when a key that doesn't exist is requested. This method is useful for providing default values. For example:

    class ListDict(dict):
        def __missing__(self, key):
            self[key] = []
            return self[key]

    thing = ListDict()
    thing[key].append(value)

    It is there to support defaultdict, where it is documented.

There are two magic methods not used by the Python interpreter, but used by the pickle module for serialization: __setstate__ and __reduce__.

UPDATE: They're documented obliquely here

Pickle can handle most, but not all, pure Python objects. For some reason ConfigObj instances are one of the object types it can't handle. (In my experimentation pickle could handle subclasses of dictionaries that also had attributes and I couldn't get a minimal sample of an object that pickle couldn't handle - ConfigObj can be pretty deeply nested objects though.)

The examples show how these methods for handling serializing objects with open file handles and sockets (not the case with ConfigObj).

Nicola Larosa and I didn't want to have to maintain pickle support in ConfigObj as it is a persistence mechanism itself and we couldn't see a use case for serializing them. Unfortunately libraries like Parallel Python that provided shared access between process use serialization. Thankfully Christian Heimes providing some code that does the job. It implements __reduce__ (along with a function called __newobj__) for persisting state and an inverse function called __setstate__ for restoring it:

def __newobj__(cls, *args):
    return cls.__new__(cls, *args)

class SomeClass(dict):

    def __setstate__(self, state):
        dict.update(self, state[0])
        self.__dict__.update(state[1])

    def __reduce__(self):
        state = (dict(self), self.__dict__)
        return (__newobj__, (self.__class__,), state)

You can see that the __reduce__ implementation stores all the values in the dictionary plus attributes stored in __dict__ in the state object. __setstate__ takes a state object and does the restore.

The pickle module source code has notes that aren't in the docs:

A __reduce__ implementation can direct protocol 2 to use the more efficient NEWOBJ opcode, while still allowing protocol 0 and 1 to work normally. For this to work, the function returned by __reduce__ should be called __newobj__, and its first argument should be a new-style class. The implementation for __newobj__ should be as follows, although pickle has no way to verify this:

def __newobj__(cls, *args):
    return cls.__new__(cls, *args)

Protocols 0 and 1 will pickle a reference to __newobj__, while protocol 2 (and above) will pickle a reference to cls, the remaining args tuple, and the NEWOBJ code, which calls cls.__new__(cls, *args) at unpickling time (see load_newobj below). If __reduce__ returns a three-tuple, the state from the third tuple item will be pickled regardless of the protocol, calling __setstate__ at unpickling time (see load_build below).

Note that no standard __newobj__ implementation exists; you have to provide your own. This is to enforce compatibility with Python 2.2 (pickles written using protocol 0 or 1 in Python 2.3 should be unpicklable by Python 2.2).

Like this post? Digg it or Del.icio.us it.

Posted by Fuzzyman on 2008-11-03 14:59:54 | |

Categories: , Tags: ,


Vista UAC and Registry Fun

emoticon:waffle A few days ago I was trying to create a Windows MSI installer that set a registry entry computed at install time. Naturally, in order to minimise intrusion and permissions difficulties, I decided to use HKEY_CURRENT_USER (the per user registry hive) rather than HKEY_LOCAL_MACHINE (the machine wide hive that needs admin privileges to write to [1]).

I was using Wix to build the msi. Wix involves plenty of XML joy to use, but once you have it setup (with most of the XML generated from Python scripts) then it isn't too bad. You can embed an executable to be run at install time as a CustomAction, which is ideal for something needing to be computed at install time.

The magic in your wxs template to create and execute a custom action looks something like:

<Binary Id="MyExecutable" SourceFile="MyExecutable.exe"/>

<CustomAction Id="MyExecutable"
          BinaryKey="MyExecutable"
          ExeCommand="command line arguments"
          Return="ignore"
          HideTarget="yes"
          Impersonate="yes" />

<InstallExecuteSequence>
    <Custom Action="MyExecutable" />
</InstallExecuteSequence>

You can use a scripting engine, with either VB or JScript, instead of embedding an executable. Everything on the intarwebz I read suggested that this was a bad idea. Many anti-virus programs decide that scripts inside an installer are dangerous and disable them (whilst running an executable is somehow fine), plus it depends on the user not having a broken scripting engine on their machine.

For this to work straightforwardly you need the Impersonate="yes" line in the CustomAction element. Installers don't run as the user who launched the installer - but as some magic user called 'Local System' instead. Impersonate makes the executable run as the user.

All is fine if the user running the installer has admin privileges. Installing anything into C:\Program Files requires admin privileges, so if the user has UAC switched on (Vista only) then it will warn them - and then install fine, writing the registry entry correctly.

If they don't have admin privileges, but work in (say) a corporate environment, then the install will require an administrator to come and enter admin credentials. (Unless they run not as an admin user but with UAC off - in which case they are stuffed. Vista won't prompt and won't install.) When the administrator enters their credentials then instead of temporarily elevating their permissions for the install (which Windows is well capable of) - instead it switches the installer to run as the admin user instead.

As a consequence the registry entry is set for the admin user and not for the user who launched the installer in the first place. As far as I can tell there is no way round this, and the only reliable thing to do is to write a machine wide registry entry instead. sigh

To ensure that this works you need to tell Wix that your embedded executable requires elevated privileges. You do this by setting Impersonate="no" along with Execute="deferred" (don't ask me why [2]). This makes the Custom Action entry in the wxs XML look like this:

<Binary Id="MyExecutable" SourceFile="MyExecutable.exe"/>

<CustomAction Id="MyExecutable"
          BinaryKey="MyExecutable"
          ExeCommand="command line arguments"
          Return="ignore"
          HideTarget="yes"
          Impersonate="no"
          Execute="deferred"/>

<InstallExecuteSequence>
    <Custom Action="MyExecutable" Before="InstallFinalize" />
</InstallExecuteSequence>

This works reliably, but if anyone has any better techniques for achieving the same goal then I would be glad to hear them.

[1]Thanks to this blog entry for setting me on the right track.
[2]Actually, if you attempt to write to it without the right privileges I think Vista will virtualise (except on 64bit systems) - meaning that attempting to access the key later from another process will fail.

Like this post? Digg it or Del.icio.us it.

Posted by Fuzzyman on 2008-11-03 14:36:53 | |

Categories: , Tags: , , , , ,


Hosted by Webfaction

Counter...