Python Programming, news on the Voidspace Python Projects and all things techie.
Resolver on Slashdot
Well, it happened. The John Udell Screencast of Resolver got slashdotted.
Lots of comments (as is the slashdot way) and much franticness in the Resolver offices ensued!
We've had lots of sign-ups for beta program as a result. Jon's website held up pretty well. The screencast was unavailable for a while, but the blog stayed up.
It is an interesting time for Resolver. We've had some good interest from investment banks, and it looks like some of these will become customers sooner rather than later.
We've also had some contact with people who are already doing lots of data analysis with Python, where Resolver provides an almost ready-made solution to problems they have right now. Some of these companies are in the bioninformatics world, which is about as far from our target market as we can imagine! Hopefully Slashdot might help us find a few more of these.
One thing that comes as a natural consequence of having your whole spreadsheet as Python code, that didn't get mentioned in the screencast, is that it is very easy to define new datatypes and use them in formulae. By implementing __repr__ they can control how they appear in the grid, and through normal operator overloading you can control how they behave in formulae. There is an example of this, with a magnitude datatype, on the Resolverhacks website.
Jon Udell: First look at Resolver
Jon Udell has just posted a blog entry, with screencast, about Resolver:
This was recorded yesterday(!) with Giles (the boss) and me. The audio was recorded over Skype, so it could be better, but I think the character of Resolver still manages to come across.
As a Python programmer Jon is very taken with Resolver. He particularly likes the fact that you can store arbitrary .NET / Python objects in the grid!
If you are interested in trying Resolver, you can still sign up for the beta program on the website. We are gradually widening the beta as fast as we can support the new users.
PyCF_DONT_IMPLY_DEDENT: Python Interactive Interpreters
There are two obstacles in the way of creating an interactive interpreter that runs in the browser, with Silverlight. Both of them are due to bugs in IronPython 2, and hopefully (now that the IronPython team are aware of them) they will both be fixed in the final version of IronPython 2 / Silverlight. This is a very geeky blog entry about what I learned in the process of uncovering these two bugs.
Both of them are related to the way that the normal Python compiler (exposed via the built-in compile function) supports the Python interactive interpreter.
compile turns Python source code into a code object that you can then pass to the exec statement to execute. Interestingly we got a massive speed benefit by re-executing compiled code objects with IronPython, which we attribute to the JIT .
The signature of compile is as follows:
compile(string, filename, kind[, flags[, dont_inherit]])
kind should be one of 'exec', 'eval' or 'single'. 'single' exists specifically to support the interactive interpreter. When you execute an expression (expressions return values) in the interactive interpreter, that value is echoed to standard out. When you exec a code block that ends with an expression, that doesn't happen unless you are exec'ing a code object compiled with 'kind' of 'single'.
Support for creating interactive interpreters from Python code is provided by the standard library code module (which in turn uses the codeop module under the hood). In my Silverlight Interactive Interpreter this doesn't happen. This is because in IronPython 2 the support for 'single' was implemented but not wired in!
You can demonstrate this with:
CPython and IronPython 1.1:
>>> exec code
>>> exec code
Fixing this in IronPython 2 is basically a one line fix.
Unfortunately, this problem notwithstanding, the code module still doesn't work with IronPython 2. This is because codeop compiles code using a mysterious flag PyCF_DONT_IMPLY_DEDENT. This flag isn't recognised by IronPython and so it throws a ValueError: Flag not recognized when you try to use the interpreters from the code module. FePy tries to get round this by simply patching this flag to be zero. Unfortunately this doesn't work, as you will discover if you try and create functions will multiple indented lines in the body in the Silverlight interpreter.
I now have a working prototype of an interpreter operating inside a a single textarea - all the text before the last prompt is read only and when you hit enter the correct text is sent to the interpreter as the current line (working for Firefox and IE). Unfortunately until the next bug is fixed it is worthless.
PyCF_DONT_IMPLY_DEDENT is defined in codeop as 0x200, but nowhere is it documented or explained. The IronPython team said that they would be perfectly happy to implement support for this flag if I could tell them what it did!
After a bit of experimentation I came up with the following difference:
>>> def compile2(source):
... return compile(source, '<input>', 'single', PyCF_DONT_IMPLY_DEDENT)
>>> def compile1(source):
... return compile(source, '<input>', 'single')
>>> source = 'def x():\n print x'
<code object ? at 00B3E720, file "<input>", line 1>
Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "compile_stuff.py", line 4, in compile2
return compile(source, '<input>', 'single', PyCF_DONT_IMPLY_DEDENT, 1)
File "<input>", line 2
SyntaxError: invalid syntax
In case you struggle to see the difference, the source 'def x():\n print x' is not terminated with a newline. By default, compile will compile this to a code object fine - as shown by the call to compile1.
With the PyCF_DONT_IMPLY_DEDENT flag used (in compile2), the same code produces a syntax error. My understanding is that the code block is not complete without a terminating dedent - which by default is implied for us, but when we are in the interactive interpreter we don't want this implied. The flag (as the name suggests) turns off this implied dedent.
The reason we don't want this implied, is so that we can tell when we are waiting for more input (the interactive interpreter gives you a different prompt depending on whether it is waiting for more input or not). What the code module does is compile with the PyCF_DONT_IMPLY_DEDENT flag. If this raises a SyntaxError, it tries again but adding a '\n' to the end. If this then compiles without error then it assumes that it should wait for more input before executing the code it has accumulated so far.
(It also has a way of checking for incomplete expressions - it compiles one more time with two terminating '\n's. If the syntax error is due to an incomplete expression rather than invalid input then the lines reported in the syntax error exception instances will be different!)
The upshot is that I need this flag implemented before I can get an interactive interpreter working properly in the browser! And the reason I'm blogging all this? Well the intarwebz has virtually nothing to say about this obscure flag, so unless my experimentation proves wildly innacurate it may prove useful to the next person who wonders what it is for. OTOH if it is wildly inaccurate, hopefully someone will let me know...
|||Jim Hugunin confirmed that it was likely that the JIT would be able to provide an effective optimization when going through the same code path in a compiled code object, even with different locals.|
Tail Call Optimisation
How hard would it be to implement tail call optimization for Python? If a return statement inside a function exits with a function call, then this function call doesn't need to return to the calling stack frame but can return directly to its caller. I guess we need to keep the frame alive in case an exception is raised, but it doesn't sound like a particularly difficult problem. (Perhaps it would be easier to do with Stackless ?)
The .NET framework has a specific 'tail call' operation in bytecode to support tail recursion for languages like Nemerle. I guess the main question is, is it worth the effort that would be required to implement this for Python? (The Python bytecode compiler could make the decision as to whether a function call is a tail call or not.)
The advantage of optimised tail call recursion is that the last call in a deeply nested recursive operation doesn't need to jump back through all the individual stack frames, but instead can jump straight back to where the result of the top-level call is needed. You can also avoid blowing the recursion limit if you don't need to keep stack frames around.
Gadgets, Python, Community, Music, Talks and Online Apps
A quick update on 'stuff'. I'm back working on the book, so no time for big entries.
DDD day is a .NET 'community' event. In general there is much less of a sense of community amongst .NET developers (a much higher proportion of 'day job' programmers) than with Python developers, but it is there and seems to be growing. The London .NET users group is about the same size as the London Python users group, but there are many more professional .NET developers in London than Python developers. The .NET guys I have met have been good folk though, and are interested in IronPython which is a good thing.
I've just started using Mozy Backup, this is one of a very few online services that I pay for (although at $4.95 a month for the personal service it is very cheap). It does silent online backup in the background and seems like a very unintrusive solution to the backup problem. Most of my data is backed up in several other ways too, but this seems like a very useful service. Its only drawback is that the first backup sucks up bandwidth in the background for quite a while, changes I make to the configuration won't then happen until this first backup has completed (at which point it will start again...). Only a minor nit.
A web application I don't pay for is Facebook, but I have friends from about six different social circles who use it - so I find myself turning to it several times a day (it certainly isn't perfect, but is 'good enough'). There are rumours that google are working on an open social networking platform. Whilst this is good news, I wonder how it will get the traction that facebook already has.
Pirate Bay has uncovered evidence that the music labels have been using illegal tactics against them and are turning the legal system against them. Good.
Patrick Logan has a blog entry on Software Transactional Memory, specifically a concrete implementation in a C/C++ compiler from Intel. STM was the subject of the most memorable talks at Mix UK, by Simon Peyton Jones (from the Cambridge Microsoft research Labs). Whilst most of the challenges of STM are in the compiler implementation (some of which could be moved to the hardware), it still introduces new language constructs. STM is a concurrency technique without locks (wrapping operations as transactions that don't modify shared objects until an operation completes - similar to a database transaction). I wonder if this technique could be an answer to the current concurrency debate in Python (although Patrick Logan is 'uninterested' in STM). Transactions could release the GIL allowing them to operate concurrently in OS level threads. This would require a special build of Python.
IronPython in Action is currently the #2 highest selling book in the Manning Early Access Program. Whilst the absolute figures aren't high, this is a great start. I'm now working on the chapter on testing, an area where dynamic languages really shine.
My boss and I have recorded a podcast with Ron Stephens for Python 411 (about Resolver and IronPython). Hopefully that will go online any day now. This week we are doing a screencast with Jon Udell, which is exciting. As soon as any of these go online I'll supply URLs of course (still waiting for the video and podcast from Mix UK to show up).
After a long time, the Chandler Project has produced a 'usable' release. I'm afraid a calendar is one of the few apps that I must have online in order to be useful, and installing / configuring the server seemed to be 'non-trivial' so I'm not using it yet. I did fire it up on the Mac and I have to say that it is a very good looking piece of software though.
The One Laptop Per Child project is finally launching their Give 1 Get 1 scheme. This is great and I will sign on the dotted line at the first possible opportunity... Fake Steve Jobs never misses an opportunity to take potshots at the freetards behind OLPC, but it is reassuring to remember the PyCon keynote from earlier this year where Ivan told us that they have smashed through so many 'impossible' problems that impossible is no longer a barrier. In this entry fake Steve manages to target both OLPC and Chandler in one entry!
Speaking of gadgets, the iPhone is arriving in the UK in November. My phone contract is up at the end of September and I do carry an iPod around with me, so it would be really nice to combine both in one device. The price (£35 a month contract for 18months) is better than expected as well (but more expensive than in the US of course) and does include unlimited data. Unfortunately an 8gig iPhone wouldn't hold all my music (16gig would be enough) and I would really prefer a 3G mobile web connection. I can get a standard phone contract for £15 a month, with enough minutes / texts but not including data, so I am torn... I don't really need it, but if I get a normal phone with data on top it might not actually be much cheaper. On principle I think it might be better for me not to buy 'the latest gadget', but wait a year. We'll see if I can hold out.
This work is licensed under a Creative Commons Attribution-Share Alike 2.0 License.