My First Screencast: Column Level Formulae in Resolver One

I've finally created a screencast. It's 2mins36 seconds and is about using column level formulae (one of the new features of version 1.3) in Resolver One. (Bonus points for working out where the names come from - but not many because it's easy.)

Column level formulas are cool to avoid duplicating formulae, but you can use them to do some interesting things like data aggregation in combination with list comprehensions; which is the subject of this screencast.

It actually took me two days of work to create this screencast! About half a day to create the spreadsheets, half a day to write and practise the script, and then a day of recording the audio and video plus editing. The next one I do will take considerably less time. Surprisingly (to me) recording the audio and video separately, and then editing them to fit together, worked very well. The trick is to know the script well enough that you don't have to read it (and therefore don't sound like you're reading a script).

For the audio I used Audactiy, and for recording the video Camtasia. Camtasia is extremely simple to use, but pretty limited in the editing that you can do.

Needless to say, it isn't 'studio quality', but it was fun to make and hopefully informative. The docs have more information on Column Level Formulae, and Jonathan did a great screencast on the other new features in Resolver One 1.3.

Ian Ozsvald of Showmedo is also doing some screencasts for us. They're a lot better than the one I did! The one on Python Objects in the Spreadsheet Grid is particularly good.

---

Updating 1st Gen iPhone to Unlocked and Jailbroken 2.0

Around October of last year a bunch of us at Resolver Systems got imported iPhones and promptly jailbroke and unlocked them. The unlock process at the time was fairly arduous, so most of us have been running the OS version 1.0.2 since then.

When the iPhone 3G came out I was sorely tempted, but my phone contract isn't up until next year. So last Sunday, when I should have been doing more sensible things like making edits to chapter one, I spent a frustrating few hours upgrading my first gen iPhone from 1.0.2 to 2.0. The first two hours were spent trying things that didn't work, so in case I save anyone two frustrating hours here's how I did it.

This is a Mac only solution. It was made complicated by the fact that the jailbreak we used rewrote the baseband to do the unlock. In order to upgrade I needed to fully 'revirginize' the phone, including restoring the baseband firmware.

The steps are pretty straightforward. It did take two hours, but most of that was unattended. To do the 'virginizing' (including restoring baseband) I followed the hackthatphone flowchart:

• iPhone modders' flowchart (first generation iPhones only)

I started at the Downgrade to 1.0.2, jailbreak and activate step. This requires the 1.0.2 OS, a specific version of iTunes (7.4.2) and a specific version of iNdependence (the Mac hacking software). Fortunately the flowcharts gives links to all of these.

In order to install the older version of iTunes you'll need to blow away MobileDevice.framework. I would also rename your iTune library folder so that you can rename it back once you have reinstalled the latest version of iTunes again.

Whilst I was running iNdependence it would periodically hang on me. Killing it and restarting it worked fine (and is the advised solution in the README!) - so long as you see the screens shown on the flowchart you know you are going in the right direction.

Once I had restored 1.0.2 and the original baseband I could upgrade back to iTunes 7.7, which promptly offered to upgrade my iPhone to 2.0. This failed (but downloaded the 2.0 OS image) as my SIM card wasn't supported - it needs to be an O2 SIM and needs to properly activate again.

This is where the PwnageTool comes in. I used PwnageTool 2.0.1, but you will also need bl39.bin / bl46.bin, which are easy enough to find via google. Some people have had problems with this, so at your own risk...

I followed a neat (and short) video tutorial on using the PwnageTool expert mode: PwnageTool Expert Mode

This creates a custom OS image for you that iTunes can 'restore' to your phone. Once loaded it activates and unlocks the phone and includes an 'installer like' application called Cydia. There is a new version of installer in the works, and many of the applications for jailbroken 1.x iPhones are still being ported over: Installer 4.

The new OS is great, with some nice touches like you can always see the last character in a password text box. This is useful because even though I am getting used to typing on the iPhone (I prefer texting on the iPhone to an ordinary mobile phone with a numeric keypad) it is hard to be 100% accurate. I've not noticed any bugs, but there is the occasional pause when opening applications.

As for applications, well. With 1000 apps in the AppStore, most of them are crap. Of the ones that aren't, or may not be, crap - most of them don't interest me. Even so, there are still some very interesting ones and they are a huge step-up from the 1.x apps.

I have only paid for two applications:

• Crash Bandicoot Nitro Kart

  I've only just got this. So far it looks fantastic.

• Solitaire

  This is the version of Solitaire that I used to play a lot (for free) as a jailbroken app on my previous version. It's very cheap (about $5) and has had two games added. It has actually lost a couple of features for the Klondike game I play (I don't use the others), but it is still very good. Solitaire and as an iPod is all I was using my iPhone for before this upgrade - now it's my main phone.

I also have a few free apps. My internet connection on my phone (an extra ten pounds a month for allegedly unlimited internet use with Orange) doesn't get switched on until next Saturday. It will only be after that that I work out which applications are really useful. The ones I am using so far are:

• Twinkle

  Twitter client, very nice.

• JirboBreak

  A 'breakout' style game. It has a nifty control mechanism, an on screen thumbpad - you roll your thumb left and right to control the paddle and the game pauses if you take your finger off the screen. It took me about three games before I really got the hang of it, but it's a great implementation of breakout and very addictive.

• Stanza

  Nice ebook reader with online repository of free books to download. Good stuff from Cory Doctorow included!

• The Facebook App

  Straightforwardly good.

• The google app

  This just redirects you to Safari for most things - but it does include google chat.

I've also installed MoPhoto and ShoZu for uploading images to Flickr and Facebook (etc) and viewing them as well. I haven't really tried them much to see if I really need them. Naturally I also have a terminal from Cydia and will be trying this technique (3proxy) for tethering an iPhone internet connection to a laptop (mine is edge rather than 3G but still worth it).

---

Error'd: To help protect your computer...

I feel very protected...

---

Report from the RuPy Conference

This was the second year for the RuPy Ruby and Python conference in Poznan, Poland. Last year there were 80 attendees and this year the conference had grown to 250. The conference is not only a great concept (great to see Ruby and Python grouped together rather than pitted against each other), but there were a high number of quality talks.

My talk was on IronPython and Silverlight 2 and it went great. I felt confident with the material, the audience laughed in all the right places and all my demos worked! (There were 80~100 in the audience - I'm rubbish at guessing numbers.) I think the talk was improved by showing two good looking Silverlight applications from the gallery. These were Tetrislight and the Image Snipper, both of which do a much better job of showing what Silverlight is capable of than my 'developer oriented' examples! (But you'll learn more from mine of course.)

As usual with a Python crowd the biggest concern was the Linux compatibility story (I believe that Moonlight will prove to be a very good 'Silverlight 2 on Linux', but time will tell). As for the adoption story, Silverlight 1 is currently being downloaded at a rate of about 1.5 million per day and is already being offered as an 'optional update' via Windows update. I'm sure that Microsoft will do the same with Silverlight 2, meaning that not long after the final release it will probably find its way onto ~90% of the world's desktop computers! Microsoft have a deal to put Silverlight 2 on the official Olympics website this summer. This gives them both a hard deadline for final release and a great driver for Silverlight 2 adoption.

The infamous Zed Shaw was there and gave a very tame talk on how to get statistically meaningful results when testing web app performance. He is a great speaker, and he did have a dig at Rails with an utterly unfair performance comparison between Rails and web.py as his example. Naturally web.py, which does a lot less, could handle ten times as many requests per second.

Interestingly, he used R to turn the raw data into meaningful results and plots. R is an odd looking language, but it did look like a good fit for the task. Of course many Pythonistas would prefer him to use matplotlib...

I missed Jay Field's talk on Domain Specific Languages. Jay is renowned in the Ruby community, both for his work with DSLs and his focus on testing. The reason I missed the talk was because I was talking to Zed (he is loud - but he isn't an asshole). As he currently works at a large financial institution I thought I would show him Resolver One. However, as that organisation is Bear Stearns I doubt much will come of it.

I did ask Zed if Python is a Ghetto. He said no.

I spent quite a bit of time talking to Carl Friedrich Bolz [1], who is a core PyPy developer and one of the writers on the PyPy Status Blog. One of the things that Carl has done is write a framework so that PyPy Garbage Collectors can be written in Python - plus he wrote the first two GC implementations. Armin has just written a new one that is more efficient - as it can avoid copying large objects - by subclassing one of the existing ones.

Carl did an excellent talk on the current state of PyPy and particularly the recent work he has been doing on the Just In Time compiler. The PyPy JIT is a tracing JIT (like Tamarin), and as well as speeding up operations on the basic types it will be able to optimise method calls (etc) which are one of the bugbears of Python performance. Techniques it uses include not generating intermediate objects that will always be thrown away and even keeping frame objects in registers where possible (without any semantic changes to Python). Very clever stuff.

For those few who still don't understand the purpose of PyPy, it is much more than just 'Python implemented in Python'. PyPy is a language agnostic compiler toolchain. Typically it is used for compiling interpreters that are written in RPython, a static subset of Python. Interpreters are analysed and compiled by PyPy, which can emit C, Java or .NET bytecode (and various other actual and possible backends).

It also compiles garbage collection and a Just in Time compiler for the interpreter (although the GC is not needed for the .NET and JVM backends). Other interesting things can be done, like experimenting with new GC and object space implementations. So PyPy is a compiler toolchain - with swappable front, middle and backends! As well as this, there is a highly compliant Python interpreter written in RPython. When translated to C it currently runs at around half the speed of CPython, but it is already faster for highly 'memory bound' operations (because the GC is better).

As the JIT makes real progress (it is the subject of Carl's thesis - so real progress is being made), some interesting things become possible. First of all, PyPy is likely to become faster than CPython. I've long joked that so far PyPy's greatest contribution to the Python community is to kill off Psyco and Stackless [2]. It finally looks like it won't be very long before I have to stop saying this.

One fun, and plausible if not very likely, consequence is that it would be entirely possible to write a Ruby interpreter in Python (RPython of course)! That might make a few flame wars moot! Porting Zope to different Pythons is an article that explores some of the more likely consequences.

On the Saturday night was the geek party, which was great fun and I had entirely too much to drink (largely at the behest of Jonathan and Susan, so it was not my fault). Jay joined the reverie and we staggered back from the centre of Poznan at 4.30 in the morning.

Speaking of Jonathan, he gave his talk on Test Driven Development. This is an excellent talk and it gets better every time he delivers it. Yet again he failed to get his Ubuntu laptop connected to the projector though, and his fonts were screwed up. No matter though, the large audience appreciated it and asked a lot of questions. If any of you have objections to my earlier blog entry on the value of test first, you need to hear Jonathan.

On the Sunday morning Konrad woke me up at 9am. I got up, which was a mistake! It did mean I was able to demo Resolver One (during the post-lunch lightning talks, which were new to RuPy this year), but I was fairly befuddled whilst doing it (actually the demo went ok but the setting up was a bit 'confused').

There was a 3 hour 'Rails 101' session, which if I had been less knackered I would have enjoyed. Instead I spent the time talking to Carl. I nearly persuaded him that he ought to be working at Resolver Systems. It is not an entirely far-fetched suggestion as he is something of an Excel/VBA expert and a great fan of what we have done with Resolver One.

All in all it was a great conference, and if you are available next year I highly recommend it.

[1]	Who stepped in at the last minute to replace Maciek who decided to stay in the US to be with his girlfriend. Some people have funny priorities.

[2]

Both of these charges are baseless. Stackless is maintained (although not mainly by Christian Tismer any more), and due to its necessary complexity Armin would not have continued to develop Psyco even without PyPy. Although Psyco is kept 'working' it doesn't optimise language features added since about Python 2.2, and it is never likely to be ported to Python 3 or to support 64 bit builds of Python.

---

Fun at PyCon 2008

I really enjoyed PyCon. It was great that the conference has grown so much but still has a real 'community' feeling where you can stop and talk to people you don't know.

Unfortunately the boss took ill at the last minute, which left Jonathan [1] and I to man the Resolver Systems sponsor stand (we had some good conversations in the exhibition hall) and to do his talk. We survived though, and being totally unprepared even enjoyed giving the talk together.

Even more problematic, Jonathan has a Linux laptop and I use a Mac - Giles was bringing the Windows laptop! Resolver One runs fine on my Macbook under Parallels, which acted as a great stand-in until Van Lindberg came to the rescue with a loan laptop. (Many thanks Van!) This was a double rescue, as in an attempt to get his Linux laptop to work with a conference projector Jonathan managed to blow away his X-server configuration and couldn't boot his laptop an hour before his talk on Test Driven Development.

I couldn't find Jonathan before doing the Resolver Systems lightning talk though, so I did it from the Mac.

I didn't attend many talks in the end (at least I don't think I did - it's all a bit of a blur), but my favourite was Raymond Hettinger on Core Python Containers (which as it is specific to CPython wasn't really relevant to me as I'm spending most of my time buried deep inside IronPython these days).

As for the great 'sponsor controversy'... personally I didn't enjoy the sponsor keynote(s?) particularly, but not everyone agrees. I'm afraid that most of the lightning talks on Friday were pretty dull, but the ones on Saturday and Sunday were much better (even the sponsor ones). As usual the issue has been blown out of proportion, but the organisers are well aware of what worked and what didn't.

I think my talk went OK. I dashed from the talk with Jonathan on 'End User Computing and Resolver One', straight into my 'Python in the Browser Talk'. Rather than having any time to prepare myself, Chris McAvoy introduced me as I walked into the room. The audience was then treated to an undignified scramble as I tried to get my computer in a fit state to give the presentation.

The best part of the talk was showing the prototype 'Interactive Interpreter in the Browser' right at the end. This used some bugfixed IronPython binaries that Dino Viehland delivered to me the morning of my talk. As if it wasn't rushed enough already! I think that once the updated binaries, the 'interpreter in the browser' will be a great tool for teaching Python.

After the talk I gave a brief (7.5 minutes podcast) interview with Dr Dobb's Journal on Silverlight and IronPython.

There were many other highlights. I talked to a lot of great folks, too numerous to mention all of them, including folks like Maciek of PyPy (who I met at the SFI conference and will also be at RuPy):

And Dino the lead IronPython developer (he gave me some great reasons why Resolver Systems should upgrade to IronPython 2 including improved performance and startup time):

His talk was amazing (showing Django on IronPython using Silverlight). In order to maintain feature parity with Jython Dino implemented from __future__ import GIL whilst I was watching! I presented the Ironclad Overview at the 'Python and .NET' Open Space organised by Feihong Hsu (who gave a great talk on Python.NET):

After the Open Space a bunch of us went to downtown Chicago (including Mahesh Prakriya who you can see lingering in the back of the picture above and Harry Pierson who would prefer I didn't call him the new commander-in-chief of dynamic languages at Microsoft). We went up the John Hancock Tower which has an astonishing view of the city.

After this it was onto the sprints. Jonathan and I went out with a crew of the 'Python-Dev' guys to a Mexican restaurant. Never have I seen an individual so excited to find good Mexican food!

Whilst hanging around in the Python-core sprint I got to meet Mark Hammond:

The sprinting was massively my favourite part of the whole event, and deserves a blog entry of its own...

[1]	Who has blogged several times about Pyglet recently, and also did a great post on Why Python.

---

Silly Snippets

Here are some fun snippets of Python, which may or may not do what you expect.

This really just illustrates Python's object orientation. Everything in Python is an object (is an instance of an object), this includes type since types are first class objects. object itself is a type - so it is an instance of type.

Just illustrating that sometimes all is not as it appears with identity checks. (I assume that is and not are both tokens to the lexer but that is and is not are actually different operators to the parser.)

This is an interesting one posted to Python-dev recently and actually reveals a subtle bug in the way Python uses reference counting to do garbage collection. It is pretty pathological though. When the 'decref' happens the finalizer (__del__) is called - which can sometimes find a route back to the now invalid object. There is a really interesting post on Python garbage collection over on the PyPy blog: PyPy Development: Python Finalizers Semantics, Part 1.

This is silliness from Michael Hudson. He says that the primary use case is for looking geeky on IRC. Actually lying to isinstance can be useful when creating proxy classes (but you can't lie to checks that use type(something)).

Back to Mac stuff. SCPlugin is nothing like as good as TortoiseSVN (yet - it is a younger project), but dropping down to the command line is kind of reassuring. I've discovered that Parallels can be made to work with multiple monitors (it basically fakes a single display the combined width of your Mac desktop - but it works fine). Looking for a straightforward IRC client for the Mac - so far Colloquy seems fine other than its unhelpful error reporting. The most useful little application I've found is OpenTerminalHere. It puts an icon on finder windows that opens a terminal window with the current directory set to the directory you are viewing (kind of like 'command window here' on Windows).

---

More Interesting Snippets

There has been a rash of great blog entries recently. In lieu of doing a real blog entry myself, here are some quotes from a few that caught my attention:

Jeff Attwood (summarising Steve Yegge), with Size is the Enemy:

Lines of code are, and always have been, the enemy. More lines of code means more to read, more to understand, more to troubleshoot, more to debug.

He cites Martin Fowler and Bruce Eckel as examples of people who have made the switch to dynamic languages as part of the solution to this problem:

A lot of incredibly smart people like Steve present a compelling case that the grass really is greener on the dynamic side.

Bruce Eckel's entry on why he is not upgrading to Vista includes a point I've not heard before. The End of the Vista Experiment:

I'm still tied to Windows because certain software tools only work on Windows. Although OpenOffice is great, its word processor can't handle documents the size of a book (initially it would crash, now it doesn't but it runs so slowly it's unusable). MS Word is still the only word processor that can produce camera-ready documents that are book-sized (Yes, I do know about all the two-step approaches: "just write it using this one tool, and lay it out using this other one." It's only efficient when you say it like that, not in reality).

I have no axe to grind here, but this is the first time I've heard anyone say that. I've just bought Pages for the Mac and it seems to handle the word documents I use fine, but IronPython in Action is being written with one document per chapter.

Bruce Schneier has a very insightful (and pleasingly short) blog entry on airport security: Airport Security Study

A team at the Harvard School of Public Health could not find any studies showing whether the time-consuming process of X-raying carry-on luggage prevents hijackings or attacks. They also found no evidence to suggest that making passengers take off their shoes and confiscating small items prevented any incidents.

""Even without clear evidence of the accuracy of testing, the Transportation Security Administration defended its measures by reporting that more than 13 million prohibited items were intercepted in one year," the researchers added. "Most of these illegal items were lighters."

This is where the TSA has it completely backwards. The goal isn't to confiscate prohibited items. The goal is to prevent terrorism on airplanes. When the TSA confiscates millions of lighters from innocent people, that's a security failure. The TSA is reacting to non-threats. The TSA is reacting to false alarms. Now you can argue that this level of failures is necessary to make people safer, but it's certainly not evidence that people are safer.

Bruce has repeatedly asked whether there is any evidence that these ridiculous security measures are actually doing anything worthwhile. I love his message that we should refuse to be terrorised.

Scott Adams (of Dilbert fame and a fantastic blogger, but off his form recently) on why we blog: Work Pleasure

It made me wonder if other people get an actual physical pleasure from doing work, at least the kind where you accomplish something, no matter how unimportant. [...]

You have to believe most bloggers have few if any actual readers. The writers are in it for other reasons. Blogging is like work, but without coworkers thwarting you at every turn. All you get is the pleasure of a completed task.

And finally... there are some great new things happening 'on the head' with Python. Lawrence Oluyede tracks recent changes in Python SVN: Updates from Python SVN, Part 19. Lots of interesting tidbits there so I won't attempt to summarise.

(That was a week without blogging - over the last three years I have averaged six posts a week!)

---

London Geeks, Scoble and Resolver

So, Friday night I had a good time chatting to Robert Scoble, Dave Sifry and Dave Winer...

Actually it's true. They're all en-route to Le Web 3 conference and stopped off in London. A bunch of London geeks got together at a geek dinner organised by Hugh MacLeod. Giles Thomas and I meant a load of interesting people (including the alpha geeks), like Mike butcher from TechCrunch UK and Nick Falstead the founder of fav.or.it. The split was about fifty-fifty between developer geeks of various sorts and those involved in social networking (not sure what I think of people who earn a living showing companies how to leverage social networks and blogs), including a guy with a bike shop round the corner who has a popular blog on cycling.

The photowalk afterwards was cool, Dave Siffry (founder of Technorati) told us about his next big idea, but kinda long - we ended up in Picadilly around half one.

Here are the two most important pictures from the photowalk:

and:

You can see more from: Dave Sifry and Tim Watt

Yesterday Scoble interviewed three of us from Resolver. It hasn't gone up yet, but he mentioned it so he can't have thought we were too bad [1]. As we just released our beta this would be great publicity.

As this blog entry has two embarrassing pictures, it might as well have a third:

This is from a few years ago. Our Church did several nightclub events called Destiny and this was some us doing publicity in Northampton town centre. I think it was before I was married - I was certainly less fuzzy back then.

[1]	Actually he seemed very impressed. As it is outside his normal web and social networking 'field of interest' we wondered what he would think.

---

We're Going to Try Agile Programming

snigger

---

LOLCODE! on the DLR

Well, John Lam has gone and done it! He has released Martin Maly's implementation of LOLCode on the Dynamic Language Runtime:

• LOLCODE! on DLR

HAI
CAN HAS STDIO?
I HAS A FISH ITZ "Yummy"
VISIBLE FISH
VISIBLE "HAI WORLD!"

I HAS A FIB
I HAS A A ITZ 1
I HAS A B ITZ 0

GIMMEH FIB

IM IN YR LOOP
  VISIBLE B

  IZ FIB SMALR 1?
  YARLY
    GTFO
  NOWAI
    VISIBLE "NOWAI"
  KTHX

  I HAS A TEMP ITZ A UP B
  LOL A R B
  LOL B R TEMP

  NERFZ FIB!!
KTHX
KTHXBYE

To get it running, you'll need to follow these steps:

• Download the source

• Download Gardens Point Parser Generator (GPPG) and Gardens Point Scanner Generator (GPLEX) [1]:

  • http://plas.fit.qut.edu.au/gppg/
  • http://plas.fit.qut.edu.au/gplex/

• Update the LolCode pre-build event as described below

• Update reference to ShiftReduceParser (part of gppg/gplex distro)

• Download the DLR, which is part of IronPython 2.0 Alpha 6 (The binaries suffice for building LolCode, but getting the source code will let you build and debug through DLR code)

• Update the references to the Microsoft.Scripting.dll in both Lc (console) and LolCode (compiler) projects

These instructions are included in the LOLCODE distribution.

[1]	The parser is generated using Gardens Point Parser Generator (GPPG) and the scanner is generated using Gardens Point Scanner Generator (GPLEX). Both these tools are developed by QUT Australia.

---

The Dynamic Language Runtime (and a new IronPython Release)

There is a new release of IronPython out: IronPython 2.0alpha6. This fixes a lot of boring bugs [1], adds relative/absolute imports and also includes an example language implemented using the Dynamic Language Runtime: ToyScript. (It is currently part of the IronPython distribution.)

Martin Maly is the developer who (other than Jim Hugunin) has been on the IronPython team longer than anyone else. Today at TechEd he did a talk on the dynamic language runtime. Here he walked through parts of the ToyScript example, which is around 3000 lines of code and pretty clear.

The DLR basically takes care of your type system and compilation. What you need to do is provide a parser and transform your Abstract Syntax Tree into a 'DLR Tree'. You also provide a set of rules that implement your language semantics. You only need to provide rules where they differ from the default CLR (.NET) behaviour, so you don't need to implement integer addition for example. A very interesting talk and ToyScript looks like a great place to start experimenting if you have a yearning for language design.

The highlight of the talk was Martin showing off his lolcode implementation. The whole thing took him only 14 hours (whilst travelling) and is a pretty complete lolcode implementation built using the DLR. One advantage of implementing a language on the DLR is the Visual Studio support, so he showed setting breakpoints in lolcode and stepping through it. Unfortunately he needs Redmond's permission before he can release it, but he says that will probably come soon...

One of the great things that this conference has confirmed is how much use Microsoft are making of IronPython and the DLR. They are building it into an enormous amount of their products, and the new version of Visual Basic (Visual Basic 10) will use the DLR. This means that the future of IronPython is secure, at least for the foreseeable future.

[1]	Which of course is unfair. The bug fix does include some Python socket module fixes, but Seo says that there are still some issues remaining unfortunately.

---