Python Programming, news on the Voidspace Python Projects and all things techie.
Movable Python and Django
If you fancy running a web application from a USB stick, then you may be interested in :
Python, Multi-core Processors and Thread Oddness
I was reading an blog entry on Multicore processors or how to choose a programming language for the next 5 years. It (rightly in my opinion) asserts that the ability to take advantage of multi-core processors is going to become an increasingly important aspect of programming languages in the not so distant future.
I decided to do a quick experiment to see how Python threads fare on my dual-core CPU.
This was tried on a Windows XP SP2 box... YMMV
from threading import Thread
x = 0
x += 1
thread1 = Thread(target=BurnCycles)
thread2 = Thread(target=BurnCycles)
print 'Starting Thread 1'
print 'Starting Thread 2'
This creates two threads running a CPU intensive task (BurnCycles). It starts the first thread, waits ten seconds and starts the second thread as well.
Watching the CPU usage, python.exe takes up about 50% after the first thread starts; as you might expect on a dual core processor.
What surprised me was that when the second thread started, I was consistently seeing CPU usage drop to around 30% or so .
I guess that around the task switching Python is releasing more CPU time back to the OS. Perhaps the lazy web can explain to me what is going on here ? (The infamous GIL at work, even though it is not explicitly used in my code ?)
Does this mean that making a CPython program multi-threaded can actually reduce efficiency ?
This is one possible reason for investigating IronPython. The 'virtual machine' it runs on (the CLR) is getting a lot more work on making it efficient at running on multi-core processors...
In case you feel I'm getting at (C)Python, have a look at MSBuild WTF: "The Error Was" for an extraordinary oddity that my boss and I discovered with MSBuild whilst programming yesterday.
|||Xtian has a theory that the time taken for thread switching (OS overhead) is not being counted as CPU activity for the process: hence the apparent drop. Because of this context switching overhead; yes, multi-threading on a single processor is less efficient.|
IronPython Book or Not ?
Well, I was keeping this under the radar; but it looks like my cover is blown.
I have a proposal under review with Manning for an IronPython book, possibly called IronPython in Action.
The proposal is quite detailed and has been a lot of work (although I guess this is nothing compared with what is to come). It goes into chapters, sections and subsections, with annotations (plus more). In the unlikely  event of Manning rejecting the proposal I will punt it around other publishers or possibly publish it online.
IronPython: 1.0 or Not ?
Ever since the release of IronPython 1.0 there have been various rumblings as to whether it really justifies the '1.0 Production' tag or not.
The latest of these comes from a developer called Christopher Baus, asking IronPython are the batteries included ?.
He cites various difficulties with using Python extension modules with IronPython.
First off, to answer one of his points, the reason that some of these issues haven't already been addressed (in particular the incompatibilities with the re module ) is that they weren't flagged until someone noticed ! The IronPython team has been very good at fixing issues once they have been raised .
More to the point, IronPython is an implementation of the Python language for .NET. It is a language implementation. Fond as you may be of them, the standard library aren't actually part of the language specification. The fact that such a lot of effort has also been put into implementing Python C modules is a bonus.
(Note that most of the standard library including most of the built-in modules, does work. The problems are with external modules implemented in C. Where parts of the standard library don't work, the IronPython team treat it as a bug and fix where possible.)
This is a cause of frustration for Python programmers who want to experiment with the new platform and the possibilities it provides. Because of the large difference between the underlying platforms, taking arbitrary CPython code which makes extensive use of CPython extension modules and trying to run it on IronPython is unfortunately a recipe for pain. Thankfully some people (notably Seo) have gone through the pain barrier, and IronPython supports more and more of the CPython libraries.
IronPython is great for new projects. The .NET framework is (despite the reputation of its progenitor) a great platform and the integration is superb.
At Resolver, where the developers other than myself had little Python experience and so just enjoyed the Python syntax and philosophy on top of the .NET framework, we had very few problems of the sort Christopher describes. Resolver is a medium sized program (over 5000 lines of production code and 15000 lines of test code) and the IronPython / .NET combination has proved a great development system for us. Filing bug reports and putting workarounds in place, used to happen every week or so, but they were rarely showstoppers. Since the 1.0 release it is (fairly) rare that we find bugs.
I understand that the differences can be frustrating, but this does not necessarily point to the fact that the language implementation is incomplete.
The situation is improving. Almost daily new wrapper modules and patches appear as part of the FePy project maintained by Seo Sanghyeon. As problems are discovered the IronPython team are working there way through them, and the widening userbase means that more and more Python systems are becoming available for use with IronPython.
What is perhaps more surprising is that .NET attributes still haven't been implemented for IronPython. This is a hole in the .NET integration and means that for some things you are forced to use C#. As far as I can tell (I may be wrong), the problem is purely one of agreeing an appropriate syntax for applying .NET attributes (which are similar to Python decorators) to classes and Python attributes. Why don't they just provide some built in functions and get on with it ?
By the way, there is now a list of IronPython Related Blog Posts on the Codeplex site. This is taken straight from Technorati, but still interesting.
|||The sys._getframe problem is more difficult because unsurprisingly IronPython doesn't use Python stack frames. The issue is flagged though and the IronPython team think that they can address it.|
|||See this thread where one of my colleagues has notified the IronPython team of a difference between code objects in Python and IronPython. This is now flagged as an issue and will undoubtedly be fixed shortly.|
Movable Python and Pywin32
import win32api was failing with an ImportError: DLL load failed message.
After a bit of digging around he discovered that the problem was due to a conflict with pythoncom24.dll and pywintypes24.dll installed in the Windows/System32 directory.
The solution (from here) is simple. If you do these imports (and in this order) the problem goes away.
Back from Italy (and Stuff)
Well, I survived Italy.
Delia and I are now back, we had a great time, and I've even managed to wade through the mountain of email awaiting my return.
Rome is a great city. It has an astounding number of remains from around two thousand year ago. The closest Britain has is (admittedly superb) castles from the middle ages. Our host Ouana works as a tour guide, so by the third day I felt like my legs had worn down to bloody stumps with all the walking.
Steve Holden points out that the UK does have some impressive Roman remains like the walls of York and the baths in Bath, not forgetting the mosaics (etc) from Roman baths in St. Albans near where I used to live. Nothing approaching the scale of Rome though...
I also got to meet up with Nicola Larosa who lives just outside Rome. (The Italian train service is about four times cheaper than the UK one.) I've programmed a fair bit with Nicola, but never met him before, so this was a great bonus to the trip. Many thanks to him and his family for coping with us on Sunday.
The only blight on the visit was the six hour delay with the return flight home . The star of the moment was my Nintendo DS Lite which provided several hours of game playing with no sign of the battery dying. This was much to the disgust of my wife who was bored silly.
Anyway, other random news. There are gradually more Python jobs arriving on the Programming Job Board. Bittorrent are advertising there, so if you fancy working on one of the most well know Python applications...
My boss has a new technical blog: GilesThomas.com. So far he describes his exploits in getting an NLSU2 network drive to perform a daily backup (I have the same device). He is an experienced programmer, so there should be some good stuff appearing there.
I got some good comments on my post about threading by the way. One of the joys about keeping a blog is that you can post in your ignorance and people teach you.
|||Many thanks to Ryanair, who gave us no information and lied about the reasons on their website. As far as we can tell they sent our plane to Spain instead. Thankfully they didn't load our luggage first !|
This work is licensed under a Creative Commons Attribution-Share Alike 2.0 License.