Python Programming, news on the Voidspace Python Projects and all things techie.
There's a new (short) article on profiling IronPython code :
It shows a couple of example timers for use in profiling for the purposes of optimising IronPython code.
Compiled Code, Angle Brackets and Optimising Resolver
For the last couple of days I've been working on another performance user story.
Resolver generates and executes a lot of code (compiled byte-code objects). We'd noticed that Resolver was slowing to a crawl when there were a lot of errors in the code (hey, blame the users; not us).
After profiling just about everywhere we finally tracked it down to the call to traceback.extract_tb(sys.exc_info()) which was part of our exception handling. It seemed to be taking a bizarrely long time, and getting worse every run through.
Luckily traceback is a pure Python module, so we had a look around.
In order to pull out the line from the source code, traceback uses the linecache module. This in turn makes several calls (at least one for each directory in sys.path) to os.stat to find the source file. It does attempt to cache this, but it is a fruitless search for a code object, hence our slowdown.
linecache does include a mechanism to try and prevent this. If you compile your code objects with a name between angle brackets then it won't search the filesystem.
I was aware of the convention of compiling code objects with names like <Some Name>, but there is no mention in the documentation that this is actually useful.
As we weren't using the line value (it was never useful and we had the line number and the source code anyway), we created our own version of extract_tb which doesn't use linecache :
tb = sys.exc_info()
stackInfo = 
f = tb.tb_frame
lineno = tb.tb_lineno
co = f.f_code
filename = co.co_filename
tb = tb.tb_next
This solved part of our problem. It was still taking slightly longer each run through (for no apparent reason) and suddenly started reporting some of our exceptions as happening in impossible places. For no good reason restoring the line line = linecache.getline(filename, lineno) (even though we throw away the results) seemed to fix that problem but slowed us down again. linecache had no side effects that we could find. Mighty puzzling.
Eventually my boss spotted it. In the traceback stacks we were retrieving there was info from previous exceptions being added every time. This is why it was taking longer each time through, the traceback list was getting longer every time. It seems that all the failed stats and the exception handling in linecache was clearing the traceback stack for us.
We solved our problem with an extra call to sys.exc_clear() inside our version of extract_tb, but it looks like there is a bug in the IronPython setting and clearing of exc_info. As soon as I can reproduce the bug I'll report it.
Developing with Python and VimCE Part II
Bandung has emailed me a few more files useful for developing Python on a PocketPC device using Vim. This is part II, see Developing with Python and Vim on the PocketPC for the first exciting instalment.
The first useful file is pythonrc.py that corrects the indentation problem with the latest python25.
Place this file in the /Temp folder.
Next are two helpful utilities :
- The "Open With" utility which attaches to MS file explorer. Really useful for opening files in Vim (etc) without having to type in long path names.
- Microsoft's latest CMD and DOS console. Useful for launching programs with nifty command arguments. Programs such as SSH tunnelling, etc. I can't find the site where I originally found these files. Powertoys allegedly has them but if you want the latest version, it doesn't seem to be in there. The path environment works better with these files than Symbolic's versions.
Using my trusty "OpenWith" utility that attaches to pocket explorer, I open the python file with Vim. What's nice about this utility is that it shows the file extension, something pocket explorer can't do. I vastly prefer to use Total Commander for everything else. Once Total Commander fixes their "alternate notepad's " ability to open files on a path that has spaces (ie \Storage Card\, then I will no longer need this utility.
The Vim script used to quickly navigate around the source code is called "python menu". This script, when placed within the "plugin" directory under Vim, adds an additional Vim sub menu called "Python" as the following image shows.
Clicking on this menu affords the programmer an additional set of choices for finding the next class or function, as an example. Also of interest is the ability to change indentation.
But what is really cool is the top most menu item called Update IM-Python menu. When you click on this, Vim scans your source code and creates another menu within Vim called IM-Python. A list of all of your functions and classes is created and stored within a submenu under IM-Python. You can easily jump to whatever function you want within the source code using this feature. It also serves as a pretty nice summary of your code's structure, by the way.
Now you can easily see all of your code structure and jump to a particular function.
This feature calls for yet another vim script called pydiction. With this script placed within the plugin directory under Vim, you can now have all of your favourite code completion commands for python. You can even add to the stock completion commands by using the utility that comes with it to scan another python module and load its command structure.
To use this feature you simply start typing part of your command and then hit <Ctrl-n> for next or <Ctrl-p> for previous. You will be presented with a completed command. If it is not the one that you want, you can scroll through all of the choices that it knows about for your partial word. I use my SIP keyboard to enter the <Ctrl-n> keystroke combination. I also have programmed a hardware keyboard combination for these combos as well. They are <Tab><Tab>n and <Tab><Tab>p.
If you type "pri" and hit <Ctrl-n>, Vim completes it with "print" If there are more than one possible choices for your word, then you can continue to cycle through them by hitting <Ctrl-n> again or <Ctrl-p> for a previous choice.
The only thing left to wish for is the ability to run the python script that I am working on while within Vim. There is a script that works for a Windows desktop but I haven't been able to get it working completely for the Pocket pc.
Now if we could only get the python debugger to work within Python25, we would be in hog heaven as they say.
Many thanks to Bandung for the files and the advice.
Oh, this is my 600'th post on this blog...
More on Python Doc Tools
It turns out that my problem with PyDoctor was nothing to do with Windows. It does need the latest version of Nevow from Subversion, which either needs Twisted SVN or the soon-to-be-released Twisted 2.5.
What we are trying to achieve is to auto-generate API docs for our public API. This would form part of our reference manual for end-users, not be used by the development team. This means we don't want too much information and need to be able to exclude private methods (etc) from the output.
I tried PythonDoc. I just couldn't co-erce it to produce the documentation for our IronPython code. Even using it as a library I could only persuade it to spit out an empty file. It works by doing static analysis, not importing, so presumably it failed to resolve the .NET imports. I tried it on ConfigObj and it worked fine. PythonDoc is specialised for Javadoc style docstrings, so it mangled the ReST in the ConfigObj docstrings, but the style of documentation it generates is pretty good.
It turns out that the default HTML renderer for PythonDoc omits documenting methods without docstrings (assuming you are in docstring mode). So we could use it, probably by calling it as a library or writing our own renderer.
Endo is very good. It also works without importing and was able to generate docs for our public API without any of my hacks to make them importable under CPython. The output is nicely cross-linked and includes a class index.
Endo does include things like imported names inside modules, which I don't think we want; but it may be configurable. It does include inherited methods on classes, which is probably good, but it also includes private methods. Again, perhaps we can configure this (I didn't try).
I commented on Pudge and Epydoc in my previous entry.
We still haven't made a decision, but we will need to soon. It currently looks like the decision is between Endo and the tool I created.
Python Documentation Tools
I've been looking at Python documentation tools. We already have a working core of our own, but I've had a look at a some other options to see if they would be less work than maintaining our own tool.
I just couldn't get this to work with Windows. It depends on Twisted (which depends on zope.interface) and Nevow. PyDoctor attempts imports from Nevow which fail on the latest stable version. I couldn't get the Nevow from SVN to install on Windows. I've emailed Michael Hudson to see if he can help.
The nice thing about PyDoctor is that it uses static code analysis rather than importing. We don't want a heavy template based system but I'll try PyDoctor if I can get it working and see if it offers substantial benefits to our current approach.
This is perhaps the 'standard' Python documentation tool. It generates very comprehensive documentation, but not in a format suitable for the reference manual we want to create. (My opinion, the other Resolver developers may disagree.)
No longer maintained. The output is quite nice, but not drastically different from the tool we already have. It would be more work to customise this than to develop our current tool.
The links to colorized source code is nice, but we'd want to disable that anyway.
Any I haven't discovered ?
The tool we already have generates API documentation for packages, including docstrings.
See the Generated Docs for Pythonutils for what it does so far.
We need to decide what to do with inherited methods, sort out formatting and generate a big index page. None of these are particularly difficult. So far we're up to 360 lines of Python.
One reason to continue developing our own tool is that for the first cut of our documentation we may want the output in 'Wiki' format rather than ReST.
Anyway, I've generated our own API docs using three tools (our current one, Pudge and Epydoc) and I'll let the other developers decide which course of action they prefer.
There are two tools I haven't yet looked at. I probably ought to try both of them.
A mature tool which works by parsing source code rather than importing. It is specialised for Javadoc style documentation in comments rather than docstrings, although it does have some support for docstrings. I'll have to see how good the support for docstrings is, and whether we can live with Javadoc in our docstrings. (We don't want to maintain the documentation twice, once for the reference and once for the docstrings.)
A potentially useful feature is that it can present the parsed documentation as an XML file (and possibly an object of used as a library) so that we could completely customise the output.
I know very little about this one, but will give it a try. It is part of the Enthought Tool Suite.
The Wing Python IDE
It's a great IDE, the autocomplete is the best I've seen in a Python IDE. It certainly beats the heck out of Textpad which we use at work.
I've finally persuaded my boss to buy a copy for work, and if the other developers like it we'll soon all be using it.
One of the problems we had with switching to Wing was ironically that it is specialised for use with Python. That means that it isn't easy to configure to run files with external tools (which is easy with tools like Textpad and my favourite text editor Ultraedit).
As you may have gathered, we use IronPython, and we have our own custom executable . When we run our tests we need to set a custom executable and set the initial directory. With Wing you can set a custom executable, but this feature is closely tied to the debugger and the Python shell that is embedded in Wing. Using a non-CPython executable just plain breaks the Python shell because it no-longer works with the debugger.
Additionally setting both the executable and initial directory doesn't work with Wing because the executable doesn't receive the full path to the file. There are ways round this , but it's not pretty and the shell is still broken.
Luckily Wing has a very sane scripting API, at least that's my conclusion after playing with it for an hour or so...
Below is my script which provides a 'custom-execute' option. Place this in the 'scripts' directory of your Wing installation. Then bind 'custom-execute' to a key in the keyboard preferences. custom-execute runs the file currently being edited.
from threading import Thread
editor = app.GetActiveEditor()
filename = editor.GetDocument().GetFilename()
directory, theFile = os.path.split(filename)
thread = Thread(target=lambda: os.system('run_python "%s"' % filename))
run_python is a batch file which inserts a pause after running the file with our executable. You can replace run_python with any executable or batch file on your path.
I use the Thread so that the file is launched asynchronously, otherwise Wing blocks until your file has executed. Wing uses its own version of Python 2.4 and the standard library (so it is not dependent on your Python installation). For some reason the subprocess module isn't included so I couldn't use that.
The next step will be to get it to prompt for save when the current file is modified.
Wingware support have come up with a better solution for a custom execute.
This one will use the modified environment that you might have set up in project properties.
from wingutils import spawn
editor = app.GetActiveEditor()
filename = editor.GetDocument().GetFilename()
directory = os.path.dirname(filename)
env = app.GetProject().GetEnvironment(filename)
cmd = env.get('COMSPEC', 'cmd.exe')
argv = [cmd, '/c', 'run_python.bat', filename]
spawn.win32_start_process(cmd, argv, env=env, child_pwd=directory, new_console=True)
There is also a PyLint Integration Script. We use PyLint for various code quality checks. As it does code analysis using the compiler module (which isn't available for IronPython) this runs under CPython as part of our test-suite.
This means that compiler.parse() should work with the FePy Project.
We'd rather not have to move the FePy source into our source code control, so I wonder if we can get Seo's work (and PyLint) working with the Microsoft IronPython DLLs. Hmmm...
Seo responds that you can use his new code with IronPython as well as FePy.
The code is contained in the single file, fepy/ast.py.
To use without FePy, you would need to replace "compiler.parse()" calls with "ast.parse()" instead.
|||As an open source developer, many thanks Stephan.|
|||One of the joys of IronPython is that creating a custom executable with an embedded IronPython engine is very easy.|
|||Don't set the initial directory and set the executable to a batch file (which launches a Python script) which sets the working directory and launches the file: as well as inserting a pause which is essential when launching files externally.|
As you may recall from my previous entries I've been looking at a custom tool to generate API documentation.
The obvious place to store API documentation is in docstrings, except a colleague of mine objects  to storing user information in our production code.
We would still need to provide docstrings in order to support the Python help.
In case you think I'm madder than necessary let me explain the motivation of this post.
We're looking at generating a reference manual for our public API, using docstrings which don't yet exist.
We need to ship docstrings in order to support Python help(object).
A colleague is worried about storing docstrings scattered through our source code rather than in a single place :
"I'm worried about keeping some stuff, that is actually useful not for us but for users in the same place we keep our sources, but maybe it's not a problem given the number of methods we have there."
I'm not convinced there is a problem, storing docstrings in code seems 'normal' to me. This post is just exploring the possibility of dynamically adding docstrings at runtime, from a central store.
Storing our documentation in a separate location sounds interesting (note I said interesting, I'm not convinced). In exploring I've discovered a restriction I wasn't aware of.
You can't dynamically assign docstrings on new style classes.
>>> Something.__doc__ = 'Some docstring'
Help on class Something in module __main__:
| Some docstring
>>> class Something(object):
>>> Something.__doc__ = 'Some docstring'
Traceback (most recent call last):
File "<stdin>", line 1, in ?
TypeError: attribute '__doc__' of 'type' objects is not writable
This restriction only applies to classes, not instances.
This isn't true of properties, you can't stick docstrings onto property instances. There is a workaround. Just create a subclass :
>>> someProperty.__doc__ = 'Some string'
Traceback (most recent call last):
File "<stdin>", line 1, in ?
TypeError: readonly attribute
>>> class PropertyWithDocstring(property):
>>> someProperty = PropertyWithDocstring(3, 4)
>>> someProperty.__doc__ = 'Some String'
Having docstrings on properties isn't useful for help, but it would be useful for my API doc generation tool.
There is also a workaround for classes, you can assign to __doc__ inside a class statement :
... __doc__ = 'Some string'
... # or some imported string
... # if we wanted to store
... # them externally
So, creating a documenting metaclass with a registry (a dictionary) of docstrings would be easy enough. Hooray, perhaps a legitimate use for metaclasses.
The metaclass would look something like :
from DocstringRegistry import DocstringRegistry
def __new__(cls, className, bases, classDict):
if className in DocstringRegistry:
classDict['__doc__'] = DocstringRegistry[className]
for name, item in classDict.items():
if (className, name) in DocstringRegistry:
item.__doc__ = DocstringRegistry[(className, name)]
return type(className, bases, classDict)
|||Well, only in theory. He accepts it may be fine, but would prefer a more organised place to store all documentation.|
Video Editing Software
I have a friend who has been experimenting with creating video clips .
You can see an example of what he has been up to at :
He's been using MovieMaker which comes free with Windows. The user interface is pretty straightforward and it does a simple job well.
You can cut sections from video sources and align them on a timeline. You can then choose to use an external soundtrack or the soundtrack from the clip.
You can't mix the external soundtrack and the audio from the video clip.
We've been looking for a free program that will let us do this, without being too complicated. Unfortunately I've drawn a blank. Anyone got any suggestions ?
These are the ones we've tried :
Didn't work at all. The audio component failed to initialise, even after downloading quicktime which it seems to depend on.
Too unstable (crashed a lot). Couldn't import MP3 audio. Even after converting an MP3 to WAV we couldn't see how to set relative volumes or do audio fades.
Took a long time to import media files (mp3s and avi files). After importing them we couldn't see how to mix audio tracks or set volumes.
Didn't work at all.
Looks like it will be good. Currently no support for music tracks.
We also considered Audacity for just creating the audio tracks externally. The problem is that we want to use the audio extracted from the clips selected in Movie Maker, so it seemed like too much hassle.
If you have any ideas of programs that we could use, please let me know.
|||It's actually for our church. We show a short clip every Sunday evening.|
O2 XDA IIi for Sale
Sigh, procrastination means I've missed the boat on this one. I'm selling my XDA IIi, and I should have got it done before Christmas.
The XDA IIi is a great PDA:
- 520 MHZ clock speed
- 128 MB RAM
- WiFi & Bluetooth
- PocketPC 2003
- 65 536 colour screen (320x240 pixels)
- Built-in phone (SIM unlocked)
- 1.3 Megapixel Camera
I'm also throwing in a bluetooth compact GPS receiver (still in warranty) and various other extras.
Of course it runs Python.
This work is licensed under a Creative Commons Attribution-Share Alike 2.0 License.