Sufficiently Small

Significant claims have been made about the performance of IronPython, notably back at its inception in 2004 when Jim Hugunin, creator of both IronPython and its cousin Jython, presented a paper on IronPython performance at PyCon. Since then, there have been numerous claims to IronPython’s supremacy over CPython in the performance stakes. The IronPython Performance Report reiterates that IronPython can turn in a good performance. According to Hugenin the standard line we’ll see is,

“IronPython is fast – up to 1.8x faster than CPython on the standard pystone benchmark.

But do these claims stand up in the face of real-world Python code?

The claims of good performance are based on synthetic micro-benchmarks which don’t accurately reflect balance of coding techniques found in complete programs.

At this point I’d like to offer my own quote:

“IronPython can be slow – 10x to 100x slower than CPython on real-world code and it has been observed to be up to 6000x slower.

Now, the unfortunate thing about real-world code is that it hasn’t been hand-crafted to highlight the performance characteristics of some aspect of the language implementation like your typical micro-benchmark. Its been written, most likely without any attention to performance. This is especially in the case of a prototype, as in my case.

Is my code really that bad?

Over the past few years I’ve been working on a hobby project called OWL BASIC, which is a compiler for the BBC BASIC lanaguage targeting the .NET Common Language Runtime (CLR). At the outset of the project I decided to write the compiler itself in Python, for much the same reasons as PyPy is written in Python — hackability. I planned specifically to use IronPython so I could benefit from access to useful .NET libraries such a Reflection.Emit, for generating Common Intermediate Language (CIL) assemblies.

During the course of developing OWL BASIC, as the code has become more complex, I’ve been consitently disappointed by the negative delta between the promise and reality of IronPython’s performance. The poor performance of IronPython has negated, for me, one of the main advantages of developing in Python — the rapid edit-run cycle.

Was my code really so inefficient? Performance was never a goal of the project, but the underwhelming performance has threatened the viability of my approach and made me question the wisdom of my chosen route of writing a compiler in Python.

The absence of profiling tools for IronPython led me down the road of getting at least the compiler front-end to work on CPython, so I could use the standard Python profilers. Fortunately, my code was portable (its just regular Python) and so I determined that with a few inconsequential tweaks to my code, the entire compiler front-end can be run on the trinity of CPython, Jython and IronPython.

I was, to say the least, somewhat surprised by the results.

The evidence: performance of CPython, Jython and IronPython

The investigation into the relative performance of the three main Python implementations was centered on running my unmodified, unprofiled and unoptimized compiler front-end on third-party BBC BASIC source-code (Acornsoft Sphinx Adventure) and measuring the execution time. All tests runs were performed five times, and the minimum time of the five chosen. Variance between the readings on successive runs was small. The following Python implementations were on the test-bench:

• Python 2.5.1 (x86)
• Jython 2.5rc2 on Java HotSpot 1.6.0_10-rc
• IronPython 2.0 on .NET 2.0 (x64)

All tests were run on a eight-core 1.86 GHz Xeon with 4 GB RAM running Vista Ultimate x64.

The following chart shows the results of running the compiler over the source code for Sphinx Adventure.

The absolute performance of CPython, Jython and IronPython on my code

Frankly, I was astounded. IronPython was left in the dust, not only by CPython but also Jython! Overall, CPython was 93× faster on the exact same code. The IronPython hyperbole, for now I could see that was what is was, had led to me expect numbers similar to those for CPython, although I had perhaps more realistic expectations that performance would be similar to Jython.

At this point I assumed I’d hit some corner case in which IronPython was performing relatively badly. I’d had a similar experience early on in the project with code from the PLY parsing package causing IronPython 1.1 to perform badly, but I’d worked around the issue by modifying parts of PLY to use pickling rather than eval-ing large list literals for the cached parsing tables. [Its worth noting in passing that this problem still exists with the IronPython and PLY combination - I'll publish my solution in another post].

I decided to dig in a little more detail and collect some timing data on the sequence of top level calls the compiler front-end makes to parse the source code followed by a sequence of transformations to the abstract syntax tree (AST). The chart below shows these top level calls, in the order in which the occur during compilation:

The ratio of performance of CPython to both IronPython and Jython when running the OWL BASIC compiler

As you can see, I have had to resort to a logarithmic scale in order to convey the huge variation in the performance of IronPython relative to CPython, ranging from buildParser with a multiple of 3.8 to convertSubroutinesToProcedures with a multiple of over 6400. Even if we ignore this outlier (maybe we are suffering from a garbage collection – but notice that Jython is also relatively much slower on this function) we can see that Jython is typically 5× to 10× slower than CPython whereas IronPython is typically 10× to 100× slower than CPython.

Notice also that there is a marked decline in the performance of IronPython from the parse function onwards; the common factor in these operations are that they are all transformations to the AST, and my AST node classes are instantiated by a Python metaclass, although its pure speculation on my part that metaclasses are the cause of this performance drop.

Conclusion

On my program at least, Jython is 6× slower than CPython and IronPython is nearly 100× slower. If you’re suffering from poor performance with IronPython, it may be well worth your time checking performance of your code on the other Python implementations, if that is an option for you.

Now need to find the root cause and boil my problem down to a short example which can form the basis of a bug report to the IronPython team. Given that the problem is pervasive in my code, that won’t be hard.

See you at EuroPython 2009 !

Jython is currently, by far the most useful of the the three main Python implementations, Jython, CPython and IronPython. It allows access to the Java standard library which often provides more standard, better defined, and certainly better documented alternatives to much of what is in the Python standard library. It is also more cross-platform than .NET bound IronPython and offers superior multithreading support to CPython.

Its well known within the Jython community that having first-class functions in Python makes development with Swing more productive in Jython than it is with Java, by allowing us to attach functions or methods directly to Swing callbacks such as actionPerformed, without the need for anonymous classes or excessive need for interface implementation. This example taken from What’s Good About Jython? shows this in action:

import javax.swing as swing

def printMessage(event):
    print "Ouch!"

if __name__== "__main__":
    frame=swing.JFrame(title="My Frame", size=(300,300))
    frame.defaultCloseOperation=swing.JFrame.EXIT_ON_CLOSE;
    button=swing.JButton("Push Me!", actionPerformed=printMessage)
    frame.contentPane.add(button)
    frame.visible=1

The most powerful result of learning different programming languages is being able to apply concepts learned in one language, in another language. This type of ‘technology transfer’ drives progress in many science and engineering fields. This cross-fertilisation has always been possible, but is becoming both more feasible and popular thanks to multiparadigm languages such as Python and Ruby, and wider use of the STL and Boost in Standard C++ which encourage and facilitate functional and higher-order programming style. There are even books on the subject of Higher Order Perl.

Although its had many notable ongoing and recent successes (e.g. Emacs, Darcs, Pugs) the functional community is unfortunately widely perceived as academic or impractical. Indeed, Guido van Rossum has recently threatened to remove some functional programming support from Python, bizarrely whilst simultaneously adding higher-order features such as decorators. This is unfortunate because functional programming style and idioms can go a long way to improving software quality in any language – functions without side-effects are far easier to test, debug and reason about.

For example, the technique of currying - fixing a function argument and returning a new function which takes the remaining parameters can be very useful for development with the Swing UI toolkit supplied with Java.

In this example extracted from one of my own projects, where the user interface features two radio buttons, we look at how some higher-order techniques can be used to aid user-interface programming with Swing. In the code below,

fixed_radio = JRadioButton(text="Regular columns of fixed width",
        actionPerformed = self.handleFixedOption)

separated_radio = JRadioButton(text="Fields separated by spaces,
        commas, or other characters",
        actionPerformed = self.handleSeparatedOption)

two bound-methods are attached to the actionPerformed event of each button.

def handleFixedOption(self, event=None):
    self.controller.setOption(self.controller.FIXED)

def handleSeparatedOption(self, event=None):
    self.controller.setOption(self.controller.SEPARATED)

We don’t actually use the event parameter in this case, but it will be passed to the handler function we provide from actionPerformed, so we must accommodate it. These two methods are identical apart from the value passed to setOption internally, so we could factor out the common code and these two functions with a single function and pass the option value FIXED or SEPARATED in as an extra argument,

def handleOption(self, option, event=None):
    self.controller.setOption(option)

However, we now have a difficulty in that we can no longer bind our new function directly to the actionPerformed handler of the buttons, since actionPerformed expects a function which takes single event argument. We know at coding time which option value should be attached to each button – but we don’t know until the code is run, which event will be passed from actionPerformed. We need to bind the option argument before the event argument. The solution to this is currying.

Rather than writing our own currying facility, we can use on off-the-shelf implementation provided by the Xoltar Toolkit which provides some functional programming support in Python. In particular we use the curry function to curry our handleOption function into some new functions that only take take a single event argument.

from functional import curry

fixed_radio = JRadioButton(text="Regular columns of fixed width",
        actionPerformed = curry(self.handleOption,
                self.controller.FIXED))

separated_radio = JRadioButton(text="Fields separated by spaces,
        commas, or other characters",
        actionPerformed = curry(self.handleOption,
                self.controller.SEPARATED))

The curry function takes the function to be curried as its first argument. In this case we pass the bound-method self.optionHandler. The second argument to curry is the value of the first parameter to the function to be curried, which want tobe fixed to this value in the created function, for example, self.controller.FIXED. The result of the curry function is a new function which accepts one less argument than the original function – so in this case we get a new function which only accepts the remaining event parameter, and is therefore equivalent to the handleFixedOption function we started with, but which we have now avoided having to write.

The material advantage with such a simple example is minimal, but when such techniques are used throughout a project the savings in space and complexity can be substantial. The combination of functional concepts from borrowed from languages such a Haskell, facilitated by tools provided by the Xoltar Tookit, developed in dynamically-typed Jython and exploiting one of the better designed user interface APIs in the form of Swing can be concise and powerful.