Consider the following procedure from Sphinx Adventure. It contains three loops, one on line 271 formed with a GOTO back to the start of the line, and two REPEAT .. UNTIL loops.

266 DEF PROCL(L)
267 LOCAL I,J:CO=0:CN=0
268 IF L=1 THEN278
269 PRINT': RESTORE L: IF O?31<>0 THEN O?31=0:DW=1
270 READ R$,R$:R$="You are "+R$
271 IF LEN(R$)+ POS>CO-CN+39 THEN R$= FNS(R$,39+CO-CN):CO=CO+39: GOTO271
272 PRINT R$: IFL=136 OR L=15 THEN O?56=L
273 IF L=16 AND FL=1 THEN PRINT"The walls are very hot!" ELSE IF L=16 THEN PRINT "The walls are steaming!"
274 IF L<>3 AND L<>142 AND L<>143 THEN PROCEX(L): IF ABS(L-19)=1 AND CH=1 THEN PRINT ELSE IF ABS(L-42)=1 AND VO=1 THEN PRINT
275 IF CH=1 AND ABS(L-19)=1 THEN PROC R(22): PRINT "chasm.":O?53=L
276 IF VO=1 AND ABS(L-42)=1 THEN PROCR(22): PRINT"glacier.":O?53=L
277 IF L=26 OR L=27 THEN O?53=L
278 J=0:I=0:CO=0
279 REPEAT:J=J+1: IF O?J=L  THEN CO=CO+1
280 UNTIL J=52: IF CO=0 AND L=1 THEN PROCR(L): GOTO 284 ELSE IF CO=0 AND L<>1 THEN 284 ELSE PRINT:MAX=CO
281 IF L=1 THEN PROCR(3) ELSE PROCR(4)
282 CN=0:CO=MAX: REPEAT I=I+1: IF O?I=L  THEN PROCOT(I,CO):CO=CO-1
283 UNTIL I=52
284 IF D<>0 THEN O?31=L
285 IF CF=1 AND L=94 THEN PRINT'"The casket is open."
286 IF L=24 AND SA=1 THEN PRINT'"The safe door is open."
287 PRINT: ENDPROC

The CFG for this code is shown below. Each program statement is shown as a purple box, with control flow to the following statement(s). Conditionals are shown in diamond boxes. The numbers in each purple box are source line numbers, where known.

Careful comparison of the source above and the diagram below will reveal some of the transformations that have been applied to the program; for example, READ R$,R$ on line 270 has been transformed into two consecutive assignment statements which actually take the form R$ = READ() where READ() is a function not available in the source language.

The statement level CFG has been analysed to identify basic-blocks, shown as yellow group nodes, thereby defining a higher level basic-block level CFG. Each basic block has only one entry point statement; none of the statement within the basic block are destinations of other jump instructions. Furthermore, each block has only one exit point.

More text follows this long diagram…

Generating the CIL code for a single basic block is straightforward enough – we can simply iterate through the statements comprising the basic block in order and generate the code for each in turn. However, there are many possible orders in which the code for the basic block themselves could be representing in the CIL, since we can branch from the end of any block to the next block, although of course we must start at the entry block for the procedure. Although any order starting with the entry block can be made to work, where possible we would like program control to flow smoothly from the end of a block to one of its successors without requiring a branch.

At first sight, some sort of topological ordering would seem to be appropriate, but a topological ordering is only well defined for a directed acyclic graph (DAG), and a DAG this program is not. The key to this conundrum is to reduce the directed graph to a DAG by identifying strongly connected components. By contracting each SCC to a single node we obtain what is called the condensation of the CFG which will be a DAG. To the resulting DAG we can apply a topological ordering. The ordering of vertices with each SCC is chosen by starting at the vertex with the greatest in-degree.

In order to identify and contract the SCCs we use an implementation of Tarjan’s algorithm during depth first traversal of the CFG. The reverse post ordering of the primary depth first traversal is used to generate the topological ordering of the condensed CFG.

The resulting ordering of basic blocks is shown in the diagram by the numeric labels to the top-left of each. This will be the order in which the CIL code for them is generated, and it can be seen that in about half of the cases, fall through from one block to the next (consecutive block numbers) without explicit branching can be exploited. Future optimisations will focus on further simplifying the generated code by removing vertices, such as block 31, which contain only jumps.

Since then, I’ve had time to reproduce the results with a small and completely unremarkable Python program, and also to run the tests on a different system. I had suspected that in the OWL BASIC compiler, my Python visitor implementation, which is used in applying transformations to the abstract syntax tree, was to blame. I set about condensing a tree visitor down to a small example, but I never got that far. It is sufficient to simply build a large binary tree to demonstrate the dramatic differences in the performance characteristics of the three main Python implementations.

The benchmark

Here is that test program, which just builds a simple binary tree of objects to the requested depth.

class Node(object):
    counter = 0

    def __init__(self, children):
        Node.counter += 1
        self._children = children

def make_tree(depth):
    if depth > 1:
        return Node ([make_tree(depth - 1), make_tree(depth - 1)])
    else:
        return Node([])

def main(argv=None):
    if argv is None:
        argv = sys.argv
    depth = int(argv[1]) if len(argv) > 1 else 10

    root = make_tree(depth)
    print Node.counter
    return 0

if __name__ == '__main__':
    import sys
    sys.exit(main())

The program builds a binary tree to the depth supplied as the only command line argument, or ten if one is not supplied. It counts the number of nodes as they a built. Remember that the merits or otherwise of this program are not the point! The point is the performance difference between the Python implementations when it is run.

My benchmarking approach has been to run this script five times for each tree depth from a depth of one, upwards to 22, or until my patience was exhausted. I’ve taken the minimum time from each run of five. Since there is a non-linear relationship between the depth of the tree and the number of nodes contained therein, logarithmic axes are used in all the charts that follow.

64 bit Windows Vista x64

Here are the results for the first test machine – with dual quad-core 1.86 GHz Xeons with 4 GB RAM running Vista x64, testing IronPython 2.0.0.0 on .NET 2.0, Jython 2.5rc2 on Java Hotspot 1.6.0 and Python 2.5.2.

Figure 1. Creation time for a binary tree including Python virtual machine startup on Windows Vista x64 with 1.86 GHz Xeon processors.

In Figure 1 we see that above 1000 nodes or so (tree depth of 10) performance for IronPython begin to degrade rapidly. CPython holds out for another two orders of magnitude before the significant costs begin to be felt . Its interesting to see that although Jython is in the middle of the pack, it scales much better than CPython, surpassing it at around half-a-million nodes (tree depth of 19).

In my application — a compiler — virtual machine (VM) start-up time is important; however, in many long-running applications this is not the case, so it is interesting to subtract VM start-up time from each series, which we see in Figure 2, below.

By subtracting VM start-up time, we get a picture more interesting for long-running processes.

Below 100 tree nodes, there is a lot of noise in these measurements. Above 100 nodes its easy to see that the blue IronPython curve is at least two chart divisions above the red CPython curve — that’s two orders of magnitude or 100× slower, and getting relatively worse as the size of the tree increases.

32 bit Windows XP x86

Responses to my earlier article suggested that trying IronPython 2.0.1 with Ngen’ed binaries on x86 may make a difference. Well, to cut a long story short, it doesn’t, but here are the details. These tests were run on a 900 MHz Pentium M Centrino laptop with 768 MB RAM, and so cannot be directly compared with those above, although its notable that a one year old workstation is only twice as fast as a five year old laptop. Moore’s law certainly isn’t delivering here!

The performance profiles are very simular with IronPython 2.0.1 on x86.

On x86, IronPython is still 100× slower than CPython, and Jython still scales better. It seems the essence of this benchmark is not dependent on which hardware or CLR platform it is run.

I’ll close by re-presenting the data in the x86 benchmarks as multiples of CPython performance, because it dramatically demonstrates the different responses to the scale of the problem size for IronPython and Jython. Again we see Jython catching up with CPython at a tree depth of 19, just we saw on x64. and IronPython delivering 6000× worse than CPython at a tree depth depth of 15. A tree of this size with thirty-thousand nodes is very similar in scale to the AST tree sizes found in the OWL BASIC during compilation of large programs.

Performance of IronPython and Jython as multiples of CPython performance.

Conclusions

• IronPython can be very slow, even on programs in the microbenchmark category, which are doing standard operations such as building trees. Presumably there are still significant optimizations to be made in IronPython to bring its performance closer to that of the other Python implementations. Hopefully, this example and the measurements can contribute to that improvement.
• Jython may scale better than Python if your application exercises Python in similar ways to this benchmark. Speculatively, that could have implications for projects such as SCons, which build large in-memory graphs.
• I suppose if nothing else we have demonstrated in passing that Java can be faster than C for some non-trivial programs (like a Python interpreter) running a trivial program, like this benchmark.