I’ve been working on OWL BASIC, a compiler for BBC BASIC for the .NET CLR. The compiler itself is written in IronPython. One of the challenges of compiling BBC BASIC is to infer the types of functions from the type of their return types. The return value of a BBC BASIC function can be an arbitrary expression, including calls to other functions or recursive calls to itself. I’ve implemented a simple type inference scheme which works well in the common cases, but for a fully capable solution my type checker and type inferencer need to be beefed up somewhat. To that end, I’ve been investigating standard type systems such as Hindley-Milner and inferencing algorithms such as Damas-Milner, sometimes known as Algorithm W. These algorithms or derivatives thereof are using in the ML family of languages (Standard ML, Ocaml, F#) and Haskell.

I managed to locate a Modula-2 implementation, a Perl implementation and a Scala implementation of the algorithm, each descended from the previous. With a view to improving my understanding of the algorithm I set about reimplementing in Python, largely guided by the Scala implementation, making mine the fourth in this sequence. I also located a Haskell implementation which seems to have independent ancestry. I’ve gone back to the companion paper (Cardelli 1987) to the original Modula-2 implementation and carried forward some of the code comments which had been omitted from its descendants to assist others who wish to understand the algorithm.

The program implements abstract syntax tree nodes for a small functional language, the type inferencing algorithm and finally exercises the algorithm by inferring the types of some canned expressions in the context of some predefined types. When executed it produces the following output:

> python hindley_milner.py
(letrec factorial = (fn n => (((cond (zero n)) 1) ((times n) (factorial (pred n))))) in (factorial 5)) :  int
(fn x => ((pair (x 3)) (x true))) :  Type mismatch: bool != int
((pair (f 4)) (f true)) :  Undefined symbol f
(let f = (fn x => x) in ((pair (f 4)) (f true))) :  (int * bool)
(fn f => (f f)) :  recursive unification
(let g = (fn f => 5) in (g g)) :  int
(fn g => (let f = (fn x => g) in ((pair (f 3)) (f true)))) :  (a -> (a * a))
(fn f => (fn g => (fn arg => (g (f arg))))) :  ((b -> c) -> ((c -> d) -> (b -> d)))

The Python code is shown below in its entirety or it can be downloaded as hindley_milner.py. It will run without modification on Python 2.6 or Python 3.

#!/usr/bin/env python
'''
.. module:: hindley_milner
   :synopsis: An implementation of the Hindley Milner type checking algorithm
              based on the Scala code by Andrew Forrest, the Perl code by
              Nikita Borisov and the paper "Basic Polymorphic Typechecking"
              by Cardelli.
.. moduleauthor:: Robert Smallshire
'''

from __future__ import print_function

#=======================================================#
# Class definitions for the abstract syntax tree nodes
# which comprise the little language for which types
# will be inferred

class Lambda(object):
    """Lambda abstraction"""

    def __init__(self, v, body):
        self.v = v
        self.body = body

    def __str__(self):
        return "(fn {v} => {body})".format(v=self.v, body=self.body)

class Ident(object):
    """Identfier"""

    def __init__(self, name):
        self.name = name

    def __str__(self):
        return self.name

class Apply(object):
    """Function application"""

    def __init__(self, fn, arg):
        self.fn = fn
        self.arg = arg

    def __str__(self):
        return "({fn} {arg})".format(fn=self.fn, arg=self.arg)

class Let(object):
    """Let binding"""

    def __init__(self, v, defn, body):
        self.v = v
        self.defn = defn
        self.body = body

    def __str__(self):
        return "(let {v} = {defn} in {body})".format(v=self.v, defn=self.defn, body=self.body)

class Letrec(object):
    """Letrec binding"""

    def __init__(self, v, defn, body):
        self.v = v
        self.defn = defn
        self.body = body

    def __str__(self):
        return "(letrec {v} = {defn} in {body})".format(v=self.v, defn=self.defn, body=self.body)

#=======================================================#
# Exception types

class TypeError(Exception):
    """Raised if the type inference algorithm cannot infer types successfully"""

    def __init__(self, message):
        self.__message = message

    message = property(lambda self: self.__message)

    def __str__(self):
        return str(self.message)

class ParseError(Exception):
    """Raised if the type environment supplied for is incomplete"""
    def __init__(self, message):
        self.__message = message

    message = property(lambda self: self.__message)

    def __str__(self):
        return str(self.message)

#=======================================================#
# Types and type constructors

class TypeVariable(object):
    """A type variable standing for an arbitrary type.

    All type variables have a unique id, but names are only assigned lazily,
    when required.
    """

    next_variable_id = 0

    def __init__(self):
        self.id = TypeVariable.next_variable_id
        TypeVariable.next_variable_id += 1
        self.instance = None
        self.__name = None

    next_variable_name = 'a'

    def _getName(self):
        """Names are allocated to TypeVariables lazily, so that only TypeVariables
        present
        """
        if self.__name is None:
            self.__name = TypeVariable.next_variable_name
            TypeVariable.next_variable_name = chr(ord(TypeVariable.next_variable_name) + 1)
        return self.__name

    name = property(_getName)

    def __str__(self):
        if self.instance is not None:
            return str(self.instance)
        else:
            return self.name

    def __repr__(self):
        return "TypeVariable(id = {0})".format(self.id)

class TypeOperator(object):
    """An n-ary type constructor which builds a new type from old"""

    def __init__(self, name, types):
        self.name = name
        self.types = types

    def __str__(self):
        num_types = len(self.types)
        if num_types == 0:
            return self.name
        elif num_types == 2:
            return "({0} {1} {2})".format(str(self.types[0]), self.name, str(self.types[1]))
        else:
            return "{0} {1}" % (self.name, ' '.join(self.types))

class Function(TypeOperator):
    """A binary type constructor which builds function types"""

    def __init__(self, from_type, to_type):
        super(Function, self).__init__("->", [from_type, to_type])

# Basic types are constructed with a nullary type constructor
Integer = TypeOperator("int", [])  # Basic integer
Bool    = TypeOperator("bool", []) # Basic bool

#=======================================================#
# Type inference machinery

def analyse(node, env, non_generic=None):
    """Computes the type of the expression given by node.

    The type of the node is computed in the context of the context of the
    supplied type environment env. Data types can be introduced into the
    language simply by having a predefined set of identifiers in the initial
    environment. environment; this way there is no need to change the syntax or, more
    importantly, the type-checking program when extending the language.

    Args:
        node: The root of the abstract syntax tree.
        env: The type environment is a mapping of expression identifier names
            to type assignments.
            to type assignments.
        non_generic: A set of non-generic variables, or None

    Returns:
        The computed type of the expression.

    Raises:
        TypeError: The type of the expression could not be inferred, for example
            if it is not possible to unify two types such as Integer and Bool
        ParseError: The abstract syntax tree rooted at node could not be parsed
    """

    if non_generic is None:
        non_generic = set()

    if isinstance(node, Ident):
        return getType(node.name, env, non_generic)
    elif isinstance(node, Apply):
        fun_type = analyse(node.fn, env, non_generic)
        arg_type = analyse(node.arg, env, non_generic)
        result_type = TypeVariable()
        unify(Function(arg_type, result_type), fun_type)
        return result_type
    elif isinstance(node, Lambda):
        arg_type = TypeVariable()
        new_env = env.copy()
        new_env[node.v] = arg_type
        new_non_generic = non_generic.copy()
        new_non_generic.add(arg_type)
        result_type = analyse(node.body, new_env, new_non_generic)
        return Function(arg_type, result_type)
    elif isinstance(node, Let):
        defn_type = analyse(node.defn, env, non_generic)
        new_env = env.copy()
        new_env[node.v] = defn_type
        return analyse(node.body, new_env, non_generic)
    elif isinstance(node, Letrec):
        new_type = TypeVariable()
        new_env = env.copy()
        new_env[node.v] = new_type
        new_non_generic = non_generic.copy()
        new_non_generic.add(new_type)
        defn_type = analyse(node.defn, new_env, new_non_generic)
        unify(new_type, defn_type)
        return analyse(node.body, new_env, non_generic)
    assert 0, "Unhandled syntax node {0}".format(type(t))

def getType(name, env, non_generic):
    """Get the type of identifier name from the type environment env.

    Args:
        name: The identifier name
        env: The type environment mapping from identifier names to types
        non_generic: A set of non-generic TypeVariables

    Raises:
        ParseError: Raised if name is an undefined symbol in the type
            environment.
    """
    if name in env:
        return fresh(env[name], non_generic)
    elif isIntegerLiteral(name):
        return Integer
    else:
        raise ParseError("Undefined symbol {0}".format(name))

def fresh(t, non_generic):
    """Makes a copy of a type expression.

    The type t is copied. The the generic variables are duplicated and the
    non_generic variables are shared.

    Args:
        t: A type to be copied.
        non_generic: A set of non-generic TypeVariables
    """
    mappings = {} # A mapping of TypeVariables to TypeVariables

    def freshrec(tp):
        p = prune(tp)
        if isinstance(p, TypeVariable):
            if isGeneric(p, non_generic):
                if p not in mappings:
                    mappings[p] = TypeVariable()
                return mappings[p]
            else:
                return p
        elif isinstance(p, TypeOperator):
            return TypeOperator(p.name, [freshrec(x) for x in p.types])

    return freshrec(t)

def unify(t1, t2):
    """Unify the two types t1 and t2.

    Makes the types t1 and t2 the same.

    Args:
        t1: The first type to be made equivalent
        t2: The second type to be be equivalent

    Returns:
        None

    Raises:
        TypeError: Raised if the types cannot be unified.
    """

    a = prune(t1)
    b = prune(t2)
    if isinstance(a, TypeVariable):
        if a != b:
            if occursInType(a, b):
                raise TypeError("recursive unification")
            a.instance = b
    elif isinstance(a, TypeOperator) and isinstance(b, TypeVariable):
        unify(b, a)
    elif isinstance(a, TypeOperator) and isinstance(b, TypeOperator):
        if (a.name != b.name or len(a.types) != len(b.types)):
            raise TypeError("Type mismatch: {0} != {1}".format(str(a), str(b)))
        for p, q in zip(a.types, b.types):
            unify(p, q)
    else:
        assert 0, "Not unified"

def prune(t):
    """Returns the currently defining instance of t.

    As a side effect, collapses the list of type instances. The function Prune
    is used whenever a type expression has to be inspected: it will always
    return a type expression which is either an uninstantiated type variable or
    a type operator; i.e. it will skip instantiated variables, and will
    actually prune them from expressions to remove long chains of instantiated
    variables.

    Args:
        t: The type to be pruned

    Returns:
        An uninstantiated TypeVariable or a TypeOperator
    """
    if isinstance(t, TypeVariable):
        if t.instance is not None:
            t.instance = prune(t.instance)
            return t.instance
    return t

def isGeneric(v, non_generic):
    """Checks whether a given variable occurs in a list of non-generic variables

    Note that a variables in such a list may be instantiated to a type term,
    in which case the variables contained in the type term are considered
    non-generic.

    Note: Must be called with v pre-pruned

    Args:
        v: The TypeVariable to be tested for genericity
        non_generic: A set of non-generic TypeVariables

    Returns:
        True if v is a generic variable, otherwise False
    """
    return not occursIn(v, non_generic)

def occursInType(v, type2):
    """Checks whether a type variable occurs in a type expression.

    Note: Must be called with v pre-pruned

    Args:
        v:  The TypeVariable to be tested for
        type2: The type in which to search

    Returns:
        True if v occurs in type2, otherwise False
    """
    pruned_type2 = prune(type2)
    if pruned_type2 == v:
        return True
    elif isinstance(pruned_type2, TypeOperator):
        return occursIn(v, pruned_type2.types)
    return False

def occursIn(t, types):
    """Checks whether a types variable occurs in any other types.

    Args:
        v:  The TypeVariable to be tested for
        types: The sequence of types in which to search

    Returns:
        True if t occurs in any of types, otherwise False
    """
    return any(occursInType(t, t2) for t2 in types)

def isIntegerLiteral(name):
    """Checks whether name is an integer literal string.

    Args:
        name: The identifier to check

    Returns:
        True if name is an integer literal, otherwise False
    """
    result = True
    try:
        int(name)
    except ValueError:
        result = False
    return result

#==================================================================#
# Example code to exercise the above

def tryExp(env, node):
    """Try to evaluate a type printing the result or reporting errors.

    Args:
        env: The type environment in which to evaluate the expression.
        node: The root node of the abstract syntax tree of the expression.

    Returns:
        None
    """
    print(str(node) + " : ", end=' ')
    try:
        t = analyse(node, env)
        print(str(t))
    except (ParseError, TypeError) as e:
        print(e)

def main():
    """The main example program.

    Sets up some predefined types using the type constructors TypeVariable,
    TypeOperator and Function.  Creates a list of example expressions to be
    evaluated. Evaluates the expressions, printing the type or errors arising
    from each.

    Returns:
        None
    """

    var1 = TypeVariable()
    var2 = TypeVariable()
    pair_type = TypeOperator("*", (var1, var2))

    var3 = TypeVariable()

    my_env = { "pair" : Function(var1, Function(var2, pair_type)),
               "true" : Bool,
               "cond" : Function(Bool, Function(var3, Function(var3, var3))),
               "zero" : Function(Integer, Bool),
               "pred" : Function(Integer, Integer),
               "times": Function(Integer, Function(Integer, Integer)) }

    pair = Apply(Apply(Ident("pair"), Apply(Ident("f"), Ident("4"))), Apply(Ident("f"), Ident("true")))

    examples = [
            # factorial
            Letrec("factorial", # letrec factorial =
                Lambda("n",    # fn n =>
                    Apply(
                        Apply(   # cond (zero n) 1
                            Apply(Ident("cond"),     # cond (zero n)
                                Apply(Ident("zero"), Ident("n"))),
                            Ident("1")),
                        Apply(    # times n
                            Apply(Ident("times"), Ident("n")),
                            Apply(Ident("factorial"),
                                Apply(Ident("pred"), Ident("n")))
                        )
                    )
                ),      # in
                Apply(Ident("factorial"), Ident("5"))
            ),

            # Should fail:
            # fn x => (pair(x(3) (x(true)))
            Lambda("x",
                Apply(
                    Apply(Ident("pair"),
                        Apply(Ident("x"), Ident("3"))),
                    Apply(Ident("x"), Ident("true")))),

            # pair(f(3), f(true))
            Apply(
                Apply(Ident("pair"), Apply(Ident("f"), Ident("4"))),
                Apply(Ident("f"), Ident("true"))),

            # let f = (fn x => x) in ((pair (f 4)) (f true))
            Let("f", Lambda("x", Ident("x")), pair),

            # fn f => f f (fail)
            Lambda("f", Apply(Ident("f"), Ident("f"))),

            # let g = fn f => 5 in g g
            Let("g",
                Lambda("f", Ident("5")),
                Apply(Ident("g"), Ident("g"))),

            # example that demonstrates generic and non-generic variables:
            # fn g => let f = fn x => g in pair (f 3, f true)
            Lambda("g",
                   Let("f",
                       Lambda("x", Ident("g")),
                       Apply(
                            Apply(Ident("pair"),
                                  Apply(Ident("f"), Ident("3"))
                            ),
                            Apply(Ident("f"), Ident("true"))))),

            # Function composition
            # fn f (fn g (fn arg (f g arg)))
            Lambda("f", Lambda("g", Lambda("arg", Apply(Ident("g"), Apply(Ident("f"), Ident("arg"))))))
    ]

    for example in examples:
        tryExp(my_env, example)

if __name__ == '__main__':
    main()

Consider the following procedure from Sphinx Adventure. It contains three loops, one on line 271 formed with a GOTO back to the start of the line, and two REPEAT .. UNTIL loops.

266 DEF PROCL(L)
267 LOCAL I,J:CO=0:CN=0
268 IF L=1 THEN278
269 PRINT': RESTORE L: IF O?31<>0 THEN O?31=0:DW=1
270 READ R$,R$:R$="You are "+R$
271 IF LEN(R$)+ POS>CO-CN+39 THEN R$= FNS(R$,39+CO-CN):CO=CO+39: GOTO271
272 PRINT R$: IFL=136 OR L=15 THEN O?56=L
273 IF L=16 AND FL=1 THEN PRINT"The walls are very hot!" ELSE IF L=16 THEN PRINT "The walls are steaming!"
274 IF L<>3 AND L<>142 AND L<>143 THEN PROCEX(L): IF ABS(L-19)=1 AND CH=1 THEN PRINT ELSE IF ABS(L-42)=1 AND VO=1 THEN PRINT
275 IF CH=1 AND ABS(L-19)=1 THEN PROC R(22): PRINT "chasm.":O?53=L
276 IF VO=1 AND ABS(L-42)=1 THEN PROCR(22): PRINT"glacier.":O?53=L
277 IF L=26 OR L=27 THEN O?53=L
278 J=0:I=0:CO=0
279 REPEAT:J=J+1: IF O?J=L  THEN CO=CO+1
280 UNTIL J=52: IF CO=0 AND L=1 THEN PROCR(L): GOTO 284 ELSE IF CO=0 AND L<>1 THEN 284 ELSE PRINT:MAX=CO
281 IF L=1 THEN PROCR(3) ELSE PROCR(4)
282 CN=0:CO=MAX: REPEAT I=I+1: IF O?I=L  THEN PROCOT(I,CO):CO=CO-1
283 UNTIL I=52
284 IF D<>0 THEN O?31=L
285 IF CF=1 AND L=94 THEN PRINT'"The casket is open."
286 IF L=24 AND SA=1 THEN PRINT'"The safe door is open."
287 PRINT: ENDPROC

The CFG for this code is shown below. Each program statement is shown as a purple box, with control flow to the following statement(s). Conditionals are shown in diamond boxes. The numbers in each purple box are source line numbers, where known.

Careful comparison of the source above and the diagram below will reveal some of the transformations that have been applied to the program; for example, READ R$,R$ on line 270 has been transformed into two consecutive assignment statements which actually take the form R$ = READ() where READ() is a function not available in the source language.

The statement level CFG has been analysed to identify basic-blocks, shown as yellow group nodes, thereby defining a higher level basic-block level CFG. Each basic block has only one entry point statement; none of the statement within the basic block are destinations of other jump instructions. Furthermore, each block has only one exit point.

More text follows this long diagram…

Generating the CIL code for a single basic block is straightforward enough – we can simply iterate through the statements comprising the basic block in order and generate the code for each in turn. However, there are many possible orders in which the code for the basic block themselves could be representing in the CIL, since we can branch from the end of any block to the next block, although of course we must start at the entry block for the procedure. Although any order starting with the entry block can be made to work, where possible we would like program control to flow smoothly from the end of a block to one of its successors without requiring a branch.

At first sight, some sort of topological ordering would seem to be appropriate, but a topological ordering is only well defined for a directed acyclic graph (DAG), and a DAG this program is not. The key to this conundrum is to reduce the directed graph to a DAG by identifying strongly connected components. By contracting each SCC to a single node we obtain what is called the condensation of the CFG which will be a DAG. To the resulting DAG we can apply a topological ordering. The ordering of vertices with each SCC is chosen by starting at the vertex with the greatest in-degree.

In order to identify and contract the SCCs we use an implementation of Tarjan’s algorithm during depth first traversal of the CFG. The reverse post ordering of the primary depth first traversal is used to generate the topological ordering of the condensed CFG.

The resulting ordering of basic blocks is shown in the diagram by the numeric labels to the top-left of each. This will be the order in which the CIL code for them is generated, and it can be seen that in about half of the cases, fall through from one block to the next (consecutive block numbers) without explicit branching can be exploited. Future optimisations will focus on further simplifying the generated code by removing vertices, such as block 31, which contain only jumps.

The first executable produced from OWL BASIC.

Above you can see the executable loaded up into .NET Reflector, which can be used to introspect the executable, and in this case attempt to disassemble it into C#. Now we see what makes .NET such a great platform for compiler construction; below is the IronPython source code for the embryonic assembly generation function. It clocks in at fewer than ten lines of code to create an assembly, create a module, create a class, add one private static field to it for each global variable, and save the result as an .exe.

def generateAssembly(name, global_symbols):
    domain = Thread.GetDomain()
    assembly_name = AssemblyName(name)
    assembly_builder = domain.DefineDynamicAssembly(assembly_name, AssemblyBuilderAccess.RunAndSave)
    module_builder = assembly_builder.DefineDynamicModule(name + ".exe")
    type_builder = module_builder.DefineType(name, TypeAttributes.Class | TypeAttributes.Public, object().GetType())

    # Add global variables to the class
    for symbol in global_symbols.symbols.values():
        field_builder = type_builder.DefineField(symbol.name, ctsType(symbol),
                                                 FieldAttributes.Private | FieldAttributes.Static)

    result = type_builder.CreateType()
    assembly_builder.Save(name + ".exe")

where global_symbols is the global symbol table constructed during traversal of the Abstract Syntax Tree and the Control Flow Graph and the ctsType function maps OWL BASIC types to their equivalent Common Type System types for .NET. Everything else is provided by Reflection.Emit and other parts of .NET.

Its interesting that no validation was applied to the variable names supplied to Reflection.Emit. As you can see, the variable names still include the sigil suffixes for variable typing (e.g. $ for string) and Reflector happily dissassembles these into invalid C# identifiers. For the final version these names will need to be mangled (Hungarian notation?), or merely de-sigiled if no conflicts result, for compatibility with other .NET languages and tools.

int x = 5;
int y = x > 4 ? x : throw std::out_of_range;

The only use I’ve found for this — and in fact the speculative attempt by which I discovered it — is range checking within constructor initializer lists.

RangeChecked::RangeChecked(int x) :
    int_member(x > 0 ? x : throw std::out_of_range)
{
}

It’s academic what the type of a throw expression is, since it will never be returned, but for type-checking purposes the compiler seems to be happy to use it in place of any type.

After far too much effort I have finally found a sequence that Works For Me™ on Windows Vista Ultimate x64.

The procedure

1. Download eclipse-SDK-3.5-win32.zip
2. Extract the zip file into a temporary directory. I used C:\Users\<usename>\Documents\tmp\eclipse
3. Within the temporary directory (important!) edit eclipse.ini, adding the line

   -configuration @user.home\.eclipse_35_config
   

   My eclipse.ini then looked like this:

   -startup
   plugins/org.eclipse.equinox.launcher_1.0.200.v20090520.jar
   --launcher.library
   plugins/org.eclipse.equinox.launcher.win32.win32.x86_1.0.200.v20090519
   -showsplash
   org.eclipse.platform
   -configuration @user.home\.eclipse_35_config
   --launcher.XXMaxPermSize
   256m
   -vmargs
   -Xms40m
   -Xmx256m
   

4. Within the temporary directory create a UTF-8 text file called eclipse.exe.manifest file side-by-side with the eclipse.exe file with the following content:

   <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
   <assembly xmlns="urn:schemas-microsoft-com:asm.v1" manifestVersion="1.0">
     <assemblyIdentity version="1.0.0.0"
        processorArchitecture="X86"
        name="eclipse"
        type="win32"/>
     <description>Eclipse</description>
     <!-- Identify the application security requirements. -->
     <trustInfo xmlns="urn:schemas-microsoft-com:asm.v2">
       <security>
         <requestedPrivileges>
           <requestedExecutionLevel level="asInvoker" uiAccess="false"/>
           </requestedPrivileges>
          </security>
     </trustInfo>
   </assembly>
   

5. Using drag and drop copy in Explorer, copy the entire eclipse directory from your temporary location to C:\Program Files (x86)\eclipse. We copy, rather than move simply so you can easily return to this step.
6. Right-click on eclipse.exe in Explorer and choose Properties
7. Click on the Security tab
8. Click Advanced in the lower right
9. In the Advanced Security Settings window that pops up, click on the
   Owner tab
10. Click Edit
11. Click Continue if you get a UAC dialog
12. Click Other users or groups
13. Click Advanced in the lower left corner
14. Click Find Now
15. Scroll through the results and click on your current user account
16. Click OK to all of the remaining windows
17. Right-click the file and select Properties (again)
18. Click on the Security tab
19. Click Edit…
20. Click on the Users group
21. Adjust the permissions for your user using the check boxes at the bottom of the dialog. e.g. enable Full Control
22. Click OK to all of the remaining windows
23. Right-click the file and select Properties (again!)
24. Click Unblock and close the dialog
25. Double click eclipse.exe to run it. You should not get an Open File – Security Warning dialog.
26. Remove the temporary eclipse directory you created when you unzipped the archive.

Credit where it is due

The above post is largely pulled together from various sources I located mixed with a good degree of trial and error. I found the following particularly useful:

• Open File – Security Warning on VistaHeads
• How do you run Eclipse 3.3 on Vista?

I’ll start out by stating that I came away from the conference a little dissappointed, although I must emphasise in no way cheated, by the experience. I’ve attended a great many professionally and community organized conferences over the years, and I felt in many regards Europython was somewhat further down the quality scale than it should be, given the rising importance of Python. I’m concious that this posting may come across as a litany of complaint, but I feel that some changes could significantly improve the experience for everybody involved. Organisers of such events are forever soliciting feedback, so here goes!

Europython 2009 was a multi-tracked conference with up to six parallel sessions, so its not possible for me to report on even a representative sample of the sessions. Nonetheless, the overview of the not-so-random sample presented below may give you a flavour of the experience. I’m not involved in web development currently so I tended to shy away from the plethora of sessions covering web frameworks. What does interest me are the alternative Python implementatons, IronPython and Jython and uses of Python for scientific computation. Unfortunately, I couldn’t make it to the SCons talk, another interest of mine.

Size, venue and organization

Europython had around 500 delegates in attendance, which is I believe a record. The chosen venue was something of a disappointment. Although the main lecture theatre (The Adrian Bolt Hall) was effective, the other rooms in use were without exception compromised in significant ways. One was quite small with insufficient capacity and little or no air conditioning – although it should be noted that Birmingham was experiencing something of a heatwave at the time. Another featured a concrete pillar between the speaker and his audience – although much better air conditioning in here – and a third had the audience seated on a raked seating arrangement that creaked and groaned with the slightest movement. I wonder how the musicians who usually inhabit the placed manage?

The lunch arrangements were wholly inadequate. Lunch involved collecting a polystyrene plate, bowl and a drink from a refectory upstairs, loading these with food canteen style and then precariously balancing these items (no trays here!) and trooping down a fire escape stairwell back into the corridors between the lecture halls where there was insufficient seating. At one point our Vice President of Product Development, had expressed an interest in attending Europython, having shown a genuine curiosity in the langauge – we’ve been working hard on internal advocacy. At the point at which I found myself having to stand up to eat my lunch standing up in a corridor, having requisitioned a corner of the coffee serving table with my colleague, I was glad he wasn’t here to witness this somewhat unprofessional scenario (Yes, I know its a community event, and not ‘professionally’ organised, but a touch of professionalism can go a long way in the Python advocacy stakes with the right people). I appreciate there were a mitigating circumstances (a broken lift) but the nature of the facilities alone will make me think twice about who I encourage to attend this event in future.

Preparation and planning

Shortly before Europython I’d returned from a conference called EAGE 2009 which had 7000 student and professional delegates in a major European convention centre (RAI in Amsterdam). I’ll confess that this may have done much to anchor my expection levels vis-à-vis conferences immediately prior to Europython. Venue differences and registration cost aside (Europython was 5× cheaper than EAGE) one clear difference was the poor level of preparedness of speakers at Europython compared to EAGE, even though at both conferences speakers were drawn from professional, student and amateur circles. Time and again at Europython speakers hurriedly arrived in the room with little or no time for setup of equipment such as projectors. At this point I must give a note — no, make that a plea — to conference organizers: Please place a printed notice in 144 point text stating the preferred projector resolution on the table or lectern at the front so we can be spared a linear search through the possible resolutions at the beginning of each talk.

Now, I don’t know about you, but on occasions that I’ve been presenting and a private meeting or a conference, especially if it involves a live demo requiring network connectivity, I’m in the room setting up, connecting, authenticating, testing, and re-testing for up to 30 minutes before hand ensuring that everything will run smoothly. Although this doesn’t ensure a professional delivery, it can usually eliminate the tiresome, and stressful for the speaker, configuration during the talk. Its also a common courtesy to your audience to be ready before they are and not keep them waiting. Doing this means you means you’ll get longer to talk, and time for questions. I’m sure most people would agree that time for questions beats staring Display Settings for the umpteenth time.

Some talks dependend on a ‘live’ Internet connection – always a bad idea but in this case it was fatal. Wireless connectivity was intermittent throughout the conference.

Talk quality

I found the quality of the material presented at Europython to be highly variable; it covered the gamut from borderline incompetent to very good indeed. Many of the talks simply did not have sufficient material behind them to justify the length of the time slot they were given. I wonder if fewer parallel tracks with shorter talks would give a better result; audiences would be larger, the pace quicker and delegates would be a able to see a larger proportion of talks, which would help in technology transfer between the different subgroups within the wider Python community.

Keynotes

The keynotes were engaging as one expects. Cory Doctorow gave the predictable but entertaining DRM is evil and copyright unenforceable polemic, in the moral vacuum we have come to expect. Bruce Eckel delivered an entertaining historical overview of language features across C++, Java, C# and Python. Susan Blackmore and Simon Greenish made an impassioned plea for funds for Saving Bletchley Park, although the talk lacked any technical content about the operations or machines there, which given the technical audience was a little disappointing. Sir Tony Hoare rounded off the keynotes with a comparative study of the disciplines of software engineering and computing science, finishing with the not unreasonable prediction that one day “software will be the most reliable component of every product which contains it”, although he didn’t venture to suggest a time-scale on which this might come to pass.

Conclusion

I’m glad I attended. The conference was good value and on the whole informative. The organisers should seriously consider a better venue for next year and raising the standard of talks would make a return visit a more compelling prospect. The easiest way to do this may be to simply reduce the number of talks, or shift more of the content into a more appropriate format such as lightning talks.

It seems clear that if you need to write code that is portable between the different Python implementations you should steer clear str and use bytes and unicode to unambigiously express your intent.

Of course, this is impossible since the Python Standard Library is littered with uses of str. For example, in IronPython pickle.dumps() returns str just like Python 2.6 but the str is actually has multibyte storage. IronPython hides this well, but the abstraction can leak, resulting in much confusion. Again Python 3.0 does what is right, and pickle.dumps() returns a bytes instance.

These difficulties are most likely to occur when interfacing with native Java or .NET APIs that expect byte arrays, for example when pickling to database blobs.

In Jython an str instance can be converted to a Java byte array as follows.

>>> import jarray
>>> a = jarray.array("This is  string", 'b')
>>> a
array('b', [84, 104, 105, 115, 32, 105, 115, 32, 32, 115, 116, 114, 105, 110, 103])

The equivalent in IronPython, as provided by Michael Foord, being,

>>> from System import Array, Byte
>>> a = Array[Byte](tuple(Byte(ord(c)) for c in "This is a string"))
>>> a
Array[Byte]((<System.Byte object at 0x000000000000002B [84]>, <System.Byte object at 0x000000000000002C [104]>, <System.Byte object at 0x000000000000002D [105]>, <System.Byte object at 0x000000000000002E [115]>, <System.Byte object at 0x000000000000002F [32]>, <System.Byte object at 0x0000000000000030 [105]>, <System.Byte object at 0x0000000000000031 [115]>, <System.Byte object at 0x0000000000000032 [32]>, <System.Byte object at 0x0000000000000033 [97]>, <System.Byte object at 0x0000000000000034 [32]>, <System.Byte object at 0x0000000000000035 [115]>, <System.Byte object at 0x0000000000000036 [116]>, <System.Byte object at 0x0000000000000037 [114]>, <System.Byte object at 0x0000000000000038 [105]>, <System.Byte object at 0x0000000000000039 [110]>, <System.Byte object at 0x000000000000003A [103]>))

Going back we can use identical code in IronPython and Jython.

>>> s = ''.join(chr(c) for c in a)
>>> s
'This is a string'

The most informative comment was from Curt Hagenlocher who works on IronPython in the Visual Studio Managed Languages group at Microsoft.

Curt suggested,

Try storing the counter as a global variable instead of a class-level member of Node — I think you’ll notice a dramatic improvement.

The modified benchmark program looks like this:

counter = 0

class Node(object):

    def __init__(self, children):
        global counter
        counter += 1
        self._children = children

def make_tree(depth):
    if depth > 1:
        return Node ([make_tree(depth - 1), make_tree(depth - 1)])
    else:
        return Node([])

def main(argv=None):
    global counter
    if argv is None:
        argv = sys.argv
    depth = int(argv[1]) if len(argv) > 1 else 10

    root = make_tree(depth)
    print counter
    return 0

if __name__ == '__main__':
    import sys
    sys.exit(main())

A dramatic improvement!

Well, Curt wasn’t wrong. This made a phenomenal difference with IronPython completing in only 12% of the time taken by CPython – over 8× faster with a binary tree depth of 20.

Let’s look in detail at the results. All results are from a dual quad-core 1.86 GHz Xeon with 4 GB RAM, and as before each benchmark was run five times, and the shortest time of the five taken.

The three test environments are:

1. Python 2.5.2 x86 32-bit
2. Jython 2.5rc2 on Java Hotspot 1.6 32-bit
3. IronPython 2.0 on .NET 2.0 x64

Figure 1. The relative performance of the three main Python implementations on a benchmark that uses a global counter, rather than mutating a class attribute.

Here we can see how IronPython’s performance has been improved hugely by this simple change. Although startup time dominates for the smaller problem size, now both Jython and IronPython surpass CPython at around half-a-million nodes.

Removing start-up time, which may be irrelevant for long-running processes, gives us the following chart:

Figure 2. The performance of the three main Python implementations excluding start-up time.

Again there is a lot of noise in the data below 1000 nodes, but it is clear that Jython scales better than IronPython, which in turn is scaling better than CPython.

Up until now I’ve been using a log-log scale in the charts because of the wide variation in performance between the different implementations, but now the performance gap is much closer, it’s difficult to get a sense of just how much faster IronPython is on the modified benchmark. Let’s throw in a log-linear plot to help us appreciate what’s going on:

Figure 3. A linear representation of the same data as in Figure 2, to highlight the performance multiple between IronPython and CPython in the larger tests.

It’s perhaps easier to see now that IronPython is doing in 14 seconds what takes CPython 114 seconds to achieve!

Finally, let’s plot those results as we did before, as multiples of CPython performance:

Figure 4. Execution time of Jython and IronPython as multiples of CPython performance.

It is easy to see that in this chart, once we pass half-a-million tree nodes (a tree depth of 19) that both Jython and IronPython are significantly beating CPython.

Explanation

Curt Hagenlocher offers the explanation in a comment in the thread on Reddit.

In this particular case, IronPython is slow because of the update to Node.counter. Currently, any update to a class will increment the version number for the class, which will have the effect of invalidating any rules compiled for that class. Effectively, the same rules are getting compiled over and over again. Moving the counter to a global should result in performance on par with that of CPython.

which is absolutely correct, except that he’s underselling the relative gain. IronPython is not only on a par with CPython, it can outperform it by a factor of eight!

With this knowledge in hand, I can now approach optimization of my OWL BASIC compiler, which lies back at the start of this illuminating tale.

Conclusion

• Avoiding mutation of Python class attributes can have significant benefits for IronPython performance.
• Both IronPython and Jython scale better than CPython by this benchmark, and have superior performance for large trees of nodes.

Since then, I’ve had time to reproduce the results with a small and completely unremarkable Python program, and also to run the tests on a different system. I had suspected that in the OWL BASIC compiler, my Python visitor implementation, which is used in applying transformations to the abstract syntax tree, was to blame. I set about condensing a tree visitor down to a small example, but I never got that far. It is sufficient to simply build a large binary tree to demonstrate the dramatic differences in the performance characteristics of the three main Python implementations.

The benchmark

Here is that test program, which just builds a simple binary tree of objects to the requested depth.

class Node(object):
    counter = 0

    def __init__(self, children):
        Node.counter += 1
        self._children = children

def make_tree(depth):
    if depth > 1:
        return Node ([make_tree(depth - 1), make_tree(depth - 1)])
    else:
        return Node([])

def main(argv=None):
    if argv is None:
        argv = sys.argv
    depth = int(argv[1]) if len(argv) > 1 else 10

    root = make_tree(depth)
    print Node.counter
    return 0

if __name__ == '__main__':
    import sys
    sys.exit(main())

The program builds a binary tree to the depth supplied as the only command line argument, or ten if one is not supplied. It counts the number of nodes as they a built. Remember that the merits or otherwise of this program are not the point! The point is the performance difference between the Python implementations when it is run.

My benchmarking approach has been to run this script five times for each tree depth from a depth of one, upwards to 22, or until my patience was exhausted. I’ve taken the minimum time from each run of five. Since there is a non-linear relationship between the depth of the tree and the number of nodes contained therein, logarithmic axes are used in all the charts that follow.

64 bit Windows Vista x64

Here are the results for the first test machine – with dual quad-core 1.86 GHz Xeons with 4 GB RAM running Vista x64, testing IronPython 2.0.0.0 on .NET 2.0, Jython 2.5rc2 on Java Hotspot 1.6.0 and Python 2.5.2.

Figure 1. Creation time for a binary tree including Python virtual machine startup on Windows Vista x64 with 1.86 GHz Xeon processors.

In Figure 1 we see that above 1000 nodes or so (tree depth of 10) performance for IronPython begin to degrade rapidly. CPython holds out for another two orders of magnitude before the significant costs begin to be felt . Its interesting to see that although Jython is in the middle of the pack, it scales much better than CPython, surpassing it at around half-a-million nodes (tree depth of 19).

In my application — a compiler — virtual machine (VM) start-up time is important; however, in many long-running applications this is not the case, so it is interesting to subtract VM start-up time from each series, which we see in Figure 2, below.

By subtracting VM start-up time, we get a picture more interesting for long-running processes.

Below 100 tree nodes, there is a lot of noise in these measurements. Above 100 nodes its easy to see that the blue IronPython curve is at least two chart divisions above the red CPython curve — that’s two orders of magnitude or 100× slower, and getting relatively worse as the size of the tree increases.

32 bit Windows XP x86

Responses to my earlier article suggested that trying IronPython 2.0.1 with Ngen’ed binaries on x86 may make a difference. Well, to cut a long story short, it doesn’t, but here are the details. These tests were run on a 900 MHz Pentium M Centrino laptop with 768 MB RAM, and so cannot be directly compared with those above, although its notable that a one year old workstation is only twice as fast as a five year old laptop. Moore’s law certainly isn’t delivering here!

The performance profiles are very simular with IronPython 2.0.1 on x86.

On x86, IronPython is still 100× slower than CPython, and Jython still scales better. It seems the essence of this benchmark is not dependent on which hardware or CLR platform it is run.

I’ll close by re-presenting the data in the x86 benchmarks as multiples of CPython performance, because it dramatically demonstrates the different responses to the scale of the problem size for IronPython and Jython. Again we see Jython catching up with CPython at a tree depth of 19, just we saw on x64. and IronPython delivering 6000× worse than CPython at a tree depth depth of 15. A tree of this size with thirty-thousand nodes is very similar in scale to the AST tree sizes found in the OWL BASIC during compilation of large programs.

Performance of IronPython and Jython as multiples of CPython performance.

Conclusions

• IronPython can be very slow, even on programs in the microbenchmark category, which are doing standard operations such as building trees. Presumably there are still significant optimizations to be made in IronPython to bring its performance closer to that of the other Python implementations. Hopefully, this example and the measurements can contribute to that improvement.
• Jython may scale better than Python if your application exercises Python in similar ways to this benchmark. Speculatively, that could have implications for projects such as SCons, which build large in-memory graphs.
• I suppose if nothing else we have demonstrated in passing that Java can be faster than C for some non-trivial programs (like a Python interpreter) running a trivial program, like this benchmark.

“IronPython is fast – up to 1.8x faster than CPython on the standard pystone benchmark.

But do these claims stand up in the face of real-world Python code?

The claims of good performance are based on synthetic micro-benchmarks which don’t accurately reflect balance of coding techniques found in complete programs.

At this point I’d like to offer my own quote:

“IronPython can be slow – 10x to 100x slower than CPython on real-world code and it has been observed to be up to 6000x slower.

Now, the unfortunate thing about real-world code is that it hasn’t been hand-crafted to highlight the performance characteristics of some aspect of the language implementation like your typical micro-benchmark. Its been written, most likely without any attention to performance. This is especially in the case of a prototype, as in my case.

Is my code really that bad?

Over the past few years I’ve been working on a hobby project called OWL BASIC, which is a compiler for the BBC BASIC lanaguage targeting the .NET Common Language Runtime (CLR). At the outset of the project I decided to write the compiler itself in Python, for much the same reasons as PyPy is written in Python — hackability. I planned specifically to use IronPython so I could benefit from access to useful .NET libraries such a Reflection.Emit, for generating Common Intermediate Language (CIL) assemblies.

During the course of developing OWL BASIC, as the code has become more complex, I’ve been consitently disappointed by the negative delta between the promise and reality of IronPython’s performance. The poor performance of IronPython has negated, for me, one of the main advantages of developing in Python — the rapid edit-run cycle.

Was my code really so inefficient? Performance was never a goal of the project, but the underwhelming performance has threatened the viability of my approach and made me question the wisdom of my chosen route of writing a compiler in Python.

The absence of profiling tools for IronPython led me down the road of getting at least the compiler front-end to work on CPython, so I could use the standard Python profilers. Fortunately, my code was portable (its just regular Python) and so I determined that with a few inconsequential tweaks to my code, the entire compiler front-end can be run on the trinity of CPython, Jython and IronPython.

I was, to say the least, somewhat surprised by the results.

The evidence: performance of CPython, Jython and IronPython

The investigation into the relative performance of the three main Python implementations was centered on running my unmodified, unprofiled and unoptimized compiler front-end on third-party BBC BASIC source-code (Acornsoft Sphinx Adventure) and measuring the execution time. All tests runs were performed five times, and the minimum time of the five chosen. Variance between the readings on successive runs was small. The following Python implementations were on the test-bench:

• Python 2.5.1 (x86)
• Jython 2.5rc2 on Java HotSpot 1.6.0_10-rc
• IronPython 2.0 on .NET 2.0 (x64)

All tests were run on a eight-core 1.86 GHz Xeon with 4 GB RAM running Vista Ultimate x64.

The following chart shows the results of running the compiler over the source code for Sphinx Adventure.

The absolute performance of CPython, Jython and IronPython on my code

Frankly, I was astounded. IronPython was left in the dust, not only by CPython but also Jython! Overall, CPython was 93× faster on the exact same code. The IronPython hyperbole, for now I could see that was what is was, had led to me expect numbers similar to those for CPython, although I had perhaps more realistic expectations that performance would be similar to Jython.

At this point I assumed I’d hit some corner case in which IronPython was performing relatively badly. I’d had a similar experience early on in the project with code from the PLY parsing package causing IronPython 1.1 to perform badly, but I’d worked around the issue by modifying parts of PLY to use pickling rather than eval-ing large list literals for the cached parsing tables. [Its worth noting in passing that this problem still exists with the IronPython and PLY combination - I'll publish my solution in another post].

I decided to dig in a little more detail and collect some timing data on the sequence of top level calls the compiler front-end makes to parse the source code followed by a sequence of transformations to the abstract syntax tree (AST). The chart below shows these top level calls, in the order in which the occur during compilation:

The ratio of performance of CPython to both IronPython and Jython when running the OWL BASIC compiler

As you can see, I have had to resort to a logarithmic scale in order to convey the huge variation in the performance of IronPython relative to CPython, ranging from buildParser with a multiple of 3.8 to convertSubroutinesToProcedures with a multiple of over 6400. Even if we ignore this outlier (maybe we are suffering from a garbage collection – but notice that Jython is also relatively much slower on this function) we can see that Jython is typically 5× to 10× slower than CPython whereas IronPython is typically 10× to 100× slower than CPython.

Notice also that there is a marked decline in the performance of IronPython from the parse function onwards; the common factor in these operations are that they are all transformations to the AST, and my AST node classes are instantiated by a Python metaclass, although its pure speculation on my part that metaclasses are the cause of this performance drop.

Conclusion

On my program at least, Jython is 6× slower than CPython and IronPython is nearly 100× slower. If you’re suffering from poor performance with IronPython, it may be well worth your time checking performance of your code on the other Python implementations, if that is an option for you.

Now need to find the root cause and boil my problem down to a short example which can form the basis of a bug report to the IronPython team. Given that the problem is pervasive in my code, that won’t be hard.

See you at EuroPython 2009 !