Before you get too excited this is an implementation of a type inference algorithm that happens to be written in Python; it has nothing to do with the Python language itself!

I’ve been working on OWL BASIC, a compiler for BBC BASIC for the .NET CLR. The compiler itself is written in IronPython. One of the challenges of compiling BBC BASIC is to infer the types of functions from the type of their return types. The return value of a BBC BASIC function can be an arbitrary expression, including calls to other functions or recursive calls to itself. I’ve implemented a simple type inference scheme which works well in the common cases, but for a fully capable solution my type checker and type inferencer need to be beefed up somewhat. To that end, I’ve been investigating standard type systems such as Hindley-Milner and inferencing algorithms such as Damas-Milner, sometimes known as Algorithm W. These algorithms or derivatives thereof are using in the ML family of languages (Standard ML, Ocaml, F#) and Haskell.

I managed to locate a Modula-2 implementation, a Perl implementation and a Scala implementation of the algorithm, each descended from the previous. With a view to improving my understanding of the algorithm I set about reimplementing in Python, largely guided by the Scala implementation, making mine the fourth in this sequence. I also located a Haskell implementation which seems to have independent ancestry. I’ve gone back to the companion paper (Cardelli 1987) to the original Modula-2 implementation and carried forward some of the code comments which had been omitted from its descendants to assist others who wish to understand the algorithm.

The program implements abstract syntax tree nodes for a small functional language, the type inferencing algorithm and finally exercises the algorithm by inferring the types of some canned expressions in the context of some predefined types. When executed it produces the following output:

> python hindley_milner.py
(letrec factorial = (fn n => (((cond (zero n)) 1) ((times n) (factorial (pred n))))) in (factorial 5)) :  int
(fn x => ((pair (x 3)) (x true))) :  Type mismatch: bool != int
((pair (f 4)) (f true)) :  Undefined symbol f
(let f = (fn x => x) in ((pair (f 4)) (f true))) :  (int * bool)
(fn f => (f f)) :  recursive unification
(let g = (fn f => 5) in (g g)) :  int
(fn g => (let f = (fn x => g) in ((pair (f 3)) (f true)))) :  (a -> (a * a))
(fn f => (fn g => (fn arg => (g (f arg))))) :  ((b -> c) -> ((c -> d) -> (b -> d)))

The Python code is shown below in its entirety or it can be downloaded as hindley_milner.py. It will run without modification on Python 2.6 or Python 3.

#!/usr/bin/env python
'''
.. module:: hindley_milner
   :synopsis: An implementation of the Hindley Milner type checking algorithm
              based on the Scala code by Andrew Forrest, the Perl code by
              Nikita Borisov and the paper "Basic Polymorphic Typechecking"
              by Cardelli.
.. moduleauthor:: Robert Smallshire
'''

from __future__ import print_function

#=======================================================#
# Class definitions for the abstract syntax tree nodes
# which comprise the little language for which types
# will be inferred

class Lambda(object):
    """Lambda abstraction"""

    def __init__(self, v, body):
        self.v = v
        self.body = body

    def __str__(self):
        return "(fn {v} => {body})".format(v=self.v, body=self.body)

class Ident(object):
    """Identfier"""

    def __init__(self, name):
        self.name = name

    def __str__(self):
        return self.name

class Apply(object):
    """Function application"""

    def __init__(self, fn, arg):
        self.fn = fn
        self.arg = arg

    def __str__(self):
        return "({fn} {arg})".format(fn=self.fn, arg=self.arg)

class Let(object):
    """Let binding"""

    def __init__(self, v, defn, body):
        self.v = v
        self.defn = defn
        self.body = body

    def __str__(self):
        return "(let {v} = {defn} in {body})".format(v=self.v, defn=self.defn, body=self.body)

class Letrec(object):
    """Letrec binding"""

    def __init__(self, v, defn, body):
        self.v = v
        self.defn = defn
        self.body = body

    def __str__(self):
        return "(letrec {v} = {defn} in {body})".format(v=self.v, defn=self.defn, body=self.body)

#=======================================================#
# Exception types

class TypeError(Exception):
    """Raised if the type inference algorithm cannot infer types successfully"""

    def __init__(self, message):
        self.__message = message

    message = property(lambda self: self.__message)

    def __str__(self):
        return str(self.message)

class ParseError(Exception):
    """Raised if the type environment supplied for is incomplete"""
    def __init__(self, message):
        self.__message = message

    message = property(lambda self: self.__message)

    def __str__(self):
        return str(self.message)

#=======================================================#
# Types and type constructors

class TypeVariable(object):
    """A type variable standing for an arbitrary type.

    All type variables have a unique id, but names are only assigned lazily,
    when required.
    """

    next_variable_id = 0

    def __init__(self):
        self.id = TypeVariable.next_variable_id
        TypeVariable.next_variable_id += 1
        self.instance = None
        self.__name = None

    next_variable_name = 'a'

    def _getName(self):
        """Names are allocated to TypeVariables lazily, so that only TypeVariables
        present
        """
        if self.__name is None:
            self.__name = TypeVariable.next_variable_name
            TypeVariable.next_variable_name = chr(ord(TypeVariable.next_variable_name) + 1)
        return self.__name

    name = property(_getName)

    def __str__(self):
        if self.instance is not None:
            return str(self.instance)
        else:
            return self.name

    def __repr__(self):
        return "TypeVariable(id = {0})".format(self.id)

class TypeOperator(object):
    """An n-ary type constructor which builds a new type from old"""

    def __init__(self, name, types):
        self.name = name
        self.types = types

    def __str__(self):
        num_types = len(self.types)
        if num_types == 0:
            return self.name
        elif num_types == 2:
            return "({0} {1} {2})".format(str(self.types[0]), self.name, str(self.types[1]))
        else:
            return "{0} {1}" % (self.name, ' '.join(self.types))

class Function(TypeOperator):
    """A binary type constructor which builds function types"""

    def __init__(self, from_type, to_type):
        super(Function, self).__init__("->", [from_type, to_type])

# Basic types are constructed with a nullary type constructor
Integer = TypeOperator("int", [])  # Basic integer
Bool    = TypeOperator("bool", []) # Basic bool

#=======================================================#
# Type inference machinery

def analyse(node, env, non_generic=None):
    """Computes the type of the expression given by node.

    The type of the node is computed in the context of the context of the
    supplied type environment env. Data types can be introduced into the
    language simply by having a predefined set of identifiers in the initial
    environment. environment; this way there is no need to change the syntax or, more
    importantly, the type-checking program when extending the language.

    Args:
        node: The root of the abstract syntax tree.
        env: The type environment is a mapping of expression identifier names
            to type assignments.
            to type assignments.
        non_generic: A set of non-generic variables, or None

    Returns:
        The computed type of the expression.

    Raises:
        TypeError: The type of the expression could not be inferred, for example
            if it is not possible to unify two types such as Integer and Bool
        ParseError: The abstract syntax tree rooted at node could not be parsed
    """

    if non_generic is None:
        non_generic = set()

    if isinstance(node, Ident):
        return getType(node.name, env, non_generic)
    elif isinstance(node, Apply):
        fun_type = analyse(node.fn, env, non_generic)
        arg_type = analyse(node.arg, env, non_generic)
        result_type = TypeVariable()
        unify(Function(arg_type, result_type), fun_type)
        return result_type
    elif isinstance(node, Lambda):
        arg_type = TypeVariable()
        new_env = env.copy()
        new_env[node.v] = arg_type
        new_non_generic = non_generic.copy()
        new_non_generic.add(arg_type)
        result_type = analyse(node.body, new_env, new_non_generic)
        return Function(arg_type, result_type)
    elif isinstance(node, Let):
        defn_type = analyse(node.defn, env, non_generic)
        new_env = env.copy()
        new_env[node.v] = defn_type
        return analyse(node.body, new_env, non_generic)
    elif isinstance(node, Letrec):
        new_type = TypeVariable()
        new_env = env.copy()
        new_env[node.v] = new_type
        new_non_generic = non_generic.copy()
        new_non_generic.add(new_type)
        defn_type = analyse(node.defn, new_env, new_non_generic)
        unify(new_type, defn_type)
        return analyse(node.body, new_env, non_generic)
    assert 0, "Unhandled syntax node {0}".format(type(t))

def getType(name, env, non_generic):
    """Get the type of identifier name from the type environment env.

    Args:
        name: The identifier name
        env: The type environment mapping from identifier names to types
        non_generic: A set of non-generic TypeVariables

    Raises:
        ParseError: Raised if name is an undefined symbol in the type
            environment.
    """
    if name in env:
        return fresh(env[name], non_generic)
    elif isIntegerLiteral(name):
        return Integer
    else:
        raise ParseError("Undefined symbol {0}".format(name))

def fresh(t, non_generic):
    """Makes a copy of a type expression.

    The type t is copied. The the generic variables are duplicated and the
    non_generic variables are shared.

    Args:
        t: A type to be copied.
        non_generic: A set of non-generic TypeVariables
    """
    mappings = {} # A mapping of TypeVariables to TypeVariables

    def freshrec(tp):
        p = prune(tp)
        if isinstance(p, TypeVariable):
            if isGeneric(p, non_generic):
                if p not in mappings:
                    mappings[p] = TypeVariable()
                return mappings[p]
            else:
                return p
        elif isinstance(p, TypeOperator):
            return TypeOperator(p.name, [freshrec(x) for x in p.types])

    return freshrec(t)

def unify(t1, t2):
    """Unify the two types t1 and t2.

    Makes the types t1 and t2 the same.

    Args:
        t1: The first type to be made equivalent
        t2: The second type to be be equivalent

    Returns:
        None

    Raises:
        TypeError: Raised if the types cannot be unified.
    """

    a = prune(t1)
    b = prune(t2)
    if isinstance(a, TypeVariable):
        if a != b:
            if occursInType(a, b):
                raise TypeError("recursive unification")
            a.instance = b
    elif isinstance(a, TypeOperator) and isinstance(b, TypeVariable):
        unify(b, a)
    elif isinstance(a, TypeOperator) and isinstance(b, TypeOperator):
        if (a.name != b.name or len(a.types) != len(b.types)):
            raise TypeError("Type mismatch: {0} != {1}".format(str(a), str(b)))
        for p, q in zip(a.types, b.types):
            unify(p, q)
    else:
        assert 0, "Not unified"

def prune(t):
    """Returns the currently defining instance of t.

    As a side effect, collapses the list of type instances. The function Prune
    is used whenever a type expression has to be inspected: it will always
    return a type expression which is either an uninstantiated type variable or
    a type operator; i.e. it will skip instantiated variables, and will
    actually prune them from expressions to remove long chains of instantiated
    variables.

    Args:
        t: The type to be pruned

    Returns:
        An uninstantiated TypeVariable or a TypeOperator
    """
    if isinstance(t, TypeVariable):
        if t.instance is not None:
            t.instance = prune(t.instance)
            return t.instance
    return t

def isGeneric(v, non_generic):
    """Checks whether a given variable occurs in a list of non-generic variables

    Note that a variables in such a list may be instantiated to a type term,
    in which case the variables contained in the type term are considered
    non-generic.

    Note: Must be called with v pre-pruned

    Args:
        v: The TypeVariable to be tested for genericity
        non_generic: A set of non-generic TypeVariables

    Returns:
        True if v is a generic variable, otherwise False
    """
    return not occursIn(v, non_generic)

def occursInType(v, type2):
    """Checks whether a type variable occurs in a type expression.

    Note: Must be called with v pre-pruned

    Args:
        v:  The TypeVariable to be tested for
        type2: The type in which to search

    Returns:
        True if v occurs in type2, otherwise False
    """
    pruned_type2 = prune(type2)
    if pruned_type2 == v:
        return True
    elif isinstance(pruned_type2, TypeOperator):
        return occursIn(v, pruned_type2.types)
    return False

def occursIn(t, types):
    """Checks whether a types variable occurs in any other types.

    Args:
        v:  The TypeVariable to be tested for
        types: The sequence of types in which to search

    Returns:
        True if t occurs in any of types, otherwise False
    """
    return any(occursInType(t, t2) for t2 in types)

def isIntegerLiteral(name):
    """Checks whether name is an integer literal string.

    Args:
        name: The identifier to check

    Returns:
        True if name is an integer literal, otherwise False
    """
    result = True
    try:
        int(name)
    except ValueError:
        result = False
    return result

#==================================================================#
# Example code to exercise the above

def tryExp(env, node):
    """Try to evaluate a type printing the result or reporting errors.

    Args:
        env: The type environment in which to evaluate the expression.
        node: The root node of the abstract syntax tree of the expression.

    Returns:
        None
    """
    print(str(node) + " : ", end=' ')
    try:
        t = analyse(node, env)
        print(str(t))
    except (ParseError, TypeError) as e:
        print(e)

def main():
    """The main example program.

    Sets up some predefined types using the type constructors TypeVariable,
    TypeOperator and Function.  Creates a list of example expressions to be
    evaluated. Evaluates the expressions, printing the type or errors arising
    from each.

    Returns:
        None
    """

    var1 = TypeVariable()
    var2 = TypeVariable()
    pair_type = TypeOperator("*", (var1, var2))

    var3 = TypeVariable()

    my_env = { "pair" : Function(var1, Function(var2, pair_type)),
               "true" : Bool,
               "cond" : Function(Bool, Function(var3, Function(var3, var3))),
               "zero" : Function(Integer, Bool),
               "pred" : Function(Integer, Integer),
               "times": Function(Integer, Function(Integer, Integer)) }

    pair = Apply(Apply(Ident("pair"), Apply(Ident("f"), Ident("4"))), Apply(Ident("f"), Ident("true")))

    examples = [
            # factorial
            Letrec("factorial", # letrec factorial =
                Lambda("n",    # fn n =>
                    Apply(
                        Apply(   # cond (zero n) 1
                            Apply(Ident("cond"),     # cond (zero n)
                                Apply(Ident("zero"), Ident("n"))),
                            Ident("1")),
                        Apply(    # times n
                            Apply(Ident("times"), Ident("n")),
                            Apply(Ident("factorial"),
                                Apply(Ident("pred"), Ident("n")))
                        )
                    )
                ),      # in
                Apply(Ident("factorial"), Ident("5"))
            ),

            # Should fail:
            # fn x => (pair(x(3) (x(true)))
            Lambda("x",
                Apply(
                    Apply(Ident("pair"),
                        Apply(Ident("x"), Ident("3"))),
                    Apply(Ident("x"), Ident("true")))),

            # pair(f(3), f(true))
            Apply(
                Apply(Ident("pair"), Apply(Ident("f"), Ident("4"))),
                Apply(Ident("f"), Ident("true"))),

            # let f = (fn x => x) in ((pair (f 4)) (f true))
            Let("f", Lambda("x", Ident("x")), pair),

            # fn f => f f (fail)
            Lambda("f", Apply(Ident("f"), Ident("f"))),

            # let g = fn f => 5 in g g
            Let("g",
                Lambda("f", Ident("5")),
                Apply(Ident("g"), Ident("g"))),

            # example that demonstrates generic and non-generic variables:
            # fn g => let f = fn x => g in pair (f 3, f true)
            Lambda("g",
                   Let("f",
                       Lambda("x", Ident("g")),
                       Apply(
                            Apply(Ident("pair"),
                                  Apply(Ident("f"), Ident("3"))
                            ),
                            Apply(Ident("f"), Ident("true"))))),

            # Function composition
            # fn f (fn g (fn arg (f g arg)))
            Lambda("f", Lambda("g", Lambda("arg", Apply(Ident("g"), Apply(Ident("f"), Ident("arg"))))))
    ]

    for example in examples:
        tryExp(my_env, example)

if __name__ == '__main__':
    main()

To compile the code comprising an OWL BASIC procedure, function or main program into CIL, we must linearise the Control Flow Graph (CFG) representing the program statements. The CFG undergoes many transformations during compilation, for example to eliminate unreachable code or convert GOSUB routines into named procedures. Generation of CIL using Reflection.Emit requires that we can define branch targets in advance of generating branch instructions or marking the target instruction and of course we want to do this in a manner which minimises the number of branches required to represent the code. The structure of the graph may be quite complex, especially for traditional BASIC spaghetti code which uses GOTO excessively rather than the more structured alternatives such as procedures and functions or the control structures introduced in BBC BASIC V .

Consider the following procedure from Sphinx Adventure. It contains three loops, one on line 271 formed with a GOTO back to the start of the line, and two REPEAT .. UNTIL loops.

266 DEF PROCL(L)
267 LOCAL I,J:CO=0:CN=0
268 IF L=1 THEN278
269 PRINT’: RESTORE L: IF O?31<>0 THEN O?31=0:DW=1
270 READ R$,R$:R$="You are "+R$
271 IF LEN(R$)+ POS>CO-CN+39 THEN R$= FNS(R$,39+CO-CN):CO=CO+39: GOTO271
272 PRINT R$: IFL=136 OR L=15 THEN O?56=L
273 IF L=16 AND FL=1 THEN PRINT"The walls are very hot!" ELSE IF L=16 THEN PRINT "The walls are steaming!"
274 IF L<>3 AND L<>142 AND L<>143 THEN PROCEX(L): IF ABS(L-19)=1 AND CH=1 THEN PRINT ELSE IF ABS(L-42)=1 AND VO=1 THEN PRINT
275 IF CH=1 AND ABS(L-19)=1 THEN PROC R(22): PRINT "chasm.":O?53=L
276 IF VO=1 AND ABS(L-42)=1 THEN PROCR(22): PRINT"glacier.":O?53=L
277 IF L=26 OR L=27 THEN O?53=L
278 J=0:I=0:CO=0
279 REPEAT:J=J+1: IF O?J=L THEN CO=CO+1
280 UNTIL J=52: IF CO=0 AND L=1 THEN PROCR(L): GOTO 284 ELSE IF CO=0 AND L<>1 THEN 284 ELSE PRINT:MAX=CO
281 IF L=1 THEN PROCR(3) ELSE PROCR(4)
282 CN=0:CO=MAX: REPEAT I=I+1: IF O?I=L THEN PROCOT(I,CO):CO=CO-1
283 UNTIL I=52
284 IF D<>0 THEN O?31=L
285 IF CF=1 AND L=94 THEN PRINT’"The casket is open."
286 IF L=24 AND SA=1 THEN PRINT’"The safe door is open."
287 PRINT: ENDPROC

The CFG for this code is shown below. Each program statement is shown as a purple box, with control flow to the following statement(s). Conditionals are shown in diamond boxes. The numbers in each purple box are source line numbers, where known.

Careful comparison of the source above and the diagram below will reveal some of the transformations that have been applied to the program; for example, READ R$,R$ on line 270 has been transformed into two consecutive assignment statements which actually take the form R$ = READ() where READ() is a function not available in the source language.

The statement level CFG has been analysed to identify basic-blocks, shown as yellow group nodes, thereby defining a higher level basic-block level CFG. Each basic block has only one entry point statement; none of the statement within the basic block are destinations of other jump instructions. Furthermore, each block has only one exit point.

More text follows this long diagram…

Generating the CIL code for a single basic block is straightforward enough – we can simply iterate through the statements comprising the basic block in order and generate the code for each in turn. However, there are many possible orders in which the code for the basic block themselves could be representing in the CIL, since we can branch from the end of any block to the next block, although of course we must start at the entry block for the procedure. Although any order starting with the entry block can be made to work, where possible we would like program control to flow smoothly from the end of a block to one of its successors without requiring a branch.

At first sight, some sort of topological ordering would seem to be appropriate, but a topological ordering is only well defined for a directed acyclic graph (DAG), and a DAG this program is not. The key to this conundrum is to reduce the directed graph to a DAG by identifying strongly connected components. By contracting each SCC to a single node we obtain what is called the condensation of the CFG which will be a DAG. To the resulting DAG we can apply a topological ordering. The ordering of vertices with each SCC is chosen by starting at the vertex with the greatest in-degree.

In order to identify and contract the SCCs we use an implementation of Tarjan’s algorithm during depth first traversal of the CFG. The reverse post ordering of the primary depth first traversal is used to generate the topological ordering of the condensed CFG.

The resulting ordering of basic blocks is shown in the diagram by the numeric labels to the top-left of each. This will be the order in which the CIL code for them is generated, and it can be seen that in about half of the cases, fall through from one block to the next (consecutive block numbers) without explicit branching can be exploited. Future optimisations will focus on further simplifying the generated code by removing vertices, such as block 31, which contain only jumps.

After a long haul, and diversions into other more important projects — including starting a family — OWL BASIC today produced its first executable. Its not much. In fact its hardly anything. Just 2048 bytes of Windows PE executable containing the global variable declarations from Acornsoft’s 1982 Sphinx Adventure. Each file of BASIC source code will be converted to a single .NET static class, with the global variables as private static fields.

The first executable produced from OWL BASIC.

Above you can see the executable loaded up into .NET Reflector, which can be used to introspect the executable, and in this case attempt to disassemble it into C#. Now we see what makes .NET such a great platform for compiler construction; below is the IronPython source code for the embryonic assembly generation function. It clocks in at fewer than ten lines of code to create an assembly, create a module, create a class, add one private static field to it for each global variable, and save the result as an .exe.

def generateAssembly(name, global_symbols):
    domain = Thread.GetDomain()
    assembly_name = AssemblyName(name)
    assembly_builder = domain.DefineDynamicAssembly(assembly_name, AssemblyBuilderAccess.RunAndSave)
    module_builder = assembly_builder.DefineDynamicModule(name + ".exe")
    type_builder = module_builder.DefineType(name, TypeAttributes.Class | TypeAttributes.Public, object().GetType())

    # Add global variables to the class
    for symbol in global_symbols.symbols.values():
        field_builder = type_builder.DefineField(symbol.name, ctsType(symbol),
                                                 FieldAttributes.Private | FieldAttributes.Static)

    result = type_builder.CreateType()
    assembly_builder.Save(name + ".exe")

where global_symbols is the global symbol table constructed during traversal of the Abstract Syntax Tree and the Control Flow Graph and the ctsType function maps OWL BASIC types to their equivalent Common Type System types for .NET. Everything else is provided by Reflection.Emit and other parts of .NET.

Its interesting that no validation was applied to the variable names supplied to Reflection.Emit. As you can see, the variable names still include the sigil suffixes for variable typing (e.g. $ for string) and Reflector happily dissassembles these into invalid C# identifiers. For the final version these names will need to be mangled (Hungarian notation?), or merely de-sigiled if no conflicts result, for compatibility with other .NET languages and tools.

Earlier today I posted the second article in what is turning out to be a short series in the investigation into why the performance of IronPython is around 100× slower than CPython, when running the front-end of my OWL BASIC compiler.

The most informative comment was from Curt Hagenlocher who works on IronPython in the Visual Studio Managed Languages group at Microsoft.

Curt suggested,

Try storing the counter as a global variable instead of a class-level member of Node — I think you’ll notice a dramatic improvement.

The modified benchmark program looks like this:

counter = 0

class Node(object):

    def __init__(self, children):
        global counter
        counter += 1
        self._children = children

def make_tree(depth):
    if depth > 1:
        return Node ([make_tree(depth - 1), make_tree(depth - 1)])
    else:
        return Node([])

def main(argv=None):
    global counter
    if argv is None:
        argv = sys.argv
    depth = int(argv[1]) if len(argv) > 1 else 10

    root = make_tree(depth)
    print counter
    return 0

if __name__ == '__main__':
    import sys
    sys.exit(main())

A dramatic improvement!

Well, Curt wasn’t wrong. This made a phenomenal difference with IronPython completing in only 12% of the time taken by CPython – over 8× faster with a binary tree depth of 20.

Let’s look in detail at the results. All results are from a dual quad-core 1.86 GHz Xeon with 4 GB RAM, and as before each benchmark was run five times, and the shortest time of the five taken.

The three test environments are:

1. Python 2.5.2 x86 32-bit
2. Jython 2.5rc2 on Java Hotspot 1.6 32-bit
3. IronPython 2.0 on .NET 2.0 x64

Figure 1. The relative performance of the three main Python implementations on a benchmark that uses a global counter, rather than mutating a class attribute.

Here we can see how IronPython’s performance has been improved hugely by this simple change. Although startup time dominates for the smaller problem size, now both Jython and IronPython surpass CPython at around half-a-million nodes.

Removing start-up time, which may be irrelevant for long-running processes, gives us the following chart:

Figure 2. The performance of the three main Python implementations excluding start-up time.

Again there is a lot of noise in the data below 1000 nodes, but it is clear that Jython scales better than IronPython, which in turn is scaling better than CPython.

Up until now I’ve been using a log-log scale in the charts because of the wide variation in performance between the different implementations, but now the performance gap is much closer, it’s difficult to get a sense of just how much faster IronPython is on the modified benchmark. Let’s throw in a log-linear plot to help us appreciate what’s going on:

Figure 3. A linear representation of the same data as in Figure 2, to highlight the performance multiple between IronPython and CPython in the larger tests.

It’s perhaps easier to see now that IronPython is doing in 14 seconds what takes CPython 114 seconds to achieve!

Finally, let’s plot those results as we did before, as multiples of CPython performance:

Figure 4. Execution time of Jython and IronPython as multiples of CPython performance.

It is easy to see that in this chart, once we pass half-a-million tree nodes (a tree depth of 19) that both Jython and IronPython are significantly beating CPython.

Explanation

Curt Hagenlocher offers the explanation in a comment in the thread on Reddit.

In this particular case, IronPython is slow because of the update to Node.counter. Currently, any update to a class will increment the version number for the class, which will have the effect of invalidating any rules compiled for that class. Effectively, the same rules are getting compiled over and over again. Moving the counter to a global should result in performance on par with that of CPython.

which is absolutely correct, except that he’s underselling the relative gain. IronPython is not only on a par with CPython, it can outperform it by a factor of eight!

With this knowledge in hand, I can now approach optimization of my OWL BASIC compiler, which lies back at the start of this illuminating tale.

Conclusion

• Avoiding mutation of Python class attributes can have significant benefits for IronPython performance.
• Both IronPython and Jython scale better than CPython by this benchmark, and have superior performance for large trees of nodes.

My previous post covering the performance problems I’ve been experiencing with IronPython raised some questions about whether the low performance was an effect peculiar to my system, or to my program — the OWL BASIC compiler — where the problem was first noticed. To briefly recap, I’d determined that IronPython was around 100× slower that CPython on the same program.

Since then, I’ve had time to reproduce the results with a small and completely unremarkable Python program, and also to run the tests on a different system. I had suspected that in the OWL BASIC compiler, my Python visitor implementation, which is used in applying transformations to the abstract syntax tree, was to blame. I set about condensing a tree visitor down to a small example, but I never got that far. It is sufficient to simply build a large binary tree to demonstrate the dramatic differences in the performance characteristics of the three main Python implementations.

The benchmark

Here is that test program, which just builds a simple binary tree of objects to the requested depth.

class Node(object):
    counter = 0

    def __init__(self, children):
        Node.counter += 1
        self._children = children

def make_tree(depth):
    if depth > 1:
        return Node ([make_tree(depth - 1), make_tree(depth - 1)])
    else:
        return Node([])

def main(argv=None):
    if argv is None:
        argv = sys.argv
    depth = int(argv[1]) if len(argv) > 1 else 10

    root = make_tree(depth)
    print Node.counter
    return 0

if __name__ == '__main__':
    import sys
    sys.exit(main())

The program builds a binary tree to the depth supplied as the only command line argument, or ten if one is not supplied. It counts the number of nodes as they a built. Remember that the merits or otherwise of this program are not the point! The point is the performance difference between the Python implementations when it is run.

My benchmarking approach has been to run this script five times for each tree depth from a depth of one, upwards to 22, or until my patience was exhausted. I’ve taken the minimum time from each run of five. Since there is a non-linear relationship between the depth of the tree and the number of nodes contained therein, logarithmic axes are used in all the charts that follow.

64 bit Windows Vista x64

Here are the results for the first test machine – with dual quad-core 1.86 GHz Xeons with 4 GB RAM running Vista x64, testing IronPython 2.0.0.0 on .NET 2.0, Jython 2.5rc2 on Java Hotspot 1.6.0 and Python 2.5.2.

Figure 1. Creation time for a binary tree including Python virtual machine startup on Windows Vista x64 with 1.86 GHz Xeon processors.

In Figure 1 we see that above 1000 nodes or so (tree depth of 10) performance for IronPython begin to degrade rapidly. CPython holds out for another two orders of magnitude before the significant costs begin to be felt . Its interesting to see that although Jython is in the middle of the pack, it scales much better than CPython, surpassing it at around half-a-million nodes (tree depth of 19).

In my application — a compiler — virtual machine (VM) start-up time is important; however, in many long-running applications this is not the case, so it is interesting to subtract VM start-up time from each series, which we see in Figure 2, below.

By subtracting VM start-up time, we get a picture more interesting for long-running processes.

Below 100 tree nodes, there is a lot of noise in these measurements. Above 100 nodes its easy to see that the blue IronPython curve is at least two chart divisions above the red CPython curve — that’s two orders of magnitude or 100× slower, and getting relatively worse as the size of the tree increases.

32 bit Windows XP x86

Responses to my earlier article suggested that trying IronPython 2.0.1 with Ngen’ed binaries on x86 may make a difference. Well, to cut a long story short, it doesn’t, but here are the details. These tests were run on a 900 MHz Pentium M Centrino laptop with 768 MB RAM, and so cannot be directly compared with those above, although its notable that a one year old workstation is only twice as fast as a five year old laptop. Moore’s law certainly isn’t delivering here!

The performance profiles are very simular with IronPython 2.0.1 on x86.

On x86, IronPython is still 100× slower than CPython, and Jython still scales better. It seems the essence of this benchmark is not dependent on which hardware or CLR platform it is run.

I’ll close by re-presenting the data in the x86 benchmarks as multiples of CPython performance, because it dramatically demonstrates the different responses to the scale of the problem size for IronPython and Jython. Again we see Jython catching up with CPython at a tree depth of 19, just we saw on x64. and IronPython delivering 6000× worse than CPython at a tree depth depth of 15. A tree of this size with thirty-thousand nodes is very similar in scale to the AST tree sizes found in the OWL BASIC during compilation of large programs.

Performance of IronPython and Jython as multiples of CPython performance.

Conclusions

• IronPython can be very slow, even on programs in the microbenchmark category, which are doing standard operations such as building trees. Presumably there are still significant optimizations to be made in IronPython to bring its performance closer to that of the other Python implementations. Hopefully, this example and the measurements can contribute to that improvement.
• Jython may scale better than Python if your application exercises Python in similar ways to this benchmark. Speculatively, that could have implications for projects such as SCons, which build large in-memory graphs.
• I suppose if nothing else we have demonstrated in passing that Java can be faster than C for some non-trivial programs (like a Python interpreter) running a trivial program, like this benchmark.

Significant claims have been made about the performance of IronPython, notably back at its inception in 2004 when Jim Hugunin, creator of both IronPython and its cousin Jython, presented a paper on IronPython performance at PyCon. Since then, there have been numerous claims to IronPython’s supremacy over CPython in the performance stakes. The IronPython Performance Report reiterates that IronPython can turn in a good performance. According to Hugenin the standard line we’ll see is,

“IronPython is fast – up to 1.8x faster than CPython on the standard pystone benchmark.

But do these claims stand up in the face of real-world Python code?

The claims of good performance are based on synthetic micro-benchmarks which don’t accurately reflect balance of coding techniques found in complete programs.

At this point I’d like to offer my own quote:

“IronPython can be slow – 10x to 100x slower than CPython on real-world code and it has been observed to be up to 6000x slower.

Now, the unfortunate thing about real-world code is that it hasn’t been hand-crafted to highlight the performance characteristics of some aspect of the language implementation like your typical micro-benchmark. Its been written, most likely without any attention to performance. This is especially in the case of a prototype, as in my case.

Is my code really that bad?

Over the past few years I’ve been working on a hobby project called OWL BASIC, which is a compiler for the BBC BASIC lanaguage targeting the .NET Common Language Runtime (CLR). At the outset of the project I decided to write the compiler itself in Python, for much the same reasons as PyPy is written in Python — hackability. I planned specifically to use IronPython so I could benefit from access to useful .NET libraries such a Reflection.Emit, for generating Common Intermediate Language (CIL) assemblies.

During the course of developing OWL BASIC, as the code has become more complex, I’ve been consitently disappointed by the negative delta between the promise and reality of IronPython’s performance. The poor performance of IronPython has negated, for me, one of the main advantages of developing in Python — the rapid edit-run cycle.

Was my code really so inefficient? Performance was never a goal of the project, but the underwhelming performance has threatened the viability of my approach and made me question the wisdom of my chosen route of writing a compiler in Python.

The absence of profiling tools for IronPython led me down the road of getting at least the compiler front-end to work on CPython, so I could use the standard Python profilers. Fortunately, my code was portable (its just regular Python) and so I determined that with a few inconsequential tweaks to my code, the entire compiler front-end can be run on the trinity of CPython, Jython and IronPython.

I was, to say the least, somewhat surprised by the results.

The evidence: performance of CPython, Jython and IronPython

The investigation into the relative performance of the three main Python implementations was centered on running my unmodified, unprofiled and unoptimized compiler front-end on third-party BBC BASIC source-code (Acornsoft Sphinx Adventure) and measuring the execution time. All tests runs were performed five times, and the minimum time of the five chosen. Variance between the readings on successive runs was small. The following Python implementations were on the test-bench:

• Python 2.5.1 (x86)
• Jython 2.5rc2 on Java HotSpot 1.6.0_10-rc
• IronPython 2.0 on .NET 2.0 (x64)

All tests were run on a eight-core 1.86 GHz Xeon with 4 GB RAM running Vista Ultimate x64.

The following chart shows the results of running the compiler over the source code for Sphinx Adventure.

The absolute performance of CPython, Jython and IronPython on my code

Frankly, I was astounded. IronPython was left in the dust, not only by CPython but also Jython! Overall, CPython was 93× faster on the exact same code. The IronPython hyperbole, for now I could see that was what is was, had led to me expect numbers similar to those for CPython, although I had perhaps more realistic expectations that performance would be similar to Jython.

At this point I assumed I’d hit some corner case in which IronPython was performing relatively badly. I’d had a similar experience early on in the project with code from the PLY parsing package causing IronPython 1.1 to perform badly, but I’d worked around the issue by modifying parts of PLY to use pickling rather than eval-ing large list literals for the cached parsing tables. [Its worth noting in passing that this problem still exists with the IronPython and PLY combination - I'll publish my solution in another post].

I decided to dig in a little more detail and collect some timing data on the sequence of top level calls the compiler front-end makes to parse the source code followed by a sequence of transformations to the abstract syntax tree (AST). The chart below shows these top level calls, in the order in which the occur during compilation:

The ratio of performance of CPython to both IronPython and Jython when running the OWL BASIC compiler

As you can see, I have had to resort to a logarithmic scale in order to convey the huge variation in the performance of IronPython relative to CPython, ranging from buildParser with a multiple of 3.8 to convertSubroutinesToProcedures with a multiple of over 6400. Even if we ignore this outlier (maybe we are suffering from a garbage collection – but notice that Jython is also relatively much slower on this function) we can see that Jython is typically 5× to 10× slower than CPython whereas IronPython is typically 10× to 100× slower than CPython.

Notice also that there is a marked decline in the performance of IronPython from the parse function onwards; the common factor in these operations are that they are all transformations to the AST, and my AST node classes are instantiated by a Python metaclass, although its pure speculation on my part that metaclasses are the cause of this performance drop.

Conclusion

On my program at least, Jython is 6× slower than CPython and IronPython is nearly 100× slower. If you’re suffering from poor performance with IronPython, it may be well worth your time checking performance of your code on the other Python implementations, if that is an option for you.

Now need to find the root cause and boil my problem down to a short example which can form the basis of a bug report to the IronPython team. Given that the problem is pervasive in my code, that won’t be hard.

See you at EuroPython 2009 !

For a useful re-implementation of BBC BASIC, especially in compiled form, a run-time system is needed to provide services to the running program which cannot be directly provided by the operating system. In the case of our OWL BASIC implementation, we are targeting .NET so we already have a very sophisticated run-time in the shape of the CLR (Common Language Runtime). Nevertheless, additional services specific to the BBC BASIC language are required, many of them based closely on old Acorn hardware from the BBC Micro and RISC OS eras. Fortunately, many of the services required to produce a functioning BBC BASIC system are fairly well abstracted from hardware details, as the many different implementations on quite different underlying hardware, such as BBC BASIC for Windows, have shown.

Many of the services, such as the VDU drivers, used extensively by BBC BASIC are not strictly part of the language; however, a fairly complete implementation is required in order that existing programs can be run successfully with little or no modification.

Its difficult to know where to draw the line when deciding on the degree of backwards compatibility. Our aim is to get the most widely used features to work well. Some of the obscure features may never be implemented. We must also strike a balance between .NET compatibility and good interoperability with other .NET languages and tools, and what I will call Acorn compatibility, for want of a better term.

Implementation details

Our OWL Runtime Library (ORL) is coding in C# and leans heavily on the existing .NET framework. We plan to support common RISCOS SWI calls and a fairly complete VDU system. Much of the work by my brother Ian and myself has to date has focussed on the VDU system, which uses GDI+ through the System.Drawing2D namespace of .NET to do most of the heavy lifting.

Compatibility Testing

The testing approach has been as follows:

1. Find write a simple but interesting BBC BASIC program using graphics commands, such as GCOL, MOVE, PLOT and VDU. Execute this program on Acorn hardware or an emulator to get the graphics output.
2. Convert all graphics commands to their equivalent VDU sequences. For example GCOL 0, 3 becomes VDU 18, 0, 3, and verify that program output is the same
3. Write a test in C# against our OWL Runtime Library using the VDU codes, and very that output is sufficienly similar.
4. Compile and run the original BBC BASIC test program using the OWL BASIC compiler and verify the output

Currently we are pursuing steps 1-3 above as the compiler is not yet ready for step 4. This enables us to work on the run-time library and compiler concurrently.

Example program

The following is a fragment from a larger example program.

REM BBC BASIC using graphics commands
size% = 220
radius% = 250
FOR t% = 1 TO 4
radius -= 30
c% = 3
cntr% = 0
cd% = 0
FOR x2% = 100 TO 450 STEP 20
cntr% += 1
GCOL 0, (c% AND 63)
TINT 2, (t% – 1) * 64
MOVE size, size
MOVE size + (COS(RAD(x2% – 20)) * radius%), size% + (SIN(RAD(x2% – 20)) * radius%)
PLOT 85, size + (COS(RAD(x2%)) * radius%), size% + (SIN(RAD(x2%)) * radius%)
CASE cntr% OF
WHEN 1: cd% = 4
WHEN 4: cd% = -1
WHEN 7: cd% = 16
WHEN 10: cd% = -4
WHEN 13: cd% = 1
WHEN 16: cd% = -16
ENDCASE
c% += cd%
NEXT
NEXT

which can be converted to the equivalent program which uses only VDU commands:

size% = 220
radius% = 250
FOR t% = 1 TO 4
radius -= 30
c% = 3
cntr% = 0
cd% = 0
FOR x2% = 100 TO 450 STEP 20
cntr% += 1
VDU 18,0,(C AND 63)
VDU 23, 17, 2, (T – 1) * 64, 0,0,0,0,0,0
VDU 25, 4, size%; size%;
VDU 25, 4, size% + (COS(RAD(x2% – 20)) * radius%); size% + (SIN(RAD(x2% – 20)) * radius%);
VDU 25, 85, size + (COS(RAD(x2%)) * radius%); size% + (SIN(RAD(x2%)) * radius%);
CASE cntr% OF
WHEN 1: cd% = 4
WHEN 4: cd% = -1
WHEN 7: cd% = 16
WHEN 10: cd% = -4
WHEN 13: cd% = 1
WHEN 16: cd% = -16
ENDCASE
c% += cd%
NEXT
NEXT

These programs both give the following output in Arculator :

Translating the second version above into C# against our run time library, we get the following:

const short size = 220;
int radius = 250;

for (int t = 1; t < 5; ++t)
{
    radius -= 30;
    int c = 3;
    int cntr = 0;
    int cd = 0;

    for (int circle = 100; circle < 450; circle+=20)
    {
        cntr++;
        // TINT action (2) color (t * 64)
        vdu.Enqueue((byte)23, (byte)17, (byte)2, (byte)((t-1) *64));
        vdu.Enqueue((byte)0, (byte)0, (byte)0, (byte)0, (byte)0, (byte)0);

        // GCOL action (0) color (c & 63)
        vdu.Enqueue((byte)18, (byte)0, (byte)(c & 63));

        short mov1x = (short)(size + (Math.Cos((circle - 20) * (Math.PI / 180)) * radius));
        short mov1y = (short)(size + (Math.Sin((circle - 20) * (Math.PI / 180)) * radius));
        short mov2x = (short)(size + (Math.Cos((circle) * (Math.PI / 180)) * radius));
        short mov2y = (short)(size + (Math.Sin((circle) * (Math.PI / 180)) * radius));

        // MOVE
        vdu.Enqueue((byte)25, (byte)4);
        vdu.Enqueue(size, size);

        // MOVE
        vdu.Enqueue((byte)25, (byte)4);
        vdu.Enqueue(mov1x, mov1y);

        // PLOT triangle fill
        vdu.Enqueue((byte)25, (byte)85);
        vdu.Enqueue(mov2x, mov2y);

        switch (cntr)
        {
            case 1:
                            cd = 4;
                            break;
            case 4:
                            cd = -1;
                            break;
            case 7:
                            cd = 16;
                            break;
            case 10:
                            cd = -4;
                            break;
            case 13:
                            cd = 1;
                            break;
            case 16:
                            cd = -16;
                            break;
        }
        c += cd;
    }
}

which renders the following display on Windows Vista,

The Windows version is of higher quality because we have double the vertical resolution since Windows is used square pixels each half the size of the original rectangular pixels used in this screen mode on RISCOS. This is an example of a difference between historical implementations and our new implementation which we do not intend to address.

However, its possible to improve quality even further using the facilities of GDI+. We have implemented a new code (in the reserved for future expansion block) VDU 23, 28, n the purpose of which is to control rendering quality. In the current version we have implemented three quality levels. With the highest quality level, we get the following output:

which was simply never possible with the original Acorn BBC BASIC.

Since BBC BASIC does not use C-like nested scopes marked by braces to delineate loops and other blocks, the structure of such blocks is not captured in the Abstract Syntax Tree (AST); that is to say that the statements within a loop will not be child nodes of the opening loop statement such as FOR or WHILE. Furthermore, there is not necessarily a one-to-one correspondence between loop entry points such as FOR, and loop ends such as NEXT – a FOR loop may contain more than one NEXT statement, and what is more, in convoluted cases involving GOTOs, a single NEXT statement may be re-used by several FOR loops. Next please! Compiling Iteration Structures in BBC BASIC for more information.

It is therefore required to pair each closing loop statement (e.g. NEXT or ENDWHILE) with its corresponding opening loop statement(s), and this is turn requires that the possible paths of execution through the program can be determined statically (i.e. at compile time) so that for each closing statement we can trace backwards through the program execution to the previous opening statement. This also makes it possible to identify illegal constructs, such as ending a WHILE loop with a NEXT statement.

To achieve this, we must build a data structure called the Control Flow Graph (CFG). For each statement in the AST we determine all possible following statement. Ordinarily this is simply the next statement in the program, but we must also resolve line number references in GOTO, GOSUB, etc. and at each IF statement control flow may go in either of two directions. We do not resolve function or procedure calls when building the CFG, so each function or procedure becomes a separate connected-component in the resulting graph.

I’ve been working with the code to Acornsoft’s Sphinx Adventure from 1982 as a real-world BBC BASIC test program for the BBC Micro.

I don’t know who the author is, but I’m willing to bet they never imagined their by now long-forgotten code would undergo the sort of analysis its about to receive! Maybe the code in Sphinx Adventure is intentionally obfuscated, or maybe its just badly written, but some of the non-structured programming techniques such as jumping out of the middle of procedures never to return, are ‘interesting’ to say the least.

Each node in this image is a single statement in the program, and the arrows between the nodes indicate the program flow. Each connected group of nodes is a separate procedure, and the largest group of nodes to the top-left of the chart is the main program starting from the first line through a long trail of DATA statements.

The Sphinx.graphml is browsable with yEd, which was also used to produce these diagrams.

Zooming in, we can see the CFG for a single procedure – in this case PROCF,

462 DEF PROCF(W): IFW=28 THEN PROCR(21): GOTO465
463 IFO?W <> L ANDO?W <> 1 THEN PROCR(7): PRINTA2$;" here.": GOTO465 ELSE IFO?W2 <> 1 THEN PROCR(8): PRINTA2$;".": GOTO465
464 IFW=27 ANDL=O?28 ANDWA=0 THEN PRINT"Your bottle is now full of water.":WA=1 ELSE IFW=27 ANDL=O?28 THEN PRINT"your bottle is already full." ELSE PROCR(21)
465 ENDPROC

The label in each node indicates its logical line number in the original code and the type of statement the node represents at the abstract syntax level.

Now we have the basic CFGs we can analyse them to match up corresponding loop statements, and optionally perform optimizations such as moving invariant code out of loops and eliminating dead (unreachable) code fragments.

Other personal projects, of both a software and familial nature have taken priority lately over my project to implement a BBC BASIC-a-like compiler for the .NET CLR, now called OWL BASIC in honour of the owl logo of the BBC’s Computer Literacy Project. Now I have some spare capacity to devote to the project, and my brother Ian, who is part-way through his Computing Science degree and is another long-time BBC/RISC OS/Acorn computing enthusiast has joined the project – he’s starting work on the optional front-end detokenizer. We have opted to handle tokenized source files by detokenizing into text, and then retokenizing with our lexer – the reason being that the tokenized BBC BASIC files do not contain all of the tokens we need for our grammar, and detokenizing even very large programs is mimimum overhead compared to the parsing and complilation.

In this post, I want to take a look at the structure of the PLY parser and the classes defining storage for the Abstract Syntax Tree that is the product of streaming the lexer tokens through the parser.

IronPython with PLY

PLY requires that the grammar rules are placed in Python docstrings within methods that follow a specific naming convention; these docstrings are then parsed during parser initialisation to build the required parsing tables. These tables are then cached in files for quicker start-up on subsequent runs. PLY stores is cached tables as large files of generated Python code containing some very long expressions. Unfortunately, these large expressions resulted in pathological behaviour of the IronPython parser, and stack overflow problems (although they were fine with regular CPython), and so some modifications to the PLY source was required to use Python pickling, rather than code generation, for storage and retrieval of the cached parsing tables. Performance is now much better, with a compiler startup time of many minutes reduced to a few seconds.

BBC BASIC grammar structure

I’ll start by pointing out that our current grammar definition is not perfect; even after the suitably caffeinated debugging session recommended in the PLY manual we still have 14 shift/reduce conflicts and 12 reduce/reduce conflicts. The former group are not serious, but the latter need to be addressed. There is no guarantee that it’s possible to parse BBC BASIC with a look-ahead left-to-right (LALR) grammar, owing the lack of a formal grammar definition for the language. The issues are resolvable, but may require stepping outside of the facilities provided by PLY – as we have done already for DATA statements.

The main structural part of our grammar is as follows:

program : statement_list;

statement_list : compound_statement
               | statement_list compound_statement;

compound_statement : lone_stmt_body stmt_terminator
                   | multi_statement stmt_terminator;

multi_statement : statement statements_tail;

statements_tail : statement_separator statement statements_tail
                | empty;

statement_separator : COLON;

stmt_terminator : EOL;

statement : stmt_body
          | empty;

lone_stmt_body : case_stmt;

stmt_body : beats_stmt
          | bput_stmt
          | call_stmt
          | circle_stmt
         ...
          | while_stmt
          | width_stmt
          | wait_stmt;

We have two recursively defined rules here: statement_list is defined in terms of itself, as is statements_tail. The former is used for collecting lists of statements or compound statements separated by line-breaks. The latter is used for accumulating statements forming compound statements separated by colons. The correct grammar for this was awkward to determine because many published grammars are for C-style languages which use statement terminators (e.g. semicolon) whereas BASIC uses Pascal-style statement separators, which can run in continguous sequences with empty statements. Some inspiration from a Pascal grammar and experimentation with UltraGram led to workable solution. A minority of statements in BBC BASIC, such as CASE must occupy their own line, which must be handled in the grammar as lone_stmt_body – the less fussy majority of statements make up stmt_body:

Abstract Syntax Tree construction

As the parser matches each rule, an Abstract Syntax Tree (AST) node is generated. I’ll use the BPUT rule as a simple example which shows many features:

def p_bput_stmt(p):
    '''bput_stmt : BPUT channel COMMA expr
                 | BPUT channel COMMA expr SEMICOLON'''
    if len(p) == 5:
        p[0] = Bput(channel = p[2], data = p[4], newline=True)
    elif len(p) == 6:
        p[0] = Bput(channel = p[2], data = p[4], newline=False)
    p[0].lineNum = p.lineno(1)

Grammar rule encoding

PLY encodes grammar rules in the doc-string of module level functions which accepts p, a list of the productions of other rules referenced such as channel, or terminals (i.e. tokens) such as COMMA. Our BPUT token represents the string “BPUT” in the source, without the following hash. The way the manual is written seems to suggest that the hash is part of the BPUT keyword, but simple experimentation with the BASIC interpreter, by inserting a space as in BPUT #f%, 65 shows that the hash is a prefix to the integer representing a channel number.

The BPUT command has two syntaxes: The first form accepts a channel number (an single value prefixed by a hash) and an expression. The second form accepts an expression followed by a semicolon. There is no mention here of the type of the expressions – type-checking is a phase distinct from parsing which comes later. These two forms of BPUT are not one-for-one equivalent with the two forms given in the BBC BASIC guide,

Syntax
        (1) BPUT#factor,numeric-expression
        (2) BPUT#factor,string-expression[;]

since our first rule matches syntaxes 1 and 2 (without the semicolon) and our second rule matches only syntax 2 (with the semicolon), illustrating how the syntax information in the manual must be converted into a YACC grammar suitable for PLY.

Creating the AST node

Crucially, our two syntax rules are of different length allowing us to use the length to determine which rule was matched. By PLY convention, the rule must place its resulting AST node in p[0]. For either match we create a Bput AST node which is a Python object representing BPUT and configure it with keyword parameters for the channel number, the data to be output, and a flag indicating whether a newline should be output following the data (n.b. this also depends on the type of data – we only output a newline for string data – by we account for that later in the type-checker once we have determined the type.) Each of these items will itself be an the root of an AST substree of nodes representing anything from a fairly mundane literal integer, to a complex expression. As a final step in configuring our new AST node, we attach line number information which is only available from terminal tokens – in this case we use the line-number from the BPUT token itself.

Declaring the AST node classes

Our requirements for our AST node classes are as follows:

• uniform interface to facilitate AST traversal for the later compiler stages
• convenient syntax for declaring the structure of the many different types of AST nodes the langauge requires
• conventient syntax for initializing new AST nodes as they are constructed by the grammar rules
• attach formal type information and other information such as line numbers to the nodes

Each AST node class started being writting out long hand, but it was clear that this was going to be a lot of boilerplate code for each class in functions which could not easily be factored out into base classes, such as __init__ methods. What was needed was a code generator for producing the classes based upon a simple specification, and this is where Python metaclasses come into their own.

Our specification for BPUT is something like this,

1. A sub-node expression evaluating to a channel integer
2. A sub-node expression evaluating to a string or a number (i.e. a non-array type – a scalar
3. A flag indicating whether a newline should be appended after the data

which we encode in Python using a declarative style as:

class AstNode(object):
    "AstNode - base class for all AST nodes"
    __metaclass__ = AstMeta

    formal_type = TypeOption()
    actual_type = TypeOption()
    line_num = IntegerOption()

    # ... Code omitted here ...

class AstStatement(AstNode):
    "AstStatement - base class for all statements"
    formal_type = TypeOption(VoidType) # All statements have a type of 'void'
    actual_type = formal_type

class Bput(AstStatement):
    "BPUT"
    channel = Node(formalType=ChannelType)
    data    = Node(formalType=IntegerType)
    newline = BoolOption(False)

The work in transforming this declarative style into the boilerplate code we need is done by the AstMeta metaclass which introspects each class as it is constructed and converts our declarative style class attributes above, such as Bput.channel into the getters, setters, properties and intializers and storage we need, in the resulting class. Our declared Nodes can accept AST subtrees, whereas our Option types store simple integer, string, or bool properties attached to the node.

Thus, the contents of the Bput that is generated are quite different to what is shown above. When the Bput class is instantiated, its metaclass actually produces something which behaves like this:

class AstNode(object):
    "AstNode - base class for all AST nodes"

    def __init__(self, lineNum = None):
        self.lineNum = lineNum

    # Line Number
    def _getLineNum(self):
        return self._getOption("line_num")

    def _setLineNum(self, line_num):
        self._setOption("line_num", line_num)

    formalType = property(getLineNum, setLineNum)

class AstStatement(AstNode):
    "AstStatement - base class for all statements"

    def __init__(self, formalType = None, actualType = None, **kwargs):
        self.formalType = None
        self.actualType = None
        super(AstStatement, self).__init__(kwargs)

    # Formal Type
    def _getFormalType(self):
        return self._getOption("formal_type")

    def _setFormalType(self, formal_type):
        self._setOption("formal_type", formal_type)

    formalType = property(getFormalType, setFormalType)

    # Actual Type
    def _getActualType(self):
        return self._getOption("actual_type")

    def _setActualType(self, actual_type):
        self._setActualType("actual_type", actual_type)

    actualType = property(getActualType, setActualType)

class Bput(AstStatement):
    "BPUT"

    # Channel
    def _getChannel(self):
        return self._getProperty("channel")

    def _setChannel(self, channel):
        self._setProperty("channel", channel)

    channel = property(getChannel, setChannel)

    # Data
    def _getData(self):
        return self._getProperty("data")

    def _setChannel(self, data):
        self._setProperty("data", data)

    data = property(getData, setData)

    # Newline
    def _getNewline(self):
        return self._getOption("newline")

    def _setNewline(self, data):
        self._setOption("newline", newline)

    newline = property(getNewline, setNewline)

This provides a uniform programming interface for the Bput object allowing simple manipulation of the Bput object itself, but also a standard interface to its child nodes allowing simple traversal by the various visitors which transform and analyse the AST in the later stages of the compiler.

Visiting the AST

Once the AST has been created by the parser, it can be analysed or manipulated by a visitor. One such visitor, which is an aid to debugging, transcribes each AST node into an XML representation. This is useful since it allows the AST to be conveniently examined using standard XML capable tools (e.g. a web browser) to aid debugging.

The following code, taken from line 374 of Acornsoft’s Sphinx Adventure,

378 IF ABS(L-26.5)<1 ANDO?34=26 THEN PROCR(27): GOTO378

results in the following AST when rendered as XML following various other visitor passes to perform some simplification and type-checking,

<If line_num="374" actual_type="Void" formal_type="Void">
  <false_clause>
    <None />
  </false_clause>
  <true_clause>
    <CallProcedure line_num="374" actual_type="Void" formal_type="Void" name="R">
      <actual_parameters>
        <LiteralInteger line_num="374" value="27" actual_type="Integer" />
      </actual_parameters>
    </CallProcedure>
    <Goto line_num="374" actual_type="Void" formal_type="Void">
      <target_logical_line formal_type="Integer">
        <LiteralInteger line_num="374" value="378" actual_type="Integer" />
      </target_logical_line>
    </Goto>
  </true_clause>
  <condition formal_type="Integer">
    <And line_num="374" actual_type="Integer" formal_type="Integer">
      <lhs formal_type="Integer">
        <LessThan line_num="374" actual_type="Integer" formal_type="Integer">
          <lhs>
            <AbsFunc line_num="374">
              <factor formal_type="Numeric">
                <Minus line_num="374" actual_type="Float">
                  <lhs formal_type="Numeric">
                    <Variable line_num="374" actual_type="Float" identifier="L" />
                  </lhs>
                  <rhs formal_type="Numeric">
                    <LiteralFloat line_num="374" value="26.5" actual_type="Float" />
                  </rhs>
                </Minus>
              </factor>
            </AbsFunc>
          </lhs>
          <rhs>
            <LiteralInteger line_num="374" value="1" actual_type="Integer" />
          </rhs>
        </LessThan>
      </lhs>
      <rhs formal_type="Integer">
        <Equal line_num="374" actual_type="Integer" formal_type="Integer">
          <lhs>
            <DyadicByteIndirection line_num="374" actual_type="Byte" formal_type="Byte">
              <base formal_type="Address">
                <Cast line_num="374" actual_type="Address" source_type="Float" target_type="Address" formal_type="Address">
                  <value>
                    <Variable line_num="374" actual_type="Float" identifier="O" />
                  </value>
                </Cast>
              </base>
              <offset formal_type="Integer">
                <LiteralInteger line_num="374" value="34" actual_type="Integer" />
              </offset>
            </DyadicByteIndirection>
          </lhs>
          <rhs>
            <LiteralInteger line_num="374" value="26" actual_type="Integer" />
          </rhs>
        </Equal>
      </rhs>
    </And>
  </condition>
</If>

This posting has been re-written from an earlier version which contained unnecessary complexity

Following on from my proposal on how to compiler BBC BASIC functions with variant return types, in response to Richard Russell’s compilation challenges, I’d like to move on to compiling BBC BASIC loop structures. Nonetheless, several readers have comments how useful even a compiler that did not support these features would be.

The claimed difficulty arises is because BBC BASIC maintains a stack at run-time for managing loop structures such as FOR..NEXT, REPEAT..UNTIL and WHILE..ENDWHILE. As the interpreter executes the program it presumably uses the instructions it encounters to manipulate this stack. They key point here is that the tokens listed above are not merely loop delimiters used by the parser, they are actually statements which modify the internal state of the interpreter.

The situation is further complicated by the quirky behaviour of the NEXT statement. First of all, the loop variable following the NEXT is optional. If it is omitted the next iteration of the FOR loop which is topmost on the stack (i.e. the inner-most loop) is executed. However, it is also possible to supply a comma-separated list of loop variables, for example NEXT i%,j%. This is equivalent to NEXT i% : NEXT j%.

Richard goes on to illustrate the point with a real-world example,

Here there are two NEXT I% statements; which gets used depends at run-time on the contents of the array. This is a real-world example (although the code could no doubt be written more elegantly).

DEF FNscan(A()) : LOCAL I%
FOR I% = 0 TO DIM(A(),1)
IF A(I%) = 0 THEN
I% = DIM(A(),1)
NEXT I%
= TRUE
ENDIF
REM Some more code here
NEXT I%
= FALSE

This time, I will use a different example, which is very similar in structure to Richard’s code, but which excludes the array, for the sake of clarity.

FOR i% = 1 TO 5
IF i% = 3 THEN
PRINT "Three!"
NEXT
ENDIF
PRINT "i% is ";i%
NEXT

Note there is one FOR statement and two NEXT statements, so it is not possible to pair them. I simply print the loop counter, or in one case print a different string and NEXT early.

Formatted like this we can see that the inner NEXT is exactly like the continue statement in C#.

for (int i = 1; i <= 5 ; ++i)
{
   if (i == 3)
   {
      Console.WriteLine("Three!");
      continue; // next
   }
   Console.WriteLine("i% is {0}", counter);
   continue; //next
}

So, we can do something isomorphic in C#, C# is compilable to CIL, ergo this particular BBC BASIC construct is compilable to C#. Obviously.

However, more nefarious constructs are possible in BBC BASIC, such as this:

FOR i% = 1 TO 5
FOR j% = 1 TO 5
PRINT "i% = ";i%,"j% = ";j%
NEXT i%

where we iterate on the outer loop, from ‘within’ the inner one. It may be possible to weed these out at compile time in simple cases such as this, and replace the FOR j%= 1 with a simple assignment. If not, its definitely possible to deal with it at run-time

The equivalent code using the C# a simple ForNext class I devised looks like this:

ForNext outer = new ForNext("i%", 1, 5, 1);
outer.execute(delegate(int counter)
        {
            ForNext inner = new ForNext("j%", 1, 5, 1);
            inner.execute(delegate(int counter2)
            {
                Console.WriteLine("i% = {0} j% = {1}", counter, counter2);
                ForNext.Next("i%");
            });
         });

For the run-time solution, I create a ForNext loop object, initialized with the counter name which is just a label and the start, end and step values. I then pass the loop body to the ForNext object as a C# anonymous delegate. NEXT statements are implemented as calls to the static method ForNext.Next() with an optional counter name.

When run, the code produces,

i% = 1     j% = 1
i% = 2     j% = 1
i% = 3     j% = 1
i% = 4     j% = 1
i% = 5     j% = 1

Of course, some machinery in the ForNext class is required to make this work. In this case we use the .NET exception handling machinery to manage the stack of FOR..NEXT loops on our behalf, calling ForNext.Next() exits the current iteration of the top-most loop on the stack and if necessary begins the next iteration.

class ForNext
{
    public delegate void Body(int counter);
    private string id;
    private int end;
    private int step;
    private int counter;

    private class NextException : System.Exception
    {
        public string id;

        public NextException(string id)
        {
            this.id = id;
        }

        public NextException() :
            this(null)
        {
        }
    }

    public ForNext(string id, int start, int end, int step)
    {
        this.id = id;
        this.counter = start;
        this.end   = end;
        this.step  = step;
    }

    public void execute(Body body)
    {
        while (counter <= end)
        {
            try
            {
                body(counter);
            }
            catch (NextException e)
            {
                if (e.id != null && e.id != this.id)
                {
                    throw;
                }
                counter += step;
            }
        }
    }

    public static void Next()
    {
        if (counter < end)
        {
            throw new NextException();
        }
    }

    public static void Next(string id)
    {
        if (counter < end)
        {
            throw new NextException(id);
        }
    }
}

Note how in this example we pass the loop counter label as ForNext.Next("i%").

Of course, relying on the .NET exception handling mechanism for all such loops in BBC BASIC would be overkill. The majority of FOR..NEXT constructs are must simpler than this. The challenge moves on to detecting the simple cases using static analysis of the program to be compiled, and generating normal loop code in these simple cases. We would only fall back to the mechanism described here when required, if multiple unlabelled NEXT statements are present. It remains to be seen how feasible detection of these cases will be.

Of course, in functional languages, explicit iteration structures such as FOR..NEXT need not be present at all, since all iteration is achieved through recursion, usually with tail-calls. Another feature invented in the functional language LISP and now commonly available in more sophisticated languages is the concept of continuations, which are basically “GOTOs with parameters”. Both exception handling, as used in this post, and other complex control structures can be implemented using continuations, as can all the iteration structures of BBC BASIC, even with mismatched FOR..NEXT statements, even when the NEXT statement is not labelled with a counter variable.