In a word, don’t. Or if you do, configure your DHCP server to always lease the same address to your NexentaStor appliance.

A week ago I installed NexentaStor CE 3.1 on a new NAS server I have thrown together. Following the install last week, everything in the web-based GUI seemed to be working fine. Following the install and a few basic tests I shut down the server, knowing that I wouldn’t have time to get back to working with it for a while.

Fast forward one week to the next opportunity I get to configure my new server. Attempts to connect to the server over HTTP on port 2000 are met with a very long wait by the browser – several minutes are needed to display the main page, which is http://nas:2000/status/general/ and once I’m in there, some pages are dog slow whereas others are quite sprightly.

After much wailing and gnashing of teeth on my part I discovered that the time since I last used the server was long enough for the DHCP lease to expire. My router duly handed out a new lease, albeit with a different address. Unfortunately, Nexenta had inserted it’s own hostname and the original IP address into its /etc/hosts file. Something in the NMV web GUI was causing a look-up here, and of course by this time my router had re-allocated the original Nexenta IP address to my wife’s iPhone.

The solution was two-fold. First of all edit /etc/hosts to contain the current IP address of the server, and secondly configure my router’s DHCP server to always dish out the same IP address to the NAS server. After a re-start everything is now working fine and it seems now seems possible that I’ll be able to use NexentaStor.

My faith isn’t quite restored though – NexentaStor 3.1 is a brand new release – and judging from the forums quite a few people have experienced problems with very long browser delays waiting for the NMV to respond. NexentaStor should work with DHCP out-of-the-box and DHCP lease expiry should be tested if Nexenta are claiming that this is an enterprise grade solution.

I’ve just released asq 1.0 – a LINQ-inspired API for performing queries over Python objects. The project had been on the unpublished one-day-I’ll-finish-this back burner for a couple of years now, but recently I found myself wanting it in the course of developing another project. I decided to go public with the incomplete version I had back in January 2011, calling what I had asq 0.5 and hoping that the harsh glare of publicity would force me to make progress. I got 18 downloads without any promotional effort and that spurred me on towards a 0.9 release two months later which was feature complete and pretty stable.

Over the course of the next three months, asq 0.9 clocked up over 200 downloads.

I decided at this point that I’d hold back on any actual marketing as such until I could produce a solid 1.0 including comprehensive documentation. Well, asq 1.0 isn’t even 24 hours old yet and it’s already made 174 downloads.

I’m in the process of writing some code which ideally would work unchanged on CPython 2.x, CPython 3.x IronPython 2.x and Jython 2.x. Given the nature of the code I’m writing, that seems eminently possible with very few workarounds.

One of the changes between Python 2 and Python 3 was the removal of the cmp() function and its corresponding __cmp__() special method. Python 3 requires the use of the rich comparison operators __lt__(), __gt__(), __ge__(), __le__(), __eq__() and __ne__(). This change is well publicised and is fine so far as it goes; it’s usually no problem to modify the code accordingly. However, the same document goes on to say,

Use __lt__() for sorting, __eq__() with __hash__(), and other rich comparisons as needed.

This is clear guidance, if not a specification, that sort routines will use __lt__() for sorting. We don’t have to look much further for a statement closer to a specification though. In the Python Sorting HOWTO we can find the following clear declaration:

The sort routines are guaranteed to use __lt__() when making comparisons between two objects. So, it is easy to add a standard sort order to a class by defining an __lt__() method.

Unfortunately, this fact is not mentioned in the documentation for list.sort() or the sorted() built-in, so it’s difficult to find. Even more unfortunately for me, the documentation for the standard library heapq module does not specify at all what protocols it requires of the objects it will be ordering.

Now all of this became an issue for me today when I implemented an important optimisation which required that I implement an incremental (i.e. lazy) partial sort rather than a complete eager sort of a sequence. Partial sorts can be efficiently implemented using heaps – hence my encounter with the heapq module. My migration from sort() to heapq worked fine on CPython 2.x and CPython 3.x but when I ran my tests with IronPython I got failures for some sorting results.

Soon I determined that the problem was that, as per the documentation for sorting, I’d implemented only __lt__() and __eq__() rather than the full set of six rich comparison operators. This worked fine with my original code and with my heapq based sort in CPython because those sort implementation do indeed only call __lt__() and __eq__(). However, the IronPython implementation of heapq also calls __gt__() and so failed. Note that IronPython hasn’t done anything wrong here – it’s following the letter of the canonical CPython documentation, if not the spirit. I’m not sure it’s fair to consider this a bug in IronPython – rather it exposes a small but important weakness in the CPython documentation.

There are perhaps two morals to this story: The first is to always implement all or nothing when in comes to the rich comparison operators just like your mother taught you. The second is that we’d avoid these sort of difficulties if the Python documentation was a little more explicit about what protocols are expected of objects by its algorithms and containers.

The existence of multiple Python implementations has been extremely effective at defining what it means to be Python as opposed to CPython. Many inconsistencies such as the one noted above have been flushed out over the years. For this reason alone I’m hesitant to suggest that the folks porting the standard library from CPython to IronPython or Jython should actually read the CPython implementations to ensure compatibility. I think it’s much preferable that such things are reimplemented from the documentation and test suite with consequential improvements for both.

Before you get too excited this is an implementation of a type inference algorithm that happens to be written in Python; it has nothing to do with the Python language itself!

I’ve been working on OWL BASIC, a compiler for BBC BASIC for the .NET CLR. The compiler itself is written in IronPython. One of the challenges of compiling BBC BASIC is to infer the types of functions from the type of their return types. The return value of a BBC BASIC function can be an arbitrary expression, including calls to other functions or recursive calls to itself. I’ve implemented a simple type inference scheme which works well in the common cases, but for a fully capable solution my type checker and type inferencer need to be beefed up somewhat. To that end, I’ve been investigating standard type systems such as Hindley-Milner and inferencing algorithms such as Damas-Milner, sometimes known as Algorithm W. These algorithms or derivatives thereof are using in the ML family of languages (Standard ML, Ocaml, F#) and Haskell.

I managed to locate a Modula-2 implementation, a Perl implementation and a Scala implementation of the algorithm, each descended from the previous. With a view to improving my understanding of the algorithm I set about reimplementing in Python, largely guided by the Scala implementation, making mine the fourth in this sequence. I also located a Haskell implementation which seems to have independent ancestry. I’ve gone back to the companion paper (Cardelli 1987) to the original Modula-2 implementation and carried forward some of the code comments which had been omitted from its descendants to assist others who wish to understand the algorithm.

The program implements abstract syntax tree nodes for a small functional language, the type inferencing algorithm and finally exercises the algorithm by inferring the types of some canned expressions in the context of some predefined types. When executed it produces the following output:

> python hindley_milner.py
(letrec factorial = (fn n => (((cond (zero n)) 1) ((times n) (factorial (pred n))))) in (factorial 5)) :  int
(fn x => ((pair (x 3)) (x true))) :  Type mismatch: bool != int
((pair (f 4)) (f true)) :  Undefined symbol f
(let f = (fn x => x) in ((pair (f 4)) (f true))) :  (int * bool)
(fn f => (f f)) :  recursive unification
(let g = (fn f => 5) in (g g)) :  int
(fn g => (let f = (fn x => g) in ((pair (f 3)) (f true)))) :  (a -> (a * a))
(fn f => (fn g => (fn arg => (g (f arg))))) :  ((b -> c) -> ((c -> d) -> (b -> d)))

The Python code is shown below in its entirety or it can be downloaded as hindley_milner.py. It will run without modification on Python 2.6 or Python 3.

#!/usr/bin/env python
'''
.. module:: hindley_milner
   :synopsis: An implementation of the Hindley Milner type checking algorithm
              based on the Scala code by Andrew Forrest, the Perl code by
              Nikita Borisov and the paper "Basic Polymorphic Typechecking"
              by Cardelli.
.. moduleauthor:: Robert Smallshire
'''

from __future__ import print_function

#=======================================================#
# Class definitions for the abstract syntax tree nodes
# which comprise the little language for which types
# will be inferred

class Lambda(object):
    """Lambda abstraction"""

    def __init__(self, v, body):
        self.v = v
        self.body = body

    def __str__(self):
        return "(fn {v} => {body})".format(v=self.v, body=self.body)

class Ident(object):
    """Identfier"""

    def __init__(self, name):
        self.name = name

    def __str__(self):
        return self.name

class Apply(object):
    """Function application"""

    def __init__(self, fn, arg):
        self.fn = fn
        self.arg = arg

    def __str__(self):
        return "({fn} {arg})".format(fn=self.fn, arg=self.arg)

class Let(object):
    """Let binding"""

    def __init__(self, v, defn, body):
        self.v = v
        self.defn = defn
        self.body = body

    def __str__(self):
        return "(let {v} = {defn} in {body})".format(v=self.v, defn=self.defn, body=self.body)

class Letrec(object):
    """Letrec binding"""

    def __init__(self, v, defn, body):
        self.v = v
        self.defn = defn
        self.body = body

    def __str__(self):
        return "(letrec {v} = {defn} in {body})".format(v=self.v, defn=self.defn, body=self.body)

#=======================================================#
# Exception types

class TypeError(Exception):
    """Raised if the type inference algorithm cannot infer types successfully"""

    def __init__(self, message):
        self.__message = message

    message = property(lambda self: self.__message)

    def __str__(self):
        return str(self.message)

class ParseError(Exception):
    """Raised if the type environment supplied for is incomplete"""
    def __init__(self, message):
        self.__message = message

    message = property(lambda self: self.__message)

    def __str__(self):
        return str(self.message)

#=======================================================#
# Types and type constructors

class TypeVariable(object):
    """A type variable standing for an arbitrary type.

    All type variables have a unique id, but names are only assigned lazily,
    when required.
    """

    next_variable_id = 0

    def __init__(self):
        self.id = TypeVariable.next_variable_id
        TypeVariable.next_variable_id += 1
        self.instance = None
        self.__name = None

    next_variable_name = 'a'

    def _getName(self):
        """Names are allocated to TypeVariables lazily, so that only TypeVariables
        present
        """
        if self.__name is None:
            self.__name = TypeVariable.next_variable_name
            TypeVariable.next_variable_name = chr(ord(TypeVariable.next_variable_name) + 1)
        return self.__name

    name = property(_getName)

    def __str__(self):
        if self.instance is not None:
            return str(self.instance)
        else:
            return self.name

    def __repr__(self):
        return "TypeVariable(id = {0})".format(self.id)

class TypeOperator(object):
    """An n-ary type constructor which builds a new type from old"""

    def __init__(self, name, types):
        self.name = name
        self.types = types

    def __str__(self):
        num_types = len(self.types)
        if num_types == 0:
            return self.name
        elif num_types == 2:
            return "({0} {1} {2})".format(str(self.types[0]), self.name, str(self.types[1]))
        else:
            return "{0} {1}" % (self.name, ' '.join(self.types))

class Function(TypeOperator):
    """A binary type constructor which builds function types"""

    def __init__(self, from_type, to_type):
        super(Function, self).__init__("->", [from_type, to_type])

# Basic types are constructed with a nullary type constructor
Integer = TypeOperator("int", [])  # Basic integer
Bool    = TypeOperator("bool", []) # Basic bool

#=======================================================#
# Type inference machinery

def analyse(node, env, non_generic=None):
    """Computes the type of the expression given by node.

    The type of the node is computed in the context of the context of the
    supplied type environment env. Data types can be introduced into the
    language simply by having a predefined set of identifiers in the initial
    environment. environment; this way there is no need to change the syntax or, more
    importantly, the type-checking program when extending the language.

    Args:
        node: The root of the abstract syntax tree.
        env: The type environment is a mapping of expression identifier names
            to type assignments.
            to type assignments.
        non_generic: A set of non-generic variables, or None

    Returns:
        The computed type of the expression.

    Raises:
        TypeError: The type of the expression could not be inferred, for example
            if it is not possible to unify two types such as Integer and Bool
        ParseError: The abstract syntax tree rooted at node could not be parsed
    """

    if non_generic is None:
        non_generic = set()

    if isinstance(node, Ident):
        return getType(node.name, env, non_generic)
    elif isinstance(node, Apply):
        fun_type = analyse(node.fn, env, non_generic)
        arg_type = analyse(node.arg, env, non_generic)
        result_type = TypeVariable()
        unify(Function(arg_type, result_type), fun_type)
        return result_type
    elif isinstance(node, Lambda):
        arg_type = TypeVariable()
        new_env = env.copy()
        new_env[node.v] = arg_type
        new_non_generic = non_generic.copy()
        new_non_generic.add(arg_type)
        result_type = analyse(node.body, new_env, new_non_generic)
        return Function(arg_type, result_type)
    elif isinstance(node, Let):
        defn_type = analyse(node.defn, env, non_generic)
        new_env = env.copy()
        new_env[node.v] = defn_type
        return analyse(node.body, new_env, non_generic)
    elif isinstance(node, Letrec):
        new_type = TypeVariable()
        new_env = env.copy()
        new_env[node.v] = new_type
        new_non_generic = non_generic.copy()
        new_non_generic.add(new_type)
        defn_type = analyse(node.defn, new_env, new_non_generic)
        unify(new_type, defn_type)
        return analyse(node.body, new_env, non_generic)
    assert 0, "Unhandled syntax node {0}".format(type(t))

def getType(name, env, non_generic):
    """Get the type of identifier name from the type environment env.

    Args:
        name: The identifier name
        env: The type environment mapping from identifier names to types
        non_generic: A set of non-generic TypeVariables

    Raises:
        ParseError: Raised if name is an undefined symbol in the type
            environment.
    """
    if name in env:
        return fresh(env[name], non_generic)
    elif isIntegerLiteral(name):
        return Integer
    else:
        raise ParseError("Undefined symbol {0}".format(name))

def fresh(t, non_generic):
    """Makes a copy of a type expression.

    The type t is copied. The the generic variables are duplicated and the
    non_generic variables are shared.

    Args:
        t: A type to be copied.
        non_generic: A set of non-generic TypeVariables
    """
    mappings = {} # A mapping of TypeVariables to TypeVariables

    def freshrec(tp):
        p = prune(tp)
        if isinstance(p, TypeVariable):
            if isGeneric(p, non_generic):
                if p not in mappings:
                    mappings[p] = TypeVariable()
                return mappings[p]
            else:
                return p
        elif isinstance(p, TypeOperator):
            return TypeOperator(p.name, [freshrec(x) for x in p.types])

    return freshrec(t)

def unify(t1, t2):
    """Unify the two types t1 and t2.

    Makes the types t1 and t2 the same.

    Args:
        t1: The first type to be made equivalent
        t2: The second type to be be equivalent

    Returns:
        None

    Raises:
        TypeError: Raised if the types cannot be unified.
    """

    a = prune(t1)
    b = prune(t2)
    if isinstance(a, TypeVariable):
        if a != b:
            if occursInType(a, b):
                raise TypeError("recursive unification")
            a.instance = b
    elif isinstance(a, TypeOperator) and isinstance(b, TypeVariable):
        unify(b, a)
    elif isinstance(a, TypeOperator) and isinstance(b, TypeOperator):
        if (a.name != b.name or len(a.types) != len(b.types)):
            raise TypeError("Type mismatch: {0} != {1}".format(str(a), str(b)))
        for p, q in zip(a.types, b.types):
            unify(p, q)
    else:
        assert 0, "Not unified"

def prune(t):
    """Returns the currently defining instance of t.

    As a side effect, collapses the list of type instances. The function Prune
    is used whenever a type expression has to be inspected: it will always
    return a type expression which is either an uninstantiated type variable or
    a type operator; i.e. it will skip instantiated variables, and will
    actually prune them from expressions to remove long chains of instantiated
    variables.

    Args:
        t: The type to be pruned

    Returns:
        An uninstantiated TypeVariable or a TypeOperator
    """
    if isinstance(t, TypeVariable):
        if t.instance is not None:
            t.instance = prune(t.instance)
            return t.instance
    return t

def isGeneric(v, non_generic):
    """Checks whether a given variable occurs in a list of non-generic variables

    Note that a variables in such a list may be instantiated to a type term,
    in which case the variables contained in the type term are considered
    non-generic.

    Note: Must be called with v pre-pruned

    Args:
        v: The TypeVariable to be tested for genericity
        non_generic: A set of non-generic TypeVariables

    Returns:
        True if v is a generic variable, otherwise False
    """
    return not occursIn(v, non_generic)

def occursInType(v, type2):
    """Checks whether a type variable occurs in a type expression.

    Note: Must be called with v pre-pruned

    Args:
        v:  The TypeVariable to be tested for
        type2: The type in which to search

    Returns:
        True if v occurs in type2, otherwise False
    """
    pruned_type2 = prune(type2)
    if pruned_type2 == v:
        return True
    elif isinstance(pruned_type2, TypeOperator):
        return occursIn(v, pruned_type2.types)
    return False

def occursIn(t, types):
    """Checks whether a types variable occurs in any other types.

    Args:
        v:  The TypeVariable to be tested for
        types: The sequence of types in which to search

    Returns:
        True if t occurs in any of types, otherwise False
    """
    return any(occursInType(t, t2) for t2 in types)

def isIntegerLiteral(name):
    """Checks whether name is an integer literal string.

    Args:
        name: The identifier to check

    Returns:
        True if name is an integer literal, otherwise False
    """
    result = True
    try:
        int(name)
    except ValueError:
        result = False
    return result

#==================================================================#
# Example code to exercise the above

def tryExp(env, node):
    """Try to evaluate a type printing the result or reporting errors.

    Args:
        env: The type environment in which to evaluate the expression.
        node: The root node of the abstract syntax tree of the expression.

    Returns:
        None
    """
    print(str(node) + " : ", end=' ')
    try:
        t = analyse(node, env)
        print(str(t))
    except (ParseError, TypeError) as e:
        print(e)

def main():
    """The main example program.

    Sets up some predefined types using the type constructors TypeVariable,
    TypeOperator and Function.  Creates a list of example expressions to be
    evaluated. Evaluates the expressions, printing the type or errors arising
    from each.

    Returns:
        None
    """

    var1 = TypeVariable()
    var2 = TypeVariable()
    pair_type = TypeOperator("*", (var1, var2))

    var3 = TypeVariable()

    my_env = { "pair" : Function(var1, Function(var2, pair_type)),
               "true" : Bool,
               "cond" : Function(Bool, Function(var3, Function(var3, var3))),
               "zero" : Function(Integer, Bool),
               "pred" : Function(Integer, Integer),
               "times": Function(Integer, Function(Integer, Integer)) }

    pair = Apply(Apply(Ident("pair"), Apply(Ident("f"), Ident("4"))), Apply(Ident("f"), Ident("true")))

    examples = [
            # factorial
            Letrec("factorial", # letrec factorial =
                Lambda("n",    # fn n =>
                    Apply(
                        Apply(   # cond (zero n) 1
                            Apply(Ident("cond"),     # cond (zero n)
                                Apply(Ident("zero"), Ident("n"))),
                            Ident("1")),
                        Apply(    # times n
                            Apply(Ident("times"), Ident("n")),
                            Apply(Ident("factorial"),
                                Apply(Ident("pred"), Ident("n")))
                        )
                    )
                ),      # in
                Apply(Ident("factorial"), Ident("5"))
            ),

            # Should fail:
            # fn x => (pair(x(3) (x(true)))
            Lambda("x",
                Apply(
                    Apply(Ident("pair"),
                        Apply(Ident("x"), Ident("3"))),
                    Apply(Ident("x"), Ident("true")))),

            # pair(f(3), f(true))
            Apply(
                Apply(Ident("pair"), Apply(Ident("f"), Ident("4"))),
                Apply(Ident("f"), Ident("true"))),

            # let f = (fn x => x) in ((pair (f 4)) (f true))
            Let("f", Lambda("x", Ident("x")), pair),

            # fn f => f f (fail)
            Lambda("f", Apply(Ident("f"), Ident("f"))),

            # let g = fn f => 5 in g g
            Let("g",
                Lambda("f", Ident("5")),
                Apply(Ident("g"), Ident("g"))),

            # example that demonstrates generic and non-generic variables:
            # fn g => let f = fn x => g in pair (f 3, f true)
            Lambda("g",
                   Let("f",
                       Lambda("x", Ident("g")),
                       Apply(
                            Apply(Ident("pair"),
                                  Apply(Ident("f"), Ident("3"))
                            ),
                            Apply(Ident("f"), Ident("true"))))),

            # Function composition
            # fn f (fn g (fn arg (f g arg)))
            Lambda("f", Lambda("g", Lambda("arg", Apply(Ident("g"), Apply(Ident("f"), Ident("arg"))))))
    ]

    for example in examples:
        tryExp(my_env, example)

if __name__ == '__main__':
    main()

To compile the code comprising an OWL BASIC procedure, function or main program into CIL, we must linearise the Control Flow Graph (CFG) representing the program statements. The CFG undergoes many transformations during compilation, for example to eliminate unreachable code or convert GOSUB routines into named procedures. Generation of CIL using Reflection.Emit requires that we can define branch targets in advance of generating branch instructions or marking the target instruction and of course we want to do this in a manner which minimises the number of branches required to represent the code. The structure of the graph may be quite complex, especially for traditional BASIC spaghetti code which uses GOTO excessively rather than the more structured alternatives such as procedures and functions or the control structures introduced in BBC BASIC V .

Consider the following procedure from Sphinx Adventure. It contains three loops, one on line 271 formed with a GOTO back to the start of the line, and two REPEAT .. UNTIL loops.

266 DEF PROCL(L)
267 LOCAL I,J:CO=0:CN=0
268 IF L=1 THEN278
269 PRINT’: RESTORE L: IF O?31<>0 THEN O?31=0:DW=1
270 READ R$,R$:R$="You are "+R$
271 IF LEN(R$)+ POS>CO-CN+39 THEN R$= FNS(R$,39+CO-CN):CO=CO+39: GOTO271
272 PRINT R$: IFL=136 OR L=15 THEN O?56=L
273 IF L=16 AND FL=1 THEN PRINT"The walls are very hot!" ELSE IF L=16 THEN PRINT "The walls are steaming!"
274 IF L<>3 AND L<>142 AND L<>143 THEN PROCEX(L): IF ABS(L-19)=1 AND CH=1 THEN PRINT ELSE IF ABS(L-42)=1 AND VO=1 THEN PRINT
275 IF CH=1 AND ABS(L-19)=1 THEN PROC R(22): PRINT "chasm.":O?53=L
276 IF VO=1 AND ABS(L-42)=1 THEN PROCR(22): PRINT"glacier.":O?53=L
277 IF L=26 OR L=27 THEN O?53=L
278 J=0:I=0:CO=0
279 REPEAT:J=J+1: IF O?J=L THEN CO=CO+1
280 UNTIL J=52: IF CO=0 AND L=1 THEN PROCR(L): GOTO 284 ELSE IF CO=0 AND L<>1 THEN 284 ELSE PRINT:MAX=CO
281 IF L=1 THEN PROCR(3) ELSE PROCR(4)
282 CN=0:CO=MAX: REPEAT I=I+1: IF O?I=L THEN PROCOT(I,CO):CO=CO-1
283 UNTIL I=52
284 IF D<>0 THEN O?31=L
285 IF CF=1 AND L=94 THEN PRINT’"The casket is open."
286 IF L=24 AND SA=1 THEN PRINT’"The safe door is open."
287 PRINT: ENDPROC

The CFG for this code is shown below. Each program statement is shown as a purple box, with control flow to the following statement(s). Conditionals are shown in diamond boxes. The numbers in each purple box are source line numbers, where known.

Careful comparison of the source above and the diagram below will reveal some of the transformations that have been applied to the program; for example, READ R$,R$ on line 270 has been transformed into two consecutive assignment statements which actually take the form R$ = READ() where READ() is a function not available in the source language.

The statement level CFG has been analysed to identify basic-blocks, shown as yellow group nodes, thereby defining a higher level basic-block level CFG. Each basic block has only one entry point statement; none of the statement within the basic block are destinations of other jump instructions. Furthermore, each block has only one exit point.

More text follows this long diagram…

Generating the CIL code for a single basic block is straightforward enough – we can simply iterate through the statements comprising the basic block in order and generate the code for each in turn. However, there are many possible orders in which the code for the basic block themselves could be representing in the CIL, since we can branch from the end of any block to the next block, although of course we must start at the entry block for the procedure. Although any order starting with the entry block can be made to work, where possible we would like program control to flow smoothly from the end of a block to one of its successors without requiring a branch.

At first sight, some sort of topological ordering would seem to be appropriate, but a topological ordering is only well defined for a directed acyclic graph (DAG), and a DAG this program is not. The key to this conundrum is to reduce the directed graph to a DAG by identifying strongly connected components. By contracting each SCC to a single node we obtain what is called the condensation of the CFG which will be a DAG. To the resulting DAG we can apply a topological ordering. The ordering of vertices with each SCC is chosen by starting at the vertex with the greatest in-degree.

In order to identify and contract the SCCs we use an implementation of Tarjan’s algorithm during depth first traversal of the CFG. The reverse post ordering of the primary depth first traversal is used to generate the topological ordering of the condensed CFG.

The resulting ordering of basic blocks is shown in the diagram by the numeric labels to the top-left of each. This will be the order in which the CIL code for them is generated, and it can be seen that in about half of the cases, fall through from one block to the next (consecutive block numbers) without explicit branching can be exploited. Future optimisations will focus on further simplifying the generated code by removing vertices, such as block 31, which contain only jumps.

After a long haul, and diversions into other more important projects — including starting a family — OWL BASIC today produced its first executable. Its not much. In fact its hardly anything. Just 2048 bytes of Windows PE executable containing the global variable declarations from Acornsoft’s 1982 Sphinx Adventure. Each file of BASIC source code will be converted to a single .NET static class, with the global variables as private static fields.

The first executable produced from OWL BASIC.

Above you can see the executable loaded up into .NET Reflector, which can be used to introspect the executable, and in this case attempt to disassemble it into C#. Now we see what makes .NET such a great platform for compiler construction; below is the IronPython source code for the embryonic assembly generation function. It clocks in at fewer than ten lines of code to create an assembly, create a module, create a class, add one private static field to it for each global variable, and save the result as an .exe.

def generateAssembly(name, global_symbols):
    domain = Thread.GetDomain()
    assembly_name = AssemblyName(name)
    assembly_builder = domain.DefineDynamicAssembly(assembly_name, AssemblyBuilderAccess.RunAndSave)
    module_builder = assembly_builder.DefineDynamicModule(name + ".exe")
    type_builder = module_builder.DefineType(name, TypeAttributes.Class | TypeAttributes.Public, object().GetType())

    # Add global variables to the class
    for symbol in global_symbols.symbols.values():
        field_builder = type_builder.DefineField(symbol.name, ctsType(symbol),
                                                 FieldAttributes.Private | FieldAttributes.Static)

    result = type_builder.CreateType()
    assembly_builder.Save(name + ".exe")

where global_symbols is the global symbol table constructed during traversal of the Abstract Syntax Tree and the Control Flow Graph and the ctsType function maps OWL BASIC types to their equivalent Common Type System types for .NET. Everything else is provided by Reflection.Emit and other parts of .NET.

Its interesting that no validation was applied to the variable names supplied to Reflection.Emit. As you can see, the variable names still include the sigil suffixes for variable typing (e.g. $ for string) and Reflector happily dissassembles these into invalid C# identifiers. For the final version these names will need to be mangled (Hungarian notation?), or merely de-sigiled if no conflicts result, for compatibility with other .NET languages and tools.

After 15 years programming in C++, I was surprised to discover today that throw in C++ is an expression rather than a statement. As a result, throw may be used as part of larger expressions.

int x = 5;
int y = x > 4 ? x : throw std::out_of_range;

The only use I’ve found for this — and in fact the speculative attempt by which I discovered it — is range checking within constructor initializer lists.

RangeChecked::RangeChecked(int x) :
    int_member(x > 0 ? x : throw std::out_of_range)
{
}

It’s academic what the type of a throw expression is, since it will never be returned, but for type-checking purposes the compiler seems to be happy to use it in place of any type.

Eclipse doesn’t come with an installer for Windows. As a result of this, installing Eclipse into Program Files is very awkward without getting into a tussle with User Account Control, and virtualisation of the Program Files directory.

After far too much effort I have finally found a sequence that Works For Me™ on Windows Vista Ultimate x64.

The procedure

1. Download eclipse-SDK-3.5-win32.zip
2. Extract the zip file into a temporary directory. I used C:\Users\<usename>\Documents\tmp\eclipse
3. Within the temporary directory (important!) edit eclipse.ini, adding the line

   -configuration @user.home\.eclipse_35_config
   

   My eclipse.ini then looked like this:

   -startup
   plugins/org.eclipse.equinox.launcher_1.0.200.v20090520.jar
   --launcher.library
   plugins/org.eclipse.equinox.launcher.win32.win32.x86_1.0.200.v20090519
   -showsplash
   org.eclipse.platform
   -configuration @user.home\.eclipse_35_config
   --launcher.XXMaxPermSize
   256m
   -vmargs
   -Xms40m
   -Xmx256m
   

4. Within the temporary directory create a UTF-8 text file called eclipse.exe.manifest file side-by-side with the eclipse.exe file with the following content:

   <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
   <assembly xmlns="urn:schemas-microsoft-com:asm.v1" manifestVersion="1.0">
     <assemblyIdentity version="1.0.0.0"
        processorArchitecture="X86"
        name="eclipse"
        type="win32"/>
     <description>Eclipse</description>
     <!-- Identify the application security requirements. -->
     <trustInfo xmlns="urn:schemas-microsoft-com:asm.v2">
       <security>
         <requestedPrivileges>
           <requestedExecutionLevel level="asInvoker" uiAccess="false"/>
           </requestedPrivileges>
          </security>
     </trustInfo>
   </assembly>
   

5. Using drag and drop copy in Explorer, copy the entire eclipse directory from your temporary location to C:\Program Files (x86)\eclipse. We copy, rather than move simply so you can easily return to this step.
6. Right-click on eclipse.exe in Explorer and choose Properties
7. Click on the Security tab
8. Click Advanced in the lower right
9. In the Advanced Security Settings window that pops up, click on the
   Owner tab
10. Click Edit
11. Click Continue if you get a UAC dialog
12. Click Other users or groups
13. Click Advanced in the lower left corner
14. Click Find Now
15. Scroll through the results and click on your current user account
16. Click OK to all of the remaining windows
17. Right-click the file and select Properties (again)
18. Click on the Security tab
19. Click Edit…
20. Click on the Users group
21. Adjust the permissions for your user using the check boxes at the bottom of the dialog. e.g. enable Full Control
22. Click OK to all of the remaining windows
23. Right-click the file and select Properties (again!)
24. Click Unblock and close the dialog
25. Double click eclipse.exe to run it. You should not get an Open File – Security Warning dialog.
26. Remove the temporary eclipse directory you created when you unzipped the archive.

Credit where it is due

The above post is largely pulled together from various sources I located mixed with a good degree of trial and error. I found the following particularly useful:

• Open File – Security Warning on VistaHeads
• How do you run Eclipse 3.3 on Vista?

Europython 2009 was my first Python conference, this being the first year that I’ve been able to use Python professionally for application development. We’d made a decision that if we were to be using Python commercially, we should be active within the Python community, so two of our team travelled from Norway to Birmingham, UK. Prior to Europython 2009 my use of Python had been confined hobby projects, occasional scripts and substantial commercial use with a several industrial-scale SCons builds.

I’ll start out by stating that I came away from the conference a little dissappointed, although I must emphasise in no way cheated, by the experience. I’ve attended a great many professionally and community organized conferences over the years, and I felt in many regards Europython was somewhat further down the quality scale than it should be, given the rising importance of Python. I’m concious that this posting may come across as a litany of complaint, but I feel that some changes could significantly improve the experience for everybody involved. Organisers of such events are forever soliciting feedback, so here goes!

Europython 2009 was a multi-tracked conference with up to six parallel sessions, so its not possible for me to report on even a representative sample of the sessions. Nonetheless, the overview of the not-so-random sample presented below may give you a flavour of the experience. I’m not involved in web development currently so I tended to shy away from the plethora of sessions covering web frameworks. What does interest me are the alternative Python implementatons, IronPython and Jython and uses of Python for scientific computation. Unfortunately, I couldn’t make it to the SCons talk, another interest of mine.

Size, venue and organization

Europython had around 500 delegates in attendance, which is I believe a record. The chosen venue was something of a disappointment. Although the main lecture theatre (The Adrian Bolt Hall) was effective, the other rooms in use were without exception compromised in significant ways. One was quite small with insufficient capacity and little or no air conditioning – although it should be noted that Birmingham was experiencing something of a heatwave at the time. Another featured a concrete pillar between the speaker and his audience – although much better air conditioning in here – and a third had the audience seated on a raked seating arrangement that creaked and groaned with the slightest movement. I wonder how the musicians who usually inhabit the placed manage?

The lunch arrangements were wholly inadequate. Lunch involved collecting a polystyrene plate, bowl and a drink from a refectory upstairs, loading these with food canteen style and then precariously balancing these items (no trays here!) and trooping down a fire escape stairwell back into the corridors between the lecture halls where there was insufficient seating. At one point our Vice President of Product Development, had expressed an interest in attending Europython, having shown a genuine curiosity in the langauge – we’ve been working hard on internal advocacy. At the point at which I found myself having to stand up to eat my lunch standing up in a corridor, having requisitioned a corner of the coffee serving table with my colleague, I was glad he wasn’t here to witness this somewhat unprofessional scenario (Yes, I know its a community event, and not ‘professionally’ organised, but a touch of professionalism can go a long way in the Python advocacy stakes with the right people). I appreciate there were a mitigating circumstances (a broken lift) but the nature of the facilities alone will make me think twice about who I encourage to attend this event in future.

Preparation and planning

Shortly before Europython I’d returned from a conference called EAGE 2009 which had 7000 student and professional delegates in a major European convention centre (RAI in Amsterdam). I’ll confess that this may have done much to anchor my expection levels vis-à-vis conferences immediately prior to Europython. Venue differences and registration cost aside (Europython was 5× cheaper than EAGE) one clear difference was the poor level of preparedness of speakers at Europython compared to EAGE, even though at both conferences speakers were drawn from professional, student and amateur circles. Time and again at Europython speakers hurriedly arrived in the room with little or no time for setup of equipment such as projectors. At this point I must give a note — no, make that a plea — to conference organizers: Please place a printed notice in 144 point text stating the preferred projector resolution on the table or lectern at the front so we can be spared a linear search through the possible resolutions at the beginning of each talk.

Now, I don’t know about you, but on occasions that I’ve been presenting and a private meeting or a conference, especially if it involves a live demo requiring network connectivity, I’m in the room setting up, connecting, authenticating, testing, and re-testing for up to 30 minutes before hand ensuring that everything will run smoothly. Although this doesn’t ensure a professional delivery, it can usually eliminate the tiresome, and stressful for the speaker, configuration during the talk. Its also a common courtesy to your audience to be ready before they are and not keep them waiting. Doing this means you means you’ll get longer to talk, and time for questions. I’m sure most people would agree that time for questions beats staring Display Settings for the umpteenth time.

Some talks dependend on a ‘live’ Internet connection – always a bad idea but in this case it was fatal. Wireless connectivity was intermittent throughout the conference.

Talk quality

I found the quality of the material presented at Europython to be highly variable; it covered the gamut from borderline incompetent to very good indeed. Many of the talks simply did not have sufficient material behind them to justify the length of the time slot they were given. I wonder if fewer parallel tracks with shorter talks would give a better result; audiences would be larger, the pace quicker and delegates would be a able to see a larger proportion of talks, which would help in technology transfer between the different subgroups within the wider Python community.

Keynotes

The keynotes were engaging as one expects. Cory Doctorow gave the predictable but entertaining DRM is evil and copyright unenforceable polemic, in the moral vacuum we have come to expect. Bruce Eckel delivered an entertaining historical overview of language features across C++, Java, C# and Python. Susan Blackmore and Simon Greenish made an impassioned plea for funds for Saving Bletchley Park, although the talk lacked any technical content about the operations or machines there, which given the technical audience was a little disappointing. Sir Tony Hoare rounded off the keynotes with a comparative study of the disciplines of software engineering and computing science, finishing with the not unreasonable prediction that one day “software will be the most reliable component of every product which contains it”, although he didn’t venture to suggest a time-scale on which this might come to pass.

Conclusion

I’m glad I attended. The conference was good value and on the whole informative. The organisers should seriously consider a better venue for next year and raising the standard of talks would make a return visit a more compelling prospect. The easiest way to do this may be to simply reduce the number of talks, or shift more of the content into a more appropriate format such as lightning talks.

Jython and IronPython run on platforms where strings are unicode capable by default. Both implementations have chosen to make str essentially an alias for unicode in Python source code. The bytes type, introduced in PEP358 as part of transition to fully unicode Python 3.0, is unambiguously a sequence of single byte values. We can see in the table below that Jython and IronPython are caught between what is on the one hand most practical for interopability with existing code and their host platforms, and on the other hand the Right Thing as delivered by Python 3.0.

It seems clear that if you need to write code that is portable between the different Python implementations you should steer clear str and use bytes and unicode to unambigiously express your intent.

Of course, this is impossible since the Python Standard Library is littered with uses of str. For example, in IronPython pickle.dumps() returns str just like Python 2.6 but the str is actually has multibyte storage. IronPython hides this well, but the abstraction can leak, resulting in much confusion. Again Python 3.0 does what is right, and pickle.dumps() returns a bytes instance.

These difficulties are most likely to occur when interfacing with native Java or .NET APIs that expect byte arrays, for example when pickling to database blobs.

In Jython an str instance can be converted to a Java byte array as follows.

>>> import jarray
>>> a = jarray.array("This is  string", 'b')
>>> a
array('b', [84, 104, 105, 115, 32, 105, 115, 32, 32, 115, 116, 114, 105, 110, 103])

The equivalent in IronPython, as provided by Michael Foord, being,

>>> from System import Array, Byte
>>> a = Array[Byte](tuple(Byte(ord(c)) for c in "This is a string"))
>>> a
Array[Byte]((<System.Byte object at 0x000000000000002B [84]>, <System.Byte object at 0x000000000000002C [104]>, <System.Byte object at 0x000000000000002D [105]>, <System.Byte object at 0x000000000000002E [115]>, <System.Byte object at 0x000000000000002F [32]>, <System.Byte object at 0x0000000000000030 [105]>, <System.Byte object at 0x0000000000000031 [115]>, <System.Byte object at 0x0000000000000032 [32]>, <System.Byte object at 0x0000000000000033 [97]>, <System.Byte object at 0x0000000000000034 [32]>, <System.Byte object at 0x0000000000000035 [115]>, <System.Byte object at 0x0000000000000036 [116]>, <System.Byte object at 0x0000000000000037 [114]>, <System.Byte object at 0x0000000000000038 [105]>, <System.Byte object at 0x0000000000000039 [110]>, <System.Byte object at 0x000000000000003A [103]>))

Going back we can use identical code in IronPython and Jython.

>>> s = ''.join(chr(c) for c in a)
>>> s
'This is a string'