Getting Hy on Python

Ready to give LWN a try?

With a subscription to LWN, you can stay current with what is happening in the Linux and free-software community and take advantage of subscriber-only site features. We are pleased to offer you a free trial subscription, no credit card required, so that you can see for yourself. Please, join us!

Lisp is one of the earliest programming languages, but unlike some of its peers, its popularity has endured. In fact, it has endured to the point that it continues to get reimplemented today, perhaps partly because it is fairly straightforward to parse and execute, but also, seemingly, because it is a fun language to work with. It certainly is clear that Paul Tagliamonte had a lot of fun implementing Lisp in Python, and he tried to impart some of that to the audience of his high-energy PyCon 2014 talk.

Tagliamonte works for the Sunlight Foundation on projects unrelated to Lisp, but the project he got hired to work on got its start at a previous PyCon, which makes the conference special to him. He is also a Debian developer and an Ubuntu member. Beyond that, he sometimes hacks on Fluxbox, he said.

Hy: Pythonic Lisp

His Lisp on Python project is called "Hy". It is actually a lot of different things, he said. To start with, Hy is an s-expression front-end to Python. It is, in some ways, similar to the way that clang is a C front-end to the LLVM compiler. Fundamentally, though, Hy is "all just Python".

He demonstrated Hy by showing a Python program that imported Hy, then imported a module called "maths". That module was contained in a file called maths.hy and was written in Lisp. The fact that Python can import Hy code is "magic and amazing", Tagliamonte said. He also demonstrated that when he used the Python debugger and stepped into Hy code, it showed Lisp code. Hy is "the most Pythonic Lisp that has ever existed", he said.

To do that magic, Hy uses a lot of Python internals "in a super Pythonic way", he said. Because of that, it makes for a good example of those internals—Hy provides a nice entry point for talking about them. Tagliamonte started working on Hy to continue some research on decentralized algorithms he had done in college. He needed a way to change the language to explore his problem space, which he did "through the healing power of macros". Lisp macros allow you to rewrite the language to add new features to it, he said.

One of the early tools he built with Hy was a domain-specific language called "snitch" that was used to monitor servers. He showed an example of snitch rules that described servers and what characteristics they should have (e.g. ping-able, http-able, have certain open ports, etc.). Using snitch, he could turn the declarative source code (in snitch) into "massive amounts of [Python] code" that uses the Python 3.4 asyncio module to do the specified monitoring.

So, if you have a statement in Lisp, such as:

    (print (+ 1 1))

    HyExpression([
         HySymbol('+'),
         HyInteger(1),
         HyInteger(1)
    ])

Once you have that arrangement of Python objects representing the Hy source code elements, you want to be able to run the code somehow. To do that in the Python virtual machine (VM), the Hy objects need to be translated into something that machine can handle. One possibility would be bytecode, which is what Python is turned into, but bytecode is "a pain". It is not stable between versions of Python and it changes constantly, he said.

Using AST

Another possibility is the Abstract Syntax Tree (ast) module, which is "totally cool", Tagliamonte said. If it is used in the right way, it can be quite powerful—it is, in some ways, like Lisp macros, he said. It allows Python to change Python code programmatically.

He went through some examples of representing various Python constructs (a list, a dict, and a function) as ASTs. Here is what a function looks like in AST:

    def square(x):
	return x * x
    # (defn square [x] (* x x))

    FunctionDef(
      name='square',
      args=arguments(
	args=[Name(id='x')],
	vararg=None, kwarg=None,
	defaults=[]),
      body=[Return(value=BinOp(
	      left=Name(id='x'),
	      op=Mult,
	      right=Name(id='x')))],
      decorator_list=[]
    )

Hy supports Python 2.6-3.4, so it has to deal with the various changes to AST in those versions. It can be somewhat painful to deal with that, but it is doable, he said. Most of the AST attribute names changed between 2.x and 3.x, but that can be handled by populating both the old and new versions of the attribute names on the AST objects, for example.

When experimenting with AST, you can expect to "segfault Python" frequently. It is easy to create nonsensical trees that crash Python when it tries to execute them. For example, AST elements have line numbers associated with them. If those numbers go backward at any point, CPython will allow it, but "PyPy explodes".

Since AST is set up for Python, it distinguishes between statements (for control flow and the like) and expressions (which return a value), but Lisp expects everything to be an expression. That made early work on Hy difficult, but he has figured out how to "fake" expressions everywhere. In answer to a question later, he said that constructs like if statements have a temporary variable to hold the "value" of each branch, which can be returned as needed for Lisp. Similar techniques are used for loops, function definitions, and so on.

The meta-information stored in AST nodes, such as line numbers and column offsets, is useful because it allows Hy to interface correctly with the Python debugger. In fact, AST allows you to do "crazy stuff" that few are actually doing, which is a shame, he said. It will allow approximating a domain-specific language in Python, for example.

Executing and importing Hy

So, Hy takes Lisp source code and turns it into an AST. The next step is to use the compile() function to turn that into bytecode. The return value from compile() is a code object, which is essentially the same information as what gets put into a .pyc file (that is put into the __pycache__ directory when running under Python 3.x). Passing that code object to exec() will actually run it.

The last piece of the puzzle from his demo was how to get Python to import .hy files correctly. It turns out that Python enhancement proposal (PEP) 302 provides import hooks that make it possible. The PEP was developed to support importing from zip files and the like, but it is generic enough to support Hy, he said.

There are two key pieces to a PEP 302 importer. The first is an Importer object that provides a find_module() method. That returns the second piece: a Loader object that actually loads the code and adds it to sys.modules. The Importer is registered with Python by appending it to sys.meta_path, which is consulted before the regular Importers are tried. That means the Hy Importer can find .hy files on sys.path, turn them into AST and then bytecode, then add them to the modules list. From that point, it looks like a normal module to Python.

As an amusing example of what can be done with importers, Tagliamonte mentioned a module that would run Python code regardless of any exceptions it raised. It essentially wraps the code in a construct like the following:

    try:
        possibly_misbehaving_code
    except:
        pass

While he does not consider Hy to be production-ready, "that ship has sailed", as there are people using it in production already. The good news is that it is much closer to being ready than it was last year, he said. Someone asked about generating Python directly, rather than AST, and he had looked into that along the way but found that it "got nasty quickly". Hy rewrites the AST it generates "on the fly", which is difficult to do with regular Python, he said.

He pointed interested people at hylang.org, while noting that he could not take credit for all of it as there are some 50 contributors to the language. Those who want to try Hy can do so on Google's App Engine (complete with a Symbolics Lisp Machine monitor). Beyond that, Tagliamonte's slides are available, as is a video of the talk.