Morgan And Grand Iron Clojure.
A functional Clojure compiler library and the start of a standalone Clojure implementation for the CLR.
The compiler is feature-complete against Clojure 1.10 and used in production (Flybot ships it on Unity, including iOS via IL2CPP). It is still maturing, not yet as battle-tested as JVM Clojure.
MAGIC is a compiler for Clojure written in Clojure targeting the Common Language Runtime. Its goals are to:
- Take full advantage of the CLR's features to produce better byte code
- Leverage Clojure's functional programming and data structures to be more tunable, composeable, and maintainable
See the top-level README and bb tasks at the repo root. bb test runs the test suite; bb dev-compiler is the iteration loop for changes under src/.
MAGIC's own analyzer (magic.analyzer, in src/magic/analyzer/) builds on clojure.tools.analyzer and adds CLR-specific passes (type inference, intrinsic rewrites, value-type handling, ...). It turns the resulting AST nodes into MAGE byte code to be compiled into executable MSIL. By building on the generic tools.analyzer and using the ClojureCLR reader (shipped in clojure-runtime), we avoid rewriting what is already high performance, high quality code. By using MAGE, we are granted the full power of Clojure to reason about generating byte code without withholding any functionality of the CLR.
In MAGIC parlance, a compiler is a function that transforms a single AST node into MAGE byte code. Previous versions of MAGIC called these symbolizers but that term is no longer used. For example, a static property like DateTime/Now would be analyzed by magic.analyzer into a hash map with a :property containing the correct PropertyInfo object. The compiler looks like this:
(defn static-property-compiler
"Symbolic bytecode for static properties"
[{:keys [property] :as ast} compilers]
(il/call (.GetGetMethod property)))It extracts the PropertyInfo from the :property key in the AST, computes the getter method, and returns the MAGE byte code for a method invocation of that method.
Note that this is not a side-effecting function, i.e. it does no actual byte code emission. It merely returns the symbolic byte code to implement the semantics of static property retrieval as pure Clojure data, and MAGE will perform the actual generation of runnable code as a final step. This makes compilers easier to write and test interactively in a REPL.
Note also that the compiler takes an additional argument compilers, though it makes no use of it. compilers is a map of keywords identifying AST node types (the :op key in the map tools.analyzer produces) to compiler functions. The basic one built into MAGIC looks like
(def base-compilers
{:const #'const-compiler
:do #'do-compiler
:fn #'fn-compiler
:let #'let-compiler
:local #'local-compiler
:binding #'binding-compiler
...Every compiler is passed such a map, and is expected it pass it down when recursively compiling. The compiler for (do ...) expressions does exactly this
(defn do-compiler
[{:keys [statements ret]} compilers]
[(map #(compile % compilers) statements)
(compile ret compilers)])do expressions analyze to hash maps containing :statements and :ret keys referring to all expressions except the last, and the last expression respectively. The do compiler recursively compiles all of these expression, passing its compilers argument to them.
Early versions of MAGIC used a multi method in place of this compiler map, but the map has several advantages. Emission can be controlled from the top level by passing in a different map. For example, compilers can be replaced:
(binding [magic/*initial-compilers*
(merge magic/base-compilers
:let #'my-namespace/other-let-compiler)]
(magic/compile-fn '(fn [a] (map inc a))))or updated
(binding [magic/*initial-compilers*
(update magic/base-compilers
:let (fn [old-let-compiler]
(fn [ast compilers]
(if-not (condition? ast)
(old-let-compiler ast compilers)
(my-namespace/other-let-compiler ast compilers)))))]
(magic/compile-fn '(fn [a] (map inc a))))Additionally, compilers can change this map before they pass it to their children if they need to. This can be used to tersely implement optimizations, and some Clojure semantics depend on it. magic.core/let-compiler implements symbol binding using this mechanism.
A spell is one such map rewrite packaged as a (fn [compilers] compilers'), applied on top of base-compilers. The built-in spells live in src/magic/spells/.
Every knob that controls compilation is a dynamic var in src/magic/flags.clj, the single configuration surface: you binding the flags, the compiler reads them.
Clients bind the optimization vars to control the codegen their build ships, chiefly *direct-linking* and *strongly-typed-invokes*. Compiler developers also bind the build-shaping options when needed: *sparse-case* for runtime-stable hashing (the bb build-magic-portable bootstrap pass), plus *legacy-dynamic-callsites* and *elide-meta*. Spells (*lift-vars*, *lift-keywords*, *sparse-case*) are flags too, so toggling one is the same idiom as any other:
(binding [magic.flags/*direct-linking* true
magic.flags/*sparse-case* true]
(compile-namespace 'my.app))During the development of Arcadia, it was found that binaries produced by the ClojureCLR compiler did not survive Unity's AOT compilation process to its more restrictive export targets, particularly iOS, WebGL, and the PlayStation. While it is understood that certain more 'dynamic' features of C# are generally not supported on these platforms, the exact cause of the failures is difficult to pinpoint. Additionally, the byte code the standard compiler generates is not as good as it can be in situations where the JVM and CLR semantics do not match up, namely value types and generics. MAGIC was built primarily to support better control over byte code, and a well reasoned approach to Arcadia export.
MAGIC stands for "Morgan And Grand Iron Clojure" (originally "Morgan And Grand IL Compiler"). It is named after the Morgan Avenue and Grand Street intersection in Brooklyn, the location of the Kitchen Table Coders studio where the library was developed. "Iron" is the prefix used for dynamic languages ported to the CLR (e.g. IronPython, IronRuby).
Copyright © 2015-2020 Ramsey Nasser and contributors.
Licensed under the Apache License, Version 2.0.