A Theory ofHygienic MacrosDavid Herman, Mitchell WandNortheastern University

A Theory of Hygienic MacrosIntroduction to macros and hygieneThe challenge of formalizing hygienem and its properties

In the beginningMACDEF SUM A,B,C LOAD A ADD B STORE CENDDEF

The essence of macrosDerived (user-defined) syntactic constructsDefined by rewriting to existing constructsTranslated at compile-time (macro expansion)

repeat s until e begin s ; while (not e) s end end

The essence of macrosDerived (user-defined) syntactic constructsDefined by rewriting to existing constructsTranslated at compile-time (macro expansion)

(repeat s until e) (begin s (while (not e) s))

Example: short-circuit or(or (foo "bar") (baz 42))

(define-macro (or e1 e2) (with (t e1) (if t t e2)))evaluate only if (foo "bar") produces #fevaluate first; evaluate only oncepattern; matches subexpressions e1 and e2template; rewrites expression using e1 and e2

The power of macrosAbstractions for control, scope, program structurePowerful meta-programming idiomsSimplifies language core (extensible languages)

local bindings, custom loop forms, test harnesses, dataconditionals,constructors,regular expressions,languagedesignerlibrary writerlittle lang- uages, short-circuit operators,

A brief history of hygiene

Nave macro expansion(define-macro (or e1 e2) (with (t e1) (if t t e2)))(with (x 1) (or #f x))(with (x 1) (with (t #f) (if t t x)))(with (t 1) (or #f t))(with (t 1) (with (t #f) (if t t t)))

Nave macros are leaky abstractions(define-macro (or e1 e2) (with (t e1) (if t t e2)))

A leaky abstraction is a brittle abstraction.Please dont refer to any variables named t in expression e2!Oh, and dont use or in a context that shadows the names with or if!

Kohlbecker: This calls for hygieneFor reasons of hygiene it will always be assumed that the bound variables that occur in a certain expression are different from the free ones. This can be fulfilled by renaming bound variables.H. P. Barendregt

Hygienic macro expansion(define-macro (or e1 e2) (with (t e1) (if t t e2)))(with (x 1) (or #f x))(with (x 1) (with (t #f) (if t t x)))(with (t 1) (or #f t))(with (t 1) (with (t #f) (if t t t)))

Hygiene and syntactic abstractionlibrarieslocal bindings, custom loop forms, test harnesses, dataconditionals,constructors,regular expressions,little lang- uages, short-circuit operators,language

Hygiene and syntactic abstractionlanguagelibraries

What do we know about hygiene?

Hygiene by example(define-macro (or e1 e2) (with (t e1) (if t t e2)))(with (x 1) (or #f x))(with (x 1) (with (t #f) (if t t x)))(with (t 1) (or #f t))(with (t 1) (with (t #f) (if t t t)))

Hygiene by informal rulesKohlbecker et al, 86: Generated identifiers that become binding instances in the completely expanded program must only bind variables that are generated at the same transcription step.

Clinger and Rees, 91: 1. It is impossible to write a macro that inserts a binding that can capture references other than those inserted by the macro. 2. It is impossible to write a macro that inserts a reference that can be captured by bindings other than those inserted by the macro.

SRFI 72: A binding for an identifier can only capture a reference to another if both were present in the source or introduced during a single evaluation of a syntax or quasisyntax form, with the understanding that the evaluation of any nested, unquoted syntax or quasisyntax forms counts as part of the evaluation of an enclosing quasisyntax.

Hygiene by implementation

But which implementation??Kohlbecker et al, 86Clinger & Rees 91Dybvig et al, 93Van Tonder, 05

Insight #1: Lexical scope(define-macro (or e1 e2) (with (t e1) (if t t e2)))(with (x 1) (or #f x))(with (x 1) (with (t #f) (if t t x)))(with (t 1) (or #f t))(with (t 1) (with (t #f) (if t t t)))==

The essence of hygienic macros

Hygienic macro expansion preserves -equivalence.

What variables are bound by with?(define-macro (with (a e1) e2) )

(with (x 1) (or #f x))(with (t 1) (or #f t))=

What is the binding structure?(define-macro (murky a e) (begin (set! a e) (lambda (a) e)))

(lambda (foo) (murky foo (+ foo 1)))

What is the binding structure?(define-macro (turing-machine e)

)

What is the binding structure?(define-macro (indecisive a e) (set! a e) (lambda (a) e))

(lambda (foo) (indecisive foo (+ foo 1)))

What variables are bound by with?(define-macro (with (a e1) e2) ) )Standard answer: look at results of expansion(with (x 1) (or #f x))((lambda (x) ((lambda (t) (if t t x)) #f) 1)(with (t 1) (or #f t))((lambda (t) ((lambda (t) (if t t t)) #f) 1)

Using expansion to define scopeTo specify hygienic expansion, we need to know the binding structure (scope) of programs.To specify the binding structure of programs, we need to know the results of hygienic expansion.

Oops.

Inside every large language is a small language struggling to get outIgarashi, Pierce, and Wadler

Insight #2: Specifying lexical scope

What variables are bound by with?(define-macro (with (a e1) e2) ) )In practice, programmers write binding specifications:with takes the form (with (a e1) e2) and produces an expression, where:a is an identifiere1 and e2 are expressionsa is bound in e2

A shape type for with

(with ( expr) expra) exprbinderexpression in same environmentexpression in scope of a

Explicitly type-annotated macroslet val x = 1 in x end(with (x 1) x )

(define-macro (with (a e1) e2) : expr ((a : var) (e1 : expr) (e2 : expr a)) )

Shape-directed -conversion

(with ( expr) expra) expr(with (x 1) (or #f x))(with (t 1) (or #f t))=

Shape-directed -conversion

(with ( expr) expra) expr(with (z 1) (or #f z))(with (t 1) (or #f t))=

Shape-directed -conversion

(with ( expr) expra) expr(with (z 1) (or #f z))(with (z 1) (or #f z))=

Quick recapMacro shape types binding structureBinding structure -equivalence-equivalence correctness of hygienic expansion

So, what of hygienic expansion?

m: a model of hygienic macros(with (x 1) x)S-expressionscompilers core language

m: a model of hygienic macrose ::=var | var.e | e e|let syntax var = m in e end|(op s )

core languagesource

Hygienic macro expansion

let syntax x = m in e end e[m/x] if BV(e) FV(m) =

((macro p => e) s ) (e) if (p) = s for some substitution and BV(s) FV(e) = and BV(e) # s

Properties of m

Confluence(with (x 1) (or #f x))(x.(with (t #f) (if t t x))) 1(x.(or #f x)) 1(with (x 1) (with (t #f) (if t t x)))

HygieneFor (well-typed) e0 = e0 and fully-expanded e and e, if e0 * e and e0 * e, then e = e.e0e0ee**

Hygienefrom confluence**ss1s1s

Hygienefrom confluence**s0s1s1s2s0s2

ContributionsSemantics for hygienic expansion, accommodating: Lexically scoped macros Binding forms of arbitrary shape Formal definitions and proofs -equivalence in presence of macros Shape-directed -conversion Confluence and hygiene

What weve gainedFormalized programmers mental model of the binding structure of macros.Showed the model has good theoretical properties.Validated intuition: hygienic macro expansion frees the programmer to -convert with impunity.Butmore features yet to model

Moral:Thank you.

Hygienic macro expansion preserves -equivalence.