a theory of hygienic macros david herman, mitchell wand northeastern university

A Theory ofHygienic Macros

David Herman, Mitchell WandNortheastern University

2

A Theory of Hygienic Macros

Introduction to macros and hygiene

The challenge of formalizing hygiene

m and its properties

3

In the beginning MACDEF SUM A,B,C LOAD A ADD B STORE CENDDEF

4

The essence of macros

Derived (user-defined) syntactic constructs Defined by rewriting to existing constructs Translated at compile-time (“macro expansion”)

repeat s until e

begin s ; while (not e) s end end

5

The essence of macros

Derived (user-defined) syntactic constructs Defined by rewriting to existing constructs Translated at compile-time (“macro expansion”)

(repeat s until e)

(begin s (while (not e) s))

6

Example: short-circuit or

(or (foo "bar") (baz 42))

(define-macro (or e1 e2) (with (t e1) (if t t e2)))

evaluate only if (foo "bar") produces #f

evaluate first; evaluate only once

pattern; matches subexpressions

e1 and e2

template; rewrites expression using

e1 and e2

7

The power of macros

Abstractions for control, scope, program structure Powerful meta-programming idioms Simplifies language core (“extensible languages”)

functions, recursion, iteration, primitive datatypes

local bindings, custom loop forms, test harnesses, data

conditionals,constructors,

regularexpressions,

languagedesigner

library writer

little lang-uages, …

short-circuit operators,

8

A brief history of hygiene

9

Naïve macro expansion

(define-macro (or e1 e2) (with (t e1) (if t t e2)))

(with (x 1) (or #f x))(with (x 1) (with (t #f) (if t t x)))

(with (t 1) (or #f t))(with (t 1) (with (t #f) (if t t t)))

10

Naïve macros are leaky abstractions(define-macro (or e1 e2) (with (t e1) (if t t e2)))

A leaky abstraction is a brittle abstraction.

Please don’t refer to any variablesnamed t in expression e2!

Oh, and don’t use or in a contextthat shadows the names with or if!

11

For reasons of hygiene it will always be assumed that the bound variables that occur in a certain expression are different from the free ones. This can be fulfilled by renaming bound variables.

—H. P. Barendregt

Kohlbecker: This calls for hygiene

12

Hygienic macro expansion

(define-macro (or e1 e2) (with (t e1) (if t t e2)))

(with (x 1) (or #f x))(with (x 1) (with (t #f) (if t t x)))

(with (t 1) (or #f t))(with (t 1) (with (t′ #f) (if t′ t′ t)))

13

Hygiene and syntactic abstraction

functions, recursion, iteration, primitive datatypes

libraries

functions, recursion, iteration, primitive datatypes

local bindings, custom loop forms, test harnesses, data

conditionals,constructors,

regularexpressions,little lang-uages, …

short-circuit operators,

language

14

Hygiene and syntactic abstraction

functions, recursion, iteration, primitive datatypes

custom loop forms, derived datatypes, first-class module systems, object-orientation, test harnesses, data

constructors, laziness, short-circuit boolean operators, iteration protocols,

static analyses, parser generators, partial evaluation, debuggers,

coroutines, exceptions, threads, type checkers,

introspection, etc…

local bindings, software contracts,

language

libraries

15

What do we know about hygiene?

16

Hygiene by example

(define-macro (or e1 e2) (with (t e1) (if t t e2)))

(with (x 1) (or #f x))(with (x 1) (with (t #f) (if t t x)))

(with (t 1) (or #f t))(with (t 1) (with (t #f) (if t t t)))

17

Hygiene by informal rules

Kohlbecker et al, ’86:

Generated identifiers that become binding instances in the completely expanded program must only bind variables that are generated at the same transcription step.

Clinger and Rees, ’91:

1. It is impossible to write a macro that inserts a binding that can capture references other than those inserted by the macro.

2. It is impossible to write a macro that inserts a reference that can be captured by bindings other than those inserted by the macro.

SRFI 72:

A binding for an identifier can only capture a reference to another if both were present in the source or introduced during a single evaluation of a syntax or quasisyntax form, with the understanding that the evaluation of any nested, unquoted syntax or quasisyntax forms counts as part of the evaluation of an enclosing quasisyntax.

18

Hygiene by implementation

19

But which implementation?

?

Kohlbecker et al, ’86 Clinger & Rees ’91 Dybvig et al, ’93 Van Tonder, ’05

20

Insight #1: Lexical scope

(define-macro (or e1 e2) (with (t e1) (if t t e2)))

(with (x 1) (or #f x))(with (x 1) (with (t #f) (if t t x)))

(with (t 1) (or #f t))(with (t 1) (with (t′ #f) (if t′ t′ t)))

=

=

21

Hygienic macro expansionpreserves -equivalence.

The essence of hygienic macros

22

What variables are bound by with?(define-macro (with (a e1) e2) )

(with (x 1) (or #f x))

(with (t 1) (or #f t))

=

…

23

What is the binding structure?

(define-macro (murky a e) (begin (set! a e) (lambda (a) e)))

(lambda (foo) (murky foo (+ foo 1)))

24

What is the binding structure?

(define-macro (turing-machine e)

)

25

What is the binding structure?

(define-macro (indecisive a e) … (set! a e) … (lambda (a) e))

(lambda (foo) (indecisive foo (+ foo 1)))

26

What variables are bound by with?(define-macro (with (a e1) e2) …) )

Standard answer: look at results of expansion

(with (x 1) (or #f x))((lambda (x) ((lambda (t) (if t t x)) #f) 1)

(with (t 1) (or #f t))((lambda (t) ((lambda (t′) (if t′ t′ t)) #f) 1)

…

27

Using expansion to define scope

To specify hygienic expansion, we need to know the binding structure (scope) of programs.

To specify the binding structure of programs, we need to know the results ofhygienic expansion.

Oops.

28

Inside every large language is a small language struggling to get out…

—Igarashi, Pierce, and Wadler

29

Insight #2: Specifying lexical scope

30

What variables are bound by with?(define-macro (with (a e1) e2) …) )

In practice, programmers write binding specifications:

with takes the form (with (a e1) e2) and produces an expression, where:

a is an identifier e1 and e2 are expressions a is bound in e2

…

31

A shape type for with

(with (<a> expr) expra) → expr

binder

expression in same

environment

expression in scope of a

32

Explicitly type-annotated macros

let val x = 1 in … x …end

(with (x 1) … x …)

(define-macro (with (a e1) e2) : expr ((a : var) (e1 : expr) (e2 : expr a)) )

…

…

33

Shape-directed -conversion

(with (<a> expr) expra) → expr

(with (x 1) (or #f x))

(with (t 1) (or #f t))

=

34

Shape-directed -conversion

(with (<a> expr) expra) → expr

(with (z 1) (or #f z))

(with (t 1) (or #f t))

=

35

Shape-directed -conversion

(with (<a> expr) expra) → expr

(with (z 1) (or #f z))

(with (z 1) (or #f z))

=

36

Quick recap

Macro shape types binding structure Binding structure -equivalence -equivalence correctness of hygienic

expansion

So, what of hygienic expansion?

37

m: a model of hygienic macros

(with (x 1) x)

(x.x) 1

S-expressions

compiler’s core language

38

m: a model of hygienic macros

e ::= var | var.e | e e| let syntax var = m in e end| (op s …)

core language

source

39

Hygienic macro expansion

let syntax x = m in e end → e[m/x]

if BV(e) ∩ FV(m) =

((macro p => e) s …) → (e)

if (p) = s for some substitution and BV(s) ∩ FV(e) = and BV(e) # s

40

Properties of m

41

Confluence

(with (x 1) (or #f x))

(x.(with (t #f) (if t t x))) 1

(x.(or #f x)) 1(with (x 1) (with (t #f) (if t t x)))

42

Hygiene

For (well-typed) e0 = e′0 and fully-expanded e and e′, if e0 →

* e and e′0 →

* e′, then e = e′.

e0

e′0

e

e′

*

*

43

Hygiene…from confluence

* *

s

s1 s′1

s′

44

Hygiene…from confluence

* *

s0

s1 s′1

s2

s′0

s′2

45

Contributions

Semantics for hygienic expansion, accommodating: Lexically scoped macros Binding forms of arbitrary shape

Formal definitions and proofs -equivalence in presence of macros Shape-directed -conversion Confluence and hygiene

46

What we’ve gained

Formalized programmers’ mental model of the binding structure of macros.

Showed the model has good theoretical properties.

Validated intuition: hygienic macro expansion frees the programmer to -convert with impunity.

But—more features yet to model…

47

Thank you.

Hygienic macro expansionpreserves -equivalence.

Moral: