growing languages with metamorphic syntax macros

PEPM 2002 Growing Languages with Metamorphic Syntax Macros January 14, 2002

Growing Languages with Metamorphic Syntax Macros

Claus Brabrand

Michael Schwartzbach

BRICS, University of Aarhus, Denmark


Outline

• Introduction

• Metamorphisms

• vDSL

• Specificity parsing

• Related and future work

• Conclusion


Lexical Macros

• MLEX: (TOKENS)n TOKENS, n 0


Lexical Macros


#define square(X) X*X#define square(X) X*X


Lexical Macros


#define square(X) X*X

square(y+1) y+1*y+1


square(y+1) y+1*y+1


Lexical Macros



square(y+1) y+1*y+1


square(y+1) y+1*y+1

#define square(X) (X)*(X)

square(y+1) (y+1)*(y+1)


square(y+1) (y+1)*(y+1)


Lexical Macros


• Problem: Independent of syntax!• Unsafe: parse errors discovered at invocation-time


square(y+1) y+1*y+1


square(y+1) y+1*y+1


square(y+1) (y+1)*(y+1)


square(y+1) (y+1)*(y+1)


Invocation Syntax

#define swap(X,Y) { int t=X; X=Y; Y=t; }#define swap(X,Y) { int t=X; X=Y; Y=t; }


Invocation Syntax


if (a>b) swap(a,b);

else b=0;

if (a>b) swap(a,b);

else b=0;*** parse error!


Invocation Syntax


if (a>b) swap(a,b);

else b=0;

if (a>b) swap(a,b);


#define swap(X,Y) do { int t=X; X=Y; Y=t; } while (0)#define swap(X,Y) do { int t=X; X=Y; Y=t; } while (0)

if (a>b) swap(a,b);

else b=0;

if (a>b) swap(a,b);

else b=0;


Invocation Syntax

• Problem: fixed invocation syntax!• same for exp / stm / …


if (a>b) swap(a,b);

else b=0;

if (a>b) swap(a,b);


#define swap(X,Y) do { int t=X; X=Y; Y=t; } while (0)#define swap(X,Y) do { int t=X; X=Y; Y=t; } while (0)

if (a>b) swap(a,b);

else b=0;

if (a>b) swap(a,b);

else b=0;

M(x,y,z)M(x,y,z)


Syntax Macro• MSYN: (AST)n AST, n 0

• Typed with nonterminals of the host grammar• Safe: no parse errors as a conseq. of expansion




stm repeat stm until ( exp ) ;stm repeat stm until ( exp ) ;




macro <stm> repeat <stm S> until ( <exp E> ) ; ::= { { bool first = true;

while (first || !<E>) {

<S>

first = false;

} }

}



<S>

first = false;

} }

}







<S>

first = false;

} }

}



<S>

first = false;

} }

}

1. Invocation syntax: grammar extension







<S>

first = false;

} }

}



<S>

first = false;

} }

}

1. Invocation syntax: grammar extension

2. Transformation: morphing into host syntax



Syntax Flexibility?

macro <decls> enum { <id X> }; ::= {…}macro <decls> enum { <id X>, <id Y> }; ::= {…} macro <decls> enum { <id X>, <id Y>, <id Z> }; ::= {…}



Syntax Flexibility?



decls enum { id } ;

enum { id , id } ;

enum { id , id , id } ;

decls enum { id } ;

enum { id , id } ;



Syntax Flexibility?



decls enum { id } ;

enum { id , id } ;


decls enum { id } ;

enum { id , id } ;


• Problems:• Only fixed (finite) arity• Highly redundant


Syntax Flexibility

• [Scheme]:• special list constructor: “...”


Syntax Flexibility


decls ( enum id * )decls ( enum id * )


Syntax Flexibility


• [MS2]:• lists +/, options ?, tuples {…}, and

token-separated lists T



Syntax Flexibility


• [MS2]:• lists +/, options ?, tuples {…}, and

token-separated lists T


decls enum { id , } ;decls enum { id

, } ; ~ E-BNF


Syntax Flexibility!

• Allow user-defined nonterminals(in invocation syntax):


Syntax Flexibility!


decls enum { id enums } ;

enums , id enums


enums , id enums


Syntax Flexibility!


• Transformation?• without compromising safety


enums , id enums


enums , id enums


Metamorphisms

• Our solution: Metamorphisms• Attach host nonterminals to a user-def’d nonterminal

metamorph <n> m();metamorph <n> m();


Metamorphisms

• Our solution: Metamorphisms• Attach host nonterminals to a user-def’d nonterminal

metamorph <n> m();

macro <stm> … <m: A> … ::= { … <A> … }

metamorph <n> m();

macro <stm> … <m: A> … ::= { … <A> … }


Metamorphisms

• Our solution: Metamorphisms• Attach host nonterminals to a user-def’d nonterminal• Specify morphing (into host syntax) inductively

metamorph <n> m();

macro <stm> … <m: A> … ::= { … <A> … }

metamorph <n> m();

macro <stm> … <m: A> … ::= { … <A> … }


Metamorphisms

• Our solution: Metamorphisms• Attach host nonterminals to a user-def’d nonterminal• Specify morphing (into host syntax) inductively

metamorph <n> m();

macro <stm> … <m: A> … ::= { … <A> … }

morph <m> … ::= { … }

metamorph <n> m();

macro <stm> … <m: A> … ::= { … <A> … }

morph <m> … ::= { … }


Metamorphisms

• Our solution: Metamorphisms• Attach host nonterminals to a user-def’d nonterminal• Specify morphing (into host syntax) inductively• Non-local transformations (multiple results)

metamorph <n> m();

macro <stm> … <m: A> … ::= { … <A> … }

morph <m> … ::= { … }

metamorph <n> m();

macro <stm> … <m: A> … ::= { … <A> … }

morph <m> … ::= { … }


Metamorphisms

• Our solution: Metamorphisms• Attach host nonterminals to a user-def’d nonterminal• Specify morphing (into host syntax) inductively• Non-local transformations (multiple results)

metamorph <n,n’> m();

macro <stm> … <m: A, B> … ::= { … <A> … <B> … }

morph <m> … ::= { … } { … }

metamorph <n,n’> m();

macro <stm> … <m: A, B> … ::= { … <A> … <B> … }

morph <m> … ::= { … } { … }


Metamorph Example: enumdecls enum { id enums } ;

enums , id enums


enums , id enums



enums , id enums


enums , id enums

enum { x, y, z };enum { x, y, z }; const int x = 0;const int y = 1;const int z = 2;

const int x = 0;const int y = 1;const int z = 2;



enums , id enums


enums , id enums

enum { x, y, z };enum { x, y, z }; const int x = 0;const int y = 1;const int z = 2;

const int x = 0;const int y = 1;const int z = 2;

• Without compile-time programminglanguage (with AST values)



enums , id enums


enums , id enums

enum { x, y, z };enum { x, y, z };

int e = 0;const int x = e++;const int y = e++;const int z = e++;

int e = 0;const int x = e++;const int y = e++;const int z = e++;

• Without compile-time programminglanguage (with AST values)


Metamorph Example: enum

metamorph <decls> enums();metamorph <decls> enums();


enums , id enums


enums , id enums



metamorph <decls> enums();

macro <decls> enum { <id X> <enums: Ds> } ; ::= { int e = 0;

const int <X> = e++;

<Ds>

}




<Ds>

}


enums , id enums


enums , id enums






<Ds>

}

morph <enums> , <id X> <enums: Ds> ::= { const int <X> = e++;

<Ds>

}




<Ds>

}


<Ds>

}


enums , id enums


enums , id enums






<Ds>

}


<Ds>

}

morph <enums> ::= { }




<Ds>

}


<Ds>

}



enums , id enums


enums , id enums


stm reserve ( res ) stm

res id res


res id res

Metamorph Example: reserve

reserve ( a b c ) ...;reserve ( a b c ) ...;

acquire(a); acquire(b); acquire(c); ...; release(c); release(b);release(a);

acquire(a); acquire(b); acquire(c); ...; release(c); release(b);release(a);


Multiple ResultsExample: reserve

metamorph <stms,stms> res();metamorph <stms,stms> res();


res id res


res id res



metamorph <stms,stms> res();

macro <stm> reserve ( <res: S1,S2> ) <stm S> ::= { { <S1> <S> <S2> }

}



}


res id res


res id res





}

morph <res> <id X> <res: S1,S2> ::= { acquire(<X>); <S1>

} {

<S2> release(<X>);

}



}


} {

<S2> release(<X>);

}


res id res


res id res





}


} {

<S2> release(<X>);

}

morph <res> ::= { } { }



}


} {

<S2> release(<X>);

}

morph <res> ::= { } { }


res id res


res id res


Metamorph Advantages

• Flexibility

• Safety

• Simplicity



• Flexibility:• Tree structures• Non-local transformations (multiple results)

• Safety

• Simplicity




• Safety:• No parse errors as a conseq. of macro expansion• Guaranteed termination

• Simplicity




• Safety:• No parse errors as a conseq. of macro expansion• Guaranteed termination

• Simplicity: • Based entirely on declarative concepts:

grammars and substitution


vDSL:very Domain Specific Languagestudies

course Math101

title “Mathematics 101”

2 point fall term

…

exclusions

Math101 <> MathA

Math102 <> MathB

prerequisites

Math101, Math102 < Math201, Math202, Math203

Math101, CS101 < CS202

studies

course Math101

title “Mathematics 101”

2 point fall term

…

exclusions

Math101 <> MathA

Math102 <> MathB

prerequisites

Math101, Math102 < Math201, Math202, Math203

Math101, CS101 < CS202


Specificity Parsing

macro <exp> select <id I> from <exp E> where <exp E2>

macro <exp> select all from <exp E> where <exp E2>




Specificity Parsing

• Challenge rounds:– Select most specific productions (wrt. FIRST sets)






Specificity Parsing

• Challenge rounds:– Select most specific productions (wrt. FIRST sets)

• Resolves many ambiguities• Independent of definition-order• Overloading• Avoids keywordification• Commit no branch explosion, no backtracking






Related Work: Macro Survey• 8x Macro languages:

• { CPP, M4, TEX, Dylan, C++ Templates, Scheme, JTS, MS2 }


Related Work: Macro Survey• 8x Macro languages:

• { CPP, M4, TEX, Dylan, C++ Templates, Scheme, JTS, MS2 }

• 31x Macro properties:• { Level of operation, Programmability, Definition scope,

Termination, Argument syntax, Error trailing, … }


Related Work

• “Extensible Syntax with Lexical Scoping” - by Cardelli, Matthes, and Abadi :

• Not a macro language, but a parser generator:– Target language (not host language)– Extend recompile parser

• Localized transformation only• Disjoint productions• Keywordification• Explicit alpha conversion


Future Work: metafront

• Extensible syntax processor, based on:• Specificity parsing and• Metamorphic syntax macros:

metafront metafrontbase grammar: Lmacros: L+ Lprogram in L+

program in L




• Safe transformation: L+ L


program in L




• Safe transformation: L+ L• Safe transformation: L’ L, (vDSL)


program in L


ConclusionMetamorphic Syntax Macros is a…

• flexible• safe• simple

…way of Growing Languages


ConclusionMetamorphic Syntax Macros is a…

• flexible• safe• simple

…way of Growing Languages

Fully implemented in <bigwig> (language for developing interactive Web services)

http://www.brics.dk/bigwig/http://www.brics.dk/bigwig/


The End

next: bonus slides


Pretty Printing & Error Reporting• Pretty Printing:

• Terminal printers:ASCII, LA EX, HTML(+/- expansion)

• Error Reporting:• stdout, HTML

*** symbol errors:*** test.wig:7: Identifier ‘inf’ not declared in macro argument ‘S’ in macro invocation ‘reader’ (test.wig:7) defined in [std.wigmac:44]

*** symbol errors:*** test.wig:7: Identifier ‘inf’ not declared in macro argument ‘S’ in macro invocation ‘reader’ (test.wig:7) defined in [std.wigmac:44]

T


-Conversion

macro <stm> repeat <stm S> until ( <exp E> ) ; ::= { {

bool first = true;


<S>

first = false;

}

}

}

macro <stm> repeat <stm S> until ( <exp E> ) ; ::= { {

bool first = true;


<S>

first = false;

}

}

}


-Conversion




<Ds>

}


<Ds>

}





<Ds>

}


<Ds>

}


decls : enum { id enums } ;

enums : , id enums

|

decls : enum { id enums } ;

enums : , id enums

|


Example: switchmetamorph <stm> swb();

macro <stm> switch ( <exp E> ) { <swb: S> } ::= { { var x = <E>; <S>}}

morph <swb> case <exp E> : <stms Ss> break; <swb: S>::= { if (x==<E>) { <Ss> } else <S>}

morph <swb> case <exp E> : <stms Ss> break; ::= { if (x==<E>) { <Ss> }}

metamorph <stm> swb();

macro <stm> switch ( <exp E> ) { <swb: S> } ::= { { var x = <E>; <S>}}

morph <swb> case <exp E> : <stms Ss> break; <swb: S>::= { if (x==<E>) { <Ss> } else <S>}

morph <swb> case <exp E> : <stms Ss> break; ::= { if (x==<E>) { <Ss> }}

stm : switch ( exp ) { swb }

swb : case exp : stms break; swb

| case exp : stms break;

stm : switch ( exp ) { swb }

swb : case exp : stms break; swb

| case exp : stms break;


Metamorph Wellformedness

• Check at definition time:• “no left recursion” (guarantees termination):

• “derivability” (metamorphisms must derive something finite) :

xlist xlist X

xlist xlist X

xlist X xlist

xlist X xlist

xlist X xlist

xlist X xlist

xlist X xlistxlist X xlist


Representationmacro <ids> MXY ( <ids Is> ) ::= { X, <Is>, Y }

A, MXY(B, C), D A, X, B, C, Y, D

macro <ids> MXY ( <ids Is> ) ::= { X, <Is>, Y }

A, MXY(B, C), D A, X, B, C, Y, D

• “Weaving” yields transparency!

weave


Syntax Macros

• MSYN: (AST)n AST , n 0

~

square( )

macro <exp> square ( <exp E> ) ::= { <E> * <E>

}

macro <exp> square ( <exp E> ) ::= { <E> * <E>

}**

exp

exp

Eexp

E

**

exp

y + 1y

+ 1 y + 1y

+ 1

exp

Eexp

Ey

+ 1y + 1

exp

growing languages with metamorphic syntax macros

Documents