dynamic language embedding with homogeneous tool support

Dynamic Language EmbeddingWith Homogeneous Tool Support

PhD DefenseLukas Renggli

AdvisorOscar Nierstrasz

1

2

SELECT email FROM usersWHERE username = 'lr'

3


Syntax

4


Semantics

General PurposeHost Language

5

+


General PurposeHost Language

6

?


SyntaxSQL SemanticsSQL

SyntaxHost SemanticsHost

ToolsHost

8

Host Language

External Language

10

Host Language

Internal Language

12

Host Language

Embedded Language

13

Non-StandardHost Language

Embedded Language

15

!" #"$%&

Mar

co Z

anol

i, cc

-by-

sa, 2

.5, w

ww

.wik

iped

ia.d

e

16

http://www.wikipedia.dem


ConventionalLanguage

17

18 Mar

co Z

anol

i, cc

-by-

sa, 2

.5, w

ww

.wik

iped

ia.d

e



Multiple Context Specific Languages

19

20 Mar

co Z

anol

i, cc

-by-

sa, 2

.5, w

ww

.wik

iped

ia.d

e



Homogeneous Code and Data Abstraction

21

§

§§

22 Mar

co Z

anol

i, cc

-by-

sa, 2

.5, w

ww

.wik

iped

ia.d

e



§

§

§

HomogeneousTool Support

23

Thesis

To support seamless integration of context-dependent languages without breaking the tools, we need

1. a host-language grammar that can be changed by language extensions,

2. a first-class language description used by the development environment, and

3. a transformation mechanism of the embedded language into a common executable representation.

24

Language Boxes

Host Language

Dyn

amic

G

ram

mar

s

Language and Tool Extensions

25

Language Boxes

Host Language

Dyn

amic

G

ram

mar

s


26

Language Boxes

Host Language

Dyn

amic

G

ram

mar

s


27

Language Boxes

Host Language

Dyn

amic

G

ram

mar

s


28

Language Boxes

Host Language

Dyn

amic

G

ram

mar

s


29

Language Boxes

Host Language

Dyn

amic

G

ram

mar

s


30

31

!" #"$%&

32

Types of Embedded Languages

33

◦ ◦ ◦◦ ◦ ●◦ ● ◦◦ ● ●● ◦ ◦● ◦ ●● ● ◦● ● ●

Syn

tax

Voca

bul

ary

Sem

antic

s

34

Host Language ◦ ◦ ◦◦ ◦ ●◦ ● ◦◦ ● ●● ◦ ◦● ◦ ●● ● ◦● ● ●

Syn

tax

Voca

bul

ary

Sem

antic

s

35

Host Language ◦ ◦ ◦◦ ◦ ●

Internal Language ◦ ● ◦◦ ● ●● ◦ ◦● ◦ ●● ● ◦● ● ●

Syn

tax

Voca

bul

ary

Sem

antic

s

36


Internal Language ◦ ● ◦Pidgin ◦ ● ●

● ◦ ◦● ◦ ●● ● ◦● ● ●

Syn

tax

Voca

bul

ary

Sem

antic

s

37


Internal Language ◦ ● ◦Pidgin ◦ ● ●

● ◦ ◦● ◦ ●● ● ◦

Creole ● ● ●S

ynta

x

Voca

bul

ary

Sem

antic

s

38

Host Language ◦ ◦ ◦Argot ◦ ◦ ●Internal Language ◦ ● ◦Pidgin ◦ ● ●

● ◦ ◦● ◦ ●● ● ◦

Creole ● ● ●S

ynta

x

Voca

bul

ary

Sem

antic

s

39

Host Language ◦ ◦ ◦Argot ◦ ◦ ●Internal Language ◦ ● ◦Pidgin ◦ ● ●— ● ◦ ◦— ● ◦ ●— ● ● ◦Creole ● ● ●

Syn

tax

Voca

bul

ary

Sem

antic

s

40

Pidgin ◦ ● ●Creole ● ● ●Argot ◦ ◦ ●

Syn

tax

Voca

bul

ary

Sem

antic

s

46

Package Name

x = 1

y = 1 (2, 1)

(2, 2)(1, 2)

x = 2

y = 2

Package Name

x = 1

y = 1 (2, 1)

(2, 2)(1, 2)

x = 2

y = 2

47

aBuilder row grow.aBuilder row fill.

aBuilder column grow.aBuilder column fill.

aBuilder x: 1 y: 1 add: (LabelShape new text: [ :each | each name ]; borderColor: #black; borderWidth: 1; yourself).aBuilder x: 1 y: 2 w: 2 h: 1 add: (RectangleShape new borderColor: #black; borderWidth: 1; width: 200; height: 100; yourself)

Package Name

x = 1

y = 1 (2, 1)

(2, 2)(1, 2)

x = 2

y = 2

48

row = grow.row = fill.

column = grow.column = fill.

(1 , 1) = label text: [ :each | each name ]; borderColor: #black; borderWidth: 1.

(1 , 2) -‐ (2 , 1) = rectangle borderColor: #black; borderWidth: 1; width: 200; height: 100.

shape { cols: #grow, #fill; rows: #grow, #fill;}label { position: 1 , 1; text: [ :each | each name ]; borderColor: #black; borderWidth: 1;}rectangle { position: 1 , 2; colspan: 2; borderColor: #black; borderWidth: 1; width: 200; height: 100;}

Package Name

x = 1

y = 1 (2, 1)

(2, 2)(1, 2)

x = 2

y = 2

50

52

Conventional Language

53


Context Specific

54


Context Specific

HomogeneousCode & Data

55


Context Specific



56


Context Specific



Language Boxes

Host Language

Dyn

amic

G

ram

mar

s


57

Renggli et al.ECOOP 2010

58 Mar

co Z

anol

i, cc

-by-

sa, 2

.5, w

ww

.wik

iped

ia.d

e



59 Mar

co Z

anol

i, cc

-by-

sa, 2

.5, w

ww

.wik

iped

ia.d

e



Editor

Compiler

Debugger

Syntax

Language 1

60

Editor

Compiler

Debugger

Syntax

Language 1

61

Language 2

Editor

Compiler

Debugger

Syntax

Language 1

62

Language 2

Editor

Compiler

Debugger

Syntax

Language 1

63

Language 2

Language Boxes

64

65

SELECT * FROM users

66

| r |

r :=

^ User fromRow: r

.SELECT * FROM users

LanguageScope

LanguageConcern

LanguageChange

LanguageBox

67

Language Scope

Active?

68

Language Scope

69

‣ System

‣ Packages

‣ Classes

‣ Methods

Language Scope

70

Language Concern

Semantics

71

Language Concern

Transformation

72

Language Concern

73

Context Menus

Navigation Search

Code Expansion

Code Completion

Error Correction

Custom Inspector

Refactorings

Code Folding

Highlighting

Language Change

74

Syntax

Language Change

75

Host Language

Language Change

76

Host Language

+ SQL Language

Language Change

77

Host Language

+ SQL Language

+ ...

Host Language

+ SQL Language

+ ...

= Custom Host Language

Language Change

78

Language Boxes

Host Language

Dyn

amic

G

ram

mar

s


80

Renggli et al.SLE 2009

Language Boxes

Host Language

Dyn

amic

G

ram

mar

s81


LanguageChange

LanguageConcern

LanguageScope

LanguageBox

82

LanguageChange = Host Language δ

83

LanguageChange =

Host Language Grammarδ

84

LanguageChange =

Grammar Transformation

85

scanIdentifier self step. ((currentCharacter between: $A and: $Z) or: [ currentCharacter between: $a and: $z ]) ifTrue: [ [ self recordMatch: #IDENTIFIER. self step. (currentCharacter between: $0 and: $9)or: [ (currentCharacter between: $A and: $Z) or: [ currentCharacter between: $a and: $z ] ] ] whileTrue. ^ self reportLastMatch ]

86

#(#[1 0 9 0 25 0 13 0 34 0 17 0 40 0 21 0 41] #[1 0 9 0 25 0 13 0 34 0 93 0 76 0 157 0 112] #[1 2 38 0 21 2 38 0 25 2 38 0 26 0 13 0 34] #[0 1 154 0 16 0 21 0 25 0 26 0 34 0 40 0 41] #[0 1 210 0 76 0 81] #[0 1 214 0 76 0 81] #[1 0 173 0 76 0 177 0 81] #[0 1 134 0 16 0 21 0 25 0 26 0 34 0 40 0 41] #[1 1 46 0 21 1 46 0 25 1 46 0 26 1 69] #[1 1 54 0 21 1 54 0 25 1 54 0 26 1 54 0 34] #[0 2 102 0 21 0 25 0 26 0 34 0 40 0 41 0 76]

#[0 2 50 0 21 0 25 0 26 0 76 0 79] #[1 1 13 0 76 2 85 0 124 1 21 0 125] #[1 2 89 0 17 2 30 0 21 2 30 0 82] #[1 2 93 0 21 2 97 0 82] )

87

ScannerlessParser Combinator

88

a..z a..z

0..9

ID ::= letter { letter | digit } ;

89

letter

letter digit

sequence

choice

many

ID ::= letter { letter | digit } ;

90

GrammarTransformation

91

letter

letter digit

sequence

choice

many

letter ! letter | "_"

92

letter

digit

sequence

choice

manychoice

_

letter

choice

_

letter ! letter | "_"

93

letter digit

sequence

choice

manychoice

_

Optimizations

94

GrammarComposition

95

Insert grammar fragment

before/after grammar production

as a choice/sequence/replacement.

96

Language Change

Insert SQL grammar

after expression production

as an additional choice

97

Language Change

98

Language Change

Variable

Literal

Parens

Expr:

99

Language Change

Variable

Literal

Parens

Expr:

SQL

Conflicts &Ambiguities

100

SELECT * FROM users

101

SELECT * FROM users

102

| r |

r :=

^ User fromRow: r

.

expr sql|

103

SELECT * FROM users<SQL: >

104

Parsing Expression Grammars

105

expr sql/

ordered

106

expr sql/

no conflict

107

expr sql/

108

surprise

/ exprsql

109

surprise

expr sql|

110

unordered

expr sql|

111

!sql expr / !expr sql

expr sql$

112

!sql expr / !expr sql / ui

expr sql$

113

!sql expr / !expr sql / ui


Language Boxes

Host Language

Dyn

amic

G

ram

mar

s

Renggli et al.DYLA 2010

114

Language Boxes

Host Language

Dyn

amic

G

ram

mar

s


115

Assignments and Swapping

Asynchronous Messages

Automaton

Brainfuck Language

Functional Pattern Matching

Grammar Definition

Message Pipes

Mondrian

Object Relationships

Positional Arguments

Program Checking

Quasiquoting

Regular Expression

Roman Numbers

SPath Expression

SQL

Schematic Tables

String Interpolation

Transactional Memory

Tuple Space

116[http://scg.unibe.ch/research/helvetia/examples]

http://scg.unibe.ch/research/helvetia/examples%5D




Automaton

Brainfuck Language


Grammar Definition

Message Pipes

✓Mondrian



Program Checking

Quasiquoting

Regular Expression

Roman Numbers

SPath Expression

✓SQL

Schematic Tables



Tuple Space






Automaton

Brainfuck Language


Grammar Definition

Message Pipes

✓Mondrian



‣ Program Checking

‣ Quasiquoting

Regular Expression

Roman Numbers

SPath Expression

✓SQL

Schematic Tables


‣ Transactional Memory

Tuple Space




applyhasChangedhasConflict

Change

object

*changes

Process0..1

currentTransaction

do: aBlockretry: aBlockcheckpointabort: anObject

escapeContext

Transaction

previousCopyworkingCopy

ObjectChange

applyBlockconflictTestBlock

CustomChange

*


119

Meta-Programming Facilities

``(`,(aString) asRegex)

120

Domain-SpecificProgram Checking

121

Language Boxes

Host Language

Dyn

amic

G

ram

mar

s


Renggli et al.

CLSS 2009 Renggli et al.

IWST 2009 Nierstrasz et al.

LNCS 2009 Renggli et al.

TOOLS 2010

122

Language Boxes

Host Language

Dyn

amic

G

ram

mar

s


123





Language Boxes

Host Language

Dyn

amic

G

ram

mar

s


124





Language Boxes

Host Language

Dyn

amic

G

ram

mar

s


125





Language Boxes

Host Language

Dyn

amic

G

ram

mar

s


126





Language Boxes

Host Language

Dyn

amic

G

ram

mar

s127


dynamic language embedding with homogeneous tool support

Technology

support seamless

common executable

sql expr

tool extensions

class language

language change

language extensions

select email