mixing source and bytecode: a case for compilation by normalization (oopsla 2008)
DESCRIPTION
Language extensions increase programmer productivity by providing concise, often domain-specific syntax, and support for static verification of correctness, security, and style constraints. Language extensions can often be realized through translation to the base language, supported by preprocessors and extensible compilers. However, various kinds of extensions require further adaptation of a base compiler's internal stages and components, for example to support separate compilation or to make use of low-level primitives of the platform (e.g., jump instructions or unbalanced synchronization). To allow for a more loosely coupled approach, we propose an open compiler model based on normalization steps from a high-level language to a subset of it, the core language. We developed such a compiler for a mixed Java and (core) bytecode language, and evaluate its effectiveness for composition mechanisms such as traits, as well as statement-level and expression-level language extensions.TRANSCRIPT
![Page 1: Mixing Source and Bytecode: A Case for Compilation By Normalization (OOPSLA 2008)](https://reader033.vdocuments.us/reader033/viewer/2022060120/55910c3d1a28ab95548b4739/html5/thumbnails/1.jpg)
Mixing Source and Bytecode
A Case for Compilation by Normalization
OOPSLA'08, Nashville, Tennessee
Lennart Kats, TU DelftMartin Bravenboer, UMassEelco Visser, TU Delft
Software Engineering Research Group
October 21, 2008
![Page 2: Mixing Source and Bytecode: A Case for Compilation By Normalization (OOPSLA 2008)](https://reader033.vdocuments.us/reader033/viewer/2022060120/55910c3d1a28ab95548b4739/html5/thumbnails/2.jpg)
2
LanguageExtensions
![Page 3: Mixing Source and Bytecode: A Case for Compilation By Normalization (OOPSLA 2008)](https://reader033.vdocuments.us/reader033/viewer/2022060120/55910c3d1a28ab95548b4739/html5/thumbnails/3.jpg)
3
DSLs and Language Extensions
Domain-specific• Database queries• Regular expressions• XML processing• Matrices• Complex numbers• ...
void browse() { List<Book> books =
<| SELECT *
FROM books
WHERE price < 100.00
|>;
…
for (int i = 0; i < books.size(); i++)
books.get(i).title =~ s/^The //;
}
![Page 4: Mixing Source and Bytecode: A Case for Compilation By Normalization (OOPSLA 2008)](https://reader033.vdocuments.us/reader033/viewer/2022060120/55910c3d1a28ab95548b4739/html5/thumbnails/4.jpg)
4
DSLs and Language Extensions
Domain-specific• Database queries• Regular expressions• XML processing• Matrices• Complex numbers• ...
General-purpose• AOP• Traits• Enums• Iterators• Case classes• Properties/accessors• ...
![Page 5: Mixing Source and Bytecode: A Case for Compilation By Normalization (OOPSLA 2008)](https://reader033.vdocuments.us/reader033/viewer/2022060120/55910c3d1a28ab95548b4739/html5/thumbnails/5.jpg)
5
Program C: Foo.class
(bytecode)
Program B: Foo.java
public class Foo { void bar(BarList list) { Iterator it = list.iterator(); while (it.hasNext()) { Bar b = it.next(); b.print(); } }}
Program A: Foo.mylang
public class Foo { void bar(BarList list) { foreach (Bar b in list) { b.print(); } }}
javac
Code generator
Example: The Foreach Statement
![Page 6: Mixing Source and Bytecode: A Case for Compilation By Normalization (OOPSLA 2008)](https://reader033.vdocuments.us/reader033/viewer/2022060120/55910c3d1a28ab95548b4739/html5/thumbnails/6.jpg)
6
The Preprocessor
+ loosely coupled+ small, modular extension+ no need to know compiler
internalsHost language
Preprocessor
Bytecode
Extendedlanguage
CompilerINTERFACE
– no parser– semantic analysis, errors?
![Page 7: Mixing Source and Bytecode: A Case for Compilation By Normalization (OOPSLA 2008)](https://reader033.vdocuments.us/reader033/viewer/2022060120/55910c3d1a28ab95548b4739/html5/thumbnails/7.jpg)
7
The Front-end Extensible Compiler
Extension
Host language
Front-end
Extendedlanguage
+ front-end reuse
Polyglot
ableJ
MetaBorg
INTERFACE
OJ
– tied to the front-end internals
Bytecode
Compiler
![Page 8: Mixing Source and Bytecode: A Case for Compilation By Normalization (OOPSLA 2008)](https://reader033.vdocuments.us/reader033/viewer/2022060120/55910c3d1a28ab95548b4739/html5/thumbnails/8.jpg)
8
public class Shape { void draw() { … } Vertices getVertices() { … }}
Traits: Composable Units of Behavior[Schärli, Ducasse, Nierstrasz, Black 2003]
Traits
public trait TDrawing { void draw() { … } require Vertices getVertices();}
public class Shape with TDrawing { … Vertices getVertices() { … }}
![Page 9: Mixing Source and Bytecode: A Case for Compilation By Normalization (OOPSLA 2008)](https://reader033.vdocuments.us/reader033/viewer/2022060120/55910c3d1a28ab95548b4739/html5/thumbnails/9.jpg)
9
• Simple idea
• Most libraries are distributed as compiled code
→ Why not do this for traits?
Foo.java TBar.java
FooBar.java
FooBar.class
Traits
javac
Traits
Not extensible
![Page 10: Mixing Source and Bytecode: A Case for Compilation By Normalization (OOPSLA 2008)](https://reader033.vdocuments.us/reader033/viewer/2022060120/55910c3d1a28ab95548b4739/html5/thumbnails/10.jpg)
10
TheFullyExtensible Compiler?
![Page 11: Mixing Source and Bytecode: A Case for Compilation By Normalization (OOPSLA 2008)](https://reader033.vdocuments.us/reader033/viewer/2022060120/55910c3d1a28ab95548b4739/html5/thumbnails/11.jpg)
11
TheFullyExtensible Compiler?
+ reuse everything+ access to back-end–– tied to all compiler internals– complexity– performance trade-offs
Bytecode
Parsing
Extendedlanguage
Parsing
ParsingName
analysis
ParsingType
analysis
ParsingCode
generation
Optimize
Parsing
ParsingEmit
Frontend
Backend
![Page 12: Mixing Source and Bytecode: A Case for Compilation By Normalization (OOPSLA 2008)](https://reader033.vdocuments.us/reader033/viewer/2022060120/55910c3d1a28ab95548b4739/html5/thumbnails/12.jpg)
12
–– tied to all compiler internals– complexity– performance trade-offs
TheWhiteBoxModel?
![Page 13: Mixing Source and Bytecode: A Case for Compilation By Normalization (OOPSLA 2008)](https://reader033.vdocuments.us/reader033/viewer/2022060120/55910c3d1a28ab95548b4739/html5/thumbnails/13.jpg)
13
The Normalizing Compiler
Extensionprocessor
Normalizing compiler
Extendedlanguage
Bytecodecore language
Source+bytecode
INTERFACE
+ simple interface+ access to back-end++ not tied to compiler internals
![Page 14: Mixing Source and Bytecode: A Case for Compilation By Normalization (OOPSLA 2008)](https://reader033.vdocuments.us/reader033/viewer/2022060120/55910c3d1a28ab95548b4739/html5/thumbnails/14.jpg)
14
Foo.java TBar.class
FooBar.classTraits, Revisited
FooBar.javaT
raits
![Page 15: Mixing Source and Bytecode: A Case for Compilation By Normalization (OOPSLA 2008)](https://reader033.vdocuments.us/reader033/viewer/2022060120/55910c3d1a28ab95548b4739/html5/thumbnails/15.jpg)
15
class FooBar {
void hello1() {
System.out.println(“Hello Java world”);
}
`hello2 (void) [
getstatic java.lang.System.out : java.io.PrintStream;
ldc “Hello bytecode world”;
invokevirtual java.io.PrintStream.println(java.lang.String : void);
]
}
Java (Foo.java)
Bytecode (TBar.class)
Java, and Bytecode, and Backticks,Oh, My!
![Page 16: Mixing Source and Bytecode: A Case for Compilation By Normalization (OOPSLA 2008)](https://reader033.vdocuments.us/reader033/viewer/2022060120/55910c3d1a28ab95548b4739/html5/thumbnails/16.jpg)
16
class FooBar {
void hello3() {
System.out.println(`ldc "Hello mixed world");
}
`hello4 (void) [
getstatic System.out : java.io.PrintStream;
push `"Hello ” + “Java".concat("bytecode world");
invokevirtual java.io.PrintStream.println (java.lang.String : void);
]
}
PP
S
Java, and Bytecode, and Backticks:Fine-grained Mixing
Typechecker
![Page 17: Mixing Source and Bytecode: A Case for Compilation By Normalization (OOPSLA 2008)](https://reader033.vdocuments.us/reader033/viewer/2022060120/55910c3d1a28ab95548b4739/html5/thumbnails/17.jpg)
17
Traits X
Partial/open classes X
Iterator generators X
Finite State Automata X
Expression-based Xextensions
Class-method / Statement / Expression level
Typical Applications
![Page 18: Mixing Source and Bytecode: A Case for Compilation By Normalization (OOPSLA 2008)](https://reader033.vdocuments.us/reader033/viewer/2022060120/55910c3d1a28ab95548b4739/html5/thumbnails/18.jpg)
18
States and transitions
Grammar
Code
Example: Finite State Automata (DFAs)
![Page 19: Mixing Source and Bytecode: A Case for Compilation By Normalization (OOPSLA 2008)](https://reader033.vdocuments.us/reader033/viewer/2022060120/55910c3d1a28ab95548b4739/html5/thumbnails/19.jpg)
19
while (true) { switch (state) { case INSIDE_QUOTE: switch (nextToken()) { case '"': consume(); state = OUTSIDE_QUOTE; // end quote found! break; … case EOF: return; default: consume(); } break; case OUTSIDE_QUOTE: switch (c) { … } break; }}
+ simple Java code– performance trade-off
Finite State Automata: Generated Java code
![Page 20: Mixing Source and Bytecode: A Case for Compilation By Normalization (OOPSLA 2008)](https://reader033.vdocuments.us/reader033/viewer/2022060120/55910c3d1a28ab95548b4739/html5/thumbnails/20.jpg)
20
InsideQuote: switch (nextToken()) { case '"': consume(); `goto OutsideQuote; case EOF: return; default: consume(); `goto InsideQuote; } }OutsideQuote: switch (c) { … }
+ performance+ low-level when necessary+ minimal changes+ maintainable
Finite State Automata: Inline Bytecode
![Page 21: Mixing Source and Bytecode: A Case for Compilation By Normalization (OOPSLA 2008)](https://reader033.vdocuments.us/reader033/viewer/2022060120/55910c3d1a28ab95548b4739/html5/thumbnails/21.jpg)
21
Method-body Extensions
• Operators: x as T, x ?? y, ...
• Embedded (external) DSLs: Database queries, XML, ...
• Closures Normalize
Lower-levelcode
![Page 22: Mixing Source and Bytecode: A Case for Compilation By Normalization (OOPSLA 2008)](https://reader033.vdocuments.us/reader033/viewer/2022060120/55910c3d1a28ab95548b4739/html5/thumbnails/22.jpg)
22
Stratego Normalization Rules
desugar-foreach: |[ foreach (t x in e) stm ]|
|[ Iterator x_iterator = e.iterator(); while (x_iterator.hasNext()) { t x = x_iterator.next(); stm; } ]| with x_iterator := <newname> “iterator”
![Page 23: Mixing Source and Bytecode: A Case for Compilation By Normalization (OOPSLA 2008)](https://reader033.vdocuments.us/reader033/viewer/2022060120/55910c3d1a28ab95548b4739/html5/thumbnails/23.jpg)
23
Normalizing ??
String property = readProperty(“x”) ?? “default”;System.out.println(property);
// In basic Java:
String temp = readProperty(“x”);if (temp == null) temp = “default”;String property = temp;System.out.println(property);
Context-sensitivetransformation
![Page 24: Mixing Source and Bytecode: A Case for Compilation By Normalization (OOPSLA 2008)](https://reader033.vdocuments.us/reader033/viewer/2022060120/55910c3d1a28ab95548b4739/html5/thumbnails/24.jpg)
24
Normalizing ??
“But placing statements in front of the assimilated expressions is easy, isn’t it?”
Requires syntactic knowledge:• for, while, return, throw, …• other expressions• other extensions
![Page 25: Mixing Source and Bytecode: A Case for Compilation By Normalization (OOPSLA 2008)](https://reader033.vdocuments.us/reader033/viewer/2022060120/55910c3d1a28ab95548b4739/html5/thumbnails/25.jpg)
25
Normalizing ??
“But placing statements in front of the assimilated expressions is easy, isn’t it? … Right?”
Also requires semantic knowledge:
Iterator<String> it = ...;
for (String e = e0; it.hasNext(); e = it.next() ?? “default”) {
...
}
![Page 26: Mixing Source and Bytecode: A Case for Compilation By Normalization (OOPSLA 2008)](https://reader033.vdocuments.us/reader033/viewer/2022060120/55910c3d1a28ab95548b4739/html5/thumbnails/26.jpg)
26
Normalizing ??
“But placing statements in front of the assimilated expressions is easy, isn’t it? … Right?”
Also requires semantic knowledge:
Iterator<String> it = ...;
Integer temp = ...;
for (String e = e0; it.hasNext(); e = temp) {
...
}
![Page 27: Mixing Source and Bytecode: A Case for Compilation By Normalization (OOPSLA 2008)](https://reader033.vdocuments.us/reader033/viewer/2022060120/55910c3d1a28ab95548b4739/html5/thumbnails/27.jpg)
27
Normalizing '??' !!
String property = readProperty(“x”) ?? “default”;System.out.println(property);
![Page 28: Mixing Source and Bytecode: A Case for Compilation By Normalization (OOPSLA 2008)](https://reader033.vdocuments.us/reader033/viewer/2022060120/55910c3d1a28ab95548b4739/html5/thumbnails/28.jpg)
28
Normalizing '??' !! - with Inline Bytecode
String property = `[ push `readProperty(“x”); dup; store temp; ifnull else; push `e2; goto end; else: load temp; end: ];System.out.println(property);
S
![Page 29: Mixing Source and Bytecode: A Case for Compilation By Normalization (OOPSLA 2008)](https://reader033.vdocuments.us/reader033/viewer/2022060120/55910c3d1a28ab95548b4739/html5/thumbnails/29.jpg)
29
Normalizing '??' !! - with Inline Bytecode
String property = `[ `String temp = readProperty(“x”); `if (temp == null) temp = “default”; push `temp];System.out.println(property);
S
![Page 30: Mixing Source and Bytecode: A Case for Compilation By Normalization (OOPSLA 2008)](https://reader033.vdocuments.us/reader033/viewer/2022060120/55910c3d1a28ab95548b4739/html5/thumbnails/30.jpg)
30
Normalizing '??' !! - with an EBlock
String property = {| String temp = readProperty(“x”); if (temp == null) temp = “default”;| temp|};System.out.println(property);
![Page 31: Mixing Source and Bytecode: A Case for Compilation By Normalization (OOPSLA 2008)](https://reader033.vdocuments.us/reader033/viewer/2022060120/55910c3d1a28ab95548b4739/html5/thumbnails/31.jpg)
31
{| String x_temp = e1; if (x_temp == null) x_temp = e2;| x_temp|}
desugar-coalescing:
'??' in a Normalization Rule
|[ e1 ?? e2 ]| |[
]|with x_temp := <newname> “temp”
![Page 32: Mixing Source and Bytecode: A Case for Compilation By Normalization (OOPSLA 2008)](https://reader033.vdocuments.us/reader033/viewer/2022060120/55910c3d1a28ab95548b4739/html5/thumbnails/32.jpg)
32
{| String x_temp = e1; if (x_temp == null) x_temp = e2;| x_temp|}
|[ e1 ?? e2 ]| |[
]|with x_temp := <newname> “temp”
desugar-coalescing:
Position Information InGenerated Code
Run-time:Exception in thread "main" java.lang.NullPointerException … at Generated.getVertices(Generated.java:10)
![Page 33: Mixing Source and Bytecode: A Case for Compilation By Normalization (OOPSLA 2008)](https://reader033.vdocuments.us/reader033/viewer/2022060120/55910c3d1a28ab95548b4739/html5/thumbnails/33.jpg)
33
trace (e1 ?? e2 @ <File>:<Line>:<Column>) {{| String x_temp = e1; if (x_temp == null) x_temp = e2;| x_temp|}}
|[ e1 ?? e2 ]| |[
]|with x_temp := <newname> “temp”
desugar-coalescing:
Maintaining Position Information With Source Tracing
Run-time:Exception in thread "main" java.lang.NullPointerException … at Shape.getVertices(Shape.java:10)
![Page 34: Mixing Source and Bytecode: A Case for Compilation By Normalization (OOPSLA 2008)](https://reader033.vdocuments.us/reader033/viewer/2022060120/55910c3d1a28ab95548b4739/html5/thumbnails/34.jpg)
34
Traits X
Partial/open classes X
Iterator generators X
Finite State Automata X
Expression-based Xextensions
Class-method / Statement / Expression level
Reflection
Aspect weaving X X X
![Page 35: Mixing Source and Bytecode: A Case for Compilation By Normalization (OOPSLA 2008)](https://reader033.vdocuments.us/reader033/viewer/2022060120/55910c3d1a28ab95548b4739/html5/thumbnails/35.jpg)
• Extension effort must be proportional to the benefits• Black Boxes vs. White Boxes• Don't throw away your primitives• Normalization rules
Mixing Source and Bytecode. A Case for Compilation by Normalization.Lennart C. L. Kats, Martin Bravenboer, and Eelco Visser. In OOPSLA'08.
http://www.programtransformation.org/Stratego/TheDryadCompilerhttp://www.strategoxt.org
Concluding Remarks