Unit Testing for Domain-Specific Languages

1Hui Wu, 1Jeff Gray and 2Marjan Mernik

1University of Alabama at Birmingham, USA{wuh,gray}@cis.uab.edu

2University of Maribor, Sloveniamarjan.mernik@uni-mb.si

DSL 2009 Oxford, UK

July 15, 2009

Programming Methodologies Laboratory

Emperors New DSLs

• As we show off our new DSLs and approaches for creating them, are there some obvious limitations that we may be ignoring?

2

But where are the tools I can use?

OOPSLA 2008 DSL Panel:The Good, the Bad, and the Ugly

• Four of the five participants listed lack of tool support as a top challenge for DSLs– Utility of a new DSL is severely diminished if

supporting tools needed by a software developer and/or end-user are not available

– Poor interoperability and reuse of such tools 3

44

What Kinds of Tools are Needed?

Domain Experts program at DSL level

DSL translated into General Purpose Language (GPL)

Domain Experts deal with DSL

Integrated Development Environment (IDE)

subsel ect mebegin l eft ri ght up downend

Transl ater

subsel ect mepubl i c cl ass Robot{ publ i c stati c voidmain(Stri ng[] args) { Robot robot =new Robot(0,0,0); / /move l eft robot.move_l eft(); / /move down robot.move_down();robot.x = 5; robot.y = 6;

EditorEditor

CompilerCompiler

Visualizer

Visualizer

DebuggerDebugger

Domain Experts deal with GPL

Test EngineTest Engine

ProfilerProfiler

55

Mismatch Between Abstraction Levels

….commands : ( c:command cs:commands | );command : ( RIGHT { fileio.print("//move right"); fileio.print("x=x+1;"); fileio.print("time=time+1;"); fileio.print(" "); } |LEFT { fileio.print("//move left"); fileio.print("x=x-1;"); fileio.print("time=time+1;"); fileio.print(" ");…

….public final void commands() throws RecognitionException, TokenStreamException {

try { // for error handling {

switch ( LA(1)) { case CALL: case INIT: case SET: case PRINT: { command(); commands(); break; } case END: { break; } default: { throw new NoViableAltException(LT(1), getFilename());

…public final void function_name() throws RecognitionException, TokenStreamException { try { // for error handling { switch ( LA(1)) { case RIGHT: { match(RIGHT); fileio.print(" //move right"); fileio.print(" move_right();"); fileio.print(" "); break; } case LEFT: { match(LEFT); sllinenumber=dsllinenumber+1; fileio.print(" //move left"); fileio.print(" move_left();"); fileio.print(" "); break; }…

ANTLR GrammarGenerated Java Parser Code

Why is basic DSL tool support lacking?

• Why are such tools scarce?– Building new DSL tools from scratch for each new DSL can be

time consuming, error prone and costly– Tooling implementation effort generally not reusable for a

different context (even in same domain)6

• Example– ANTLRStudio

http://www.placidsystems.com/antlrstudio.aspx

– Fixed tool support customized for specific DSL

Idea of this paper

• Introduce the idea of a framework for building DSL tools that reuse existing GPL tools

• Assumptions/Limitations: – Only applicable to source-to-source translation (DSL

reuses GPL compiler and other tools of GPL context)– DSL tool implementation dependent on available GPL

mappings; requires adaptation to existing grammars – Does not consider case when one DSL statement is

mapped to non-contiguous GPL code (e.g., an aspect language)

– Paper uses simple examples for illustration; scalability of approach not certain

7

88

DSL to GPL Source Mapping…

3 knight:

4 position(+0,+1);

5 position(+0,+1);

6 position(+1,+0);

7 knight:

8 …

9 Init position(0,0);

10 left;

11 down;

12 knight;

13 Set position(5,6);

14 up;

15 right;

16 Print position;

…

…

6 public static void move_knight(){

7 x=x+0;

8 y=y+1;

9 x=x+0;

10 y=y+1;

11 x=x+1;

12 y=y+0;}

13 public static void main(String[] args) {

14 x=0;

15 y=0;

…

18 move_knight();

…

20 x = 5;

21 y = 6;

…

26 System.out.println("x coordinate="+x+""+

27 "y coordinate= " + y);}

…

{13, "Robot.java", 20, 21, "main", "none"}

99

Reusing GPL Tools in DSL Context

Re-interpreter GPL Tool

Source CodeMapping

Tool Results Mapping

Tool Methods Mapping

ANTLR Translator

GPL

DSL Tool Actions

DSL Variable View

DSL Level

GPL Level

End-User

GPL Tool Commands

DSL

Background Context: DSL Debuggers

• Our initial work for DSL tool support focused on debuggers

• The DSL Debugging Framework (DDF) provides a context for understanding the source mapping necessary to reuse GPL tools in a DSL context

10

1111

Debugging Methods Mapping

Mapping DSL GPL

Source Coden_i maps to m_i to m_j

Line Number:n_1n_2n_...n_in_i+1n_...n_jn_j+1n_...

Line Number: m_1m_2m_...m_im_i+1m_...m_jm_j+1m_...

Breakpoint Set breakpoint at n_i Set breakpoint at m_i

Step Over Step over line at n_i Step Over algorithm

Step Into Step into line at n_i Step Into algorithm

Terminate Terminate at line n_i Terminate at line m_i

Resume Resume at line n_i Resume at line m_i

1212

Debugging Results Mapping:Tangled Concerns within Grammar

1 | INIT var:VARIABLES LPAREN init_num1:NUMBER COMMA init_num2:NUMBER RPAREN

2 {

3 dsllinenumber=dsllinenumber+1;

4 fileio.print("x="+init_num1.getText()+";");

5 fileresult.print("x_coordinate=print x");

6 gplbeginline=fileio.getLinenumber();

7 fileio.print("y="+init_num2.getText()+";");

8 fileresult.print("y_coordinate=print y");

9 fileio.print("time=0"+";");

10 gplendline=fileio.getLinenumber();

11 filemap.print("mapping.add(new Map("+dsllinenumber+",\"Robot.java\","+

12 gplbeginline +","+gplendline+","+"\""+funcname+"\""+","+"\""+funcall+"\""+"));");

13 fileresult.print(variable=var.getText()+"(x_coordinate,y_coordinate)");

14 }

Black: Basic functionalityBlue: Source code mappingRed: Debugging results mapping

1313

Imperative DSL Debugger

User has access to traditional debugging

operations

Variables defined in terms of DSL, not GPL

1414

Declarative DSL Debugger (FDL)

1 Car : all (Carbody, Transmission, Engine, Horsepower, opt(pullsTrailer))2 Transmission : oneof (automatic, manual)3 Engine : moreof (electric, gasoline)4 Horsepower : oneof (lowPower, mediumPower, highPower)5 include pullsTrailer6 pullsTrailer requires highPower

1515

Declarative DSL Debugger (BNF)

1616

Hybrid Robot Debugger (Java within DSL)

1717

Hybrid SWUL Debugger (DSL within Java)

18

Current Focus: DSL Testing

• Before locating software errors how do we know there are bugs inside a DSL program?

1919

• Complement to the DSL Debug Framework (DDF) – the DUTF assists in identifying the presence of errors and the DDF assists in isolating the specific location of the error

• Architecture and process of construction is similar to the DDF architecture (Figure 1 in paper); source code mapping similar

DSL Unit Testing Framework (DUTF)

2020

DSL Unit Testing Framework (DUTF)

5

...21 Init position(0,0);22 down;23 knight;24 Set position(5,6);25 right;26 Print position; ...

public class Robot{ public static void main(String[] args) { …… //move left x=x-1; time=time+1;

//move down y=y-1; time=time+1; …… } }

import java.util.ArrayList;

public class Mapping { ArrayList mapping; public Mapping(){ mapping=new ArrayList(); mapping.add(new Map(1, "Robot.java",2,8)); mapping.add(new Map(2, "Robot.java",10,14)); …… } }

Test Results Mapping

Robot DSL Grammar Source Code Mapping

Robot DSL

Generated Lexer, and Parser by ANTLR

Robot.java and Mapping.java

Robot DSL Unit Test Perspective in Eclipse

Robot DSL Unit Test Script GrammarTest Case Mapping

...2 Init position(0,0);3 Expectedposition (1,2);4 knight;5 AssertEqual(Expectedposition, position);...

Robot DSL Unit Test Script public class TestRobot extends TestCase{ ... public void testknight() { i=1; j=2; robot.knight(); assertEquals(i, robot.x); assertEquals(j, robot.y); } ...}

TestRobot.java

Test Results View

Unit Test Script Editor

1 2

3

4

2121

Robot DSL Unit Test Case1 TestCase testknight {2 Init position(0,0);3 Expectedposition(1,2);4 knight;5 AssertEqual (Expectedposition, position);6 }…

GPL Unit Test Case (JUnit)11 public void testknight() {12 robot.x = 0;13 robot.y =0;14 int x=1;15 int y=2;16 robot.move_knight();17 assertEquals(x, robot.x);18 assertEquals(y, robot.y);19 }…

{1, “TestRobot.java”,11,“testknight”}

• A testing assertion at the DSL level may result in multiple assertions at the GPL level depending on variable mapping

DSL Unit Testing Framework (DUTF):Test Cases Mapping

2222

Car FDL Unit Test Case1 TestCase testFeatures {2 Expectedfeature:(carbody, manual, highPower);3 use Car.FDL(All);4 Constraint C1: include pullsTrailer;5 AssertTrue(contain(Expectedfeature, feature));6 AssertEqual(6, numberof feature);7 }

GPL Unit Test Case (JUnit)11 public void testFeatures () {12 testFeatures.add("carbody");13 testFeatures.add("manual");14 testFeatures.add("highPower");…27 assertTrue(compareFeatures(testFeatures,parse(fc,root,cons)));28 assertEquals(6,getFeatureListNumber(parse(fc,root,cons)));…

DSL Unit Testing Framework (DUTF):Test Cases Mapping

2323

DSL Unit Testing Framework (DUTF)Robot Language Unit Test Engine

Correct knight method

1 begin knight:

2 position (+0,+1);

3 position (+0,+1);

4 position (+1,+0);

5 end knight:

Incorrect knight method

1 begin knight:

2 position (+0,+1);

3 position (+1,+1);

4 position (+1,+0);

5 end knight:

User sees results as typical JUnit test report, but with DSL test cases

2424

DSL Unit Testing Framework (DUTF)FDL Unit Test Engine

Video Demonstration

• DUTF Testing Demo 2.1 available at

http://www.cis.uab.edu/softcom/DDF/

25

2626

Adapting Existing DSL Grammars for DSL Tool Support

• Crosscutting concerns emerge in different software artifacts (e.g., model, grammar, and source code)

• AspectG: A domain-specific aspect language for language grammars written in ANTLR– Program transformation rules are generated from a

domain-specific aspect language focused on language specification in ANTLR

– Generated rules modify grammars, not source code

2727

dsllinenumber=dsllinenumber+1;

gplbeginline=fileio.getLinenumber();

gplendline=fileio.getLinenumber();

filemap.print(" mapping.add(new Map("+dsllinenumber+",\"Robot.java\","+gplbeginline+","+gplendline+"));");

Addition of DSL Tool Support: A Crosscutting Grammar Concern

dsllinenumber=dsllinenumber+1;

gplbeginline=fileio.getLinenumber();

gplendline=fileio.getLinenumber();

filemap.print(" mapping.add(new Map("+dsllinenumber+",\"Robot.java\","+gplbeginline+","+gplendline+"));");

…command :( RIGHT {

fileio.print(" //move right");fileio.print(" x=x+1;");

fileio.print(" time=time+1;");

fileio.print(" ");

} |LEFT {

fileio.print(" //move left");fileio.print(" x=x-1;");

fileio.print(" time=time+1;");

fileio.print(" ");

}|…

Base Grammar

Duplicate Tool Aspect Code

What if this line changes?Change here

Change here

Change ……

2828

pointcut productions(): within(command.*);

pointcut count_gpllinenumber(): within(command.*) && match (fileio.print("time=time+1;"));

before(): productions() {dsllinenumber=dsllinenumber+1;}

before(): count_gpllinenumber() {gplbeginline=fileio.getLinenumber();}

after(): count_gpllinenumber() {gplendline=fileio.getLinenumber();}

after(): productions() {filemap.print(" mapping.add(new Map("+ dsllinenumber+", \"Robot.java\","+gplbeginline+ ","+gplendline+"));"); }

…command :( RIGHT { dsllinenumber=dsllinenumber+1; fileio.print("x=x+1;"); gplbeginline=fileio.getLinenumber(); fileio.print("time=time+1;"); gplendline=fileio.getLinenumber(); fileio.print(" "); filemap.print(" mapping.add(new Map("+ dsllinenumber+", \"Robot.java\","+gplbeginline+ ","+gplendline+"));"); }…

Join Point Model for AspectGANTLR Grammar AspectG

2929

Summary and Lessons Learned

• Tool development requires deep understanding of DSL, our framework, and the required mappings

• JUnit tests for equality in assertion checks may not match semantics of domain and need to be adapted

• Among 22 software components in DUTF, there are 3,001 lines of code that are generalized and can be reused to generate different DSL unit test engines

DSL Category DSL Name Number of Specific Functions or Classes

Customized Lines of Code

Imperative DSL Robot Language 2 239

Declarative DSL FDL 4 482

3030

Future Work

• An extension of the current framework that enables DSL profiling

• Adaptation of DUTF to address more complex DSLs

• Investigation into extending DUTF to other platforms– Application of different IDE platforms

(Microsoft Visual Studio .Net) and GPLs (C# and C++) using Nunit and other testing tools

3131

Questions?

Video demonstrations and papers available at:http://www.cis.uab.edu/softcom/DDF/

Acknowledgement:The work presented in this paper was supported in part by NSF CAREER grant (CCF-0643725) and the IBM Eclipse Innovation Grant (EIG).

Programming Methodologies Laboratory