equivalence testing for javascript transformations
TRANSCRIPT
![Page 1: Equivalence Testing for JavaScript Transformations](https://reader034.vdocuments.us/reader034/viewer/2022042707/5868b8171a28ab29668b492d/html5/thumbnails/1.jpg)
Equivalence Testing for JavaScriptTransformations
Johannes Kloos, Rupak Majumdar, Frank McCabe
October 23, 2015
![Page 2: Equivalence Testing for JavaScript Transformations](https://reader034.vdocuments.us/reader034/viewer/2022042707/5868b8171a28ab29668b492d/html5/thumbnails/2.jpg)
Reducing JavaScript file sizes
JavaScript programs often send a lot of unnecessary code.We would like to eliminate “dead code” from pages, to sendsmaller JavaScript files.
The DOLOTO tool tries to do this.
Getting the transformation right turns out to be hard. . .
![Page 3: Equivalence Testing for JavaScript Transformations](https://reader034.vdocuments.us/reader034/viewer/2022042707/5868b8171a28ab29668b492d/html5/thumbnails/3.jpg)
Reducing JavaScript file sizes
JavaScript programs often send a lot of unnecessary code.We would like to eliminate “dead code” from pages, to sendsmaller JavaScript files.
The DOLOTO tool tries to do this.
Getting the transformation right turns out to be hard. . .
![Page 4: Equivalence Testing for JavaScript Transformations](https://reader034.vdocuments.us/reader034/viewer/2022042707/5868b8171a28ab29668b492d/html5/thumbnails/4.jpg)
Reducing JavaScript file sizes
JavaScript programs often send a lot of unnecessary code.We would like to eliminate “dead code” from pages, to sendsmaller JavaScript files.
The DOLOTO tool tries to do this.
Getting the transformation right turns out to be hard. . .
![Page 5: Equivalence Testing for JavaScript Transformations](https://reader034.vdocuments.us/reader034/viewer/2022042707/5868b8171a28ab29668b492d/html5/thumbnails/5.jpg)
How the transformation works
The transformation replaces functions that are not usedimmediately with “stub code” that loads the actualimplementation from the network.
function func(arg) {
... /* lots and lots of code */
}
becomes (roughly)
function func_AZ123(arg) {
/* get(’x’) loads body of x as string */
func_AZ123 = eval(get(’func’));
return func_AZ123(arguments);
}
function func(arg) {
return func_AZ123(arguments);
}
![Page 6: Equivalence Testing for JavaScript Transformations](https://reader034.vdocuments.us/reader034/viewer/2022042707/5868b8171a28ab29668b492d/html5/thumbnails/6.jpg)
How the transformation works
The transformation replaces functions that are not usedimmediately with “stub code” that loads the actualimplementation from the network.
function func(arg) {
... /* lots and lots of code */
}
becomes (roughly)
function func_AZ123(arg) {
/* get(’x’) loads body of x as string */
func_AZ123 = eval(get(’func’));
return func_AZ123(arguments);
}
function func(arg) {
return func_AZ123(arguments);
}
![Page 7: Equivalence Testing for JavaScript Transformations](https://reader034.vdocuments.us/reader034/viewer/2022042707/5868b8171a28ab29668b492d/html5/thumbnails/7.jpg)
What went wrong?
Excerpt of code from the site:
c = /xyz/.test(function () { xyz }) ?
/\ b_super\b/ : /.*/
Before running the transformation, this is equivalent to:
c = /\ b_super\b/
But after running the transformation, we get:
c = /.*/
Note that /xyz/.test(s) implicitely transforms s to a string!
![Page 8: Equivalence Testing for JavaScript Transformations](https://reader034.vdocuments.us/reader034/viewer/2022042707/5868b8171a28ab29668b492d/html5/thumbnails/8.jpg)
JavaScript challenges
This work is based on a commercial implementation of thetransformation.
It has to deal with:
I Correct variable scopes
I toString shenanigans
I Handling of anonymous and nested functions
I . . .
Are we doing it right?Have we even covered everything?
![Page 9: Equivalence Testing for JavaScript Transformations](https://reader034.vdocuments.us/reader034/viewer/2022042707/5868b8171a28ab29668b492d/html5/thumbnails/9.jpg)
JavaScript challenges
This work is based on a commercial implementation of thetransformation.
It has to deal with:
I Correct variable scopes
I toString shenanigans
I Handling of anonymous and nested functions
I . . .
Are we doing it right?
Have we even covered everything?
![Page 10: Equivalence Testing for JavaScript Transformations](https://reader034.vdocuments.us/reader034/viewer/2022042707/5868b8171a28ab29668b492d/html5/thumbnails/10.jpg)
JavaScript challenges
This work is based on a commercial implementation of thetransformation.
It has to deal with:
I Correct variable scopes
I toString shenanigans
I Handling of anonymous and nested functions
I . . .
Are we doing it right?Have we even covered everything?
![Page 11: Equivalence Testing for JavaScript Transformations](https://reader034.vdocuments.us/reader034/viewer/2022042707/5868b8171a28ab29668b492d/html5/thumbnails/11.jpg)
Verifying the transformation – the problem
We want to make sure that the transformation changes pages insuch a way that the change is invisible to the user.Questions:
I What does “invisible to the user” actually mean?
I How do we verify that?
One possible approach:“Just run the interpreter test suite” – we tried that.Not sufficient, doesn’t catch the toString problem
![Page 12: Equivalence Testing for JavaScript Transformations](https://reader034.vdocuments.us/reader034/viewer/2022042707/5868b8171a28ab29668b492d/html5/thumbnails/12.jpg)
Verifying the transformation – the problem
We want to make sure that the transformation changes pages insuch a way that the change is invisible to the user.Questions:
I What does “invisible to the user” actually mean?More on this later!
I How do we verify that?
One possible approach:“Just run the interpreter test suite” – we tried that.Not sufficient, doesn’t catch the toString problem
![Page 13: Equivalence Testing for JavaScript Transformations](https://reader034.vdocuments.us/reader034/viewer/2022042707/5868b8171a28ab29668b492d/html5/thumbnails/13.jpg)
Verifying the transformation – the problem
We want to make sure that the transformation changes pages insuch a way that the change is invisible to the user.Questions:
I What does “invisible to the user” actually mean?More on this later!
I How do we verify that?One possible approach:“Just run the interpreter test suite” – we tried that.Not sufficient, doesn’t catch the toString problem
![Page 14: Equivalence Testing for JavaScript Transformations](https://reader034.vdocuments.us/reader034/viewer/2022042707/5868b8171a28ab29668b492d/html5/thumbnails/14.jpg)
Another approach: Formal verificationIdea: Use formal, mathematical proof that the transformation isalways correct, like in CompCert.
JavaScript semantics make this extremely hard; consider eval,Function.toString, the Function constructor, . . . .
![Page 15: Equivalence Testing for JavaScript Transformations](https://reader034.vdocuments.us/reader034/viewer/2022042707/5868b8171a28ab29668b492d/html5/thumbnails/15.jpg)
Another approach: Formal verificationIdea: Use formal, mathematical proof that the transformation isalways correct, like in CompCert.JavaScript semantics make this extremely hard; consider eval,Function.toString, the Function constructor, . . . .
![Page 16: Equivalence Testing for JavaScript Transformations](https://reader034.vdocuments.us/reader034/viewer/2022042707/5868b8171a28ab29668b492d/html5/thumbnails/16.jpg)
Another approach: Formal verificationIdea: Use formal, mathematical proof that the transformation isalways correct, like in CompCert.JavaScript semantics make this extremely hard; consider eval,Function.toString, the Function constructor, . . . .
![Page 17: Equivalence Testing for JavaScript Transformations](https://reader034.vdocuments.us/reader034/viewer/2022042707/5868b8171a28ab29668b492d/html5/thumbnails/17.jpg)
Another idea: CSmith
The question that CSmith attempts to answer: How well doexisiting C compilers follow the language specification?Idea:
1. Generate many small test programs (with only clearly definedbehavior)
2. Compile each program with multiple compilers.
3. Run all programs. For each program declare the mostcommon output as “correct”; all other versions are consideredmis-compiled.
Results: Many bona-fide bugs were found in all tested C compilers;false positives not a problem.Question: Can we apply a similar testing approach?
![Page 18: Equivalence Testing for JavaScript Transformations](https://reader034.vdocuments.us/reader034/viewer/2022042707/5868b8171a28ab29668b492d/html5/thumbnails/18.jpg)
Another idea: CSmith
The question that CSmith attempts to answer: How well doexisiting C compilers follow the language specification?Idea:
1. Generate many small test programs (with only clearly definedbehavior)
2. Compile each program with multiple compilers.
3. Run all programs. For each program declare the mostcommon output as “correct”; all other versions are consideredmis-compiled.
Results: Many bona-fide bugs were found in all tested C compilers;false positives not a problem.Question: Can we apply a similar testing approach?
![Page 19: Equivalence Testing for JavaScript Transformations](https://reader034.vdocuments.us/reader034/viewer/2022042707/5868b8171a28ab29668b492d/html5/thumbnails/19.jpg)
Another idea: CSmith
The question that CSmith attempts to answer: How well doexisiting C compilers follow the language specification?Idea:
1. Generate many small test programs (with only clearly definedbehavior)
2. Compile each program with multiple compilers.
3. Run all programs. For each program declare the mostcommon output as “correct”; all other versions are consideredmis-compiled.
Results: Many bona-fide bugs were found in all tested C compilers;false positives not a problem.Question: Can we apply a similar testing approach?
![Page 20: Equivalence Testing for JavaScript Transformations](https://reader034.vdocuments.us/reader034/viewer/2022042707/5868b8171a28ab29668b492d/html5/thumbnails/20.jpg)
Another idea: CSmith
The question that CSmith attempts to answer: How well doexisiting C compilers follow the language specification?Idea:
1. Generate many small test programs (with only clearly definedbehavior)
2. Compile each program with multiple compilers.
3. Run all programs. For each program declare the mostcommon output as “correct”; all other versions are consideredmis-compiled.
Results: Many bona-fide bugs were found in all tested C compilers;false positives not a problem.Question: Can we apply a similar testing approach?
![Page 21: Equivalence Testing for JavaScript Transformations](https://reader034.vdocuments.us/reader034/viewer/2022042707/5868b8171a28ab29668b492d/html5/thumbnails/21.jpg)
Another idea: CSmith
The question that CSmith attempts to answer: How well doexisiting C compilers follow the language specification?Idea:
1. Generate many small test programs (with only clearly definedbehavior)
2. Compile each program with multiple compilers.
3. Run all programs. For each program declare the mostcommon output as “correct”; all other versions are consideredmis-compiled.
Results: Many bona-fide bugs were found in all tested C compilers;false positives not a problem.
Question: Can we apply a similar testing approach?
![Page 22: Equivalence Testing for JavaScript Transformations](https://reader034.vdocuments.us/reader034/viewer/2022042707/5868b8171a28ab29668b492d/html5/thumbnails/22.jpg)
Another idea: CSmith
The question that CSmith attempts to answer: How well doexisiting C compilers follow the language specification?Idea:
1. Generate many small test programs (with only clearly definedbehavior)
2. Compile each program with multiple compilers.
3. Run all programs. For each program declare the mostcommon output as “correct”; all other versions are consideredmis-compiled.
Results: Many bona-fide bugs were found in all tested C compilers;false positives not a problem.Question: Can we apply a similar testing approach?
![Page 23: Equivalence Testing for JavaScript Transformations](https://reader034.vdocuments.us/reader034/viewer/2022042707/5868b8171a28ab29668b492d/html5/thumbnails/23.jpg)
Our chosen approach: Model-based testing
Based on CSmith.
Adapted approach:
1. Generate random JavaScript programs (“unmodifiedprograms”)So far, we use existing test suites. Some pitfalls mentionedlater.
2. Run unmodified programs through the transformationThis gives “transformed programs”.
3. Compare results of running unmodifed and transformedprograms.The rest of the talk focuses on this!
![Page 24: Equivalence Testing for JavaScript Transformations](https://reader034.vdocuments.us/reader034/viewer/2022042707/5868b8171a28ab29668b492d/html5/thumbnails/24.jpg)
Our chosen approach: Model-based testing
Based on CSmith.Adapted approach:
1. Generate random JavaScript programs (“unmodifiedprograms”)
So far, we use existing test suites. Some pitfalls mentionedlater.
2. Run unmodified programs through the transformationThis gives “transformed programs”.
3. Compare results of running unmodifed and transformedprograms.The rest of the talk focuses on this!
![Page 25: Equivalence Testing for JavaScript Transformations](https://reader034.vdocuments.us/reader034/viewer/2022042707/5868b8171a28ab29668b492d/html5/thumbnails/25.jpg)
Our chosen approach: Model-based testing
Based on CSmith.Adapted approach:
1. Generate random JavaScript programs (“unmodifiedprograms”)So far, we use existing test suites. Some pitfalls mentionedlater.
2. Run unmodified programs through the transformationThis gives “transformed programs”.
3. Compare results of running unmodifed and transformedprograms.The rest of the talk focuses on this!
![Page 26: Equivalence Testing for JavaScript Transformations](https://reader034.vdocuments.us/reader034/viewer/2022042707/5868b8171a28ab29668b492d/html5/thumbnails/26.jpg)
Our chosen approach: Model-based testing
Based on CSmith.Adapted approach:
1. Generate random JavaScript programs (“unmodifiedprograms”)So far, we use existing test suites. Some pitfalls mentionedlater.
2. Run unmodified programs through the transformationThis gives “transformed programs”.
3. Compare results of running unmodifed and transformedprograms.The rest of the talk focuses on this!
![Page 27: Equivalence Testing for JavaScript Transformations](https://reader034.vdocuments.us/reader034/viewer/2022042707/5868b8171a28ab29668b492d/html5/thumbnails/27.jpg)
Our chosen approach: Model-based testing
Based on CSmith.Adapted approach:
1. Generate random JavaScript programs (“unmodifiedprograms”)So far, we use existing test suites. Some pitfalls mentionedlater.
2. Run unmodified programs through the transformationThis gives “transformed programs”.
3. Compare results of running unmodifed and transformedprograms.The rest of the talk focuses on this!
![Page 28: Equivalence Testing for JavaScript Transformations](https://reader034.vdocuments.us/reader034/viewer/2022042707/5868b8171a28ab29668b492d/html5/thumbnails/28.jpg)
Outline
I What are valid test cases?
I Overview of the result comparison
I Experimental results
![Page 29: Equivalence Testing for JavaScript Transformations](https://reader034.vdocuments.us/reader034/viewer/2022042707/5868b8171a28ab29668b492d/html5/thumbnails/29.jpg)
Outline
I What are valid test cases?
I Overview of the result comparison
I Experimental results
![Page 30: Equivalence Testing for JavaScript Transformations](https://reader034.vdocuments.us/reader034/viewer/2022042707/5868b8171a28ab29668b492d/html5/thumbnails/30.jpg)
Not every JavaScript program is a test case!
// Array containing expected property names of the
// window object , i.e. the names described in the
// ECMAScript standard and by the DOM.
var exp = new Array (...);
// The names that are actually contained.
var real = Object.getOwnPropertyNames(window);
// Names that were not expected
var extra = exp.filter(function (name) {
return Array.prototype.indexOf(real , name) == -1;
});
console.log("Letter -of -the -law compliant: " +
(extra.length = 0));
This code would break when transformed!Reason: This code performs very heavy introspection.
![Page 31: Equivalence Testing for JavaScript Transformations](https://reader034.vdocuments.us/reader034/viewer/2022042707/5868b8171a28ab29668b492d/html5/thumbnails/31.jpg)
Not every JavaScript program is a test case!
// Array containing expected property names of the
// window object , i.e. the names described in the
// ECMAScript standard and by the DOM.
var exp = new Array (...);
// The names that are actually contained.
var real = Object.getOwnPropertyNames(window);
// Names that were not expected
var extra = exp.filter(function (name) {
return Array.prototype.indexOf(real , name) == -1;
});
console.log("Letter -of -the -law compliant: " +
(extra.length = 0));
This code would break when transformed!
Reason: This code performs very heavy introspection.
![Page 32: Equivalence Testing for JavaScript Transformations](https://reader034.vdocuments.us/reader034/viewer/2022042707/5868b8171a28ab29668b492d/html5/thumbnails/32.jpg)
Not every JavaScript program is a test case!
// Array containing expected property names of the
// window object , i.e. the names described in the
// ECMAScript standard and by the DOM.
var exp = new Array (...);
// The names that are actually contained.
var real = Object.getOwnPropertyNames(window);
// Names that were not expected
var extra = exp.filter(function (name) {
return Array.prototype.indexOf(real , name) == -1;
});
console.log("Letter -of -the -law compliant: " +
(extra.length = 0));
This code would break when transformed!Reason: This code performs very heavy introspection.
![Page 33: Equivalence Testing for JavaScript Transformations](https://reader034.vdocuments.us/reader034/viewer/2022042707/5868b8171a28ab29668b492d/html5/thumbnails/33.jpg)
Introspection in JavaScript
Introspection is widespread:
I Feature checks:
if (Array.prototype.filter == null) { /* apply
polyfill */ }
I Object-oriented hacks using Function.toString:
if (/super /.test(f.toString)) { /* provide
super argument */ }
I for..in
I Object.getOwnProperty and the like
![Page 34: Equivalence Testing for JavaScript Transformations](https://reader034.vdocuments.us/reader034/viewer/2022042707/5868b8171a28ab29668b492d/html5/thumbnails/34.jpg)
Introspection in JavaScript
Introspection is widespread:
I Feature checks:
if (Array.prototype.filter == null) { /* apply
polyfill */ }
I Object-oriented hacks using Function.toString:
if (/super /.test(f.toString)) { /* provide
super argument */ }
I for..in
I Object.getOwnProperty and the like
![Page 35: Equivalence Testing for JavaScript Transformations](https://reader034.vdocuments.us/reader034/viewer/2022042707/5868b8171a28ab29668b492d/html5/thumbnails/35.jpg)
Introspection in JavaScript
Introspection is widespread:
I Feature checks:
if (Array.prototype.filter == null) { /* apply
polyfill */ }
I Object-oriented hacks using Function.toString:
if (/super /.test(f.toString)) { /* provide
super argument */ }
I for..in
I Object.getOwnProperty and the like
![Page 36: Equivalence Testing for JavaScript Transformations](https://reader034.vdocuments.us/reader034/viewer/2022042707/5868b8171a28ab29668b492d/html5/thumbnails/36.jpg)
The open world assumption
In a web application, many scripts and libraries must co-exist. Thisis an open world: Things can be added to the environment withoutour knowledge.
Therefore, a script must not make assumptions about parts of theenvironment that it does not control.
In particular, only tame introspection: Only apply introspection toobjects it controls (except some feature checks).
![Page 37: Equivalence Testing for JavaScript Transformations](https://reader034.vdocuments.us/reader034/viewer/2022042707/5868b8171a28ab29668b492d/html5/thumbnails/37.jpg)
The open world assumption
In a web application, many scripts and libraries must co-exist. Thisis an open world: Things can be added to the environment withoutour knowledge.
Therefore, a script must not make assumptions about parts of theenvironment that it does not control.
In particular, only tame introspection: Only apply introspection toobjects it controls (except some feature checks).
![Page 38: Equivalence Testing for JavaScript Transformations](https://reader034.vdocuments.us/reader034/viewer/2022042707/5868b8171a28ab29668b492d/html5/thumbnails/38.jpg)
The open world assumption
In a web application, many scripts and libraries must co-exist. Thisis an open world: Things can be added to the environment withoutour knowledge.
Therefore, a script must not make assumptions about parts of theenvironment that it does not control.
In particular, only tame introspection: Only apply introspection toobjects it controls (except some feature checks).
![Page 39: Equivalence Testing for JavaScript Transformations](https://reader034.vdocuments.us/reader034/viewer/2022042707/5868b8171a28ab29668b492d/html5/thumbnails/39.jpg)
Outline
I What are valid test cases?
I Overview of the result comparison
I Experimental resuls
![Page 40: Equivalence Testing for JavaScript Transformations](https://reader034.vdocuments.us/reader034/viewer/2022042707/5868b8171a28ab29668b492d/html5/thumbnails/40.jpg)
The process in a picture
function () { blah;}
function f() { blah;}
function () { return f();}
Test case generation Transform with JSStreaming
Execute unmodifedand transformedtest cases,generating traces
Compare traces
Execute unmodifedand transformedtest cases,generating traces
![Page 41: Equivalence Testing for JavaScript Transformations](https://reader034.vdocuments.us/reader034/viewer/2022042707/5868b8171a28ab29668b492d/html5/thumbnails/41.jpg)
What is a trace?
A trace is a sequence of events that describe the execution of aprogram.Some of the events we consider:
Read(ref , val) Read value val from ref
Write(ref , val) Write value val to ref
FunPre(f , this, args) Call function f on thiswith arguments args
We also track variable declarations, exception handling, expressionevaluation, . . . .
![Page 42: Equivalence Testing for JavaScript Transformations](https://reader034.vdocuments.us/reader034/viewer/2022042707/5868b8171a28ab29668b492d/html5/thumbnails/42.jpg)
What is a trace?
A trace is a sequence of events that describe the execution of aprogram.Some of the events we consider:
Read(ref , val) Read value val from ref
Write(ref , val) Write value val to ref
FunPre(f , this, args) Call function f on thiswith arguments args
We also track variable declarations, exception handling, expressionevaluation, . . . .
![Page 43: Equivalence Testing for JavaScript Transformations](https://reader034.vdocuments.us/reader034/viewer/2022042707/5868b8171a28ab29668b492d/html5/thumbnails/43.jpg)
An example of a trace
console.log("x")
givesRead(console, obj:42)Read(obj:42.log, func:23)Literal(”x”)FunPre(func:23, obj:42, obj:105) // obj:105 looks like ["x"]
. . .
![Page 44: Equivalence Testing for JavaScript Transformations](https://reader034.vdocuments.us/reader034/viewer/2022042707/5868b8171a28ab29668b492d/html5/thumbnails/44.jpg)
A simple equivalence criterion
Declare some events as externally visible:
I Calls to external functions.A function is external if it is implemented using native code.
I Writes to global variables and objects.A variable/object is global if it is not declared/created insidethe script.
Then two traces can be defined as equivalent when thesubsequences consisting of global events are the same.
![Page 45: Equivalence Testing for JavaScript Transformations](https://reader034.vdocuments.us/reader034/viewer/2022042707/5868b8171a28ab29668b492d/html5/thumbnails/45.jpg)
An observation on trace structures
Original trace Transformed trace
e1 e1
init1
init2e2 e2
init3e3 e3
call call
enterwrap1
call
enter entere4 e4e5 e5
exit exit
return
exit
return returne6 e6
![Page 46: Equivalence Testing for JavaScript Transformations](https://reader034.vdocuments.us/reader034/viewer/2022042707/5868b8171a28ab29668b492d/html5/thumbnails/46.jpg)
Our final equivalence criterion
Basically, two traces A and B are equivalent (for thetransformation) if A is a sub-trace of B “in a nice way”.
More formally: Two traces A and B are equivalent (for thetransformation) if:
I B can be divided into subsequences consisting of pairedevents, initialization events and wrapper events.
I The events in A correspond to the subsequence of pairedevents in B1.
I Wrapper and initialization events allow only events that arenot externally visible.
I Restrictions on how to split trace B: See paper.
1Up to renaming of memory cells and the like
![Page 47: Equivalence Testing for JavaScript Transformations](https://reader034.vdocuments.us/reader034/viewer/2022042707/5868b8171a28ab29668b492d/html5/thumbnails/47.jpg)
Sketching the main stepsThe analysis consists of three main steps:
function () { blah;}
function f() { blah;}
function () { return f();}
Test case generation Transform with JSStreaming
Execute unmodifedand transformedtest cases,generating traces
Compare enriched traces
Execute unmodifedand transformedtest cases,generating traces
Enrich traces
![Page 48: Equivalence Testing for JavaScript Transformations](https://reader034.vdocuments.us/reader034/viewer/2022042707/5868b8171a28ab29668b492d/html5/thumbnails/48.jpg)
Sketching the main stepsThe analysis consists of three main steps:
function () { blah;}
function f() { blah;}
function () { return f();}
Test case generation Transform with JSStreaming
Execute unmodifedand transformedtest cases,generating traces
Compare enriched traces
Execute unmodifedand transformedtest cases,generating traces
Enrich traces
DynamicAnalysis
(Jalangi2)
![Page 49: Equivalence Testing for JavaScript Transformations](https://reader034.vdocuments.us/reader034/viewer/2022042707/5868b8171a28ab29668b492d/html5/thumbnails/49.jpg)
Sketching the main stepsThe analysis consists of three main steps:
function () { blah;}
function f() { blah;}
function () { return f();}
Test case generation Transform with JSStreaming
Execute unmodifedand transformedtest cases,generating traces
Compare enriched traces
Execute unmodifedand transformedtest cases,generating traces
Enrich traces Resolve aliasing,Reconstruct heap
![Page 50: Equivalence Testing for JavaScript Transformations](https://reader034.vdocuments.us/reader034/viewer/2022042707/5868b8171a28ab29668b492d/html5/thumbnails/50.jpg)
Sketching the main stepsThe analysis consists of three main steps:
function () { blah;}
function f() { blah;}
function () { return f();}
Test case generation Transform with JSStreaming
Execute unmodifedand transformedtest cases,generating traces
Compare enriched traces
Execute unmodifedand transformedtest cases,generating traces
Enrich traces
DFS-stylebacktracking
with heuristics
![Page 51: Equivalence Testing for JavaScript Transformations](https://reader034.vdocuments.us/reader034/viewer/2022042707/5868b8171a28ab29668b492d/html5/thumbnails/51.jpg)
Outline
I What are valid test cases?
I Overview of the result comparison
I Experimental results
![Page 52: Equivalence Testing for JavaScript Transformations](https://reader034.vdocuments.us/reader034/viewer/2022042707/5868b8171a28ab29668b492d/html5/thumbnails/52.jpg)
Trying it out
First round of experimental results: Run on test cases from theECMA262 test suite.
ECMA262 test suite
“Good” test cases
“Mediocre” test cases
“Clean” traces
“Good” traces
“Mediocre” traces
“Good” result set
“Mediocre” result set
New transformer
Old transformer
Trace generation
Trace generation
Trace generation
Matching
Matching
Two questions:
1. How stable is testing? How many false positives/falsenegatives do we get?
2. How well does error detection work?
![Page 53: Equivalence Testing for JavaScript Transformations](https://reader034.vdocuments.us/reader034/viewer/2022042707/5868b8171a28ab29668b492d/html5/thumbnails/53.jpg)
Stability
Matching foundMatching not found
In total: 4972 test cases; 3883 found matches.
Observations: No false positives found (and a sample of traceslooks fine), but the rate of potential false negatives is still prettyhigh.Further analysis: centered around handling of higher-orderfunctions (“chapter 15”) and introspection/reflection.
![Page 54: Equivalence Testing for JavaScript Transformations](https://reader034.vdocuments.us/reader034/viewer/2022042707/5868b8171a28ab29668b492d/html5/thumbnails/54.jpg)
Stability
Matching foundMatching not found
In total: 4972 test cases; 3883 found matches.Observations: No false positives found (and a sample of traceslooks fine), but the rate of potential false negatives is still prettyhigh.
Further analysis: centered around handling of higher-orderfunctions (“chapter 15”) and introspection/reflection.
![Page 55: Equivalence Testing for JavaScript Transformations](https://reader034.vdocuments.us/reader034/viewer/2022042707/5868b8171a28ab29668b492d/html5/thumbnails/55.jpg)
Stability
Matching foundMatching not found
In total: 4972 test cases; 3883 found matches.Observations: No false positives found (and a sample of traceslooks fine), but the rate of potential false negatives is still prettyhigh.Further analysis: centered around handling of higher-orderfunctions (“chapter 15”) and introspection/reflection.
![Page 56: Equivalence Testing for JavaScript Transformations](https://reader034.vdocuments.us/reader034/viewer/2022042707/5868b8171a28ab29668b492d/html5/thumbnails/56.jpg)
Error detection
All Without 150
500
1000
1500
2000
2500
3000
3500
Bug finding power
Both pos.
Neg → Pos
Both neg.
Pos → Neg
The additional non-matches (red bars) point to actual errors!
![Page 57: Equivalence Testing for JavaScript Transformations](https://reader034.vdocuments.us/reader034/viewer/2022042707/5868b8171a28ab29668b492d/html5/thumbnails/57.jpg)
Summary
We are working on verifying a JavaScript source-to-sourcetransformation through systematic testing.The test approach is different from traditional testing: Wedecouple the decision whether a test case has passed from the testinput.This approach seems to be working reasonably well for for ourtransformation; can it be applied for othertransformations/applications?
![Page 58: Equivalence Testing for JavaScript Transformations](https://reader034.vdocuments.us/reader034/viewer/2022042707/5868b8171a28ab29668b492d/html5/thumbnails/58.jpg)
How we do trace collection
Jalangi2: Dynamic analysis framework for JavaScript. It Works byinstrumenting JavaScript code.
We simply dump all events. Some special handling to give objectsa unique identifier.
Outputs a JSON file containing:
I The actual trace.
I The initial state of each object.
I The body of all functions, as string (if internal) or with uniqueID (if external).
I Information about global objects.
![Page 59: Equivalence Testing for JavaScript Transformations](https://reader034.vdocuments.us/reader034/viewer/2022042707/5868b8171a28ab29668b492d/html5/thumbnails/59.jpg)
What a refined trace file contains
Refined trace files are very similar to traces from trace collection.Main differences:
I The trace is modified to resolve all aliasing in reads and writes.
I Variable shadowing is eliminated using a SSA-like transform.
I Information is introduced to provide a snapshot of the heapafter each step.
![Page 60: Equivalence Testing for JavaScript Transformations](https://reader034.vdocuments.us/reader034/viewer/2022042707/5868b8171a28ab29668b492d/html5/thumbnails/60.jpg)
An example of an obstacle
function f(x,y) {
x=42;
y=42;
console.log(arguments [0] + "," + arguments [1]);
}
f(1,2);
f(1);
Output:
42, 42
42, undefined
![Page 61: Equivalence Testing for JavaScript Transformations](https://reader034.vdocuments.us/reader034/viewer/2022042707/5868b8171a28ab29668b492d/html5/thumbnails/61.jpg)
An example of an obstacle
function f(x,y) {
x=42;
y=42;
console.log(arguments [0] + "," + arguments [1]);
}
f(1,2);
f(1);
Output:
42, 42
42, undefined
![Page 62: Equivalence Testing for JavaScript Transformations](https://reader034.vdocuments.us/reader034/viewer/2022042707/5868b8171a28ab29668b492d/html5/thumbnails/62.jpg)
How we match traces
Basically, split the transformed trace using a backtrackingalgorithm.On the trace level, various heuristics for fast search.The algorithm has to match objects; this uses the heap informationfrom trace enrichment.