autoflox: an automatic fault localizer for client-side javascript

AutoFLox: An Automatic Fault Localizer for Client-Side

JavaScript

Frolin S. Ocariza, Jr.

Karthik Pattabiraman

Ali Mesbah

University of British Columbia

Web 2.0 Applications: JavaScript JavaScript: Implementation of ECMAScript

standard Client-Side JavaScript: used to develop web apps

Executes at client’s browser Web application’s functionality Dependability problem: Average of 4 errors

per web app in the wild [ISSRE11]

2

JavaScript Dependability: Document Object Model (DOM)

3

html

body head

scriptdiv p

Text: “Hello world”

table

tr p

Element added by JavaScript

Element removed by JavaScript

JavaScript Dependability: Document Object Model (DOM)

4

div

id: elem

JavaScript code:

DOM:

var x = document.getElementById(“elem”);var x = document.getElementById(“elme”);

Inexistent IDWill return null

DOM-related JavaScript error

Fault Localization Web application testing

What to do after we find errors? Need to fix them…

Fault localization: Find the root cause of the error

5

Contributions/Outline1. Discussion of JavaScript fault localization

challenges2. Automated technique to localize DOM-

related JavaScript faults3. Empirical studies

Highlight prevalence of DOM-related JavaScript errors

Evaluate accuracy and performance of technique

6

1 function changeBanner(bannerID) {

2 clearTimeout(changeTimer);

3 changeTimer = setTimeout(changeBanner, 5000);

4

5 prefix = “banner_”;

6 currBannerElem = document.getElementById(prefix+currentBannerID);

7 bannerToChange = document.getElementById(prefix + bannerID);

8 currBannerElem.removeClassName(“active”);

9 bannerToChange.addClassName(“active”);

10 currentBannerID = bannerID;

11 }

12 currentBannerID = 1;

13 changeTimer = setTimeout(changeBanner, 5000);

Running Example

7

1 function changeBanner(bannerID) {

2 clearTimeout(changeTimer);

3 changeTimer = setTimeout(changeBanner, 5000);

4

5 prefix = “banner_”;

6 currBannerElem = document.getElementById(prefix+currentBannerID);

7 bannerToChange = document.getElementById(prefix + bannerID);

8 currBannerElem.removeClassName(“active”);

9 bannerToChange.addClassName(“active”);

10 currentBannerID = bannerID;

11 }

12 currentBannerID = 1;

13 changeTimer = setTimeout(changeBanner, 5000);

Goal: Show a fading banner that cycles through four images every 5 seconds

Passed with no argument (even though changeBanner needs one argument)

bannerID will be set randomly

Would return null if bannerID out of range

NULL EXCEPTION!setTimeout call

1 function changeBanner(bannerID) {

2 clearTimeout(changeTimer);

3 changeTimer = setTimeout(changeBanner, 5000);

4

5 prefix = “banner_”;

6 currBannerElem = document.getElementById(prefix+currentBannerID);

7 bannerToChange = document.getElementById(prefix + bannerID);

8 currBannerElem.removeClassName(“active”);

9 bannerToChange.addClassName(“active”);

10 currentBannerID = bannerID;

11 }

12 currentBannerID = 1;

13 changeTimer = setTimeout(changeBanner, 5000);

JavaScript Fault Localization: Challenges Multiple JS execution sequences executing

asynchronously

8

1 function changeBanner(bannerID) {

2 clearTimeout(changeTimer);

3 changeTimer = setTimeout(changeBanner, 5000);

4

5 prefix = “banner_”;

6 currBannerElem = document.getElementById(prefix+currentBannerID);

7 bannerToChange = document.getElementById(prefix + bannerID);

8 currBannerElem.removeClassName(“active”);

9 bannerToChange.addClassName(“active”);

10 currentBannerID = bannerID;

11 }

12 currentBannerID = 1;

13 changeTimer = setTimeout(changeBanner, 5000);

1 function changeBanner(bannerID) {

2 clearTimeout(changeTimer);

3 changeTimer = setTimeout(changeBanner, 5000);

4

5 prefix = “banner_”;

6 currBannerElem = document.getElementById(prefix+currentBannerID);

7 bannerToChange = document.getElementById(prefix + bannerID);

8 currBannerElem.removeClassName(“active”);

9 bannerToChange.addClassName(“active”);

10 currentBannerID = bannerID;

11 }

12 currentBannerID = 1;

13 changeTimer = setTimeout(changeBanner, 5000);Sequence 1

Sequence 2

1 function changeBanner(bannerID) {

2 clearTimeout(changeTimer);

3 changeTimer = setTimeout(changeBanner, 5000);

4

5 prefix = “banner_”;

6 currBannerElem = document.getElementById(prefix+currentBannerID);

7 bannerToChange = document.getElementById(prefix + bannerID);

8 currBannerElem.removeClassName(“active”);

9 bannerToChange.addClassName(“active”);

10 currentBannerID = bannerID;

11 }

12 currentBannerID = 1;

13 changeTimer = setTimeout(changeBanner, 5000);

1 function changeBanner(bannerID) {

2 clearTimeout(changeTimer);

3 changeTimer = setTimeout(changeBanner, 5000);

4

5 prefix = “banner_”;

6 currBannerElem = document.getElementById(prefix+currentBannerID);

7 bannerToChange = document.getElementById(prefix + bannerID);

8 currBannerElem.removeClassName(“active”);

9 bannerToChange.addClassName(“active”);

10 currentBannerID = bannerID;

11 }

12 currentBannerID = 1;

13 changeTimer = setTimeout(changeBanner, 5000);

JavaScript Fault Localization: Challenges (2) DOM-related JS Errors – fault can be in JS code

or DOM – find direct DOM access

9

direct DOM access

element = $(“elem”);

b = element.getAttribute(“badAttr”)

element.innerHTML = “text”;

b.value = “newValue”;

Scope of Technique Types of DOM-related JS errors

10

Code-terminating DOM-related JS errors

exception

Output DOM-related JS errorsfunction changeToBlue(elem) {

elem.style.color = “red”;

}Wrong colour change

Block Diagram

11

Instrument JS Code

Run Web Application

Generate Traces

Analyze backward slice

Extract Relevant Sequence

Partition into Sequences

Trace file

Web Application

direct DOM access

Trace Collection

Phase

Trace Analysis Phase

Trace Collection

1. Intercept JS code in web app and instrument each line

2. Run web app – instrumentation will execute for each line that executes

3. Generate a dynamic trace of all executed lines

12

- Trace contains line number, function name, and values of in-scope variables

- Error marker also in trace

Instrument JS Code

Run Web Application

Generate Traces

Trace file

Web Application1 function changeBanner(bannerID) {

2 clearTimeout(changeTimer);

3 changeTimer = setTimeout(changeBanner, 5000);

4

5 prefix = “banner_”;

6 currBannerElem = document.getElementById(prefix+currentBannerID);

7 bannerToChange = document.getElementById(prefix + bannerID);

8 currBannerElem.removeClassName(“active”);

9 bannerToChange.addClassName(“active”);

10 currentBannerID = bannerID;

11 }

12 currentBannerID = 1;

13 changeTimer = setTimeout(changeBanner, 5000);

1 function changeBanner(bannerID) {

2 clearTimeout(changeTimer); trace();

3 changeTimer = setTimeout(changeBanner, 5000); trace();

4

5 prefix = “banner_”; trace();

6 currBannerElem = document.getElementById(prefix+currentBannerID);trace();

7 bannerToChange = document.getElementById(prefix + bannerID); trace();

8 currBannerElem.removeClassName(“active”); trace();

9 bannerToChange.addClassName(“active”); trace();

10 currentBannerID = bannerID; trace();

11 }

12 currentBannerID = 1; trace();

13 changeTimer = setTimeout(changeBanner, 5000); trace();

1 function changeBanner(bannerID) {

2 clearTimeout(changeTimer); trace();

3 changeTimer = setTimeout(changeBanner, 5000); trace();

4

5 prefix = “banner_”; trace();

6 currBannerElem = document.getElementById(prefix+currentBannerID);trace();

7 bannerToChange = document.getElementById(prefix + bannerID); trace();

8 currBannerElem.removeClassName(“active”); trace();

9 bannerToChange.addClassName(“active”); trace();

10 currentBannerID = bannerID; trace();

11 }

12 currentBannerID = 1; trace();

13 changeTimer = setTimeout(changeBanner, 5000); trace();

Trace Record Prefix:

changeBanner:::4

Variables:

currentBannerID (global): 1

changeTimer (global): 2

bannerID (local): -11

prefix (local): none

currBannerElem (local): none

bannerToChange (local): none

Trace Analysis

1. Trace is divided into sequences2. Relevant sequence – execution sequence that

contains error marker3. Analyze backward slice of last variable that held

null to see where null value originated. The line where null value originated is the direct DOM access.

13

Instrument JS Code

Run Web Application

Generate Traces

Analyze backward slice

Extract Relevant Sequence

Partition into Sequences

Trace file

direct DOM access

Web Application

1 function changeBanner(bannerID) {

2 clearTimeout(changeTimer);

3 changeTimer = setTimeout(changeBanner, 5000);

4

5 prefix = “banner_”;

6 currBannerElem = document.getElementById(prefix+currentBannerID);

7 bannerToChange = document.getElementById(prefix + bannerID);

8 currBannerElem.removeClassName(“active”);

9 bannerToChange.addClassName(“active”);

10 currentBannerID = bannerID;

11 }

12 currentBannerID = 1;

13 changeTimer = setTimeout(changeBanner, 5000);

Sequences.: (1) line2 -> line3 -> line5 -> line6 -> line7 -> line8 -> line9

(2) line12 -> line13

Trace Analysis: Example1 function changeBanner(bannerID) {

2 clearTimeout(changeTimer);

3 changeTimer = setTimeout(changeBanner, 5000);

4

5 prefix = “banner_”;

6 currBannerElem = document.getElementById(prefix+currentBannerID);

7 bannerToChange = document.getElementById(prefix + bannerID);

8 currBannerElem.removeClassName(“active”);

9 bannerToChange.addClassName(“active”);

10 currentBannerID = bannerID;

11 }

12 currentBannerID = 1;

13 changeTimer = setTimeout(changeBanner, 5000);

14

Relevant Seq.: line12 -> line13 -> line2 -> line3 -> line5 -> line6 -> line7 -> line8 -> line9

1 function changeBanner(bannerID) {

2 clearTimeout(changeTimer);

3 changeTimer = setTimeout(changeBanner, 5000);

4

5 prefix = “banner_”;

6 currBannerElem = document.getElementById(prefix+currentBannerID);

7 bannerToChange = document.getElementById(prefix + bannerID);

8 currBannerElem.removeClassName(“active”);

9 bannerToChange.addClassName(“active”);

10 currentBannerID = bannerID;

11 }

12 currentBannerID = 1;

13 changeTimer = setTimeout(changeBanner, 5000);

1 function changeBanner(bannerID) {

2 clearTimeout(changeTimer);

3 changeTimer = setTimeout(changeBanner, 5000);

4

5 prefix = “banner_”;

6 currBannerElem = document.getElementById(prefix+currentBannerID);

7 bannerToChange = document.getElementById(prefix + bannerID);

8 currBannerElem.removeClassName(“active”);

9 bannerToChange.addClassName(“active”);

10 currentBannerID = bannerID;

11 }

12 currentBannerID = 1;

13 changeTimer = setTimeout(changeBanner, 5000);

bannerToChange being assigned return value of DOM access function

Error Marker

Technique outputs this line as direct DOM access

Last variable to take on null value

Implementation of AutoFLox Trace Collection: Modified versions of existing

tools (InvarScope [Groeneveld et al.] and Crawljax [Mesbah et al.])

Trace Analysis: Written from scratch

15

Evaluation: Research Questions RQ1: How prominent are DOM-related

JavaScript errors in web applications? RQ2: What is the fault localization accuracy of

AutoFLox? RQ3: What is the performance overhead of

AutoFLox?

16

Prominence of DOM-related JS Errors RQ1 Approach: Study bug reports of four web apps

(TUDU, TaskFreak, WordPress, and Google) Only JS-related bugs marked valid or fixed are

studied

17

17%

4%

79%

DOM-related

Non-DOM-relatedUnknown Type

65%

22%

13%

Code-terminatingDOM-related

Output DOM-related

Unknown DOM-related

Research Questions RQ1: How prominent are DOM-related

JavaScript errors in web applications? RQ2: What is the fault localization accuracy of

AutoFLox? RQ3: What is the performance overhead of

AutoFLox?

18

Accuracy of AutoFLox RQ2 Approach: Fault injection experiment on

TUDU, TaskFreak, and WordPress Faults injected: Null exceptions

Mutate the parameters of getElementById and getAttribute methods

19

Accuracy of AutoFLox: Results

WordPress: anonymous functions not supported by AutoFLox at the moment

Direct DOM access in Tumblr also found

20

Web App Total Number of Mutations

Number of direct DOM accesses identified

Percentage identified

TaskFreak 29 29 100%

TUDU 24 24 100%

WordPress 13 7 53.8%

Overall 66 60 90.9%

Research Questions How prominent are DOM-related JavaScript

errors in web applications? What is the fault localization accuracy of

AutoFLox? What is the performance overhead of

AutoFLox?

21

Performance RQ3 Approach: Measure trace collection

overhead in Tumblr; measure time it takes for trace analysis to complete Tumblr used because production code more

complex than development code Results

Trace collection incurred 35% overhead Trace analysis took 0.115 seconds to

complete

22

Conclusion DOM-related JS errors are prevalent Created fault localization tool for JS called

AutoFLox http://ece.ubc.ca/~frolino/projects/autoflox/

Evaluated accuracy and performance of AutoFLox

Future work Relax simplifying assumptions Target Output DOM-related JS errors Automatically fixing DOM-related JS errors

23

Backup Slides

25

Web 2.0 Applications: Document Object Model

- Defines elements contained in webpage

Dependability Problems in JavaScript Big contributing factor: JS interaction with

Document Object Model (DOM)

26

DIV

id: elem

DOM JavaScript Code

var x = document.getElementById(“elem”);var x = document.getElementById(“elme”);

Will return null Inexistent ID

DOM-related JavaScript error

Fault Localization Web application testing

What to do after we find errors? Need to fix them…

Fault localization: Find the root cause of the error

27

x = document.getElementById(“elme”);

changeText(x);

…..

function changeText(domElement) {

y = domElement;

y.innerHTML = ‘New Text’;

}exception

root cause

Running Example

28

1 function changeBanner(bannerID) {

2 clearTimeout(changeTimer);

3 changeTimer = setTimeout(changeBanner, 5000);

4

5 prefix = “banner_”;

6 currBannerElem = document.getElementById(prefix+currentBannerID);

7 bannerToChange = document.getElementById(prefix + bannerID);

8 currBannerElem.removeClassName(“active”);

9 bannerToChange.addClassName(“active”);

10 currentBannerID = bannerID;

11 }

12 currentBannerID = 1;

13 changeTimer = setTimeout(changeBanner, 5000);

Passed with no argument (even though changeBanner needs one argument)

bannerID will be set randomly

Would return null if bannerID out of range

NULL EXCEPTION!

AutoFLox: Scope Types of DOM-related JS errors

Code-terminating DOM-related JavaScript Errors: Errors that occur when DOM access function returns null, undefined, or incorrect value

Output DOM-related JavaScript Errors: Errors that occur when DOM update function sets DOM element’s property to wrong value

29

AutoFLox is concerned with code-terminating DOM-related JavaScript errors

Limitations User has to know how to replicate error User required to specify elements to click

during crawling Multiple JS errors currently not supported

30