![Page 1: Finding Presentation Failures Using Image Comparison Techniques Sonal Mahajan and William G.J. Halfond Department of Computer Science University of Southern](https://reader030.vdocuments.us/reader030/viewer/2022032723/56649d0a5503460f949dd1f0/html5/thumbnails/1.jpg)
Finding Presentation Failures Using Image Comparison Techniques
Sonal Mahajan and William G.J. Halfond
Department of Computer Science
University of Southern California
![Page 2: Finding Presentation Failures Using Image Comparison Techniques Sonal Mahajan and William G.J. Halfond Department of Computer Science University of Southern](https://reader030.vdocuments.us/reader030/viewer/2022032723/56649d0a5503460f949dd1f0/html5/thumbnails/2.jpg)
2
Web Applications
Online music service
It takes users only 50 ms to form opinion about your website (Google research - 2012)
![Page 3: Finding Presentation Failures Using Image Comparison Techniques Sonal Mahajan and William G.J. Halfond Department of Computer Science University of Southern](https://reader030.vdocuments.us/reader030/viewer/2022032723/56649d0a5503460f949dd1f0/html5/thumbnails/3.jpg)
3
Motivation
• Presentation of a website– Crucial to make first impression– Capture users’ interest
• What is a presentation failure?– Web page rendering ≠ expected appearance
![Page 4: Finding Presentation Failures Using Image Comparison Techniques Sonal Mahajan and William G.J. Halfond Department of Computer Science University of Southern](https://reader030.vdocuments.us/reader030/viewer/2022032723/56649d0a5503460f949dd1f0/html5/thumbnails/4.jpg)
Example
Web page rendering Expected appearance (oracle)≠ 4
![Page 5: Finding Presentation Failures Using Image Comparison Techniques Sonal Mahajan and William G.J. Halfond Department of Computer Science University of Southern](https://reader030.vdocuments.us/reader030/viewer/2022032723/56649d0a5503460f949dd1f0/html5/thumbnails/5.jpg)
Example
Difference 1: Alignment problem
Web page rendering Expected appearance (oracle)≠ 5
![Page 6: Finding Presentation Failures Using Image Comparison Techniques Sonal Mahajan and William G.J. Halfond Department of Computer Science University of Southern](https://reader030.vdocuments.us/reader030/viewer/2022032723/56649d0a5503460f949dd1f0/html5/thumbnails/6.jpg)
Example
Difference 2: Color problem
Web page rendering Expected appearance (oracle)≠6
![Page 7: Finding Presentation Failures Using Image Comparison Techniques Sonal Mahajan and William G.J. Halfond Department of Computer Science University of Southern](https://reader030.vdocuments.us/reader030/viewer/2022032723/56649d0a5503460f949dd1f0/html5/thumbnails/7.jpg)
Example
Difference 3: Style problem
Web page rendering Expected appearance (oracle)≠ 7
![Page 8: Finding Presentation Failures Using Image Comparison Techniques Sonal Mahajan and William G.J. Halfond Department of Computer Science University of Southern](https://reader030.vdocuments.us/reader030/viewer/2022032723/56649d0a5503460f949dd1f0/html5/thumbnails/8.jpg)
8
Motivation
• Presentation of a website– Crucial to make first impression– Capture users’ interest
• What is a presentation failure?– Web page rendering ≠ expected appearance
• Impact of presentation failures– Gives negative impression of your business
• Affects branding efforts
– Reduces usability
End user – no penalty to move to another websiteBusiness – loses out on valuable customers
![Page 9: Finding Presentation Failures Using Image Comparison Techniques Sonal Mahajan and William G.J. Halfond Department of Computer Science University of Southern](https://reader030.vdocuments.us/reader030/viewer/2022032723/56649d0a5503460f949dd1f0/html5/thumbnails/9.jpg)
9
Limitations of Related Techniques
• Manual– Labor-intensive and error-prone
• Invariant specification techniques– Selenium, Sikuli, Cucumber, Crawljax– Required to exhaustively specify correctness invariants
• Tree-based comparison techniques– XBT, GUI differencing, automated oracle comparators– Cannot be used if DOM has changed significantly
• Fighting Layout Bugs– Application independent correctness checker
![Page 10: Finding Presentation Failures Using Image Comparison Techniques Sonal Mahajan and William G.J. Halfond Department of Computer Science University of Southern](https://reader030.vdocuments.us/reader030/viewer/2022032723/56649d0a5503460f949dd1f0/html5/thumbnails/10.jpg)
10
Our Approach
1. Detection 2. Localization
Oracle image
Test web page
Visual differences
Pixel-HTML mapping
Report
Goal – Automatically detect and localize presentation failures in web pages
![Page 11: Finding Presentation Failures Using Image Comparison Techniques Sonal Mahajan and William G.J. Halfond Department of Computer Science University of Southern](https://reader030.vdocuments.us/reader030/viewer/2022032723/56649d0a5503460f949dd1f0/html5/thumbnails/11.jpg)
11
Our Approach
1. Detection: determine whether a presentation failure has occurred
– Use image comparison to find visual differences between test web page and oracle
Model as image processing problem
![Page 12: Finding Presentation Failures Using Image Comparison Techniques Sonal Mahajan and William G.J. Halfond Department of Computer Science University of Southern](https://reader030.vdocuments.us/reader030/viewer/2022032723/56649d0a5503460f949dd1f0/html5/thumbnails/12.jpg)
12Oracle Test web page
![Page 13: Finding Presentation Failures Using Image Comparison Techniques Sonal Mahajan and William G.J. Halfond Department of Computer Science University of Southern](https://reader030.vdocuments.us/reader030/viewer/2022032723/56649d0a5503460f949dd1f0/html5/thumbnails/13.jpg)
13
![Page 14: Finding Presentation Failures Using Image Comparison Techniques Sonal Mahajan and William G.J. Halfond Department of Computer Science University of Southern](https://reader030.vdocuments.us/reader030/viewer/2022032723/56649d0a5503460f949dd1f0/html5/thumbnails/14.jpg)
14
Visual differencesDifference
pixels
![Page 15: Finding Presentation Failures Using Image Comparison Techniques Sonal Mahajan and William G.J. Halfond Department of Computer Science University of Southern](https://reader030.vdocuments.us/reader030/viewer/2022032723/56649d0a5503460f949dd1f0/html5/thumbnails/15.jpg)
15
Our Approach
2. Localization: identify the faulty HTML element
– Use R-tree to map pixel visual differences to HTML elements
– “R”ectangle-tree: height-balanced tree, popular to store multidimensional data
Use rendering maps to find faulty HTML elements corresponding to visual differences
![Page 16: Finding Presentation Failures Using Image Comparison Techniques Sonal Mahajan and William G.J. Halfond Department of Computer Science University of Southern](https://reader030.vdocuments.us/reader030/viewer/2022032723/56649d0a5503460f949dd1f0/html5/thumbnails/16.jpg)
16
![Page 17: Finding Presentation Failures Using Image Comparison Techniques Sonal Mahajan and William G.J. Halfond Department of Computer Science University of Southern](https://reader030.vdocuments.us/reader030/viewer/2022032723/56649d0a5503460f949dd1f0/html5/thumbnails/17.jpg)
R1
R2
R3
R4 Sub-tree of R-tree
17
![Page 18: Finding Presentation Failures Using Image Comparison Techniques Sonal Mahajan and William G.J. Halfond Department of Computer Science University of Southern](https://reader030.vdocuments.us/reader030/viewer/2022032723/56649d0a5503460f949dd1f0/html5/thumbnails/18.jpg)
(100, 400)
Map pixel visual differences to HTML elements18
R1
R2 R3 R4
div form
div input
Report:
/html/body/div[1]/html/body/div[1]/div[2]/form/div
/html/body/div[1]/div[2]/form/html/body/div[1]/div[2]/form/div/input[3]
![Page 19: Finding Presentation Failures Using Image Comparison Techniques Sonal Mahajan and William G.J. Halfond Department of Computer Science University of Southern](https://reader030.vdocuments.us/reader030/viewer/2022032723/56649d0a5503460f949dd1f0/html5/thumbnails/19.jpg)
19
Case Study
Subject Application Size #T Localization
Gmail 161 53 79%
Craigslist Autos 70 41 66%
Virgin America 1,016 41 78%
PayPal 317 51 84%
Average detection = 100%Average localization = 77%
• Detection accuracy: % of test cases in which our approach could detect that a presentation failure had occurred
• Localization accuracy: % of test cases in which the expected faulty element was reported in the result set
![Page 20: Finding Presentation Failures Using Image Comparison Techniques Sonal Mahajan and William G.J. Halfond Department of Computer Science University of Southern](https://reader030.vdocuments.us/reader030/viewer/2022032723/56649d0a5503460f949dd1f0/html5/thumbnails/20.jpg)
20
Future Work
• Problem 1: Handle only static pages– Dynamic regions e.g.: advertisements, user account
information, text from database– Check visual properties, not content
• Problem 2: Oracle image == test web page rendering– Pixel-perfect match not always needed– Allow for a “close enough” match with tolerance level
• Problem 3: Large result sets– Provide ranked result set to the user– Heuristics based on parent-child relationships
![Page 21: Finding Presentation Failures Using Image Comparison Techniques Sonal Mahajan and William G.J. Halfond Department of Computer Science University of Southern](https://reader030.vdocuments.us/reader030/viewer/2022032723/56649d0a5503460f949dd1f0/html5/thumbnails/21.jpg)
21
Summary
• Technique for automatically detecting and localizing presentation failures
• Use image processing techniques for detection
• Use rendering maps for localization
• Preliminary results validate feasibility of the approach
![Page 22: Finding Presentation Failures Using Image Comparison Techniques Sonal Mahajan and William G.J. Halfond Department of Computer Science University of Southern](https://reader030.vdocuments.us/reader030/viewer/2022032723/56649d0a5503460f949dd1f0/html5/thumbnails/22.jpg)
22
Thank you
Finding Presentation Failures Using Image Comparison Techniques
Sonal Mahajan
![Page 23: Finding Presentation Failures Using Image Comparison Techniques Sonal Mahajan and William G.J. Halfond Department of Computer Science University of Southern](https://reader030.vdocuments.us/reader030/viewer/2022032723/56649d0a5503460f949dd1f0/html5/thumbnails/23.jpg)
23
Need to Debug Presentation Failures
Requirements Gathering
Design
Development
Testing
And
Maintenance
SDLCSoftware Development
Life Cycle
![Page 24: Finding Presentation Failures Using Image Comparison Techniques Sonal Mahajan and William G.J. Halfond Department of Computer Science University of Southern](https://reader030.vdocuments.us/reader030/viewer/2022032723/56649d0a5503460f949dd1f0/html5/thumbnails/24.jpg)
24
Need to Debug Presentation Failures
Requirements Gathering
Design
Development
Testing
And
Maintenance
SDLCSoftware Development
Life Cycle
Presentation Development
Testing
![Page 25: Finding Presentation Failures Using Image Comparison Techniques Sonal Mahajan and William G.J. Halfond Department of Computer Science University of Southern](https://reader030.vdocuments.us/reader030/viewer/2022032723/56649d0a5503460f949dd1f0/html5/thumbnails/25.jpg)
25
1. Presentation Development Testing
• Front-end developers– Convert oracle images to “pixel-perfect” HTML
template pages• Back-end developers
– Change templates by adding dynamic content
• Both continuously test if the implemented page is consistent with the oracle
![Page 26: Finding Presentation Failures Using Image Comparison Techniques Sonal Mahajan and William G.J. Halfond Department of Computer Science University of Southern](https://reader030.vdocuments.us/reader030/viewer/2022032723/56649d0a5503460f949dd1f0/html5/thumbnails/26.jpg)
26
Need to Debug Presentation Failures
Requirements Gathering
Design
Development
Testing
And
Maintenance
SDLCSoftware Development
Life Cycle
Refactoring Debugging
![Page 27: Finding Presentation Failures Using Image Comparison Techniques Sonal Mahajan and William G.J. Halfond Department of Computer Science University of Southern](https://reader030.vdocuments.us/reader030/viewer/2022032723/56649d0a5503460f949dd1f0/html5/thumbnails/27.jpg)
27
2. Refactoring Debugging
• Applicable during regression testing• Changes to code after initial implementation
– E.g.: Refactoring page from <table> based layout to <div> based layout
• Changes not intended to change appearance• Change may have direct or indirect impact
• Test for presentation failures and debug to find responsible HTML elements
![Page 28: Finding Presentation Failures Using Image Comparison Techniques Sonal Mahajan and William G.J. Halfond Department of Computer Science University of Southern](https://reader030.vdocuments.us/reader030/viewer/2022032723/56649d0a5503460f949dd1f0/html5/thumbnails/28.jpg)
28
Need to Debug Presentation Failures
Requirements Gathering
Design
Development
Testing
And
Maintenance
SDLCSoftware Development
Life Cycle
Standard Debugging
![Page 29: Finding Presentation Failures Using Image Comparison Techniques Sonal Mahajan and William G.J. Halfond Department of Computer Science University of Southern](https://reader030.vdocuments.us/reader030/viewer/2022032723/56649d0a5503460f949dd1f0/html5/thumbnails/29.jpg)
29
3. Standard Debugging
• Make corrective code changes based on bug reports– E.g.: Resolve user-reported failures
• Reproduce the failure in-house and debug
![Page 30: Finding Presentation Failures Using Image Comparison Techniques Sonal Mahajan and William G.J. Halfond Department of Computer Science University of Southern](https://reader030.vdocuments.us/reader030/viewer/2022032723/56649d0a5503460f949dd1f0/html5/thumbnails/30.jpg)
30
Mockup Driven Development
• Mockups generated by graphic designers• Front-end developers must create “pixel-
perfect” template pages• Cursory search for front-end developer job
postings shows this is very common
![Page 31: Finding Presentation Failures Using Image Comparison Techniques Sonal Mahajan and William G.J. Halfond Department of Computer Science University of Southern](https://reader030.vdocuments.us/reader030/viewer/2022032723/56649d0a5503460f949dd1f0/html5/thumbnails/31.jpg)
31
![Page 32: Finding Presentation Failures Using Image Comparison Techniques Sonal Mahajan and William G.J. Halfond Department of Computer Science University of Southern](https://reader030.vdocuments.us/reader030/viewer/2022032723/56649d0a5503460f949dd1f0/html5/thumbnails/32.jpg)
32
![Page 33: Finding Presentation Failures Using Image Comparison Techniques Sonal Mahajan and William G.J. Halfond Department of Computer Science University of Southern](https://reader030.vdocuments.us/reader030/viewer/2022032723/56649d0a5503460f949dd1f0/html5/thumbnails/33.jpg)
33
![Page 34: Finding Presentation Failures Using Image Comparison Techniques Sonal Mahajan and William G.J. Halfond Department of Computer Science University of Southern](https://reader030.vdocuments.us/reader030/viewer/2022032723/56649d0a5503460f949dd1f0/html5/thumbnails/34.jpg)
34
![Page 35: Finding Presentation Failures Using Image Comparison Techniques Sonal Mahajan and William G.J. Halfond Department of Computer Science University of Southern](https://reader030.vdocuments.us/reader030/viewer/2022032723/56649d0a5503460f949dd1f0/html5/thumbnails/35.jpg)
35
![Page 36: Finding Presentation Failures Using Image Comparison Techniques Sonal Mahajan and William G.J. Halfond Department of Computer Science University of Southern](https://reader030.vdocuments.us/reader030/viewer/2022032723/56649d0a5503460f949dd1f0/html5/thumbnails/36.jpg)
36
![Page 37: Finding Presentation Failures Using Image Comparison Techniques Sonal Mahajan and William G.J. Halfond Department of Computer Science University of Southern](https://reader030.vdocuments.us/reader030/viewer/2022032723/56649d0a5503460f949dd1f0/html5/thumbnails/37.jpg)
37
![Page 38: Finding Presentation Failures Using Image Comparison Techniques Sonal Mahajan and William G.J. Halfond Department of Computer Science University of Southern](https://reader030.vdocuments.us/reader030/viewer/2022032723/56649d0a5503460f949dd1f0/html5/thumbnails/38.jpg)
38
![Page 39: Finding Presentation Failures Using Image Comparison Techniques Sonal Mahajan and William G.J. Halfond Department of Computer Science University of Southern](https://reader030.vdocuments.us/reader030/viewer/2022032723/56649d0a5503460f949dd1f0/html5/thumbnails/39.jpg)
39
![Page 40: Finding Presentation Failures Using Image Comparison Techniques Sonal Mahajan and William G.J. Halfond Department of Computer Science University of Southern](https://reader030.vdocuments.us/reader030/viewer/2022032723/56649d0a5503460f949dd1f0/html5/thumbnails/40.jpg)
40
![Page 41: Finding Presentation Failures Using Image Comparison Techniques Sonal Mahajan and William G.J. Halfond Department of Computer Science University of Southern](https://reader030.vdocuments.us/reader030/viewer/2022032723/56649d0a5503460f949dd1f0/html5/thumbnails/41.jpg)
41
![Page 42: Finding Presentation Failures Using Image Comparison Techniques Sonal Mahajan and William G.J. Halfond Department of Computer Science University of Southern](https://reader030.vdocuments.us/reader030/viewer/2022032723/56649d0a5503460f949dd1f0/html5/thumbnails/42.jpg)
42