![Page 1: Browserbite: Accurate Cross-Browser Testing via Machine Learning Over Image Features Nataliia Semenenko*, Tõnis Saar** and Marlon Dumas* *{nataliia,marlon.dumas}@ut.ee,](https://reader035.vdocuments.us/reader035/viewer/2022062320/56649c995503460f949568e5/html5/thumbnails/1.jpg)
Browserbite: Accurate Cross-Browser
Testing via Machine Learning Over
Image Features Nataliia Semenenko*, Tõnis Saar** and
Marlon Dumas*
*{nataliia,marlon.dumas}@ut.ee, Institute of Computer Science,
University of Tartu, Estonia**[email protected],
Browsrbite and STACC, Tallinn, Estonia
![Page 2: Browserbite: Accurate Cross-Browser Testing via Machine Learning Over Image Features Nataliia Semenenko*, Tõnis Saar** and Marlon Dumas* *{nataliia,marlon.dumas}@ut.ee,](https://reader035.vdocuments.us/reader035/viewer/2022062320/56649c995503460f949568e5/html5/thumbnails/2.jpg)
Outline
• Introduction• Visual cross-browser testing• Machine learning model• Results and future work
![Page 3: Browserbite: Accurate Cross-Browser Testing via Machine Learning Over Image Features Nataliia Semenenko*, Tõnis Saar** and Marlon Dumas* *{nataliia,marlon.dumas}@ut.ee,](https://reader035.vdocuments.us/reader035/viewer/2022062320/56649c995503460f949568e5/html5/thumbnails/3.jpg)
Cross-browser visual testing
Internet Explorer 9 Internet Explorer 8
Where’s that button?
![Page 4: Browserbite: Accurate Cross-Browser Testing via Machine Learning Over Image Features Nataliia Semenenko*, Tõnis Saar** and Marlon Dumas* *{nataliia,marlon.dumas}@ut.ee,](https://reader035.vdocuments.us/reader035/viewer/2022062320/56649c995503460f949568e5/html5/thumbnails/4.jpg)
Goal
• Develop method for cross-browser visual layout testing
• Replace human labor in visual testing• Evaluate detected errors
![Page 5: Browserbite: Accurate Cross-Browser Testing via Machine Learning Over Image Features Nataliia Semenenko*, Tõnis Saar** and Marlon Dumas* *{nataliia,marlon.dumas}@ut.ee,](https://reader035.vdocuments.us/reader035/viewer/2022062320/56649c995503460f949568e5/html5/thumbnails/5.jpg)
Methods• DOM (Document Object Model) based:
Mogotest (www.mogotest.com), Browsera (www.browsera.com)
• Image processing – non-invasive black box testing – Our current approach
Web page Static image
![Page 6: Browserbite: Accurate Cross-Browser Testing via Machine Learning Over Image Features Nataliia Semenenko*, Tõnis Saar** and Marlon Dumas* *{nataliia,marlon.dumas}@ut.ee,](https://reader035.vdocuments.us/reader035/viewer/2022062320/56649c995503460f949568e5/html5/thumbnails/6.jpg)
Cross-Browser Visual testing
![Page 7: Browserbite: Accurate Cross-Browser Testing via Machine Learning Over Image Features Nataliia Semenenko*, Tõnis Saar** and Marlon Dumas* *{nataliia,marlon.dumas}@ut.ee,](https://reader035.vdocuments.us/reader035/viewer/2022062320/56649c995503460f949568e5/html5/thumbnails/7.jpg)
Web page visual segmentation
• Image segmentation into regions of interest (ROI)
• ROI comparison
www.htcomp.ee
![Page 8: Browserbite: Accurate Cross-Browser Testing via Machine Learning Over Image Features Nataliia Semenenko*, Tõnis Saar** and Marlon Dumas* *{nataliia,marlon.dumas}@ut.ee,](https://reader035.vdocuments.us/reader035/viewer/2022062320/56649c995503460f949568e5/html5/thumbnails/8.jpg)
ROI comparison
• Position• Size• Geometry• Correlation
ROI from WIN7 Chrome
ROI from WIN7 IE8
VS
![Page 9: Browserbite: Accurate Cross-Browser Testing via Machine Learning Over Image Features Nataliia Semenenko*, Tõnis Saar** and Marlon Dumas* *{nataliia,marlon.dumas}@ut.ee,](https://reader035.vdocuments.us/reader035/viewer/2022062320/56649c995503460f949568e5/html5/thumbnails/9.jpg)
Visual testing results
• Test set of 140 web pages from alexa.com• 98% recall• 66% precision
Web page Static imageImage
segmentation(into ROIs)
ROI comparison
Example of true positive Example of false positive
![Page 10: Browserbite: Accurate Cross-Browser Testing via Machine Learning Over Image Features Nataliia Semenenko*, Tõnis Saar** and Marlon Dumas* *{nataliia,marlon.dumas}@ut.ee,](https://reader035.vdocuments.us/reader035/viewer/2022062320/56649c995503460f949568e5/html5/thumbnails/10.jpg)
ROI comparison + ML
Web page Static imageImage
segmentation(into ROIs)
ROI comparison
Classification
![Page 11: Browserbite: Accurate Cross-Browser Testing via Machine Learning Over Image Features Nataliia Semenenko*, Tõnis Saar** and Marlon Dumas* *{nataliia,marlon.dumas}@ut.ee,](https://reader035.vdocuments.us/reader035/viewer/2022062320/56649c995503460f949568e5/html5/thumbnails/11.jpg)
Machine learning
• 140 most popular websites of Estonia according to www.alexa.com
• 1200 potential incompatibilities • 40 subjects from 6 countries• Two classes :False positive vs True postive• Each ROI pair had 8 judgments• Inter-rater reliability 0,94
![Page 12: Browserbite: Accurate Cross-Browser Testing via Machine Learning Over Image Features Nataliia Semenenko*, Tõnis Saar** and Marlon Dumas* *{nataliia,marlon.dumas}@ut.ee,](https://reader035.vdocuments.us/reader035/viewer/2022062320/56649c995503460f949568e5/html5/thumbnails/12.jpg)
ROI features
• 10 histogram bins• Correlation index• Horizontal and vertical position• Horizontal and vertical size• Configuration index• Mismatch Density
![Page 13: Browserbite: Accurate Cross-Browser Testing via Machine Learning Over Image Features Nataliia Semenenko*, Tõnis Saar** and Marlon Dumas* *{nataliia,marlon.dumas}@ut.ee,](https://reader035.vdocuments.us/reader035/viewer/2022062320/56649c995503460f949568e5/html5/thumbnails/13.jpg)
Machine learning
• Neural network• Three layers• 11 neurons in hidden layer• Five-fold cross-validation
• Classification tree
![Page 14: Browserbite: Accurate Cross-Browser Testing via Machine Learning Over Image Features Nataliia Semenenko*, Tõnis Saar** and Marlon Dumas* *{nataliia,marlon.dumas}@ut.ee,](https://reader035.vdocuments.us/reader035/viewer/2022062320/56649c995503460f949568e5/html5/thumbnails/14.jpg)
Results and Conclusions
Measure Plain Browserbite
Mogotest Classification tree
Neural network
Precision 0.66 0.75 0.844 0.964
Recall 0.98 0.82 0.792 0.886
F-score 0.79 0.78 0.81 0.923
![Page 15: Browserbite: Accurate Cross-Browser Testing via Machine Learning Over Image Features Nataliia Semenenko*, Tõnis Saar** and Marlon Dumas* *{nataliia,marlon.dumas}@ut.ee,](https://reader035.vdocuments.us/reader035/viewer/2022062320/56649c995503460f949568e5/html5/thumbnails/15.jpg)
Results and conclusions
1. Choudhary, S.R., Prasad, M.R., and Orso, A. (2012). CrossCheck: Combining Crawling and Differencing to Better Detect Cross-browser Incompatibilities in Web Applications. (ICST), 2012 IEEE Fifth International Conference On, pp. 171–180.
2. Choudhary, S.R., Versee, H., and Orso, A. (2010). WEBDIFF: Automated identification of cross-browser issues in web applications. (ICSM), pp. 1–10.
Tool Mogotest CrossCheck [1] WebDiff [2] BB+ML
Precision 75% 36% 21% 96%
![Page 16: Browserbite: Accurate Cross-Browser Testing via Machine Learning Over Image Features Nataliia Semenenko*, Tõnis Saar** and Marlon Dumas* *{nataliia,marlon.dumas}@ut.ee,](https://reader035.vdocuments.us/reader035/viewer/2022062320/56649c995503460f949568e5/html5/thumbnails/16.jpg)
Future work
• Combination of image processing and DOM methods
• Dynamic content suppression
![Page 17: Browserbite: Accurate Cross-Browser Testing via Machine Learning Over Image Features Nataliia Semenenko*, Tõnis Saar** and Marlon Dumas* *{nataliia,marlon.dumas}@ut.ee,](https://reader035.vdocuments.us/reader035/viewer/2022062320/56649c995503460f949568e5/html5/thumbnails/17.jpg)
Thank You!