automated visual testing that does not suck
TRANSCRIPT
Adam Carmi (@carmiadam)
Co-Founder and VP R&D at Applitools
AUTOMATED VISUAL TESTING THAT DOES
NOT SUCK
YOU CAN AND SHOULD
AUTOMATE YOUR
VISUAL TESTS!
AGENDA
Why automated visual testing?
Tools & Technology
Where does it fit?
WHAT IS VISUAL TESTING?
A quality assurance activity aimed to verify that a Graphical User Interface appears correctly to users
A VISUAL BUG
WHY SHOULD IT BE AUTOMATED?
THE TEST MATRIX IS TOO BIG TO APPROXIMATE MANUALLY Web browsers
Operating systems
Screen resolutions
Responsive designs
L10n
WHY SHOULD IT BE AUTOMATED?
MANUALVISUALTESTINGISERRORPRONE
WHY SHOULD IT BE AUTOMATED?
WHY SHOULD IT BE AUTOMATED?
Many are already doing it (and sharing)…
PhantomCSS Fighting Layout Bugs
CSSCritique
Wraith
Needle
Grunt PhotoBox
dpdxt
WebdriverCSS
EyesHuxley
FBSnapshotTestCase
GeminiSelenium Visual Diff
VisualCeption
Specter
Snap And Compare
AGENDA
Why automated visual testing?
Tools & Technology
Where does it fit?
THE WORKFLOW
Drive the AUT and take screenshots
Compare screenshots with baseline images
Report differences
Update the baseline
THE WORKFLOW
Drive the AUT and take screenshots
Compare screenshots with baseline images
Report differences
Update the baseline
DRIVING AND CAPTURING
OTHER
PhantomCSS (CasperJS)
Fighting Layout Bugs (Java)CSSCritique (URLs)Wraith (URLs)
Needle (Python)Grunt PhotoBox (URLs)
WebdriverCSS (JS)
Eyes (All)
Huxley (RP)
FBSnapshotTestCaseSelenium Visual Diff (Java)
VisualCeption (PHP)
Specter (JS)
Wraith-Selenium (Ruby)
Snap And Compare (URLs)
Eyes (All)
Gemini (JS)
dpdxt (URLs)
Gemini (JS)
SCREENSHOTS
Real browsers? Full page? Frames? Regions? Viewport size? Page stabilization? Page preparation?
THE WORKFLOW
Drive the AUT and take screenshots
Compare screenshots with baseline images
Report differences
Update the baseline
WHY NOT?
FALSE POSITIVES
ANTI-ALIASING?
ANTI ALIASING 1/2
ANTI ALIASING 2/2
BRIGHTNESS 1/2
BRIGHTNESS 2/2
SUB PIXEL SHIFTS 1/2
SUB PIXEL SHIFTS 2/2
SUB PIXEL SHIFTS 1/2
SUB PIXEL SHIFTS 2/2
ANDMORE…
1 pixel offsets in element positioning
Dynamic content
Moving elements
Images of different size
Image Comparison APIs
ImageMagick A powerful command line tool for image processing. APIs are available for most programming languages. Fuzzing is used to eliminate slight color differences An error ratio is usually used to determine a match
$ compare –metric AE –fuzz 5% img1.png img2.png diff.png
2246
Resemble.js An image comparison tool implemented in Javascript
Used by PhantomCSS and other tools.
Good de-antialiasing
An error ratio is usually used to determine a match
http://huddle.github.io/Resemble.js/
Applitools Eyes A specialized image processing stack designed to compare
computer generated UI images
Anti-aliasing
Partial and full pixel offsets
Images of different size
Dynamic content
Moving elements
Layout matching
Available as a cloud service
DEMO
THE WORKFLOW
Drive the AUT and take screenshots
Compare screenshots with baseline images
Report differences
Update the baseline
REPORT DIFFERENCESAs files on the file system (combined with source control)
REPORT DIFFERENCESAs a Gallery (example from Selenium Visual Diff)
THE WORKFLOW
Drive the AUT and take screenshots
Compare screenshots with baseline images
Report differences
Update the baseline
UPDATE THE BASELINE
Rename or commit individual image files
Overwrite mode
UPDATE THE BASELINE
GUI (Gemini GUI)
Automatic maintenance (demo)
AGENDA
Why automated visual testing?
Tools & Technology
Where does it fit?
WHERE DOES IT FIT?
• Component
s
• Code
review
• Developers
• Designers
• QA
WHERE DOES IT FIT?
• Pages
• Page
sections
• Developers
• Designers
• QA
• Others
WHERE DOES IT FIT?
• Staging vs.
Production
• Ops
• QA
WHERE DOES IT FIT?
• Monitoring
• Ops
• QA
QUESTIONS?
Adam Carmi (@carmiadam)
Co-Founder and VP R&D at Applitools
THANK YOU