automated visual testing that doesn't suck!
TRANSCRIPT
Adam Carmi (@carmiadam)Co-Founder and VP R&D at Applitools
AUTOMATED VISUAL TESTING THAT DOES
NOT SUCK
YOU CAN AND SHOULD AUTOMATE YOUR VISUAL
TESTS!
AGENDA
Why automated visual testing?Tools & TechnologyWhere does it fit?
WHAT IS VISUAL TESTING?
A quality assurance activity aimed to verify that a
Graphical User Interface appears correctly to users
A VISUAL BUG
WHY SHOULD IT BE AUTOMATED?
THE TEST MATRIX IS TOO BIG TO APPROXIMATE MANUALLY
Web browsers Operating systems Screen resolutions Responsive designs L10n
WHY SHOULD IT BE AUTOMATED?
MANUALVISUALTESTINGISERRORPRONE
WHY SHOULD IT BE AUTOMATED?
WHY SHOULD IT BE AUTOMATED?
Many are already doing it (and sharing)…
PhantomCSS Fighting Layout Bugs
CSSCritiqueWraith
Needle
Grunt PhotoBox
dpdxt
WebdriverCSS
EyesHuxley
FBSnapshotTestCase
GeminiSelenium Visual Diff
VisualCeption
Specter
Snap And Compare
AGENDA
Why automated visual testing?Tools & TechnologyWhere does it fit?
THE WORKFLOW Drive the AUT and take screenshots
Compare screenshots with baseline images
Report differences
Update the baseline
THE WORKFLOW Drive the AUT and take screenshots
Compare screenshots with baseline images
Report differences
Update the baseline
DRIVING AND CAPTURING
OTHER
PhantomCSS (CasperJS)
Fighting Layout Bugs (Java) CSSCritique (URLs)Wraith (URLs)
Needle (Python)Grunt PhotoBox (URLs)
WebdriverCSS (JS)
Eyes (All)
Huxley (RP)
FBSnapshotTestCaseSelenium Visual Diff (Java)
VisualCeption (PHP)
Specter (JS)
Wraith-Selenium (Ruby)
Snap And Compare (URLs)
Eyes (All)
Gemini (JS)
dpdxt (URLs)
Gemini (JS)
SCREENSHOTS
Real browsers?Full page?Frames?Regions?Viewport size?Page stabilization?Page preparation?
THE WORKFLOW Drive the AUT and take screenshots
Compare screenshots with baseline images
Report differences
Update the baseline
WHY NOT?
FALSE POSITIVES
ANTI-ALIASING?
ANTI ALIASING 1/2
ANTI ALIASING 2/2
BRIGHTNESS 1/2
BRIGHTNESS 2/2
SUB PIXEL SHIFTS 1/2
SUB PIXEL SHIFTS 2/2
SUB PIXEL SHIFTS 1/2
SUB PIXEL SHIFTS 2/2
AND MORE…1 pixel offsets in element positioning
Dynamic contentMoving elementsImages of different size
Image Comparison APIs
ImageMagick A powerful command line tool for image processing. APIs are available for most programming languages. Fuzzing is used to eliminate slight color differences An error ratio is usually used to determine a match
$ compare –metric AE –fuzz 5% img1.png img2.png diff.png
2246
Resemble.js An image comparison tool
implemented in Javascript Used by PhantomCSS and other
tools. Good de-antialiasing An error ratio is usually used to
determine a match http://huddle.github.io/Resemble.js/
Applitools Eyes A specialized image processing stack
designed to compare computer generated UI images Anti-aliasing Partial and full pixel offsets Images of different size Dynamic content Moving elements Layout matching
Available as a cloud service
DEMO
THE WORKFLOW Drive the AUT and take screenshots
Compare screenshots with baseline images
Report differences
Update the baseline
REPORT DIFFERENCESAs files on the file system (combined with source control)
REPORT DIFFERENCESAs a Gallery (example from Selenium Visual Diff)
THE WORKFLOW Drive the AUT and take screenshots
Compare screenshots with baseline images
Report differences
Update the baseline
UPDATE THE BASELINE
Rename or commit individual image files
Overwrite mode
UPDATE THE BASELINE
GUI (Gemini GUI)
Automatic maintenance (demo)
AGENDA
Why automated visual testing?Tools & TechnologyWhere does it fit?
WHERE DOES IT FIT?
• Components
• Code review
• Developers
• Designers• QA
WHERE DOES IT FIT?
• Pages• Page
sections
• Developers
• Designers• QA• Others
WHERE DOES IT FIT?
• Staging vs. Production
• Ops• QA
WHERE DOES IT FIT?
• Monitoring
• Ops• QA
QUESTIONS?
Adam Carmi (@carmiadam)Co-Founder and VP R&D at Applitools
THANK YOU