leveraging visual testing with your functional tests
TRANSCRIPT
![Page 1: Leveraging Visual Testing with Your Functional Tests](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a64964a7f8b9a36568b5107/html5/thumbnails/1.jpg)
Peter Kim 2017
![Page 2: Leveraging Visual Testing with Your Functional Tests](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a64964a7f8b9a36568b5107/html5/thumbnails/2.jpg)
INTRO – H20Dragon Peter Kim Email: [email protected] https://www.linkedin.com/in/peterkim777
![Page 3: Leveraging Visual Testing with Your Functional Tests](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a64964a7f8b9a36568b5107/html5/thumbnails/3.jpg)
Agenda
� Share how visual testing benefited our functional test automation strategy
� Share where visual testing successfully provided coverage to other test types
� Share how to get started with using visual testing
� Demos � Q&A
![Page 4: Leveraging Visual Testing with Your Functional Tests](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a64964a7f8b9a36568b5107/html5/thumbnails/4.jpg)
What is Visual Testing?
![Page 5: Leveraging Visual Testing with Your Functional Tests](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a64964a7f8b9a36568b5107/html5/thumbnails/5.jpg)
What is Visual Testing?
� Verifies the UI/UX � Test coverage of what was actually rendered
on the UI device
� Test coverage for responsive app. behaviors
Verifies as though you are using your eyes.
![Page 6: Leveraging Visual Testing with Your Functional Tests](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a64964a7f8b9a36568b5107/html5/thumbnails/6.jpg)
Automated Visual Testing
![Page 7: Leveraging Visual Testing with Your Functional Tests](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a64964a7f8b9a36568b5107/html5/thumbnails/7.jpg)
A need to change because .... � Catch defects earlier in the SDLC � Verify the UI/UX across target browsers/UI devices � Verify the UI/UX across different resolutions � Existing automation strategy didn’t include “visual” coverage. � Current process of asserting validations are too static � Brittle Selenium locators (e.g. XPATH) � Page Objects aren’t scaling quickly enough � Customers are reporting defects, in PROD, instead of us � Inability to scale functional (Regression) test coverage � Uncertainty of what type of assertions are needed � Lack domain knowledge in the app. � Test reports lack depth � Test reports need meaningful visuals � Customers feel that your UI/UX is as important as the functionality � QA team has limited resources � It’s not 2007 anymore, but your test automation design/strategy
thinks it is. � You want to repurpose your functional tests for other test types –
e.g. Localization / Accessibility
![Page 8: Leveraging Visual Testing with Your Functional Tests](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a64964a7f8b9a36568b5107/html5/thumbnails/8.jpg)
You need it because ..
Although, we were unable to create assertions to catch potential defects, with visual testing, we now at least have the ability to report and share with others what has changed. Here, you’ve empowered others, or possibly held accountable, that they too were in a position to catch (or see) potential defects or comment on changes (whether intended or not per requirements/user story).
![Page 9: Leveraging Visual Testing with Your Functional Tests](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a64964a7f8b9a36568b5107/html5/thumbnails/9.jpg)
Automation with Selenium
![Page 10: Leveraging Visual Testing with Your Functional Tests](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a64964a7f8b9a36568b5107/html5/thumbnails/10.jpg)
Automation with Selenium � Challenges
� Takes time and resources to learn effective
Selenium/WebDriver
� Selenium is an automation library. You’re responsible for creating and/or piecing together the framework, designs, and strategies that will leverage it.
� Poor test frameworks/design will mean brittle tests and will most likely hinder your ability to scale – you will be playing “catch-up” with the current versions/releases of your app.
![Page 11: Leveraging Visual Testing with Your Functional Tests](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a64964a7f8b9a36568b5107/html5/thumbnails/11.jpg)
Automation with Selenium � Challenges
� Assertions are typically “static”, based on the
requirements/user stories/pre-canned queries. E.g. Limited number of assertions used to verify and validate your test case. This means, if your application behaves differently and none of your assertions cover that change .. you are now exposed to risks (defects, false confidence in your findings).
![Page 12: Leveraging Visual Testing with Your Functional Tests](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a64964a7f8b9a36568b5107/html5/thumbnails/12.jpg)
Are You Automating the “Expected”?
“Obvious is the most dangerous word in
mathematics.” - Eric Temple Bell (Mathematician) 1883-1960
![Page 13: Leveraging Visual Testing with Your Functional Tests](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a64964a7f8b9a36568b5107/html5/thumbnails/13.jpg)
Are You Automating the ‘Expected’?
![Page 14: Leveraging Visual Testing with Your Functional Tests](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a64964a7f8b9a36568b5107/html5/thumbnails/14.jpg)
Are You Automating the ‘Expected’?
� Pesticide Paradox Over the course of time, your automated tests will diminish it’s ability to catch defects. Especially, the tests that seem to always pass.
![Page 15: Leveraging Visual Testing with Your Functional Tests](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a64964a7f8b9a36568b5107/html5/thumbnails/15.jpg)
Improving automation
� What was our criteria to advance our existing automation strategy?
![Page 16: Leveraging Visual Testing with Your Functional Tests](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a64964a7f8b9a36568b5107/html5/thumbnails/16.jpg)
Automation Requirements � Need a technical solution that plays well with Selenium/WebDriver
since all our end-to-end tests are Selenium based.
� Minimal impact, e.g. refactoring, of our existing automation test suites.
� Avoid re-inventing designs based on old designs, thinking they’re new designs.
� Be cautious of “best practices” that have been around since 2007.
� Short and simple learning curve, with powerful results.
� Technology/tool must be well supported
� Technology/tool must scale and be secure (e.g. cloud, multiple device support)
� Powerful and customizable test reports
![Page 17: Leveraging Visual Testing with Your Functional Tests](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a64964a7f8b9a36568b5107/html5/thumbnails/17.jpg)
Lessons Learned � .. Be cautious when investing in resources
with strategies which ultimately repurpose the same old approach and processes.
� Consider leveraging a new design/strategy
� Consider leveraging proven technologies that integrate with your existing testware.
![Page 18: Leveraging Visual Testing with Your Functional Tests](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a64964a7f8b9a36568b5107/html5/thumbnails/18.jpg)
DEMO – Functional Automated Test
Filters, for advanced search, returns correct set of Porsche 911s.
![Page 19: Leveraging Visual Testing with Your Functional Tests](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a64964a7f8b9a36568b5107/html5/thumbnails/19.jpg)
DEMO – Functional Automated Test
Filters, for advanced search, returns correct set of Porsche 911s.
![Page 20: Leveraging Visual Testing with Your Functional Tests](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a64964a7f8b9a36568b5107/html5/thumbnails/20.jpg)
DEMO I. Create a baseline visual test
The visual baseline will be used for validation of future visual test results.
![Page 21: Leveraging Visual Testing with Your Functional Tests](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a64964a7f8b9a36568b5107/html5/thumbnails/21.jpg)
DEMO I. Create a baseline visual test
The visual baseline will be used for validation of future visual test results.
![Page 22: Leveraging Visual Testing with Your Functional Tests](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a64964a7f8b9a36568b5107/html5/thumbnails/22.jpg)
Demo I. Re-run the test. Visually compare the new run against the baseline
![Page 23: Leveraging Visual Testing with Your Functional Tests](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a64964a7f8b9a36568b5107/html5/thumbnails/23.jpg)
Demo I. Recap � Created a visual baseline
� Execute end-to-end with visual coverage � Specified matchType (e.g. Strict, Content, Layout) � Specified target page/regions
� Automated coverage scales with more assertions.
� Easy to “see” visual results
� Ability to manage/update assertion criteria
![Page 24: Leveraging Visual Testing with Your Functional Tests](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a64964a7f8b9a36568b5107/html5/thumbnails/24.jpg)
Demo II. Visual Defect
![Page 25: Leveraging Visual Testing with Your Functional Tests](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a64964a7f8b9a36568b5107/html5/thumbnails/25.jpg)
Demo II. Visual Defect Analysis
![Page 26: Leveraging Visual Testing with Your Functional Tests](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a64964a7f8b9a36568b5107/html5/thumbnails/26.jpg)
Demo II. View the results
![Page 27: Leveraging Visual Testing with Your Functional Tests](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a64964a7f8b9a36568b5107/html5/thumbnails/27.jpg)
What was actually demonstrated?
� Little resources are needed to add test coverage (assertions) if you have a simple, yet powerful, ability to leverage visual testing.
� Coverage is dramatically improved � Context (text) � Layout � Graphics
� Less brittleness in your automated Selenium tests, while decreasing coding efforts
� Forgiveness of ones’ lack of insight of “what should be covered?” or “what’s the scope?”
� UI (Visual) assertions can facilitate functional testing � Minimize pesticide paradox � Fred Flintstone has an awesome car!
![Page 28: Leveraging Visual Testing with Your Functional Tests](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a64964a7f8b9a36568b5107/html5/thumbnails/28.jpg)
Leveraging VT
Localization Test Automation (Regression)
![Page 29: Leveraging Visual Testing with Your Functional Tests](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a64964a7f8b9a36568b5107/html5/thumbnails/29.jpg)
Leveraging VT
� Localization Testing Ensuring that all targeted and/or served parties, regardless of language, nationality, and culture are equally and fairly respected and presented with the same information and content.
![Page 30: Leveraging Visual Testing with Your Functional Tests](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a64964a7f8b9a36568b5107/html5/thumbnails/30.jpg)
Localization Test Automation � Content � Images � Layout
![Page 31: Leveraging Visual Testing with Your Functional Tests](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a64964a7f8b9a36568b5107/html5/thumbnails/31.jpg)
Localization Test Automation � Content
� Headings � Data formats ○ Date ○ Currency
� Translation Errors � Images � Layout
![Page 32: Leveraging Visual Testing with Your Functional Tests](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a64964a7f8b9a36568b5107/html5/thumbnails/32.jpg)
Localization Test Automation � Content
� Headings � Data formats ○ Date ○ Currency
� Translation Errors � Images � Layout
Get more details on automating your Localization tests at https://applitools.com/concur/
![Page 33: Leveraging Visual Testing with Your Functional Tests](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a64964a7f8b9a36568b5107/html5/thumbnails/33.jpg)
Localization Test Auto
![Page 34: Leveraging Visual Testing with Your Functional Tests](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a64964a7f8b9a36568b5107/html5/thumbnails/34.jpg)
Tools –Applitools Eyes � Visual Test Automation
� Advanced Visual/Digital Analysis (AI) ○ Mitigates false positives (e.g. floating or shifting of elements) ○ Create/manage regions per inclusion/exclusion
� Applitools Account (It’s free)
https://applitools.com/sign-up/
� Applitools API Key
� Install the SDK (based on your programming language) for your Selenium tests
� Understanding resolutions – e.g. “viewport” Reference: https://applitools.com/videotutorials/
![Page 35: Leveraging Visual Testing with Your Functional Tests](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a64964a7f8b9a36568b5107/html5/thumbnails/35.jpg)
Summary � Functional testing can be improved by leveraging visual testing.
� It’s simple to conduct Visual Test Automation E.g. Applitools Eyes with Selenium/WebDriver based framework
� Heavy reliance, using the pattern of asserting the “obvious”, being the “master of the obvious”, or “happy path” is a great way to ship defects into PROD.
� Run and validate your tests on the actual devices your end-users are using.
� Run and validate your tests on the same resolutions (UI devices) based on the “rendered” pages (not just the DOM).
� Automate responsive apps visually.
� Create a plan to deal with Pesticide Paradox
� Advance your test automation framework with new designs and strategies (it’s not 2007) – also, leverage new technologies that are proven to be reliable and scalable, while being simple yet powerful.
� For QA/QE teams – leverage “CYA”
![Page 36: Leveraging Visual Testing with Your Functional Tests](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a64964a7f8b9a36568b5107/html5/thumbnails/36.jpg)
Thank You
Peter Kim Email: [email protected] https://www.linkedin.com/in/peterkim777
https://dcast.io