build your own mad-lab - for mobile test automation
TRANSCRIPT
MAD-LABMobile Automation Devices LAB
Anand BagmarSoftware Quality Evangelist
About Me
@BagmarAnand
Blog – essenceoftesting
about.me/anand.bagmar
What do you expect from this session?
Context
EcosystemPlatforms
• Android (> v4.4.2)
• iPhones (> v5)
• Desktop / WAP
Regions• Indonesia
• Malaysia
• India
• Middle-East
• ...
Dynamic Content
Search
Download
Offers
ADs
What this means ... Reality!• Unique installs (Android) - > 1.75M
• Unique # of devices (Android) - > 13K
What this means ... Product!• One product, One experience
• Unique / Regional / Local content
• User detection & offers based on origin
• Region
• Carrier network
• ...
What this means ... Testing!• Video Quality, Dynamic Content, Download Expiry
• Simulating Carriers / Networks
• Consumer focus!• Experience
• Device / OS combinations
Business Requirements• Provide visibility into -
• What is Tested
• Overall Quality of Product
APIs
Payment partners
CMS
....
....
Architecture
APIs
Payment partners
CMS
....
....
Where are these?
How would you approach
Automation for such a product?
What Metrics make sense for
determining Quality?
Automation Strategy
Automation Criteria• E2E tests should focus on Business Rules & User Scenarios
• Reports should reflect the above
• Understood by all Business & Technical stakeholders
• Generated automatically
• Should run frequently
• Every new change in the product-under-test
• On a variety of device / OS combinations
What to Automate?Validate what was working, continues to work!
• Identify user journeys / scenarios
• Prioritize it!
Run Automation against...Usage patterns from Analytics / PlayStore / App Store
• Identify devices that matter
• Identify devices that are “automation-friendly”
Automation Tech Stack• Cucumber-JVM à For specifying Business Rules
• Appium à To allow integration with Android / iOS
• Gradle / Groovy à As build tool scripting language
• Jenkins à Continuous Integration Server
• TTA - Test Trend Analyzer à Trend & Failure Analysis
MVP / POC
Scale&
Repeat
Experiment, Learn & Evolve
Experiments• Run tests on Emulators
• Use Cloud-based services (Local Vs Remote)
• SauceLabs
• TestObject
• Amazon Device Farm
• pCloudy
• Devices
Some Experiments didn’t work• Emulators
• Video Player
• Location
• Cloud-based services• Location
• Tech Stack
• Devices
MAD-LAB
MAD-LAB – Mobile Automation Devices LAB
Innovations in MAD-LAB• Device management (selection, cleanup, app install and
uninstall)
• Parallel test execution (at Cucumber scenario level) -maximising device utilisation)
• Appium server management
• Adb utilities (periodic ADB server disconnects)
• Custom reporting using cucumber-reports
• Video recording of each scenario and embedding in the custom reports
• Analytics Automation
Conclusion
Learnings (so far)Blue-print is important!
• Stepping-stone approach
Experiment!
• Know your objective(s)
• Fail-fast!
Make the Product Testable!!
References• Sample code
• https://github.com/anandbagmar/cucumber-jvm-appium-infra
• Blog posts• https://essenceoftesting.blogspot.com/search/label/madlab
• TTA (Test Trend Analyzer)• https://essenceoftesting.blogspot.com/search/label/tta
Thank you!
@BagmarAnand
Blog – essenceoftesting
about.me/anand.bagmar