designing a robust test strategy for mobile apps
TRANSCRIPT
4/23/15
1
Designing a Robust Test Strategy for
Mobile Apps
Parimala Hariprasad Tester, Speaker, Author
Who am I?
u A Curious Tester
u 12+ years experience in eCommerce, Mobile and UX
u Domain Expertise in CRM, Remote Automation, eCommerce, Real Estate and Healthcare
u Contributing Author to ‘More Agile Testing’ book
u First Recipient of ‘Bach Brothers Legion of Testing Merit’ award from James and Jon Bach
u Rock Star Mom of 2 girls, Foodie and a Horrible Cook
4/23/15
2
Agenda
• Mobility Trends • Four Dimensional Mobile Test Strategy
• Functional Tests • Code Level Tests • Specialized Tests • Analytical Tests
• Fifth Dimension • Real World Testing
• Summary u Q & A
4/23/15
3
Mobile Vs. Desktop
What is Test Strategy?
“Set of ideas that guide test design and execution“
~ James Bach
4/23/15
4
Four Dimensional Mobile Test Strategy
Functional
• Installation • Features • Platforms • Networks • Guidelines
Code Level
• Memory Footprint Analysis • Performance Benchmark • APK De-Compilation • IPA Analysis • Others
Analytical
• Field Testing • Analytics Driven • Social Media • Competitor Analysis • Customer Touch Points
Specialized
• Load / Performance • User Experience • Accessibility • Security • Localization
Functional
8
Installation Features
• App Exploration
• Feature Coverage
• LONG FUN CUP
Platforms
• Platforms / OS • iOS • Android • Windows
• Device Manufacturers & Models
• Screen Sizes & Resolutionn
• Fragmentation
Network
• Network Speed • 2G, 3G, 4G • WiFi • No Network • Switching • Shared
Network • Cellular • Dead Spots
• Network Carriers
Guidelines
• HIG Design Guidelines
• Android Design Guidelines
• UI Design Do's and Dont's
• Installability • Onboarding • Uninstallation
4/23/15
5
Functional > Features > App Exploration
Functional > Features > LONG FUN CUP
4/23/15
6
11
• LONG FUN CUP is a heuristic based test model E.g. If an app doesn’t support orientation, tester can skip that step based on the context
• LONG FUN CUP is a good way to test User Interfaces of mobile apps
• LONG FUN CUP is the topmost layer of mobile app testing
Functional > Features > LONG FUN CUP
Browsers
Opera2ng Systems Device Models
Functional > Platforms > Fragmentation
4/23/15
7
Fragmentation
Functional > Platforms > Selecting from an ocean of devices?
• Singular Approach ü One Device
• Proportional Approach ü Multiple Devices
• Shotgun approach
• Outsourced Approach
hBp://www.tes2ngcircus.com/tes2ng-‐trapeze-‐2014-‐december-‐edi2on/
4/23/15
8
Functional > Network
• Hardware Ø Smartphone
Ø Tablet
Ø Others?
• Operating System Versions Ø Main and Sub Versions
• Network Providers
15
• iOS HIG Guidelines
• Android Design Guidelines
• Android UI Design Do’s and Don’ts
Functional > Guidelines
4/23/15
9
Code Level
17
Memory Footprint Analysis
• Memory • CPU • Crash • Network
Performance Benchmark
• Battery Usage • Network
Performance
Android De-Compilation
Testing
• Manifest File Testing
• White box Testing • Apps Settings • Developer
Options
Code Level > Android Decompilation Testing
4/23/15
10
Code Level > Memory Footprint Analysis
Analytical
20
Field Testing
• Field Testing • Mobile
Personas • User Scenarios
Analytics Driven Testing
• User Analytics • Usage
Analytics • Analytical Tools
• Google Analytics
• Mix Panel • Omniture • Others
Social Media Testing
• Social Sharing • Social
Sentiments
Competitor Analysis
• Feature Comparison
• Platform Comparison
• Technology Comparison
• Search • SWOT
Customer Touch Points
• Supportability Factors
4/23/15
11
Analytical > Field Testing
21
• 80 kms bike trip while the app was streaming music all along the journey
• 3 hour road trip between Bangalore and Mysore
• Using the app in Garuda Mall, ladies toilet where none of the streaming apps work on 3G
• Streaming music in Old Airport Road, Bangalore where network gets disconnected automatically in one specific location
• Testing the app on Wi-Fi about 50m away from the building that hosted the Wi-Fi infrastructure
• Moving from 2G tower to 3G tower and vice versa over a distance of several kilometers
22
• Novice
• Trespasser
• First time user
• Impatient user
• Tech Savvy • User of competitor apps / different platform
• Who else?
Analytical > Field Testing > Mobile Personas
4/23/15
12
23
A person looking to rent on a <Real Estate Mobile App> is a
male graduate, between 25- 30 years on
age looking for a place in Bangalore. He is a
salaried employee of a private company,
earning between <XXX> USD to Rs <YYY> USD
Analytical > Field Testing > User Scenarios
24
Analytical > User Analytics
4/23/15
13
25
• What does data tell? ü Analytics
• How are apps used? ü Surveys
• What do users say? ü Reviews
Analytical > Usage Analytics
u Social Sharing u Context-driven sharing
u Login/Logout – App Login vs. Social Media Login
u App Sharing Compatibility across Platforms
u Data Privacy & Security
u Social Sentiments u How do users feel about the product?
u How do users feel about the social media presence?
Analytical > Social Media Testing
4/23/15
14
27
• Platform level comparison • Feature level comparison
• Technology level comparison
• Quality Criteria level comparison
Analytical > Competitor Analysis
What’s Sticky About YOUR App when compared to other apps?
4/23/15
15
29
Analytical > Customer Touch Points Testing
Supportability Factors
• Calls / SMSes
• Chat
• Service Requests
• Feedback
• Field Visits
• Operational Documentation
• Service Level Agreements 30
4/23/15
16
Measuring Customer Touch points
Specialized Tests Load /
Performance
• Concurrency • Throughput • Response
Times • Network
Utilization • Server
Utilization • Page Load &
Rendering Times
• Performance across Networks
User Experience
• Usability / UX • Real User
Surveys • Think Aloud
Videos • Emotions
Testing • Supportability
Testing • Claims Testing • Paper
Prototyping
Security
• OWASP Guidelines
• Code Review • Threat
Modeling • Vulnerability
Scanning • Tools based
Security Testing
• Crowd Security Experts
Accessibility
• Feature Coverage
• Language Coverage
• Serves Different Types of Specially-abled Users
• User Demographics
• Feature and Behavior Consistency
Localization
• Location based customization
• Language testing
4/23/15
17
Introducing Fifth Dimension
Real World Testing
• Platforms • Networks • App Behavior • Localization • User Groups • Security
App ready to Test
Ready to Go-Live
Five dimensional Mobile Test Strategy Framework
Real World Field Testing • Platforms • Networks • App Behavior • Localization • User Groups • Security
4/23/15
18
How do you know your Test Strategy is Good Enough?
What type of fish you catch depends on the type of net you use
Selber Schult
Never, under any circumstances, blame yourself for feeling confused by the technology
~ Jonathan Kohl ~
4/23/15
19
• Dhanasekar Subramaniam
• Jonathan Kohl
• Karen Johnson
• Julian Harty
• Rob Sabourin • Mayank Mittal
• Anne Marie Charrett
• Pradeep Soundararajan
• Image Credits – The World Wide Web
Acknowledgements
38
@CuriousTester
hBp://curioustester.blogspot.in
Thank You