it's a phone first! how to test your five-star mobile apps
DESCRIPTION
Mobile application development shares many similarities-and some stark differences-with traditional web-based development. To build, test, and deploy five-star mobile applications, your organization needs-from inception-a focused test strategy to drive quality. Employing the wrong approaches and tools can leave your business sponsors and clients wondering what went wrong. Will Hurley outlines the current mobile landscape and explains what can and cannot be controlled in the mobile lifecycle. He explores the current landscape and limitations on tools for testing mobile apps, and offers guidance on what-and what not to-automate. With Will's guidance, you’ll learn how to establish a mobile lifecycle test strategy that is both leading edge and practical. From high-level feature classifications that are meaningful to your business to low-level test types and approaches that practitioners need to know, Will shares the knowledge you need to release a five-star app.TRANSCRIPT
BW4 Concurrent Session 11/7/2012 10:15 AM
"It's a Phone First! How to Test Your Five-star Mobile Apps"
Presented by:
Will Hurley Quality and Security Services
Brought to you by:
340 Corporate Way, Suite 300, Orange Park, FL 32073 888‐268‐8770 ∙ 904‐278‐0524 ∙ [email protected] ∙ www.sqe.com
Will Hurley Quality and Security Services
A practice leader of quality management and security for Idea Integration Corporation, Will Hurley has been at the crest of numerous software development waves during his twenty-four year career. Will represented the US Air Force in the DoD’s first joint service system architecture, led CMM assessments, and supported the development of the CMMi. After leaving the Air Force, he consulted for industry-leading companies, helping them adopt and apply new development methods. In his current position, Will consults with clients in North America on achieving desired system and product lifecycle characteristics.
28/09/2012
1
©2012 Idea. All rights reserved 1
It’s a Phone First! How to Test Your Five‐star Mobile Apps
Today’s discussionIdea and I
Acknowledgements
Five Star Mobile Application
©2012 Idea. All rights reserved 2
improving mobile project outcomes and time to market.
28/09/2012
2
Since 1986, we’ve helped some of the world’s largest and most respected brands thrive in the digital spacedigital space.
©2012 Idea. All rights reserved 3
The Presenter
Will HurleyPractice ExecutiveQuality Mgmt. & SecurityIdea
USAF Systems EngineerTBMCS Core Systems LeadJoint Service Architect GCCS
©2012 Idea. All rights reserved 4
Idea
[email protected](781) 831-0389
Joint Service Architect GCCSUSAF Data AdministratorConsultantSmall Business OwnerPractice Leader
28/09/2012
3
This discussion captures mobile testing insights developed at Idea’s Global Solution Centers in Nashua, NH and Houston, TX.
Idea’s insights are supported by more than 10 years of mobile
Acknowledgements
g pp y ydevelopment and testing.
Insights are not SDLC or tool specific.
Data is updated on the quarter through a verity of sources.
ComScore®
Netbiscuits®
©2012 Idea. All rights reserved 5
Harris Interactive®
OPA
Crittercism
5
5 Star Mobile applications
©2012 Idea. All rights reserved 6
6
28/09/2012
4
5 Star Mobile radar chart
©2012 Idea. All rights reserved 7
Ecosystem insights
o Capture more than 80% of smartphone users i h 3 k i
Understanding the mobile ecosystem and trends increase test effectiveness and product quality characteristics.
with 3 network carriers.
o Capture more than 85% of smart phone users with Android and iOS platforms.
o Application use is growing faster than browser use.
o Android users spend more time on carrier networks.
©2012 Idea. All rights reserved 8
o iOS users spend more time on wireless networks.
28/09/2012
5
Ecosystem carrier
As of Aug 2012, 114 million Americans own smart phones.
US Carrier Smart Phone Market Share
p
60 percent of all mobile devices acquired in the U.S. are smartphones.
Verizon market share has grown fastest driven by iOS.
Consider market share when signing contracts for cloud services.
Verizon 31.60%
AT&T 26.30%
SprintT‐Mobile 9 50%
Other 15%
©2012 Idea. All rights reserved 9
Sprint 10.50%
9.50%Tracfone 7.10%
Ecosystem oem
Samsung Galaxy S III leading 2012 sales may be passed by iPhone 5.
US Smartphone OEM
iPhone 4 took the top US device sales spot in 2011.
iPhone 3GS took 2nd place.
iPhone 4S took 3rd place.
BlackBerry Curve 8530 4th place.
HTC Evo 4G 5th place.
Samsung, 25.6%
LG, 18.4%
Apple, 16 3%
RIM, 6.3%
Other, 15.8%
©2012 Idea. All rights reserved 10
10
HTC is the most active OEM for Android smartphones.
16.3%
Motorola, 11.2%
HTC, 6.4%
28/09/2012
6
Ecosystem platform
66 percent of Americans ages 24-35 own a smartphone
US OS Share
own a smartphone.
Apple and Android market share are growing.
Market share for all other platforms is shrinking.
Android devices out number iPhones by 3-to-2 in the U.S. market.
Android52.2%
Apple33.4%
RIM9.5%
Microsoft3.6%
Symbian0.8%
Other0.5%
©2012 Idea. All rights reserved 11
Platform fragmentation
Fragmentation is a real but manageable issue for all platforms.
A d id f i iAndroid faces more issues since it resides in an open ecosystem.
Android OS adoption is slower than iOS.
Trades reporting >½ of all iPhone users that can upgrade to iOS5 have.
iOS6 expected to drive largest upgrade population to date.
©2012 Idea. All rights reserved 12
upg ade popu at o to date
Crash data are normalized as crashes as a percentage of each application launch.
28/09/2012
7
Ecosystem tablet
An estimated 74.1 million consumers are using tablets.
Tablet Share (Platform)
iPads account for more than 70% of tablet network traffic.
More than 80% of tablet traffic is wireless, not cellular.
Tablet adoption is balanced between males and females, trending older with households earning $50K or more.
Apple46%
Android42%
Other12%
©2012 Idea. All rights reserved 13
Kindle Fire 28%
iPad 2 31%
Customer Experience insights
o Good design is not accidental. Get help if d i
Mobile users expect more.
you need it.
o Good design is sticky.
o Unstructured or unsolicited feedback has more action value than a survey.
o Analytics portray the customer voice.
©2012 Idea. All rights reserved 14
28/09/2012
8
Experience components
How easy is it for your customer to use your application.
How does the li ti k
The application does what your customer wants it to do.
©2012 Idea. All rights reserved 15
application make your customer feel.
Customer experience
Four in ten mobile users are disappointed with applications from their favorite brands.
Text
Smartphone Usage
Three-quarters of users surveyed believe a company's mobile app should be easier to use than its Web site.
Users spend more time with mobile applications than their mobile browser.
82% of iOS and Android Apps crash once out of every 10 loads
Text message, 74.60%
Used browser, 48.60%
Used App, 48.60%Social net
or Blog,
Games, 31.80%
Music, 24.50%
©2012 Idea. All rights reserved 16
Less than 5% of mobile apps are still in use 3 months after downloading.
35.70%
28/09/2012
9
Experience sticky
©2012 Idea. All rights reserved 17
Experience design
©2012 Idea. All rights reserved 18
28/09/2012
10
Experience themes
Based on conversations, we found several common themes:
• All purchasing because it’s required by their apartment complexapartment complex.
• All participants were referred to site via a brochure.
• Participants pleased overall with the simplicity of the online process
• Liked being able to select from presented options in order to simplify and speed up the process.
• Appreciated the ability to get additional
©2012 Idea. All rights reserved 19
Appreciated the ability to get additional details where necessary.
• A majority didn’t use these features. • All participants pay annually for their
policy.
o Functional requirements are only part of the customer experience.
o Key moments in customer interaction demand good
Experience maturity
design.o If you don’t have experienced designers on your team
bring them in. o Understand accessible and emotional goals at every key
moment.o Don’t predict how people will respond to a new
experience test it
©2012 Idea. All rights reserved 20
experience, test it. o Allot the time to make incremental changes on all key moments.
28/09/2012
11
Functional insights
o Test while preforming common smart phone k
It’s a phone first.
tasks.
o It’s cheaper, faster and easier to test in emulated environments first.
o Testing types are the same as Web based n-tier applications
o Manage functional and non-functional insights across projects.
©2012 Idea. All rights reserved 21
g p j
o Create a mobile test strategy document.
o Reduce tool and process resistance between groups.
Manage scope
Carriers
Operating Systems
OEM Devices
©2012 Idea. All rights reserved 22
Com & Sensors
28/09/2012
12
Manage knowledge
©2012 Idea. All rights reserved 23
• Display and Graphics• Input Devices
Manage modularity
• Sensors• Permissions• Intents• Phone and messaging• Deploy and Upgrade
A l i
©2012 Idea. All rights reserved 24
• Analytics• Your application’s key moments
28/09/2012
13
Emulation(Simulation) testing
Does not require the physical device.
Can still use network services.
Emulation requires experience with specific platform development tools.
Static security and performance testing can be performed on source code.
Emulation is best for:
Investigating architectures and application layers
Agile testing of key features and
©2012 Idea. All rights reserved 25
g g yinteractions
Evaluating code quality characteristics
Research HTML5 characteristics.
Planning for automation
Device testing
Does require a physical device.
Testing on the device is best for:
U E i th l t 100 ilUser Experience – the last 100 miles.
Acceptance testing – working with gestures, sensors and hardware.
Soak Testing – how will the device perform over time.
Provisioning – installation and removal.
©2012 Idea. All rights reserved 26
Cloud providers and plugins for popular automation tools support both functional and performance testing.
No need to run out and buy all platforms.
28/09/2012
14
Automated testing
Browser automationTools can be used for ‘closest‐match’, browser testing on or off the device.
‘Closest‐match’ browser automation possible on some devices ‐ Safari 3 for the iPhone driven on a PC.
User‐Agent header data to drive compatibility testing logic.
Work closely with development team to manage and control Web 2.0 features.
Third party libraries can speed
©2012 Idea. All rights reserved 27
implementations.
Physical devices and Cloud services can be used to test physical platforms when defects are detected.
Automated testing
Application automationTools can be used in emulated environments and on the device.
Installation of some type of automation controller is required on the device.
Controllers consume resources.
Plug‐in and Cloud support is growing and stable.
Cloud services may not provide object recognition.
Bluetooth/Near Fiel
©2012 Idea. All rights reserved 28
Additional fixtures may be required for sensor (GPS, Accelerometer) and hardware (camera, battery) automation.
Applications may incorporate browser engine.
Field
28/09/2012
15
Functional details
Create a mobile test strategy document.Tight control of manifest and OS support.
Identify development and testing tool stack.
Explain how you plan to manage fragmentation – OS, device, sensors, UI, marketplace.
Test teams should be working in emulated, physical and cloud based environments.
Test teams should manage insights across releases. HTML5 platform insights
Carrier and Device insights
©2012 Idea. All rights reserved 29
OS / Platform insights
Security insights
Marketplace
Security insights
o Address security issues programmatically and start now!
Protect your brand.
o Static analysis is a highly effect approach to address coding issues.
o Dynamic analysis, Security Penetration testing, is your last line of defense.
o SD memory should be tested
©2012 Idea. All rights reserved 30
28/09/2012
16
Security testing
Static Application Security Testing (SAST)
Static analysis tools can address unsafe
Dynamic Application Security Testing
Mobile devices are susceptible toStatic analysis tools can address unsafe practices earlier in the SDLC.
Clang; AppValidator; AnalytiX
SAST should be integrated with build and reporting practices.
Evaluate application permissions.
Don’t build on or deliver on compromised platforms
Mobile devices are susceptible to common browser exploits and more.
Emulators or devices can be used to ensure applications don’t bleed sensitive data on to the device or to the network.
Keep up to date with known exploits and add test cases to qualify the impact.
©2012 Idea. All rights reserved 31
platforms.
Performance insights
o Static analysis is a effect approach to address performance issues.
Protect your brand.
p
o Dynamic analysis, Security Penetration testing is your last line of defense before delivery.
o SD memory should be tested.
©2012 Idea. All rights reserved 32
28/09/2012
17
Performance testing
Static Application Performance Testing (SAPT)
Static Analysis tools find issues earlier in
Performance Testing
Load stress and longevity testing areStatic Analysis tools find issues earlier in the SDLC and improve coding practices.
Tools address code that slow application performance.
In a 53K line project we found 328 instances of code that would contribute to performance issues.
SAPT should be integrated with build and
Load, stress and longevity testing are critical to the customer experience.
Load and stress testing are focused on middle and backend service layers.
Longevity testing should look at both backend services and the mobile device.
Performance tools should support bandwidth throttling and common RESTful
©2012 Idea. All rights reserved 33
reporting. protocols.
Final thoughts
o Cloud based testing services are valuable but…
o Will not achieve expected ROI without proper investigationproper investigation.
o Lead to Scrumfall application lifecycles.
o Can slow delivery times.
o Functional, performance, security and test automation should be considered for each mobile project.
o Simplify the testing space through active
©2012 Idea. All rights reserved 34
p y g p gmanagement.
o Diagram test environments.
o Support agile approaches for all team members.
28/09/2012
18
Final thoughts
o Develop sticky applications with the 5 star insights.
o Fill out our five start survey and see how your team stacks up!how your team stacks up!
©2012 Idea. All rights reserved 35