five ways to improve your mobile testing

T22 Mobile Testing

10/16/2014 3:00:00 PM

Five Ways to Improve Your Mobile Testing

Presented by:

Dennis Schultz

IBM

Brought to you by:

340 Corporate Way, Suite 300, Orange Park, FL 32073 888-268-8770 ∙ 904-278-0524 ∙ [email protected] ∙ www.sqe.com

Dennis Schultz

IBM In IBM’s Emerging Technologies organization, Dennis Schultz is a solution architect, tester, and software engineer. As a child, Dennis had an insatiable curiosity of how things worked and a desire to take them apart. The all-too-common outcome was that he could not put them back together again, thus leading to an adult career in testing. Wanting to make amends for these transgressions eventually lead to his current position as a Solution Architect, helping clients assemble products and tools into integrated solutions to meet their challenges. Dennis has held roles in IBM from marketing to sales enablement to technical sales.

© 2014 IBM Corporation

5 Ways to Improve the Quality and Efficiency of your Mobile Testing

Dennis Schultz, IBM Solution Architect Rational Emerging Technologies Team [email protected] dennisschultz.wordpress.com

© 2014 IBM Corporation 2

Mobile

State of the Market


Mobile

48% of mobile consumers reported

feeling frustrated and annoyed, with

sites that are not designed with

mobile in mind.

The Mobile Imperative "Only 16% of consumers

would give a mobile app

more than two attempts.

Poor mobile app

experience is therefore

likely to discourage users

from using an app again."

No Mobile Site = Lost Customers Two-thirds of smart phone users say a mobile-friendly site makes them more likely to buy a company’s product or service

Mobile internet usage doubles in growth in the U.S.

By 2014, mobile internet

should take over desktop

internet usage

“By 2014, mobile internet should take over desktop internet usage”




Mobile Shoppers in the US projected to spend $37.44 billion in 2013, up from $23.72 billion last year

"In 2013 consumers are expected to spend over $1.25 trillion dollars online, $38.4 billion will come from mobile"

"In 2013 consumers are expected to spend over $1.25 trillion dollars online, $38.4 billion will come from mobile""In 2013 consumers are expected to spend over $1.25 trillion dollars online, $38.4 billion will come from mobile“"In 2013 consumers are expected to spend over $1.25 trillion dollars online, $38.4 billion will come from mobile“ "In 2013 consumers are expected to spend over $1.25 trillion dollars online, $38.4 billion will come from mobile""In 2013 consumers are expected to spend over $1.25 trillion dollars online, $38.4 billion will come from

Spending on mobile

applications development will

grow by 50% in 2013 to nearly

2% of total IT spend.

"Gartner predicts that mobile AD

projects targeting smartphones and

tablets will outnumber native PC projects

by a ratio of 4:1 by 2015."

"Mobile users will increase by 91 million, over the next four years" (IDC)

“Sales of smart mobile devices including smartphones and tablets will grow by 20%, generate 20% of all IT sales, and drive a whopping 57% of all IT market growth" “Sales of smart mobile devices including smart phones and tablets will grow by 20%, generate 20% of all IT sales, and drive a whopping 57% of all

market growth“ “Sales of smart mobile devices including smart phones and tablets will grow by 20%, generate 20% of all IT sales, and drive

Mobile web growth: 1 in 5

internet users don't use a

computer

Mobile traffic has climbed to more than 16 percent of overall web traffic, increasing 27 percent this year.

eMarketer forecasts the number of mobile shoppers in the US will increase by 24% in 2013 to 118 million consumers and represent 62% of digital shoppers.

"Since 2009, Mobile Internet usage

has doubled every year"

"A poor mobile experience can have

a negative impact on the

transaction; and in the long term, it

could be even more detrimental to a

brand's reputation"

A Poor Mobile Email Experience Leads 30

percent of Consumer to Unsubscribe


Mobile

Question:

Are the apps you are developing and testing “mission critical”? Would your management agree?


Mobile

Challenges of Testing Mobile Applications

DEVICE DIVERSITY APP DIVERSITY ORGANIZATION

DIVERSITY

•  Multiple diverse teams •  ALM tools •  Automation tools •  Business process tester •  Automation engineers

and programmers •  Manual testers

MOBILE PAYMENT

APP

INVESTMENT APP

MAIN WEBSITE

INTERNAL CORPORATE

WEBSITE

INTERNAL EXPENSE

APP

INTERNAL TRADING

APP COMMERCIAL BANKING

APP

•  Software Platforms •  Connectivity •  Hardware

Native Web Hybrid


Mobile

Question:

What type(s) of mobile apps are you developing? 1.  Native 2.  Web 3.  Hybrid


Mobile

Question:

What is your role? 1.  All I do is test and I only test mobile apps. 2.  All I do is test. I test mobile apps, but my day job is testing

other stuff. 3.  I’m a developer and they are forcing me to test what I write.


Mobile

The Impact of Mobile

Development

Testing

Data Management / Security

Area / Discipline

PM/Requirements Management

UI Design Low Large

Low Medium

Huge Huge

Low Low

Process Changes Tool Changes

Low None


Mobile

5 Ways to Improve the Quality and Efficiency

of your Mobile Testing


Mobile

1.  Implement a collaborative, agile process using ALM tools



Mobile

! Software development involves a team of people with diverse skills using different tools

–  Design, development, test, operations, etc.

! Tools can make or break collaboration –  Fragmented tools create silos –  Integrated tools break down barriers

! Effective development depends on … –  Defined goals –  Transparency of progress and quality trends –  Aligning teams across the entire application

development lifecycle

Mobile Software Development is a Team Sport


Mobile

Application Lifecycle Management (ALM) is the discipline of overcoming organizational silos to realize a whole-team, whole-view approach to the software delivery cycle.

1. Business

2. Development

Decide

3. Operations*

ALM manages the flow of: People Process

Information

Mobile development requires the adoption of Agile practices to keep up with demand for frequent releases


Mobile

Question

Do your current mobile development projects use Agile methods and practices? 1.  Yes, all of them use Agile 2.  No, none of them use Agile 3.  Mix, some of them use Agile


Mobile

What does an Agile paradigm offer?

Agile succeeds three times more often than non-agile projects

The Chaos Manifesto, Standish Group 2012


Mobile

1.  Implement a collaborative agile process using ALM tools

2.  Automate repetitive tests to speed time to market and improve quality



Mobile

!  As with software QA, test automation of mobile apps improves the effectiveness of your QA test process

!  Improve quality !  Increase device coverage

!  Save money !  Execute more tests in less time

Automate, Automate, Automate


Mobile

Goals

Test Cases

Frequency

Methodology

Basic acceptance, build acceptance

Functional testing across all existing areas of product

Compatibility across devices

Very high value to automate!

High value to automate

Medium value to automate

# of Devices

Optimal Test Strategy

Automated Testing

Automated Testing

Manual or Automated Testing

Smoke Testing

Regression Testing

Compatibility Testing

Deep testing of new areas of functionality

Low value to automate

Manual Testing

New Feature Testing

Not automated

Exploratory Testing

Exploration of functionality from customer viewpoint

Manual Testing

Efficient and Targeted Automation Strategy


Mobile

Question

Do you use test automation tools today?


Mobile



3.  Adopt a strategy for Continuous Integration Testing



Mobile

Service virtualization makes the unavailable become available for testing

!  Virtual components simulate the behavior of a service or application during testing

!  System dependencies are a key challenge in setting up test environments

!  Unavailable/inaccessible services slow down the testing process

!  Costly 3rd party access fees can significantly impact testing availability

!  Virtual components run on commodity hardware, private cloud, public cloud

!  Each developer and tester can easily have their own test environment

!  Developers and testers can continue to use current testing procedures and tools

!  Streamline creation of production-like environments and negative testing

Heterogeneous Environments

Public Cloud Private Cloud

Data Warehouse Mainframe Enterprise Service Bus

Directory Identity

File systems

Collaboration

App Under Test Routing Service

Third-party Services Portals

Content Providers

EJB

Shared Services

Archives

Business Partners

Messaging Services

Databases Mainframe applications

App Under Test

Third-party Services

Packaged apps, messaging services, etc.

Virtual Components


Mobile

Continuous Integration Testing - by example

DevOps Foundation

Continuous deployment and testing across all development stages

Pass/Fail App M1 M2 ERP WSDL 3rd party Database

Real V V V " Test /w back end systems

Real " Test my own piece

•  Test Mobile App with three virtualized services. •  Quick to setup and low-cost. •  Author integration tests early

#  Service Virtualization is an enabler for continuous integration testing

#  Services, applications, systems are introduced into the continuous integration cycle in a prioritized, controlled fashion.

# Controlled integration helps isolate defects for faster resolution.


Mobile


DevOps Foundation


Pass/Fail App M1 M2 ERP WSDL 3rd party Database



Real V V V R

Real V V V V

Real V V V Test Data V V

Real V V V Test Data V R

$ " " "

Integrate w/another This won’t

slow me down!

First Complete System Test

All components Integrated!

Real V V V Test Data R R " Test integrations with control


Mobile


DevOps Foundation


Pass/Fail

Real R R V Test Data R R

Real R R R R R R

" The expensive backend

All Systems UP! "

App M1 M2 ERP WSDL 3rd party Database

Real V V V R

Real V V V V

Real V V V Test Data V V

Real V V V Test Data V R

$ " " "

Integrate w/another This won’t

slow me down!

First Complete System Test

All components Integrated!



Real V V V Test Data R R " Test integrations with control

•  Accelerate test environment refresh •  Integrate and test new functionality with control •  Gradually test integrations with expensive

backend systems •  Combine continuous test execution with

automated build deployment – Validate not only the code being released but also the deployment process

•  Discover and isolate defects faster! •  Accelerate delivery of higher quality software!


Mobile

Question

Are you using service virtualization or continuous integration testing today?


Mobile




4.  Utilize a combination of real and emulated devices



Mobile

Different screen resolution / screen size

Unreadable text, blurred images, misalignment of screen elements, and items that fall off the screen.

Android Customizations

On-screen and physical controls that function differently across devices Customized handling of inputs and events (e.g. Samsung Swype vs. default Android keyboard). !

Memory / CPU

Low or insufficient memory or processing power. Impacts of other services running on the device.

Mobile device characteristics that impact the quality of your application or website, but are not easily verifiable using an emulated phone or browser, include:

Why Test on Real Devices?


Mobile

!  Device platforms, fragmentation, and growth •  iOS, Android, BlackBerry, Windows Phone,

Symbian, etc. •  Multiple form factors and screen resolutions •  Device refresh every 24 months •  In 2011, over 300 new devices released

!  New capabilities to test •  Camera, GPS, direction, orientation, voice, etc.

!  More network considerations •  Multiple carriers, variable throughput and latency,

disconnected use, network switching, etc.

Testing on Real Devices is Challenging


Mobile

Question

How much of your mobile testing is done using real devices? 1.  0 – 1/3 2.  1/3 – 2/3 3.  2/3 – All of it


Mobile

Smok

e Te

stin

g U

nit

Test

ing

Mobile Product Development Timeline

Test

Pha

se

Emulators Real Devices/ Device Cloud

Emulators Real Devices/ Device Cloud

New

Fe

atur

e Te

stin

g Emulators Real Devices

When to use Emulators/Simulators vs Real Devices

Dev QA

OS

&

Com

patib

ility

Te

stin

g

Device Cloud


Mobile

Real Devices in the cloud Your computer, connected to the internet

Your key presses and mouse clicks sent to the device

The device’s screen sent back to your

computer

SaaS-based Remote Device Products


Mobile

Devices

Shared Devices !  24X7 access to hundreds of smart devices !  Use for compatibility testing, “untrusted device” testing

Private Devices !  24X7 access to your devices !  Enables geographically diverse team

!  Inside corporate firewall or external

Local Devices !  Plug your smart devices on hand directly into your local computer !  Make use of existing assets, great for local teams

Use a Combination of Local and Remote Devices


Mobile





5.  Focus on the user’s experience



Mobile

Concerns that make Mobile App Quality different

Tester End Users Developer LOB/Digital Marketer

�When someone leaves a negative review in the app store, it scars your app for life, you can’t respond to it, and you can’t learn more about the problem in order to fix it quickly.�

- Mobile Orchard

•  How effective is our mobile app at engaging customers?

•  How can I stay on top of current quality metrics, and turn them into an actionable strategy?

•  How could I get visibility to issues that customers experience in production so that I can address them quickly?

•  How can I get more people to test the apps so that we can cover different devices, network carriers, and OS versions?

•  How can I get the latest and greatest build without wasting my time to track that down?

•  How can I quickly submit bug reports with screenshots and device details right from the application?

•  How can I provide direct and constructive feedback?


Mobile

Tester End Users Developer

Over the air build distribution 1 In app bug reporting 2

Crash log reporting

3

In app user feedback 4

LOB/Digital Marketer

Sentiment Analysis 5

Builds

User Feedback Crash logs

Bugs Bugs vs. Crashes

IBM Mobile Quality Assurance Continuously deliver high quality mobile apps


Mobile





5.  Focus on the user’s experience



Mobile


Mobile

© Copyright IBM Corporation 2012. All rights reserved. The information contained in these materials is provided for informational purposes only, and is provided AS IS without warranty of any kind, express or implied. IBM shall not be responsible for any damages arising out of the use of, or otherwise related to, these materials. Nothing contained in these materials is intended to, nor shall have the effect of, creating any warranties or representations from IBM or its suppliers or licensors, or altering the terms and conditions of the applicable license agreement governing the use of IBM software. References in these materials to IBM products, programs, or services do not imply that they will be available in all countries in which IBM operates. Product release dates and/or capabilities referenced in these materials may change at any time at IBM�s sole discretion based on market opportunities or other factors, and are not intended to be a commitment to future product or feature availability in any way. IBM, the IBM logo, Rational, the Rational logo, Telelogic, the Telelogic logo, and other IBM products and services are trademarks of the International Business Machines Corporation, in the United States, other countries or both. Other company, product, or service names may be trademarks or service marks of others.

five ways to improve your mobile testing

Technology

mobile consumers

mobile users

mobile site

mobile state

mobile imperative

t22 mobile testing

mobile testing dennis

mobile ad projects