seven elements of a mobile test strategy

7/27/2019 Seven Elements of a Mobile Test Strategy

http://slidepdf.com/reader/full/seven-elements-of-a-mobile-test-strategy 1/13

DEPLOYMENT. ASSURED. ™

SEVEN ELEMENTS OF A

MOBILE TEST STRATEGY

An Olenick & Associates White Paper

July 2013



Deployment. Assured. ™

Contents

Executive Summary .................................................................................................................... 1

Mobile Vision .............................................................................................................................. 3 QA Methodology and Process .................................................................................................... 4

Deployment Process ................................................................................................................... 5

Information Security ................................................................................................................... 6

Mobile Development and Test Environment............................................................................... 7

Test Tooling ................................................................................................................................ 8

Team and Skills ......................................................................................................................... 10

Conclusion ................................................................................................................................ 11

About Olenick & Associates ...................................................................................................... 11

Exhibits

EXHIBIT 1 SEVEN ELEMENTS OF A MOBILE TEST STRATEGY ................................................................ 2

EXHIBIT 2 MOBILE VISION DECISION POINTS ..................................................................................... 3

EXHIBIT 3 QA METHODOLOGY AND PROCESS DECISION POINTS ......................................................... 4

EXHIBIT 4 DEPLOYMENT PROCESS DECISION POINTS ......................................................................... 5

EXHIBIT 5 MOBILE SECURITY DECISION POINTS ................................................................................. 6

EXHIBIT 6 MOBILE TEST ENVIRONMENT DECISION POINTS.................................................................. 7EXHIBIT 7 TEST TOOLING DECISION POINTS ...................................................................................... 8

EXHIBIT 8 MOBILE TEST AUTOMATION TOOLS .................................................................................. 9

EXHIBIT 9 STAFFING DECISION POINTS ........................................................................................... 10

Copyright © 2013 Olenick & Associates, Inc. All rights reserved.

205 W. Wacker Drive, Suite 2000

Chicago, IL 60606

312 332 0096

http://www.olenick.com

@OlenickIT

Additional copies of this white paper can be downloaded from

http://www.olenick.com/

Seven Elements of a Mobile Test Strategy



http://twitter.com/olenickit









EXECUTIVE SUMMARYWhile the importance of mobile applications rapidly grows for both consumers and enterprises,

mobile information technology presents significant new risks for IT professionals and their

organizations. An explicit mobile quality management strategy can offset these risks.

CIOs and IT managers cannot assume that traditional quality practices will achieve the same

stability and predictability for mobile apps as they have for traditional systems.

Most organizations will have to stretch to design effective mobile apps and then establish

entirely new development environments, processes, and players to produce them. But that isn’t

all that’s needed. An effective, organization-wide mobile test strategy can offset the new risks

of mobile IT and help to realize the promise of this transformative technology.

This white paper offers a checklist for planning your mobile testing initiative:

1. Mobile technology vision

2. Quality assurance methodology and process

3. Deployment process

4. Mobile development and testing environment

5. Information security

6. Tooling

7. Team and skills

These elements, listed in Exhibit 1, reflect lessons we’ve learned in testing many mobile apps

and working with our clients as they do the same.

In the following sections, the key decision points for each element are discussed. Your answers

to them will outline a mobile testing strategy for your organization.

Seven Elements of a Mobile Test Strategy 1




Exhibit 1 Seven Elements of a Mobile Test Strategy

• User needs

• Business objectives

• Product roadmapMobile Vision

• QA methodology

• QA process

• Metrics, standards, templates

QAMethodology &

Process

• Development process

• Deployment process

DeploymentProcess

• Mobile devices and connectivity

• Server-side environment

• Maintenance and support process

MobileEnvironment

• Information security policy

• Mobile device policy

• Mobile device management

InformationSecurity

•Configuration managmenet

• Testing and test management

• Deployment managementTooling

• Staff experience

• Staff core skill set

• Training programTeam & Skills





MOBILE VISIONA mobile vision defines key business

drivers for using mobile technology.

The mobile vision defines a general

approach to mobile application

development and support, a mobile

application portfolio, and targeted

mobile platforms and devices.

A mobile test strategy based on it will

be aligned with the company’s

direction as well as current and future

testing needs.

Exhibit 2 lists key decision points toconsider in defining a mobile

technology vision.

Mobile Vision Decision Points

1 What are the business drivers for developing mobile application(s)?

2 What mobile applications are planned?

3 What platforms will be supported and why?

3 How will the mobile applications be developed and supported?

4 Who are the users and stakeholders for the mobile applications?

5 What are the stakeholders’ success criteria for the mobile applications?

6What level of risk is the organization willing to accept with the deployment of mobile

applications?

7 How critical is the quality of the applications?

Exhibit 2 Mobile Vision Decision Points

Mobile

Vision

Users

BusinessObjectives

Product

Roadmap





QA METHODOLOGY AND PROCESSYour organization’s current quality assurance (QA) methodology and processes should be

reviewed in light of how you plan to develop, test, deploy, support, and maintain mobile apps.

In response to the urgent need to build and deploy mobile applications, many enterprises are

relying on new vendors and technologies. Existing QA methodology and processes should be

revised to align your approach these new resources.

Mobile projects can be measured, monitored,

and reported using existing test management

tools, reports, and dashboards. However,

some adjustments will probably be needed to

get meaningful feedback about mobile app

quality. For example, you may want to closely

monitor the “stars” your apps garner in

Google Play or iTunes, in addition to

traditional quality metrics.

QA Methodology and Process Decision Points

1 What is the development process: agile, waterfall, or your own?

2 What are the QA processes that are currently in place?

3 What testing standards and governance models need to be followed?

4Are your QA methodology and processes aligned with the mobile development

process?

5If vendors are being used, are their processes integrated or aligned with the

organization?

Exhibit 3 QA Methodology and Process Decision Points

“You may want to closely

monitor the “stars” your

apps garner in Google

Play or iTunes, in additionto traditional quality

metrics.





DEPLOYMENT PROCESSAn organization’s traditional build and deployment process will need to be extended to support

mobile applications. In particular, it is important to define an efficient and scalable build/

deployment process for systems that span server-side components and mobile platforms.

A mature mobile application deployment process effectively supported by application

development and complete platform testing is vital for:

• Managing and controlling application

updates.

• Improving the user experience.

• Increasing user adoption of the

application.

• Reducing application user support.

Deployment Process Decision Points

1 How are developers managing and delivering code?

2Will this process accommodate mobile applications or is a new approach and

infrastructure needed?

3 What is the current deployment process?

4Can it be adjusted to accommodate the mobile applications or is a new process and

related infrastructure required?

5For native mobile applications, how will they be deployed to mobile devices in the test

and in the production environments?

Exhibit 4 Deployment Process Decision Points

“Define an efficient and

scalable build/deployment

process for systems that

span server-side

components and mobile platforms.





INFORMATION SECURITY

Traditional enterprise applications run on

enterprise desktops / laptops and are typically

secured following an enterprise IT security policy.The expansion of enterprise applications into the

mobile space and flexibility now given to

employees to connect with mobile devices

presents significant new challenges to

information security.

• Support for BYOD (bring your own device) and access to core enterprise applications

outside of a secure corporate network creates exposure to many new kinds of attacks.

• The physical security that fixed location desktop and server computers provide cannot

be assumed. Mobile devices are routinely lost, stolen, or surreptitiously compromised

outside of an organization’s premises.

• Mobile malware and mobile-specific social engineering now pose threats at the same

level as web and desktop exploits.

It is therefore essential to consider security policy, implementation, and verification early in the

mobile development and test planning process. The National Institute of Standards and

Technology provides many recommendations for securing mobile technology.* We see three

additional device management policy issues:

• How will Mobile Device Management (MDM) support authentication, auto-lock,

application installation, and remote restore/clean for a fleet of corporate and BYOD

mobiles?

• What is data management policy for data encryption and persistence?

• What is data access policy including roles, logging, auditing, and remote wiping?

Mobile Security Decision Points

1 Does the IT security policy address mobile applications and devices?

2 What is the security concerns associated with the current mobile applications? Future?

3 Will the enterprise applications be allowed on personal devices?

Exhibit 5 Mobile Security Decision Points

*Related publications may be downloaded from

http://csrc.nist.gov/publications/PubsTC.html#Communications & Wireless

“ Consider security policy,

implementation, and verification early in the

mobile development and

test planning process.


http://csrc.nist.gov/publications/PubsTC.html%23Communications%20&%20Wireless






MOBILE DEVELOPMENT AND TEST ENVIRONMENT

A mobile test environment has different characteristics than traditional desktop and web

application test environment(s).

Not only does the mobile environment

need to integrate with existing systems for

end-to-end business flow testing but it also

needs to accommodate mobile

compatibility and connectivity

requirements. Exhibit 6 lists mobile testing

environment decision points.

Exhibit 6 Mobile Test Environment Decision Points

It is usually not possible to test even a small subset of every combination of device type, form

factor, platform (OS), locale, and carrier. We’ve found that a hybrid approach is practical:

provide your own basic set of device form factors and platforms, then as necessary, use cloud-

based device sharing and crowd-sourced mobile user experience services.

“ We’ve found that a hybrid

approach is practical.

Mobile Test Environment Decision Points

Physical Devices Actual physical devices.

Device Emulators Software that simulates actual devices.

Mobile Web Browser

EmulatorsBrowsers that simulate a mobile browser environment.

Cloud ServicesMobile cloud service vendors provide fee-based remote

desktop access to a wide range of devices and carriers.

Crowd-sourced ServicesCrowd-sourced services provide rapid feedback from thousands of

users over many devices and locales.





TEST TOOLINGMany new design, programming, and

deployment tools are needed for mobile apps.

A wide range of new mobile-specific tools and

services are available to automate functional

and performance testing. These tools are

typically platform-specific, so any

organization developing for multiple mobile

platforms (iOS, Android, mobile web, etc.) will

have to resolve questions of coverage and

integration.

Existing test management tools can support hands-on functional testing and user-experience

evaluation. Configuration and process management tools like Jenkins and Jira or their

commercial counterparts may be used to good effect for mobile app development.

Testing tools—newer mobile tools in particular— tend to be technology islands. To establish a

complete environment, it’s useful to consider their fit with your overall tool chain, as Exhibit 7

suggests.

Well-defined criteria to evaluate tooling for functional and performance testing can lead to a

configuration that best supports your mobile vision. We’ve found that an incremental approach

with proof of concept and pilot projects is a good way to find the best fit.

Tooling Decision Points

1 What tools will be used to support requirements, be they user stories or line items?

2 What tools will be used for test management and bug tracking?

3 What tools will be used to support the configuration management process?

4 What tools will be used to support the deployment process?

5 For automation and performance, what are the current supported platforms? Future?

Exhibit 7 Test Tooling Decision Points

“ An incremental approach

with proof of concept and pilot projects is a good way

to find the best fit.





There are many choices to make once you’ve settled on an environment. Exhibit 8 lists mobile

functional testing tools that we have used in our mobile testing lab.

Tool Hosting Mobile Platforms TestingSupported

Borland

SilkMobile

Desktop Android, iOS, Blackberry, Windows

Phone

Functional

Device

Anywhere

Desktop and Cloud Android, BlackBerry, iOS, Windows

Phone

Functional,

Performance

Jamo M-uex Desktop Android, BlackBerry, iOS,

Windows Phone

Functional

Perfecto Mobile Cloud Android, iOS, Blackberry, Windows

Mobile, Symbian

Functional,

Performance

Ranorex Desktop Android, iOS Functional

SOASTA Cloud and local Android, iOS Functional,

Performance

ZapFix Cloud Mobile Web, Android, BlackBerry,

iOS, Windows Phone

Functional,

Performance

Exhibit 8 Mobile Test Automation Tools





TEAM AND SKILLS

To define, manage, and implement a robust

and scalable mobile strategy, it is critical

that the right people are involved and givennecessary training.

Mobile testing requires strong software

testing skills along with a mobile-savvy

mindset.

When assembling a testing team, tester

background, skill, and testing specialties should be taken into account. If testing will be

automated, testers adept in test environment design, development, maintenance, test data

selection, and experienced in the selected automation tool should be included in the team.

We’ve found it is equally important to get the right cross-section of users involved. They can

communicate the broadest spectrum of user scenarios to drive effective functional,

performance, and usability testing.

Staffing Decision Points

1 What level of experience does your staff have with mobile testing?

2 How technical is the existing testing staff?

3 What is the training approach and budget?

4 Do you have a mobile test strategist?

Exhibit 9 Staffing Decision Points

“ We’ve found it is equally

important to get the right

cross-section of users

involved.





CONCLUSIONCIOs and enterprise IT managers can establish a broad and robust strategy to manage risk as

mobile apps are added to an IT portfolio. A comprehensive mobile test strategy can enable your

organization to deliver, test, and support high-quality and high-value mobile apps. This can

also help to right-size mobile testing budgets and staffing.

A mobile test strategy should address process, technology, and staffing requirements. It should

guide and empower project teams to plan effective application-specific functional and

performance test plans. It should also define an IT organization’s development, deployment,

and environment management approaches for mobile apps.

The seven key elements and their decision points discussed in this white paper provide a

framework for such a mobile testing strategy.

ABOUT OLENICK & ASSOCIATESWe prepared this white paper to help CIOs and IT managers understand how to set an approach

for mobile app testing.

In our experience, many managers underestimate this challenge. To find out more about how

Olenick & Associates can support your mobile testing initiatives, please contact us at 1 (312)

332-0096 or via email at [email protected].

To learn more about our full range of deployment assurance services, please visit us at

http://www.olenick.com/.


mailto:[email protected]?subject=The%20Seven%20Elements%20of%20a%20Mobile%20Test%20Strategy







seven elements of a mobile test strategy

Documents