MONITORING PE

RFORMANCE

AUT

OMAT

ION

DEVELOP BUILD TEST

DEP

LOY

M

ONITOR

Embed quality and accelerate development throughout your mobile SDLC

ContinuousQuality

Deliver Better Mobile Apps Faster with

2

MONITORING PE

RFORMANCE

AUT

OMAT

ION

DEVELOP BUILD TEST

DEP

LOY

M

ONITOR

Table of Contents

Defining Continuous Quality ....................................................................................................................................................................5

Mobile is Now Mission-Critical ................................................................................................................................................................6

Components of the Continuous Quality Lab ..............................................................................................................................................11

The 4 Phases of Continuous Quality ........................................................................................................................................................18

Summary ..............................................................................................................................................................................................23

3

4

Executive SummaryMobile apps have dramatically changed the way users interact with brands. But with the rapid pace of mobile hardware and software innovation, many organizations struggle to meet user expectations. These challenges are well understood, it’s the solutions which have proven to be elusive. In order to ensure application quality in the modern age of mobile, a new approach is required. But what?

We believe that the answer can be found in the principles and practice of Continuous Quality (CQ). Written in plain language – but with enough technical details to satisfy seasoned development and QA professionals – this short book provides a definition of Continuous Quality in theory and gives a practical overview on how to implement its methodology.

Topics Covered:

• New Dynamics and Challenges of Mobile Apps

• An Introduction to the Principles of Continuous Quality

• Components of the Continuous Quality Lab (CQ Lab)

• The 4 Key Phases of Continuous Quality in Practice

5

Continuous Quality (CQ) is a methodology for embedding quality activities into every step of the SDLC process from design through build to production.

CQ shrinks time-to-market, drives faster release cycles and minimizes escaped defects to production.

Definition

MT

6

Mobile adoption is growing 8 times faster than web adoption did in the 1990s and 2000s

Today, 60% of online traffic now originates from mobile devices

More than 80% of mobile time is spent within native apps

73% of Millennials are already transacting directly on their mobile devices. 75% of Millennials have their mobile devices glued to their palm while in store as a trusted personal shopping assistant.

No Longer a Nice-To-Have, Mobile is Now Mission-CriticalMobile is becoming the main product platform for enterprises and is driving SDLC transformation

This time, it really is different. We’re talking, of course, about the rapid adoption of mobile devices and applications – a claim backed up by the numbers:

60%

8x

7

These are just a few (of many) recent statistics that have led brands to reach two important conclusions:

1. Mobile is now the preferred consumer brand engagement point.

2. As consumer preferences change, so too must corporate business models: keep up or fall behind.

This brief book is a primer for those seeking to stay ahead of the innovation curve by adopting the principles of Continuous Quality.

We wrote this as a guide for Dev, QA, Operations and other departments in charge of application development and quality, but believe it offers useful insights for anyone involved in the design or production of mobile apps.

8

New Dynamics = New Dev/QA StrategiesNew dynamics raise the stakes when balancing velocity and qualityTime is not on the side of today’s Dev and QA teams – they must adapt to a number of emerging dynamics including:

Rapid Innovation New devices, OS enhancements, wearables, sensors and third-party plug-ins are continuously being released into the market.

Mobile app development teams must constantly update their release plans to ensure compatibility with a growing number of variables.

These external pressures have compressed release schedules to weeks instead of months.

Feedback

Cloud Deploy

Client Deploy

Story backlog

Code Committed

Code BuiltVer X+1

Client in-AppDiscovery Upgrade

Publish &Review

In-storeGeneral

Availability

9

44%

44% OF DEFECTSARE FOUND BY USERS

24% Direct User Feedback

20% Public Users Reviews

“After two bad experiences, I’ll try something else”Users uncover half of application defects

http://info.perfectomobile.com/wpr-why-apps-fail.html

Skyrocketing User Expectations Typical commerce websites will fully load pages in about 6-10 seconds. Mobile users expect faster results. Although few enterprises are achieving Google’s oft-cited one second (above the fold) content load target, studies show that users will quickly abandon applications with any response delays.

Increased Complexity The desired mobile experience is dependent on a combination of factors – battery, memory, CPU / GPU, network connectivity, screen size, sensors and even app style (Web vs. Native). Ensuring quality in mobile environments requires enhanced testing, complex analysis and client side visibility.

10

Continuous Quality Is NOT Another BuzzwordTo effectively handle the increased velocity, quality and complexities associated with mobile development, industry leaders are turning to an innovative practice: Continuous Quality.

Continuous Quality is a methodology for embedding quality-activities into every step of the SDLC process from design to deployment based on automated processes, tools and a lab infrastructure tailored for mobile.

Successful Continuous Quality optimizes time-to-market, drives faster and more frequent releases and minimizes the defects that escape to production.

The Continuous Quality (CQ) methodology is built on the notion of a robust test lab capable of supporting the full end-to-end development, continuous integration and delivery cycle.

It specifies a model in which a single lab solution is capable of supporting all aspects of testing – unit, functional, build verification, performance testing and proactive UX monitoring. In addition, it is fully accessible to every tool and user in a company’s application development workflow. The result is an automation-ready iterative dev/test workflow capable of supporting a full range of mobile initiatives.

11

Anywhere, anytime, on-demand access to a CQ Lab capable of supporting all development activities from end-to-end is critical to the success of a CQ Program.

The CQ Lab provides the required infrastructure and tools to conduct tests, measurements and analysis which are fully embedded and integrated into the Agile / DevOps process and able to replicate end user conditions throughout the lifecycle.

The CQ Lab supports a continuous, 24/7 operation with a production grade SLA and consistency across the different phases in the lifecycle, allowing reproducibility and minimizing time-to-resolution to maximize time-to-market.

The key building blocks of an effective CQ Lab solution for mobile app development include:

1. Real environments

2. Real devices

3. Coverage across the SDLC

4. Integration with existing Tools

Making Continuous Quality a Reality

Key building blocks of an effective CQ Lab solution includes

1. Real Environments

2. Real Devices

3. SDLC Coverage

4. Tool integration

12

As mobile requirements increase, companies face greater challenges in the way they develop, build, test and deploy end-user applications – forcing them to:

Rethink Methodologies Enterprises that currently base their entire SDLC on traditional waterfall or pre-agile practices are quickly realizing that such solutions, which worked well in the web and desktop era, are inadequate in today’s mobile space.

Evolve Testing Coverage means more than just the code. For mobile, coverage necessitates implementing a device coverage strategy that addresses the current Android and growing iOS market fragmentation.

Effective app testing requires mimicking real-world conditions including varied network conditions, device stress (e.g., background streaming media apps) and back-end load.

Dev teams therefore need access to a variety of devices running varied OS profiles, actively connected to networks in target geographies.

What Got You Here Won’t Get You There EXPERTSAGREE

“The rapid pace of change in the mobile market is putting pressure on development and operations teams to adopt rapid development and deployment practices that constantly iterate their mobile applications as expectations change.”

– Gartner, Traditional Development Practices Will Fail for Mobile Apps, 2014

13

Eliminate Obstacles Companies looking to capitalize on mobile opportunities cannot afford barriers that impede the velocity of development – especially those created by ineffectual internal processes.

Typical challenges that organizations must overcome to effectively manage mobile app development include:

• Insufficient test coverage due to time constraints

• Lack of stability and test consistency of the in-house test lab

• Inability to enable cross-team collaboration

• Gaps in production feedback being passed to product teams

“Poor app quality is the primary reason behind the weak adoption, usage and even failure of many mobile apps. Enterprises that are developing mobile apps need to employ mobile app testing tools and services to help mitigate the risks associated with testing increasingly sophisticated contextual mobile applications.”

– Gartner, IT Market Clock for Enterprise Mobility, 2014

EXPERTSAGREE

14

CQ Lab Component #1 Real EnvironmentsDesign and development teams cannot effectively create mobile apps without an understanding of how server-network-device “wind tunnel” load factors impact performance.

The CQ Lab must be able to execute tests on actively connected devices capable of reporting data from different network profiles (3G, LTE, WiFi) as implemented across the globe.

Server Domain

Back-end Servers

3rd Party / CloudServices

ISPs Customers

Web Services Content DeliveryNetworks

Mobile Carriers EmployeesEnd-User Devices

Network Domain Device Domain

15

CQ Lab Component #2 Real Devices One of the greatest challenges for CQ initiatives is ensuring that tests can be executed on a wide range of device/OS profiles.

Not only must the CQ Lab provide robust device pools, it must also ensure quick access to the newest hardware and operating system releases to keep proactive, externally-driven release plans on schedule.

MT

This is doubly true for iOS updates.

Upwards of 50% of Apple users will upgrade to the latest operating systems within the first 48 hours

of availability.

16

CQ Lab Component #3 Coverage Across the SDLC The CQ Lab must be able to support all test scenarios from sprint zero to post-production including individual unit tests, functional testing on target devices, load testing for of production readiness and proactive user experience monitoring on live devices.

Those evaluating mobile apps must be able to provide both object properties on all elements as well as visual results that allow testers to see exactly what the user sees on their device. With consistent results from an SLA-backed test environment, operations and development groups can more effectively collaborate throughout the SDLC.

17

CQ Lab Component #4 Integration With Existing Tools Test labs that require users to leave their known environments or learn new skills inherently restrain the velocity of a Continuous Quality initiative.

To be effective, the CQ Lab must provide seamless, out-of-the-box integration with an organization’s existing development stack including open source and commercial programming languages, IDEs, CI servers and test frameworks.

The days of enterprises having a single standard tool set is waning. Open source tools are changing the game and development teams expect integration with industry standard interfaces as table stakes.

CQ requires integration with market leading quality tools

18

In Practice The 4 Phases of Continuous Quality CQ Enhances Every Phase of Software DevelopmentIntegrated quality enhances every aspect of software development, including design, code, debug, build, stage, deploy and production management.

Regardless of individual implementation variances, CQ provides overarching benefits across the SDLC.

19

Phase #1 Development and Debugging Fast feedback Fuels Velocity: Mobile apps behave differently when run on different device/OS profiles and across varied network and location conditions which can introduce additional performance and usability issues.

Therefore, developers need actionable data on performance impacting factors as early as possible during the app development phase.

To provide developers with immediate feedback throughout the development process, build verification testing on target devices must be integrated into the organization’s chosen development environment.

Executing unit tests on actual devices not only ensures early defect detection, but also provides critical data on device vitals, network bottlenecks, response times and back-end connectivity.

From a workflow perspective, CQ best practice recommends integrating test code development and execution into the developer’s preferred IDE (Eclipse, Android Studio, Visual Studio, etc.).

It is also important to select the CQ Lab environment that supports the programmer’s preferred programming language (C#, Java, Ruby etc.) to reduce learning curves. Any incompatibility that delays the dev/test feedback loop will impact quality and/or delay release schedules.

20

Phase #2Code Builds Automation Drives Efficiencies: Because Continuous Integration (CI) is critical to agility, organizations need to integrate mobile testing into their individual code build process. To ensure efficiency, CQ practice recommends integrating the CQ Lab tests into CI process, regardless of which server – Jenkins, Team City, TFS – is used.

When developers commit code and trigger the build process, they need a full report to quickly correct any issues that may impact user experience. Reports should include both test details (device OS logs, device vitals, network conditions, back-end delays) as well as visual recordings that enable developers to easily pinpoint the source of any failures.

A key piece of any CQ program is the ability to reproduce problems and ensure that they have been corrected before changes are recommitted into the master code repository.

Because the CQ Lab supports the entire SDLC, developers can take any test code created during the during development and automatically include it in the build verification process. Even if teams have to perform additional interoperability manual tests to assure specific functions, a true CQ Lab solution will support those capabilities.

MT

21

Phase #3Integration & Pre-Production Scalability Ensures Readiness: In a CQ workflow, developers (either Android or iOS) perform code commits into a master code repository (SCM like Git, SVN, etc.) and each performs local build acceptance testing and debugging through unit tests and other verifications, usually on a small number of mobile devices.

As development progresses to the point where the quality and robustness of the mobile app is expected to deliver against all functional specifications, any artifacts must be tested at scale for higher functionality and performance. To ensure quality under tight deadlines, the CQ Lab should serve as the common foundation for automating and performing these tests.

With seamless support from within the build environment for functional and non-functional performance testing, standard industry tools such as Selenium, HP UFT, Appium and Calabash, allow teams to validate quality and performance at scale across multiple device/OS/network profiles. Without this level of testing, companies cannot verify the release readiness of a mobile app with confidence.

22

Phase #4 Proactive Assurance Preempts Complaints Proactive Assurance Preempts Complaints: Production is a critical phase which can determine business success and end-user adoption.

To stay ahead of issues, the product team needs pro-active user experience monitoring embedded into the CQ process. Such capability will assure that the team gets production insights and real time alerts as soon as something goes wrong.

When proactive monitoring is performed on real devices in a governed, secured CQ Lab, mobile DevOps teams have full control of the entire workflow from the early warning through resolution, typically before end-users are impacted.

23

EXPERTSAGREE

Continuous Quality A Framework For GrowthSummaryAs development work evolves to encompass mobile environments, technical and business challenges mount. Release cycles accelerate and user expectations climb.

Poor app quality undermines adoption, conversion and ultimately becomes a barrier to the achievement of business objectives.

To meet rapid release schedules without compromising quality, it is imperative that development groups transition to CQ practices by employing a robust CQ Lab. This provides a consistent quality foundation across the entire Build-Test-Deploy workflow. More importantly, it ensures that performance issues do not impact adoption, conversion or innovation KPIs.

Recommendations

EMBRACE THE MOVE TO“CONTINUOUS QUALITY”

In this first decade of mobile mass adoption, many developers have snuck by without comprehensive approaches to quality. With app and device complexity increasing, and consumers less forgiving of clumsy mobile apps, quality is quickly moving from a position near the end of the development life cycle to an embedded and continuous process throughout the

cycle.

Source: Forrester, Market Overview: Mobile app

Testing, 2014

24

Perfecto Mobile provides a hybrid cloud-based Continuous Quality Lab that enables mobile development and testing teams to deliver better apps faster.

The Continuous Quality Lab supports testing processes earlier and more often in the development cycle, giving way to faster feedback and improved time-to-market. Users can access an exhaustive selection of real mobile devices connected to live networks around the world and leverage them for testing and monitoring throughout the mobile application development lifecycle – from development, functional and performance testing to monitoring and support. More than 1,500 customers, including 50% of the Fortune 500 across the banking, insurance, retail, telecommunications and media industries rely on Perfecto Mobile to deliver optimal mobile app functionality and end user experiences, ensuring their brand’s reputation, establishing loyal customers, and attracting new users. For more information about Perfecto Mobile visit www.perfectomobile.com or follow us on Twitter at @PerfectoMobile.

About Perfecto Mobile

25

Notes

© Copyright is 2015, Perfecto Mobile Inc.

Deliver Better Mobile Apps FasterMobile apps have dramatically changed the way users interact with brands. But with the rapid pace of mobile hardware and software innovation, many organizations struggle to meet user expectations. These challenges are well understood; it’s the solutions which have proven to be elusive.

We believe that the answer can be found in the principles and practice of Continuous Quality.

This short book provides an introduction to Continuous Quality for the mobile executive, developer and QA manager including an overview of the framework behind Continuous Quality and a practical methodology for implementation.

About Perfecto MobilePerfecto Mobile, the world’s leader in mobile app quality, provides a hybrid cloud-based Continuous Quality Lab that enables mobile app development and testing teams to deliver better apps faster.

Learn more at www.perfectomobile.com