performance testing in mobility-2 - techarcis solutions · performance testing in mobility is both...

Performance Testing in Mobility


Driving QualityInspiring Transformation

CHAPTER

02

CHAPTER

03

CHAPTER

01 Performance Testing in Mobility ................................................................................... 03

Importance for Performance Testing for Mobile Applications ............................ 03

Consequences Ignoring Performance Testing .......................................................... 03

CHAPTER

05

CHAPTER

06

CHAPTER

04 Challenges with Mobile Performance Testing .......................................................... 03

Classification of Applications which undergo Performance Testing ................. 04

Ideal takeaways for an Application undergoing Performance Testing ............ 04

CHAPTER

08

CHAPTER

07 Common approach to test Mobile Application ......................................................... 05

Tools assisting Performance Testing in Mobility....................................................... 06

Topics Covered

Confidential and Proprietary

3


Importance for Performance Testing for Mobile Applications

To break through the performance of mobile applications, one has to break through the technology being used.

After all, it’s not the display but the performance that matters.

Studies say that the number of mobile users reached over 1800 million by the end of 2014, thus surpassing the

number of desktop users. Even the global mobile traffic rose swiftly to 11% of total Internet traffic. This rapidly

growing use of mobile applications makes Performance Testing necessary for smooth operating in this mobile era.

Performance is the bridge for cordial User Association. Mobile users look forward to sturdy performance and

serviceability on the move. As per WQR report, the enormous increase in mobile users has also impacted the new

development resources, about 34% of them are focusing on mobile and customer guide solutions in the year

2015-16. So, the need of the hour is to emphasize on End-to-End Mobile Performance Verification. The focus on

Mobile Performance QA is expected to be based on 3 major specs:

first is emphasizing on measuring the mobile end user experience, so that the performance is skewed towards

end user requirements;

second is the app should execute on arrays of devices and networks to ensure seamless performance across

different platforms and parameters;

and lastly, to verify the impact of server load on performance of the apps so that there aren’t any hitches with

exponentially increasing mobile traffic.

Consequences Ignoring Performance TestingConsequence of excluding Performance Testing can be hazardous and lead to unsatisfied customers and downfall

of business. Depending upon just functional testing is an unreliable option for any full-fledged intended testing.

Though functional test scenarios provide drilled down testing, they don’t include testing in scenarios of increasing

loads and traffics for each mobile, which later on may lead to slow loading time, page crashing in the middle of

transaction etc. All these will have a major adverse effect on the conversion rate of the app. This is some of the ROI

justification that compels us to think that it is pretty much advisable to invest into a mobile performance test project.

Challenges with Mobile Performance TestingThough the desktop applications and mobile apps use the same underlying technology, common testing strategies

are limited and can’t be adequate to test the mobile apps. Some existing challenges faced by the domain of mobile

performance testing are:


4

Difference in bandwidths for wireless protocols: The bandwidth has an effect on response time of the end user

as well as server. Thus, anticipating the effect of bandwidth variation is quite challenging.

Recording test scenarios especially for secured apps that use HTTPS: This is still an open challenge for testers.

Before running a test, Android mobile clients are expected to put their device in recording ready mode and the

tab settings has Playback on Instrumented app option, which needs to be selected. No such configuration is

required for iOS mobile clients.

Need for more Shift Left Testing: With the inclusion of fuzzy logics in the app schemas, more shift left testing will

upgrade the performance of mobile applications.

Emphasis on Network Virtualization Test: This is driven by the fact that the range of users varies from residents

of urban areas to those of rural areas. Other than this scenario, even the user use their mobiles in elevators and

in tunnels, which makes it necessary to ensure a flawless experience that the test should emphasize upon.

Classification of Applications which undergo Performance TestingBeing pro, we are aware of basic types of mobile applications, i.e. Native Applications, Hybrid Applications and

Web-based applications.

Native applications or the client server applications which gets entrenched into your smart phone device, Native

Applications vouch for secured applications and faster access.

Web based applications are accessible by web browsers even by smart phones.

Hybrid Applications as the name suggests are combination of Native applications and Web based applications,

all the platform specific tasks or transformations are performed at Native App then, and few applications are

directed to web based application for publishing the outcomes.

Talking of response time, response time is computed for all types of applications. For Hybrid Apps and Web Apps,

response time is calculated involuntarily. However for native apps, Response time is based on Synchronization

policies. But again the question is when is a native app synchronized? The answer is when the response for native

app is being received and the app waits on for user action. During testing, synchronization policies are set

automatically for each step.

Few synchronization policies are

Wait for idle where progress indicators are ignored and the playback awaits for new user’s input,

Wait not busy which points that the playback waits until the background task especially the progress indicator

is complete and

Do not wait when response time calculation is independent.

Ideal takeaways for an Application undergoing Performance TestingIt is very difficult to assess any application before performance testing but making a log of some specific information

from app during runtime can make the testing more perfect:


5

Crash and Error Reports: to determine root cause of any issue

Device performance (CPU usage, memory usage, etc.)

Application response time

Server response time

Demographic data (operating system, device type, carrier, etc.)

To determine if the crashes are due to OS issue or device issue or systematic issue due to any specific system or

interactions of many systems, capturing following information will be helpful:

Operating System and version

Device Type

Network Carrier

Connection Type (3G, 4G, Wi-Fi, etc.)

Application Version

User Geography

This granularity of information will be of great help to conduct efficient Performance testing.

Common approach to test Mobile ApplicationPerformance testing of mobile application starts with comprehending the user load profile. Users accessing any app

have options like variances in modem systems, UI, device performance, network conditions. So, sampling of the

user profile is carried out.

Another most important stride is maintenance of performance metrics through continuous feedback loop. This step

demands study of choices of sample customers thoroughly, what is their common paths, variance in load

conditions, the type of access the users enjoy in terms of native app, mobile site, full site etc.

Network Virtualization is another approach to test performance of mobile applications. Using real devices,

characteristics of a particular network is simulated, especially the bandwidth and latency. Even the mobile

conditions are altered from good to poor and to complete network loss. To simulate network conditions, load

generators are put behind network virtualization open source proxy tools.

When performance testing is to be carried out in cloud, the best approach is ‘Cloud-in-a-box’ approach which is

speeding at a good pace which enables to set up a virtual testing lab, though its assess is limited within a company’s

corporate network. The lab offers inside the firewall access to local cloud of smartphones. Cloud based performance

testing is done to test the performance of any application for users in diverse geographic locations. Cloud based

solutions for performance testing generate load from same geographic locations that the users are coming from. In

practice, a large percentage of loads are generated and then user experience on real devices is tested while the

application is under load.


6

Tools assisting Performance testing in MobilityMost of the companies including the top layered ones like HP, IBM indirectly proceed for mobile test automation

by partnering with niche and small vendors to provide overall solutions.

Following is the overview of the tools with the performance parameters:

Performance

Parameters

Overview

Application type

support

Recording any Mobile

App

Network Virtualization

Performance testing tool

for mobile and web

apps, covers all

performance testing

from component tests to

hybrid cloud tests.

Native and Hybrid

Applications

Direct

Yes Yes No

Yes Yes No

Yes Yes N/A

Yes

Yes

Yes Yes

Yes(with extension) Yes

9/10 10/10 9/10

Very Active Very Active Active

App & device both App & device both App & device both

Java N/A N/A

More than 30+ Gbps More than 30+ Gbps 30+ Gbps

Direct in handset Indirect using recording

proxy

Native applications

Perfect to strengthen the

digital interaction with

quality first approach. It

is a hardware and

software solution.

Native applications

Appropriate tool for

testing performance of

real world conditions

using networks around the

world. Efficient in testing

even the complex script

application.

Neotys Perfecto Mobile Apica

Geographic realism

creation

Bandwidth capability

Simulators/Emulators

Real Devices

Community Support

Supported context

Scripting Language

Supports Visual Objects

Technical support

Driving QualityInspiring Transformation

TechArcis Solutions, Inc.Quality Assurance l Testing Transformation l Outsourcing

101 Colony Park Drive, Suite 300, Cumming, GA 30040, USAwww.techarcis.com l [email protected] l Tel: 770-415-4815 l Fax: 770-415-4816

ConclusionPerformance testing in mobility is both creative and challenging. It develops a relationship between the virtual

aspects i.e. the technology and the real aspect i.e. the device of mobility. And the outcome evolves as a dialogue

between the two. The following sums it up beautifully: ‘An ounce of Mobility performance is worth pounds of

challenges.

Other than these tools, there are Automation tools assisting Performance testing in mobility:

- Robotium: It is Android test automation framework that supports native and hybrid applications. It is an open

source automation tool. It works with Android JUnit Eclipse platforms and can also be executed using

instrumentation test runner using a utility called adb.

- Squish: It is cross-platform/cross-technology GUI test automation tool for performance GUI tests. It automates

data driven testing and BDD & ensures smooth testing.

Mobile Testing tools assists in building test case scenarios for mobile devices. They even give an option to execute

and record automated UI for mobile apps and assist in testing responsive and fluid web interfaces.

Stay Connected Join Our Mailing Listwww.techarcis.com

performance testing in mobility-2 - techarcis solutions · performance testing in mobility is both...

Documents