Dynamics of Mobile Application Testing

Main Author Name: Shinny Singla Designation: Technical Leader

Co – Author Name: Pradeep Bhandari Designation: Technical Leader

Aricent Technologies

Introduction

The demand for mobility has fuelled an impetus for wide scale development of consumer and enterprise mobile applications. This has led to the development of apps at rapid pace and with it arises, a paramount need of testing them and ensure that they run on an increasing array of platforms and devices. To test these mobile applications one can proceed with traditional QA practices, however the rapid pace at which they are developed and needed to be rolled out, this calls for application of different techniques which are more efficient. Nowadays, mobile devices offer everything that a consumer wishes to do, in fact now the apps are built not just for smartphones anymore, but also for wearables devices, connected homes, smart cars, IoT-enabled devices etc. With such enormous availability of mobile devices based on their characteristics such as physical dimension, hardware and software compatibility like tablets, smartphone, e-book readers etc., there arises an indispensable obligation of testing these Mobile Applications quickly with astute results and considering the complexities involved, it becomes supreme to decide the scope of the testing.

Dissecting the dynamics involved

To identify the correct testing strategy, it is imperative to first comprehend the environment on which the mobile devices run.

Mobile Operating Systems:To begin with, first we need to look up onto the platforms on which most of these mobile devices operate and based on the popular platforms, our testing strategy should be focused more around it. The following table gives an overview of some of the popular mobile operating systems available in market with their latest version available.

Operating System Developed by Popularity (Low, Medium, High)

Latest available version

Android Google Inc High7.0-Nougat, 8.0 Oreo

iOS Apple Inc High iOS 10.X

Blackberry Blackberry Ltd Low Blackberry 10.3.3

Windows Microsoft Inc Medium Windows 10 Mobile

Tizen Samsung electronics

Medium 3.0

Based on some generic survey, the usage of different operating systems in the market can be depicted as shown below.

Android; 64%

Apple (iOS); 33%

Windows; 1%

Blackberry; 0%

Samsung; 0%

Others; 2%

Operating System

Android OS is the market leader with iOS as nearest competitor and these two Operating System has almost absolute duopoly in platform used in mobile devices.

Below is the Browser share for mobile /tablet:

Chrome; 57%

Safari; 33%

Android Brow.; 5%Opera; 3%

Firefox; 1%

Internet Explorer; 1%

Others; 1%

Browser

Chrome browser is majorly used in mobile devices as it is the default browser for Android platform. Other popular browsers like firefox, internet explorer etc. are also the choice for some significant users.

Now, when the testing timeline is dependent on time to market then important emphasis must be on Testing on android platforms and for web apps chrome should be preferred.

Smartphones, Tablets and E-Book readers constitute majority of mobile devices and are being used primarily. The following table gives an overview about it.

Device Tablets E-book Readers Smartphones

What it is

Tablets are portable computer devices.Unlike traditionalcomputers, they don’t have keyboards or mouse, however the entire screen is touch sensitive.

E-book readers—also called e-readers—are similar to tablet computers, except they are mainly designed for reading e-books (digital, downloadable books).

A smartphone is a powerful mobile phone that is designed to run a variety of applications in addition to providing phone service.

Used for

Almost all the jobs which we can do with traditional computers or desktops.

Reading e-booksWeb browsing, watching videos, reading e-books, and playing games

Example Samsung Tablets Amazon Kindle, Barnes & Noble Nook.

Sony smartphones, Samsung smartphones, Apple iPhone.

Within this environment, bristling with differing hardware, software and connectivity capabilities are applications in three distinct flavours:

Mobile Web Apps: These are typical server side apps that conglomerate the vast capabilities of the web and the functionality of touch-enabled smartphones. These apps are quite easy and quick to deploy as they don’t need to go through approvals since they don’t depend on mobile app store or marketplace for distribution.

Native Mobile Apps: These are custom built platform-specific applications designed and targeted towards a particular device’s OS and firmware. They leverage the device’s sensors and capabilities. Typically, a native app developed for one mobile device has to be re-worked and adapted for another

device. Native apps may enjoy a higher level of security than web apps.

Hybrid Mobile Apps: These apps are designed using latest web technologies, however they run inside the device’s native container. A hybrid app is powerful as it exposes the device’s capabilities (say a camera or an accelerometer using an abstraction layer) to what is essentially a web app. A typical hybrid mobile app has different usability, capability and security across devices.

The demand for the hybrid apps has put large burden on testing mobile applications across devices and device versions. Let’s consider some of the key challenges thrown up when testing mobile applications to understand the complexity of mobile testing:

Challenges

The targeted devices, platforms and OS versions with which the application must be tested?

How quickly can the app be rolled out in the market?

How can performing full-fledged comprehensive test now, be leveraged to prepare for future device and OS upgrades?

What all kinds of tests needed to be run?

How can the test scripts be reused across different platforms?

How can automation be implemented to run the scripts? Which automation tool to opt for, given that the plethora of options available.?

How quickly can the test be done?

How to guarantee the security of the application against any vulnerability?

How to measure the performance of an application and then evaluate whether it doesn’t

Security

AutomationDevice Proliferation

Performance &

Usability

Key Focus Areas

Upgrades

Functionality

hamper the devices performance or other installed apps?

How to test the performance of the device environment (server side)?

What will be the overall cost associated in performing the testing for the mobile app?

Test Coverage (What to Test)

Installation

▪ Install App via App Store▪ Over Wi-Fi▪ Over

data network

▪ Install Ad Hoc build after tailoring it for device

▪ App behaviour with preinstalled apps

▪ First time launch behaviour

▪ Un-installation and device clean-up

Installation

▪ Install App via App Store▪ Over Wi-Fi▪ Over

data network

▪ Install Ad Hoc build after tailoring it for device

▪ App behaviour with preinstalled apps

▪ First time launch behaviour

▪ Un-installation and device clean-up

Installation

▪ Install App via App Store▪ Over Wi-Fi▪ Over

data network

▪ Install Ad Hoc build after tailoring it for device

▪ App behaviour with preinstalled apps

▪ First time launch behaviour

▪ Un-installation and device clean-up

Installation

▪ Install App via App Store▪ Over Wi-Fi▪ Over

data network

▪ Install Ad Hoc build after tailoring it for device

▪ App behaviour with preinstalled apps

▪ First time launch behaviour

▪ Un-installation and device clean-up

Installation

▪ Install App via App Store▪ Over Wi-Fi▪ Over

data network

▪ Install Ad Hoc build after tailoring it for device

▪ App behaviour with preinstalled apps

▪ First time launch behaviour

▪ Un-installation and device clean-up

Installation

▪ Install App via App Store▪ Over Wi-Fi▪ Over

data network

▪ Install Ad Hoc build after tailoring it for device

▪ App behaviour with preinstalled apps

▪ First time launch behaviour

▪ Un-installation and device clean-up

Installation

▪ Install App via App Store▪ Over Wi-Fi▪ Over

data network

▪ Install Ad Hoc build after tailoring it for device

▪ App behaviour with preinstalled apps

▪ First time launch behaviour

▪ Un-installation and device clean-up

GUI & Usability

▪ Portrait and Landscape

▪ Interface▪ Pop-ups▪ Copy bleed▪ Overlay▪ Layout

and alignment

▪ Header & Footer▪ Interactions▪ Multi-touch▪ TILT▪ Real world usage▪ Environment▪ Conventions▪ Sitting/standing▪ Single handed

GUI & Usability

▪ Portrait and Landscape

▪ Interface▪ Pop-ups▪ Copy bleed▪ Overlay▪ Layout

and alignment

▪ Header & Footer▪ Interactions▪ Multi-touch▪ TILT▪ Real world usage▪ Environment▪ Conventions▪ Sitting/standing▪ Single handed

GUI & Usability

▪ Portrait and Landscape

▪ Interface▪ Pop-ups▪ Copy bleed▪ Overlay▪ Layout

and alignment

▪ Header & Footer▪ Interactions▪ Multi-touch▪ TILT▪ Real world usage▪ Environment▪ Conventions▪ Sitting/standing▪ Single handed

GUI & Usability

▪ Portrait and Landscape

▪ Interface▪ Pop-ups▪ Copy bleed▪ Overlay▪ Layout

and alignment

▪ Header & Footer▪ Interactions▪ Multi-touch▪ TILT▪ Real world usage▪ Environment▪ Conventions▪ Sitting/standing▪ Single handed

GUI & Usability

▪ Portrait and Landscape

▪ Interface▪ Pop-ups▪ Copy bleed▪ Overlay▪ Layout

and alignment

▪ Header & Footer▪ Interactions▪ Multi-touch▪ TILT▪ Real world usage▪ Environment▪ Conventions▪ Sitting/standing▪ Single handed

GUI & Usability

▪ Portrait and Landscape

▪ Interface▪ Pop-ups▪ Copy bleed▪ Overlay▪ Layout

and alignment

▪ Header & Footer▪ Interactions▪ Multi-touch▪ TILT▪ Real world usage▪ Environment▪ Conventions▪ Sitting/standing▪ Single handed

GUI & Usability

▪ Portrait and Landscape

▪ Interface▪ Pop-ups▪ Copy bleed▪ Overlay▪ Layout

and alignment

▪ Header & Footer▪ Interactions▪ Multi-touch▪ TILT▪ Real world usage▪ Environment▪ Conventions▪ Sitting/standing▪ Single handed

Functional

▪ Emulators and Devices

▪ Data validation for forms

▪ Localization Testing

▪ Native language first

▪ Layout▪ Text alignment▪ Image &

text captions

▪ Text-on-buttons▪ Text Encoding▪ Interrupt Testing▪ Phone calls▪ SMS▪ Calendar alerts▪ Volume▪ Ringer▪ Screenshot

▪ Headphones

Functional

▪ Emulators and Devices

▪ Data validation for forms

▪ Localization Testing

▪ Native language first

▪ Layout▪ Text alignment▪ Image &

text captions

▪ Text-on-buttons▪ Text Encoding▪ Interrupt Testing▪ Phone calls▪ SMS▪ Calendar alerts▪ Volume▪ Ringer▪ Screenshot

▪ Headphones

Functional

▪ Emulators and Devices

▪ Data validation for forms

▪ Localization Testing

▪ Native language first

▪ Layout▪ Text alignment▪ Image &

text captions

▪ Text-on-buttons▪ Text Encoding▪ Interrupt Testing▪ Phone calls▪ SMS▪ Calendar alerts▪ Volume▪ Ringer▪ Screenshot

▪ Headphones

Functional

▪ Emulators and Devices

▪ Data validation for forms

▪ Localization Testing

▪ Native language first

▪ Layout▪ Text alignment▪ Image &

text captions

▪ Text-on-buttons▪ Text Encoding▪ Interrupt Testing▪ Phone calls▪ SMS▪ Calendar alerts▪ Volume▪ Ringer▪ Screenshot

▪ Headphones

Functional

▪ Emulators and Devices

▪ Data validation for forms

▪ Localization Testing

▪ Native language first

▪ Layout▪ Text alignment▪ Image &

text captions

▪ Text-on-buttons▪ Text Encoding▪ Interrupt Testing▪ Phone calls▪ SMS▪ Calendar alerts▪ Volume▪ Ringer▪ Screenshot

▪ Headphones

Functional

▪ Emulators and Devices

▪ Data validation for forms

▪ Localization Testing

▪ Native language first

▪ Layout▪ Text alignment▪ Image &

text captions

▪ Text-on-buttons▪ Text Encoding▪ Interrupt Testing▪ Phone calls▪ SMS▪ Calendar alerts▪ Volume▪ Ringer▪ Screenshot

▪ Headphones

Functional

▪ Emulators and Devices

▪ Data validation for forms

▪ Localization Testing

▪ Native language first

▪ Layout▪ Text alignment▪ Image &

text captions

▪ Text-on-buttons▪ Text Encoding▪ Interrupt Testing▪ Phone calls▪ SMS▪ Calendar alerts▪ Volume▪ Ringer▪ Screenshot

▪ Headphones

Compatibility

▪Devices and networks

▪Firmware versions▪Backward

compatibility

▪Beta firmware▪Device screens▪Resolution▪Size and dpi▪Scaling of

images▪Device features▪Flip, Slider

or Holster▪Camera▪Accelerometer▪Battery Usage▪LBS▪Default or 3rd

party browser▪Link navigation▪Open/close pop-ups

Non-functional

▪Performance Testing

▪Emulators▪Devices▪Device Cloud▪Security Testing▪Code Review▪Forensics▪Static Analysis▪Dynamic Analysis▪Penetration Testing▪Device Behaviour▪Battery drain▪Memory Usage▪Network

carrier testing

Compatibility

▪Devices and networks

▪Firmware versions▪Backward

compatibility

▪Beta firmware▪Device screens▪Resolution▪Size and dpi▪Scaling of

images▪Device features▪Flip, Slider

or Holster▪Camera▪Accelerometer▪Battery Usage▪LBS▪Default or 3rd

party browser▪Link navigation▪Open/close pop-ups

Compatibility

▪Devices and networks

▪Firmware versions▪Backward

compatibility

▪Beta firmware▪Device screens▪Resolution▪Size and dpi▪Scaling of

images▪Device features▪Flip, Slider

or Holster▪Camera▪Accelerometer▪Battery Usage▪LBS▪Default or 3rd

party browser▪Link navigation▪Open/close pop-ups

Compatibility

▪Devices and networks

▪Firmware versions▪Backward

compatibility

▪Beta firmware▪Device screens▪Resolution▪Size and dpi▪Scaling of

images▪Device features▪Flip, Slider

or Holster▪Camera▪Accelerometer▪Battery Usage▪LBS▪Default or 3rd

party browser▪Link navigation▪Open/close pop-ups

Compatibility

▪Devices and networks

▪Firmware versions▪Backward

compatibility

▪Beta firmware▪Device screens▪Resolution▪Size and dpi▪Scaling of

images▪Device features▪Flip, Slider

or Holster▪Camera▪Accelerometer▪Battery Usage▪LBS▪Default or 3rd

party browser▪Link navigation▪Open/close pop-ups

Compatibility

▪Devices and networks

▪Firmware versions▪Backward

compatibility

▪Beta firmware▪Device screens▪Resolution▪Size and dpi▪Scaling of

images▪Device features▪Flip, Slider

or Holster▪Camera▪Accelerometer▪Battery Usage▪LBS▪Default or 3rd

party browser▪Link navigation▪Open/close pop-ups

Compatibility

▪Devices and networks

▪Firmware versions▪Backward

compatibility

▪Beta firmware▪Device screens▪Resolution▪Size and dpi▪Scaling of

images▪Device features▪Flip, Slider

or Holster▪Camera▪Accelerometer▪Battery Usage▪LBS▪Default or 3rd

party browser▪Link navigation▪Open/close pop-ups

Non-functional

▪Performance Testing

▪Emulators▪Devices▪Device Cloud▪Security Testing▪Code Review▪Forensics▪Static Analysis▪Dynamic Analysis▪Penetration Testing▪Device Behaviour▪Battery drain▪Memory Usage▪Network

carrier testing

Non-functional

▪Performance Testing

▪Emulators▪Devices▪Device Cloud▪Security Testing▪Code Review▪Forensics▪Static Analysis▪Dynamic Analysis▪Penetration Testing▪Device Behaviour▪Battery drain▪Memory Usage▪Network

carrier testing

Non-functional

▪Performance Testing

▪Emulators▪Devices▪Device Cloud▪Security Testing▪Code Review▪Forensics▪Static Analysis▪Dynamic Analysis▪Penetration Testing▪Device Behaviour▪Battery drain▪Memory Usage▪Network

carrier testing

Non-functional

▪Performance Testing

▪Emulators▪Devices▪Device Cloud▪Security Testing▪Code Review▪Forensics▪Static Analysis▪Dynamic Analysis▪Penetration Testing▪Device Behaviour▪Battery drain▪Memory Usage▪Network

carrier testing

Non-functional

▪Performance Testing

▪Emulators▪Devices▪Device Cloud▪Security Testing▪Code Review▪Forensics▪Static Analysis▪Dynamic Analysis▪Penetration Testing▪Device Behaviour▪Battery drain▪Memory Usage▪Network

carrier testing

Non-functional

▪Performance Testing

▪Emulators▪Devices▪Device Cloud▪Security Testing▪Code Review▪Forensics▪Static Analysis▪Dynamic Analysis▪Penetration Testing▪Device Behaviour▪Battery drain▪Memory Usage▪Network

carrier testing

Intricacies of TestingThere are several areas that need to be looked upon while formulating a strategy for mobile application testing.

Functionality Testing: Functionality testing is essential as it tests the overall functional aspect of the application by evaluating all valid use cases and boundary / special conditions.

Some important features that must be evaluated includes, ease of navigation, screen transitions, response speed of the device and application, ability to customise and personalise the application and the intuitiveness of the touch interface. Such testing must be done among various user groups against benchmarks to ensure adaption and quality feedback about the application’s functionality.

Performance and Usability Testing: While application stability and performance are of prime importance, they are dependent on a variety of other factors ranging from the mobile device to load

and application response, server utilisation and network conditions. A wide variety of test tools are available for performance testing. However, to select the right tool based on the scope of the test and costs is critical.

Automation: Test solutions should be platform agnostic. In order to ensure that testing time is reduced, the solutions should be reusable and scalable. Automation leads to shorter test cycles and productivity improvements of around 25 to 30 per cent.

Security Testing: The mobile and personal nature of smartphones makes it a very vulnerable device to security threats. Security testing for mobile applications comes in three tiers – physical end-point device security, enterprise level security for devices (Mobile Device Management - MDM) and application security. Security check lists include password locked keypads, policy enforcements via MDM solution, remote data wipe in the event of non-compliance / theft / accidental loss of the device, application security checks via tools and vulnerability exploitation exercises. Security testing requirements of the mobile application includes evaluating OS specific vulnerabilities, application specific vulnerabilities, network vulnerabilities, storage vulnerabilities, data vulnerability and physical security of devices.

Device Proliferation: With such severe fragmentation in the device space, application testing has to be done across a variety of devices. This means having a very large inventory of handsets available for tests.

Upgrades: Rapid platform upgrades means keeping pace with them and validating the impact of the upgrades on the applications as soon as possible. Though platform owners don’t wish to impact any public application interfaces leveraged by 3rd party developers, at times there could be impact on the application due to an upgrade. It is a good practice to check the release notes of the platform upgrade to look for any potential impact / enhancement opportunity for the application. For businesses this means identifying the core regression suite to validate the impact of the change. This must be executed every time there is a platform / application upgrade. Tests should be based on upgrade release notes that identify the impact areas. Finally, this calls for automation of core regression suite for quick execution with increasing coverage to include more features.

Infrastructure and Connectivity: Mobile testing requires dedicated infrastructure. To function, most mobile applications require connecting to the application servers. This creates the need for having Wi-Fi connectivity to the enterprise backbone (limited to application servers). There are several other ways to gain connectivity to application servers and it is important from a scaling perspective to ensure that test environments can be leveraged by geographically distributed testing team. This, in turn, may impact compliance to IT security policies and require intervention from the IT department.

Addressing the Intricacies of TestingTo ensure a comprehensive and reliable mobile application testing, one requires an effective tool analysis based on the type of testing needed for the particular environment. A wide array of tools are available for performing test automation, performance testing and security testing.

Now due to availability of huge variety of platforms and OS flavours, a same use case has to be run and validated on a growing number of devices and platforms, thus to manage it a robust Test automation tool is a must for mobility testing.

Automation solutions available can be broadly classified into two categories – cloud based and desktop based. Since tool reusability across platforms and devices is of paramount importance, hence many organizations prefer to partner with a technology provider who has a deep understanding of mobile test automation and provides cloud based solutions (Sauce labs, perfecto, App Thwack etc).

Another way to manage mobile application testing is to set up an in-house mobile test automation lab. This means equipping the lab with devices (various platforms and form factors), mobile infrastructure (secure Wi-Fi, internet, backend, automation stations, PI servers), tools (test, performance, security) and data (test data, benchmarks etc.). This is an expensive exercise. Most enterprises also find it difficult to retain the skills and expertise required to manage and maintain such facilities that are not core to their respective business.

The thumb rule of mobile in getting the best outcome for your mobile app testing is as follows:

Mobile Automation tool comparison table:

Criteria Countdown Device Anywhere

PerfectoMobile Robotium

Hardware Integration

Data Port Electrical integration(PCB level)

Data Port+ Camera Data Port

Testing Interface Web based Client

Java Swing Client Download For win,Max and Linux

Web based interface

Only through API’s based on Android framework and Junit3

Operating System

Web Client Windows, MAC and PC

Any Flash Capable Browser

Windows+Eclipse+Java+Android SDK+ADT for Eclipse

Platform Support Android, Blackberry, Windows Mobile, Symbian

Iphone,Android, Blackberyy, Windows Mobile, Symbian

iPhone,Android, Blackberry,Symbian

Only Android

Third party tool integration

No HP,IBM, Custom Test Harness

QTP Mave, Ant and Cucumber

Device Connectivity

yes Only remote location connectivity

Only remote location connectivity

USB connectivity

Test Reporting Yes Yes Yes Yes

SummaryIn today’s highly fragmented, fast growing and dynamic mobile market it is tough for enterprises to own and operate mobile application testing infrastructure with high utilisation levels to justify the investment. It is best for businesses to rely on cloud based solutions for all the hardware and platform needs. The following points must be considered while selecting the best solution:

Providing almost all major mobile devices simulators/emulators with different OS versions. Automation capabilities to ensure best quality in shortest possible time. Scalable test infrastructure. Ready-to-customize business processes and use cases.

The major benefit of opting for cloud based solution is that an enterprise can remain platform agnostic and focus on the application rather than on the device. Rapid automated testing of applications across platforms and versions will ensure faster time to market, with a high degree of functional and security assurance in addition to deep test coverage - all this without the headache of managing the test infrastructure or making CAPEX investments with uncertain ROI.

References https://tecordeon.com/wp-content/uploads/2015/06/Mobile-Application-Testing.jpg

https://www.pcloudy.com/mobile-app-testing-trends-of-2016/

https://bitbar.com/testdroid-showcasing-at-mobile-dev-test-and-iot-dev-test/

Author Biography

Shinny Singla has 9+ years’ experience of working on Automation Framework and Manual testing of medium to large sized projects in Telecom domain. she is currently working as a Technical Leader in Aricent Technologies. She has a very good hands-on experience on various flavours of Mobile application testing. Also, she has strong exposure of various software development life cycle including Requirement Understanding, Design, Feature Analysis, Test Strategy, Test Planning, Test Execution and Test Automation of assigned features.

Pradeep Bhandari has 6 + years of working experience in QA Capacity wherein majority of the assignments were related to Implementing Automation in the Telecom Domain Applications.

He has strong exposure in creating automation frameworks using tools like QTP/UFT, Selenium, Appium and has worked on numerous Automation projects and been an integral part of various software development life cycle including Requirement Understanding, Design, Feature Analysis, Test Strategy, Test Planning, Test Execution and Test Automation of assigned features.

THANK YOU!