Understanding the “Testing Challenges" in Mobile Application

Presentation Objectives

Understanding the "Testing Challenges" in Mobile Applications by

• differentiating between the different types of Mobile Applications,

• the HCI Guidelines,

• User Expectations,

• Testing Methodologies and some practical examples.

Setting Up the Testing Environment

FACTS

ACCORDING TO SDTIMES SURVEY 6 IN EVERY 7 COMPANIES ARE PLANNING TO DEVELOP A MOBILE APP WITHIN THE NEXT 18 MONTHS

SMART PHONES ARE RAPIDLY BECOMING THE PRIMARY METHOD OF INTERACTION BETWEEN BUSINESS AND CONSUMERS

So What is at Stake?

Types of Mobile Apps1. Mobile Native Apps

Types of Mobile Apps2. Mobile Web Apps

Types of Mobile Apps3. Hybrid Mobile App

Pros and Cons of Mobile Apps

Apple’s HUMAN COMPUTER INTERFACE GUIDELINES

Users have one view at a time

Re-organize how information is to be displayed from the desktop

Minimize number of user controls and input elements

Do not display UI elements that are not absolutely necessary

Provide tap-able items with a minimum dimension of 44 x 44 points

Make it easier for the end user to interact with the system

Apple’s HUMAN COMPUTER INTERFACE GUIDELINES

User standard controls with which users have had prior experience

Steps with which users can go back or retrace their steps.

Users are going to use the application under various distractions

Design process to enable users to complete their tasks as quickly

Minimize text inputs

Use dropdowns/sliders/checkbox where applicable

Android UI Design GUIDELINES

Pictures are faster than words

Consider using pictures to explain ideas

Give me tricks that work everywhere

Make it easier to learn by leveraging design patterns from other Android apps

Sprinkle Encouragement

Give feedback on actions, even if it's a subtle glow or a slight vibration.

The power of App UsersUSERS CAN AND WILL PROVIDE REVIEWS OF YOUR APPLICATION PUBLIC ON APP STORE.

Users Control ReputationFROM THE LIST BELOW WHICH APP IS THE MOST APPEALING TO THE USER?

WHAT FACTORS MAKE THE USERS WANT TO DOWNLOAD THEIR APPLICATION?

USABILITY. USER INTERFACE HAVE TO BE TOP NOTCH. APP SHOULD PERFORM ITS MAIN PURPOSE SEAMLESSLY WITH GREAT ANIMATIONS.

APP SPEED . FUNCTIONAL SPEED ON DIFFERENT TYPES OF MOBILE NETWORKS. [USERS ARE IN GENERAL IMPATIENT.]

DRAINAGE. DRAIN TOO MUCH BATTERY, USERS WILL UNINSTALL THE APPLICATION.

SECURITY. USERS ARE SENSITIVE ON PRIVACY AND TRACKING.

APP AVAILABILITY. IN ALL TYPES OF HANDHELD DEVICES.

UPDATES. USERS WANT THE APP TO BE UPDATED REGULARLY FOR IMPROVEMENTS AND BUG FIXES.

What does the Users Want?

What is the Ultimate Goal?

•A COMPANY’S BRAND STRENGTH AND REPUTATION IS DIRECTLY CO-RELATIONAL TO THEIR MOBILE APPLICATIONS.

Type of App Content

Productivity Apps.

Users using app to organize data such as Personal Finance Apps requiring multiple views.

Type of App Content

Utility Apps.

Users performing single tasks like watching videos on an attractive UI.

Type of App Content

Immersive Apps.

Gaming/simulation Applications providing custom GUI.

TESTING ON MOBILE DEVICES

UI/USABILITY TESTING

The key to this is to make sure that HCI guidelines set forward by Apple/Google are imposed on the applications.

Along with many questions, here are few we should consider asking: Are the buttons too small to click on?*

Does the UI utilize good animations and transition effects?*

Is the application smooth and fast to use?*

Does the UI renders correctly on different OS versions?

Are proper error handling messages displayed*

Supporting Different Screen Resolutions

Using Adaptable Layouts for different screen sizes.

Is the Adaptable Layout justifiable?

There is more space to display more options to the user

What would be the maximum/minimum screen resolution?

The layouts are working properly on different OS versions?

Usability Issues -- 1

A user turns of the app and tries to log in. However, the internet connection is disabled.

Usability Issues -- 1

Let the user know that the internet is turned off, else he/she will think that something is wrong with the app.

Usability Issues -- 2

The user purchased over 150 Courses from the Great Courses website.

Via the API call during Log in to the app, all the Courses and the Course Details were Loaded.

Usability Issues -- 2

This user faced a big problem. It took him over 4/5 minutes to load all the courses.

On top of that, when tested on low configuration devices, the app ended up crashing.

Usability Issues -- 2

Hence in order to rectify the issue, the API call were modified to only send the Course List first.

Afterwards, on course click, each individual courses returned the Lecture list via another API call.

This improved the loading time significantly to less than 30 seconds

Trade-off: App now has to make more calls to the server to retrieve data.

UI/USABILITY TESTING -- Events

TESTING ON MOBILE DEVICES

FUNCTIONAL TESTING

• Are there too many functionalities on-screen to distract the end users?*

• Can the tasks be broken down into simpler parts?

• Are the functionalities working cross platform?*

• Are the functionalities working even when the app is resumed/restarted?

Functional Testing Issue -1

A video is being played by the user.

Then decides to click on the Back Button or Log Out Button.

The video should stop playing for both cases

Functional Testing Issue -2

Usually, during app load various JS files are downloaded from the server.

When the internet was turned off when app is launches many of the JS files were not loaded.

The biggest and most major problem was BrightcoveExperiences.js was not loaded and hence

• No Video or Audio Files were played

• User had to kill app and restart app after activating internet connection to be able to watch videos.

TESTING ON MOBILE DEVICES

INTERRUPT TESTING

How does the application handle:

• Incoming Phone Calls

• Text Messages

• Notifications

• Low Battery Alerts

TESTING ON MOBILE DEVICES

NETWORK TESTING

A decisive factor on App’s performance

• Hence, testing under different/strained networks is a must.*

o Wi-Fi Network [High Speed/low Speed]

o 3G/2G

o Different Carriers

TESTING ON MOBILE DEVICES

CROSS-OS / CROSS-Device TESTING

Does the app performs satisfactorily in

• IOS 5.0/5.1/6.0/6.1/7.0/7.1 in both iPhone & iPad?*

• Android 4.0/4.1.4.2/4.3/Company Specific Rip?*

• Other necessary OS versions on Windows Phones, Blackberry?

TESTING ON MOBILE DEVICES

PERFORMANCE AND STRESS TESTING

• Does the application perform in low configuration devices?*

• What if the user turns on this application while a large number of apps are working on the background?*

• Is there any graphics lag?*

• Does animation slow down the application?*

• Are all user information saved before crashing?

TESTING ON MOBILE DEVICES

RESOURCE ISSUES.

• Does App cache over time leads to UI lags and application crashing?**

• Test application on low configuration emulator/devices*

• Check reliability of the application*

EmulatorsADVANTAGES OF TESTING ON EMULATORS

•Cost effective solution as actual devices are expensive

•Faster internet and has less latency than real devices

•Can be easier to develop and test app in the initial development stages

•Real-time debugging available on iOS Apps and Android Kit Kat

EmulatorsDISADVANTAGES OF TESTING ON EMULATORS

• Can be unreliable as it can cause false impression

• Mobile computing resources are not emulated accurately to be able to make judgment calls

• Mobile/3G Networks affect application behavior

• Can be very slow to use for testing. Can loose chain of thought.

Actual DevicesADVANTAGES OF TESTING ON ACTUAL DEVICES

• Allows Feel and touch sensations which is a big for understanding usability

• Only true way of understanding the user experience, taking into account the CPU, memory, screen size, etc.

• Can test under different mobile Networks as it will affect the app.

• Real-time debugging available ono iOS Apps and,o All upcoming Android versions starting from Android Kit-Kat

Actual DevicesDISADVANTAGES OF TESTING ON ACTUAL DEVICES

• Physical Resources which need to be managed.

• The logistics and costs involved in procuring and managing these devices are significantly high. 

Android ChallengesDEVICE SPECIFIC TESTING IS A MUST

Android ChallengesDEVICE SENSITIVE TESTING IS A MUST

• SAMSUNG/NEXUS/HTC ALL USE MODIFIED OS VERSION

• NETWORK/MEMORY CAPACITY/OTHER RESOURCES ARE DIFFERENT ON EACH DEVICE

• DIFFERENT DEVICES USES LOCAL RESOURCES DIFFERENTLY

Android ChallengesDEVICE SENSITIVE TESTING IS A MUST

• IMPRACTICAL TO TEST ON ALL DEVICES.

• HENCE, TEST ON POPULAR MOBILE PHONES –

SAMSUNG S4 / NEXUS 5 / IPHONE 5

• ALSO TEST ON POPULAR TABS –NEXUS 10/SAMSUNG GALAXY TAB/ IPAD 2

Bug Reporting Options

1. SCREENSHOTS CAN BE USED ALONG WITH FACE TO FACE COMMUNICATION WITH DEVELOPERS TO VISUALLY GENERATE THE GLITCH FOR BETTER UNDERSTANDING.

PROS AND CONS

• IT IS THE FASTEST WAY TO REPORT BUGS.

• TOUCH AND FEEL SENSATIONS CANNOT BE REPRESENTED PROPERLY.

• GENERATE ISSUE IN FRONT OF DEVELOPER TO EXPLAIN THE TOUCH AND FEEL FACTOR PROPERLY.

WHAT ARE THE OPTIONS?

Bug Reporting Options

2. USING ANDROID SCREEN RECORD APPS

A SHORT STORY OF THE SCENARIO FOLLOWED BY THE SCREEN RECORDED SCENE.

PROS AND CONS

• TIME CONSUMING AND VIDEOS ARE LARGE AND LAGGY

• CAN GENERATE REAL SCENARIO FOR DEVELOPER TO VISUALIZE WITH TOUCH GESTURE RECORDING

• HOWEVER, CAN BE CONFUSING TO INTERPRET.

PROBABLE OPTIONS

Setting Up the Test Environment1. Java Installation

• Download and Install Java JRE from

• http://www.oracle.com/technetwork/java/javase/downloads/jre7-downloads-1880261.html

• Add java JRE bin folder to the Path Variables

• Download and Install Java JDK from

https://jdk7.java.net/download.html

• Add java JDK bin folder to the Path Variables

Setting Up the Test Environment2. ANDROID-SDK

• Download SDK from

http://developer.android.com/sdk/index.html?hl=sk

• Unzip the download on C:\

Setting Up the Test Environment

• Add Android SDK tools folder to the Path Variables

Setting Up the Test Environment

3. Apache Ant

[Java-based build tool]

Download from http://ant.apache.org/bindownload.cgi

Unzip to any location. Eg. E:\Tools\Project\Build\apache-ant-x.x.x

Open environment variables (Control Panel » System » Advanced » Environment Variables) add a new system variable as displayed below, 

Setting Up the Test Environment

Set the %ANT_HOME%\bin; to the path as displayed below,

Verify the installation of Ant by typing the following command in the command prompt, 

Setting Up the Test Environment

4. NODEJS

[ Software platform for scalable server-side and networking applications]

Download NODEJS from http://nodejs.org/download/

Run command prompt

If nodejs is installed then npm command will be recognized

Phone-Gap /Cordova

Phone-Gap/Cordova are application framework that allows developers to use HTML, JavaScript and CSS to create apps for Mobile Market Place.

Adobe donated the code-base of Phone-Gap to Apache, which then renamed it Cordova.

This was done so that PhoneGap can reap the benefits of open source, while at the same time having the ability to be different.

Setting Up the Test Environment

5. Cordova/Phone-Gap

Open command prompt and type

Setting Up the Test Environment

Already have a MAC VM with all the necessary environments pre-installed to be able to start testing immediately

Thank You