mobile app testing by mark wilson

© Planit Software Testing

Mobile App Testing & Other Considerations

Presenter: Mark Wilson


Our knowledge is your power

Software Testing

Consulting, Advisory, Delivery,

Managed Services, Training


Why is Mobile Computing Important?

People worldwide now spend more time on their smartphones than on traditional PCs.

Smartphone sales surpassed PC sales in 2011.


64%

15 - 65

The Mobile Landscape In New Zealand

Have a smartphone now


90%

15 - 65


Will have a smartphone

by 2018


44%

2013


Have a tablet at home now


78%

2018


Will have a tablet by 2018


Apps vs Web


The Future Of Mobile Apps


What Is Mobile Computing Used For?

• Apps continue to grow

• Gaming and Social Media are over 60% of apps usage


How Successful Are most Mobile Apps?

Apps on our smart phones

• Average 25-26 apps

• 1 in 4 apps downloaded are never used

• On average 22% of Apple users who download an app

only use it once.

• On average only 34% of Apple users who download an

app will use it more than 11 times.

• Over 1 million applications are available from the app

store alone


What Spurs Mobile App Growth?

Mobile app usage worldwide.

What will be the next big thing?


Do Mobile Apps Have Problems?

Have you had problems with a mobile app within the

last 6 months?

NO

44%

YES

56%


Types Of Problems Encountered?

•The mobile app crashed/froze/ displayed an error

•The mobile app was slow to launch

•The mobile app would not launch

•The mobile app didn’t function as expected

•None of these


What The Customer Expects

•Things to work first time

•They expect things to work fast

•They expect things to be intuitive

•They expect instructions to be clear and self explanatory

•Your app generally has to be usable by people with differing

abilities

•Most apps don’t come with a 1-2 day training course! It’s a

Web App not SAP


So Why Do Mobile Apps Fail??

Customers Can’t Make It Work!!

1. They have problems installing

2. They don’t run properly

3. They freeze or lock


The App Won’t Install, Won’t Work Or Freezes!

Why is this? 3 Answers:

• They have not been functionally tested properly!

• They have not been useability tested on that platform combination.

• They have bugs which were not found in testing.

• This application documentation clearly states that this device in this

configuration is not supported.

People Love to

Complain

http://kotenks.wordpress.com/2012/01/01/fat-dogs-dont-go-to-heaven/


Scope Targeted Platforms

What platforms will you target and test your

application on?

• Operating Systems?

• Browsers ?

• Hardware?


Scope Targeted Platforms?

Each one has multiple versions of the

operating system


Targeted Platforms?


Targeted Platforms?

What Devices?

Phones – Iphone 3-6, Galaxy S3-S5, Nexus 4-7

Blackberry ??, HTC ?, Nokia windows phone,

Huawei, Redme + hundreds of clones etc.

Tablets – IPad 1-4, Galaxy Note 1-4, Windows

Surface ??. Le-Novo, Sony, Acer, Asus, Amazon

Kindle, HP, IPOD Touch + Hundreds of clones.

Different screen sizes

Different resolutions

Different memory

Different processing power

Different operating systems

Different quality antennas


153

Targeted Platforms?


Lessons Learnt – Devices

Manage your platform scope

• How many devices, operating systems, versions?

o iOS?

o Android?

o Windows Phone?

o Blackberry, Symbian, . . .?

o Smart Phones, Tablets, . . . ?

• Your systems future?

o Device & OS fragmentation

Requirements


Problems with Look Feel and Function

1. The application doesn’t do what the customer thinks it should

2. The application is just too complicated or confusing

3. It doesn’t look very good or is isn’t nice to use (Subjective but

unfortunately customers are)


The Application Is Not Intuitive

• Make your app as clear and easy to use as possible.

• Keep a common look and feel throughout the application.

• Get your kids to trial it, then get you mum and gran to have a play.

• There are guides around usability – look at them. Government have

usability and accessibility guidelines which are being enforced more

rigorously.

• If you are targeting a bigger audience then look at the issues around

accessibility. http://blindfoundation.org.nz/learn/information-for-

businesses/accessible-websites is a good place to start.

http://blindfoundation.org.nz/learn/information-for-businesses/accessible-websites









Lessons Learnt – Look/Feel/Function

Have a clear vision for the application

• Put together clearly defined requirements (‘Just like our website’ is

not good enough!!)

• Target your market demographic

• Look at what is successful in that market

• Look and feel can be as important as function

• But look and feel won’t keep your customers happy if the

application doesn’t do what it’s supposed to

Requirements Requirements


Be Prepared To Speed Up

First to market is a big advantage!


Focus on time – everything moves quicker on mobile projects

• The market is moving incredibly fast

• The technology is moving very rapidly

• The fashions and trends are moving fast

The Application Takes Too Long To Get To Market

http://williesun.com/tag/ramble-away/?doing_wp_cron=1402289455.7205860614776611328125

Remember you are better off being

second to market with a product that

works than being first with one that

doesn’t


A Newer, Quicker Way Of Working!

• Manage your stakeholders

• Get marketing and the business involved early

• Gear up for more releases


Focus on time

Look at continuous integration

Automated functional regression testing

Automated regression performance testing

Automated penetration testing/vulnerability scanning

Service virtualisation to allow parallel development

Farm out the bulk of your device usability testing

Look To Reduce Cycle Times – Not Quality!


Prepare For Regular Release Cycles

• Plan for your ongoing release cycles

o Business will see new opportunities

o Expect Public APIs to change (e.g. Google Maps, TradeMe)

o Technologies will change

o Taste and fashion change

• How will you deploy future releases?

o Plan for availability of infrastructure


Lessons Learnt – Be Prepared To Speed Up

• Be prepared for small projects

o With frequent follow-ons

• Build your team early

o Get the right mobile expertise in at the start

• Connect early to the business stakeholders

• Get your scope nailed early

o Especially what devices and OSs you will support

• Start small

o Plan for measured rollouts

• Plan your device management and your security early

Requirements Requirements Requirements!


Test Improvement

Consider:

• Testing smarter

• Device emulation tools

• Test automation


Test Smarter

Consider

• Do you need to run full functional regression tests against

every device combination?

• Could you use Bluetooth keyboards to reduce fatigue?

• Could you use an adaptor and a full size screen?


Device Emulation

Pros:

• Makes many device combinations available quickly and affordably

• Testers use a proper desktop to drive the application

• Defect creation is faster using cut and paste, screen shots etc.

Cons:

• It’s a step away from production like. You have introduced another

layer that can create and/or mask problems.


Test Automation

Consider

• You may need test automation to enable you to perform

regression testing on each release in a timely manner

• Do an ROI calculation before you invest in this area

• If you are going to invest in this area do it early


Lessons Learnt – Test Improvement

Emulation can speed up testing but is not a substitute for

testing on the actual device.

Look in to test automation early.

Devices:

• Track carefully – they can go walkies . . .

• Testing infrastructure – rent or buy, Planit Mobile Test Lab

• Bluetooth keyboards can save the thumb

• Screen adaptors can save eyes

• Invest in antibacterial wipes


Users Expectation Mobile Apps vs Mobile Web

Performance is important in the

mobile world


Users Expectation Mobile Apps vs Mobile Web

So what are your customers

expectations?


Users Performance Expectation Mobile Apps vs Mobile Web


User Expectations For Mobile App Performance


The Apps Are Too Slow!

• Performance is a really big deal.

• How do you test mobile performance?

• What is good performance?

• What is the total impact on the performance of your

system?



We are not fully in control • The users location determines the strength of signal they have at the

time

• The network provider and plan they use affects performance

• The base performance of their device can vary

• There could be other applications running on their device at the same

time

As a general rule – we cannot control these!



Mobile Networks

System Under Test

F5

F5

APP

APP

APP

APP

Legacy 1

Address FWK

Account System Internet



System Under Test

F5

F5

APP

APP

APP

APP

Legacy 1

Address FWK

Account System

x 220 Staff

x 10 Manager

x 20 Accounts

Mobile Networks

Internet



To get production like response times you need:

• A production like environment to test against

• To create a production like load on all components

• A production like test data selection

If it isn’t production like then you introduce the risk of missing

a problem or creating a problem! Also, the response times you

see during testing most likely wont be representative of

production.



You can’t control what people have running on their devices, where they

use it or who their provider is!

So keep it mean and lean!

http://kotenks.wordpress.com/2012/01/01/fat-dogs-dont-go-to-heaven/



What is good performance?

1 second response time?

2 second response time?

Whatever enables your

customers and staff to work

efficiently without getting

frustrated.


Lessons Learnt – Performance

• Think about performance testing early

• Consider the whole system not just the mobile component

• As production like as possible: o Environment, hardware, configuration, application versions

o Data populated to production like levels, test cases selected in a production like

way

o Production like transaction mix and durations

o Production like load on all system components.

• Look to test at lower levels earlier e.g. web services.

• Look to introduce regression performance testing if appropriate.

• Be aware of the performance of any third party services such as payment

gateways and try to ensure you can see the performance costs in your logs


Security Issues?

90% of top mobile banking apps

have security issues (IOActive, Jan 2014)

90% of mobile apps have at least 1

vulnerability (ZDNet, Nov 2013)


Security Security Security

The top 10 security issues faced by Mobile devices:

1. Lack of passwords

2. Lack of 2 factor authentication

3. Wireless transmissions not being encoded

4. Mobile devices are susceptible to malware

5. Mobile devices often don’t have security software

6. OS is out of date and doesn’t contain latest security patches

7. App software is often out of date

8. Unsecure ports on devices can be open

9. Mobile devices can be modified, jail broken, or rooted which can allow unauthorised access

10. Unsecure Wi-Fi can allow access to the device.

Consider BYOD


Lessons Learnt - Security

• Don’t forget mobile security

• Address BYOD

o Strategy to manage apps

on devices

o Security policies

o User agreements

www. webhostinggeeks.com


Mobile Projects Growing in NZ

• Telcos

• Banks

• Insurance

• Media

• Government

• Entertainment


How Our Projects Are Changing

Business changes lead to new projects

• Mobile projects are increasing

o In quantity

o In business importance

o And in speed – “First-mover advantage”


Summary

• Growing business pressures on schedule

• The user / customer experience is critical

• Mobile projects are increasing

o In quantity, in business importance, and in speed

• Lessons Learnt

o How many devices, versions?

o Define requirements

o Involve the business early

o Involve testing early

o Device management


Mark Wilson

Principal Consultant

[email protected]

+64 27 367 1939

Lessons Learnt In Managing And Delivering Mobile Technology Projects

Contact Planit

www. planittesting.co.nz

+64 4 815 8199

mobile app testing by mark wilson

Technology

mobile app usage

mobile landscape

mobile app crashedfroze

mobile app growth

future of mobile apps

mobile app didnt function

web app

app store