QA tools / Mobile testing

Basic Desktop tools

Bug tracker (jira, mantis ..)

Bug tracker hooks (browser extension, screen capture ..)

Communication tools (emails, chat, wiki ..)

Test plans (excel, testrail, google doc ..)

Video capture (quicktime ..)

Image editing (paint ..)

Network monitoring (shark)

Others

Dbgview

Process explorer

Text editors

..

04/04/2013 2

Advanced needs

We have more advanced needs, such as:

Kill / relaunch application easily

« Autofill » strings that we type hundred of times daily

Set a .ini file with some specific attributes to provide meaningfull logs to devs

Easily find users profiles on local machine

Clean local settings to simulate a « fresh install »

Get some informations on any given test account

Premium status and history

A/B testing test cell

Data transactions

..

Act on the above account’s details

Being able to simulate a premium purchase, just as a user would do

04/04/2013 3

Problems

Developpers are busy working on actual user facing problems / features

Developpers are busy developping

QA is busy QAing

It’s never the right moment

It’s not so important after all, is it ?

No one cares but you

04/04/2013 4

How we got it done

Use our talented QA people to do a windows tool for all client side needs

Accessible languages: C#, python, nodejs

Fullfills the following needs

.ini

Kill / relaunch

Profiles

Autofill strings

User support needed a server side tool too

When problem arised, everyone quickly understood how critical it was to have a real server side tools

04/04/2013 5

It’s all great, but now that I have to test on an iPhone / Android ..

There are many devices to test (Android screen sizes ..)

It’s difficult to get the build on the device

How do I take a screenshot and add it to a bug ?

How do I make a video of a bug ?

Tests take 5x the time because of touch & tiny keyboard

Someone sent me a link on my desktop, I need it on the test device. How to send it quickly ?

I don’t have access to the file system, and thus cannot change someconfiguration files or reset local settings

How to test play/apple store payments ?

How do I simulate a poor connexion ?

04/04/2013 6

There are ways ..

Use VMs (genymotion) or simulators

Some bugs will be VM only and some will be real device only ..

Mutualize devices with other teams (extQA)

Hockeyapp, testflight to distribute the application

Beware of the auto update integration in the app. This can cause bugs that no user will ever have

Dropbox to sync screenshots to desktop in a breeze

If working with shared devices, can bring a lot of screenshot / photos from other teams

Use the same email account on all devices, and send links to this address

Video

Record a video with another device

Or use tools such as reflexion (sends iphone screen to a desktop client)

Code in your app some test specific behaviors when using test accounts

Beware not shipping a test enabled app !

Beware not missing a bug because of test mode !

Apple has some built in dev tools to simulate poor connexion

04/04/2013 7

But not for everything

Apple and Android both feature test payments, but with limitations

On android you need an actual google account with a valid CC saved. This prevents externalQA from testing purchases

On iOS you might make it work if using the correct app ID. But test purchaseswon’t work on the live build, thus you cannot reproduce a live payment bug

iOS is a closed system

Testers are in full black box mod. But not able to replicate real usersconditions

You may implement a debug menu in your app, but can be tedious to dev, maintain and use

04/04/2013 8

Conclusion

We are yet to find better solutions to ease the tests on mobile

We are very curious as to how it goes on your side, are you facing the same problems ? Please react!

04/04/2013 9