Human testing of mobile apps

Human testing of mobile appsFor Vistacon 2013 conference25 April 2013JULIAN HARTYContact me: julianharty@gmail.comRev: 29 April 2013Creative Commons LicenseHow to design your mobile apps by Julian Harty is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.

http://creativecommons.org/licenses/by-sa/3.0/deed.en_US 1automationDrive for efficiencyState of affairsThe industry is driving towards automated testsManual testing is downgradedAgile projects need automated tests to provide fast & frequent feedbackDevelopment environments for Mobile Apps (SDKs) include Test Automation frameworksWindows Phone 8 Test Automationhttp://blogs.msdn.com/b/windowsappdev/archive/2012/09/04/automating-the-testing-of-windows-8-apps.aspx (from BJ Rollisons answer on SO site)http://mobile.dzone.com/articles/windows-phone-8-unity-testing

3Automated testsUnit TestsContinuous BuildsUI Test AutomationScale out to run tests on multiple devicesVirtual devices, remote devices, local devices

Complex, sometimes fragile, ecosystemsChallenges: device security models, limitations in frameworks, connectivity, immature domain, lack of support for multi-modal interfaces, etc.People need People tooHuman-centric testingOr mobile apps need people too?5Beyond the reach of automated testsTest frameworks focus on code, API, or GUI,

Not:Movement: acceleration, rotation, locationTactile, Haptic, audio Inputs or OutputsAesthetics: look & feel, transitionsUser Experience (UX)Highly dynamic updates: video, audio outputComplex inputs: Gestures, multi-touch, shakingLight: e.g. camera as an input device

So how do we test these?QOXQuality Of Experience (QOX)

From the standpoint of a users holistic experience of using the mobile app.

Essentially subjectivehttp://en.wikipedia.org/wiki/Quality_of_experienceUX Research, Evaluation, and Testing: strategies, tips and tools Mark J. Williams

Ive got feelings too6 hats of software testingThe red hat emotions, intuitions, feelingsI've Got a Feeling: Emotions in Testinghttp://www.stickyminds.com/s.asp?F=S17922_COL_2 Here's an article I wrote a few years ago on the six hats of software testing http://archive.newsweaver.com/qualtech/newsweaver.ie/qualtech/e_article000666359.html and if you really have lots of time on your hands, here's a video of a talk I gave on the same topic at StarWest 2008 http://www.stickyminds.com/Media/Video/Detail.aspx?WebPage=117 An inconsistency between the product and the document might be surprising, frustrating, or annoying to that person, but what really matters is inconsistency between the product and the desire.What do we need to do?SoFast, efficient interactive testingBugfestsMove and rotate the phoneBe mobileBugfest: A group of people get together to test in parallel and in common. Developers on-hand for direct feedback.Lunch outtesting a video appAround 2 hoursAround 10 testersTest the app while going out for lunch

Examples of what the testers didTourist trip on the top of a double-decker busLondon UndergroundOut for a walk with sandwichesEstablish strong foundationsLearn:The platforms, the norms, what users expectHow the platform was designed to be usedThe application lifecycleThe devicesCommon, and unusual problemsBlack BoxWhite Box12Insightful testingRAPID softwareHeuristicsMnemonics to guide the testing

Session-Based Test Management techniquesCharterTime-boxedSimple, lightweight test reportingDebrief

http://moolya.com/blog/2012/04/18/test-mobile-applications-with-cop-who-flung-gun/Extended testingExtended testingRemote devices

Remote testing

Remote testers

Credit: "Ken Banks, kiwanja.net"

http://www.kiwanja.net/gallery/texting/kiwanja_kenya_texting_17.jpghttp://qz.com/49916/indian-smartphone-users-are-gobbling-up-mobile-data/http://www.imedicalapps.com/2012/09/researchers-smartphone-spirometer-mobile-app/16Challenges of extended testingLoss of informationSensory deprivationMissing or incomplete data

Loss of fidelityLags, delays

Loss of controlHow do we test rich interactive apps unless you have the device in your hand?

In theory, theory and practice are the same. In practice, they are not. A device in your hand is worth 2 in the cloud, and 100 virtual devices. More contextMore contextAdditional considerationsCultural norms & differencesGenerationsWhats important to them?How can we empathize?Mobile Apps & PlatformsInternationalCulture?Culture?Culture?GSMA REVEALS FEARS OVER MOBILE PRIVACY ARE HOLDING BACK THE GROWTH OF MOBILE APPS IN LATIN AMERICAhttp://www.gsma.com/newsroom/gsma-reveals

A South Korean woman using her mobile phone to buy a product at the virtual retail shop in Seoul.www.guardian.co.ukExamples of what can go wrongWhat this means is that unlike the iPhone, which is most certainly the closest competitor on the market to this phone, the Storm's UI is not custom built for touch navigation -- touch navigation is added after the fact. Things which flow naturally on an iPhone -- flicking through lists, scrolling for a contact, moving around in a webpage or looking through photos -- feel inelegant and uncomfortable on the Storm. There's no inertia to movement, no assurance that your finger is the lynchpin to control of the device. The screen is sensitive enough, surely, but how its software reacts to those touches makes all the difference, and here the feeling is that you're never completely in charge of the phone.http://www.engadget.com/2008/11/19/blackberry-storm-review/

Image: http://worldwide.blackberry.com/gettingstarted/storm-9500.png! XOR[*]Practical outcomes[*] Not Exclusive ORCombining strengthsAutomated TestsFast feedback, technology and developer centric

Interactive TestsExploratory, rich interactions and dynamicsRemote testing e.g. in other countries

CombinedAugmented, Cyborg testing e.g. automated preparation, data collection, etc.Learn to delegate & learn when to take controlSome further readings

http://www.wipconnector.com/download/GuideToTheParallelUniverse_3rdEdition.pdfhttp://www.enough.de/fileadmin/uploads/dev_guide_pdfs/Guide_12thEdition_WEB.pdf

Testing and Test Automation for Mobile AppsJulian Harty

Summer 2013CRC PressDevelopmentMarketingTesting(2009)Testing(2013)Q & A ?Misc references & linkshttp://techcrunch.com/2012/06/02/android-qa-testing-quality-assurance/http://techcrunch.com/2012/03/19/applover-android-testing-platform/

24If you want the draft bookDraft book available for review & feedback

Email me julianharty@gmail.comAgree not to share or distributeComment to get the next available draft