Simplify MobilePhone testingby Randy Kemp and RogerRichardson

2© 2012 by Roger Richardson and Randy Kemp

AbstractToday companies need a solid method for testingsmartphone firmware, application upgrades, softwareupgrades and new releases.New features are constantly introduced into smartphones.Interactions between software upgrades, firmware, softwareand user interactions are increasing in complexity. Newphone models are constantly introduced into themarketplace. Users keep demanding new features and aregiven more marketplace choices. The demand on testingteams is stressful.Inadequate testing is a constant challenge. The world

population has an increasing demand for smartphones. It'sestimated that about three­quarters of the US population willown one by 2013. Users prefer to use smartphone camerasover their own cameras. The phone system API encouragesthird­party application development. A constant demand fornew features, phone models and applications, increases therisks of testing errors.Testing teams must have a systematic smartphone testingapproach. Users must adapt an approach that looks at threetesting categories:

• Phone states: There are dynamic states to consider,where the software actively works on data. There arestatic states, where the user is left at a specific place forfunctional control.• External interruptions: Interruptions beyond a userscontrol• Orientation: Adjustment of the physical positioning ofthe device, that can affect both the display and changefunctional actions

This paper will introduce you to a testing method, designedto cut user calls to call centers, test the largest smartphonepossible scenarios, and increase testing team success.

3© 2012 by Roger Richardson and Randy Kemp

Mobile phone Complexity and testing:The big pictureBrief historyThere are two types of phones testers need to test,according to Phone Scoop

• Feature phones – According to Wiki, it's a “ mobilephone which at the time of manufacture is notconsidered to be a smartphone, but has added functionsover and above standard mobile services. It is intendedfor customers who want a lower­price phone without allthe features of a smartphone.” In addition,it is operated by proprietary firmware.• Smartphones – According to Wiki, “Asmartphone is a mobile phone built on amobile operating system, with moreadvanced computing capability andconnectivity than a feature phone.”

A comprehensive testing approach should cover both.According to Don Kellogg,”in 2011, feature phonesaccounted for 60 percent of the mobile telephones in theUnited States and 70 percent ofmobile phones sold worldwide.”

Yet smartphones are increasing in popularity each year:• venturebeat.com conducted a survey in 2012.According to the survey, about 50% ofUS  mobleconsumers owned smartphones. By 2013this number would increase to 70%.• According to telecomsmarketresearch.com, thenumber of active European mobile subscribers is860  million.

A good sign of increasing mobile phone usage, is the numberof application downloads. The mobile phone applicationscontinue to multiply. Let's look at some data from aSeptember 2012 research report by Gartner:

Feature phones are still popularTesting teams must still focus onfeature phones. These lower costdevices are popular with seniorcitizens and other groups.

4© 2012 by Roger Richardson and Randy Kemp

• Free Mobile applications make up 89% of alldownloads – most others are under $3..• They estimate in 2012 there will be 46billion mobile application downloads. This isalmost twice the 2011 figures of 25 billiondownloads.• They are predicting by 2016, there will be310 billion app downloads – 93% being freeapps.

Let's look at a Table Gardner supplied:Mobile Application store downloads – Worldwide(millions of downloads), 2011­2016

During September 2012, Hugo Barra, Google’s AndroidDirector of Product Management , shared some interestingdata on Android. He said this:

• 500 million Android devices are globally active• Each day users add 1.3 million Android devices

Operating systems introduce complexityTo keep up with demands, manufacturers created varioussmartphone operating systems. The best know are IOS fromApple and Android from Google. Microsoft attempts topartner with manufacturers to gain market share.

Downloads add complexityThey also increase consumerdemand for mobile phones.Increased demand brings increasedtesting needs.

5© 2012 by Roger Richardson and Randy Kemp

Operating systems introduce complexityTo keep up with demands, manufacturers created varioussmartphone operating systems. The best know are IOS fromApple and Android from Google. Microsoft attempts topartner with manufacturers to gain market share.Just as PC operating systems can multi­task, so cansmartphone operating systems. This means testers need tolook at combinations ­ not just serial events. Can we capturevarious multi­task operationsUser InteractionOne thing missing from many testingapproaches is user interaction. Perhaps theuser is downloading email, running a weatherapplication, checking on stock picks andlooking at his calendar. These user tasks are inan active state. The user can interrupt thesetasks or start new ones. Do current testingapproaches take user interaction into account?Consider testing a new smartphone. New applications inplace must work with old software and co­exist with a user'sexpected behavior. We're not just testing a simple phone.Now we're testing smart phones with more capabilities andpossibilities to include. What happens if the user changes aphone's orientation? The user could change the physicalphone's position. This can effect both the display andchange functional approaches.Here's an example scenario to look at: the application mustwork properly with all user actions. Possible user actions arebutton , screen presses and orientation adjustments.Challenges testers faceIn 2011, IDC did a comprehensive study of softwaredevelopers. One study aspect focused on the top challengesthey faced, when developing and testing mobile application.Here are the challenges and percentages of respondents:

Are smartphones easy to use?It appears so on the surface. Butunderneath with all the softwareand hardware, many surprises canchallenge your testing team.

6© 2012 by Roger Richardson and Randy Kemp

Challenges and percentages of respondents

Looking at combinationcomplexity

• How would you approach testingthe following variables?• Suppose you had forty applications­ twenty to fifty possible user interactions.• Interruptions from users (includes all screen actions,camera and buttons, plus reaction to user actions) ­about one hundred possibilities.• Interruptions from external events beyond the userscontrol (i.e. network changes WIFI, 3G and GSM,incoming audio/ video calls, Emails, MMS, etc.).­ aboutsixty possibilities.• Interruptions from accessories, like external powerand Bluetooth ­ about 20 possible.

Tip: Testers encounter new challenges ­continued

7© 2012 by Roger Richardson and Randy Kemp

• Orientation adjustment with six possible orientationsper single screen• Orientation adjustment with six possibleorientations, with both screens open ­rotated right 90*, left 90*, normal, 180degrees and face up/down.• If we considering both the starting andending orientations in the two previousorientation adjusts, it means each testcase has 144 possible orientationcombinations.

This is a staggering number to ponder.An approach to testingTest case designers try to explore all testing possibilities.But thinking of possible interactions is difficult. Such anapproach requires a structure. For practical purposes, wewill include software applications (.ie GPS) under the featureumbrella.Next we will look at some test case categories. Thesuggested categories are static states, dynamic states,external events, user actions, actions and accessories.

Tip: Don't be overwhelmedThe actual number of test caninteractions might surprise you. Buta solid testing approach will leadtesting teams down the right road.

8© 2012 by Roger Richardson and Randy Kemp

Tip: Test effectively

9© 2012 by Roger Richardson and Randy Kemp

Now we will introduce a method called Feature InteractionMatrix. It gives testers and management a way of perceivingsystem level and behavioral issues not seen by normaltesting methods.By arranging the data into rows (states) and columns

(Events), multiple interactions can be observed. Test casescan be selected, where row and column met. Testers makingtest cases no longer have to rely on their memory. Thisapproach will choose interactions to work with.Others can accumulate the events and add to the tool. Youcan create this method with a simple Excel sheet or an onlinesystem build in platforms like .NET, PHP, Mysql andPostegeSql. We will illustrate this systemthough a series of diagrams.What it solvesTest case selection becomes a new issue.There are hundreds of combinations to selectfrom. Just choosing the right combination tofind issues, can bury the tester. It's hard torestrict selecting test cases, when they are so easily created.Are matched pairs an answer? Some pairs are moreinteresting than others.Data interruptionsWhen a data interruption interferes with a data action, theprogram controls it. The programmer is quite ready to solvethose issues. Crossing the functional edge of the controlstructure, leaves the system with different considerations.Audio hand offsAudio hand offs are particularly sensitive and may find somevery unusual issues. Some examples of incomplete hand offsare:

• Muting that reaches into areas where they shouldn't• Volume adjustments that are either maintained or lostwhen switching sources

Tip: Let FIM find what you can'tIt's hard for the human mind to thinkof all possibilities. But a goodtesting methodl can do this easily.

10© 2012 by Roger Richardson and Randy Kemp

• Audio control that gets locked with the interruptions controland never returns to system control.Network switch oversNetwork switch overs happen in the background, but can beof real interest. This is especially true when multiple RFsources switch back and forth. Consider concentrations ondoing the possible RF switching between GSM, LTE,3G,WIFI and Bluetooth. Concurrency issues can surface,where each action and program by itself works fine. Whenwe join them together, it becomes inconsistent.Cross audio control and data transfer itemsCross audio control and data transfer items area great source of issues. Your ears can pick upa single missed note or slur in the replay, whiledata transfers area active. Look at it this way:data interruptions are used to handling thedata control. When the effect of audio comesinto the play, the data control might take toomuch processing power. It could drop audio orsuspend the audio actions. Users are very sensitive tomissing the beat. So these types of transitions are a primaryinterest.Architectural visualizationUser initiated background applications ­ like a music player ­adds another diminution to testing. T Picture the music playerin the background, with another non­audio application,working in the foreground. Then put an incoming call inplace. You would think that it is the same thing as testing themusic player in the foreground, with an interruption ­ but itisn't. Problems found can include:

• Loss of control of the audio• A complete lack of being able to return to thebackground player.

Tip: Save money with soundtesting

11© 2012 by Roger Richardson and Randy Kemp

This same style of testing should also extend to streamingaudio on a browser if it can play in the background.Using the methodLet's explore the method through some visual displays.Orientation adjustments for each state/interruption.

25 possible combinations for each orientation

Tip: Automation is key!Testing this manually would be quitea chore! Imagine how much easierthis is using automation?

12© 2012 by Roger Richardson and Randy Kemp

Matrix LayoutThis diagram shows how to select an individual test case

from the matrix. The intersection of S1 and EI1 creates a testcase. The user starts with an S1 state, at a selectedorientation. It is adjusted to the interruption orientation, andthen the EI1 interruption occurs.

BenefitsWe hope you see how this automated approach is superior tocurrent manual approaches. Automation ofwriting the test cases saves immense amounts oftime. Typical text case output can easily be 200– 300 test cases ­ per tester ­per day. More testcases can be written than there is time to test.The problem is now making the correct test caseselections.The benefits of this approach are immense.

• Because more test case interactions can be accountedfor, there should be a significant reduction in phonesbeing returned.• The tester is free to focus on other tasks, since thecombinations are automatically calculated.• Management and marketing can focus on new mobilephone releases, without sacrificing quality.

Tip: Have the right smartphonetesting compass

13© 2012 by Roger Richardson and Randy Kemp

• Users are happier with the phone products, as theyshould have a better communication experience withothers.

Why wait?Mobiel usage keeps increasing each year,as the following graph illustrates:

Cellphone subscribers by technology

Wouldn't you like to have a sound testing methodology inplace?

Tip: Take the next stepContact the authors. Find out whatFIM can do for you smartphonetesting.

14© 2012 by Roger Richardson and Randy Kemp

About the AuthorsFor more information on the FIM test case approach, pleasecontact the authors at their given website and/or social me­dia outlets.Roger Richardson spend several years doing US Navy com­puter and hardware repairs. He then spend several yearsworking with Motorola Mobility – now owned by Google ­ onleading cell phone technology. Motorola granted him sixpatents for ideas he submitted. He's available as a techno­logy consultant, as well as an expert on various web­basedprogramming languages. You can find out more about him athttp://tinyurl.com/rr­il­linkedin.Randy Kemp spent 30 years as an employee and consultantfor several software environments, like Motorola, Allstate,Discover Card and United Airlines. He's now a copywriter,writer and ghostwriter, who specializes in software, direct re­sponse and viral marketing. Randy is a Motorola trained, sixsigma black belt. He spent several years developing soft­ware for Motorola Mobility, now owned by Google. There hehad a chance to work with Roger on his test case method.You can find out more about him at http://tinyurl.com/rlk­googleplus.