clasen sample usability test
DESCRIPTION
Sample Usability TestTRANSCRIPT
LISCI310‐Clasen‐IndividualProject Page1of15
UW‐MilwaukeeL&ISCI310
CommonIndustryFormatforUsabilityTestReportv2.0iPhone3Gver.4.1forElderlyUsers
TestLeadandReportPreparedbyWilliamClasenTestConducted:April4‐6,2011ReportPrepared:April7‐8,2011
Contents PageExecutiveSummary 2
Introduction 3
FullProductDescription 3
TestObjectives 3
Methodology 4‐9
ParticipantProfile 4
ContextofProductUseintheTest 4
TaskList 5
TestFacility 6
DeviceEnvironment 6
Design,Procedure,andInstructions 7
UsabilityMetrics 8
Results 9‐12
DataAnalysis 9
Task1:AccessWeb 9
Task2:SendTextMessage 10
Task3:SendEmail 12
Recommendations/Discussion 13‐14
Bibliography 15
Contact:WilliamClasen(608)921‐8324
Janesville,WI53546
LISCI310‐Clasen‐IndividualProject Page2of15
ExecutiveSummary
Manyservicesreadilyutilizedbyseniorsaremigratingonline.
Marketinganalystshaveidentifiedthreecommongoalswithrespectto
usersofsmartphonesoverfifty‐yearsofage:textmessaging,email,and
webconnectivity.AusabilitytestwasconductedtargetingtheiPhone
3Gver.4.1(pictured)ontheAT&Tcellularnetworkforusebyelderlyclientele.Thisdocument
containstestmethodology,efficiency/effectivenessratings,taskcompletionrates,timeontask,
errors,participantfeedback,anddiscussionforimprovementsbasedonapplicabletheory.
Fourparticipantsovertheageoffifty‐fivewereselectedtoparticipateinthreetasksthat
targetedthefunctionsrequiringevaluation.Halfoftheparticipantshadprevioussmartphone
experiencewithadifferentdevice.Allparticipantsweredailyusersoftheweb,textmessaging,and
email.Overall,theparticipantsappearedveryreceptivetosmartphone
technology.Onlyahandfulofmodificationsarerequiredtomitigatesensoryand
cognitivechallengesexperiencedbythetargetusergroup:
Change Justification Severity1. Emphasizezoomandmagnifyin
marketing,sales,andhelp2. Add'magnify'buttonin
supportingapplicationstoincreasefontsizeall‐at‐once
Mitigationofsensorychallengesthatresultfromaging.
Allfourparticipantsfailedtoacknowledgethezoomandmagnifyfeaturestoreduceeyestrain.
25%ofparticipantsfailedtocompletewebbrowsingtaskasaresult
High
3. Auto‐correctiondefaultstatusto'off'orprovidelinkdirectlytothissettingonthehomescreen
Mitigationofcognitivestabilitychallengesthatresultfromaging.
50%ofparticipantsindicatedthisfeaturewas"irritating"or"distracting",butdidnotknowhowtoturnitoff
Moderate
4. Addcontextuallabelto'createnew'icon
5. ModifycontextuallabelforSafarito"WebBrowser"
6. Change'plus'iconto'contact'iconforconsistencyacrossallplatformapplications
Mitigationofcognitivesimplicityandmemorabilitychallengesthatresultfromaging.
75%ofusersfailedtorecognizeSafariasawebbrowserapplication
100%ofusersfailedtorecognizethe'plus'icontoaccessthecontactlist
75%failedtorecognizethe'plus'iconinsubsequenttasksafterassistance
High
LISCI310‐Clasen‐IndividualProject Page3of15
Introduction
ProductDescription
TheiPhone3GiOSver.4.1(picturedbelow)asmanufacturedbyAppleandoperatedonthe
AT&Tcellularnetworkisbeingmarketedasasmartphoneformainstreamusers.Functionssuchas
textmessaging,email,andwebconnectivityofferedbysuchdevicesareofcurrentinterestto
smartphonemarketinganalystswithregardtoseniorcitizens.(Greengard,p21)
Manycommerce,healthcare,andgovernmentservicesreadilyutilizedbyseniorsare
migratingonline.Greengardmaintainsthatelderlyclientsareentirelyreceptivetosmartphone
technologysolongasitusefulandeasytooperate(Greengard,p20).Inordertoachieveusefulness
andeaseofoperation,thedesignmustmitigatechallengespresentedbydegradationofvision,
tactileperception,andcognitionastheageofclienteleadvancesbeyondfifty‐fiveyearsofage.
(Greengard,p20‐1).
TestObjectives
Thisreportassessesusabilityanddiscussesimprovementforthisdevicewithrespectto
seniorcitizensandtheircommongoalsforfunctionality.Inparticular,findingsregardingweb
navigation,sendingtextmessages,andsendingemailsarereported.Audio,asitisunrelatedto
theseparticularfunctions,isnotevaluated.
LISCI310‐Clasen‐IndividualProject Page4of15
Methodology
ParticipantProfile
Allfourparticipantswerechosenasarepresentativesampleofthetargeteduser
population.Criteriaincludedageandprevioussmartphoneexperience.Allparticipantsareolder
thanfifty‐fiveyearsofage.Participantsweresegmentedbasedonprevioussmartphoneexperience.
TwoparticipantseachhadtwoyearsofexperienceusingwebenabledBlackberrydevices.Two
participantshadnopreviousexperiencewithsmartphones.
Priortoevaluation,eachparticipantwasaskedabouttheiraveragedailyinvolvementwith
thefollowingfunctions:hoursspentaccessingtheweb,numberoftextmessagessent,andnumber
ofemailssent.Thequestionsdidnotspecifycellulardevices.Thegoalwassimply
toascertainoverallawarenessandexperiencewiththesefunctionsviaany
availabletechnologythattheymayhavehad.Itwasfurtherdeterminedthatnone
oftheparticipantshadanyexperiencewithAppledevicesorsoftware.
Key Name Age Gender SmartphoneExperience(Years)
DailyWebAccess(Hours)
DailyTextMessagesSent
DailyEmailsSent
P1 JerryClasen 66 Male 2 2 12 8P2 CarolClasen 68 Female 2 4 10 15P3 JamesWuetrich 70 Male None 3 0 4P4 DorothyWuetrich 68 Female None 2 3 12
ContextofProductUseinTest
Thereisonesignificantdifferencebetweentheevaluatedcontextandtheexpectedcontext
ofuse.Theaudiocapabilitiesofthissmartphonedevicearenotcriticaltoevaluatingwebaccess,
textmessaging,andemailfunctionsofthetargetusergroup.Audiocapabilities,asseenlargelya
componentoftraditionalcellularphonesandmultimediausesassociatedwiththesmartphone,are
notevaluatedinthisreport.
Usebythetargetgroupwillbepersonalratherthanprofessionalandevaluatedassuch.
LISCI310‐Clasen‐IndividualProject Page5of15
TaskList
Taskswereselectedtoevaluatethemostcommongoalsassociatedwiththedevicerelative
tothetargetaudienceasrepresentedbytheparticipants.
Task1Task Viewapple.comwebsiteandnavigateto[support]State HomescreenSSC Participant
1. openssafariapplication(33%)2. entersapple.cominaddressbar(33%)3. activateslinktosupporttablocatedonapple.comhomepage(33%)
Benchmark 0.5minute.TaskFailureat2.5minutes.Script Pleaseopentheinternetbrowserandgo toapple.com.Oncetheapple.comwebpage
hasloaded,findthesupportlinkonthatpageandclickit.
Task2Task Sendtextmessagetocontact JohnDoeState HomescreenSSC Participant
1. opensmessageapplication(25%)2. selectscontact(25%)3. typesmessage(25%)4. sendsmessage(25%)
Benchmark 0.5minute.TaskFailureat2.5minutes.Script JohnDoe'scontactinformationhasbeenstoredinthephone.Pleasesendthe
followingtextmessagetoJohnDoe:Hello
Task3Task Send30wordemailtocontact John DoeState Homescreen.Participant isgivenhardcopyof50wordemailtoduplicateandsend.SSC Participant
1. opensemailapplication(25%)2. selectscontact(25%)3. typesemailcontent(25%)4. sends(25%)
Benchmark 1.5minutes.TaskFailureat7.5minutes.Script JohnDoe'scontactinformationhasbeenstoredinthephone.Pleasesendthe
followingemailtoJohnDoe:DearJohn,YouarereceivingthefirstemailthatIamsendingfromanIPhone.IamhelpingtoconductausabilitystudyforAT&T.Ihopealliswell.
LISCI310‐Clasen‐IndividualProject Page6of15
TestFacility
Participantswereobservedperformingthetaskssetforthbythisevaluationattheir
respectivehomesinordertoreplicatetypicaluserenvironment.
DeviceEnvironment
Allparticipantsusedthesamedevice(Serial#87926MD8Y7H)inordertonegatehardware
specificvariables.
AllparticipantswereevaluatedintheirhomesinthesamegenerallocationwithintheAT&T
cellularnetworkinordertonegatecellularsignalstrengthvariables.
Thefollowingtechnicalspecificationsrepresentaspectsofthedeviceenvironmentthatare
subjecttothisreport:
1. Sizeandweight
a. Height: 4.5inches(115.5mm)
b. Width: 2.4inches(62.1mm)
c. Depth: 0.48inch(12.3mm)
d. Weight: 4.7ounces(133grams)
2. Capacity
a. 8GBflashdrive
3. Cellularandwireless
a. AT&T3Gcellularnetwork
b. GSM/EDGE(850,900,1800,1900MHz)
4. Display
a. 3.5‐inch(diagonal)widescreenMulti‐Touchdisplay
b. 480‐by‐320‐pixelresolutionat163ppi
c. 16M(full)color
d. Supportfordisplayofmultiplelanguagesandcharacterssimultaneously
iPhone3Gver.4.1onAT&Tcellularnetwork
LISCI310‐Clasen‐IndividualProject Page7of15
Task Evaluation
Section 1
Experience
Section 2
Orientation
Section 3
Tests
Section 4
Debriefing
TestMonitorRole
Thetestadministratorgreetedparticipants,explainedeachtaskandprovidesupporting
materialaseachtaskisintroduced,andrespondedtorequestsforassistanceonlyifaskeddirectly
bytheparticipant.AssistanceistobelimitedtothecurrentSSCsegmentforthetaskonhand.
Design,Procedure,andInstructions
Thetaskevaluationwillconsistofa
performancetestmadeoffoursections:
1. Participantgreetingandbackground
Eachparticipantwillbepersonally
greetedbythetestmonitorandmade
tofeelcomfortableandrelaxed.The
testmonitorwillrecordparticipant
responsestothefollowquestions:
a. Whatisyourname?
b. Whatisyourage?
c. Haveyoueverusedasmartphone?Ifso,forhowlong?
d. Howmuchtimeperdaywouldyouestimatethatyouaveragebrowsingthewebon
cellphones,computers,TVbrowsers,etc.?
e. Howmanyinstantortextmessagesperdaywouldyouestimatethatyousendon
averageviacellphones,computers,TVbrowsers,etc.?
f. Howmanyemailsperdaywouldyouestimatethatyousendonaverageviacell
phones,computers,TVbrowsers,etc.?
FourSectionTaskEvaluationModel
LISCI310‐Clasen‐IndividualProject Page8of15
2. Orientation
Participantswilleachreceiveascripted,verbalintroductionandorientationwhichwill
explainthepurposeandgoalsofthetest.Theywillalsobeassuredthatthedeviceiswhat
isbeingtested,notthemselves.
3. PerformanceTest
Theperformancetestconsistsofaseriesoftasksthattheparticipantswillcarryoutwhile
beingobserved.Thesetaskswillbeconductedinthefollowingmanner:
a. Participantwillbehandedascript,whichthetestmonitorwillreadoutloud,
detailingthetasktobeaccomplished.Theywillthenattemptthattask.
b. Duringtheperformancetestthemonitorwillmakenotesonelapsedtimeand
participanterrors.Thetestmonitorwillrespondonlytodirectrequestsfor
assistancebyverballyofferingproceduralhelp.Thetestmonitorwilldocument
numberofrequestsandatwhatstageduringthetasktheyoccurred.Themonitor
willalsomakenoteofanyunusualcircumstances,suchasincorrectkeystrokes,that
maynotbepresentedasdirectmeasuresoftheSSC.
c. Afterconcludingthetasks,theparticipantwillbedebriefedbythetestmonitor.
4. ParticipantDebriefing
Eachparticipantwillbedebriefedbythetestmonitor.Eachsessionwillbedocumentedby
thetestmonitor.Thequestionisasfollows:
a. DoyouhaveanyparticularfeedbackorrecommendationsfortheiPhonebasedon
thetaskyoujustperformed?
UsabilityMetrics
1. Measures of effectiveness include percent of task
completion, quantity of errors, and frequency of assists.
2. Measures of efficiency includes time taken to achieve task.
Usabilitymetricstodeterminesatisfaction
LISCI310‐Clasen‐IndividualProject Page9of15
Results
DataAnalysis
Taskcompletionissegmentedasapercentageinthetasklistforthepurposeofquantifying
results.Assistsrepresenteachdirectrequestfromtheusertothetestmonitorforassistance.
Tasktimebenchmarks,aspresentedinthetasklist,aremeasuresoftimerepresentingthe
durationofthetask,roundeduptothenearest0.5minuteinterval,ascompletedbythetest
monitor,WilliamClasen.WilliamClasenhasbeenadailymainstreamuserofthedevicefortwo
years.
Tasktimefailurearebenchmarkstimesmultipliedbyfive.Taskfailuretimesrepresent
time‐outduetofrustration.
Keystrokeerrorsrepresenteverykeystrokeonthekeypadthatdidnotleaddirectlytoward
taskcompletion.
Task1AccessWebSite
Key UnassistedTaskEffectiveness
(%TaskComplete)
AssistedTaskEffectiveness
(%TaskComplete)
Assists TaskTime(min)
KeystrokeErrors
P1 0 100 2 1.25 6P2 66 100 1 1.00 0P3 0 66 0 2.50 8P4 0 100 2 1.75 2
P1,P3,andP4failedtorecognizethedefaultwebbrowser
application,Safari.Asaresult,twooutoffourparticipantsasked,"How
doIgettotheweb?"Oneparticipantasked,"WhereisExplorer?"
Allparticipantsfailedtoimplementpinch‐zoomormagnifyas
toolstoexpandtextontheapple.comhomepage.Thesefeatureswere
necessarytorenderthesupportlinklegible.Activatingthesupportlink
representedthefinal33%oftheSSC.AllparticipantsthataskedforTask 1 ‐ Web Access
LISCI310‐Clasen‐IndividualProject Page10of15
assistanceregardingthisfeatureasked,"HowdoIzoom?"P3didnotseekassistancewiththezoom
featureandtimed‐outwhilerandomlytappingthescreeninhopesoffindingthecorrectlink.All
otherkeystrokeerrorsresultedfromsinglecharactertypographicalerrorsandthesubsequent
deletionoftheerrantcharacter.
Figure 1 ‐ Task 1 Time and Assists
Allparticipantsexceededthebenchmarktimeof0.5minutesforexperiencedmainstream
users.P1andP2,bothexperiencedsmartphoneusers,askedmoreimmediatequestionswhile
requestingassistanceandcompletedthetaskfasterthanP3andP4,non‐experiencedsmartphone
users.Duringthetaskdebriefing,P2andP3expressedfrustrationwiththedurationofthetask.P1
andP2suggestedthatthezoomfeaturewasnotveryintuitive.
Task2SendTextMessage
Key UnassistedTaskEffectiveness
(%TaskComplete)
AssistedTaskEffectiveness
(%TaskComplete)
Assists TaskTime(min)
KeystrokeErrors
P1 25 100 1 1.00 4P2 25 100 1 0.75 2P3 0 100 2 2.00 10P4 0 100 2 1.50 8
0
0.5
1
1.5
2
2.5
Benchmark P1 P2 P3 P4
Time (min)
Assists
LISCI310‐Clasen‐IndividualProject Page11of15
P3andP4didnotknowhowtoaccessthetextmessaging
applicationonthedevice.Thetestmonitornoticedthatthesetwo
participantsheldthephonecloseasifstrainedtoreadthetextlabelsthat
accompaniedicons.P3,after25secondsand4randomkeystrokes,and
P4,after15secondsand2randomkeystrokes,askedthetestmonitor
howtoaccesstheapplication.P1andP2accessedtheapplicationdirectly
fromthehomepageratherthanbrowsingthroughthecontacts.
Noneoftheparticipantsrecognizedthe'plus'iconasthewayto
accessthecontactlist.Eachparticipantaveraged15secondssearchingthescreenbeforeaskingthe
testmonitorhowtoaccessthecontactlist.Similarobservationsweremaderegardingthe'create
newicon'.Allfourparticipantsexpressedfrustrationregardingtheiconasunintuitive.P2
suggestedthatthecontactsicononthehomepagewasmorerecognizable.
Keystrokeerrors,otherthanthosereportedabove,wereduetotypographicalerrorsand
subsequentcorrection.P2statedthekeypadresponsiveness,"seemedslow".P1andP3statedthat
theautocorrectfeature,"wasannoying".P4offeredanunsolicitedstatementthattheautocorrect
feature,"washandy",thoughitappearedtotakeP4moretimetoevaluatetheoutput.
Figure 2 ‐ Task 2 Time and Assists
Allparticipantsexceededthebenchmarktimeof0.5minutesforexperiencedmainstream
users.P1andP2,bothexperiencedsmartphoneusers,completedthetaskfasterthanP3andP4,
0
0.5
1
1.5
2
2.5
Benchmark P1 P2 P3 P4
Time (min)
Assists
Task 2 ‐ Text Message
LISCI310‐Clasen‐IndividualProject Page12of15
non‐experiencedsmartphoneusers.Allparticipantscompletedthetaskwellbeforethetaskfailure
timeof2.5minutes.
Duringthedebriefing,P1andP2feltthattheprocessoverallwas"simple".P3offeredthat
thetaskmadehimfeel,"empowered".P4suggested,"Icouldgetusedtothis."
Task3SendEmail
Key UnassistedTaskEffectiveness
(%TaskComplete)
AssistedTaskEffectiveness
(%TaskComplete)
Assists TaskTime(min)
KeystrokeErrors
P1 100 100 0 3.00 8P2 100 100 0 2.00 4P3 25 100 2 5.50 20P4 25 100 1 4.50 12
P1andP2,presumablyfromhavinghadpreviousexperience
emailingfromsmartphonedevices,completedthetaskwithout
assistance.P3andP4bothrecognizedthemailapplication,butfailedto
recallthatthe'plus'iconprovidedaccesstothecontactlist.P3,after10
secondsand4randomkeystrokes,andP4,after15secondsand6
randomkeystrokes,askedthetestmonitorhowtoaccessthecontactlist.
Theremainingkeystrokeerrorswereduetotypographicalerrorsand
subsequentcorrection.Again,P1andP3expressedfrustrationregarding
theautocorrectfeature.
Figure 3 ‐ Task 3 Time and Assists
0
1
2
3
4
5
6
7
Benchmark P1 P2 P3 P4
Time (min)
Assists
Task 3 ‐ Email
LISCI310‐Clasen‐IndividualProject Page13of15
Allparticipantsexceededthebenchmarktimeof2.5minutesforexperiencedmainstream
users.P1andP2,bothexperiencedsmartphoneusers,againcompletedthetaskfasterthanP3and
P4,non‐experiencedsmartphoneusers.Allparticipantscompletedthetaskwellbeforethetask
failuretimeof7.5minutes.Duringthedebriefing,allparticipantscommentedthattheaccesstothe
contactlistinthemailprogramwasnotintuitive.P1andP2suggestedthatthecontacticononthe
homepagebeutilizedinsteadofthe'plus'icon.
Recommendations/Discussion
Asidefromsomesensoryandcognitivechallenges,theparticipantsexpressedeagerness
regardingthedevice.25%oftheparticipantsfelt"empowered"bythetasks.Thefollowing
guidelineswilloffersupportforassistingelderlyclientswithcellularweb‐basedinteractionbased
ontheaforementionedobservations.
Sensorydesignprinciplesrevolvearoundvisibilityoffonts,
pointers,andiconsinconsiderationoftheelderlymarket.Thefont
sizeusedonboththescreenandkeypadshouldsupportvisual
limitationsthatresultfromaging(Nielsen,2002).Nielsenfurther
stipulatesthataminimumtwelvepointfontshouldbeutilized.The
iPhone3Gprovidesuserswithwaystomitigatesensoryrestraints
andincreaseoverallusability.Asaresultofthistest,itappearsthattheexistenceofthesefeatures
isnotapparent.Itisrecommendedthatmarketingtothetargetgroupemphasizesaccesstothe
pinch‐zoomandmagnifyfeatures.Asafurthersuggestion,aniconrepresentinga
magnifyingglass(pictured)couldbeincludedinpredominantlytextdriven
applicationsasawaytocallattentiontothemagnifyfeature.Alternatively,theicon
couldservetoincreaseonscreenfontsizeallatonce.
Inordertomaximizeusability,thecognitiveloadrequiredtocarryouttasksmustbe
reducedtoaminimumwithregardtoelderlyclients(Greengard,p21).Cognitiveloadrefersto
Recommendmagnifyicon
Marketing and help should emphasize zoom feature
LISCI310‐Clasen‐IndividualProject Page14of15
stability,simplicity,memorability,andpredictabilityindeterminingtheusefulnessofdesign
principles(Heim,pp198‐208).
Intermsofstability,theelderlyrequireinterfacesthatareforgivingtothepointofallowing
multipleformatsfordataentrysuchastelephoneandcreditcardnumbers(Nielsen,2002).During
thedebriefingfortasks3and4,50%ofparticipantsindicatedthattheinherentauto‐correct
featurethatprovidessuggestionswhiletypingwas"irritating"or
"distracting".Whenmarketingtoelderlyusers,specialattention
shouldbegiventowardasolutionthatwillmakethelocationofthesettingtoturnoffthisfeature
(pictured)morereadilyaccessible.Alternatively,thedefaultsettingcouldbestetto'off'.
Mainstreamuserswillbemoreadeptatfindingthesettingtoturnitbackonifdesired.Forgiving
interfacesrequirethaterrormessagesbeeliminated,ifpossible,duetoobscurityandimprecision
thatmayresultincognitivedisconnect(Nielsen,2002).
Intermsofsimplicity,windowsshouldutilizeasingledocument
interface(SDI)andshortcontextualmenusifastaticinterfaceis
unattainable(Heim,pp368‐69,376‐86).TheiPhone3Gpresentsanfairly
straightforwardSDI.Iconspresentedcontextualdilemmasforallofthe
elderlyparticipants,however.Intask2,allparticipantsfailedto
immediatelyrecognizethe'createnew'icon.Thisfunction(pictured)is
representedbyarelativelysmallsquarecontainingahashmark.The
usersexpressedfrustrationregardingthemeaningoftheicon.Inorder
toincreaseusabilityforseniors,itisrecommendedthatthisiconbegivena
contextuallabel.Alongsimilaranalysis,threeoutoffourusersdidnotrecognize
theSafariapplicationicon(pictured)asaccesstoaninternetbrowser.Moreissues
regardingiconrecognitionaredescribedbelow.
Browsericonnotrecognizeable
'CreateNew'iconnotidentifiable
LISCI310‐Clasen‐IndividualProject Page15of15
Memorabilityrequireseventstobepronounced(Nielsen,2002).Asthetest
resultsindicate,fiftypercentofuserscannotbeexpectedtorecallicon
identificationevenafterthefunctionoftheiconisillustrated.25%ofparticipants
suggestedthatthecontactsiconvisibleonthehomescreenwasmorerecognizable.
Inthesamevein,predictabilityrequirestheinterfacetobefamiliar(Greengard,
p20).Inordertoovercometheobstaclesapparenttoaccessingthecontactlist,itisrecommended
thatthe'plus'iconbereplacedwiththemorecommonlyrecognizedcontactlisticonaspictured.
Inordertomaintainutility(Heim,p198)anysmartphonedeviceofferedtotheelderly
shouldincorporatebroadbandinternetaccess.Inordertosatisfythedemandsforwebaccess,text
messaging,andemailbyusersoverfifty‐fiveyearsofage,theinterfacemustmitigatesensoryand
cognitiveabilitiesasageprogresses.(Greengard,p21)Shouldphoneandmulti‐mediamarketing
analysisbedesired,audiousabilitytestingmayberequired.
Bibliography
Nielsen,J.2002.UsabilityforSeniorCitizens[online].Available:http://www.useit.com/alertbox/seniors.html(accessedMarch12,2011).
Greengard,Samuel."FacinganAge‐OldProblem"CommunicationsoftheACM.Sep.2009:20‐22.
Heim,Steven.TheResonantInterface.NewYork:PearsonEducation,Inc.2008.Print.
Contact icons