remote user testing
TRANSCRIPT
Evaluation for 2008
Michael GaiggMarch, 18th 2009
Remote User Testing
Remote User Testing 2
Agenda
• Show how – User-centered Design helps teams to save:
• Time• Money• Aggravation from clients
– Usability testing early in the design process identifies:• Areas of confusion and frustration • Where the user can’t complete a task • While it is still cheap and easy to mitigate these problems
03/18/2009
Remote User Testing 303/18/2009
Agenda
• User-centered Design• User Testing• Output (Qualitative & Quantitative)• Software & Usage• Time Estimates & Cost• Advantages & Disadvantages
Remote User Testing 4
Introduction to User-centered Design
• UCD is a design philosophy and a process in which the needs, wants, and limitations of the end user of an interface or document are given extensive attention at each stage of the design process.
• Its measures are:– Effectiveness (accuracy and completeness to achieve goals)– Efficiency (resources expended in relation to the accuracy
and completeness)– Satisfaction (comfort and acceptability of the work system to
its users)
03/18/2009
Remote User Testing 5
User Testing is a UCD Method
03/18/2009
Method Cost Output Sample Size When to Use
Competitive Studies Medium Stat. & Non-Stat. 5 Requirements
Focus Groups High Non-Statistical 6-9 Requirements
Field Studies High Non-Statistical 2-3 Requirements
Heuristic Evaluation Low Statistical 2-3 Design
Paper Prototyping Medium Stat. & Non-Stat. 5 Design
Card Sorting High Statistical 15-20 Design
Participatory Design Low Non-Statistical n/a Design
User Testing Medium Stat. & Non-Stat. 5 Design & Evaluation
Surveys Low Statistical 20+ Requirements & Evaluation
Interviews High Non-Statistical 3-5 Requirements & Evaluation
Server Traffic Log A. Low Statistical n/a Evaluation
Search Log Analysis Low Statistical n/a Evaluation
Remote User Testing 6
Overview
• User testing involves: – A design– A user (participant) – An observer (facilitator).
• Observe user behavior & receive user feedback.• Test tasks (not functions).
03/18/2009
Observer
UserDesign
Remote User Testing 7
Qualitative Output
• Understand mental model of the user (navigation).• Validate requirements and design assumptions
(focus on important functionality early).• Discover non-functional design mistakes and areas
of improvement (unforeseen user requirements).• Evaluate effectiveness of design elements.
03/18/2009
Remote User Testing 8
Quantitative Output
• Success rates / Error rates• Task time• Satisfaction ratings
03/18/2009
Task Success Task Time
Remote User Testing 9
User Testing Software
• TechSmith Morae• Manager: $ 1,295.-• Bundle: $ 1,495.-
• Morae Manager uses the collected data (observation markers and notes, video, keyboard and mouse inputs) to analyze and calculate task times, error quotes and other common measurements.
03/18/2009
Remote User Testing 10
Remote User Testing Software
• TechSmith UserVue• $ 149.- / Month or• $ 1,495. / Year
• Web-based app enabling a Facilitator to remotely observe (and record) a Participant using a phone line for communication.
• Multiple Observers can passively join the Session and share their observations with the Facilitator.
03/18/2009
Remote User Testing 11
The Session
• Setting up a session.• Connecting (download of
software bundle required).• Calling participant.• Starting the session.• Viewing (and recording) the participant.• Removing software bundle from computers of
involved parties.
03/18/2009
Remote User Testing 12
Time Estimates & Cost
• Preparation (18 hours)– User screening: 8 hours– Task creation: 8 hours– Environment: 2 hours
• User testing (10 hours)– 2 hours per user– 5 users per round
• Post-test (32 hours)– Test report: 16 hours– Implementation: 12 hours– Communication: 4 hours
03/18/2009
Our time:1 Round = 60 hours2 Rounds = 110 hrs.3 Rounds = 160 hrs.
+ User comps+ Time additional observers
Remote User Testing 13
Advantages of remote UT
• Reduced (or no) travel time and expenses.• Higher exposure through easy screen sharing
(managers can sneak in easily).• Actual user environment, familiar and comfortable.• Possibly fewer drop-outs.
03/18/2009
Remote User Testing 14
Disadvantages of remote UT
• Facilitator not physically present (degree of separation can be challenging).
• Can't see facial expressions or non-verbal cues.• Difficult to build rapport and trust.• Difficult to control environment.• Possibly technical difficulties (firewall, etc.).• Setup and use of software or usability lab might be
challenging and requires a liaison.
03/18/2009
Remote User Testing 15
Summary
• Increased usability– Higher degree of client satisfaction– Continued business = Higher revenues– Lesser client aggravation (“This is completely unusable”)
• Assist PM during requirements and design– Focus and prioritize on important functionality– Identify unforeseen user requirements early!
• Reduced costs– Training and service– Viewer Help-Desk calls
03/18/2009