discount usability engineering marti hearst (ucb sims) sims 213, ui design & development march...
TRANSCRIPT
Discount Usability Engineering
Marti Hearst (UCB SIMS)SIMS 213, UI Design &
DevelopmentMarch 2, 1999
Adapted from slide by James Landay
Last Time
UI tools good for testing more developed UI ideas
Two styles of tools– “Prototyping” vs. UI builders
Most ignore the “insides” of application
Evaluation Techniques
UI Specialists– User Studies /
Usability Testing– Heuristic
Evaluation
Interface Designers and Implementors– Software Guidelines– Cognitive
Walkthroughs
What is Heuristic Evaluation?
A “discount” usability testing method UI experts inspect an interface
– prototype initially– later, the full system
Check the design against a list of design guidelines or heuristics
Present a list of problems to the UI designers and developers, ranked by severity
Adapted from slide by James Landay
Phases of Heuristic Evaluation
1) Pre-evaluation training– give evaluators needed domain knowledge and
information on the scenario
2) Evaluation– individuals evaluate and then aggregate
results
3) Severity rating– determine how severe each problem is
(priority)
4) Debriefing– discuss the outcome with design team
Heuristic Evaluation Developed by Jakob Nielsen Helps find usability problems in a UI design Small set (3-5) of evaluators examine UI
– independently check for compliance with usability principles (“heuristics”)
– different evaluators will find different problems– evaluators only communicate afterwards
» findings are then aggregated
Can perform on working UI or on sketches
Why Multiple Evaluators?
Every evaluator doesn’t find every problem Good evaluators find both easy & hard ones
Heuristics (original)
H1-1: Simple & natural dialog
H1-2: Speak the users’ language
H1-3: Minimize users’ memory load
H1-4: Consistency H1-5: Feedback
H1-6: Clearly marked exits
H1-7: Shortcuts H1-8: Precise &
constructive error messages
H1-9: Prevent errors H1-10: Help and
documentation
Adapted from slide by James Landay
How to Perform H.E. At least two passes for each evaluator
– first to get feel for flow and scope of system– second to focus on specific elements
Assistance from implementors/domain experts– If system is walk-up-and-use or evaluators are
domain experts, then no assistance needed– Otherwise might supply evaluators with
scenarios and have implementors standing by
Adapted from slide by James Landay
How to Perform H.E. Each evaluator produces list of
problems– Explain each problem with reference to
heuristics or other information– Be specific– List each problem separately
Adapted from slide by James Landay
Example Problem Descriptions
Can’t copy info from one window to another– Violates “Minimize the users’ memory load” (H1-
3)– Fix: allow copying
Typography uses mix of upper/lower case formats and fonts– Violates “Consistency and standards” (H2-4)– Slows users down– Probably wouldn’t be found by user testing– Fix: pick a single format for entire interface
Adapted from slide by James Landay
How to Perform Evaluation
Where problems may be found– single location in UI– two or more locations that need to be
compared– problem with overall structure of UI– something that is missing
» this part is hard to find with paper prototypes so work extra hard on those
Adapted from slide by James Landay
Debriefing
Conduct with evaluators, observers, and development team members
Discuss general characteristics of UI Suggest potential improvements to
address major usability problems Dev. team rates how hard things are
to fix Make it a brainstorming session
– little criticism until end of session
Adapted from slide by James Landay
Severity Rating Used to allocate resources to fix problems Estimates of need for more usability efforts Combination of
– frequency– impact– persistence (one time or repeating)
Should be calculated after all evaluations are done
Should be done independently by all evaluators
Adapted from slide by James Landay
Severity Ratings (from Nielsen & Mack 94)
0 - don’t agree that this is a usability problem
1 - cosmetic problem 2 - minor usability problem3 - major usability problem; important to fix4 - usability catastrophe; imperative to fix
Adapted from slide by James Landay
Severity Ratings Example
1. [H1-4 Consistency] [Severity 3]
The interface used the string "Save" on the first screen for saving the user's file, but used the string "Write file" on the second screen. Users may be confused by this different terminology for the same function.
Adapted from slide by James Landay
Results of Using HE
Discount: benefit-cost ratio of 48 [Nielsen94]– cost was $10,500 for benefit of $500,000– value of each problem ~15K (Nielsen &
Landauer)– how might we calculate this value?
» in-house productivity; open market sales
There is a correlation between severity & finding w/ HE
Adapted from slide by James Landay
Results of Using HE (cont.)
Single evaluator achieves poor results– only finds 35% of usability problems– 5 evaluators find ~ 75% of usability
problems– why not more evaluators? 10? 20?
»adding evaluators costs more»adding more evaluators doesn’t increase
the number of unique problems found
Adapted from slide by James Landay
Decreasing Returns
problems found benefits / cost
Caveat: graphs for a specific example
Adapted from slide by James Landay
Summary of H.E. Heuristic evaluation is a “discount” method Procedure
– Evaluators go through the UI twice– Check to see if it complies with heuristics– Note where it doesn’t and say why
Follow-up– Combine the findings from 3 to 5 evaluators– Have evaluators independently rate severity– Discuss problems with design team
Alternate with user testing
Study by Jeffries et al. 91
Compared the four techniques– Evaluating the HP-VUE windows-like
OS interface – Used a standardized usability
problem reporting form
Study by Jeffries et al. 91
Compared the four techniques– Usability test
»conducted by a human factors professional»six participants (subjects)»3 hours learning, 2 hours doing ten tasks
– Software guidelines»62 internal guidelines»3 software engineers (had not done the
implementation, but had relevant experience)
Study by Jeffries et al. 91
Compared the four techniques– Cognitive Walkthrough
»3 software engineers, as a group (had not done the implementation but had relevant experience)
»procedure: developers “step through” the interface in the context of core tasks a typical user will need to accomplish
»used tasks selected by Jeffries et al.»a pilot test was done first
Study by Jeffries et al. 91
Compared the four techniques– Heuristic Evaluation
»4 HCI researchers»two-week period (interspersed evaluation
with the rest of their tasks)
Results of Study
H.E. group found the largest number of unique problems
H.E. group found more than 50% of all problems found
Results of Study Problems Discovered
H.E. U.T. S.G. C.W.
Technique 105 30 13 30
Side Eff ect -- 1 8 5
PriorExperience
-- 0 12 0
Time Required (in person-hours)
H.E. U.T. S.G. C.W.
EvaluationTime
20 199 17 27
TechniqueTrainingTime
-- -- 5 27
HP-VUETrainingTime
15 -- 64 6
Average Severity Ratings
Partition into three groups Classify results into two of these
– most severe / least severe– average >=3.86 / average <=2.86
HE 28/52 UT 18/2 SG 12/11 CW 9/10
Benefit/Cost Ratios
Average severity rating/time– HE: 12– UT: 1– SG: 2 – CW: 3– (S.G. becomes 6 if you factor in the
time required for HP-VUE training)
Adapted from slide by James Landay
HE vs. User Testing HE is much faster
– 1-2 hours each evaluator vs. days-weeks– finds some problems user testing does not find
HE doesn’t require interpreting user’s actions User testing is more accurate (by def.)
– takes into account actual users and tasks– finds some problems HE doesn’t find– HE may miss problems & find “false positives”
Good to alternate between HE and user testing– find different problems– don’t waste participants