Discount Evaluation

Evaluating with experts

Agenda

Part 4 preview Heuristic Evaluation Perform HE on each other’s

prototypes Cognitive Walkthrough Perform CW on each other’s

prototypes

Project part 4

I’ll make comments on your evaluation plans by Tuesday

Perform the evaluationsClearly inform your users what you

are doing and why. If you are audio or video recording, I

prefer you use a consent form.Pilot at least once – know how long its

going to take.If you need equipment, ask me

Part 4 write up

State exactly what you did (task list, how many, questionnaires etc.)

Summarize data collected Summarize usability conclusions

based on your data Discuss implications for the prototype

based on those conclusions

Discount Evaluation Techniques

Basis:Observing users can be time-

consuming and expensiveTry to predict usability rather than

observing it directlyConserve resources (quick & low cost)

Approach - inspections

Expert reviewers usedHCI experts interact with system and

try to find potential problems and give prescriptive feedback

Best if• Haven’t used earlier prototype• Familiar with domain or task• Understand user perspectives

Does not require working system

Heuristic Evaluation

Developed by Jakob Nielsen

Several expert usability evaluators assess system based on simple and general heuristics (principles or rules of thumb)

(Web site: www.useit.com)

How many experts?

Nielsen found thatabout 5 evaluations found 75% of the problems

Above that you get more, but at decreasing efficiency

Evaluate System

Reviewers need a prototype May vary from mock-ups and storyboards to a

working system

Reviewers evaluate system based on high-level heuristics.

Where to get heuristics? http://www.useit.com/papers/heuristic/ http://www.asktog.com/basics/firstPrinciples.html

Heuristics

use simple and natural dialog

speak user’s language

minimize memory load

be consistent provide feedback

provide clearly marked exits

provide shortcuts provide good error

messages prevent errors

Neilsen’s Heuristics

visibility of system status

aesthetic and minimalist design

user control and freedom

consistency and standards

error prevention

recognition rather than recall

flexibility and efficiency of use

recognition, diagnosis and recovery from errors

help and documentation

match between system and real world

Groupware heuristics

Provide the means for intentional and appropriate verbal communication

Provide the means for intentional and appropriate gestural communication

Provide consequential communication of an individual’s embodiment

Provide consequential communication of shared artifacts (i.e. artifact feedthrough)

Provide Protection Manage the transitions between tightly and loosely-coupled

collaboration Support people with the coordination of their actions Facilitate finding collaborators and establishing contact

Baker, Greenberg, and Gutwin, CSCW 2002

Ambient heuristics

Useful and relevant information “Peripherality” of display Match between design of ambient display

and environments Sufficient information design Consistent and intuitive mapping Easy transition to more in-depth information Visibility of state Aesthetic and Pleasing Design

Mankoff, et al, CHI 2003

Process

Perform two or more passes through system inspectingFlow from screen to screenEach screen

Evaluate against heuristics Find “problems”

Subjective (if you think it is, it is)Don’t dwell on whether it is or isn’t

Debriefing

Organize all problems found by different reviewersAt this point, decide what are and

aren’t problemsGroup, structureDocument and record them

Severity Rating

Based on frequency impact persistence market impact

Rating scale 0: not a problem 1: cosmetic issue, only fixed if extra time 2: minor usability problem, low priority 3: major usability problem, high priority 4: usability catastrophe, must be fixed

Advantages

Few ethical issues to consider Inexpensive, quick

Getting someone practiced in method and knowledgeable of domain is valuable

Challenges

Very subjective assessment of problemsDepends of expertise of reviewers

Why are these the right heuristics?Others have been suggested

How to determine what is a true usability problemSome recent papers suggest that many

identified “problems” really aren’t

Let’s practice: PAL

Heuristics

use simple and natural dialog

speak user’s language

minimize memory load

be consistent provide feedback

provide clearly marked exits

provide shortcuts provide good error

messages prevent errors

Your turn

Prepare Determine your heuristics Demo your prototype to all of us

Evaluate HA eval Mini-e Mini-e eval GroupOne GroupOne eval HA

Debrief On your own, include in Part 4

Cognitive Walkthrough

More evaluation without users

Cognitive Walkthrough

Assess learnability and usability through simulation of way novice users explore and become familiar with interactive system

A usability “thought experiment” Like code walkthrough (s/w

engineering) From Polson, Lewis, et al at UC Boulder

CW: Process

Construct carefully designed tasks from system spec or screen mock-up

Walk through (cognitive & operational) activities required to go from one screen to another

Review actions needed for task, attempt to predict how users would behave and what problems they’ll encounter

CW: Assumptions

User has rough plan User explores system, looking for

actions to contribute to performance of action

User selects action seems best for desired goal

User interprets response and assesses whether progress has been made toward completing task

CW: Requirements

Description of users and their backgrounds

Description of task user is to perform Complete list of the actions required

to complete task Prototype or description of system

CW: Methodology

Step through action sequenceAction 1Response A, B, ..Action 2Response A...

For each one, ask four questions and try to construct a believability story

CW: Questions

1. Will users be trying to produce whatever effect action has?

2. Will users be able to notice that the correct action is available? (is it visible)

3. Once found, will they know it’s the right one for desired effect? (is it correct)

4. Will users understand feedback after action?

CW: Answering the Questions

1. Will user be trying to produce effect?Typical supporting evidence

• It is part of their original task• They have experience using the system• The system tells them to do it

No evidence?• Construct a failure scenario• Explain, back up opinion

CW: Next Question

2.Will user notice action is available?Typical supporting evidence

• Experience• Visible device, such as a button • Perceivable representation of an action

such as a menu item

CW: Next Question

3.Will user know it’s the right one for the effect?Typical supporting evidence

• Experience• Interface provides a visual item (such as

prompt) to connect action to result effect • All other actions look wrong

CW: Next Question

4.Will user understand the feedback?Typical supporting evidence

• Experience• Recognize a connection between a

system response and what user was trying to do

CW: Questions

1. Will users be trying to produce whatever effect action has?

2. Will users be able to notice that the correct action is available? (is it visible)

3. Once found, will they know it’s the right one for desired effect? (is it correct)

4. Will users understand feedback after action?

Let’s practice: PAL

User characteristics

Technology savy users Familiar with computers Comfortable with basic cell phone

operations Familiar with many features of cell

phone, but not expert

Task:Party helper

Heather goes to a reception with some conference attendees. Heather doesn’t know many of them but wants to make a good impression since she’ll be looking for a job in a few months, so she wants to use PAL to remind her of names of folks she meets during the evening.

Action list

Assume PAL is running…

1. When introduced to someone new, press “earmark” button and repeat the name.

2. Assume no intervening introductions…

3. When there is a break, press and hold the back navigation button until bar reaches earmark.

4. Listen to name being repeated

CW Summary

Advantages Explores important

characteristic of learnability

Novice perspective Detailed, careful

examination Working prototype

not necessary

Disadvantages Can be time

consuming May find problems

that aren’t really problems

Narrow focus, may not evaluate entire interface

Your turn

Prepare: Describe users and their experience Determine task(s) and action lists

Evaluation: HA evaluate GroupOne GroupOne evaluate Mini-e Mini-e evaluate HA

As a group, for each action, answer the 4 questions with evidence

CW: Questions

1. Will users be trying to produce whatever effect action has?

2. Will users be able to notice that the correct action is available? (is it visible)

3. Once found, will they know it’s the right one for desired effect? (is it correct)

4. Will users understand feedback after action?