why keep it to yourself? teaching everyone on the team to do usability testing
DESCRIPTION
Techniques for teaching people on your development team how to gather data from users.TRANSCRIPT
?
♫
Why keep it to yourself?Getting everyone on the team to do usability testing
Dana ChisnellIA Summit 2010
1
2
Data GutInfluence = +
3
Influence = Data < Gut
4
Influence = Data x Gut
5
Influence = Data^Gut
Throughput is a problem.
6
Influencing design decisions
7
8
Dev
DevI A
IX
D
?
U
♫
D
9
U
?
10
Dev
♫
Dev
IXDD
D
I A
11
Not either/or
12
Do it together
13
U
I
A
D
D
D
DEV
DEV
U
U
14
DEV
DEVIA D
UIX
D
Yay!
15
16
Give up control (you don’t have it, anyway)
Get the team to value data over gut
Research by lots of amateurs produces better results
Otherwise, you’ll burn out
Coach and advise instead
Work on more interesting, harder questions
Influencing design decisions
DEV
DEVIA D
UIX
D
Yay!
Improving the Gut
17
18
Tasks
Moderating
Analysis
19
Tasks: Seeing through the users’ eyes
Help each team member think of a user scenario they care about
Act out the scenario as the user
App: Church social network ✤ Roles:
✤ Senior Pastor
✤ Social Networking Pastor
✤ Volunteer
✤ Lay Leader
✤ Member
✤ First time visitor
✤ Administrative Assistant
20
✤ App: Organizing students to help homebound people vote
✤ User roles:
✤ Student
✤ Voter
✤ Organizer
✤ Trainer
✤ Scheduler
21
22
What happened?
Task scenarios set the scene, give context
You want to create realistic and relatable stories
23
Moderating: Active presence
Role play practice (in dry runs)
Video of the moderator
24
25
Moderating, not training
Impartial, unbiased observing
No teaching!
Listen and watch
Ask open-ended questions: Why? How? What?
Correct and train at the end
Good listenerExcellent memory Develops rapport, empathetic
Quick learner
Who should moderate?
26
Planning, managing data, producing reports
Scientist
Echoing with play-by-playMaximizing flow of information to observers
Sportscaster
Ensuring the safety and comfort of the participant
Flight attendant
27
Joan Dorsey, American Airlines: http://aviationblog.dallasnews.com/archives/aviation-historytrivia/
John Madden, http://www.forbes.com/2008/01/28/television-sports-madden-biz-sports-cx_lh_0128broadcasters_slide_3.html?thisSpeed=15000
Marie Curie, Smithsonian Institution http://photography.si.edu/SearchImage.aspx?t=5&id=3523&q=SIL14-C6-05
Moderators: Run the session
Observers: Note behaviors and quotes
Participant: Do what you normally would do, try to think aloud
28
29
What to look for
Hesitating
Comments
Questions
Body language
Behaviors
Technique
Think aloud - ask:
“Tell me what you’re doing”
“Tell me what you’re thinking”
30
Review at end - ask:
“Walk me through what you did”
“How’d that go?” Use the ballot as a guide for the discussion
“What was confusing or frustrating?”
31
Review
✤ Participant: What was it like?
✤ Observers: What did you see?
✤ Moderators: What questions do you have?
Mid-session reminders
“You are not being tested”
“That’s useful feedback, thank you”
“Please tell me what you’re thinking”
32
Meanwhile, you track time and tasks remaining in the session
Don’t be afraid to move on if something is taking too long
Decide ahead what is low priority or optional
Narration
Don’t be afraid to interact
Say what you are observing - don’t interpret
33
Examples:
“I see you just clicked on the Zit button.”
“Sorry, I didn’t catch that. Would you say that again?”
“That really, really stinks? Could you say what about it stinks?”
Maximizing data
✤ If the participant says “hmmm” or “oops” or “I wonder...”Say: “What questions do you have right now?”
✤ If the participant is silent for 10 - 20 seconds (count!) Say: “What are you thinking?”
✤ If the participant stops because she thinks she’s done or she’s stuck- Summarize what you saw her do - Ask what she will do next
34
Analysis: Structured discussion
Tell stories
KJ - to set priorities
Guess the reason - to exercise inference-making
Observation-inference-opinion-theory
35
KJ Analysis
reach consensus from subjective data
similar to affinity diagramming
invented by Jiro Kawakita
objective, quick
8 simple steps
36
1. Focus questionWhat needs to be fixed in Product X to improve the user experience?(observations, data)
What obstacles do teams face in implementing UE practices? (opinion)
37
2. Organize the groupCall together everyone concerned
For user research, only those who observed
Typically takes an hour
38
3. Put opinions or data on notesFor a usability test, ask for observations
(not inferences, not opinion)
No discussion
39
4. Put notes on a wallRandom
Read others’
Add items
No discussion
40
5. Group similar items In another part of the room
Start with 2 items that seem like they belong together
Place them together, one below the other
Move all stickies
Review and move among groups
Every item in a group
No discussion
41
6. Name each groupUse a different color
Each person gives each group a name
Names must be noun clusters
Split groups
Join groups
Everyone must give every group a name
No discussion
42
7. Vote for the most important groupsDemocratic sharing of opinions
Independent of influence or coercion
Each person writes their top 3
Rank the choices from most to least important
Record votes on the group sticky
No discussion
43
8. Rank the groupsPull notes with votes
Order by the number of votes
Read off the groups
Discuss combining groups
Agreement must be unanimous
Add combined groups’ votes together
Stop at 3-5 clear top priorities
44
45
Observations to directionQuantifying subjective data in 4 easy steps
46
Observations
Sources:
usability testing
user research
sales feedback
support calls and emails
training
47
What you saw
What you heard
Inferences
Judgements
Conclusions
Guesses
Intuition
48
Opinions
Review the inferences
What are the causes?
How likely is this inference to be the cause?
How often did the observation happen?
Are there any patterns in what kinds of users had issues?
49
Direction
What’s the evidence for a design change?
What does the strength of the cause suggest about a solution?
Test theories
50
Resistance from the team
51
I don’t have time
It’s not my job
I don’t know how
What if I screw up
Resistance is futile.
52
Resistance from the team
53
Exchange time in meetings for time with users
It will make you better at your job
I’ll help you learn
You won’t; it doesn’t matter
Stop pouting.
54
Yep, you’re qualified and they aren’t.
Go peddle your fish.
Haven’t you wanted more visibility?
55
A lot of sloppy data is better than a little excellent data
Trust your team to be delegated to
Shift your mindset from your butt to their Gut
Improving the Gut
U
-+ Desktop
D
UsersData
X?
Y?
?
Recommendations
Direction
56
This?
DEV
DEVIA D
UIX
D
Yay!
57
Or this?
Dev I A D DevUIX
D
Why keep it to yourself?
58
Where to learn more
59
Dana’s blog: http://usabilitytestinghowto.blogspot.com/
Download templates, examples, and links to other resources from www.wiley.com/go/usabilitytesting