usability testing & think aloud
DESCRIPTION
Usability Testing & Think Aloud. 2 /20/14 HCC 729. In-class Homework Critiques. What happened after the UARs? Pair up and share your HEs with another group (5 minutes each) Identify good examples, questions . Sharing example HEs. Good examples Questions / unsure?. UAR Awards. - PowerPoint PPT PresentationTRANSCRIPT
Usability Testing & Think Aloud
2/20/14HCC 729
In-class Homework Critiques
What happened after the UARs?
• Pair up and share your HEs with another group (5 minutes each)
• Identify good examples, questions
Sharing example HEs
• Good examples• Questions / unsure?
UAR Awards
• Best written UAR• Funniest usability problem• Interesting edge case
Discussion
Inspirations or UI news?Readings
For next time: I’ll pick selected examples (to save time)
Recap from last week
• HTAs & personas feedback coming soon (Saturday)
• Heuristic evaluation process
Phases of Heuristic Evaluation
0) Pre-evaluation training (optional)Give evaluators needed domain knowledge & information on the scenario
1) Evaluate the interface to find usability problems 2) Record the problem 3) Aggregate problems 4) Assign severity rating5) Find a solution complexity rating
Find which heuristic is violated1. Simple & Natural Dialog2. Speak User’s Language3. Minimize User’s Memory Load4. Consistency 5. Feedback 6. Clearly Marked Exits7. Shortcuts 8. Good Error Messages 9. Prevent Errors 10. Help & Documentation
Nielsen’s 10 Heuristics
http://www.nngroup.com/articles/ten-usability-heuristics/
Marie Silverstrim
Today: Working with Real Users
When should you user test?
Early and often
Categories of user testing
• Formative user testing• Usually happens early in the design process• Identify problems for the next round of design• Can be done with non-functional prototypes
• Summative user testing• Done later in the development process• Can be part of QA
Assumptions of user testing
1. Even an expert can’t find all the usability challenges
1. Expertise can blind us problems2. Domain experts, technical experts
2. Understanding the system image can help us to find bugs
3. People focus on aesthetics / other issues
Think Aloud Method
• Usually used as a summative evaluation technique:• Use it to validate/asses an interface
• Could be used as a Formative one:• Use Think Aloud to discover process
Goals for Today
• Things you will learn:• How to conduct a Think Aloud • When to conduct a Think Aloud • What data can & should be captured • Preparation: picking users & tasks
• Think Alouds are useful to discover how users think about doing a task• Interface evaluation, problem solving, learning, etc.• Follows detailed usability testing
How to conduct a Think Aloud Evaluation
What is the Think Aloud Method?
• “[Think-Aloud]...may be the single most valuable usability engineering method” [Nielsen, p. 195]
• An empirical technique where you ask users to “think aloud” as they work on a task you believe is interesting or important, using the interface you are interested in improving
Think Aloud Metrics
• Think Aloud is about observing• how users think about tasks• where users have problems with a system
• What kind of information do you get?• Qualitative Insights (from a few users)
• What kind of information do you NOT get? • Performance Measurement (time, accuracy)• This comes from a usability test
Think Aloud Process
1. Preparation: Design a test plan 2. Preparation: Choose what to observe 3. Preparation: Prepare for the user test 4. Run the user test 5. Aggregate, organize, and analyze your notes
Think Aloud Process
1. Preparation: Design a test plan 2. Preparation: Choose what to observe 3. Preparation: Prepare for the user test 4. Run the user test 5. Aggregate, organize, and analyze your notes
Preparation: Design a Test Plan
• What problem is your system trying to solve?
• Who are your target users?
• What scenarios are you evaluating?
• What are your usability goals?
• What is your hypothesis?
Think Aloud Process
1. Preparation: Design a test plan 2. Preparation: Choose what to observe 3. Preparation: Prepare for the user test 4. Run the user test 5. Aggregate, organize, and analyze your notes
Choose Tasks to Observe
• Choose what to observe• Don’t pick obvious or impossible tasks • Should be more than just “try out the
interface” • Pick tasks that are: representative, critical,
frequent • Often better to integrate small tasks as a
sequence
Example Task Sheet• Company ABC’s photo website• Task 1: A friend mentions ABC’s website. You Google it & reach its front
page. Without leaving this page, think aloud about your thoughts. What would you click next?
• Task 2: You’re preparing a talk about Paris & need some photos for illustrations. Use ABC’s website to find a photo from Paris. Remember to think aloud as you work.
• Task 3: You’ve found good photos from Paris, and you’d like to bookmark them for future use. How would you bookmark a photo that you are viewing now? Remember to think aloud as you work.
• Task 4: Your friends have asked for your Rome photos from last year. Publish one photo from Rome using ABC’s website. The photos are on your desktop in the “Rome” folder. Remember to think aloud as you work.
Notes about the Task Sheet
• Give each user multiple tasks to perform• Break complex tasks into small parts
• Not all tasks need to start in the beginning• Clearly structured tasks help the user focus on
evalution• Make the tasks independent from each other so you
can skip tasks and move around• i.e. make the tasks independent
• Always remind them to think aloud
Think Aloud Process
1. Preparation: Design a test plan 2. Preparation: Choose what to observe 3. Preparation: Prepare for the user test 4. Run the user test 5. Aggregate, organize, and analyze your notes
#3 Piloting your user test
• It is always good to “pilot” your user test before you recruit real users• Practice your think aloud • Learn the script • Test out the tasks, see how long it takes
Think Aloud Process
1. Preparation: Design a test plan 2. Preparation: Choose what to observe 3. Preparation: Prepare for the user test 4. Run the user test 5. Aggregate, organize, and analyze your notes
Run a Think Aloud: Warm-up
• Introduce yourself• Describe Think Aloud:
• May initially be awkward • It’s important that they talk • You may prompt them • Best to provide examples
• Promise confidentiality • Get permission to record
Run a Think Aloud: Tasks
• Introduce your system• Don’t rationalize or defend
system• Remind them this isn’t your
system• Describe the tasks
• Use your Task Sheet • User can read these aloud • Ask for questions
Run a Think Aloud: Tasks• Keep the user talking
• A few phrase you might use• “Please keep talking”• “Tell me what you are
thinking”• “Tell me what you’re trying
to do" • “Can you tell me more?”• “What is your opinion?”
Run a Think Aloud: Tasks• You are only an observer
• Don’t help do the task • Unless they are really struggling
and taking too long• Do respond to & note questions
• Take good notes• Record all problems, not just
what you were looking for• Helpful: watch, A/V recordings• Must: take notes
Notes about your notes
• Keep track of notes for each task (for each user)
• Keep track of the actions and the behaviors you saw for each action
• Include important quotes• Find the fastest note taking medium that isn’t
distracting
User Test Notes
• Example: Company ABC’s photo website• User 1: Task 2 (search for photos)
• Scrolled through several pages of photos • Found Paris photos but didn’t use search button
• User 1: Task 3 (bookmark photos)• Successfully clicked on bookmark button with no delay• Used copy and paste to enter bookmark data• Confused by the fact that fields aren’t necessarily in the same order as
they would normally be entered• User 2: Task 2 (search for photos)
• “so...it’s not one of the categories...so I’ll search for it”• “I want to search for the phrase Paris, but don’t see a way to do it other
than paging” (finds photos by clicking Next button)
Run a Think Aloud: Tasks• When it’s over, it’s helpful to ask
overall impressions• What were the best three things
about it?• What were the worst three
things about it?• Thank the user • Provide any compensation
Think Aloud Process
1. Preparation: Design a test plan 2. Preparation: Choose what to observe 3. Preparation: Prepare for the user test 4. Run the user test 5. Aggregate, organize, and analyze your notes
Organize your notes
• Aggregate, organize, and analyze your notes• This is not just about transcribing what you
recorded. • You need to analyze both what they say & do.
• Establish criteria for “critical incidents”• Record these critical incidents on UAR forms
Critical Incidents
• First introduced by Flanagan, 1954• “To be critical, an incident must occur in a
situation where the purpose or intent of the act seems fairly clear to the observer and where its consequences are sufficiently definite to leave little doubt concerning its effects.”
• In other words: describes a discrete event • Observable behavior leads to specific consequence
• Can be good or bad!
Bad Critical Incidents• Possible criteria:• User articulates goal but doesn’t attain goal within [3
minutes]• User articulates goal, tries several things (or same thing)
before giving up• User articulates goal but tries multiple ways before
finding a solution• User accomplishes task but in a suboptimal way• User expresses surprise or hesitation• User expresses some negative sentiment, either about
interface or about their skills
Good Critical Incidents
• Possible criteria:• User expressed some positive affect or says
something is really easy• User expresses happy surprise• Some previous analysis has predicted a
usability problem, but user has no difficulty with that aspect of the system
Use the data
• Make sure your reflections are supported directly by the data
• Don’t include your own hunches (there are other opportunities to do this)
Record your evidence using UARs!
Critical Incident UARs
1. Summarize user test notes2. List critical incidents: both good & bad3. Complete UARs: refer back to raw data4. Try to judge why each difficulty occurred
User Test Report
• What did Think Aloud tell you? • Does the UI work the way you thought it would? • Do the users take the approaches you expected?• Is something missing from your UI?
• Rethink your design • Based on UAR rankings, what fixes will you work
on? • Redesign is often more than just fixing bugs
Live demo
• Need two volunteers
Pro Tips
Notes• Search “snow boots”!• Jump to bottom• What do widths mean?• Repeat size• Colors• Heel styles – what were they?• Walking through hierarchy• Didn’t see “womens”• Hard to see pictures• Positive – said what was missing• Backtracking• Out of stock
Notes on think aloud
• Letting her know when to stop• Hard not to call out• Not clear when to end• Hard to keep track of both
Pro tips from our demo?
Think Aloud Introspection
• Think Alouds can disrupt a user’s thought process • When people are trying to figure something
out • When a problem requires “insight”
• If you notice your participant looking confused, now might not be exactly the time to push them to “Keep Talking”
Think Alouds: Avoid “Why”• “Why” questions also lead to introspection
• “Why did you do that?”• “What does that mean?”
• During Think Aloud, best not to ask “why”• Introspection can change your process• If you ask a question, people will always give an
answer…• Remember: TA is about observing raw process, a
contextual inquiry is about understanding & interpreting context
For next week
AssignmentReadings
Reading
• None! Focus on your think aloud evaluation.
• Don’t wait until the last minute!
Assignment
• Perform a think aloud with 3 people (4 if in a group) who represents someone from your user analysis
• Complete CI UARs based on what you saw• Fill out the top part for all users first• Aggregate across all users• Complete the bottom half• Have them do 5 tasks (ideally from your TA)
• Everyone: Write 200 words about what you learned
In-class exercise
• Work in pairs (with someone who didn’t do a UAR for your site) and conduct a think aloud for the task you mapped out in your HTA.• Take notes on the back of the UAR sheet• Fill out a CI UAR together