What qualitative researchers should know about usability testing

QRCA - New York City Chapter

P r e s e n t e r : K a y C o r r y A u b r e y J u l y 2 0 , 2 012

Background

3 3 3

Agenda – what you will learn

1. This presentation will be an “appetizer” on usability testing (1 hr 15 minutes)

• Overview of how to plan, run, analyze & report on a usability study*

• Pointers to where you can learn more 2. Hands-on demonstration

• We’ll usability test the Android, iPhone, & iPad versions of the new QRCA VIEWS app (45 minutes)

3

You will learn the process of usability testing, its language, and when to use it in a qualitative research project

4 4 4

What is a usability test?

• Qualitative study where typical users try to accomplish typical tasks on their own with the product

• Point is to see how clearly the product “speaks” to them, meets their expectations, fits into their typical work and task flow

• Moderator & team watch participant working and keep score of task success & failure, comments, body language

5 5 5

What can be usability tested? • Web sites, software applications

• Consumer products (e.g., vacuum cleaners, ovens, mobile phones – examples from projects I’ve done to illustrate the range of where you can apply this technique)

• Packaging

• Customer service or ordering procedures

• Training & documentation

• Basically you can usability test any product or service where there is user interaction

6 6 6

Some Good reasons for running a usability study

• Make interaction with a product as fluid & intuitive as possible • Avoid embarrassment – expose usability problems • Test design concepts • Compare design approaches • Challenge assumptions • To compare your product with a competitor’s • Improve ease of use and learning • To better understand users • To understand training and documentation needs • To increase sales, improve your product’s reputation, decrease need for

technical support • To save money and time (less need to rework the design, fewer calls to

customer support)

7 7 7

Where does usability testing fit in with other qualitative research methods?

• Similar to in-depth interviews, focus groups, and ethnographies. Listen to participant, note their body language, learn their work and task flow, needs, attitudes, and perceptions

• Different from other forms of qualitative research because goal is to watch participant perform tasks against a product design and keep track of success and failure

8 8 8

Where does usability testing fit in with other qualitative research methods?

Shown with permission

Stages of a usability test

10 10 10

Major phases of a usability study

1. Planning the study 2. Running the study 3. Analyzing results 4. Reporting results

A typical soup-to-nuts usability study takes about 70 hours

Planning a usability study

12 12 12

Planning the study

• Determine study objectives • Identify target user characteristics, write recruiting

screener, oversee participant recruiting & scheduling • Create usability study plan and task list* • Make sure prototypes & other stimuli are ready • Manage logistics

*Bolded items are unique to usability testing

13 13 13

Usability objectives should focus on a specific aspect of the product design

Aspects of an interactive product design that affect usability

• Are the features, functions, and content appropriate?

• Does the organization and navigation align with the user’s task and work flow, their “mental model”?

• Do users understand labeling & terminology?

• Do they know where to start, what to do next, and the range of available choices?

14 14 14

Need an analysis framework for an interactive product

15 15 15

Sync testing approach to product’s development stage

Test method Project stage Focus Paper prototype Early Overall product concept,

terminology, navigation Electronic prototype Design Task flow, visual design, page

layout, specific features, validate redesign

Functioning product Development & QA Defaults, online help, feature integration, performance

Comparison Post-release Product features, performance benchmarks, can be within your own product or against competitor’s

16 16 16

Venue choices for a study Venue Pros Cons Facility Can invite lots of observers,

fewer logistical headaches, can test wider range of products

Artificial environment, costs more

Conference room Saves money - no travel, can test wider range of products

More logistics, artificial environment, observers want to sit in same room as tester

Live online More natural (participant is in their environment, saves money, no travel, fewer logistics, easy for team to observe

Can only test Web-based products, hear voice but don’t see body language, need to recruit more tech savvy participants, firewall issues

Native habitat (mobile studies) Person is in context of use - people use mobile phones when they are “on the go”, cheap & quick

Cannot easily record, only have 1-2 observers

17 17 17

Tips on recruiting for a usability study

• Usability studies typically involve only a few participants (8-10) • Choose confident and articulate participants who care about

the product category • Aim to recruit people with varying levels of experience • Define the required foundation skills • Make sure participants understand what it means to be in

usability study

18 18 18

Task list development EXAMPLE TASK LIST “You are the owner of a family-owned business that has just deployed the Acme Web Conferencing product. Today you are working from home but need to hold several meetings with your managers and employees. For the first time, you will be using your iPad to participate.” • Task 1 – Your Administrative Assistant, Kathleen, has set up an Acme Web

conferencing meeting for you. Go to your e-mail and find the invitation. • Task 2 - Join the conference. • Task 3 – Enter your information: Chris Doe, DoeChris173@myemail.com,

987-123-9876. • Task 4 - Enter the meeting room. • Task 5 - Once you are in the meeting room, connect your phone to the Web

conference so you can hear what is going on.

19 19 19

Planning a study is very involved

Create project plans and check lists to keep your ducks in a row.

20 20 20

Forget a step and you are dead

Just kidding

Moderating a usability study

22 22 22

Running the study Sessions follow a structure that is similar to any qualitative research session: • Moderate sessions – steps are the same • Keep track of participants’ task success and failure • Note comments, facial expressions, and body language • Hold debrief discussions with participants and observers • Administer surveys between tasks and after session has ended

During a usability study you are watching people’s behavior while listening closely to what they say

23 23 23

What type of data do you collect? • Collect objective results (can they do it? Define up front what

“success” means for each task)

• Emotional reactions

• Practical information (how does this product design fit into their world? What kind of training would a person need to be productive with this technology?)

• Typical measures (task success/failure, time on task)

• When they struggle, note why

• You see patterns after 3-4 sessions but new stuff always emerges (depending on diversity of participant backgrounds)

Analyzing results

25 25 25

Analyzing the study results

• Keep score of task success/failure by participant and task

• Identify trends – What was easy? What was hard? How did the product design play into these results?

• Find verbatim comments and video clips that encapsulate key findings

• Debrief continuously with clients – gaining consensus on issues is critical to your success

26 26 26

Use affinity diagramming to help team reach consensus on issues

Reporting results

28 28 28

Writing the report

• Use a conventional approach to structuring the report (e.g., executive summary, methodology, findings)

• Group findings by product features and functions • Where possible, tell the story with numbers (e.g.,

number of participants who succeeded at each task) • Layer the information – place summarized information

in the report, and details in a spreadsheet

29 29 29

Example of a usability test report

30 30 30

Look for opportunities to collect quantitative data

Product Reaction Cards Participants quickly select 5

attributed from among 118 choices The attributes are balanced between

positive and negative

System Usability Scale Participants answer 10 questions on

key aspects of usability Survey produces a score between 0

and 100; a score <60 is considered poor

31 31 31

Seek results that can be expressed in pictures

Source: Moxie Software http://www.moxiesoft.com/tal_products/employee-spaces-does-design-matter.aspx

32 32 32

Product Reaction Card results add human interest

Text size indicates number of times that attribute was chosen

33 33 33

System usability Scale (SUS) scores cut to the chase

Product ABC Product XYZ

0 0 0 0 0 0 0

10

0-10 11-20 21-30 31-40 41-50 51-60 61-70 > 71

1

0

1

0

1 1

3 3

0-10 11-20 21-30 31-40 41-50 51-60 61-70 > 71

SUS scores < 60 indicate poor usability. See the study spreadsheet for details.

XYZ’s’ SUS scores from all 10 participants were between 75 and 100

34 34 34

Reporting – each discipline expects specific insights from the study results

Executives want a distilled version, an encapsulation of “the problems” and to understand what is perceived as the premium this product offers

Product managers and marketing people seek insights on segmentation, product identity, competitive information, participant reaction to feature sets

Product designers want detailed usability feedback to guide refinements to the product’s interface and behavior. Is the design intuitive? How does it fit into the user’s work and task flow?

Engineering needs to understand the usability bugs so they can prioritize them and fix them (often their input is the bottom line)

Training and documentation people want to know which content to include in their work

QRCA VIEWS for Android and iPhone

Hands-on exercise Usability testing a

mobile app

36 36 36

Hands-on exercise – usability test QRCA VIEWS mobile app

STEPS 1. Translate the objectives into a task list 2. Break into groups of 2-3 people, one person is the administrator

another the participant, others are observers who will take notes 3. Run the study 4. Observers will record impressions on colored sticky notes (each

group will have its own color) 5. We’ll reconvene and do a mini affinity diagramming session to

tabulate and discuss the results and recommendations for improvement. We’ll hold a debrief.

6. Kay present our results to Laurie and Eddie 7. Then we’ll pat ourselves on the back!!!

37 37 37

Additional resources

• Review a bibliography on user experience plus VIEWS and Quirks articles on how to run a usability study

• Watch Camille Carlin and my QRCA QCast • Sign up for my newsletter “Usability Tidbits”

The End Thank You!!

39 39 39

About your presenter Kay Corry Aubrey - UX researcher and designer Kay Corry Aubrey is the owner of Usability Resources, which specializes in user-centered research and design. Kay has over 20 years of experience in applying qualitative research methods and usability testing to technology-oriented products and collaborative software. She has led user research, usability, and design efforts for dozens clients including AT&T, the Broad Institute, Affinnova, Constant Contact, Monster Worldwide, the Massachusetts Medical Society, the Mayo Clinic, and iRobot. Kay teaches usability and design for Northeastern University. She is a Managing Editor of the QRCA VIEWS magazine, a market research journal. Kay has an MSW from Boston University’s School of Social Work, an MS in information systems from Northeastern University’s Graduate School of Engineering, and a BA from McGill University. She is a RIVA-certified Master Moderator who enjoys doing research with both groups and individuals. Contact information e-mail: Kay@UsabilityResources.net phone: 781-275-3020 www.UsabilityResources.net