Usability Testing and QA

A practical approachtowards creating better user interfaces

Who am I?Shilpa Thanawala

@skthana

And you?Improving usability is about listening, shifting your

perspective, empathy.

Interview your neighbor, then introduce her/him.

Name

Role at your company or agency

Background (eg. designer? developer? etc.)

Experience with usability testing

How you think you’ll use what you learn today

Why this class?

Housekeeping

Initial and sign the roster

Evaluations at the end of the day

Cell phones on silent, calls outside

Breaks & lunch

Usability TestingSome thoughts to start us off...

Designers know too much about their product to be objective judges;the features they have come to love and prefer may not be understood

or preferred by future customers.Don Norman, nngroup.com

Image credit: templatemonster.com

The test of the machine is the satisfaction it gives you. There isn't anyother test. If the machine produces tranquility it's right. If it disturbs

you it's wrong until either the machine or your mind is changed.Robert M. Pirsig, Zen and the Art of Motorcycle Maintenance: An Inquiry Into Values

Image credit: Optimum7.com

Supposing is good, but finding out is better.Mark Twain

Image credit: Mediamatic

Usability TestingSome historical highlights...

Early 20th CenturyImprovements in industrial efficiency

In WW I, reduced work motions for soldiers to assemble &take apart weapons in the dark.

http://www.measuringusability.com/blog/usability-history.php

1947

John Karlin - how short phone cords should be

“It is not so much that Mr. Karlin trainedmidcentury Americans how to use the telephone.

It is, rather, that by studying the psychologicalcapabilities and limitations of ordinary people, he

trained the telephone...”Excerpt from New York Times article

1980’s

New methodologies developed and published

Usability becomes a profession

Defined as a function of efficiency, effectiveness, andsatisfaction

first publishedPsychology of Everyday Things

1990’s

Usability matures

Diminishing returns from testing more than 3-5 participants

Books by Jakob Nielsen

2000 - todaySteve Krug’s Don’t Make Me Think

Remote testing

Statistical analysis of usability data

Mobile devices, touch and gestures

http://www.measuringusability.com/blog/usability-history.php

Why?

Why spend valuable time and money doingusability tests?

It's the best way to learn how users actually interact withyour website. Watching > interviewing

You don't really know your users' needs. Users are not all thesame. And they're not your Mom.

All websites have problems, and the serious ones are easy tofind.

Why spend valuable time and money doingusability tests?

You'll learn things you didn't know that you didn't know.

Watching users gets everyone on the same page.

Watching users will make you a better designer / developer /product manager / content writer / etc.

Usability testing doesn't have to be expensive or time-consuming.

What Testing Can't Tell You

Limitations of Usability Testing

Won't fix your site's problems (it's not a user training session)

Results are dependent on the questions you ask (tasks)

Can't model social interactions very well

Isn't always representative of real situations

For more on issues testing the social web see Dana Chisnell's talk

When?

When should usability testing bedone?

Before you design or build anything

Before you implement a change

Throughout the project

After implementing a fix

Throughout the life of the website

As soon as possible & repeat often!

Demo: A short usability test

Qualitative vs. Quantitative

Qualitative vs. Quantitative

Quantitative Usability StudiesDesigned and conducted to produce data for statistical analysis

Each measurement represents a count or an amount

Distances, weights, quantities, etc.

Qualitative Usability StudiesFocused on verbal descriptions of users' experiences

Each measurement is a description or category

Words, sentences, feelings, yes/no

Categories are not quantitative, even if they are assignednumbers

userfocus.co.uk

“...qualitative methods are much better suited for answeringquestions about why or how to fix a problem, whereas quantitative

methods do a much better job answering ‘how many’ and ‘how much’types of questions.”

More on Quantitative methods at

"When to Use Which User Experience Research Methods", nngroup.com

measuringusability.com

Usability Metrics

Effectiveness, Efficiency, andSatisfaction

EffectivenessAccuracy, correctness. Are users able to achieve their goals?

EfficiencyEase of use, speed. How much effort (time) does it take to

complete a task?

SatisfactionUsers' perception. What does the user think about their

experience?

More at , usability.gov wikipedia.org/wiki/Usability

Some Metrics for Common User Tasks

From Measuring the User Experience – Tullis & Albert

Performance-based Metrics

Task SuccessCommonly used, pass / fail, or degrees of success, shouldhave clear end-state.

Task TimeParticularly important for repetitive tasks

ErrorsIncorrect actions leading to significant time loss, additionalcosts, or task failure

EfficiencyEffort and time needed to complete the task

LearnabilityEffort and time needed for the user to learn how to use theinterface

Issue Metrics

Usability findings based on users' behavior

Consistent / repeatable, or idiosyncratic?

Remain open-minded to the unexpected

Other Metrics

Self-Reporting: Users' perceptions, feelings

Behavioral / Psychological: Eye-tracking, stress,

unprompted verbal expressions

Combinations / Comparisons: Interpretations based on

more than one metric

Card-Sorting: How to organize information in a way that

makes sense to users.

A/B Testing: Comparison of two alternate designs

Accessibility: Usability for those with disabilities (

, )

CCVA of

2010 WCAG

Bias

Every study contains some level of bias.

Participant background, knowledge, comfort level

Task definitions

Methodology (testing process, session length, how muchtalking)

Artifacts of the prototype or product

Environment (lighting, noise, cameras, distractions)

The Facilitator (experience, skill, style)

ExpectationsMeasuring the User Experience – Tullis & Albert

Number of Test Participants

What's the right number of users to recruit for a study?

3-5? 4-6? 10? 20? 100?Those who favor a small number feel 80% of the mostimportant usability problems are detected by the first fewusers.

Cases in which more users may be required:You need to be sure you've captured as many problems aspossible

You have more than one user group (persona)

You have a large number of screens, complex design, widevariety of tasks

You'd like to account for evaluator bias

Why we'll focus on qualitative testing today

Anyone can do itNo knowledge of statistics or data analysis needed; based onverbal feedback, observation, and empathy

Cheap, fast, and easyGreat for small budgets, busy teams, minimal resources...and frequent repeats

It gets the job doneMore than sufficient to identify a website's most serioususability problems

Break

DIY Usability TestsBased largely on Rocket Surgery Made Easy – Steve Krug

Establish your GoalsWhat do you want to learn about your site?

Can users find products easily?

Is it clear to users what the site is about?

Is the information organized intuitively?

Is this new feature distracting?

Keep an open mind to allow for unexpected results.

Define your TasksTasks → Activities

Determine which activities best target what you want to learn.Create one or more scenarios based on

essential, real-world user goals.Then, describe the activities in the context of your scenario.

Activities should be realistic

Activities should be actionable

Phrase them so as not to give away the answer

Print each activity on its own piece of paper. (Avoid labels or numbers.)

"Turn User Goals into Task Scenarios for Usability Testing" — nngroup.com

Recruiting Test ParticipantsWhat kinds of participants?

Where to find them?

How many?

How to compensate them?

What kinds of participants?How representative of actual users?

Most serious UI problems will be discovered by non-

representative users.

Testing with non-representative users is better than

obsessing over finding representative users.

Key requirements: not part of the team, fluent in the UI's

language

Where to find participants?Where do your users hang out?

Advertise on the site, message boards, social networks

Craigslist

How many users?

For most of us doing DIY testing, 3-5 participants is fine.

No-shows or last-minute cancellations are common.

Have 1-2 backup participants lined up.

Phone-screen, scheduling, day-before reminders, directions,

parking, NDA's... use a checklist

CompensationShows appreciation for participant's time and effort

Encourages participants to be engaged and enthusiastic.

Gift cards, free product or service.

Equipment & SetupHint: you don't need a lab

Bare minimum (if you're the only one): a computer in a quiet

room, a pen and notepad, and 2 chairs

For a team, add a remote observation room, a good mic, and

screen-sharing (Skype or GHangout). And provide

irresistable snacks.

Can use screen-capture software (but most of the time it

won't be watched)

Can record the participant (but it's not really needed)

Make it a Team EffortTry to get everyone to attend.

Assign a trustworthy team member to manage theobservation room

The Session ScriptStart with a template & edit to suit your needs

Welcome / introduction

Logistics, release forms

Necessary background information

Test: general questions

Test: scenario activities

Wrapup, follow-up questions

Conclude and provide honorarium

Some Test Session Script Templates

Free templates at: , , infodesign.com Steve Krug's site Book siteguide by Rubin, Chisnell & Spool

DebriefOnly for those who attended at least one session

Schedule it on the same day (next day latest), 60-90 min.

Provide food!

Get all observers to state (or stick) UX problems they

observed

Get general agreement on which are most serious & will be

tackled first (2-3 maximum)

Remind everyone when the next round of testing will take

place

Run your own usability test1. Select a website to test

2. Determine goals: What's the question you're trying to answerin this round of testing?

3. Create a scenario & 3 activities that will give you informationpertaining to your goal

Lunch1 hour

Run your own usability test1. Prepare your script (links to resources )

2. Rehearse and refine

3. Print scenario(s) and activities on 1 sheet (well-spaced so youcan cut them apart)

j.mp/ux-dec-11

Run your own usability test1. Test two participants (~20 min each)

2. Take notes

Run your own usability test1. Debrief within your original group

2. Identify 2-3 usability issues

ToolsTesting session checklists

, ,

Usability testing software , ,

Online / Remote services , , ,

Usability.gov Krug's checklist checklist on book site byRubin, Chisnell & Spool

Silverback (Mac) Morae (Win) Camtasia

Usertesting.com Loop11 usabilityhub.com many more

So how do we fix it?

Fixing usability issuesThe bad news: most issues never get fixed

Make the smallest change possible to "stop the bleeding"

Large changes might introduce new issues

It's not fixed until you verify by ... testing again!

Demo: Prototyping a UI change usingPhotoshop + InVision

One method (of many) to prototype a change:

Create some static mockups using screenshots and visual

editing software

Model interactions using the InVision app

A/B Testing

A/B TestingA simple experiment testing two cases.

Before vs. after, one fix vs. another

Qualitative or quantitative

A/B Testing PossibilitiesLayout and design

Relative sizes of UI elements

UI controls

Form elements (how many, field labels, helper-text)

Images

Copy length or content

Calls to action

...many more

A/B Testing ActivityPrototyping your proposed fixes

A Usability Test using PaperPrototypes

Paper prototype of a kids’ website

Credit: BlueDuckLabs / YouTube.com

View online

A/B Testing Activity1. Select one usability issue from your previous activity

2. Brainstorm at least two ways to fix it

3. Print the page and use either Balsamiq or paper prototypingto mock up your two solutions.

4. Test each case on a separate participant in the other group

5. Debrief in your group. Which worked better, A or B?More detail at: SM's Ultimate Guide to A/B Testing

Testing Accessibility

Why is accessibility important?

“The Web is an increasingly important resource in many aspects oflife: education, employment, government, commerce, health care,

recreation, and more. It is essential that the Web be accessible in orderto provide equal access and equal opportunity

to people with disabilities. ”From The W3C's Web Accessibility Initiative

Is my site accessible?Unplug your mouse / turn off your trackpad

Why - blind, low-vision, compromised motor-control, no

hands

How to test - start at url, use tab / shift-tab and arrows to see

whether you can access all controls, links, and fields on the

page, in the right order. Popups and dialogs should be

usable.

How to fix - proper HTML heirarchy, tabindex attribute,

javascript plugins

Is my site accessible?Check color contrast

Why - low-vision; 1 out of 12 have color deficiency

How to test - online tools ( , ),browser plugins ( )

How to fix - shift color palette to improve contrast, increasefont sizes for low contrast areas, avoid colorful backgroundimages

Check my Colors Contrast RatioGrayscale Tool

Is my site accessible?Turn off images

Why - images useless to low-vision or blind users, lowbandwidth

How to test - block images in browser, WebAIM tool( )

How to fix - proper alt attributes, don't use images when textcan be used

WAVE

Accessibility Resources and ToolsWebAIM

W3C's Web Accessibility Initiative

Quick accessibility tests & fixes

SitePoint's Easy accessibility checks

Mobile / Touch

Mobile Usability

Context / environment (noise, lighting, glare)Many more contexts beyond desktop

BandwidthMobile devices force us to think about bandwidth again

Touch target sizeAverage fingertip size = 10mm Adjustments in spacing, button size Touch is also being used on large screens

Driving / attention What usability issues contribute to driver distraction?

Mobile Usability Testing Resources

NNGroup recommendations

Article on UXMagazine

Break10 minutes

Accessibility Testing Activity1. Click the URL at the top of the browser window

2. Without using the mouse again, try using only the keyboardto navigate

3. Can you access all the links, menus, and controls?

4. Can you determine where you are, or is it easy to lose yourplace?

Wrap-upUsability testing is an essential tool for improving yourwebsite or product

Anyone can do a simple usability study

Test as early as possible, as frequently as possible

Tailor the type of testing you conduct (qualitative /quantitative, metrics, number of participants) to give you theinformation you're after

Use the available tools and resources to help you run yoursessions

Wrap-upGet your whole team involved

Small adjustments are preferable (and faster) than a fullredesign

Remember to test your fixes, too

Stay aware of non-standard users (accessibility) and thechanging technological landscape (new platforms, new uses)

ResourcesWebsites / Blogs: , ,

,

Quantitative Testing: , ,

nngroup.com usability.gov sensible.com(Krug) UIE

measuringusability.com Measuring theUser Experience – Tullis & Albert usability.gov

Thank you!

Shilpa Thanawala | @skthana