remote testing methods & tools webinar

Webinar:

Remote Testing Methods & Tools

December 13th, 2011

Bill Albert, Director of Design & Usability Center at Bentley University presents a complimentary webinar on Remote Testing Methods and Tools.

With special guests:

2

Webinar Overview

1. Brief introductions & goal of the webinar

2. Updated overview of competitive landscape, updated figures, etc

3. The different methods / categories / tools. When to use what?

4. Types of research goals

1. Categories of tools available

2. Overview of some of those tools

5. Examples:

– Usertesting.com

– Optimal Workshop

– UserZoom

6. Takeaways, Q&A

3

• Speakers today:o Bill Albert, Director of Design & Usability Center, Bentley Universityo Dave Garr, Co-Founder, UserTesting.como Andrew Mayfield, CEO, Optimal Workshopo Matt Paulus, Director of US Sales, UserZoom

• Goals of today’s session

Introduction

walbert@bentley.edu 781.891.2500 | www.bentley.edu/usability

Bill AlbertDecember 13, 2011

Remote Testing Methods and Tools

5

• Introduction to remote usability testing

• Quantitative-based tools

• Qualitative-based tools

Agenda

6

• Director, Design and Usability Center, Bentley University• Experience with remote testing tools• I am data driven!

About me

7

I am a big fan of usability testing in the lab!

Introduction to remote usability testing

9

What do we mean by “remote testing tools”?• A remote testing tool is any technology that allows a researcher to collect

data from users about their experience in using an interface, without direct, face-to-face contact.

There are two ways of organizing remote testing tools• Moderated vs. Unmoderated• Qualitative vs. Quantitative-based

Defining Remote Testing Tools

Moderated Unmoderated

Qualitative-based Qualitative-based

Quantitative-based Quantitative-based

10

Unmoderated Testing Tools

Unmoderated tools

Qualitative-based

Video tools

Reporting

Expert Reviews

Quantitative-based

Full-service

Self-service

Card Sorting/IA

Surveys

Click/Mouse

11

Our UX Toolkit

12

• Saves timeo Lab study takes 2-4 weeks from start to finish, unmoderated typically takes

hours to a few days*

• Saves moneyo Participants compensation typically a lot less ($10 vs. $100)o Tools are becoming very inexpensive

• Reliable metricso Only (reasonable) way to collect UX data from large sample sizes

• Geography is not a limitationo Collect feedback from customers all over the world

• Greater Customer insighto Richest dataset about the customer experience

Why Should You Care?

13

• Questions from senior management are becoming more complex and time sensitive

• Traditional usability is no longer enough

• Push to measure the UX

• Convergence with market research and web analytics to paint a more complete picture of the UX

• Budgets/timeline constraints

Why is this important NOW?

14

Which one goes first?

Lab first, then Unmoderated Unmoderated first, then Lab

Identify/fix “low hanging fruit”, then focus on remaining tasks with large sample size

Identify the most significant issues online through metrics, then use lab study to gather deeper qualitative understanding of those issues

Generate new concepts, ideas, questions through lab testing, then test/validate online

Collect video clips or more quotes of users to help bring metrics to life

Validate attitudes/preferences observed in lab testing

Gather all the metrics to validate design – if it tests well, then no need to bring users into the lab

Quantitative-based tools

16

Unmoderated Testing Tools - Quantitative

Unmoderated tools

Qualitative-based

Video tools

Reporting

Expert Reviews

Quantitative-based

Full-service

Self-service

Card Sorting/IA

Surveys

Click/Mouse

17

Common Research Questions:• What are the usability issues, and how big?• Which design is better, and by how much?• How do customer segments differ?• What are user design preferences? • Is the new design better than the old design?• Where are users most likely to abandon a transaction?

Types of Studies:• Comprehensive evaluation• UX benchmark• Competitive evaluation• Live site vs. prototype comparison• Feature/function test• Discovery

Overview

Typical Metrics:• Task success• Task time• Self-report ratings such as ease of

use, confidence, satisfaction• SUS• Click paths• Abandonment

18

Strengths:• Comparing products• Measuring user experience• Finding the right participants• Focusing on design improvements• Insight into the user’s real

experience

Limitations:• Not well suited to rapid, iterative

design• Need a deep understanding of

issues• Studies that require long sessions• Lose control over prototypes• Internet access

Strengths / Limitations / Myths

Myths:• Only test with websites• Very expensive• Only gather quantitative data• A lot of noise in the data• Same as any market research study• Comparing products

19

Full Service Tools – Online Usability Testing

20

Self Service Tools – Online Usability Testing

21

Card Sorting / IA Tools

22

Online Surveys

23

Click and Mouse Tools

Qualitative-based tools

25

Unmoderated Testing Tools - Qualitative

Unmoderated tools

Qualitative-based

Video tools

Reporting

Expert Reviews

Quantitative-based

Full-service

Self-service

Card Sorting/IA

Surveys

Click/Mouse

26

OverviewCommon Research Questions:• What are the big pain usability pain points, and

wins?• How do users think, and feel about the design?• Are we on the right (design) track? • What is the overall experience like? • Why might users abandon a transaction?

Types of Studies:• Low fidelity prototype• High fidelity prototype• Competitive evaluation• Comprehensive evaluation• Feature/function test• Discovery

Common Metrics:• Frequency of issues• Verbatim comments• Task success (via video)• Task time (via video)

27

Video Tools

28

Report Tools

29

Expert Review

30

Online:www.measuringUX.comwww.measuringusability.comhttp://usability.bentley.eduhttp://www.usefulusability.com/24-usability-testing-tools/http://remoteresear.ch/tools/

Follow me:@UXMetrics@BentleyDUC

LinkedIn Group:Bentley Design and Usability Center

Additional Resources

Dave GarrCo-Founder

UserTesting.com

In Cyberspace, no one can hear your visitor scream.

What’s your visitor thinking?

33

Solution?

How It Works

“I don’t know why it keeps doing this.This is definitely very confusing and frustrating.”

Write an Annotation

“I can’t figure out which cartridge goes with my printer”

How to share clips

View Tester’s Answers

Follow up with users

Export to Excel

42

Mobile Testing

• Test mobile apps

• Test mobile-friendly sites

43

What do you want to test?

“I’d like to look at these three star reviews, … it doesn’t let me do that… I’d have to go through all of the

reviews.”

44

Presenting Your Findings

Andrew MayfieldCEO

Optimal Workshop

Quantitative analysis to assess the

effectiveness of an Information

Architecture is

hard work.

Peter Morville’sUX Honeycomb

Optimal Workshop

The Suite

by Optimal Workshop

Qualitativevs

Quantitative

www.optimalworkshop.com/blogSee latest blog post on this topic

Par

ticip

ant

Cen

tric

Ana

lysi

s

Safe orexistingoptions

Radicaloption

?

BENCHMARK

Remote studies enable near unlimitediteration and participation

Matt PaulusDirector of US Sales

UserZoom

UserZoom’s Software Solutions

• On-demand, web-based SaaS

• 100% self-service

• All-in-one quantitative unmoderated online research

• Enterprise level software

• Feature rich & intuitive

• Quantitative remote testing

• Natural context

• International/domestic testing

• No moderation needed

• Behavior (analytics) + Feedback (online Surveys)

Unmoderated Remote Testing Methodology

UserZoom Products

UserZoom Products

60

Live Websites Wireframes & Prototypes

Multimedia content

Mobile Interfaces & Apps

Compatible Platforms

UserZoom Task-Based Usability Testing

UserZoom Results – Open Ended/Text Clouds

UserZoom Results – Net Promotor Score

UserZoom Results – Filters/Segmentation

UserZoom Results – Effectiveness & Efficiency

UserZoom Results – Clickstreams

UserZoom Results- Click Heatmaps

68

• Remote testing tools are becoming more advanced every day, and are a critical part of the UX toolkit

• Find the right type of tool based on your research question

• Demo a select number of remote testing tools to find the one that will deliver the results you need, within budget and timeframe

Key Takeaways

69

Quick Poll!

Thank You!

Floor open to Q&A

Bill AlbertDirector of Design & Usability CenterBentley University

Dave GarrCo-Founder

UserTesting.com

Andrew MayfieldCEO

Optimal Workshop

Matt PaulusDirector of US

SalesUserZoom