remote testing methods & tools webinar

70
Webinar: Remote Testing Methods & Tools December 13th, 2011 Bill Albert, Director of Design & Usability Center at Bentley University presents a complimentary webinar on Remote Testing Methods and Tools. With special guests:

Upload: userzoom

Post on 27-Jan-2015

114 views

Category:

Education


0 download

DESCRIPTION

On Tuesday, December 13th, Bill Albert, co-author of Beyond the Usability Lab, and one of the most knowledgable experts in Remote Usability Testing, presented an update on Remote User Testing methods and tools to optimize your UX Roadmap. If you were not able to attend the live webinar please feel free to view the slideshare below! The webinar was co-hosted with 3 special guests: Dave Garr, Co-Founder, UserTesting.com Andrew Mayfield, CEO, Optimal Workshop Matt Paulus, Director of US Sales, UserZoom

TRANSCRIPT

Page 1: Remote Testing Methods & Tools Webinar

Webinar:

Remote Testing Methods & Tools

December 13th, 2011

Bill Albert, Director of Design & Usability Center at Bentley University presents a complimentary webinar on Remote Testing Methods and Tools.

With special guests:

Page 2: Remote Testing Methods & Tools Webinar

2

Webinar Overview

1. Brief introductions & goal of the webinar

2. Updated overview of competitive landscape, updated figures, etc

3. The different methods / categories / tools. When to use what?

4. Types of research goals

1. Categories of tools available

2. Overview of some of those tools

5. Examples:

– Usertesting.com

– Optimal Workshop

– UserZoom

6. Takeaways, Q&A

Page 3: Remote Testing Methods & Tools Webinar

3

• Speakers today:o Bill Albert, Director of Design & Usability Center, Bentley Universityo Dave Garr, Co-Founder, UserTesting.como Andrew Mayfield, CEO, Optimal Workshopo Matt Paulus, Director of US Sales, UserZoom

• Goals of today’s session

Introduction

Page 4: Remote Testing Methods & Tools Webinar

[email protected] 781.891.2500 | www.bentley.edu/usability

Bill AlbertDecember 13, 2011

Remote Testing Methods and Tools

Page 5: Remote Testing Methods & Tools Webinar

5

• Introduction to remote usability testing

• Quantitative-based tools

• Qualitative-based tools

Agenda

Page 6: Remote Testing Methods & Tools Webinar

6

• Director, Design and Usability Center, Bentley University• Experience with remote testing tools• I am data driven!

About me

Page 7: Remote Testing Methods & Tools Webinar

7

I am a big fan of usability testing in the lab!

Page 8: Remote Testing Methods & Tools Webinar

Introduction to remote usability testing

Page 9: Remote Testing Methods & Tools Webinar

9

What do we mean by “remote testing tools”?• A remote testing tool is any technology that allows a researcher to collect

data from users about their experience in using an interface, without direct, face-to-face contact.

There are two ways of organizing remote testing tools• Moderated vs. Unmoderated• Qualitative vs. Quantitative-based

Defining Remote Testing Tools

Moderated Unmoderated

Qualitative-based Qualitative-based

Quantitative-based Quantitative-based

Page 10: Remote Testing Methods & Tools Webinar

10

Unmoderated Testing Tools

Unmoderated tools

Qualitative-based

Video tools

Reporting

Expert Reviews

Quantitative-based

Full-service

Self-service

Card Sorting/IA

Surveys

Click/Mouse

Page 11: Remote Testing Methods & Tools Webinar

11

Our UX Toolkit

Page 12: Remote Testing Methods & Tools Webinar

12

• Saves timeo Lab study takes 2-4 weeks from start to finish, unmoderated typically takes

hours to a few days*

• Saves moneyo Participants compensation typically a lot less ($10 vs. $100)o Tools are becoming very inexpensive

• Reliable metricso Only (reasonable) way to collect UX data from large sample sizes

• Geography is not a limitationo Collect feedback from customers all over the world

• Greater Customer insighto Richest dataset about the customer experience

Why Should You Care?

Page 13: Remote Testing Methods & Tools Webinar

13

• Questions from senior management are becoming more complex and time sensitive

• Traditional usability is no longer enough

• Push to measure the UX

• Convergence with market research and web analytics to paint a more complete picture of the UX

• Budgets/timeline constraints

Why is this important NOW?

Page 14: Remote Testing Methods & Tools Webinar

14

Which one goes first?

Lab first, then Unmoderated Unmoderated first, then Lab

Identify/fix “low hanging fruit”, then focus on remaining tasks with large sample size

Identify the most significant issues online through metrics, then use lab study to gather deeper qualitative understanding of those issues

Generate new concepts, ideas, questions through lab testing, then test/validate online

Collect video clips or more quotes of users to help bring metrics to life

Validate attitudes/preferences observed in lab testing

Gather all the metrics to validate design – if it tests well, then no need to bring users into the lab

Page 15: Remote Testing Methods & Tools Webinar

Quantitative-based tools

Page 16: Remote Testing Methods & Tools Webinar

16

Unmoderated Testing Tools - Quantitative

Unmoderated tools

Qualitative-based

Video tools

Reporting

Expert Reviews

Quantitative-based

Full-service

Self-service

Card Sorting/IA

Surveys

Click/Mouse

Page 17: Remote Testing Methods & Tools Webinar

17

Common Research Questions:• What are the usability issues, and how big?• Which design is better, and by how much?• How do customer segments differ?• What are user design preferences? • Is the new design better than the old design?• Where are users most likely to abandon a transaction?

Types of Studies:• Comprehensive evaluation• UX benchmark• Competitive evaluation• Live site vs. prototype comparison• Feature/function test• Discovery

Overview

Typical Metrics:• Task success• Task time• Self-report ratings such as ease of

use, confidence, satisfaction• SUS• Click paths• Abandonment

Page 18: Remote Testing Methods & Tools Webinar

18

Strengths:• Comparing products• Measuring user experience• Finding the right participants• Focusing on design improvements• Insight into the user’s real

experience

Limitations:• Not well suited to rapid, iterative

design• Need a deep understanding of

issues• Studies that require long sessions• Lose control over prototypes• Internet access

Strengths / Limitations / Myths

Myths:• Only test with websites• Very expensive• Only gather quantitative data• A lot of noise in the data• Same as any market research study• Comparing products

Page 19: Remote Testing Methods & Tools Webinar

19

Full Service Tools – Online Usability Testing

Page 20: Remote Testing Methods & Tools Webinar

20

Self Service Tools – Online Usability Testing

Page 21: Remote Testing Methods & Tools Webinar

21

Card Sorting / IA Tools

Page 22: Remote Testing Methods & Tools Webinar

22

Online Surveys

Page 23: Remote Testing Methods & Tools Webinar

23

Click and Mouse Tools

Page 24: Remote Testing Methods & Tools Webinar

Qualitative-based tools

Page 25: Remote Testing Methods & Tools Webinar

25

Unmoderated Testing Tools - Qualitative

Unmoderated tools

Qualitative-based

Video tools

Reporting

Expert Reviews

Quantitative-based

Full-service

Self-service

Card Sorting/IA

Surveys

Click/Mouse

Page 26: Remote Testing Methods & Tools Webinar

26

OverviewCommon Research Questions:• What are the big pain usability pain points, and

wins?• How do users think, and feel about the design?• Are we on the right (design) track? • What is the overall experience like? • Why might users abandon a transaction?

Types of Studies:• Low fidelity prototype• High fidelity prototype• Competitive evaluation• Comprehensive evaluation• Feature/function test• Discovery

Common Metrics:• Frequency of issues• Verbatim comments• Task success (via video)• Task time (via video)

Page 27: Remote Testing Methods & Tools Webinar

27

Video Tools

Page 28: Remote Testing Methods & Tools Webinar

28

Report Tools

Page 29: Remote Testing Methods & Tools Webinar

29

Expert Review

Page 30: Remote Testing Methods & Tools Webinar

30

Online:www.measuringUX.comwww.measuringusability.comhttp://usability.bentley.eduhttp://www.usefulusability.com/24-usability-testing-tools/http://remoteresear.ch/tools/

Follow me:@UXMetrics@BentleyDUC

LinkedIn Group:Bentley Design and Usability Center

Additional Resources

Page 31: Remote Testing Methods & Tools Webinar

Dave GarrCo-Founder

UserTesting.com

Page 32: Remote Testing Methods & Tools Webinar

In Cyberspace, no one can hear your visitor scream.

What’s your visitor thinking?

Page 33: Remote Testing Methods & Tools Webinar

33

Solution?

Page 34: Remote Testing Methods & Tools Webinar

How It Works

Page 35: Remote Testing Methods & Tools Webinar

“I don’t know why it keeps doing this.This is definitely very confusing and frustrating.”

Page 36: Remote Testing Methods & Tools Webinar

Write an Annotation

Page 37: Remote Testing Methods & Tools Webinar

“I can’t figure out which cartridge goes with my printer”

Page 38: Remote Testing Methods & Tools Webinar

How to share clips

Page 39: Remote Testing Methods & Tools Webinar

View Tester’s Answers

Page 40: Remote Testing Methods & Tools Webinar

Follow up with users

Page 41: Remote Testing Methods & Tools Webinar

Export to Excel

Page 42: Remote Testing Methods & Tools Webinar

42

Mobile Testing

• Test mobile apps

• Test mobile-friendly sites

Page 43: Remote Testing Methods & Tools Webinar

43

What do you want to test?

“I’d like to look at these three star reviews, … it doesn’t let me do that… I’d have to go through all of the

reviews.”

Page 44: Remote Testing Methods & Tools Webinar

44

Presenting Your Findings

Page 45: Remote Testing Methods & Tools Webinar

Andrew MayfieldCEO

Optimal Workshop

Page 46: Remote Testing Methods & Tools Webinar

Quantitative analysis to assess the

effectiveness of an Information

Architecture is

hard work.

Page 47: Remote Testing Methods & Tools Webinar

Peter Morville’sUX Honeycomb

Page 48: Remote Testing Methods & Tools Webinar

Optimal Workshop

Page 49: Remote Testing Methods & Tools Webinar

The Suite

by Optimal Workshop

Page 50: Remote Testing Methods & Tools Webinar

Qualitativevs

Quantitative

www.optimalworkshop.com/blogSee latest blog post on this topic

Page 51: Remote Testing Methods & Tools Webinar
Page 52: Remote Testing Methods & Tools Webinar
Page 53: Remote Testing Methods & Tools Webinar

Par

ticip

ant

Cen

tric

Ana

lysi

s

Page 54: Remote Testing Methods & Tools Webinar
Page 55: Remote Testing Methods & Tools Webinar

Safe orexistingoptions

Radicaloption

?

BENCHMARK

Remote studies enable near unlimitediteration and participation

Page 56: Remote Testing Methods & Tools Webinar

Matt PaulusDirector of US Sales

UserZoom

Page 57: Remote Testing Methods & Tools Webinar

UserZoom’s Software Solutions

• On-demand, web-based SaaS

• 100% self-service

• All-in-one quantitative unmoderated online research

• Enterprise level software

• Feature rich & intuitive

Page 58: Remote Testing Methods & Tools Webinar

• Quantitative remote testing

• Natural context

• International/domestic testing

• No moderation needed

• Behavior (analytics) + Feedback (online Surveys)

Unmoderated Remote Testing Methodology

Page 59: Remote Testing Methods & Tools Webinar

UserZoom Products

Page 60: Remote Testing Methods & Tools Webinar

UserZoom Products

60

Live Websites Wireframes & Prototypes

Multimedia content

Mobile Interfaces & Apps

Compatible Platforms

Page 61: Remote Testing Methods & Tools Webinar

UserZoom Task-Based Usability Testing

Page 62: Remote Testing Methods & Tools Webinar

UserZoom Results – Open Ended/Text Clouds

Page 63: Remote Testing Methods & Tools Webinar

UserZoom Results – Net Promotor Score

Page 64: Remote Testing Methods & Tools Webinar

UserZoom Results – Filters/Segmentation

Page 65: Remote Testing Methods & Tools Webinar

UserZoom Results – Effectiveness & Efficiency

Page 66: Remote Testing Methods & Tools Webinar

UserZoom Results – Clickstreams

Page 67: Remote Testing Methods & Tools Webinar

UserZoom Results- Click Heatmaps

Page 68: Remote Testing Methods & Tools Webinar

68

• Remote testing tools are becoming more advanced every day, and are a critical part of the UX toolkit

• Find the right type of tool based on your research question

• Demo a select number of remote testing tools to find the one that will deliver the results you need, within budget and timeframe

Key Takeaways

Page 69: Remote Testing Methods & Tools Webinar

69

Quick Poll!

Page 70: Remote Testing Methods & Tools Webinar

Thank You!

Floor open to Q&A

Bill AlbertDirector of Design & Usability CenterBentley University

Dave GarrCo-Founder

UserTesting.com

Andrew MayfieldCEO

Optimal Workshop

Matt PaulusDirector of US

SalesUserZoom