beyond the usability lab - bentley university · pdf filekeys to success agenda. design and...

Design and Usability Center

175 Forest Street, Waltham, MA 02452 Tel: 1-800-523-2354

Beyond the Usability Lab: Exploring Large-Scale User Experience Research

Bill AlbertDirector, Design and Usability Center

[email protected]

Lena DmitrievaSenior Consultant, Design and Usability Center

[email protected]

2


Acknowledgments

• Thanks to Tom Tullis and Donna Tedesco, Fidelity

Investments

• Based on upcoming book: “Beyond the Usability Lab:

Conducting Large-Scale User Experience Studies” (published by Morgan Kauffman/Elsevier in January, 2010)

3


• Основан в 1999 году

• Оказывает консалтинговые услуги для корпоративных клиентов

• Работает с Master’s program - Human Factors in Information Design

• Консультанты

Bill Albert, PhD – Director Bill Gribbons, PhD - Sr. Consultant

Rich Buttiglieri - Usability Consultant Lena Dmitrieva - Usability Consultant

Kris Engdahl – Usability Consultant Chris Hass – Usability Consultant

9 Research Associates

О центре

4


Наши клиенты

http://www.metlife.com/Applications/Corporate/WPS/CDA/PageGenerator

http://www.bcbs.com/

http://www.discovercard.com/

https://www.fidelity.com/?bar=c

http://www.metlife.com/Applications/Corporate/WPS/CDA/PageGenerator

http://www.bcbs.com/

http://www.brighamandwomens.org/

5


Наши технические возможности

• Две юзабилити лаборатории

• Прямая трансляция юзабилити сессий

• Eye tracking

• Возможность удаленного тестирования

• Оборудование, предназначенное для

тестирования мобильных и медицинских

устройств

6


Мы предлагаем

Изучение потребительских нужд

• Анализ пользователей - Персоны

• Интернет опросы

• Исследования вне лаборатории

• Фокус-группы

Тестирование, оценки, дизайн

• Количественные и качественные исследования

• Исследования по-русски

• Экспертная оценка интерфейсов

• Юзабилити-тестирование: в лаборатории, удаленное, без участия ведущего

• Информационная архитектура

• Доступность информационно-коммуникационных технологий

User Еxperience Обучение

• Корпоративные треннинги

• Интенсивные треннинги

7


1. Context of UX research

2. What is large-scale UX research?

3. Strengths, limitations and myths

4. Complements with other UX research methods

5. How to conduct a study (planning, design, pilot/launch,

and analysis)

6. Keys to success

Agenda


Context of UX Research

Beyond the Usability Lab:

9


The Goal of UX Research

• Making smart and quick decisions

• Informing design and strategy

• Sparking innovation

• Methods don’t matter

10


I am a big fan of traditional usability

11


Missing Piece in our Toolkit

12


Simplified Toolkit

13


Big Picture


What is Large-Scale

UX Research?


15


What do you call this?

Unmoderated Usability

Online UsabilitySelf-Guided Usability

Remote UsabilityAutomated Usability

Asynchronous Usability

Large-Scale Usability

16


5 basic ingredients

1. Interactive system (usually a website)

2. A lot of participants (n>50 per segment)

3. An online tool to moderate study and collect data

4. Tasks (not just attitudes/opinions)

5. Capturing data about their experience

17


Welcome Page

18


Screener Questions

19


Task Instruction

20


Dialog

21


Post Task Questions

22


Follow-up Questions

23


Open-ended Questions

24


Types of Studies

• Comprehensive evaluation

• UX benchmark

• Competitive evaluation

• Live site vs. prototype comparison

• Feature/function test

• Discovery

25


Types of Data

• Performance

• Success, time, efficiency

• Self-reported

• Task ratings, SUS, expectations, verbatims, etc

• Clickstream

• Pages, times, clicks, data entry, abandonment, etc

26


Outcomes

• What are the usability issues, and how big?

• Which design is better, and by how much?

• How do different customer segments differ?

• What is the user experience like?


Strengths, Limitations & Myths


28


Strengths

1. Comparing products

2. Measuring user experience

3. Finding the right participants

4. Focusing on design improvements

5. Insight into the user’s real experience

29


Limitations

1. Not well suited to rapid, iterative design

2. Need a deep understanding of issues

3. Studies that require long sessions

4. Lose control over prototypes

5. Internet access

30


Myths

1. Only test with websites

2. It is very expensive

3. Only gather quantitative data

4. A lot of noise in the data

5. Same as any market research

study


Compliments to Other

UX Methods


32


• Use lab test to identify issues, iterate, and the run

online study to validate

• After an online study, use lab to do a deep dive on

issues

Traditional Lab Testing

33


• Use focus groups to generate design concepts, then

use online study to validate

Focus Groups

34


• Conduct an expert review to identify the most

significant issues, then test online

Expert Reviews

35


Web Traffic Analysis

• Web traffic analysis can be used to identify areas of

concern, followed by an online test as a deep dive into

the problems

• Web traffic can

also follow an

online test to

validate design

changes


Planning a Large-Scale

UX Study


37


Setting Goals

• Compare prototype designs

• Benchmark

• Identify issues and prioritize

• Validate new design against

current design

• Compare specific design

features/functions

38


Budget

• Recruiting costs• $2 - $20 per complete (panel)

• No cost if using internal list

• Incentives• $5 - $10 per complete

• Drawing to win

• No incentive

• Technology• Annual license or per study for commercial service

• Discount approach

39


Timelines

ID Task Name Start Finish Duration

Jan 3 2010 Jan 10 2010 Jan 17 2010

4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

1 3d01/06/201001/04/2010Plan Goals and Structure of Study

2 5d01/11/201001/05/2010Build prototypes (2 Wizard Flows, with

slight difference)

3 3d01/11/201001/07/2010Build tasks and questions into existing

online study tool

4 3d01/13/201001/11/2010Pilot and Test tool (2 iterations)

2d01/15/201001/14/2010Launch Study5

2d01/19/201001/18/2010Analysis

7

6

1d01/20/201001/20/2010Present Results

21 22

ID Task Name Start Finish Duration

Mon Jan 4

8 9 10 11 12 1 2 3

1 1h01/04/201001/04/2010Point online study tool to prototypes

and create tasks and questions

2 1h01/04/201001/04/2010Pilot with team members and iterate

3 2h01/04/201001/04/2010Launch study internally

4 2h01/04/201001/04/2010Analyze and report back results

40


Commercial Services

41


Discount Approach


Designing a Study


43


Introducing the Study

• Purpose

• Sponsor/contact

information

• Time estimate

• Incentive

• Technical requirements

• Legal stuff

• Instructions

44


Screeners

• Screener questions focus study on target users

• Most common screens based on product experience

and demographics

• Screening questions need to be grouped together so

they are hard to guess

• Careful about

fraudulent

participants

45


Starter Questions

• Customer

segmentation

• Product experience

• Expectations

• Avoid sensitive

questions

• Conditional logic

46


Tasks

• Open-ended (discovery)

• “Learn about how carbon offsets work”

• Close-ended (measuring success)

• “What is the ISBN number of the book “Measuring the User

Experience”

• Self-generated (discovery)

• “What would you like learn on this website?...”

• Self-selected

• “Choose 5 tasks that are most important to you”

47


Mental Cheaters

• About 10% of participants go through the study as fast

as possible to get the incentive

• Speed traps

• Consistency checks

48


Post-Task Questions

• Ease of use

• Confidence

• Open-ended

• Proxy measures

49


Post Session Questions

• Overall

assessment

(SUS)

• Demographics

• Open-ended

(focus on

specifics)


Piloting and Launching


51


Piloting and Launching

Technical checks

Usability checks

Full pilot with data checks

Timing the launch

Singular launch

Phased launches

Monitoring results


Data Analysis


53


Data Cleanup

Removing participants

Incomplete data

Fraudulent participants

Mental cheaters

Removing or modifying data for individual tasks

Outliers

Contradictory responses

Removing a task for all participants

Modifying task success

54


Data Analysis

• Task performance

• Self-reported

• Clickstream

• Combined metrics

55


Task Performance

• Task success

• Task time

• Efficiency (tasks per minute)

• Learnability (efficiency over time)

Number of Correct Tasks per Minute

(Error bars represent the 90% confidence interval)

1.80

1.85

1.90

1.95

2.00

2.05

2.10

2.15

2.20

Static TOC Expand-Collapse TOC Auto-collapse TOC

56


Self-Reported Data

• Rating Scales

• Likert Scale (disagree – agree)

• Semantic Differential (easy –

difficult)

• Open-ended questions

• Based on conditional logic Before and After Task Ease Ratings

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0

Before Rating

Aft

er

Rati

ng

Harder Easier

Easier

Harder

Harder Easier

Easier

Harder

T1

T2

T3 T4

T5

Fix It FastBig Opportunity

Promote It Don’t Touch It

Task Ease Ratings (1-5, Higher=Easier)


1.0

1.5

2.0

2.5

3.0

3.5

4.0

4.5

5.0

Ferret Food Store Location Gift Card Shipping Pine Cat Litter

Site A

Site B

57


Clickstream Analysis

• Percentage of participants who used specific paths

• Intended paths, acceptable paths, unacceptable paths

• Abandonment (rates and pages)

• Total page visits (compared to minimum)

• Lostness metric# Pages Beyond Minimum

0

2

4

6

8

10

12

Task 1 Task 2 Task 3 Task 4

58


Combined Metrics

• Convert each

metric to a

percentage, then

average all

percentages

• Z-score

transformation

• Jeff Sauro’s SUM

method

Participant Condition

Mean Task

Ease Rating

Percent

Correct

Mean Task

Time (sec) Time %

Task

Rating %

Combined

Task %

P382 Static TOC 4.0 100% 20 50% 75% 75%

P383 Static TOC 4.4 100% 24 42% 85% 76%

P384 Auto-collapse TOC 4.4 100% 19 51% 85% 79%

P385 Expand/Collapse TOC 4.0 100% 47 21% 75% 65%

P386 Static TOC 3.8 90% 26 39% 70% 66%




P390 Static TOC 4.9 100% 23 43% 98% 80%


P392 Static TOC 4.3 90% 22 46% 83% 73%

P393 Static TOC 4.4 100% 17 60% 85% 82%


P395 Static TOC 3.2 100% 25 40% 55% 65%







Overall Task % (Accuracy, Speed, Task Ease)


67%

68%

69%

70%

71%

72%

Static TOC Expand/Collapse TOC Auto-collapse TOC


Keys to Success


60


Keys to Success

1. Choosing the right tool

2. Thinking beyond the web

3. Test early

4. Compare alternatives

5. Consider the entire user experience

6. Use your entire toolkit

7. Explore your data

8. Sell your results

9. Trust your data (within limits)

10. Just dive in

61


Resources

• www.measuringux.com

• www.remoteusability.com

• www.userzoom.com

• www.measuringusability.com

• http://keynote.com/product

s/customer_experience/

http://www.measuringux.com/

http://www.remoteusability.com/

http://www.userzoom.com/

http://www.measuringusability.com/

http://keynote.com/products/customer_experience/

http://keynote.com/products/customer_experience/


175 Forest Street, Waltham, MA 02452 Tel: 1-800-523-2354

Спасибо Thank you

Bill AlbertDirector, Design and Usability Center

[email protected]

Lena DmitrievaSenior Consultant, Design and Usability Center

[email protected]

beyond the usability lab - bentley university · pdf filekeys to success agenda. design and...

Documents