ux by the numbers: discovering the why from numbers

©2013 Webnographer Limited

Discovering WHY from numbers

1234567890123456789012345678901234567890123456789012345678901234567890

1


Why Numbers?Really, the question is: Why not?

2


20th Anniversary of Nielson’s Law.

20 years ago it was expensive.

There was a diminishing return.

3

N (1-(1- .31 ) n )


0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%0%

20%

40%

60%

80%

100%

Lab Testing with 10 users

Remote Testing with 80 users

Visibility of the issue

Lik

eli

ho

od

of

dis

co

ve

rin

g a

n i

ss

ue

Discoverability Gap based on the Theory

The above Lab testing numbers mean that 10 users will find 85% of the issues with a visibility of 17%. The these numbers use

the formula from Nielsen, Jakob, and Landauer, Thomas K.: "A mathematical model of the finding of usability problems,“.

N (1-(1- .31 ) n )

4


0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%0%

5%

10%

15%

20%

25%

30%

27%

21%

14%

8%

5%6%

2%3% 3%

1%

2%1% 1%

0%

2%1%

0%1%

1% 1% 1%

Likelihood of an issue occurring

Iss

ue

Dis

trib

uti

on

Issues that can be reliably found in the Lab(5 respondents)

18%56%

Issu

e d

istr

ibu

tion

is t

he

% o

f is

sue

s th

at

fall

into

ea

ch c

ate

go

ry o

f is

sue

vis

ibili

ty

5 respondents find 80% of issues with 30% visibilityBUT that is only 18% of ALL issues

18% of interactions with issues identified27% of interactions have no issue= 45.5% coverage

56% of interactions that have issues get missed.

Discoverability Gap based on data

5


0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%0%

5%

10%

15%

20%

25%

30%

27%

21%

14%

8%

5%6%

2%3% 3%

1%

2%1% 1%

0%

2%1%

0%1%

1% 1% 1%

Likelihood of an issue occurring

Iss

ue

Dis

trib

uti

on

Issues that can be reliably found in the Lab(10 respondents)

28%

10 respondents find 85% of issues with 17% visibilityBUT that is only 28% of ALL issues

28% of interactions with issues identified27% of interactions have no issue= 55% coverage

45% of interactions that have issues get missed.

45%

Discoverability Gap based on dataIs

sue

dis

trib

utio

n is

th

e %

of

issu

es

tha

t fa

ll in

to e

ach

ca

teg

ory

of

issu

e v

isib

ility

6

©2013 Webnographer Limited 7

Benefits: Prioritise findings

0 50 100 150 200 250 300 3500%

5%

10%

15%

20%

25%

30%

35%

40%

45%

40%

9%

Because Remote Un-moderated

Testing evaluates designs with

large numbers of users,

actionable metrics have a small

margin of error. This allows the

prioritisation of insights and high

level decision making.

The findings from the lab have a

large margin of error because

they are evaluated with only a few

users. This makes it hard to

prioritise the findings.Number of participants

Margin of error

Remote Un-moderated TestingLab Testing

The chart uses Sauro’s 20/20 rule. The rule states that Testing with 20 users will produce a margin of error of +/-20%. Also see http://www.measuringusability.com/test-margin.php for an explanation of the 20/20 rule.

http://www.measuringusability.com/test-margin.php


Why care about WHY in quantitative research?

Why, why?


Quant research is powerless without why

• If you know 50% of customers can’t find what they need. What can you change?

• If you cannot explain a finding, it is powerless.


C P Snow

The 2 CulturesScience vs. humanitiesOne without the other is dangerous


How to get WHY from numbers?

11


WISDOM

KNOWLEDGE

INFORMATION

DATA

WHY

HOW

WHAT

THAT


Theory first

Karl PopperA hypothesis is proven in the negative.

SocratesThe Socratic method influenced scientific method where you start with a hypothesis.


Finding out WHY customers

want your product

14


Nobel Prize

Winners of the Nobel prize in science were able to explain phenomenons in the world through theory first


Design as hypothesis

Stakeholder Views

as hypothesis

Review methods

as hypothesis

Other dataas hypothesis

4 types of hypothesis for UX research

Hypothesis for why you are RIGHT

Hypothesis for why you are WRONG


1 2

Optimum Path: • Is the shortest possible path to the target page.• Is the hypothesis of the designer of how customers behave.

Click on Broadband & Internet in Help and Support

Reached the Help hub page

Click on My connection is slow

Reach the “Slow connection” page

3 4


17


1 2 3 4

x

x

x

x

x

x

x

x

There may be other alternative routes to the target page which are longer.

However, the interface should help people find the shortest path to the information. Customers do not want the scenic route. They want to get stuff done.

Longer routes can be an indication for the interface facilitating error recovery.

Lostness

Click on Broadband & Internet in Help and Support

Reached the Help hub page

Click on My connection is slow

Reach the “Slow connection” page

18



Other Data as Hypothesis

Survey data Web analytics dataLab test findings


Stakeholder Hypothesis

CONFUSION UNDERSTANDING


Review Methods as Hypothesis

Heuristic Evaluation, Cognitive Walkthrough, or GOMS for each step of the design hypothesis


Create a UX Hypothesis

The hypothesis is the theory WHY something happens.

• Write down a hypothesis for how customers behave

• What evidence do you need to support your hypothesis?

• What evidence do you need to disprove your hypothesis?


To evaluate the research questions, participants are asked to complete a Webtask on the website.

The Webtask presents participants with a task to perform using the website. While they navigate through the site their interactions are recorded with our remote usability testing tool.

Remote usability testing records all interactions with the website, such as clicks, hovering, keystrokes, back press, page scrolling, time in each form item, errors per item, time on page, pages visited, task completion rate, time on task.

Participant’s interaction with the website allows us to collect comprehensive information regarding website usability, and customer behaviours.

Webtasks provide thus valuable insights on where and why customers are experiencing problems, and casts light on ways to improve website usability.

1

43

2

Test your hypothesis through un-moderated remote usability testing


Remote Usability Testing CanWhen using the right method

25


[email protected]@sabrinamach

[email protected]@jamespage

Thank you.

ux by the numbers: discovering the why from numbers

Technology

reliably found

broadband

target page

hypothesis

customers

issues identified

margin

hypothesis