Beyond Measure

@mulegirl

!#? !#?

:-)

DEEP THOUGHT,

TELL US THE ANSWER…

OK

7,500,000 YEARS LATER…

???

42!

42!!#!?&wtf!

¯\_(ツ)_/¯

IT WOULD HAVE BEEN SIMPLER TO

KNOW THE ACTUAL QUESTION.

ONLY WHEN YOU KNOW THE

QUESTION, WILL YOU KNOW WHAT THE ANSWER MEANS.

I am a rational biped. All rational bipeds rule.

So, I rule.

:-)

:-(

We have to work together to make better decisions.

Quantitative approaches feel

better, but are often actually worse.

We are imperfect humans

designing things for other imperfect humans.

DON’T PANIC

Fe

Flickr/fungleo

Kirk Siang/Flickr

Fe

Flickr/eskimo_jo

[dramatization]

[a dramatization]

The Nourishing Home / Kelly Smith

[dramatization]

Lucky!

luckyironfish.com

oo

O

O

O

OHO

HO

OH

OHH2C

H2CCH2

CH2CH2C

C

CH2 C

C

:N

N:

Human problems require illogical

approaches.

Stories have

power

Flickr/ Billy Wilson

Only 12 people?!?

2,500,000 000,000, 000,000 bytes

90%?

01101000011001010110110001110000001000000110110101100101

010010010010000001100001011011010010000001100010011001010110100101101110011001110010000001110111011000010111010001100011011010000110010101100100

01101000011001010110110001110000001000000110110101100101

:-)

01101000011001010110110001110000001000000110110101100101

010010010010000001100001011011010010000001100010011001010110100101101110011001110010000001110111011000010111010001100011011010000110010101100100

$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$

This is pretty creepy when you think about it.

Flickr/ thevlue

Flickr/ thevlue

This is pretty creepy when you think about it.

I am a rational biped.

one line is definitely longer

Original image: Olly Moss

We are blind to our minds.

1 2System System

1System

2System

Fast Intuitive

Emotional1SystemSystem

1

2SystemSlow

Deliberative Effortful

Data doesn’t change minds.

Ease

Clear Display

Related Experience

Primed Idea

Good Mood

Feels True

Feels Familiar

Feels Good

Feels Effortless

26%

Feeling confident?

You might have a case of Dunning-Kruger

“The incompetent are often blessed with an inappropriate confidence, buoyed by something that feels to them like knowledge.”

—David Dunning

Our access to data has evolved,

our brains haven’t.

Data doesn’t have meaning.

42

Obviously, what I can

count should

count more.

We have a lot of data. We make a lot of decisions.

We like stories. We aren’t good at math.

The hard decisions aren’t math problems, anyway,

DON’T PANIC

QuantitativeQualitative

QuantityQuality

What is going on? How are people behaving?

Why? What does it mean?

What is the problem? Any ideas?

Where? When?

How many? How much? How often?

Which yields more?

Usability Studies Field Studies

Interviews

Clickstream Analysis A/B Testing

Surveys*

Are you a statistician?

Flickr/ Billy Wilson

Tight control.

42!Hi!37banana672

No fishing trips.

Know your POV first.

Flickr/gabbahey

The limit of split testing

Your glorious potential.

A B

Push Push

Traffic Quality

Safe Exciting

Attention $$$

New Familiar!

Mine Ours

More Enough

Yes

What is.

What ought to

be

?

DecisionsDecisions

EVIDENCE

EVIDENCE

What should we do?

Surveys

Surveys are the most dangerous research tool.

Strongly disagree Disagree Neither agree nor disagree Agree Strongly agree

It is too easy to run a survey.

Easy feels true.

Bad surveys don’t smell bad.

Direct interaction with users is prohibited by my organization, but I have

been allowed to conduct a simple survey by email to identify usability issues.

[a dramatization]

The Nourishing Home / Kelly Smith

My boss is a convert to Foresee. She was apparently very skeptical

of it at first, but she's a very analytical person and was

converted by its promise of being able to quantify unquantifiable data

—like "satisfaction".

Customer satisfaction is a lie.

“Changes in customers’ satisfaction levels explain less than 1% of the variation in changes in their share of category spending.”

—MIT Sloan Management Review

“Customer-service scores have no relevance to stock market returns… the most-hated companies perform better than their beloved peers.”

—Bloomberg Businessweek

“Over a one year period 6% of ‘extremely satisfied customers’ closed their accounts whilst 5.8% of ‘unsatisfied customers’ closed their accounts.”

—HBR, 1995

Will the people I’m surveying be willing and able to provide a truthful answer to my question?

Never ask people

what they like.

Never ask people

to remember.

Never ask people

to predict.

How likely are you to attend the opera?

Very!

How concerned are you about looking smart?

Very!

How likely is it you are telling the truth?

Not Very!

Lies Poppycock Irrelevancies

Be Better

ONLY WHEN YOU KNOW THE

QUESTION, WILL YOU KNOW WHAT THE ANSWER MEANS.

“At its core, all business is about making bets on human behavior.”

—Ben Wiseman, WSJ

Define success

first.

The most measurable data is not the most

valuable.

Get actionable insights.

Get useful

information.

Make better choices.

We are imperfect humans

designing things for other imperfect humans.

Know your reason.

What and Why

Flickr/johnwiechecki

Thank you.