how to design with science: and not destroy the magic (joe leech)

49
@mrjoe How to design with science January 8, 2015 Joe Leech Waving einstein How to design with science and not destroy the magic @mrjoe

Upload: future-insights

Post on 27-Jul-2015

8.736 views

Category:

Design


4 download

TRANSCRIPT

@mrjoe

How to design with scienceJanuary 8, 2015

Joe Leech

Waving einstein

How to design with science and not destroy the magic

@mrjoe

Hi, I’m @mrjoe but you can call me Joe

@mrjoe

⅓ ⅓ ⅓ Research DesignDigital

Strategy

@mrjoe

[Newton] has destroyed all the poetry of the rainbow, by reducing it to the prismatic colours. – John Keats

@mrjoe

The product goes live and it's lost

it’s magic

The team / client

continually questioning your

choices

Being micromanaged by your boss / client

Ever heard these issues? Hands-up? I had the first one so badly for about a year, I called it the curse of Joe.

@mrjoe

It started with a need

6

http://www.cxpartners.co.uk/cxblog/the_myth_of_the_page_fold_evidence_from_user_testing/

It started with a need. I had heard the same thing again and again.

I wrote this, have you read it? Almost half a million of you have.

@mrjoe

Here’s the data, from an eyetracking study. It shows 2 designs, one to the left with stuff crammed above the fold. The one on the right where things are spaced out more. People explored more when there was less stuff crammed above the fold.

It showed me there was a need for evidence, reasoning and science to support the design process.

@mrjoe

Design is not so much a design issue as a power struggle. – Alan Cooper

Alan cooper, created visual basic amongst other things, teaches design leadership. Harsh words! So what do we do?

@mrjoe

Design is the process of choosing and organising code words, images and messages

into a form that communicates and influences its audience – @mrjoe + The UK Design

council

Talk stories.

@mrjoe

Here’s some research I did many years ago for the trainline. What does this photo show?

This one photo saved countless time for thousands of people.

@mrjoe

2.

1.

It tells us two things. 1. When people travel they are often carrying loads of stuff 2. most people have a phone and are looking for the conformation email. They don’t print it out. We took the confirmation number and put it as the email subject line.

@mrjoe

I didn’t see them [the asterisks]. There’s nothing that explains what they mean.

How did you hear about us?*

How old are you? *

I’ve heard this a few times in user research. What does it mean? Not enough people to make it statistically significant but enough to change my approach to forms.

@mrjoe

How did you hear about us?

How old are you?Optional

We don’t mark mandatory, we mark optional. It’s enough to get me to change my design approach. Is there enough evidence to prove it from just a handful of users? No, it requires another set of supporting data. Which is why we need to use each approach in combination with another.

Talk business.

@mrjoe

hמp://www.slate.com/articles/technology/technology/2014/06/online_advertising_effectiveness_for_large_brands_online_ads_may_be_worthless.single.html?utm_content=bufferfa367&utm_medium=social&utm_source=twiמer.com&utm_campaign=buffer

http://carolinavega.com/web%20documents%20and%20pics/oreo.jpg

In 2003 Mel Karmazin, a Viacom executive. Viacom are a huge media company where the bulk of their income is from advertising, visited the Google offices. He met Sergey and Larry who preceded to tell him all the ways they could measure and track advertising online. His one comment:

@mrjoe

You’re f---ing with the magic! – Mel Karmazin, Viacom executive

hמp://www.slate.com/articles/technology/technology/2014/06/online_advertising_effectiveness_for_large_brands_online_ads_may_be_worthless.single.html?utm_content=bufferfa367&utm_medium=social&utm_source=twiמer.com&utm_campaign=buffer

http://carolinavega.com/web%20documents%20and%20pics/oreo.jpg

@mrjoe

http://justinmrao.com/lewis_rao_nearimpossibility.pdfOn the Near Impossibility of Measuring the

Returns to Advertising⇤

Randall A. LewisGoogle, Inc.

[email protected]

Justin M. RaoMicrosoft Research

[email protected]

April 23, 2013

Abstract

Classical theories of the firm assume access to reliable signals to measurethe causal impact of choice variables on profit. For advertising expenditurewe show, using twenty-five online field experiments (representing $2.8 mil-lion) with major U.S. retailers and brokerages, that this assumption typicallydoes not hold. Statistical evidence from the randomized trials is very weakbecause individual-level sales are incredibly volatile relative to the per capitacost of a campaign—a “small” impact on a noisy dependent variable can gen-erate positive returns. A concise statistical argument shows that the requiredsample size for an experiment to generate su�ciently informative confidenceintervals is typically in excess of ten million person-weeks. This also impliesthat heterogeneity bias (or model misspecification) unaccounted for by ob-servational methods only needs to explain a tiny fraction of the variation insales to severely bias estimates. The weak informational feedback means mostfirms cannot even approach profit maximization.

Keywords: advertising, field experiments, causal inference, electronic commerce,return on investment, informationJEL Codes: L10, M37, C93

⇤Previous versions circulated under the name “On the Near Impossibility of Measuring Adver-tising E↵ectiveness.” We especially thank David Reiley for his contributions to this work. He alsocame up with the title. Ned Augenblick, Arun Chandrasekhar, Garrett Johnson, Clara Lewis, R.Preston McAfee, Markus Mobius, Michael Schwarz, and Lars Lefgren gave us valuable feedbackas well. We also thank attendees at Brigham Young University’s Economics Seminar, the BeckerFriedman Institute Advances with Field Experiments Conference, and other venues where we havepresented this work. We also thank countless engineers, sales people, and product managers atYahoo, Inc. Much of this work was done when the authors were at Yahoo! Research, Santa Clara,CA.

This paper was published. The study was undertaken written when Lewis and Rao were at Yahoo.

What does this tell us? There is no proof this works. It’s built on faith.

@mrjoe

The 5 basic business models.

19

http://mrjoe.uk

Advertising Affiliate feesSelling Subscription

/ Licensing Market place

What does that mean for you. You need to understand some business basics. I wrote an article. more coming soon. These are the 5 basic income streams for organisations. Understand how they generate money and you have more science to base your designs on. Read more at http://mrjoe.uk

@mrjoe

Strong ideas, weakly held.

http://mrjoe.uk/strong-ideas-weakly-held/

Talk data.

@mrjoe 22

5 years ago. I was a single man. At a point in life where it was becoming harder to meet new people. Someone suggested internet dating. I thought wow, ok, I can do this. My UX background is perfect for meeting the perfect person.

@mrjoe

http://blog.okcupid.com/index.php/the-4-big-myths-of-profile-pictures/

I went in search of data, OK Cupid has a blog. Full of great insights, like what is the perfect profile picture. I set about getting all UX on it, user researching my profile. At one point I emailed OK Cupid and asked if I could drop a google analytics tag my page. I learned a lot about how data can inform design. I also met my wife.

@mrjoe

Classic -Conversion (sales / visits)-Product page conversion (sales / visits to product page)-Funnel analysis-Basket abandonment-Bounce rate-Sales-Leads-Subscribers-Unique visitors-Returning visitors-Page views per visit -Visit to order ratio-Load time-Registrations-Visit / session length -Page views per visit-Time on page-Time on site-Form abandonment-Failed internal search-Referring pages / links-Geographic locations -Print page

Banners:-Click through rate-Impressions

Financial / sales / business -Average order value-Basket value-Profit margin-Average sales price-Cross sell-Gross margin-Category margin-Cost per lead-Customer acquisition cost-Lifetime customer value-Average customer value-Membership / subscription churn

RSS -Feedburner subscriptions-Shares on Google Reader

Call centre calls -Average call length-Support vs sales calls-Inbound vs outbound calls-Web generated calls (unique number on website)-Web fulfilled information calls

SEO -SEM keyword value-SEO positioning-Changes in SERP results/rankings-Top entry pages-Number of keywords triggering results for your site-Number of clicks to your site from keywords-Google trends-Inbound links (back link discovery)-Percentage share of each engine-Branded vs non-branded searches-Affiliate links-Affiliate fees

Social media -Facebook referrals-Incoming Twitter links-Facebook sends/shares/mentions-Facebook likes-Facebook fans-Facebook fan rates-Tweets-Retweets-@s on Twitter

-Twitter followers-Twitter follow rate-Google +1s-Bookmarks on Delicious-StumbleUpon thumbs up-StumpleUpon reviews-Diggs-Google BlogSearch links-Blog comments-Blog articles-Video views-Youtube favourites-Youtube channel subscriptions-Youtube channel comments-Youtube video reviews-Slideshare views-Forum mentions-Thread size-Online review mentions-Stars in reviews-Bit.ly / URL shorteners usage / clicks

Third party / benchmarks -Comscore -Hitwise-Alexa-Compete (US)

Email & campaigns -Email newsletter churn-Email sign-up-Emails sent-Emails bounced (bad address)-Email forwards-Email campaigns-Open rate-Delivery rate (sent - bounces)-Click through rate (CTR)-Email related to conversion / other metric-Unsubscribe rate

-SMS subscribers-Via print publication / 3rd party (unique URLs)

Internal search -Search no results-# Search 1 - 10 results -# Search 10 - 25 results-# Search over 50 results

IA: -GOMS technique-Time to content-Clicks to content -Back button clicks -Task completion-Errors

Testing metrics: -Words Recognition Rate-Reported expectations and performance-Facial reaction-Number of back presses-Gap satisfaction-Path's taken as a measure of scent-Work-flow matches mental model or not 

Satisfaction measurements: -Net Promoter Score-ASQ PDF: After Scenario Questionnaire (3 Questions) -NASA-TLX : NASA's task load index is a measure of mental effort (5 Questions) -SMEQPDF: Subjective Mental Effort Questionnaire -UMEPDF : Usability Magnitude Estimation -SEQ PDF: Single Ease Question

A bewildering amount of metrics. Here’s some I collected over a couple of years. This can be dangerous, organisations can be tied to data, refusing to make a decision without it.

@mrjoe

I had a recent debate over whether a border should be 3, 4 or 5 pixels wide, and was asked to prove my case. I can’t operate in an environment like that… … I won’t miss a design philosophy that lives or dies strictly by the sword of data. – Doug Bowmen

http://stopdesign.com/archive/2009/03/20/goodbye-google.html

This is taken from Doug Bowmen’s blog, after he left Google, he was their first visual designer. Pretty damning.

@mrjoe

Classic -Conversion (sales / visits)-Product page conversion (sales / visits to product page)-Funnel analysis-Basket abandonment-Bounce rate-Sales-Leads-Subscribers-Unique visitors-Returning visitors-Page views per visit -Visit to order ratio-Load time-Registrations-Visit / session length -Page views per visit-Time on page-Time on site-Form abandonment-Failed internal search-Referring pages / links-Geographic locations -Print page

Banners:-Click through rate-Impressions

Financial / sales / business -Average order value-Basket value-Profit margin-Average sales price-Cross sell-Gross margin-Category margin-Cost per lead-Customer acquisition cost-Lifetime customer value-Average customer value-Membership / subscription churn

RSS -Feedburner subscriptions-Shares on Google Reader

Call centre calls -Average call length-Support vs sales calls-Inbound vs outbound calls-Web generated calls (unique number on website)-Web fulfilled information calls

SEO -SEM keyword value-SEO positioning-Changes in SERP results/rankings-Top entry pages-Number of keywords triggering results for your site-Number of clicks to your site from keywords-Google trends-Inbound links (back link discovery)-Percentage share of each engine-Branded vs non-branded searches-Affiliate links-Affiliate fees

Social media -Facebook referrals-Incoming Twitter links-Facebook sends/shares/mentions-Facebook likes-Facebook fans-Facebook fan rates-Tweets-Retweets-@s on Twitter

-Twitter followers-Twitter follow rate-Google +1s-Bookmarks on Delicious-StumbleUpon thumbs up-StumpleUpon reviews-Diggs-Google BlogSearch links-Blog comments-Blog articles-Video views-Youtube favourites-Youtube channel subscriptions-Youtube channel comments-Youtube video reviews-Slideshare views-Forum mentions-Thread size-Online review mentions-Stars in reviews-Bit.ly / URL shorteners usage / clicks

Third party / benchmarks -Comscore -Hitwise-Alexa-Compete (US)

Email & campaigns -Email newsletter churn-Email sign-up-Emails sent-Emails bounced (bad address)-Email forwards-Email campaigns-Open rate-Delivery rate (sent - bounces)-Click through rate (CTR)-Email related to conversion / other metric-Unsubscribe rate

-SMS subscribers-Via print publication / 3rd party (unique URLs)

Internal search -Search no results-# Search 1 - 10 results -# Search 10 - 25 results-# Search over 50 results

IA: -GOMS technique-Time to content-Clicks to content -Back button clicks -Task completion-Errors

Testing metrics: -Words Recognition Rate-Reported expectations and performance-Facial reaction-Number of back presses-Gap satisfaction-Path's taken as a measure of scent-Work-flow matches mental model or not 

Satisfaction measurements: -Net Promoter Score-ASQ PDF: After Scenario Questionnaire (3 Questions) -NASA-TLX : NASA's task load index is a measure of mental effort (5 Questions) -SMEQPDF: Subjective Mental Effort Questionnaire -UMEPDF : Usability Magnitude Estimation -SEQ PDF: Single Ease Question

Measure 2 to 3 things.

Each has: 1. Timescale

2. Benchmark 3. Reason to be

reported 4. Associated

action

http://mrjoe.uk/ux-and-roi-what-to-measure-and-what-to-expect/

Truth is it’s important to measure. It shows confidence in your design. Here’s some tips to get it right. See a talk by me on this here http://mrjoe.uk/ux-and-roi-what-to-measure-and-what-to-expect/

@mrjoe 27

Data shows confidence in the design

@mrjoe http://www.cxpartners.co.uk/cxblog/multi_variant_testing_and_the_flying_squirrel/

The squirrel evolved bit by bit to fly. Small incremental improvements that allowed him to jump further, land better, and eventually to glide. The squirrel has reached a point where he isn’t going to get better at flying (called a local maximum).

@mrjoe

The soaring eagle. Elegantly drifting through the air with little effort. The eagle also evolves, little by little to its own maxim. It’s a superb flyer, soaring to 15,000 feet instead of making long leaps between trees. The eagle started from a better set of basics: front limbs to wings, reptile scales to feathers.

Good design gives a good starting point for evolution. MVT can get you to a maxim, but it can’t help you choose the best starting point to get to the highest maxim. A strong design leads to elegant flight. A weak design leads to hopping between branches. We need to combine MVT with craft. Read more: http://www.cxpartners.co.uk/cxblog/multi_variant_testing_and_the_flying_

@mrjoe

Sore like an eagle don’t leap like a squirrelhttp://www.cxpartners.co.uk/cxblog/multi_variant_testing_and_the_flying_squirrel/

Talk theory.

@mrjoe

James Vicary, in 1957, flashed up hungry eat popcorn, thirsty drink coke as single frames during a movie showing.

@mrjoe

58%A reported 58% increase in sales of popcorn. But it was later found out to be a hoax. He made it up to prop up his failing advertising business.

@mrjoe

Some examples, but there is no proof. http://www.badabingrecords.com/2011/03/the-subliminal-scares-a-short-history-of-an-american-obsession/ http://www.businessinsider.com/subliminal-ads-2011-5?op=1

@mrjoe

Strikingly, there was no significant effect.

http://www.bbc.co.uk/news/magazine-30878843

The media’s obsession. They were shown the same three-minute clip from the BBC/Kudos drama Spooks twice, but each time one group wore the blindfolds. The clip watched by the group with red blindfolds contained a 10-millisecond flash, every five seconds, of the word “Lipton", http://www.bbc.co.uk/news/magazine-30878843

So what can we do?

@mrjoe

There are resources out there that help with theory. This is from Ribot, alumni of my psychology for designers workshop. http://coglode.com/

Workshop: http://mrjoe.uk/psychology-design-ux-workshop/

@mrjoe PsychologyForDesigners.com

Or you can buy my book, I show you how to find a theory and apply it to design. How to use psychology to advocate design.

PsychologyForDesigners.com

@mrjoe

Title Text

38

A designer who doesn't understand psychology

is going to be no more successful

than an architect who doesn't

understand physics Image: http://victorenrich.com/archives/155

@mrjoe

@mrjoe

The biggest myth ever perpetuated in the design field is that good design sells itself. – Mike

Montiero

Talk objectively.

@mrjoe

'I feel the design, minimalist in style, reflects a modern design aesthetic’

- The Designer

@mrjoe

‘Well I feel more that it’s a website and not

a f#cking coffee table.’

- The Product Manager

@mrjoe

'I feel the design, minimalist in style, reflects a modern design aesthetic’

- The DesignerWhat was wrong with that statement? Anytime you say I feel, it leaves the door open for the ‘Well I feel’. It becomes a battle of wills, there is no subjectivity to judge the design against.

@mrjoe http://www.gensleron.com/storage/post-images/cities/reinventingaclassicreuniontowerobservationdeckreopens/1.jpg

Which reminds me, I was once looking over a beautiful view, but I got annoyed at the poor information design of the support panel. My wife simply said, chill the f$ck out and enjoy the view, the view is the focus not the design of the information panel.

@mrjoe

Art: …works to be appreciated primarily for their beauty or

emotional power. – Merriam Webster Dictionary

@mrjoe

http://www.ise.bgu.ac.il/faculty/noam/papers/00_nt_ask_di_iwc.pdf

Art does support design. Pretty things are easier to use.

@mrjoe

What is beautiful is usable.

1. Talk stories 2. Talk business

3. Talk data 4. Talk theory

5. Talk objectively

@mrjoe

mrjoe.uk