ab testing the most powerful way

8/12/2019 AB Testing the Most Powerful Way

1/21


2/21


3/21

A/BTESTING


4/21


5/21

A/BTESTING

T H E M O S T P O W E R F U L W A Y T O

T U R N C L I C K S I N T O C U S T O M E R S

DAN SI ROKER PETE KOOMEN

W I T H CARA HARSHMAN


6/21

C o ve r d e si g n: R y an M ye r s

Copyrigh t# 2 0 1 3 b y D a n S i r ok e r an d P e t e K o o me n . A l l r i g h ts r es e r v ed .

P u bl i sh ed b y J oh n W i le y & S on s , In c ., H ob o ke n, N e w J e rs e y.

P u bl i sh ed s i mu l t an e ou s ly i n C a na d a.

N o p a rt o f th i s p u bl i ca t io n m a y b e re p ro du c ed , st o re d i n a r e tr i ev a l s ys t em , o r tr a ns mi t te d i n a n y f o rmo r b y a n y m e a ns , e l e c t r o ni c , m e c h a n ic a l , p h o t o c op y i ng , r e c o r d i n g, s c a nn i n g, o r o t h e r w i se , e x c e p t a sp e r m it t e d u n de r S e c t i o n 1 0 7 o r 1 0 8 o f t h e 1 9 7 6 U n i t e d S t a t e s C o p y ri g h t A c t , w i t ho u t e i t h er t h e p r i o r

written permission of th e Publish er, or auth orization th rough paymen t of th e appropriate per-copy feet o t h e C o py r i g ht C l e a ra n c e C e nt e r , 2 2 2 R o s ew o o d D r i v e, D a n ve r s , M A 0 1 9 2 3, ( 9 7 8 ) 7 5 08400,fax (978 ) 6468600 , o r o n t h e w e b a t www.copyrigh t.com. Requests to th e Publish er for permissions h o ul d b e a dd r e ss e d t o t h e Pe r m i ss i o n s D e p a rt m e nt , J o h n W i l e y & S o n s, I n c. , 1 1 1 R i v e r S t r e et ,Hoboken , NJ 07030, (201 ) 7486011, fax (201) 7486 0 0 8 , o r o nl i n e a t www.wiley.com/go/permission s.

L i m i t o f L i a b i li t y / Di s c l a im e r o f W a r r a nt y : W h i l e t h e p ub l i s he r a n d a u t h or h a v e u s e d t h e ir b e s t e f f or t s

i n p r e p a r i n g t h is b o o k , t he y m a k e n o r e p r es e n t a t io n s o r w a r ra n t i e s w it h t h e r e s pe c t t o t h e a c c ur a c y o rc o m pl e t e n es s o f t h e c o n t e nt s o f t h i s b o o k a n d s p e cic a ll y d i sc l ai m a n y i m pl i ed w a rr a nt i e s o f m e r c ha n t a b il i t y o r t n e ss fo r a pa r t i c ul a r p u r po s e . N o w a r r a nt y m a y b e c r e a t ed o r e x t e nd e d b y s a l e sr e p r es e n t a t i v e s o r w r i t te n s a l e s m a te r i a l s. T h e a d v i c e a n d s t r a t eg i e s c o n t ai n e d h e r e in m a y n o t b e s u i t a b l ef o r y o u r s i t u a ti o n . Y o u s h o ul d c o n su l t w i t h a p r o f e s si o n a l w h e re a p p r o p r ia t e . N e i t h e r t h e p u b li s h er n o rth e auth or sh all be liable for damages arising h erefrom.

F o r g e ne r a l i n f or m a t i on a b o ut o u r o t he r p r o du c t s a nd s e r v ic e s , p l ea s e c o nt a c t o u r C u st o m er C a r eD e pa r tm e nt w i t hi n t h e U n i t ed S t a t e s a t ( 8 00 ) 7 6 22 9 7 4 , ou t s i de t h e U n it e d S t a t es a t ( 3 1 7 ) 57 23993 orfax (317 ) 5724002.

Wiley publish es in a variety of prin t an d electron ic formats an d by prin t-on -de man d. S ome materiali nc lu de d w it h s ta nd ar d p ri nt v er si on s o f t hi s b oo k m ay n ot b e i nc lu de d i n e -b oo ks o r i n p ri nt -o n-

d ema nd. If t hi s bo ok r efe rs t o me di a suc h a s a C D or DVD t ha t i s no t i nc lu de d i n t he v er si on yo u p u rc h as e d , y o u m a y d o wn l oa d t h is m a te r ia l a t h ttp://booksupport.wiley.com. F o r m o re i n f o r ma t i o na b o u t W i l e y p r o du c t s , v i s i t www.wiley.com.

Library of Congress Cataloging-in-Publication Data:S i r o ke r , D a n .

A /B testin g : th e most powerful way to turn clicks in to customers / Dan S iroker, Pete Koomen .p ag es c mI n c lu d e s i n d ex .

I S BN 9 7 8 - 1- 1 1 8 -5 3 6 0 9- 4 ( c l o th ) ; 9 7 8 - 1 -1 1 8 - 65 9 1 7 -5 ( e b k) ; I S B N 9 7 8 - 1- 1 1 8 - 65 9 2 0 -5 ( e b k )1. Or ga ni za ti on al e ffe ct iv enes s. 2 . Mul ti med ia s ys tem sS o c ia l a s p ec t s .

3 . App li ca ti on so ft wa reTestin g. I. Koomen , Pete, 1982- II. Title.

H D 5 8. 9 . S5 4 4 7 2 0 1 3658.80 3402854678dc23

2013016038

P ri nt ed i n t he U ni t ed S ta t es o f A me ri c a

10 9 8 7 6 5 4 3 2 1
http://www.copyright.com/http://www.wiley.com/go/permissionshttp://booksupport.wiley.com/http://www.wiley.com/http://www.wiley.com/go/permissionshttp://www.copyright.com/http://www.wiley.com/http://booksupport.wiley.com/


7/21

Contents

Chapter 1 How A/B Testing Helped Win the White

HouseTwice 1

PART I LESSONSLEARNED FROM 200,000 A/B TESTS

(ANDCOUNTING) 13

Chapter 2 What to Test 17

Chapter 3 Seek the Global Maximum 33

Chapter 4 Less Is More: Reduce Choices 47

Chapter 5 Words Matter: Focus on Your Call to Action 59

Chapter 6 Fail Fast and Learn 71

PART II IMPLEMENTINGA/B TESTING:

PLAY-BY-PLAYGUIDE 83

Chapter 7 Choose the Solution That's Right for Your

Organization 87

v
http://0.0.0.0/http://0.0.0.0/http://0.0.0.0/http://0.0.0.0/http://0.0.0.0/http://0.0.0.0/http://0.0.0.0/http://0.0.0.0/http://0.0.0.0/http://0.0.0.0/http://0.0.0.0/http://0.0.0.0/http://0.0.0.0/http://0.0.0.0/http://0.0.0.0/http://0.0.0.0/http://0.0.0.0/http://0.0.0.0/http://0.0.0.0/http://0.0.0.0/http://0.0.0.0/http://0.0.0.0/http://0.0.0.0/http://0.0.0.0/http://0.0.0.0/http://0.0.0.0/http://0.0.0.0/http://0.0.0.0/http://0.0.0.0/http://0.0.0.0/http://0.0.0.0/http://0.0.0.0/http://0.0.0.0/http://0.0.0.0/http://0.0.0.0/


8/21


9/21

CHAPTER

1How A/B Testing Helped Win

the White HouseTwice

The $57 Million Button


10/21


11/21

It was 2007 when then-Senator Barack Obama was running for

President, and no one but the Des Moines Registerseemed tothink he had a chance of winning the Democratic primary.

DAN: I was a product manager at Google at the time, and Id

seen Obama speak at our headquarters several weeks prior to

the primary election. I am a big believer in reason and facts

and evidence and science and feedbackeverything that

allows you to do what you do. Thats what we should be doing

in our government,Obama told the packed auditorium. I

think that many of you can help me, so I want you to be

involved.He probably meant that he wanted donations, ormaybe votes, but I took him literally. I took a leave of absence

from Google initially and eventually quit my job to move from

California to Chicago to join the campaign.

I joined what was being called the new mediateam. They

used the phrase new mediabecause it encompassed every-

thing that didnt typicallyt into traditional political cam-

paigns: email, social media, blogging, SMS, and the web.

The team had competent bloggers, designers, and email

copywriters; I wondered where I might be able to make an

impact.

One thing stood out to me: a red button.

Online donations to the campaign came from subscribers

to the email newsletter; subscriptions for this came from the

campaign websites signup form; and the signup form came as

a result of clicking a red button that said Sign Up.This was

3


12/21

the gateway through which all of Obamas email supporters

had to pass; it all came down to one button. So, one simple,humble question immediately became pivotal.

Is This the Right Button?

Is this ourbestchance to get every single supporter, and every

single dollar, that we possibly can?

I had zero political experience at the time, and little clout

within the organization. I didnt have a politicos intuition

about what the button or the image above it should look

likenor the persuasive rhetoric required to run any pro-

posed improvements up the chain of command. All I had was

one insistent question:Is this button the absolute best?and the

desire to nd the answer. There was only one way to know

for certain.

Knowing little about politics or why certain words and

images might be more moving or more effective than others,

I suggested experimenting to gure out what worked to drive

the most signups. Our team tested four different labels for the

button (Sign Up, Sign Up Now, Join Us Now, and

Learn More) and six different media (images and videos)

above it to see which combination induced the most visitors

to engage and sign up.

Our team took bets onwhich variation (Figures 1.1 through 1.3)

would perform best at garnering email signups. Most folks put

their money on Sams Video,a compilation of some of the

most powerful moments in Obamas speeches. We assumed any

video

with not just Obamas image, but his voice and mes-

sagewould lead more people to enter their email addresses

than a simple static image would.

Boy, were we wrong.

4 HOW A/B TESTING HELPED WIN THE WHITE HOUSETWICE


13/21

FIGURE 1.2 The button variations we tested.

Source: Optimizely.

FIGURE 1.1 The original splash page we set out to optimize at the

Obama campaign in 2008.

Source: Optimizely.

How A/B Testing Helped Win the White HouseTwice 5


14/21

FIGURE 1.3 The media variations we tested.

Source: Optimizely.



15/21

In fact, not only Sams Videobuteveryvideo dramatically

underperformedevery image. Even more dramatically, oneimage-and-button combination in particular stood head and

shoulders above the original (Figure 1.4).

A combination of the Family Imageand the Learn More

button improved the signup rate by a staggering 40.6 percent.

Over the course of the campaign, that 40.6 percent lift in

signups translated to 2.8 million more email subscribers,

288,000 more volunteers, andperhaps most important ofallan additional$57 million in donations.

Obama went on, with an enormous lead in dollars and

supporters raised online, to win the election. He was buoyed

by a team willing totest everythingand tolisten to the dataeven

when it surprised them the most.

A small, simple question about a small red button had beenanswered conclusively with a straightforward experiment. But

in its place loomed another question, just as simple and just as

insistent:

Why arent more people doing this?

FIGURE 1.4 A side-by-side comparison of the original and winning

variation of the splash page at the 2008 Obama campaign.

Source: Optimizely.



16/21

The Age of Testing

The answer, in short, was that the commercially available

tools at the time required heavy involvement from software

engineers to run experiments. For all of the spectacular

gains that website testing enabled at the Obama campaign

as well as at big-tech players like Google and Amazonit

was still a highly technical practice. It was simply out of

reach for most businesses that didnt have the know-how

and a dedicated in-house team, and prohibitively difcult

even for many that did. But why did this have to remain

the case? Why couldnt every organization have access to

these tools?

I joined up with my fellow Google product manager, Pete

Koomen, and in 2010 the two of us struck off on our own to

help do just that. What we built as a result is Optimizely, awebsite optimization platform that makes it easy for any

organization, from a one-person startup to a Fortune 100

rm, to do what the Obama team did on the road to the

White Housewith no degrees in statistics or dedicated

engineering team required.

Over the past several years, a range of new tools has emergedto make this online testing and optimization practiceA/B test-

ing, as it is knowneasier and faster. The concept of A/B testing is

simple: show different variations of your website to different

people and measure which variation is the most effective at turning

them into customers. If each visitor to your website is randomly

shown one of these variations and you do this over the same period

of time, then youve created a controlled experiment known as anA/B test. A/B testing has gone from a secret weapon within thepurview of only a handful of tech companies to an increasingly

ubiquitous and critical part of doing business online.



17/21

This sea change in the way companies are conducting online

business and marketing is perhaps best illustrated by taking aglance at the election cycle that came next: the 2012 presidential

race. There were some key differences between 2012 and 2008:

the Obama campaign team had an intense testing program in

place from day one and didnt need to be persuaded as to the

mission-critical value of A/B testing. The other key difference: so

did the opponent, Mitt Romney.

Leading publications from TIME to The Atlantic to Busi-nessweek to Forbeswrote about the 2012 presidential campaignfundraising machines as being the most sophisticated, data-

driven, and efcient organizations that politics had ever seen.

And at the heart of this new reality was A/B testing.

Optimization for Everyone

Whether or not you have plans to run for ofce in the near future,

whether you come from a huge organization or a team of one, and

whether your background is in computer science or marketing,

you can make these same changes happenin your own workplace.

In this book well tell the story of a number of different

companieswhat they tested, what they learned, and how testing

has grown to become part of their mindset, values, and culture.

We at Optimizely have been fortunate to witness this transfor-

mation take place time and again. Part of what we want to change

is the notion that testing is something that only IT staff or

developers can do, because nothing could be further from the

truth. Anyone can conduct A/B testing, and it can affectandinfectan entire organization.

This book is for anyone interested in building a culture of

testing within his or her organization by being curious enough,



18/21

brave enough, and humble enough to ask simple questions like Is

this red button the best?and to experiment to nd the answer.Through our experiences building Optimizely and through

extensive contact with our own customers, weve come to under-

stand very well the challenges that this group facesa group that

includes the following members at its core:

Digital marketers and marketing managers

Designers

Product managers

Software engineers

Entrepreneurs

Copywriters

Growth hackers Data scientists

No matter your role or experience level, your seniority or

budget, youll come away from this book with a host of new ideas,

as well as a seasoned understanding of the challengesand

advantagesinvolved in using data to drive your organizations

growth and revenue. To help you get going, well start with some

of the most common questions:

How should we choose what to test?

How can my team and I adopt a sustainable testing

process?

Where do we begin?

This book takes you through dozens of real-world examples

that show specically how teams like yours have embraced A/Btesting as a fundamental part of not only their day-to-day work-

ow, but their creative process as well. We examine dramaticsuccesses, subtle differences, and surprising failures, and we reveal



19/21

a wealth of ideas and actionable insights by looking through the

eyes of the people and teams who discovered them.Each case study serves to highlight different best (and worst)

practices, and weve grouped our studies broadly based on com-

mon underlying themes. You can read them one after the other,

or pick and choose targeted examples based on what youre

looking for. At the end of each chapter, weve included a brisk

TL;DR (too long; didnt read) condensation of the chapters

main points and key takeaways.

The tools are out there, and theyre not hard to get or hard to

use. All you need to do is start asking questions. Who knows

where they might lead you?



20/21


21/21

ab testing the most powerful way

Documents