evolve your a/b testing skills

77
@caraharshman Uplevel Your A/B Testing Skills

Upload: cara-harshman

Post on 15-Apr-2017

4.126 views

Category:

Marketing


2 download

TRANSCRIPT

Page 1: Evolve Your A/B Testing Skills

@ c a r a h a r s h m a n

U p l e v e l Y o u r A / B T e s t i n g S k i l l s

Page 2: Evolve Your A/B Testing Skills

August 2nd 2016, Central Park, NYC

Page 3: Evolve Your A/B Testing Skills
Page 4: Evolve Your A/B Testing Skills

@ c a r a h a r s h m a n

Y o u r A / B T e s t i n g S k i l l sE v o l v e

Page 5: Evolve Your A/B Testing Skills

POKÉMARKETÉRS

Page 6: Evolve Your A/B Testing Skills

A/B TESTING SKILLSEVOLVE YOUR

@CaraHarshman #MozCon

Page 7: Evolve Your A/B Testing Skills

27%

3% 70%

70% OF MARKETERS SAID CONVERSION OPTIMIZATION IS MORE OF A PRIORITY FOR 2016.

Source: ConversionXL

SAME

LESS MORE

GREAT NEWS @CaraHarshman #MozCon

Page 8: Evolve Your A/B Testing Skills

41%56%

4%56% OF TEAMS WILL ALLOCATE MORE BUDGET TO CONVERSION OPTIMIZATION IN 2016.

Source: ConversionXL

SAME

MORE

LESS

DECENT NEWS @CaraHarshman #MozCon

Page 9: Evolve Your A/B Testing Skills

BIGGEST OPTIMIZATION CHALLENGES: KNOWLEDGE RESOURCES KNOWING WHAT TO TEST

NOT GREAT NEWS @CaraHarshman #MozCon

Page 10: Evolve Your A/B Testing Skills

IDEAS EXECUTION TEAM

EVOLVE YOUR A/B TESTING

@CaraHarshman #MozCon

Page 11: Evolve Your A/B Testing Skills

IDEAS EXECUTION TEAM

EVOLVE YOUR A/B TESTING

@CaraHarshman #MozCon

HYPOTHESES AUTOMATION ADVOCATES

Page 12: Evolve Your A/B Testing Skills

1. LEARN RATE = 100% 2. BUSINESS GROWTH, HIGHER REVENUE 3. DELIGHTED CUSTOMERS 4. STRONGER DECISIONS & CULTURE

OPTIMAL OUTCOME OF A/B TESTING EVOLUTION

@CaraHarshman #MozCon

Page 13: Evolve Your A/B Testing Skills

AN EXAMPLE FROM

KYLE RUSH

Page 14: Evolve Your A/B Testing Skills

AN EXAMPLE FROM

KYLE RUSH+ HILLARY

Page 15: Evolve Your A/B Testing Skills

HYPOTHESIS

HILLARY CLINTON CAMPAIGN

@CaraHarshman #MozCon@CaraHarshman #MozCon

IF WE MAKE IT AS EASY AS POSSIBLE TO SAVE YOUR CREDIT CARD INFO,

Page 16: Evolve Your A/B Testing Skills

HYPOTHESIS

HILLARY CLINTON CAMPAIGN

@CaraHarshman #MozCon@CaraHarshman #MozCon

IF WE MAKE IT AS EASY AS POSSIBLE TO SAVE YOUR CREDIT CARD INFO, THEN MORE PEOPLE WILL DONATE,

Page 17: Evolve Your A/B Testing Skills

HYPOTHESIS

HILLARY CLINTON CAMPAIGN

@CaraHarshman #MozCon@CaraHarshman #MozCon

IF WE MAKE IT AS EASY AS POSSIBLE TO SAVE YOUR CREDIT CARD INFO, THEN MORE PEOPLE WILL DONATE, BECAUSE WE’VE REMOVED THE FRICTION OF PAYMENT.

Page 18: Evolve Your A/B Testing Skills

@CaraHarshman #MozConORIGINAL FLOW

Page 19: Evolve Your A/B Testing Skills

@CaraHarshman #MozConORIGINAL FLOW

VARIATION FLOW

Page 20: Evolve Your A/B Testing Skills

RESULTS

@CaraHarshman #MozCon

CONVERSION LIFT: 239% ORIGINAL CR: 9%

VARIATION CR: 31%

Page 21: Evolve Your A/B Testing Skills

VARIATION

RESULTS

@CaraHarshman #MozCon

Page 22: Evolve Your A/B Testing Skills

IDEAS HYPOTHESES EVOLVE YOUR A/B TESTING

@CaraHarshman #MozCon

Page 23: Evolve Your A/B Testing Skills

IDEAS HYPOTHESES “LET’S… ADD THIS MOVE THAT SWAP THIS FOR THAT SHORTEN THIS CHANGE THIS”

@CaraHarshman #MozCon

Page 24: Evolve Your A/B Testing Skills

IDEAS HYPOTHESES “LET’S… ADD THIS MOVE THAT SWAP THIS FOR THAT SHORTEN THIS CHANGE THIS”

@CaraHarshman #MozCon

“IF WE DO THIS THEN THIS WILL HAPPEN BECAUSE OF THIS REASON.”

Page 25: Evolve Your A/B Testing Skills

@CaraHarshman #MozCon

X

X

IDEAS HYPOTHESES

Page 26: Evolve Your A/B Testing Skills

IDEAS EXECUTION TEAM

EVOLVE YOUR A/B TESTING

@CaraHarshman #MozCon

HYPOTHESES AUTOMATION ADVOCATES

Page 27: Evolve Your A/B Testing Skills

@CaraHarshman #MozCon

X

X

ANALYTICSVOICE OF THE CUSTOMER

IDEAS HYPOTHESES

Page 28: Evolve Your A/B Testing Skills

GREAT A/B TESTS HYPOTHESES ARE BORN OUT OF REASON & DATA, NOT WHITEBOARD SESSIONS

@CaraHarshman #MozCon

Page 29: Evolve Your A/B Testing Skills

ANALYTICS

@CaraHarshman #MozCon

QUANTITATIVE / QUALITATIVE INFORMATION ABOUT YOUR WEBSITE/APP

Page 30: Evolve Your A/B Testing Skills

TOP ANALYTICS SOURCES

@CaraHarshman #MozCon

1. FUNNEL REPORT (GA) 2. HEAT MAP ON HOME & HIGH INTENT PAGES 3. FORM ERROR SUBMISSION REPORT (GA) 4. EXIT / PAGE BOUNCE REPORT (GA) 5. HIGH TRAFFIC / LOW CONVERSION LANDING PAGES

Page 31: Evolve Your A/B Testing Skills

VOICE OF THE CUSTOMER

@CaraHarshman #MozCon

EMPATHY BASED ANALYSIS, UNDERSTAND THE NEEDS OF THE END USER AS A HUMAN

Page 32: Evolve Your A/B Testing Skills

VOICE OF THE CUSTOMER LISTENING CHANNELS

@CaraHarshman #MozCon

1. CUSTOMER SUCCESS TEAM 2. RECORDED USER TESTING SESSIONS 3. COFFEE SHOPS 4. ON-PAGE SURVEYS 5. EMAIL SURVEYS

Page 33: Evolve Your A/B Testing Skills

VOICE OF THE “HUMAN”

@CaraHarshman #MozCon

1. LIFT MODEL BY WIDER FUNNEL 2. CONVERSION EQUATION FROM OLI GARDNER 3. PSYCHOLOGY 4. DESIGN PRINCIPLES 5. NEUROMARKETING

Page 34: Evolve Your A/B Testing Skills

MVMT WATCHESA HYPOTHESIS CASE STUDY

Page 35: Evolve Your A/B Testing Skills

HYPOTHESIS

MVMT WATCHES

@CaraHarshman #MozCon@CaraHarshman #MozCon

IF WE ADD THE NEW PRODUCT UP-SELL ON THE PRODUCT DETAIL PAGE,

Page 36: Evolve Your A/B Testing Skills

HYPOTHESIS

MVMT WATCHES

@CaraHarshman #MozCon@CaraHarshman #MozCon

IF WE ADD THE NEW PRODUCT UP-SELL ON THE PRODUCT DETAIL PAGE, THEN SALES WILL INCREASE

Page 37: Evolve Your A/B Testing Skills

HYPOTHESIS

MVMT WATCHES

@CaraHarshman #MozCon@CaraHarshman #MozCon

IF WE ADD THE NEW PRODUCT UP-SELL ON THE PRODUCT DETAIL PAGE, THEN SALES WILL INCREASE BECAUSE THIS PLACEMENT MIMICS THE IN-STORE BUYING EXPERIENCE.

Page 38: Evolve Your A/B Testing Skills

VARIATION

MVMT WATCHES

@CaraHarshman #MozCon@CaraHarshman #MozCon

Page 39: Evolve Your A/B Testing Skills

RESULTS

@CaraHarshman #MozCon

CONVERSION LIFT DESKTOP: 2.2%

MOBILE: 5.5%

Page 40: Evolve Your A/B Testing Skills

@CaraHarshman #MozCon

Page 41: Evolve Your A/B Testing Skills

A/B TEST HACKATHON

@CaraHarshman #MozCon

1. GIVE PEOPLE ACCESS TO DATA 2. INVITE CROSS FUNCTIONAL TEAMS 3. GIVE PRIZES 4. FOLLOW UP

Page 42: Evolve Your A/B Testing Skills

TESTING HACKATHON POSTER

DOWNLOAD

OPTIMIZELY.COM/TESTING-TOOLKIT/

Page 43: Evolve Your A/B Testing Skills

@CaraHarshman #MozCon

X

X

ANALYTICSVOICE OF THE CUSTOMER

IDEAS HYPOTHESES

Page 44: Evolve Your A/B Testing Skills

EXECUTION AUTOMATION

EVOLVE YOUR A/B TESTING

@CaraHarshman #MozCon

GET SHIT DONE

Page 45: Evolve Your A/B Testing Skills

EXECUTION AUTOMATION

@CaraHarshman #MozCon

GET SHIT DONEGET SHIT DONE

Page 46: Evolve Your A/B Testing Skills

EXECUTION AUTOMATION

@CaraHarshman #MozCon

GET SHIT DONE MAKE A PLAN SO WE CAN SCALE

Page 47: Evolve Your A/B Testing Skills

@CaraHarshman #MozCon

GET SHIT DONEX

X

PRIORITIZATION

STANDARDIZATION

EXECUTION AUTOMATION

Page 48: Evolve Your A/B Testing Skills

IF YOU WANT TO SCALE, THEN YOU HAVE TO TAKE OFF YOUR GET-SHIT-DONE-HAT

Image source: http://teeeshop.com/products/get-shit-done-bucket-hat

Page 49: Evolve Your A/B Testing Skills

PRIORITIZATION

@CaraHarshman #MozCon

AN EMOTION-FREE SYSTEM TO DECIDE WHAT TO TEST

Page 50: Evolve Your A/B Testing Skills

PRIORITIZATION CHART: EASE VS. IMPACT

@CaraHarshman #MozCon

IF YOU CAN MAKE TIMEDEPRIORITIZE

NEEDS A VERY SOLID

HYPOTHESISDO IT!

EASE

IMPACT

Page 51: Evolve Your A/B Testing Skills

PRIORITIZATION SCORE CARD

@CaraHarshman #MozCon

RULE 1 point 0 points

page part of funnel outside funnel

audience 100% of visitors

specific audience

success metric primary goal secondary

goal

A RULE & POINT BASED SYSTEM WHERE EVERY HYPOTHESIS GETS A SCORE

Page 52: Evolve Your A/B Testing Skills

“IF YOU COME TO ME WITH AN IDEA AND IT’S NOT LIVE IN 2 WEEKS, IT’S NOT BECAUSE IT’S A BAD IDEA – IT’S BECAUSE I HAVE BETTER THINGS TO TEST.”-Pauline, PM at Hotwire

Page 53: Evolve Your A/B Testing Skills

PRIORITIZATION WORKSHEETS

DOWNLOAD

OPTIMIZELY.COM/TESTING-TOOLKIT/

Page 54: Evolve Your A/B Testing Skills

STANDARDIZATION

@CaraHarshman #MozCon

A BROADLY AGREED-UPON SET OF PRIORITIES, METRICS, DEFINITIONS

Page 55: Evolve Your A/B Testing Skills

3 PIECES TO STANDARDIZE

@CaraHarshman #MozCon

1. MAIN SUCCESS METRICS 2. STOPPING CRITERIA FOR TESTS 3. IMPORTANT AUDIENCES

Page 56: Evolve Your A/B Testing Skills

STANDARDIZE SUCCESS METRICS

@CaraHarshman #MozCon

WHAT ARE THE ACTIONS YOU ARE TRYING TO MOVE?

COMPANY GOALS

TEAM GOALS

TESTING GOALS

TEST HYPOTHESIS

GOAL

Page 57: Evolve Your A/B Testing Skills

STANDARDIZE STOPPING CRITERIA

@CaraHarshman #MozCon

HOW LONG WILL A TEST RUN BEFORE YOU STOP IT?

STOP

Page 58: Evolve Your A/B Testing Skills

STANDARDIZE YOUR AUDIENCES

@CaraHarshman #MozCon

WHICH UNIQUE TRAFFIC SEGMENTS DESERVE MORE ATTENTION?

NewGoogle

Mobile

DesktopReturning

Referral

Page 59: Evolve Your A/B Testing Skills

BROOKS RUNNINGA STANDARDIZATION CASE STUDY

Page 60: Evolve Your A/B Testing Skills

HYPOTHESIS

BROOKS RUNNING

@CaraHarshman #MozCon@CaraHarshman #MozCon

IF WE DELIVER UNIQUE CONTENT TO SHOPPERS BUYING SIMILAR SIZE SHOES,

Page 61: Evolve Your A/B Testing Skills

HYPOTHESIS

BROOKS RUNNING

@CaraHarshman #MozCon@CaraHarshman #MozCon

IF WE DELIVER UNIQUE CONTENT TO SHOPPERS BUYING SIMILAR SIZE SHOES, THEN THEY WILL BE LESS LIKELY TO RETURN

Page 62: Evolve Your A/B Testing Skills

HYPOTHESIS

BROOKS RUNNING

@CaraHarshman #MozCon@CaraHarshman #MozCon

IF WE DELIVER UNIQUE CONTENT TO SHOPPERS BUYING SIMILAR SIZE SHOES, THEN THEY WILL BE LESS LIKELY TO RETURN BECAUSE THEY DECIDED ON A SIZE.

Page 63: Evolve Your A/B Testing Skills

@CaraHarshman #MozCon

EXPERIENCE FOR INDECISIVE SHOPPERS

Page 64: Evolve Your A/B Testing Skills

RESULTS

@CaraHarshman #MozCon

RETURN RATE FOR THIS AUDIENCE: -80%

CONVERSION RATE: +2%

Page 65: Evolve Your A/B Testing Skills

@CaraHarshman #MozCon

GET SHIT DONEX

X

PRIORITIZATION

STANDARDIZATION

EXECUTION AUTOMATION

Page 66: Evolve Your A/B Testing Skills

TEAM ADVOCATES

EVOLVE YOUR A/B TESTING

@CaraHarshman #MozCon

Page 67: Evolve Your A/B Testing Skills

TEAM ADVOCATES

@CaraHarshman #MozCon

PEOPLE WHO DO WORK

Page 68: Evolve Your A/B Testing Skills

TEAM ADVOCATES

@CaraHarshman #MozCon

PEOPLE WHO DO WORK

PEOPLE WHO PROMOTE YOUR WORK

Page 69: Evolve Your A/B Testing Skills

@CaraHarshman #MozCon

X X

COMMUNICATIONDIVERSE SKILLS

TEAM ADVOCATES

Page 70: Evolve Your A/B Testing Skills

30%

29%

20%

21%WHO DOES OPTIMIZATION AT YOUR ORGANIZATION?

Source: ConversionXL

(PRETTY) GOOD NEWS @CaraHarshman #MozCon

NOBODY

A TEAM OF PEOPLE

ONE PERSON PEOPLE ACROSS TEAMS

Page 71: Evolve Your A/B Testing Skills

TOP 5 SKILL SETS TO HAVE

@CaraHarshman #MozCon

1. PROJECT MANAGER 2. DEVELOPER 3. DESIGNER 4. ANALYST 5. FEARLESSNESS

Page 72: Evolve Your A/B Testing Skills

2 TEAM EXAMPLES

@CaraHarshman #MozCon

JOSEPHINE AT HOTWIRE

DESIGN DEVELOPERCUSTOMER SUCCESS

PRODUCT OWNER, HOTELS

PRODUCT OWNER, FLIGHTS

CRAIG HEAD OF GROWTH

JOSEPHINE PRODUCT MANAGER

FOR TESTING

CRAIG AT EARNEST

Page 73: Evolve Your A/B Testing Skills

COMMUNICATION CHANNELS

@CaraHarshman #MozCon

1. OPT-IN EMAIL NEWSLETTER 2. WEEKLY MEETINGS FOR CORE CREW 3. INTERNAL WIKI / WORKFLOW TOOL 4. LUNCH N’ LEARNS 5. ALL HANDS MEETINGS

Page 74: Evolve Your A/B Testing Skills

RESULTS SHARING TEMPLATES + DECKS

DOWNLOAD

OPTIMIZELY.COM/TESTING-TOOLKIT/

Page 75: Evolve Your A/B Testing Skills

IDEAS HYPOTHESES EXECUTION AUTOMATION TEAM ADVOCATES

EVOLVE YOUR A/B TESTING

@CaraHarshman #MozCon

Page 76: Evolve Your A/B Testing Skills

1. LEARN RATE = 100% 2. BUSINESS GROWTH, HIGHER REVENUE 3. DELIGHTED CUSTOMERS 4. STRONGER DECISIONS & CULTURE

OPTIMAL OUTCOME OF A/B TESTING EVOLUTION

@CaraHarshman #MozCon

Page 77: Evolve Your A/B Testing Skills

C A R A H A R S H M A N

c a r a h a r s h m a n . m e @ c a r a h a r s h m a n