a/b testing - what your mother didn't tell you
DESCRIPTION
Many people think A/B testing is merely showing customers two versions of something and choosing the one that customers respond to better. It's much, much more complex than that.TRANSCRIPT
![Page 1: A/B Testing - What your mother didn't tell you](https://reader033.vdocuments.us/reader033/viewer/2022052618/554f5f3fb4c905c8088b48c5/html5/thumbnails/1.jpg)
P.O.P. Copyright 2013, All Around The World January 14, 2014
![Page 2: A/B Testing - What your mother didn't tell you](https://reader033.vdocuments.us/reader033/viewer/2022052618/554f5f3fb4c905c8088b48c5/html5/thumbnails/2.jpg)
A/B TesAng
What your mother didn’t tell you hDp://www.allaroundtheworld.fr/
January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
![Page 3: A/B Testing - What your mother didn't tell you](https://reader033.vdocuments.us/reader033/viewer/2022052618/554f5f3fb4c905c8088b48c5/html5/thumbnails/3.jpg)
What is A/B TesAng?
• TesAng customers, not code • Show different versions • Pick the best one
January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
![Page 4: A/B Testing - What your mother didn't tell you](https://reader033.vdocuments.us/reader033/viewer/2022052618/554f5f3fb4c905c8088b48c5/html5/thumbnails/4.jpg)
Who Relies On A/B TesAng?
January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
![Page 5: A/B Testing - What your mother didn't tell you](https://reader033.vdocuments.us/reader033/viewer/2022052618/554f5f3fb4c905c8088b48c5/html5/thumbnails/5.jpg)
Transform Your Company
• Test your customers, not your code • Stop guessing what works
January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
![Page 6: A/B Testing - What your mother didn't tell you](https://reader033.vdocuments.us/reader033/viewer/2022052618/554f5f3fb4c905c8088b48c5/html5/thumbnails/6.jpg)
January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
Should your paycheck depend on my hunch?
![Page 7: A/B Testing - What your mother didn't tell you](https://reader033.vdocuments.us/reader033/viewer/2022052618/554f5f3fb4c905c8088b48c5/html5/thumbnails/7.jpg)
Conversion Funnel
January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
![Page 8: A/B Testing - What your mother didn't tell you](https://reader033.vdocuments.us/reader033/viewer/2022052618/554f5f3fb4c905c8088b48c5/html5/thumbnails/8.jpg)
Conversion Funnel
January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
→
![Page 9: A/B Testing - What your mother didn't tell you](https://reader033.vdocuments.us/reader033/viewer/2022052618/554f5f3fb4c905c8088b48c5/html5/thumbnails/9.jpg)
A Basic Test
• Your current behavior is the base • A variant is a new version (just one change) • You can have one or more variants
• But we assume only one for this talk
January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
![Page 10: A/B Testing - What your mother didn't tell you](https://reader033.vdocuments.us/reader033/viewer/2022052618/554f5f3fb4c905c8088b48c5/html5/thumbnails/10.jpg)
January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
![Page 11: A/B Testing - What your mother didn't tell you](https://reader033.vdocuments.us/reader033/viewer/2022052618/554f5f3fb4c905c8088b48c5/html5/thumbnails/11.jpg)
Which Generates More Clicks?
January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
![Page 12: A/B Testing - What your mother didn't tell you](https://reader033.vdocuments.us/reader033/viewer/2022052618/554f5f3fb4c905c8088b48c5/html5/thumbnails/12.jpg)
Which Generates More Clicks?
January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
10,017 Visits 9,983 Visits
125 Conversions 132 Conversions
![Page 13: A/B Testing - What your mother didn't tell you](https://reader033.vdocuments.us/reader033/viewer/2022052618/554f5f3fb4c905c8088b48c5/html5/thumbnails/13.jpg)
Which Generates More Clicks?
January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
9,950 Visits 10,550 Visits
125 Conversions 155 Conversions
![Page 14: A/B Testing - What your mother didn't tell you](https://reader033.vdocuments.us/reader033/viewer/2022052618/554f5f3fb4c905c8088b48c5/html5/thumbnails/14.jpg)
The Numbers (95% Confidence)
January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
hDp://getdelve.com/2011/12/math-‐behind-‐ab-‐tesAng-‐for-‐curious-‐execuAves-‐part-‐1/
Version Visits Conversion Rate Not Random
Red 10,017 125 1.25% —
Blue 9,983 132 1.32% 51%
![Page 15: A/B Testing - What your mother didn't tell you](https://reader033.vdocuments.us/reader033/viewer/2022052618/554f5f3fb4c905c8088b48c5/html5/thumbnails/15.jpg)
The Numbers (95% Confidence)
January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
hDp://getdelve.com/2011/12/math-‐behind-‐ab-‐tesAng-‐for-‐curious-‐execuAves-‐part-‐1/
Version Visits Conversion Rate Not Random
Red 9,950 125 1.26% —
Blue 10,550 155 1.47% 94%
![Page 16: A/B Testing - What your mother didn't tell you](https://reader033.vdocuments.us/reader033/viewer/2022052618/554f5f3fb4c905c8088b48c5/html5/thumbnails/16.jpg)
Which Sells More Product?
January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
![Page 17: A/B Testing - What your mother didn't tell you](https://reader033.vdocuments.us/reader033/viewer/2022052618/554f5f3fb4c905c8088b48c5/html5/thumbnails/17.jpg)
Which Creates More Profit?
January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
![Page 18: A/B Testing - What your mother didn't tell you](https://reader033.vdocuments.us/reader033/viewer/2022052618/554f5f3fb4c905c8088b48c5/html5/thumbnails/18.jpg)
What Confidence Level?
Is 99% beDer than 90%?
January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
![Page 19: A/B Testing - What your mother didn't tell you](https://reader033.vdocuments.us/reader033/viewer/2022052618/554f5f3fb4c905c8088b48c5/html5/thumbnails/19.jpg)
January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
The Cycle
![Page 20: A/B Testing - What your mother didn't tell you](https://reader033.vdocuments.us/reader033/viewer/2022052618/554f5f3fb4c905c8088b48c5/html5/thumbnails/20.jpg)
Create Experiment
Only one difference per version!
January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
![Page 21: A/B Testing - What your mother didn't tell you](https://reader033.vdocuments.us/reader033/viewer/2022052618/554f5f3fb4c905c8088b48c5/html5/thumbnails/21.jpg)
Roll the Dice
January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
![Page 22: A/B Testing - What your mother didn't tell you](https://reader033.vdocuments.us/reader033/viewer/2022052618/554f5f3fb4c905c8088b48c5/html5/thumbnails/22.jpg)
Track Performance
Sta:s:cs
January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
![Page 23: A/B Testing - What your mother didn't tell you](https://reader033.vdocuments.us/reader033/viewer/2022052618/554f5f3fb4c905c8088b48c5/html5/thumbnails/23.jpg)
Choose the Best
Pa:ence
January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
![Page 24: A/B Testing - What your mother didn't tell you](https://reader033.vdocuments.us/reader033/viewer/2022052618/554f5f3fb4c905c8088b48c5/html5/thumbnails/24.jpg)
Create Experiment
Wash, rinse, repeat
January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
![Page 25: A/B Testing - What your mother didn't tell you](https://reader033.vdocuments.us/reader033/viewer/2022052618/554f5f3fb4c905c8088b48c5/html5/thumbnails/25.jpg)
Choose the Best
• Ignore the first day • Full-‐business cycle • What if you’re wrong?
• False posiAves
hDp://www.einarsen.no/is-‐your-‐ab-‐tesAng-‐effort-‐just-‐chasing-‐staAsAcal-‐ghosts/
January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
![Page 26: A/B Testing - What your mother didn't tell you](https://reader033.vdocuments.us/reader033/viewer/2022052618/554f5f3fb4c905c8088b48c5/html5/thumbnails/26.jpg)
LimitaAons of A/B TesAng
• Doesn’t tell you why • Side effects • Doesn’t tell you loyalty
January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
![Page 27: A/B Testing - What your mother didn't tell you](https://reader033.vdocuments.us/reader033/viewer/2022052618/554f5f3fb4c905c8088b48c5/html5/thumbnails/27.jpg)
Benefits
• Hyundai: 62% increase in test drive requests • Artsy Editor: 47% increase in click-‐throughs • Visual Website OpAmizer: 50% signup increase • AssessmentDay: 62% increase in sales • SmileyCookie: 41% increase in sales • BaseKit: 25% increase in conversions • Mobal: 27% increase in sales • You Need A Budget: 85% increase in downloads
hDp://visualwebsiteopAmizer.com/case-‐studies.php
![Page 28: A/B Testing - What your mother didn't tell you](https://reader033.vdocuments.us/reader033/viewer/2022052618/554f5f3fb4c905c8088b48c5/html5/thumbnails/28.jpg)
What Are They TesAng?
• Add-‐to-‐cart buDon placement • Call to acAon buDons • Using popup forms • Removing content • Link color • Adding tesAmonials • NavigaAon menus • Removing signup forms
January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
![Page 29: A/B Testing - What your mother didn't tell you](https://reader033.vdocuments.us/reader033/viewer/2022052618/554f5f3fb4c905c8088b48c5/html5/thumbnails/29.jpg)
Gemng Started
• Vendor or in-‐house? • Go for quick wins • Run an AA test • Be pa:ent!
January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
![Page 30: A/B Testing - What your mother didn't tell you](https://reader033.vdocuments.us/reader033/viewer/2022052618/554f5f3fb4c905c8088b48c5/html5/thumbnails/30.jpg)
Problems
Ego January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
![Page 31: A/B Testing - What your mother didn't tell you](https://reader033.vdocuments.us/reader033/viewer/2022052618/554f5f3fb4c905c8088b48c5/html5/thumbnails/31.jpg)
Problems
• Forgemng caching • Non-‐random selecAon
• Forgemng selecAon
• TesAng only the view • Performance
• Dead code • Profit versus conversion
January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
![Page 32: A/B Testing - What your mother didn't tell you](https://reader033.vdocuments.us/reader033/viewer/2022052618/554f5f3fb4c905c8088b48c5/html5/thumbnails/32.jpg)
Tips
• Most experiments fail • Don’t roll the dice prematurely
• A/B test bugs • Test “stupid” ideas • Don’t require rollouts!
January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
![Page 33: A/B Testing - What your mother didn't tell you](https://reader033.vdocuments.us/reader033/viewer/2022052618/554f5f3fb4c905c8088b48c5/html5/thumbnails/33.jpg)
SegmentaAon
• New versus returning customers • Different browsers • Accept-‐Language • Time of day
• IP country • Gender? Age? Something else?
January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
![Page 34: A/B Testing - What your mother didn't tell you](https://reader033.vdocuments.us/reader033/viewer/2022052618/554f5f3fb4c905c8088b48c5/html5/thumbnails/34.jpg)
See Also
• The MulA-‐Armed Bandit hDp://stevehanov.ca/blog/index.php?id=132
January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
![Page 35: A/B Testing - What your mother didn't tell you](https://reader033.vdocuments.us/reader033/viewer/2022052618/554f5f3fb4c905c8088b48c5/html5/thumbnails/35.jpg)
Conclusion
• Lots of money lying on the table • Egos are bad • Data is good • Customer behavior trumps expert opinion
• Customer behavior trumps code behavior
January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
![Page 36: A/B Testing - What your mother didn't tell you](https://reader033.vdocuments.us/reader033/viewer/2022052618/554f5f3fb4c905c8088b48c5/html5/thumbnails/36.jpg)
QuesAons?
hDp://www.allaroundtheworld.fr/
January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/