a/b testing: you might just be doing it wrong
TRANSCRIPT
PRESENTED BY
+
A/B Testing You Might Just Be Doing it Wrong
+
Tomasz Borys Director of Marketing,
KISSmetrics
@tbcali
Andrew Caravella VP of Marketing,
Sprout Social
@andrewcaravella
+
Table of Contents
• The optimum strategies for A/B testing
• How we A/B Test at KISSmetrics
• A/B Testing beyond click conversions - having a pulse on the entire funnel and why it’s important
• The influence social traffic has in the funnel
+
The Optimum Strategies for A/B Testing
• Set a Goal
• Baby Steps
• Aim for Statistical Significance
• Never Lose Sight
+
Set a Goal
+
Baby Steps
People want to move the needle and see results fast, by implementing multiple changes at once.
But how can you be sure what element had an impact?
+
Aim for Statistical Significance SET A GOAL
Aimfor Statistical Significance
getdatadriven.com
+
Never Lose Sight
We can caught up on what we think is best to optimize conversions or traffic… but data doesn’t lie
+
How We A/B Test at KISSmetrics
• 4,000 and above sample size
• 99% Statistical Confidence
• Blind Eye for 1 Week
+
4,000 and Above Sample Size
significant or decent amount of data is needed when you’re testing beyond click conversions
+
4,000 and Above Sample Size
significant or decent amount of data is needed when you’re testing beyond click conversions
99% Statistical Confidence
+
Blind Eye for 1 Week
Data can be very erratic the first several days, so it’s easy to hit the panic button
+
A/B Testing Beyond Click Conversions
+
Original
+
Variant
+
Signups
+
Signups
+
Signups
+
Signup Install JS Custom Data Opps
+
Signups
Installed JS
+
Signups
Install JS
+
Signups
Install JS
+
Custom Data
+
Custom Data
+
Custom Data
+
Custom Data
Opportunities Created
+
Custom Data
Opportunities Created
+
Custom Data
Opportunities Created
+
Social Ad Funnel
