5 Steps to Your First A/B Testing Win

Sean Ellis, CEO, Qualaroo (@seanellis)Sean Ellis is the CEO of Qualaroo and founder of GrowthHackers.com. Prior to Qualaroo, Sean held marketing leadership roles with breakout companies including Dropbox, LogMeIn (IPO), Uproar (IPO), Eventbrite and Lookout.

Ryan Lillis, Strategic Optimization Consultant, Optimizely

(@Optimizely)Ryan Lillis is a web conversion optimization specialist who helps Optimizely clients get the most out of their A/B, multivariate testing and targeting campaigns. He has worked in the website optimization space for eight years and has worked with companies as small as 5 people all the way up to Fortune 100 companies.

#winningtests

Why Spend Time on A/B Testing?

$92 dollars spent on acquisition for every $1 on optimization

87% of companies (currently testing) run 1-5 tests per month

Best companies in the world run hundreds per month

Source: Econsultancy & Redeye Conversion Rate Optimization Report 2014

#winningtests

Optimization is a Process

QuantitativeInsights

QualitativeInsightsTesting

#winningtests

Understanding Conversion Optimization Goals

image via: http://www.shayhowe.com/web-design/defining-multivariate-testing/

Brainstorm Test Ideas Make a Small Change See Mixed

or No Results

Typical Testing Process

#winningtests

Ideal Testing Process

User Research Lots of Testing Ideas

QuantitativeInsights

QualitativeInsightsTesting

Continuous Process of Learning

#winningtests

Guessing is Expensive

Top 5 highest bounce rate pages

Top 5 abandonment points in your funnel

Top 5 most valuable pages to your business

#1: Find Your Best Opportunities

#winningtests

1. Look at top landing pages

2. Identify those with high bounce rates

3. Conduct research on those pages

4. Develop tests to improve page performance

Image via: SmartInsights.com

Example: High Bounce Rates

#winningtests

#2: Understand Visitor Needs

Ask and Answer Four Key Questions:!

Why did they come to your website?

What stopped them from converting?

Did they find what they were looking for?

If they did convert, what almost stopped them?

#winningtests

Example: Post Conversion Surveys

#winningtests

Use Data & Insights to Drive Testing

Start small to get a win and validation for process

Test for impact, avoid ‘Meek Tweaking’

Use test result data to inform the next test

#3: Use Data & Insights to Inform Testing

#winningtests

Image via Conversion Rate Experts

Meek Tweaking

#winningtests

Running Your First Tests

Testing Ideas to Start:!

4 Message Tests

4 "Aha Moment" Tests

2 Large Scale Design Tests

#4: Where to Start?

#winningtests

Measure impact through the entire funnel, not just at the page level.

Bigger changes yield faster results.

Different levels of statistical significance will speed up or slow down decision making.

#5: Measuring Success

#winningtests

Putting It All Together

QuantitativeInsights

QualitativeInsightsTesting

Optimization is a Process

#winningtests

1. Plan and run your first five to ten tests.

2. Get your first win.

3. Start the cycle of continuous improvement.

Next Steps

#winningtests

qualaroo.com/beginners-guide-to-cro/

Resources

optimize.ly/datadna

#winningtests

Q & A

#winningtests