Running Experiments for Online and Offline Marketing

for marketing

(a.k.a “A/B” testing)

© 2013 Steven Atkinson

This is a personal presentation. Any views or opinions represented in this presentation are personal and belong solely to the author and do not represent those of people, institutions or organizations that the author may or may not be associated with in professional or personal capacity, unless explicitly stated. Any views or opinions are not intended to malign any religion, ethnic group, club, organization, company, or individual.

All content provided on this presentation is for informational purposes only. The author of this presentation makes no representations as to the accuracy or completeness of any information on this presentation or found by following any link on this presentation. The author will not be liable for any errors or omissions in this information nor for the availability of this information. The owner will not be liable for any losses, injuries, or damages from the display or use of this information.

Overview

• what are experiments?• how can existing marketing tools inform experiments?• case studies: linkedin email tests, commerce site• but how does the math work?

– sample low-sample-size analysis• caveats vendor

Long, windy road driving with experiments…

types of experiments

• online and offline – online: website ux,

online-clickthru, ad partner testing

– offline: (e)mailings, surveys, media spots

• channel

experiment terminology• tests…• … have cells

– control or test cells• visitors, customers,

members• conversion events for the

business• business metrics

– e.g. time-based rates of events (download/day)

– facets: stuff tracked for everyone to slice by during analysis

Bonus points: Track Test Meta-Data and Resultsin a Journal / Database so you can review past experiments later.

tools for experimental marketing

• site tracking– gives site engagement, page

falloff, time spent– can help answer questions

about why A/B test cells fail, was there a systemic problem poisoning a test cell?

– can help show where your site is fighting your goals -> design new tests that change page flow for example

tools for experimental marketing

• online surveys– gives qualitative data

without hosting qual sessions

– use as pre-flight gut check for future ideas to experiment with• e.g. “what video game

system do you own”

tools for experimental marketing

• Online Marketing / Creative Experiments– channels are a great way to slice test cell

results and tests are a way to know how effective a channel is for a particular task

– send multiple creative to same partner to run in parallel

– channels must be tracked throughout visitor sessions

• When site has more traffic: Consider excluding those arriving by ONLINE or PAID SEARCH channels from your tests (they are not walk-ons / potentially different intent)

before you set up an experiment

• make your metric choices upfront– know what you are measuring for each test– have a clear hypothesis stated

• know what your conversion event is

email testing

• you are in control of volume

• measurable metrics (open, click-thru)

• Invitation emails subject to rigorous testing over 12 months

• Then new marketing folks tested the simple “Please join my professional network on LinkedIn.”

• Major increase in accepted invitations

Does this work for low-traffic?

• statistical test design– Choose 3 inputs:

• confidence level (often 95%)• #cells (start with 2)• conversion difference of interest

– (e.g. >= 1% difference interests me)

– Get back:• required volume per cell• time required for that volume to be

reached

• it’s OK to have smaller sample sizes for each cell if the observed conversion difference is big enough – so be bold in variation design for situations where sample size is smaller – stop running stupid tests

sample test for analysis

Example: Changed Home Page on Commerce Site

“conversion” == a click on a email or call link indicating interest in ordering

analysis: assuming binomial approximation to normal distribution

• Control Existing Homepage cell: – 145 visitors, 35 clicks conversion: 24% (0.24)– standard error for 95% confidence +-6.9%– range: 0.069 == 1.96*square root (0.24 * (1-0.24) / 145)

• Menu-as-Homepage cell:– 123 visitors / 49 clicks conversion: 40% (0.40)– standard error for 95% confidence +-8.6%– range: 0.086 == 1.96*square root (0.40 * (1-0.40) / 123)

• Analysis: No Confidence overlap:– 17.1%...30.9% vs 31.4%...48.6%– So Menu-as-Homepage cell is a clear winner– Moral: Even with low traffic you can get significant results with big enough

difference in conversion rate.– Moral 2: if results are not conclusive go with the variation best for long term

strategy of the business

(statistical numbers based on http://visualwebsiteoptimizer.com/split-testing-blog/using-ab-split-testing-reduce-bounce-rate-ecommerce-store/)

experiment procedure: be careful• start simple• not too many variations• wait until you have large enough sample size for

your desired conversion improvement before doing analysis

• accept that sometimes it will not be possible to give a definitive answer

• think about other factors that could affect your metrics / poison your customer– customer or visitor sees > 1 experience

• kick visitors out of test during analysis when detected

– seasonal effects– site performance

• assign lower %ages of traffic if you are worried about downside effects of testing

poisoned tests: bad performance

• Your cells may ALL fail to convert– If your site or a cell is not performant, you

will dampen all possible resultsof AB tests

• Need a fair playing field for cells– Google: Speed Matters

• Need to have SOME traffic to get results– Make sure site doesn’t fail basic perf optimizations– Run through GooglePageSpeed or Y-Slow.

summary• Perform Visitor (and later

Customer) testing• Consider Online and Offline testing• Decide on metrics/facets business

cares about• Test clear understandable

hypotheses• Perform careful analysis with

statistics• Journal what was tested and the

results• Many online tools to assist – no

need for in-house inventions.

references• Online tools

– Free tool set– http://www.testsignificance.com/

• Companies:– optimizely.com – built by original product managers from Chrome browser, members of

Obama’s “smart” online fundraising team– foresee.com – surveys built in to your site– omniture.com – site tracking (expensive)

• Reading Articles– Basic mathematics guide

• Example case study showing low traffic ok

– Advanced mathemetics slideshow– AB Testing for Executives– “Stop running stupid tests” advice