A/B Testing on the Obama Campaign
Winning With Data
Winning with A/B Testing
What impact can testing have?
Testing = constant improvement Little improvements add up
Improving 1% here and 2% there isn’t a lot at first, but over time it adds up
Tests upon tests upon tests
Subject & draft tests Full-list tests Background personalization tests
Review: Every piece of communication was an opportunity to test
A single email often had many tests attached
Lessons
Click icon to add picture
Don’t Trust Your Gut
Lesson #1
Don’t trust your gut
We don’t have all the answers Conventional wisdom is often wrong Long-held best practices are often wrong You are not your audience
There was this thing called the Email Derby… If even the experts are bad at predicting a
winning message, it shows just how important testing is.
Foster a culture of testing
Lesson #2
The culture of testing
Check your ego at the door
Use every opportunity to test something
Compare against yourself, not against your competitors or “the industry” Are you doing better this month than last
month? Are you doing better than you would have
otherwise?
Use data to make the user experience more personal
Lesson #3
Big data ≠ big brother
Testing allows you to listen to your user base Let them tell you what they like Whether through A/B testing or behavioral
segmentation, optimization gives them a better experience
Usually, the interactions that are the most human are the ones that win
Be human!
In general, we founds shorter, less formal emails and subject lines did best. Classic example: “Hey”
When we dropped a mild curse word into a subject line, it usually won “Hell yes, I like Obamacare” “Let’s win the damn election” “Pretty damn cool”
Good segmentation: behavioral Behavioral segmentation was much more
effective than demographic segmentation Donor vs. non-donor High-dollar vs. low-dollar Volunteer status What issues do people say they care about?
After using A/B tests to create a winning message, we could tweak it slightly for various behavioral groups and get better results
Conclusions
Conclusions
Test everything, especially your gut instinct
Foster a culture of testing
Use data to make it personal