the online laboratory: perfecting your outreach through email testing
DESCRIPTION
Presentation delivered at Netroots Nation 2013 by CCAH's Damien Shirley and PFAW's Ben Betz.TRANSCRIPT
The Online Laboratory:
Perfecting Your Outreach
Through Email Testing
#NNOnlineLab
WHO ARE YOU?
• Ben Betz
– Online Strategy Manager, People For the
American Way
• Damien Shirley
– Account Executive, Chapman Cubine
Adams + Hussey
WHAT IS AN EMAIL TEST?
• Send multiple versions of an email
• Identical except for a single test variable
• To a random, representative sample of your
email list
• Measuring which version performs better
• Goal: Try a new approach and see if it works
VS.
WHY SHOULD I TEST?
• Every email list is different
• There are best practices, but they don’t always apply to every email list.
• You may know how your supporters respond to your emails, but not why
• If you’re not testing, you’re guessing
• You might be leaving something on the table
• Bottom line: Test to optimize your email engagement.
WHAT DO I NEED TO GET
STARTED?
• Email list
• Email platform (CRM)– Many have built-in testing modules
• Email messaging program
• Confidence calculator
WHAT SHOULD I TEST?
• Anything
• Test approaches that could improve
your email performance
– Let your data guide you
• Make your tests scalable
• Think outside the inbox
WHAT SHOULD I TEST: TESTING
IDEAS
• Sender
• Send time
• Send date
• Subject line
• Personalization
• Callout box position
• Callout box image
• Salutation
• Signer
• Length
• Tone
• Landing page
• Landing page
layout
HOW DO I TEST?
• Components:
– Control and test email versions
• Only difference: your test variable
• Only one test variable at a time
– Control and test audiences
• Random, representative samples of your list
– Bottom line: the only difference between
email versions should be the variable
you’re testing
HOW DO I TEST: TEST TYPES• A/B split test
– One control email variant & one test email
variant
– Most common
– Easiest to set up
Control Test
• Multivariate Test
– One control email variant & multiple test
email variants
– Still only one test variable
– Better for larger lists
Control Test 1 Test 2
HOW DO I TEST: TEST TYPES
• Pre-Launch Test
– Send your control and test variant(s) to
small segments of your email list
– Launch the winning version to the rest of
your list
– Takes more time to set up
Control TestTo Full List
HOW DO I TEST: TEST TYPES
HOW OFTEN SHOULD I TEST?
• Always
• Or, as often as you can
• Continue until you get significant
results
HOW DO I KNOW WHAT WON?
• Check your data
• Use a confidence calculator
–Determine statistical significance
–Calculate your confidence interval
HOW DO I KNOW WHAT WON?
• What if neither variant wins?
–No significant difference: go with
your preference
–Mixed results: prioritize your metrics
WHAT NOW?
• Apply your results
• Plan your next test
• Build a testing library
• Embrace failure
• You’re not done yet: retesting
– No set schedule
– Let your data guide you
WHAT TO TEST: MEMBER DRIVE
• Test idea: Test including a member
drive versus standard renewal
messaging.
• Test type: A/B split test
• Control: Standard renewal messages.
• Test: Messages included the member
drive goal for new or renewed
members.
Standard Renewal Member Drive
WHAT TO TEST: MEMBER DRIVE
• Results: – The member drive test had a higher
clickthrough rate.
– The member drive also test generated more revenue per donor.
– Using a confidence calculator, we determined that both of these results were statistically significant.
• Recommendation: – Include member drives in future renewal
campaigns.
WHAT TO TEST: MEMBER DRIVE
WHAT TO TEST: EMAIL LAYOUT
Personalized Member Invoice
vs. Standard Email Format
Invoice was the winner
WHAT TO TEST: CALLOUT BOX
IMAGE
Member Card had a higher
clickthrough rate
Thermometerhad a higher response
rate (winner)
WHAT TO TEST: SENDER
• Testing campaign-y names as sender
versus names of staff
– Stop GOP Witch Hunts vs. NAME, PFAW
– Overturning Citizens United vs. NAME,
PFAW
– PFAW Membership Director vs. NAME,
PFAW
WHAT TO TEST: MESSAGE
• Message A vs. Message B
• Roll out a message to smaller segment
of the email list to test its strength prior
to full-list launch
QUESTIONS?