10 tips to improve testing
TRANSCRIPT
10 Tips for Better Testing
2
About Us
Jen Boland
Rosa Del Angel
Data Analyst@jenboland
Senior Marketing Consultant
3
About Beaconfire RedEngine
Integrated Fundraising & Marketing
Web Design & Development
Strategy, Testing & Analytics
Acquisition, Cultivation
and Retention
Team of 40+ fundraising, marketing, UX and technology experts focused on optimizing clients’ digital programs.
#1 Run an A/A Test
4Confidential - Presented to [Client Name] - ©2014
#2 Statistical Significance
• Make sure your results are statistically significant.
Sample Size Calculator
“A minimum detectable effect (MDE) is the smallest true effect that has a ‘good chance’ of being found to be statistically significant.”
Minimum Detectable Effect
Sorry! – You’re not Amazon
Minimum Detectable Effect
10
11
12
Test across thank you pages and emailsC: Honor Gift
B: Sustainer w/ Premium
A:Sustainer
“ I hate math. But I love counting my money.”
- unknown
#3 Use the right stats
Online Calculator
Statistical Significant Gift Array Test Similar conversion rate, but
55% higher
avg. gift and 13%
more revenue
T-test
Using Excel to Conduct t-Testt-Test: Two-Sample Assuming Unequal Variances
Variable 1 Variable 2Mean 96.69892 113.956Variance 15711.8 70387.51Observations 93 91Hypothesized Mean Difference 10df 128t Stat -0.88788P(T<=t) one-tail 0.188135t Critical one-tail 1.656845P(T<=t) two-tail 0.37627t Critical two-tail 1.978671
Results are significant if p < .05.
“You Need a Hypothesis. Period.”
#4 Hypothesis
18Confidential - Presented to [Client Name] - ©2014
Test against your control
2X Conversio
n Rate22% More Revenue
CONTROL VARIANT
20
Hypothesis: By improving the user experience and integrating the memorial and honor giving options into one donation form, we can improve conversion.Control Test
Hypothesis: Sending paid media traffic straight to donation form will generate more conversions, more revenue
Visitors sent to the splash page were 47% more likely to donate
and had an average gift $69
higher than visitors sent
straight to the donation form
22
Keep variables the same7% Higher Revenue 11% Higher Avg. Gift
23
#5 Segment your results
New Users vs Returning
Mobile vs Desktop
Donors vs Non Donors
Acquisition vs Cultivation
#6 Validate Your Results
Keep an eye on your conversion rates.
Make sure your leads convert
27
• Global Navigation Menu
• Carousel Banners and Sliders
• Stories Strip• Header and
Footer Navigation• Donate Page and
Quick Donate boxes
• CTA buttons within content
#7 Advanced Analytics
28
HRW form validation tracking
29
Become an EX registration form
30
AfterReduced error rate by
28%
#8 Learn from your failures
32
Winning Search Text Ad Facebook Ad
#9 Use test results to inform other channels
Nearly 30 Million Battle Diabetes. Your Donation Will Improve Lives. Donate Today.
33
#10 Test across channels
Supporters who received integrated emails turned out to be three times more likely to donate compared to those who did not. Also, a 5% lift was seen in offline giving for people who received the integrated emails.
Connecting Offline & Online Audiences
• Special Olympics EOY Facebook Custom Audience Test • Facebook CRM ads targeting direct mail donors, lapsed donors
and non-donors during EOY matching gift campaign • Test Group A featured donors who received a Matching Gift direct
mail package AND were served Facebook ads• Test Group B included lapsed donors and non-donors who
received a non-Matching Gift direct mail piece AND were served Facebook ads
• Match rate from Facebook was 50%
• DM lapsed and non-donors who saw FB ads (test) and who did not see ads (
Facebook EOY CRM Retargeting
• Facebook CRM ads targeting direct mail donors, lapsed donors and non-donors
• Test Group A featured donors who received a Matching Gift direct mail package AND were served Facebook ads
• Test Group B included lapsed donors and non-donors who received a non-Matching Gift direct mail piece AND were served Facebook ads
• 2 Control groups (donor and non) who did not see Facebook ads
Facebook EOY CRM Retargeting• For donors, FB ads generated 153% ROI
with online gifts, plus small avg gift increase in mail
• For non-donors, FB ads lifted avg gift in mail by nearly 20% and generated 21% more revenue than DM group who did not see ads. For every $1 spent online, we raised $8.50 in the mail.
• Match rate from Facebook was 50%
37
1. Run an A/A test2. Make sure your test results are statistically
significant3. Use the right stats4. You need a hypothesis5. Segment your results6. Validate your results7. Use Advanced Analytics8. Learn from your failures9. Use test results to inform other channels10.Test across channels
Recap
38
Questions?www.beaconfire.com | @beaconfirered
Jen Boland
Rosa Del Angel
Data [email protected]@Beaconfire-red.com
Senior Marketing [email protected]@Beaconfire-red.com