a culture of testing
DESCRIPTION
In the past few years, organizations like the Obama campaign and Upworthy have demonstrated how data and analytics can be used to run hugely successful digital programs. Many of the same principles used by these groups can also be applied at smaller organizations to significantly improve the impact of online campaigns. impact of online campaigns. This training will describe the “culture of testing” that’s needed at your own organization to run a data-driven program that maximizes engagement from your supporters. Presenter: Jiim Pugh, Share Progress RELATED LINKS: http://www.shareprogress.org/ https://www.salsalabs.com/support-community/blog/six-testing-ideas-your-next-email-campaignTRANSCRIPT
CULTURE OF TESTING
MY STORY OF SELF
• Not just a set of rules or best practices
• A commitment to:– Measure the effectiveness of your choices– Make decisions using data, not intuition
• So… how do we do that?
WHAT IS A CULTURE OF TESTING?
• Metric: numerical measure
of performance
• Email metrics:– Open rate– Click rate– Action rate– Unsubscribe rate
• Web page metrics:– Conversion rate (online actions per visitor)
METRICS, METRICS, METRICS
• Q: You have two ideas for an email – which one do you choose?
• A: Both!
• Steps to run an email A/B test:– Select two randomized email groups from your list– Send email A to the first group and email B to the second– Look at email metrics to determine which email did better– Send better email to the remainder of your list
METRICS IN ACTION:EMAIL A/B TESTS
• Example test:– Email A subject line about
puppies, email B subject line about cats
– List of 100,000 emails; select 10,000 randomly for A, 10,000 randomly for B
– Send emails, measure results:• 1,276 open A and 1,403 open B• 267 donate to A, 195 donate to B
– Which do you choose?
METRICS IN ACTION:EMAIL A/B TESTS
REAL EMAIL A/B TEST EXAMPLE
71% more in contributions from Version 1
• Higher test performance doesn’t always mean better – needs to be statistically significant
• Make sure test groups large enough for significance
• CALCULATOR DEMO
STATISTICAL SIGNIFICANCE
• Emails aren’t free – lose people who unsubscribe
• Need to balance benefit of actions against cost of unsubscribes
• “Trial balloon” test:– Send email to small, randomized group– Evaluate metrics and decide if benefits outweigh costs– If yes, send to remainder; if no, don’t send
TO SEND OR NOT TO SEND:UNSUBSCRIBES & TRIAL
BALLOONS
• Like email, possible to A/B test web pages
• Web page A/B test:– Create several variations of web page– For new users coming to page, randomly show one
variation– Record if visitor takes action– Calculate conversion rate for each variation – highest
conversation rate is best
• Need enough time for statistical significance
WEB PAGE A/B TESTING
29% increasedconversion rate
WEB PAGE A/B TESTING
• Until recently, not possible to do A/B testing on social media sharing
• With ShareProgress, can now test Facebook and Twitter share posts:– Create several variations of Facebook post (title,
description, thumbnail) or Twitter text– For users sharing from your website, randomly give them
one variation– Record if users’ friends click share link and take action– Calculate recruitment rate for each variation – share post
variation most likely to drive action is best
SOCIAL A/B TESTING
SOCIAL A/B TESTING
Facebook TitleShares Driving Action / Total
Shares
Success Rate
Improvement
Monsanto shouldn’t be above the law 341/1041
32.8% (±2.9%) --
“One of the most outrageous special interest provisions in years”
322/96233.5%
(±3.0%) 2.2%
Tell the Senate: Repeal the Monsanto Protection Act
458/97247.1%
(±3.1%) 43.8%
Sign the Petition: Repeal the Monsanto Protection Act
555/1037 53.5% (±3.0%) 63.4%
Facebook headline test by CREDO Action – 63% increased recruitment rate from Version 4
• Emails and web pages
• Social sharing
• Online ads
• Even offline actions
CULTURE OF TESTING INEVERYTHING YOU DO
MAINTAINING ACULTURE OF TESTING
• Get buy-in across the organization
• Run at least one test for every campaign you do
• Schedule periodic check-ins on what you've learned and accomplished
• Email A/B testing:– Check your email
management system
• Webpage A/B testing:– Google Website Optimizer– Optimizely
• Social A/B testing:– ShareProgress
• A/B Test Calculator
HELPFUL TOOLS
QUESTIONS?