What Your Crystal Ball Isn’t Telling You Using Predictive Analytics to Turn Data Into ROISteve Beshuk
DirectorBusiness Intelligence [email protected]
• Overview of the analytics framework
• Learn about the process for implementing predictive analytics
• Review real-life case study
Goals for Today
Predictive Analytics Framework
• Make it a project
• Involve the business folks from the beginning
• Create milestones with goal dates
• Begin preliminary data review (if necessary)
Prepare
• Define your “analytics” goal
– What is the behavior of interest?
– What are we trying to affect?
– What have we done to affect it in the past?
• Understand the current process/strategies related to the goal
• What are the current metrics?
• How will you measure success?
• Identify key stakeholders and don’t ignore communication
Understand
• How do we make more money?
• How do we grow our donor base?
• How do we increase first-year renewal rates?
Analytics Question
• Converting data from raw to richly informative
• What is it telling you?
– What are the natural groupings?
– What are the central tendencies?
• Get your data ready
– Do you need to transform it?
– Do you have all you need?
– Do you need to improve it?
– This can be time-consuming
Explore
“When you cannot express it in numbers, your knowledge is of a meager and unsatisfactory kind…” - Lord Kelvin, British Physicist
• Look at the data you are trying to predict (aka the “dependent” or “response” variable)
• Visualize it
• Establish the baseline
• Prepares you for modeling and maybe a quick win
Explore
• How did our response rates perform over time?
• What are our groups? “The Bucket”
– First-timers (dating)
– Second-timers (engaged)
– Renewers (married)
– Flip-Floppers
– Hibernators
• Assemble the Explore data dataset
Explore Example
• Bucket Report
The Bucket Report
• You have more than one
• Used the Bucket segments as context
• Time series and control charts
What Are Our Response Rates?
Acquisition Over Time
Change Over Time
Response Rates Over Time
• You have more than one – different groups and changes over time
• Is it stable?
• Basic concepts of variation
– Common cause variation: stable and random
– Special cause variation: violates the laws of probability, a single cause
• Tampering: are you really making a difference?
• For this job, you need a control chart
What’s Your Response Rate?
What’s Your Response Rate?
What is Your Response Rate?
When do Donors Lapse?
• Brainstorm the factors that affect the “behavior of interest”
• Create the analytics dataset: Select – collect – transform
• Select the modeling technique
• Build, iterate and improve
Building the Model
• Evaluate each factor and determine if it is statistically significant
• Examples from brainstorming
Are the Factors Significant?
Prior Giving (Y/N) Gift of membershipGift Amount ListTime since first gift Member LevelChannel Acknowledgement TurnaroundExpiration month ExhibitDistance to charity
Are the Factors Significant?
0.34
5
0.54
0
0.00.10.20.30.40.50.60.70.80.91.0
None(5870) All Others(521)
Like
lihoo
d to
Ren
ew
Effect of List
0.53
8
0.55
5
0.44
4
0.11
8
0.36
2
0.00.10.20.30.40.50.60.70.80.91.0
Like
lihoo
d t
o R
enew
Effect of Channel
0.315
0.441
0.384
0.00.10.20.30.40.50.60.70.80.91.0
Family(2673) Individual(675) All Others(3043)
Likeli
hood
to Re
new
Effect of Member Level
0.40
3
0.80
7
0.35
3
0.00.10.20.30.40.50.60.70.80.91.0
Likel
ihoo
d to
Rene
w
Effect of Acknowledgement Timing
Not Significant• Prior gift• Prior length• Distance• Price• Exhibits
First Pass = 44% - Find and Explain the Error
Second Pass = 66%
7.55.02.50.0-2.5-5.0
99.99
99
95
80
50
20
5
1
0.01
Standardized Residual
Perc
ent
Normal Probability Plot( e ns i A
Final Model = 85%
43210-1-2-3-4
99.99
99
95
80
50
20
5
1
0.01
Standardized Residual
Perc
ent
Normal Probability Plot(response is ADM)
• Plan the deployment
• How will you measure the effectiveness?
• Experiment
– Identify the specific change you will test
– Verify how you will segment the groups to adjust for known effects
Deploying the Model
Case StudyAnalyzing the Annual Fund
• Goal was to improve the return on the annual fund
• 60% of gifts were in response to direct appeals: direct mail, telefund, onsite sales, and renewals
• 40% of revenue to the Annual Fund was unsolicited
• Had ability to affect only 60% of gifts
• Mailing size, frequency, and spend seemed to have no impact on net revenue
• Analytics goal: control expenses to increase net
Project Profile
• Studied:
– Demographics: age, marital status, income, distance from organization
– Behavioral: past giving, loyalty, visitation, giving level, point of sale
– Environmental: exhibitions on view, average temperature
Explore
• Propensity to give
• Size of gift
• Treated each as its own analysts project
Decided on Two Models
Propensity: Longevity
Propensity: Point of Sale
Propensity: Geography
• Gender
• Age
• Marital status
• Home ownership
What Didn’t Affect Propensity
Gift Size: Longevity
Gift Size: Distance from Organization
Gift Size: Household Income
Gift Size: Home Ownership
• Gender
• Age
• Marital status
• Presence of children in household
What Didn’t Affect Size of Gift
• Some characteristics that increase propensity to give (e.g. longevity) also correlate with increased gift size…but not all of them!
• Geography and distance from organization are actually slightly different factors
• Demographic and psychographic data obtained as part of database appends are reliable indicators of both propensity and size of gift, and should therefore be updated regularly
Gift Size Takeaways
• From key learnings, built formulas that:
– Produced a unit of measurement that combined propensity to give with predicted size of gift
– Customized the formula to time of year (December and March have significantly different giving patterns)
– Produced revenue projections used calculate incremental expense; used to identify the point of maximum net
Deployment Strategy
After a year of using these analytics…
• Expenses are down by six figures thanks to significantly smaller mailings
• Yet revenue has increased
• On deck: find “on the verge” donors
Results
Questions?
?