What Your Crystal Ball Isn’t Telling You Using Predictive Analytics to Turn Data Into ROISteve Beshuk

DirectorBusiness Intelligence GroupJCASteve.Beshuk@Jcainc.com

• Overview of the analytics framework

• Learn about the process for implementing predictive analytics

• Review real-life case study

Goals for Today

Predictive Analytics Framework

• Make it a project

• Involve the business folks from the beginning

• Create milestones with goal dates

• Begin preliminary data review (if necessary)

Prepare

• Define your “analytics” goal

– What is the behavior of interest?

– What are we trying to affect?

– What have we done to affect it in the past?

• Understand the current process/strategies related to the goal

• What are the current metrics?

• How will you measure success?

• Identify key stakeholders and don’t ignore communication

Understand

• How do we make more money?

• How do we grow our donor base?

• How do we increase first-year renewal rates?

Analytics Question

• Converting data from raw to richly informative

• What is it telling you?

– What are the natural groupings?

– What are the central tendencies?

• Get your data ready

– Do you need to transform it?

– Do you have all you need?

– Do you need to improve it?

– This can be time-consuming

Explore

“When you cannot express it in numbers, your knowledge is of a meager and unsatisfactory kind…” - Lord Kelvin, British Physicist

• Look at the data you are trying to predict (aka the “dependent” or “response” variable)

• Visualize it

• Establish the baseline

• Prepares you for modeling and maybe a quick win

Explore

• How did our response rates perform over time?

• What are our groups? “The Bucket”

– First-timers (dating)

– Second-timers (engaged)

– Renewers (married)

– Flip-Floppers

– Hibernators

• Assemble the Explore data dataset

Explore Example

• Bucket Report

The Bucket Report

• You have more than one

• Used the Bucket segments as context

• Time series and control charts

What Are Our Response Rates?

Acquisition Over Time

Change Over Time

Response Rates Over Time

• You have more than one – different groups and changes over time

• Is it stable?

• Basic concepts of variation

– Common cause variation: stable and random

– Special cause variation: violates the laws of probability, a single cause

• Tampering: are you really making a difference?

• For this job, you need a control chart

What’s Your Response Rate?

What’s Your Response Rate?

What is Your Response Rate?

When do Donors Lapse?

• Brainstorm the factors that affect the “behavior of interest”

• Create the analytics dataset: Select – collect – transform

• Select the modeling technique

• Build, iterate and improve

Building the Model

• Evaluate each factor and determine if it is statistically significant

• Examples from brainstorming

Are the Factors Significant?

Prior Giving (Y/N) Gift of membershipGift Amount ListTime since first gift Member LevelChannel Acknowledgement TurnaroundExpiration month ExhibitDistance to charity

Are the Factors Significant?

0.34

5

0.54

0

0.00.10.20.30.40.50.60.70.80.91.0

None(5870) All Others(521)

Like

lihoo

d to

Ren

ew

Effect of List

0.53

8

0.55

5

0.44

4

0.11

8

0.36

2

0.00.10.20.30.40.50.60.70.80.91.0

Like

lihoo

d t

o R

enew

Effect of Channel

0.315

0.441

0.384

0.00.10.20.30.40.50.60.70.80.91.0

Family(2673) Individual(675) All Others(3043)

Likeli

hood

to Re

new

Effect of Member Level

0.40

3

0.80

7

0.35

3

0.00.10.20.30.40.50.60.70.80.91.0

Likel

ihoo

d to

Rene

w

Effect of Acknowledgement Timing

Not Significant• Prior gift• Prior length• Distance• Price• Exhibits

First Pass = 44% - Find and Explain the Error

Second Pass = 66%

7.55.02.50.0-2.5-5.0

99.99

99

95

80

50

20

5

1

0.01

Standardized Residual

Perc

ent

Normal Probability Plot( e ns i A

Final Model = 85%

43210-1-2-3-4

99.99

99

95

80

50

20

5

1

0.01

Standardized Residual

Perc

ent

Normal Probability Plot(response is ADM)

• Plan the deployment

• How will you measure the effectiveness?

• Experiment

– Identify the specific change you will test

– Verify how you will segment the groups to adjust for known effects

Deploying the Model

Case StudyAnalyzing the Annual Fund

• Goal was to improve the return on the annual fund

• 60% of gifts were in response to direct appeals: direct mail, telefund, onsite sales, and renewals

• 40% of revenue to the Annual Fund was unsolicited

• Had ability to affect only 60% of gifts

• Mailing size, frequency, and spend seemed to have no impact on net revenue

• Analytics goal: control expenses to increase net

Project Profile

• Studied:

– Demographics: age, marital status, income, distance from organization

– Behavioral: past giving, loyalty, visitation, giving level, point of sale

– Environmental: exhibitions on view, average temperature

Explore

• Propensity to give

• Size of gift

• Treated each as its own analysts project

Decided on Two Models

Propensity: Longevity

Propensity: Point of Sale

Propensity: Geography

• Gender

• Age

• Marital status

• Home ownership

What Didn’t Affect Propensity

Gift Size: Longevity

Gift Size: Distance from Organization

Gift Size: Household Income

Gift Size: Home Ownership

• Gender

• Age

• Marital status

• Presence of children in household

What Didn’t Affect Size of Gift

• Some characteristics that increase propensity to give (e.g. longevity) also correlate with increased gift size…but not all of them!

• Geography and distance from organization are actually slightly different factors

• Demographic and psychographic data obtained as part of database appends are reliable indicators of both propensity and size of gift, and should therefore be updated regularly

Gift Size Takeaways

• From key learnings, built formulas that:

– Produced a unit of measurement that combined propensity to give with predicted size of gift

– Customized the formula to time of year (December and March have significantly different giving patterns)

– Produced revenue projections used calculate incremental expense; used to identify the point of maximum net

Deployment Strategy

After a year of using these analytics…

• Expenses are down by six figures thanks to significantly smaller mailings

• Yet revenue has increased

• On deck: find “on the verge” donors

Results

Questions?

?