the business of predictive modeling december 17, 2013 christine hofbeck, fsa, maaa centroid...
Post on 16-Dec-2015
215 Views
Preview:
TRANSCRIPT
The Business ofPredictive Modeling
December 17, 2013
Christine Hofbeck, FSA, MAAACentroid Analytics, LLC
AGENDA
PART I -- INTRODUCTION
PART II – MODELING 101 (Basic Steps)
PART III – “GOLDEN QUESTION”
PART IV – OPERATIONAL CONSIDERATIONS
2
INTRODUCTION
Predictive modelling [sic] is the process by which a model is created or chosen to try to best predict the probability of an outcome. -- Wikipedia
In practice:
3
VISUALIZECustomers
Most Profitable Lines
OTHERyou are only limited by
your creativity
Identify patterns/ segment
risks
Develop business
rules
Improved decision making
Potential Applications
4
OPTIMIZE Operational Efficiency Distribution ChannelsClaims Management
Pricing / Reserves
DATA
VISUALIZECustomers
Most Profitable Lines or Products
Target Marketing
OTHERyou are only limited by
your creativity
MINIMIZERisk
Fraud
Potential Applications (Life)
5
1. Triage UW decisions; implement STP for (more) applicants
2. Decrease purchase of traditional UW requirements by determining when they may not be necessary
3. Identify & target customers more likely to buy
4. Identify customers more likely to lapse – intervene if profitable, allow unhealthies to lapse
5. Inforce book management
6. Identify most desirable agents
7. Smart customer handling
Predictors
A predictive model is made up of a number of predictors (“independent variables”), which are data elements likely to influence future behavior or results (“dependent variable”).
6 SEEK PARSIMONY
DON’T USE ONE VARIABLE DON’T USE ALL VARIABLES the mean predicts the future but doesn’t tell us why…(“underfit”)
exactly replicates the past… cannot predict the future (“overfit”)
BASIC STEPS (Modeling 101)
7
Define & Scope
Data Prep
Model Build
Model Validation
Implementation
Review & Refine
Define & Scope
8
What is the budget?
Consider IT, staff, data purchase,
training, etc.
Do we have the systems capacity to
implement?
How long do we have to build? To
implement?
Insource or outsource?
How will the results be used?For whom/what
are we trying to predict this? (“unit
of exposure”)
Exactly what are we trying
to predict?
Data Prep* sometimes the most time intensive step of modeling
9
MODELING DATASET
INTERNAL DATAo # yearso accuracyo ability to accesso primary key
EXTERNAL DATAo match rateo cost – to modelo cost – to useo frequency of update
Consider both expected & unexpected relationships – creativity in data exploration can be the key
to your competitive edge!
Data Prep (cont’d)1. COMBINE various data sources2. CONVERT to desired exposure unit or format3. CORRECT inaccurate data4. INSPECT to remove variables:
- Too many blank values that cannot be imputed- All/most values the same- Data cannot be relied upon- Data will not be captured going forward- Legal advice not to use
5. BUCKET (“bin”) values10
Model Build (cont’d)
UNIVARIATE ANALYSIS – test each variable one by one to see which ones may be predictive.
11
MULTIVARIATE ANALYSIS – examine multiple variables in different groups to obtain the best, USABLE results – remember parsimony!
INTERACTIONS – which variables can be combined into a “mega variable” to improve results (i.e., does 1+1 = 1.5? does 1+1 = 3?)
Complicate the model (add variables, interactions) and simplify the model (remove variables, bin)
to find the preferred combination.
Model Build (cont’d)
Various tests can be used to determine variable inclusion:
12
STATISTICAL CONSISTENCY JUDGMENT
P-valuesCramer’s VConfidence
intervalsType III tests
Apply business knowledge to
assess whether suggested
relationships make sense
Of patterns -
Over timeOver random
parts of a dataset
Model Validation
ACTUAL vs. EXPECTED-- how close did we get?
Generally, a subset of the data is withheld during the modeling process for validation:
Model validation graphs are useful for communicating model performance to non-technical audiences.
13
OUT OF TIMEwithhold most
recent data
OUT OF SAMPLEwithhold randomly
generated % of records
Model Validation – Sample Chart
14
1 2 3 4 5 6 7 8 9 100.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
ActualExpectedO
utco
me
Decile
Implementation
BUSINESS RULESWhat decisions will be made based on the prediction?
May vary by location, business, rate group, etc.
SYSTEM BUILDScoring engine (collects data & calculates predictions)
Decision tool (executes business rules)User interface
TRAININGAnyone who will interact with the model must
understand what it does and why15
Review & RefineREPORTING
How close did we get to the goal? How far did we exceed it?
Multiple reporting packages required for varied audiences, for example:
Executives – highlights in aggregate by zone, business unit, product
Actuaries – detailed results by variable, state, rate group Marketing – by broker/agent, location Underwriting – by underwriter as a performance measure
Frequency of update – weekly, monthly, quarterly, yearly?
Method of calculation – automated? ad hoc?
Review & Refine (cont’d)MODEL UPDATES
WHY? As target customer is attained, characteristics of inforce
book will change Business goals/strategies may change New data may become available Tolerance for certain characteristics may change
HOW? Update current variable relativities (“recalibrate”) Start over - search for more predictive variables (“recast”)
HOW OFTEN?17
Advantages of Modeling Over
18
1. Many additional and often unconventional variables may be examined
2. Modeling a particular variable controls for the effects of other included variables – we don’t risk double counting or attributing effects to the wrong variables
3. Traditional approaches segment data into smaller categories which impact credibility
4. Interactions are introduced
The above advantages can lead to improved accuracy, enhanced business and strategic benefits, more reliable
assumptions, improved risk mitigation, etc.
Traditional Approaches
THE “GOLDEN QUESTION”
Through brainstorming, feedback loops, and data review, determine what single characteristic (“golden question”) will define your target
19
OPERATIONAL CONSIDERATIONS
1. Executive & cross-functional support
2. Time/cost versus depth of investigation
3. Strategic modeling process
4. Cross-functional involvement throughout build
5. Thorough training
20
Executive & Cross-Functional Support
21
If target users don’t support the model, they will resist using it.
Gaining complete support can be difficult:1. Resistance to change2. Concern that model results will highlight
current deficiencies3. Lack of understanding of predictive models
Support (cont’d)
22
We’ve always done it this way, and it’s worked.
I don’t see a reason to change
anything.
I will have to take
on additional work associated with
new processes. My workflow will
double (triple).
I don’t know how to explain this to a broker/agent so I don’t want
to use it.
I found one outlier so the
model must be wrong.
I already have an established plan. I
know who our target customer is. The model will suggest
that my current methodis incorrect, which will reflect poorly on my
performance/reputation.
My position will be eliminated if a model is now used to select risks. My expertise
must not be important to the company.
Time/Cost vs. Depth of Investigation
23
The process of building and implementing a model can typically be quite lengthy – longer than most expect
OR
Remember that a simple model does not necessarily indicate a simple study!
• Simpler Study (3-12 months)• Results more conservative• Perhaps internal data only• Generous binning• Limited interactions• May be appropriate if goal is
a general sense of direction
• More thorough investigation• Additional time• Additional development cost• Possible greater payoff
through enhanced segmentation and data exploration
Strategic Modeling ProcessTARGET PREDICTION/USEo Ensure target is appropriate for the intended useo While many ideas are interesting, you may wish to focus on those
which are actionable
STATISTICAL SIGNIFICANCE vs. ULTIMATE IMPACTo The most statistically significant model may not be the most
impactfulo Consider ease of implementation, repeatability, updateso Identify when “less is more”!
FLEXIBILITYo Allow for unexpected insights which could lead to unanticipated
changes in business strategy or processo Sometimes the insights gained from the journey will prove more
important than the planned goal
Cross-Functional InvolvementData, product & IT experts, legal advisors, and model
users must remain engaged throughout the model build
25
Insight from functions
Insight to functions
Eases training and implementation
Keep modelers apprised of changes in strategy
Legal considerations around certain variables
Thorough Training
26
The model isn’t done when it’s done.
Who will provide the training?Who is most appropriate to provide training?
Modeling teamGeneral training team
Functional expertsConsulting team*
Other
No clear answer – but this must be thoughtfully considered and appropriately executed to reap the full benefits of the model
which was built
*Consider what information may be shared (non-proprietary)
Discussion/Q&A
27
Remember…Modeling is a complete business strategy
NOT just a mathematical process
So how will YOU use predictive modeling to improve your business?
top related