best practices in risk model development - etouches · good / bad good/ bad stay / attrite ......

Best practices in risk model development

Introducing:

Keith Tanaka Experian

Jeff Meli Experian

©Experian 3

• Market conditions

• Regulatory compliance

• More and better data

• Stronger tools

Risk modeling landscape

5/1/2017 Experian Public Vision 2017

©Experian 4

Model applications

Account acquisition

Account management

Account loss

Ris

k

man

ag

em

en

t • Front-end risk

• Prescreen risk

• Fraud

• Bankruptcy

• First payment default

• Behavioral

• CLI

• Roll rate

• Delinquency

• Transactional fraud

• Early stage collections

• Late stage collections

• Recovery

Mark

eti

ng

eff

ort

s

• Response

• Activation

• Conversion

• Purchase propensity

• Profitability

• Usage

• Activation / reactivation

• Balance transfer

• Cross-sell

• Attrition/churn

• Pre-payment

• Retention

• Front-end risk


©Experian 5

Risk modeling Drivers of success

Strong project team Preparation

Execution excellence


Seamless Best-in-class On time

©Experian 6

Model development process


Implementation Documentation Model development

Segmentation Inference Data prep Project inception

©Experian 7

Assemble project team

Project inception



Segmentation Inference Data prep

Project inception

©Experian 8

Comprehensive

Collaborative

Committed

Project inception Key participants


• Project sponsor

• Line of business

• Risk managers

• Analysts

• IT

• Compliance

• Project manager

• Third party consultants

©Experian 9

• Assemble project team

• Establish clear objectives

• Evaluate lending environment

• Define model development parameters

Project inception



Segmentation Inference Data prep

Project inception

©Experian 10

Project inception Define model development parameters


• Sample parameters

Target population

Sample timeframe

Performance flag

Performance window

Exclusion

• Data sources and attribute candidates

• Model design

Inference

Segmentation

Attribute selection

Modeling technique

Model performance measurement

• Implementation considerations

©Experian 11

Prescreen

solicitation

Account

acquisition

Credit line

authorizations

reissue

Attrition Collections

Tool Prescreen scoring New applicant

scoring Behavior scoring Behavior scoring Collection scoring

Data sources at observation

Credit bureau

demographic

Application credit

bureau customer

Master file

credit bureau

Master file

credit bureau

demographic

Master file

credit bureau

Performance window 2–3 months 12–24 months 6–12 months 3–6 months 3–6 months

Performance flags Response / non-

response profitable / unprofitable

Good / bad Good / bad Stay / attrite $ collected /

no collection

Project inception Example parameters by model application


©Experian 12

78.4 75.1 78.7 75.5

0

10

20

30

40

50

60

70

80

90

100

Auto Bankcard

Gin

i C

oeff

icie

nt

90+ DPD Performance Flag

Model 60+ DPD Model 90+ DPD

Project inception Model robustness across performance flags

77.0 74.2 76.8 74.5

0

10

20

30

40

50

60

70

80

90

100

Auto Bankcard

Gin

i C

oeff

icie

nt

60+ DPD Performance Flag

Model 60+ DPD Model 90+ DPD


©Experian 13

77.8 75.8 78.7 75.5

0

10

20

30

40

50

60

70

80

90

100

Auto Bankcard

Gin

i C

oeff

icie

nt

24-Month Performance Window

Model 12 Months Model 24 Months

Project inception Model robustness across performance windows

78.7 77.7 79.1 76.8

0

10

20

30

40

50

60

70

80

90

100

Auto Bankcard

Gin

i C

oeff

icie

nt

12-Month Performance Window

Model 12 Months Model 24 Months


©Experian 14

Project inception Define model development parameters


• Sample parameters

Target population

Sample timeframe

Performance flag

Performance window

Exclusion

• Data sources and attribute candidates

• Model design

Inference

Segmentation

Attribute selection

Modeling technique

Model performance measurement

• Implementation considerations

©Experian 15

Project inception Roadblocks and potential risks



• Lack of representation from key stakeholders

• Poor stakeholder communication and / or

competing objectives

• Weak project management is counterproductive

• Don’t move forward with design decisions

without stakeholder consensus

• Poorly designed samples may limit model shelf

life

• Consider implementation planning now

Progress impeded Potential risks

http://www.trafficsign.us/650/mark/m4-8.gif

©Experian 16

• Data extraction

• Standard data integrity checks and multi-dimensional EDA

– Portfolio reports

– Population Stability Index (PSI)

– Characteristic analysis

– Attribute treatment

Data preparation



Segmentation Inference Project inception

Data prep

©Experian 17

Data preparation Population stability

40.7

59.3

11.4

88.6

0

10

20

30

40

50

60

70

80

90

100

Missing 1+

% o

f D

istr

ibu

tio

n

Externally Sourced Attribute

Development Out of Time

PSI = 49.2

• Reveals attribute stability concerns

• Is marginal added predictive value justified?


©Experian 18

5,001 +

2,501 - 5,000

501 - 2,500

1 - 500

0

1.50 1.50 1.00

Bad Good

Good / bad index

Data preparation Characteristic analysis: Revolving balance ($) – all trades*


* Originations model

©Experian 19

Data preparation Characteristic analysis: Revolving balance ($) – one account*


* Behavioral line management model for a store credit card

2,001 +

1,001 - 2,000

201 - 1,000

1 - 200

0

1.50 1.50 1.00

Bad Good

Good / bad index

©Experian 20

• Data extraction

• Standard data integrity checks and multi-dimensional EDA

– Portfolio reports

– Population Stability Index (PSI)

– Characteristic analysis

– Attribute treatment

• Waterfall statistics

• Sample selection

Data preparation



Segmentation Inference Project inception

Data prep

©Experian 21

Data preparation Roadblocks and potential risks



• Lack of specifications for data extraction

• Insufficient attention given to EDA

• Lack of sufficient attribute vetting

• Delaying extraction and preparation

of out-of-time validation sample



©Experian 22

• Assess the need for inference

• Determine appropriate technique

• Evaluate impact on booked population performance

Inference



Segmentation Data prep Project inception

Inference

©Experian 23

Inference The need for inference


To correct for sample bias

Without inference on declined applications the scorecard would only be valid for a small proportion of the application universe

Example:

Approved applications

Declined applications

Score 640

cut-off

Modeled

population

Application

population

©Experian 24

Re-weighting

• Based on a score interval, “weight-up” accounts with known performance to the through the door distribution

Reclassification

• High risk applications, typically based on a score or high risk profile, are assigned a “bad” performance

Parceling

• Rejected applications are assigned good and bad performance based on a risk score

Bureau inference

• Using bureau data assign good and bad performance based on performance of similar trades with other creditors

Inference Inference techniques


©Experian 25

Inference Pre-diction vs. post-diction

Pre-diction and post-diction model development and assessment:

• Booked trade suppressed from post-diction attributes

• Models built on booked accounts only

• Non-booked performance assigned based on performance of a similar trade on the bureau

• Model applied to non-booked applicants

Observation

Point

(Pre-diction

Attributes)

Outcome

Point

(Post-diction

Attributes)

Outcome Window


63.9

45.4

83.5

70.2

0

10

20

30

40

50

60

70

80

90

100

Known Performance Inferred Performance

Gin

i C

oeff

icie

nt

Prediction Postdiction

©Experian 26

Inference Inferred applicant weighting and impact on booked account performance


51.93

55.76

52.64

55.61

40

45

50

55

60

Existing benchmark

BG model (100% known)

BGI model 60/40 known/inf)

BGI model (85/15 known/inf)

Model

Gin

i C

oeff

icie

nt

©Experian 27

Inference Roadblocks and potential risks



• Gaps in sample design and data

preparation

• Overcoming the inference design

• Too little inference may increase

risk exposure

• Too much inference compromises

performance of traditional book

of business



©Experian 28

• Determine candidate schemes

• Build master model

• Build out niche models for each candidate scheme

• Compare master vs. niche model performance

• Assess trade-offs of candidate schemes

Segmentation



Inference Data prep Project inception

Segmentation

©Experian 29

Segmentation Segment evaluation


Gin

i C

oe

ffic

ien

t

0

20

40

50

60

Segment 1 Segment 2 Segment 3 Overall

Niche

Master

10

30

70

90

80

©Experian 30

Segmentation Roadblocks and potential risks



• Lack of attribute vetting during project initiation

• Potential segment schemes omitted

from consideration

• Segmentation overkill

• Failure to review simple statistics to help

eliminate non-viable segments



©Experian 31

• Finalize attribute selection and modeling approach

• Develop preliminary models

• Conduct stakeholder review, and test alternate attributes

• Refine and finalize models

• For each iteration

– Test model vs. benchmarks

– Conduct in- and out-of-time validations

Model development


Implementation Documentation Segmentation Inference Data prep Project inception

Model development

©Experian 32

Model development Roadblocks and potential risks



• Lack of stakeholder sign-off on

all prior phases

• Insufficient stratification of estimation

and in-sample validation samples

during data preparation

• Insufficient attribute vetting

• ‘Science’ only gets you so

far- expert intuition is key

• Delaying out-of-time validations



©Experian 33

• Primary focus on compilation rather than construction

• Compliance and governance

• Document key decisions alongside the rationale

Documentation


Model development

Segmentation Inference Data prep Project inception Implementation

Documentation

©Experian 34

Documentation Roadblocks and potential risks



• Decisions made that were

not recorded

• Insufficient documentation

from prior phases

• Do not delay!

• Avoid large gaps –

end-to-end documents

• Ensure document is tailored

to complete audience



©Experian 35

• Coding / testing / auditing

• Strategy design

• Compliance and governance

• Monitoring

Implementation


Documentation Model development


Implementation

©Experian 36

Implementation Roadblocks and potential risks



• Unrealistic implementation timeframe

• Key participants not consulted

• System limitations not explored

• Use scorecard for designed purpose

• Monitor regularly

• Do not neglect model maintenance



©Experian 37

37

• Chose the right partners for the journey

• Use the right fuel

• Plan your route and your milestones

• Look ahead and stay focused

• Maintain a travel log

• Be prepared for the unexpected

Rules of the road




©Experian 40

Share your thoughts about Vision 2017!


Please take the time now to give us your feedback about this session.

You can complete the survey at the kiosk outside.

How would you rate both the Speaker and Content?

best practices in risk model development - etouches · good / bad good/ bad stay / attrite ......

Documents