proven practices for predictive · pdf fileproven practices for predictive modeling...

PROVEN PRACTICES FOR PREDICTIVE MODELING

MARY-ELIZABETH (“M-E”) EDDLESTONEPRINCIPAL SYSTEMS ENGINEER, ANALYTICS

CONTRIBUTIONS FROM:DARIUS BAERDAVID OGDENDOUG WIELENGA

BROUGHT TO YOU BY SAS CUSTOMER LOYALTY

BEST PRACTICES DISCLAIMERS

• The choice of “Best Practices” is highly subjective.

• Certain suggested practices may not be suitable for a particular situation.

• It is the responsibility of a data mining practitioner to critically evaluate methods and select the best method for a particular situation.

• This presentation represents the opinions of the contributors.

BEST PRACTICES TO HELP YOU MEET AND EXCEED YOUR GOALS

Faster model developmentMore useful modelsSuperior models

BEST PRACTICES FORMAT OF PRESENTATION

• Background & General Guidance• Developing the Data• Developing & Delivering the Model

BACKGROUND &GENERAL GUIDANCEANALYTICS CYCLE AND THE MODELING PROCESS

ANALYTICS LIFECYCLE

Formulate Problem

Data Preparation

Data Exploration

Transform & Select

Develop Models

Validate Models

Deploy Model

Evaluate & Monitor

BEST PRACTICE IT’S ALL ABOUT BALANCE

• Many factors need to be considered and optimized:• Time• People• Money• IT Resources

People

TechnologyBusiness Process

• BI reporting• Web portals /

dashboards• Information

management• Problem-specific

business solutions• Predictive analytics• Hardware

ANALYTICS INFRASTRUCTURE

TECHNOLOGY

BUSINESS PROCESS

DECISION ANALYTICS

Fact-based decision making requires the right technology, talent, processes and culture

• Continuous Process Improvement

• Planning• Project

methodology• Standards

• Vision & Leadership

• Team composition

• Enterprise authorityPEOPLE

Domain ExpertMakes DecisionsEvaluates Processes & ROI

BUSINESS MANAGER

Data ExplorationData VisualizationReport Creation

BUSINESS ANALYST

Exploratory AnalysisDescriptive SegmentationPredictive ModelingModel Validation & Registration

DATA MINER

Model ValidationModel DeploymentModel MonitoringData Preparation

IT/SYSTEMS MANAGEMENT

Formulate Problem

Data Preparation

Data Exploration

Transform & Select

Develop Models

Validate Models

Deploy Model

Evaluate & Monitor

LIFECYCLE BEST PRACTICE

INVOLVE ALL THE RELEVANT PEOPLE/ROLES

BEST PRACTICE WEAR MANY HATS

Have a passion to understand not just

analytics, but the business and

technology.

BEST PRACTICE USE THE TECHNOLOGY AND METHOD THE FITS THE JOB

Every tool and method has advantages and disadvantages.Whenever possible, select the tool or method that balances long-term goals for the entire process.

BEST PRACTICE BEGIN WITH THE END IN MIND

•What?•How?•Who?•When?

• What is the overarching strategic objective/initiative?

• How will the model be used?• How will it be put into production?

• Who will be affected by the use of the model?

• Who needs to be convinced of the value of the model?

• When will the model be used?

BEST PRACTICES BUSINESS CONSIDERATIONS BEFORE YOU MODEL

• Thoroughly understand the business/marketing objectives• Detail the precise (planned) usage for the output• Define the target variable (the outcome being modeled /

predicted)• Formulate a theoretical model: Y = f (X1, X2, …) fill-in

the likely X’s

BEST PRACTICE

SEMMA Process for Model

Development

BEST PRACTICE MODELING APPROACH

1. Sample training set(s), validation set(s), holdout test set

2. Explore min, max, mean, median, missing values, levels (categorical cardinality)

3. Modify filtering outliers, reducing cardinality, correcting multicolinearity, imputations, non-linear transformations

BEST PRACTICE MODELING APPROACH

4. Model variable selection, various model formulations, iterative cycle, insights & client reviews

5. Assess performance criteria and review

BEST PRACTICE MODELING APPROACH (CONTINUED)

7. Final Assessment & Testing8. Profile characteristics & indicators9. Document results10. Prepare (production-ready) data collection and

score code11. Monitor model performance

DEVELOPING THE DATA

BEST PRACTICES OPTIMIZING DATA

Determining DataSelecting TargetPreparing Variables

DETERMINING DATA

BEST PRACTICES TECHNICAL CONSIDERATIONS BEFORE MODELING

• Brainstorm all potential input data elements• Identify source systems, specific data fields, availability/priority/level-of-effort of data

• Finalize data to be collected

BEST PRACTICES TECHNICAL CONSIDERATIONS BEFORE MODELING

• Formulate structure and layout of modeling dataset to be built

• Devil-in-the-details: filters, timeframe of history, etc…

• Build modeling dataset

BEST PRACTICE ALLOW SUFFICIENT TIME FOR ALL ASPECTS

BEST PRACTICE

SEMMA Process for Model

Development

SAMPLE

• (Over) Sampling• Decisioning• Partitioning

SAMPLING

SAMPLE TO SAMPLE OR NOT?

• Sampling is a valuable tool that can be used to great effect.

• If computing resources are no object, it’s possible to use all data.

• When resource constrained, try increasing sample sizes as model development progresses.

• When model is nearly finalized, try different seeds for samples to ensure model stability.

SAMPLE

ALL DATA

SAMPLE WHAT ABOUT OVERSAMPLING?

• It depends.• Frequently one needs to oversample in order to allow

algorithm(s) to identify effect, especially with rare targets.• Only oversample as much as you need to in order to

obtain a model that makes sense from a business perspective. This is highly subjective.

SELECTING TARGET

CHOOSING YOUR TARGET

• Choosing the Target• Response vs. Propensity• Number of Models

DECISIONING

WEIGHTING YOUR DECISIONS

• Expected Profit• Decision Boundaries

UNDERSTANDING EXPECTED PROFIT

• Consider this game• Flip a fair coin one time• If it is heads, you win $10.00• Cost of playing one time is $1.00

Do you want to play this game?

UNDERSTANDING EXPECTED PROFIT

• Consider this game• Flip a fair coin one time• If it is heads, you win $10.00• Cost of playing one time is $1.00

E(Profit) = 0.5 * (10 - 1) + 0.5 * (-1) = 4.50 + (-0.50) = 4.00

DECISIONS COMBINING THE WEIGHTS WITH PRIORS

DECISIONS DETERMINING DECISION WEIGHTS

• To determine the amount of weight to assign to the rare event in a binary target, calculate this ratio:

• Specify the weight of the rare event to be this ratio• http://support.sas.com/kb/47/965.html

DECISIONS DETERMINING DECISION WEIGHTS: EXAMPLE

• Consider a binary event where Prob(Yes) = 0.1 and Prob(No) = 0.9

• To determine the amount of weight to assign to the rare event in a binary target, calculate this ratio:

• Specify the weight of the rare event to be this ratio

DECISIONS INCORPORATING PRIORS

• Before fitting model• Decision Profile

• After fitting model• Decision Node

PARTITIONING

SAMPLE DATA PARTITIONING

PARTITION ROLE

Training Used to fit the model

Validation Used to validate the model and prevent over-fitting

Test Used to provide unbiased estimate of model performance

SAMPLE SAMPLE: DATA PARTITIONING

WHAT IS OPTIMAL PARTITION?

Training

Validation

BEST PRACTICE SAMPLE: DATA PARTITIONING

Training

Validation

BEST PRACTICE SAMPLE: DATA PARTITIONING

Training

Validation

It depends!

SAMPLE DATA PARTITIONING CONSIDERATIONS

• How much data is available?• Is an unbiased measure of model performance required?

• Should test data be in-sample or out-of-sample?• How many test samples are needed? (e.g. different time

periods, different geographies, etc.)• When should test data be used in the process?

BEST PRACTICE DATA PARTITIONING

• Percentages: frequently used percentages are 50/50/0, 60/40/0 and 70/30/0 with a completely separate Test partition.

• Do not bring Test data into process until model is complete. It should not influence modeling process, merely used to report performance.

• Multiple Test data can be used – consider how model will be deployed and create representative samples.

PREPARING DATA

EXPLORE & MODIFY ITERATIVE RELATIONSHIP WITH DATA PREPARATION

Data Prep

Data Exploration

Data Modification

BEST PRACTICE EXPLORE & MODIFY: GETTING THE MOST OUT OF DATA

• Once you have an analytics-ready table:• Examine Categorical Variables• Examine Continuous Variables• Explore Missing Values• Cluster Variables

EXPLORE & MODIFY CATEGORICAL VARIABLES

EXPLORE & MODIFY CONTINUOUS VARIABLES

EXPLORE & MODIFY MISSING DATA

• Why is data missing?• Are there patterns to the missing data within or across variables?

• Imputation methods to consider

EXPLORE & MODIFY CLUSTER VARIABLES

• There is no single answer for clusters

• Design clusters and profiles around themes using smaller set of related variables

SELECTING VARIABLES

EXPLORE & MODIFY VARIABLE SELECTION/REDUCTION TECHNIQUES

• Stepwise Regression• Variable Selection Node• Decision Tree Node• Variable Clustering • Combined Approach

• Multicollinearity

• Interactions

BEST PRACTICES OPTIMIZING DATA

Selecting TargetDetermining DataPreparing Variables

DEVELOPING & DELIVERING THE MODEL

MODEL & ASSESS DELIVERING THE MODEL

• Developing Your Model• Choosing a Model• Deploying the Model

DEVELOPING THE MODEL

MODEL MODEL DEVELOPMENT

• Regression• Decision Trees• Neural Networks• Ensemble• Rule Induction • Something Else?

BEST PRACTICE MODEL DEVELOPMENT

• Try various techniques and combinations of techniques.

CHOOSING A MODEL

BEST PRACTICES MODEL SELECTION

• Evaluate model metrics• Consider business knowledge• Recognize constraints

ASSESS CUMULATIVE CHARTS

ASSESS NON-CUMULATIVE CHARTS

DEPLOYING THE MODEL

BEST PRACTICES MODEL DEPLOYMENT

• Reporting Results• Clean up and back up• Monitor performance

BEST PRACTICES MODEL DEPLOYMENT

• Incorporate and share knowledge• Automate ETL (Extract, Transform, Load)• Automate process

ULTIMATE GOAL

SAS MODEL FACTORYSOURCE /

OPERATIONAL SYSTEMS

MODEL MANAGEMENT

MODEL DEVELOPMENT

DATA PREPARATION

MODEL DEPLOYMENT

MODEL & ASSESS DELIVERING THE MODEL

• Developing Your Model• Choosing a Model• Deploying the Model

BEST PRACTICES FORMAT OF PRESENTATION

• Background & General Guidance• Developing the Data• Developing & Delivering the Model

BEST PRACTICE BE ANALYTICALLY SAVVY AND CREATIVE

analytical creative

It’s both science and

THANK YOU FOR USING SAS!

proven practices for predictive · pdf fileproven practices for predictive modeling...

Documents

best practices in cyber supply chain risk...

ibm spss predictive analytics workshop · explore multiple...

fda's predictive toxicology roadmap · fda’s predictive...

proven autothermal reforming technology for modern · pdf...

accounting day big data and predictive...

global best practices in insurance claim cycle time & · pdf...

best practices research - automation solutions asset...

science - predictive index | the predictive index system(r)

best practices for using predictive analytics to extract...

predictive modeling upper delaware valley practices€¦ ·...

good practices in the application of predictive analytics

completing learning support: using predictive modeling to...

model predictive control for shunt active filters with...

manuela ekowo and iris palmer predictive … · manuela...

aintso scm...

you have a predictive lead score. now what? | best practices...

predictive analytics demystified agenda •introduction...

predictive maintenance & service and predictive quality...

fastrack supreme wireless cpu development kit quick ......

1 chapter 4: predictive modeling 4.1 introduction to...