October 20, 2009

Raising the Bar in Cross-

Sell Marketing with Uplift

Modeling

Michael D. Grundhoefer

U.S. Bank

Company Background

$266 billion in assets, the 6th largest

commercial bank in the United States

Operates 2,850 banking offices and

5,173 ATMs serving 15.8 million

customers.

Headquarters: Minneapolis, MN

Provides a comprehensive line of

banking, brokerage, insurance,

investment, mortgage, trust and payment

services products to consumers,

businesses and institutions

Regional

Consumer and Business Banking

Wealth Management

National

Wholesale Banking

Trust Services

Global

Payments

Regional

Consumer and Business Banking

Wealth Management

National

Wholesale Banking

Trust Services

Global

Payments

What is Uplift (Incremental) Modeling?

Review:

– Predicting who‟s likely to respond only when they are given a treatment (direct marketing) as opposed to responding on their own without a treatment.

Direct Marketing Segmentation

No Yes

Yes

No

PERSUADABLES

SLEEPING DOGS

SURE THINGS

LOST CAUSES

BUY IF

MAILED

BUY IF

NOT MAILED

Decile Example

-0.40%

-0.30%

-0.20%

-0.10%

0.00%

0.10%

0.20%

0.30%

0.40%

0.50%

0.60%

1 2 3 4 5 6 7 8 9 10

Model Score Decile

Incre

men

tal U

plift

Direct Marketing Segmentation

PERSUADABLESLOST CAUSES/

SURE THINGSSLEEPING

DOGS

Why Incremental Matters

Provides most accurate measurement of ROI

Quantifies the true value of marketing spend

Is what our product managers are measured on

As modelers – we are showing our true worth by adding incremental lift to campaigns – and not just able to correctly rank people in likelihood to buy products.

How did we try to do uplift modeling?

Where we‟ve been ...

– Straight Response Modeling and hope the offer lifts response

“that‟s so... „90‟s”

– Two Model – Decile Matrix

– Campaign Response Model overlaid with Natural Buy Rate Model

– Look for biggest differences (10 x 10) (Matrix Model)

– Two Models (rank on difference)

– Campaign Mail Response model

– Campaign Control “response” model

– Decision Tree (manual) (my great idea!)

– Find the field that has the biggest uplift/incremental, next ..

and so on ...

How we do Uplift modeling - Now

Where we are today:

– Portrait Software – Uplift Optimizer (Automated Decision Tree)

– Automated variable selection – uplift based

– Bootstrapping/bagging/Boosting (resampling)

– Multiple tree averaging

– Variable restrictions – (holding out some variables)

– Auto pruning of unstable nodes

– Model “Gini” score comparison

– Visual Variable Effect exploration

– Model Comparison: Training vs Validation

Customer Incremental Response (Uplift) Rate

0.25%

0.16%

0.25% 0.25%0.28%

0.20% 0.19%

0.12%

0.32% 0.31%0.28%

0.50%

0.35%

0.22%

0.06%

0.47%

0.08%

0.23%

0.19%

0.18%

0.00%

0.10%

0.20%

0.30%

0.40%

0.50%

0.60%

0204 0504 0804 0105 0405 0805 1005 0206 0406 0706 1006 0207 0407 0807 0108 0408 0708 1008 0109 0409

Campaign Date

Inc

rem

en

tal (U

plift

) R

es

po

ns

e R

ate

Results – Past and Present

Customer Home Equity Cross-Sell

Matrix

Model

Improved Tree

Model ??

Pilot Testing

with Portrait

Full Roll out of Portrait Uplift Models

Gen 1 to 3 Models

Back to Matrix

and other testing

Uplift Modeling

Why all the Portrait techniques help

– Help with small sample size and small response rate

– Takes a lot of manual processes and batches together

– Presents reports for easy review and comparison of models

– Allows for many runs to find best and robust model

Nuances of this approach

– Specific variable interpretation may be hard – average of many trees

– Observed some volatility in the middle deciles

DDA Cross-sell Uplift Model

Original Training/Validation Data

-2.00%

-1.50%

-1.00%

-0.50%

0.00%

0.50%

1.00%

1.50%

10987654321

Score Decile

Up

lift

Training

Validation

Example of Volatility of Middle Deciles

Score Decile Report (DDA Example)

Some instability in middle deciles

PERSUADABLESLOST CAUSES/

SURE THINGSSLEEPING

DOGS

Starting to build an Uplift Modeling

Prepare a “Good” Data Set – 80% of the work!

– Campaign Specific Response(booked/opened) Data

– Use the optimal response observation period

– Too short = not full impact of mailing;

– Too long = will trend towards control group.

– Entire populations should be sampled

– Need to sample lower deciles - if model used to select

– Understand the unique selection of campaign and weight accordingly

– If lower deciles were sampled, they should be weighted back up to

represent the full population.

– If unique selections were made – e.g. over or under-sampling an area –

those should be weighted back too.

Considerations with Uplift Modeling

Data Set (cont.)

– Large Sample Size is King

– It helps to have a large training data set and an equally large validation

data set.

– We typically have 500k+ raw count and 1.2m when weighted back

– For this type of modeling, the quantity of the control group is equally as

important as the mailed group – ideally we could have 50% mail and 50%

control.

– We typically have a 10% hold out control group (may have to negotiate!)

Considerations with Uplift Modeling (cont.)

To help with small Control Group quantity

– May be able to tap into the lower decile unselected control (“Extra

Control”)

For example:% (90%) (10%) Total Extra

Selected Mailed Control Quantity Control

in Campaign Decile Quantity Quantity Selected Available

100% 1 -Best 90,000 10,000 100,000 0

100% 2 90,000 10,000 100,000 0

100% 3 90,000 10,000 100,000 0

10% 4 9,000 1,000 10,000 9,000

10% 5 9,000 1,000 10,000 9,000

10% 6 9,000 1,000 10,000 9,000

10% 7 9,000 1,000 10,000 9,000

10% 8 9,000 1,000 10,000 9,000

10% 9 9,000 1,000 10,000 9,000

10% 10-Worst 9,000 1,000 10,000 9,000

333,000 37,000 370,000 63,000

Valid

ation

Universal Hold-Out

Considerations with Uplift Modeling (cont.)

Uplift Models tend to need refreshing more often than

standard response models.

– Home Equity – refresh every three quarters

– More sensitive in this economic environment?

– Consumer opinions change more often?

– DDA is new but has held up for four quarters – but watching closely

Example: Home Equity Cross-Sell Uplift Model

Background

Previous Campaign Data

– Data set size ~ 500,000

– After weighting ~ 1,000,000

~ 300+ Variables selected down to ~40 eligible for inclusion in model

Including: Product Ownership Product Balances

Product Usage Demographics

Geographic Channel Usage

Final Uplift Score was an average of 10 Decision Trees with a total of

10 selected variables.

Test Uplift vs Matrix Model

Model Prediction

Home Equity Uplift Gen 1 - Prediction

Home Equity Uplift - Gen 1

-1.00%

-0.80%

-0.60%

-0.40%

-0.20%

0.00%

0.20%

0.40%

1 2 3 4 5 6 7 8 9 10

Decile

Incre

men

tal

Up

lift

Training

Validation

(weighted) Actual Incremental Cummulative Total Cummulative

Decile Quantity Uplift Accounts Accounts Revenue1

Revenue

1 100,000 0.36% 360 360 360,000$ 360,000$

2 100,000 0.31% 310 670 310,000$ 670,000$

3 100,000 0.04% 35 705 35,000$ 705,000$

4 w 100,000 0.03% 32 737 32,000$ 737,000$

5 w 100,000 0.00% 0 737 100$ 737,100$

6 w 100,000 -0.05% (50) 687 (50,000)$ 687,100$

7 w 100,000 0.06% 60 747 60,000$ 747,100$

8 w 100,000 -0.05% (50) 697 (50,000)$ 697,100$

9 w 100,000 -0.11% (110) 587 (110,000)$ 587,100$

10 w 100,000 0.10% 100 687 100,000$ 687,100$

1,000,000 0.07% 687 687,100$

Examination of Actual Campaign Results

Home Equity Campaign - Uplift Score Decile Report (example)

Model Ranked well in top deciles, but had some volatility in the lower

deciles.

– May be due to the small sample size validation-lower deciles

1 $1000 per opened account

Model Comparison: Uplift vs Matrix

New Model was tested against old Matrix Model (new campaign)

Model Comparison

Uplift vs Matrix

Q107

0.00%

0.05%

0.10%

0.15%

0.20%

0.25%

0.30%

0.35%

0.40%

1 2 3 4 5 6 7 8 9 10Decile

Cu

mm

ula

tive L

ift

Uplift Model

Matrix Model

Mailing focused

on top 3 deciles

Model Comparison: Uplift vs Matrix

The Top 3 deciles performed well in both the Matrix and the new

Uplift Model.

However – there were significant gains in ROI to be shown by the

Uplift Model – Gen 1 of over 190%

Uplift IM

500,000 500,000

0.23% 0.11%

1,175 550

625

$0.45

$1,000

$225,000 $225,000

$1,174,550 $549,750

$949,550 $324,750

$624,800

192%

Difference (Uplift - IM)

SUMMARY (Top 3 Deciles)

Mailing volume

Lift

Incr. bookings

Percenatge improvement in ROI

Difference in Revenue

Cost Per Piece Mailed

Revenue per Account

Total Maling Cost

Total Value of Accounts Booked

(Example)

Models Deployed and Planned

Other Uplift Models Deployed:

– Checking Cross-sell

– Other Credit Product (2nd type)

– DDA Incentive Decision model ($A vs $B) (testing)

– Money Market Cross-sell (testing)

Uplift Models to be implemented in the near term:

– Home Equity Account Utilization

– Auto Loans

– Retool of Money Market

– Retool of DDA Incentive

Ongoing Monitoring and Retooling

We examine the results from each campaign and decide to whether

to retool

We know the models lose their strength sooner than traditional

propensity models so we mail lower decile validation samples often.

We look for new ways to use this methodology in other aspects of

marketing – e.g. Incentive assignment.

Questions

Questions?

Thank you.