modeling choice with limited data - celect at strata + hadoop world 2015

Bringing Science to the Art of Retail SM

Modeling Choice with Limited DataVivek Farias, CTO @ CelectRobert N. Noyce Professor @ MIT

©2015 Celect, Inc. All Rights Reserved.2

An Important Question

©2015 Celect, Inc. All Rights Reserved.3

An Important Question

Comparisons are eas(ier) …

… but also, comparisons are everywhere.

©2015 Celect, Inc. All Rights Reserved.4

Comparisons Are Everywhere

Traditional Reviews: Peter prefers Tamarind to Cafe Boulud

Peter G.

Peter G.’s Profile

About Peter G.

Peter G.

©2015 Celect, Inc. All Rights Reserved.

Comparisons Are Everywhere

5

Click Stream Data

©2015 Celect, Inc. All Rights Reserved.

Comparisons Are Everywhere

6

Click Stream Data: Vinay is more in the mood for The Expendables 3 vs. Cosmos or The Bank Job or ... ?

Comparisons Are Everywhere

©2015 Celect, Inc. All Rights Reserved.7

(Brick and Mortar) Retail Data

©2015 Celect, Inc. All Rights Reserved.

Comparisons Are Everywhere

8

(Brick and Mortar) Retail Data: I prefer Neutrogena to Dove, and Aveeno, and …

©2015 Celect, Inc. All Rights Reserved.

Two Applications• Helping a department store build “hyper-local”

experiences‒ Fully data driven approach to modeling choice

• Helping a top 10 US Retail website personalize to intent‒ A new view of what personalization online means

9

“Harvesting and Learning From Comparisons”

©2015 Celect, Inc. All Rights Reserved.

A Mental Picture Of A Choice Model

10

Representative Customer

©2015 Celect, Inc. All Rights Reserved.

A Generic Choice Model

11

30% 30% 7% 7% 20% 6%

©2015 Celect, Inc. All Rights Reserved.

A Generic Choice Model

12

= 30+30 = 60%

30% 30% 7% 7% 20% 6%

©2015 Celect, Inc. All Rights Reserved.

A Generic Choice Model

13

30% 30% 7% 7% 20% 6%

©2015 Celect, Inc. All Rights Reserved.

A Generic Choice Model

14

= 30+30+20 = 80%

30% 30% 7% 7% 20% 6%

©2015 Celect, Inc. All Rights Reserved.

A Generic Choice Model

15

$200

$300

$100

30% 30% 7% 7% 20% 6%

©2015 Celect, Inc. All Rights Reserved.

A Generic Choice Model

16

$100 $100$200 $200 $300 $30030% 30% 7% 7% 20% 6%

©2015 Celect, Inc. All Rights Reserved.

A Generic Choice Model

17

30% 30% 7% 7% 20% 6%$100 $100$200 $200 $300 $300 $200 $300

©2015 Celect, Inc. All Rights Reserved.

A Generic Choice Model

18

$200 $300$200 $200 $300 $30030% 30% 7% 7% 20% 6%

©2015 Celect, Inc. All Rights Reserved.

Revenue Estimation

19

?

Challenge: How to deal with sparse data & a high dim model? Traditional Approach: Assume comes from a parametric family

Generic Model Limited Data

Predicted Rev Rate

©2015 Celect, Inc. All Rights Reserved.

Revenue Estimation

20

Challenge: How to deal with sparse data & a high dim model? Our Approach: Assume that comes from a sparse model

Generic Model Limited Data

?Predicted Rev Rate

©2015 Celect, Inc. All Rights Reserved.

Our Approach

21

“Simple” “Conservative”

What do we know?

©2015 Celect, Inc. All Rights Reserved.

Case Study: Choice Modeling + Assortments• Mid-Size Retailer (~$4B in revenue. Close to 300 stores)• Task

‒ Optimize assortment using our approach to learn choice patterns at individual stores

• Data‒ In-store inventory + transactions over time‒ Online browse/ transactions

• Controlled experiment at 10 representative stores (Q3 14)

22

23

Case Study: Choice Modeling + Assortments

©2015 Celect, Inc. All Rights Reserved.

Test Stores

Control Stores

08/03 - 09/27

06/15 - 07/26

Test Period

24

Case Study: Choice Modeling + Assortments

©2015 Celect, Inc. All Rights Reserved.

Test Stores

Control Stores

08/03 - 09/27

06/15 - 07/26Trailing 6

Weeks

25

Case Study: Choice Modeling + Assortments

©2015 Celect, Inc. All Rights Reserved.

5.71% -2.91%

-.54% -2.51%

Test Stores

Control Stores

08/03 - 09/27

06/15 - 07/26

Test Stores Net Relative5.71 - -0.54 = 6.25%

26

Case Study: Choice Modeling + Assortments

©2015 Celect, Inc. All Rights Reserved.

5.71% -2.91%

-.54% -2.51%

Test Stores

Control Stores

08/03 - 09/27

06/15 - 07/26

Control Stores Net Relative

-2.91 - -2.51 = -0.40%

27

Case Study: Choice Modeling + Assortments

5.71% -2.91%

-.54% -2.51%

Test Stores

Control Stores

08/03 - 09/27

06/15 - 07/26

Adjusted Net Relative6.25 - -0.40 = +6.65%

©2015 Celect, Inc. All Rights Reserved.

©2015 Celect, Inc. All Rights Reserved.28

Case Study Summary• Celect modifiers were applied to a limited portion of each

test store• Adjusting against control stores and for trailing 6-wk

trends:‒ Celect modified departments yielded an incremental 6.65%

increase in revenue‒ We predicted 6.43%, so our performance is predictable

• Test stores in total yielded an incremental 2.98% increase in revenue potentially due to positive spillover effects of a better assorted store

©2015 Celect, Inc. All Rights Reserved.

Personalization

29

“If you liked this then you will also like these other things”

©2015 Celect, Inc. All Rights Reserved.

Personalization

30

“If you liked this then you will also like these other things”

• What does it mean to personalize online?• Explosion of ad-hoc heuristics -- tailor made to application• All have a ‘nearest neighbors’ flavor• How does choice play a role?

©2015 Celect, Inc. All Rights Reserved.

Personalization

31

“If you liked this then you will also like these other things”

• What does it mean to personalize online?• Explosion of ad-hoc heuristics -- tailor made to

application• All have a ‘nearest neighbors’ flavor• Our Approach: Build a choice model, and optimize

assuming choice

©2015 Celect, Inc. All Rights Reserved.

Implementation at a Gigantic Retailer• Retailer (~$36B in revenue. ~ 10 Monthly Uniques)• Task

‒ Personalized offerings for search and checkout‒ < 5 ms RTT tolerance

• Data‒ Online browse/ transactions‒ CRM

• Live on traffic

32

©2015 Celect, Inc. All Rights Reserved.33

Implementation at a Gigantic RetailerCategory Incumbents Celect Increase

Appliances $4.38 $5.57 27.17%

Clothing $0.64 $0.77 20.31%Lawn & Garden $3.26 $4.50 38.04%

Tools $1.36 $1.56 14.71%

Toys $0.41 $0.67 63.41%

Outdoor Living $4.49 $5.03 12.03%

Total $2.86 $3.43 19.93%

20% increase in monetizing site

traffic.

Distinct incumbents in each category

©2015 Celect, Inc. All Rights Reserved.

In Summary• Understanding choice is valuable• We can model choice by learning from atomic comparisons

‒ Comparisons are everywhere• Data-Driven Choice modeling can re-shape offline retail

functions‒ Hyper-Localized Buying, Planning and Allocation

• Data-Driven Choice modeling can re-shape personalization‒ Understanding choice beyond nearest-neighbor algorithms

34

35

A Top Strategic Initiative"Finally, our localization initiative is important to our growth plan as we look to offer increasingly local-centric assortments. We are testing new software called Celect, a predictive modeling tool that allows us to leverage customers' omnichannel purchase and browse behavior to identify merchandise localization opportunities. We believe that localization can allow us to further increase market share in our low-volume doors.”

Kathy Bufano, CEO, Bon-Ton StoresBONT Earnings Call, May 21, 2015

©2015 Celect, Inc. All Rights Reserved.

©2015 Celect, Inc. All Rights Reserved.36

Thank you!

Live Webcast – October 15th at 1pm ETwww.celect.com/screaming-data-webcast

Visit blog.celect.com

Follow us on Twitter! @celect