machine learning at criteo - paris datageeks

Shallow learning @ Criteo

Nicolas Le RouxScientific Program Manager – R&D

I N T R O D U C T I O N

Copyright © 2013 Criteo

WHAT DOES CRITEO DO?

3

• We buy advertising space on websites

• We display ads for our partners

• We get paid if the user clicks on the ad


HOW DO WE BUY ADVERTISING SPACE?

4

• Real-time bidding (RTB): it is an auction

• Second-price

• Optimal strategy: bid the expected gain

• Expected gain = CPC * CTR


WHAT TO DO ONCE WE WIN THE DISPLAY?

5

• Choose the best products

• Choose the color, the font, the layout…

• Generate the banner


WHERE IS THE ML AT CRITEO?

6

• Click prediction model

• Product recommendation

• Testing, …


WHERE IS THE ML AT CRITEO?

7

• Click prediction model

• Product recommendation

• Testing, …


INFRASTRUCTURE AT CRITEO

8

• 5 data centers on 3 continents

• More than 3 500 servers

• More than 30B HTTP requests per day

• More than 2B banners displayed per day

• Peak traffic: 500K HTTP requests per second


C L I C K P R E D I C T I O N M O D E L


CTR PREDICTION MODEL

10

𝑃 (𝑐𝑙𝑖𝑐𝑘|𝑎𝑖 )= 𝑓 (𝑎𝑖 ,𝜃 )Copyright © 2013 Criteo


11

• : features computed for this advertiser



12


• : parameters of the model



13


• : parameters of the model

• : prediction model



14

• : [TimeOfDay, AdvertiserID, SizeOfAd]

• : learnt using an optimisation algorithm

• : dot product


15-SECOND ML CRASH COURSE

15

• Get a massive amount of (, click) pairs

• Find the value of which gives the best prediction on these pairs.


NEW HIRE REACTION

16

𝑃 (𝑐𝑙𝑖𝑐𝑘|𝑎𝑖 )= 𝑓 (𝑎𝑖 ,𝜃 )Copyright © 2013 Criteo

NEW HIRE REACTION

17

• « You should use [his PhD topic] for f, it’s awesome! »


NEW HIRE REACTION

18


• « To f, you should use [other part of his PhD] »


NEW HIRE REACTION

19


• « To f, you should use [other part of his PhD] »

• « I think you should use [a particular optimizer he used during his PhD] »


REAL TIME CONSTRAINTS FOR

20

• Bid must happen within a few ms

• Includes extracting variables, scoring tens of campaigns

• Leaves s per prediction

• Process is extremely noisyCopyright © 2013 Criteo

IS WEIRD

21

• Around features

• Extremely sparse

• Website URL, Merchant ID, ...


OPTIMIZING

22

• Merchant ID is too raw

• We can learn fancy features for each merchant: demographics, size, ...

• Hard to not penalize large merchants


OPTIMIZING THE MODEL

23

• Computing is extremely fast

• The botteleneck is the streaming of the data

• Not all points are created equal

• Best optimizer is the one that needs the fewest data


DEALING WITH MANY VARIABLES

24

• can be extremely large

• It makes learning and prediction complicated


DEALING WITH MANY VARIABLES

25

• can be extremely large

• It makes learning and prediction complicated

• Solution: hash into

• New model is


OTHER CHALLENGES

26

• How long do you wait for sales after a click?

• How much in the past do you go to learn the model?

• Do you value more long term or short term variables?

• How to choose the best variables?


P R O D U C T R E C O M M E N D A T I O N


THE CHALLENGES OF PRODUCT RECOMMENDATION

28

• We have the list of products seen

• A catalog can contain products

• How do we choose the right products to show?


THE CHALLENGES OF PRODUCT RECOMMENDATION

29

• We have the list of products seen

• A catalog can contain products

• How do we choose the right products to show?

• In less than 20ms


NEW HIRE REACTION

30

• « You should use [his PhD topic], it’s awesome! »


NEW HIRE REACTION

31


• « It just takes a week to train. »


NEW HIRE REACTION

32


• « It just takes a week to train. »

• « In real-time, you just have to score every product. »


MAJOR HURDLES

33

• Products come and go

• There might be a sale (Black Friday)

• The « panties syndrome »

• Complementary vs. similar products


OTHER CHALLENGES

34

• Multiple products in a banner

• Interaction between products and layout

• Different timeframes for different products


A FIRST RECIPE FOR SUCCESS

35

• There are many sources of success/failure

• It is often suboptimal to focus on one

• The first step to address each source is often manual


CONCLUSION

36

• Prediction is at the core of our business

• Huge engineering constraints

• Different bottlenecks than in academia

• Build from the ground up, not the other way around


Nicolas: [email protected]: [email protected] Clem: [email protected]

QUESTIONS ?

mailto:[email protected]

machine learning at criteo - paris datageeks

Technology