recommendation architecture - opentable - recsys 2014 - large scale recommender systems
DESCRIPTION
Conventionally when we talk about Recommender Systems, we talk about collaborative filtering. While providing personalized recommendations through collaborative filtering is an essential aspect to providing effective recommendations, it is but a piece of a much broader ecosystem of functionality, tools, and development pipelines. This presentation will discuss an holistic approach to building recommendation systems including 1) iterating towards better recommendations, 2) the data pipelines required, 3) a machine-learned ranking approach based on an Information Retrieval formulation that leverages collaborative filtering, 4) ways to make recommendations more relevant and interpretable.TRANSCRIPT
Recommendation Architecture
Jeremy Schiff
RecSys 2014Large Scale Recommender Systems10/10/2014
BEFORE DURING AFTER
DIN
ERS
REST
AURA
NTS
Sharing & Remembering
Understanding& Evolving
Attracting &Planning
Discovery &Convenience
DelightfulDining
OpenTable:Becoming an Experience Company
Proprietary 2
BEFORE DURING AFTER
DIN
ERS
REST
AURA
NTS
Understanding& Evolving
Attracting &Planning
Deliver great experiences at every step, based on who you are
Proprietary 3
OpenTable in Numbers
•Our network connects diners with approximately 32,000 restaurants worldwide.•Our diners have spent more than $25 billion at our partner restaurants.•OpenTable seats more than 15 million diners each month.• Every month, OpenTable diners write more than 400,000 restaurant reviews
4
Recommendations ≠
Collaborative Filtering
5
So what are recommendations?
6
Building Recommendation Systems
• Importance of A/B Testing
• Generating Recommendations
• Recommendation Explanations
• Recommendation Infrastructure
7
What’s the Goal
Minimizing Engineering Time to Improve The Metric that Matters
•Make it Easy to Measure•Make it Easy to Iterate• Reduce Iteration Cycle Times
8
Importance of A/B Testing
• If you don’t measure it, you can’t improve it
•Metrics Drive Behavior
• Continued Forward Progress
9
Pick Your Business Metric
Revenue, Conversions or Satisfaction• OpenTable • Amazon
Engagement• Netflix• Pandora• Spotify
10
Satisfaction Metric
Recommend RestaurantUser Query
User Review
Verify / Refine
Models
11
Measuring & The Iteration Loop
A/B Testing
Weeks
Measure
12
Measuring & The Iteration Loop
Optimize Models
A/B Testing
Days Weeks
Predict Measure
13
Measuring & The Iteration Loop
Analyze & Introspect
Optimize Models
A/B Testing
Hours Days Weeks
Insights Predict Measure
14
Fundamental Differences in Usage
Right now vs. Planning
Search vs. Recommendations
Cost of Being Wrong
15
Recommendation Stack
Query Interpretation
Retrieval
Ranking – Item & Explanation
Index Building
Context for Query & User
Model Building
Explanation Content
Visualization
CollaborativeFilters
Item / User Metadata
16
Query Interpretation & Retrieval
•Get User Intent• Two Solutions
- Spelling Correction- Auto Complete
•One Box, Many Types
- Name- Cuisine
17
Ranking Objectives
Objectives: • Training Error
- RMSE
•Generalization Error- Precision at K
• A/B Metric- Conversion
18
Ranking
Phase 1: Bootstrap through heuristics
Phase 2: Learn to Rank• E [ Revenue | Query, Position, Item, User ]• E [ Engagement | Query, Position, Item,
User ]• Modeling Diversity is Important
19
Modeling Support: Example
20
Modeling Support: Example
21
Modeling Support: Example
22
Modeling Confidence
•Understand Intersection of
- Support of User- Support of Item
•How does support affect variability of prediction?
- 23
Frequency, Sentiment, and Context
• High End Restaurant for Dinner
- High Sentiment, Low Frequency
• Fast, Mediocre Sushi for Lunch
- High Frequency, Moderate Sentiment
24
How to use this data
• Frequency Data: - General: Popularity- Personalized: Implicit CF
• Sentiment Data: - General: Good Experience- Personalized: Explicit CF
• Good Recommendation- Use both to drive your Business Metric
25
Collaborative Filtering Architecture
Hyper-Parameter Tuning
(Many Days)
Predicted Rating
Full Trainer(Many hours)
Incremental Trainer
(A few seconds)Model
(User, Item)
26
Analyzing Review Content
27
This really is a hidden gem and I'm not sure I want to share but I will. :) The owner, Claude, has been here for 47 years and is all about quality, taste, and not overcharging for what he loves. My husband and I don't often get into the city at night, but when we do this is THE place. The Grand Marnier Souffle' is the best I've had in my life - and I have a few years on the life meter. The custard is not over the top and the texture of the entire dessert is superb. This is the only family style French restaurant I'm aware of in SF. It also doesn't charge you an arm and a leg for their excellent quality and that also goes for the wine list. Soup, salad, choice of main (try the lamb shank) and choice of dessert - for around $42 w/o drinks.
“SUPERB!”
Bay Area ReviewsPost Jan 2013
Reviews come in all shapes and sizes!
28
29
The ingredients of a spectaculardining experience…
30
… and a spectacularly bad one
Content Features
Pandora•Music Genome Project
Natural Language Processing• Topics & Tags
31
Generating Topic Features
• Stop Words & Stemming• Bag of Words Model• TF/IDF • Topic Modeling•Describe Restaurants as Topics
32
Stop Words & Stemming
The food was great! I loved the view of the sailboats.
33
Stop Words & Stemming
The food was great! I loved the view of the sailboats.
34
The food was great! I loved the view of the sailboats.
Bag of Words Model
food great chicken sailboat view service
1 1 0 1 1 0
35
TF-IDF
• Term Frequency - Inverse Document Frequency
• Final Value = TF(t) IDF(t)36
TF-IDF Example
food great chicken sailboat view service
.02 0.05 0 0.5 0.25 0
The food was great! I loved the view of the sailboats.
37
Topic Modeling Methods
We applied two main topic modeling methods:• Latent Dirichlet Allocation (LDA)
- (Blei et al. 2003)
•Non-negative Matrix Factorization (NMF)
- (Aurora et al. 2012)38
Topics with NMF using TF-IDFWord 1 Word … Word N
Review 1 0.8 0.9 0
Review … 0.6 0 0.8
Review N 0.9 0 0.8
ReviewsX
Words
ReviewsX
Topics
TopicsX
Words
39
Describing Restaurants as Topics
Each review for a given restaurant has certain topic distribution
Combining them, we identify the top topics for that restaurant.
Topic 01 Topic 02 Topic 03 Topic 04 Topic 05
Topic 01 Topic 02 Topic 03 Topic 04 Topic 05
Topic 01 Topic 02 Topic 03 Topic 04 Topic 05
review 1
review 2
review N
...
Topic 01 Topic 02 Topic 03 Topic 04 Topic 05
Rest
aura
nt
40
Examples of Topics
41
Varying Topic By Region• San Francisco
• `
• London
• Chicago
• New York
42
Compelling Recommendations
43
Recommendation Explanations• Amazon
• Ness
• Netfl
• ix
• Ness - Social
44
Summarizing Content
• Essential for Mobile• Balance Utility With Trust?
- Summarize, but surface raw data
• Example: - Initially, read every review- Later, use average star
rating
45
Summarizing Restaurant Attributes
46
Active Learning for Summarization
• Incremental Supervised Learning• Know Precision & Recall• Always Forward Progress• Generate Dataset: False Positive/Negative or Difficult to
Discriminate
Provide Labels
Train Model
Evaluate AccuracyOn Full Dataset
Generate New
Dataset
47
Devil is in the Details
Attribute Tag – Dim Lighting
“I love the relaxed feel of this place – dark, small, and cozy – like a comfortable living room.”
48
Dish Recommendation•What to try once I have arrived?
49
Edit via the Header & Footer menu in PowerPoint
5050
Infrastructure
User Interactions
Service Logs
51
Infrastructure
User Interactions
Service Logs
Queue
52
Infrastructure
User Interactions
Service LogsBatched Data For Analysis
Real-Time Processing
Queue
53
Infrastructure
User Interactions
Service Logs
Queue
Batched Data For Analysis
Real-Time Processing
Model Training
A/B Testing
Analytics
54
Multi-Datacenter Infrastructure
55
Building Recommendation Systems
• Importance of A/B Testing
• Generating Recommendations
• Recommendation Explanations
• Recommendation Infrastructure
56
Team Composition
Team• Data Scientist
- Math & Applied Machine Learning- Relevancy and Accuracy
• Data Science Engineer- Software Development- Infrastructure, Speed and Maintainability
Everyone works on production systems57
Questions?
Jeremy [email protected]