an architecture for evaluating recommender systems in real world scenarios
Post on 15-Jul-2015
1.709 Views
Preview:
TRANSCRIPT
An architecture for evaluating recommender systems in real world scenarios
Master Thesis Manuel Blechschmidt 2011
SupervisorProf. Dr. Christoph MeinelM.Sc. Rehab Alnemr
Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
2
Christmas 2009 ...
Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
3
Agenda
■ Motivation and Current Research■ Solution□ Use Cases & Requirements□ Wireframes□ Implementation
■ Related Work■ Conclusion
Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
4
Motivation and Current Research
Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
7
Motivation
■ The choice overload problem is well known in psychology
□ It is necessary to do a preselection for the customer
■ Recommender systems are already very successful to decrease the choice overload problem in some domains
□ Product-to-Product Recommendation Amazon.com→□ Movie Recommendation NetFlix→
■ Algorithms already produce great results
■ Already research in soft factores like: Diversity, Serendepity, Trust, Explanations
not a lot of emprical studies how these influences customers → → no cross domain data sets not a lot of business intereset integration→
Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
8
Current Algorithms and Developments
■ Matrix Factorization (best RMSE 0.855 for NetFlix Dataset)
□ SVD
□ SVD++ R.M.Bell, Y. Koren, and C. Volinsky
□ TimeSVD++ R.M.Bell, Y. Koren, and C. Volinsky
■ Collaborative Filtering
□ Item based
□ User based
■ Performance gains
□ ALS1 István Pilászy, Dávid Zibriczky, Domonkos Tikk
■ Some of the algorithms already implemented in a distributed manner Mahout, MyMedia
Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
9
Empirical Studies
■ Current empirical studies (RecSys 2010)□ Understanding Choice Overload in Recommender Systems
174 participants□ Eye-Tracking Product Recommendersʼ Usage
18 participants□ Recommender Algorithms in Activity Motivating Games
180 participants□ Group-Based Recipe Recommendations: Analysis of Data Aggregation
Strategies170 participants
□ A User-Centric Evaluation Framework of Recommender Systems807 participants
□ Information Overload and Usage of Recommendations466 participants
□ ...
Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
10
Current Problems
■ Not a lot of big empirical studies how recommender quality influence consumer behavior especially
□ Acurarcy
□ Familiarity
□ Serendipity
□ Attractiveness
□ Enjoyability
□ Novelty
□ Diversity
□ Context Compatibility
■ Taken from A User-Centric Evaluation Framework of Recommender Systems
Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
11
Evaluating in real world
■ Most of the academia persons do not know enough persons which are willing to test the algorithms. Therefore the following things are difficult:
□ Evaluating User Interfaces
□ Evaluating Maintenance
□ Evaluating Scalibility
□ Evaluating Performance
Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
13
Master Thesis
■ Building and maintaining an evaluation platform for recommender systems in real world scenarios
■ Maintenance challenges in running a recommender system
■ Empirical study about user behavior
□ Brand loyalty
□ Pricing
□ Timing
Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
14
Solution: Use Cases
Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
15
Roles
■ 5 Roles with different point of views and different interests and goals
■ The roles are describeded with description and goals
■ Example:
□ Provider
□ A provider is a legal personality which has as primary goal to optimize a particular objective. In an economic context this is most of the time a business goal like raise profit or optimize conversion rates. …
□ Goals:– optimizing an objective– get forecasts– ensure privacy of his data
Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
16
Use Cases and Requirements
■ Use Cases and Requirements are described based on IEEE 830
■ A use case is defined by:
□ Id
□ Name
□ Summary
□ Roles
□ Preconditions
□ Postconditions
□ Wireframes
□ More optional attributes
Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
17
Use Case Example C1 Design User Interaction
■ Id: C1 Name: Design User Interaction
■ Summary: When a user interaction should be run like a newsletter or an item-to-item recommendation the consultant has to do the following steps: …
■ Roles: Consultant
■ Preconditions□ User is logged in
□ User has the Consultant role
□ At least one user interaction is implemented
□ At least one provider is associated with the consultant
□ The provider has the necessary data which is needed for the user interaction
■ Postconditions□ Provider received an email for approving the user interaction
□ User interaction is created in the system
Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
18
C1 Design User Interaction
Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
19
C1 Design User Interaction
Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
20
C1 Design User Interaction
Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
21
C1 Design User Interaction
Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
22
Implemented Architecture
Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
23
Logical Modularization
Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
24
Survey Module Entities
Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
25
Survey Module Services
Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
27
Implemented User Interaction chocStore
Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
29
Related Work: Competition
■ NetFlix Grand Prize 2006 – 2009
□ 1.000.000 $ to make CineMatch 10% better
□ Lots research of papers
■ KDD Cup 2011 Recommending Music Itemsbased on the Yahoo! Music Dataset
■ ECML/PKDD’2007 DISCOVERY CHALLENGE
□ User 1 User’s behaviour prediction
Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
30
Related Work: Platforms
■ GroupLens Research of University of Minnesota
□ MovieLens 1997 http://movielens.umn.edu/
■ RichRelevance RecLab 2011
□ RecLab: A System For eCommerce Recommender Research with Real Data, Context and Feedback
■ Knowledge and Data Engineering Group of Uni Kassel
□ 2006 BibSonomy is a system for sharing bookmarks and lists of literature.
Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
31
Further Research
■ Implement more user interactions
□ Item-to-Item recommender
■ Prove that the platform is scalable
■ Run the platform for a long time and evaluate usage
■ Integrate more companies
■ Promote plattform in science and economics
■ Take part at research projects together with companies
Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
33
Conclusion
■ An enterprise ready platform was defined and implemented
■ Companies already applied for using
■ One example user interaction was implemented
□ chocStore
■ Statistical test can be applied to the data to give scientific results
Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
34
Questions
Questions?
Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
35
Backup: What is a recommender?
top related