Kris Jack and Maya Hristakeva16/12/2014

Modern Perspectives on Recommender Systems and their Applications in Mendeley

Kris Jack, Chief Data Scientisthttp://www.mendeley.com/profiles/kris-jack/

Maya Hristakeva, Senior Data Scientisthttp://www.mendeley.com/profiles/maya-hristakeva/

Phil Gooch, Senior Data Scientisthttp://www.mendeley.com/profiles/phil-gooch/

Overview• The what and why of recommenders• Evolution of the recommender problem• Recommender algorithms • Evaluating a recommender• Recommender systems @ Mendeley

Overview• The what and why of recommenders• Evolution of the recommender problem• Recommender algorithms • Evaluating a recommender• Recommender systems @ Mendeley

What is a recommender?A recommendation system (recommender) is a push system that presents users with the most relevant content for their context and needs• helps users to deal with information overload• recommenders are complementary to search

search enginepull

recommendation enginepush

requestinfers context

and needs

Information Retrieval Information

Filtering

Recommenders @ Linkedin50% of LinkedIn connections are from recommendations

Recommenders @ Linkedin

Recommenders @ NetflixStop 1% of users from cancelling subscription = $500M/yearNetflix invests $150M/year (300 people) in their content rec team

Recommenders @ ResearchGate

Why recommenders?• Search and recommendations are complementary, have arms and legs!• Higher usability, user satisfaction and engagement• Increase product stickiness• Monetise them

...and in the context of research...Help researchers keep up-to-date with latest research, connect with researchers in their field, contextualise their work within the global body of research (articles, researchers, conferences, research groups, etc.)

Overview• The what and why of recommenders• Evolution of the recommender problem• Recommender algorithms • Evaluating a recommender• Recommender systems @ Mendeley

Evolution of recommender problem

Problem: We have a massive collection of items (e.g. > 1 million).We want to recommend 5 items that the user will like.

Evolution of recommender problem

First, seen as a ratings prediction problem. So, given some knowledge of the user, estimate how much they will appreciate each item on scale of 1-5.

4.9

choose top 5 items with highest predicted ratings

4.7 4.7 4.6 4.5

Evolution of recommender problem

But do predicted ratings give the best order? Improve the recommender by reranking a selection of items with high predicted ratings.

rerank items that are highly predicted

4.7 4.9 4.6 4.6 4.8

Evolution of recommender problem

Let’s improve the recommendations by optimizing the page in which they appear.

deliver them in style

Evolution of recommender problem

Take the user’s context into account.

new to this topic?

yesno

Evolution of recommender problem

Actively researching how to take other properties into account in context: trustworthiness; freshness; diversity; serendipity; novelty; recency.

at work? yesno

Rating predictionRerankingPage optimisationContext-awareFuture: trustworthiness; freshness; diversity; serendipity; novelty; recency.

How to make recommendations?On to the algorithms...

Evolution of recommender problem

time

Overview• The what and why of recommenders• Evolution of the recommender problem• Recommender algorithms • Evaluating a recommender• Recommender systems @ Mendeley

Recommender algorithms

A recommender processes information and transforms it into actionable knowledge. Here we’ll focus on the algorithms that make this possible.

information flow (components often built in parallel)

Recommender algorithms• Collaborative filtering (similarity and model-based)• Content-based filtering• Hybrid• Non-traditional

Collaborative filtering

Formal representation

• User-based CF finds users who have similar appreciations for items as you and recommends new items based on what they like.

• Item-based CF finds items that are similar to the ones you like. Similarity is based on item cooccurrences (e.g. the users who bought x also bought y).

Similarity-based CF

• ti: rating of user xi for item yi.• Infer prediction function

Collaborative filtering

Formal representation of MF

• X: user-item ratings matrix• U: user-latent factors matrix• S: latent factor diagonal matrix• V: latent factor-item matrix

• Matrix Factorisation (SVD++)• Clustering (K-means to LDA)• LSH (Locality sensitive hashing)• Restricted Boltzmann Machines

Model-based CF

Collaborative filtering

Pros• Minimal domain knowledge

required• User and item features are not

required• Produces good enough results

in most cases

• Cold start problem• Requires high user:item ratio (1:

10)• Needs standardised products • Popularity bias (doesn’t play

well with the long tail)

Cons

• User-based CF• Item-based CF• Model-based CF

Content-based filtering• Determine item similarity based on item content not usage data• Recommend items similar to those that a user is known to like• The user model:

• explicitly provided features/keywords of interest• can be a classifier (e.g Naive Bayes, SVM, Decision trees)

Formal representation

• ti: rating of user xi for item yi, where xi and yi are feature vectors• Infer prediction function

Content-based filtering

Pros• No cold start problem• No need for usage data• No popularity bias, can

recommend items with rare features

• Item content needs to be machine readable and meaningful

• Easy to pigeonhole the user• Difficult to implement serendipity• Difficult to combine multiple item’

s features together

Cons

• Determine item similarity based on item content not usage data• Recommend items similar to those that a user is known to like• The user model:

• explicitly provided features/keywords of interest• can be a classifier (e.g Naive Bayes, SVM, Decision trees)

Hybrid approaches

Method Description

Weighted Outputs from several techniques (in the form of scores or votes) are combined with different degrees of importance to offer final recommendations

Switching Depending on situation, the system changes from one technique to another

Mixed Recommendations from several techniques are presented at the same time

Feature combination Features from different recommendation sources are combined as input to a single technique

Cascade The output from one technique is used as input of another that refines the result

Feature augmentation The output from one technique is used as input features to another

Meta-level The model learned by one recommender is used as input to another

Hybrid approachesCombining user and item features and usage to benefit from both

Pros• Often outperforms CF and CB

alone

Cons• Can be a lot of work to get the

right balance

Non-traditional approaches• Deep learning• Social recommendations• Learning to rank• ...

Pros Cons• Good for eking out those final

performance percentage points• You can say you’re working with

current edge approaches ;)

• Less well understood• Less supported in

recommendation toolkits• Not recommended approaches

for your first recommender

Is your recommender doing well?

• Typically employ collaborative filtering• May need to use content-based filtering particularly to bootstrap• Go advanced with a hybrid• Do all of that before getting adventurous with state-of-the-art

You don’t really know unless you evaluate it...

Algorithms

Overview• The what and why of recommenders• Evolution of the recommender problem• Recommender algorithms • Evaluating a recommender• Recommender systems @ Mendeley

• Offline testing• Online testing (A/B testing)

Evaluating a recommender

Offline testing• Test offline before deploying

• Parameter sweep is quick• Doesn’t offend real users

• n-fold cross validation:• Take the users, items and

relationships between them (e.g. clicked on, bought)

• Split into n folds, for training (n-1) and testing (1)

• Attempt to predict the testing data based on the training data

• Popularity as baseline

Metrics• Precision, recall and f-measure• Receiver operating characteristic

(ROC) curve• Normalised discounted cumulative

gain (NDCG)• Mean reciprocal rank (MRR)• Fraction of Concordant Pairs (FCP)• ...

Online testing• Offline performance isn’t a very

precise indicator• Offline test is good sanity

check• Online test gives real

performance• A/B testing

• Deploy your systems that perform ‘well enough’

• Compare them with each other in real world

• Mind the pitfalls

Metrics• The offline metrics +

• Conversion rate• Open, view, click through rates• Usage data (e.g. reordered item,

completed reading book)• Hard to evaluate: trustworthiness;

freshness; diversity; serendipity; novelty; recency.

• Start with offline testing• Perform A/B testing but be aware of the common pitfalls• Hard to evaluate performance in terms of: trustworthiness; freshness;

diversity; serendipity; novelty; recency.

How do we use recommenders?On to a few of our use cases...

Evaluating a recommender

Overview• The what and why of recommenders• Evolution of the recommender problem• Recommender algorithms • Evaluating a recommender• Recommender systems @ Mendeley

Recommenders @ Mendeley

Recommenders @ MendeleyRelated research for an article

Recommenders @ MendeleyRelated research for multiple articles

Recommenders @ MendeleyMendeley Suggest - personalised batch of recommended reading

Recommenders @ MendeleyResearchers to follow on Mendeley

Recommenders @ MendeleyInteresting activity from your social network

• Recommenders are employed for a number of use cases• Recommenders deliver different kinds of value depending upon use case• Can reuse the same underlying recommender system and framework for all

Recommenders @ Mendeley

• Recommenders are complementary to search and becoming mainstream• although arguably can cater for a wider range of use cases

• When building a recommender, it’s common to predict ratings, rerank, optimise the page and then introduce context-awareness

• In building a recommender, start with collaborative filtering if you can, content-based if you need to bootstrap and then explore hybrids

• Open research questions remain as recommenders are used to tackle trustworthiness; freshness; diversity; serendipity; novelty; recency

Conclusions

References• Xavier Amatriain, The Recommender Problem Revisited (http://www.

slideshare.net/xamat/recsys-2014-tutorial-the-recommender-problem-revisited)

• Rec Sys 2014 (http://recsys.acm.org/recsys14/)

Thank youwww.mendeley.com