advances in content-based recommender systems explanation...

@cataldomusto [email protected]

Advances in Content-based Recommender Systems

Explanation StrategiesCATALDO MUSTO

UNIVERSITÀ DEGLI STUDI DI BARI ‘ALDO MORO’ - ITALY

Recommender Systems

Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies.

ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 2

The Explanation Problem

Recommendation


ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019

I suggest you…

3

The Explanation Problem



Recommendation

4

A possible solution: descriptive properties



Recommendation

I suggest you The Ring because you

often like movies with Naomi Watts

as 21 grams and Mulholland Drive.

Furthermore, you like films about

ghosts such as The Sixth Sense.

5



Another solution: review-based features

I recommend you The Ring because

people who liked the movie think that

it delivers some bone-chilling terror.

Moreover, people liked The Ring

since the casting is pretty good.

6

An overview of content-based strategies

to build a domain-agnostic and

algorithm-agnostic explanation

supporting the recommendation.



In this talk

7

An overview of content-based strategies

to build a domain-agnostic and

algorithm-agnostic explanation

supporting the recommendation.



In this talk

8

1. Content-based Explanations exploiting

the Linked Open Data cloud

2. Review-based Explanation exploiting

Sentiment Analysis techniques

3. Review-based Explanations exploiting

Automatic Text Summarization



Agenda

9

1. Content-based Explanations exploiting

the Linked Open Data cloud



Agenda

Cataldo Musto, Fedelucio Narducci, Pasquale Lops, Marco de Gemmis,

Giovanni Semeraro: ExpLOD: A Framework for Explaining Recommendations

based on the Linked Open Data Cloud.

Proceedings of RecSys 2016: pp. 151-154 (Best Paper Nominee)

10

2. Review-based Explanation exploiting




Agenda

Cataldo Musto, Pasquale Lops, Marco de Gemmis, Giovanni Semeraro:

Justifying Recommendations through Aspect-based Sentiment Analysis of

Users Reviews.

Proceedings of ACM UMAP 2019: pp. 4-12

11

3. Review-based Explanations exploiting




Agenda

Cataldo Musto, Gaetano Rossiello, Marco de Gemmis, Pasquale Lops, Giovanni

Semeraro: Combining Text Summarization and Aspect-based Sentiment

Analysis of Users’ Reviews to Justify Recommendations.

To be presented at ACM RecSys 2019☺

12

Content-based Explanations exploiting the

Linked Open Data cloud



1.

13

Intuition



Descriptive features of

the items can be freely

gathered from

knowledge graphs as

DBpedia

(http://dbpedia.org)

14

http://dbpedia.org/

Properties from DBpedia

The Ring

Ghost Films

Hans

Zimmer

Naomi

Watts

Psychological

Horror Films

Films shot

in California

Horror

Movies

Japanese

MoviesGore

Verbinski

dcterms:subject dbo:starring

dcte

rms:s

ub

ject

dcte

rms:s

ub

ject



Methodology



ExpLOD Framework



ExpLOD: Mapper



ExpLOD: Mapper

Mapper

Profile Recommendations

dbp:The_Ring_(2002_film)dbp:21_grams

Profile Recommendation



ExpLOD: Builder



ExpLOD: Builder

Recommendation

American

Films

Psychological

Movies

Films about

Ghosts

Naomi

Watts



ExpLOD: Ranker



ExpLOD: Ranker

items in the

user profile and in

the recommendation list

property

number of edges

connecting the property c

with the items in

the user profile

number of edges

connecting the property c

with the items in

the recommendation set



ExpLOD: Ranker

Recommendation

American

Films

Psychological

Movies

Films about

Ghosts

Naomi

Watts



ExpLOD: Generator



ExpLOD: Generator



Naomi

Watts

I suggest you The Ring because you

often like movies with Naomi Watts

as 21 grams and Mulholland Drive.

27

ExpLOD: Generator



Naomi

Watts

Films about

Ghosts

Furthermore, you like films about

ghosts such as The Sixth Sense.

28

ExpLOD: final output

Recommendation

I suggest you The Ring because

you often like movies with

Naomi Watts as 21 grams and

Mulholland Drive. Furthermore,

you like films about ghosts such

as The Sixth Sense.



Experimental EvaluationResearch Question How does our explanations perform with respect to other explanation strategies?

Experimental DesignUser Study with a Web Application

308 subjects, Movie Domain

Metrics: Transparency, Engagement, Persuasion, Trust, Effectiveness [^]

Between-subjects experiment

Configurations: ExpLOD, popularity-based baseline, non-personalized baseline

[^] Tintarev, N., & Masthoff, J. Designing and evaluating

explanations for recommender systems. In Recommender

systems handbook. pp. 479-510. Springer, Boston, MA. 2011



Experimental Protocol



1. Gathering movie preferences

Users rated their favourite movies

31




2. Recommendation is obtained

Personalized PageRank as algorithm



32






3. Explanation is generated

Random Configuration (users not aware)



33








3. Explanation is generated

Random Configuration (users not aware)

4. Metrics are calculated

Through Questionnaires

34

Explanations - Results



MOVIES ExpLOD Non-personalized Popularity

transparency 4.18 3.04 3.01

persuasion 3.41 2.84 2.59

engagement 3.48 3.28 2.31

trust 3.39 2.81 2.67

effectiveness 0.72 0.66 0.93

35








trust 3.39 2.81 2.67


«I recommend you The Ring since you should like movies by

Gore Verbinski whose music composer is Hans Zimmer»

36








trust 3.39 2.81 2.67


«I recommend you The Ring since it is very

popular in the community»

37








trust 3.39 2.81 2.67


Significant improvement for 4 out of 5 metrics

38




Aim Question

transparencyI understood why this movie was

recommended to me

topic

director

distributor

composer

persuasionThe explanation made the

recommendation more convincing

awards

director

location

producer

engagementThe explanation helped me discover new

information

writer

director

producer

distributor

trustThe explanation increased my trust in the

recommender system

awards

composer

producer

topic

effectiveness I like this recommendationdirector

writer

location

composer

39

Review-based Explanation exploiting




2.

40

Intuition

To identify relevant and distinguishing

characteristics of the recommended

item by mining users’ reviews



Intuition

Intense thriller

Pretty good casting

Well-plotted investigation

Impressive horror

......

To identify relevant and distinguishing

characteristics of the recommended

item by mining users’ reviews



Workflow

43Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies.


Aspect Extraction



Aspect ExtractionGoal: to identify the aspects that are

discussed when people talk about the item

Strategy: to use natural language

processing techniques (specifically, part-

of-speech tagging) to extract the names

mentioned in users’ reviews



Aspect Extraction

reviews aspects

Input: reviews of the item i R = {ri1, ri2 … rin}

Output: aspects A = {ai1, ai2 … aik}



Aspect Extraction

reviews aspects

Input: reviews of the item i R = {ri1, ri2 … rin}

Output: aspects A = {ai1, ai2 … aik}

Why only names?

Findings from previous

work in the area

Why no bigrams?

No significant

improvement emerged



Aspect Ranking



Aspect Ranking

Goal: to identify distinguishing aspects that are discussed

with a positive sentiment when people talk about the item



Aspect Ranking



How many times aspect ‘a’ appears in the

reviews of item ‘i’ (frequency of the aspect)



Aspect Ranking





How positive is the opinion of the users

when they talk about aspect ‘a’ (opinion

towards the aspect)



Aspect Ranking





How positive is the opinion of the users

when they talk about aspect ‘a’ (opinion

towards the aspect)

How distinguishing is

the aspect ‘a’ (inverse

popularity)



Aspect Ranking



Intuition: our formula gives an higher score to the aspects that are

frequently mentioned in the reviews with a positive sentiment.

Moreover, it also rewards less popular aspects (higher IAF).



Aspect Ranking

aspects top-k aspects

Input: aspects A = {ai1, ai2 … aim}

Output: top-k aspects A = {ai1, ai2 … aik}



Generation



GenerationGoal: to generate a template-based natural language

justification that relies on the most relevant aspects

identified by the ASPECT RANKING module.



GenerationGoal: to generate a template-based natural language

justification that relies on the most relevant aspects

identified by the ASPECT RANKING module.

For each aspect ’a’ returned by the ASPECT RANKING moduleBrowse the available reviews

Look for a compliant excerpt containing ‘a’If the sentence has a positive sentiment

Add the sentence to the justification

Strategy



GenerationQuestion: when does an excerpt is a compliant sentence?

Answer: an excerpt is compliant if it follows one of the 18

justification patterns we defined

Example: the excerpt must have a third personal

singular verb



GenerationQuestion: when does an excerpt is a compliant sentence?

“I really liked the cast” Not compliant

“The cast was great” Compliant

Example: the excerpt must have a third personal

singular verb



Answer: an excerpt is compliant if it follows one of the 18

justification patterns we defined

Generation – Final Output

I recommend you The Ring because people who

liked the movie think that it delivers some bone-

chilling terror. Moreover, people liked The Ring

since the casting is pretty good.

LegendaRed: randomized template sentences

Green: recommendation

Blue: aspects (k=2)

Black: compliant excerpts



Experimental EvaluationResearch Question 1How effective are the justifications generated through the pipeline, on varying of different

combinations of the parameters?

Research Question 2How does our justifications perform with respect to a classic feature-based explanation?


286 subjects

Movie and Books Domain

Metrics: Transparency, Engagement, Persuasion, Trust, Effectiveness

Between-subjects for Research Question 1, Within-subjects for Research Question 2



Experimental EvaluationParameters of the system- Length of the justifications (short vs. long justifications)

short→ top-2 aspects long -> top-4 aspects

- Vocabulary of aspects (statics vs. complete)

static→ bounded to a fixed and pre-defined set of relevant aspects. No aspect

extraction, just aspect ranking

complete→ not bounded. All the aspects are discovered by the Aspect Extractor

- Four different configurations

Implementation DetailsRecommendations generated through Personalized PageRank, aspect extraction through

CoreNLP POS-tagger and sentiment analysis through Stanford algorithm




Recommendation



Experimental Protocol (Research Question 1)

Recommendation

Review-based

Explanation

Questionnaire



Experimental Protocol (Research Question 2)

I propose you “Aliens”

because you sometimes like

movies edited by Canadian

film editors, American fiction

films and 1980s films, as The

Terminator.

I recommend you “Aliens”

because people who liked this

movie think that the Alien

series is one of the best sci-fi

movies and that the ending is

awesome with some fantastic

action scenes.

Review-based

Explanation

Feature-based

Explanation



Results (Research Question 1)

MOVIES Transparency Persuasion Engagement Trust Effectiveness

Static Short 3.40 3.13 3.09 3.23 0.64

Static Long 3.77 3.68 3.55 3.73 0.55

Complete Short 3.91 3.60 3.25 3.70 0.53

Complete Long 3.74 3.48 3.35 3.46 0.59




Finding 1With the ‘complete’ set of

aspects, shorter justifications

have the best results

Finding 2With the ‘static’ set of

aspects, longer justifications

have the best results

OverallLong justifications based on

static aspects have the best

results in the Movie Domain


Static Short 3.40 3.13 3.09 3.23 0.64

Static Long 3.77 3.68 3.55 3.73 0.55

Complete Short 3.91 3.60 3.25 3.70 0.53

Complete Long 3.74 3.48 3.35 3.46 0.59




MOVIESReview-

based

Feature-

basedIndiffer.

Transparency 47.4% 38.6% 14.0%

Persuasion 51.7% 43.3% 5.0%

Engagement 66.7% 25.0% 8.3%

Trust 53.3% 35.5% 11.7%

Effectiveness 57.9% 35.0% 7.1%

Outcome: Users preferred Review-based JustificationsConfirmed for all the metrics and both the domains



Review-based Explanations exploiting




3.

70

Why do we need another

approach that exploits

users’ reviews?



Motivations

71

Why do we need another

approach that exploits

users’ reviews?



Motivations

Our first methodology has

two main weaknesses

• Very naïve strategy for ASPECT EXTRACTION

• Very static template-based GENERATION

72

To exploit automatic text summarization

techniques to build an higher-quality justifications.



Intuition

73

To exploit automatic text summarization

techniques to build an higher-quality justifications.

We conceive our justification as a summary of the

information conveyed by all the available reviews.



Intuition

74

Workflow



Workflow



Same conceptual

workflow, different

implementations of

the modules!

76

Aspect ExtractionStatistical approach based on the Kullback-Leibler

(KL) Divergence



Measures the difference between the distribution of a term

in a generic corpus (e.g. BNC) and its distribution in a domain corpus

(e.g. movie reviews)

77

Aspect Extraction




Insight: the higher the divergence, the higher the

importance of the term in the domain

t = term

ca = corpus A

cb = corpus B



Statistical approach based on the Kullback-Leibler

(KL) Divergence

78

Aspect Extraction








KL(cast, BNC, movie-reviews) >> 0

KL(actor, BNC, movie-reviews) > 0

KL(city, BNC, movie-reviews) ~ 0

KL(woman, BNC, movie-reviews) ~ 0

We label as ‘aspects’ the

nouns whose

KL-divergence is higher

than zero


(KL) Divergence

79

Aspect Extraction








KL(cast, BNC, movie-reviews) >> 0 YES

KL(actor, BNC, movie-reviews) > 0 YES

KL(city, BNC, movie-reviews) ~ 0 NO

KL(woman, BNC, movie-reviews) ~ 0 NO


(KL) Divergence

80

Aspect Ranking



Aspect Ranking



Novelty: KL-divergence is used as relevance score rela,Ri



Generation



Same conceptual

workflow, different

implementations of

the modules!

83

Generation

Intuition: we conceive our justification as a summary of the

information conveyed by all the available reviews



Generation

Intuition: we conceive our justification as a summary of the

information conveyed by all the available reviews

Approach: we exploited a centroid-based method for automatic text

summarization. Very good performance in multi-document

summarization scenarios.



Assumption: each review can be considered as ‘document’ thus the

corpus of the reviews can be used to feed the algorithm

85

Generation

Generation process is in turn split into two steps

• Sentence Filtering

• Text Summarization

Sentence Filtering is used to feed the summarization algorithm

with compliant sentences. We selected sentences that matched

the following criterions:



Generation

Generation process is in turn split into two steps

• Sentence Filtering

• Text Summarization

Sentence Filtering is used to feed the summarization algorithm

with compliant sentences. We selected sentences that matched

the following criterions:



• The sentence contains a main aspect

• The sentence is longer than 5 tokens

• The sentence expresses a positive sentiment

• The sentence does not contain first-person personal or possessive pronouns

87

Generation

Text Summarization Algorithm



Input: item i, sentences s1…sn, word limit kOutput: summary for item i consisting of k words

1. Build a vector space representation for each sentence2. Merge all the sentences in a pseudo-document that represents the

item (centroid vector)3. Until the word limit is not reached

3.1 Go through the sentences and calculate cosine similarity3.2 Pick the one with the highest cosine similarity to the

centroid (and not that similar to the sentences previously picked)

3.3 Add it to the summary

88

Generation










89

Generation










90

Generation










91

Generation










92

Generation – Final Output

“If you like or love the blood and gore kinds of films,

this movie will certainly disappoint you as the focus is

on character, story, mood and unique special effects.

The Ring is a story about supernatural evil therefore,

it is a horror film, done very much in the style of the

psychological thriller.”

LegendaRed: aspects (k=4)

Black: compliant excerpts



Experimental EvaluationResearch Question 1How effective are the justifications generated through the pipeline, on varying of different

combinations of the parameters?

Research Question 2How does our justifications perform with respect to a simple review-based explanation?


141 subjects

Movie Domain. 300 movies. ~150k reviews.

Metrics: Transparency, Engagement, Persuasion, Trust, Effectiveness [^]

Parameters: Justification Length (Short=50 words, Long=100) and #Aspects (10 and 30).

Between-subjects for Research Question 1, Within-subjects for Research Question 2

[^] Tintarev, N., & Masthoff, J. Designing and evaluating

explanations for recommender systems. In Recommender

systems handbook. pp. 479-510. Springer, Boston, MA. 2011





Top-10 Short 2.83 3.06 3.06 2.83 0.89

Top-30 Long 3.16 3.06 2.69 3.19 0.94

Top-10 Short 3.95 3.64 3.37 3.55 0.55

Top-30 Long 3.24 3.18 3.12 3.22 0.38



Finding 1

Long justifications better

than short justifications,

on average

Finding 2

Top-10 aspect provide

better explanations than

Top-30 aspects

Finding 3

Long explanations based

on Top-10 aspects lead to

the best results

95


MOVIESReview+

Summary

Review-

basedIndiffer.

Transparency 54.5% 40.9% 4.6%

Persuasion 77.3% 13.6% 9.1%

Engagement 63.6% 27.3% 9.1%

Trust 68.2% 4.5% 27.3%

Outcome: automatic Text Summarization provides users with the best explanation

Confirmed for all the metrics and both the domains





Recap and Take Home Messages

Recap



“If you like or love the blood and gore kinds of films,

this movie will certainly disappoint you as the focus

is on character, story, mood and unique special

effects. The Ring is a story about supernatural evil

therefore, it is a horror film, done very much in the

style of the psychological thriller.”

I recommend you The Ring because people who

liked the movie think that it delivers some bone-

chilling terror. Moreover, people liked The Ring since

the casting is pretty good.

I suggest The Ring because you

often like movies with Naomi Watts as 21 grams

and Mulholland Drive. Furthermore, you like films

about ghosts such as The Sixth Sense.

Feature-based

explanation

exploiting DBpedia

Review-based

explanation using

sentiment analysis

Review-based

explanation using

automatic text

summarization

98

Take-home Messages



99

1.

2.

All the methodologies can provide the users with satisfying

explanations, that can support the suggestions returned by a

generic recommendation algorithm

How to choose the most suitable one?

Available data and explanation aims have to drive the choice!

Feature-based: easier approach, good transparency;

Review-based: improves the persuasion and the engagement;

Summarization-based: more sophisticated generation, good for

long-term usage of the explanation facilities.

Thank you!

[email protected]

@cataldomusto

Contacts



advances in content-based recommender systems explanation...

Documents