v jornadas emadrid sobre “educación digital”. roberto centeno, universidad nacional de...

On the inaccuracy of numerical ratings: A pairwise based reputation

mechanism in MOOCs

July 1st, 2015

Roberto [email protected]

Dpto. Lenguajes y Sistemas InformáticosUNED

mailto:[email protected]

Outline

1. Introduction

2. Motivation

3. From opinion ratings to pairwise queries: PWRM

4. Towards ranking resources in MOOCs

5. Conclusions & future work

2

Introduction

3

Social Networks (Online Review Systems) & Reputation

4

1. Introduction


4

1. Introduction

Social Networks (Online Review Systems) Reputation

many types: professional links, friendships, purchases, ...complex: dynamism, complexity of the social structure, many nodes (users, entities, ..)how can we identify and locate appropriate entities/services to consume? (more and more available information online)Not enough experience so.. online review systems a (Yelp, Tripadvisor, ..) as a means to obtain opinions, rankings, etc…


4

1. Introduction

Social Networks (Online Review Systems)

Reputation

many types: professional links, friendships, purchases, ...complex: dynamism, complexity of the social structure, many nodes (users, entities, ..)how can we identify and locate appropriate entities/services to consume? (more and more available information online)Not enough experience so.. online review systems a (Yelp, Tripadvisor, ..) as a means to obtain opinions, rankings, etc…


4

1. Introduction


Reputation

TrustReputation

opinions of third parties

Confidencelocal experiences

many typescomplexnodes (users, entities, ..)how can we identify and locate appropriate entities/services to consume?Not enough experience so.. Tripadvisor, ..) as a means to obtain opinions, rankings, etc…


4

1. Introduction


Reputation

TrustReputation

opinions of third parties

Confidencelocal experiences

objective: extract reputation of entities (users, objects, …)how: gathering and aggregating opinionsexamples:

5

1. Introduction

Reputation Mechanisms in Social Networks

Reputation Mechanisms

objective: how: examples:

5

1. Introduction

Reputation Mechanisms in Social Networks

Reputation Mechanisms

Motivation

6

passive: expecting users’ opinionscapturing opinions through numerical ratings + textual informationestimate reputation based on aggregating ratings (average ratings)

7

2. Motivation

Reputation Mechanisms (traditionally)…

Capturing preferences through numerical opinions


7

2. Motivation


PROBLEM 1

‣DIFICULT TO MAP PREFERENCES INTO NUMERICAL OPINIONS‣SUBJECTIVITY!!!



7

2. Motivation


AGR



7

2. Motivation


AGRPROBLEM 2

BIAS PROBLEMS!!!



7

2. Motivation


AGRDie Hard 1

Gone with the wind 2

Ben-Hur 3

Ben-Hur ≻ Gone with the wind ≻ Die Hard

Die Hard 3


Ben-Hur 4

Ben-Hur ≻ Die Hard ≻ Gone with the wind



7

2. Motivation


AGRDie Hard 1


Ben-Hur 3


Die Hard 3


Ben-Hur 4





7

2. Motivation


AGRDie Hard 1


Ben-Hur 3

Ben-Hur ≻ Gone with the wind ≻ Die Hard Ben-Hur ≻ Die Hard ≻ Gone with the wind


Die Hard 0.2

Gone with the wind 0.1

Ben-Hur 1.0


8

2. Motivation

best model fitting VS best fit to a normal distribution

Bias problems derived from numerical ratings

Analyzing the distribution of average ratings from HetRec-2011 dataset

8

2. Motivation




average rating distribution of the 100, 1.000 and 4.000 most reviewed movies by users

8

2. Motivation




average rating distribution of the 100, 1.000 and 4.000 most reviewed movies by users

clearly biased to positive ratings

8

2. Motivation




average rating distribution of the 100, 1.000 and 4.000 most reviewed movies given by top critics

8

2. Motivation





not influenced and biased by others’ ratings

8

2. Motivation





not influenced and biased by others’ ratings

PROBLEM CONFIRMATION

potential bias problems when mapping opinions onto numerical values, reputation

rankings may vary; and likely to cause differences between the true quality of an

entity and its rating aggregated from opinions

9

2. Motivation

FROMReputation Rankings:

passive method + numerical opinions + opinions aggregation (average ratings)

Solution proposed

9

2. Motivation



Reputation Rankings:

pro-active method + comparative opinions + comparative aggregationTO

Solution proposed

9

2. Motivation



Reputation Rankings:

pro-active method + comparative opinions + comparative aggregation

Pairwise preference elicitation

Aggregation mechanism

TO

Solution proposed

From opinion ratings to pairwise queries: PWRM

10

11

3. From opinion ratings to pairwise queries: PWRMComparative opinions: Pairwise preference elicitation

Based on pairwise queries:

11



FROMBen-Hur

[0..1][Awful, fairly bad, It’s OK,

Will enjoy, Must see]

Gone withthe wind

:15. Ben-Hur 4.3

:23. Gone with the wind 4.1

:

11



FROMBen-Hur

[0..1][Awful, fairly bad, It’s OK,

Will enjoy, Must see]

Gone withthe wind

:15. Ben-Hur 4.3

:23. Gone with the wind 4.1

:

TO Which movie do you prefer, Ben-Hur or Gone with the wind?

:15. Ben-Hur

:23. Gone with the wind

:

easier for users to state opinions when the queries compare objects in a pairwise fashion…

“… between these two objects, which one do you prefer?”

12

3. From opinion ratings to pairwise queries: PWRMPairwise comparison dynamics (I)

Reputation (opinions aggregation) as an iterative process based on …

12


Knock-Out tournaments:


12


Knock-Out tournaments:


A

B

C

D

A

D

D

Match: pairwise comparison between two entities

Dynamics: every match sent to a set of users that reply to the query

Policies:‣ Entity selection‣ Tournament schedule‣ Users selection‣ Winner determination

13

3. From opinion ratings to pairwise queries: PWRMPairwise comparison dynamics (II)

Policies

Entities selection: which pair of entities should I select to be compared?‣ Mixed: current ranking vs new objects (Exploitation Vs Exploration)‣ Random‣ Domain-dependent: objects with no information/fuzzy positions

Tournament schedule: how to initialize the tournament‣ Random schedule (iterative process)

14

3. From opinion ratings to pairwise queries: PWRMPairwise comparison dynamics (III)

Policies

Users selection: who receives the queries (matches)?‣ Random selection‣Clustering of users by their preferences (representative users)‣ Using (social) network properties: degree distribution, centrality of nodes, …

Winner determination: how to decide which entity wins in a match‣ Voting procedures: preference replies from users count as votes‣ Alternatives: absolute majority / full agreement (voting protocols)‣ If there is no winner, no object gets through the next round

15

3. From opinion ratings to pairwise queries: PWRMComparative aggregation: from matches to a ranking

When: After each match, the ranking is updated (iterative method)How: Adaptation of a method for aggregating partial pairwise comparison results into a ranking (Negahban et al., 2012)

‣Ranking approximation = random walk on G (weighted graph):‣ An edge <ei,ej> if the pair has already been compared‣ The weights define the outcome of the comparisons‣ Random walk uses a transition matrix P where:‣ It moves from state ei to state ej with probability equal to the chance that entity ej is preferred over entity ei

‣ Under these conditions, a vector w is a valid stationary distribution for matrix P (wT

t+1 = wT · P)‣ w defines the scores for each entity => ranking

16

3. From opinion ratings to pairwise queries: PWRMPWRM’s iterative process for building a reputation ranking

Require: a social network G = (U,E,LU , LE)

Require: a subset of E

0 ✓ E entities to be evaluated

1: for t 2 � time do

2: Ei EntitiesSelectionPolicy.selectEntitiesToEvaluate(E

0)

3: KTEi scheduleTournament(Ei)

4: for m 2 matches(KTEi) do

5: nb UsersSelectionPolicy.getUsersToAsk(U)

6: send(m,nb)

7: votes receive()

8: winner WinnerDeterminationPolicy.getWinner(votes)

9: Ri AggregationMechanism.updateRanking(m,winner)

10: setWinnerNextRound(winner,KTE0)

11: end for

12: end for

13: return Ri where E

0are ranked by their estimated reputation

Towards ranking resources in MOOCs

17

18

4. Towards ranking resources in MOOCsOpinions in MOOCs?

18

4. Towards ranking resources in MOOCsOpinions in MOOCs?

opinions to rank courses/resources

19

4. Towards ranking resources in MOOCsApplying PWRM into MOOCs

‣MOOCs modeled as a Social Network (Online Review Systems)

‣Apply PWRM for ranking learning resources in MOOCs

‣Allowing users (students/teachers) to find the best resources

‣Formalize a MOOC from a peer based system point of view

Idea:

Let M = hU,R,LR, LU i be a MOOC, where:

• U = {u1, . . . , un} is a set of users (teachers or students);

• R = {r1, . . . , rm} is the set of learning resources uploaded in the course;

• LR = {hui, rji/ui 2 U ; rj 2 R} is the set of links among users and re-sources, representing that user ui has uploaded the resource rj in thecourse;

• LU = {huk, rmi/uk 2 U ; rm 2 R} is the set of links also between usersand resources representing that user uk has used the resource rm.

20

4. Towards ranking resources in MOOCsPWRM as a function

‣ Objective: to query users about resources to give their opinions in order to build a global ranking of resources regarding their reputation, so..

M = hU,R,LR, LU , ranki

• rank : R0 ⇥ O ! {1, . . . , |R0|} is a function in charge of defining a totalordering (ranking) over a subset of resources R0 2 R, taking into accountthe set of opinions O given by users;

• oi 2 O represents an opinion given into a MOOC, modeled as oi = hri, rjiand representing a pairwise query sent to a set of users participating inthe MOOC, where learning resources ri and rj are compared.

20

4. Towards ranking resources in MOOCsPWRM as a function

‣ Objective: to query users about resources to give their opinions in order to build a global ranking of resources regarding their reputation, so..

M = hU,R,LR, LU , ranki

• rank : R0 ⇥ O ! {1, . . . , |R0|} is a function in charge of defining a totalordering (ranking) over a subset of resources R0 2 R, taking into accountthe set of opinions O given by users;

• oi 2 O represents an opinion given into a MOOC, modeled as oi = hri, rjiand representing a pairwise query sent to a set of users participating inthe MOOC, where learning resources ri and rj are compared.

rank = PWRM

21

4. Towards ranking resources in MOOCsPWRM algorithm in MOOCs

Require: a MOOC M = hU,R,LR, LU , rankiRequire: a subset of R0 ✓ R of learning resources to be ranked1: for t 2 � time do

2: Ri ResourcesSelectionPolicy.selectResourcesToEvaluate(R0)3: KTRi scheduleTournament(Ri)4: for m 2 matches(KTRi) do5: Ui UsersSelectionPolicy.getUsersToAsk(U)6: send(m,Ui)7: Oi ReceiveOpinions()8: winner WinnerDeterminationPolicy.getWinner(votes,Oi)9: Ranki AggregationMechanism.updateRanking(Oi, winner)

10: promoteResourceWinnerToNextRound(winner,KTRi)11: end for

12: end for

13: return Ranki where the subset R0 of learning resources are ranked by their

reputation

22

4. Towards ranking resources in MOOCsPWRM algorithm in MOOCs: Policies

‣ Resource selection policy:

- resources clustered regarding their typology (e.g. videos, recorded class…)

- regarding the number of opinions received by each resource (lowest/highest number)

- opinions in terms of the result of each match (matches with tight results)

‣ User selection policy:

- taking advantage of the underlying structure generated by interactions between users and resources

‣ Winner determination policy:

- voting theory: simple majority, complete agreement, …

Conclusions & Future Work

23

24

Contributions

Current reputation mechanisms

✓ follow a very passive/static and quantitative dependent approach

‣ easy to manipulate

‣ bias problems due to difficulty/subjectivity to map opinions into numerical values

Our Approach: PWRM (1) based on comparative(2) preference aggregation in reputation rankings (iterative process - tournaments)(3) applied to MOOCs (ranking learning resources)

5. Conclusions & Future work

24

Contributions

Current reputation mechanisms

✓ follow a very passive/static and quantitative dependent approach

‣ easy to manipulate

‣ bias problems

Our Approach: PWRM (1) based on comparative opinions, elicited through pairwise preference request(2) preference aggregation in reputation rankings (iterative process - tournaments)(3) applied to MOOCs (ranking learning resources)

5. Conclusions & Future work

25

Future work5. Conclusions & Future work

‣ Adding social network properties:

- cluster users, centrality, betweenness, …

‣ Partial cooperative users:

- incentive mechanisms fostering cooperation (“what do you think users prefer, A or B?”)

‣ Reputation of MOOCs:

- resources = courses, finding opinions in other opinions sites: twitter, Facebook, forums, etc..

‣ Individual recommendation:

- resources/courses: from global reputation ranking to individual recommendations

That’s all

Thank you for your attention!!

26

On the inaccuracy of numerical ratings: A pairwise based reputation

mechanism in MOOCs

July 1st, 2015

Roberto [email protected]

Dpto. Lenguajes y Sistemas InformáticosUNED

mailto:[email protected]

v jornadas emadrid sobre “educación digital”. roberto centeno, universidad nacional de...

Education

users opinions

numerical opinions

based reputation mechanism

social structure

aggregating opinions

opinion ratings

types complex nodes

available information