1 algorithms for large data sets ziv bar-yossef lecture 7 april 20, 2005

1

Algorithms for Large Data Sets

Ziv Bar-YossefLecture 7

April 20, 2005

http://www.ee.technion.ac.il/courses/049011

2

Rank Aggregation

3

Outline

The rank aggregation problem Applications Desired properties Arrow’s impossibility theorem Rank aggregation methods

4

The Rank Aggregation Problem

m candidates (a.k.a. “alternatives”) M = {1,…,m}: set of candidates

n voters (a.k.a. “agents” or “judges”) N = {1,…,n}: set of voters

Each voter i, has an ranking i on M i(a) < i(b) means i-th voter prefers a to b Ranking may be a total or partial order

The rank aggregation problem:Combine 1,…,n into a single ranking on M, which represents the “social choice” of the voters. Rank aggregation function: f(1,…,n) = may be a total or partial order

5

Examples

m small, n large: elections (multi-party parliament, academies, boards,...)

m modest, n small: program committees, sports

m large, n small: meta-search, travel plans, restaurant selection

6

Applications to Web Search

Meta search Combine results of different search engines into a better overall

ranking Combat spam

Spam results unlikely to rank high in aggregate ranking, even though they can rank high in one or two search engines.

Search for multiple terms AND: bad recall OR: bad precision Complex boolean queries: too complicated for average user Solution: search for small subsets of terms and aggregate results

Combine multiple ranking functions Use different ranking functions (e.g., VSM, PageRank, HITS, …)

and aggregate them into a single function

7

Applications to Databases

Rank items in a database according to multiple criteriaEx: Choose a restaurant by cuisine, distance,

price, quality, etc.Ex: Choose a flight ticket by price, # of stops,

date and time, frequent flier bonuses, etc.

8

Desired Properties: Unanimity

Unanimity (a.k.a. Pareto optimality):

If all voters prefer candidate a to candidate b (i.e., i(a) < i(b) for all i), then also should prefer a to b (i.e., (a) < (b)).

aca

bac

cbb

a:b = 3:0

9

Desired Properties: Condorcet Condorcet Criterion [Condorcet, 1785]:

Condorcet winner: a candidate a, which is preferred by most voters to any other candidate b (i.e., for all b, # of i s.t. i(a) < i(b) is at least n/2).

Condorcet criterion: If Condorcet winner exists, should rank it first (i.e., (a) = 1).

abc

baa

ccb

a:b = 2:1, a:c = 2:1

abc

bca

cab

No Condorcet winner

10

Desired Properties: XCC Extended Condorcet Criterion (XCC):

If most voters prefer candidate a to candidate b (i.e., # of i s.t. i(a) < i(b) is at least n/2), then also should prefer a to b (i.e., (a) < (b)).

Not always realizable

abc

baa

ccb

a) < (b) < (c)

abc

bca

cab

Not realizable

11

XCC and Spam [Dwork et al. 2001]

Definition: a page p is said “spam” to a ranking , if there is a page q ranked lower than p, which most human evaluators will think should be ranked higher than p.

Assumption: for any two pages p,q, majority of human evaluators agrees with majority of search engine rankings on the order of p,q.

Conclusion: Spam pages are always “Condorcet losers” If rank aggregation function respects XCC, it eliminates

spam.

12

Desired Properties: Independence from Irrelevant Alternatives

Independence from Irrelevant Alternatives:

Relative order of a and b in should depend only on relative order of a and b in 1,…,n.Ex: if i = (a b c) changes to (a c b), relative

order of a,b in should not change.

13

Desired Properties: Neutrality and Anonymity

NeutralityNo candidate should be favored to others. If two candidates switch positions in 1,…,n, they

should switch positions also in .

AnonymityNo voter should be favored to others. If two voters switch their orderings, should remain

the same.

14

Desired Properties: Monotonicity and Consistency Monotonicity

If the ranking of a candidate is improved by a voter, its ranking in can only improve.

ConsistencyIf voters are split into two disjoint sets, S and T, and both the aggregation of voters in S and the aggregation of voters in T prefer a to b, then also the aggregation of all voters should prefer a to b.

15

Dictatorship and Democracy

Dictatorship: f(1,…,n) = i

Democracy (a.k.a. Majoritian aggregation):Use extended Condorcet Criterion to rank candidates.Always works for m = 2.Not always realizable for m ≥ 3.Theorem [May, 1952]: For m = 2, Democracy is

the only rank aggregation function which is monotone, neutral, and anonymous.

16

Arrow’s Impossibility Theorem [Arrow, 1951] Theorem: If m ≥ 3, then the only rank

aggregation function that is unanimous and independent from irrelevant alternatives is dictatorship.Won Nobel prize (1972)

17

Positional Rank Aggregation Methods Plurality

score(a) = # of voters who chose a as #1 : order candidates by decreasing scores

Top-k approval score(a) = # of voters who chose a as one of the top k : order candidates by decreasing scores

Borda’s rule [Borda, 1781] score(a) = i i(a) : order candidates by increasing scores

Violate independence from irrelevant alternatives

18

Positional Methods: Example

aacb

bdbd

ccdc

dbaa

PluralityTop-2 ApprovalBorda

a221+1+4+4=10

b132+4+2+1=9

c113+3+1+3=10

d024+2+3+2=11

19

Optimal Rank Aggregation

d: distance measure among rankings Definition: The optimal rank aggregation

for 1,…,n w.r.t. d is the ranking which minimizes i d(,i).

2

1n

20

Distance Measures

Kendall tau distance (a.k.a. “bubble sort distance”)K(,) = # of pairs of candidates (a,b) on which

and disagreeEx: K( (a b c d), (a d c b)) = 0 + 2 + 1 = 3

Spearman footrule distanceF(,) = a |(a) - (a)|Ex: F((a b c d), (a d c b)) = 0 + 2 + 0 + 2 = 4

21

Kemeny Optimal Aggregation[Kemeny 1959] Optimal aggregation w.r.t. Kendall-tau distance

Theorem [Young & Levenglick, 1978] [Truchon 1998]: Kemeny optimal aggregation is the only rank aggregation function, which is neutral, consistent, and satisfies the Extended Condorcet principle. Effective for fighting spam

Generative model: * is the “correct” ranking 1,…,n are generated from by swapping every pair with probability < ½. Then: Kemeny optimal aggregation gives the maximum likelihood given

1,…,n. [Young 1988]

22

Complexity of Kemeny Optimal Aggregation NP-hard, even for n = 4 [Dwork et al. 2001]

In P, for n = 2. Unknown for n = 3.

Can be approximated using Spearman footrule: Proposition [Diaconis-Graham]:

K(,) ≤ F(,) ≤ 2 K(,)

What is the complexity of footrule optimal aggregation?

23

Footrule Optimal Aggregation

Theorem [Dwork et al. 2001]Footrule optimal aggregation can be computed in polynomial time.

Proof Want to find which minimizes i a |(a) - i(a)| Define a weight bipartite graph G = (L,R,W) as follows:

L = M (the candidates) R = {1,…,m}: the available ranks W(a,r) = i |r - i(a)|

A matching in G = ranking Cost of a matching: i a |(a) - i(a)| Hence, reduced to finding a minimum cost matching in

a bipartite graph

24

Local Kemenization [Dwork et al. 2001] Definition: A ranking is locally Kemeny optimal

aggregation for 1,…,n if there is no other ranking ’, which: Can be obtained from by flipping one pair Satisfies i K(’, i) < i K(,i)

Features: Every Kemeny optimal aggregation is also locally

Kemeny optimal, but converse is not necessarily true. Locally Kemeny optimal aggregations satisfy XCC. Locally Kemeny optimal aggregations can be

computed in O(n m log m) time.

25

Markov Chain Techniques[Dwork et al. 2001] Markov Chain states = candidates Transitions depend on the voter rankings Basic idea: probabilistically switch to a

“better” candidate Final ranking: induced by stationary

distribution

26

Four MC Methods

Current state is candidate a. MC1: Choose uniformly from multiset of all candidates that

were ranked at least as high as a by some voter. Probability to stay at a: ~ average rank of a.

MC2: Choose a voter i u.a.r. and pick u.a.r. from among the candidates that the i-th voter ranked at least as high as a.

MC3: Choose a voter i u.a.r. and pick u.a.r. a candidate b. If i-th voter ranked b higher than a, go to b. Otherwise, stay in a.

MC4: Choose a candidate b u.a.r. If most voters ranked b higher than a, go to b. Otherwise, stay in a. Rank of a ~ # of “pairwise contests” a wins.

27

End of Lecture 7