seesaw personalized web search jaime teevan, mit with susan t. dumais and eric horvitz, msr

SeesawPersonalized Web Search

Jaime Teevan, MITwith Susan T. Dumais

and Eric Horvitz, MSR

Query expansion

Personalization Algorithms

Standard IR

Document

Query

User

Server

Client

Query expansion

Personalization Algorithms

Standard IR

Document

Query

User

Server

Client

v. Result re-ranking

Result Re-Ranking

Ensures privacyGood evaluation frameworkCan look at rich user profileLook at light weight user models

Collected on server side Sent as query expansion

Seesaw Search EngineSeesawSeesaw

dog 1cat 10india 2mit 4search 93amherst 12vegas 1

Seesaw Search Engine

query



query


dog cat monkey banana

food

baby infant

child boy girl

forest hiking

walking gorp

baby infant

child boy girl

csail mit artificial research

robotweb

search retrieval ir

hunt


query


1.6 0.26.0

0.2 2.7

1.3

Search results page

web search

retrieval ir hunt

1.3

Calculating a Document’s Score

Based on standard tf.idf

web search

retrieval ir hunt

1.3

Calculating a Document’s Score

Based on standard tf.idf

(ri+0.5)(N-ni-R+ri+0.5)

(ni-ri+0.5)(R-ri+0.5)wi = log

1.30.1 0.5

0.05 0.35 0.3

User as relevance feedback Stuff I’ve Seen index More is better

Finding the Score Efficiently

Corpus representation (N, ni) Web statistics Result set

Document representation Download document Use result set snippet

Efficiency hacks generally OK!

Evaluating Personalized Search

15 evaluatorsEvaluate 50 results for a query

Highly relevant Relevant Irrelevant

Measure algorithm quality

DCG(i) = { Gain(i),DCG(i–1) + Gain(i)/log(i),

if i = 1otherwise

Evaluating Personalized Search

Query selection Chose from 10 pre-selected queries Previously issued query

cancerMicrosofttraffic…

bison friseRed Soxairlines…

Las VegasriceMcDonalds…

Pre-selected

53 pre-selected (2-9/query)

Total: 137

JoeMary

Seesaw Improves Text Retrieval

0

0.1

0.2

0.3

0.4

0.5

0.6

Rand RF SS Web Combo

DC

G

RandomRelevance

FeedbackSeesaw

Text Features Not Enough

0

0.1

0.2

0.3

0.4

0.5

0.6


DC

G

Take Advantage of Web Ranking

0

0.1

0.2

0.3

0.4

0.5

0.6


DC

G

Further Exploration

Explore larger parameter spaceLearn parameters

Based on individual Based on query Based on results

Give user control?

Making Seesaw Practical

Learn most about personalization by deploying a system

Best algorithm reasonably efficientMerging server and client

Query expansion Get more relevant results in the set to be re-ranked

Design snippets for personalization

User Interface Issues

Make personalization transparentGive user control over personalization

Slider between Web and personalized results Allows for background computation

Creates problem with re-finding Results change as user model changes Thesis research – Re:Search Engine

Thank [email protected]

Personalizing Web Search

MotivationAlgorithmsResultsFuture Work

Study of Personal Relevancy

15 participants Microsoft employees Managers, support staff, programmers, …

Evaluate 50 results for a query Highly relevant Relevant Irrelevant

~10 queries per person

Study of Personal Relevancy

Query selection Chose from 10 pre-selected queries Previously issued query

cancerMicrosofttraffic…

bison friseRed Soxairlines…

Las VegasriceMcDonalds…

Pre-selected

53 pre-selected (2-9/query)

Total: 137

JoeMary

Relevant Results Have Low Rank

1 5 9 13 17 21 25 29 33 37 41 45 49

Rank

Highly Relevant

Relevant

Irrelevant

1 5 9 13 17 21 25 29 33 37 41 45 49

Rank

Relevant Results Have Low Rank

Highly Relevant

Relevant

Irrelevant

Rater 1 Rater 2

Same Results Rated Differently

Average inter-rater reliability: 56%Different from previous research

Belkin: 94% IRR in TREC Eastman: 85% IRR on the Web

Asked for personal relevance judgmentsSome queries more correlated than others

Same Query, Different Intent

Different meanings “Information about the astronomical/astrological

sign of cancer” “information about cancer treatments”

Different intents “is there any new tests for cancer?” “information about cancer treatments”

Same Intent, Different Evaluation

Query: Microsoft “information about microsoft, the company” “Things related to the Microsoft corporation” “Information on Microsoft Corp”

31/50 rated as not irrelevant Only 6/31 do more than one agree All three agree only for www.microsoft.com Inter-rater reliability: 56%

Search Engines are for the Masses

Joe Mary

Much Room for Improvement

Group ranking Best improves on

Web by 38% More people

Less improvement

1.2

1.25

1.3

1.35

1.4

1 2 3 4 5 6

Number of People

DC

G

Group

Much Room for Improvement

Group ranking Best improves on

Web by 38% More people

Less improvement

Personal ranking Best improves on

Web by 55% Remains constant

1.2

1.25

1.3

1.35

1.4

1 2 3 4 5 6

Number of People

DC

G

Personalized Group

- Seesaw Search Engine- See- Seesaw



BM25

N

ni

N

ni

wi = log

ri

R

with Relevance Feedback

Score = Σ tfi * wi

N

ni

(ri+0.5)(N-ni-R+ri+0.5)

(ni-ri+0.5)(R-ri+0.5)

ri

R

wi = log

Score = Σ tfi * wi

BM25 with Relevance Feedback

(ri+0.5)(N-ni-R+ri+0.5)

(ni- ri+0.5)(R-ri+0.5)

User Model as Relevance Feedback

N

ni

R

ri

Score = Σ tfi * wi

(ri+0.5)(N’-ni’-R+ri+0.5)

(ni’- ri+0.5)(R-ri+0.5)wi = log

N’ = N+R

ni’ = ni+ri


N

ni

R

ri

World

User

Score = Σ tfi * wi


R

ri

User

N

ni

World

World related to query

Nni

Score = Σ tfi * wi


N

ni

R

ri

World

UserWorld related to query

User related to query

R

Nni

ri

Query Focused Matching

Score = Σ tfi * wi


N

ni

R

ri

World

UserWeb related to query

User related to query

R

N ri

Query Focused Matching

ni

World Focused Matching

Score = Σ tfi * wi

Parameters

Matching

User representation

World representation

Query expansion

Parameters

Matching

User representation


Query expansion

Query focused

World focused

User Representation

Stuff I’ve Seen (SIS) index MSR research project [Dumais, et al.] Index of everything a user’s seen

Recently indexed documentsWeb documents in SIS indexQuery historyNone

Parameters

Matching

User representation


Query expansion

Query focused

World focused All SISRecent SISWeb SISQuery historyNone

Parameters

Matching

User representation


Query expansion

Query Focused

World Focused All SISRecent SISWeb SISQuery HistoryNone

World Representation

Document Representation Full text Title and snippet

Corpus Representation Web Result set – title and snippet Result set – full text

Parameters

Matching

User representation


Query expansion

Query focused


Full textTitle and snippet

WebResult set – full textResult set – title and snippet

Query Expansion

All words in document

Query focused

The American Cancer Society is dedicated to eliminating cancer as a major health problem by preventing cancer, saving lives, and diminishing suffering through ...

The American Cancer Society is dedicated to eliminating cancer as a major health problem by preventing cancer, saving lives, and diminishing suffering through ...

Parameters

Matching

User representation


Query expansion

Query focused




All words

Query focused

Best Parameter Settings

Matching

User representation


Query expansion

Query focused




All words

Query focused

All SISRecent SISWeb SIS

All SISRecent SISWeb SISQuery historyNone

Full text

All words

Query focused

World focused

Result set – title and snippet

Web

Query focused

All SIS

Title and snippet

Result set – title and snippet

Query focused

Seesaw Improves Retrieval

0

0.1

0.2

0.3

0.4

0.5

0.6

None Rand RF SS Web Combo

DC

G

No user model

RandomRelevance

FeedbackSeesaw

Text Alone Not Enough

0

0.1

0.2

0.3

0.4

0.5

0.6


DC

G

Incorporate Non-text Features

0

0.1

0.2

0.3

0.4

0.5

0.6


DC

G

Summary

0

0.2

0.4

0.6

0.8

1

None SS Web Group ?

Rich user model important for search personalization

Seesaw improves text based retrievalNeed other featuresto improve WebLots of room for improvement

future



Further exploration Making Seesaw practical User interface issues

Further Exploration

Explore larger parameter spaceLearn parameters

Based on individual Based on query Based on results

Give user control?

Making Seesaw Practical

Learn most about personalization by deploying a system

Best algorithm reasonably efficientMerging server and client

Query expansion Get more relevant results in the set to be re-ranked

Design snippets for personalization

User Interface Issues

Make personalization transparentGive user control over personalization

Slider between Web and personalized results Allows for background computation

Creates problem with re-finding Results change as user model changes Thesis research – Re:Search Engine

Thank you!

Search Engines are for the Masses

Best common ranking

DCG(i) = { Sort results by number marked highly relevant,

then by relevant

Measure distance with Kendall-TauWeb ranking more similar to common

Individual’s ranking distance: 0.469 Common ranking distance: 0.445

Gain(i), if i = 1DCG(i–1) + Gain(i)/log(i), otherwise

seesaw personalized web search jaime teevan, mit with susan t. dumais and eric horvitz, msr

Documents

query expansion slide

search engine slide

msr slide

end slide

better slide

future work slide

advantage of web ranking

seesaw search engine