learning user interaction models for predicting web search result preference
DESCRIPTION
Learning User Interaction Models for Predicting Web Search Result Preference. Eugene Agichtein et al. Microsoft Research SIGIR ‘06. Objective. Provide a rich set of features for representing user behavior Query-text Browsing Clickthough Aggregate various feature RankNet. - PowerPoint PPT PresentationTRANSCRIPT
Learning User Interaction Models for Predicting Web Search Result
PreferenceEugene Agichtein et al.
Microsoft Research
SIGIR ‘06
Objective
• Provide a rich set of features for representing user behavior– Query-text– Browsing– Clickthough
• Aggregate various feature– RankNet
Browsing feature
• Related work
• The amount of reading time could predict– interest level on news articles– rating in recommender system
• The amount of scrolling on a page also have strong relationship with interest
Browsing feature
• How to collect browsing feature?– Obtain the information via opt-in client-side
instrumentation
Browsing feature
• Dwell time
Browsing feature
• Average & Deviation
• Properties of the click event
Clickthrough feature
• 1. Clicked VS. Unclicked– Skip Above (SA)– Skip Next (SN)
• Advantage– Propose preference pair
• Disadvantage– Inconsistency– Noisiness of individual
Clickthrough feature
• 2. Position-biased
Clickthrough feature
Clickthrough feature
Clickthrough feature
• Disadvantage of SA & SN– User may click some irrelevant pages
Clickthrough feature
• Disadvantage of SA & SN– User often click part of relevant pages
Clickthrough feature
• 3. Feature for learning
Feature set
Feature set
Evaluation
• Dataset– Random sample 3500 queries and their top
10 results– Rate on a 6-point scale manually– 75% training, 25% testing– Convert into pairwise judgment– Remove tied pair
Evaluation
• Pairwise judgment
• Input– UrlA, UrlB
• Outpur– Positive: rel(UrlA) > rel(UrlB)
– Negative: rel(UrlA) ≤ rel(UrlB)
• Measurement– Average query precision & recall
Evaluation
1. Current– Original rank from search engine
• 2. Heuristic rule without parameter– SA, SA+N
• 3. Heuristic rule with parameter– CD, CDiff, CD + CDiff
• 4. Supervised learning– RankNet
Evaluation
Evaluation
Evaluation
Conclusion
• Recall is not a important measurement
• Heuristic rule– very low recall and low precision
• Feature set– Browsing features have higher precision
Discussion
• Is user interaction model better than search engine– Small coverage– Only pairwise judgment
• Given the same training data, which one is better, traditional ranking algorithm or user interaction?
• Which feature is more useful?