click models for web search - lecture 1 · lecture 1 lecture 2 lecture 4 practical 2 lecture 5...
Post on 04-Jun-2018
227 Views
Preview:
TRANSCRIPT
Introduction Basic click models Click probabilities
Click Models for Web SearchLecture 1
Aleksandr Chuklin§,¶ Ilya Markov§ Maarten de Rijke§
a.chuklin@uva.nl i.markov@uva.nl derijke@uva.nl
§University of Amsterdam¶Google Research Europe
AC–IM–MdR Click Models for Web Search 1
Introduction Basic click models Click probabilities
Aleksandr Chuklin
Currently at Google Zurich
Previously at Yandex Moscow
Research interests: user experience evaluation and modelling
Participated at RuSSIR 2009 in Petrozavodsk and RuSSIR2011 in Saint Petersburg
AC–IM–MdR Click Models for Web Search 2
Introduction Basic click models Click probabilities
Ilya Markov
Postdoctoral researcher at the University of Amsterdam
PhD at the University of Lugano
Research interests: heterogeneous search environments
Distributed IR, federated search, aggregated searchUser behavior, user-oriented evaluation
Teach MSc courses on IR and Web Search
AC–IM–MdR Click Models for Web Search 3
Introduction Basic click models Click probabilities
Long-term relations with RuSSIR
RuSSIR 2007, student
RuSSIR 2010, lecturer on Distributed IR (with Fabio Crestani)
RuSSIR 2011, member of organizing committee
RuSSIR 2015, chair of program committee
RuSSIR 2016, lecturer
AC–IM–MdR Click Models for Web Search 4
Introduction Basic click models Click probabilities
Course on Information Retrieval in St. Petersburg
http://compsciclub.ru/courses/
information-retrieval/2016-autumn/
AC–IM–MdR Click Models for Web Search 5
Introduction Basic click models Click probabilities
Maarten de Rijke
Currently at the University of Amsterdam
Ongoing collaborations with Bloomberg Labs, Google,Microsoft Research, Yandex Moscow
Research interests: semantic search, online learning to rank
Always looking for strong new PhD students
AC–IM–MdR Click Models for Web Search 6
Introduction Basic click models Click probabilities
The book
http://clickmodels.weebly.com/the-book.html
AC–IM–MdR Click Models for Web Search 7
Introduction Basic click models Click probabilities
Other course materials
clickmodels.weebly.com/russir-2016-course.html
Demos and practical sessions:
clickmodels.weebly.com/russir-2016-setup.html
github.com/markovi/PyClick
AC–IM–MdR Click Models for Web Search 8
Introduction Basic click models Click probabilities
Course content
Basic Click Models
Parameter Estimation Evaluation
Data and ToolsResultsApplications
Advanced Models
Recent Studies
Future Research
AC–IM–MdR Click Models for Web Search 9
Introduction Basic click models Click probabilities
Lectures
Basic Click Models
Parameter Estimation Evaluation
Data and ToolsResultsApplications
Advanced Models
Recent Studies
Future Research
Lecture 1 Lecture 2
Lecture 4Practical 2 Lecture 5
Lecture 3Practical 1Lecture 2
AC–IM–MdR Click Models for Web Search 10
Introduction Basic click models Click probabilities
Course overview
Basic Click Models
Parameter Estimation Evaluation
Data and ToolsResultsApplications
Advanced Models
Recent Studies
Future Research
AC–IM–MdR Click Models for Web Search 11
Introduction Basic click models Click probabilities
This lecture
Basic Click Models
Parameter Estimation Evaluation
Data and ToolsResultsApplications
Advanced Models
Recent Studies
Future Research
AC–IM–MdR Click Models for Web Search 12
Introduction Basic click models Click probabilities
Lecture outline
1 Introduction
2 Basic click models
3 Click probabilities
AC–IM–MdR Click Models for Web Search 13
Introduction Basic click models Click probabilities
Web search
AC–IM–MdR Click Models for Web Search 14
Introduction Basic click models Click probabilities
Why clicks?
AC–IM–MdR Click Models for Web Search 15
Introduction Basic click models Click probabilities
Why clicks?
Reflect user interests
Help to improve search
Help to evaluate search
Ongoing and future research: other user search interactions
mouse movementsscrollingtouch gestures
AC–IM–MdR Click Models for Web Search 16
Introduction Basic click models Click probabilities
What can we do with clicks?
AC–IM–MdR Click Models for Web Search 17
Introduction Basic click models Click probabilities
What can we do with clicks?
countclick-through rate (CTR)
Global CTR = # clicks# shown docs
Rank-based CTR = # clicks at rank r# shown docs at rank r
Query-document CTR = # u is clicked for q# u is shown for q
Some notation: u – URL (or document), q – query
AC–IM–MdR Click Models for Web Search 18
Introduction Basic click models Click probabilities
Why click models?
AC–IM–MdR Click Models for Web Search 19
Introduction Basic click models Click probabilities
Why click models?
Scientific modelling is a scientific activity, the aim of which is tomake a particular part or feature of the world easier to understand,define, quantify, visualize, or simulate by referencing it to existingand usually commonly accepted knowledge.
Wikipedia, Scientific modelling
AC–IM–MdR Click Models for Web Search 20
Introduction Basic click models Click probabilities
Why click models?
Click models make user clicks in web searcheasier to understand, define, quantify, visualize, or simulate
using (mostly) probabilistic graphical models.
AC–IM–MdR Click Models for Web Search 21
Introduction Basic click models Click probabilities
Click log
Yandex Relevance Prediction Challengehttp://imat-relpred.yandex.ru/en
AC–IM–MdR Click Models for Web Search 22
Introduction Basic click models Click probabilities
Why do we need click models?
Understand users
Simulate users
Approximate document relevance
Evaluate search
AC–IM–MdR Click Models for Web Search 23
Introduction Basic click models Click probabilities
Lecture outline
1 Introduction
2 Basic click modelsRandom click modelCTR modelsPosition-based modelCascade modelDynamic Bayesian network modelUser browsing model
3 Click probabilities
AC–IM–MdR Click Models for Web Search 24
Introduction Basic click models Click probabilities
Lecture outline
2 Basic click modelsRandom click modelCTR modelsPosition-based modelCascade modelDynamic Bayesian network modelUser browsing model
AC–IM–MdR Click Models for Web Search 25
Introduction Basic click models Click probabilities
Random click model
Pclick
Pclick
Pclick
Pclick
Pclick
AC–IM–MdR Click Models for Web Search 26
Introduction Basic click models Click probabilities
Random click model
Terminology
Cu – binary random variable denoting a click on document uDocument u is clicked: Cu = 1Document u is not clicked: Cu = 0P(Cu = 1) – probability of click on document uP(Cu = 0) = 1− P(Cu = 1)
Random click model (RCM)
Any document can be clicked with the same (fixed) probability
P(Cu = 1) = const = ρ
AC–IM–MdR Click Models for Web Search 27
Introduction Basic click models Click probabilities
Random click model
P(Cu1 = 1) = ρ
P(Cu2 = 1) = ρ
P(Cu3 = 1) = ρ
P(Cu4 = 1) = ρ
P(Cu5 = 1) = ρ
ρ =# clicks
# shown docs= Global CTR
AC–IM–MdR Click Models for Web Search 28
Introduction Basic click models Click probabilities
Lecture outline
2 Basic click modelsRandom click modelCTR modelsPosition-based modelCascade modelDynamic Bayesian network modelUser browsing model
AC–IM–MdR Click Models for Web Search 29
Introduction Basic click models Click probabilities
Rank-based CTR model
P(Cu1 = 1) = ρ1
P(Cu2 = 1) = ρ2
P(Cu3 = 1) = ρ3
P(Cu4 = 1) = ρ4
P(Cu5 = 1) = ρ5
P(Cur = 1) = ρr =# clicks at rank r
# shown docs at rank r
AC–IM–MdR Click Models for Web Search 30
Introduction Basic click models Click probabilities
Query-document CTR model
P(Cu1 = 1) = ρu1q
P(Cu2 = 1) = ρu2q
P(Cu3 = 1) = ρu3q
P(Cu4 = 1) = ρu4q
P(Cu5 = 1) = ρu5q
P(Cu = 1) = ρuq =# u is clicked for q
# u is shown for q
AC–IM–MdR Click Models for Web Search 31
Introduction Basic click models Click probabilities
CTR models: summary
Random click model (global CTR):
P(Cu = 1) = ρ
Rank-based CTR:
P(Cur = 1) = ρr
Query-document CTR:
P(Cu = 1) = ρuq
AC–IM–MdR Click Models for Web Search 32
Introduction Basic click models Click probabilities
CTR models: demo
Demo
AC–IM–MdR Click Models for Web Search 33
Introduction Basic click models Click probabilities
Lecture outline
2 Basic click modelsRandom click modelCTR modelsPosition-based modelCascade modelDynamic Bayesian network modelUser browsing model
AC–IM–MdR Click Models for Web Search 34
Introduction Basic click models Click probabilities
Position-based model
Pread(1) , Pclick(u1q)
Pread(2), Pclick(u2q)
Pread(3) , Pclick(u3q)
Pread(4) , Pclick(u4q)
Pread(5) , Pclick(u5q)
AC–IM–MdR Click Models for Web Search 35
Introduction Basic click models Click probabilities
Position-based model: examination
TerminologyExamination = reading a snippet
Er – binary random variable denoting examinationof a snippet at rank r
Snippet at rank r is examined: Er = 1
Snippet at rank r is not examined: Er = 0
P(Er = 1) – probability of examination of rank r
P(Er = 0) = 1− P(Er = 1)
Position-based model (PBM)Examination depends on rank
P(Er = 1) = γr
AC–IM–MdR Click Models for Web Search 36
Introduction Basic click models Click probabilities
Position-based model
γ1 , Pclick(u1q)
γ2 , Pclick(u2q)
γ3 , Pclick(u3q)
γ4 , Pclick(u4q)
γ5 , Pclick(u5q)
AC–IM–MdR Click Models for Web Search 37
Introduction Basic click models Click probabilities
Position-based model: attractiveness
TerminologyAttractiveness = a user wants to click on a documentafter examining (reading) its snippet
Au – binary random variable showing whether document uis attractive to a user, given query q
Document u is attractive: Au = 1
Document u is not attractive: Au = 0
P(Au = 1) – probability of attractiveness of document u
P(Au = 0) = 1− P(Au = 1)
Position-based model (PBM)Attractiveness depends on a query-document pair
P(Auq = 1) = αuq
AC–IM–MdR Click Models for Web Search 38
Introduction Basic click models Click probabilities
Position-based model
γ1 , αu1q
γ2 , αu2q
γ3 , αu3q
γ4 , αu4q
γ5 , αu5q
AC–IM–MdR Click Models for Web Search 39
Introduction Basic click models Click probabilities
Position-based model: summary
P(Eru = 1) = γru
P(Au = 1) = αuq
P(Cu = 1) = P(Eru = 1) · P(Au = 1)
AC–IM–MdR Click Models for Web Search 40
Introduction Basic click models Click probabilities
Position-based model: probabilistic graphical model
document u
Eu
Cu
Au
↵uq�ru
AC–IM–MdR Click Models for Web Search 41
Introduction Basic click models Click probabilities
Position-based model: exercises
P(Eru = 1) = γru
P(Au = 1) = αuq
P(Cu = 1) = P(Eru = 1) · P(Au = 1)
Eru = 0⇒ Cu = 0
Au = 0⇒ Cu = 0
Eru = 1⇒ (Cu = 1 ⇐⇒ Au = 1)
Au = 1⇒ (Cu = 1 ⇐⇒ Eru = 1)
AC–IM–MdR Click Models for Web Search 42
Introduction Basic click models Click probabilities
Position-based model: demo
Demo
AC–IM–MdR Click Models for Web Search 43
Introduction Basic click models Click probabilities
Lecture outline
2 Basic click modelsRandom click modelCTR modelsPosition-based modelCascade modelDynamic Bayesian network modelUser browsing model
AC–IM–MdR Click Models for Web Search 44
Introduction Basic click models Click probabilities
Position-based model
P(Eru = 1) = γru
P(Au = 1) = αuq
P(Cu = 1) = P(Eru = 1) · P(Au = 1)
AC–IM–MdR Click Models for Web Search 45
Introduction Basic click models Click probabilities
Cascade model
1 Start from the first document
2 Examine documents one by one
3 If click, then stop
4 Otherwise, continue
AC–IM–MdR Click Models for Web Search 46
Introduction Basic click models Click probabilities
Cascade model
Er = 1 and Aur = 1⇔ Cr = 1
P(Aur = 1) = αurq
P(E1 = 1)︸ ︷︷ ︸start from first
= 1
P(Er = 1 | Er−1 = 0)︸ ︷︷ ︸examine one by one
= 0
P(Er = 1 | Cr−1 = 1)︸ ︷︷ ︸if click, then stop
= 0
P(Er = 1 | Er−1 = 1,Cr−1 = 0)︸ ︷︷ ︸otherwise, continue
= 1
AC–IM–MdR Click Models for Web Search 47
Introduction Basic click models Click probabilities
Cascade model: probabilistic graphical model
document urdocument ur�1
Er�1
Cr�1
Ar�1
Er
Cr
Ar
......
↵ur�1q ↵urq
AC–IM–MdR Click Models for Web Search 48
Introduction Basic click models Click probabilities
Click models so far
CTR models
+ count clicks (simple and fast)– do not distinguish examination and attractiveness
Position-based model (PBM) −→ User browsing model+ examination and attractiveness– examination of a document at rank r does not depend
on examinations and clicks above r
Cascade model (CM) −→ Dynamic Bayesian network+ cascade dependency of examination at r
on examinations and clicks above r– only one click is allowed
AC–IM–MdR Click Models for Web Search 49
Introduction Basic click models Click probabilities
Lecture outline
2 Basic click modelsRandom click modelCTR modelsPosition-based modelCascade modelDynamic Bayesian network modelUser browsing model
AC–IM–MdR Click Models for Web Search 50
Introduction Basic click models Click probabilities
Dynamic Bayesian network model
Cascade model −→ Dynamic Bayesian network
AC–IM–MdR Click Models for Web Search 51
Introduction Basic click models Click probabilities
Dynamic Bayesian network model
1 Start from the first document
2 Examine documents one by one
3 If click, read actual documentand can be satisfied
4 If satisfied, stop
5 Otherwise, continue with fixedprobability
AC–IM–MdR Click Models for Web Search 52
Introduction Basic click models Click probabilities
Dynamic Bayesian network model: satisfaction
TerminologySatisfaction = a user reads the clicked documentand satisfies his/her information needSu – binary random variable showing whether document uis satisfactory for query qP(Su = 1) – probability of satisfactoriness of document u,P(Su = 0) = 1− P(Su = 1)
Dynamic Bayesian network model (DBN)If a user is satisfied, he/she stopsOtherwise, continues with fixed probability
P(Er = 1 | Sr−1 = 1)︸ ︷︷ ︸if satisfied, stop
= 0
P(Er = 1 | Er−1 = 1, Sr−1 = 0)︸ ︷︷ ︸otherwise, continue
= γ
AC–IM–MdR Click Models for Web Search 53
Introduction Basic click models Click probabilities
Dynamic Bayesian network model: summary
Er = 1 and Aur = 1⇔ Cr = 1
P(Aur = 1) = αurq
P(E1 = 1) = 1
P(Er = 1 | Er−1 = 0) = 0
P(Sr = 1 | Cr = 1)︸ ︷︷ ︸if click, can be satisfied
= σurq
P(Er = 1 | Sr−1 = 1)︸ ︷︷ ︸if satisfied, stop
= 0
P(Er = 1 | Er−1 = 1, Sr−1 = 0)︸ ︷︷ ︸otherwise, continue
= γ
AC–IM–MdR Click Models for Web Search 54
Introduction Basic click models Click probabilities
Dynamic Bayesian network: probabilistic graphical model
document urdocument ur�1
Er�1
Cr�1
Ar�1
Er
Cr
Ar
......
↵ur�1q ↵urq
Sr�1 Sr
�
�ur�1q �urq
AC–IM–MdR Click Models for Web Search 55
Introduction Basic click models Click probabilities
Dynamic Bayesian network model: demo
Demo
AC–IM–MdR Click Models for Web Search 56
Introduction Basic click models Click probabilities
Lecture outline
2 Basic click modelsRandom click modelCTR modelsPosition-based modelCascade modelDynamic Bayesian network modelUser browsing model
AC–IM–MdR Click Models for Web Search 57
Introduction Basic click models Click probabilities
User browsing model
Position-based model −→ User browsing model
AC–IM–MdR Click Models for Web Search 58
Introduction Basic click models Click probabilities
Position-based model
γ1 , αu1q
γ2 , αu2q
γ3 , αu3q
γ4 , αu4q
γ5 , αu5q
P(Er = 1 | Cr ′ = 1,Cr ′+1 = 0, . . . ,Cr−1 = 0) = γrr ′
AC–IM–MdR Click Models for Web Search 59
Introduction Basic click models Click probabilities
User browsing model
γ10 , αu1q
γ21 , αu2q
γ31 , αu3q
γ41 , αu4q
γ54 , αu5q
P(Er = 1 | Cr ′ = 1,Cr ′+1 = 0, . . . ,Cr−1 = 0) = γrr ′
AC–IM–MdR Click Models for Web Search 60
Introduction Basic click models Click probabilities
User browsing model: summary
P(Cu = 1) = P(Eru = 1) · P(Au = 1)
P(Au = 1) = αuq
P(Er = 1 |Cr ′ = 1,
Cr ′+1 = 0, . . . ,Cr−1 = 0) = γrr ′
AC–IM–MdR Click Models for Web Search 61
Introduction Basic click models Click probabilities
User browsing model: probabilistic graphical model
document ur
Er
Cr
Ar
...
↵urq
�rr0
AC–IM–MdR Click Models for Web Search 62
Introduction Basic click models Click probabilities
Basic click models summary
CTR models: counting clicks
Position-based model (PBM): examination and attractiveness
Cascade model (CM): previous examinations and clicks matter
Dynamic Bayesian network model (DBN): satisfactoriness
User browsing model (UBM): rank of previous click
AC–IM–MdR Click Models for Web Search 63
Introduction Basic click models Click probabilities
Probability theory
Partitioned probability: A = A1 ∪ A2, A1 ∩ A2 = ∅
P(A) = P(A1,A2) = P(A1) + P(A2)
Bayes’ rule
P(A | B) · P(B) = P(B | A) · P(A)
B causes A: B → A
P(B) = P(B | A) · P(A)
AC–IM–MdR Click Models for Web Search 64
Introduction Basic click models Click probabilities
Probability theory (cont’d)
B → A, A = A1 ∪ A2, A1 ∩ A2 = ∅
P(B) = P(B | A) · P(A)
= P(B | A1,A2) · P(A1,A2)
= P(B | A1,A2) · (P(A1) + P(A2))
= P(B | A1,A2) · P(A1) + P(B | A1,A2) · P(A2)
= P(B | A1) · P(A1) + P(B | A2) · P(A2)
P(B) = P(B | A1) · P(A1) + P(B | A2) · P(A2)
AC–IM–MdR Click Models for Web Search 65
Introduction Basic click models Click probabilities
Lecture outline
1 Introduction
2 Basic click models
3 Click probabilities
AC–IM–MdR Click Models for Web Search 66
Introduction Basic click models Click probabilities
Click probabilities
Full probability – probabilitythat a user clickson a document at rank r
P(Cr = 1)
Conditional probability –probability that a user clickson a document at rank rgiven previous clicks
P(Cr = 1 | C1, . . . ,Cr−1)
AC–IM–MdR Click Models for Web Search 67
Introduction Basic click models Click probabilities
Dependency between examination and clicks
document u
Eu
Cu
Au
↵uq�ru
AC–IM–MdR Click Models for Web Search 68
Introduction Basic click models Click probabilities
Full click probability
P(Cr = 1) = +P(Cr = 1 | Er = 1) · P(Er = 1)
P(Cr = 1 | Er = 0) · P(Er = 0)
= P(Aur = 1) · P(Er = 1) + 0
= αurqεr
AC–IM–MdR Click Models for Web Search 69
Introduction Basic click models Click probabilities
Cascade models: dependency between examinations
document urdocument ur�1
Er�1
Cr�1
Ar�1
Er
Cr
Ar
......
↵ur�1q ↵urq
AC–IM–MdR Click Models for Web Search 70
Introduction Basic click models Click probabilities
Full click probability
P(Cr = 1) = P(Aur = 1) · P(Er = 1) = αurqεr
εr+1 = P(Er+1 = 1)
= +P(Er = 1) · P(Er+1 = 1 | Er = 1)
P(Er = 0) · P(Er+1 = 1 | Er = 0)
= εr · P(Er+1 = 1 | Er = 1) + 0
= εr ·
(+P(Er+1 = 1 | Er = 1,Cr = 1) · P(Cr = 1 | Er = 1)
P(Er+1 = 1 | Er = 1,Cr = 0) · P(Cr = 0 | Er = 1)
)
AC–IM–MdR Click Models for Web Search 71
Introduction Basic click models Click probabilities
Full click probability: Dynamic Bayesian network model
Dynamic Bayesian network model: satisfactoriness
document urdocument ur�1
Er�1
Cr�1
Ar�1
Er
Cr
Ar
......
↵ur�1q ↵urq
Sr�1 Sr
�
�ur�1q �urq
P(Cr+1 = 1) = αur+1qεr ·
(+P(Er+1 = 1 | Er = 1,Cr = 1) · P(Cr = 1 | Er = 1)
P(Er+1 = 1 | Er = 1,Cr = 0) · P(Cr = 0 | Er = 1)
)
P(Cr+1 = 1) = αur+1qεr ·
(+
(1− σurq)γ · αurq
γ · (1− αurq)
)AC–IM–MdR Click Models for Web Search 72
Introduction Basic click models Click probabilities
Conditional click probability
P(Cr = 1 | C1, . . . ,Cr−1) = P(Cr = 1 | C<r )
= +P(Cr = 1 | Er = 1,C<r ) · P(Er = 1 | C<r )
P(Cr = 1 | Er = 0,C<r ) · P(Er = 0 | C<r )
= P(Aur = 1) · P(Er = 1 | C<r ) + 0
= αurqεr
εr+1 = +
P(Er+1 = 1 | Er = 1,Cr = 1) · c(s)r
P(Er+1 = 1 | Er = 1,Cr = 0) · εr (1− αurq)
1− αurqεr· (1− c(s)
r )
c(s)r – a click on rank r in query session s
AC–IM–MdR Click Models for Web Search 73
Introduction Basic click models Click probabilities
Click probabilities summary
Full probability
P(Cr+1 = 1) =
αur+1qεr ·
(+P(Er+1 = 1 | Er = 1,Cr = 1) · P(Cr = 1 | Er = 1)
P(Er+1 = 1 | Er = 1,Cr = 0) · P(Cr = 0 | Er = 1)
)
Conditional probability
P(Cr+1 = 1 | C1, . . . ,Cr )
= αur+1q ·
+
P(Er+1 = 1 | Er = 1,Cr = 1) · c(s)r
P(Er+1 = 1 | Er = 1,Cr = 0) · εr (1− αurq)
1− αurqεr· (1− c(s)
r )
AC–IM–MdR Click Models for Web Search 74
Introduction Basic click models Click probabilities
Lecture 1 summary
CTR models: counting clicks
Position-based model (PBM): examination and attractiveness
Cascade model (CM): previous examinations and clicks matter
Dynamic Bayesian network model (DBN): satisfactoriness
User browsing model (UBM): rank of previous click
AC–IM–MdR Click Models for Web Search 75
Introduction Basic click models Click probabilities
Lecture 1 summary
What do click models give us?
General
Understanding of user behavior
Specific
Conditional click probabilitiesFull click probabilitiesAttractiveness and satisfactoriness for query-document pairs
AC–IM–MdR Click Models for Web Search 76
Introduction Basic click models Click probabilities
Course overview
Basic Click Models
Parameter Estimation Evaluation
Data and ToolsResultsApplications
Advanced Models
Recent Studies
Future Research
AC–IM–MdR Click Models for Web Search 77
Introduction Basic click models Click probabilities
Next lecture
Basic Click Models
Parameter Estimation Evaluation
Data and ToolsResultsApplications
Advanced Models
Recent Studies
Future Research
AC–IM–MdR Click Models for Web Search 78
Introduction Basic click models Click probabilities
Acknowledgments
All content represents the opinion of the authors which is not necessarily shared orendorsed by their respective employers and/or sponsors.
AC–IM–MdR Click Models for Web Search 79
top related