Modeling and Predicting the Task-by-Task Behavior of Search Engine

Users

Gabriele TolomeiUniversità Ca’ Foscari Venezia, Italy

Claudio LuccheseISTI-CNR, Pisa, Italy

Salvatore OrlandoUniversità Ca’ Foscari Venezia, Italy

Fabrizio SilvestriISTI-CNR, Pisa, Italy

Raffaele PeregoISTI-CNR, Pisa, Italy

May, 23 2013 - Lisbon, Portugal

10th International Conference in the RIAO series

2

Outline

• Motivation• Research Challenges• Experiments and Results• Conclusion and Future Work

3

Outline

• Motivation• Research Challenges• Experiments and Results• Conclusion and Future Work

4

A New Way of Search

May, 23 2013 - Lisbon, Portugal

Alice

Bob

Same Task! “Reserving a hotel room in New York”

5

… and Search Engines?

• Roughly, they are still Web document retrieval tools– answering on a per-query basis– ten-blue links to relevant Web pages

May, 23 2013 - Lisbon, Portugal

6

Information Need Hierarchy

• Web Task: any (atomic) activity that a user performs through Web search– “find a recipe”, “book a flight”, “read

news”, etc.– distinct users may use different queries to

accomplish the same Web task

• Web Mission: composition of Web tasks to achieve complex goals – distinct users may use different Web tasks

to accomplish the same Web mission

May, 23 2013 - Lisbon, Portugal

[Jones and Klinkner, CIKM ‘08]

7

Goals

• Mine Search Engine logs to detect Web tasks

• Provide a user model for task-oriented search– from query-by-query to task-by-task

• Show how such model can be used to design a real-world application– from query to task recommendation

May, 23 2013 - Lisbon, Portugal

8

Outline

• Motivation• Research Challenges• Experiments and Results• Conclusion and Future Work

9

The Big Picture

• Bottom-up, 2-stage clustering solution:– User Task Discovery from “raw” queries issued

by the same user and stored in query logs– Collective Task Discovery from distinct User

Tasks

• Graph-based representation of Collective Tasks and their relatedness (TRG)May, 23 2013 - Lisbon, Portugal

10

User Task Discovery

• User Task– set of possibly non contiguous queries (multi-

tasking), issued by a single user, whose aim is to carry out a specific Web task

• QC-HTC– Graph-based query clustering solution

proposed in our previous work [Lucchese et al., WSDM’11]

– outperforms other techniques for session boundary detection in query logs (e.g., QFG [Boldi et al., CIKM’08])

May, 23 2013 - Lisbon, Portugal

11

User Task Discovery: QC-HTC

• Splits long-term user session into shorter time-based sessions

• Builds a weighted undirected graph for each time-based session– nodes in each graph are the queries of a time-based

session

• Weight-links consecutive pairs of queries with their content-based similarity:– lexical (query character n-grams)– semantic (query “wikification”)

• Merges any two sequential clusters if their first (head) and last (tail) queries are similar enough

May, 23 2013 - Lisbon, Portugal

12

Task-oriented User Sessions

May, 23 2013 - Lisbon, Portugal

13

Collective Task Discovery

• Collective Task– group of distinct user tasks (i.e., distinct sets of

queries performed by several users) to represent the same Web task

• Identify similar user tasks by clustering their “bag of words” representations – Each user query is a sentence– Each user task is a concatenation of possibly

many sentences (i.e., a text document)

• T = {T1, …, TK} is the final set of Collective Tasks

May, 23 2013 - Lisbon, Portugal

14

Mapping User to Collective Tasks

… … … …

May, 23 2013 - Lisbon, Portugal

15

Task Relation Graph (TRG)

• Task-oriented model of user search behavior• TRG(T, E, w, η) is a weighted directed graph

– nodes are the set of collective tasks T={T1, …, TK}

– edges E represent task relatedness– w: TxT [0,1] is the weighting-edge function– ηis a weight threshold

• Ti and Tj are linked together iff w(Ti, Tj) > η

May, 23 2013 - Lisbon, Portugal

16

Outline

• Motivation• Research Challenges• Experiments and Results• Conclusion and Future Work

User Task Discovery

18

Data Set: AOL 2006 Query Log

May, 23 2013 - Lisbon, Portugal

19

Results

Results were evaluated on a manually-built ground-truth of user tasks [Lucchese et al., TOIS 2013]

May, 23 2013 - Lisbon, Portugal

Collective Task Discovery

21

Data Set: AOL 2006 Query Log

May, 23 2013 - Lisbon, Portugal

22

Training Set vs. Test Set

May, 23 2013 - Lisbon, Portugal

23

Clustering User Tasks

• Algorithm: Repeated Bisections vs. Agglomerative

• Similarity Measure: Cosine similarity vs. Pearson’s correlation

• Objective Function: maximize intra-cluster similarity

• Stop Criterion: choose heuristically the final number K of clusters through the “elbow method”

• We select K = 1,024

May, 23 2013 - Lisbon, Portugal

24

Results and Example

Results were evaluated on a manually-built ground-truth of collective tasks [Lucchese et al., TOIS 2013]

May, 23 2013 - Lisbon, Portugal

Task Relation Graph

26

Building TRG: Task Relatedness

• Use the training set to compute w(Ti,Tj)

• Frequent Sequential Patterns– η= support (i.e., probability) of Ti and Tj co-

occurring in a specified sequence: P(<Ti, Tj>)

– task order matters!

• Association Rules Ti Tj – η= support: P({Ti, Tj})

– η= confidence: P(Tj|Ti)

– task order doesn’t matter!

May, 23 2013 - Lisbon, Portugal

27

Task Recommendation

• One out of many possible applications of TRG

• A user is performing (or has just performed) a task Ti

– indeed a user task which is similar to a known Ti

• Retrieve from TRG the set Rm(Ti) including the m-top related nodes/tasks to Ti

– tasks in Rm(Ti) are those having the m highest edge weights among all the adjacent nodes to Ti

May, 23 2013 - Lisbon, Portugal

28

Task Recommendation: Experiments

• Use TRGs built from training set to generate task recommendations for the test set

• Original user sessions in test set are split in 1/3 prefix and 2/3 suffix sets of user tasks

• Each user task is mapped to a candidate collective task Tc (cosine similarity)

• From all the Tc in prefix retrieve the union-set

of recommendations U Rm(Tc) from TRG

May, 23 2013 - Lisbon, Portugal

29

Task Recommendation: Evaluation

Coverage is affected by the edge weighting function and by the threshold η

May, 23 2013 - Lisbon, Portugal

30

Task Recommendation: Results (top-1)

May, 23 2013 - Lisbon, Portugal

31

Task Recommendation: Results (top-3)

May, 23 2013 - Lisbon, Portugal

32

Task Recommendation: Examples

May, 23 2013 - Lisbon, Portugal

33

Task Recommendation: Examples

May, 23 2013 - Lisbon, Portugal

34

Task vs. Query Recommendation

• To show that task recommendation is different from well-known query recommendation

• TRG vs. QFG– 83.8% of top-3 query suggestions generated by

QFG live in the same (collective) task– Only 15.1% of top-3 query suggestions generated

by QFG lead to 2 separate (collective) tasks

• QFG is great if user wants to stay in the same task

• TRG allows user to switch and jump to other tasks

May, 23 2013 - Lisbon, Portugal

35

Outline

• Motivation• Research Challenges• Experiments and Results• Conclusion and Future Work

36

The “Take-Away” Message

• Web Search Engines should handle user requests from “query-by-query” to “task-by-task”

• New models for user search behavior are needed: from Query Flow Graph to Task Relation Graph

• Task Relation Graph may be exploited for several applications (e.g., Task Recommendation)

May, 23 2013 - Lisbon, Portugal

37

Future Work

• Advanced Task Representation– E.g., linked data, as opposed to simple bag-of-

queries

• Automatic Task Labeling (taxonomy of Web tasks):– Linking queries of collective tasks with referent

entities in a knowledge base– Exploit entity categories to label the whole task

• Use TRG for other applications– Task-based advertising, Mission discovery, etc.

• New SERP to render task-oriented results

May, 23 2013 - Lisbon, Portugal

Thank You!Questions?