Contextual IR:Participations in TREC Contextual Suggestion Tracks 2012 and 2013

Gilles Huberthttp://bit.ly/wlvHubert2013

Wolverhampton TalkDecember 2013

Summary

1. Information Retrieval

2. Contextual Information Retrieval

3. TREC

4. TREC Contextual Track 20121. Our approach2. Results

5. TREC Contextual Track 20131. Our approach2. Results

6. Result analysis and Future work

2

Information RetrievalUsual IR process

Query

Queryrepresentation Document

representation

Documentrepresentation

Documentrepresentation

DocumentDocument

Document

Matching

Indexing Indexing

List of estimated relevant

documents

3

Contextual Information RetrievalNotion of context in IR

How to consider context in IR process ?Q1 : Retrieve items corresponding to the context

Q2 : Retrieve the context corresponding to items

Information

User

DeviceSoftware

4

Contextual IRContext integration in Q1

Query

Queryrepresentation Document

representation

Documentrepresentation

Documentrepresentation

DocumentDocument

Document

Matching

Indexing Indexing

List of estimated relevant documents

List of estimated relevant documents

Reranking

Context

5

TRECText Retrieval Conference

Organized by the NIST (USA) since 1992

Based on the Cranfield paradigm of retrieval system evaluation

A set of documents (Collection)A set of information needs (Topics/Queries)A set of relevance judgments (Qrels)

Various tracks: AdHoc, Robust, Web…

Evaluation measures

6

relevant, retrieved(True positive)

irrelevant, retrieved(False positive)

relevant, not retrieved(False negative)

irrelevant, not retrieved(True negative)

A

C

B

D

Document collection

System output:retrieved documents

Information need:relevant documents

AP (Average Precision),MAP (Mean Average Precision),P@5 (Precision at 5 retrieved documents)…

TRECCampaign principles

7(Voorhees, 2007)

TREC Contextual Suggestion Track 2012

8

Great summer !!!

Where to go around here on this Sunday afternoon?

TREC Contextual Suggestion Track 2012Retrieve items corresponding to the context (Q1)Items = Suggestions

Places to visit (shops, restaurants, parks…) around the user (5 hours by car max.)

Collection = Open Web (Websites)Context =

Spatiotemporal data

User preferences

9

<context number=”1”> <city>Portland</city> <state>Oregon</state> <lat>45.5</lat> <long>-122.7</long> <day>weekday</day> <time>evening</time> <season>fall</season></context>

<example number=”1”> <title> Dogfish Head Alehouse </title> <description>Craft Brewed Ales and tasty woodgrilled food </description> <url>http://www.dogfishalehouse.com/</url> </example> <example number=”2”> <title>The Flaming Pit</title> <description> The Flaming Pit Restaurant and Piano Lounge,home of Tyrone DeMonke. </description> <url>http://www.flamingpitrestaurant.com/</url> </example>

<profile number=”1”> <example number=”1” initial=”1” final=”1”/> <example number=”2” initial=”0” final=”-1”/> </profile >

TREC Contextual Suggestion Track 2012Two subtasks

S1 : Suggestions corresponding to spatiotemporal data

List of suggestions for each contextS2 : S1 + user preferences

List of suggestions for each profile (user) and each contextSuggestion = Title + Description + Url

2 “runs” maximum

Our participation

Team : G. Cabanac & G. Hubert (IRIT – Univ. of Toulouse)2 runs submitted to S2 subtask

<context2012 groupid=”waterloo” runid=”watcs12a”> <suggestion profile=”1” context=”1” rank=”1”> <title>Deschutes Brewery Portland Public House</title> <description> Deschutes Brewery’s distinct Northwest brew pub in Portland’s Pearl District has become a convivial gathering spot of beer and food lovers since it’s 2008 opening. </description> <url>http://www.deschutesbrewery.com</url> </suggestion> etc.</context2012>

10

TREC Contextual Suggestion Track 2012: Our approachContextual IRS framework 2012

Input data Internal process

External resource

Intermediate data

Database

ContextiPlace selection Google Places

API

Place sets

Place query

Contextual list of places

Place description enrichment

Contextual list of

detailed places

Bing Google

Useri

Context processing

Output data

Personalization

Personalized suggestions

Examples

Profilei Preference definition

Positive

preferencesi

Negative

preferencesi

Preference processing

11

TREC Contextual Suggestion Track 2012: Our approachSpatiotemporal data

User preferencesCoarse-grained approach : iritSplit3CPv1

Merging of descriptions of examples with initial and final = 1 -> Pref+(P)Merging of descriptions of examples with initial and final = -1 -> Pref-(P)score(P,r) = cosine(Pref+(P),R) − cosine(Pref−(P),R)

Fine-grained approach : iritSplit3CPv2

Example description with initial and final = 1 -> Pref+l(P)Example description with initial and final = -1 -> Pref-m(P)score(P, r) = max(cosine(Pref+l (P), r))− max(cosine(Pref−m(P), r))

12

TREC Contextual Suggestion Track 2012: ResultsEvaluations

For each profile and each context

Different dimensions : W (Website), G (Geographical), T (Temporal), and D (Description), and combinations (WGT and GT)

Two measures : P@5 and MRR (Mean Reciprocal Rank)

13

iritSplit3CPv1

iritSplit3CPv2

TREC Contextual Suggestion Track 2012: ResultsP@5

14

TREC Contextual Suggestion Track 2012: ResultsMRR

15

TREC Contextual Suggestion Track 2013

16

Where to go around here?

TREC Contextual Suggestion Track 2013Context =

Spatial only

User preferences

{ "1": { "lat": "40.71427", "city": "New York City", "state": "NY", "long": "-74.00597” }, …}

{ "1": { "url": http://www.freshrestaurants.ca, "description": "Our vegan menu boasts an array of exotic starters, multi-layered salads, filling wraps, high protein burgers and our signature Fresh bowls.”, "title": "Fresh on Bloor” }, “2": { "url": http://www.flamingpitrestaurant.com/, "description": "The Flaming Pit Restaurant and Piano Lounge, home of Tyrone DeMonke.”, "title": "The Flaming Pit” }, …}

{ "1": [ {"attraction_id": 1, "website": 1, "description": 0}, ... ], "2": [ {"attraction_id": 1, "website": 4, "description": 3}, … ], ”3": [ {"attraction_id": 1, "website": -1, "description": 2}, … ], …}

17

TREC Contextual Suggestion Track 2013Two subtasks

Open Web

Same question: Suggest places items corresponding to the context (Q1)Places to visit (restaurants, museums…) around the user (5 hours by car)Collection = Open Web (Websites)

ClueWeb

ClueWeb12 (Same question as OpenWeb)ClueWeb12 Contextual suggestion subcollection

Sets of ClueWeb12 documents per context

Question: Personalization per user profile

2 “runs” maximumOur participation

Team: G. Cabanac, G. Hubert & K. Pinel-Sauvagnat (IRIT – Univ. of Toulouse)

C. Sallaberry (LIUPPA – Univ. of Pau)

D. Palacio (GeoComp – Univ. of Zurich)

1 “run” Open Web

1 “run” ClueWeb (Contextual suggestion subcollection)

18

TREC Contextual Suggestion Track 2013: Our approachContextual IRS framework 2013

19Open Web ClueWeb

20

Example of suggestion in 2013

Title: Celtic Mist Pub

Description:Place types: bar, establishment. This place is about .3 Km West from here (2 min by car with no traffic). Address: 117 South 7th Street, Springfield.There are 11 POIs around: 2 Hotels, 3 Libraries, 3 Parks, 1 PostOffice, 2 Religious.Snippet: Located in Springfield, IL the Celtic Mist is your home away from home with over 16 imported beers on tap and a friendly staff ready to serve you…

URL: http://www.celticmistpub.com/

Example of suggestion in 2012

Title: Oakley Pub and Grill

DescriptionOakley Pub and Grill - Located in Oakley Square, Cincinnati, Ohio. Local pub with pleasant atmoshpere and great food. Voted #1 Best Burger in Cincinnati. Outdoor ...PUB and GRILL OAKLEYOAKLEY Oakley Pub and Grill ~ 3924 Isabella Avenue ~ Cincinnati, Ohio 45209 On Oakley Square ~ (513) 531-2500 www.oakleypub.com Used with permission…

URL: http://oakleypubandgrill.com/

21

22

Final ResultsOpen Web

23

Final ResultsClueWeb

24

Result AnalysisFirst edition (2012)

All the participants discovered the track principles

Worst results: Descriptions of suggestions

Second edition (2013)OpenWeb

Focus on suggestion descriptionsChanges in relevance judgments

ClueWeb

Misunderstanding of guidelines or insufficient details

Next edition: TREC Contextual Suggestion Track 2014Close to TREC Contextual Suggestion Track 2013

Future workExperiment framework variants on 2013 data

Replace limited online tools/services

Process larger collection: ClueWeb12 (870 millions pages, ~27TB)

Next edition of TREC Contextual Suggestion Track 2014Contexts outside USA

Future workExperiment framework variants on 2013 data

Replace limited online tools/services

Process larger collection: ClueWeb12 (870 millions pages, ~27TB)

Thank you for your attentionQuestions?

Wolverhampton TalkDecember 2013