ecdl 2010 - measuring effectiveness of geographic ir systems in digital libraries: evaluation and...

Damien Palacio - [email protected] 1

ECDL 20106-10 september 2010

Measuring Effectiveness ofGeographic IR Systems

in Digital Libraries:Evaluation and Case Study

Damien Palacio, Guillaume Cabanac,

Christian Sallaberry, Gilles Hubert

2

Outline

1. Motivation Topical IR Geographic IR→Hypothesis: GIRS > IRS

2. Context IRS evaluation Issue Current evaluation frameworks

= partial

3. Contribution GIRS evaluation framework

4. Experiments Case study with PIV GIRSHypothesis validated

5. Conclusion and Future Works

3

Outline



= partial




4

1. Motivation – Why Geographic IR?

Geographic Information Retrieval➔ Query = ''trip around Glasgow in summer 2010''

➔ Search Engines➔ Topical term ∈ {trip, Glasgow, summer, 2010}

spatial ∈ {citiesNearGlasgow ...}➔ Geographic temporal ∈ {21june .. 22sept 2010}

term ∈ {trip, Glasgow, summer, 2010}

➔ ≈ 1/6 Queries = Geographic Queries➔ Excite (Sanderson et al., 2004)

➔ AOL (Gan et al., 2008)

➔ Yahoo! (Jones et al., 2008)

➔ Current Issue and Realistic

5


A Geographic IRS: How Does It Work?➔ 3 Dimensions to Process:

➔ Spatial, temporal and topical

➔ 1 Index per Dimension➔ Topical bag of words, vector space model, ...➔ Spatial named entity recognition, ...➔ Temporal named entity recognition, ...

6


A Geographic IRS: How Does It Work?➔ Spatial Processing

7


A Geographic IRS: How Does It Work?➔ 3 Dimensions to Process:

➔ Spatial, temporal and topical

➔ 1 Index per Dimension➔ Topical bag of words, vector space model, ...➔ Spatial named entity recognition, ...➔ Temporal named entity recognition, ...

➔ Retrieval➔ Usually by filtering (STEWARD, SPIRIT, CITER, …)

➔ Issue: Performance of GIRS vs. topical IRS➔ Hypothesis: Geographic IRS better than topical IRS

8

Outline



= partial




9

2. Context and Issue: IRS Partial Evaluation

Evaluating an IR System➔ System = efficiency + effectiveness

➔ Effectiveness Evaluation

Geo IR litterature Topical IR litterature

10





Computation time

Storage needed

11




Computation time

Storage needed Quality


12




Computation time



TopicalTemporal

Spatial

13




Computation time



TopicalTemporal

Spatial

TREC, CLEF, ...

14




Computation time



TopicalTemporal

Spatial

TREC, CLEF, ...TempEval

15




Computation time



TopicalTemporal

Spatial


Bucher et al. (2005)GeoClef

16




Computation time



TopicalTemporal

Spatial


Bucher et al. (2005)GeoClefEvaluation

framework proposed

17

Outline



= partial




18

➔ Goal: measuring GIRS quality

➔ Means: building on TREC framework (1992-)

➔ ''Cranfield'' methodology➔ Test collection

➔ Corpus➔ ≥ 25 Topics➔ Qrels

➔ Measures: P@X, MAP, NDCG, ...

[Voorhees, 2007]

3. Proposition – GIRS Evaluation Framework

Evaluation Framework for the 3 Dimensions (1/2)

19


Evaluation Framework for the 3 Dimensions (2/2) ➔ TREC Framework Extension

➔ Test collection➔ ≥ 25 Topics➔ Corpus➔ Gradual qrels➔ + geographic ressources

Covering the 3 dimensions

20




➔ About qrels … ➔ Relevance (doc, topic) ∈ {0;1;2;3;4}➔ Principle: ''the more satisfied dimensions there are, the

better it is''


No dimension

3 dimensions: Topic: ''trip around Glasgow'' Doc: trip + Bob born in Dumbarton

3 dimensions + global=

Satisfied topic

21




➔ About qrels … ➔ Relevance (doc, topic) ∈ {0;1;2;3;4}➔ Principle: ''the more satisfied dimensions there are, the

better it is''

➔ Gradual qrels aware measure:Normalized Discounted Cumulative Gain [Järvelin & Kekäläinen, 2002]

➔ By topic: NDCG for each topic

➔ Global: meanNDCG for the system


No dimension

3 dimensions: Topic: ''trip around Glasgow'' Doc: trip + Bob born in Dumbarton

3 dimensions + global=

Satisfied topic

22

Outline



= partial




23

4. Experiments – Case Study with PIV GIRS

Case Study: PIV System➔ Indexing: 1 index per dimension

➔ Topical = Terrier IRS [Ounis et al, 2005] ➔ Spatial = map segmentation into tiles➔ Temporal = timeline segmentation into tiles

➔ Retrieval➔ Result document list for each index➔ Results combination with CombMNZ [Fox & Shaw, 1993; Lee, 1997]

CombMNZ

24


CombMNZ Principle [Fox & Shaw, 1993; Lee 1997]

25



26



27


Case Study: MIDR_2010 collection➔ Building Qrels: 12 volunteers (thanks!!!)

31 topics

5645 documents

=

paragraphs

Map for tracking spatial

information

Qrels

Relevance judgments

{0;1;2;3;4}

28

4. Experiments – Hypothesis Validated

Analysis of Collected Data➔ IRS Evaluation

➔ ResultsList × Qrels NDCG

➔ Results: geographic IRS most effective

trec_eval

Hypothesis

29

4. Experiments – Hypothesis Validated

Analysis of Collected Data➔ Results: geographic IRS most effective

30

Outline



= partial




31

Evaluation framework for Geographic IR Systems

Conclusions and Future Works (1/2)➔ Evaluation Framework for Geographic IR Systems

➔ Reusable➔ Generalizable for more dimensions: confidence,

freshness, ... [Costa Pereira et al., 2009]

➔ Not gradual relevance per dimension

➔ Case Study with PIV System➔ Creation of a specific test collection (≥ 25 topics)➔ French test collection➔ Limited collection (number of documents)

32

Evaluation Framework for Geographic IR Systems

Conclusions and Future Works (2/2)➔ Hypothesis Validated

➔ The 3 dimensions improve IR (+66.5%)

➔ Future Works➔ More precise analysis: by query➔ Quantify PIV improvements: various indexes combinations➔ Organize a GIRS evaluation campaign: anyone interested?

Damien Palacio - [email protected] 33

ECDL 20106-10 september 2010

Thank you!

34

Spatial Interface

35

Spatial Interface

36

Temporal Interface

37

Temporal Interface

38

Spatial Tiling

ecdl 2010 - measuring effectiveness of geographic ir systems in digital libraries: evaluation and...

Technology