benchmarking the effectiveness of associating chains of links for exploratory semantic search

Benchmarking the Effectiveness of Associating

Chains of Links for Exploratory Semantic Search

Laurens De Vocht Selver Softic, Ruben Verborgh, Erik Mannens, Martin Ebner, Rik Van de Walle

:Paris

?

2

:Paris :Anne_Hidalgo

:mayor

3

:Paris :Anne_Hidalgo

:mayor

:Bethlehem,_PA

?

4

:Anne_Hidalgo

?

5

:Anne_Hidalgo

?

:Bethlehem,_PA

:Anne_Hidalgo

:Bethlehem,_PA

Exploratory Semantic Search Engine

7

?

8

:Paris

9 :mayor :Anne_Hidalgo

< :birthPlace :San_Fernando,_Caldiz

9 :country :Spain

< :birthPlace :Edward_Ferrero

9 :battle :Battle_of_Roanoke_Island

< :battle :Charles_Adam_Heckman

9 :birthPlace :Easton,_Pennsylvania

9 :mouthMountain :Lehigh_River

9 :city

:Bethlehem,_Pennsylvania

A

9

:Paris

< :capital :France

< :citizenship :Cyril_Bourlon_de_Rouvre

9 :education :Aerospace_engineering

< :occupation :Dick_Johnson_(glider_pilot)

9 :almaMater :Mississippi_State_University

< :almaMater :Clara_Southmayd_Ludlow

9 :birthPlace :Easton,_Pennsylvania

< :mouthMountain :Lehigh_River

9 :city

:Bethlehem,_Pennsylvania

B

10

B A

?

How effective does an exploratory semantic

search engine reveal initially hidden associations,

as chains of links between interlinked resources?

Introduction

Exploratory Search

Benchmark Model

Motivating Example

Discussion and Conclusion

Introduction

Exploratory Search

Benchmark Model

Motivating Example

Discussion and Conclusion

[EXPLORATORY SEARCH: FROM FINDING TO UNDERSTANDING, Machionini, 2006]

Lookup Learn Investigation

Exploratory Search

`Learning searches involve multiple iterations and return sets of

objects that require cognitive processing and interpretation’

`Searches that support investigation involve multiple iterations that take place

over perhaps very long periods of time and may return results that are critically

assessed before being integrated into personal and professional knowledge bases’

Definition

15

1. Lookup

2. Relate/Expand

Lookup and learn: interpretation

16

lookup

expand relate

An exploratory semantic search engine

17

lookup

:Paris

Paris

18

expand

:Paris

:Paris

19

relate

20

Introduction

Exploratory Search

Benchmark Model

Motivating Example

Discussion and Conclusion

Iterative Exploratory Queries

Exploratory Semantic Search Engine

Datasets Baseline

Effectiveness

22

Effectiveness

The effectiveness E indicates the overall perception of the results by

the users taking into account expert-user feedback.

# user marked relevant objects

E =

# retrieved objects

Note:

E can be interpreted as precision in traditional IR.

Typical IR examine both precision and recall.

23

[TALKEXPLORER, Verbert et al., 2013]

Introduction

Exploratory Search

Benchmark Model

Motivating Example

Discussion and Conclusion

Motivating Example

ResXplorer.org

Everything Is Connected Engine

Virtuoso

User Study Extracted Queries

25

User Study Extracted Queries

1. lookup; 2. expand; 3. relate

26

LDOW

P(0)

P(1)

P(2)

P(3)

Effectiveness : Interpretation

27

Sample Results

Everything Is Connected Engine

28

Sample Results

Virtuoso

29

Introduction

Exploratory Search

Benchmark Model

Motivating Example

Discussion and Conclusion

Limitations

Only indicate comparisons to baseline within the same use case.

Not possible to use the benchmark as a leverage to compare different

approaches across use cases

Could better demonstrate in which aspects an exploratory approach

excels traditional systems.

31

Future Work

Put the results in perspective by indicating the nuances among different

expert user ratings.

Especially when there is expert disagreement or inconsistencies.

Facilitate generalization of the preliminary search context,

so results for engines can be reusable across datasets: avoiding that a

certain engine’s results differ strongly when changing the data and queries.

Make sure that the approach is generic and can be applied to other search

contexts with different data and use cases.

32

Benefits

Compare exploratory search engines to a baseline:

show use cases when the baseline can be outperformed;

for which queries the ‘engine under test’ is relatively more effective.

Sensitive to initial query keywords as inputted by the user:

when there are inconsistencies or vague terms,

even mismatches in the query context, or when expert users disagreed.

33

Contact

@laurens_d_v

laurens.devocht@ugent.be

http://slideshare.net/laurensdv

http://semweb.mmlab.be/