pragmatic evaluation of folksonomies

Knowledge Management Institute

Pragmatic Evaluation of Folksonomies

20th International World Wide Web Conference (WWW2011)Hyderabad, India

D. Helic, M. Strohmaier, C. Trattner, M. Muhr, K. Lerman

Markus StrohmaierAssistant Professor, Graz University of Technology, Austria

Visiting Scientist, (XEROX) PARC, USA

1

Markus Strohmaier 2011


Taxonomies: Categorization by Experts

Taxonomy: Usually produced and maintained byfew (e g dozens of) domain expertsfew (e.g. dozens of) domain experts.

Alternative: Folk-generated taxonomies( F lk i “)(„Folksonomies“)

But how useful are such hierarchical structures? How can they be evaluated?

2



Outline of this talk

1. FolksonomiesC t ti d E l tiConstruction and Evaluation

2 Decentralized Search2. Decentralized SearchJ. Kleinberg‘s algorithm

3. Pragmatic Evaluation FrameworkPresentation and discussion

4. Results & Findings

3








4



Tagging: Social classification by users

ResourcesUsers label and categorize

resources with concepts (tags)

U

Tags

is a tuple V:= (U, T, R, Y) whereth th di j i t fi it t U T R d t user 1

Users

• the three disjoint, finite sets U, T, R correspond to– a set of persons or users u ∈ U – a set of tags t ∈ T and

user 1

– a set of resources or objects r ∈ R

• Y ⊆ U ×T ×R, called set of tag assignmentstag 1 res. 1

Tag similarity based on

5


users and resources


Construction of FolksonomiesF t t lit t t litFrom tag centrality to tag generality:high tag centrality:

more abstract

low tag centrality:more specific

Other existing folksonomy algorithms: k-means, affinity propagation, …

6


[Heyman and Garcia-Molina 2006]


Semantic Evaluation of FolksonomiesSemantic Evaluation of Folksonomies

Emerging Hierarchy Expert Hierarchyg g y(Emergent)via e.g. hierarchical clustering

p y(Golden Standard)WordNet: a lexical DB for English

computers

programmingProgramming

Map-ping

Synset Hierarchy

distance d = 1 distance

Design Python

languages

distance d1 = 1 distance d2 = 2

gpatterns

g g

java python

Semanticgrounding

abs. difference |d1 - d2| a simple proxy for the quality

8


j pythonp p y q yof emergent semantics







9



Decentralized SearchDecentralized Search

Background knowledge:

Idea: use folksonomies as background knowledge Then, the performance of decentralized search

Shortest path to targetBackground knowledge:(a tag hierarchy)

g gpdepends on the suitability of folksonomies.

In other words, we can evaluate the suitability of

Folksonomy1

Folksonomy...

Folksonomyn

folksonomies for decentralized search through simulations.

shortest path found with l l k l d 4A (tag-tag) network:

Goal: Navigate from START to TARGETusing local and background knowledge

local knowledge pLK = 4

Δ = pLK-pGK

start target

using local and background knowledge only

shortest path with candidates

10

Markus Strohmaier 2011J. Kleinberg. The small-world phenomenon: An algorithmic perspective. Proc. 32nd ACM Symposium on Theory of Computing, 2000. Also appears as Cornell Computer Science Technical Report 99-1776 (October 1999)

pglobal knowledge pGK = 3







11



Pragmatic Evaluation Framework

General idea:• Use the OUTPUT produced by folksonomy algorithms

(hi hi l t t ) INPUT (b k d(hierachical structures) as INPUT (background knowledge) for decentralized search.

Framework Instantiation1. Generate n folksonomies K-means, Aff.Prop.,

DegCentrality, CloCentrality

2. Model navigational task exploratory navigation

3. Select evaluation metrics success rate, stretch

4 Sim late na igation decentralized search4. Simulate navigation decentralized search

5. Evaluate performance comparative evaluation

12



Simulating Exploratory NavigationTopically

tags

related tagsSTART TARGET

resources

Random R d

Topically related

resourcesRandom start

page: e.g. landing

page from

Random resource

resources

Usefulness of:

page from search engine

F lk F lk F lk

We generate 100.000 search pairs (start, target) for each dataset, and run simulations

13


Folksonomy1

Folksonomy...

Folksonomyn

run simulations







14



Success Rates Across Different FolksonomiesTag generalityflickr dataset

k-means /

Tag generality approaches

Random

affinity propagation

Random folksonomy

Success rate: The number of times an agent is successful

All approaches outperform a random folksonomy

The number of times an agent is successful in finding a path using a particular folksonomy as background knowledge

y

Tag generality approaches outperform k-means / Aff. Propagation

max hops n: the maximal number of steps an agent is allowed to perform before stopping (a tunable

t t l f ll li k )

n

16


Propagationparameter e.g., an agent only follows n links).


Success Rates Across Different Datasets

Holds for all datasets(to diff

But how efficient are

those(to diff. extents)

those folksonomies

during search?

17


search?


Stretch Δ = p pStretch Δ = pLK-pGKShortest Paths found with Local Knowledge

Bib K M

Finds no path: Δ = infinite

Bibsonomy K-Means

Δ infiniteFinds paths that is +1 longer:Δ = 1

T litHolds for all

d t t Finds shortest possible path:Δ = 0

Tag generality approaches (d+e) find much shorter

paths!

datasets(to diff. extents)

paths!

18



Pragmatic Evaluation Framework

Framework Instantiation Alternatives1. Generate n folksonomies K-means, Aff.Prop.,

DegCentrality, other folksonomy

algorithms orCloCentrality expert taxonomies

2. Model navigational task exploratorynavigation

other tasks

3 Select evaluation metrics success rate, stretch other evaluation metrics3. Select evaluation metrics4. Simulate navigation decentralized search actual click data

5. Evaluate performance comparativeevaluation

other evaluationapproachesevaluation approaches

Pragmatic evaluation produces different results for different tasks and different assumed or observed navigation behavior.

The evaluation framework is completely general with regard to

19


the task, data and evaluation metrics adopted.


Results & Findings: Summary

1 F lk i f l b k d k l d f1. Folksonomies are useful background knowledge for navigation.

2. Existing folksonomy algorithms are more useful than a random baselinethan a random baseline.

3 Tag generality approaches perform better than3. Tag generality approaches perform better than existing hierarchical clustering approaches.

4. Pragmatic results support theoretical analysis (not presented in talk – included in paper).

20


(not presented in talk included in paper).


Th k YThank You.

Markus [email protected]

D. Helic, M. Strohmaier, C. Trattner, M. Muhr, K. LermanPragmatic Evaluation of Folksonomies

20th International World Wide Web Conference (WWW2011)Hyderabad, India, March 28 - April 1, ACM, 2011.

http://kmi.tugraz.at/staff/markus/documents/2011_WWW2011_Folksonomies.pdf

21
