from expert finding to entity search on the web

From Expert Finding to Entity Search on the Web

Full-day Tutorial at ECIR 20121st April 2012

Gianluca Demartini, Peter Mika,Thanh Tran, Arjen P. de Vries

http://diuf.unifr.ch/main/xi/EntitySearchTutorial

Presenters

• Dr. Gianluca Demartini– eXascale Infolab, University of Fribourg, Switzerland– Research Interests:

• Entity Search• IR Evaluation• Semantic Web

01 Apr 2012 2ECIR 2012 Tutorial - From Expert Finding to Entity Search on the Web

Presenters

• Dr. Peter Mika – Senior Researcher, Yahoo! Research, Barcelona– Semantic Search group at Yahoo! Barcelona– Semantic Search, Web Object Retrieval, Natural

Language Processing


Presenters

• Dr. Thanh Tran – (Institut AIFB, Universität

Karlsruhe, Germany)– Semantic Search group at AIFB– Semantic Search, Semantic Data

Management, Linked Data Query Processing


Presenters

• Prof.dr.ir. Arjen P. de Vries– Interactive Information Access research group,

Centrum Wiskunde & Informatica (CWI); Delft University of Technology; Spinque

– Research interest: the intersection between information retrieval and databases

01 Apr 2012 5Van Rijsbergen, 1979ECIR 2012 Tutorial - From Expert Finding to

Entity Search on the Web

Entity

• An entity is a “proper noun”, “something that is referred to”

Outcome of “definition” discussion reported in SIGIR Workshop Report on The First International Workshop on Entity-Oriented Search (EOS), Krisztian Balog, Arjen P. de Vries, Pavel Serdyukov, Ji-Rong Wen, ACM SIGIR Forum, Vol. 45, No. 2, Dec. 2011, pp 43-50


Entity Search

• All those search tasks that aim at retrieving as answer to a user query an entity instead of a document– People, Countries, Movies, Restaurants, etc.


Motivation

• Information is entity-centric• Search for information is often conducted

around entities (Query log analysis)– Many queries (50%) search for specific entities

instead of documents [Kumar&Tomkins09]

• Traditional search retrieves a list of blue links• Novel web experiences may be designed

around entities instead01 Apr 2012 8ECIR 2012 Tutorial - From Expert Finding to


Here, for one search query “Nicole Kidman”,various entities make up the answer:

biophotosmoviestriviaquotes(...)

01 Apr 2012 9

Entity-centric Applications

• Enterprise applications• News portals• Movie portals• Product reviews• Search Engines


Entities in SERP


Entity Search: The Pipeline

• Entity Representation (DB/SW)• Entity Extraction (NLP)• Entity Linking and De-duplication (DB/SW)• Entity Storage and Indexing (DB/SW)• Entity Search and Ranking (IR)• Result presentation (HCI)• Evaluation (HCI/IR)


Outline

• FULL DAY Tutorial (sorry, this is not a joke :)• Morning

– Data (Peter)– Data Management (Thanh)

• Afternoon– Search and Ranking (Gianluca & Thanh)– Evaluation (Arjen)


Morning

• Data– Structured vs. Unstructured:– Entity Profiles: data models, entity identifiers,

standards – Datasets (Desktop, Enterprise, Wikipedia, Web, RDF)

• Data Management– Entity Extraction – Entity de-duplication / data fusion– Entity storage & indexing


Afternoon

• Search and Ranking– Expert Finding models– Entity Ranking in Wikipedia– Web Entity Retrieval– Entity Search over Structured Data– Relational Entity Search over Structured Data

• Evaluation– TREC Enterprise– INEX Entity Ranking– TREC Entity– SemSearch, Ad-hoc Object Retrieval


Data for Semantic Search

ECIR 2012 Tutorial - From Expert Finding to Entity Search on the Web

18

Data

• Web data– Information Extraction– Semantic Web

• Non-web data– Enterprise data– Desktop data– Email– ...

01 Apr 2012


19

Data on the Web

• Most web pages on the Web are generated from structured data– Data is stored in relational databases (typically)– Queried through web forms– Presented as tables or simply as unstructured text

• The structure and semantics (meaning) of the data is not directly accessible to search engines

• Two solutions– Information Extraction [see Part 2]– Relying on publishers to use Semantic Web formats

• Linked Data vs. metadata in HTML

01 Apr 2012


20

Semantic Web

• Sharing structured data across the Web– Standard graph-based data model

• RDF

– A number of syntaxes (file formats)• RDF/XML, RDFa

– Powerful, logic-based schema languages• OWL, RIF

– Query languages and protocols• HTTP, SPARQL

01 Apr 2012

21

Resource Description Framework (RDF)

• Each resource (thing, entity) is identified by a URI or otherwise it’s a blank node– URIs are globally unique

• Data is broken down into individual facts– Triples of (subject, predicate, object)

• A set of triples is published together in an RDF document

example:roi

“Roi Blanco”

name

typefoaf:Person

RDF document

01 Apr 2012

22

An RDF graph

peter#123

“Peter Mika”

namefoaf:Person

sameAs

peter#456worksWith

roi#234

“[email protected]”

email

type

type

01 Apr 2012


23

OWL, the Web Ontology Language

• The schema language for the Semantic Web– Classes, properties and restrictions on their usage– Allows validation and inference

• Schema is also data– Published just like any other RDF document– Queries can refer to both schema and data

• e.g. taxonomy expansion: retrieve instances of a class and instances of all subclasses

01 Apr 2012


24

Publishing RDF and OWL

• Linked Data– Data published as RDF documents linked to other RDF

documents• Typically RDF/XML or Turtle• Keep an eye on JSON-LD

– Community effort to (re-)publish open datasets • Embedded metadata

– RDFa, microdata, microformats annotations inside webpages– Recommended for site owners by Yahoo, Google, Facebook

• SPARQL endpoints– Triple stores (RDF databases) that can be queried through the

web01 Apr 2012


25

Linked Data

• Interlinked datasets on the Web– Often data from existing databases or APIs

• The four rules of Linked Data:– Use URIs to identify things.– Use HTTP URIs so that these things can be referred to

and accessed by people and crawlers.– Use standard formats such as RDF to provide useful

information about the thing when its URI is accessed– Include links to other datasets

• Most importantly: links to entities in other datasets that describe the same entity

01 Apr 2012

http://en.wikipedia.org/wiki/Uniform_Resource_Identifier

26

Peter’s homepage

Yahoo!

Friend-of-a-Friend ontologyLinked Data

peter#123

“Peter Mika”

namefoaf:Person

sameAs

peter#456worksWith

roi#234

“[email protected]”

email

type

type

01 Apr 2012


27

Linked (Open) Data = LOD• Advantages:

– No change to the publishing of the HTML documents– Data can be published by third party (e.g. Dbpedia)

• Disadvantages:– Web servers need to be configured to properly handle URIs that identify concepts

instead of documents– Not favored by search engines

• Lack of use cases• Crawling needs to be changed• Authority is difficult to determine

• Tools– Triple stores (Virtuoso, Oracle etc.) and front-ends (Pubby)– RDB-to-RDF mappers (e.g. D2RQ, Triplify)– Validators (Vapour)– Linked Data browsers (many)

01 Apr 2012

http://www4.wiwiss.fu-berlin.de/pubby

http://www4.wiwiss.fu-berlin.de/bizer/d2rq

http://triplify.org/Overview

http://validator.linkeddata.org/vapour

http://www.w3.org/wiki/TaskForces/CommunityProjects/LinkingOpenData/SemWebClients

Linked Data community

• Community effort to (re)publish open datasets as Linked Data– In particular, scientific and government data– see linkeddata.org and ckan.org for developer

information and datasets

http://ckan.org/


29

Linked Data in practice

• Fetching data dumps– See catalogs such as thedatahub.org, linkeddata.org

• Crawling Linked Data– Similar to HTML crawling, but the the crawler needs to parse

RDF/XML (and others) to extract URIs to be crawled– Semantic Sitemap/VOID descriptions– Existing crawls

• Billion Triples Challenge (2009-2011) datasets• LOD cache

• Querying SPARQL endpoints– See catalogs such as thedatahub.org, linkeddata.org– Semantic Sitemap/VOID descriptions

01 Apr 2012

http://sw.deri.org/2007/07/sitemapextension/

http://vocab.deri.ie/void/guide

http://sw.deri.org/2007/07/sitemapextension/

http://vocab.deri.ie/void/guide


30

Datasets

• Broad coverage datasets are linking hubs– Dbpedia– Freebase– Starting in 2012: Wikidata

• Domain-specific datasets form clusters– Biology– Government– Library– Entertainment – …

01 Apr 2012

31

Wikipedia

32

Dbpedia

Usingthe Dbpediaontology

Raw data


33

Metadata in HTML• 1995: HTML meta tags• 1998: RDF/XML• 2003: Web 2.0

– Tagging– Microformats– Metadata in Wikipedia– Machine tags in Flickr

• 2005: eRDF • 2008: RDFa 1.0• 2011: RDFa 1.1• 2012: Microdata

01 Apr 2012


34

HTML meta tags<HTML><HEAD profile="http://dublincore.org/documents/dcq-html/"><META name="DC.author" content="Peter Mika"><LINK rel="DC.rights copyright"

href="http://www.example.org/rights.html" /> <LINK rel="meta" type="application/rdf+xml" title="FOAF"

href= "http://www.cs.vu.nl/~pmika/foaf.rdf"> </HEAD> …</HTML>

01 Apr 2012


35

Microformats (μf)• Agreements on the way to encode certain kinds metadata in HTML

– Reuse of semantic-bearing HTML elements– Based on existing standards– Minimality

• Microformats exist for a limited set of objects– hCard (persons and organizations)– hCalendar (events)– hResume– hProduct– hRecipe

• Varying degrees of support and stability– hCard and rel-tag are widely supported

• Community centered around microformats.org– Specifications and discussions are hosted there

01 Apr 2012


36

Microformats: limitations

• No shared syntax– Each microformat has a separate syntax tailored to the

vocabulary • No formal schemas

– Limited reuse, extensibility of schemas– Unclear which combinations are allowed

• No datatypes• No namespaces, unique identifiers (URIs)

– no interlinking– mapping between instances is required

• Always appears in the HTML <body> 01 Apr 2012


37

Example: the hCard microformat

<cite class="vcard"><a class="fn url" rel="friend colleague met” href="http://meyerweb.com/">Eric Meyer</a> </cite> wrote a post (<cite><a href="http://meyerweb.com/eric/thoughts/2005/12/16/tax-relief/">Tax Relief</a></cite>) about an unintentionally humorous letter he received from the <span class="vcard”> <a class="fn org url" href="http://irs.gov/">Internal Revenue Service</a> </span>.

<div class="vcard"> <a class="email fn" href="mailto:[email protected]">Joe Friday</a> <div class="tel">+1-919-555-7878</div> <div class="title">Area Administrator, Assistant</div> </div>

01 Apr 2012


38

RDFa• W3C standard for embedding RDF data in HTML documents

– A set of new HTML attributes to be used in head or body– A specification of how to extract the data from these attributes

• RDFa is just a syntax, you have to choose a vocabulary separately• RDFa 1.0 is a W3C Recommendation since October, 2008

– RDFa Primer• RDFa 1.1 currently under standardization

– RDFa Core & RDFa Lite Working Draft as of January 31, 2012– Updated version of the RDFa Primer

• RDFa API for accessing RDFa data in a webpage in the browser from JavaScript– Currently Working Draft (April 19, 2011)

01 Apr 2012

http://www.w3.org/TR/rdfa-core/

http://www.w3.org/TR/rdfa-lite/

http://www.w3.org/TR/rdfa-primer/

http://www.w3.org/TR/rdfa-api/


39

RDFa 1.1

• Changes– New vocab attribute to define the default

namespace for the document or subtree– Syntax changes for ease of use– RDFa Lite profile

• RDFa 1.1 is backward compatible with RDFa 1.0– RDFa 1.1 is recommended if you want to use

HTML5

01 Apr 2012


40

Microdata

• Currently under standardization at the W3C– Working Draft (May 25, 2011)

• Microdata vs. RDFa– Microdata is simpler to author– Lacking some extension features such as co-typing

• HTML5 also has a number of “semantic” elements such as <time>, <video>, <article>, <section>…

01 Apr 2012

http://www.w3.org/TR/microdata/


41

Microdata example<div itemscope itemid=“http://www.yahoo.com/resource/person”> <p>My name is <span itemprop="name">Neil</span>.</p> <p>My band is called <span itemprop="band">Four Parts Water</span>. I was born on <time itemprop="birthday" datetime="2009-05-10">May 10th 2009</time>. <img itemprop="image" src=”me.png" alt=”me”> </p></div

01 Apr 2012


42

Example: Facebook’s Like and the Open Graph Protocol

• The ‘Like’ button provides publishers with a way to promote their content on Facebook and build communities – Shows up in profiles and news feed– Site owners can later reach users who have liked an object– Facebook Graph API allows 3rd party developers to access the data

• Open Graph Protocol is an RDFa-based format that allows to describe the object that the user ‘Likes’

01 Apr 2012

43

Example: Facebook’s Open Graph Protocol

• RDF vocabulary to be used in conjunction with RDFa– Simplify the work of developers by restricting the freedom in RDFa

• Activities, Businesses, Groups, Organizations, People, Places, Products and Entertainment

• Only HTML <head> accepted

<html xmlns:og="http://opengraphprotocol.org/schema/"> <head>

<title>The Rock (1996)</title> <meta property="og:title" content="The Rock" /> <meta property="og:type" content="movie" /> <meta property="og:url" content="http://www.imdb.com/title/tt0117500/" /> <meta property="og:image" content="http://ia.media-imdb.com/images/rock.jpg" /> …

</head> ...


44

Fragmentation of web markup

• Multiple schemas– Yahoo!’s SearchMonkey – June, 2008– Google announces Rich Snippets – June, 2009

• Faceted search for recipes – Feb, 2011

– Facebook’s Open Graph Protocol – April, 2010• ‘Verbs’ added to OGP – September, 2010

– Bing tiles – Feb, 2011• Different syntax

– Microformats, RDFa, microdata

01 Apr 2012


45

Schema.org

• Agreement between Bing, Google, and Yahoo on what markup webmasters should use– Help adoption by reducing fragmentation– Pre-competitive: each party will continue to build

competing products independently• Schema.org covers areas of interest to all three

parties– Business listings (local), creative works (video), recipes,

reviews– Expected to open up also to external contributions for non-

core areas01 Apr 2012


46

Example: schema.org

01 Apr 2012


47

Embedded metadata in practice

• 5-10% of webpages contain some explicit metadata– Statistics computed from commoncrawl.org give

different results• Schema.org helped resolve fragmentation

– Except Facebook• RDFa and microdata likely to co-exist

01 Apr 2012

Non-web Data


49

Enterprise Data

• Unstructured– Technical reports, Product Specification, etc.

• Semi-structured– E-mail, Spreadsheets

• Structured– Databases, Repositories

01 Apr 2012


50

Enterprise Search

• Challenges– Deal with data and format diversity– Index/search diverse datasets

• Vertical vs Centralized systems

– Deal with security and access control– Specific informational needs

• Expert Finding• Writing an overview

01 Apr 2012


51

Desktop Data

• Textual– Unstructured

• Txt documents

– Semi-Structured• E-mails, PDFs, Word files, etc. contain much metadata

• Multi-media– Pictures, Videos, Audio– Metadata

01 Apr 2012

Desktop Search

• IR techniques over unstructured data• Exploit

– the structure and metadata available– user activity logs (browsing history, file access patterns,

etc.)• Beagle++

– Hybrid search over inverted index and RDF store– E-mail context and attachments– Folder structure– Browser cache

Enrico Minack, Raluca Paiu, Stefania Costache, Gianluca Demartini, Julien Gaugaz, Ekaterini Ioannou, Paul-Alexandru Chirita, Wolfgang Nejdl: Leveraging personal metadata for Desktop search: The Beagle++ system. J. Web Sem. 8(1): 37-54 (2010)

01 Apr 2012


53

Tutorial Outline

• Morning– Data (Peter)– Data Management (Thanh)


01 Apr 2012

Data Management


55

Agenda

• Knowledge/Entity Extraction• Entity Linking• Entity De-duplication• Entity Storage & Indexing… very high-level overview of problems and solutions!… see tutorials on the specific problems!

01 Apr 2012

Knowledge/Entity Extraction

Source: Tadej Steiner from Jozef Stefan Institute, Ljubljana, Slovenia


57

Problem definition

• Knowledge extraction: – Extracting information from data and – Adding it to a knowledge base

01 Apr 2012


58

Problem definition

• Information extraction + knowledge acquisition

(textual) dataextracted infomation

knowledge base

Informationextraction

Knowledge acquisition

01 Apr 2012


59

Information extraction

• From the advent of the WWW, there are huge quantities of unstructured textual data, where manual information extraction would be infeasible

• How to extract information from text automatically with human-comparable quality

01 Apr 2012


60

Information extraction: early solutions

• Match manually defined patterns against text• Example:

– Patterns like “Pay ? from ? in favor of ?”– ATRANS (1986) inter-banking message exchange

01 Apr 2012


61


• How to transform a world (or domain) model from existing forms into a computer-friendly form– Conceptual knowledge (classes, rules, T-Box) VS.– Instance information (instance data, resource

descriptions, data records, A-Box) • Use cases for knowledge bases:

– Answering complex entity search queries / questions in general:

• “which scientists are also politicians?”01 Apr 2012


62


• Constructing a knowledge base is expensive– The Cyc KB was mostly manually constructed over

the last 20 years

• Coupling information extraction and knowledge acquistion lets us construct a knowledge base with no or little human effort

01 Apr 2012


63

Challenges

• Human effort:– Defining (domain-specific and domain-

independent) extraction patterns– Especially, in case of bootstrapping approaches:

• Specifying relations• Construction of training examples

– Maintaining knowledge base consistency

01 Apr 2012


64

Related research areas

• Natural language processing• Information extraction• Machine learning• Knowledge management

Knowledge extraction tools can be compared by these perspectives

01 Apr 2012


65

General knowledge extraction tools

• WebKB• TextRunner • Cyc• SOFIE with the corresponding YAGO

knowledge base• Read The Web• EntityCube

01 Apr 2012


66

Natural language processing

• Employed by most modern approaches• Part-of-speech tagging• Noun phrase chunking, used for entity

extraction• Abstraction of text

– From: “Slovenia borders Italy” – To:“noun – verb – noun”

01 Apr 2012


67

Information extraction: entities

• Entity extraction / Named Entity Recognition– “Slovenia borders Italy”

• Entity resolution– “Apple released a new Mac”.– From “Apple”, “Mac”– To Apple_Inc., Macintosh_(computer)

• Entity classification– Into a set of predefined categories of interest– Person, location, organization, date/time, e-mail address,

phone number, etc.– E.g. <“Slovenia”, type, Country>

01 Apr 2012


68

Some NER tools

• Java– Stanford Named Entity Recognizer

• http://nlp.stanford.edu/software/CRF-NER.shtml

– GATE• http://gate.ac.uk/

– LingPipe• http://alias-i.com/lingpipe/

• C– SuperSense Tagger

• http://sourceforge.net/projects/supersensetag/

• Python– NLTK

• http://www.nltk.org

01 Apr 2012

NER – list lookup

• Entities stored in lists (gazetteers)– E.g., Countries and cities

• Plus: Simple, fast, cross-language• Minus: list update, name variants (UPF,

Universitat Pompeu Fabra), ambiguity

01 Apr 2012 ECIR 2012 Tutorial - From Expert Finding to Entity Search on the Web 69


70

List lookup – ambiguities• Term level

– E.g. capitalized words: [All American Bank] vs. All [State Police]• Structure level

– “[Dolce and Gabbana]” vs “[Microsoft] and [Yahoo!]”• Type level

– John Smith (organization vs. person)– May (person vs. date vs. verb)– Washington (person vs. location)– 2015 (date vs. time)

Gazetteers not end solution but sources of background knowledge

01 Apr 2012

NER methods• Rule Based

– Regular expressions, e.g. capitalized word + {street, boulevard, avenue} indicates location

– Engineered vs. learned rules• NER can be formulated as classification tasks

– NE extraction: assign word mentions to tags (B beginning of an entity, I continues the entity, O word outside the entity)

– NE classification: assign entity mentions to categories (Person, Organization, etc.)– Use ML methods for classification: Decision trees, SVM, AdaBoost– Standard classification assumes cases are disconnected (i.i.d)

• Probabilistic sequence models: HMM, CRF– Each token in a sequence is assigned a label – Labels of tokens are dependent on the labels of other tokens in the sequence

particularly their neighbors (not i.i.d).


Naïve Bayes Classification• Determine category of xk by computing for each yi

• Priors P(Y=yi) and conditionals P(X=xk | Y=yi) estimated from data (via MLE), – E.g. If ni of the examples in D are in yi then P(Y=yi) = ni / |D|

• When categories are complete and disjoint, P(X=xk):

)(

)|()()|(

k

ikiki xXP

yYxXPyYPxXyYP

m

i k

ikim

iki xXP

yYxXPyYPxXyYP

11

1)(

)|()()|(

m

iikik yYxXPyYPxXP

1

)|()()(



73

Classification via Logistic Regression

• Instead of generative models, a descriminative model can be used to specifically focus on the conditional distribution P(Y | X)

• Assumes a parametric form for directly estimating P(Y | X)

• Basically a linear model

01 Apr 2012

n

i iiXwwXYP

10 )exp(1

1)|1(

n

i ii

n

i ii

Xww

Xww

10

10

)exp(1

)exp(

)|1(

)|0(1 iff 0 labelAssign

XYP

XYPY

n

i iiXww10 )exp(1

n

i iiXww10ly equivalentor

n

i iiXww100

n

i iiXww10ly equivalentor

)|1(1)|0( XYPXYP

ClassificationY

X1 X2 … Xn

Y

X1 X2 … Xn

Naïve Bayes

LogisticRegression

Conditional

Generative

Discriminative


Sequence LabelingY2

X1 X2 … XT

HMM

Linear-chain CRF

Conditional

Generative

Discriminative

Y1 YT

..

Y2

X1 X2 … XT

Y1 YT..


Sunita Sarawagi and William W. Cohen. Semi-Markov Conditional Random Fields for Information Extraction. In NIPS, 2005.

NER features • Gazetteers (background knowledge)

– location names, first names, surnames, company names• Word

– Orthographic• initial-caps, all-caps, all-digits, contains-hyphen, contains-dots, roman-

number, punctuation-mark, URL, acronym

– Word type• Capitalized, quote, lowercased, capitalized

– Part-of-speech tag• NP, noun, nominal, VP, verb, adjective

• Context– Text window: words, tags, predictions– Trigger words

• Mr, Miss, Dr, PhD for person and city, street for location


Exploiting Query Logs / Click-Through Data • Weakly-supervised entity Extraction from queries /

click-through data – A small set of seed instances for each entity type

• Learn – Patterns captured by LDA-based topic model: Probabilities

of query contexts and click websites of named entities for each class

– Template: common query prefix and postfix, e.g. “how did country gain independence”

• Apply patterns / templates to click-through data / query logs to mine new named entities Marius Pasca: Weakly-supervised discovery of named entities using web search queries. CIKM 2007:683-690

Gu Xu, Shuang-Hong Yang, Hang Li: Named entity mining from click-through data using weakly supervised latent dirichlet allocation. KDD 2009:1365-1374



78

Information extraction: relations

• Relation extraction– <“Slovenia”, “borders”, “Italy”>

• Relation resolution– <“Slovenia”, borders, “Italy”>– <“Slovenia”, next_to, “Italy”>

• We distinguish between open and bootstraped approaches

01 Apr 2012


79

Relation Extraction• Extracting relations

– Typical paraphrase problem: identify all the ways a relation may be expressed • Formulated as classification task, e.g. uses SVM

– Training data: parse tree, with nodes associate with a type as well as a role (e.g. role=member, role=affiliation to capture a person-affiliation relation)

– Tree-based kernel: two trees are similar if roots have same type and role, and each has a subsequence of children (not necessarily consecutive) with the same types and roles

– Examples are converted into such parse trees with role labels, and used to train the system

– SVM can then classify new examples of possible relations• Formulate as sequence labeling (semantic role labeling)• Joint inference: considers different types of features (syntactic,

semantic) and problems (extraction, resolution)

01 Apr 2012


80

Bootstrapped information extraction

• Provide examples for relationships which we want to extract

• Compromise: lower coverage, higher quality• Example: Sofie, ReadTheWeb

01 Apr 2012


81

Open information extraction

• We do not want to put constraints on the types of relationships we want to extract

• Very interesting for open-domain WWW datasets

• Example: TextRunner• Compromise: higher coverage, lower quality• Hybrid approaches:

– EntityCube combines both bootstrapped and open extraction

01 Apr 2012


82

Knowledge management

• Organization• Consistency management• Strictness

01 Apr 2012


83

Knowledge organization

• Lexicon: A set of entities and statements

• Ontology: A complex graph of formal concepts– Not only concrete entities, but also abstract

classes– Sofie/Yago, WebKB, ReadTheWeb, TextRunner

• Full world model: A Context-sensitive complex graph

• Cyc

01 Apr 2012


84

Knowledge consistency

• Consistency management– Not all extracted information is accurate– Inaccurate information leads to inconsistencies in

the knowledge base– Example:

• Having pattern “?x is mayor of ?y” and knowledge that <x,mayorOf,y> requires <x,type,Person> and <y,type,City>, we can filter out inconsistent information

01 Apr 2012


85

Knowledge consistency

• Examples:– SOFIE:

• Select the subset of statements which have the maximum satisfiability with regard to constraints

– ReadTheWeb:• Learns new constraints via semi-supervised boostrap

learning• Accuracy grows with ontology complexity

01 Apr 2012


86


• Bootstrapping– Using existing manually prepared knowledge to

generate new knowledge– While the knowledge base grows, the rules for

extraction also change

01 Apr 2012


87


• Strictness:– When do we consider entity and relationship

resolution important?

• Depends on use case:– Reasoning and data integration requires well-

formed and unambigouous entities and relations– Information retrieval can cope with not-well

formed relations and entities

01 Apr 2012


88

Machine learning

• Used in NLP, IE as well as knowledge acquisition

• Various approaches– Self-supervised– Semi-supervised– Supervised

01 Apr 2012


89

Machine learning

• Natural language processing– Part-of-speech learning

• Information extraction– Pattern learning

• ReadTheWeb, TextRunner, WebKB

• Knowledge acquisition– Rule learning (WebKB)– Constraint learning (ReadTheWeb)

01 Apr 2012


90

Summary

• Cyc– Full world model knowledge base

• WebKB– First attempt of automatically constructing a

knowledge base• TextRunner

– Open information extraction

01 Apr 2012


91

Summary

• EntityCube– Hybrid bootstrapped and open IE

• SOFIE/YAGO– Tight integration of natural language processing,

disambiguation and acquisition• Read The Web

– Constraint learning

01 Apr 2012

Entity Linking

Source: Tadej Steiner from Jozef Stefan Institute, Ljubljana, Slovenia


93

Basic situation

01 Apr 2012


94

Pipeline

1. Identify named entity mentions in source text using a named entity recognizer

2. Given the mentions, gather candidate KB entities that have that mention as a label

3. Rank the KB entities4. Select the best KB entity for each mention

01 Apr 2012


95

Pipeline

1. Identify named entity mentions in source text using a named entity recognizer

2. Given the mentions, gather candidate KB entities that have that mention as a label

3. Rank the KB entities4. Select the best KB entity for each mention

01 Apr 2012


96

Linking approaches - pair-wise linking

• Pair-wise linking: for each in-text entity, choose the candidate entity which is the best w.r.t. description similarity and textual features

• Is each disambiguation choice independent?– Pair-wise vs. collective disambiguation

01 Apr 2012


97

Important ranking features

• Mention popularity – P(entity|mention) – P(dbpedia:Kashmir_(song)|”Kashmir”) = 0.54 – P(dbpedia:Kashmir_(region)|”Kashmir”) = 0.91– Distribution of links and anchors in Wikipedia

Context similarity - sim(ctx(mention), ctx(entity))Context of a mention is the surrounding sentencesContext of an entity is the description of the entity (Wiki article)

Coherence / CollectiveEntities that appear together tend to be related to one anotherUsually solved by a greedy graph pruning algorithm

01 Apr 2012

98

Collective linking• For each in-text entity, choose the candidate entity

which is the most similar to the in-text entity and related to other entities that are already chosen.

01 Apr 2012

Tadej Stajner, Dunja Mladenic: Entity Resolution in Texts Using Statistical Learning and Ontologies. ASWC 2009:91-104Xianpei Han, Le Sun, Jun Zhao: Collective entity linking in web text: a graph-based method. SIGIR 2011:765-774



99

Relatedness

• Intuition: entities that co-occur in the same context tend to be more related

• How can we express relatedness of two entities in a numerical way?– Statistical co-occurrence– Similarity of entities’ descriptions– Relationships in the ontology

01 Apr 2012

100

Semantic relatedness

• If entities have an explicit assertion connecting them (or have common neighbours), they tend to be related

Elvis

Memphis

Elvis Presley

Memphis, Egypt

Memphis,TN

origin

Person

Location

type

type

type

St. Elvistype

01 Apr 2012


101

Co-occurrence as relatedness

• If distinct entities occur together more often than by chance, they tend to be related

Document

FC Barcelona

Bayern

FC Barcelona

Bavaria

Bayern München

Mutual information

Mutual information

x

y

x

y

x

y

z

01 Apr 2012

102

Content similarity as relatedness

• If distinct entities have higher similarity of their descriptions, they tend to be related

Document

a

b

x

y

z

similarity = 0.35

similarity = 0.25

similarity = 0.35

similarity = 0,7

similarity = 0,1

similarity = 0,2

01 Apr 2012

103

Architecture

Input textPreprocessing (entity extraction and consolidation)

.. with in-text

entities

Background knowledge(ontology)Match

retrieval

Entity description vectors

Assertion type informativeness

Entity co-occurences

.. with resolved entities

RelatednessEntity linking

104

Crowdsourcing for Entity Linking

Gianluca Demartini, Djellel Eddine Difallah, and Philippe Cudré-Mauroux.ZenCrowd: Leveraging Probabilistic Reasoning and Crowdsourcing Techniques for Large-Scale Entity Linking. In: 21st International Conference on World Wide Web (WWW 2012), Lyon, France, April 2012.


105


• Matching micro-task– Unclear (i.e., low confidence) matches are

crowdsourced– Top algorithmic results are presented to the

workers– Answers from the crowd are input to a

probabilistic network

01 Apr 2012


• Probabilistic Graph– Worker prior probability (from previous tasks)– Link prior probability (from algo matchers)– Link factors connect worker clicks and links

– SameAs constraints– Dataset unicity contstraints

01 Apr 2012

Entity De-duplication

“Entity Consolidation”“Entity Resolution”“Record Linkage”

“Instance Matching”Sources: Yongtao Ma from Karlsruhe Institute of Technology, Samur Araujo from

The Delft Bioinformatics Lab and Aidan Hogan from Digital Enterprise Research Institute


108

Structure

• Motivation• Problem and task overview• Consider only explicit owl:sameAs• Consider some lightweight reasoning• Inductive / instance matching methods

– Effectiveness– Efficiency

01 Apr 2012


109

Motivation 340， 000 Results

01 Apr 2012


110

Motivation• 2% of customer records obsolete in 1 month, due to deaths, name changes • $611B/year loss in US due to poor customer data• An average company has 49 different databases and spends 35% of its IT

dollars on integration efforts

01 Apr 2012


111

Motivation

01 Apr 2012


112

Hetereogenity in naming…

Tim Berners-Lee: URIs

…

timbl:i

dblp:100007

identica:45563

adv:timblfb:en.tim_berners-lee

db:Tim-Berners_Lee

= owl:sameAs

01 Apr 2012

http://richard.cyganiak.de/2007/10/lod/imagemap.html

http://images.google.com.hk/imgres?imgurl=http://www2005.org/keynotes/tim.jpg&imgrefurl=http://www2005.org/keynotes/&usg=__TWfydY0sN3IuwPp3y3Zb_oBkCNc=&h=2050&w=1411&sz=278&hl=zh-CN&start=5&zoom=1&itbs=1&tbnid=x8q3A6HSeRapDM:&tbnh=150&tbnw=103&prev=/images?q=tim+berners+lee&hl=zh-CN&safe=strict&gbv=2&tbs=isch:1

http://images.google.com.hk/imgres?imgurl=http://4.bp.blogspot.com/_cMDe93PF3Q0/Sw1DbCbYSVI/AAAAAAAAAKk/Us-BMFFje28/s1600/tim2_g1.7.gif&imgrefurl=http://blinking-links.blogspot.com/2009/11/tech-hacking-history.html&usg=__LaYkUQ8282f_od6N88ApjIxNDxU=&h=633&w=378&sz=143&hl=zh-CN&start=2&zoom=1&itbs=1&tbnid=7F-qcKdeY-6DnM:&tbnh=137&tbnw=82&prev=/images?q=tim+berners+lee&hl=zh-CN&safe=strict&gbv=2&tbs=isch:1

http://images.google.com.hk/imgres?imgurl=http://www.samogden.com/images/tim_berners-lee.jpg&imgrefurl=http://www.samogden.com/tim_berners-lee_photo.html&usg=__beqADInuZWHziJIyV8AxsVAxT2w=&h=451&w=432&sz=27&hl=zh-CN&start=12&zoom=1&itbs=1&tbnid=LTRieFKuHi4CzM:&tbnh=127&tbnw=122&prev=/images?q=tim+berners+lee&hl=zh-CN&safe=strict&gbv=2&tbs=isch:1

http://images.google.ie/imgres?imgurl=http://cdn.mikeabundo.com/wp-content/uploads/2009/03/tim-berners-lee.jpg&imgrefurl=http://mikeabundo.com/2009/03/14/twenty-years-of-the-world-wide-web/&usg=___idDcJPse_rC7jpEeO5VGxkfutM=&h=320&w=500&sz=31&hl=en&start=17&zoom=1&itbs=1&tbnid=5_qmVxHYPp5v6M:&tbnh=83&tbnw=130&prev=/images?q=tim+berners+lee&hl=en&newwindow=1&gbv=2&tbs=isch:1

http://images.google.ie/imgres?imgurl=http://i.telegraph.co.uk/telegraph/multimedia/archive/01175/berners-lee_1175434c.jpg&imgrefurl=http://www.telegraph.co.uk/news/uknews/road-and-rail-transport/6560308/Downing-Street-plan-to-put-transport-information-at-passengers-fingertips.html&usg=__vdiAPdeRBmo4CgYUGy2WkQ23zpk=&h=288&w=460&sz=12&hl=en&start=24&zoom=1&itbs=1&tbnid=39FQnj8z865C0M:&tbnh=80&tbnw=128&prev=/images?q=tim+berners+lee&start=18&hl=en&newwindow=1&sa=N&gbv=2&ndsp=18&tbs=isch:1

http://www.freebase.com/

http://images.google.ie/imgres?imgurl=http://upload.wikimedia.org/wikipedia/commons/6/62/Tim_Berners-Lee_CP.jpg&imgrefurl=http://commons.wikimedia.org/wiki/File:Tim_Berners-Lee_CP.jpg&usg=__P7vPVZGY9qdc7fGNQ0I1hI2Hxuk=&h=3215&w=2148&sz=1845&hl=en&start=21&zoom=1&itbs=1&tbnid=LtYcwmrU2diAGM:&tbnh=150&tbnw=100&prev=/images?q=tim+berners+lee&start=18&hl=en&newwindow=1&sa=N&gbv=2&ndsp=18&tbs=isch:1


113113

De-duplication for Web data

01 Apr 2012

http://richard.cyganiak.de/2007/10/lod/imagemap.html


Problem and Task Overview


115

Data integration – big picture• Ontology matching

– Widely studied in Semantic Web research, see e.g. list of publications at ontologymatching.org

• Entity de-duplication– Logic-based approaches in the Semantic Web– Studied as record linkage in the database literature – Machine learning based approaches, focusing on attributes– Graph-based approaches, see e.g. the work of Lisa Getoor are applicable to

RDF data• Improvements over only attribute based matching

• Blending / data fusion– Merging objects that represent the same real world entity and reconciling

information from multiple sources– Information quality / redundance

01 Apr 2012

http://ontologymatching.org/publications.html

http://www.cs.umd.edu/~getoor/

http://www.cs.umd.edu/~getoor/


116

De-duplication

• The problem of determining if two instances refer to the small real-world entity.

owl:sameas

Source Instances Target Instances

01 Apr 2012


117

1. Find equivalences in the data• Consider only explicit owl:sameAs (baseline)• Consider some lightweight reasoning (extended)• Inductive / instance matching methods

2. Rewrite data according to equivalences (data fusion)

De-duplication – task overview

01 Apr 2012


Consider only explicit owl:sameAs


119

• Use provided owl:sameAs mappings in the data

timbl:i owl:sameas identica:45563 .dbpedia:Berners-Lee owl:sameas identica:45563 .

• Store “equivalences” found

timbl:i ->identica:45563 ->dbpedia:Berners-Lee->

timbl:i

identica:45563

dbpedia:Berners-Lee

De-duplication

01 Apr 2012


120

• For each set of equivalent identifiers, choose a canonical term

timbl:i

identica:45563

dbpedia:Berners-Lee

De-duplication

01 Apr 2012


121

• Afterwards, rewrite identifiers to their canonical version:

De-duplication

timbl:i rdf:type foaf:Person .identica:48404 foaf:knows identica:45563 .

dbpedia:Berners-Lee dpo:birthDate “1955-06-08”^^xsd:date .

dbpedia:Berners-Lee rdf:type foaf:Person .identica:48404 foaf:knows dbpedia:Berners-Lee .

dbpedia:Berners-Lee dpo:birthDate “1955-06-08”^^xsd:date .

timbl:i

identica:45563

dbpedia:Berners-Lee

01 Apr 2012


Consider some lightweight reasoning


123

• Infer owl:sameAs through reasoning (OWL 2 RL/RDF)1. explicit owl:sameAs (again)2.owl:InverseFunctionalProperty3.owl:FunctionalProperty4.owl:cardinality 1 / owl:maxCardinality 1

foaf:homepage a owl:InverseFunctionalProperty .timbl:i foaf:homepage w3c:timblhomepage .adv:timbl foaf:homepage w3c:timblhomepage .

⇒timbl:i owl:sameas adv:timbl .

…then apply data fusion as before

Extended de-duplication

01 Apr 2012


Inductive / Instance Matching Methods


125

Agenda

• Problem overview• Attribute level

– (see term matching)• Instance level

– Effectiveness: learning – Efficiency: blocking

• Dataset level– (see collective entity linking)

01 Apr 2012


126

Problem overview effectiveness vs. efficiency

Instance Matching

Effectivity

Find correct matches!

Efficiency

Do it fast!

01 Apr 2012


127

Efficiency

O(NxM)

Source Target

Not efficient

01 Apr 2012

128

“Diclofenac” occurrence on DBPEDIA

01 Apr 2012


129

Problem overview – attribute level<A1, ‘Dave White’, ‘Intel’, ‘Male’> <P1, ‘Database…’, ‘John Black’, ‘Don White’><A2, ‘Don White’, ‘CMU’, ‘Male’> <P2, ‘Multimedia…’, ‘Sue Grey’, ‘D. White’><A3, ‘Susan Grey’, ‘MIT’, ‘Female’> <P3, ‘Title3…’, ‘Dave White’><A4, ‘John Black’, ‘MIT’, ‘Male’> <P4, ‘Title5…, ‘Don White’, ‘Joe Brown’><A5, ‘Joe Brown’, unknown, ‘Male’><P5, ‘Title6…’, ‘Joe Brown’, ‘Liz Pink’><A6, ‘Liz Pink’, unknown, ‘Female’> <P6, ‘Title7…’, ‘Liz Pink’, ‘D. White’>

Attribute level‘Don White’ , ‘D. White’

‘Don White’, ‘Dave, White’

• What (values?)

• How • Similarity metrics• Similarity threshold• Matching techniques

01 Apr 2012


130

Problem overview – instance level<A1, ‘Dave White’, ‘Intel’, ‘Male’> <P1, ‘Database…’, ‘John Black’, ‘Don White’><A2, ‘Don White’, ‘CMU’, ‘Male’> <P2, ‘Multimedia…’, ‘Sue Grey’, ‘D. White’><A3, ‘Susan Grey’, ‘MIT’, ‘Female’> <P3, ‘Title3…’, ‘Dave White’><A4, ‘John Black’, ‘MIT’, ‘Male’> <P4, ‘Title5…, ‘Don White’, ‘Joe Brown’><A5, ‘Joe Brown’, unknown, ‘Male’><P5, ‘Title6…’, ‘Joe Brown’, ‘Liz Pink’><A6, ‘Liz Pink’, unknown, ‘Female’> <P6, ‘Title7…’, ‘Liz Pink’, ‘D. White’>

Instance level <A1, ‘Dave White’, ‘Intel’, ‘Male’><A2, ‘Don White’, ‘CMU’, ‘Male’>

• What (attributes?)


01 Apr 2012

131


Problem overview – dataset level<A1, ‘Dave White’, ‘Intel’, ‘Male’> <P1, ‘Database…’, ‘John Black’, ‘Don White’><A2, ‘Don White’, ‘CMU’, ‘Male’> <P2, ‘Multimedia…’, ‘Sue Grey’, ‘D. White’><A3, ‘Susan Grey’, ‘MIT’, ‘Female’> <P3, ‘Title3…’, ‘Dave White’><A4, ‘John Black’, ‘MIT’, ‘Male’> <P4, ‘Title5…, ‘Don White’, ‘Joe Brown’><A5, ‘Joe Brown’, unknown, ‘Male’><P5, ‘Title6…’, ‘Joe Brown’, ‘Liz Pink’><A6, ‘Liz Pink’, unknown, ‘Female’> <P6, ‘Title7…’, ‘Liz Pink’, ‘D. White’>

Dataset level

• What (instances?)

01 Apr 2012


132

Agenda

• Problem overview• Attribute Level

– (see term matching)• Instance Level


• Dataset Level– (see collective entity linking)

01 Apr 2012


133

Character-based

• [see term matching in Part 3 on search & ranking]• Edit Distance [G98]

– Character Operations: insert, delete, replace– Given two strings, s and t, edit(s,t):

• Minimum cost of operations transforming s to t• Exp.: edit(Eorror, Eror)=1, edit(great,grate)=2

– Aiming at: common typing errors– Problem: works not well with other type of errors

• Exp.: D. White vs Dave White

• Jaro Rule

01 Apr 2012


134

Token-based

• Q-gram– The q-grams are short character substrings of

length q of the string – Example: 3-gram(White)={ ‘Whi’, ‘hit’, ‘ite’ }– set similarity then can be applied to the grams

01 Apr 2012


135

Agenda





01 Apr 2012


136

Questions

• Given instance attributes {Name, Institute, Gender, Publish}– Which ones are more important?– Which similarity measures should be adopted?– What is the threshold of similarity that should be

adopted?

01 Apr 2012

137

Bayes Decision Rule

• Notation– A,B are two tables, of n comparable fields– tuple pairs – classes: M (match) and U (non-match)– random vector , xi shows the level of

agreement of the ith field for

• Decision rule:

01 Apr 2012 called likelihood ratio


138

Bayes Decision Rule • Given training data, assume p(xi|M) and p(xj|

M) are independent for i≠j[5]

• Extension:– Using an expectation maximization (EM) algorithm

to estimate likelihood– Relax independent assumption– Decision with reject class

01 Apr 2012


139

Agenda





01 Apr 2012


140

Blocking strategies

Source Target

• Used to reduce the number of instance comparison• Non-overlapping partitions• Canopies and clustering – overlapping partitions

01 Apr 2012


141

Blocking strategies

Blocking

Attribute dependent

Attribute agnostic

When the source and target schema match.

Otherwise

01 Apr 2012


142

Attribute dependent

• Blocking Key Value (BKV)– Sorted Neighborhood approach – Q-grams blocking technique

• Blocking keys are highly discriminating attributes (e.g. last name, phone number)

• Targeting homogeneous datasets

b

a (blocking key)

c

d

01 Apr 2012

Sorted Neighborhood

• Motivation：– similar records have similar values– multiple “cheap” passes faster than an “expensive” one

• Goal: sort feature by a key to bring matching records close to each other

• Methodology:– Create a key for every record (e.g. first 3 characters of last

name)– Sort data by the key– Pair-wise comparison within a small sliding window– Multiple passes based on distinct key


Sorted Neighborhood

• Example:

ID Name SS Birthday ZIP

r1 David Black 123-45 01.05.1985 76137

r2 Dauid Black 123-45 01.06.1985 76137

r3 David White 325-52 23.09.1984 84212

r4 David B. 126-53 30.10.1983 84123

r5 David B. 745-32 07.05.1973 84212

r1

r2

r4

r3

r5

r2

r1

r3

r4

r5

ZIP[1..3]

Name[1..3]


Q-gram blocking

• Motivation: similar matches have high overlaps of q-grams• Goal: relaxes the edit distance constraint to a weaker count

constraint on the number of matching q-grams • Methodology:

given two strings s and t, and a edit distance constraint k– Count Filtering: s and t must share LBs,t=max(|s|,|t|)-1-(k-1)*q q-

grams– Position Filtering: s and t must share at least LBs,t positional q-grams– Length Filtering: ||s|-|t||≤k


Q-gram blocking

• Example:

3-gram

s=abaxabaaba ##a,#ab,aba,bax,axa,xab,aba,baa,aab,aba,ba$,a$$

t=abaabaaba ##a,#ab,aba,baa,aab,aba,baa,aab,aba,ba$,a$$

ED(s,t)≤k → |Q(s) ∩ Q(t)| ≥ max(|s|,|t|)-1-(k-1)*q

ED(s,t)=1, |Q(s) ∩ Q(t)|=9



147

Attribute dependent

• Learning the attributes (blocking keys)– Decision tree– Maximum hyper-rectangles

DrugBank DBPEDIA

LabelDrugnameSideeffectPage

TitleNameProducerComposition

01 Apr 2012


148

Attribute dependent

• Learning functions of similarity (e.g., Jaccard, Jaro, Levenshtein, Hamming, Cosine, etc.)

DrugBank DBPEDIA

Label= TitleDiclofenac Diclofenac Sodium =≈

01 Apr 2012


149

Attribute agnostic

• Designed for heterogeneous information space. (i.e., loose schema binding, noise, missing or inconsistent values, as well as an unprecedented level of heterogeneity)

• No knowledge about the schema

software

Corp. (blocking key)

radio

film

01 Apr 2012


150

Attribute agnostic

• “All tokens”• Reduce comparison space

– Block purging, – Block scheduling, – Block enumeration, – Duplicate propagation, – Comparisons propagation, and– Comparisons pruning.

software

Corp. (blocking key)

radio

film

01 Apr 2012

Entity Storage & Indexing


152

Indexing

• Search requires matching and ranking– Matching selects a subset of the elements to be scored

• The goal of indexing is to speed up matching– Retrieval needs to be performed in milliseconds– Without an index, retrieval would require scanning

through the collection• The type of index depends on the types of data

and queries to be supported– DB-style indexing– IR-style indexing

01 Apr 2012


153

IR-style indexing

• Index data as text– Create virtual documents from data– One virtual document per subgraph, resource or

triple• typically: resource

• Key differences to Text Retrieval– RDF data is structured– Minimally, queries on property values are required

01 Apr 2012

154

Horizontal index structure• Two fields (indices): one for terms, one for properties• For each term, store the property on the same position

in the property index– Positions are required even without phrase queries

• Query engine needs to support the alignment operator• Dictionary is number of unique terms + number of

properties

01 Apr 2012

155

Vertical index structure• One field (index) per property• Positions are not required

– But useful for phrase queries• Query engine needs to support fields• Dictionary is number of unique terms• Number of fields could be a problem for merging, query

performance

01 Apr 2012


156

Indexing using MapReduce

• MapReduce is the perfect model for building inverted indices– Map creates (term, {doc1}) pairs– Reduce collects all docs for the same term: (term,

{doc1, doc2…}– Sub-indices are merged separately

• Term-partitioned indices

01 Apr 2012

Search and Ranking


158

Outline

• Expert Finding models• Entity Ranking in Wikipedia• Web Entity Retrieval• Entity Search over Structured Data• Relational Entity Search over Structured Data

01 Apr 2012


159

From Documents to Entities

• Document Search

01 Apr 2012


160

From Documents to Entities

• Entity Search

Entities

1. Ent12. Ent23. Ent3

01 Apr 2012

161

A taxonomy of Entity Search tasks

01 Apr 2012


162

Expert Finding - Motivation

• Scenario– In large companies competencies and

skills are spread– Executives need to create a team for a

new project: find staff with the right expertise

– Someone needs to solve a problem– Example: I need an expert on ontology

engineering

01 Apr 2012


163

Expert Finding - Motivation

• Goal– Use the digital content available in the

enterprise– Create a ranking of people who are

experts in the given topic

01 Apr 2012


164

Motivation for System Support

• Busy experts do not have time to maintain adequate descriptions of their continuously changing specialized skills

• Expert seekers have poorly articulated requirements and are not fully enabled to judge a good expert from a bad one

01 Apr 2012


165

Complicating factors• Volume of communication/publication is not a

reliable indication of expertise• Certain topics engender more opinion than facts• Lack of information about past performance of

experts• New employees don’t know about informal social

networks• Access to expertise is often controlled (informally or

formally, by the experts or their management)• Solutions to complex problems require diverse

ranges of expertise

01 Apr 2012

166

Evidence of Expertise• Email or bulletin board messages• Corporate communications• Shared folders in file system• Resumes and homepages• Employee database• Email flow• Bibliographic information• Software library usage• Search and publication history• Project time charges

See also bibliography on TREC-ENT wiki:http://www.ins.cwi.nl/projects/trec-ent/wiki/index.php/Bibliography

Content

Social networks

Activities

01 Apr 2012

http://www.ins.cwi.nl/projects/trec-ent/wiki/index.php/Bibliography


167

Assumptions

• Content– Experts are mentioned in relevant documents– Experts author relevant documents

• Social networks– People that interact are likely to share expertise– Evidence in records of information exchange (and

co-authorship, co-work) and/or organizational structure

01 Apr 2012


168

Two Basic ApproachesWho should I ask about the copyright forms?

• Document-based: rank docs, extract experts

Copyright forms

LoriLori

LoriEllen

IanLori

Lori

EllenLori

1.

2. 1.

4.

5.

3.

01 Apr 2012


169

Document-based Expert Finding

• Find and score documents about the topic– Title about topic– Abstract about topic

• Aggregate scores for each distinct author

01 Apr 2012


170

Two Basic ApproachesWho should I ask about the copyright forms?

• Document-based: rank docs, extract experts

• Candidate-based: rank candidate profiles

Copyright forms

LoriLori

LoriEllen

IanLori

Lori

EllenLori

1.

2. 1.

4.

5.

3.

Lori

Copyright forms

Ellen

Ian

01 Apr 2012

Voting model

• Data fusion techniques• Each ranked document represents a vote for

the expertise of a candidate• Vote aggregation:

– Number of docs voting for each candidate– Scores of retrieved documents– Ranks of retrieved documents

01 Apr 2012

Craig Macdonald, Iadh Ounis: Voting for candidates: adapting data fusion techniques for an expert search task. CIKM 2006: 387-396


172

User-Oriented Model

• Additional real-world constraints

• Distance between user and expert– User previous knowledge on the topic– Contact time (organizational hierarchy, geo

location, collaboration)

01 Apr 2012

Elena Smirnova, Krisztian Balog: A User-Oriented Model for Expert Finding. ECIR 2011: 580-592


173

Additional TechniquesResearch Systems

• Combine the two basic approaches• Estimate the quality of the evidence• Use of collection/structural knowledge

– Treat emails different from documents– Treat email’s subject/sender/receiver different

from body– Locate homepages

See also TREC proceedings 2005-200701 Apr 2012


174

Additional TechniquesResearch Systems

• Use social network extracted from co-authorship or email lists

• Relevance propagation over expertise graph• Use Web Search evidence

01 Apr 2012


175

Expert Finding - References

– P@noptic Expert [Craswell et al. Ausweb01]– Balog’s Model 1 and 2 [Balog et al. SIGIR06]– Voting Model [Macdonald and Ounis CIKM06,

ECIR07, ECIR08]– Expertise evidence [Macdonald et al. ECIR08]– Vector Space Model [Demartini et al. ECIR09]– Web evidence [Serdyukov et al. TREC08]– User-Oriented Model [Smirnova and Balog ECIR11]

01 Apr 2012

Entity Ranking


177

Ranking…

• People• Actors• … Car companies

[i.e., insert your fav entity type here]

Entity Ranking!!!

01 Apr 2012

Wikipedia• Encyclopedia

– multilingual, Web-based, free-content, openly-editable: errors are promptly corrected

• Articles:– balanced, neutral, and encyclopedic, containing

notable verifiable knowledge• Categories / sub-categories• Links, anchor text (Germany -> Albert Einstein)


179

Entities in Wikipedia

• Art museums• Countries• Actors, Singers• Monarchs• Artists• Magicians• ...01 Apr 2012 ECIR 2012 Tutorial - From Expert Finding to



180

Example Entity Ranking Scenarios

• Impressionist art museums in Holland• Countries with the Euro currency• German car manufacturers• Artists related to Pablo Picasso• Countries involved in WWI• Actors who played Hamlet• English monarchs who married French women

Many examples on http://www.ins.cwi.nl/projects/inex-xer/topics/

01 Apr 2012

http://www.ins.cwi.nl/projects/inex-xer/topics/


181

Entity Ranking

• Topical query Q• Entity (result) type TX

• A list of entity instances Xs

• An entity is represented by its Wikipedia page• Systems employ categories, structure, links

01 Apr 2012


182

Tasks

• Entity Ranking (ER)– Given Q and T, provide Xs

• List Completion (LC)– Given Q and Xs[1..m]– Return Xs[m+1..N]

01 Apr 2012

183ECIR 2012 Tutorial - From Expert Finding to Entity Search on the Web

Topic 60Titleolympic classes dinghy sailing

Entities470 (dinghy) (#816578) 49er (dinghy) (#1006535) Europe (dinghy) (#855087)

Categoriesdinghies (#30308)DescriptionThe user wants the dinghy classes that are or have been olympic classes, such as Europe and 470.NarrativeThe expected answers are the olympic dinghy classes, both historic and current. Examples include Europe and 470.

TX

Q

Xs

01 Apr 2012

http://infao5501.ag5.mpi-sb.mpg.de:8080/topx/archive?link=Wikipedia-Lip6-2/816578.xml&style



Formal Model for Entity Ranking

– Indexing• Entities• Data Sources

“Alexandre Pato”ID: ap12dH5a(born in; 1989)(playing with; acm15hDJ)

Formal Model for Entity Ranking• Searching

– Users' Information Need– Entity Ranking System


186

Approaches to ES in Wikipedia

• Exploit and refine the category structure– Wordnet to find entity types (e.g., a professor is a

person)• Extend the query

– Synonyms and related words (Wordnet synsets)• Exploit the link structure

– Links in Wikipedia are usually entities– Search Keywords also in anchor text of outLinks

01 Apr 2012

YAGO– Suchanek et al. 2007– Highly accurate ontology

(>95%)– Extracted from Wikipedia +

WordNet– Provides semantic concepts

describing Wikipedia entities

Married... With

Children

Sitcoms

WordNet Synset

WikipediaCategory

Wikipedia Taxonomy

YAGO subClassOf

Situation Commedy


Category Based Search• Query expansion by modifying category

information– Subcategories

• Extracted from Wikipedia

– “Children” Categories• Filtered using the YAGO subClassOf relation

– “Sibling” Categories• Extracted from Wikipedia• Having with the same YAGO type


Subcategories

Sitcoms

WikipediaSubcategories

Latino Sitcoms

Sitcoms in Canada

BBC Television Sitcoms

Sitcom Characters

WikipediaCategory


“Children” Categories

Sitcoms

YAGO subClassOf

Latino Sitcoms

Sitcoms in Canada


Sitcom Characters

Situation Comedy

Fictional Character

WikipediaSubcategories

WikipediaCategory


“Sibling” Categories

Sitcoms

YAGO subClassOf

Latino Sitcoms

Sitcoms in Canada


Situation Comedy

WikipediaCategory

YAGOsubClassOf

...


192

Entity Search over Wikipedia

• Search for many different entity types with one system!

• Main observations– Link information is important– Cleaning the category structure of Wikipedia is critical

(YAGO)– NLP-based techniques on the user query improve

effectiveness

• Open issues– No temporal evolution of content is considered– Wikipedia is meant to be objective

01 Apr 2012 ECIR 2012 Tutorial - From Expert Finding to Entity Search on the Web

Gianluca Demartini, Claudiu S. Firan, Tereza Iofciu, Ralf Krestel, and Wolfgang Nejdl. Why Finding Entities in Wikipedia is Difficult, Sometimes. In: "Information Retrieval" 13(5): 534-567, Springer, October 2010.


193

Time-Aware Entity Retrieval

• In some cases the time dimension is available– News collections– Blog postings

• News stories evolve over time– Entities appear/disappear– Analyse and exploit relevance evolution– Decide about relevance at document level

• An Entity Search system can exploit the past to find relevant entitiesGianluca Demartini, Malik Muhammad Saad Missen, Roi Blanco, Hugo Zaragoza. TAER: Time

Aware Entity Retrieval. CIKM 2010, Toronto, Canada.01 Apr 2012


194


01 Apr 2012


195


• Evaluation– P3, P5, AvgPrec– Ties aware measures [McSherry and Najork, ECIR08]

• Paired t-test– ** p<<0.01– * p<0.05

• Related considered NonRelevant

01 Apr 2012


196

History Features

• We also tried– Weight history features with doc length– Weight history features with BM25

Feature P3 P5 MAP

F(e,d) .65 .56 .60

F(e,d1) .58 .53 .56

F(e,d-1) .64 .56 .62*

F(e,H) .66 .59** .66**

CoOcc(e,H) .62 .57 .65**

DF(e,H) .63 .57* .65**

01 Apr 2012

197

Dataset and Analysis

• TREC Novelty Track 2004– 25 event topics– 779 relevant news

• Entity annotations (7481 entities)• Relevance judgements

• How useful is to find relevant sentences?– P(e is Rel) 0.411 [0.404-0.417]– P(e is NotRel) 0.168 [0.163-0.173]– P(e is Rel | s is Rel) 0.547 [0.534-0.559]– Sentences:

21727 total 1.46 entity occurences 5122 relevant 1.88 entity occurences


198

Data Analysis

• How useful is looking at the past?– P(e|d1) 0.893 [0.881-0.905]

– P(e|d-1) 0.701 [0.677-0.726]

• Is useful to consider sentence co-occurence?

P(e1,e2) Relevant Related NotRelevant NotAnEntityRelevant 0.24 0.08 0.03 0.07Related 0.07 0.03 0.03

NotRelevant 0.07 0.05NotAnEntity 0.04


Approach

• Entity Ranking features for News articles– Local Features

F(e,d) FirstSenLen FirstSenPos Fsubj

AvgBM25(q,s) SumBM25(q,s)

History Features• Feature combination

– Linear and Machine Learning01 Apr 2012 ECIR 2012 Tutorial - From Expert Finding to

Entity Search on the Web 199

200

Local Features

Feature P5 MAP

F(e,d) .56 .60

FirstSenLen .36 .45

FirstSenPos .31 .43

Fsubj .44 .50

AvgBM25(q,s) .30 .41

SumBM25(q,s) .44 .52

Feature P5 MAP

All Tied .34 .42


201

Is the past useful?

• Looking at previous documents– Entity occurences so far F(e,H)– Docs where the entity appeared so far

DF(e,H)– Entity occurences in the previous doc

F(e,d-1)

– Frequency of entity the first time F(e,d1)

– Number of other entities with which the entity co-occured so far CoOcc(e,H)


202

History Features

• * t-test p value < 0.05 as compared with F(e,d)• ** t-test p value < 0.01 as compared with F(e,d)

Feature P5 MAP

F(e,d) .56 .60

F(e,d1) .53 .56

F(e,d-1) .56 .62*

F(e,H) .59** .66**

CoOcc(e,H) .57 .65**

DF(e,H) .57* .65**


2030 10 20 30 40 50 60

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Using the History

i-th document (i.e., history size+1)

AvgP

rec

Using the History

• Conclusion– Evidence from past documents is very important– Effectiveness should improve over time (run F(e,H))

01 Apr 2012

204

Discussion

• New search task: Time-Aware Entity Retrieval

• Constructed evaluation benchmark• Experimental Evaluation

– Investigated some features and combinations

– Information from the past helps most– Obtain 15% improvement over F(e,d)


Ranking Entities on the Web

206


• TREC Entity Track 2009-2010– 50M web pages (including Wikipedia)– Find related entities (return homepages)



207


• Approaches– Use Wikipedia (and infoboxes) as background info– Extract entities from tables and lists– Find the homepage given the entity name (see

ENS)• Barack Obama -> www.barackobama.com

• Since 2010: 1 billion web pages

01 Apr 2012

http://www.barackobama.com/


208

Related Entity Finding

• Approaches– Kaptein et al., CIKM10

• Exploits Wikipedia to improve entity retrieval effectiveness• Identifies entity types• Wikipedia external links as source for entity homepage• Anchor text index for entity search

– Bron et al, CIKM10• Entity co-occurence• Entity type filtering• Context (relation type)• Wikipedia-based homepage finding

01 Apr 2012


209

Discussion

01 Apr 2012


210

Expert Finding - Key Requirements

• Identify experts via self-nomination and/or automated analysis of expert communications, publications, and activities

• Classify the type and level of expertise of individuals and communities

• Validate the breadth and depth of expertise of an individual

• Recommend experts, including the ability to rank order experts on multiple dimensions including skills, experience, certification and reputation

01 Apr 2012


211

Current systems

• Hardly validate the breadth and depth of expertise– Count mentions– Weight with relevance score– Sometimes weight with authority of document

containing candidate mention• Do not really attempt to classify the type and

level of expertise

01 Apr 2012


212

Evidence of Expertise

• Information about true expertise is often not explicit in artifacts (as opposed to factual knowledge)

• Information about expertise is expressed using specialized terms and concepts

01 Apr 2012


213

How to improve?

• Integrate more sources of evidence– CV information– Project related data

• Including temporal information

– Training data (HR dept)• Cost of achieving this evidence for expert vs.

non-expert as weighting factor– Participation in TREC, authoring a book, getting a

PhD in IR, ... Raymond D'Amore, Expert Finding in Disparate Environments, 2008

01 Apr 2012


214

However...

• Two types of challenges to be overcome:– System challenge– Evaluation challenge

01 Apr 2012


215

System Challenges

• Multi-lingual entity extraction• Privacy management

– E.g., Tacit can email top N experts with private profiles (only recipient knows)

• Interoperability with heterogeneous data sources– IMAP, Exchange, Lotus Notes– LDAP, JDBC/ODBC, XML repositories, Peoplesoft,

Oracle Financials, Word/Excel/PDF, ...

01 Apr 2012


216

Where is my data?

• > 80% of data not in relational databases– Documents, spreadsheets, presentations– Web pages– Email, instant messages, news feeds– Images, audio, video

01 Apr 2012


217

Dataspaces

• The complete set of information belonging to one organization or task

• Examples:– Personal dataspace – Enterprise dataspace– Community dataspace

• E.g., scientific, sports club, ...

“From Databases to Dataspaces: A New Abstraction for Information Management”, Michael Franklin, Alon Halevy, David Maier, SIGMOD Record, December 2005.

01 Apr 2012


218

Conclusions so far...

• Expert finding could in principle use many more resources that indicate expertise, possibly more reliably, but it is difficult to setup the research– System challenges– Data availability

• Motivates research in operational setting– E.g., Raymond D'Amore, Expert Finding in

Disparate Environments, 2008

01 Apr 2012


219

Entity Search - Discussion

• Similar challenges as Expert Finding• Entity information is spread over the Web

– In different formats (HTML, RDF, images)– It is redundant (Wikipedia, DBPedia, homepage)– It varies over time (e.g., population of a country)– It is inconsistent (neutrino vs light speed)

01 Apr 2012


220

Entity Search - References

• Approaches exploit– Wikipedia structure (links, categories)

• Kaptein et al., CIKM10 (REF)• Demartini et al., IRJ 2010 (XER)

– Entity relations• Bron et al., CIKM10 (REF)• Demartini et al., CIKM10 (TAER)

– Graph-based methods• Rode et al., INEX08 (XER)• Iofciu et al., ECIR11 (XER)

– Probabilistic Models• Weerkamp et al., INEX08 (XER)• Balog et al., ECIR10 (XER)

01 Apr 2012


221

Entity Search - Discussion

• Structured data may be the way to improve search effectiveness– Entity identifiers– Entity relations

01 Apr 2012

Ad-hoc Object Retrieval


223

Introduction

• Unstructured or hybrid search over RDF data– Supporting end-users

• Users who can not express their need in SPARQL

– Dealing with large-scale data• Giving up query expressivity for scale

– Dealing with heterogeneity• Users who are unaware of the schema of the data• No single schema to the data

– Example: 2.6m classes and 33k properties in Billion Triples 2009

• Entity search– Queries where the user is looking for a single entity named or

described in the query– e.g. kaz vaporizer, hospice of cincinnati, mst3000

01 Apr 2012

224

Use cases in web search

Top-1 entity with structured data

Related entitiesStructured dataextracted from HTML


225

Architecture overview

Doc

1. Download, uncompress, convert (if needed)

2. Sort quads by subject

3. Compute Minimal Perfect Hash (MPH)

map

map

reduce

reduce

map reduce

Index

3. Each mapper reads part of the collection

4. Each reducer builds an index for a subset of the vocabulary

5. Optionally, we also build an archive (forward-index)

5. The sub-indices are merged into a single index

6. Serving and Ranking

1st part of the talk 2nd part

01 Apr 2012


226

Vertical index structure (reminder)• One field (index) per property• Positions are not required• Query engine needs to support fields✓ Dictionary is number of unique terms✓ Occurrences is number of tokens

✗ Number of fields is a problem for merging, query performance• In experiments we index the N most common properties

01 Apr 2012


227

BM25F RankingBM25(F) uses a term-frequency (tf) that accounts for the decreasing marginal contribution of terms

where vs is the weight of the field

tfsi is the frequency of term i in field s

Bs is the document length normalization factor:ls is the length of field s

avls is the average length of s bs is a tunable parameter

01 Apr 2012

Roi Blanco, Peter Mika, Sebastiano Vigna: Effective and Efficient Entity Search in RDF Data. International Semantic Web Conference 2011:83-97

228

BM25F ranking cont.

• Final term score is a combination of tf and idf

wherek1 is a tunable parameter

wIDF is the inverse-document frequency:

• Finally, the score of a document D is the sum of the scores of query terms q

01 Apr 2012


229

Hierarchical entity model

• Unstructured, structured and hierarchical entity model • Hierrachical entity model

– Predicate type generation– Predicate generation: importance of a predicate within its type– Term generation: importance of a term is determined by the predicate

in which it occurs and all other predicates of that type in the entity

01 Apr 2012

Robert Neumayer, Krisztian Balog, and Kjetil Nørvåg. On the modeling of entities for ad-hoc entity search in the web of data. In ECIR'12.


230

Query Independent Ranking

• The question is not which answer is more relevant; i.e. all answers are relevant

• The task is finding out which of the answers should be ranked higher

• Importance is subjective• Closely related to the popularity of a resource?

01 Apr 2012

Lorand Dali, Blaz Fortuna, Thanh Tran Duc and Dunja MladenicLearning the Query-Independent Ranking of RDF Entity Search Results In Proceedings of 9th Extended Semantic Web Conference (ESWC'12)


231

Towns from Andhra Pradesh

• Hyderabad• Srisailam• Chittoor• Masulipatnam• Chandavaram• Mahbubnagar• Gooty• Vijaywada• …

1. All answers are relevant2. Ranking is important3. Ranking is static4. Hard to obtain the true ranking

01 Apr 2012


232

Learning to Rank• Machine learning approach to building a ranking

model• We know the true ranking (golden standard)• We represent each answer (resource) as a feature

vector• The final score is a linear combination of the features,

and the weights have to be learned

A

B

C

Pairwise preferences

A better than B

A better than C

B better than C

true ranking

01 Apr 2012


233

Ranking Features

• Importance derived – from Graph analysis– from Wikipedia– from Web search engine– from other external sources (N-gram databases)

01 Apr 2012


234

Graph Features

01 Apr 2012


235

Graph Features

• Pagerank• Hubs and Authorities• RDF graph features

– nRSubj - number of relations where this resource appears as the subject

– nRObj - number of relations where this resource appears as the object

– nLiteral - number of relations where this resource appears as the subject and the object is a literal

01 Apr 2012


236

Importance of Wikipedia Pages

• Popularity– How many people visited a particular page during June-

January 2010– Data obtained from the Wikipedia access logs available at:

http://dammit.lt/wikistats– Captures importance from the point of view of users

• Page length– How much text a Wikipedia page contains– Importance from the authors’ perspective

• Number of edits– Importance from the editors’ perspective

01 Apr 2012

http://dammit.lt/wikistats

http://dammit.lt/wikistats


237

Features Approximating Importance Correlate Well

• Compare rank based on page length and based number of edits with page popularity

Spearman’s CC NDCG

Page length 0.60 0.84

Number of edits 0.78 0.93

01 Apr 2012


238

Web Search Features

• How many search results do we get in a web search if we search for:– The answer’s name– Keywords from the answer’s description

• We used Yahoo! BOSS services to do the search

• Querying the web for many resources is expensive

01 Apr 2012


239

N-gram features

• Similar to web search features• We look how many times the name of a

resource appears in a large N-gram database (e.g. Google N-grams, Google Book N-grams, etc.)

• A cheaper way to see how many times a resource appears on the web or in books

01 Apr 2012

Relational Entity Search


241

Introduction• Intuitive keyword search interface over databases• “A direction” of semantic search, which employs semantics of

– Relational information (structured data) in – Different datasets to produce complex structured, aggregated results to answer complex information needs

• Short version of the Semantic Search tutorial at ESSIR’11– Matching Techniques – Ranking Techniques

• Complementary to DB keyword search tutorial, emphasizes – The role of textual data: data graphs with textual content nodes– Ranking

[Chen et al, SIGMOD09]

01 Apr 2012


Matching


243

Structure• Keyword search: keywords over data graphs

– Term matching – Content matching – Structure matching

• Schema-based keyword search• Schema-agnostic keyword search

– Online search algorithms– Index-based approaches

01 Apr 2012


244

Keyword search approaches• Finding “substructures” matching keyword nodes• Different result semantics for different types of data

– Textual data (Web pages connected via hyperlink)– DB (tuple connected via foreign keys)– XML (elements/attributes via parent-child edges)

• Commonly used results: Steiner tree / subgraph– Connect keyword matching elements– Contain one keyword matching element for every query keyword – Minimal substructures: closely connected keyword nodes

• Query is ambiguous, lacks explicit structure constraints– NP-hard, thus efficiency of matching is a problem– Large amounts on candidate matches, thus ranking is a problem

01 Apr 2012


245

Keyword search on hybrid data graphs

Alice

Bob is a good friend of mine. We went to the same university, and also shared an apartment in Berlin in 2008. The trouble with Bob is that he takes much better photos than I do:

trouble with bob

creator

Bob

knowssunset.jpg

creator

titleBeautiful Sunset

knows Thanhworks

author

KIT locate

d

Germany

Semantic Search

year

2009

Germany

author

Peterworks

FluidOps 34age

knows someone works at KITapartment shared Berlin Alice

Example information need“Information about a friend of Alice, who shared an apartment with her in Berlin and knows someone working at KIT.”

Term matching

Content matching

Structure matching

01 Apr 2012


246

Term matching

• Distance-based (syntax)– Levenshtein distance (edit distance)– Hamming distance– Jaro-Winkler distance

• Dictionary-based (semantics)– Taxonomy– Dictionary of similar words– Translation memory– Ontologies

01 Apr 2012

247

Content matching• Retrieve partial matches

• Inverted list (inverted index) ki {< d1, pos, score, ...>,

< d2, pos, score, ...>, ...}

• Combine partial matches: union or join

shared

shared berlin alice= =

shared Berlin Alice shared Berlin Alice

D1 D1 D1

01 Apr 2012


248

Structure matching• Retrieve structured data given patterns (e.g. triple patterns)

• Index on tables• Multiple “redundant” indexes to cover different access patterns

• Combine: union or join• Blocking, e.g. linear merge join (required sorted input)• Non-blocking, e.g. symmetric hash-join• Materialized join indexes

SP-index PO-index

==

=

?x ns:knows ?y. ?x ns:knows ?z. ?z ns: works ?v. ?v ns:name “KIT”

Per1 ns:works ?v ?v ns:name “KIT”

Per1 ns:works Ins1 Ins1 ns:name KIT

Per1 ns:works Ins1 Ins1 ns:name KIT

Structure not explicitly given in query exploration / other kinds of join

01 Apr 2012


249





01 Apr 2012


250

Matching in keyword search – schema-based

Alice Bob KIT

• Operate on schema graph • Query interpretation

– Compute queries instead of results– Query presentation – Query processing by DB engine

• Leverage the power of underlying DB query engine

Result 1

Result 2

[Tran et al, ICDE09]

[Hristidis et al, VLDB02] [Agrawal et al, ICDE02]

[Qin et al, SIGMOD09]

01 Apr 2012


251





01 Apr 2012

252

Matching in keyword search – schema-agnostic

Alice Bob KIT

• Operate on data graph– No schema needed– Flexibly support different types of data e.g. hybrid data graphs– Native tailored optimization

• Online in-memory graph search • Using materialized indexes

Result 1

Result 2

[He et al, SIGMOD07] [Li et al, SIGMOD08][Tran et al, CIKM11]

[Kacholia et al, VLDB05]

01 Apr 2012

253

Online search – top-k exploration • Compute Steiner tree with distinct roots• Backward expansion strategy• Run Dijkstra’s single-source-shortest-path algorithms

– Explore shortest keyword-root paths– To find root (an answer)– Until k answers are found– Approximate: no top-k guarantee, i.e. further answers found later from other

expansion paths may have higher score• Complete top-k: terminate safely when lower bound of top-k candidate is

higher than upper bound of what can be achieved with remaining inputs

[Bhalotia et al, ICDE02]

Alice Bob KIT

Result 1

01 Apr 2012


254

Taxonomy of matching approaches• Schema-based vs. schema-agnostic • Online search

– Complete top-k– Approximate top-k – Backward expansion, bidirectional search, undirected

subgraph exploration, dynamic programming• Indexing for retrieval + join for combine

– Path retrieval, then path join – Graph retrieval, then graph pruning– Graph retrieval, then neighborhood / graph join

(neighborhood indexed as a set of paths)01 Apr 2012


Ranking


256

Structure• Ranking paradigms

– Explicit model of relevance– No notion of relevance

• Features– Content-based– Structure-based– Structured-content-based

01 Apr 2012

257

Ranking paradigms• No explicit notion of relevance: similarity between the query

and the document model– Vector space model (cosine similarity)

– Language models (KL divergence)

• Explicit relevance model– Foundation: probability ranking principle– Ranking results by the posterior probability (odds) of being

observed in the relevant class:

)),...,(,),...,((),( ,,1,,1 qkqdtd wwwwCosdqSim

)|(

)|(log()|()||(),(

d

qq

Vtdq tP

tPtPKLdqSim

))|(1()|( )|(

DtDt

NtPRtPRDP

01 Apr 2012


258

Features• Features are orthogonal to retrieval models

– Weights for query / document vectors?– Language models for document / queries?– Relevance models?– What to use for learning to rank?

01 Apr 2012


259

Features Dealing with ambiguities

• Content features– Co-occurrences

• Terms K that often co-occur form a contextual interpretation, i.e. topics (cluster hypothesis, distributional semantics)

• “Berlin” and “apartment” geographic context Berlin as city

– Frequencies: d more likely to be “about” a query term k when d more often, mentions k (probabilistic IR)

• Structure features– Structured-content-based: consider relevance at fine-

grained level of attributes– Link-based popularity– Proximity-based

Termambiguity

Contentambiguity

Structureambiguity

01 Apr 2012


260

Content-based features – frequency • Document statistics, e.g.

– Term frequency – Document length

• Collection statistics, e.g. – Inverse document frequency

– Background language models)|()1(

||)|( CtP

d

tftP d

idfd

tfw dt

||,

• An object is more likely about “Berlin”?

• When it contains a relatively high number of mentions of the term “Berlin”

• When number of mentions of term in the overall collection is relatively low

01 Apr 2012


261

Structure-based features – links • PageRank

– Link analysis algorithm– Measuring relative importance of nodes– Link counts as a vote of support – The PageRank of a node recursively depends on the number

and PageRank of all nodes that link to it (incoming links)• ObjectRank

– Types and semantics of links vary in structured data– Authority transfer schema graph specifies connection

strengths – Recursively compute authority transfer data graph

• An object (about “Berlin”) is more important? • When a relatively large number of objects are linked to it

[Hristidis et al, TDS08]

How to incorporate it into a content-based retrieval model?

01 Apr 2012

262

• EASE, XRANK, BLINKS, etc.• EASE

– Proximity between a pair of keywords

– Overall score of a JRT is aggregation on the score of keyword pairs

• XRANK– Ranking of XML documents / elements– Proximity of n is defined based on w, the smallest text window in n

that contains all search keywords

Structure-based features – proximity

• A structured result (e.g. Steiner tree) is more relevant? • When it is more compact s.t. elements are closely related

[Li et al, SIGMOD08]

[Guo et al, SIGMOD03]

adopted from: [Chen et al, SIGMOD09]

How to incorporate it into a content-based retrieval model?


263

Structured-content-based model• Consider structure of objects during content-based

modeling, i.e., to obtain structured content-based model– Content-based model for structured objects, structured

documents, database tuples…

)|()|( fFf

fd

d

tPtP

• An object is more likely about “Berlin”? • When its (important) fields / attributes contain a relatively

high number of mentions of the term “Berlin”

01 Apr 2012


264

P(w|Q) w

.077 palestinian

.055 israel

.034 jerusalem

.033 protest

.027 raid

.011 clash

.010 bank

.010 west

.010 troop

…

sample probabilities

UMM

k

iik MqPMwPMPqqwP

11 )|()|()()...,(

Palestinian

Israeli

raids

???

q1q2q3

w

M

M

M

)...(

)...,()...|()|(

1

11

k

kk qqP

qqwPqqwPRwP

Structured-content-based model Relevance model [Lavrenko et al, SIGIR01]

01 Apr 2012


265

Structured-content-based model Edge-specific relevance model

• Given a query Q={q1,…,qn}, a set of resources (FR) are retrieved– E.g. Q={Hepburn, Holiday}, FR = {m1, p1, p4,m2, p2m2,m3}

• Based on FR results, an edge specific RMFR is constructed for each unique edge e:

[Bicer et al, CIKM11]

01 Apr 2012


266

• Edge-specific resource model:

– Smoothing with model for the entire resource

• Use RM for query expansion: the score of a resource calculated based on cross-entropy of edge-specific RMFR and edge-specific RMr:

– Alpha allows to control the importance of edges

Structured-content-based model Edge-specific relevance model

01 Apr 2012


267

Ranking Steiner tree / join result tuples (JRT)

• Ranking aggregated JRTs: – The cross entropy between the edge-specific RMFR (query model) and

geometric mean of combined edge-specific RMJRT:

• The proposed ranking function is monotonic with respect to the individual resource scores (a necessary property for using top-k keyword search algorithms)

01 Apr 2012


268

Taxonomy of ranking approaches

• Explicitly vs. non-explicitly relevance-based• Different approaches for model construction

– Content-based ranking– Structure-based ranking– Structured-content-based ranking

01 Apr 2012


269

• Agrawal, S., Chaudhuri, S., and Das, G. (2002). DBXplorer: A system for keyword-based search over relational databases. In ICDE, pages 5-16.

• Amer-Yahia, S. and Shanmugasundaram, J. (2005). XML full-text search: Challenges and opportunities. In VLDB, page 1368.

• Bao, Z., Ling, T. W., Chen, B., and Lu, J. (2009). Effective xml keyword search with relevance oriented ranking. In ICDE, pages 517-528.

• Bhalotia, G., Nakhe, C., Hulgeri, A., Chakrabarti, S., and Sudarshan, S. (2002). Keyword Searching and Browsing in Databases using BANKS. In ICDE, pages 431-440.

• Bicer, V., Tran, T. (2011): Ranking Support for Keyword Search on Structured Data using Relevance Models. In CIKM.

• Bizer, G., Lehmann, J., Kobilarov, G., Auer, S., Becker, C., Cyganiak, R., Hellmann, S. (2009): DBpedia - A crystallization point for the Web of Data. J. Web Sem. (WS) 7(3):154-165

• Chen, Y., Wang, W., Liu, Z., Lin, X.(2009): Keyword search on structured and semi-structured data. SIGMOD 2009:1005-1010

• Ding, B., Yu, J. X., Wang, S., Qin, L., Zhang, X., and Lin, X. (2007). Finding top-k min-cost connected trees in databases. In ICDE, pages 836-845.

• Golenberg, K., Kimelfeld, B., and Sagiv, Y. (2008). Keyword proximity search in complex data graphs. In SIGMOD, pages 927-940.

• Guo, L., Shao, F., Botev, C., and Shanmugasundaram, J. (2003). XRANK: Ranked keyword search over XML documents. In SIGMOD.

• He, H., Wang, H., Yang, J., and Yu, P. S. (2007). BLINKS: Ranked keyword searches on graphs. In SIGMOD, pages 305-316.

• Hristidis, V., Hwang, H., and Papakonstantinou, Y. (2008). Authority-based keyword search in databases. ACM Trans. Database Syst., 33(1):1-40

01 Apr 2012


270

• Hristidis, V. and Papakonstantinou, Y. (2002). Discover: Keyword search in relational databases. In VLDB.

• Kacholia, V., Pandit, S., Chakrabarti, S., Sudarshan, S., Desai, R., and Karambelkar, H. (2005). Bidirectional expansion for keyword search on graph databases. In VLDB, pages 505-516.

• Kimelfeld, B. and Sagiv, Y. (2006). Finding and approximating top-k answers in keyword proximity search. In PODS, pages 173-182.

• Ladwig, G., Tran, T. (2011): Index Structures and Top-k Join Algorithms for Native Keyword Search Databases. In CIKM.

• Lavrenko, V. Croft, W.B. (2001): Relevance-Based Language Models. In SIGIR, pages 120-127.• Li, G., Ooi, B. C., Feng, J., Wang, J., and Zhou, L. (2008). EASE: an effective 3-in-1 keyword search

method for unstructured, semi-structured and structured data. In SIGMOD.• Liu, F., Yu, C., Meng, W., and Chowdhury, A. (2006). Effective keyword search in relational

databases. In SIGMOD, pages 563-574.• Luo, Y., Lin, X., Wang, W., and Zhou, X. (2007). SPARK: Top-k keyword query in relational

databases. In SIGMOD, pages 115-126.• Qin, L., Yu J. X., Chang, L. (2009) Keyword search in databases: the power of RDBMS. In SIGMOD,

pages 681-694.• Tran, T., Herzig, D., Ladwig, G. (2011): SemSearchPro: Using Semantics throughout the Search

Process. In Journal of Web Semantics, 2011. • Tran, T., Wang, H., Rudolph, S., Cimiano, P. (2009): Top-k Exploration of Query Graph Candidates

for Efficient Keyword Search on RDF. In ICDE. • Vagelis Hristidis, L. G. and Papakonstantinou, Y. (2003). Efficient ir-style keyword search over

relational databases. In VLDB.

01 Apr 2012


271

Entity Search - Conclusions

• Historically:• Expert Finding came first• Generalized to Entity Search

– First on Wikipedia (easier)– Then on the Web (harder)

• Over structured data– AOR– Relational Entity Search

01 Apr 2012


272

Entity Search - Conclusions

• What helps for Expert Finding?– Good entity extraction– Diverse evidence (Enterprise, web, user profiles)

• What helps for Entity Search?– Wikipedia / DBPedia– Entity types– Anchors

01 Apr 2012


273

Next Steps for Entity Search

• Improve effectiveness for existing tasks– Errors propagate along the pipeline– Improve individual components (extraction,

linking, de-duplication, etc.)– Improve ranking models by considering additional

evidence (as for expert finding)• Work on top of Entity Search

– New User experiences (based on entities)

01 Apr 2012


274

Next Steps for Entity Search

• Novel entity-oriented search tasks– Summary generation– Entity Attribute Search (“At which age Nobel prize

winners in physics died?”)– Entity Popularity (rank Nobel laureates by

popularity)– Relation Search (“how are Michelle Obama,

Barack Obama, George Bush related?”)– Event Search (“Jazz concerts in Barcelona”)

01 Apr 2012


Tutorial Outline

• Morning– Data (Peter)– Data Management (Thanh)


01 Apr 2012 275

Evaluation


277

Outline

• TREC Enterprise• INEX Entity Ranking• TREC Entity• INEX Data Centric• SemSearch challenge

01 Apr 2012


278

Laboratory Evaluation in IR

• An IR test collection:– A collection of documents– A collection of search requests– Relevance assessments for each search request

against the collection of documents

• Compare mean effectiveness (e.g., ratio of relevant vs. non-relevant documents retrieved) on ‘many’ search requests

01 Apr 2012


279

Relevance

TREC working definition:If you were writing a report on the subject of the topic and would use the information contained in the document in the report, then the document is relevant.

01 Apr 2012


280

Relevance – issues

• Binary relevance– Regardless of how small the piece of relevant

information is in relation to the rest of the document!

• Considered only in isolation– Regardless of what other relevant information the

user has already seen

01 Apr 2012


281

Question:

• How well does this working definition of relevance capture ‘real’ searcher behaviour?– Ideal PhD student – Intelligence officer– Medical doctor– Patent lawyer– My mum

01 Apr 2012


282

Question:

• What happens if we change the definition of relevance to reflect different types of searches?

01 Apr 2012


283

Entity Ranking

• Many users search for specific entitiesinstead of just any type of documents

01 Apr 2012


284

Ranking People

• Expert Finding in TREC-ENT (Enterprise Track)• Collection:

– Corpus: crawl of *.w3.org sites– People: names of 1092 people who may be experts

• Query: – ‘information retrieval’

• Results: – A list of people who know about information retrieval

01 Apr 2012


285

C-TEST

• TODO• C-TEST

– http://es.csiro.au/C-TEST

01 Apr 2012

http://es.csiro.au/C-TEST


286

Expert Finding

01 Apr 2012


287

Evaluation Challenge

• How to get realistic data for research evaluation?

• So far: build corpora from externally facing parts of company intranets

01 Apr 2012


288

Evaluation: Enterprise Track

• Goal– Build a test collection for Enterprise Search

• Corpora– W3C corpus (2005, 2006)– CSIRO corpus (2007)

• Tasks– Email search (2005, 2006)– Expert finding (2005-2007)– Document search (2007)

01 Apr 2012

W3C Collection

• A crawl of the W3C web, containing email lists, wiki, web pages, etc.– 5.7 Gb crawl of w3.org

sites (in June 2004) – 331,037 documents,

including ~175,000 emails

• A candidate list



290

W3C Limitations

• Topics and assessments developed by researchers instead of users

• Difficult to situate information needs in an organization you are not part of – representative of new members?

• Public e-mail lists; no private/personal communication, and receivers unknown

01 Apr 2012


291

CERC

• Real enterprise– CSIRO is the Commonwealth Scientific and

Industrial Research Organisation– 50 sites in Australia and around the world – 17 divisions, conducting research in areas such as

industrial physics, mining, and ICT• Real users

– Science communicators

01 Apr 2012


292

CERC Limitations

• Real data, but...• TREC has to use externally-facing data

– Publicly available pages from csiro.au only– Can’t address security, databases, click data

• No candidate list!

– Participants do the assessments

01 Apr 2012

293

UvT Expert Collection

• Based on a publicly accessible database of UvT employees who are involved in research or teaching

• ~38,000 documents– Research descriptions– Course descriptions– Publications– Personal homepages

Balog et al., Broad Expertise Retrieval in Sparse Data Environments, SIGIR 2007

01 Apr 2012

294

UvT Expert Collection

• Clean, heterogeneous, structured, and focused (but comprises a limited number of documents)

• Contains hierarchical information from an organization (institutes and faculties)

• List of expertise areas provided by the employees themselves– http://ilk.uvt.nl/uvt-expert-collection/

Balog et al., Broad Expertise Retrieval in Sparse Data Environments, SIGIR 2007

01 Apr 2012

http://ilk.uvt.nl/uvt-expert-collection/

‘In Situ’ Evaluation

Thomas and Hawking, Evaluation by comparing result sets in context, CIKM 2006

01 Apr 2012 295

296

‘In Situ’ Evaluation

• Advantages– Evaluation on live search system– Click-through data useful in inferring preference,

even in the absence of explicit user judgements.• Limitations

– Experimenter effect– Experiments not repeatable– A is better than B, but by how much? Multi-way

comparisons?

Thomas and Hawking, Evaluation by comparing result sets in context, CIKM 2006

01 Apr 2012


297

Alternative benchmark model?

• Standard benchmark distributes the data + queries + relevance information

• Q: Could we distribute the systems instead?– Need organization(s) willing to evaluate in own

intra-net, with their own users– Need participants with robust systems that run

off-site, unmonitored...

01 Apr 2012


298

INEX Entity Ranking

01 Apr 2012


299

Entity Ranking

• Topical query Q• Entity (result) type TX

• A list of entity instances Xs

01 Apr 2012


300

Topic 60Titleolympic classes dinghy sailingEntities470 (dinghy) (#816578) 49er (dinghy) (#1006535) Europe (dinghy) (#855087) Categoriesdinghies (#30308)DescriptionThe user wants the dinghy classes that are or have been olympic classes, such as Europe and 470.NarrativeThe expected answers are the olympic dinghy classes, both historic and current. Examples include Europe and 470.

TX

Q

Xs

01 Apr 2012





301

INEX-XER

• INEX XML Entity Ranking Track• Assumptions:

– Entities (Xs) are represented as Wikipedia pages– Binary relevance

01 Apr 2012


302

Examples of Wikipedia Entities (TX)

• Art museums and galleries• Countries• Famous people• Monarchs of the British Isles• Artists• Magicians

01 Apr 2012


303

Entity Ranking Scenarios (Q,TX)

• Impressionist art museums in Holland• Countries with the Euro currency• German car manufacturers• Artists related to Pablo Picasso• Countries involved in WWI• Actors who played Hamlet• Films shot in Venice• Paul Auster novels

Many more examples on http://www.ins.cwi.nl/projects/inex-xer/topics/

01 Apr 2012

http://www.ins.cwi.nl/projects/inex-xer/topics/


304

Tasks

• Entity Ranking (ER)– Given Q and TX, provide Xs

• List Completion (LC)– Given Q and Xs[1..m]– Return Xs[m+1..N]

01 Apr 2012


305

Topic 60Titleolympic classes dinghy sailingEntities470 (dinghy) (#816578) 49er (dinghy) (#1006535) Europe (dinghy) (#855087) Categoriesdinghies (#30308)DescriptionThe user wants the dinghy classes that are or have been olympic classes, such as Europe and 470.NarrativeThe expected answers are the olympic dinghy classes, both historic and current. Examples include Europe and 470.

01 Apr 2012





306

Easy Baseline?

//article[about(Category:T and Q)]/title

01 Apr 2012


307

But not an easy task…

• Categories related to car companies:– Peugeot_vehicles – Citroën_vehicles– Mercedes-Benz platforms– DaimlerChrysler– Motor vehicle manufacturers of Germany – List_of_automobile_manufacturers – Companies established in 1926 – …

01 Apr 2012

http://wikipedia.hiit.fi/alvis-wikipedia/front?query=car+manufacturers+citroen+mercedes+bmw+renault&categories=entity-cat=%22Category:Peugeot_vehicles%22

http://en.wikipedia.org/wiki/Category:DaimlerChrysler

http://en.wikipedia.org/wiki/Category:Motor_vehicle_manufacturers_of_Germany

http://wikipedia.hiit.fi/alvis-wikipedia/front?query=car+manufacturers+citroen+mercedes+&categories=entity-list=%22List_of_automobile_manufacturers%22

http://en.wikipedia.org/wiki/Category:Companies_established_in_1926


308

XML Structure to Rescue?• Q = ‘James Bond’• Xs = [Sean Connery, Roger Moore, …]

• Corpus data:– Category:James Bond Actors– Passage

Over 40 years after he first played the role, Connery is still widely regarded as the definitive cinematic incarnation of James Bond, despite popular interpretations by the likes of Roger Moore, Pierce Brosnan and Timothy Dalton, the latter often considered closer to the Bond of the novels.

– Passage To date, six actors have portrayed James Bond in the official series. They are:

• Sean Connery (1962–1967; 1971) • George Lazenby (1969) • Roger Moore (1973–1985) • Timothy Dalton (1987–1989) • Pierce Brosnan (1995–2002) • Daniel Craig (2006–present)

• So, should exploit XML Structure!!!

01 Apr 2012

http://en.wikipedia.org/wiki/Roger_Moore

http://en.wikipedia.org/wiki/Pierce_Brosnan

http://en.wikipedia.org/wiki/Timothy_Dalton

http://en.wikipedia.org/wiki/Sean_Connery

http://en.wikipedia.org/wiki/George_Lazenby

http://en.wikipedia.org/wiki/Roger_Moore

http://en.wikipedia.org/wiki/Timothy_Dalton

http://en.wikipedia.org/wiki/Pierce_Brosnan

http://en.wikipedia.org/wiki/Daniel_Craig


309

Structure in the Collection

• Wikipedia contains semantically very rich annotations:– Wikipedia Categories– Wikipedia Lists– Times (1930, 1931, 1932, etc. etc.)– Names– Disambiguation pages

01 Apr 2012

31001 Apr 2012


311

So... Recall our question:

• What happens if we change the definition of relevance to reflect different types of searches?

01 Apr 2012


312

INEX-XER Results

• XER Topics created by participants:– Today, 38 genuine XER topics with completed

assessments

– ER: 8 groups, 18 runs submitted– LC: 6 groups, 10 runs submitted– Pool depth 50

01 Apr 2012


313

Ad-hoc vs. XER

• Many ad-hoc topics can serve as entity ranking topics (but articles containing relevant passages must be re-assessed)– INRIA&RMIT created 30 training topics based on

the INEX 2006 ad-hoc collection– XER Organizers created a similar set of 18 topics

based on the INEX 2007 ad-hoc collection

01 Apr 2012


314

Not relevant for XER…

• Articles on topic are not necessarily relevant entities– Actually, they are surprisingly often not!

• Articles that are or contain hubs (e.g., ‘list of …’ pages) are not entities either, so they are also not considered relevant

01 Apr 2012


315

Ad-hoc vs. XER

• On the 18 ad-hoc ’07 topics that were re-assessed as XER topics, only about 35% out of original relevant documents have been assessed as relevant– Most relevant Steven Seagul results are movies

(17/20); but most ‘French president in the 5th republic’ results are not presidents or not presidents in the 5th republic (6/129), and most Bob Dylan songs are not songs (32/267)

01 Apr 2012


316

Ad-hoc vs. XER

• E.g., ‘Bob Dylan songs’ relevant articles/passages in ad-hoc, but non-relevant entities in XER:– Bob Dylan– The Band– Albums (however… what about track list?)– Don’t look back (film)– March on Washington for Jobs and Freedom (speech)– 19XX [in music], 19XX grammy awards…– Singer-songwriters that play covers/tributes/…– Cities where Bob Dylan livedEtc.etc.

01 Apr 2012


317

XER Close(r) to ‘Real’ User Need?

• Example: Ad-hoc topic 509, ‘songs of Bob Dylan’– Relevant passage…

The theme song is called This Wheel's on Fire written by Bob Dylan and performed in this instance by Julie Driscoll.

…however hidden in middle of lengthy page about the ‘Absolutely fabulous’ TV series

• Clearly, the song’s article itself is the preferable result, even for the ad-hoc task

01 Apr 2012




318

Perceived pool quality

• Participants missed only some entities they know of…– Toyota (‘hybrid cars’)– Goat (‘herbivorous mammals’)– Hillary Clinton (‘women in politics’)

• More of a problem for ad-hoc derived pools?

01 Apr 2012


319

Using hubs for validation

• Steven Seagal entity page– Film-o-graphy:

• Prince of Central Park not in assessments• Further web search: producer, not actor!

01 Apr 2012

320

Q: Non-topic-authors as judge?

Q: Use external information for judging?

01 Apr 2012

Q: Assessor’s Topic Knowledge not in Collection?



322

TREC (Web) Entity (Search)

01 Apr 2012


323

Related Entity Finding (REF)

• Topics:– Input Entity:

Name + Homepage– Target Type:

Person | Organisation | Product | Location– Narrative:

Description of the relation in free text

01 Apr 2012


324

Entity Representation(on the Web)

• Primary Entity Homepage:devoted to and in control of the entity– Gain = 3

• Relevant Entity Homepage: devoted to the entity, but not in its control– Gain = 1

• Not an Entity Homepage:– Wikipedia Page– News articles and blog posts, even if exclusively about the

entity

01 Apr 2012


325

The 2009 Pilot

• 20 topics– Target Type:

Person | Organisation | Product– 13 participants

01 Apr 2012


326

Example Topics

• Organizations that award Nobel prizes• CDs released by the King's Singers• Companies that John Hennessey serves on the

board of• Universities that are members of the SEC

conference for football• Scotch whisky distilleries on the island of Islay

01 Apr 2012


327

Lessons Learned

• Not that many entities in ClueWeb B– Makes it difficult to define good topics, especially

product topics• Wikipedia/DBPedia dominate approaches and

results

01 Apr 2012


328

2010• ClueWeb A English• 50 new REF topics

– Target Type: Person | Organisation | Product | Location

– 47 topics judged, pool depth 20• 20 2009 topics re-assessed• Wikipedia pages not accepted as valid answers,

primary homepages rewarded more– Evaluation script:

http://trec.nist.gov/data/entity/10/eval-entity.pl • 14 participants

01 Apr 2012

http://trec.nist.gov/data/entity/10/eval-entity.pl


329

2011

• Modifications to simplify the evaluation and to improve cross-system comparison– Binary relevance (only Primary HP)– Groups using Web Search Engines should also submit a run

using the Lemur ClueWeb Online Query Service• 50 new REF topics

– Target Type not limited (extracted from Narrative)– Support document per answer (could be WP page)

• 10 participants (only 4 doing REF )

01 Apr 2012

http://www.google.com/url?q=http://lemurproject.org/clueweb09.php/index.php


330

Entity List Completion (2010)

• BTC-2009 – Billion Triple Collection – LOD sample, 1.14B statements, 17GB zipped

• 13 topics– Out of 20 REF 2009, skipping topics without

results or without example entities– Mapped type to most specific class in DBPedia

ontology– Mapped known relevant entity HPs to URIs in BTC-

2009

01 Apr 2012


331

ELC Assessments (2010)

• By track coordinators(yes, I have read my share of the semantic web )

• Many answers of correct type, but not in correct relation to example entities (and, many descriptors of input entity had been retrieved)

• 5 topics without relevant entities, 1 topic “un-judgeable”

01 Apr 2012


332

“Semantic” Sub-tasks 2011

• Sindice-2011 dataset– Various Linked Data RDF dumps– 1.7B entities, 11B RDF statements (230M docs)– Web documents containing RDF, RDFa and

Microformats documents (crawled by the Sindice Bot, since 2009)

http://data.sindice.com/trec2011/ S. Campinas, D. Ceccarelli, T. E. Perry, R. Delbru, K. Balog, and G. Tummarello. The Sindice-2011 Dataset for Entity-Oriented Search in the Web of Data. In EOS, SIGIR 2011 workshop, July 28, Beijing, China.

01 Apr 2012

http://data.sindice.com/trec2011/

http://research.microsoft.com/en-us/um/beijing/events/eos2011/13.pdf


333

“Semantic” Sub-tasks 2011

• REF-LOD– Like REF, but results are URIs in Sindice instead of

docids in ClueWeb• But, not a re-usable collection – only 1 participant

• Entity List Completion– Like ELC 2010, but examples not given as URI, but

only as ClueWeb identifier

01 Apr 2012


334

Meanwhile at SemEval...

01 Apr 2012


Web People Search

• WePS1 (2007) focused on Disambiguation of person names in search results– determine #people for that person name– assign documents to each referent

• WePS2 (2009) introduced Attribute Extraction– 18 attribute types including birthdate and –place,

occupation, affiliation, ...

• WePS3 (2010) introduced disambiguation for organization names (motivated by Online Reputation Management)

http://nlp.uned.es/weps/01 Apr 2012

http://nlp.uned.es/weps/


336

Other Entity Oriented Evaluations

• Where to go, now that TREC Entity is “on a break”?!

01 Apr 2012


337

TREC Knowledge Base Acceleration

• Given– Incoming text stream (news and social media

content)• First month w/ human-generated labels as training data

– A target entity from a knowledge base (for now: people, specified by their Freebase and Wikipedia entries)

• Score each item (“document”) based on how “pertinent” it is to the target KB node

01 Apr 2012


338

TAC Knowledge Base Population• Tasks related to extracting information about entities with

reference to an external knowledge source (Wikipedia infoboxes)

• KBP 2011 had three tasks: – entity-linking: given an entity name (person, organization, or

geopolitical entity) and a document containing that name, determine the KB node for that entity or add a new node for the entity if it is not already in the KB

– slot-filling: given a named entity and a pre-defined set of attributes (“slots”) for the entity type, augment a KB node for that entity by extracting all new learnable slot values from a large corpus of documents

– temporal slot-filling: similar to the regular slot-filling task, but also requests time intervals to be specified for each extracted slot value.

01 Apr 2012


339

CLEF RepLab

• Online reputation management (ORM)– Continuation of the WePS3 ORM task– Two tasks, evaluated on Twitter data:

• Monitoring task: thematically cluster tweets including a company’s name

• Profiling task: annotate tweets according to their polarity

01 Apr 2012


340

INEX Data-Centric Track 2011

• IMDB data collection– XML generated from plain text files published on

the IMDB web site on April 10, 2010– Movies and persons involved in movies, e.g.

actors/actresses, directors, producers etc.• ~4M objects, including 1.5M movies and 1.8M actors• Each object is richly structured

– Each movie has title, rating, directors, actors, plot, keywords, genres, release dates, trivia, etc.

– Each person has name, birth date, bio, filmography, etc.

01 Apr 2012


341

INEX Data-Centric Track 2011• Ad-hoc

– Return a ranked list of results, i.e. objects, or equivalently documents in the IMDB collection, estimated relevant to the user’s information need

– Known-item, list and informational topics• Faceted search task

– Recommend appropriate facet-values for the user to refine the query and thus quickly identify what he/she really wants in the large set of results.

– A User Simulation System (USS) interacts with a so-called FSM to simulate a faceted search session; using simulated interaction cost as evaluation measure (alongside NDCG of facet values)

01 Apr 2012


342

INEX Data-Centric Track 2012

• 2012 uses DBPedia and Wikipedia

01 Apr 2012

Evaluation

Ad-hoc Object Retrieval


344

Semantic Search Evaluation

• Started at SemSearch 2010• Two tasks

– Entity Search• Queries where the user is looking for a single real world

object• Pound et al., Ad-hoc Object Retrieval in the Web of

Data, WWW 2010.

– List search (new in 2011)• Queries where the user is looking for a class of objects

01 Apr 2012


345

Semantic Search Evaluation

• Billion Triples Challenge 2009 dataset• Evaluated using Amazon’s Mechanical Turk

– See Halpin et al. Evaluating Ad-Hoc Object Retrieval, IWEST 2010

• Prizes sponsored by Yahoo! Labs (2011)

01 Apr 2012


346

SemSearch Entity Search Track

• Retrieval of data related to a single entity• 50 queries explicitly naming an entity(possibly w/

context)– Selected from Search Query Tiny Sample v1.0 (i.e., Yahoo!

Webscope): web search queries sampled from US query log of January, 2009.

– Queries asked by at least three different users; queries with long number sequences removed (privacy reasons)

• 2011: same setup, but queries derived from a mix of Microsoft and Yahoo! Logs

01 Apr 2012


347

SemSearch List Query Track

• List queries–Queries that describe a set of entities–The answer is a closed set –Relatively small number of possible

answers–The answer is not likely to change

01 Apr 2012


348

SemSearch List Query Track

• Hand-picked but not hand-written–Yahoo! Search logs

• Queries from the Tiny Sample v1.0 dataset

• Queries with clicks on Wikipedia–TrueKnowledge

• Recent queries

01 Apr 2012

Data set

• Billion Triples Challenge 2009 data set– Blank nodes encoded as URIs

• A data set combining crawls of multiple semantic search engines– doesn’t necessarily match the current state

of the Web– doesn’t necessarily match the coverage of

any particular search engine

http://km.aifb.uni-karlsruhe.de/ws/dataset_semsearch2010/000-CONTENTS 01 Apr 2012

http://km.aifb.uni-karlsruhe.de/ws/dataset_semsearch2010/000-CONTENTS


350

Collecting the results• Submissions via semsearch.yahoo.com

– NTNU, IIIT Hyderabad, DERI, U of Delaware, Daiict– max. 3 submissions per team per track

• Pooling of results– Top 20 results are evaluated– Despite validation, still problems

• e.g. N-Triples encoded URIs, lowercased URIs• Collecting triples for each result

– All triples where the URI is the subject– Discarded URIs that didn’t appear as subject

• Rendering result display– Values are clipped at 300 chars (last # or / for object-properties)– RDF built-ins shown first– Preference to English language values

01 Apr 2012

semsearch.yahoo.com

352

Assessment with Amazon Mechanical Turk

– Evaluation using non-expert judges• Paid $0.2 per 12 results

– Typically done in 1-2 minutes– $6-$12 an hour– Sponsored by the European SEALS project

• Each result is evaluated by 5 workers

– Workers are free to choose how many tasks they do

• Makes agreement difficult to compute

A2QXD8JQK5BVEZ

ATFFJK

Q1CSJRS

A2EB82V1YT

IUOQ

A1S5QX7B2MG2PO

A25H6GCMT4T6

R9

AI9SPC59ZL

YXO

A2FPET

U5MAKSR4

A3B69OWW

DNZFBY

A2NI0FI4RV6VE3

A1YFF3

HA9TVIO

E

A2PIZ5876TY

ZJI

AZCIY1

ZZ5QJM

Z

A3QU3KM3IYWHIF

A12JZR4MW

0PBO2

A171JY5QW

KQR8X

A1MGYBAFM

J98R3

A36Y503MT3

33BI

A118BQBBYH32X4

A2FHMBKGCIRXKV

A4VLRHP3EIL

X4

A2WOUJSF

XLK4F8

A1VZGADFR

HKK7T

A1I8LNCA4EF

UBD

A1QKU9MCGCVL35

A1J8TV

ICSRC70W

A283NW6KTH

H7DH

A2PHQZ724QJM

J5

ARQNHVDMAVYO9

A3MPG7NMNMO0XW

A25ZA4IBIEZ

GNC

A3MTA1M6SS

2ZEX

0

50

100

150

200

250

300

Number of tasks completed per worker (2010)


353

Evaluation form

01 Apr 2012


354

Evaluation form

01 Apr 2012

Catching the bad guys– Payment can be rejected for workers who try to

game the system• An explanation is commonly expected, though cheaters

rarely complain

– We opted to mix control questions into the real results

• Gold-win cases that are known to be perfect• Gold-loose cases that are known to be bad

– Metrics• Avg. and std. dev on gold-win and gold-loose results• Time to complete


356

Catching the bad guys

01 Apr 2012

Worker Known bad

Real Known Good Total N

Time to complete (sec)N Mean N Mean N Mean

badguy 20 2.556 200 2.738 20 2.684 240 29.6

goodguy 13 1 130 2.038 13 3 156 95

whoknows 1 1 21 1.571 2 3 24 83.5


357

Lessons learned• Finding complex queries is not easy

– Query logs from web search engines contain simple queries– Semantic Web search engines are not used by the general public

• Computing agreement is difficult– Each judge evaluates a different number of items

• Result rendering is critically important• The Semantic Web is not necessarily all DBpedia

– sub30-RES.3 40.6% DBpedia– sub31-run3 93.8% Dbpedia

• Follow up experiments validated the Mechanical Turk approach– Blanco et al. Repeatable and Reliable Search System Evaluation using

Crowd-Sourcing, SIGIR2011

01 Apr 2012


358

SemSearch Next Steps

• Achieve repeatability– Simplify our process and publish our tools– Automate as much as possible… except the

Turks ;)• Web site for evaluation

– Continuous submission?• Join the discussion at

[email protected]

01 Apr 2012


359

Entity Search Evaluation

• What’s next?– Continuous evaluation for AOR– INEX Data-Centric Track 2012– Knowledge Base Acceleration Track at TREC 2012

– New entity-centric search tasks?– New data collections?

01 Apr 2012


36001 Apr 2012