entity extraction for query interpretation patrick pantel ǂ

Entity Extractionfor Query Interpretation

Patrick Pantelǂ

Query Representation and Understanding SIGIR

July 23, 2010

Collaborators:Alpa Jain, Ana-Maria Popescu, Arkady Borkovsky, Eric Crestan, Hadar Shemtov, Marco Pennacchiotti, Nicolas Torzec, Vishnu Vyas

ǂ Now at Microsoft Research

- 2 -Entity Extraction July 2010

The Ensemble Effect

1 23 41 59 77 95 114 132 150 168 186 205 223 241 259 277 295 314 332 350 368 386 405 423 441 459 477 4950.4

0.5

0.6

0.7

0.8

0.9

1

pattern-based distributional combined ml-combined ensemble

Rank

Prec

isio

n

(chart refers to class Athlete)

+19% average precision over state of

the art


Ensemble Semantics

S1

SK

S2

KB

(Pennacchiotti and Pantel; EMNLP 2009)


Ensemble Semantics

S1

SK

S2

KE

nK

E2

KE

1

FG1 FG2 FGm

KB

FEATURE GENERATORSK

NO

WLE

DG

E E

XTR

AC

TOR

S



Ensemble Semantics

S1

SK

S2

KE

nK

E2

KE

1

FG1 FG2 FGm

KB

FEATURE GENERATORSK

NO

WLE

DG

E E

XTR

AC

TOR

S

AG

GR

EG

ATO

R



Ensemble Semantics

S1

SK

S2

KE

nK

E2

KE

1

FG1 FG2 FGm

KB

FEATURE GENERATORS

RANKER

KN

OW

LED

GE

EX

TRA

CTO

RS

AG

GR

EG

ATO

R

MODELER

DECODER



S1

SK

S2

KE

nK

E2

KE

1

FG1 FG2 FGm

KB

FEATURE GENERATORS

RANKER

KN

OW

LED

GE

EX

TRA

CTO

RS

AG

GR

EG

ATO

R

MODELER

DECODER

Knowledge Extractors


1) DRIVING IDEA

3) IN ACTION

How are Jubil Sarch and Kirov Chob related?

• Jubil Sarch starring Kirov Chob• Jubil Sarch featuring the beautiful Kirov Chob• Steven Soderbergh-directed Jubil Sarch which

earned star Kirov Chob an Oscar

2) ALGORITHM• State-of-the-art pattern-based

algorithm for relation extraction [Pasca et al., 2006]

• Instantiate typical relations of the class at hand

• (e.g. act-in(Actor,Movie))

Seed tuples

Learn patterns

Append reliable tuples

Find new tuples

Web

act-in(Tom Hanks, The Terminal)act-in(Nicole Kidman, Eyes Wide Shut)

Johnny DeppDenis LawsonBrad PittMorgan Freeman…

Pattern-based Extractor


1) DRIVING IDEA

3) IN ACTION

What is tezgüno?– A bottle of tezgüno is on the table.– Everyone likes tezgüno.– Tezgüno makes you drunk.– We make tezgüno out of grapes.

2) ALGORITHM• State-of-the-art distributional entity

extractor [Pantel et al., EMNLP 2009]• Given a small set of seeds for a

given class, find distributionally similar candidate instances

Distributional Extractor

Nicole KidmanAl PacinoTom Hanks

Web

anna gunnolivier gueriteetomas von bromssenharry jonesjudy mathesonrobert keithmariah o'brienstarring dennis quaidnoah beery jrfederico castelluccioadienne shellybetty morangeorge takaijo anne worleyruth hampton

rex hagonalex fonggene burkemiguel hermoso arnaoeiko andocharles mccaughanyukijiro hotarualec christiedame wendy hillerjohn waynearthur lakesir herbert beerbohm treetonya wrightlori saunders


S1

SK

S2

KE

nK

E2

KE

1

FG1 FG2 FGm

KB

FEATURE GENERATORS

RANKER

KN

OW

LED

GE

EX

TRA

CTO

RS

AG

GR

EG

ATO

R

MODELER

DECODER

Feature Generators


Feature sets• 4 feature families• 5 feature types• 402 features

Web600M pages web crawl Query log1 year of queries (top 1M)Web TableFrom 600M pages web crawlWikipedia2M articles 2008 dump


Query Log FeaturesJodie Foster Humphrey BogartRobert DuvallJodie Foster Humphrey BogartRobert Duvall

starringfilmographymoviesOscar

Johnny DepppicturesstarringDonnie BrascofilmographyTim BurtonQuotesmovies

Barack Obamabiographybirth certificatespeechquotespicturesinaugurationwiki

starringCydney BernardfilmographyThe AccusedmoviesfilmOscar

westernsfilmographyTangofilmsmoviesstarringApocaplypse Now

filmographyquotesmoviesOscarLauren Bacall starringImdb

top-PMI attributes


Web-table Features

Harrison FordBurt LanacasterIan HartNicholas Jones

SeedsDenis LawsonCandidate


Ranker

S1

SK

S2

KE

nK

E2

KE

1

FG1 FG2 FGm

KB

FEATURE GENERATORS

RANKER

KN

OW

LED

GE

EX

TRA

CTO

RS

AG

GR

EG

ATO

R

MODELER

DECODER


EXPERIMENTS

S1

SK

S2

KE

nK

E2

KE

1

FG1 FG2 FGm

KB

FEATURE GENERATORS

RANKER

KN

OW

LED

GE

EX

TRA

CTO

RS

AG

GR

EG

ATO

R

MODELER

DECODER


Experimental Setup• Task : Entity extraction• Tested classes : Actors, Athletes, Musicians

• Gold Standard : - 500 manually annotated instances per class

- 10 annotators, Kappa=0.88

• Evaluation : - Metric: average precision- 10-fold cross validation

• Comparisons : (B1) pattern-based extractor(B2) distributional extractor(E1) combined extractor(E2) ML-combined extractor [Mirkin et al.,2006]


Experimental Results

FEATURESs = Source featuresw = Webcrawl (600M docs)q = Query Logs (1 year)t = Web tables (600M docs)k = Wikipedia


Standalone Extractors gain : Actors

0

0.2

0.4

0.6

0.8

1

Prec

isio

n

Rank

Distributional : Precision vs. Rank analysis

distributionalensemble

0

0.2

0.4

0.6

0.8

1

Prec

isio

n

Rank

Pattern-based: Precision vs. Rank analysis

pattern-basedensemble


Standalone Extractors gain : Athletes

0

0.2

0.4

0.6

0.8

1

Prec

isio

n

Rank



0

0.2

0.4

0.6

0.8

1

Prec

isio

n

Rank




Standalone Extractors gain : Musicians

0

0.2

0.4

0.6

0.8

1

Prec

isio

n

Rank



0

0.2

0.4

0.6

0.8

1

Prec

isio

n

Rank




Ensemble Effect: Athletes

1,15328,835

56,5180.00

0.10

0.20

0.30

0.40

0.50

0.60

0.70

0.80

0.90

1.00Precision vs. Rank Analysis

upper_boundbaselinet-gbdt-t300

Rank

Prec

isio

n


free

googleyou tubegirlspeopledisneybabytexasairspearsnew york … … … …

100 Most Frequent

Entities in QL

<10% Precision

Takeaway: Harvesting knowledge requires different metrics than using the knowledge

• Editorial cleaning of entities covering 80% of queries• Automatic cleaning of torso/tail• Impact on query and document Interpretation?


Conclusions• Ensemble Semantics:

– Draw from many knowledge sources– Apply different extraction biases– Leverage many signals of knowledge

• Significant gains in both coverage and precision on entity extraction

• BUT, is the knowledge useful??– 90% precision huge list of actors results in many many query

interpretation errors!• Frequent terms tend to be ambiguous• Some highly accurate databases contain very bad errors (e.g., Texas and

Baby are both actors in Y! Movies)– Further processing is necessary to make use of the knowledge…

entity extraction for query interpretation patrick pantel ǂ

Documents