patrick pantel - akbcwekex2012.files.wordpress.com · 1x listen to music (on a creativework or...

Patrick Pantel Joint work with: Tom Lin (UW), Michael Gamon (MSR), Anitha Kannan (MSR), Ariel Fuxman (MSR)

June 2012

Structured Data

Entity Detection

Play trailer

Structured Recommendations

Price prediction

Task completion

Task completion

Aggregate ratings

Active Objects 4

Direct Answer Structured Data

Current Experience

Active Objects 21

Better experience

Active Objects 23

Rec

ogn

ize

en

tity

in q

uer

y A

ctio

ns

easi

ly

acce

ssib

le

Better experience

Active Objects 24

Rec

ogn

ize

en

tity

in q

uer

y A

ctio

ns

easi

ly

acce

ssib

le

Better experience

Active Objects 25

Rec

ogn

ize

en

tity

in q

uer

y A

ctio

ns

easi

ly

acce

ssib

le

Politicians

Active Objects 29

Rec

ogn

ize

en

tity

in q

uer

y A

ctio

ns

easi

ly

acce

ssib

le

Films

Active Objects 30

Rec

ogn

ize

en

tity

in q

uer

y A

ctio

ns

easi

ly

acce

ssib

le

32

tt

QQ

tt

n22

qq

TT

aae

wwKK

css s

f f

KK

y y

TT

KK

Active Objects

Actions vs Intents

Active Objects 36

User Intents and Goals

Query Intent

plan vacation

Query hilton orlando reviews sea world location

Finer-grained Intents

Actions on Entities

read reviews(hotel)

get address(landmark) add to Netflix queue(film)

buy(camera)

Navigational Informational Transactional [Broder, 2002]

Advice Locate … Download Obtain Interact …

[Rose and Levinson, 2004]

get in shape

how to lose weight

Do web queries contain entities?

Active Objects 38

43% entity (e.g., “GoldenEye”, “Horne Auto”)

14% entity category (e.g., “golf cart battery”, “global sim card”)

15% no entity (e.g., “xxx”, “good reading quotes”)

28% website (e.g., “yahoo mail”, “girlybox.com”)

category 4%

category + refiner

10%

entity 29%

entity + refiner 14%

website 28%

other 15%

Entity Distribution in Web Search Queries

* From a query traffic-weighted sample

creativework 40%

organization 37%

product 9%

person 8%

event 3% other

3%

Schema.org types for entity-bearing queries

Ontology of Actions

Active Objects 43

Navigational 10x Login Action (on a Website entity) 4x Search Action (on a Website entity) Informational (need satisfied by reading content, or could be satisfied by written transcript of content) 1x Find Location(s) (on an Organization entity) 1x Find Lyrics (on a CreativeWork / MusicalTrack entity) 2x Find Recipe For (on a food) 1x Find Where to Buy (on a Product entity) 2x Get Contact Information (on an Organization entity) 1x Get Directions To (on an Organization / Location entity) 2x Get Domain Information (on a Website entity) 1x Get Event Details (on an Event entity) 2x Get Event Results (on an Event entity) 4x Product Detail (on a Product entity) 29x Learn (on any entity) 6x Learn / Educational (on a Person / Product / Organization entity) 1x Learn / Trivia (on any entity) 1x Operating Hours (on an Organization entity) 3x Read Articles (on a News / Magazine entity) 1x Read Guide (on a Product entity) 1x Read Help (on a Product entity) 8x Read News About (on any entity) 3x Read Reviews (Shopping on a CreativeWork / Product / Service entity) 1x Read Spoilers (on a CreativeWork) 8x Research (focused information gathering, on any entity) 8x Search Database of (e.g., obituaries, on an Organization / Website) See Menu (on a Restaurant) 3x See Pictures (on a Person / Product / Organization entity) Side Effects / Safety (on a Product entity) Stock Price (on an Organization entity)

Transactional (navigating to a web-mediated action)

1x Apply for Job (on a LocalBusiness / Organization entity)

Buy (Shopping on a Product entity)

Buy Tickets (on an Event / Product / Person entity)

3x Content Creation (on a Website entity)

Discuss Online (on any entity)

5x Download (on a CreativeWork or Software entity)

1x Listen to Music (on a CreativeWork or Website entity)

Manage Account (on a Local Business / Website / Org entity)

Pay Bill (on a Website / Organization entity)

14x Play Game (on a Game entity)

Rent (on a CreativeWork / Product entity)

2x Reservation (on a Hotel entity)

Schedule Appointment (on a LocalBusiness entity)

Sell (Shopping on a Product entity)

1x Use Service On (e.g., translate, on a Website)

6x Watch Video About (on any entity)

1x Web Chat

Other

13x Shopping (category of actions including reviews and buying)

19x Various/Unknown

Actions are tied to entity types 47 actions in current list

Note: No existing Actions equivalent for Schema.org

How many Actions should there be?

Active Objects 44

0

1

2

3

4

5

6

7

8

9

10

0 50 100 150 200 250

Ne

w A

ctio

ns

pe

r 1

0 A

nn

ota

tio

ns

(ave

rage

)

Annotations

Discovery Rate of New Actions

Rapidly decreasing

discovery rate

45

tt

QQ

tt

n22

qq

TT

aae

wwKK

css s

f f

KK

y y

TT

KK

Active Objects

Learning Actions from Web Usage Logs

Active Objects 49

21 types

58,123 hosts

2,164,579 (query, host) pairs

over 3 months 129,088 contexts

235,385 entities • Three months of us-en

web logs

• Annotate with Freebase entities

• Keep queries with an entity in set of 21 types

• Filter out navigational queries

• Filter out clicked hosts that weren’t clicked at least 100 times

Orlando hotel reviews

Does Hope Solo have a boyfriend?

Free Winzip download

watch family guy online

get reviews

read biography

download software

watch shows online

←

Model 1

Active Objects 52

Action Context

Goal: Define a theory for how actionable queries are generated.

The story for p(actionable query), or more formally The story for p(f,q,a,n | a,b) For each action a

fa ~ Dirichlet(b) For each query q

q ~ Dirichlet(a) For each context position in q (pre or post)

action a ~ Multinomial(q) ngram n ~ Multinomial(fa)

Model 1.01Model 1.01

bb

f f

KK

a a

q q

QQ

a1a1

n1

a2a2

n2

Star Wars “ebert review"

read reviews action

“______ ______”

read reviews action

(action→contexts die)

Active Objects 53

Model 2

Action Context

The story for p(f,q,a,n | a,b) For each action a

fa ~ Dirichlet(b) For each query q

q ~ Dirichlet(a) For each context position in q (pre or post)

action a ~ Multinomial(q) ngram n ~ Multinomial(fa)

action a ~ Multinomial(q) ngram n1 ~ Multinomial(fa) ngram n2 ~ Multinomial(fa)

a a

q q

n1 n2

QQ

bb

f f

KK


aa

Star Wars “ebert review"

read reviews action

Active Objects 54

Model 3

Clicked hosts matter…

Action Context

buy action read reviews action

amazon.com

ebay.com

walmart.com

rottentomatoes.com

metacritic.com

efilmcritic.com

“______ ______”

read reviews action

Active Objects - 55 -


Model 3

The story for P(f,q,w,a,n,c | a,b,) For each action a

fa ~ Dirichlet(b)

For each query q q ~ Dirichlet(a) action a ~ Multinomial(q) ngram n1 ~ Multinomial(fa) ngram n2 ~ Multinomial(fa)

“ebert review”

Action Host Context

bb

f f

KK

ww

KK

a a

q q

QQ

n22

c

aa

click c ~ Multinomial(wa)

wa ~ Dirichlet(i)

www.rottentomatoes.com

w, ,c ,

(to contexts) (to clicks)

Star Wars

Active Objects 57

Model 3

The type matters…

Action Host Context

Active Objects 58


Model 4

Action Host Context Type


bb

f f

KKn

22

aa

ww

KKc

g g

t t

TT

a a

q q

QQ

tt

The story for P(f,q,t,w,t,a,n,c | a,b,g,) For each action a

fa ~ Dirichlet(b) wa ~ Dirichlet(i)

For each query q

q ~ Dirichlet(a) action a ~ Multinomial(q) ngram n1 ~ Multinomial(fa) ngram n2 ~ Multinomial(fa) click c ~ Multinomial(wa)

a a

q q

QQ

For each type t tt ~ Dirichlet(g)

type t ~ Multinomial(q) action a ~ Multinomial(q) action a ~ Multinomial(tt)

“ebert review” “______ ______”

film type

read reviews action

www.rottentomatoes.com

t, t, g,

(to contexts) (to clicks)

(to actions)

Star Wars

Active Objects 60

Model 3

We also have entity data…

Action Host Context Type


a a

q q

QQ

bb

f f

KK

Model 1.05bModel 1.05b

tt

n22

g g

t t

TT

aa

ww

KKc

Model 5

Action Host Context Type Entity

h h

y y

TTe

The story for P(f,q,t,y,w,t,a,e,n,c | a,b,g,h,) For each action a

fa ~ Dirichlet(b) wa ~ Dirichlet(i)

For each type t tt ~ Dirichlet(g)

For each query q q ~ Dirichlet(a) type t ~ Multinomial(q) action a ~ Multinomial(tt) ngram n1 ~ Multinomial(fa) ngram n2 ~ Multinomial(fa) click c ~ Multinomial(wa)

yt ~ Dirichlet(h)

entity e ~ Multinomial(yt)

y, e, h,


Model 6

Action Host Context Type Entity Empty

The story for P(f,q,t,y,w,s,t,a,e,s,n,c | a,b,g,h,,e) For each action/type pair, {a, t}

fa ~ Dirichlet(b) wa ~ Dirichlet() sa,t ~ Beta(e)

For each type t tt ~ Dirichlet(g) yt ~ Dirichlet(h)

For each query q q ~ Dirichlet(a) type t ~ Multinomial(q) action a ~ Multinomial(tt) entity e ~ Multinomial(yt) switch s1 ~ Bernou (sa) switch s2 ~ Bernoulli(sa,t) if (s1) ngram n1 ~ Multinomial(fa) if (s2) ngram n2 ~ Multinomial(fa) click c ~ Multinomial(wa)

The story for P(f,q,t,y,w,s,t,a,e,s,n,c | a,b,g,h,,e) For each action/type pair, {a, t}

fa ~ Dirichlet(b) wa ~ Dirichlet() sa ~ Beta(e)

switch s1 ~ Bernoulli(sa) switch s2 ~ Bernoulli(sa) if (s1) if (s2)

s,

a a

q q

QQ


tt

n22

g g

t t

TT

aa

e

ww

KKc

bb

f f KK

h h

y y

TT

e e

s s

KK

s

Apply Generative Model

Active Objects 66

66

Action (hidden)

Clicked Host (observed)

Query Context (observed)

Type (observed)

P(Action|Type)

P(Host|Action) P(Context|Action)

A new query comes in: (e.g., “New York City hotels”) Entity Recognition(query) → entity (“New York City”) entity → types (“city”, “employer”, “travel destination”) (query, entity) → context (“Ø”, “hotels”) Historical Data(query) → distribution over hosts • EM Posterior Probabilities to give us likelihood of

each action cluster. • action cluster → action phrase (“book hotel in”) • (action cluster, action phrase, historical data) →

best hosts (“travel.bing.com”)

query: jetbeam rrt-0

Active Objects 69

Action words from Web Trigrams

Active Objects 70

• Patterns (similar to Hearst patterns) on a Web Trigram corpus to get actions.

Web Trigrams

Pattern Match “want to (x)” “have to (x)” “you can (x)” “I can (x)”

Filter Adverbs (e.g., “honestly”, “quickly”)

Filter noise (the 25% with

lowest 𝑓𝑟𝑒𝑞𝑢𝑒𝑛𝑐𝑦

𝑢𝑛𝑖𝑔𝑟𝑎𝑚 𝑐𝑜𝑢𝑛𝑡

e.g., “a”, “boy”)

13,417 action words make download find torrent say eBay Pay login Buy podcast help … …

Method scales to longer actions, e.g., 4-grams for 2-word actions (“read review”)

Finds modern/web actions that older annotated corpora might miss.

↑ ↑

Web Action words from Trigrams

Active Objects 71

• Not all actions can be recommended over the Web (e.g., “shock” or “kill”). How do we find the ones that can?

Web Action ngram pattern “(x) at (y)” where y has the form of a web site URL “buy at Amazon.com” “download at cnet.com”

13,417 action words make download find torrent say eBay Pay login Buy podcast help … …

1,279 web actions buy review shop unsubscribe book download …

Human Annotation of Action Phrases

Active Objects 73

19, Read biography of 19, See pictures of 19, Read blog of 19, Contact 19, Read interview with 19, Watch video of

12, Download 12, Find reviews of 12, Update 12, Get help for

24, Apply for jobs at 24, View career options at 24, View map of 24, Read news about 24, Find locations of 24, Find address of 24, Get stock quote of

From each model, we first automatically generate:

Use as training data

Evaluation Setup

Active Objects 79

a a

q q

n1 n2

QQ

bb

f f

KK


aa

Model 2 Model 3 Model 4 Model 5 Model 6

+clicked host +entity types +entity +empty switch action → context

a a

q q

QQ

bb

f f

KK


n22

aa

ww

KKc

a a

q q

QQ

bb

f f

KK


tt

n22

g g

t t

TT

aa

ww

KKc

a a

q q

QQ

bb

f f

KK


tt

n22

g g

t t

TT

aah h

y y

TTe

ww

KKc

a a

q q

QQ


tt

n22

g g

t t

TT

aa

e

ww

KKcs

e e

s s

KK

bb

f f KK

h h

y y

TT

21 types

58,123 hosts

2,164,579 (query, host) pairs

over 3 months 129,088 contexts

235,385 entities

Data

+ Filter out Navigational

Queries

50 clusters, 2-step learning over 100 total EM iterations, 2 folds per model

Evaluation Framework

Active Objects 82

Co

mb

ined

set

of

acti

on

s o

ver

all m

od

els

x 7

UHRS Fair κ

agreement

Performance on HEAD vs. TAIL vs. Type-Balanced queries

Active Objects 87

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

HEAD TAIL Type-Balanced

nD

CG

nDCG vs. Query Sets (with 95% confidence bounds)

Model 2

Model 3

Model 4

Model 5

Model 6

Tail is dominated by “People” type

+ click

+ type

+ entity + switch

Action Discovery, Diversity

Active Objects 89

0%

100%

200%

300%

400%

500%

600%

700%

800%

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

Tota

l P(a

ctio

n |

typ

e)

Cluster Rank

Model 4

0%

100%

200%

300%

400%

500%

600%

700%

800%

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

Tota

l P(a

ctio

n |

typ

e)

Cluster Rank

Model 4

Model 5

0%

100%

200%

300%

400%

500%

600%

700%

800%

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

Tota

l P(a

ctio

n |

typ

e)

Cluster Rank

Model 4

Model 5

Model 6

Examples

Active Objects 92

Query: Webster University

Entity: Webster University

Context: (“”, “”)

Types: /business/employer, education/university, /location/location

Model 2 (context) Model 3 (+click) Model 4 (+type) Model 5 (+entity) Model 6 (+switch)

1. Torrent 2. Read biography 3. Find adult pictures of 4. Watch videos 5. See picture of 6. Get quotes from 7. Apply for jobs at

1. Torrent 2. Read biography 3. Read news about 4. See pictures of 5. Apply for jobs at 6. Get quotes from 7. See videos with

1. Read reviews of 2. See map of 3. Follow sports teams of 4. Get weather in 5. Apply for jobs at 6. Find address of 7. See rankings of

1. Read reviews of 2. See map of 3. Follow sports teams of 4. Get weather in 5. Apply for jobs at 6. Find address of 7. See tuition of

1. Find address 2. See pictures of 3. Find map of 4. Read news about 5. Apply for jobs at 6. See cost of 7. See ranking of

Models 4, 5, 6 automatically generate reasonable actions for this query


1. Torrent 2. Read biography 3. Find adult pictures of 4. Watch videos 5. See picture of 6. Get quotes from 7. Apply for jobs at

1. Torrent 2. Read biography 3. Read news about 4. See pictures of 5. Apply for jobs at 6. Get quotes from 7. See videos with

1. Read reviews of 2. See map of 3. Follow sports teams of 4. Get weather in 5. Apply for jobs at 6. Find address of 7. See rankings of

1. Read reviews of 2. See map of 3. Follow sports teams of 4. Get weather in 5. Apply for jobs at 6. Find address of 7. See tuition of

1. Find address 2. See pictures of 3. Find map of 4. Read news about 5. Apply for jobs at 6. See cost of 7. See ranking of

+ User Model?

Examples

Active Objects 93

Query: download Skype

Entity: Skype

Context: (“download”, “”)

Types: /computer/software,/business/employer,/business/business_operation


1. Download 2. Login to 3. Read celebrity gossip about 4. Watch videos with 5. Hack 6. Find games with 7. Watch movies with

1. Download 2. Play games 3. Hack 4. Chat at 5. Create account at 6. Torrent 7. Read biography of

1. Download 2. Login to 3. Hack

1. Find on social networks 2. Download 3. Hack

1. Download 2. Find reviews of 3. Update 4. Get help for

Again, we can now automatically generate reasonable actions for queries!

144

tt

QQ

tt

n22

qq

TT

aae

wwKK

css s

f f

KK

y y

TT

KK

Active Objects

Active Objects (Model IM)Active Objects (Model IM)

QQn

22

qqTT

aae

wwKK

css s

f f KK

y y TT

KK

Inferring Type Distributions

• Entity types are modeled as latent variables, jointly with the intended actions.

• Extensions: – Given admissible types from a

KB such as Freebase, learn their priors and contextual disambiguation

– Given a new term, induce the types of the term

– Automatically induce type list and admissible types for arbitrary entities

Active Objects 145

tt

tt

Prior Art (Guo, Xu, Cheng, and Li, SIGIR’09)

Active Objects 146

Generative process for entity bearing queries. For each query q

entity e ~ Multinomial(y) type t ~ Multinomial(te) ngram n1 ~ Multinomial(ft) ngram n2 ~ Multinomial(ft)

QQ

Guo’09Guo’09y y

e

ttttEE

n22

f f TT

Active Objects 147

y y

QQ

tt

n22

e

f f TT

ttEE

Guo’09Guo’09

+ Empty Switch

y y

QQ

tt

n22

e

f f TTtt

EE

Model M0Model M0

ss s TT

Active Objects 148

y y

QQ

tt

n22

e

f f TT

ttEE

Guo’09Guo’09

+ Empty Switch + Click

wwTT

css s TT

y y

QQ

tt

n22

e

f f TTtt

EE

Model M1Model M1

Active Objects 149

y y

QQ

tt

n22

e

f f TT

ttEE

Guo’09Guo’09

+ Empty Switch + Click + Action

Model IMModel IMtt

QQ

tt

n22

e f f KK

y y TT

qqTT

aa

wwKK

css s KK

Experimental Setting

• Training – Queries from 3 months of US Bing search usage logs

– Entities from 73 Freebase types, accounting for 50% query traffic in US market

– Model parameters trained using 2-step learning over 100 EM iterations, 2 folds per model

• Testing – Query-weighted random sample of 500 HEAD and 500 TAIL entity-

bearing queries

– 7 paid independent annotators identified all applicable Freebase types to the entities in the queries • Two annotators per query

• Fleiss’ was 0.445, moderate agreement

Active Objects 150

Performance Analysis

Active Objects 151

• † indicates statistical significance over BFB, and ‡ over both BFB and Guo’09. • Bold indicates statistical significance over all non-bold models in the

column. • M1 (empty context + click signal) significantly outperforms baseline and

Guo’09, on HEAD. • IM significantly better over all models and across all metrics

– Biggest gains in first position of its ranking (Prec@1 metric).

Switch Parameter Analysis

Active Objects 153

• Switch improves performance across all models

• More expressive models benefit more from switch

0

0.02

0.04

0.06

0.08

0.1

0.12

0.14

0.16

M0 M1 IM

Re

lati

ve g

ain

of

swit

ch v

s. n

o s

wit

ch

Effect of Empty Switch Parameter (s) on HEAD

No switch

nDCG

MAP

MAPW

Prec@1

Discussion

• Why performance on TAIL lower than expected? – TAIL is skewed towards the PEOPLE types

– Latent actions are over-expressive and they do not help in differentiating PEOPLE types • Inspection of latent Action parameter in IM shows that most PEOPLE

types have all their mass distributed to three generic and common intents (see pictures of, find biographical information about, and see video of)

• Success case in the TAIL – “ymca” -> {song, place, educational_institution}

• Marginalizing out the context words gives the following intent priors: 0.63, 0.29, 0.08

– q1 = “jamestown ymca ny” IM correctly classified “ymca” as a place

– q2 = “ymca palomar” IM correctly classified “ymca” as a educational_institution

Active Objects 155

Wrap-Up

Active Objects 157

Active Objects 159

• A hodgepodge of related strings

• Only actionable through search

Active Objects 160

2009

2010

Active Objects 161

Big Wins

• Typed relations

2009

2010

Active Objects 162

Big Wins

• Typed relations

• User Interface accesses structured data

2009

2010

Active Objects 163

Big Wins

• Typed relations


• Click through experience can now leverage strongly-typed identifier

2009

2010

Active Objects 164

Big Wins

• Typed relations


• Click through experience can now leverage strongly-typed identifier

• Brokered Actions (one click conversions)

2009

2010

Add to Amazon cart

Connect on Twitter

Play track

Active Objects 165

2009

2010

To Do

• Fixed actions

• Model tasks

• Annotate URLs/Apps with actions

• Active Objects as an entity-centric UI

patrick pantel - akbcwekex2012.files.wordpress.com · 1x listen to music (on a creativework or...

Documents