row-less universal schema · tech pioneer bill gates microsoft, the seattle-steve balmer, will take...

Row-less Universal Schema

Patrick Verga and Andrew McCallum

January 15, 2000

Tech pioneer Bill Gates stepped down today as chief executive officer of Microsoft, the Seattle-headquartered software giant. His long-time friend, Steve Balmer, will take over as CEO of Microsoft. Gates will now focus on the charitable foundation he runs with his wife Melinda French Gates. Bill and Melinda were married in a ceremony in Hawaii, rather than her hometown of Dallas.Gates moved his family into their 55,000-square-foot, $54 million house on the shore of Lake Washington, just outside Seattle.

Relation Types

spou

se

born

-in

frien

dpe

r:co-

work

er

per:l

ives-

in

org:t

op-m

embe

rs

org:m

embe

r

Melinda, Bill

Steve, Microsoft

Bill, MicrosoftBill, Steve

Microsoft, SeattleMelinda, Dallas

Bill, SeattleObama, Hawaii

Obama, US

January 15, 2000


CEO

chair

man

pr

esid

ent

leade

r-of

head

-of

head

-of-s

tate

m

arrie

d to

structured textual 50k columns

2

January 15, 2000


Entity Types

pers

on

locat

ion

orga

niza

tion

per:w

orke

r pe

r:scie

ntist

loc

:city

org:n

on-p

rofit

Melinda

Steve

MicrosoftBill

SeattleDallas

USAHawaii

Obama

January 15, 2000


coun

try

prov

iden

ce

chair

man

pr

esid

ent

athl

ete

polit

ician

he

ad-o

f-sta

te

structured textual

3

Steve, Microsoft


Melinda, BillMicrosoft, Seattle

Melinda, Dallas


Obama, US

####

###

####

###

####

###

####

###

####

###

####

###

####

###

####

###

####

###

####

###

####

###

####

###

####

###

####

###

vectorembeddingdistributedrepresentation

ofsemantics

learnedparameters

#######

~x

~y

January 15, 2000


Relation Typesstructured textual 50k columns

CEO

chair

man

pr

esid

ent

leade

r-of

head

-of

head

-of-s

tate

m

arrie

d to

spou

se

born

-in

frien

dpe

r:co-

work

er

per:l

ives-

in

org:t

op-m

embe

rs

org:m

embe

r

4

CEO

chair

man

pr

esid

ent

leade

r-of

head

-of

head

-of-s

tate

m

arrie

d to

spou

se

born

-in

frien

dpe

r:co-

work

er

per:l

ives-

in

org:t

op-m

embe

rs

org:m

embe

r

Steve, Microsoft



Melinda, Dallas


Obama, US


ofsemantics

learnedparameters

#######

~x

####

###

####

###

####

###

####

###

####

###

####

###

####

###

####

###

####

###

####

###

####

###

####

###

####

###

####

###~y

January 15, 2000


Gates moved his family into their 55,000-square-foot, $54 million house on the shore of Lake Washington, just outside Seattle.

5

January 15, 2000


####

###

####

###

####

###

####

###

####

###

####

###

####

###

####

###

####

###

####

###

####

###

####

###

####

###

####

###

Gates moved his family into their 55,000-square-foot, $54 million house on the shore of Lake Washington, just outside Seattle.

CEO

chair

man

pr

esid

ent

leade

r-of

head

-of

head

-of-s

tate

m

arrie

d to

spou

se

born

-in

frien

dpe

r:co-

work

er

per:l

ives-

in

org:t

op-m

embe

rs

org:m

embe

r

Steve, Microsoft



Melinda, Dallas


Obama, US


ofsemantics

learnedparameters

#######

~x

~y

6

January 15, 2000


####

###

####

###

####

###

####

###

####

###

####

###

####

###

####

###

####

###

####

###

####

###

####

###

####

###

####

###

Gates moved his family into their 55,000-

square-foot, $54 million house on the shore

of Lake Washington, just outside Seattle.

CEO

chair

man

pr

esid

ent

leade

r-of

head

-of

head

-of-s

tate

m

arrie

d to

spou

se

born

-in

frien

dpe

r:co-

work

er

per:l

ives-

in

org:t

op-m

embe

rs

org:m

embe

r

Steve, Microsoft



Melinda, Dallas


Obama, US


ofsemantics

learnedparameters

#######

~x

~y

7

Pattern Encoder

####

###

####

###

####

###

####

###

####

###

####

###

####

###

####

###

####

###

####

###

####

###

####

###

####

###

####

###

####

###

####

###

####

###

####

###

####

###

####

###

####

###

####

###

####

###

####

###

####

###

####

###

####

###

####

###




####

###

CEO

chair

man

pr

esid

ent

leade

r-of

head

-of

head

-of-s

tate

m

arrie

d to

spou

se

born

-in

frien

dpe

r:co-

work

er

per:l

ives-

in

org:t

op-m

embe

rs

org:m

embe

r

Steve, Microsoft



Melinda, Dallas


Obama, US


ofsemantics

learnedparameters

#######

~x

~y

####

###

####

###

####

###

####

###

####

###

####

###

####

###

8

January 15, 2000


Joe NewEntity worked with his wife Jane NewEntity on relation extraction.

JoeJane


9

####

###

####

###

####

###

####

###

####

###

####

###

####

###

####

###

####

###

####

###

####

###

####

###

####

###

####

###

####

###

####

###

####

###

####

###

####

###

####

###

####

###

####

###

####

###

####

###

####

###

####

###

####

###

####

###




####

###

CEO

chair

man

pr

esid

ent

leade

r-of

head

-of

head

-of-s

tate

m

arrie

d to

spou

se

born

-in

frien

dpe

r:co-

work

er

per:l

ives-

in

org:t

op-m

embe

rs

org:m

embe

r

Steve, Microsoft



Melinda, Dallas


Obama, US


ofsemantics

learnedparameters

#######

~x

~y

Pattern Encoder

####

###

####

###

####

###

####

###

####

###

####

###

####

###





CEO

chair

man

pr

esid

ent

leade

r-of

head

-of

head

-of-s

tate

m

arrie

d to

spou

se

born

-in

frien

dpe

r:co-

work

er

per:l

ives-

in

org:t

op-m

embe

rs

org:m

embe

r

Pattern Encoder

Joe NewEntity worked with his wife Jane

NewEntity on relation extraction.

####

###

####

###

####

###

####

###

####

###

####

###

####

###

####

###

####

###

####

###

####

###

####

###

####

###

####

###

####

###

####

###

####

###

####

###

####

###

####

###

####

###

####

###

####

###

####

###

####

###

####

###

####

###

####

###

####

###

####

###

Steve, Microsoft



Melinda, Dallas


Obama, US


ofsemantics

learnedparameters

#######

~x

~y

####

###

####

###

####

###

####

###

####

###

####

###

####

###

10

JoeJane

####

###





CEO

chair

man

pr

esid

ent

leade

r-of

head

-of

head

-of-s

tate

m

arrie

d to

spou

se

born

-in

frien

dpe

r:co-

work

er

per:l

ives-

in

org:t

op-m

embe

rs

org:m

embe

r

Steve, Microsoft



Melinda, Dallas


Obama, US


ofsemantics

learnedparameters

#######

~x

~y

Pattern Encoder

Joe Jane


####

###

####

###

####

###

####

###

####

###

####

###

####

###

####

###

####

###

####

###

11



####

###

####

###

####

###

####

###

spousequery:

####

###

#################################################

#####################

####

###





CEO

chair

man

pr

esid

ent

leade

r-of

head

-of

head

-of-s

tate

m

arrie

d to

spou

se

born

-in

frien

dpe

r:co-

work

er

per:l

ives-

in

org:t

op-m

embe

rs

org:m

embe

r


ofsemantics

learnedparameters

#######

~x

~y

Pattern Encoder

Steve, Microsoft



Melinda, Dallas


Obama, US

Joe Jane


####

###

####

###

####

###

####

###

####

###

####

###

####

###

####

###

####

###

####

###

12



####

###

####

###

####

###

####

###

spousequery:

####

###

Aggregation Function

#################################################

#####################

####

###





CEO

chair

man

pr

esid

ent

leade

r-of

head

-of

head

-of-s

tate

m

arrie

d to

spou

se

born

-in

frien

dpe

r:co-

work

er

per:l

ives-

in

org:t

op-m

embe

rs

org:m

embe

r

~x

~y

Pattern Encoder

Steve, Microsoft



Melinda, Dallas


Obama, US

Joe Jane


####

###

####

###

####

###

####

###

####

###

####

###

####

###

####

###

####

###

####

###

13



####

###

####

###

####

###

####

###

spousequery:

####

###


####

###

####

###

####

###

#################################################

#####################

####

###





CEO

chair

man

pr

esid

ent

leade

r-of

head

-of

head

-of-s

tate

m

arrie

d to

spou

se

born

-in

frien

dpe

r:co-

work

er

per:l

ives-

in

org:t

op-m

embe

rs

org:m

embe

r

~x

~y

Pattern Encoder

Steve, Microsoft




Obama, US

Joe Jane


####

###

####

###

####

###

####

###

####

###

####

###

####

###

####

###

####

###

14



####

###

####

###

####

###

spousequery:


Melinda, Bill

spou

se

per:c

o-wo

rker

mar

ried t

o

####

###

####

###

####

###

Query Independent1. Mean Pool

2. Max Pool

#######

####

###

####

###

####

###

####

###

####

###

####

###

#################################################

#####################

####

###




CEO

chair

man

pr

esid

ent

leade

r-of

head

-of

head

-of-s

tate

m

arrie

d to

spou

se

born

-in

frien

dpe

r:co-

work

er

per:l

ives-

in

org:t

op-m

embe

rs

org:m

embe

r

~x

~y

Pattern Encoder

Steve, Microsoft




Obama, US

Joe Jane##

####

###

####

#

####

###

####

###

####

###

####

###

####

###

####

###

####

###

15



####

###

####

###

####

###


Melinda, Bill

Query Dependent

co-founderquery: #######

spou

se

per:c

o-wo

rker

mar

ried t

o

####

###

####

###

####

###

-0.1 0.025.0

1. Max Column

####

###

####

###

####

###

####

###

#######

####

###

####

###

####

###

#################################################

#####################

####

###




CEO

chair

man

pr

esid

ent

leade

r-of

head

-of

head

-of-s

tate

m

arrie

d to

spou

se

born

-in

frien

dpe

r:co-

work

er

per:l

ives-

in

org:t

op-m

embe

rs

org:m

embe

r

~x

~y

Pattern Encoder

Steve, Microsoft




Obama, US

Joe Jane##

####

###

####

#

####

###

####

###

####

###

####

###

####

###

####

###

####

###

16



####

###

####

###

####

###


Melinda, Bill

Query Dependent

co-founderquery: #######

spou

se

per:c

o-wo

rker

mar

ried t

o

####

###

####

###

####

###

-0.1 0.025.0

1. Max Column

2. Attention

####

###

####

###

####

###

####

###

.006 .008.986

####

###

####

###

####

###

#######

####

###

####

###

####

###

#################################################

#####################

####

###




CEO

chair

man

pr

esid

ent

leade

r-of

head

-of

head

-of-s

tate

m

arrie

d to

spou

se

born

-in

frien

dpe

r:co-

work

er

per:l

ives-

in

org:t

op-m

embe

rs

org:m

embe

r

~x

~y

Pattern Encoder

Steve, Microsoft




Obama, US

Joe Jane##

####

###

####

#

####

###

####

###

####

###

####

###

####

###

####

###

####

###

17



####

###

####

###

####

###


Melinda, Bill

Training

spou

se

per:c

o-wo

rker

mar

ried t

o

####

###

####

###

####

###

#######

Entity Type Prediction

• Freebase + Clueweb

• 1.1k FB types, 5k textual types, ~140k entities

• MAP over 707 test types

• 100 negatives randomly generated for each positive

18

19

/education/academic, /music/artist,/music/group member, /people/person

/sports/pro athlete,/sports/sports award winner, /people/person /award/award winner, /tv/tv actor

/baseball /baseball player

/architecture /engineer

/computer/computer scientist

engineer, /book/author, sir/projects/project focus , /people/person

Example Max Columns

Model MAP

Entity Embeddings

3.1

Mean Pool 35.5

Max Pool 31.0

Attention 54.5

Max Relation 54.7

Model MAP

Entity Embeddings 54.8

Mean Pool 39.5

Max Pool 32.6

Attention 55.7

Max Column 55.4

20

Entity Type Prediction

seen at train

unseen at train

Relation Extraction

• Subset of FB15k-237

• Freebase + Clueweb

21

Model MRR Hits@10

Entity Embeddings

5.2 11.9

Mean Pool 18.1 35.8

Max Pool 20.8 40.3

Attention 29.8 49.7

Max Relation 28.5 48.1

Model MRR Hits@10

Entitiy Pair Embeddings 31.9 51.7

Mean Pool 25.9 45.9

Max Pool 29.6 49.9

Attention 31.9 51.7

Max Column 31.7 51.9

22

Relation Extraction

seen at train

unseen at train

Model MRR Hits@10Entity Embeddings

5.2 11.9

5.2 11.9

Attention 29.8 49.7

27.8 51.0

Max Relation 28.5 48.1

29.6 54.2

Model MRR Hits@10

Entity Embeddings 31.9 51.7

Entity Embeddings-LSTM 33.4 54.4

Attention 31.9 51.7

Attention-LSTM 30.0 53.4

Max Column 31.7 51.9

Max Column-LSTM 30.8 54.8

23

Relation ExtractionRowless + Columnless

seen at train

unseen at train

Future Work

24

• Improvements to Attention model

• Discern between irrelevant and counter evidence

Melinda, Bill

spou

sepe

r:co-

work

erm

arrie

d to

####

###

####

###

####

###

think

s is a

wful

####

###

hatesquery:

• Row-less Universal Schema forgoes explicit row representations for aggregation over observed columns

• Performs as well as old model using fewer parameters

• Naturally extends to unseen entities

• Aggregation based on query-specific attention outperforms query-independent aggregations

25

Summary

Thank you!

row-less universal schema · tech pioneer bill gates microsoft, the seattle-steve balmer, will take...

Documents