traversing knowledge graphs in vector space
TRANSCRIPT
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
1/149
Traversing Knowledge Graphs inVector Space
Kelvin Guu, John Miller, Percy Liang
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
2/149
Knowledge graphs
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
3/149
Knowledge graphs
What languages are spoken by people in Portugal?
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
4/149
Knowledge graphs
portugal/location/language
What languages are spoken by people in Portugal?
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
5/149
Knowledge graphs
portugal/location/language
What languages are spoken by people in Portugal?
portugal
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
6/149
Knowledge graphs
portugal/location/language
What languages are spoken by people in Portugal?
portugal
fernando_pessoa
jorge_sampaio
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
7/149
Knowledge graphs
french
portuguese
english
portugal/location/language
What languages are spoken by people in Portugal?
portugal
fernando_pessoa
jorge_sampaio
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
8/149
Knowledge graphs
french
portuguese
english
fernando_pessoa
jorge_sampaio
portugallocation
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
9/149
Knowledge graphs
french
portuguese
english
fernando_pessoa
jorge_sampaio
portugal
(fernando_pessoa, location, portugal)
location
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
10/149
Knowledge graphs
french
portuguese
english
fernando_pessoa
jorge_sampaio
portugal
(fernando_pessoa, location, portugal)
location
fact = edge = triple
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
11/149
Knowledge graphs
french
portuguese
english
fernando_pessoa
jorge_sampaio
portugallocation
93.8%of persons from Freebase have no placeof birth, and 78.5%have no nationality [Min et al, 2013]
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
12/149
Knowledge graphs
french
portuguese
english
fernando_pessoa
jorge_sampaio
portugallocation
93.8%of persons from Freebase have no placeof birth, and 78.5%have no nationality [Min et al, 2013]
strength:compositionality
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
13/149
Knowledge graphs
french
portuguese
english
fernando_pessoa
jorge_sampaio
portugallocation
93.8%of persons from Freebase have no placeof birth, and 78.5%have no nationality [Min et al, 2013]
weakness:
incompleteness
strength:compositionality
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
14/149
Vector space models
fernando
pessoa
portugal
barackobama
united
states
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
15/149
Vector space models
fernandopessoa
portugal
barackobama
united
states
location
location
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
16/149
Vector space models
fernandopessoa
portugal
barackobama
united
states
location
location
large # facts > low-dimensional space
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
17/149
Vector space models
fernandopessoa
portugal
barackobama
united
states
location
locationcompression > generalization
large # facts > low-dimensional space
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
18/149
Vector space models
Tensor factorization [Nickel et al., 2011]
Neural Tensor Network [Socher et al., 2013]
TransE [Bordes et al., 2013]
Universal Schema [Riedel et al., 2013]
General framework + comparison [Yang et al.,2015]
Compositional embedding of paths [Neelakantan etal., 2015]
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
19/149
Vector space models
Tensor factorization [Nickel et al., 2011]
Neural Tensor Network [Socher et al., 2013]
TransE [Bordes et al., 2013]
Universal Schema [Riedel et al., 2013]
General framework + comparison [Yang et al.,2015]
Compositional embedding of paths [Neelakantan etal., 2015]
strength:handle incompleteness
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
20/149
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
21/149
Graph databases Vector space models
Compositionalqueries
Handleincompleteness
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
22/149
Graph databases Vector space models
Compositionalqueries
Handleincompleteness
This talk
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
23/149
Graph databases Vector space models
Compositionalqueries
Handleincompleteness
This talk
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
24/149
Graph databases Vector space models
Compositionalqueries
Handleincompleteness
This talk
PART I
PART II
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
25/149
Outline
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
26/149
Outline
PART I
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
27/149
Outline
PART I
Interpret many existing vector space models as each
implementing a traversal operator.
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
28/149
Outline
PART I
Interpret many existing vector space models as each
implementing a traversal operator.
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
29/149
Outline
PART I
Interpret many existing vector space models as each
implementing a traversal operator.
Proposehow all these models can be generalized to answer
path queries.
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
30/149
Outline
PART I
Interpret many existing vector space models as each
implementing a traversal operator.
Proposehow all these models can be generalized to answer
path queries.
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
31/149
Outline
PART I
Interpret many existing vector space models as each
implementing a traversal operator.
Proposehow all these models can be generalized to answer
path queries.
PART II
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
32/149
Outline
PART I
Interpret many existing vector space models as each
implementing a traversal operator.
Proposehow all these models can be generalized to answer
path queries.
PART II
Showthat when we train these models to answer path queries,they actually become better at predicting missing facts.
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
33/149
Outline
PART I
Interpret many existing vector space models as each
implementing a traversal operator.
Proposehow all these models can be generalized to answer
path queries.
PART II
Showthat when we train these models to answer path queries,they actually become better at predicting missing facts.
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
34/149
Outline
PART I
Interpret many existing vector space models as each
implementing a traversal operator.
Proposehow all these models can be generalized to answer
path queries.
PART II
Showthat when we train these models to answer path queries,they actually become better at predicting missing facts.
Explainwhy this new path-based training procedurehelps.
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
35/149
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
36/149
Path queries
portugal/location/language
What languages are spoken by people in Portugal?
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
37/149
Path queries
{portugal}portugal
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
38/149
Path queries
{portugal}portugal / location
{fernando_pessoa,
jorge_sampaio,
vasco_da_gama}
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
39/149
Path queries
{portugal}portugal / location
/ language{fernando_pessoa,
jorge_sampaio,
vasco_da_gama}
{portuguese,
spanish,
english}
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
40/149
Path queries in vector space
{portugal}portugal / location
/ language{fernando_pessoa,
jorge_sampaio,
vasco_da_gama}
{portuguese,
spanish,
english}
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
41/149
Path queries in vector space
{portugal}portugal / location
/ language{fernando_pessoa,
jorge_sampaio,
vasco_da_gama}
{portuguese,
spanish,
english}
sparse
vectors
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
42/149
Path queries in vector space
{portugal}portugal / location
/ language{fernando_pessoa,
jorge_sampaio,
vasco_da_gama}
{portuguese,
spanish,
english}
sparse
vectors
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
43/149
Path queries in vector space
{portugal}portugal / location
/ language{fernando_pessoa,
jorge_sampaio,
vasco_da_gama}
{portuguese,
spanish,
english}
sparse
vectorsx ->
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
44/149
Path queries in vector space
{portugal}portugal / location
/ language{fernando_pessoa,
jorge_sampaio,
vasco_da_gama}
{portuguese,
spanish,
english}
sparse
vectorsx -> x ->
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
45/149
Path queries in vector space
{portugal}portugal / location
/ language{fernando_pessoa,
jorge_sampaio,
vasco_da_gama}
{portuguese,
spanish,
english}
sparse
vectorsx -> x ->
dense,
low-dim
vectors
x -> x ->
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
46/149
Path queries in vector space
{portugal}portugal / location
/ language{fernando_pessoa,
jorge_sampaio,
vasco_da_gama}
{portuguese,
spanish,
english}
sparse
vectorsx -> x ->
dense,
low-dim
vectors
x -> x ->
set vectors!
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
47/149
Path queries in vector space
{portugal}portugal / location / language
{fernando_pessoa,
jorge_sampaio,
vasco_da_gama}
{portuguese,
spanish,
english}
sparse
vectorsx -> x ->
dense,
low-dim
vectors
x -> x ->
traversaloperators
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
48/149
Path queries in vector space
{portugal} {fernando_pessoa,
jorge_sampaio,
vasco_da_gama}
{portuguese,
spanish,
english}
sparse
vectorsx -> x ->
dense,
low-dim
vectors
x -> x ->
membership operator:dot product
portugal / location / language
xenglish
xenglish
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
49/149
Path queries in vector space
q = s / r1 / r2 / / rk
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
50/149
Path queries in vector space
q = s / r1 / r2 / / rk
x>
sWr1Wr2 . . .W rk
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
51/149
Path queries in vector space
q = s / r1 / r2 / / rk
x>
sWr1Wr2 . . .W rk
is tan answer to q?
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
52/149
Path queries in vector space
q = s / r1 / r2 / / rk
x>
sWr1Wr2 . . .W rk
is tan answer to q?
xtscore(q, t) =
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
53/149
Path queries in vector space
q = s / r1 / r2 / / rk
x>
sWr1Wr2 . . .W rk
is tan answer to q?
xtscore(q, t) =
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
54/149
Training examples:
Margin:
Objective:
Algorithm:
Training
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
55/149
Training examples:
Margin:
Objective:
Algorithm:
Training
(q, t)
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
56/149
Training examples:
Margin:
Objective:
Algorithm:
Training
(q, t)
margin (q, t, t0) = score (q, t) score (q, t0)
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
57/149
Training examples:
Margin:
Objective:
Algorithm:
Training
(q, t)
margin (q, t, t0) = score (q, t) score (q, t0)
nX
i=1
X
t0N(qi)
[1margin (qi, ti, t0)]+
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
58/149
Training examples:
Margin:
Objective:
Algorithm:
Training
(q, t)
margin (q, t, t0) = score (q, t) score (q, t0)
nX
i=1
X
t0N(qi)
[1margin (qi, ti, t0)]+
SGD
T i i
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
59/149
Training examples:
Margin:
Objective:
Algorithm:
Training
(q, t)
margin (q, t, t0) = score (q, t) score (q, t0)
nX
i=1
X
t0N(qi)
[1margin (qi, ti, t0)]+
SGD
Next:generalize existing models
E i i d l
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
60/149
Existing vector space models
Tensor factorization [Nickel et al., 2011] Neural Tensor Network [Socher et al., 2013]
TransE [Bordes et al., 2013]
Universal Schema [Riedel et al., 2013] General framework + comparison [Yang et al.,
2015]
E i i d l
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
61/149
Existing vector space models
E i i d l
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
62/149
Existing vector space models
score (subject,predicate, object)
E i ti t d l
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
63/149
Existing vector space models
score (subject,predicate, object)
score (obama, nationality, united states)
core (obama,nationality, germany)
= HIGH
= LOW
E i ti t d l
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
64/149
Existing vector space models
score (subject,predicate, object)
score (obama, nationality, united states)
core (obama,nationality, germany)
= HIGH
= LOW
score (obama,nationality, france)?
E i ti t d l
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
65/149
Existing vector space models
score (subject,predicate, object)
score (obama, nationality, united states)
core (obama,nationality, germany)
= HIGH
= LOW
score (obama,nationality, france)?
special case ofanswering
path queries
E i ti t d l
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
66/149
Existing vector space models
r
s
t
Predict whether the triple (s, r, t)is true
N th d l
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
67/149
New path query models
r3r1 r2
r4
Predict whether a path existsbetween sand t
s
t
N th d l
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
68/149
New path query models
r3r1 r2
r4
Predict whether a path existsbetween sand t
s
t
single-hop > multi-hop
G li i ti d l
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
69/149
Original model
(single edge)
Generalize existing models
New model
(path)
G li i ti d l
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
70/149
Original model
(single edge)
Generalize existing models
New model
(path)
Bilinear
(Nickel+, 2012)
Generalize existing models
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
71/149
Original model
(single edge)
Generalize existing models
New model
(path)
x>
sWrxt
Bilinear
(Nickel+, 2012)
Generalize existing models
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
72/149
Original model
(single edge)
Generalize existing models
New model
(path)
x>
sWrxt x
>
sWr1Wr2 . . .W rkxt
Bilinear
(Nickel+, 2012)
Generalize existing models
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
73/149
Original model
(single edge)
Generalize existing models
New model
(path)
x>
sWrxt x
>
sWr1Wr2 . . .W rkxt
Bilinear
(Nickel+, 2012)
Generalize existing models
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
74/149
Original model
(single edge)
Generalize existing models
New model
(path)
x>
sWrxt x
>
sWr1Wr2 . . .W rkxt
Bilinear
(Nickel+, 2012)
Generalize existing models
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
75/149
Original model
(single edge)
Generalize existing models
New model
(path)
x>
sWrxt x
>
sWr1Wr2 . . .W rkxt
Bilinear
(Nickel+, 2012)
TransE
(Bordes+, 2013)
Generalize existing models
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
76/149
Original model
(single edge)
Generalize existing models
New model
(path)
x>
sWrxt x
>
sWr1Wr2 . . .W rkxt
Bilinear
(Nickel+, 2012)
TransE
(Bordes+, 2013) kxs + wr xtk
2
2
Generalize existing models
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
77/149
Original model
(single edge)
Generalize existing models
New model
(path)
x>
sWrxt x
>
sWr1Wr2 . . .W rkxt
Bilinear
(Nickel+, 2012)
TransE
(Bordes+, 2013) kxs + wr1 . . .+ wrk xtk
2
2kxs + wr xtk2
2
Generalize existing models
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
78/149
Original model
(single edge)
Generalize existing models
New model
(path)
x>
sWrxt x
>
sWr1Wr2 . . .W rkxt
Bilinear
(Nickel+, 2012)
TransE
(Bordes+, 2013) kxs + wr1 . . .+ wrk xtk
2
2kxs + wr xtk2
2
Generalize existing models
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
79/149
Original model
(single edge)
Generalize existing models
New model
(path)
x>
sWrxt x
>
sWr1Wr2 . . .W rkxt
Bilinear
(Nickel+, 2012)
TransE
(Bordes+, 2013) kxs + wr1 . . .+ wrk xtk
2
2kxs + wr xtk2
2
More generally
(??, 2015)
Generalize existing models
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
80/149
Original model
(single edge)
Generalize existing models
New model
(path)
x>
sWrxt x
>
sWr1Wr2 . . .W rkxt
Bilinear
(Nickel+, 2012)
TransE
(Bordes+, 2013) kxs + wr1 . . .+ wrk xtk
2
2kxs + wr xtk2
2
More generally
(??, 2015)M (Tr(xs), xt)
Generalize existing models
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
81/149
Original model
(single edge)
Generalize existing models
New model
(path)
x>
sWrxt x
>
sWr1Wr2 . . .W rkxt
Bilinear
(Nickel+, 2012)
TransE
(Bordes+, 2013) kxs + wr1 . . .+ wrk xtk
2
2kxs + wr xtk2
2
More generally
(??, 2015)M (Trk(. . .Tr1(xs)), xt)M (Tr(xs), xt)
Generalize existing models
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
82/149
Original model
(single edge)
Generalize existing models
New model
(path)
x>
sWrxt x
>
sWr1Wr2 . . .W rkxt
Bilinear
(Nickel+, 2012)
TransE
(Bordes+, 2013) kxs + wr1 . . .+ wrk xtk
2
2kxs + wr xtk2
2
More generally
(??, 2015)M (Trk(. . .Tr1(xs)), xt)M (Tr(xs), xt)
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
83/149
Generalize existing models
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
84/149
Original model
(single edge)
Generalize existing models
New model
(path)
x>
sWrxt x
>
sWr1Wr2 . . .W rkxt
Bilinear
(Nickel+, 2012)
TransE
(Bordes+, 2013) kxs + wr1 . . .+ wrk xtk
2
2kxs + wr xtk2
2
More generally
(??, 2015)M (Trk(. . .Tr1(xs)), xt)M (Tr(xs), xt)
answer path queries
a newway to trainexisting models
Old: single-edge training
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
85/149
Old: single-edge training
r
s
t
Predict whether the triple (s, r, t)is true
New: path training
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
86/149
New: path training
r3r1 r2
r4
Predict whether a path existsbetween sand t
s
t
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
87/149
Outline
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
88/149
Outline
PART I
Interpret many existing vector space models as each
implementing a traversal operator.
Proposehow all these models can be generalized to answer
path queries.
PART II
Showthat when we train these models to answer path queries,
they actually become better at predicting missing facts.
Explainwhy this new path-based training procedurehelps.
Experiments
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
89/149
Experiments
PATHSINGLE-
EDGE
PATH
SINGLE-EDGE
TRAINING
TASK
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
90/149
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
91/149
Experiments
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
92/149
Experiments
PATH PATH-1
PATH
PATH-1
TRAINING
TASK
portugal/location/language?
(pessoa, language, english)?
Train and test distributions match
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
93/149
Train and test distributions match
PATHSINGLE-
EDGE
PATH
SINGLE-EDGE
TRAINING
TASK
portugal/location/language?
(pessoa, language, english)?
Datasets
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
94/149
Datasets
[Chen et al, 2013] WordNet Freebase
Entities 39,000 75,000
Relations 11 13
Train edges 113,000 316,000
Test edges 11,000 24,000
Datasets
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
95/149
Datasets
[Chen et al, 2013] WordNet Freebase
Entities 39,000 75,000
Relations 11 13
Train edges 113,000 316,000
Test edges 11,000 24,000
(laugh, has_instance, giggle)
Datasets
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
96/149
Datasets
[Chen et al, 2013] WordNet Freebase
Entities 39,000 75,000
Relations 11 13
Train edges 113,000 316,000
Test edges 11,000 24,000
(laugh, has_instance, giggle)
(snort, derivationally_related, snorter)
Datasets
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
97/149
Datasets
[Chen et al, 2013] WordNet Freebase
Entities 39,000 75,000
Relations 11 13
Train edges 113,000 316,000
Test edges 11,000 24,000
(laugh, has_instance, giggle)
(snort, derivationally_related, snorter)
(abraham_lincoln, place_of_birth, kentucky)
Path training: way more data
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
98/149
Path training: way more data
~ WordNet Freebase
Single-edge training 113,000 316,000
Path training 2,000,000 6,000,000
Single-edge test 11,000 24,000
Path test 45,000 110,000
Path sampling
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
99/149
Path sampling
r3r1
r2r4
1. start at a random node in the knowledge graph2. traverse a random sequence of relations3. The final destination marked as one answer to the query.
s
t
Several models that we generalize
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
100/149
g
Evaluation metric
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
101/149
Evaluation metric
portugal / location / language
Evaluation metric
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
102/149
Evaluation metric
portugal / location / language
1. klingon2. portuguese3. latin
4. javascript5. chinese
Evaluation metric
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
103/149
a uat o et c
portugal / location / language
Quantile:percentage of negatives ranked aftercorrect answer (100 is best, 0 is worst)
1. klingon2. portuguese3. latin
4. javascript5. chinese
Evaluation metric
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
104/149
portugal / location / language
Quantile:percentage of negatives ranked aftercorrect answer (100 is best, 0 is worst)
75%
1. klingon2. portuguese3. latin
4. javascript5. chinese
Experiments
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
105/149
p
PATHSINGLE-
EDGE
PATH
SINGLE-EDGE
TRAINING
TASK
portugal/location/language?
(pessoa, language, english)?
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
106/149
Results: path querying
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
107/149
p q y g
Freebas
e
50
60
70
80
90
100
Bilinear Bilinear-Diag TransE
Edge Path Edge Path Edge Path
WordNet
50
6070
80
90
100
Bilinear Bilinear-Diag TransE
Edge Path Edge Path Edge Path
Why does path training help?
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
108/149
y p g p
Why does path training help?
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
109/149
Train and test distributions match:If you test onpath queries, you should train on path queries, not
single edges.
y p g p
Why does path training help?
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
110/149
Train and test distributions match:If you test onpath queries, you should train on path queries, not
single edges.
Why should single-edge training be so bad at
path queries?
y p g p
Why does path training help?
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
111/149
Train and test distributions match:If you test onpath queries, you should train on path queries, not
single edges.
Why should single-edge training be so bad at
path queries? Path query training cuts down on cascading errors
y p g p
Cascading errors
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
112/149
g
Who is Tad Lincolns grandparent?
Cascading errors
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
113/149
g
Who is Tad Lincolns grandparent?
Cascading errors
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
114/149
g
Who is Tad Lincolns grandparent?
pathtraining
Example
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
115/149
p
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
116/149
PATHSINGLE-
EDGE
PATH
SINGLE-EDGE
TRAINING
TASK
portugal/location/language?
(pessoa, language, english)?
Results: single-edge task
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
117/149
Results: single-edge task
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
118/149
Freebas
e
50
60
70
80
90
100
Bilinear Bilinear-Diag TransE
Edge Path Edge Path Edge Path
WordNet
50
6070
80
90
100
Bilinear Bilinear-Diag TransE
Edge Path Edge Path Edge Path
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
119/149
PATHSINGLE-
EDGE
PATH
SINGLE-EDGE
TRAINING
TASK
portugal/location/language?
(pessoa, language, english)?
Why does path training helpon the single edge task?
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
120/149
on the single-edge task?
Why does path training helpon the single edge task?
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
121/149
on the single-edge task?
Paths in the graph can help infer other edges.
Why does path training helpon the single edge task?
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
122/149
on the single-edge task?
Paths in the graph can help infer other edges.
Lao et al, 2010 (Path Ranking Algorithm)
Why does path training helpon the single edge task?
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
123/149
on the single-edge task?
Paths in the graph can help infer other edges.
Lao et al, 2010 (Path Ranking Algorithm)
Nickel et al, 2014
Why does path training helpon the single edge task?
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
124/149
on the single-edge task?
Paths in the graph can help infer other edges.
Lao et al, 2010 (Path Ranking Algorithm)
Nickel et al, 2014
Neelakantan et al, 2015
Why does path training helpon the single edge task?
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
125/149
on the single-edge task?
Paths in the graph can help infer other edges. Lao et al, 2010 (Path Ranking Algorithm)
Nickel et al, 2014
Neelakantan et al, 2015spouse
child
A B
C
Why does path training helpon the single edge task?
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
126/149
on the single-edge task?
Paths in the graph can help infer other edges. Lao et al, 2010 (Path Ranking Algorithm)
Nickel et al, 2014
Neelakantan et al, 2015spouse
child
A B
C
child?
Why does path training helpon the single edge task?
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
127/149
on the single-edge task?
Paths in the graph can help infer other edges. Lao et al, 2010 (Path Ranking Algorithm)
Nickel et al, 2014
Neelakantan et al, 2015spouse
child
A B
C
child?
spouse (A,B) child (B,C) = child (A,C)
Why does path training helpknowledge base completion?
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
128/149
knowledge base completion?
Why does path training helpknowledge base completion?
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
129/149
knowledge base completion?
If you cannot model paths correctly, you cannotuse them to infer edges.
Why does path training helpknowledge base completion?
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
130/149
knowledge base completion?
If you cannot model paths correctly, you cannotuse them to infer edges.
PATH SINGLE-EDGE
PATH
SINGLE-EDGE
TRAINING
TASK
Path training helps learn Horn clauses
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
131/149
Path training helps learn Horn clauses
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
132/149
parent (A,B) place of birth (B,C) = nationality (A,C)High-precision
Path training helps learn Horn clauses
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
133/149
parent (A,B) place of birth (B,C) = nationality (A,C)
High-precision
location (A,B) borders (B,C) = nationality (A,C)
Low-precision
Path training helps learn Horn clauses
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
134/149
parent (A,B) place of birth (B,C) = nationality (A,C)
High-precision
location (A,B) borders (B,C) = nationality (A,C)
Low-precision
child (A,B) gender (B,C) = nationality (A,C)
Zero-coverage
Path training helps learn Horn clauses
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
135/149
parent (A,B) place of birth (B,C) = nationality (A,C)
High-precision
location (A,B) borders (B,C) = nationality (A,C)
Low-precision
child (A,B) gender (B,C) = nationality (A,C)
Zero-coverage
WparentWplace of birth Wnationality
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
136/149
To recap
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
137/149
Graph databases Vector space models
Compositional
queries
Handleincompleteness
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
138/149
Connections
and
speculative thoughts
RNNs
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
139/149
RNNs
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
140/149
Repeated application of traversal operators can bethought of as implementing a recurrent neuralnetwork.
RNNs
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
141/149
Repeated application of traversal operators can bethought of as implementing a recurrent neuralnetwork.
Trk . . . Tr2 Tr1(xs)
RNNs
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
142/149
Repeated application of traversal operators can bethought of as implementing a recurrent neuralnetwork.
Trk . . . Tr2 Tr1(xs)
Initial hidden state: xs
Inputs: r1, r2, , rk
RNNs
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
143/149
Repeated application of traversal operators can bethought of as implementing a recurrent neuralnetwork.
Trk . . . Tr2 Tr1(xs)
Initial hidden state: xs
Inputs: r1, r2, , rk
[Neelakantan et al, 2015], [Graves et al, 2013]
Matrix factorization
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
144/149
x x
W
E
ET
[Nickel et al, 2013], [Bordes et al, 2011]
Matrix factorization
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
145/149
x x
W
E
ET
3
x x
WW
Graph-like data
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
146/149
All kinds of datasets are graphs
finished_giving
open domain relations
kelvin
his_talk
okay
was_considered
[Riedel et al, 2013], [Fader et al, 2011], [Banko et al 2007]
Graph-like data
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
147/149
All kinds of datasets are graphs
the boy happily ate ice cream
the child enjoyed dessert
the child hated dessert
entailment
negation
textual entailment
[Bowman et al, 2015], [Dagan et al, 2009]
Graph-like data
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
148/149
All kinds of datasets are graphs
0.1
word co-occurrences
king
throne
queen
0.4
[Levy, 2014], [Pennington, 2014], [Mikolov, 2013]
-
7/23/2019 Traversing Knowledge Graphs in Vector Space
149/149
Thank you!