chains of reasoning over entities, relations, and text ...1. no reasoning about entities in path,...
TRANSCRIPT
![Page 1: Chains of Reasoning over Entities, Relations, and Text ...1. No reasoning about entities in path, just relations 2. Reasoning from single path 3. Train a separate model for each relation-type](https://reader034.vdocuments.us/reader034/viewer/2022052102/603c8c43ca78222bad630c0c/html5/thumbnails/1.jpg)
Chains of Reasoning over Entities, Relations, and Text
using RNNs Authors:
Rajarshi Das, Arvind Neelakantan, David Belanger, Andrew McCallum Presented by:
Assma Boughoula
![Page 2: Chains of Reasoning over Entities, Relations, and Text ...1. No reasoning about entities in path, just relations 2. Reasoning from single path 3. Train a separate model for each relation-type](https://reader034.vdocuments.us/reader034/viewer/2022052102/603c8c43ca78222bad630c0c/html5/thumbnails/2.jpg)
Motivation
● Knowledge Base completion through relation inference ● Infer probability of relation “Lives in” between entities “Melinda” and “Seattle” ● Given the other paths between them in the knowledge graph
Target relation
![Page 3: Chains of Reasoning over Entities, Relations, and Text ...1. No reasoning about entities in path, just relations 2. Reasoning from single path 3. Train a separate model for each relation-type](https://reader034.vdocuments.us/reader034/viewer/2022052102/603c8c43ca78222bad630c0c/html5/thumbnails/3.jpg)
Previous work: Compositional VSM for KB Completion (2015)
Previously:
1. No reasoning about entities in path, just relations
2. Reasoning from single path 3. Train a separate model for each
relation-type
This work:
1. Jointly reason about relation-types, entities and entity-types
2. Multiple paths 3. Single RNN that can predict all
relation types
Compositional Vector Space Models for Knowledge Base Completion
![Page 4: Chains of Reasoning over Entities, Relations, and Text ...1. No reasoning about entities in path, just relations 2. Reasoning from single path 3. Train a separate model for each relation-type](https://reader034.vdocuments.us/reader034/viewer/2022052102/603c8c43ca78222bad630c0c/html5/thumbnails/4.jpg)
Per-relation Model: Path-RNN
● Train a separate RNN for each relation type (CountryOfHeadquarters)
● Only relation vectors are taken into account
![Page 5: Chains of Reasoning over Entities, Relations, and Text ...1. No reasoning about entities in path, just relations 2. Reasoning from single path 3. Train a separate model for each relation-type](https://reader034.vdocuments.us/reader034/viewer/2022052102/603c8c43ca78222bad630c0c/html5/thumbnails/5.jpg)
Single-Model
Target Relation: “country of HQ” Start: “Microsoft” Target: “USA”
Concatenation + RNN step
ej Learned during training
rk
![Page 6: Chains of Reasoning over Entities, Relations, and Text ...1. No reasoning about entities in path, just relations 2. Reasoning from single path 3. Train a separate model for each relation-type](https://reader034.vdocuments.us/reader034/viewer/2022052102/603c8c43ca78222bad630c0c/html5/thumbnails/6.jpg)
Single-Model
RNN hidden state at step t in the path
Now Previous
No dependency on target relation r here!
f = sigmoid function
![Page 7: Chains of Reasoning over Entities, Relations, and Text ...1. No reasoning about entities in path, just relations 2. Reasoning from single path 3. Train a separate model for each relation-type](https://reader034.vdocuments.us/reader034/viewer/2022052102/603c8c43ca78222bad630c0c/html5/thumbnails/7.jpg)
Single-Model: incorporating entities
1. Learn Entity Vector Representation
2. Get annotated entity types from
FreeBase: “Melinda Gates”
CEO Duke University Alumni Philanthropist American Citezen
Entity Representation
![Page 8: Chains of Reasoning over Entities, Relations, and Text ...1. No reasoning about entities in path, just relations 2. Reasoning from single path 3. Train a separate model for each relation-type](https://reader034.vdocuments.us/reader034/viewer/2022052102/603c8c43ca78222bad630c0c/html5/thumbnails/8.jpg)
So far …
![Page 9: Chains of Reasoning over Entities, Relations, and Text ...1. No reasoning about entities in path, just relations 2. Reasoning from single path 3. Train a separate model for each relation-type](https://reader034.vdocuments.us/reader034/viewer/2022052102/603c8c43ca78222bad630c0c/html5/thumbnails/9.jpg)
Single-Model: score pooling Assigns 0 weight to some paths Each path gets same share of gradient regardless of whether it’s more/less important Each path gets share of gradient proportional to its score since:
Top K
Average
LogSumExp
![Page 10: Chains of Reasoning over Entities, Relations, and Text ...1. No reasoning about entities in path, just relations 2. Reasoning from single path 3. Train a separate model for each relation-type](https://reader034.vdocuments.us/reader034/viewer/2022052102/603c8c43ca78222bad630c0c/html5/thumbnails/10.jpg)
Single-Model: score pooling Assigns 0 weight to some paths Each path gets same share of gradient regardless of whether it’s more/less important Each path gets share of gradient proportional to its score since:
Top K
Average
LogSumExp
JFK – locatedIn – NYC – locatedIn – NY YankeeStadium – locatedIn – NYC – locatedIn – NY Target relation: airport_serves
![Page 11: Chains of Reasoning over Entities, Relations, and Text ...1. No reasoning about entities in path, just relations 2. Reasoning from single path 3. Train a separate model for each relation-type](https://reader034.vdocuments.us/reader034/viewer/2022052102/603c8c43ca78222bad630c0c/html5/thumbnails/11.jpg)
Experiments Setup
Dataset:
● Triples: (es, r, et )
● Set of paths S connecting (es, et ) in the knowledge graph
Data is from FreeBase (KB) and ClueWeb(Text)
![Page 12: Chains of Reasoning over Entities, Relations, and Text ...1. No reasoning about entities in path, just relations 2. Reasoning from single path 3. Train a separate model for each relation-type](https://reader034.vdocuments.us/reader034/viewer/2022052102/603c8c43ca78222bad630c0c/html5/thumbnails/12.jpg)
Experiments: Effect of Pooling Techniques
● LogSumExp is best
○ Important to include all paths
● Average is worst
○ Important to weigh path scores according to their values
![Page 13: Chains of Reasoning over Entities, Relations, and Text ...1. No reasoning about entities in path, just relations 2. Reasoning from single path 3. Train a separate model for each relation-type](https://reader034.vdocuments.us/reader034/viewer/2022052102/603c8c43ca78222bad630c0c/html5/thumbnails/13.jpg)
Experiments Results
Effect of Pooling
Other Multi-Hop Algorithms
Effect Including Entities
![Page 14: Chains of Reasoning over Entities, Relations, and Text ...1. No reasoning about entities in path, just relations 2. Reasoning from single path 3. Train a separate model for each relation-type](https://reader034.vdocuments.us/reader034/viewer/2022052102/603c8c43ca78222bad630c0c/html5/thumbnails/14.jpg)
Take-aways:
● Complete Knowledge Graph by inferring relations between entities using existing paths
● Single-Model: trains a single RNN to handle multiple relation types
● Incorporating Entity vectors improves results