study of end to end memory networks
TRANSCRIPT
A presentation on
End to End Memory Networks (MemN2N)
Slides: 26Time: 15 minutes
IE 594 Data Science 2University of Illinois at Chicago, February 2017
Under the guidance of,Prof. Dr. Ashkan Sharabiani
By,Ashish Menkudale
2
The kitchen is north of the hallway.
The bathroom is west of the bedroom.
The den is east of the hallway.
The office is south of the bedroom.
How do you go from den to kitchen?
3
The kitchen is north of the hallway.
The bathroom is west of the bedroom.
The den is east of the hallway.
The office is south of the bedroom.
How do you go from den to kitchen?
Kitchen
Hallway
Bathroom Bedroom
Den
Office
West, North.
West
North
4
Brian is frog.Lily is grey.
Brian is yellow.Julius is green.
What color is Greg?
Greg is frog.
5
Brian is frog.
Lily is grey.
Brian is yellow.
Julius is green.
What color is Greg?
Greg is frog.Br
ian
Lily
Lily
Juliu
s
Juliu
s
Bria
n
Greg
Yellow.
6
External Global Memory
Memory Module
Controller Module
Output
Read
Write
Input
Dedicated separate memory module.
Memory can be
stack or list/set of vectors.
Control module accesses memory
(read, write).
Advantage: stable, scalable.
Charles BabbageInvented
analytical engineConcept.
1791-1871
Konrad ZuseInvented
stored-program concept.
1910-1995
7
Warren Sturgis McCulloch.
Computational model for neural
networks1898-1969
Memory Networks
• Memory network with large external memory.required for low level tasks like object recognition.
• Writes everything to the memory, but reads only relative information.• Attempts to add long term memory component to make it more like artificial intelligence.
• Two types:• Strongly supervised memory network: Hard addressing.• Weekly supervised memory network: Soft addressing.
• Hard addressing: max of the inner product between the internal state and memory contents.
Mary is in garden. John is in office. Q: Where is John? Bob is in kitchen.
Walter Pitts.Computational
model for neural networks
1923-1969
8
Memory VectorsExample: Constructing memory vectors with bag of words (BoW)
Embed each wordSum embedding vectors
“Sam drops apple” apple =
Embedding vectorsmemory vector
Example: Temporal structure – special words for time and include them in bag of words.
1. Sam moved to garden2. Sam moved to kitchen.3. Sam drops apple.
apple + time =
Time EmbeddingTime Stamp
9
Bob is in kitchen. Mary is in garden. John is in office. Where is John?
Embed Embed Embed Embed
X X X
Max
Internal State vector
John is in office
Embed +
Decoder
OfficeOutput
Supervision
Memory Controller
Memory NetworksInput
10
Issues with Memory Network
• Requires explicit supervision of attention during training. Need to say which memory the model should use.
• Need a model that just requires supervision at output.No supervision of attention required.
• Only feasible for simple tasks. Severely limits application of model.
11
End-To-End Memory Networks
• Soft attention version of MemN2N.• Flexible read-only memory.
• Multiple memory lookups (hops).• Can consider multiple memory before deciding output.• More reasoning power.
• End-to-end training.• Only needs final output for training.• Simple back-propagation.
Sainbayar Sukhbaatar
Arthur Szlam
Jason Weston
Rob Fergus
12
Tanh / ReLU
Dot product
Softmax
Weighted Sum
Mem
ory
cont
ent
Sum
Linear
State
StateMemory module
Output TargetLossFunction
Input
Controller moduleE.g. RNN
MemN2N architecture
Supervision
13
MemN2N in action : Single memory lookup
Sentences {Xi}
Softmax
Question q
Embedding BInner product
Embedding A
Embedding C
Probability
Weighted Sum
∑
Inpu
tsW
eigh
tsO
utpu
t
O
u
Weight
Softmax
PredictedAnswer
Mary is in garden. John is in office. Bob is in kitchen.
Where is John?
OfficeTraining: estimate embedding matrices A, B & C and output matrix W.
14
A3
C3
A2
C2
C1
Multiple Memory Lookups: Multiple Hops
Sentences {Xi}
Input 1
Output 1
∑
Question q
Input 2
Output 2
∑
Input 3
Output 3
∑
A1
O1
O1
O3W �̂�
Predicted Answer
u1
u2
u3
15
Components
I (Input): No conversion keep original text X.
G (Generalization): Stores I (X) in next available memory slot.
O (Output): Loops over all memories.Find best match of with X.Find best match of with ( , X)Can be extended to multiple number of hops.
R (Response) : Ranks all words in dictionary given o and returns best single word. Infact, RNN can be used here for better sentence correction.
16
Weight Tying
Weight tying : Indicates how weight vectors are multiplied to input and output component.
Two Methods:Adjacent:
Similar to stack layersOutput embedding of one layer are input embedding of the next layer.
Layer wise: Input embedding remains the same for every layer in architecture.
17
Scoring function
Question : Answers are mapped to story using word embedding.
Word Embedding : Maps different words in low dimensional vector space with advantage to calculate distance between word vectors.
Allow us to find similarity score between different sentence to understand maximum correlation between them.
Match (‘Where is football?’, ‘John picked up the football’).
qTUTUd : This model is default word embedding used in memory networks.q – Question.U – matrix by which word embedding are obtained. d – Answer.
18
Model Selection
Model Selection: Determines how to model story, questions and answer vectors for word embedding.
Two possible approach:
Bag of words model: Considers each word in a sentence.Embeds each word and sums resulting vector.Does not take into account context for each word.
Position Encoding: Considers position/context of sentence/words.Takes care of preceding and forwarding words.Maps it to low dimensional vector space.
Model RefiningAddition of noise.increasing training dataset.
19
Decisions for Configuration
• Number of hops
• Number of epochs
• Embedding size
• Training dataset
• Validation dataset
• Model selection
• Weight tying
20
RNN viewpoint of MemN2N
Plain RNN Memory Network
RNN
Input Sequence
Memory
RNN
All Input
Selected input Addressing signal
Inputs are fed to RNN one-by-one in order. RNN has only one chance to look at a certain input symbol.
Place all inputs in the memory. Let the model decide which part it reads next.
21
• More generic input format • Any set of vectors can be input • Each vector can be
o BOW of symbols (including location) o Image feature + feature position
• Location can be 1D, 2D, … • Variable size
Advantages of MemN2N over RNN
• Out-of-order access to input data
• Less distracted by unimportant inputs
• Longer term memorization
• No vanishing or exploding gradient problems
bAbi Project: Task CAtegories
Training dataset: 1000 questions for each tasks. Testing dataset: 1000 questions for each tasks.
23
Demo for bAbi tasks
24
bAbi Project: Benchmark results
25
1. GitHub project archives: https://github.com/vinhkhuc/MemN2N-babi-python2. https://www.msri.org/workshops/796/schedules/20462/documents/2704/assets/247343. Dynamic Neural Turing Machine with Soft and Hard Addressing Schemes: https://arxiv.org/pdf/1607.00036.pdf4. bAbi answers: https://arxiv.org/pdf/1502.05698.pdf5. Memory Networks by Microsoft research: https://www.youtube.com/watch?v=ZwvWY9Yy76Q&t=1s6. Memory Networks (Jenil Shah): https://www.youtube.com/watch?v=BN7Kp0JD04o7. N gram – SVM – generative models difference.
http://stackoverflow.com/questions/20315897/n-grams-vs-other-classifiers-in-text-categorization8. Paper on results for bAbi tasks by Facebook AI team:
https://papers.nips.cc/paper/5846-end-to-end-memory-networks.pdf9. Towards AI-complete question answering : a set of prerequisite toy tasks https://arxiv.org/pdf/1502.05698.pdf
References
26
Questions