syntactic parsing with recursive neural...
TRANSCRIPT
![Page 1: Syntactic parsing with Recursive Neural Networkscompling.hss.ntu.edu.sg/courses/hg7017/pdf/rrn.pdf · Syntactic parsing with Recursive Neural Networks. ... Motivation 2. Review of](https://reader033.vdocuments.us/reader033/viewer/2022053011/5f0e5f3b7e708231d43eecec/html5/thumbnails/1.jpg)
Syntactic parsing with Recursive Neural
Networks
![Page 2: Syntactic parsing with Recursive Neural Networkscompling.hss.ntu.edu.sg/courses/hg7017/pdf/rrn.pdf · Syntactic parsing with Recursive Neural Networks. ... Motivation 2. Review of](https://reader033.vdocuments.us/reader033/viewer/2022053011/5f0e5f3b7e708231d43eecec/html5/thumbnails/2.jpg)
Agenda
1. Motivation2. Review of word vectors3. Recursive NN for syntactic parsing4. Resources & summary
![Page 3: Syntactic parsing with Recursive Neural Networkscompling.hss.ntu.edu.sg/courses/hg7017/pdf/rrn.pdf · Syntactic parsing with Recursive Neural Networks. ... Motivation 2. Review of](https://reader033.vdocuments.us/reader033/viewer/2022053011/5f0e5f3b7e708231d43eecec/html5/thumbnails/3.jpg)
![Page 4: Syntactic parsing with Recursive Neural Networkscompling.hss.ntu.edu.sg/courses/hg7017/pdf/rrn.pdf · Syntactic parsing with Recursive Neural Networks. ... Motivation 2. Review of](https://reader033.vdocuments.us/reader033/viewer/2022053011/5f0e5f3b7e708231d43eecec/html5/thumbnails/4.jpg)
![Page 5: Syntactic parsing with Recursive Neural Networkscompling.hss.ntu.edu.sg/courses/hg7017/pdf/rrn.pdf · Syntactic parsing with Recursive Neural Networks. ... Motivation 2. Review of](https://reader033.vdocuments.us/reader033/viewer/2022053011/5f0e5f3b7e708231d43eecec/html5/thumbnails/5.jpg)
Motivation for research
Lexical information is needed for correct parsing
![Page 6: Syntactic parsing with Recursive Neural Networkscompling.hss.ntu.edu.sg/courses/hg7017/pdf/rrn.pdf · Syntactic parsing with Recursive Neural Networks. ... Motivation 2. Review of](https://reader033.vdocuments.us/reader033/viewer/2022053011/5f0e5f3b7e708231d43eecec/html5/thumbnails/6.jpg)
● Naive○ add lexical information into syntactic category
● Discriminative parsing● Advanced feature engineering
○ Refining each category: Ex. NP -> {NP-1, NP-2,NP-3,...}
Lexicalization approaches
![Page 7: Syntactic parsing with Recursive Neural Networkscompling.hss.ntu.edu.sg/courses/hg7017/pdf/rrn.pdf · Syntactic parsing with Recursive Neural Networks. ... Motivation 2. Review of](https://reader033.vdocuments.us/reader033/viewer/2022053011/5f0e5f3b7e708231d43eecec/html5/thumbnails/7.jpg)
Problem
- No clear way of representing category/phrase- All approaches require complicated feature engineering process
![Page 8: Syntactic parsing with Recursive Neural Networkscompling.hss.ntu.edu.sg/courses/hg7017/pdf/rrn.pdf · Syntactic parsing with Recursive Neural Networks. ... Motivation 2. Review of](https://reader033.vdocuments.us/reader033/viewer/2022053011/5f0e5f3b7e708231d43eecec/html5/thumbnails/8.jpg)
Is it possible to learn features automatically?
![Page 9: Syntactic parsing with Recursive Neural Networkscompling.hss.ntu.edu.sg/courses/hg7017/pdf/rrn.pdf · Syntactic parsing with Recursive Neural Networks. ... Motivation 2. Review of](https://reader033.vdocuments.us/reader033/viewer/2022053011/5f0e5f3b7e708231d43eecec/html5/thumbnails/9.jpg)
Solution
1. Use continuous word vectors as input2. Train recursive neural network for structure
prediction
![Page 10: Syntactic parsing with Recursive Neural Networkscompling.hss.ntu.edu.sg/courses/hg7017/pdf/rrn.pdf · Syntactic parsing with Recursive Neural Networks. ... Motivation 2. Review of](https://reader033.vdocuments.us/reader033/viewer/2022053011/5f0e5f3b7e708231d43eecec/html5/thumbnails/10.jpg)
What we achieve
+ Only 1,9% behind Stanford Parser(2003) + Good results on short sentences + No manual engineering, No POS tagging
![Page 11: Syntactic parsing with Recursive Neural Networkscompling.hss.ntu.edu.sg/courses/hg7017/pdf/rrn.pdf · Syntactic parsing with Recursive Neural Networks. ... Motivation 2. Review of](https://reader033.vdocuments.us/reader033/viewer/2022053011/5f0e5f3b7e708231d43eecec/html5/thumbnails/11.jpg)
Word representation
Symbolic representation of the word(one-hot)
0 0 0 1 … 0 0 0
0 0 0 0 … 1 0 0
chopsticks
fork
![Page 12: Syntactic parsing with Recursive Neural Networkscompling.hss.ntu.edu.sg/courses/hg7017/pdf/rrn.pdf · Syntactic parsing with Recursive Neural Networks. ... Motivation 2. Review of](https://reader033.vdocuments.us/reader033/viewer/2022053011/5f0e5f3b7e708231d43eecec/html5/thumbnails/12.jpg)
Limitations of one-hot model
● Rare words in training data -> poorly estimated
● Not seen in training data -> model cannot handle it
![Page 13: Syntactic parsing with Recursive Neural Networkscompling.hss.ntu.edu.sg/courses/hg7017/pdf/rrn.pdf · Syntactic parsing with Recursive Neural Networks. ... Motivation 2. Review of](https://reader033.vdocuments.us/reader033/viewer/2022053011/5f0e5f3b7e708231d43eecec/html5/thumbnails/13.jpg)
Word vectors
Figure from Y. Bengio tutorial 2012 “Recursive neural networks”
![Page 14: Syntactic parsing with Recursive Neural Networkscompling.hss.ntu.edu.sg/courses/hg7017/pdf/rrn.pdf · Syntactic parsing with Recursive Neural Networks. ... Motivation 2. Review of](https://reader033.vdocuments.us/reader033/viewer/2022053011/5f0e5f3b7e708231d43eecec/html5/thumbnails/14.jpg)
Problem
How to get representations for phrases if we have only word vectors?
![Page 15: Syntactic parsing with Recursive Neural Networkscompling.hss.ntu.edu.sg/courses/hg7017/pdf/rrn.pdf · Syntactic parsing with Recursive Neural Networks. ... Motivation 2. Review of](https://reader033.vdocuments.us/reader033/viewer/2022053011/5f0e5f3b7e708231d43eecec/html5/thumbnails/15.jpg)
Compositionality principle
Figure from Y. Bengio tutorial 2012 “Recursive neural networks”
![Page 16: Syntactic parsing with Recursive Neural Networkscompling.hss.ntu.edu.sg/courses/hg7017/pdf/rrn.pdf · Syntactic parsing with Recursive Neural Networks. ... Motivation 2. Review of](https://reader033.vdocuments.us/reader033/viewer/2022053011/5f0e5f3b7e708231d43eecec/html5/thumbnails/16.jpg)
Recursive NN
Figure from Socher et al. 2010
![Page 17: Syntactic parsing with Recursive Neural Networkscompling.hss.ntu.edu.sg/courses/hg7017/pdf/rrn.pdf · Syntactic parsing with Recursive Neural Networks. ... Motivation 2. Review of](https://reader033.vdocuments.us/reader033/viewer/2022053011/5f0e5f3b7e708231d43eecec/html5/thumbnails/17.jpg)
Architecture
Figure from Socher et al. 2010
![Page 18: Syntactic parsing with Recursive Neural Networkscompling.hss.ntu.edu.sg/courses/hg7017/pdf/rrn.pdf · Syntactic parsing with Recursive Neural Networks. ... Motivation 2. Review of](https://reader033.vdocuments.us/reader033/viewer/2022053011/5f0e5f3b7e708231d43eecec/html5/thumbnails/18.jpg)
Parsing a sentence
Figure from Y. Bengio tutorial 2012 “Recursive neural networks”
![Page 19: Syntactic parsing with Recursive Neural Networkscompling.hss.ntu.edu.sg/courses/hg7017/pdf/rrn.pdf · Syntactic parsing with Recursive Neural Networks. ... Motivation 2. Review of](https://reader033.vdocuments.us/reader033/viewer/2022053011/5f0e5f3b7e708231d43eecec/html5/thumbnails/19.jpg)
Parsing a sentence
Figure from Y. Bengio tutorial 2012 “Recursive neural networks”
![Page 20: Syntactic parsing with Recursive Neural Networkscompling.hss.ntu.edu.sg/courses/hg7017/pdf/rrn.pdf · Syntactic parsing with Recursive Neural Networks. ... Motivation 2. Review of](https://reader033.vdocuments.us/reader033/viewer/2022053011/5f0e5f3b7e708231d43eecec/html5/thumbnails/20.jpg)
Training RNN
● Collect correct parse trees from the corpora● For each collapses of 2 words into phrase
we assign score:○ Correct collapse -> higher score○ Incorrect collapse -> lower score
● Thats how we learn W and Wscore
![Page 21: Syntactic parsing with Recursive Neural Networkscompling.hss.ntu.edu.sg/courses/hg7017/pdf/rrn.pdf · Syntactic parsing with Recursive Neural Networks. ... Motivation 2. Review of](https://reader033.vdocuments.us/reader033/viewer/2022053011/5f0e5f3b7e708231d43eecec/html5/thumbnails/21.jpg)
Some types of RNN
1. Greedy RNN2. Context aware RNN3. Context aware RNN + category classifier4. Max-Margin Framework with Beam-Search
![Page 22: Syntactic parsing with Recursive Neural Networkscompling.hss.ntu.edu.sg/courses/hg7017/pdf/rrn.pdf · Syntactic parsing with Recursive Neural Networks. ... Motivation 2. Review of](https://reader033.vdocuments.us/reader033/viewer/2022053011/5f0e5f3b7e708231d43eecec/html5/thumbnails/22.jpg)
Context aware RNN
Figure from Socher et al. 2010
![Page 23: Syntactic parsing with Recursive Neural Networkscompling.hss.ntu.edu.sg/courses/hg7017/pdf/rrn.pdf · Syntactic parsing with Recursive Neural Networks. ... Motivation 2. Review of](https://reader033.vdocuments.us/reader033/viewer/2022053011/5f0e5f3b7e708231d43eecec/html5/thumbnails/23.jpg)
Materials
1. CW08 “A Unified Architecture for Natural Language Processing: Deep Neural Networks with Multitask Learning”
2. TRB10, “Word representations: A simple and general method for semi-supervised learning”
![Page 24: Syntactic parsing with Recursive Neural Networkscompling.hss.ntu.edu.sg/courses/hg7017/pdf/rrn.pdf · Syntactic parsing with Recursive Neural Networks. ... Motivation 2. Review of](https://reader033.vdocuments.us/reader033/viewer/2022053011/5f0e5f3b7e708231d43eecec/html5/thumbnails/24.jpg)
Summary● Learned how to represent phrases having
only word vectors● Build framework for syntactic parsing
○ Can be used also for paraphrase detection