cs224n research highlight - stanford university
TRANSCRIPT
CS224N Research Highlight
A Simple but Tough-to-beat Baseline for Sentence Embeddings
Sanjeev Arora, Yingyu Liang, Tengyu MaPrinceton University
Presenter: Danqi Chen
In submission to ICLR 2017
linguistics =
2
Word Sentence ?
0
BBBBBBBBBB@
0.2860.792�0.177�0.1070.109�0.5420.3490.271
1
CCCCCCCCCCA
linguistics =
2
Word Sentence ?
Natural language processing is fun. =
0
BBBBBBBBBB@
�0.1321.1290.8270.110�0.5270.1560.349�0.286
1
CCCCCCCCCCA
0
BBBBBBBBBB@
0.2860.792�0.177�0.1070.109�0.5420.3490.271
1
CCCCCCCCCCA
3
Sentence embedding
• Compute sentence similarity using the inner product:
S1: Mexico wishes to guarantee citizen’s safety.
S2: Mexico wishes to avoid more violence.Score: 4 (/5)
S1: Iranians Vote in Presidential Election.
S2: Keita Wins Mali Presidential Election. Score: 0.4 (/5)
4
Sentence embedding
Natural language processing is fun. =
0
BBBBBBBBBB@
�0.1321.1290.8270.110�0.5270.1560.349�0.286
1
CCCCCCCCCCA
• Use as features for sentence classification (e.g., sentiment analysis):
• Bag-of-words (BoW) v(“natural language processing”) =
1/3 (v(“natural”) + v(“language”) + v(“processing”))
5
From Bag-of-words to Complex Models…
• Bag-of-words (BoW) v(“natural language processing”) =
1/3 (v(“natural”) + v(“language”) + v(“processing”))
5
From Bag-of-words to Complex Models…
• Recurrent neural networks, recursive neural networks, convolutional neural networks..
• A VERY SIMPLE unsupervised method
• weighted Bag-of-words + remove some special direction
6
This paper
• A VERY SIMPLE unsupervised method
• weighted Bag-of-words + remove some special direction
6
This paper
• Step 1:
• A VERY SIMPLE unsupervised method
• weighted Bag-of-words + remove some special direction
6
This paper
• Step 1:
• Step 2: