![Page 1: The Language Resource Spectrum: A Perspective from …lrec2016.lrec-conf.org/media/filer_public/2017/01/27/ryanmcdonaldl... · The Language Resource Spectrum: A Perspective from Google](https://reader030.vdocuments.us/reader030/viewer/2022013006/5b02c5237f8b9ab9598e5346/html5/thumbnails/1.jpg)
Ryan McDonald
Google NLU team Google Linguistics team
The Language Resource Spectrum: A Perspective from Google
![Page 2: The Language Resource Spectrum: A Perspective from …lrec2016.lrec-conf.org/media/filer_public/2017/01/27/ryanmcdonaldl... · The Language Resource Spectrum: A Perspective from Google](https://reader030.vdocuments.us/reader030/viewer/2022013006/5b02c5237f8b9ab9598e5346/html5/thumbnails/2.jpg)
Ryan McDonald
Google NLU team Google Linguistics team
The Language Resource Spectrum: A Perspective from Google
![Page 3: The Language Resource Spectrum: A Perspective from …lrec2016.lrec-conf.org/media/filer_public/2017/01/27/ryanmcdonaldl... · The Language Resource Spectrum: A Perspective from Google](https://reader030.vdocuments.us/reader030/viewer/2022013006/5b02c5237f8b9ab9598e5346/html5/thumbnails/3.jpg)
LREC, May 2016
Language Resource Spectrum
search logs
![Page 4: The Language Resource Spectrum: A Perspective from …lrec2016.lrec-conf.org/media/filer_public/2017/01/27/ryanmcdonaldl... · The Language Resource Spectrum: A Perspective from Google](https://reader030.vdocuments.us/reader030/viewer/2022013006/5b02c5237f8b9ab9598e5346/html5/thumbnails/4.jpg)
LREC, May 2016
Language Resource Spectrum
search logs
unsu
pervis
ed
weakly
super
vised
fully
super
vised
![Page 5: The Language Resource Spectrum: A Perspective from …lrec2016.lrec-conf.org/media/filer_public/2017/01/27/ryanmcdonaldl... · The Language Resource Spectrum: A Perspective from Google](https://reader030.vdocuments.us/reader030/viewer/2022013006/5b02c5237f8b9ab9598e5346/html5/thumbnails/5.jpg)
LREC, May 2016
amount of
data
supervision
![Page 6: The Language Resource Spectrum: A Perspective from …lrec2016.lrec-conf.org/media/filer_public/2017/01/27/ryanmcdonaldl... · The Language Resource Spectrum: A Perspective from Google](https://reader030.vdocuments.us/reader030/viewer/2022013006/5b02c5237f8b9ab9598e5346/html5/thumbnails/6.jpg)
LREC, May 2016
amount of
data
supervision
![Page 7: The Language Resource Spectrum: A Perspective from …lrec2016.lrec-conf.org/media/filer_public/2017/01/27/ryanmcdonaldl... · The Language Resource Spectrum: A Perspective from Google](https://reader030.vdocuments.us/reader030/viewer/2022013006/5b02c5237f8b9ab9598e5346/html5/thumbnails/7.jpg)
LREC, May 2016
amount of
data
supervision
![Page 8: The Language Resource Spectrum: A Perspective from …lrec2016.lrec-conf.org/media/filer_public/2017/01/27/ryanmcdonaldl... · The Language Resource Spectrum: A Perspective from Google](https://reader030.vdocuments.us/reader030/viewer/2022013006/5b02c5237f8b9ab9598e5346/html5/thumbnails/8.jpg)
LREC, May 2016
amount of
data
supervision
which is better?
![Page 9: The Language Resource Spectrum: A Perspective from …lrec2016.lrec-conf.org/media/filer_public/2017/01/27/ryanmcdonaldl... · The Language Resource Spectrum: A Perspective from Google](https://reader030.vdocuments.us/reader030/viewer/2022013006/5b02c5237f8b9ab9598e5346/html5/thumbnails/9.jpg)
LREC, May 2016
High quality annotations Crowd-sourced
Software Engineer Auto resources
Models Active Learning
Pre-existing resources
![Page 10: The Language Resource Spectrum: A Perspective from …lrec2016.lrec-conf.org/media/filer_public/2017/01/27/ryanmcdonaldl... · The Language Resource Spectrum: A Perspective from Google](https://reader030.vdocuments.us/reader030/viewer/2022013006/5b02c5237f8b9ab9598e5346/html5/thumbnails/10.jpg)
LREC, May 2016
High quality annotations Crowd-sourced
Software Engineer Auto resources
Models Active Learning
Pre-existing resources
α β γ δ
![Page 11: The Language Resource Spectrum: A Perspective from …lrec2016.lrec-conf.org/media/filer_public/2017/01/27/ryanmcdonaldl... · The Language Resource Spectrum: A Perspective from Google](https://reader030.vdocuments.us/reader030/viewer/2022013006/5b02c5237f8b9ab9598e5346/html5/thumbnails/11.jpg)
LREC, May 2016
High quality annotations Crowd-sourced
Software Engineer Auto resources
Models Active Learning
Pre-existing resources
α β γ δ
![Page 12: The Language Resource Spectrum: A Perspective from …lrec2016.lrec-conf.org/media/filer_public/2017/01/27/ryanmcdonaldl... · The Language Resource Spectrum: A Perspective from Google](https://reader030.vdocuments.us/reader030/viewer/2022013006/5b02c5237f8b9ab9598e5346/html5/thumbnails/12.jpg)
LREC, May 2016
The Task Matters
✤ ML is really good at the head
POS tagging
QA
![Page 13: The Language Resource Spectrum: A Perspective from …lrec2016.lrec-conf.org/media/filer_public/2017/01/27/ryanmcdonaldl... · The Language Resource Spectrum: A Perspective from Google](https://reader030.vdocuments.us/reader030/viewer/2022013006/5b02c5237f8b9ab9598e5346/html5/thumbnails/13.jpg)
LREC, May 2016
Pipelined / multi-Component Systems
Indexer Text Extractor
Segmentation Morphosyntax
Shallow Semantics MT QA
SearchEntity resolution Relation extraction
![Page 14: The Language Resource Spectrum: A Perspective from …lrec2016.lrec-conf.org/media/filer_public/2017/01/27/ryanmcdonaldl... · The Language Resource Spectrum: A Perspective from Google](https://reader030.vdocuments.us/reader030/viewer/2022013006/5b02c5237f8b9ab9598e5346/html5/thumbnails/14.jpg)
LREC, May 2016
Pipelined / multi-Component Systems
Indexer Text Extractor
Segmentation Morphosyntax
Shallow Semantics MT QA
SearchEntity resolution Relation extraction
End User Task
![Page 15: The Language Resource Spectrum: A Perspective from …lrec2016.lrec-conf.org/media/filer_public/2017/01/27/ryanmcdonaldl... · The Language Resource Spectrum: A Perspective from Google](https://reader030.vdocuments.us/reader030/viewer/2022013006/5b02c5237f8b9ab9598e5346/html5/thumbnails/15.jpg)
LREC, May 2016
Pipelined / multi-Component Systems
Indexer Text Extractor
Segmentation Morphosyntax
Shallow Semantics MT QA
SearchEntity resolution Relation extraction
End User Task
Upstream Task
![Page 16: The Language Resource Spectrum: A Perspective from …lrec2016.lrec-conf.org/media/filer_public/2017/01/27/ryanmcdonaldl... · The Language Resource Spectrum: A Perspective from Google](https://reader030.vdocuments.us/reader030/viewer/2022013006/5b02c5237f8b9ab9598e5346/html5/thumbnails/16.jpg)
Upstream: Morphosyntactic Tagging
![Page 17: The Language Resource Spectrum: A Perspective from …lrec2016.lrec-conf.org/media/filer_public/2017/01/27/ryanmcdonaldl... · The Language Resource Spectrum: A Perspective from Google](https://reader030.vdocuments.us/reader030/viewer/2022013006/5b02c5237f8b9ab9598e5346/html5/thumbnails/17.jpg)
LREC, May 2016
Feature-based Classification
![Page 18: The Language Resource Spectrum: A Perspective from …lrec2016.lrec-conf.org/media/filer_public/2017/01/27/ryanmcdonaldl... · The Language Resource Spectrum: A Perspective from Google](https://reader030.vdocuments.us/reader030/viewer/2022013006/5b02c5237f8b9ab9598e5346/html5/thumbnails/18.jpg)
LREC, May 2016
Feature-based Classification
word=entendrez suffix3=rez word-1=n word+1=jamais cluster=124 cluster-1=53 cluster+1=210
![Page 19: The Language Resource Spectrum: A Perspective from …lrec2016.lrec-conf.org/media/filer_public/2017/01/27/ryanmcdonaldl... · The Language Resource Spectrum: A Perspective from Google](https://reader030.vdocuments.us/reader030/viewer/2022013006/5b02c5237f8b9ab9598e5346/html5/thumbnails/19.jpg)
LREC, May 2016
Resource Trade-Off
Annotated data
Model
![Page 20: The Language Resource Spectrum: A Perspective from …lrec2016.lrec-conf.org/media/filer_public/2017/01/27/ryanmcdonaldl... · The Language Resource Spectrum: A Perspective from Google](https://reader030.vdocuments.us/reader030/viewer/2022013006/5b02c5237f8b9ab9598e5346/html5/thumbnails/20.jpg)
LREC, May 2016
Resource Trade-Off
Annotated data
ModelDictionaries / Lexicons
![Page 21: The Language Resource Spectrum: A Perspective from …lrec2016.lrec-conf.org/media/filer_public/2017/01/27/ryanmcdonaldl... · The Language Resource Spectrum: A Perspective from Google](https://reader030.vdocuments.us/reader030/viewer/2022013006/5b02c5237f8b9ab9598e5346/html5/thumbnails/21.jpg)
LREC, May 2016
Resource Trade-Off
Annotated data
ModelDictionaries / Lexicons
![Page 22: The Language Resource Spectrum: A Perspective from …lrec2016.lrec-conf.org/media/filer_public/2017/01/27/ryanmcdonaldl... · The Language Resource Spectrum: A Perspective from Google](https://reader030.vdocuments.us/reader030/viewer/2022013006/5b02c5237f8b9ab9598e5346/html5/thumbnails/22.jpg)
LREC, May 2016
Morphosyntactic Lexicons via Graph-Propagation
![Page 23: The Language Resource Spectrum: A Perspective from …lrec2016.lrec-conf.org/media/filer_public/2017/01/27/ryanmcdonaldl... · The Language Resource Spectrum: A Perspective from Google](https://reader030.vdocuments.us/reader030/viewer/2022013006/5b02c5237f8b9ab9598e5346/html5/thumbnails/23.jpg)
LREC, May 2016
Morphosyntactic Lexicons via Graph-Propagation
![Page 24: The Language Resource Spectrum: A Perspective from …lrec2016.lrec-conf.org/media/filer_public/2017/01/27/ryanmcdonaldl... · The Language Resource Spectrum: A Perspective from Google](https://reader030.vdocuments.us/reader030/viewer/2022013006/5b02c5237f8b9ab9598e5346/html5/thumbnails/24.jpg)
LREC, May 2016
Morphosyntactic Lexicons via Graph-Propagation
![Page 25: The Language Resource Spectrum: A Perspective from …lrec2016.lrec-conf.org/media/filer_public/2017/01/27/ryanmcdonaldl... · The Language Resource Spectrum: A Perspective from Google](https://reader030.vdocuments.us/reader030/viewer/2022013006/5b02c5237f8b9ab9598e5346/html5/thumbnails/25.jpg)
LREC, May 2016
Morphosyntactic Lexicons via Graph-Propagation
![Page 26: The Language Resource Spectrum: A Perspective from …lrec2016.lrec-conf.org/media/filer_public/2017/01/27/ryanmcdonaldl... · The Language Resource Spectrum: A Perspective from Google](https://reader030.vdocuments.us/reader030/viewer/2022013006/5b02c5237f8b9ab9598e5346/html5/thumbnails/26.jpg)
LREC, May 2016
candies dish
dishes
candy
cat
cats
suff:ies:y pref
clust:10
suff:es: pref
clust:20
suff:s
suff:s: pref
Number=Plur
clust:35
Morphosyntactic Lexicons via Graph-Propagation
![Page 27: The Language Resource Spectrum: A Perspective from …lrec2016.lrec-conf.org/media/filer_public/2017/01/27/ryanmcdonaldl... · The Language Resource Spectrum: A Perspective from Google](https://reader030.vdocuments.us/reader030/viewer/2022013006/5b02c5237f8b9ab9598e5346/html5/thumbnails/27.jpg)
LREC, May 2016
candies dish
dishes
candy
cat
cats
suff:ies:y pref
clust:10
suff:es: pref
clust:20
suff:s
suff:s: pref
Number=Plur
clust:35
+1 -1
Morphosyntactic Lexicons via Graph-Propagation
![Page 28: The Language Resource Spectrum: A Perspective from …lrec2016.lrec-conf.org/media/filer_public/2017/01/27/ryanmcdonaldl... · The Language Resource Spectrum: A Perspective from Google](https://reader030.vdocuments.us/reader030/viewer/2022013006/5b02c5237f8b9ab9598e5346/html5/thumbnails/28.jpg)
LREC, May 2016
candies dish
dishes
candy
cat
cats
suff:ies:y pref
clust:10
suff:es: pref
clust:20
suff:s
suff:s: pref
Number=Plur
clust:35
Reinforce (1) suff:s clust:35 clust:20 Flip (-1) suff:ies:y suff:es: suff:s: Neutral (0) pref clust:10
-1
-1
1
11
0
-1
+1 -1
Morphosyntactic Lexicons via Graph-Propagation
![Page 29: The Language Resource Spectrum: A Perspective from …lrec2016.lrec-conf.org/media/filer_public/2017/01/27/ryanmcdonaldl... · The Language Resource Spectrum: A Perspective from Google](https://reader030.vdocuments.us/reader030/viewer/2022013006/5b02c5237f8b9ab9598e5346/html5/thumbnails/29.jpg)
LREC, May 2016
candies dish
dishes
candy
cat
cats
suff:ies:y pref
clust:10
suff:es: pref
clust:20
suff:s
suff:s: pref
Number=Plur
clust:35
Reinforce (1) suff:s clust:35 clust:20 Flip (-1) suff:ies:y suff:es: suff:s: Neutral (0) pref clust:10
-1
-1
1
11
0
-1
+1 -1-1
-1
+1
+1
Morphosyntactic Lexicons via Graph-Propagation
![Page 30: The Language Resource Spectrum: A Perspective from …lrec2016.lrec-conf.org/media/filer_public/2017/01/27/ryanmcdonaldl... · The Language Resource Spectrum: A Perspective from Google](https://reader030.vdocuments.us/reader030/viewer/2022013006/5b02c5237f8b9ab9598e5346/html5/thumbnails/30.jpg)
LREC, May 2016
candies dish
dishes
candy
cat
cats
suff:ies:y pref
clust:10
suff:es: pref
clust:20
suff:s
suff:s: pref
Number=Plur
clust:35
Reinforce (1) suff:s clust:35 clust:20 Flip (-1) suff:ies:y suff:es: suff:s: Neutral (0) pref clust:10
-1
-1
1
11
0
-1
+1 -1-1
-1
+1
+1
Faruqui et al ’16:Ising Mean Field Approximation
Morphosyntactic Lexicons via Graph-Propagation
![Page 31: The Language Resource Spectrum: A Perspective from …lrec2016.lrec-conf.org/media/filer_public/2017/01/27/ryanmcdonaldl... · The Language Resource Spectrum: A Perspective from Google](https://reader030.vdocuments.us/reader030/viewer/2022013006/5b02c5237f8b9ab9598e5346/html5/thumbnails/31.jpg)
LREC, May 2016
Universal Lexicons
✤ Seed with Universal Dependencies (Nivre et al. ’16)
John saw Mary VERB
Tense=Past The saw broke
NOUN Number=Sing
saw
NOUN:Number=Sing VERB:Tense=Past
![Page 32: The Language Resource Spectrum: A Perspective from …lrec2016.lrec-conf.org/media/filer_public/2017/01/27/ryanmcdonaldl... · The Language Resource Spectrum: A Perspective from Google](https://reader030.vdocuments.us/reader030/viewer/2022013006/5b02c5237f8b9ab9598e5346/html5/thumbnails/32.jpg)
LREC, May 2016
Resource Trade-off
90
92
94
96
98
Cs Fi Hu
Baseline* +Auto Lexicon +Gold + Gold + Auto + 100% data
91
91.7
95.5
![Page 33: The Language Resource Spectrum: A Perspective from …lrec2016.lrec-conf.org/media/filer_public/2017/01/27/ryanmcdonaldl... · The Language Resource Spectrum: A Perspective from Google](https://reader030.vdocuments.us/reader030/viewer/2022013006/5b02c5237f8b9ab9598e5346/html5/thumbnails/33.jpg)
LREC, May 2016
Resource Trade-off
90
92
94
96
98
Cs Fi Hu
Baseline* +Auto Lexicon +Gold + Gold + Auto + 100% data
93.293.6
96.8
91
91.7
95.5
![Page 34: The Language Resource Spectrum: A Perspective from …lrec2016.lrec-conf.org/media/filer_public/2017/01/27/ryanmcdonaldl... · The Language Resource Spectrum: A Perspective from Google](https://reader030.vdocuments.us/reader030/viewer/2022013006/5b02c5237f8b9ab9598e5346/html5/thumbnails/34.jpg)
LREC, May 2016
Resource Trade-off
90
92
94
96
98
Cs Fi Hu
Baseline* +Auto Lexicon +Gold + Gold + Auto + 100% data
95.195.2
97
93.293.6
96.8
91
91.7
95.5
![Page 35: The Language Resource Spectrum: A Perspective from …lrec2016.lrec-conf.org/media/filer_public/2017/01/27/ryanmcdonaldl... · The Language Resource Spectrum: A Perspective from Google](https://reader030.vdocuments.us/reader030/viewer/2022013006/5b02c5237f8b9ab9598e5346/html5/thumbnails/35.jpg)
LREC, May 2016
Resource Trade-off
90
92
94
96
98
Cs Fi Hu
Baseline* +Auto Lexicon +Gold + Gold + Auto + 100% data
95.895.7
97.4
95.195.2
97
93.293.6
96.8
91
91.7
95.5
![Page 36: The Language Resource Spectrum: A Perspective from …lrec2016.lrec-conf.org/media/filer_public/2017/01/27/ryanmcdonaldl... · The Language Resource Spectrum: A Perspective from Google](https://reader030.vdocuments.us/reader030/viewer/2022013006/5b02c5237f8b9ab9598e5346/html5/thumbnails/36.jpg)
LREC, May 2016
Resource Trade-off
90
92
94
96
98
Cs Fi Hu
Baseline* +Auto Lexicon +Gold + Gold + Auto + 100% data
91.7
92.8
96.8
95.895.7
97.4
95.195.2
97
93.293.6
96.8
91
91.7
95.5
![Page 37: The Language Resource Spectrum: A Perspective from …lrec2016.lrec-conf.org/media/filer_public/2017/01/27/ryanmcdonaldl... · The Language Resource Spectrum: A Perspective from Google](https://reader030.vdocuments.us/reader030/viewer/2022013006/5b02c5237f8b9ab9598e5346/html5/thumbnails/37.jpg)
LREC, May 2016
Part-Of-Speech Tagging: Queries
![Page 38: The Language Resource Spectrum: A Perspective from …lrec2016.lrec-conf.org/media/filer_public/2017/01/27/ryanmcdonaldl... · The Language Resource Spectrum: A Perspective from Google](https://reader030.vdocuments.us/reader030/viewer/2022013006/5b02c5237f8b9ab9598e5346/html5/thumbnails/38.jpg)
LREC, May 2016
Search Logs
![Page 39: The Language Resource Spectrum: A Perspective from …lrec2016.lrec-conf.org/media/filer_public/2017/01/27/ryanmcdonaldl... · The Language Resource Spectrum: A Perspective from Google](https://reader030.vdocuments.us/reader030/viewer/2022013006/5b02c5237f8b9ab9598e5346/html5/thumbnails/39.jpg)
LREC, May 2016
Search Logs
Click
![Page 40: The Language Resource Spectrum: A Perspective from …lrec2016.lrec-conf.org/media/filer_public/2017/01/27/ryanmcdonaldl... · The Language Resource Spectrum: A Perspective from Google](https://reader030.vdocuments.us/reader030/viewer/2022013006/5b02c5237f8b9ab9598e5346/html5/thumbnails/40.jpg)
LREC, May 2016
POS Taggers From Clicks Ganchev et al. (2012)
70
76.25
82.5
88.75
95
MS-251-NVX MS-251 Long Tail
Baseline +Click
74.3
81.9
92.8
![Page 41: The Language Resource Spectrum: A Perspective from …lrec2016.lrec-conf.org/media/filer_public/2017/01/27/ryanmcdonaldl... · The Language Resource Spectrum: A Perspective from Google](https://reader030.vdocuments.us/reader030/viewer/2022013006/5b02c5237f8b9ab9598e5346/html5/thumbnails/41.jpg)
LREC, May 2016
POS Taggers From Clicks Ganchev et al. (2012)
70
76.25
82.5
88.75
95
MS-251-NVX MS-251 Long Tail
Baseline +Click
77.5
84.5
93.5
74.3
81.9
92.8
![Page 42: The Language Resource Spectrum: A Perspective from …lrec2016.lrec-conf.org/media/filer_public/2017/01/27/ryanmcdonaldl... · The Language Resource Spectrum: A Perspective from Google](https://reader030.vdocuments.us/reader030/viewer/2022013006/5b02c5237f8b9ab9598e5346/html5/thumbnails/42.jpg)
LREC, May 2016
Morphosyntax Conclusions
✤ Money on more supervised data not necessarily optimal
✤ Better alternative: lexical resources (auto, manual & both)
✤ Better alternative: correlate usage statistics (click logs)
![Page 43: The Language Resource Spectrum: A Perspective from …lrec2016.lrec-conf.org/media/filer_public/2017/01/27/ryanmcdonaldl... · The Language Resource Spectrum: A Perspective from Google](https://reader030.vdocuments.us/reader030/viewer/2022013006/5b02c5237f8b9ab9598e5346/html5/thumbnails/43.jpg)
End User: Machine Translation
![Page 44: The Language Resource Spectrum: A Perspective from …lrec2016.lrec-conf.org/media/filer_public/2017/01/27/ryanmcdonaldl... · The Language Resource Spectrum: A Perspective from Google](https://reader030.vdocuments.us/reader030/viewer/2022013006/5b02c5237f8b9ab9598e5346/html5/thumbnails/44.jpg)
LREC, May 2016
Pipelined Machine Translation
John saw Mary
nsubj dobj
John Mary saw
nsubj dobj
Preprocess Reorder
Translate
Preprocess Reorder Translate Postorder Postprocess
![Page 45: The Language Resource Spectrum: A Perspective from …lrec2016.lrec-conf.org/media/filer_public/2017/01/27/ryanmcdonaldl... · The Language Resource Spectrum: A Perspective from Google](https://reader030.vdocuments.us/reader030/viewer/2022013006/5b02c5237f8b9ab9598e5346/html5/thumbnails/45.jpg)
LREC, May 2016
Pipelined Machine Translation
John saw Mary
nsubj dobj
John Mary saw
nsubj dobj
Preprocess Reorder
Translate
Preprocess Reorder Translate Postorder Postprocess
![Page 46: The Language Resource Spectrum: A Perspective from …lrec2016.lrec-conf.org/media/filer_public/2017/01/27/ryanmcdonaldl... · The Language Resource Spectrum: A Perspective from Google](https://reader030.vdocuments.us/reader030/viewer/2022013006/5b02c5237f8b9ab9598e5346/html5/thumbnails/46.jpg)
LREC, May 2016
Reordering Data VS. BL
EU Δ
0
1
2
3
4
5
En->Ar En->Iw En->Ja
3.7
1.41.2
3.5
1.51
Human Auto
Syntax-based reordered (Lerner & Petrov ’13)
![Page 47: The Language Resource Spectrum: A Perspective from …lrec2016.lrec-conf.org/media/filer_public/2017/01/27/ryanmcdonaldl... · The Language Resource Spectrum: A Perspective from Google](https://reader030.vdocuments.us/reader030/viewer/2022013006/5b02c5237f8b9ab9598e5346/html5/thumbnails/47.jpg)
LREC, May 2016
Reordering Data VS. BL
EU Δ
0
1
2
3
4
5
En->Ar En->Iw En->Ja
3.7
1.41.2
3.5
1.51
Human Auto
Syntax-based reordered (Lerner & Petrov ’13)
BLEU
Δ
0
1
2
3
4
5
En->Ja
4.8
3.73.5
Human AutoRule-based
![Page 48: The Language Resource Spectrum: A Perspective from …lrec2016.lrec-conf.org/media/filer_public/2017/01/27/ryanmcdonaldl... · The Language Resource Spectrum: A Perspective from Google](https://reader030.vdocuments.us/reader030/viewer/2022013006/5b02c5237f8b9ab9598e5346/html5/thumbnails/48.jpg)
LREC, May 2016
Better ParsersM
T Q
UA
LIT
Y
72
74.5
77
79.5
82
Time
Structured Training
Greedy transition-based
SSL
Better features
![Page 49: The Language Resource Spectrum: A Perspective from …lrec2016.lrec-conf.org/media/filer_public/2017/01/27/ryanmcdonaldl... · The Language Resource Spectrum: A Perspective from Google](https://reader030.vdocuments.us/reader030/viewer/2022013006/5b02c5237f8b9ab9598e5346/html5/thumbnails/49.jpg)
LREC, May 2016
Better ParsersM
T Q
UA
LIT
Y
72
74.5
77
79.5
82
Time
Structured Training
Greedy transition-based
SSL
1pt improvement is significant to humans
Better features
![Page 50: The Language Resource Spectrum: A Perspective from …lrec2016.lrec-conf.org/media/filer_public/2017/01/27/ryanmcdonaldl... · The Language Resource Spectrum: A Perspective from Google](https://reader030.vdocuments.us/reader030/viewer/2022013006/5b02c5237f8b9ab9598e5346/html5/thumbnails/50.jpg)
LREC, May 2016
More Data
EN->
JA R
eord
erin
g Sc
ore
75
76.5
78
1x 2x 10x
Syntax Reordering
Katz-Brown et al; Hall et al. 2011
![Page 51: The Language Resource Spectrum: A Perspective from …lrec2016.lrec-conf.org/media/filer_public/2017/01/27/ryanmcdonaldl... · The Language Resource Spectrum: A Perspective from Google](https://reader030.vdocuments.us/reader030/viewer/2022013006/5b02c5237f8b9ab9598e5346/html5/thumbnails/51.jpg)
LREC, May 2016
Machine Translation
✤ Human vs. auto data: about the same
✤ Human models sometimes better than learned
✤ Better parsing models = better translation
✤ Better to spend on targeted resources — reordering
vs.
![Page 52: The Language Resource Spectrum: A Perspective from …lrec2016.lrec-conf.org/media/filer_public/2017/01/27/ryanmcdonaldl... · The Language Resource Spectrum: A Perspective from Google](https://reader030.vdocuments.us/reader030/viewer/2022013006/5b02c5237f8b9ab9598e5346/html5/thumbnails/52.jpg)
End User: Sentence Compression
![Page 53: The Language Resource Spectrum: A Perspective from …lrec2016.lrec-conf.org/media/filer_public/2017/01/27/ryanmcdonaldl... · The Language Resource Spectrum: A Perspective from Google](https://reader030.vdocuments.us/reader030/viewer/2022013006/5b02c5237f8b9ab9598e5346/html5/thumbnails/53.jpg)
LREC, May 2016
Sentence Compression @Google
Former Los Angeles Lakers head coach Phil Jackson won eleven
NBA championships. He won six titles with the Chicago Bulls and
five titles with the Lakers.
![Page 54: The Language Resource Spectrum: A Perspective from …lrec2016.lrec-conf.org/media/filer_public/2017/01/27/ryanmcdonaldl... · The Language Resource Spectrum: A Perspective from Google](https://reader030.vdocuments.us/reader030/viewer/2022013006/5b02c5237f8b9ab9598e5346/html5/thumbnails/54.jpg)
LREC, May 2016
Google News
Headline
First sentence
Filippova & Altun ’13✤ Can extract millions of pairs✤ Quality ~= expert annotations✤ 81.4 -> 84.3 F1 (10% -> 100% data)
![Page 55: The Language Resource Spectrum: A Perspective from …lrec2016.lrec-conf.org/media/filer_public/2017/01/27/ryanmcdonaldl... · The Language Resource Spectrum: A Perspective from Google](https://reader030.vdocuments.us/reader030/viewer/2022013006/5b02c5237f8b9ab9598e5346/html5/thumbnails/55.jpg)
LREC, May 2016
Need For High Quality Annotations?
?
![Page 56: The Language Resource Spectrum: A Perspective from …lrec2016.lrec-conf.org/media/filer_public/2017/01/27/ryanmcdonaldl... · The Language Resource Spectrum: A Perspective from Google](https://reader030.vdocuments.us/reader030/viewer/2022013006/5b02c5237f8b9ab9598e5346/html5/thumbnails/56.jpg)
LREC, May 2016
Need For High Quality Annotations?
?Filippova et al. ’15: LSTM compression
by deletion 1/0
![Page 57: The Language Resource Spectrum: A Perspective from …lrec2016.lrec-conf.org/media/filer_public/2017/01/27/ryanmcdonaldl... · The Language Resource Spectrum: A Perspective from Google](https://reader030.vdocuments.us/reader030/viewer/2022013006/5b02c5237f8b9ab9598e5346/html5/thumbnails/57.jpg)
LREC, May 2016
Need For High Quality Annotations?
?Filippova et al. ’15: LSTM compression
by deletion 1/0
[ ] [ ] [ ] [ ]word
Accuracy
30
![Page 58: The Language Resource Spectrum: A Perspective from …lrec2016.lrec-conf.org/media/filer_public/2017/01/27/ryanmcdonaldl... · The Language Resource Spectrum: A Perspective from Google](https://reader030.vdocuments.us/reader030/viewer/2022013006/5b02c5237f8b9ab9598e5346/html5/thumbnails/58.jpg)
LREC, May 2016
Need For High Quality Annotations?
?Filippova et al. ’15: LSTM compression
by deletion 1/0
[ ] [ ] [ ] [ ]word[ ] [ ] [ ] [ ]parent
Accuracy
31
30
![Page 59: The Language Resource Spectrum: A Perspective from …lrec2016.lrec-conf.org/media/filer_public/2017/01/27/ryanmcdonaldl... · The Language Resource Spectrum: A Perspective from Google](https://reader030.vdocuments.us/reader030/viewer/2022013006/5b02c5237f8b9ab9598e5346/html5/thumbnails/59.jpg)
LREC, May 2016
Need For High Quality Annotations?
?Filippova et al. ’15: LSTM compression
by deletion 1/0
[ ] [ ] [ ] [ ]word[ ] [ ] [ ] [ ]parent[ ] [ ] [ ] [ ]syn struct
Accuracy
34
31
30
![Page 60: The Language Resource Spectrum: A Perspective from …lrec2016.lrec-conf.org/media/filer_public/2017/01/27/ryanmcdonaldl... · The Language Resource Spectrum: A Perspective from Google](https://reader030.vdocuments.us/reader030/viewer/2022013006/5b02c5237f8b9ab9598e5346/html5/thumbnails/60.jpg)
LREC, May 2016
The Resource Trade-off
High quality annotations Crowd-sourced Auto resources
Data + modelData + model
![Page 61: The Language Resource Spectrum: A Perspective from …lrec2016.lrec-conf.org/media/filer_public/2017/01/27/ryanmcdonaldl... · The Language Resource Spectrum: A Perspective from Google](https://reader030.vdocuments.us/reader030/viewer/2022013006/5b02c5237f8b9ab9598e5346/html5/thumbnails/61.jpg)
End User: QA & Knowledge Extraction
![Page 62: The Language Resource Spectrum: A Perspective from …lrec2016.lrec-conf.org/media/filer_public/2017/01/27/ryanmcdonaldl... · The Language Resource Spectrum: A Perspective from Google](https://reader030.vdocuments.us/reader030/viewer/2022013006/5b02c5237f8b9ab9598e5346/html5/thumbnails/62.jpg)
LREC, May 2016
QA & Knowledge Extraction
![Page 63: The Language Resource Spectrum: A Perspective from …lrec2016.lrec-conf.org/media/filer_public/2017/01/27/ryanmcdonaldl... · The Language Resource Spectrum: A Perspective from Google](https://reader030.vdocuments.us/reader030/viewer/2022013006/5b02c5237f8b9ab9598e5346/html5/thumbnails/63.jpg)
LREC, May 2016
QA & Knowledge Extraction
as of 2014
![Page 64: The Language Resource Spectrum: A Perspective from …lrec2016.lrec-conf.org/media/filer_public/2017/01/27/ryanmcdonaldl... · The Language Resource Spectrum: A Perspective from Google](https://reader030.vdocuments.us/reader030/viewer/2022013006/5b02c5237f8b9ab9598e5346/html5/thumbnails/64.jpg)
LREC, May 2016
Weakly Sup. Knowledge Extraction (West et al. 2014)
R=parents
parent of __ __’s parent __ father
mother of __ …
Extract relation templates/queries
search logs
QA system
![Page 65: The Language Resource Spectrum: A Perspective from …lrec2016.lrec-conf.org/media/filer_public/2017/01/27/ryanmcdonaldl... · The Language Resource Spectrum: A Perspective from Google](https://reader030.vdocuments.us/reader030/viewer/2022013006/5b02c5237f8b9ab9598e5346/html5/thumbnails/65.jpg)
LREC, May 2016
Weakly Sup. Knowledge Extraction (West et al. 2014)
R=parents
parent of __ __’s parent __ father
mother of __ …
Extract relation templates/queries
search logs
QA system
Q=Frank Zappa
parent of Frank Zappa Frank Zappa’s parent Frank Zappa father
mother of Frank Zappa …
Mothers of Inversion Ray Collins
Rose Marie Colimore Francis Zappa
Gail Zappa Rose Marie
Score entities in result snippets & aggregateIssue queries
QA system
![Page 66: The Language Resource Spectrum: A Perspective from …lrec2016.lrec-conf.org/media/filer_public/2017/01/27/ryanmcdonaldl... · The Language Resource Spectrum: A Perspective from Google](https://reader030.vdocuments.us/reader030/viewer/2022013006/5b02c5237f8b9ab9598e5346/html5/thumbnails/66.jpg)
LREC, May 2016
Does it Work?
![Page 67: The Language Resource Spectrum: A Perspective from …lrec2016.lrec-conf.org/media/filer_public/2017/01/27/ryanmcdonaldl... · The Language Resource Spectrum: A Perspective from Google](https://reader030.vdocuments.us/reader030/viewer/2022013006/5b02c5237f8b9ab9598e5346/html5/thumbnails/67.jpg)
LREC, May 2016
Does it Work?
![Page 68: The Language Resource Spectrum: A Perspective from …lrec2016.lrec-conf.org/media/filer_public/2017/01/27/ryanmcdonaldl... · The Language Resource Spectrum: A Perspective from Google](https://reader030.vdocuments.us/reader030/viewer/2022013006/5b02c5237f8b9ab9598e5346/html5/thumbnails/68.jpg)
LREC, May 2016
The Resource Trade-off
High quality annotations Crowd-sourced Auto resources
![Page 69: The Language Resource Spectrum: A Perspective from …lrec2016.lrec-conf.org/media/filer_public/2017/01/27/ryanmcdonaldl... · The Language Resource Spectrum: A Perspective from Google](https://reader030.vdocuments.us/reader030/viewer/2022013006/5b02c5237f8b9ab9598e5346/html5/thumbnails/69.jpg)
LREC, May 2016
The Resource Trade-off
High quality annotations Crowd-sourced Auto resources?
![Page 70: The Language Resource Spectrum: A Perspective from …lrec2016.lrec-conf.org/media/filer_public/2017/01/27/ryanmcdonaldl... · The Language Resource Spectrum: A Perspective from Google](https://reader030.vdocuments.us/reader030/viewer/2022013006/5b02c5237f8b9ab9598e5346/html5/thumbnails/70.jpg)
LREC, May 2016
Top-level Conclusion
Syntax
Semantics
![Page 71: The Language Resource Spectrum: A Perspective from …lrec2016.lrec-conf.org/media/filer_public/2017/01/27/ryanmcdonaldl... · The Language Resource Spectrum: A Perspective from Google](https://reader030.vdocuments.us/reader030/viewer/2022013006/5b02c5237f8b9ab9598e5346/html5/thumbnails/71.jpg)
Thanks