the language resource spectrum: a perspective from...

Ryan McDonald

Google NLU team Google Linguistics team

The Language Resource Spectrum: A Perspective from Google

Ryan McDonald

Google NLU team Google Linguistics team

The Language Resource Spectrum: A Perspective from Google

LREC, May 2016

Language Resource Spectrum

search logs

LREC, May 2016

Language Resource Spectrum

search logs

unsu

pervis

ed

weakly

super

vised

fully

super

vised

LREC, May 2016

amount of

data

supervision

LREC, May 2016

amount of

data

supervision

LREC, May 2016

amount of

data

supervision

LREC, May 2016

amount of

data

supervision

which is better?

LREC, May 2016

High quality annotations Crowd-sourced

Software Engineer Auto resources

Models Active Learning

Pre-existing resources

LREC, May 2016

High quality annotations Crowd-sourced

Software Engineer Auto resources

Models Active Learning

Pre-existing resources

α β γ δ

LREC, May 2016

High quality annotations Crowd-sourced

Software Engineer Auto resources

Models Active Learning

Pre-existing resources

α β γ δ

LREC, May 2016

The Task Matters

✤ ML is really good at the head

POS tagging

QA

LREC, May 2016

Pipelined / multi-Component Systems

Indexer Text Extractor

Segmentation Morphosyntax

Shallow Semantics MT QA

SearchEntity resolution Relation extraction

LREC, May 2016

Pipelined / multi-Component Systems

Indexer Text Extractor

Segmentation Morphosyntax

Shallow Semantics MT QA

SearchEntity resolution Relation extraction

End User Task

LREC, May 2016

Pipelined / multi-Component Systems

Indexer Text Extractor

Segmentation Morphosyntax

Shallow Semantics MT QA

SearchEntity resolution Relation extraction

End User Task

Upstream Task

Upstream: Morphosyntactic Tagging

LREC, May 2016

Feature-based Classification

LREC, May 2016

Feature-based Classification

word=entendrez suffix3=rez word-1=n word+1=jamais cluster=124 cluster-1=53 cluster+1=210

LREC, May 2016

Resource Trade-Off

Annotated data

Model

LREC, May 2016

Resource Trade-Off

Annotated data

ModelDictionaries / Lexicons

LREC, May 2016

Resource Trade-Off

Annotated data

ModelDictionaries / Lexicons

LREC, May 2016

Morphosyntactic Lexicons via Graph-Propagation

LREC, May 2016

Morphosyntactic Lexicons via Graph-Propagation

LREC, May 2016

Morphosyntactic Lexicons via Graph-Propagation

LREC, May 2016

Morphosyntactic Lexicons via Graph-Propagation

LREC, May 2016

candies dish

dishes

candy

cat

cats

suff:ies:y pref

clust:10

suff:es: pref

clust:20

suff:s

suff:s: pref

Number=Plur

clust:35

Morphosyntactic Lexicons via Graph-Propagation

LREC, May 2016

candies dish

dishes

candy

cat

cats

suff:ies:y pref

clust:10

suff:es: pref

clust:20

suff:s

suff:s: pref

Number=Plur

clust:35

+1 -1

Morphosyntactic Lexicons via Graph-Propagation

LREC, May 2016

candies dish

dishes

candy

cat

cats

suff:ies:y pref

clust:10

suff:es: pref

clust:20

suff:s

suff:s: pref

Number=Plur

clust:35

Reinforce (1) suff:s clust:35 clust:20 Flip (-1) suff:ies:y suff:es: suff:s: Neutral (0) pref clust:10

-1

-1

1

11

0

-1

+1 -1

Morphosyntactic Lexicons via Graph-Propagation

LREC, May 2016

candies dish

dishes

candy

cat

cats

suff:ies:y pref

clust:10

suff:es: pref

clust:20

suff:s

suff:s: pref

Number=Plur

clust:35

Reinforce (1) suff:s clust:35 clust:20 Flip (-1) suff:ies:y suff:es: suff:s: Neutral (0) pref clust:10

-1

-1

1

11

0

-1

+1 -1-1

-1

+1

+1

Morphosyntactic Lexicons via Graph-Propagation

LREC, May 2016

candies dish

dishes

candy

cat

cats

suff:ies:y pref

clust:10

suff:es: pref

clust:20

suff:s

suff:s: pref

Number=Plur

clust:35

Reinforce (1) suff:s clust:35 clust:20 Flip (-1) suff:ies:y suff:es: suff:s: Neutral (0) pref clust:10

-1

-1

1

11

0

-1

+1 -1-1

-1

+1

+1

Faruqui et al ’16:Ising Mean Field Approximation

Morphosyntactic Lexicons via Graph-Propagation

LREC, May 2016

Universal Lexicons

✤ Seed with Universal Dependencies (Nivre et al. ’16)

John saw Mary VERB

Tense=Past The saw broke

NOUN Number=Sing

saw

NOUN:Number=Sing VERB:Tense=Past

LREC, May 2016

Resource Trade-off

90

92

94

96

98

Cs Fi Hu

Baseline* +Auto Lexicon +Gold + Gold + Auto + 100% data

91

91.7

95.5

LREC, May 2016

Resource Trade-off

90

92

94

96

98

Cs Fi Hu

Baseline* +Auto Lexicon +Gold + Gold + Auto + 100% data

93.293.6

96.8

91

91.7

95.5

LREC, May 2016

Resource Trade-off

90

92

94

96

98

Cs Fi Hu

Baseline* +Auto Lexicon +Gold + Gold + Auto + 100% data

95.195.2

97

93.293.6

96.8

91

91.7

95.5

LREC, May 2016

Resource Trade-off

90

92

94

96

98

Cs Fi Hu

Baseline* +Auto Lexicon +Gold + Gold + Auto + 100% data

95.895.7

97.4

95.195.2

97

93.293.6

96.8

91

91.7

95.5

LREC, May 2016

Resource Trade-off

90

92

94

96

98

Cs Fi Hu

Baseline* +Auto Lexicon +Gold + Gold + Auto + 100% data

91.7

92.8

96.8

95.895.7

97.4

95.195.2

97

93.293.6

96.8

91

91.7

95.5

LREC, May 2016

Part-Of-Speech Tagging: Queries

LREC, May 2016

Search Logs

LREC, May 2016

Search Logs

Click

LREC, May 2016

POS Taggers From Clicks Ganchev et al. (2012)

70

76.25

82.5

88.75

95

MS-251-NVX MS-251 Long Tail

Baseline +Click

74.3

81.9

92.8

LREC, May 2016

POS Taggers From Clicks Ganchev et al. (2012)

70

76.25

82.5

88.75

95

MS-251-NVX MS-251 Long Tail

Baseline +Click

77.5

84.5

93.5

74.3

81.9

92.8

LREC, May 2016

Morphosyntax Conclusions

✤ Money on more supervised data not necessarily optimal

✤ Better alternative: lexical resources (auto, manual & both)

✤ Better alternative: correlate usage statistics (click logs)

End User: Machine Translation

LREC, May 2016

Pipelined Machine Translation

John saw Mary

nsubj dobj

John Mary saw

nsubj dobj

Preprocess Reorder

Translate

Preprocess Reorder Translate Postorder Postprocess

LREC, May 2016

Pipelined Machine Translation

John saw Mary

nsubj dobj

John Mary saw

nsubj dobj

Preprocess Reorder

Translate

Preprocess Reorder Translate Postorder Postprocess

LREC, May 2016

Reordering Data VS. BL

EU Δ

0

1

2

3

4

5

En->Ar En->Iw En->Ja

3.7

1.41.2

3.5

1.51

Human Auto

Syntax-based reordered (Lerner & Petrov ’13)

LREC, May 2016

Reordering Data VS. BL

EU Δ

0

1

2

3

4

5

En->Ar En->Iw En->Ja

3.7

1.41.2

3.5

1.51

Human Auto

Syntax-based reordered (Lerner & Petrov ’13)

BLEU

Δ

0

1

2

3

4

5

En->Ja

4.8

3.73.5

Human AutoRule-based

LREC, May 2016

Better ParsersM

T Q

UA

LIT

Y

72

74.5

77

79.5

82

Time

Structured Training

Greedy transition-based

SSL

Better features

LREC, May 2016

Better ParsersM

T Q

UA

LIT

Y

72

74.5

77

79.5

82

Time

Structured Training

Greedy transition-based

SSL

1pt improvement is significant to humans

Better features

LREC, May 2016

More Data

EN->

JA R

eord

erin

g Sc

ore

75

76.5

78

1x 2x 10x

Syntax Reordering

Katz-Brown et al; Hall et al. 2011

LREC, May 2016

Machine Translation

✤ Human vs. auto data: about the same

✤ Human models sometimes better than learned

✤ Better parsing models = better translation

✤ Better to spend on targeted resources — reordering

vs.

End User: Sentence Compression

LREC, May 2016

Sentence Compression @Google

Former Los Angeles Lakers head coach Phil Jackson won eleven

NBA championships. He won six titles with the Chicago Bulls and

five titles with the Lakers.

LREC, May 2016

Google News

Headline

First sentence

Filippova & Altun ’13✤ Can extract millions of pairs✤ Quality ~= expert annotations✤ 81.4 -> 84.3 F1 (10% -> 100% data)

LREC, May 2016

Need For High Quality Annotations?

?

LREC, May 2016

Need For High Quality Annotations?

?Filippova et al. ’15: LSTM compression

by deletion 1/0

LREC, May 2016

Need For High Quality Annotations?

?Filippova et al. ’15: LSTM compression

by deletion 1/0

[ ] [ ] [ ] [ ]word

Accuracy

30

LREC, May 2016

Need For High Quality Annotations?

?Filippova et al. ’15: LSTM compression

by deletion 1/0

[ ] [ ] [ ] [ ]word[ ] [ ] [ ] [ ]parent

Accuracy

31

30

LREC, May 2016

Need For High Quality Annotations?

?Filippova et al. ’15: LSTM compression

by deletion 1/0

[ ] [ ] [ ] [ ]word[ ] [ ] [ ] [ ]parent[ ] [ ] [ ] [ ]syn struct

Accuracy

34

31

30

LREC, May 2016

The Resource Trade-off

High quality annotations Crowd-sourced Auto resources

Data + modelData + model

End User: QA & Knowledge Extraction

LREC, May 2016

QA & Knowledge Extraction

LREC, May 2016

QA & Knowledge Extraction

as of 2014

LREC, May 2016

Weakly Sup. Knowledge Extraction (West et al. 2014)

R=parents

parent of __ __’s parent __ father

mother of __ …

Extract relation templates/queries

search logs

QA system

LREC, May 2016

Weakly Sup. Knowledge Extraction (West et al. 2014)

R=parents

parent of __ __’s parent __ father

mother of __ …

Extract relation templates/queries

search logs

QA system

Q=Frank Zappa

parent of Frank Zappa Frank Zappa’s parent Frank Zappa father

mother of Frank Zappa …

Mothers of Inversion Ray Collins

Rose Marie Colimore Francis Zappa

Gail Zappa Rose Marie

Score entities in result snippets & aggregateIssue queries

QA system

LREC, May 2016

Does it Work?

LREC, May 2016

Does it Work?

LREC, May 2016

The Resource Trade-off

High quality annotations Crowd-sourced Auto resources

LREC, May 2016

The Resource Trade-off

High quality annotations Crowd-sourced Auto resources?

LREC, May 2016

Top-level Conclusion

Syntax

Semantics

Thanks