grounded language understanding: speakers: from the world
TRANSCRIPT
Task Training Prediction Design choices Monroe et al. (2016) Related tasks
Grounded language understanding:Speakers: From the world to language
Christopher Potts
Stanford Linguistics
CS224u: Natural language understanding
1 / 8
Task Training Prediction Design choices Monroe et al. (2016) Related tasks
Color describer: Task formulation and data
Color Utterance
xxxx greenxxxx purplexxxx grapexxxx turquoisexxxx moss greenxxxx pinkish purplexxxx light blue greyxxxx robin’s egg bluexxxx british racing greenxxxx baby puke green
2 / 8
McMahan and Stone 2015
Task Training Prediction Design choices Monroe et al. (2016) Related tasks
Color describer: Training with teacher forcing
DecoderEncoder
208.3, 60, 88.2
3 / 8
Task Training Prediction Design choices Monroe et al. (2016) Related tasks
Color describer: Training with teacher forcing
DecoderEncoder
208.3, 60, 88.2
color embedding
3 / 8
Task Training Prediction Design choices Monroe et al. (2016) Related tasks
Color describer: Training with teacher forcing
DecoderEncoder
208.3, 60, 88.2
color embedding
color rep
3 / 8
Task Training Prediction Design choices Monroe et al. (2016) Related tasks
Color describer: Training with teacher forcing
DecoderEncoder
208.3, 60, 88.2
color embedding
color rep
<s>
3 / 8
Task Training Prediction Design choices Monroe et al. (2016) Related tasks
Color describer: Training with teacher forcing
DecoderEncoder
208.3, 60, 88.2
color embedding
color rep
<s>
x1
3 / 8
Task Training Prediction Design choices Monroe et al. (2016) Related tasks
Color describer: Training with teacher forcing
DecoderEncoder
208.3, 60, 88.2
color embedding
color rep
<s>
x1
h1
3 / 8
Task Training Prediction Design choices Monroe et al. (2016) Related tasks
Color describer: Training with teacher forcing
DecoderEncoder
208.3, 60, 88.2
color embedding
color rep
<s>
x1
h1
dark
3 / 8
Task Training Prediction Design choices Monroe et al. (2016) Related tasks
Color describer: Training with teacher forcing
DecoderEncoder
208.3, 60, 88.2
color embedding
color rep
<s>
x1
h1
dark
light
error signal
3 / 8
Task Training Prediction Design choices Monroe et al. (2016) Related tasks
Color describer: Training with teacher forcing
DecoderEncoder
208.3, 60, 88.2
color embedding
color rep
<s>
x1
h1
dark
light
error signal
embedding
derived from x1 and color rep, the initial hidden state
predicted probability distribution over the vocab
one-hot encoding for next word
3 / 8
Task Training Prediction Design choices Monroe et al. (2016) Related tasks
Color describer: Training with teacher forcing
DecoderEncoder
208.3, 60, 88.2
color embedding
color rep
light<s>
x1
h1
dark
3 / 8
Task Training Prediction Design choices Monroe et al. (2016) Related tasks
Color describer: Training with teacher forcing
DecoderEncoder
208.3, 60, 88.2
color embedding
color rep
light<s>
x1 x37
h1
dark
3 / 8
Task Training Prediction Design choices Monroe et al. (2016) Related tasks
Color describer: Training with teacher forcing
DecoderEncoder
208.3, 60, 88.2
color embedding
color rep
light<s>
x1 x37
h1 h2
dark
3 / 8
Task Training Prediction Design choices Monroe et al. (2016) Related tasks
Color describer: Training with teacher forcing
DecoderEncoder
208.3, 60, 88.2
color embedding
color rep
light<s>
x1 x37
h1 h2
dark blue
3 / 8
Task Training Prediction Design choices Monroe et al. (2016) Related tasks
Color describer: Training with teacher forcing
DecoderEncoder
208.3, 60, 88.2
color embedding
color rep
light<s>
x1 x37
h1 h2
dark blue
blue
error signal
3 / 8
Task Training Prediction Design choices Monroe et al. (2016) Related tasks
Color describer: Training with teacher forcing
DecoderEncoder
208.3, 60, 88.2
color embedding
color rep
light<s> blue
x1 x37
h1 h2
dark blue
3 / 8
Task Training Prediction Design choices Monroe et al. (2016) Related tasks
Color describer: Training with teacher forcing
DecoderEncoder
208.3, 60, 88.2
color embedding
color rep
light<s> blue
x1 x37 x11
h1 h2
dark blue
3 / 8
Task Training Prediction Design choices Monroe et al. (2016) Related tasks
Color describer: Training with teacher forcing
DecoderEncoder
208.3, 60, 88.2
color embedding
color rep
light<s> blue
x1 x37 x11
h1 h2 h3
dark blue
3 / 8
Task Training Prediction Design choices Monroe et al. (2016) Related tasks
Color describer: Training with teacher forcing
DecoderEncoder
208.3, 60, 88.2
color embedding
color rep
light<s> blue
x1 x37 x11
h1 h2 h3
dark blue green
3 / 8
Task Training Prediction Design choices Monroe et al. (2016) Related tasks
Color describer: Training with teacher forcing
DecoderEncoder
208.3, 60, 88.2
color embedding
color rep
light<s> blue
x1 x37 x11
h1 h2 h3
dark blue green
</s>
error signal
3 / 8
Task Training Prediction Design choices Monroe et al. (2016) Related tasks
Color describer: Prediction
DecoderEncoder
208.3, 60, 88.2
color embedding
color rep
4 / 8
Task Training Prediction Design choices Monroe et al. (2016) Related tasks
Color describer: Prediction
DecoderEncoder
208.3, 60, 88.2
color embedding
color rep
<s>
4 / 8
Task Training Prediction Design choices Monroe et al. (2016) Related tasks
Color describer: Prediction
DecoderEncoder
208.3, 60, 88.2
color embedding
color rep
<s>
x1
4 / 8
Task Training Prediction Design choices Monroe et al. (2016) Related tasks
Color describer: Prediction
DecoderEncoder
208.3, 60, 88.2
color embedding
color rep
<s>
x1
h1
4 / 8
Task Training Prediction Design choices Monroe et al. (2016) Related tasks
Color describer: Prediction
DecoderEncoder
208.3, 60, 88.2
color embedding
color rep
<s>
x1
h1
dark
4 / 8
Task Training Prediction Design choices Monroe et al. (2016) Related tasks
Color describer: Prediction
DecoderEncoder
208.3, 60, 88.2
color embedding
color rep
dark<s>
x1 x20
h1 h2
dark
4 / 8
Task Training Prediction Design choices Monroe et al. (2016) Related tasks
Color describer: Prediction
DecoderEncoder
208.3, 60, 88.2
color embedding
color rep
dark<s>
x1 x20
h1 h2
dark blue
4 / 8
Task Training Prediction Design choices Monroe et al. (2016) Related tasks
Color describer: Prediction
DecoderEncoder
208.3, 60, 88.2
color embedding
color rep
dark<s> blue
x1 x20 x11
h1 h2 h3
dark blue
4 / 8
Task Training Prediction Design choices Monroe et al. (2016) Related tasks
Color describer: Prediction
DecoderEncoder
208.3, 60, 88.2
color embedding
color rep
dark<s> blue
x1 x20 x11
h1 h2 h3
dark blue </s>
4 / 8
Task Training Prediction Design choices Monroe et al. (2016) Related tasks
Miscellaneous design choices
• The Encoder and Decoder could have more hiddenlayers. We would expect the layer counts to match tofacilitate the hand-off between Encoder and Decoder,though pooling or copying might work too.
• It seems very common at present for researchers to tiethe embedding and classifier parameters (Press and Wolf2017)
• During training, one might drop teacher forcing a smallpercentage of the time to encourage the model toexplore.
5 / 8
Task Training Prediction Design choices Monroe et al. (2016) Related tasks
Color describer of Monroe et al. (2016)
DecoderEncoder
HSV
Fourier transform
color rep
light<s> blue
x1 x37 x11
h1 h2 h3
w2 w3 w4
6 / 8
Task Training Prediction Design choices Monroe et al. (2016) Related tasks
Related tasks
Non-linguistic representation ⇒ Language
• Image captioning
• Scene description
• Visual Question Answering(Image + Question-text ⇒ Answer-text)
• Instruction giving (State ⇒ Language)
• . . .
7 / 8
References
References I
Brian McMahan and Matthew Stone. 2015. A Bayesian model of grounded color semantics. Transactions of the Associationfor Computational Linguistics, 3:103–115.
Will Monroe, Noah D. Goodman, and Christopher Potts. 2016. Learning to generate compositional color descriptions. InProceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2243–2248,Stroudsburg, PA. Association for Computational Linguistics.
Ofir Press and Lior Wolf. 2017. Using the output embedding to improve language models. In Proceedings of the 15thConference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages157–163, Valencia, Spain. Association for Computational Linguistics.
8 / 8