![Page 1: Grounded language understanding: Speakers: From the world](https://reader035.vdocuments.us/reader035/viewer/2022081701/62e1516d2eafe528ea3ac98d/html5/thumbnails/1.jpg)
Task Training Prediction Design choices Monroe et al. (2016) Related tasks
Grounded language understanding:Speakers: From the world to language
Christopher Potts
Stanford Linguistics
CS224u: Natural language understanding
1 / 8
![Page 2: Grounded language understanding: Speakers: From the world](https://reader035.vdocuments.us/reader035/viewer/2022081701/62e1516d2eafe528ea3ac98d/html5/thumbnails/2.jpg)
Task Training Prediction Design choices Monroe et al. (2016) Related tasks
Color describer: Task formulation and data
Color Utterance
xxxx greenxxxx purplexxxx grapexxxx turquoisexxxx moss greenxxxx pinkish purplexxxx light blue greyxxxx robin’s egg bluexxxx british racing greenxxxx baby puke green
2 / 8
McMahan and Stone 2015
![Page 3: Grounded language understanding: Speakers: From the world](https://reader035.vdocuments.us/reader035/viewer/2022081701/62e1516d2eafe528ea3ac98d/html5/thumbnails/3.jpg)
Task Training Prediction Design choices Monroe et al. (2016) Related tasks
Color describer: Training with teacher forcing
DecoderEncoder
208.3, 60, 88.2
3 / 8
![Page 4: Grounded language understanding: Speakers: From the world](https://reader035.vdocuments.us/reader035/viewer/2022081701/62e1516d2eafe528ea3ac98d/html5/thumbnails/4.jpg)
Task Training Prediction Design choices Monroe et al. (2016) Related tasks
Color describer: Training with teacher forcing
DecoderEncoder
208.3, 60, 88.2
color embedding
3 / 8
![Page 5: Grounded language understanding: Speakers: From the world](https://reader035.vdocuments.us/reader035/viewer/2022081701/62e1516d2eafe528ea3ac98d/html5/thumbnails/5.jpg)
Task Training Prediction Design choices Monroe et al. (2016) Related tasks
Color describer: Training with teacher forcing
DecoderEncoder
208.3, 60, 88.2
color embedding
color rep
3 / 8
![Page 6: Grounded language understanding: Speakers: From the world](https://reader035.vdocuments.us/reader035/viewer/2022081701/62e1516d2eafe528ea3ac98d/html5/thumbnails/6.jpg)
Task Training Prediction Design choices Monroe et al. (2016) Related tasks
Color describer: Training with teacher forcing
DecoderEncoder
208.3, 60, 88.2
color embedding
color rep
<s>
3 / 8
![Page 7: Grounded language understanding: Speakers: From the world](https://reader035.vdocuments.us/reader035/viewer/2022081701/62e1516d2eafe528ea3ac98d/html5/thumbnails/7.jpg)
Task Training Prediction Design choices Monroe et al. (2016) Related tasks
Color describer: Training with teacher forcing
DecoderEncoder
208.3, 60, 88.2
color embedding
color rep
<s>
x1
3 / 8
![Page 8: Grounded language understanding: Speakers: From the world](https://reader035.vdocuments.us/reader035/viewer/2022081701/62e1516d2eafe528ea3ac98d/html5/thumbnails/8.jpg)
Task Training Prediction Design choices Monroe et al. (2016) Related tasks
Color describer: Training with teacher forcing
DecoderEncoder
208.3, 60, 88.2
color embedding
color rep
<s>
x1
h1
3 / 8
![Page 9: Grounded language understanding: Speakers: From the world](https://reader035.vdocuments.us/reader035/viewer/2022081701/62e1516d2eafe528ea3ac98d/html5/thumbnails/9.jpg)
Task Training Prediction Design choices Monroe et al. (2016) Related tasks
Color describer: Training with teacher forcing
DecoderEncoder
208.3, 60, 88.2
color embedding
color rep
<s>
x1
h1
dark
3 / 8
![Page 10: Grounded language understanding: Speakers: From the world](https://reader035.vdocuments.us/reader035/viewer/2022081701/62e1516d2eafe528ea3ac98d/html5/thumbnails/10.jpg)
Task Training Prediction Design choices Monroe et al. (2016) Related tasks
Color describer: Training with teacher forcing
DecoderEncoder
208.3, 60, 88.2
color embedding
color rep
<s>
x1
h1
dark
light
error signal
3 / 8
![Page 11: Grounded language understanding: Speakers: From the world](https://reader035.vdocuments.us/reader035/viewer/2022081701/62e1516d2eafe528ea3ac98d/html5/thumbnails/11.jpg)
Task Training Prediction Design choices Monroe et al. (2016) Related tasks
Color describer: Training with teacher forcing
DecoderEncoder
208.3, 60, 88.2
color embedding
color rep
<s>
x1
h1
dark
light
error signal
embedding
derived from x1 and color rep, the initial hidden state
predicted probability distribution over the vocab
one-hot encoding for next word
3 / 8
![Page 12: Grounded language understanding: Speakers: From the world](https://reader035.vdocuments.us/reader035/viewer/2022081701/62e1516d2eafe528ea3ac98d/html5/thumbnails/12.jpg)
Task Training Prediction Design choices Monroe et al. (2016) Related tasks
Color describer: Training with teacher forcing
DecoderEncoder
208.3, 60, 88.2
color embedding
color rep
light<s>
x1
h1
dark
3 / 8
![Page 13: Grounded language understanding: Speakers: From the world](https://reader035.vdocuments.us/reader035/viewer/2022081701/62e1516d2eafe528ea3ac98d/html5/thumbnails/13.jpg)
Task Training Prediction Design choices Monroe et al. (2016) Related tasks
Color describer: Training with teacher forcing
DecoderEncoder
208.3, 60, 88.2
color embedding
color rep
light<s>
x1 x37
h1
dark
3 / 8
![Page 14: Grounded language understanding: Speakers: From the world](https://reader035.vdocuments.us/reader035/viewer/2022081701/62e1516d2eafe528ea3ac98d/html5/thumbnails/14.jpg)
Task Training Prediction Design choices Monroe et al. (2016) Related tasks
Color describer: Training with teacher forcing
DecoderEncoder
208.3, 60, 88.2
color embedding
color rep
light<s>
x1 x37
h1 h2
dark
3 / 8
![Page 15: Grounded language understanding: Speakers: From the world](https://reader035.vdocuments.us/reader035/viewer/2022081701/62e1516d2eafe528ea3ac98d/html5/thumbnails/15.jpg)
Task Training Prediction Design choices Monroe et al. (2016) Related tasks
Color describer: Training with teacher forcing
DecoderEncoder
208.3, 60, 88.2
color embedding
color rep
light<s>
x1 x37
h1 h2
dark blue
3 / 8
![Page 16: Grounded language understanding: Speakers: From the world](https://reader035.vdocuments.us/reader035/viewer/2022081701/62e1516d2eafe528ea3ac98d/html5/thumbnails/16.jpg)
Task Training Prediction Design choices Monroe et al. (2016) Related tasks
Color describer: Training with teacher forcing
DecoderEncoder
208.3, 60, 88.2
color embedding
color rep
light<s>
x1 x37
h1 h2
dark blue
blue
error signal
3 / 8
![Page 17: Grounded language understanding: Speakers: From the world](https://reader035.vdocuments.us/reader035/viewer/2022081701/62e1516d2eafe528ea3ac98d/html5/thumbnails/17.jpg)
Task Training Prediction Design choices Monroe et al. (2016) Related tasks
Color describer: Training with teacher forcing
DecoderEncoder
208.3, 60, 88.2
color embedding
color rep
light<s> blue
x1 x37
h1 h2
dark blue
3 / 8
![Page 18: Grounded language understanding: Speakers: From the world](https://reader035.vdocuments.us/reader035/viewer/2022081701/62e1516d2eafe528ea3ac98d/html5/thumbnails/18.jpg)
Task Training Prediction Design choices Monroe et al. (2016) Related tasks
Color describer: Training with teacher forcing
DecoderEncoder
208.3, 60, 88.2
color embedding
color rep
light<s> blue
x1 x37 x11
h1 h2
dark blue
3 / 8
![Page 19: Grounded language understanding: Speakers: From the world](https://reader035.vdocuments.us/reader035/viewer/2022081701/62e1516d2eafe528ea3ac98d/html5/thumbnails/19.jpg)
Task Training Prediction Design choices Monroe et al. (2016) Related tasks
Color describer: Training with teacher forcing
DecoderEncoder
208.3, 60, 88.2
color embedding
color rep
light<s> blue
x1 x37 x11
h1 h2 h3
dark blue
3 / 8
![Page 20: Grounded language understanding: Speakers: From the world](https://reader035.vdocuments.us/reader035/viewer/2022081701/62e1516d2eafe528ea3ac98d/html5/thumbnails/20.jpg)
Task Training Prediction Design choices Monroe et al. (2016) Related tasks
Color describer: Training with teacher forcing
DecoderEncoder
208.3, 60, 88.2
color embedding
color rep
light<s> blue
x1 x37 x11
h1 h2 h3
dark blue green
3 / 8
![Page 21: Grounded language understanding: Speakers: From the world](https://reader035.vdocuments.us/reader035/viewer/2022081701/62e1516d2eafe528ea3ac98d/html5/thumbnails/21.jpg)
Task Training Prediction Design choices Monroe et al. (2016) Related tasks
Color describer: Training with teacher forcing
DecoderEncoder
208.3, 60, 88.2
color embedding
color rep
light<s> blue
x1 x37 x11
h1 h2 h3
dark blue green
</s>
error signal
3 / 8
![Page 22: Grounded language understanding: Speakers: From the world](https://reader035.vdocuments.us/reader035/viewer/2022081701/62e1516d2eafe528ea3ac98d/html5/thumbnails/22.jpg)
Task Training Prediction Design choices Monroe et al. (2016) Related tasks
Color describer: Prediction
DecoderEncoder
208.3, 60, 88.2
color embedding
color rep
4 / 8
![Page 23: Grounded language understanding: Speakers: From the world](https://reader035.vdocuments.us/reader035/viewer/2022081701/62e1516d2eafe528ea3ac98d/html5/thumbnails/23.jpg)
Task Training Prediction Design choices Monroe et al. (2016) Related tasks
Color describer: Prediction
DecoderEncoder
208.3, 60, 88.2
color embedding
color rep
<s>
4 / 8
![Page 24: Grounded language understanding: Speakers: From the world](https://reader035.vdocuments.us/reader035/viewer/2022081701/62e1516d2eafe528ea3ac98d/html5/thumbnails/24.jpg)
Task Training Prediction Design choices Monroe et al. (2016) Related tasks
Color describer: Prediction
DecoderEncoder
208.3, 60, 88.2
color embedding
color rep
<s>
x1
4 / 8
![Page 25: Grounded language understanding: Speakers: From the world](https://reader035.vdocuments.us/reader035/viewer/2022081701/62e1516d2eafe528ea3ac98d/html5/thumbnails/25.jpg)
Task Training Prediction Design choices Monroe et al. (2016) Related tasks
Color describer: Prediction
DecoderEncoder
208.3, 60, 88.2
color embedding
color rep
<s>
x1
h1
4 / 8
![Page 26: Grounded language understanding: Speakers: From the world](https://reader035.vdocuments.us/reader035/viewer/2022081701/62e1516d2eafe528ea3ac98d/html5/thumbnails/26.jpg)
Task Training Prediction Design choices Monroe et al. (2016) Related tasks
Color describer: Prediction
DecoderEncoder
208.3, 60, 88.2
color embedding
color rep
<s>
x1
h1
dark
4 / 8
![Page 27: Grounded language understanding: Speakers: From the world](https://reader035.vdocuments.us/reader035/viewer/2022081701/62e1516d2eafe528ea3ac98d/html5/thumbnails/27.jpg)
Task Training Prediction Design choices Monroe et al. (2016) Related tasks
Color describer: Prediction
DecoderEncoder
208.3, 60, 88.2
color embedding
color rep
dark<s>
x1 x20
h1 h2
dark
4 / 8
![Page 28: Grounded language understanding: Speakers: From the world](https://reader035.vdocuments.us/reader035/viewer/2022081701/62e1516d2eafe528ea3ac98d/html5/thumbnails/28.jpg)
Task Training Prediction Design choices Monroe et al. (2016) Related tasks
Color describer: Prediction
DecoderEncoder
208.3, 60, 88.2
color embedding
color rep
dark<s>
x1 x20
h1 h2
dark blue
4 / 8
![Page 29: Grounded language understanding: Speakers: From the world](https://reader035.vdocuments.us/reader035/viewer/2022081701/62e1516d2eafe528ea3ac98d/html5/thumbnails/29.jpg)
Task Training Prediction Design choices Monroe et al. (2016) Related tasks
Color describer: Prediction
DecoderEncoder
208.3, 60, 88.2
color embedding
color rep
dark<s> blue
x1 x20 x11
h1 h2 h3
dark blue
4 / 8
![Page 30: Grounded language understanding: Speakers: From the world](https://reader035.vdocuments.us/reader035/viewer/2022081701/62e1516d2eafe528ea3ac98d/html5/thumbnails/30.jpg)
Task Training Prediction Design choices Monroe et al. (2016) Related tasks
Color describer: Prediction
DecoderEncoder
208.3, 60, 88.2
color embedding
color rep
dark<s> blue
x1 x20 x11
h1 h2 h3
dark blue </s>
4 / 8
![Page 31: Grounded language understanding: Speakers: From the world](https://reader035.vdocuments.us/reader035/viewer/2022081701/62e1516d2eafe528ea3ac98d/html5/thumbnails/31.jpg)
Task Training Prediction Design choices Monroe et al. (2016) Related tasks
Miscellaneous design choices
• The Encoder and Decoder could have more hiddenlayers. We would expect the layer counts to match tofacilitate the hand-off between Encoder and Decoder,though pooling or copying might work too.
• It seems very common at present for researchers to tiethe embedding and classifier parameters (Press and Wolf2017)
• During training, one might drop teacher forcing a smallpercentage of the time to encourage the model toexplore.
5 / 8
![Page 32: Grounded language understanding: Speakers: From the world](https://reader035.vdocuments.us/reader035/viewer/2022081701/62e1516d2eafe528ea3ac98d/html5/thumbnails/32.jpg)
Task Training Prediction Design choices Monroe et al. (2016) Related tasks
Color describer of Monroe et al. (2016)
DecoderEncoder
HSV
Fourier transform
color rep
light<s> blue
x1 x37 x11
h1 h2 h3
w2 w3 w4
6 / 8
![Page 33: Grounded language understanding: Speakers: From the world](https://reader035.vdocuments.us/reader035/viewer/2022081701/62e1516d2eafe528ea3ac98d/html5/thumbnails/33.jpg)
Task Training Prediction Design choices Monroe et al. (2016) Related tasks
Related tasks
Non-linguistic representation ⇒ Language
• Image captioning
• Scene description
• Visual Question Answering(Image + Question-text ⇒ Answer-text)
• Instruction giving (State ⇒ Language)
• . . .
7 / 8
![Page 34: Grounded language understanding: Speakers: From the world](https://reader035.vdocuments.us/reader035/viewer/2022081701/62e1516d2eafe528ea3ac98d/html5/thumbnails/34.jpg)
References
References I
Brian McMahan and Matthew Stone. 2015. A Bayesian model of grounded color semantics. Transactions of the Associationfor Computational Linguistics, 3:103–115.
Will Monroe, Noah D. Goodman, and Christopher Potts. 2016. Learning to generate compositional color descriptions. InProceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2243–2248,Stroudsburg, PA. Association for Computational Linguistics.
Ofir Press and Lior Wolf. 2017. Using the output embedding to improve language models. In Proceedings of the 15thConference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages157–163, Valencia, Spain. Association for Computational Linguistics.
8 / 8