![Page 1: CS 7016: TOPICS IN DEEP LEARNINGmiteshk/CS7016/QA/charembeds.pdf · CS 7016: TOPICS IN DEEP LEARNING CONVOLUTIONAL NEURAL NETWORKS FOR CHARACTER EMBEDDINGS . ... Split it into characters](https://reader033.vdocuments.us/reader033/viewer/2022051510/5ffdb7d61b163540ea5d5079/html5/thumbnails/1.jpg)
CS 7016: TOPICS IN DEEP LEARNING
CONVOLUTIONAL NEURAL NETWORKS FOR CHARACTER EMBEDDINGS
![Page 2: CS 7016: TOPICS IN DEEP LEARNINGmiteshk/CS7016/QA/charembeds.pdf · CS 7016: TOPICS IN DEEP LEARNING CONVOLUTIONAL NEURAL NETWORKS FOR CHARACTER EMBEDDINGS . ... Split it into characters](https://reader033.vdocuments.us/reader033/viewer/2022051510/5ffdb7d61b163540ea5d5079/html5/thumbnails/2.jpg)
I r e a l l y l o v e t h i s s h o w ! !
1
0
0
0
…
0
0
0
0
0
0
0
0
…
0
0
0
0
0
0
0
0
…
0
1
0
0
One-hot encoding
0
0
1
0
…
0
0
0
0
0
0
0
0
…
0
0
1
0
0
0
0
0
…
0
0
1
0
0
0
0
…
0
0
1
0
0
1
0
…
0
0
0
0
0
0
0
0
…
0
0
0
0
0
0
0
0
…
0
0
0
zero pads
Input: “I really love this show!!"Split it into characters and turn it into a list
![Page 3: CS 7016: TOPICS IN DEEP LEARNINGmiteshk/CS7016/QA/charembeds.pdf · CS 7016: TOPICS IN DEEP LEARNING CONVOLUTIONAL NEURAL NETWORKS FOR CHARACTER EMBEDDINGS . ... Split it into characters](https://reader033.vdocuments.us/reader033/viewer/2022051510/5ffdb7d61b163540ea5d5079/html5/thumbnails/3.jpg)
Max Length: 140 charactersVocabulary = 70( characters + digits + special characters)
ONE-HOT REPRESENTATION
140
70
![Page 4: CS 7016: TOPICS IN DEEP LEARNINGmiteshk/CS7016/QA/charembeds.pdf · CS 7016: TOPICS IN DEEP LEARNING CONVOLUTIONAL NEURAL NETWORKS FOR CHARACTER EMBEDDINGS . ... Split it into characters](https://reader033.vdocuments.us/reader033/viewer/2022051510/5ffdb7d61b163540ea5d5079/html5/thumbnails/4.jpg)
2D CONVOLUTIONS NOT SUITED FOR TEXT!
![Page 5: CS 7016: TOPICS IN DEEP LEARNINGmiteshk/CS7016/QA/charembeds.pdf · CS 7016: TOPICS IN DEEP LEARNING CONVOLUTIONAL NEURAL NETWORKS FOR CHARACTER EMBEDDINGS . ... Split it into characters](https://reader033.vdocuments.us/reader033/viewer/2022051510/5ffdb7d61b163540ea5d5079/html5/thumbnails/5.jpg)
1
0
0
0
…
0
0
0
0
0
0
0
0
…
0
0
0
0
0
0
0
0
…
0
1
0
0
Input: “I really love this show!!"Split it into characters and turn it into a list
0
0
1
0
…
0
0
0
0
0
0
0
0
…
0
0
1
0
0
0
0
0
…
0
0
0
1
0
0
0
0
…
0
0
0
1
…
0
0
0
0
…
0
0
1
0
0
0
0
0
…
0
0
1
0
0
0
0
…
0
0
0
1
0
0
1
0
…
0
0
0
0
0
0
0
0
…
0
0
0
0
0
0
0
0
…
0
0
0
0
0
0
0
0
…
0
0
0
0
140
Input Channel =
Alphabet size = 70
1 1-D Kernel
1 0 1
![Page 6: CS 7016: TOPICS IN DEEP LEARNINGmiteshk/CS7016/QA/charembeds.pdf · CS 7016: TOPICS IN DEEP LEARNING CONVOLUTIONAL NEURAL NETWORKS FOR CHARACTER EMBEDDINGS . ... Split it into characters](https://reader033.vdocuments.us/reader033/viewer/2022051510/5ffdb7d61b163540ea5d5079/html5/thumbnails/6.jpg)
1
0
0
0
…
0
0
0
0
0
0
0
0
…
0
0
0
0
0
0
0
0
…
0
1
0
0
Input: “I really love this show!!"Split it into characters and turn it into a list
0
0
1
0
…
0
0
0
0
0
0
0
0
…
0
0
1
0
0
0
0
0
…
0
0
0
1
0
0
0
0
…
0
0
0
1
…
0
0
0
0
…
0
0
1
0
0
0
0
0
…
0
0
1
0
0
0
0
…
0
0
0
1
0
0
1
0
…
0
0
0
0
0
0
0
0
…
0
0
0
0
0
0
0
0
…
0
0
0
0
0
0
0
0
…
0
0
0
0
140
Input Channel =
Alphabet size = 70
1 1-D Kernel
1 0 1
![Page 7: CS 7016: TOPICS IN DEEP LEARNINGmiteshk/CS7016/QA/charembeds.pdf · CS 7016: TOPICS IN DEEP LEARNING CONVOLUTIONAL NEURAL NETWORKS FOR CHARACTER EMBEDDINGS . ... Split it into characters](https://reader033.vdocuments.us/reader033/viewer/2022051510/5ffdb7d61b163540ea5d5079/html5/thumbnails/7.jpg)
1
0
0
0
…
0
0
0
0
0
0
0
0
…
0
0
0
0
0
0
0
0
…
0
1
0
0
Input: “I really love this show!!"Split it into characters and turn it into a list
0
0
1
0
…
0
0
0
0
0
0
0
0
…
0
0
1
0
0
0
0
0
…
0
0
0
1
0
0
0
0
…
0
0
0
1
…
0
0
0
0
…
0
0
1
0
0
0
0
0
…
0
0
1
0
0
0
0
…
0
0
0
1
0
0
1
0
…
0
0
0
0
0
0
0
0
…
0
0
0
0
0
0
0
0
…
0
0
0
0
0
0
0
0
…
0
0
0
0
140
Input Channel =
Alphabet size = 70
1 1-D Kernel
1 0 1
![Page 8: CS 7016: TOPICS IN DEEP LEARNINGmiteshk/CS7016/QA/charembeds.pdf · CS 7016: TOPICS IN DEEP LEARNING CONVOLUTIONAL NEURAL NETWORKS FOR CHARACTER EMBEDDINGS . ... Split it into characters](https://reader033.vdocuments.us/reader033/viewer/2022051510/5ffdb7d61b163540ea5d5079/html5/thumbnails/8.jpg)
1
0
0
0
…
0
0
0
0
0
0
0
0
…
0
0
0
0
0
0
0
0
…
0
1
0
0
Input: “I really love this show!!"Split it into characters and turn it into a list
0
0
1
0
…
0
0
0
0
0
0
0
0
…
0
0
1
0
0
0
0
0
…
0
0
0
1
0
0
0
0
…
0
0
0
1
…
0
0
0
0
…
0
0
1
0
0
0
0
0
…
0
0
1
0
0
0
0
…
0
0
0
1
0
0
1
0
…
0
0
0
0
0
0
0
0
…
0
0
0
0
0
0
0
0
…
0
0
0
0
0
0
0
0
…
0
0
0
0
140
Input Channel =
Alphabet size = 70
1 1-D Kernel
1 0 1
![Page 9: CS 7016: TOPICS IN DEEP LEARNINGmiteshk/CS7016/QA/charembeds.pdf · CS 7016: TOPICS IN DEEP LEARNING CONVOLUTIONAL NEURAL NETWORKS FOR CHARACTER EMBEDDINGS . ... Split it into characters](https://reader033.vdocuments.us/reader033/viewer/2022051510/5ffdb7d61b163540ea5d5079/html5/thumbnails/9.jpg)
1
0
0
0
…
0
0
0
0
0
0
0
0
…
0
0
0
0
0
0
0
0
…
0
1
0
0
Input: “I really love this show!!"Split it into characters and turn it into a list
0
0
1
0
…
0
0
0
0
0
0
0
0
…
0
0
1
0
0
0
0
0
…
0
0
0
1
0
0
0
0
…
0
0
0
1
…
0
0
0
0
…
0
0
1
0
0
0
0
0
…
0
0
1
0
0
0
0
…
0
0
0
1
0
0
1
0
…
0
0
0
0
0
0
0
0
…
0
0
0
0
0
0
0
0
…
0
0
0
0
0
0
0
0
…
0
0
0
0
140
Input Channel =
Alphabet size = 70
1 1-D Kernel
1 0 1
Batch: 1 “image” Input Channel = 70 Output Channel = 1
Output Shape = (1,1,138)
1 0 11 0 11 0 1
1 0 1
1 0 1
1 0 1
200 1-D Kernels
Batch: 1 “image” Input Channel = 70
Output Channel = 200 Output Shape =
(1,200,138)
Note that this kernel covers tri-gram, there could be other which can cover bi-gram, five-grams etc.
Max pooling operation will result in a 200 dimensional embedding for each “image”
![Page 10: CS 7016: TOPICS IN DEEP LEARNINGmiteshk/CS7016/QA/charembeds.pdf · CS 7016: TOPICS IN DEEP LEARNING CONVOLUTIONAL NEURAL NETWORKS FOR CHARACTER EMBEDDINGS . ... Split it into characters](https://reader033.vdocuments.us/reader033/viewer/2022051510/5ffdb7d61b163540ea5d5079/html5/thumbnails/10.jpg)
ACKNOWLEDGMENTS
• The material for these slides has been taken from https://www.youtube.com/watch?v=CNY8VjJt-iQ