deep learning intro - github pages · deep learning? • subfield of ml: learning representations...
TRANSCRIPT
![Page 1: Deep Learning Intro - GitHub Pages · Deep Learning? • Subfield of ML: learning representations of data. – Attempt to learn (multiple levels of) representation by using a hierarchy](https://reader033.vdocuments.us/reader033/viewer/2022052719/5f07e8e27e708231d41f5e30/html5/thumbnails/1.jpg)
Deep Learning Intro
2018. 5. 11.Lee, Gyeongbok
CSI4108-01 ARTIFICIAL INTELLIGENCE 1
![Page 2: Deep Learning Intro - GitHub Pages · Deep Learning? • Subfield of ML: learning representations of data. – Attempt to learn (multiple levels of) representation by using a hierarchy](https://reader033.vdocuments.us/reader033/viewer/2022052719/5f07e8e27e708231d41f5e30/html5/thumbnails/2.jpg)
Contents
• Machine Learning and Deep Learning• Neural Network Architectures
– Convolutional Neural Network (CNN)– Recurrent Neural Network (RNN)
• …and some practices later (with pytorch)
Deep Learning Intro 2
※ Most of the material is from [1], [2], [3] in References slide.
![Page 3: Deep Learning Intro - GitHub Pages · Deep Learning? • Subfield of ML: learning representations of data. – Attempt to learn (multiple levels of) representation by using a hierarchy](https://reader033.vdocuments.us/reader033/viewer/2022052719/5f07e8e27e708231d41f5e30/html5/thumbnails/3.jpg)
Machine Learning?
• Machine learning: a field of computer science that gives computers the ability to learn without being explicitly programmed– Can learn from and make predictions on data
Labeled Data
Labeled Data
Machine Learningalgorithm
Learned model Prediction
Training
Prediction
Deep Learning Intro 3
![Page 4: Deep Learning Intro - GitHub Pages · Deep Learning? • Subfield of ML: learning representations of data. – Attempt to learn (multiple levels of) representation by using a hierarchy](https://reader033.vdocuments.us/reader033/viewer/2022052719/5f07e8e27e708231d41f5e30/html5/thumbnails/4.jpg)
Types of Learning
• Supervised: Learning with a labeled training set– email classification with already labeled emails
• Unsupervised: Discover patterns in unlabeled data– cluster similar documents based on text
• Reinforcement learning: learn to act based on feedback/reward– Go agent (alphaGo) - reward: win or lose
Deep Learning Intro
Regression
class A
class A
ClassificationAnomaly DetectionSequence labeling
…
Clustering
http://mbjoseph.github.io/2013/11/27/measure.html
4
![Page 5: Deep Learning Intro - GitHub Pages · Deep Learning? • Subfield of ML: learning representations of data. – Attempt to learn (multiple levels of) representation by using a hierarchy](https://reader033.vdocuments.us/reader033/viewer/2022052719/5f07e8e27e708231d41f5e30/html5/thumbnails/5.jpg)
ML vs. Deep Learning
• Still needs human works– Most machine learning methods work well because of human-
designed representations and input features– ML becomes just optimizing weights to best make a final
prediction (tuning)
Deep Learning Intro 5
![Page 6: Deep Learning Intro - GitHub Pages · Deep Learning? • Subfield of ML: learning representations of data. – Attempt to learn (multiple levels of) representation by using a hierarchy](https://reader033.vdocuments.us/reader033/viewer/2022052719/5f07e8e27e708231d41f5e30/html5/thumbnails/6.jpg)
https://www.xenonstack.com/blog/static/public/uploads/media/machine-learning-vs-deep-learning.png
Deep Learning?
• Subfield of ML: learning representations of data. – Attempt to learn (multiple levels of) representation by
using a hierarchy of multiple layers– If you provide the system tons of information, it begins to
understand it and respond in useful ways.– Exceptional effective at learning patterns!
Deep Learning Intro 6
![Page 7: Deep Learning Intro - GitHub Pages · Deep Learning? • Subfield of ML: learning representations of data. – Attempt to learn (multiple levels of) representation by using a hierarchy](https://reader033.vdocuments.us/reader033/viewer/2022052719/5f07e8e27e708231d41f5e30/html5/thumbnails/7.jpg)
Why is DL useful?
• Existing ML uses manually designed features– often over-specified and incomplete– take a long time to design and validate
• Learned Features are easy to adapt, fast to learn• Deep learning provides a very flexible, (almost?)
universal, learnable framework for representing world, visual and linguistic information.– For both unsupervised and supervised
• Effective end-to-end joint system learning• Utilize large amounts of training data
Deep Learning Intro 7
![Page 8: Deep Learning Intro - GitHub Pages · Deep Learning? • Subfield of ML: learning representations of data. – Attempt to learn (multiple levels of) representation by using a hierarchy](https://reader033.vdocuments.us/reader033/viewer/2022052719/5f07e8e27e708231d41f5e30/html5/thumbnails/8.jpg)
In Google Trend…
Deep Learning Intro 8
![Page 9: Deep Learning Intro - GitHub Pages · Deep Learning? • Subfield of ML: learning representations of data. – Attempt to learn (multiple levels of) representation by using a hierarchy](https://reader033.vdocuments.us/reader033/viewer/2022052719/5f07e8e27e708231d41f5e30/html5/thumbnails/9.jpg)
State of the art in …
Deep Learning Intro 9
![Page 10: Deep Learning Intro - GitHub Pages · Deep Learning? • Subfield of ML: learning representations of data. – Attempt to learn (multiple levels of) representation by using a hierarchy](https://reader033.vdocuments.us/reader033/viewer/2022052719/5f07e8e27e708231d41f5e30/html5/thumbnails/10.jpg)
State of the art in …
Deep Learning Intro 10
Ends in 2017Image from https://blogs.nvidia.com/blog/2016/06/29/deep-learning-6/
![Page 11: Deep Learning Intro - GitHub Pages · Deep Learning? • Subfield of ML: learning representations of data. – Attempt to learn (multiple levels of) representation by using a hierarchy](https://reader033.vdocuments.us/reader033/viewer/2022052719/5f07e8e27e708231d41f5e30/html5/thumbnails/11.jpg)
State of the art in …
• Several big improvements in recent years in NLP – Machine Translation– Sentiment Analysis– Dialogue Agents– Question Answering– Text Classification
Deep Learning Intro 11
![Page 12: Deep Learning Intro - GitHub Pages · Deep Learning? • Subfield of ML: learning representations of data. – Attempt to learn (multiple levels of) representation by using a hierarchy](https://reader033.vdocuments.us/reader033/viewer/2022052719/5f07e8e27e708231d41f5e30/html5/thumbnails/12.jpg)
Demo (tensorflow playground) How do we train?
𝒉𝒉 = 𝝈𝝈(𝐖𝐖𝟏𝟏𝒙𝒙 + 𝒃𝒃𝟏𝟏)𝒚𝒚 = 𝝈𝝈(𝑾𝑾𝟐𝟐𝒉𝒉 + 𝒃𝒃𝟐𝟐)
4 + 2 = 6 neurons (not counting inputs)[3 x 4] + [4 x 2] = 20 weights
4 + 2 = 6 biases26 learnable parameters
Neural Network Basis
Deep Learning Intro 12
𝒙𝒙𝒉𝒉
𝒚𝒚
Weights & Activation Functions
hidden
output
input
![Page 13: Deep Learning Intro - GitHub Pages · Deep Learning? • Subfield of ML: learning representations of data. – Attempt to learn (multiple levels of) representation by using a hierarchy](https://reader033.vdocuments.us/reader033/viewer/2022052719/5f07e8e27e708231d41f5e30/html5/thumbnails/13.jpg)
Optimize (min. or max.) objective/cost function 𝐽𝐽(𝜃𝜃)Generate error signal that measures difference between predictions and target values
https://medium.com/@ramrajchandradevan/the-evolution-of-gradient-descend-optimization-algorithm-4106a6702d39
Training Process
Deep Learning Intro 13
1. Sample labeled data(batch input)
2. Forward it through the network, get predictions
3. Back-propagatethe errors
4. Update the network weights
![Page 14: Deep Learning Intro - GitHub Pages · Deep Learning? • Subfield of ML: learning representations of data. – Attempt to learn (multiple levels of) representation by using a hierarchy](https://reader033.vdocuments.us/reader033/viewer/2022052719/5f07e8e27e708231d41f5e30/html5/thumbnails/14.jpg)
Problems?
Deep Learning Intro 14
https://www.neuraldesigner.com/images/learning/selection_error.svg
https://www.neuraldesigner.com/images/learning/selection_error.svg
Over-fittingLearned hypothesis may fit the trainingdata very well, even for outliers (noise)but fail to generalize to new examples(test data)
Solution: regularization, etc
![Page 15: Deep Learning Intro - GitHub Pages · Deep Learning? • Subfield of ML: learning representations of data. – Attempt to learn (multiple levels of) representation by using a hierarchy](https://reader033.vdocuments.us/reader033/viewer/2022052719/5f07e8e27e708231d41f5e30/html5/thumbnails/15.jpg)
Neural Network Architectures
Deep Learning Intro 15
CNN
RNNImages from CS231n lecture slides
![Page 16: Deep Learning Intro - GitHub Pages · Deep Learning? • Subfield of ML: learning representations of data. – Attempt to learn (multiple levels of) representation by using a hierarchy](https://reader033.vdocuments.us/reader033/viewer/2022052719/5f07e8e27e708231d41f5e30/html5/thumbnails/16.jpg)
Convolution Neural Network (CNN)
Deep Learning Intro 16
Images from CS231n lecture slidesAnimation from https://github.com/vdumoulin/conv_arithmetic
![Page 17: Deep Learning Intro - GitHub Pages · Deep Learning? • Subfield of ML: learning representations of data. – Attempt to learn (multiple levels of) representation by using a hierarchy](https://reader033.vdocuments.us/reader033/viewer/2022052719/5f07e8e27e708231d41f5e30/html5/thumbnails/17.jpg)
Convolution Neural Network (CNN)
Deep Learning Intro 17
![Page 18: Deep Learning Intro - GitHub Pages · Deep Learning? • Subfield of ML: learning representations of data. – Attempt to learn (multiple levels of) representation by using a hierarchy](https://reader033.vdocuments.us/reader033/viewer/2022052719/5f07e8e27e708231d41f5e30/html5/thumbnails/18.jpg)
Pooling layer
Deep Learning Intro 18
Images from CS231n lecture slides
![Page 19: Deep Learning Intro - GitHub Pages · Deep Learning? • Subfield of ML: learning representations of data. – Attempt to learn (multiple levels of) representation by using a hierarchy](https://reader033.vdocuments.us/reader033/viewer/2022052719/5f07e8e27e708231d41f5e30/html5/thumbnails/19.jpg)
Max Pooling
Deep Learning Intro 19
Images from CS231n lecture slides
![Page 20: Deep Learning Intro - GitHub Pages · Deep Learning? • Subfield of ML: learning representations of data. – Attempt to learn (multiple levels of) representation by using a hierarchy](https://reader033.vdocuments.us/reader033/viewer/2022052719/5f07e8e27e708231d41f5e30/html5/thumbnails/20.jpg)
ConvNet
Deep Learning Intro 20
Images from https://blog.floydhub.com/building-your-first-convnet/
![Page 21: Deep Learning Intro - GitHub Pages · Deep Learning? • Subfield of ML: learning representations of data. – Attempt to learn (multiple levels of) representation by using a hierarchy](https://reader033.vdocuments.us/reader033/viewer/2022052719/5f07e8e27e708231d41f5e30/html5/thumbnails/21.jpg)
CNN Applications
Deep Learning Intro 21
Images from CS231n lecture slides
![Page 22: Deep Learning Intro - GitHub Pages · Deep Learning? • Subfield of ML: learning representations of data. – Attempt to learn (multiple levels of) representation by using a hierarchy](https://reader033.vdocuments.us/reader033/viewer/2022052719/5f07e8e27e708231d41f5e30/html5/thumbnails/22.jpg)
CNN Applications
Deep Learning Intro 22
Images from CS231n lecture slides
![Page 23: Deep Learning Intro - GitHub Pages · Deep Learning? • Subfield of ML: learning representations of data. – Attempt to learn (multiple levels of) representation by using a hierarchy](https://reader033.vdocuments.us/reader033/viewer/2022052719/5f07e8e27e708231d41f5e30/html5/thumbnails/23.jpg)
CNN Applications
Deep Learning Intro 23
Style Transfer
Images from CS231n lecture slides
![Page 24: Deep Learning Intro - GitHub Pages · Deep Learning? • Subfield of ML: learning representations of data. – Attempt to learn (multiple levels of) representation by using a hierarchy](https://reader033.vdocuments.us/reader033/viewer/2022052719/5f07e8e27e708231d41f5e30/html5/thumbnails/24.jpg)
Recurrent Neural Network (RNN)
Deep Learning Intro 24
![Page 25: Deep Learning Intro - GitHub Pages · Deep Learning? • Subfield of ML: learning representations of data. – Attempt to learn (multiple levels of) representation by using a hierarchy](https://reader033.vdocuments.us/reader033/viewer/2022052719/5f07e8e27e708231d41f5e30/html5/thumbnails/25.jpg)
RNN Applications
Deep Learning Intro 25
Images from CS224n lecture slides
Language Model
![Page 26: Deep Learning Intro - GitHub Pages · Deep Learning? • Subfield of ML: learning representations of data. – Attempt to learn (multiple levels of) representation by using a hierarchy](https://reader033.vdocuments.us/reader033/viewer/2022052719/5f07e8e27e708231d41f5e30/html5/thumbnails/26.jpg)
RNN Applications
Deep Learning Intro 26
Images from CS224n lecture slides
![Page 27: Deep Learning Intro - GitHub Pages · Deep Learning? • Subfield of ML: learning representations of data. – Attempt to learn (multiple levels of) representation by using a hierarchy](https://reader033.vdocuments.us/reader033/viewer/2022052719/5f07e8e27e708231d41f5e30/html5/thumbnails/27.jpg)
RNN Applications
Deep Learning Intro 27
Images from:http://www.wildml.com/2016/04/deep-learning-for-chatbots-part-1-introduction/
https://medium.com/botsupply/generative-model-chatbots-e422ab08461e
Question Answering, Conversation (Chatbot)
![Page 28: Deep Learning Intro - GitHub Pages · Deep Learning? • Subfield of ML: learning representations of data. – Attempt to learn (multiple levels of) representation by using a hierarchy](https://reader033.vdocuments.us/reader033/viewer/2022052719/5f07e8e27e708231d41f5e30/html5/thumbnails/28.jpg)
RNN Applications
Deep Learning Intro 28
Images from https://research.googleblog.com/2014/11/a-picture-is-worth-thousand-coherent.html
Image/Video Caption
![Page 29: Deep Learning Intro - GitHub Pages · Deep Learning? • Subfield of ML: learning representations of data. – Attempt to learn (multiple levels of) representation by using a hierarchy](https://reader033.vdocuments.us/reader033/viewer/2022052719/5f07e8e27e708231d41f5e30/html5/thumbnails/29.jpg)
References
[1] UIUC CS 510 Course Material made by Ismini Lourentzou– http://times.cs.uiuc.edu/course/510f17/ppt/deep-learning.pptx
[2] Stanford CS231n lecture slides (CNN/Visual Recognition)– http://cs231n.stanford.edu/syllabus.html– https://www.youtube.com/playlist?list=PL3FW7Lu3i5JvHM8ljYj-
zLfQRF3EO8sYv (2017 Lecture Videos)
[3] Stanford CS224n lecture slides (RNN/Language)– http://web.stanford.edu/class/cs224n/syllabus.html– https://www.youtube.com/playlist?list=PLqdrfNEc5QnuV9RwUAhoJco
Qvu4Q46Lja (2017 Lecture Videos)
Deep Learning Intro 29