deep learningce.aut.ac.ir/~keyvanrad/download/dl961/dl_1(introduction).pdf · 9/24/2017 m.a...
Post on 12-Sep-2020
0 Views
Preview:
TRANSCRIPT
Deep Learning
Mohammad Ali Keyvanrad
Lecture 1:Introduction
OUTLINE • Recent success with Deep Learning
• Deep Learning definition
• History
• Course plan
• Resources
• Grading Policy
9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 2
OUTLINE • Recent success with Deep Learning
• Deep Learning definition
• History
• Course plan
• Resources
• Grading Policy
9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 3
9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 4
Recent success with Deep Learning
• Over the last few years Deep Learning was applied tohundreds of problems. Computer vision and pattern recognition
Speech recognition and speech synthesis
Natural language processing
Computer games, robots & self-driving cars
…
• In many problems they have established the state ofthe art Often exceeding previous benchmarks by large margins
9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 5
Learning Lip Sync from Audio(University of Washington, 2017)
9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 6
Restore colors in B&W photos (Waseda University, 2016)
Input [Larsson et al. 2016] [Zhang et al. 2016a] [Zhang et al. 2016b]
9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 7
Pixel restoration(Google Brain, 2017)
• Take very low resolution images and predict what eachimage most likely looks like.
9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 8
Describing photos(Stanford University, 2015)
• Computers can automatically classify our photos Facebook can automatically tag your friends
• Deep Learning not only learned to classify the elementsin the photo, but to actually describe them.
9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 9
Translation(Google Translate, 2015)
• Google Translate app now does real-time visualtranslation of 20 more languages. A photo taken by the phone, and Google Translate "reads"
the text and replaces it with a text in English in real-time.
9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 10
Create new images(University of California, 2017)
• Deep Learning network to create other types of newimages
9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 11
Reading text in the Wild(University of Oxford, 2014)
• An attempt to read text from photos and videos Search for text from BBC News videos
9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 12
Teach a computer to play(DeepMind, 2015)
• Google's DeepMind used a Deep Learning technique toteach a computer to play Control of the keyboard while watching the score, and its
goal was to maximize the score
9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 13
Beating people in dozens of computer games
Computer program playing Doom using only raw pixel data.
9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 14
Self-driving cars(Tesla, 2016)
• A Tesla electric vehicle drives without humanintervention Notice how it distinguishes different type of objects,
including people and road signs.
9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 15
Robotics(BostonDynamics, 2016)
• Deep Learning is also heavily used in robotics thesedays SpotMini and Atlas
9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 16
Voice generation(Google WaveNet, 2016)
• Deep Learning is taking us a step closer to givingcomputers the ability to speak like humans do. Google released WaveNet and Tacotron
Baidu released Deep Speech
9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction)
• Tacotron learns pronunciations based on phrasesemantics. “He has read the whole thing.”
“He reads books.”
• Tacotron is sensitive to punctuation. “This is your personal assistant, Google Home.”
• Tacotron learns stress and intonation. “The buses aren't the problem, they actually provide a solution.”
“The buses aren't the PROBLEM, they actually provide a SOLUTION.”
• Tacotron's prosody changes in a question. “The quick brown fox jumps over the lazy dog.”
“Does the quick brown fox jump over the lazy dog?”
17
Voice generation(Google Tacotron, 2017)
9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 18
Restoring sound in videos(MIT, Berkeley, Google, 2016)
• Deep Learning network was trained on videos in whichpeople were hitting and scratching objects
• After several iterations learning, the scientists asked thecomputer to regenerate the sound
9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 19
LIPNET(Oxford, DeepMind, 2016)
• LipNet reached 93% success in reading people's lipswhere an average lipreader succeeds 52% of the time.
9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 20
Automatically writing(Stanford University, 2016)
• Let a Deep Learning network "read" Shakespeare,Wikipedia, math papers and computer code.
9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 21
Handwriting(University of Toronto, 2014)
• Today the computer can also handwrite.
OUTLINE • Recent success with Deep Learning
• Deep Learning definition
• History
• Course plan
• Resources
• Grading Policy
9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 22
9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 23
Deep Learning definition
• Deep learning is a class of machine learning algorithmsthat: Definition 1: They use a cascade of many layers of nonlinear processing units for
feature extraction and transformation Each successive layer uses the output from the previous layer as
input. The algorithms may be supervised or unsupervised. Applications include pattern analysis (unsupervised) and
classification (supervised).
Definition 2: They are part of the broader machine learning field of learning
representations of data.
9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 24
Deep Learning definition
Definition 3: These are based on the (unsupervised) learning of multiple levels
of features or representations of the data.
Higher level features are derived from lower level features to forma hierarchical representation.
Definition 4: They learn multiple levels of representations that correspond to
different levels of abstraction; the levels form a hierarchy ofconcepts.
9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 25
Deep Learning definition
• Common in definitions Models consisting of multiple layers or stages of nonlinear
information processing.
The supervised or unsupervised learning of featurerepresentations in each layer, with the layers forming ahierarchy from low-level to high-level features.
9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 26
Deep Learning definition
• Deep or Shallow? Credit Assignment Path (CAP) A chain of transformations from input to output.
CAPs describe potentially causal connections between input andoutput.
CAP depth Number of hidden layers plus one
as the output layer is also parameterized
For recurrent neural networks the CAP depth is potentiallyunlimited.
a signal may propagate through a layer more than once.
9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 27
Deep Learning definition
• Deep/shallow?
• No universally agreed upon threshold of depth divides shallowlearning from deep learning
• Most researchers agree that deep learning has multiple nonlinearlayers (CAP > 2).
OUTLINE • Recent success with Deep Learning
• Deep Learning definition
• History
• Course plan
• Resources
• Grading Policy
9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 28
9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 29
History
9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 30
Evolution of DepthLeNet
1998
7 Layers
AlexNet
2012
8 Layers
GoogLeNet
2014
22 Layers
ResNet
2016
152 Layers
9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 31
Evolution of Depth
OUTLINE • Recent success with Deep Learning
• Deep Learning definition
• History
• Course plan
• Resources
• Grading Policy
9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 32
9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 33
Course plan
• Introduction
• A Review of Artificial Neural Networks Perceptron
Stochastic Gradient Descent
Backpropagation
Rectified Linear Function
Root Mean Square Propagation
Dropout
L1 and L2 Regularization
9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 34
Course plan
• Deep Belief Network PGM
MRF
Sampling
RBM
• Auto-Encoder Linear Auto-Encoder
Denoising Auto-Encoder
• Computational Network
9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 35
Course plan
• Selected Applications in Object Recognition and ComputerVision Convolutional Neural Networks Region Based CNN Generative Adversarial Network GoogLeNet and Microsoft ResNet
• Selected Applications in Language Modeling and NaturalLanguage Processing Word2Vec Recurrent Neural Networks and Language Models Machine translation and advanced recurrent LSTMs and GRUs
9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 36
Course plan
• Selected Applications in Speech and Audio Processing Speech recognition and bi-directional RNN
Speech synthesis and WaveNet or Tacotron
• Deep Reinforcement Learning
OUTLINE • Recent success with Deep Learning
• Deep Learning definition
• History
• Course plan
• Resources
• Grading Policy
9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 37
9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 38
Resources
• Ian Goodfellow, Yoshua Bengio, Aaron Courville, “Deep Learning (AdaptiveComputation and Machine Learning series)”, MIT Press, 2016.
• Dong Yu, Li Deng, “Automatic Speech Recognition: A Deep LearningApproach”, Springer, 2015
• L. Deng and D. Yu, “Deep Learning: Methods and Applications”, NowPublishers Inc, 2014.
• Pattern Recognition and Machine Learning, Christopher M. Bishop, 2006.
• Stanford (CS224n: Natural Language Processing with Deep Learning, 2017)
• Stanford (CS231n: Convolutional Neural Networks for Visual Recognition,2017)
• Related papers
OUTLINE • Recent success with Deep Learning
• Deep Learning definition
• History
• Course plan
• Resources
• Grading Policy
9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 39
9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 40
Grading Policy
• Assignments: 20%
• Presentation: 15%
• Final Exam: 35%
• Final Project: 30%
9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 41
9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 42
top related