intro deep learning
TRANSCRIPT
M. R. Avendi CPCC, UC Irvine
Nov. 2014
Deep Learning, Trends, and Advances
Outline Introduction and Motivations Machine Learning and Challenges Neuroscience Experiments Neural Networks and Optimization Deep Networks and Advances Summary
Machine Learning Supervised: labeled data, eg. spam filtering Unsupervised: unlabeled data, eg. clustering
Typical Applications
Images, Behind the Scene
Image Classification
Image Classification
What are features!?
Feature Extraction Hard , time consuming, requires knowledge Human brain does feature extraction
Neuroscience Experiment, (1992)
Auditory cortex learns to see!
Roe, Anna W., et al. "Visual projections routed to the auditory pathway in ferrets: receptive fields of
visual neurons in primary auditory cortex." The Journal of neuroscience 12.9 (1992): 3651-3664.
Seeing With Tongue Blind people can see using tongue http://www.wicab.com/en_us/press.html
One Learning Algorithm Hypothesis
We want: Automatic feature learning Training data Unlabeled: whatever, we have a lot! Labeled: small!
Mimicking Brain: Neural Networks Perceptron: one-layer NN Parameters : w, not known, training Activation function: f(x) =f(wi xi +w0)
Training One Layer Network Training data: input x, output y
• Reference: Deep Learning and Neural Networks, by Kevin Duh, http://cl.naist.jp/~kevinduh/a/deep2014/
Training: Gradient Descent
• Reference: Deep Learning and Neural Networks, by Kevin Duh, http://cl.naist.jp/~kevinduh/a/deep2014/
Two-Layer Network
• Reference: Deep Learning and Neural Networks, by Kevin Duh, http://cl.naist.jp/~kevinduh/a/deep2014/
Training: Backpropagation
• Reference: Deep Learning and Neural Networks, by Kevin Duh, http://cl.naist.jp/~kevinduh/a/deep2014/
Training: Backpropagation
• Reference: Deep Learning and Neural Networks, by Kevin Duh, http://cl.naist.jp/~kevinduh/a/deep2014/
Multi-Layer Network: Dark Ages
• Reference: Deep Learning and Neural Networks, by Kevin Duh, http://cl.naist.jp/~kevinduh/a/deep2014/
Breakthrough, [Hinton, et al., 2006] Layer-Wise Pre-Training, unsupervised Optimize likelihood of data, P(x)
• Reference: Deep Learning and Neural Networks, by Kevin Duh, http://cl.naist.jp/~kevinduh/a/deep2014/
Breakthrough, cnt. Fine-tune using labeled data, supervised
• Reference: Deep Learning and Neural Networks, by Kevin Duh, http://cl.naist.jp/~kevinduh/a/deep2014/
Deep Learning Approaches Deep Belief Networks RBM: learns data likelihood Stacked RBMs
Deep Learning Approaches Stacked Autoencoders Autoencoders: learns to reconstruct input data Easier to train
• Reference: Deep Learning tutorial, Andrew Ng
Convolutional Networks
• Reference: Deep Learning tutorial, Andrew Ng
Extracted Features Convolutional Deep Belief Networks for Scalable Unsupervised Learning of Hierarchical Representations[Lee et
al., 2009]
Deep Learning: advances Microsoft real-time speech translation https://www.youtube.com/watch?v=NhxCg2PA3ZI
Deep Learning: advances Google artificial brain learns to find cat and face NN, 1 billion connection, 16000 computers, browse YouTube
for 3 days
Others Google+ Image Search, no-tag image search Handwriting recognition Android speech to text Medical Diagnosis
Summary
• Reference: Deep Learning and Neural Networks, by Kevin Duh, http://cl.naist.jp/~kevinduh/a/deep2014/