python for image understanding: deep learning with convolutional neural nets
TRANSCRIPT
![Page 1: Python for Image Understanding: Deep Learning with Convolutional Neural Nets](https://reader034.vdocuments.us/reader034/viewer/2022051113/55c12d29bb61eb893d8b45b3/html5/thumbnails/1.jpg)
for Image Understanding:Deep Learning with Convolutional Neural Nets
Roelof PietersPhD candidate at KTH & Data Science consultant at Graph Technologies
@graphific
London [email protected]
![Page 2: Python for Image Understanding: Deep Learning with Convolutional Neural Nets](https://reader034.vdocuments.us/reader034/viewer/2022051113/55c12d29bb61eb893d8b45b3/html5/thumbnails/2.jpg)
What is Deep Learning?
![Page 3: Python for Image Understanding: Deep Learning with Convolutional Neural Nets](https://reader034.vdocuments.us/reader034/viewer/2022051113/55c12d29bb61eb893d8b45b3/html5/thumbnails/3.jpg)
“Deep learning is a set of algorithms in machine learning that attempt to learn in multiple levels, corresponding to different levels of abstraction.” (much debated definition)
A Definition
![Page 4: Python for Image Understanding: Deep Learning with Convolutional Neural Nets](https://reader034.vdocuments.us/reader034/viewer/2022051113/55c12d29bb61eb893d8b45b3/html5/thumbnails/4.jpg)
• A host of statistical machine learning techniques
• Enables the automatic learning of feature hierarchies
• Generally based on artificial neural networks
A typology
Deep learning is:
![Page 5: Python for Image Understanding: Deep Learning with Convolutional Neural Nets](https://reader034.vdocuments.us/reader034/viewer/2022051113/55c12d29bb61eb893d8b45b3/html5/thumbnails/5.jpg)
Manually designed features are often over-specified, incomplete and take a long time to design and validate
Learned features are easy to adapt, fast to learn
Deep learning provides a very flexible, (possibly?) universal, learnable framework for representing world, visual and linguistic information.
Deep learning can learn unsupervised (from raw text/audio/images/whatever content) and supervised (with specific labels like positive/negative)
Summary by Richard Socher.
Old vs new school?
![Page 6: Python for Image Understanding: Deep Learning with Convolutional Neural Nets](https://reader034.vdocuments.us/reader034/viewer/2022051113/55c12d29bb61eb893d8b45b3/html5/thumbnails/6.jpg)
No More Handcrafted Features !
6
![Page 7: Python for Image Understanding: Deep Learning with Convolutional Neural Nets](https://reader034.vdocuments.us/reader034/viewer/2022051113/55c12d29bb61eb893d8b45b3/html5/thumbnails/7.jpg)
“Brain”-like: Feature Hierarchies
![Page 8: Python for Image Understanding: Deep Learning with Convolutional Neural Nets](https://reader034.vdocuments.us/reader034/viewer/2022051113/55c12d29bb61eb893d8b45b3/html5/thumbnails/8.jpg)
input layer
output layer
hidden layers
Feature Hierarchies: Vision
![Page 9: Python for Image Understanding: Deep Learning with Convolutional Neural Nets](https://reader034.vdocuments.us/reader034/viewer/2022051113/55c12d29bb61eb893d8b45b3/html5/thumbnails/9.jpg)
input layer
output layer
hidden layers
Feature Hierarchies: Audio
![Page 10: Python for Image Understanding: Deep Learning with Convolutional Neural Nets](https://reader034.vdocuments.us/reader034/viewer/2022051113/55c12d29bb61eb893d8b45b3/html5/thumbnails/10.jpg)
cars elephants chairsFeature Hierarchies: And so on…
![Page 11: Python for Image Understanding: Deep Learning with Convolutional Neural Nets](https://reader034.vdocuments.us/reader034/viewer/2022051113/55c12d29bb61eb893d8b45b3/html5/thumbnails/11.jpg)
History: audio recognition
chart by Clarifai
![Page 12: Python for Image Understanding: Deep Learning with Convolutional Neural Nets](https://reader034.vdocuments.us/reader034/viewer/2022051113/55c12d29bb61eb893d8b45b3/html5/thumbnails/12.jpg)
History: image recognition
chart by Clarifai
![Page 13: Python for Image Understanding: Deep Learning with Convolutional Neural Nets](https://reader034.vdocuments.us/reader034/viewer/2022051113/55c12d29bb61eb893d8b45b3/html5/thumbnails/13.jpg)
History: image recognition
Krizhevsky et al. ImageNet Classification with Deep Convolutional Neural Networks, ILSVRC2010
![Page 14: Python for Image Understanding: Deep Learning with Convolutional Neural Nets](https://reader034.vdocuments.us/reader034/viewer/2022051113/55c12d29bb61eb893d8b45b3/html5/thumbnails/14.jpg)
![Page 15: Python for Image Understanding: Deep Learning with Convolutional Neural Nets](https://reader034.vdocuments.us/reader034/viewer/2022051113/55c12d29bb61eb893d8b45b3/html5/thumbnails/15.jpg)
![Page 16: Python for Image Understanding: Deep Learning with Convolutional Neural Nets](https://reader034.vdocuments.us/reader034/viewer/2022051113/55c12d29bb61eb893d8b45b3/html5/thumbnails/16.jpg)
16
Karpathy, A., Fei Fei, L. (2015) Deep Visual-Semantic Alignments for Generating Image Descriptions
Image-Text: Joint Visual Semantic embeddings
![Page 17: Python for Image Understanding: Deep Learning with Convolutional Neural Nets](https://reader034.vdocuments.us/reader034/viewer/2022051113/55c12d29bb61eb893d8b45b3/html5/thumbnails/17.jpg)
Video Scene Detection
![Page 18: Python for Image Understanding: Deep Learning with Convolutional Neural Nets](https://reader034.vdocuments.us/reader034/viewer/2022051113/55c12d29bb61eb893d8b45b3/html5/thumbnails/18.jpg)
![Page 19: Python for Image Understanding: Deep Learning with Convolutional Neural Nets](https://reader034.vdocuments.us/reader034/viewer/2022051113/55c12d29bb61eb893d8b45b3/html5/thumbnails/19.jpg)
http://googleresearch.blogspot.co.uk/2015/06/inceptionism-going-deeper-into-neural.html
![Page 20: Python for Image Understanding: Deep Learning with Convolutional Neural Nets](https://reader034.vdocuments.us/reader034/viewer/2022051113/55c12d29bb61eb893d8b45b3/html5/thumbnails/20.jpg)
http://googleresearch.blogspot.co.uk/2015/06/inceptionism-going-deeper-into-neural.html
![Page 21: Python for Image Understanding: Deep Learning with Convolutional Neural Nets](https://reader034.vdocuments.us/reader034/viewer/2022051113/55c12d29bb61eb893d8b45b3/html5/thumbnails/21.jpg)
![Page 22: Python for Image Understanding: Deep Learning with Convolutional Neural Nets](https://reader034.vdocuments.us/reader034/viewer/2022051113/55c12d29bb61eb893d8b45b3/html5/thumbnails/22.jpg)
How does Deep Learning work?
![Page 23: Python for Image Understanding: Deep Learning with Convolutional Neural Nets](https://reader034.vdocuments.us/reader034/viewer/2022051113/55c12d29bb61eb893d8b45b3/html5/thumbnails/23.jpg)
![Page 24: Python for Image Understanding: Deep Learning with Convolutional Neural Nets](https://reader034.vdocuments.us/reader034/viewer/2022051113/55c12d29bb61eb893d8b45b3/html5/thumbnails/24.jpg)
![Page 25: Python for Image Understanding: Deep Learning with Convolutional Neural Nets](https://reader034.vdocuments.us/reader034/viewer/2022051113/55c12d29bb61eb893d8b45b3/html5/thumbnails/25.jpg)
Activation Functions
Sigmoid
![Page 26: Python for Image Understanding: Deep Learning with Convolutional Neural Nets](https://reader034.vdocuments.us/reader034/viewer/2022051113/55c12d29bb61eb893d8b45b3/html5/thumbnails/26.jpg)
Activation Functions
Tanh
![Page 27: Python for Image Understanding: Deep Learning with Convolutional Neural Nets](https://reader034.vdocuments.us/reader034/viewer/2022051113/55c12d29bb61eb893d8b45b3/html5/thumbnails/27.jpg)
Activation Functions
ReLU
often approximated by just
![Page 28: Python for Image Understanding: Deep Learning with Convolutional Neural Nets](https://reader034.vdocuments.us/reader034/viewer/2022051113/55c12d29bb61eb893d8b45b3/html5/thumbnails/28.jpg)
optimization strategies
![Page 29: Python for Image Understanding: Deep Learning with Convolutional Neural Nets](https://reader034.vdocuments.us/reader034/viewer/2022051113/55c12d29bb61eb893d8b45b3/html5/thumbnails/29.jpg)
So what’s there for Python?
![Page 30: Python for Image Understanding: Deep Learning with Convolutional Neural Nets](https://reader034.vdocuments.us/reader034/viewer/2022051113/55c12d29bb61eb893d8b45b3/html5/thumbnails/30.jpg)
python has a wide range of deep learning-related libraries available
Deep Learning with Python
Low level
High level
(efficient gpu-powered math)
(computer-vision oriented DL framework, model-zoo, prototxt model definitions) pythonification ongoing!
(wrapper for theano, yaml, experiment-oriented)
(theano-wrapper, models in python code, abstracts theano away)
(theano-extension, models in python code, theano not hidden)
and of course:
![Page 31: Python for Image Understanding: Deep Learning with Convolutional Neural Nets](https://reader034.vdocuments.us/reader034/viewer/2022051113/55c12d29bb61eb893d8b45b3/html5/thumbnails/31.jpg)
python has a wide range of deep learning-related libraries available
Deep Learning with Python
Low level
High level
deeplearning.net/software/theano
caffe.berkeleyvision.org
deeplearning.net/software/pylearn2
keras.io
lasagne.readthedocs.org/en/latest
and of course:
![Page 32: Python for Image Understanding: Deep Learning with Convolutional Neural Nets](https://reader034.vdocuments.us/reader034/viewer/2022051113/55c12d29bb61eb893d8b45b3/html5/thumbnails/32.jpg)
python has a wide range of deep learning-related libraries available
Deep Learning with Python
Low level
High level
deeplearning.net/software/theano
caffe.berkeleyvision.org
deeplearning.net/software/pylearn2
keras.io
lasagne.readthedocs.org/en/latest
and of course:
we will use lasagne in our examples
![Page 33: Python for Image Understanding: Deep Learning with Convolutional Neural Nets](https://reader034.vdocuments.us/reader034/viewer/2022051113/55c12d29bb61eb893d8b45b3/html5/thumbnails/33.jpg)
Doing Deep Learning?
![Page 34: Python for Image Understanding: Deep Learning with Convolutional Neural Nets](https://reader034.vdocuments.us/reader034/viewer/2022051113/55c12d29bb61eb893d8b45b3/html5/thumbnails/34.jpg)
1. Preprocess the data
2. Choose architecture
3. Train
4. Optimize/Regularize
5. Tips/Tricks
Training a (deep) Neural Network
![Page 35: Python for Image Understanding: Deep Learning with Convolutional Neural Nets](https://reader034.vdocuments.us/reader034/viewer/2022051113/55c12d29bb61eb893d8b45b3/html5/thumbnails/35.jpg)
1. Preprocess the data
2. Choose architecture
3. Train
4. Optimize/Regularize
5. Tips/Tricks
Training a (deep) Neural Network
![Page 36: Python for Image Understanding: Deep Learning with Convolutional Neural Nets](https://reader034.vdocuments.us/reader034/viewer/2022051113/55c12d29bb61eb893d8b45b3/html5/thumbnails/36.jpg)
• Mean subtraction
• Normalization
• PCA and Whitening
1. Preprocess the data
![Page 37: Python for Image Understanding: Deep Learning with Convolutional Neural Nets](https://reader034.vdocuments.us/reader034/viewer/2022051113/55c12d29bb61eb893d8b45b3/html5/thumbnails/37.jpg)
• Mean subtraction
• Normalization
• PCA and Whitening
1. Preprocess the data: Normalization
(mean image visualised of cifar-10)
![Page 38: Python for Image Understanding: Deep Learning with Convolutional Neural Nets](https://reader034.vdocuments.us/reader034/viewer/2022051113/55c12d29bb61eb893d8b45b3/html5/thumbnails/38.jpg)
• Mean subtraction
• Normalization
• PCA and Whitening
1. Preprocess the data: PCA & Whitening
![Page 39: Python for Image Understanding: Deep Learning with Convolutional Neural Nets](https://reader034.vdocuments.us/reader034/viewer/2022051113/55c12d29bb61eb893d8b45b3/html5/thumbnails/39.jpg)
1. Preprocess the data: PCA & Whitening
![Page 40: Python for Image Understanding: Deep Learning with Convolutional Neural Nets](https://reader034.vdocuments.us/reader034/viewer/2022051113/55c12d29bb61eb893d8b45b3/html5/thumbnails/40.jpg)
![Page 41: Python for Image Understanding: Deep Learning with Convolutional Neural Nets](https://reader034.vdocuments.us/reader034/viewer/2022051113/55c12d29bb61eb893d8b45b3/html5/thumbnails/41.jpg)
1. Preprocess the data, the right way
Warning:
• compute preprocessing statistics on training data
• apply on all (training / validation / test) data
![Page 42: Python for Image Understanding: Deep Learning with Convolutional Neural Nets](https://reader034.vdocuments.us/reader034/viewer/2022051113/55c12d29bb61eb893d8b45b3/html5/thumbnails/42.jpg)
1. Preprocess the data
2. Choose architecture
3. Train
4. Optimize/Regularize
5. Tips/Tricks
Training a (deep) Neural Network
![Page 43: Python for Image Understanding: Deep Learning with Convolutional Neural Nets](https://reader034.vdocuments.us/reader034/viewer/2022051113/55c12d29bb61eb893d8b45b3/html5/thumbnails/43.jpg)
• Deep Belief Network (DBN)
• Convolutional Net (CNN)
• Recurrent Net (RNN)
2. Choosing the right architecture
![Page 44: Python for Image Understanding: Deep Learning with Convolutional Neural Nets](https://reader034.vdocuments.us/reader034/viewer/2022051113/55c12d29bb61eb893d8b45b3/html5/thumbnails/44.jpg)
• Deep Belief Network (DBN)
• Convolutional Net (CNN)
• Recurrent Net (RNN)
2. Choosing the right architecture
![Page 45: Python for Image Understanding: Deep Learning with Convolutional Neural Nets](https://reader034.vdocuments.us/reader034/viewer/2022051113/55c12d29bb61eb893d8b45b3/html5/thumbnails/45.jpg)
• Deep Belief Network (DBN)
• Convolutional Net (CNN)
• Recurrent Net (RNN)
2. Choosing the right architecture
![Page 46: Python for Image Understanding: Deep Learning with Convolutional Neural Nets](https://reader034.vdocuments.us/reader034/viewer/2022051113/55c12d29bb61eb893d8b45b3/html5/thumbnails/46.jpg)
• Deep Belief Network (DBN)
• Convolutional Net (CNN)
• Recurrent Net (RNN)
2. Choosing the right architecture
![Page 47: Python for Image Understanding: Deep Learning with Convolutional Neural Nets](https://reader034.vdocuments.us/reader034/viewer/2022051113/55c12d29bb61eb893d8b45b3/html5/thumbnails/47.jpg)
Convolutional Neural Net
Pbird
Psunset
Pdog
Pcat
![Page 48: Python for Image Understanding: Deep Learning with Convolutional Neural Nets](https://reader034.vdocuments.us/reader034/viewer/2022051113/55c12d29bb61eb893d8b45b3/html5/thumbnails/48.jpg)
Convolutional Neural Net
![Page 49: Python for Image Understanding: Deep Learning with Convolutional Neural Nets](https://reader034.vdocuments.us/reader034/viewer/2022051113/55c12d29bb61eb893d8b45b3/html5/thumbnails/49.jpg)
![Page 50: Python for Image Understanding: Deep Learning with Convolutional Neural Nets](https://reader034.vdocuments.us/reader034/viewer/2022051113/55c12d29bb61eb893d8b45b3/html5/thumbnails/50.jpg)
![Page 51: Python for Image Understanding: Deep Learning with Convolutional Neural Nets](https://reader034.vdocuments.us/reader034/viewer/2022051113/55c12d29bb61eb893d8b45b3/html5/thumbnails/51.jpg)
DrawCNN: visualizing the units' connections
Agrawal, et al. Analyzing the performance of multilayer neural networks for object recognition. ECCV, 2014 Szegedy, et al. Intriguing properties of neural networks.arXiv preprint arXiv:1312.6199, 2013
Zeiler, M. et al. Visualizing and Understanding Convolutional Networks, ECCV 2014
![Page 52: Python for Image Understanding: Deep Learning with Convolutional Neural Nets](https://reader034.vdocuments.us/reader034/viewer/2022051113/55c12d29bb61eb893d8b45b3/html5/thumbnails/52.jpg)
1. Preprocess the data
2. Choose architecture
3. Train
4. Optimize/Regularize
5. Tips/Tricks
Training a (deep) Neural Network
![Page 53: Python for Image Understanding: Deep Learning with Convolutional Neural Nets](https://reader034.vdocuments.us/reader034/viewer/2022051113/55c12d29bb61eb893d8b45b3/html5/thumbnails/53.jpg)
1. Preprocess the data
2. Choose architecture
3. Train (Code Finally!)
4. Optimize/Regularize
5. Tips/Tricks
Training a (deep) Neural Network
![Page 54: Python for Image Understanding: Deep Learning with Convolutional Neural Nets](https://reader034.vdocuments.us/reader034/viewer/2022051113/55c12d29bb61eb893d8b45b3/html5/thumbnails/54.jpg)
Training a (deep) Neural Network
(…)
layer definitions
layerparameters
![Page 55: Python for Image Understanding: Deep Learning with Convolutional Neural Nets](https://reader034.vdocuments.us/reader034/viewer/2022051113/55c12d29bb61eb893d8b45b3/html5/thumbnails/55.jpg)
Training a (deep) Neural Network
hyperparameters
![Page 56: Python for Image Understanding: Deep Learning with Convolutional Neural Nets](https://reader034.vdocuments.us/reader034/viewer/2022051113/55c12d29bb61eb893d8b45b3/html5/thumbnails/56.jpg)
Training a (deep) Neural Network
Lift off!
![Page 57: Python for Image Understanding: Deep Learning with Convolutional Neural Nets](https://reader034.vdocuments.us/reader034/viewer/2022051113/55c12d29bb61eb893d8b45b3/html5/thumbnails/57.jpg)
Training a (deep) Neural Network
1. Preprocess the data
2. Choose architecture
3. Train
4. Optimize/Regularize
5. Tips/Tricks
Debug
![Page 58: Python for Image Understanding: Deep Learning with Convolutional Neural Nets](https://reader034.vdocuments.us/reader034/viewer/2022051113/55c12d29bb61eb893d8b45b3/html5/thumbnails/58.jpg)
Debug Training: Visualize Loss Curve
![Page 59: Python for Image Understanding: Deep Learning with Convolutional Neural Nets](https://reader034.vdocuments.us/reader034/viewer/2022051113/55c12d29bb61eb893d8b45b3/html5/thumbnails/59.jpg)
Debug Training: Visualize Loss Curve
![Page 60: Python for Image Understanding: Deep Learning with Convolutional Neural Nets](https://reader034.vdocuments.us/reader034/viewer/2022051113/55c12d29bb61eb893d8b45b3/html5/thumbnails/60.jpg)
Debug Training: Visualize Loss Curve
This looks (too) linear: lower your learning rate!
This looks (too) wide: increase your batch size!
![Page 61: Python for Image Understanding: Deep Learning with Convolutional Neural Nets](https://reader034.vdocuments.us/reader034/viewer/2022051113/55c12d29bb61eb893d8b45b3/html5/thumbnails/61.jpg)
Debug Training: Visualize Accuracy
big gap: overfitting: regularize!
no gap: underfitting (increase model size)
![Page 62: Python for Image Understanding: Deep Learning with Convolutional Neural Nets](https://reader034.vdocuments.us/reader034/viewer/2022051113/55c12d29bb61eb893d8b45b3/html5/thumbnails/62.jpg)
Debug Training: Visualize Weights
(usually: first layer)
complete mess, doesn't get past the random initialisation
![Page 63: Python for Image Understanding: Deep Learning with Convolutional Neural Nets](https://reader034.vdocuments.us/reader034/viewer/2022051113/55c12d29bb61eb893d8b45b3/html5/thumbnails/63.jpg)
Debug Training: Visualize Weights
(usually: first layer)
better, but still “noisy” weights
mostly solvable by stronger
regularisation
![Page 64: Python for Image Understanding: Deep Learning with Convolutional Neural Nets](https://reader034.vdocuments.us/reader034/viewer/2022051113/55c12d29bb61eb893d8b45b3/html5/thumbnails/64.jpg)
Debug Training: Visualize Weights
(usually: first layer)
good: now operates as “edge detector”
![Page 65: Python for Image Understanding: Deep Learning with Convolutional Neural Nets](https://reader034.vdocuments.us/reader034/viewer/2022051113/55c12d29bb61eb893d8b45b3/html5/thumbnails/65.jpg)
Training a (deep) Neural Network
1. Preprocess the data
2. Choose architecture
3. Train
4. Optimize/Regularize
5. Tips/Tricks
Debug
![Page 66: Python for Image Understanding: Deep Learning with Convolutional Neural Nets](https://reader034.vdocuments.us/reader034/viewer/2022051113/55c12d29bb61eb893d8b45b3/html5/thumbnails/66.jpg)
• Tweak Hyperparameters / Architecture
• Data Augmentation
• Dropout
• Batch Normalization
Optimize / Regularize
![Page 67: Python for Image Understanding: Deep Learning with Convolutional Neural Nets](https://reader034.vdocuments.us/reader034/viewer/2022051113/55c12d29bb61eb893d8b45b3/html5/thumbnails/67.jpg)
• Tweak Hyperparameters / Architecture
• Data Augmentation
• Dropout
• Batch Normalization
Optimize / Regularize
![Page 68: Python for Image Understanding: Deep Learning with Convolutional Neural Nets](https://reader034.vdocuments.us/reader034/viewer/2022051113/55c12d29bb61eb893d8b45b3/html5/thumbnails/68.jpg)
• Grid search won't work on your millions + parameters
• Random Search? Mwah…
• Bayesian Optimization: Yeh baby!
• Spearmint: https://github.com/HIPS/Spearmint
• Hypergrad: https://github.com/HIPS/hypergrad
Choosing Hyperparameters
![Page 69: Python for Image Understanding: Deep Learning with Convolutional Neural Nets](https://reader034.vdocuments.us/reader034/viewer/2022051113/55c12d29bb61eb893d8b45b3/html5/thumbnails/69.jpg)
• Tweak Hyperparameters / Architecture
• Data Augmentation
• Dropout
• Batch Normalization
Overfitting
![Page 70: Python for Image Understanding: Deep Learning with Convolutional Neural Nets](https://reader034.vdocuments.us/reader034/viewer/2022051113/55c12d29bb61eb893d8b45b3/html5/thumbnails/70.jpg)
Data Augmentation
![Page 71: Python for Image Understanding: Deep Learning with Convolutional Neural Nets](https://reader034.vdocuments.us/reader034/viewer/2022051113/55c12d29bb61eb893d8b45b3/html5/thumbnails/71.jpg)
Data Augmentation
http://benanne.github.io/2015/03/17/plankton.html
(realtime data augmentation at Kaggle’s #1 National Data Science Bowl≋ Deep Sea ≋ team)
rotation: random with angle between 0° and 360° (uniform)translation: random with shift between -10 and 10 pixels (uniform)rescaling: random with scale factor between 1/1.6 and 1.6 (log-uniform)flipping: yes or no (bernoulli)shearing: random with angle between -20° and 20° (uniform)stretching: random with stretch factor between 1/1.3 and 1.3 (log-uniform)
![Page 72: Python for Image Understanding: Deep Learning with Convolutional Neural Nets](https://reader034.vdocuments.us/reader034/viewer/2022051113/55c12d29bb61eb893d8b45b3/html5/thumbnails/72.jpg)
• Tweak Hyperparameters / Architecture
• Data Augmentation
• Dropout
• Batch Normalization
Optimize / Regularize
![Page 73: Python for Image Understanding: Deep Learning with Convolutional Neural Nets](https://reader034.vdocuments.us/reader034/viewer/2022051113/55c12d29bb61eb893d8b45b3/html5/thumbnails/73.jpg)
Dropout as Regularization
(naively trained net)
Overfits !
![Page 74: Python for Image Understanding: Deep Learning with Convolutional Neural Nets](https://reader034.vdocuments.us/reader034/viewer/2022051113/55c12d29bb61eb893d8b45b3/html5/thumbnails/74.jpg)
Dropout as Regularization
(naively trained net)
Overfits !
Dropout!
![Page 75: Python for Image Understanding: Deep Learning with Convolutional Neural Nets](https://reader034.vdocuments.us/reader034/viewer/2022051113/55c12d29bb61eb893d8b45b3/html5/thumbnails/75.jpg)
Dropout as Regularization
(naively trained net)
Overfits !
Dropout!
![Page 76: Python for Image Understanding: Deep Learning with Convolutional Neural Nets](https://reader034.vdocuments.us/reader034/viewer/2022051113/55c12d29bb61eb893d8b45b3/html5/thumbnails/76.jpg)
Dropout as Regularization
(naively trained net) (net with dropout)
less strongly overfitted & can run for more epochs higher accuracy
![Page 77: Python for Image Understanding: Deep Learning with Convolutional Neural Nets](https://reader034.vdocuments.us/reader034/viewer/2022051113/55c12d29bb61eb893d8b45b3/html5/thumbnails/77.jpg)
Overfitting
• Tweak Hyperparameters / Architecture
• Data Augmentation
• Dropout
• Batch Normalization
![Page 78: Python for Image Understanding: Deep Learning with Convolutional Neural Nets](https://reader034.vdocuments.us/reader034/viewer/2022051113/55c12d29bb61eb893d8b45b3/html5/thumbnails/78.jpg)
• Normalize the activations in each layer within a minibatch
• Learn the mean and variance of each layer as parameters
Batch Normalization as regularization
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift S Ioffe and C Szegedy (2015)
85%
50%
15%
![Page 79: Python for Image Understanding: Deep Learning with Convolutional Neural Nets](https://reader034.vdocuments.us/reader034/viewer/2022051113/55c12d29bb61eb893d8b45b3/html5/thumbnails/79.jpg)
Training a (deep) Neural Network
1. Preprocess the data
2. Choose architecture
3. Train
4. Optimize/Regularize
5. Further Tips & Tricks to improve Model Accuracy
Debug
![Page 80: Python for Image Understanding: Deep Learning with Convolutional Neural Nets](https://reader034.vdocuments.us/reader034/viewer/2022051113/55c12d29bb61eb893d8b45b3/html5/thumbnails/80.jpg)
• Ensembles
• Finetuning pre-trained/earlier-trained net
• Sticking extracted layer features in another classifier (ie SVM)
Other “Tricks”
![Page 81: Python for Image Understanding: Deep Learning with Convolutional Neural Nets](https://reader034.vdocuments.us/reader034/viewer/2022051113/55c12d29bb61eb893d8b45b3/html5/thumbnails/81.jpg)
• Ensembles
• Finetuning pre-trained/earlier-trained net
• Sticking extracted layer features in another classifier (ie SVM)
Other “Tricks”
![Page 82: Python for Image Understanding: Deep Learning with Convolutional Neural Nets](https://reader034.vdocuments.us/reader034/viewer/2022051113/55c12d29bb61eb893d8b45b3/html5/thumbnails/82.jpg)
Ensembles
• majority vote when hard predictions (ie classes)• average vote for soft predictions (continious scale)• make sure classifiers are uncorrelated• cross validate ensemble weights (by grid search, or
rank average)• stacked• blending
![Page 83: Python for Image Understanding: Deep Learning with Convolutional Neural Nets](https://reader034.vdocuments.us/reader034/viewer/2022051113/55c12d29bb61eb893d8b45b3/html5/thumbnails/83.jpg)
Ensembles
(10 similar nets with varying hyperparameters on same tiny-imagenet dataset)
![Page 84: Python for Image Understanding: Deep Learning with Convolutional Neural Nets](https://reader034.vdocuments.us/reader034/viewer/2022051113/55c12d29bb61eb893d8b45b3/html5/thumbnails/84.jpg)
avg: 0.3647
Ensembles
(10 similar nets with varying hyperparameters on same tiny-imagenet dataset)
![Page 85: Python for Image Understanding: Deep Learning with Convolutional Neural Nets](https://reader034.vdocuments.us/reader034/viewer/2022051113/55c12d29bb61eb893d8b45b3/html5/thumbnails/85.jpg)
predict by mean of all: 0.4244
avg: 0.3647
Ensembles
(10 similar nets with varying hyperparameters on same tiny-imagenet dataset)
![Page 86: Python for Image Understanding: Deep Learning with Convolutional Neural Nets](https://reader034.vdocuments.us/reader034/viewer/2022051113/55c12d29bb61eb893d8b45b3/html5/thumbnails/86.jpg)
(10 similar nets with varying hyperparameters on same tiny-imagenet dataset)
predict by mean of all: 0.4244
leave out model9: 0.4259
avg: 0.3647
Ensembles
![Page 87: Python for Image Understanding: Deep Learning with Convolutional Neural Nets](https://reader034.vdocuments.us/reader034/viewer/2022051113/55c12d29bb61eb893d8b45b3/html5/thumbnails/87.jpg)
(10 similar nets with varying hyperparameters on same tiny-imagenet dataset)
0.4259 0.4244
0.3647
Ensembles
![Page 88: Python for Image Understanding: Deep Learning with Convolutional Neural Nets](https://reader034.vdocuments.us/reader034/viewer/2022051113/55c12d29bb61eb893d8b45b3/html5/thumbnails/88.jpg)
danielnouri.org/notes/2014/12/17/using-convolutional-neural-nets-to-detect-facial-keypoints-tutorial/(ensembling specialist nets by Daniel Nouri, Kaggle facial keypoint tutorial)
Ensembles
“Specialists” Ensemble
![Page 89: Python for Image Understanding: Deep Learning with Convolutional Neural Nets](https://reader034.vdocuments.us/reader034/viewer/2022051113/55c12d29bb61eb893d8b45b3/html5/thumbnails/89.jpg)
https://www.kaggle.com/c/facial-keypoints-detection/
try it yourself :)3 similar nets trained on the same data but with different hyper parameters.
RMSE’s: •2,08449 •2,04575 •2.01565
together: 1.93397
disclaimer: Kaggle is not real life, people…
![Page 90: Python for Image Understanding: Deep Learning with Convolutional Neural Nets](https://reader034.vdocuments.us/reader034/viewer/2022051113/55c12d29bb61eb893d8b45b3/html5/thumbnails/90.jpg)
Machine learning systems can easily be
fooled
but beware… / no free lunch:
![Page 91: Python for Image Understanding: Deep Learning with Convolutional Neural Nets](https://reader034.vdocuments.us/reader034/viewer/2022051113/55c12d29bb61eb893d8b45b3/html5/thumbnails/91.jpg)
Saliency Maps
K. Simonyan, A. Vedaldi, A. Zisserman , "Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps", ICLR Workshop 2014
first we predict on a pixel level
![Page 92: Python for Image Understanding: Deep Learning with Convolutional Neural Nets](https://reader034.vdocuments.us/reader034/viewer/2022051113/55c12d29bb61eb893d8b45b3/html5/thumbnails/92.jpg)
Fooling ConvNets
Szegedy, Christian, et al. "Intriguing properties of neural networks." arXiv preprint, 2013. Nguyen, Anh, Jason Yosinski, and Jeff Clune. "Deep Neural Networks are Easily Fooled: High
Confidence Predictions for Unrecognizable Images." arXiv preprint
then we do our “magic”
![Page 93: Python for Image Understanding: Deep Learning with Convolutional Neural Nets](https://reader034.vdocuments.us/reader034/viewer/2022051113/55c12d29bb61eb893d8b45b3/html5/thumbnails/93.jpg)
Fooling ConvNets
Szegedy, Christian, et al. "Intriguing properties of neural networks." arXiv preprint, 2013. Nguyen, Anh, Jason Yosinski, and Jeff Clune. "Deep Neural Networks are Easily Fooled: High
Confidence Predictions for Unrecognizable Images." arXiv preprint
then we do our “magic”
![Page 94: Python for Image Understanding: Deep Learning with Convolutional Neural Nets](https://reader034.vdocuments.us/reader034/viewer/2022051113/55c12d29bb61eb893d8b45b3/html5/thumbnails/94.jpg)
Failing ConvNets
![Page 95: Python for Image Understanding: Deep Learning with Convolutional Neural Nets](https://reader034.vdocuments.us/reader034/viewer/2022051113/55c12d29bb61eb893d8b45b3/html5/thumbnails/95.jpg)
Failing ConvNets
“Suddenly, a leopard print sofa appears”, rocknrollnerd.github.io
![Page 96: Python for Image Understanding: Deep Learning with Convolutional Neural Nets](https://reader034.vdocuments.us/reader034/viewer/2022051113/55c12d29bb61eb893d8b45b3/html5/thumbnails/96.jpg)
Failing ConvNets
“Suddenly, a leopard print sofa appears”, rocknrollnerd.github.io
![Page 97: Python for Image Understanding: Deep Learning with Convolutional Neural Nets](https://reader034.vdocuments.us/reader034/viewer/2022051113/55c12d29bb61eb893d8b45b3/html5/thumbnails/97.jpg)
Failing ConvNets
“Suddenly, a leopard print sofa appears”, rocknrollnerd.github.io
![Page 98: Python for Image Understanding: Deep Learning with Convolutional Neural Nets](https://reader034.vdocuments.us/reader034/viewer/2022051113/55c12d29bb61eb893d8b45b3/html5/thumbnails/98.jpg)
Failing ConvNets
![Page 99: Python for Image Understanding: Deep Learning with Convolutional Neural Nets](https://reader034.vdocuments.us/reader034/viewer/2022051113/55c12d29bb61eb893d8b45b3/html5/thumbnails/99.jpg)
thanks for listening ;)questions?or find me @graphific
![Page 100: Python for Image Understanding: Deep Learning with Convolutional Neural Nets](https://reader034.vdocuments.us/reader034/viewer/2022051113/55c12d29bb61eb893d8b45b3/html5/thumbnails/100.jpg)
• Computer Vision: Fei-Fei Li & Andrej Karpathy, Stanford course “Convolutional Neural Networks for Visual Recognition” http://vision.stanford.edu/teaching/cs231n
• Natural Language Processing: Richard Socher, Stanford course “Deep Learning for Natural Language Processing”, http://cs224d.stanford.edu/
• Neural Nets: Geoffrey Hinton, Coursera/Toronto, “Neural Networks for Machine Learning"https://www.coursera.org/course/neuralnets
Wanna Play?
• Bunch of tutorials: http://deeplearning.net/tutorial/
• Book: Yoshua Bengio, et al, “Deep Learning” http://www.iro.umontreal.ca/~bengioy/dlbook/
• UFLDL Tutorial http://deeplearning.stanford.edu/tutorial/
• Reading Lists: http://deeplearning.net/reading-list/ http://memkite.com/deep-learning-bibliography/
• PodcastTalking Machines, http://www.thetalkingmachines.com/