pooling and invariance in convolutional neural networkstheorycenter.cs.uchicago.edu/reu/2014/... ·...
TRANSCRIPT
![Page 1: Pooling and Invariance in Convolutional Neural Networkstheorycenter.cs.uchicago.edu/REU/2014/... · CNN Presentation.pptx Author: Stuart Kurtz Created Date: 10/3/2014 12:59:37 PM](https://reader035.vdocuments.us/reader035/viewer/2022071116/5ffc87f1138cee62851f020e/html5/thumbnails/1.jpg)
Pooling and Invariance in Convolutional Neural Networks
![Page 2: Pooling and Invariance in Convolutional Neural Networkstheorycenter.cs.uchicago.edu/REU/2014/... · CNN Presentation.pptx Author: Stuart Kurtz Created Date: 10/3/2014 12:59:37 PM](https://reader035.vdocuments.us/reader035/viewer/2022071116/5ffc87f1138cee62851f020e/html5/thumbnails/2.jpg)
General Neural Networks Compositions of linear maps and component-wise non-linearities
![Page 3: Pooling and Invariance in Convolutional Neural Networkstheorycenter.cs.uchicago.edu/REU/2014/... · CNN Presentation.pptx Author: Stuart Kurtz Created Date: 10/3/2014 12:59:37 PM](https://reader035.vdocuments.us/reader035/viewer/2022071116/5ffc87f1138cee62851f020e/html5/thumbnails/3.jpg)
Neural Networks
![Page 4: Pooling and Invariance in Convolutional Neural Networkstheorycenter.cs.uchicago.edu/REU/2014/... · CNN Presentation.pptx Author: Stuart Kurtz Created Date: 10/3/2014 12:59:37 PM](https://reader035.vdocuments.us/reader035/viewer/2022071116/5ffc87f1138cee62851f020e/html5/thumbnails/4.jpg)
Common Non-Linearities Rectifier Linear Unit Sigmoid Hyperbolic tangent
![Page 5: Pooling and Invariance in Convolutional Neural Networkstheorycenter.cs.uchicago.edu/REU/2014/... · CNN Presentation.pptx Author: Stuart Kurtz Created Date: 10/3/2014 12:59:37 PM](https://reader035.vdocuments.us/reader035/viewer/2022071116/5ffc87f1138cee62851f020e/html5/thumbnails/5.jpg)
Biological Inspiration Neurons diagram Rectified Linear Unit
![Page 6: Pooling and Invariance in Convolutional Neural Networkstheorycenter.cs.uchicago.edu/REU/2014/... · CNN Presentation.pptx Author: Stuart Kurtz Created Date: 10/3/2014 12:59:37 PM](https://reader035.vdocuments.us/reader035/viewer/2022071116/5ffc87f1138cee62851f020e/html5/thumbnails/6.jpg)
Sparse Connections Not required to be fully connected
![Page 7: Pooling and Invariance in Convolutional Neural Networkstheorycenter.cs.uchicago.edu/REU/2014/... · CNN Presentation.pptx Author: Stuart Kurtz Created Date: 10/3/2014 12:59:37 PM](https://reader035.vdocuments.us/reader035/viewer/2022071116/5ffc87f1138cee62851f020e/html5/thumbnails/7.jpg)
Backpropagation
![Page 8: Pooling and Invariance in Convolutional Neural Networkstheorycenter.cs.uchicago.edu/REU/2014/... · CNN Presentation.pptx Author: Stuart Kurtz Created Date: 10/3/2014 12:59:37 PM](https://reader035.vdocuments.us/reader035/viewer/2022071116/5ffc87f1138cee62851f020e/html5/thumbnails/8.jpg)
Representations Learnt
![Page 9: Pooling and Invariance in Convolutional Neural Networkstheorycenter.cs.uchicago.edu/REU/2014/... · CNN Presentation.pptx Author: Stuart Kurtz Created Date: 10/3/2014 12:59:37 PM](https://reader035.vdocuments.us/reader035/viewer/2022071116/5ffc87f1138cee62851f020e/html5/thumbnails/9.jpg)
Representation Learning Images far apart in Euclidean space Need to find representation such that members of same class are mapped to similar values
![Page 10: Pooling and Invariance in Convolutional Neural Networkstheorycenter.cs.uchicago.edu/REU/2014/... · CNN Presentation.pptx Author: Stuart Kurtz Created Date: 10/3/2014 12:59:37 PM](https://reader035.vdocuments.us/reader035/viewer/2022071116/5ffc87f1138cee62851f020e/html5/thumbnails/10.jpg)
Convolutional Neural Network Compositions of: Convolutions by Linear Filter Thresholding Non-Linearities Spatial Pooling
![Page 11: Pooling and Invariance in Convolutional Neural Networkstheorycenter.cs.uchicago.edu/REU/2014/... · CNN Presentation.pptx Author: Stuart Kurtz Created Date: 10/3/2014 12:59:37 PM](https://reader035.vdocuments.us/reader035/viewer/2022071116/5ffc87f1138cee62851f020e/html5/thumbnails/11.jpg)
Convolution by Linear Filter
![Page 12: Pooling and Invariance in Convolutional Neural Networkstheorycenter.cs.uchicago.edu/REU/2014/... · CNN Presentation.pptx Author: Stuart Kurtz Created Date: 10/3/2014 12:59:37 PM](https://reader035.vdocuments.us/reader035/viewer/2022071116/5ffc87f1138cee62851f020e/html5/thumbnails/12.jpg)
Convolution by Linear Filter
![Page 13: Pooling and Invariance in Convolutional Neural Networkstheorycenter.cs.uchicago.edu/REU/2014/... · CNN Presentation.pptx Author: Stuart Kurtz Created Date: 10/3/2014 12:59:37 PM](https://reader035.vdocuments.us/reader035/viewer/2022071116/5ffc87f1138cee62851f020e/html5/thumbnails/13.jpg)
Example
![Page 14: Pooling and Invariance in Convolutional Neural Networkstheorycenter.cs.uchicago.edu/REU/2014/... · CNN Presentation.pptx Author: Stuart Kurtz Created Date: 10/3/2014 12:59:37 PM](https://reader035.vdocuments.us/reader035/viewer/2022071116/5ffc87f1138cee62851f020e/html5/thumbnails/14.jpg)
Convolutional Neural Networks 1) Convolution by Linear Filter 2) Apply non-linearity 3) Pooling
![Page 15: Pooling and Invariance in Convolutional Neural Networkstheorycenter.cs.uchicago.edu/REU/2014/... · CNN Presentation.pptx Author: Stuart Kurtz Created Date: 10/3/2014 12:59:37 PM](https://reader035.vdocuments.us/reader035/viewer/2022071116/5ffc87f1138cee62851f020e/html5/thumbnails/15.jpg)
Convolutional Neural Network
![Page 16: Pooling and Invariance in Convolutional Neural Networkstheorycenter.cs.uchicago.edu/REU/2014/... · CNN Presentation.pptx Author: Stuart Kurtz Created Date: 10/3/2014 12:59:37 PM](https://reader035.vdocuments.us/reader035/viewer/2022071116/5ffc87f1138cee62851f020e/html5/thumbnails/16.jpg)
Convolutional Neural Networks
![Page 17: Pooling and Invariance in Convolutional Neural Networkstheorycenter.cs.uchicago.edu/REU/2014/... · CNN Presentation.pptx Author: Stuart Kurtz Created Date: 10/3/2014 12:59:37 PM](https://reader035.vdocuments.us/reader035/viewer/2022071116/5ffc87f1138cee62851f020e/html5/thumbnails/17.jpg)
Imagenet Network
![Page 18: Pooling and Invariance in Convolutional Neural Networkstheorycenter.cs.uchicago.edu/REU/2014/... · CNN Presentation.pptx Author: Stuart Kurtz Created Date: 10/3/2014 12:59:37 PM](https://reader035.vdocuments.us/reader035/viewer/2022071116/5ffc87f1138cee62851f020e/html5/thumbnails/18.jpg)
Successes Computer vision Speech Chemistry
![Page 19: Pooling and Invariance in Convolutional Neural Networkstheorycenter.cs.uchicago.edu/REU/2014/... · CNN Presentation.pptx Author: Stuart Kurtz Created Date: 10/3/2014 12:59:37 PM](https://reader035.vdocuments.us/reader035/viewer/2022071116/5ffc87f1138cee62851f020e/html5/thumbnails/19.jpg)
Object Classification a
![Page 20: Pooling and Invariance in Convolutional Neural Networkstheorycenter.cs.uchicago.edu/REU/2014/... · CNN Presentation.pptx Author: Stuart Kurtz Created Date: 10/3/2014 12:59:37 PM](https://reader035.vdocuments.us/reader035/viewer/2022071116/5ffc87f1138cee62851f020e/html5/thumbnails/20.jpg)
Segmentation
![Page 21: Pooling and Invariance in Convolutional Neural Networkstheorycenter.cs.uchicago.edu/REU/2014/... · CNN Presentation.pptx Author: Stuart Kurtz Created Date: 10/3/2014 12:59:37 PM](https://reader035.vdocuments.us/reader035/viewer/2022071116/5ffc87f1138cee62851f020e/html5/thumbnails/21.jpg)
Object Detection
![Page 22: Pooling and Invariance in Convolutional Neural Networkstheorycenter.cs.uchicago.edu/REU/2014/... · CNN Presentation.pptx Author: Stuart Kurtz Created Date: 10/3/2014 12:59:37 PM](https://reader035.vdocuments.us/reader035/viewer/2022071116/5ffc87f1138cee62851f020e/html5/thumbnails/22.jpg)
Speech
![Page 23: Pooling and Invariance in Convolutional Neural Networkstheorycenter.cs.uchicago.edu/REU/2014/... · CNN Presentation.pptx Author: Stuart Kurtz Created Date: 10/3/2014 12:59:37 PM](https://reader035.vdocuments.us/reader035/viewer/2022071116/5ffc87f1138cee62851f020e/html5/thumbnails/23.jpg)
Physical Chemistry Successfully predict atomization energy, polarizability, frontier orbital eigenvalues, ionization potential, electron affinity and excitation energies from molecular structure.
![Page 24: Pooling and Invariance in Convolutional Neural Networkstheorycenter.cs.uchicago.edu/REU/2014/... · CNN Presentation.pptx Author: Stuart Kurtz Created Date: 10/3/2014 12:59:37 PM](https://reader035.vdocuments.us/reader035/viewer/2022071116/5ffc87f1138cee62851f020e/html5/thumbnails/24.jpg)
Visualization of First Layer
![Page 25: Pooling and Invariance in Convolutional Neural Networkstheorycenter.cs.uchicago.edu/REU/2014/... · CNN Presentation.pptx Author: Stuart Kurtz Created Date: 10/3/2014 12:59:37 PM](https://reader035.vdocuments.us/reader035/viewer/2022071116/5ffc87f1138cee62851f020e/html5/thumbnails/25.jpg)
Standard Pooling Mechanisms Ave pooling Max pooling
![Page 26: Pooling and Invariance in Convolutional Neural Networkstheorycenter.cs.uchicago.edu/REU/2014/... · CNN Presentation.pptx Author: Stuart Kurtz Created Date: 10/3/2014 12:59:37 PM](https://reader035.vdocuments.us/reader035/viewer/2022071116/5ffc87f1138cee62851f020e/html5/thumbnails/26.jpg)
Example
![Page 27: Pooling and Invariance in Convolutional Neural Networkstheorycenter.cs.uchicago.edu/REU/2014/... · CNN Presentation.pptx Author: Stuart Kurtz Created Date: 10/3/2014 12:59:37 PM](https://reader035.vdocuments.us/reader035/viewer/2022071116/5ffc87f1138cee62851f020e/html5/thumbnails/27.jpg)
Heterogenous Pooling Some filters passed to Ave pooling Others filters passed to Max pooling
![Page 28: Pooling and Invariance in Convolutional Neural Networkstheorycenter.cs.uchicago.edu/REU/2014/... · CNN Presentation.pptx Author: Stuart Kurtz Created Date: 10/3/2014 12:59:37 PM](https://reader035.vdocuments.us/reader035/viewer/2022071116/5ffc87f1138cee62851f020e/html5/thumbnails/28.jpg)
Pooling Continuum
Accordingly, LeCun et al. 2012 ran experiment with variety of “p” values.
![Page 29: Pooling and Invariance in Convolutional Neural Networkstheorycenter.cs.uchicago.edu/REU/2014/... · CNN Presentation.pptx Author: Stuart Kurtz Created Date: 10/3/2014 12:59:37 PM](https://reader035.vdocuments.us/reader035/viewer/2022071116/5ffc87f1138cee62851f020e/html5/thumbnails/29.jpg)
Results along spectrum
Optimal for this SVHN dataset was p = 4.
![Page 30: Pooling and Invariance in Convolutional Neural Networkstheorycenter.cs.uchicago.edu/REU/2014/... · CNN Presentation.pptx Author: Stuart Kurtz Created Date: 10/3/2014 12:59:37 PM](https://reader035.vdocuments.us/reader035/viewer/2022071116/5ffc87f1138cee62851f020e/html5/thumbnails/30.jpg)
L_p learnt pooling Why not learn optimal p for each filter map?
![Page 31: Pooling and Invariance in Convolutional Neural Networkstheorycenter.cs.uchicago.edu/REU/2014/... · CNN Presentation.pptx Author: Stuart Kurtz Created Date: 10/3/2014 12:59:37 PM](https://reader035.vdocuments.us/reader035/viewer/2022071116/5ffc87f1138cee62851f020e/html5/thumbnails/31.jpg)
Stochastic Pooling
![Page 32: Pooling and Invariance in Convolutional Neural Networkstheorycenter.cs.uchicago.edu/REU/2014/... · CNN Presentation.pptx Author: Stuart Kurtz Created Date: 10/3/2014 12:59:37 PM](https://reader035.vdocuments.us/reader035/viewer/2022071116/5ffc87f1138cee62851f020e/html5/thumbnails/32.jpg)
Stochastic Pooling Expectation at Test Time
![Page 33: Pooling and Invariance in Convolutional Neural Networkstheorycenter.cs.uchicago.edu/REU/2014/... · CNN Presentation.pptx Author: Stuart Kurtz Created Date: 10/3/2014 12:59:37 PM](https://reader035.vdocuments.us/reader035/viewer/2022071116/5ffc87f1138cee62851f020e/html5/thumbnails/33.jpg)
Entropy Pooling Extend to variable p In particular, Alternative:
![Page 34: Pooling and Invariance in Convolutional Neural Networkstheorycenter.cs.uchicago.edu/REU/2014/... · CNN Presentation.pptx Author: Stuart Kurtz Created Date: 10/3/2014 12:59:37 PM](https://reader035.vdocuments.us/reader035/viewer/2022071116/5ffc87f1138cee62851f020e/html5/thumbnails/34.jpg)
Max-Out pooling Pooling across filters Substantial Improvement in performance and allowed depth
![Page 35: Pooling and Invariance in Convolutional Neural Networkstheorycenter.cs.uchicago.edu/REU/2014/... · CNN Presentation.pptx Author: Stuart Kurtz Created Date: 10/3/2014 12:59:37 PM](https://reader035.vdocuments.us/reader035/viewer/2022071116/5ffc87f1138cee62851f020e/html5/thumbnails/35.jpg)
Example
![Page 36: Pooling and Invariance in Convolutional Neural Networkstheorycenter.cs.uchicago.edu/REU/2014/... · CNN Presentation.pptx Author: Stuart Kurtz Created Date: 10/3/2014 12:59:37 PM](https://reader035.vdocuments.us/reader035/viewer/2022071116/5ffc87f1138cee62851f020e/html5/thumbnails/36.jpg)
Compete Neurons Neurons can suppress other responses
![Page 37: Pooling and Invariance in Convolutional Neural Networkstheorycenter.cs.uchicago.edu/REU/2014/... · CNN Presentation.pptx Author: Stuart Kurtz Created Date: 10/3/2014 12:59:37 PM](https://reader035.vdocuments.us/reader035/viewer/2022071116/5ffc87f1138cee62851f020e/html5/thumbnails/37.jpg)
Visualizations of Filters Early Layers
![Page 38: Pooling and Invariance in Convolutional Neural Networkstheorycenter.cs.uchicago.edu/REU/2014/... · CNN Presentation.pptx Author: Stuart Kurtz Created Date: 10/3/2014 12:59:37 PM](https://reader035.vdocuments.us/reader035/viewer/2022071116/5ffc87f1138cee62851f020e/html5/thumbnails/38.jpg)
Visualizations
![Page 39: Pooling and Invariance in Convolutional Neural Networkstheorycenter.cs.uchicago.edu/REU/2014/... · CNN Presentation.pptx Author: Stuart Kurtz Created Date: 10/3/2014 12:59:37 PM](https://reader035.vdocuments.us/reader035/viewer/2022071116/5ffc87f1138cee62851f020e/html5/thumbnails/39.jpg)
Visualizations
![Page 40: Pooling and Invariance in Convolutional Neural Networkstheorycenter.cs.uchicago.edu/REU/2014/... · CNN Presentation.pptx Author: Stuart Kurtz Created Date: 10/3/2014 12:59:37 PM](https://reader035.vdocuments.us/reader035/viewer/2022071116/5ffc87f1138cee62851f020e/html5/thumbnails/40.jpg)
Invariance under Rigid Motion
Goodfellow et al. 2009 demonstrated the CNN are invariant under Indeed, depth of NN critical to establishing such invariance
![Page 41: Pooling and Invariance in Convolutional Neural Networkstheorycenter.cs.uchicago.edu/REU/2014/... · CNN Presentation.pptx Author: Stuart Kurtz Created Date: 10/3/2014 12:59:37 PM](https://reader035.vdocuments.us/reader035/viewer/2022071116/5ffc87f1138cee62851f020e/html5/thumbnails/41.jpg)
Unstable under Deformation Szegedy et al.
![Page 42: Pooling and Invariance in Convolutional Neural Networkstheorycenter.cs.uchicago.edu/REU/2014/... · CNN Presentation.pptx Author: Stuart Kurtz Created Date: 10/3/2014 12:59:37 PM](https://reader035.vdocuments.us/reader035/viewer/2022071116/5ffc87f1138cee62851f020e/html5/thumbnails/42.jpg)
Lipshitz Bounds for Layers Max and ReLU are contractive FC Layers: usual linear operator norm Conv Layers: Parseval’s and DFT yield explicit formula
![Page 43: Pooling and Invariance in Convolutional Neural Networkstheorycenter.cs.uchicago.edu/REU/2014/... · CNN Presentation.pptx Author: Stuart Kurtz Created Date: 10/3/2014 12:59:37 PM](https://reader035.vdocuments.us/reader035/viewer/2022071116/5ffc87f1138cee62851f020e/html5/thumbnails/43.jpg)
Solutions? Regularize Lipshitz operator
![Page 44: Pooling and Invariance in Convolutional Neural Networkstheorycenter.cs.uchicago.edu/REU/2014/... · CNN Presentation.pptx Author: Stuart Kurtz Created Date: 10/3/2014 12:59:37 PM](https://reader035.vdocuments.us/reader035/viewer/2022071116/5ffc87f1138cee62851f020e/html5/thumbnails/44.jpg)
Coding Symmetry Convolutional Wavelet Networks
![Page 45: Pooling and Invariance in Convolutional Neural Networkstheorycenter.cs.uchicago.edu/REU/2014/... · CNN Presentation.pptx Author: Stuart Kurtz Created Date: 10/3/2014 12:59:37 PM](https://reader035.vdocuments.us/reader035/viewer/2022071116/5ffc87f1138cee62851f020e/html5/thumbnails/45.jpg)
Architecture Wavelet convolutions composed with modulus operator
![Page 46: Pooling and Invariance in Convolutional Neural Networkstheorycenter.cs.uchicago.edu/REU/2014/... · CNN Presentation.pptx Author: Stuart Kurtz Created Date: 10/3/2014 12:59:37 PM](https://reader035.vdocuments.us/reader035/viewer/2022071116/5ffc87f1138cee62851f020e/html5/thumbnails/46.jpg)
Gabor Wavelets Trigonometric function in Gaussian Envelope
![Page 47: Pooling and Invariance in Convolutional Neural Networkstheorycenter.cs.uchicago.edu/REU/2014/... · CNN Presentation.pptx Author: Stuart Kurtz Created Date: 10/3/2014 12:59:37 PM](https://reader035.vdocuments.us/reader035/viewer/2022071116/5ffc87f1138cee62851f020e/html5/thumbnails/47.jpg)
Group Convolution
![Page 48: Pooling and Invariance in Convolutional Neural Networkstheorycenter.cs.uchicago.edu/REU/2014/... · CNN Presentation.pptx Author: Stuart Kurtz Created Date: 10/3/2014 12:59:37 PM](https://reader035.vdocuments.us/reader035/viewer/2022071116/5ffc87f1138cee62851f020e/html5/thumbnails/48.jpg)
Group Invariant Scattering Main Result: Conv. Wavelet Networks are translation invariant functions in L_2(R^2) Furthermore, CWN can be made invariant to action under any compact lie group.
![Page 49: Pooling and Invariance in Convolutional Neural Networkstheorycenter.cs.uchicago.edu/REU/2014/... · CNN Presentation.pptx Author: Stuart Kurtz Created Date: 10/3/2014 12:59:37 PM](https://reader035.vdocuments.us/reader035/viewer/2022071116/5ffc87f1138cee62851f020e/html5/thumbnails/49.jpg)
Textures Sifre and
Mallat
![Page 50: Pooling and Invariance in Convolutional Neural Networkstheorycenter.cs.uchicago.edu/REU/2014/... · CNN Presentation.pptx Author: Stuart Kurtz Created Date: 10/3/2014 12:59:37 PM](https://reader035.vdocuments.us/reader035/viewer/2022071116/5ffc87f1138cee62851f020e/html5/thumbnails/50.jpg)
Basis for images Learns similar representation as Imagenet CNN
![Page 51: Pooling and Invariance in Convolutional Neural Networkstheorycenter.cs.uchicago.edu/REU/2014/... · CNN Presentation.pptx Author: Stuart Kurtz Created Date: 10/3/2014 12:59:37 PM](https://reader035.vdocuments.us/reader035/viewer/2022071116/5ffc87f1138cee62851f020e/html5/thumbnails/51.jpg)
Learnt Invariances
![Page 52: Pooling and Invariance in Convolutional Neural Networkstheorycenter.cs.uchicago.edu/REU/2014/... · CNN Presentation.pptx Author: Stuart Kurtz Created Date: 10/3/2014 12:59:37 PM](https://reader035.vdocuments.us/reader035/viewer/2022071116/5ffc87f1138cee62851f020e/html5/thumbnails/52.jpg)
Optimal network 1) Encoded symmetry 2) Regularize Lipshitz coefficients 3) Compete Neurons in final layers
![Page 53: Pooling and Invariance in Convolutional Neural Networkstheorycenter.cs.uchicago.edu/REU/2014/... · CNN Presentation.pptx Author: Stuart Kurtz Created Date: 10/3/2014 12:59:37 PM](https://reader035.vdocuments.us/reader035/viewer/2022071116/5ffc87f1138cee62851f020e/html5/thumbnails/53.jpg)
Final words