case study of convolutional neural network
Post on 21-Apr-2017
1.786 Views
Preview:
TRANSCRIPT
Case Study of CNN from LeNet to ResNet
NamHyuk Ahn @ Ajou Univ. 2016. 03. 09
Convolutional Neural Network
Convolution Layer
- Convolution (3-dim dot product) image and filter
- Stack filter in one layer (See blue and green output, called channel)
Convolution Layer- Local Connectivity
• Instead connect all pixels to neurons, connect only local region of input (called receptive field)
• It can reduce many parameter
- Parameter sharing
• To reduce parameter, each channel have same filter. (# of filter == # of channel)
Convolution Layer- Example) 1st conv layer in AlexNet
• Input: [224, 224], filter: [11x11x3], 96, output: [55, 55]
- Each filter extract different features (i.e. horizontal edge, vertical edge…)
Pooling Layer- Downsample image to reduce parameter
- Usually use max pooling (take maximum value in region)
ReLU, FC Layer
- ReLU
• Sort of activation function (e.g. sigmoid, tanh…)
- Fully-connected Layer
• Same as normal neural network
Convolutional Neural Network
\
Training CNN1. Calculate loss function with foward-prop
2. Optimize parameter w.r.t loss function with back-prop
• Use gradient descent method (SGD)
• Gradient of weight can calculate with chain rule of partial derivate
ILSVRC trend
AlexNet (2012) (ILSVRC 2012 winner)
AlexNet
- ReLU
- Data augmentation
- Dropout
- Ensemble CNN (1-CNN 18.2%, 7-CNN 15.4%)
AlexNet
- Other methods (but will not mention today)
• SGD + momentum (+ mini-batch)
• Multiple GPU
• Weight Decay
• Local Response Normalization
Problems of sigmoid
- Gradient vanishing
• when gradient pass sigmoid, it can vanish because local gradient of sigmoid can be almost zero.
- Output is not zero-centered
• cause bad performance
ReLU
- Converge of SGD is faster than sigmoid-like
- Computationally cheap
Data augmentation- Randomly crop [256, 256] images to [224, 224]
- At test time, crop 5 images and average to predict
Dropout- Similar to bagging (approximation of bagging)
- Act like regularizer (reduce overfit)
- Instead of using all neurons, “dropout” some neurons randomly (usually 0.5 probability)
Dropout• At test time, not “dropout” neurons, but use
weighted neurons (usually 0.5)
• Weight is expected value of each neurons
Architecture
- conv - pool - … - fc - softmax (similar to LeNet)
- Use large size filter (i.e. 11x11)
Architecture
- Weights must be initalized randomly
• If not, all gradients of neurons will be same
• Usually, use gaussian distribution, std = 0.01
- Use mini-batch SGD and momentum SGD to update weight
VGGNet (2014) (ILSVRC 2014 2nd)
VGGNet
- Use small size kernel (always 3x3)
• Can use multiple non-linearlity (e.g. ReLU)
• Less weights to train
- Hard data augmentation (more than AlexNet)
- Ensemble 7 model (ILSVRC submission 7.3%)
Architecture
- Most memory needs in early layers, most parameters increase in fc layers.
GoogLeNet - Inception v1 (2014) (ILSVRC 2014 winner)
GoogLeNet
Inception module- Use 1x1, 3x3 and 5x5 conv
simultaneously to capture variety of structure
- Capture dense structure to 1x1, more spread out structure to 3x3, 5x5
- Computational expensive
• Use 1x1 conv layer to reduce dimension (explain details in later in ResNet)
Auxiliary Classifiers- Deep network raises concern about effectiveness
of graident in backprop
- Loss of auxiliary is added to total loss (weighted by 0.3), remove at test time
Average Pooling
- Proposed in Network in Network (also used in GoogLeNet)
- Problems of fc layer
• Needs lots of parameter, easy to overfit
- Replace fc to average pooling
Average Pooling- Make channel as same as # of class in last conv
- Calc average on each channel, and pass to softmax
- Reduce overfit
MSRA ResNet (2015) (ILSVRC 2015 winner)
before ResNet..
- Have to know about
• PReLU
• Xavier Initalization
• Batch Normalization
PReLU- Adaptive version of ReLU
- Train slope of function when x < 0
- Slightly more parameter (# of layer x # of channel)
Xavier Initalization- If init with gaussian distribution, output of neurons
will be nearly zeros when network is deeep
- If increase std (1.0), output will saturate to -1 or 1
- Xavier init decide initial value by number of input neurons
- Looks fine, but this init method assume linear activation so can’t use in ReLU-like network
output is saturated
output is vanished
Xavier Initalization / 2
Xavier Initalization
Xavier Initalization / 2
Batch Normalization- Make output to be gaussian distribution, but
normalization cost a lot
• Calc mean, variance in each dimension (assume each dims are uncorrelated)
• Calc mean, variance in mini-batch (not entire set)
- Normalize constrain non-linearlity and constrain network by assume each dims are uncorrelated
• Linear transform output (factors are parameter)
Batch Normalization- When test, calc mean, variance using entire set (use
moving average)
- BN act like regularizer (don’t need Dropout)
ResNet
ResNet
Problem of degradation- More depth, more accurate but deep network can
vanish/explode gradient • BN, Xavier Init, Dropout can handle (~30 layer)
- More deeper, degradation problem occur • Not only overfit, but also increase training error
Deep Residual Learning
- Element-wise addition with F(x) and shortcut connection, and pass through ReLU non-linearlity
- Dim of x, F(x) are unequal (changing of channel), linear project x to match dim (done by 1x1 conv)
- Similar to LSTM
Deeper Bottleneck
- To reduce training time, modify as bottleneck design (just for economical reason)
• (3x3x3)x64x64 + (3x3x3)x64x64=221184 (left)
• (1x1x3)x256x64 + (3x3x3)x64x64 + (1x1x3)x64x256=208896 (right)
• More width(channel) in right, but similar parameter
• Similar method also used in GoogLeNet
ResNet
- Data augmentation as AlexNet does
- Batch Normalization (no dropout)
- Xavier / 2 initalization
- Average pooling
- Structure follows VGGNet style
Conclusion
Top-
5 Er
ror
0%
4%
8%
12%
16%
AlexNet
(2012)
VGGNet
(2014)
Inception-V1
(2014)
Human
PReLU-net
(2015)
BN-Inception
(2015)
ResNet-152
(2015)Inception-ResNet
(2016)
3.1%3.57%4.82%4.94%5.1%
6.66%7.32%
15.31%
Conclusion
- Dropout, BN
- ReLU-like activation (e.g. PReLU, ELU..)
- Xavier initalization
- Average pooling
- Use pre-trained model :)
Reference- Krizhevsky, Alex, Ilya Sutskever, and Geoffrey E. Hinton. "Imagenet classification with deep
convolutional neural networks." Advances in neural information processing systems. 2012.
- Simonyan, Karen, and Andrew Zisserman. "Very deep convolutional networks for large-scale image recognition." arXiv preprint arXiv:1409.1556 (2014).
- Lin, Min, Qiang Chen, and Shuicheng Yan. "Network in network." arXiv preprint arXiv:1312.4400 (2013).
- He, Kaiming, et al. "Delving deep into rectifiers: Surpassing human-level performance on imagenet classification." Proceedings of the IEEE International Conference on Computer Vision. 2015.
- He, Kaiming, et al. "Deep Residual Learning for Image Recognition." arXiv preprint arXiv:1512.03385 (2015).
- Szegedy, Christian, Sergey Ioffe, and Vincent Vanhoucke. "Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning." arXiv preprint arXiv:1602.07261 (2016).
- Gu, Jiuxiang, et al. "Recent Advances in Convolutional Neural Networks." arXiv preprint arXiv:1512.07108 (2015). (good for tutorial)
- Also Thanks to CS231n, I used some figures in CS231n lecture slides. see http://cs231n.stanford.edu/index.html
top related