Upload File

convolutional neural networks & deep learning16385/s17/slides/9.6 cnns... · convolutional...

  • Home
  • Documents
  • Convolutional Neural Networks & Deep Learning16385/s17/Slides/9.6 CNNs... · Convolutional Neural Networks & Deep Learning. Pre deep learning era Cons: • Hand crafted features are
32
Convolutional Neural Networks & Deep Learning

Upload: others

Post on 21-May-2020

18 views

Category:

Documents


0 download

Report

TRANSCRIPT

Page 1: Convolutional Neural Networks & Deep Learning16385/s17/Slides/9.6 CNNs... · Convolutional Neural Networks & Deep Learning. Pre deep learning era Cons: • Hand crafted features are

ConvolutionalNeuralNetworks&DeepLearning

Page 2: Convolutional Neural Networks & Deep Learning16385/s17/Slides/9.6 CNNs... · Convolutional Neural Networks & Deep Learning. Pre deep learning era Cons: • Hand crafted features are

Predeeplearningera

Cons:• Hand crafted features are difficult to engineer!

• Time consuming process.

• Which set of features maximizes accuracy?

• Tends to overfit.

Page 3: Convolutional Neural Networks & Deep Learning16385/s17/Slides/9.6 CNNs... · Convolutional Neural Networks & Deep Learning. Pre deep learning era Cons: • Hand crafted features are

WhatisDeepLearning?Composition ofnon-linear transformationofdataWhy“deep”?Findcomplex patternsbylearninghierarchical features

Page 4: Convolutional Neural Networks & Deep Learning16385/s17/Slides/9.6 CNNs... · Convolutional Neural Networks & Deep Learning. Pre deep learning era Cons: • Hand crafted features are

Butdeeplearningissimple!• Deep Learning builds an end-to-end recognition system. • Non linear transformation of raw pixels directly to labels. • Build a complex non-linear system by combining 4 simple building blocks.

Convolutions

Softmax

Pooling

Activationfunctions

Page 5: Convolutional Neural Networks & Deep Learning16385/s17/Slides/9.6 CNNs... · Convolutional Neural Networks & Deep Learning. Pre deep learning era Cons: • Hand crafted features are

Convolutions

Page 6: Convolutional Neural Networks & Deep Learning16385/s17/Slides/9.6 CNNs... · Convolutional Neural Networks & Deep Learning. Pre deep learning era Cons: • Hand crafted features are

Convolutions

Page 7: Convolutional Neural Networks & Deep Learning16385/s17/Slides/9.6 CNNs... · Convolutional Neural Networks & Deep Learning. Pre deep learning era Cons: • Hand crafted features are

Convolutions– Indeeplearning

FigurefromS-1716-824CMU

Weneedtolearnthesefilters.

Page 8: Convolutional Neural Networks & Deep Learning16385/s17/Slides/9.6 CNNs... · Convolutional Neural Networks & Deep Learning. Pre deep learning era Cons: • Hand crafted features are

FigurefromS-1716-824CMU

Convolutions– Indeeplearning

Page 9: Convolutional Neural Networks & Deep Learning16385/s17/Slides/9.6 CNNs... · Convolutional Neural Networks & Deep Learning. Pre deep learning era Cons: • Hand crafted features are

FigurefromS-1716-824CMU

Convolutions– Indeeplearning

Page 10: Convolutional Neural Networks & Deep Learning16385/s17/Slides/9.6 CNNs... · Convolutional Neural Networks & Deep Learning. Pre deep learning era Cons: • Hand crafted features are

FigurefromS-1716-824CMU

Convolutions– Indeeplearning

Page 11: Convolutional Neural Networks & Deep Learning16385/s17/Slides/9.6 CNNs... · Convolutional Neural Networks & Deep Learning. Pre deep learning era Cons: • Hand crafted features are

Convolutions– Indeeplearning

Page 12: Convolutional Neural Networks & Deep Learning16385/s17/Slides/9.6 CNNs... · Convolutional Neural Networks & Deep Learning. Pre deep learning era Cons: • Hand crafted features are

Convolution– SpatialDimensions

FigurefromS-1716-824CMU

Page 13: Convolutional Neural Networks & Deep Learning16385/s17/Slides/9.6 CNNs... · Convolutional Neural Networks & Deep Learning. Pre deep learning era Cons: • Hand crafted features are

Convolution– SpatialDimensions

FigurefromS-1716-824CMU

Page 14: Convolutional Neural Networks & Deep Learning16385/s17/Slides/9.6 CNNs... · Convolutional Neural Networks & Deep Learning. Pre deep learning era Cons: • Hand crafted features are

Convolution:Example

Page 15: Convolutional Neural Networks & Deep Learning16385/s17/Slides/9.6 CNNs... · Convolutional Neural Networks & Deep Learning. Pre deep learning era Cons: • Hand crafted features are

WhynotuseFCsforlearningimagefeatures?

- Huge number of parameters in Fully connected network.

- Full connectivity is wasteful. Leads to overfitting.

- (200x200x3) x 5 neurons = 120,000x5 parameters in FC!

- No spatial relation in FCs.

- Just learn several filters (weights in CNNs).

- 5x5x100 = 2500 parameters for learning 100 filters in CNNs.

Page 16: Convolutional Neural Networks & Deep Learning16385/s17/Slides/9.6 CNNs... · Convolutional Neural Networks & Deep Learning. Pre deep learning era Cons: • Hand crafted features are

Max-pooling

Figure from Fei-Fei Li & Andrej Karpathy & Justin Johnson (CS231N)

• Non-linear down sampling.

• Input is partitioned into non-overlapping patches and maximum value in each

partition is chosen.

Page 17: Convolutional Neural Networks & Deep Learning16385/s17/Slides/9.6 CNNs... · Convolutional Neural Networks & Deep Learning. Pre deep learning era Cons: • Hand crafted features are

Max-pooling

Figure from Fei-Fei Li & Andrej Karpathy & Justin Johnson (CS231N)

Depth doesn’t change!

Page 18: Convolutional Neural Networks & Deep Learning16385/s17/Slides/9.6 CNNs... · Convolutional Neural Networks & Deep Learning. Pre deep learning era Cons: • Hand crafted features are

WhyMax-pool?

• Reduce spatial size of representation.

• Reduce the number of parameters drastically.

• 2x2 filter with stride = 2 discards 75% of the activations!

• Control overfitting.

• Provides translation invariance.

Page 19: Convolutional Neural Networks & Deep Learning16385/s17/Slides/9.6 CNNs... · Convolutional Neural Networks & Deep Learning. Pre deep learning era Cons: • Hand crafted features are

LinearActivations

SlidefromCMU16-720

Page 20: Convolutional Neural Networks & Deep Learning16385/s17/Slides/9.6 CNNs... · Convolutional Neural Networks & Deep Learning. Pre deep learning era Cons: • Hand crafted features are

Whynon-linear activationfunctions?Weneedanon-lineartransformationofdatasuchthattheoutputisacomplex,non-linear

transformationoftheinput.

Page 21: Convolutional Neural Networks & Deep Learning16385/s17/Slides/9.6 CNNs... · Convolutional Neural Networks & Deep Learning. Pre deep learning era Cons: • Hand crafted features are

HistoryofActivationFunctions

Page 22: Convolutional Neural Networks & Deep Learning16385/s17/Slides/9.6 CNNs... · Convolutional Neural Networks & Deep Learning. Pre deep learning era Cons: • Hand crafted features are

Sigmoid

SlidefromCMU16-720

Page 23: Convolutional Neural Networks & Deep Learning16385/s17/Slides/9.6 CNNs... · Convolutional Neural Networks & Deep Learning. Pre deep learning era Cons: • Hand crafted features are

Sigmoid

• Squashes numbers to range [0,1] – can kill gradients. (Vanishing gradient)

• Best for learning “logical” functions – i.e. functions on binary inputs.

• Not as good for image networks (replaced by RELU)

Page 24: Convolutional Neural Networks & Deep Learning16385/s17/Slides/9.6 CNNs... · Convolutional Neural Networks & Deep Learning. Pre deep learning era Cons: • Hand crafted features are

RectifiedLinearUnit

SlidefromCMU16-720

Page 25: Convolutional Neural Networks & Deep Learning16385/s17/Slides/9.6 CNNs... · Convolutional Neural Networks & Deep Learning. Pre deep learning era Cons: • Hand crafted features are

WhyReLu?

• Inexpensive computations. (Almost 6x faster than sigmoid!)

• No vanishing gradient!

• Leaky ReLus used to prevent “dying” neurons.

• Sparse gradients. (Skip computations where input < 0)

Page 26: Convolutional Neural Networks & Deep Learning16385/s17/Slides/9.6 CNNs... · Convolutional Neural Networks & Deep Learning. Pre deep learning era Cons: • Hand crafted features are

SoftmaxFunction

• Allpositivevalueswhichsumto1.

• Finallayerafteroutputlayer.

• Neatprobabilisticinterpretation – givesprobabilitiesofeachclass.

Page 27: Convolutional Neural Networks & Deep Learning16385/s17/Slides/9.6 CNNs... · Convolutional Neural Networks & Deep Learning. Pre deep learning era Cons: • Hand crafted features are

DeepLearningisjustacombinationofConvolutions+Pooling+ReLu

Page 28: Convolutional Neural Networks & Deep Learning16385/s17/Slides/9.6 CNNs... · Convolutional Neural Networks & Deep Learning. Pre deep learning era Cons: • Hand crafted features are

NetworkInitialization

Howdoyouinitializealltheweightsinthenetwork?

Wedonotknowthefinalvaluesoftheweights..

Page 29: Convolutional Neural Networks & Deep Learning16385/s17/Slides/9.6 CNNs... · Convolutional Neural Networks & Deep Learning. Pre deep learning era Cons: • Hand crafted features are

Allweights=0?

• Nolearning.

• Alloutputsare0.

• Errorsarenotbackpropagated.

• Noupdates.

Page 30: Convolutional Neural Networks & Deep Learning16385/s17/Slides/9.6 CNNs... · Convolutional Neural Networks & Deep Learning. Pre deep learning era Cons: • Hand crafted features are

Initializedtosmallrandomvalues

• Wewanttheweightscloseto0,butnotexactly0.• Initializetosmallrandomvaluestobreaksymmetry.• Recommended:SamplefromUniform(-r,r)

𝑟 = 46

𝑖𝑛 + 𝑜𝑢𝑡�

Page 31: Convolutional Neural Networks & Deep Learning16385/s17/Slides/9.6 CNNs... · Convolutional Neural Networks & Deep Learning. Pre deep learning era Cons: • Hand crafted features are

Topdeeplearninglibraries

Page 32: Convolutional Neural Networks & Deep Learning16385/s17/Slides/9.6 CNNs... · Convolutional Neural Networks & Deep Learning. Pre deep learning era Cons: • Hand crafted features are

Terminologies

• Iteration : 1 forward pass

• Epochs : 1 full training cycle on data set

• Batch-size : Number of samples trained per iteration

• Learning Rate : Update = Learning Rate x Gradient

• Max-Epochs : Usually 20. (Depends on data set)


Radiologists versus Deep Convolutional Neural Networks: A

ImageNet Classification with Deep Convolutional Neural ... · Convolutional neural networks Output Hidden Data Here's a one-dimensional convolutional neural network Each hidden neuron

Deep convolutional neural networks - Computer …web.cs.ucdavis.edu/~yjlee/teaching/ecs189g-spring2015/lee_lecture... · Deep convolutional neural networks ... in the visual cortex,

ImageNet Classification with Deep Convolutional Neural Networks€¦ · ImageNet Classification with Deep Convolutional Neural Networks Alex Krizhevsky, Ilya Sutskever, Geoffrey

Convolutional Neural Networks Analyzed via Convolutional ... · PDF fileConvolutional Neural Networks Analyzed via Convolutional Sparse Coding ... Deep Learning, Convolutional Neural

Deep convolutional neural networks, features, and

ImageNet Classification with Deep Convolutional Neural ...usman/courses/cs675_summer20/...ImageNet Classification with Deep Convolutional Neural Networks Alex Krizhevsky University

Deep convolutional neural networks - Computer Science- …web.cs.ucdavis.edu/.../lee_lecture19_deeplearning_notes.pdf · Deep convolutional neural networks ... “Deep” architecture

Refining Architectures of Deep Convolutional Neural Networksmi.eng.cam.ac.uk/~cipolla/publications/inproceedings/2016-CVPR... · Refining Architectures of Deep Convolutional Neural

Deep Convolutional Neural Network for Image Deconvolution

A ROBUST DEEP CONVOLUTIONAL NEURAL NETWORK MODEL …

ImageNet Classification with Deep Convolutional Neural …vision.stanford.edu/teaching/cs231b_spring1415/slides/alexnet... · ImageNet Classification with Deep Convolutional Neural

Training Deep Photonic Convolutional Neural Networks With

"Deep Learning" Chap.6 Convolutional Neural Net

ImageNet Classification with Deep Convolutional Neural ...rogerioferis.com/.../presentations/GuangnanAndMajaDeepLearning.pdf · ImageNet Classification with Deep Convolutional Neural

Handwritten Hangul recognition using deep convolutional ...xhx/publications/HHR.pdf · Handwritten Hangul recognition using deep convolutional neural networks ... deep neural networks

Convolutional Neural Network Models - Deep Learning

Face Template Protection Using Deep Convolutional Neural

ImageNet Classification with Deep Convolutional Neural ...web.cs.ucdavis.edu/~yjlee/teaching/ecs289g-fall2016/yuanzhe.pdf · ImageNet Classification with Deep Convolutional Neural

ImageNet Classification with Deep Convolutional Neural Networks

Deep convolutional neural networks for multiplanar lung

DEEP CONVOLUTIONAL SPIKING NEURAL NETWORKS FOR …

Conditional Deep Convolutional Neural Networks for

Deep Convolutional Neural Networks on Multichannel Time ... · Deep Convolutional Neural Networks On Multichannel Time Series ... as a common input for the neural network ... Deep

Self-supervised deep convolutional neural network for

Exploring the Design Space of Deep Convolutional … the Design Space of Deep Convolutional Neural Networks at Large ... Exploring the Design Space of Deep Convolutional Neural Networks

Training deep convolutional neural networks for ...jdh4/churchill_deep_learning_user... · Training deep convolutional neural networks for classification of multi-scale, nonlocal

Convolutional Neural Networks for Deep Learning: An Introduction · 2017-10-05 · Introduction: Convolutional Neural Networks (CNN) Convolutional Neural Networks: A deep learning

Deep Neural Networks Convolutional Networks II - Deep Learningdeeplearning.cs.cmu.edu/document/slides/lec9.CNN.pdf · Deep Neural Networks Convolutional Networks II Bhiksha Raj Spring

DEEP CONVOLUTIONAL NEURAL NETWORK FOR AUTOMATIC …

Empirical Evaluation of Deep Convolutional Neural Networks

Very Deep Convolutional Neural Networks for Noise …...convolutional layers with carefully tuned architecture. •ASR: Very Deep Convolutional Neural Networks uses up to 10 convolutional

ImageNet Classification with Deep Convolutional Neural Networks › ~hinton › absps › imagenet.pdf · ImageNet Classification with Deep Convolutional Neural Networks Alex Krizhevsky

Deep Convolutional Neural Networks - Cjoint.com

Ensemble of Deep Convolutional Neural Networks for