Upload File

csce 636 neural networks (deep learning)what a neural network does: learn a function x neural...

  • Home
  • Documents
  • CSCE 636 Neural Networks (Deep Learning)What a neural network does: learn a function x Neural Network value of f(x) The neural network learns the function f(x), either exactly or approximately
42
CSCE 636 Neural Networks (Deep Learning) Lecture 2: Mathematical Building Blocks of Neural Networks Anxiao (Andrew) Jiang

Upload: others

Post on 16-Oct-2020

5 views

Category:

Documents


0 download

Report

TRANSCRIPT

Page 1: CSCE 636 Neural Networks (Deep Learning)What a neural network does: learn a function x Neural Network value of f(x) The neural network learns the function f(x), either exactly or approximately

CSCE636NeuralNetworks(DeepLearning)

Lecture2:Mathematical BuildingBlocksofNeuralNetworks

Anxiao (Andrew)Jiang

Page 2: CSCE 636 Neural Networks (Deep Learning)What a neural network does: learn a function x Neural Network value of f(x) The neural network learns the function f(x), either exactly or approximately

Chapter2

Beforewebegin:themathematicalbuildingblocksofneuralnetworks

Page 3: CSCE 636 Neural Networks (Deep Learning)What a neural network does: learn a function x Neural Network value of f(x) The neural network learns the function f(x), either exactly or approximately

Whataneuralnetworkdoes:learnafunction

NeuralNetworkx

valueoff(x)

Theneuralnetworklearnsthefunctionf(x),eitherexactlyorapproximately.

Page 4: CSCE 636 Neural Networks (Deep Learning)What a neural network does: learn a function x Neural Network value of f(x) The neural network learns the function f(x), either exactly or approximately

Application:HandwrittenDigitRecognition

NeuralNetwork 4

Task:Classifygrayscaleimagesofhandwrittendigits(28x28pixels)intotheir10categories(0through9).

Page 5: CSCE 636 Neural Networks (Deep Learning)What a neural network does: learn a function x Neural Network value of f(x) The neural network learns the function f(x), either exactly or approximately

Howtostart?

Page 6: CSCE 636 Neural Networks (Deep Learning)What a neural network does: learn a function x Neural Network value of f(x) The neural network learns the function f(x), either exactly or approximately

Step1:Loadthedataset

MNISTDataset:60,000 trainingimagesand10,000 testimages,alongwiththeirlabels.

Page 7: CSCE 636 Neural Networks (Deep Learning)What a neural network does: learn a function x Neural Network value of f(x) The neural network learns the function f(x), either exactly or approximately

Step1:Loadthedataset

train_images: 60,000x28x28array,whereeachelement (pixel) isaninteger in[0,255]train_labels: vectoroflength60,000,whereeachelement (label) isaninteger in[0,9]

test_images: 10,000x28x28array,whereeachelement (pixel) isaninteger in[0,255]test_labels: vectoroflength10,000,whereeachelement (label)isaninteger in[0,9]

Rule:trainingdataandtestdataaredisjoint.Onlyusetrainingdatatotrainneuralnetwork!

Page 8: CSCE 636 Neural Networks (Deep Learning)What a neural network does: learn a function x Neural Network value of f(x) The neural network learns the function f(x), either exactly or approximately

4

Page 9: CSCE 636 Neural Networks (Deep Learning)What a neural network does: learn a function x Neural Network value of f(x) The neural network learns the function f(x), either exactly or approximately

Step2:Buildneuralnetworkarchitecture

428x282-d array

0

1

2

28x28-1 =783

0

1

2

511

512neurons

Page 10: CSCE 636 Neural Networks (Deep Learning)What a neural network does: learn a function x Neural Network value of f(x) The neural network learns the function f(x), either exactly or approximately

Whatisaneuron

Activation function(manypossible forms)

ReLU (mostpopularActivationfunction)

Page 11: CSCE 636 Neural Networks (Deep Learning)What a neural network does: learn a function x Neural Network value of f(x) The neural network learns the function f(x), either exactly or approximately

Step2:Buildneuralnetworkarchitecture

428x282-d array

0

1

2

28x28-1 =783

0

1

2

511

512neurons

0

1

2

9

10neurons

Page 12: CSCE 636 Neural Networks (Deep Learning)What a neural network does: learn a function x Neural Network value of f(x) The neural network learns the function f(x), either exactly or approximately

Softmax (popularactivationfunctionforthelastlayerofaclassificationnetwork)

Page 13: CSCE 636 Neural Networks (Deep Learning)What a neural network does: learn a function x Neural Network value of f(x) The neural network learns the function f(x), either exactly or approximately

Step2:Buildneuralnetworkarchitecture

428x282-d array

0

1

2

28x28-1 =783

0

1

2

511

512neurons

0

1

2

9

10neurons

Probability oflabel “0”

Probability oflabel “1”

Probability oflabel “2”

Probability oflabel “9”

Page 14: CSCE 636 Neural Networks (Deep Learning)What a neural network does: learn a function x Neural Network value of f(x) The neural network learns the function f(x), either exactly or approximately

Step3:chooselossfunction,optimizer,andtargetmetrics

Page 15: CSCE 636 Neural Networks (Deep Learning)What a neural network does: learn a function x Neural Network value of f(x) The neural network learns the function f(x), either exactly or approximately

Categoricalcross-entropy(apopularlossfunctionformulti-classclassification)

Numberofsamples

Numberofclasses

Trueprobability(1or0)thisinputbelongstoclassj

probabilitypredictedbyneuralnetworkthatthisinputbelongsto

classj

Page 16: CSCE 636 Neural Networks (Deep Learning)What a neural network does: learn a function x Neural Network value of f(x) The neural network learns the function f(x), either exactly or approximately

RMSProp (apopularoptimizer,detailstobeintroduced later)

Page 17: CSCE 636 Neural Networks (Deep Learning)What a neural network does: learn a function x Neural Network value of f(x) The neural network learns the function f(x), either exactly or approximately

Accuracy: fractionoftimesthattheneuralnetworkmakescorrectionpredictions• Ifwecareaboutaccuracy,whydoweoptimizecategoricalcross-entropyduringtraining?• Answer:lossfunctionneedstobedifferentiable.(Andthelossfunctioniscloselyrelatedtothetargetmetric.Minimizingthelossfunctionis(approximatelyorprecisely)optimizingthetargetmetric.

Page 18: CSCE 636 Neural Networks (Deep Learning)What a neural network does: learn a function x Neural Network value of f(x) The neural network learns the function f(x), either exactly or approximately

428x282-d array

0

1

2

28x28-1 =783

0

1

2

511

512neurons

0

1

2

9

10neurons

Probability oflabel “0”

Probability oflabel “1”

Probability oflabel “2”

Probability oflabel “9”

The“Teacher”:

Loss function: categoricalcross-entropyOptimizer: RMSPropTargetMetric:Accuracy

Page 19: CSCE 636 Neural Networks (Deep Learning)What a neural network does: learn a function x Neural Network value of f(x) The neural network learns the function f(x), either exactly or approximately

Step4:PreparetrainingandtestdataHere:Reshapeandnormalizeinputtrainingdata

train_images:Originally: 3-dimensional arrayofsize60000 x28x28,whereeach element isanintegerin [0,255]Afterreshaping: 2-dimensional arrayofsize60000 x784,whereeach element isanintegerin [0,255]Afternormalization: 2-dimensional arrayofsize60000 x784,where eachelement is arealnumber in[0,1]

28x282-d array

1-d array oflength28*28 =784

Normalize values inthearraytobetween0and 1

Page 20: CSCE 636 Neural Networks (Deep Learning)What a neural network does: learn a function x Neural Network value of f(x) The neural network learns the function f(x), either exactly or approximately

428x282-d array

0

1

2

0

1

2

511

512neurons

0

1

2

9

10neurons

Probability oflabel “0”

Probability oflabel “1”

Probability oflabel “2”

Probability oflabel “9”

The“Teacher”:

Loss function: categoricalcross-entropyOptimizer: RMSPropTargetMetric:Accuracy

783

0

1

2

783

Page 21: CSCE 636 Neural Networks (Deep Learning)What a neural network does: learn a function x Neural Network value of f(x) The neural network learns the function f(x), either exactly or approximately

“Reshape”outputtrainingdata:categoricallyencodeeachlabelusingone-hotencoding

Label One-hotencoding0 1,0,0,0,0,0,0,0,0,01 0,1,0,0,0,0,0,0,0,02 0,0,1,0,0,0,0,0,0,03 0,0,0,1,0,0,0,0,0,04 0,0,0,0,1,0,0,0,0,0

Label One-hotencoding5 0,0,0,0,0,1,0,0,0,06 0,0,0,0,0,0,1,0,0,07 0,0,0,0,0,0,0,1,0,08 0,0,0,0,0,0,0,0,1,09 0,0,0,0,0,0,0,0,0,1

Page 22: CSCE 636 Neural Networks (Deep Learning)What a neural network does: learn a function x Neural Network value of f(x) The neural network learns the function f(x), either exactly or approximately

428x282-d array

0

1

2

0

1

2

511

512neurons

0

1

2

9

10neurons

The“Teacher”:

Loss function: categoricalcross-entropyOptimizer: RMSPropTargetMetric:Accuracy

783

0

1

2

783

0

1

2

9

Page 23: CSCE 636 Neural Networks (Deep Learning)What a neural network does: learn a function x Neural Network value of f(x) The neural network learns the function f(x), either exactly or approximately

Step5:Traintheneuralnetwork

428x282-d array

0

1

2

0

1

2

511

512neurons

0

1

2

9

10neurons

The“Teacher”:Loss function, Optimizer, TargetMetric:Accuracy

783

0

1

2

783

0

1

2

9

Page 24: CSCE 636 Neural Networks (Deep Learning)What a neural network does: learn a function x Neural Network value of f(x) The neural network learns the function f(x), either exactly or approximately

Batchsize:thenumberofsamplestouseeachtimeforcomputingthelossfunctionandupdatingtheweights.

Epochs:thenumberoftimesthetrainingprocessusesthewholetrainingdataset.

Page 25: CSCE 636 Neural Networks (Deep Learning)What a neural network does: learn a function x Neural Network value of f(x) The neural network learns the function f(x), either exactly or approximately

Andsoon(totally5epochs).

Accuracyontrainingdata:97.8%

Page 26: CSCE 636 Neural Networks (Deep Learning)What a neural network does: learn a function x Neural Network value of f(x) The neural network learns the function f(x), either exactly or approximately

Step6:Testthetrainedneuralnetwork

Comparetotrainingaccuracy:0.989

Test accuracyis(clearly) lowerthantrainingaccuracy.Maybethere issome over-fittingtodata.

Butstill, performance isnice!

Page 27: CSCE 636 Neural Networks (Deep Learning)What a neural network does: learn a function x Neural Network value of f(x) The neural network learns the function f(x), either exactly or approximately

Summary

Page 28: CSCE 636 Neural Networks (Deep Learning)What a neural network does: learn a function x Neural Network value of f(x) The neural network learns the function f(x), either exactly or approximately
Page 29: CSCE 636 Neural Networks (Deep Learning)What a neural network does: learn a function x Neural Network value of f(x) The neural network learns the function f(x), either exactly or approximately

Step1:Loadthedataset

4

Page 30: CSCE 636 Neural Networks (Deep Learning)What a neural network does: learn a function x Neural Network value of f(x) The neural network learns the function f(x), either exactly or approximately

Step2:Buildneuralnetworkarchitecture

428x282-d array

0

1

2

28x28-1 =783

0

1

2

511

0

1

2

9

Page 31: CSCE 636 Neural Networks (Deep Learning)What a neural network does: learn a function x Neural Network value of f(x) The neural network learns the function f(x), either exactly or approximately

Step3:chooselossfunction,optimizer,andtargetmetrics

428x282-d array

0

1

2

28x28-1 =783

0

1

2

511

0

1

2

9

Probability oflabel “0”

Probability oflabel “1”

Probability oflabel “2”

Probability oflabel “9”

The“Teacher”:Loss function, Optimizer, TargetMetric

Page 32: CSCE 636 Neural Networks (Deep Learning)What a neural network does: learn a function x Neural Network value of f(x) The neural network learns the function f(x), either exactly or approximately

Step4:Preparetrainingandtestdata

428x282-d array

0

1

2

0

1

2

511

0

1

2

9783

0

1

2

783

0

1

2

9

The“Teacher”:Loss function, Optimizer, TargetMetric

Page 33: CSCE 636 Neural Networks (Deep Learning)What a neural network does: learn a function x Neural Network value of f(x) The neural network learns the function f(x), either exactly or approximately

Step5:Traintheneuralnetwork

428x282-d array

0

1

2

0

1

2

511

512neurons

0

1

2

9

10neurons

The“Teacher”:Loss function, Optimizer, TargetMetric:Accuracy

783

0

1

2

783

0

1

2

9

Page 34: CSCE 636 Neural Networks (Deep Learning)What a neural network does: learn a function x Neural Network value of f(x) The neural network learns the function f(x), either exactly or approximately

Step6:Testthetrainedneuralnetwork

HowdidIdo?

Well…

Page 35: CSCE 636 Neural Networks (Deep Learning)What a neural network does: learn a function x Neural Network value of f(x) The neural network learns the function f(x), either exactly or approximately

MiscellaneousBasicConcepts

Page 36: CSCE 636 Neural Networks (Deep Learning)What a neural network does: learn a function x Neural Network value of f(x) The neural network learns the function f(x), either exactly or approximately

Datarepresentation:Tensor(Array)

• Scalarnumbers(0-dimensionaltensors)• Vectors(1-dtensors)• Matrices(2-dtensors)• 3-dtensors,andhigher-dimensionaltensors

• Keyattributesforatensor:• (1)numberofaxes• (2)shape• (3)datatype

Page 37: CSCE 636 Neural Networks (Deep Learning)What a neural network does: learn a function x Neural Network value of f(x) The neural network learns the function f(x), either exactly or approximately

Somebasictensoroperations

• Addtwotesnors (ofthesameshape):element-wiseaddition• ApplyaReLU activationfunctiontoatensor:element-wiseoperation

• TensorProduct(alsocalledtensordot)

Page 38: CSCE 636 Neural Networks (Deep Learning)What a neural network does: learn a function x Neural Network value of f(x) The neural network learns the function f(x), either exactly or approximately

Reshapetensor

Page 39: CSCE 636 Neural Networks (Deep Learning)What a neural network does: learn a function x Neural Network value of f(x) The neural network learns the function f(x), either exactly or approximately

Basictermsforaneuralnetwork

• Layers:thebuildingblocksinaneuralnetwork• Model:networkoflayers• Lossfunctionandoptimizer:keystoconfiguringthelearningprocess

Page 40: CSCE 636 Neural Networks (Deep Learning)What a neural network does: learn a function x Neural Network value of f(x) The neural network learns the function f(x), either exactly or approximately

Keras:adeeplearninglibraryforPython

PyTorchisgettingpopulartoday

Page 41: CSCE 636 Neural Networks (Deep Learning)What a neural network does: learn a function x Neural Network value of f(x) The neural network learns the function f(x), either exactly or approximately

Keras:adeeplearninglibraryforPython

UseaGPUwhenpossible

Page 42: CSCE 636 Neural Networks (Deep Learning)What a neural network does: learn a function x Neural Network value of f(x) The neural network learns the function f(x), either exactly or approximately

Jupyter notebook:anicewaytoeditandrundeeplearningexperiments


CSCI 5561: Assignment #4 Convolutional Neural Network 1 ...hspark/csci5561_F2019/hw4.pdf · CSCI 5561: Assignment #4 Convolutional Neural Network def fc(x, w, b)... return y Input:

TRANSFER LEARNING BASED CONVOLUTIONAL NEURAL NETWORK FOR … · 2021. 1. 24. · TRANSFER LEARNING BASED CONVOLUTIONAL NEURAL NETWORK FOR COVID-19 DETECTION WITH X-RAY IMAGES Kevser

Biological Neural Network & Nonlinear Dynamics Biological Neural Network Similar Neural Network to Real Neural Networks Membrane Potential Potential of

02 Fundamentals of Neural Network - myreaders.info · 1.1 Why Neural Network SC - Neural Network – Introduction Neural Networks follow a different paradigm for computing.

Neural Network

Neural Network Modeling of Speech and Music Signals · Neural Network Modeling of Speech and Music Signals k(i) Control x(i+1) x(i+ TL----- x(i-3Tj -x(i-T) x(i-2T) x(i) Fig. I: Input/Output

Deep neural network based forensic automatic speaker ... · Deep neural network based forensic automatic speaker recognition in VOCALISE using x-vectors Finnian Kelly 1, Oscar Forth

Creation of Synthetic X-Rays to Train a Neural Network to ... · Creation of Synthetic X-Rays to Train a Neural Network to Detect Lung Cancer Abhishek Moturu and Alex Chang Department

Compositional Neural-Network Modeling of …Compositional Neural-Network Modeling of Complex Analog Circuits Ramin M. Hasani , Dieter Haerle , Christian F. Baumgartner x, Alessio R

Bachelor / Master Thesis: Neural Network Pre-Distortion x

Neural Network Modeling of Spectral Embeddingpdfs.semanticscholar.org/a6d0/f4e824e9fc65bbec5d1ada2cd4a1c4939411.pdf2 Neural Network for Manifold Learning Neural network can be expressed

Recurrent Neural Network (RNN) - 國立臺灣大學speech.ee.ntu.edu.tw/~tlkagk/courses/ML_2017/Lecture/RNN.pdf · Recurrent Neural Network (RNN) x 1 x 2 y 1 y 2 a 1 a 2 Memory can

Neural Network Language Models - The University of Edinburgh · Hybrid neural network / HMM acoustic models Neural network features { Tandem, posteriorgrams Deep neural network acoustic

A convolutional neural network-based screening …A convolutional neural network-based screening tool for X-ray serial crystallography Tsung-Wei Ke,a Aaron S. Brewster,b Stella X

Artificial Neural Network (Back-Propagation Neural Network)

CS456: Machine Learning · Outlines A brief history of neural network Multilayer neural network Backpropagation Deep neural network Convolutional neural networks (CNNs) Long-Short

Chapter 10 Chapter 10 Neural Network Neural Network

Neural network for Neural Networks

02 Fundamentals of Neural Network - myreaders.info€¦ · 1.1 Why Neural Network SC - Neural Network – Introduction Neural Networks follow a different paradigm for computing.

- Tutorial - Neuromorphic Computing with Spintronics...• Deep neural network • Recurrent neural network –Reservoir computing • Hopfield network • Stochastic neural network

Cross Validation and WAIC in Layered Neural Networkswatanabe- Neural Network is Nonidentifiable Input x Para-meter w Output f(x,w) w → f( ,w) is not injective { ∂wjf(x,w) } is

Neural Network Society, Japanese Neural Network Society 15

Evaluation of a Dual Convolutional Neural Network ... · Evaluation of a Dual Convolutional Neural Network Architecture for Object-wise Anomaly Detection in Cluttered X-ray Security

Chapter 9 Chapter 9 Neural Network Neural Network

Deep Neural Networks (1) Hidden layers; Back-propagation · Single layer network Single-layer network, 1 output, 2 inputs + x 1 x 2 MLP Lecture 3 / 1 October 2019 Deep Neural Networks

ELEC 677: Recurrent Neural Network Applications & Recurrent Neural Network … · 2016-11-08 · Recurrent Neural Network Applications & Recurrent Neural Network Language Models Lecture

Neural Network Toolbox - Hacettepe Universitysolen/Matlab/MatLab/Matlab - Neural... · Neural Network Toolbox For Use with MATLAB ... Neural Network Design Book.....xxii Acknowledgments.....xxiii

Recurrent Neural Network (RNN) - NTU Speech …speech.ee.ntu.edu.tw/~tlkagk/courses/ML_2016/Lecture/RNN...Recurrent Neural Network (RNN) x 1 x 2 y 1 y 2 a 1 a 2 Memory can be considered

Recurrent Neural Networks - Stanford Universitycs231n.stanford.edu/slides/2016/winter1516_lecture10.pdf · Recurrent Neural Network x RNN y We can process a sequence of vectors x

Statistical & Data Analysis Using Neural Network ·  · 2015-03-12Statistical & Data Analysis Using Neural Network ... The Biological Perspective of Neural Networks Neural Network