nelson intro ann

Introduction to Artificial Neural NetworksAndrew L. Nelson

Visiting Research FacultyUniversity of South Florida

OverviewOutline to the leftCurrent topic in redIntroductionHistory and OriginsBiologically Inspired Applications PerceptronActivation FunctionsHidden Layer NetworksTraining with BPExamples

ReferencesIntroductionHistoryBiologically InspiredApplicationsThe PerceptronActivation FunctionsHidden Layer NetworksTraining with BPExamples

ReferencesW. S. McCulloch, W. Pitts, "A logical calculus of the ideas immanent in nervous activity," Bulletin of Mathematical Biophysics, vol. 5 pp. 115-133, 1943.

J. L. McClelland, D. E. Rumelhart, Parallel Distributed Processing: Explorations in the Microstructure of Cognition, The MIT Press, 1986.

C. M. Bishop, Neural Networks for Pattern Recognition, Oxford University Press, 1995.


IntroductionArtificial Neural Networks (ANN)Connectionist computationParallel distributed processingComputational modelsBiologically Inspired computational modelsMachine LearningArtificial intelligence ReferencesIntroductionHistoryBiologically InspiredApplicationsThe PerceptronActivation FunctionsHidden Layer NetworksTraining with BPExamples

HistoryMcCulloch and Pitts introduced the Perceptron in 1943.Simplified model of a biological neuronFell out of favor in the late 1960's (Minsky and Papert) Perceptron limitations Resurgence in the mid 1980'sNonlinear Neuron FunctionsBack-propagation training


Summary of ApplicationsFunction approximationPattern recognitionSignal processingModelingControlMachine learning


Biologically InspiredElectro-chemical signalsThreshold output firing ReferencesIntroductionHistoryBiologically InspiredApplicationsThe PerceptronActivation FunctionsHidden Layer NetworksTraining with BPExamples

The PerceptronBinary classifier functionsThreshold activation functionReferencesIntroductionHistoryBiologically InspiredApplicationsThe PerceptronActivation FunctionsHidden Layer NetworksTraining with BPExamples

The Perceptron: Threshold Activation FunctionBinary classifier functionsThreshold activation function


Linear Activation functionsOutput is scaled sum of inputsReferencesIntroductionHistoryBiologically InspiredApplicationsThe PerceptronActivation FunctionsHidden Layer NetworksTraining with BPExamples

Linear

Nonlinear Activation FunctionsSigmoid Neuron unit functionReferencesIntroductionHistoryBiologically InspiredApplicationsThe PerceptronActivation FunctionsHidden Layer NetworksTraining with BPExamples

Layered Networks.ReferencesIntroductionHistoryBiologically InspiredApplicationsThe PerceptronActivation FunctionsHidden Layer NetworksTraining with BPExamples

SISO Single Hidden Layer NetworkCan represent and single input single output functions: y = f(x)ReferencesIntroductionHistoryBiologically InspiredApplicationsThe PerceptronActivation FunctionsHidden Layer NetworksTraining with BPExamples

Training Data SetAdjust weights (w) to learn a given target function: y = f(x)Given a set of training data XYReferencesIntroductionHistoryBiologically InspiredApplicationsThe PerceptronActivation FunctionsHidden Layer NetworksTraining with BPExamples

Training Weights: Error Back-Propagation (BP)Weight update formula: ReferencesIntroductionHistoryBiologically InspiredApplicationsThe PerceptronActivation FunctionsHidden Layer NetworksTraining with BPExamples

Error Back-Propagation (BP)Training error term: eReferencesIntroductionHistoryBiologically InspiredApplicationsThe PerceptronActivation FunctionsHidden Layer NetworksTraining with BPExamples

whid,1

yhid(uhid,1)

wout,1

N1

yout(uout,n)

yhid(uhid,2)

N2

yhid(uhid,n)

Nn

x

e(yout, ytrain)

whid,2

wout,2

whid,n

wout,n

uhid,1

x

yhid

uout,1

yout

ytrain

Hidden Neurons

Output Neuron

e

BP FormulationReferencesIntroductionHistoryBiologically InspiredApplicationsThe PerceptronActivation FunctionsHidden Layer NetworksTraining with BPExamples

whid,1

yhid(uhid,1)

wout,1

N1

yout(uout,n)

yhid(uhid,2)

N2

yhid(uhid,n)

Nn

x

e(yout, ytrain)

whid,2

wout,2

whid,n

wout,n

uhid,1

x

yhid

uout,1

yout

ytrain

Hidden Neurons

Output Neuron

e

Example: The XOR problem: Single hidden layer: 3 Sigmoid neurons2 inputs, 1 output Desired I/O table (XOR):ReferencesIntroductionHistoryBiologically InspiredApplicationsThe PerceptronActivation FunctionsHidden Layer NetworksTraining with BPExamples

x1x2yExample 1000Example 2011Example 3101Example 4110

Example: The XOR problem: Training error over epochReferencesIntroductionHistoryBiologically InspiredApplicationsThe PerceptronActivation FunctionsHidden Layer NetworksTraining with BPExamples

Example: The XOR problem: initial_weights =0.0654 0.2017 0.0769 0.1782 0.0243 0.0806 0.0174 0.1270 0.0599 0.1184 0.1335 0.0737 0.1511

final_weights =4.6970 -4.6585 2.0932 5.5168 -5.7073 -3.2338 -0.1886 1.6164 -0.1929 -6.8066 6.8477 -1.6886 4.1531ReferencesIntroductionHistoryBiologically InspiredApplicationsThe PerceptronActivation FunctionsHidden Layer NetworksTraining with BPExamples

Example: The XOR problem: Mapping produced by the trained neural net:ReferencesIntroductionHistoryBiologically InspiredApplicationsThe PerceptronActivation FunctionsHidden Layer NetworksTraining with BPExamples

x1x2yExample 100 0.0824Example 201 0.9095Example 310 0.9470Example 411 0.0464

Example: Overtraining Single hidden layer: 10 Sigmoid neurons1 input, 1 outputReferencesIntroductionHistoryBiologically InspiredApplicationsThe PerceptronActivation FunctionsHidden Layer NetworksTraining with BPExamples

nelson intro ann

Documents