neural networks mse 2400 ealicara spring 2015 dr. tom way
TRANSCRIPT
![Page 1: Neural Networks MSE 2400 EaLiCaRA Spring 2015 Dr. Tom Way](https://reader030.vdocuments.us/reader030/viewer/2022032803/56649e425503460f94b34200/html5/thumbnails/1.jpg)
Neural Networks
MSE 2400 EaLiCaRA
Spring 2015 Dr. Tom Way
![Page 2: Neural Networks MSE 2400 EaLiCaRA Spring 2015 Dr. Tom Way](https://reader030.vdocuments.us/reader030/viewer/2022032803/56649e425503460f94b34200/html5/thumbnails/2.jpg)
Background
Neural Networks can be :
- Biological models
- Artificial models
Desire to produce artificial systems capable of sophisticated computations similar to the human brain.
MSE 2400 Evolution & Learning 2
![Page 3: Neural Networks MSE 2400 EaLiCaRA Spring 2015 Dr. Tom Way](https://reader030.vdocuments.us/reader030/viewer/2022032803/56649e425503460f94b34200/html5/thumbnails/3.jpg)
Biological analogy and some main ideas
• The brain is composed of a mass of interconnected neurons– each neuron is connected to many other neurons
• Neurons transmit signals to each other
• Whether a signal is transmitted is an all-or-nothing event (the electrical potential in the cell body of the neuron is thresholded)
• Whether a signal is sent, depends on the strength of the bond (synapse) between two neurons
MSE 2400 Evolution & Learning 3
![Page 4: Neural Networks MSE 2400 EaLiCaRA Spring 2015 Dr. Tom Way](https://reader030.vdocuments.us/reader030/viewer/2022032803/56649e425503460f94b34200/html5/thumbnails/4.jpg)
How Does the Brain Work ?
NEURON
- The cell that performs information processing in the brain.
- Fundamental functional unit of all nervous system tissue.
MSE 2400 Evolution & Learning 4
![Page 5: Neural Networks MSE 2400 EaLiCaRA Spring 2015 Dr. Tom Way](https://reader030.vdocuments.us/reader030/viewer/2022032803/56649e425503460f94b34200/html5/thumbnails/5.jpg)
Brain vs. Digital Computers
- Computers require hundreds of cycles to simulate a firing of a neuron.
- The brain can fire all the neurons in a single step. Parallelism
- Serial computers require billions of cycles to perform some tasks but the brain takes less than a second.
e.g. Face Recognition MSE 2400 Evolution & Learning 5
![Page 6: Neural Networks MSE 2400 EaLiCaRA Spring 2015 Dr. Tom Way](https://reader030.vdocuments.us/reader030/viewer/2022032803/56649e425503460f94b34200/html5/thumbnails/6.jpg)
Comparison of Brain and computer
Human Computer
ProcessingElements
100 Billionneurons
10 Milliongates
Interconnects 1000 perneuron
A few
Cycles per sec 1000 500 Million
2Ximprovement
200,000Years
2 Years
MSE 2400 Evolution & Learning 6
![Page 7: Neural Networks MSE 2400 EaLiCaRA Spring 2015 Dr. Tom Way](https://reader030.vdocuments.us/reader030/viewer/2022032803/56649e425503460f94b34200/html5/thumbnails/7.jpg)
Biological neuron
• A neuron has– A branching input (dendrites)– A branching output (the axon)
• The information circulates from the dendrites to the axon via the cell body
• Axon connects to dendrites via synapses– Synapses vary in strength– Synapses may be excitatory or inhibitory
axon
cell body
synapse
nucleus
dendrites
MSE 2400 Evolution & Learning 7
![Page 8: Neural Networks MSE 2400 EaLiCaRA Spring 2015 Dr. Tom Way](https://reader030.vdocuments.us/reader030/viewer/2022032803/56649e425503460f94b34200/html5/thumbnails/8.jpg)
What is an artificial neuron ?
• Definition : Non linear, parameterized function with restricted output range
1
10
n
iiixwwfy
x1 x2 x3
w0
y
MSE 2400 Evolution & Learning 8
![Page 9: Neural Networks MSE 2400 EaLiCaRA Spring 2015 Dr. Tom Way](https://reader030.vdocuments.us/reader030/viewer/2022032803/56649e425503460f94b34200/html5/thumbnails/9.jpg)
Learning in Neural Networks• The procedure that consists in estimating the parameters
of neurons so that the whole network can perform a specific task
• 2 types of learning– The supervised learning– The unsupervised learning
• The Learning process (supervised)– Present the network a number of inputs and their corresponding
outputs– See how closely the actual outputs match the desired ones– Modify the parameters to better approximate the desired outputs
MSE 2400 Evolution & Learning 9
![Page 10: Neural Networks MSE 2400 EaLiCaRA Spring 2015 Dr. Tom Way](https://reader030.vdocuments.us/reader030/viewer/2022032803/56649e425503460f94b34200/html5/thumbnails/10.jpg)
Supervised learning
• The desired response of the neural network in function of particular inputs is well known.
• A “Professor” may provide examples and teach the neural network how to fulfill a certain task
MSE 2400 Evolution & Learning 10
![Page 11: Neural Networks MSE 2400 EaLiCaRA Spring 2015 Dr. Tom Way](https://reader030.vdocuments.us/reader030/viewer/2022032803/56649e425503460f94b34200/html5/thumbnails/11.jpg)
Unsupervised learning
• Idea : group typical input data in function of resemblance criteria un-known a priori
• Data clustering• No need of a professor
– The network finds itself the correlations between the data
– Examples of such networks :• Kohonen feature maps
MSE 2400 Evolution & Learning 11
![Page 12: Neural Networks MSE 2400 EaLiCaRA Spring 2015 Dr. Tom Way](https://reader030.vdocuments.us/reader030/viewer/2022032803/56649e425503460f94b34200/html5/thumbnails/12.jpg)
Zipcode Example
Examples of handwritten postal codes drawn from a database available from the US Postal service
MSE 2400 Evolution & Learning 12
![Page 13: Neural Networks MSE 2400 EaLiCaRA Spring 2015 Dr. Tom Way](https://reader030.vdocuments.us/reader030/viewer/2022032803/56649e425503460f94b34200/html5/thumbnails/13.jpg)
MSE 2400 Evolution & Learning 13
Character recognition example
• Image 256x256 pixels• 8 bits pixels values
(grey level)
• Necessary to extract features
imagesdifferent 102 1580008256256
![Page 14: Neural Networks MSE 2400 EaLiCaRA Spring 2015 Dr. Tom Way](https://reader030.vdocuments.us/reader030/viewer/2022032803/56649e425503460f94b34200/html5/thumbnails/14.jpg)
Neural Networks
• A mathematical model to solve engineering problems– Group of highly connected neurons to realize compositions of
non linear functions• Tasks
– Classification– Discrimination– Estimation
• 2 types of networks– Feed forward Neural Networks– Recurrent Neural Networks
MSE 2400 Evolution & Learning 14
![Page 15: Neural Networks MSE 2400 EaLiCaRA Spring 2015 Dr. Tom Way](https://reader030.vdocuments.us/reader030/viewer/2022032803/56649e425503460f94b34200/html5/thumbnails/15.jpg)
Feed-forward Networks- Arranged in layers.
- Each unit is linked only in the unit in next layer.
- No units are linked between the same layer, back to the previous layer or skipping a layer.
- Computations can proceed uniformly from input to output units.
- No internal state exists.
MSE 2400 Evolution & Learning 15
![Page 16: Neural Networks MSE 2400 EaLiCaRA Spring 2015 Dr. Tom Way](https://reader030.vdocuments.us/reader030/viewer/2022032803/56649e425503460f94b34200/html5/thumbnails/16.jpg)
Feed Forward Neural Networks
• Information is propagated from inputs to outputs• Can pass through one or more hidden layers
x1 x2 xn…..
1st hidden layer
2nd hiddenlayer
Output layer
MSE 2400 Evolution & Learning 16
![Page 17: Neural Networks MSE 2400 EaLiCaRA Spring 2015 Dr. Tom Way](https://reader030.vdocuments.us/reader030/viewer/2022032803/56649e425503460f94b34200/html5/thumbnails/17.jpg)
Feed-Forward Example
I1
I2
t = -0.5W24= -1
H4
W46 = 1t = 1.5
H6
W67 = 1
t = 0.5
I1
t = -0.5W13 = -1
H3
W35 = 1t = 1.5
H5
O7
W57 = 1
W25 = 1
W16 = 1
MSE 2400 Evolution & Learning 17
![Page 18: Neural Networks MSE 2400 EaLiCaRA Spring 2015 Dr. Tom Way](https://reader030.vdocuments.us/reader030/viewer/2022032803/56649e425503460f94b34200/html5/thumbnails/18.jpg)
Recurrent Network (1)
- The brain is not and cannot be a feed-forward network.
- Allows activation to be fed back to the previous unit.
- Internal state is stored in its activation level.
- Can become unstable
- Can oscillate.
MSE 2400 Evolution & Learning 18
![Page 19: Neural Networks MSE 2400 EaLiCaRA Spring 2015 Dr. Tom Way](https://reader030.vdocuments.us/reader030/viewer/2022032803/56649e425503460f94b34200/html5/thumbnails/19.jpg)
Recurrent Network (2)
- May take long time to compute a stable output.
- Learning process is much more difficult.
- Can implement more complex designs.
- Can model certain systems with internal states.
MSE 2400 Evolution & Learning 19
![Page 20: Neural Networks MSE 2400 EaLiCaRA Spring 2015 Dr. Tom Way](https://reader030.vdocuments.us/reader030/viewer/2022032803/56649e425503460f94b34200/html5/thumbnails/20.jpg)
Recurrent Neural Networks• Can have arbitrary topologies• Can model systems with
internal states (dynamic ones)• Delays are associated to a
specific weight• Training is more difficult• Performance may be
problematic– Stable Outputs may be more
difficult to evaluate– Unexpected behavior
(oscillation, chaos, …)
x1 x2
1
010
10
00
MSE 2400 Evolution & Learning 20
![Page 21: Neural Networks MSE 2400 EaLiCaRA Spring 2015 Dr. Tom Way](https://reader030.vdocuments.us/reader030/viewer/2022032803/56649e425503460f94b34200/html5/thumbnails/21.jpg)
Multi-layer Networks and Perceptrons
- Have one or more layers of hidden units.
- With two possibly very large hidden layers, it is possible to implement any function.
- Networks without hidden layer are called perceptrons.
- Perceptrons are very limited in what they can represent, but this makes their learning problem much simpler.
MSE 2400 Evolution & Learning 21
![Page 22: Neural Networks MSE 2400 EaLiCaRA Spring 2015 Dr. Tom Way](https://reader030.vdocuments.us/reader030/viewer/2022032803/56649e425503460f94b34200/html5/thumbnails/22.jpg)
Perceptrons- First studied in the late 1950s.
- Also known as Layered Feed-Forward Networks.
- The only efficient learning element at that time was for single-layered networks.
- Today, used as a synonym for a single-layer, feed-forward network.
MSE 2400 Evolution & Learning 22
![Page 23: Neural Networks MSE 2400 EaLiCaRA Spring 2015 Dr. Tom Way](https://reader030.vdocuments.us/reader030/viewer/2022032803/56649e425503460f94b34200/html5/thumbnails/23.jpg)
MSE 2400 Evolution & Learning 23
![Page 24: Neural Networks MSE 2400 EaLiCaRA Spring 2015 Dr. Tom Way](https://reader030.vdocuments.us/reader030/viewer/2022032803/56649e425503460f94b34200/html5/thumbnails/24.jpg)
MSE 2400 Evolution & Learning 24
Perceptrons
![Page 25: Neural Networks MSE 2400 EaLiCaRA Spring 2015 Dr. Tom Way](https://reader030.vdocuments.us/reader030/viewer/2022032803/56649e425503460f94b34200/html5/thumbnails/25.jpg)
What can Perceptrons Represent?
Some complex Boolean function can be represented. For example:
Majority function - will be covered in this lecture.
Perceptrons are limited in the Boolean functions they can represent.
MSE 2400 Evolution & Learning 25
![Page 26: Neural Networks MSE 2400 EaLiCaRA Spring 2015 Dr. Tom Way](https://reader030.vdocuments.us/reader030/viewer/2022032803/56649e425503460f94b34200/html5/thumbnails/26.jpg)
Need for hidden units
• If there is one layer of enough hidden units, the input can be recoded (perhaps just memorized)
• This recoding allows any mapping to be represented
• Problem: how can the weights of the hidden units be trained?
MSE 2400 Evolution & Learning 26
![Page 27: Neural Networks MSE 2400 EaLiCaRA Spring 2015 Dr. Tom Way](https://reader030.vdocuments.us/reader030/viewer/2022032803/56649e425503460f94b34200/html5/thumbnails/27.jpg)
Backpropagation
- In 1969 a method for learning in multi-layer network, Backpropagation, was invented by Bryson and Ho.
- The Backpropagation algorithm is a sensible approach for dividing the contribution of each weight.
- Works basically the same as perceptrons
MSE 2400 Evolution & Learning 27
![Page 28: Neural Networks MSE 2400 EaLiCaRA Spring 2015 Dr. Tom Way](https://reader030.vdocuments.us/reader030/viewer/2022032803/56649e425503460f94b34200/html5/thumbnails/28.jpg)
Backpropagation flow
MSE 2400 Evolution & Learning 28
![Page 29: Neural Networks MSE 2400 EaLiCaRA Spring 2015 Dr. Tom Way](https://reader030.vdocuments.us/reader030/viewer/2022032803/56649e425503460f94b34200/html5/thumbnails/29.jpg)
Backpropagation Network training
• 1. Initialize network with random weights• 2. For all training cases (called examples):
– a. Present training inputs to network and calculate output
– b. For all layers (starting with output layer, back to input layer):• i. Compare network output with correct output (error function)• ii. Adapt weights in current layer This is
what you want
MSE 2400 Evolution & Learning 29
![Page 30: Neural Networks MSE 2400 EaLiCaRA Spring 2015 Dr. Tom Way](https://reader030.vdocuments.us/reader030/viewer/2022032803/56649e425503460f94b34200/html5/thumbnails/30.jpg)
Backpropagation Algorithm – Main Idea – error in hidden layers
The ideas of the algorithm can be summarized as follows :
1. Computes the error term for the output units using the observed error.
2. From output layer, repeat - propagating the error term back to the previous layer
and - updating the weights between the two layers until the earliest hidden layer is reached.
MSE 2400 Evolution & Learning 30
![Page 31: Neural Networks MSE 2400 EaLiCaRA Spring 2015 Dr. Tom Way](https://reader030.vdocuments.us/reader030/viewer/2022032803/56649e425503460f94b34200/html5/thumbnails/31.jpg)
How many hidden layers?
• Usually just one (i.e., a 2-layer net)
• How many hidden units in the layer?– Too few ==> can’t learn– Too many ==> poor generalization
MSE 2400 Evolution & Learning 31
![Page 32: Neural Networks MSE 2400 EaLiCaRA Spring 2015 Dr. Tom Way](https://reader030.vdocuments.us/reader030/viewer/2022032803/56649e425503460f94b34200/html5/thumbnails/32.jpg)
How big a training set?• Determine your target error rate, e• Success rate is 1- e• Typical training set approx. n/e, where n is the
number of weights in the net• Example:
– e = 0.1, n = 80 weights– training set size 800
trained until 95% correct training set classification
should produce 90% correct classification
on testing set (typical)
MSE 2400 Evolution & Learning 32
![Page 33: Neural Networks MSE 2400 EaLiCaRA Spring 2015 Dr. Tom Way](https://reader030.vdocuments.us/reader030/viewer/2022032803/56649e425503460f94b34200/html5/thumbnails/33.jpg)
MSE 2400 Evolution & Learning 33
![Page 34: Neural Networks MSE 2400 EaLiCaRA Spring 2015 Dr. Tom Way](https://reader030.vdocuments.us/reader030/viewer/2022032803/56649e425503460f94b34200/html5/thumbnails/34.jpg)
Summary
• Neural network is a computational model that simulate some properties of the human brain.
• The connections and nature of units determine the behavior of a neural network.
• Perceptrons are feed-forward networks that can only represent linearly separable (very simple) functions.
MSE 2400 Evolution & Learning 34
![Page 35: Neural Networks MSE 2400 EaLiCaRA Spring 2015 Dr. Tom Way](https://reader030.vdocuments.us/reader030/viewer/2022032803/56649e425503460f94b34200/html5/thumbnails/35.jpg)
Summary (cont’d)
• Given enough units, any function can be represented by Multi-layer feed-forward networks.
• Backpropagation learning works on multi-layer feed-forward networks.
• Neural Networks are widely used in developing artificial learning systems.
MSE 2400 Evolution & Learning 35