back propagation learning algorithm

Dr.-Ing. Erwin SitompulPresident University

Lecture 5

Introduction to Neural Networksand Fuzzy Logic

President University Erwin Sitompul NNFL 5/1

http://zitompul.wordpress.com

President University Erwin Sitompul NNFL 5/2

Back Propagation Learning AlgorithmMLP for System ModelingNeural Networks

Backwardpropagation

f(.)

f(.)

f(.)

•Set the weights•Calculate output

1

1

( )( ) ( ) ( )

pl l lk k jl

ikj

Ei f net i y i

w

w

1 21

1

( )( ) ( ) ( ) ( )

pl l l l lk k kj j il

iji

Ei f net i w f net i y i

w

w

Forwardpropagation

1( ), ( )l lj ky i y i

•Calculate error•Calculate gradient

vector

•Update the weights

1

1

( ) ( )

( ) ( )

l lk k

ml l lk kj j

j

y i f net i

net i w x i

1 1

1 1 2

1

( ) ( )

( ) ( )

l lj j

nl l lj ji i

i

y i f net i

net i w y i

1

( ) ( ),

l lkj ji

E E

w w

w w

( )lk i

President University Erwin Sitompul NNFL 5/3

Feedforward Network

InputNeuronLayer

NeuronLayer

Output

f(.)

f(.)

f(.)

MLP for System ModelingNeural Networks

President University Erwin Sitompul NNFL 5/4

Feedforward Network

01 2 1 0 2 0y

02 3 1 3 1 0y

21 17 3 9 9 0d

01 ( )y i

02 ( )y i

21 ( )y i

MLP for System ModelingNeural Networks

President University Erwin Sitompul NNFL 5/5

Recurrent NetworksExternal Recurrence

Internal Recurrence

Input NeuronLayer

NeuronLayer

Output

Time Delay

Element

Time Delay

Element

Input NeuronLayer

NeuronLayer

Output

Time Delay

Element

MLP for System ModelingNeural Networks

President University Erwin Sitompul NNFL 5/6

InputDynamicSystem

Output

( )u k ( )y k

Dynamic System

( ) ( , )y k m g

a b( 1), , ( ), ( 1), , ( )y k y k n u k u k n g

System parameter

Input-output data vector

MLP for System ModelingNeural Networks

President University Erwin Sitompul NNFL 5/7

InputDynamic

Model

Output

( )u k ˆ( )y k

Dynamic Model

ˆ( ) ( , , )y k w b g

a b( 1), , ( ), ( 1), , ( )y k y k n u k u k n g

weightsbias

input-output data vector

MLP for System ModelingNeural Networks

President University Erwin Sitompul NNFL 5/8

Neural Network Dynamic Model

Feedforward

ˆ( )y k : model output,estimate of system output

( )y k : system output. . .

. . .

. . .

ˆ( )y k

. . .

( 1)u k

b( )u k n

( 1)y k

a( )y k n

MLP for System ModelingNeural Networks

President University Erwin Sitompul NNFL 5/9

Neural Network Dynamic Model

Recurrent

. . .

. . .

. . .

ˆ( )y k

. . .

( )u k

1z

anz

1z

bnz

MLP for System ModelingNeural Networks

President University Erwin Sitompul NNFL 5/10

Tapped Delay Line (TDL)

( )u k

( 1)u k ( 2)u k

( 3)u k ( )u k n

1z 1z 1z 1z .....

( )u k

( 1)u k ( )u k n

T D L

.....

MLP for System ModelingNeural Networks

Unit 1 Unit 2 Unit 3 Unit n

President University Erwin Sitompul NNFL 5/11

Implementation

InputDynamicSystem

Output

( )u k ( )y k

ˆ( )y k. . .

. . .

T D L T D L

feedforward

external recurrence

MLP for System ModelingNeural Networks

President University Erwin Sitompul NNFL 5/12

ExampleSingle Tank System

2

20.4 m0.012 m

Aa

outq

inq

hLearning Data Generation

A : cross-sectional area of the tanka : cross-sectional area of the pipe

Area of operation

Save data to workspace

MLP for System ModelingNeural Networks

in a

1 ah q v

A A

in

12

ah q gh

A A

President University Erwin Sitompul NNFL 5/13

Example

( 1)u k ( )y k

( 1)y k

Data size : 201 from 200 seconds of

simulation

0 20 40 60 80 100 120 140 160 180 2000

0.02

0.04

0.06

0.08

0.1

0.12

0 20 40 60 80 100 120 140 160 180 2000

0.02

0.04

0.06

0.08

0.1

0.12

Feedforward Network External Recurrent Network

MLP for System ModelingNeural Networks

President University Erwin Sitompul NNFL 5/14

Homework 4

( 1)u k

y k( 2)u k

( 1)y k

( 2)y k

0 20 40 60 80 100 120 140 160 180 200-0.025

-0.02

-0.015

-0.01

-0.005

0

0.005

0.01

0.015

0.02

Delta of 2–2–1 network

4–4–1 Network

MLP for System ModelingNeural Networks

A neural network with 2 inputs and 2 hidden neurons seems not to be good enough to model the Single Tank System. Now, design a neural network with 4 inputs and 4 hidden neurons to model the system. Use bias in all neurons and take all a = 1.

Be sure to obtain decreasing errors.

Submit the hardcopy and softcopy of the m-file.