1 artificial neural networks: an introduction s. bapi raju dept. of computer and information...

1

Artificial Neural Networks: An Introduction

S. Bapi Raju Dept. of Computer and Information Sciences,

University of Hyderabad

ANN-Intro (Jan 2010)

2 of 29

OUTLINEBiological Neural NetworksApplications of Artificial Neural NetworksTaxonomy of Artificial Neural NetworksSupervised and Unsupervised Artificial Neural NetworksBasis function and Activation functionLearning RulesApplications OCR, Load Forecasting, Condition Monitoring


3 of 29

Biological Neural NetworksStudy of Neural Networks originates in biological systems

Human Brain: contains over 100 billion neurons, number of synapses is approximately 1000 times that

in electronic circuit terms: synaptic fan-in fan-out is 1000, switching time of a neuron is order of milliseconds But on a face recognition problem brain beats fastest

supercomputer in terms of number of cycles of computation to arrive at answer

Neuronal Structure

• Cell body• Dendrites for input• Axon carries output to other dendrites• Synapse-where they meet• Activation signal (voltage) travels along axon


4 of 29

Need for ANNStandard Von Neumman Computing as existing

presently has some shortcomings. Following are some desirable characteristics in ANN

Learning AbilityGeneralization and AdaptationDistributed and Parallel representationFault ToleranceLow Power requirements

Performance comes not just from the computational elements themselves but the manner of networked interconnectedness of the decision process.


5 of 29VonNeuman

n

versus

Biological

Computer


6 of 29

ANN Applications

Pattern Classification Speech Recognition, ECG/EEG classification, OCR


7 of 29

ANN Applications

Clustering/Categorization Data mining, data compression


8 of 29

ANN Applications

Function Approximation Noisy arbitrary function needs to be approximated


9 of 29

ANN Applications

Prediction/Forecasting Given a function of time, predict the function

values for future time values, used in weather prediction and stock market predictions


10 of 29

ANN ApplicationsOptimization Several scientific and other problems can be

reduced to an optimization problem like the Traveling Salesman Problem (TSP)


11 of 29

ANN ApplicationsContent Based Retrieval Given the partial description of an object

retrieve the objects that match this


12 of 29

ANN ApplicationsControl Model-reference adaptive control, set-point control Engine idle-speed control


13 of 29

Characteristics of ANNBiologically inspired computational unitsAlso called as Connectionist Models or Connectionist ArchitecturesLarge number of simple processing elements Very large number of weighted connections between elements. Information in the network is encoded in the weights learned by the connectionsParallel and distributed controlConnection weights are learned by automatic training techniques


14 of 29Artifical Neuron Working Model

Objective is to create a model of functioning of biological neuron to aid computation

• All signals at synapses are summed i.e. all the excitatory and inhibitory influences and represented by a net value h(.)• If the excitatory influences are dominant, then the neuron fires, this is modeled by a simple threshold function (.)• Certain inputs are fixed biases • Output y leads to other neurons

McCulloch Pitts Model


15 of 29

More about the ModelActivation Functions play a key role Simple thresholding (hard limiting) Squashing Function (sigmoid) Gaussian Function Linear Function

Biases are also learnt


16 of 29

Different Kinds of Network Architectures


17 of 29

Learning AbilityMere Architecture is insufficientLearning Techniques also need to be formulatedLearning is a process where connection weights are adjustedLearning is done by training from labeled examples. This is the most powerful and useful aspect of neural networks in their use as “Black Box” classifiers.Most commonly an input-output relationship is learntLearning Paradigm needs to be specifiedWeight update in learning rules must be specifiedLearning Algorithm specifies step by step procedure


18 of 29

Learning TheoryMajor Factors Learning Capacity: This concerns the

number of patterns that can be learnt and the functions and kinds of decision boundaries that can be formed

Sample Complexity: This concerns the number of the samples needed to learn with generalization. “Overfitting” problem is to be avoided

Computational Complexity: This concerns the computation time needed to learn the concepts embedded in the training samples. Generally the computational complexity of learning is high.


19 of 29

Learning Issues


20 of 29

Major Learning RulesError Correction: Error signal (d–y) used to adjust the weights so that eventually desired output d is produced

Perceptron Solving “AND”

Problem


21 of 29

Major Learning RulesError Correction: in Mutlilayer Feedforward Network

Geometric interpretation of the role of hidden units in a 2D input space


22 of 29

Major Learning RulesHebbian:weights are adjusted by a factor proportional to the activities of the neurons associated

Orientation Selectivity of a Single Hebbian Neuron


23 of 29

Major Learning RulesCompetitive Learning: “winner take all”

(a) Before Learning (b) After Learning


24 of 29Summary of ANN Algorithms


25 of 29


26 of 29

Application to OCR SystemThe main problem in the Handwritten Letter recognition is that characters with variation in thickness shape, rotation and different nature of strokes need to be recognized as of being in the different categories for each letter. Sufficient number of sample training data is required for each character to train the networks

A Sample set of characters in the NIST Data


27 of 29

OCR Process


28 of 29

OCR Example (continued)

Two schemes shown at right• First makes use of the feature extractors• Second uses the image pixels directly


29 of 29

References

A. K. Jain, J.Mao, K.Mohiuddin, “ANN a Tutorial”, IEEE Computer, 1996 March, pp 31-44 (Figures and Tables taken from this reference)

B. Yegnanarayana, Artificial Neural Networks, Prentice Hall of India, 2001.Y. M. Zurada, Inroduction to Artificial Neural Systems, Jaico, 1999.MATLAB neural networks toolbox and manual

1 artificial neural networks: an introduction s. bapi raju dept. of computer and information...

Documents

characteristics of ann

ann standard

ann applications optimization

ann applications content

ocr slide

axon slide

approximated slide

data compression slide