[ieee 2010 international conference on computer and communication technology (iccct) - allahabad,...

Int’l Conf. on Computer & Communication Technology��

___________________________________978-1-4244-��-/10/$26.00©2010 IEEE 526

A Comparative Analysis of Different Neural Networks for Face Recognition Using Principal Component Analysis, Wavelets and Efficient Variable

Learning Rate

Raman Bhati, Sarika Jain, Nilesh MaltareAcropolis Institute of Technology and Research,

Indore, [email protected]

Durgesh Kumar Mishra Acropolis Institute of Technology and Research,

Indore, [email protected]

Abstract: This paper proposes a new way to find the optimum learning rate that reduces the training time and increases the recognition accuracy of the back propagation neural network as well as single layer feed forward Neural Network. It involves feature extraction using principal component analysis and wavelet decomposition and then classification by creation of back propagation neural network. Paper gives acomparative analysis of performance of back propagation neural network and single layer feed forward neural network. In this approach variable learning rate is used and its superiority over constant learning rate is demonstrated. Different inner epochs for different input patterns according to their difficulty of recognition are assigned to patterns. The effect of optimum numbers of inner epochs, best variable learning rate and numbers of hidden neurons on training time and recognition accuracy are also shown.We run our algorithm for face recognition application using Coiflet wavelets, principal component analysis,neural network and demonstrate the effect of numbers of hidden neurons on training time and recognition accuracy for given numbers of input patterns. We use ORL database for all the experiments.

Keywords: Back Propagation neural network, Single layer feed forward neural network, Wavelet Decomposition, Principle Component Analysis, variable learning rate.

I. INTRODUCTION

The task of recognition of human faces is quite complex. The human face is full of information but working with all the information associated with the face is time consuming and less efficient. It is better to use some unique and important information (facial feature vectors) and discard other useless information in order to make system efficient. Face recognition systems can be widely used in areas where more security is needed. For example on Air ports, Military bases, Government offices etc. Also, these systems

can help in places where unauthorized access of persons is prohibited. Sirovich and Kirby [1] had efficiently represented human faces using principal component analysis. M.A. Turk and Alex P. Pentland [2] developed a near real time Eigen faces system for face recognition using Euclidean distance. A face recognition system can be considered as a good system if it can fetch the important features, without making the system complex and can make use of those features for recognizing the unseen faces. For feature extraction we use Principle Component Analysis and for recognition back propagation Neural Network and single layer feed forward network are used. In this paper we give a new approach to recognize the faces in less training time and less training patterns (images). The experiments are carried out on 8 input patterns and 60% training images per input pattern.

A). Principal Component Analysis:

The Principal Component Analysis or PCA is used to extract the features from an intensity image of human frontal face. All the training patterns of same size and configuration are described as the basic face database. In this paper we have used the ORL database for recognition of faces. We apply PCA on this database and get the unique feature vectors using the following method [3]

Suppose there are P patterns and each pattern has t training images of m x n configuration.

.

1. The database is rearranged in the form of a matrix where each column represents an image.

2. With the help of Eigen values and Eigen vectors covariance matrix is computed.

3. Feature vector for each image is then computed. This feature vector represents the signature of the image. Signature matrix for whole database is then computed [4-7]

4. Euclidian distance of the image is computed with all the signatures in the database

.

[8-9].


527

5. Image is identified as the one which gives least distance with the signature of the image to recognize.

B). Wavelet Transformation:

Wavelet analysis allows us to isolate and manipulate some specific type of patterns hidden in masses of data [13]. Although wavelets are popular for image compression but they are also used for pattern recognition [14]

Wavelet transformation allows the generation of such facial characteristics that are invariant to lighting conditions and overlapping. In this paper we use coiflet wavelets for decomposition of face image and a comparison of second level wavelet decomposition and fourth level decomposition is given

. Wavelet analysis is about analyzing signal with short duration finite energy function. They transform the signal under investigation into another representation which represents the signal in more useful form.

[15-16]

The transformation is done using following method:.

The feature size we use in this paper after second wavelet decomposition is 806 and after 4th level decomposition is 110. We make the feature vector by using approximations of coiflet wavelet transform.

C). Neural Network

A Neural Network is made up of neurons residing in various layers of network. These neurons of different layers are connected with each other via links and those links have some values called weights. These weights store the information. Basically the neural network is composed of 3 types of layers: first is Input layer, which is responsible for inserting the information to the network. Second is Hidden layer. It may consist of one or more layers as needed but it has been observed that one or two hidden layers are sufficient to solve difficult problems. The hidden layer is responsible for processing the data and training of the network. Last layer is the output layer which is used to give the network’s output to a comparator which compares the output with predefined target value [10]

Neural networks require training. We give some input patterns for training and some target values and the weights of neural networks get adjusted

.

[11-12]. A Neural network is said to be good and efficient if it requires less training patterns, takes less time for training and is able to recognize more unseen patterns.

The problem with neural network is slow convergence of its weights. Also some more problems have been encountered as the local minima and uncertain behavior of neural networks. For same learning parameters (learning rate) and structuring parameters (hidden neurons), the neural network shows different results.

[13]

In this paper we show the effect of the use of variable learning rate. In each outer epoch we increase the value of learning rate by a very less value. By one outer epoch we mean a single presentation of the all training patterns to the neural network. (All training patterns are applied to the network for one time). This step helps in fast convergence of weights. Using this method the learning rate, started from 0.7, reaches to a high value of 4 to 20 without making system unstable. In this paper we also discuss a balancing parameter (Inner training epoch) which has a great significance of training the network in less time as well as responsible for fast convergence of weights. By n inner epochs we mean n iterations of one training pattern. (A single input pattern applied to the neural network for n times in one outer epoch.)

II. IMPLEMENTATION AND RESULTS

PCA and Neural Networks: In this paper we introduce a new approach to select the learning rate for face recognition. We increase the learning rate by a small value (0.000-0.008) in each outer epoch. This requires less learning time and gives comparatively better recognition accuracy. We run the system for 8 patterns and get 100% recognition accuracy with 60% training images. Due to uncertain behavior of Neural Networks we run a single configuration 15 times, and in each time we get the 100% recognition accuracy. In the experiment we use 10 hidden neurons and the size of feature vectors is set to N=300. Initial learning rate is set to 0.7 and then it is incremented by 0.004. The momentum is fixed to 0.85.Table 1 shows some comparative results. Using back

propagation method, this new approach gives 100% accuracy with 60% training images and takes 290 seconds and 1300 epochs, while the constant learning rate gives 100% accuracy with 60% training images and takes 440 seconds and 3900 epochs. While using single layer feed forward neural network, new approach gives 93.75% of accuracy and constant learning rate gives 68.5% of recognition accuracy. We also give a new analysis of inner epochs. First, we find the optimum inner epochs and then we compare the optimum inner epochs with 1 inner epoch. For 8 patterns, while using 1 epoch we get 100% accuracy in 440 seconds and 3900 outer epochs, using optimum inner epochs the system gives 100% accuracy in 290 seconds. Here in table 1,


528

Complex means that the input pattern is difficult to recognize from other input patterns. Thus more epochs (70) are assigned to that pattern. Similarly normal means that the input pattern is easy to recognize. Thus less inner epochs (12) are assigned to that pattern.

Figure 2 shows the effect of varying hidden neurons over recognition accuracy for different input patterns. To get 100% recognition accuracy for 8 patterns, the range of hidden neurons should be in between 5 to 65. After 65 neurons, the accuracy goes down and training time increases rapidly. The second graph shows the training time for different number of input patterns. Apparently we can see that back propagation method is very sensitive to number of hidden neurons but other network is less sensitive to hidden neurons.

In back propagation method, if we use 20 to 40 hidden neurons, we get 100% recognition accuracy in very less time. When we increase hidden neurons from 40 to 60 the recognition accuracy remains 100% but training time increases. And when we increase the hidden neurons more than 65 the accuracy starts decreasing and after 75, it reaches to zero. The training time shown for feed forward neural network is only for 93.75% accuracy but training time shown for back propagation method.

Table 2 shows a comparative analysis of different neural networks over various step size of learning rate. Apparently we can see that back propagation algorithm is more sensitive to variable learning rate. When step size is increased, the back propagation neural network becomes unstable. While feed forward neural network remains stable.

Table 3 shows the results of different experiments in terms of recognition rate and training time. In the first experiment, 40% of the database is used for training of the neural network and 60% is used for testing. When 60% database is used for training, 93.75% accuracy is achieved in 241 seconds and when 70% database is used for training the same recognition rate is achieved in only 160 seconds.

Wavelets and Neural Networks:In this paper we introduce a new approach to select the

learning rate for face recognition. We increase the learning rate by a small value (0.00-0.1) in each outer epoch. We run the system for 8 patterns and get 100% recognition accuracy with 60% training images. In the experiment we use 20 hidden neurons and the size of feature vectors is set to N=806 (2nd

Table 1 shows that wavelets with single layer neural network give 100% recognition accuracy while PCA and single layer neural networks give only 93%

recognition accuracy. In table 1 we can see that single layer neural network very less sensitive to learning rate. High learning rate does not make the system unstable but is responsible for faster training. When wavelets are used for image processing, this approach takes less time for training in comparison to PCA. Results show that wavelets and neural networks take only 179 seconds but PCA and neural networks take250 seconds to train the network completely. Figure 2 shows that wavelets and PCA both give almost same recognition accuracy on different hidden neurons. But their training times are shown in figure 3 which describes that when wavelets are used for feature extraction, Single layer feed forword neural network takes less time for convergence of weights. Table 3 shows a comparison of different training patterns for wavelets and PCA and two neural networks. Results show that if more training patterns will be given to neural network higher recognition accuracy will be achieved in less training time. The table also shows that in all experiments wavelets have performed better in comparison to PCA. With wavelets if 70% training images are given to neural network, it is trained just in 60 seconds.

level Decomposition). Initial learning rate is set to 0.3 andthen it is incremented by 0.05. The momentum is fixed to 0.85.

III. CONCLUSION

In this paper we used Eigen faces and Coiflet wavelets to represent the feature vectors. This paper introduced a new approach to select the learning rate for back propagation neural network as well as single layer feed forward neural network. The new approach gave better results in all aspects including recognition rate, training time and mean square error. The paper shown that single layer neural network can give 100% recognition accuracy if correct learning rate is assigned to it. This paper shown that back propagation neural network is more sensitive to structuring parameter (hidden neurons) and learning parameters (learning rate). The paper shown that when wavelets are used for feature extraction both neural networks perform better. The paper also gave some comparative analysis like � Effect of inner epochs on training time and

recognition rate.� Effect of various learning rate step sizes on

training time and recognition rate.� Effect of various training patterns on

training time and recognition rate.� Effect of numbers of hidden neurons on

training time.� Effect of numbers of hidden neurons on

recognition accuracy. With all the results shown above we can conclude that this new approach performs better.


529

REFERENCES:[1]. Kirby and Sirovich, 1990. Application of Karhunen-Loeve procedure for the characterization of human faces. IEEE Trans. pattern analysis and machine intelligence, 12:103-108. [2]. Turk, M.A. and A.L. Pentland, 1991. Face recognition using Eigen faces. Proc. IEEE computer society Conf Computer Vision and pattern recognition, pp: 586-591.[3]. Zhujie, Y.L.Y., 1994. Face recognition with Eigen faces. Proc. IEEE Intl. Conf. Industrial Technol. Pp: 434-438.[4]. Firdaus M., 2005.An approach for feature extraction for face recognition. Proc. Conf. Intelligent systems and Robotics(iCISAR2005).[5]. Firdaus M., 2005.Face recognition using neural networks. Proc. Intl. Conf. Intelligent Systems (ICIS), CD-ROM.[6]. Firdaus M.. 2006. Dimensions reductions for face recognition using principal component analysis. Proc. 11th International Symp artificial life and robotics (AROB 11th 06). CD-ROM .[7]. Mathew, J.H., 2003. Eigen values and Eigen Vectors, pp: 612-621.[8]. Demmel, J. and K. veselic, 1989. Jacobi’s method is more accurate than QR. Technical Report: UTCS-89-88.1-60.[9]. Health M.T., 2002. Scientific Computing: An Introductory survey. McGraw hill, pp: 191-200.

[10]. Debipersand, S.C. and A.D Broadhurst, 1997. Face recognition using neural networks. Proc. IEEE Commun. Signal Processing (COMSIG’97), pp: 33-36.[11]. Nazish, 2001. Face recognition using neural networks. Proc. IEEE INMIC 2001, pp: 277-281.[12]. Saad, P., 2001. A comparison of feature normalization techniques on complex image recognition. Proc. 2nd Conf. Information Technology in Asia, pp: 39-409.[13]. D.E Rumelhart, G.E. Hinton and R.J. Williams, Learning internal representation by error propagation, In D.E. Rumelhart and J.L. Mcclelland, eds, parallel distributed processing. Exploration in microstructure in cognition.1 pp:318-362. MIT press, Cambridge, Massachusetts, (1986).[13]. An introduction to wavelets:http://www.amara.com/current/wavelet.html[14]: Daubechies, I., Where do wavelet come from? A personal point of view, Proceedings of IEEE, special issue on wavelets, vol. 84, no 4, pp. 510-513, April 1996.[15]: Feris R., Krueger V., Ceasor R.”A Wavelet subspace method for real time face tracking, real time imaging” 10, pp. 339-350, 2004.[16]: Zhang, D., Biometric Authentication in the e-world, Kluwer Academic Publishers, ch. 16, 2003.

Figure: 1 ORL database images

0 10 20 30 40 50 60 70 80 90 1000

10

20

30

40

50

60

70

80

90

100

Hidden Neurons

Recogn

ition Ac

curacy

Hidden neurons Vs. Recognition Accuracy

PCA+ FFNNWavelets+FFNNPCA+BackpWavelets+Backp

Figure 2: Recognition Accuracy of 2 Neural Networks for different hidden Neurons

0 10 20 30 40 50 60 70 80 90 1000

500

1000

1500

2000

2500

3000

Hidden Neurons

Trainin

g Time

Hidden Neurons Vs. Training time

PCA+FFNNWavelets+FFNNPCA+BackpWavelets+Backp

Figure 3: Training time of 2 Neural Networks for different hidden neuron


530

sTable 1: A comparative analysis of different neural networks over training time and recognition accuracy

Table 2: A comparative analysis of different neural networks over training time and Recognition Accuracy

Feed forward Neural Network Back Propagation Neural NetworkWavelet PCA Wavelet PCA

Learning rate Step

size

Accuracy

Training Time (Sec)

Learning rate Step

size

Accuracy

Training

Time (Sec)

Learning rate Step

size

Accuracy

Training

Time (Sec)

Learning rate Step

size

Accuracy

Training Time (Sec)

0(Constant)

93.75% 1645 0(Constant)

68% 430 0(Constant)

98.4% 262 0(Constant)

80% 410

0.05 96.87% 1336 0.0008 93.75% 380 0.05 100% 236 0.0008 100% 387

0.10 100% 1275 0.001 93.75% 370 0.10 100% 146 0.001 100% 332

1.0 100% 600 0.004 93.75% 350 0.15 100% 150 0.004 100% 290

10.0 100% 308 0.008 93.75% 390 0.20 97% 190 0.008 100% 348

30.0 100% 179 0.01 93.75% 410 0.25 95% 203 0.01 90% 390

50.0 100% 155 0.03 93.75% 413 0.30 94% 210 0.03 40% 420

70.0 100% 136 0.05 93.75% 428 0.35 91.5 222 0.05 24% 434

90.0 90% 250 0.07 93.75% 433 0.4 91.5% 241 0.07 24% 470

100.0 90% 430 0.09 90% 439 0.45 86% 281 0.09 24% 520

110.0 90% 470 0.1 81% 446 0.5 86% 360 0.1 24% 567

Neural Network Patterns Inner Epochs Outer epochs Learning Rate Seconds Recognition Accuracy

Single Layer Feed forward

Neural Network

8 1 5000 Constant 655 68.75%

8 1 3800 Variable 430 90%

8 Complex Pattern 70Normal Pattern 12

1600 Constant 243 93.75%


1000 Variable 148 93.75%

Back Propagation

Neural Network

8 1 3900 Constant 440 100%

8 1 2750 Variable 380 100%


2200 Constant 300 100%


1300 Variable 290 100%


531

Table: 3 Recognition rate and training time for different training sets

Neural Networks Different Percentage of Training

PCA Wavelets

Type Training Set Recognition Rate Training Time Recognition Rate

Training Time

Single Layer Feed forward Neural Network

40% Training 60% Testing 79.6667% 630 Seconds 86.00% 163 Seconds


60% Training 40% Testing 93.7599% 241 Seconds 100% 85 Seconds

70% Training 30% Testing 93.7599% 160 Seconds 100% 60 Seconds

Back Propagation Neural Network

40% Training 60% Testing 87.75% 441 Seconds 92.7%/ 331 Seconds


60% Training 40% Testing 100% 290 Seconds 100% 230 Seconds

70% Training 30% Testing 100% 265 Seconds 100% 105 Seconds

[ieee 2010 international conference on computer and communication technology (iccct) - allahabad,...

Documents