NAFIPS 2005 - 2005 Annual Meeting of the North American Fuzzy Information Processing Society

Iterative Image Fusion Technique using Fuzzyand Neuro Fuzzy logic and ApplicationsRahul Ranjan, Harpreet Singh Thomas Meitzler, Grant R. Gerhart

Department ofElectrical and Computer Engineering, U.S. Army TACOM Survivability Technology AreaWayne State University, Detroit, M7 48202 Warren, Ml, 48397-5000

avl069@wayne. edu , hsingh@ece. eng.wayne. edu MeitzleT@tacom.army. mil,gerhartg@tacom.army.mil

Abstract- Image fusion has attracted a widespread attentionowing to applications in medical imaging, automotive andremote sensing. Image fusion deals with integrating dataobtained from different sources of information for intelligentsystems. Image Fusion provides output as a single image froma set of input images obtained from different sources ortechniques. Different approaches in image fusion providedifferent type of results for different applications. Fuzzy andNeuro-Fuzzy algorithms have already been proposed forimage fusion process. The work here further explores theimage fusion technique in iterative fashion using Fuzzy andNeuro Fuzzy approach. We found this technique very usefulin medical imaging and other areas, where quality of image ismore important than the real time application. The work issupplemented by algorithms, its simulation and qualitativeanalysis using Entropy.

I. INTRODUCTION

Fuzzy Logic approach has extensively been used for anumber of applications. Fuzzy and Neuro Fuzzy techniquesfor image fusion have already been proposed andimplemented [9]. Image Fusion has become a topic of greatinterest to a variety of engineers working in differentdisciplines. It is being used for medical applications so as toget a better image. Image fusion has the advantages ofimproving the reliability by removing redundantinformation and improving the capability by keepingcomplementary information. Image fusion and variousmethods of implementing it have been covered in literatureand many journals [1][2][3].Fuzzy logic was proposed by Zadeh [4] and has applicationin large number of fields. Some of the applications ofFuzzy Logic listed in [6] are in the areas of robotics,automobiles and target detection [5]. The subject of imagefusion is developing at a very fast rate as it has applicationsin a number of different fields including medical sciences.Iterative fuzzy and neuro fuzzy approach here is a firtherextension to the approach described for image fusion in [9].[13] explores neuro fuzzy models for dynamic systems incontrol applications.Real time application and quality of the images are twoimportant considerations for the image to be used inindustrial applications. Real time image requires lesserimage data processing and faster computing, so it requiresfast computing platform as well as less complex algorithm.

Fuzzy and Neuro fiuzzy approach can be used for suchapplications along with many other methods suggested inliterature. But for critical applications like medical imagingor satellite imaging or automatic target guidance system,information content need to be reliable and should reflectvarious aspects of the image. This paper explores thesecond requirement so as to get a better image quality andrepresenting better information content. Although timetaken to compute the fused image will be high compared tomethods suggested earlier [9], it is worthwhile to gothrough this method of image fusion.Once we get a fused image from input images, we canfurther use the same image for fusion with the one or theother input images to get a better image quality dependingon the application. We have further extended theapplication ofthe method by using it for fusing medical andlandmine images.

II. FUZZy AND NEURO FUZZY APPROACH TO IMAGEFUSION

Depending on the requirement of industrial applications,new techniques are needed to get a better method of imagefusion. Fuzzy approaches are used where there isuncertainty and no mathematical relations are easilyavailable. These approaches are comparatively easy andsoftware tools are available so that these approaches couldbe implemented.Approach for pixel level image fusion was introduced in[9]. This approach forms an alternative to a large number ofconventional approaches, which are based on a host ofempirical relations. Fuzzy logic approach is based onsimple rules, which are easy to apply and take lesser time.Recent efforts on this topic prove the increasing popularityofthe method.After the image fusion as suggested in [9], between inputimages, we can funrther fuse it with one or both of the inputimages to get a better quality image. When two or morethan two images are given as inputs to the fuzzy or neurofuzzy system they have equal share pixel wise in final fusedoutput. But in the iterative approach, we give priority tosome of the images and the prioritized image is fused morethan once for the final output image. Suppose we have threeimage sources and one is a normal image, the second is IRimage and the third is image from some other source and in

0-7803-91 87-X/05/$20.00 ©2005 IEEE. 706

the final image, we want to have more effect from IRimage. Then we fuse the new output image with the secondIR input image once again and we can do it again if it isrequired. In the algorithm, we have adopted a generalprocess in which we have taken n images, and images havebeen given priority with some index of priority. Index ofpriority for an image decides how many times a image hasto be fused for the final output image. From practical pointof view, we should get the better image at two to threeiterations, when taken into consideration the time taken toget the fused image.

Following algorithms and .M file for pixel levelimage fusion using iterative Fuzzy and Neuro Fuzzy Logicuses FIS editor and ANFIS (Fuzzy Inference System) editorof Fuzzy Logic toolbox in Matlab.

A. ITERATIVEFUZZYLOGICAPPROACHFORIMAGEFUSION:

Algorithm for iterative image fusion using Fuzzy Logicusing FIS editor:

1. Read n images M1, M2, M3..... Mn with priorityindex tI,t2, t3..... tn respectively. Size of each imageshould be same; if not then it should be madesame. Priority index is such thattl.t2 > t3..... .tn >1A priority index tk of a image Mk signifies thatthe particular image should be fused 'k' times forthe final output image.

2. Make a 'fis' file for fusing n input images. Asdescribed in [9], such file has been made withmembership function as 'gaussian' and number ofmembership function '6' for all inputs and outputimages. Make rules for input images, whichresolve the n antecedents to a single number fromOto 255.

3. After that rule of fuzzification based on pixelvalues is applied the same way as described in [9]to get a final image output column with n inputcolumns and using 'evalfis' command.

4. Make a 'fis' file for fusing 2 input images. Asdescribed in [9], such file has been made withmembership function as 'gaussian' and number ofmembership function '6' for both input and outputimages. Make rules for input images, whichresolve the two antecedents to a single numberfrom 0 to 255.

5. If tk > 1 , then for each image it, has to be fusedwith the output image obtained in step (3)

(tk -1) times more for the final output image.If one image Mi undergoes one fusion then

tj=tj+ 1

6. We continue the fusion process with two inputs,in which one of the inputs is the latest output

image column and second is the required inputimage satisfying the condition in (5) until tj < tk.

7. New output column is converted to image matrix.

8. Entropies of all the images are calculated.

For implementing the above algorithm we are taking oneexample where n = 4 andt = 3, t2= 2, t3= 1,t4=1.Testing images are landmine images from four differentsources.

B. MATLAB FILEFOR FUZZYLOGIC

function Y=fusemfile(MI,M2,M3...Mn)%Y = fusemfile (MI, M2) image fusion withIterative Fuzzy Logic method% MI - input image #1 M2 - input image #2%....Mn- input image #n%Size of the input images should be same%Colormapm = gray(256);f=readfis('fuse4.fis');M1=double(M1);M2=double(M2);M3=double(M3);M4=double(M4);im=evalfis([M1 (:) M2(:) M3(:) M4(:)],f);k-1;for i=l:1 :ilrowforj=l:1:ilcol

img(j,i)=im(k);k-k+1;

end;end;fl=readfis('fuse.fis');iml=evalfis([Ml (:) im(:)],fl);k=1;for i=1:1:ilrowforj=l:l:ilcol

imgl(j,i)=iml(k);k=k+1;

end;end;im2=evalfis([M2(:) iml (:)],fl );k=1;for i=1:1:ilrowforj=l:1:ilcol

img2{j,i)=im2(k);k=k+1;

end;end;im3=evalfis([Ml(:) im2(:)],fl);

k=1;for i=1:1:ilrowforj=l:l:ilcol

707

img3(j,i)=im3(k);k-k+1;

end;end; Y= img3;

entropy(M1); entropy(M2); entropy(M3);entropy(M4);entropy(img); entropy(imgl); entropy(img2);entropy(img3);

C. ITERATIVE NEURO-FUZZY APPROACH FORIMAGE FUSION:

Algorithm for iterative image fusion using Neuro FuzzyLogic using ANFIS editor:

1 Read n images M1, M2, M3.,* Mn with priorityindex tI,t2, t3,,,,, tn respectively. Size of each imageshould be same; if not then it should be madesame. Priority index is such that

tI.t2t3..t. 1.A priority index tk of a image Mk signifies thatthe particular image should be fused 'k' times forthe final output image.

2 Training data is made such that there are (n+1)columns corresponding to n input columns andone output column. Each row of the training datacorresponds to the gray value range 0-255.

3 Make a Neuro fuzzy structure using 'fismat' and'anfis' command as described in [9] with thespecified membership function type and number.

4 Checkdata of input image columns(n) is formedand output image after the fusion is obtained asdescribed in [9]

5 Repeat steps of 2 and 3 for n=2. and this newneuro fuzzy structure formed for fusing two inputimages is used for fuirther fusion process.

6 If tk> 1, then for each image Mi, it has to befused with the output image obtained in step (4) (tk-1) times more for the final output image. If oneimage Mi undergoes one fusion thentj=tj+ 1

7 We continue the fusion process with two inputs,in which one of the input is the latest output imagecolumn and second is the required input imagesatisfying the condition in (6) until t1 < tk.

8 Convert the column form to matrix form anddisplay the fused image.

9 Calculate entropies of all the images.

For implementing the above algorithm we are taking oneexample where n = 2 andt==2, t2=2Testing images are medical images.

D. M4TLAB FILEFOR ITERATIVENEURO FUZZYfunction Y= anfism(M1,M2)%Y = anfism(MI, M2) image fusion with

% Iterative Neuro Fuzzy method Y - fused%image% M1 - inputimage #1 M2 - input image #2[zI si] = size(M1);[z2 s2] = size(M2);if(zl = z2) (siI=s2) % check inputserror('Input images are not of same size');

end;m =gray(256);[ilrow ilcol] = size(M1);trnData=[0: 1:255;0: 1:255;0: 1:255];trnData=trnData';%/Ousing images as one columnchkData=[M1(:) M2(:)];epoch n=20;numMFs=5;mfType= 'gaussmf;% to start training we need FIS structurefismat=genfisl (trData,numMFs,mfType);out_fismat=anfis(trnData,fismat,20);im = evalfis(chkData,out_fismat);k=1;for i= l:1 :ilrow

forj=l:1:ilcolimg(j,i)=im(k);k=k+1;

end;end;Y=img;chkData=[im(:) Ml(:)];iml = evalfis(chkData,out_fismat);

%converting the column into imagek=0;for i=1:1:zl

forj=l:1:slk=k+1;imgl(j,i)=iml(k);

end;end;chkData=[M2(:) iml(:)];im2 = evalfis(chkData,out_fismat);

%converting the column into imagek=0;for i=l:1:zl

forj=l:l:slk=k+1;img2(j,i)=im2(k);

end;end;Y=img2;

%Computing entropy for input and fused imagesentropy(M1); entropy(M2); entropy(img);entropy(iml); entropy(im2);

708

III. RESULTS

FusedMed2.bmp

ti = I ,t2=2 t1 =2 ,t2=2Fig. 3 Result of medical images after second and third iteration ofNeuro

Fuzzy approach

tauunnnJ= ev.unhUh--,Fig. 1 Landmine Input images for iterative Fuzzy approach TABLE I

ENTROPY VALUES AT DIFFERENT ITERATION LEVEL WITH FUZZY LOGICAPPROACH( FOUR IMAGES)

LI - I , L2 1, 13 I,14=IFused lb

Iterative Fuzzy App oach for 4 imagesIteration Level Entropy(Image 1) Hi 7.4019(Image 2 ) H2 7.5630(Image 3) H3 7.6681(Image 4 )H4 7.5165

ti = I t2= 1, t3= 1,t4=1 4.9046t1 = 2 t2= 1, t3 = I,t4 =1 6.1862t1 = 2, t2= 2, t3 = l,t4=l 6.5379t1 = 3, t2= 2, t3= 1,14=1 6.6266

ti= 2, t2= 2, t3= I,t4=I ti = 3, t2= 2, t3= I,t4=1Fig. 2 Result ofLandmine images at various iteration level with Fuzzy

approach ofImage fusion.

TABLE IIENTROPY VALUES AT DIFFERENT ITERATION LEVEL WITH NEURO FUZZY

APPROACH (TWO IMAGES)

Iterative Neuro Fuzzy Approach for 2 imagesIteration Level Entropy(Image 1) HI 1.7126(Image 2 ) H2 5.6561tl = 1, t2= 1 4.8794tl = 1, t2= 2 4.2910t = 2, t2= 2 5.0510

IV. QUALITATIVE ANALYSIS OF FUSED IMAGES

It is very difficult to get a unique and universally acceptedquantitative measure for the quality of the image. Textureof an image could be used as a guiding criterion forassessing its quality [8].

709

VI. CONCLUSIONThe concept of entropy is well known in

communication and was proposed by Shannon. Entropy forimages is defined in [12], as "The entropy of an image is ameasure of information content. It is the average number ofbits needed to quantize the intensities in the image. It isdefined as:

L-1H=- p(g)log2 P(g)

g=O(1)

where p(g) is the probability of gray level (g), and therange of g is [0, ...L-l]". Entropy tells us about quality ofthe image if compared for the same type of images as donehere, but for different types of images entropy should notbe used as a tool of comparison.

V. APPLICATIONS

Image fusion has widespread applications in differentdisciplines like medical imaging, navigation and missileguidance for improved accuracy.Fuzzy approaches are used where there is uncertainty andno mathematical relations are easily available. Inapplications where precise and better images for the varietyof information contents are always required, Iterative fuzzyand neuro fuzzy logic produce better results in case of suchapplications. The method suggested here takes more timeto get the fused image compared to the earlier suggestedfuzzy and neuro fuzzy logic approach[9], so real timeapplications like automobile vision enhancement is difficultto implement. But non real time applications where qualityand information content is important, this method will bevery useful.Also, in satellite imaging of the earth, battlefieldsurveillance and automated target guidance, whereprecision in the information content is very important, thesemethods will find a great deal of utility. Automated mobilerobot and its actions based on imaging is one of thepossible areas, which should be explored.Images of mine fields generated using light of differentwavelengths, and/or at different time of the day are fusedtogether to closely estimate the probability of a mineexistence at a particular location [10].

Remote sensing applications [10][11] include thefusion of multi-sensor and multi-temporal images of thesame site acquired at different times, by using neuralnetworks.

Fuzzy and Neuro-Fuzzy algorithms had already beenimplemented to fuse a variety of images. Furtherimprovement in the quality of image can be improved withiterative approach. The results of proposed fusion processare given in terms of Entropy. The image fusion has beenimplemented for medical and landmine images. Thistechnique will be useful in applications which requirequality and critical information imaging. The futureextension of this work could be video image processing forreal time processing. The requirement for the same is fastmicroprocessor speed and implementation of thealgorithm in assembly language or core language, so thatvast image digital data can be processed in real time. Onceimplemented in either embedded systems or VLSI chip, thisalgorithm can be used in many other applications likeautomated mobile robot and automatic navigation system.

REFERENCES

[1] Dasarathy Belur, "Sensor Fusion Potential Exploitation - InnovativeArchitectures and Illustrative Applications," Proceedings ofthe IEEE. vol.85. NO. I January1997.[21 http://www.rockinger.purespace.de/icip97.pdf[31 http:/www.eecs.lehigh.edu/SPCRIJIF/image fusion.htm[4] L. Zadeh "Fuzzy Sets," Inform. Contr., vol. 8,no. 3,pp. 338-353,1965.[5] T. Meitzler, L. Arafeh, H. Singh and G. Gerhart ," Fuzzy logicapproach for computing the probability of target detection in clutteredenvironments," Opt. Eng., vol. 35, pp. 3623-3636, Dec 1996.[6] Labib Arafeh, Harpreet Singh, Sushil Putatunda "A neuro fuzzy logicapproach to material processing," Proceedings ofIEEE. Vol.29, No. 3August 1999.[7] Thomas Meitzler, David Bednarz, E. J. Sohn, Kimberly Lane, DarrylBryk, Gulsheen Kaur, Harpreet Singh, Samuel Ebenstein, Greogory Smith,Yelena Rodin, James Rankin "Fuzzy Logic based Image Fusion,"Aerosense 2002 Orlando, April 2-5, 2002.[8] Rafael C Gonzalez, Richard E Woods. Digital Inage Processing. 3rdEd. Addison Wesley.[9] Harpreet Singh, Jyoti Raj, Gulsheen Kaur, Thomas Meitzler, 4"Imagefusion using fizzy logic and applications", Budapest-04[10] Thomas Meitzler, Darryl Bryk, E.J. Sohn, Kimberly Lane, Jyoti Raj,Hapreet Singh.,"Fuzzy logic based sensor fusion for mine and concealedweapon detection Image fusion techniqes for remote sensing applications"[11] Simone, Giovanni and Farina, Alfonso and Morabito, Francesco andSerpico, Sebastiano Bruno and Bruzzone, Lorenzo (2002). TechnicalReport DIT-02-025, Informatica e Telecomunicazioni, University ofTrento.[12] Yaonon Wang"Multi sensor image fusion: Concept, Method andApplications,"[13] W. Barada, H. Singh, "Generating optimal adaptive fuzzy-neuralmodels of dynamical systems with applications to control", IEEE Trans.Systems Man Cybernet. Part C 28 (1998) 371 391.

710