Method of combining information from a multichannel system, using wavelet spectra

V. V. Teterin, V. A. Pavlova, and V. A. Aleksandrov

S. I. Vavilov State Optical Institute All-Russia Scientific Center, St. Petersburg�Submitted March 29, 2006�Opticheski� Zhurnal 73, 47–51 �October 2006�

This paper proposes a method of sorting the information contained in video streams formed bysensors based on various physical principles. The method consists of estimating the degree ofsignificance of local information from each sensor according to a definite information-contentcriterion, followed by blending of the discriminated most significant information into a single,combined image. The results of computer modeling showed that the proposed model has advan-tages over known methods in terms of response rate, comfort of presentation, and the absence ofartifacts. © 2006 Optical Society of America.

INTRODUCTION

Intense development of combined multichannel systemshas been recently observed in which channels are used toobtain video information based on various physical prin-ciples �visual, laser, radar, daylight television �TV�, low-level TV, thermal vision �ThV�, and other channels�.

The main reason for combining optoelectronic systems�OESs� is that interfering factors of natural and artificial ori-gin have different effects on different channels for obtaininginformation, since each of the indicated channels, takenseparately, cannot satisfy the technical requirements on theapparatus under conditions of poor visibility, careful mask-ing, and active electronic countermeasures.

The purpose of combining channels is to form the result-ant images by analyzing the information in each channel,separating out the maximum of the local information accord-ing to a definite information-content criterion, and blendingthe selected information in a separate combined channel �thefusion problem�. In this case, the efficiency of the system isgreater in terms of the basic parameters �range of action,recognition probability, tracking accuracy, etc.� than that ofeach of the separate channels �the synergetic effect�.1

Preliminary studies have shown that the use of synthe-sized images in solving the problems of the automatic detec-tion and recognition of specified objects as well as in au-totracking systems usually provides increased correct-recognition probability and improves the reliability of theautoselection, decreases the probability of mistracking, etc.

The information of the various channels in combinedOESs can be blended at various levels of information pro-cessing: one can blend data, blend attributes, or blend deci-sions.

Each of the indicated levels of blending has its own ad-vantages and disadvantages. Blending �of image elements� atthe data level is carried out at the stage of obtaining infor-mation from various sensors. This method of complexinginformation is most suitable for sensors that have identicalworking parameters: the rate at which the data are formedand their dimension and format.

A typical example of data blending at the level of imageelements is the joint processing of information obtained fromTV and ThV channels or from two ThV channels that operate

698 J. Opt. Technol. 73 �10�, October 2006 1070-9762/2006/1

in different spectral ranges. The joint processing and sortingof the information of TV and ThV channels make it possibleto form combined information that contains informationfrom both channels.

Before the procedure of blending the images, they mustbe normalized and brought into coincidence in order to beable to combine the corresponding image elements. Whenblending images, one should pay special attention to match-ing the fields of view of the individual channels. Reference 2proposed a formal semantic apparatus for combining the datafrom different sensors and implemented a method of blend-ing the images at the level of the segmentation data, usinginformation concerning the intensity of and the distance tothe object of observation. The proposed algorithm corre-sponds to intuitive methods for combining information.

Methods of blending images are based on traditionalmethods of digital image processing. Reference 3 proposed aneural network for blending the data of TV and ThV chan-nels.

In order to completely utilize the possibilities of blend-ing at the image level, the images must be brought into co-incidence pixel by pixel. This condition is automatically sat-isfied in multichannel optical systems with coincident opticalfields �with a common optical axis�. The images obtainedfrom each individual channel must be appropriately cali-brated, amplified, and matched in order that the informationfrom each channel can be blended into a single pattern inorder to evaluate it efficiently. It is required in practical ap-plications that the resultant image not contain any spuriousartifacts and that it be comfortable to observe.

ALGORITHMS FOR BLENDING AT THE LEVEL OF THEIMAGE ELEMENTS

The blending process must not only combine the infor-mation of the different channels into a single output imagebut must also discriminate the most essential parts of theimages on the basis of the informational attributes. In theresultant image, those elements of the initial images must beselected that have greater information content in terms of achosen local attribute. By a local attribute in this case ismeant the computation of a certain numerical criterion in agiven aperture that includes the element being sought.

69800698-04$15.00 © 2006 Optical Society of America

at use

To estimate the local information content, the presencewas determined in a certain neighborhood �aperture� withsize n�n around the image element under consideration �thepixel� of a sufficient number of brightness falloffs. This pres-ence was determined either by estimating the strength of thecontours �in terms of the rms deviation� in the given aper-ture, or simply from the mean modulus of the deviation ofthe brightness relative to the central point of the aperture.

A block diagram of the blending algorithm using theattribute of the rms deviation of the brightness in shown inFig. 1. Both of the initial images are simultaneously scannedby a window with size n�n, and the rms deviation of thebrightness in the aperture is read at each step �i , j� of thescanning, with the result being stored in auxiliary arraysCA�ij� and CB�ij� at an address corresponding to the coordi-nates of the center of the aperture. Two auxiliary matricesCA�ij� and CB�ij� are thus formed that have the meaning ofthe local rms deviations for the initial images. After this, apair-by-pair comparison is carried out of the elements of thetwo matrices in order to determine the degree of local infor-mation content at a given position of the sliding aperture.The radiance of the pixel from the initial images for whichthe rms deviation in the auxiliary matrix is larger is selectedin the resultant �combined� image.

The results of computer modelling using actual imagesshowed that the individual features of each sensor cause the

FIG. 2. Block diagram of a blending algorithm that uses t

FIG. 1. Block diagram of a blending algorithm th

699 J. Opt. Technol. 73 �10�, October 2006

images formed by them to be significantly different fromeach other in their brightness characteristics, while the TVand ThV images as a rule are mutually negative. Therefore,the problem arises of normalizing �standardizing� the imagesof the different channels before the blending procedure.Without normalization of the images, the blending procedureoften causes the combined image to display artifacts thathave the character of an appliqué.

To eliminate these difficulties, we developed a blendingalgorithm based on the use of the wavelet spectra of theinitial images.

METHOD OF COMBINING INFORMATION USING WAVELETSPECTRA

The idea of using a discrete wavelet transformation tosolve the fusion problem springs from the well-known JPEGmethod of data compression. In this algorithm, the Fouriercoefficients are analyzed by applying the discrete Fouriertransformation to segments of the image and then eliminat-ing the zero Fourier coefficients, as well as the low-information coefficients whose modulus is less than a givenvalue.

Unlike Fourier transformations, where an infinitely os-cillating basis function �a sine wave� makes it impossible toobtain localized information, wavelet basis functions are

iterion of the greatest modulus of the wavelet coefficient.

s the criterion of the greatest local rms deviation.

he cr

699Teterin et al.

well localized, and this makes it possible to carry out localspectral analysis.4 The spectral wavelet coefficients corre-spond not only to the amplitudes of various frequencies butalso to various spatial sections on the image. This makes itunnecessary to do a preliminary breakup of the image intolocal regions, since this possibility is already included in thewavelet transformation; moreover, there are no side effects�artifacts� in this case. It also becomes unnecessary to use asliding window to carry out local analysis of the informationcontent of segments of the image, as was done in the blend-ing method considered above at the level of the image ele-ments �see Fig. 1�.

The blending algorithm developed here for blendingvideo data from different physical channels on the basis of ananalysis of the information content of the wavelet coeffi-cients is thus the following sequence of operations �see theblock diagram in Fig. 2�:

• synchronous wavelet transformation of the images of thetwo channels and the formation of matrices WA�ij� andWB�ij� of the spectral wavelet coefficients for the initialimages A�ij� and B�ij�, respectively;

• comparison of the corresponding pairs of wavelet coeffi-cients in terms of the modulus, and choosing the larger ofthem;

• formation of the matrix of the resultant wavelet spectrum;• inverse wavelet transformation to obtain a combined

image.

To carry out a rapid algorithm for computing the wavelettransformation, a system of Haar wavelets is chosen as thebasis, possessing the properties of orthogonality and com-pactness of the carrier. In this case, in accordance with therequirements of the Haar basis,5 the size of the initial imagesmust be a multiple of 2n.

The analytical expression for the Haar wavelet functionis

�kj = �1 when j/2k � t � �j + 1�/2k,

− 1 when �j + 1�/2k � t � �j + 2�/2k,

otherwise 0.� �1�

The most widely used method for computing the two-dimensional wavelet transformation is to apply a one-dimensional wavelet transformation to each row of the imageand then apply a one-dimensional wavelet transformation toeach column of the image. This method is easy to use, sinceit requires repeated implementation of the one-dimensionalwavelet transformation.

FIG. 3. Blocks of coefficients of the wavelet spectrum W�ij�.

700 J. Opt. Technol. 73 �10�, October 2006

In matrix form, the wavelet transformation can be writ-ten as

�H�A = W, �2�

FIG. 4. The result of the informational blending of the information of TV�a� and ThV �b� images, using the criterion of the greatest modulus of thecoefficients of their wavelet spectra �c�.

700Teterin et al.

where A is the column vector of the initial image, with 2n

elements, W is a column vector of coefficients with dimen-

701 J. Opt. Technol. 73 �10�, October 2006

sion 2n�1, and �H� is the unit matrix of the Haar transfor-n n

mation, with dimension 2 �2 :

�3�

The upper half of the transformation matrix in Eq. �3�corresponds to obtaining the averaging wavelet coefficients,and the lower corresponds to obtaining the detailed coeffi-cients. Therefore, the wavelet transformation is usually rep-resented as the use of low-frequency and high-frequency fil-ters. The low-frequency filter transmits informationcontaining a small number of details �averaging coefficients�,and the low-frequency filter contains information concerningfine details �detailing coefficients�.

The successive application of a one-dimensional wavelettransformation to each row and then to each resulting columnwas thus carried out at each step in the general case of amultiple wavelet transformation. As a result, the followingblocks of coefficients are formed: HHk is the result of theaction of a high-frequency filter �H� to a row and then to acolumn, where k is the step of the wavelet transformation,LHk is the result of the action of a low-frequency filter �L� toa column that has already undergone the action of a high-frequency filter, HLk is the result of the action of high-frequency processing of columns that have already under-gone the action of low-frequency processing, and LLk is theresult of the application of only low-frequency processing.This is illustrated in Fig. 3 for two steps �k=2� of the wavelettransformation.

The information of two channels �images A and B in Fig.2� are directly blended by pairwise comparison of the spec-tral coefficients of the corresponding wavelet spectra WA�ij�and WB�ij�. The significance of the coefficients is deter-mined by their absolute magnitude; i.e., the coefficient

whose absolute value is greater is chosen in the resultant�combined� spectrum W�ij�:

W�ij� = �WA�ij�, if �WA�ij�� � �WB�ij�� ,otherwise WB�ij� .

�4�

The desired combined image C�ij� is formed as a resultof the inverse wavelet transformation.

It should be pointed out that various other versions of thecombination of the resultant wavelet spectrum W�ij� fromthe initial WA�ij� and WB�ij� are possible, and this is deter-mined by the requirements of the specific applied problems.We carried out computer modelling using actual TV and ThVimages according to the selection criterion of Eq. �4� andwith a single-step �k=1� wavelet transformation.

As an illustration, Fig. 4 shows the result of the infor-mational blending of the information of TV �a� and ThV �b�images according to the criterion of the greatest value of themodulus of the coefficients of their wavelet spectra �c�.

1R. McDaniel and D. Scribner, “Image fusion for tactical application,”Proc. SPIE 3436, 685 �1998�.

2C. Pohl and J. Van Genderen, “Multisensor image in remote sensing,” Int.J. Remote Sens. 19, 823 �1998�.

3A. Pongsak and L. Terranse, “Neural network model for fusion of visibleand infrared sensor outputs,” Proc. SPIE 1003, 153 �1988�.

4A. P. Petukhov, Introduction to the Theory of Bases of Bursts �SPbGTU,St. Petersburg, 1999�.

5N. M. Astaf’eva, “Wavelet analysis: Foundations of theory and examplesof its use,” Usp. Fiz. Nauk 166, 1145 �1996� �Phys. Usp. 39, 1085�1996��.

701Teterin et al.