a novel framework for image fusion based on improved pal ... · complex wavelet transform, curvelet...
TRANSCRIPT
International Journal of Engineering Studies.
ISSN 0975-6469 Volume 8, Number 1 (2016), pp. 47-61
© Research India Publications
http://www.ripublication.com
A Novel Framework for Image Fusion Based on
Improved Pal-King’s Algorithm and Transform
Domain Techniques
H. V. Patil1, S D Shirbahadurkar2
1Associate Professor, NDMVP’s COE, Nashik, Maharashtra, India.
E-mail: [email protected] 2Principal, D Y Patil College of Engineering, Ambi, Pune, India.
E-mail: [email protected]
Abstract
In image fusion literature, transform domain fusion is one of the popular
techniques for efficient image fusion. Although transform domain techniques
are computationally heavy, it has been evident through rigorous
experimentation that in most of the cases, they are superior to spatial domain
methods. Medical images contain very minute edges that have poorer
resolution in spatial domain. If such edges are directly subjected to transform
domain techniques, the quality of resultant fused image may be affected. To
address this problem, a novel approach that utilizes improved Pal-King’s
algorithm for spatial domain enhancement along with subsequent transform
domain fusion has been presented in this paper. The results indicate that
proposed approach yields superior results than conventional transform domain
techniques.
Keywords: Image fusion, Transform domain, Pal-King’s algorithm,
Performance Metrics, Medical Imaging.
1. INTRODUCTION
Up until now, many sensors have been developed for image sensing and subsequent
processing. However, in many scenarios, a single imaging sensor is unable to capture
the complete information about the scene in question [1], [2]. Development of the
image fusion techniques started when the need is felt about merging source images to
yield an amalgamated image that could be utilized to better perform the tasks such as
object detection and recognition [3], [4]. The target image utilizes the complementary
48 H. V. Patil and S D Shirbahadurkar
information available in source images which are captured from same scene [5]. The
resultant image is of higher quality than the source images. The term ‘higher quality’
varies as per the need of the targeted application. Image fusion methods are
extensively used for medical image processing, biometrics, and satellite image
processing [4], [5]. The state-of-the-art image fusion algorithms are categorized into
three major techniques: pixel-level fusion, feature-level fusion, and decision-level
fusion methods [6], [7]. Pixel level fusion techniques depend upon only raw pixel
values either in spatial domain or transform domain [8]. Feature level fusion
techniques extract specific features from the input images to obtain resultant image.
The decision level fusion techniques depend on output of classifier for fusion. This
paper focuses on transform domain pixel level fusion. Most of the latest pixel level
fusion techniques are based on multi-scale transforms such as discrete wavelet
transform [4], [9], Laplacian pyramid [10], stationary wavelet transform [11], [12],
curvelet transform [13], dual tree complex wavelet transform [3]. In general, these
approaches perform forward decomposition of source images to obtain forward
coefficients. Subsequently, merging of coefficients is performed using predefined
fusion rule. The resultant image is yielded upon computation of inverse
transformation [8], [14]. Mankar and Daimiwal [15] proposed a technique which uses
non-subsampled contourlet transform along with pixel and decision level fusion of CT
and MRI images. They have employed contrast enhancement to preserve edge like
features at curves of the image in addition to image smoothing, Gabor directional
filters and derivative based fusion. Singh et al. [16] implemented a Daubechies
complex wavelet transform based fusion scheme which include separate rules for
average and detailed feature coefficients. They have claimed higher quality fused
image because of multi-scale edge information, phase information and shift
invariance of complex wavelet basis function. Biswas et al. [17] computed fused
images from MR and CT images of human spine. They have collected anatomical
details of CT image and functional data of MR image into resultant fused image. A
wiener filtering approach in shearlet domain has been devised. Sharmila et al. [18]
proposed a technique that involves discrete wavelet transform, averaging entropy and
KL transform to yield a fused image. They have concluded that entropy contributes
good quality of information in resultant image. Malarvizhi and Raviraj [19]
implemented a framework based on frequency domain image combination and
standard dynamic range adjustment. The gradients are amalgamated with overall
luminance levels which results in fused image.
Rest of the paper is organized as follows: In section 2, the overview of the dual tree
complex wavelet transform, curvelet transform, non-sub-sampled contourlet
transform, Improved Pal-King’s image enhancement algorithm, and performance
metrics for evaluation of quality of images is presented. A novel combination of
Improved Pal-King’s enhancement algorithm and transform domain fusion techniques
is presented in section 3. In Section 4, results of experimentation using novel
approach are benchmarked with conventional fusion methods. Finally section 5
presents conclusion.
A Novel Framework for Image Fusion Based on Improved Pal-King’s Algorithm 49
2. METHODS AND MATERIALS
This section concisely presents the so far published fundamental literature on which
the proposed approach is based.
2.1 The Dual Tree Complex Wavelet Transform
Despite of remarkable success of the Discrete Wavelet Transform (DWT) for gamut
of image processing applications such as compression; it does not perform well for
image enhancement and restoration due to shift variance and reduced directional
selectivity [20]. Wavelet transforms substitute sinusoidal basis functions of the
Fourier transform which has infinite time duration with set of small oscillating basis
maps called wavelets [21]. Any measurable signal )(tx can be represented in terms of
wavelet scaling functions upon forward decomposition. Due to paucity of shift
invariance, any slight shift in input signal results in dominant variations in magnitude
of energy amongst DWT coefficients at various scales. Reduced directional selectivity
due to orthogonally separable and real wavelet filters produces poor diagonal features.
Kingsbury [20], [21] proposed the dual tree complex wavelet transform (DTCWT)
which improves directional selectivity in two or more dimensions as well as it
maintains near shift invariance. The DTCWT utilizes a dual tree representation of
wavelet filter banks to yield a pair of real and imaginary wavelet coefficients. In
addition, this representation helps to reduce redundancy ( 1:2m ) than the standard
discrete wavelet transform based representation in its maximally decimated form [20].
A general dual tree complex wavelet transform forward decomposition is depicted in
Fig. 1.
Fig.1. The dual tree complex wavelet transform (DTCWT) consisting of Tree 1 and
Tree 2 which yield real and imaginary parts of wavelet coefficients [22].
50 H. V. Patil and S D Shirbahadurkar
2.2 Curvelet Transform
Starck et al. [23] proposed digital domain implementation of the curvelet transform
along with perfect reconstruction capability, perturbation resistance and reduced
number of computations. Large magnitude wavelet coefficients for smooth way
images at fine level scales is one of the major limitations of the discrete wavelet
transform (DWT). Therefore, a need was felt to develop novel representations which
can characterize smooth away images with small nonzero coefficients. The curvelet
transform was developed in order to represent smooth curves with sparse
representation [23]. The curvelet transform is a combination of multilevel ridgelet
transform along with band-pass filtering in spatial domain. Curvelet transform
coefficients possess properties such as variable length, variable width and flexible
anisotropy. The forward decomposition of continuous time signal using discrete
curvelet transform follows dyadic representation ( m2 ) of scales. In addition, filter
bank notations for forward curvelet transform is ,...),,( 21 ggLg where, q is saturated
near ]2,2[ 222 qq as shown in (1).
)2(ˆ)(ˆ , 222 q
qqq g (1)
Subsequently, every single sub-band is processed for smooth partitioning yielding
squares of suitable level (scale) as in (2).
qQQqQq grg )( (2)
Every consequent square is regularized to unit scale to obtain Qd as shown in (3). Unit
scaled squares are decomposed using the discrete ridgelet transform to obtain the
discrete curvelet transform [23].
qqQQQ QQgrTd ),()( 1 (3)
The curvelet transform coefficients have 116 m redundancy at m scales.
2.3 Non Subsampled Contourlet Transform
Cunha et al. [24] developed the non-subsampled contourlet transform (NSCT) for
multilayer, manifold directional decomposition with shift invariance. They have
utilized pyramidal structures and two channel directional filter banks with non-
separable properties for design of over complete NSCT. It offers superior frequency
selection and regularity than state-of-the-art directional transforms [24]. It is effective
for capturing directional edges with less number of linearly dependent basis functions.
Fig.2 illustrates forward decomposition stages involved in NSCT.
A Novel Framework for Image Fusion Based on Improved Pal-King’s Algorithm 51
Fig.2. Forward decomposition of non-subsampled contourlet transform [24].
The non-subsampled pyramid using two channel 2-D filter banks yields various multi-
level decomposition images. Bamberger and Smith [25] proposed directional filter
banks using fan shaped filter structures and re-samplers. NSCT utilizes the directional
filter banks to extract directionally filtered images in the form of wedges [24].
2.4 Improved Pal-King’s Image Enhancement Algorithm
Image enhancement is employed to boost certain features in an image while abating
secondary features in the background without any depreciation in the image quality.
Fuzzy logic based image enhancement algorithms give very interesting results for
medical image enhancement [26], [27], [28], [29]. Pal-King algorithm [30] has been
extensively used by researchers for spatial domain image enhancement using fuzzy
logic. The prominent steps of the Pal-King algorithm [30] are as follows:
(i) Any image I which contains M rows and N columns is represented as a
sampled set of fuzzy points as given in (4). The term ijij / denotes degree
of membership of image pixel ),( jiI with respect to gray level intensity ij .
In general, maxIij . The fuzzy logic characteristic plane is denoted by ij .
NjMiIM
i
N
j
ijij ,...,2,1,,...,2,1 /1 1
(4)
(ii) The value of ij in (4) is computed with the help of fuzzy logic
membership function as in (5). Where, eF stands for exponent fuzzy factor
and dF denotes a reciprocal fuzzy factor.
52 H. V. Patil and S D Shirbahadurkar
eF
dijijij FIF
)/)((1)( max (5)
(iii) A non-linear transformation function ij' is given by (6).
,...4,3,2,1 )),(()( 11' rIII ijrijrij (6)
(iv) An inverse transformation 1F that maps fuzzy enhancement to spatial
domain is stated in (7).
])'(1[)'('
1
max1 eF
ijdijij FIFI
(7)
Shiwei et al. [31] identified the limitations of Pal-King’s approach such as (a) loss of
edge details with low gray scale magnitudes, (b) Histogram or Otsu based crossover
point selection which is computationally complex, (c) parameter optimization
problem of dF , eF . In order to address these shortcomings, they have proposed
‘improved Pal-King’s algorithm’ which is explained in the following steps.
(i) A novel fuzzy membership function has been defined as shown in (8).
))/()(1(log)( minmaxmin2 IIIF ijijij (8)
(ii) A novel transformation relationship has been proposed in (9).
,....5,4,3,2,1 )),(()(' 11 rTTT ijrijrij (9)
Where,
10.5 ,2
1)1(
4
1))5.0(sin(
5.00 ,4
1))5.0(sin(
)(2
2
1
ijij
ij
ijij
ij
ijT
(iii) Reverse mapping of image from fuzzy point domain is required to achieve
enhanced image using improved Pal-King’s algorithm. The grey scale
values of enhanced image are given by (10).
)12)((I'
minmaxmin'ij ijIII
(10)
2.5 Performance Metrics
Various popular performance metrics utilized to access quality of resultant image are
enlisted in this subsection. Quantitative analysis yields better performance when the
application demands autonomous processing of images.
2.5.1 PSNR
PSNR is a ratio between number of grey levels of set of pixels in an image divided by
respective pixels in the fused and reference image. If the value of PSNR is high, it
indicates higher quality of fused image. The formula for computation of PSNR is
given in (11).
A Novel Framework for Image Fusion Based on Improved Pal-King’s Algorithm 53
1
0
1
0
2
2
10
)),(),((1
log20M
i
N
j
fusedref jiIjiIMN
LPSNR (11)
2.5.2 MSE
The intensity level changes between the reference image and resultant fused image are
estimated by subtractive method during computation of MSE. The spectral quality of
resultant fused image can be judged with the help of mean square error (MSE). Lower
the magnitude of MSE, better the fused image. The mathematical expression of MSE
is given as in (12).
1
0
1
0
2)),(),((1 M
i
N
j
fusedref jiIjiIMN
MSE (12)
2.5.3 CC
The degree of similarity of spectral domain features amongst reference image and
resultant fused image is given by CC. If the value of CC is near to 1, it indicates
higher similarity between reference and fused image. The formula for calculation of
CC is given in (13).
fr
r
CC
CCC
f
2 (13)
2.5.4 MI
Mutual Information (MI) is used to compute the degree of similarity between
reference image and fused image. If the value of MI is high, it indicates better quality
of fused image. The mathematical expression for calculation of MI is given by (14).
),(),(
),(log),( 2
1
0
1
0 jihjih
jiIhIjihMI
fr
fr
II
frM
i
N
j
II (14)
2.5.5 SSIM
The comparison between regional patterns of pixel values amongst reference and
resultant fused image is given by SSIM as in (15). Higher the value of SSIM, greater
is the matching between the two images.
))((
)2)(2(
222
122
21
CIICII
CCSSIM
frfr
IIII frfr
(15)
54 H. V. Patil and S D Shirbahadurkar
2.5.6 UIQI
The amount of transformation that an image undergoes from reference image after
performing fusion is computed using UIQI metric. +1 value of UIQI signifies that
both reference and resultant fused image has greater overlap. The mathematical
expression for UIQI is presented in (16).
))((
)(4
2222frfr
IIII
IIIIUIQI
frfr
(16)
3. PROPOSED IMAGE FUSION METHOD
In this section, we illustrate the medical image fusion using transform domain
techniques in details. In the proposed method, two source images are first subjected to
fuzzy logic based contrast enhancement algorithm, and then enhanced images are
subjected to transform domain methods for image fusion. We have enhanced images
in spatial domain using improved Pal-King’s algorithm. The working mechanism of
proposed image fusion method is depicted in Fig. 3. The various methods used for
computing forward transform domain decomposition are dual tree complex wavelet
transform, curvelet transform and non-sub-sampled contourlet transform.
Fig.3. Proposed image fusion method
A Novel Framework for Image Fusion Based on Improved Pal-King’s Algorithm 55
Algorithm used for proposed image fusion is illustrated as follows:
Input: Source image 1, Source image 2, Number of decomposition levels d
Output: Resultant fused image, performance metrics
Algorithm:
Step 1: Read source image 1S and source image 2S .
Step 2: Obtain boosted images 1S and 2S using improved Pal-King’s algorithm
Step 3: Perform forward decomposition using transform domain methods (Dual tree
complex wavelet transform, curvelet transform, non-subsampled contourlet
transform) up to d levels.
Step 4: Perform averaging of respective transform domain coefficients and
reconstruct the resultant fused image 12F using inverse transform.
Step 5: Compute quality of fused image using performance metrics such as PSNR,
MSE, CC, MI, SSIM, and UIQI.
4 EXPERIMENTS AND DISCUSSION
In the following section, we provide details on experimentation using proposed
method and benchmark it with image fusion without enhancement.
4.1 Image fusion using the dual tree complex wavelet transform
In order to reveal the image fusion performance using proposed image fusion
technique, we have performed experimentation on eight image pairs using the dual
tree complex wavelet transform. Each pair is decomposed with four decomposition
levels. Fig. 4 depicts representative images and their resultant fused images without
enhancement and with enhancement. Table 1 represents performance metrics which
are useful for benchmarking purpose.
Table 1: Performance Metrics for DTCWT based fusion
Method PSNR MSE CC MI SSIM UIQI
Without
enhancement
17.04 1465.4 0.6428 0.0647 0.5910 0.377
With
enhancement
(Proposed)
14.54 2510.09 0.5783 0.4591 0.5008 0.3284
56 H. V. Patil and S D Shirbahadurkar
Source Image 1 Source Image 2 Resultant Image
without Enhancement
(a)
Resultant Image with
Enhancement
(Proposed) (b)
Fig.4. Image fusions using the dual tree complex wavelet transform (a) without
enhancement (b) with enhancement (proposed)
4.2 Image fusion using the curvelet transform
The curvelet transform forward decomposition at four levels has been employed to
extract the performance using proposed technique. The experimentation is performed
on 8 image pairs. Fig. 5 illustrates resultant fused image without enhancement and
with enhancement. Comparison of performance metrics between with enhancement
and without enhancement is presented in Table 2.
A Novel Framework for Image Fusion Based on Improved Pal-King’s Algorithm 57
Source Image 1 Source Image 2 Resultant Image
without
Enhancement
(a)
Resultant Image
with Enhancement
(Proposed) (b)
Fig.5. Image fusion using curvelet transform (a) without enhancement (b) with
enhancement (proposed)
Table 2: Performance Metrics for curvelet transform based fusion
Method PSNR MSE CC MI SSIM UIQI
Without
enhancement
17.35 1347.01 0.6276 0.0012 0.5876 0.3771
With
enhancement
(Proposed)
15.10 2227.36 0.5542 0.2843 0.5007 0.2675
4.3 Image fusion using the non-sub-sampled contourlet transform
Image fusion has been implemented with non-sub-sampled contourlet transform at
four decomposition levels. The performance metrics are obtained for eight image
pairs. Fig. 6 shows representative images without and with enhancement. The
performance metrics of resultant images are presented in Table 3.
58 H. V. Patil and S D Shirbahadurkar
Source Image 1 Source Image 2 Resultant Image
without
Enhancement
(a)
Resultant Image
with Enhancement
(Proposed) (b)
Fig.6. Image fusion using non-sub-sampled contourlet transform (a) without
enhancement (b) with enhancement (proposed)
Table 3: Performance Metrics for NSCT based fusion
Method PSNR MSE CC MI SSIM UIQI
Without
enhancement
12.97 3376.84 0.4846 1.7929 0.4342 0.2429
With
enhancement
(Proposed)
14.38 2667.73 0.6025 0.6402 0.5103 0.3346
5. CONCLUSION
The major contribution of presented work lies in proposing a novel image fusion
method which is a combination of pre-processing in spatial domain and averaging
A Novel Framework for Image Fusion Based on Improved Pal-King’s Algorithm 59
based coefficient combination in transform domain and subsequent reconstruction to
yield a resultant fused image. It is evident from the experimentation that the proposed
fusion technique which combines improved Pal-King’s algorithm and conventional
transform domain methods yields superior performance in terms of quality of
resultant fused image.
REFERENCES
1. Zhang, Q. and M.D. Levine, Robust Multi-Focus Image Fusion Using Multi-
Task Sparse Representation and Spatial Context. IEEE Transactions on Image
Processing, 2016. 25(5): p. 2045-2058.
2. Sahu, D.K. and M. Parsai, Different image fusion techniques–a critical review.
International Journal of Modern Engineering Research (IJMER), 2012. 2(5): p.
4298-4301.
3. Lewis, J.J., et al., Pixel- and region-based image fusion with complex
wavelets. Information Fusion, 2007. 8(2): p. 119-130.
4. Pajares, G. and J. Manuel de la Cruz, A wavelet-based image fusion tutorial.
Pattern Recognition, 2004. 37(9): p. 1855-1872.
5. Ardeshir Goshtasby, A. and S. Nikolov, Image fusion: Advances in the state of
the art. Information Fusion, 2007. 8(2): p. 114-118.
6. Pohl, C. and J.L. Van Genderen, Review article Multisensor image fusion in
remote sensing: Concepts, methods and applications. International Journal of
Remote Sensing, 1998. 19(5): p. 823-854.
7. Xu, M., H. Chen, and P.K. Varshney, An Image Fusion Approach Based on
Markov Random Fields. IEEE Transactions on Geoscience and Remote
Sensing, 2011. 49(12): p. 5116-5127.
8. Li, S., B. Yang, and J. Hu, Performance comparison of different multi-
resolution transforms for image fusion. Information Fusion, 2011. 12(2): p.
74-84.
9. Li, H., B.S. Manjunath, and S.K. Mitra, Multisensor Image Fusion Using the
Wavelet Transform. Graphical Models and Image Processing, 1995. 57(3): p.
235-245.
10. Burt, P. and E. Adelson, The Laplacian Pyramid as a Compact Image Code.
IEEE Transactions on Communications, 1983. 31(4): p. 532-540.
11. Beaulieu, M., S. Foucher, and L. Gagnon. Multi- spectral image resolution
refinement using stationary wavelet transform. in Geoscience and Remote
Sensing Symposium, 2003. IGARSS '03. Proceedings. 2003 IEEE
International. 2003.
12. LI, S., MULTISENSOR REMOTE SENSING IMAGE FUSION USING
STATIONARY WAVELET TRANSFORM: EFFECTS OF BASIS AND
DECOMPOSITION LEVEL. International Journal of Wavelets,
Multiresolution and Information Processing, 2008. 06(01): p. 37-50.
13. Nencini, F., et al., Remote sensing image fusion using the curvelet transform.
Information Fusion, 2007. 8(2): p. 143-156.
60 H. V. Patil and S D Shirbahadurkar
14. Liu, Y., S. Liu, and Z. Wang, A general framework for image fusion based on
multi-scale transform and sparse representation. Information Fusion, 2015.
24: p. 147-164.
15. Mankar, R. and N. Daimiwal. Multimodal medical image fusion under
nonsubsampled contourlet transform domain. in Communications and Signal
Processing (ICCSP), 2015 International Conference on. 2015.
16. Singh, R., et al. Mixed scheme based multimodal medical image fusion using
Daubechies Complex Wavelet Transform. in Informatics, Electronics & Vision
(ICIEV), 2012 International Conference on. 2012.
17. Biswas, B., A. Chakrabarti, and K.N. Dey. Spine medical image fusion using
wiener filter in shearlet domain. in Recent Trends in Information Systems
(ReTIS), 2015 IEEE 2nd International Conference on. 2015.
18. Sharmila, K., S. Rajkumar, and V. Vijayarajan. Hybrid method for
multimodality medical image fusion using Discrete Wavelet Transform and
Entropy concepts with quantitative analysis. in Communications and Signal
Processing (ICCSP), 2013 International Conference on. 2013.
19. Malarvizhi, D. and P. Raviraj. Medical image fusion using enhanced gradient
exposure bracketed pairs. in Green Computing Communication and Electrical
Engineering (ICGCCEE), 2014 International Conference on. 2014.
20. Kingsbury, N. The dual-tree complex wavelet transform: A new efficient tool
for image restoration and enhancement. in Signal Processing Conference
(EUSIPCO 1998), 9th European. 1998.
21. Selesnick, I.W., R.G. Baraniuk, and N.C. Kingsbury, The dual-tree complex
wavelet transform. IEEE Signal Processing Magazine, 2005. 22(6): p. 123-
151.
22. Kingsbury, N., Complex Wavelets for Shift Invariant Analysis and Filtering of
Signals. Applied and Computational Harmonic Analysis, 2001. 10(3): p. 234-
253.
23. Jean-Luc, S., E.J. Candes, and D.L. Donoho, The curvelet transform for image
denoising. IEEE Transactions on Image Processing, 2002. 11(6): p. 670-684.
24. Cunha, A.L.D., J. Zhou, and M.N. Do, The Nonsubsampled Contourlet
Transform: Theory, Design, and Applications. IEEE Transactions on Image
Processing, 2006. 15(10): p. 3089-3101.
25. Bamberger, R.H. and M.J.T. Smith, A filter bank for the directional
decomposition of images: theory and design. IEEE Transactions on Signal
Processing, 1992. 40(4): p. 882-893.
26. Chaira, T. Medical image enhancement using intuitionistic fuzzy set. in Recent
Advances in Information Technology (RAIT), 2012 1st International
Conference on. 2012.
27. Chaira, T. Contrast enhancement of medical images using type II fuzzy set. in
Communications (NCC), 2013 National Conference on. 2013.
28. Wen, H. and W. Qi. Enhancement and Denoising Method of Medical
Ultrasound Image Based on Wavelet Analysis and Fuzzy Theory. in 2015
Seventh International Conference on Measuring Technology and
Mechatronics Automation. 2015.
A Novel Framework for Image Fusion Based on Improved Pal-King’s Algorithm 61
29. Ping, W., et al. A Multi-scale Enhancement Method to Medical Images Based
on Fuzzy Logic. in TENCON 2006 - 2006 IEEE Region 10 Conference. 2006.
30. Pal, S. and R. King, Image Enhancement Using Smoothing with Fuzzy Sets.
IEEE Transactions on Systems, Man, and Cybernetics, 1981. 11(7): p. 494-
501.
31. Shiwei, T., Z. Guofeng, and N. Mingming. An improved image enhancement
algorithm based on fuzzy sets. in Information Technology and Applications
(IFITA), 2010 International Forum on. 2010. IEEE.
62 H. V. Patil and S D Shirbahadurkar