Ic

PS

a

ARA

KIIWLDJ

1

rtrd

Cbiltwts

tamt[aw

o

0d

Optik 123 (2012) 932– 936

Contents lists available at ScienceDirect

Optik

j o ur nal homepage: www.elsev ier .de / i j leo

mage displacement detection using double phase-encoded joint transformorrelator based on wavelet denoising

eng Ge, Qi Li ∗, Huajun Feng, Zhihai Xutate Key Lab of Modern Optical Instrumentation, Zhejiang University, Hangzhou, China, 310027

r t i c l e i n f o

rticle history:eceived 15 January 2011ccepted 17 June 2011

a b s t r a c t

Since double phase-encoded joint transform correlator (DPEJTC) cannot directly detect the motionbetween image frames of a video sequence with inadequate exposure, an image displacement detectionmethod using DPEJTC based on log-Gabor wavelet denoising is proposed. The method uses a log-Gaborwavelet transform to denoise the reference and the target image obtained in the condition lack of enoughexposure, preserving the phase information of them. Experimental results show that the method can

eywords:mage processmage stabilization

avelet transformack of exposure

successfully accomplish the motion detection. The technology improves the detection ability in condi-tions of low illumination and low contrast, showing great promise for motion detection under thesecircumstances. A possible hybrid electronic–optic setup is proposed.

© 2011 Elsevier GmbH. All rights reserved.

ouble phase-encoded

oint transform correlator (JTC)

. Introduction

Airborne and space-borne imaging systems are often limited inesolution by image degradation resulting from mechanical vibra-ions during exposure. This problem also brings eye fatigue. Highesolution imaging systems are especially sensitive to such degra-ation of the modulation transfer function [1].

Research into image stabilization has been widely studied.ommon methods include mechanical, optical and electronic sta-ilization. Mechanical stabilization is designed to fix the whole

maging system but with the focal plane still moving. Optical stabi-ization adds a number of optical components to the optical systemo compensate the image system. Options include variable opticaledge image stabilization, lens compensation and CCD compensa-

ion image stabilization. And digit image stabilization achieves thetabilization using signal processing methods.

The most important factors involved in digital image stabiliza-ion are motion vector detection and motion compensation. Manylgorithms, such as block matching algorithms (BMA) [2], bit-planeatching algorithms (BPM) [3], projection algorithms (PA) [4], fea-

ure tracking algorithms (FTA) [5], optical flow based algorithms6], phase correlation [7] et al. have been proposed. However, theselgorithms require a huge amount of computation, so they are not

ell suited for real-time usage.

Janschek et al. use an auxiliary matrix image sensor and annboard joint transform optical correlation processor (JTOC) to

∗ Corresponding author. Tel.: +86 571 87951182; fax: +86 571 87951182.E-mail address: liqi@zju.edu.cn (Q. Li).

030-4026/$ – see front matter © 2011 Elsevier GmbH. All rights reserved.oi:10.1016/j.ijleo.2011.07.008

measure real-time 2D image motion, following which the imagemotion of the focal plane is stabilized by a 2-axis piezo-driveassembly [1,8–10]. Because this is a hardware-related method, itcan greatly speed up the process, potentially updating a thousandtimes or more per second so meeting the requirements of real-timeoperation.

As imaging equipment on vehicles, ships, airborne and othermoving systems often have to work in low-light, low contrast envi-ronments, the robustness of JTOC for detection of displacementshas been investigated [11]. In [11] the experimental results showedthat with 1/23 of the normal exposure, the method can detect outthe displacement precisely. However, with 1/24 of the requiredexposure, the technique shows some randomness. Even this motiondetection technique fails with illumination below this level.

The purpose of this paper is to solve the above-mentioned prob-lem, i.e., improve precision of image displacement’s detection atlow exposure and low contrast when illumination is not sufficient,without replacing an ordinary CCD camera with ultra-sensitivecamera. The whole process includes taking the original pictures,carrying out the wavelet transformation, wiping out the noise toenhance the feature of them and sending them to the DPEJTC sys-tem to detect the motion vector.

2. Theory

2.1. The theory of motion vector detection by DPEJTC

Considering a target image having displacement relative to thereference image. Phase-encoding was first proposed by Refregierand Javidi and widely applied in optical image encryption anddecryption [12]. We modify and apply it to image displacement

k 123

md

�

�

tit

r

tttg

f

F

J

*s

P

cmttodgta

H

H

wmbF

P. Ge et al. / Opti

easurement. Define a random phase function �(u, v) in Fourieromain and a phase mask ˚(u, v)

(u, v) = exp[−j˚(u, v)] (1)

(x, y) = ifft2[�(u, v)] (2)

In Eq. (1), �(u, v) is a random phase function uniformly dis-ributed between 0 and 2� in Fourier domain, j denotes anmaginary unit. In Eq. (2) ifft2 is 2D reverse Fourier transform func-ion. We use �(x, y) to encode the reference image r(x,y) as followed

′′(x, y) = r(x, y) ⊗ �(x, y) (3)

Because the target image has a displacement (xi, yi) with regardo the reference image, the target image could be expressed as(x + xi, y + yi). We overlap the phase-encoded reference with thearget image to form the total input image, so the input image isiven by

(x, y) = r′′(x, y) + t(x + xi, y + yi) (4)

Its Fourier spectrum is expressed as followed

(u, v) = fft2[r′′(x, y) + t(x + xi, y + yi)] = fft2[r(x, y) ⊗ �(x, y)]

+ fft2[t(x + xi, y + yi)] = R(u, v)�(u, v)

+ T(u, v) exp(−iuxi − ivyi) (5)

And joint power spectrum (JPS) is obtained by

PS(u, v) = F(u, v)2 = R(u, v)2 + T(u, v)2

+ R(u, v)�(u, v)T(u, v)∗ exp(iuxi + ivyi)

+ R(u, v)∗�(u, v)∗T(u, v) exp(−iuxi − ivyi) (6)

denotes conjugate operator. Then we multiply Eq. (6) with theame phase mask �(u, v), getting phase-encoded JPS (PJPS),

JPS = JPS(u, v)�(u, v) = R(u, v)2�(u, v) + T(u, v)2�(u, v)

+ R(u, v)�2(u, v)T(u, v)∗ exp(iuxi + ivyi)

+ R(u, v)∗T(u, v) exp(−iuxi − ivyi) (7)

After applying a reverse Fourier transformation to Eq. (7) theorrelation output will be obtained. Sometimes the size of phaseask is not big enough relative to the input image or the con-

ent of the input image is too complex, so the noises generated byhe phase mask will affect the cross-correlation. And the side-lobef the cross-correlation peak is too wide. Alam proposed a filterefined as H(u, v) = |R(u, v)|−2 in classic JTC [13]. Because the tar-et image is not exactly the same as the reference image due tohey have displacement with respect to each other, here we define

filter

(u, v) = |R(u, v)|−1 (8)

Eq. (7) is multiplied with Eq. (8), we get HPJPS as

PJPS(u, v) = R(u, v)ϕR(u, v)�(u, v) + 1|R(u, v)|T(u, v)2�(u, v)

+ ϕR(u, v)�2(u, v)T(u, v)∗ exp(iuxi + ivyi)

+ ϕR(u, v)∗T(u, v) exp(−iuxi − ivyi)] (9)

here ϕR(u, v) = 1|R(u,v)| R(u, v), is the phase part of R(u, v). The for-

er three items in Eq. (9) have a random phase function. They wille scattered into system noises in the output plane when a reverseourier transform is applied to them. The fourth item in Eq. (9) is

(2012) 932– 936 933

the cross-correlation between the reference and the target image.We decompose ϕR(u, v) into two independent components as,

ϕR(u, v) = exp(iuϕx + ivϕy) (10)

where ϕx and ϕy are phases part along u,v axis. So the Fourth itemin Eq. (9) is expressed as

C(u, v) = T(u, v) exp[−iu(xi + ϕx) − iv(yi + ϕy)] (11)

Finally we apply a reverse Fourier transformation to Eq. (11) andget correlation output as

c(x, y) = t(x + xi + ϕx, y + yi + ϕy) (12)

By the filter the cross-correlation peak will become very sharpfor displacement detection and noise effect will be lessened. Firstwe let the target image be the same as the reference when xi = 0 andyi = 0 to fix the ideal position where cross-correlation peak appearsat (xideal,yideal) given by{

xideal = −ϕx

yideal = −ϕy(13)

Then the target image is not the same as the reference imagewhen the translation is not zero, the cross-correlation peak’s posi-tion (xpeak, ypeak) is detected by{

xpeak = −(xi + ϕx)ypeak = −(yi + ϕy)

(14)

So the displacement (xi, yi) is given by{xi = xideal − xpeak

yi = yideal − ypeak(15)

2.2. The principle of wavelet denoising

A wavelet transform is a transformation over the time and fre-quency domain. It is effective in extracting information from thesignal, and can analyse a function or signal in multi-scale by stretch-ing and shifting them, etc. The wavelet transform proves veryeffective in distinguishing signal and noise components separately.Two-dimensional wavelet transform of f(x, y) is defined as

WTf (cx, cy, dx, dy)

= (cxcy)−1/2∫ ∫

f (x, y)h∗(

x − dx

cx,

y − dy

cy

)dxdy

=∫ ∫

f (x, y)h∗(x, y)dxdy (16)

In Eq. (16), cx, cy represents the expanding factor, and dx, dy rep-resents the shifting factor. hcd(x, y) is the mother wavelet function,while h(x, y) is the function that expands and shifts the motherwavelet function.

The main steps required to use the wavelet to denoise the imageare as follows:

Step 1: Calculate the two-dimensional wavelet transformationof the images and decompose the coefficients of the results of thewavelet transform.

Multi-resolution is the most important concept in wavelet anal-ysis. The image is expressed as a low-frequency component anddifferent resolution under a combination of high frequency com-ponents. After a two-dimensional wavelet transform, four subimages of the same size will be obtained, denoted as the horizon

low-frequency and vertical low-frequency Ci, the horizon low-frequency vertical high-frequency D1

i, the horizon high-frequency

vertical high-frequency D2i

and the horizon high-frequency ver-tical high-frequency D3

i. Of these, Ci can be decomposed further.

9 k 123

W[a

Fdt

34 P. Ge et al. / Opti

avelets based on complex valued log-Gabor functions is used here14,15]. These functions have a Gaussian transfer function whenre viewed on the logarithmic frequency scale. Two functions in

ig. 1. (a) Image with normal exposure; (b) reference image; (c) target image; (d)enoised image of (b); (e) denoised image of (c); (f) phase-encoded of (d); (g) thearget image overlaid with (f); (h) correlation output.

(2012) 932– 936

quadrature are used. We convolve the image f(x, y) with each of thequadrature pairs of wavelets. Me

n, Mon denote the even-symmetric

and odd-symmetric wavelets at a scale n separately, so we get

[en(x, y), on(x, y)] = [f (x, y) ∗ Men, f (x, y) ∗ Mo

n] (17)

where * denotes convolution. en(x, y), on(x, y) denote real and imagi-nary parts of complex valued frequency component. The amplitudeof the transform at a given wavelet scale is given by

An(x, y) =√

en(x, y)2 + on(x, y)2 (18)

And the phase is given by

˚n(x, y) = tan−1(

on(x, y)en(x, y)

)(19)

At each position (x,y) in an image we have an array of theseresponse vectors, one vector for each scale of filter.

Step 2: Modify the transformed coefficients.The coefficients resulting from the signal are usually centralized,

while the coefficients resulting from the noise are decentralized.So the energy from the signal is concentrated on a few of coef-ficients which different from the noise. The first step is to selectan appropriate threshold. Then shrink the magnitudes of all thecoefficients by the threshold while leaving the phase unchanged.Estimated image can be reconstructed by summing the remainedcoefficients over all scales and orientations. Here a self-adaptive

threshold scheme is adopted [16].

Step 3: Transform the inverse wavelet and reconstruct thewavelet coefficients. Reconstruct the images according to the coef-ficients achieved in step 2. The denoised image will be obtained.

Fig. 2. Performacne over a video.

P. Ge et al. / Optik 123

Tab

le

1Pe

rfor

man

ce

over

vid

eos

un

der

dif

fere

nt

shu

ttle

tim

e.

Shu

tter

tim

e

RM

SE

on

H

wit

hou

td

enoi

sin

gR

MSE

on

Hw

ith

den

oisi

ng

RM

SE

on

V

wit

hou

td

enoi

sin

gR

MSE

on

Vw

ith

den

oisi

ng

Shu

tter

tim

e

RM

SE

on

H

wit

hou

td

enoi

sin

gR

MSE

on

Hw

ith

den

oisi

ng

RM

SE

on

V

wit

hou

td

enoi

sin

gR

MSE

on

Vw

ith

den

oisi

ng

1/18

0s0.

0176

0.00

82

0.02

33

0.01

32

1/10

00s

0.09

65

0.04

54

0.02

49

0.01

271/

250s

0.03

06

0.01

95

0.01

27

0.00

78

1/20

00s

1.94

62

0.14

86

1.92

38

0.09

221/

350s

0.02

59

0.01

29

0.01

39

0.00

74

1/30

00s

8.66

55

0.10

61

13.3

775

0.11

921/

500s

0.01

18

0.00

69

0.01

74

0.00

73

1/40

00s

6.63

34

0.08

16

10.7

410

0.15

441/

750s

0.03

24

0.01

03

0.04

70

0.01

52

1/80

00s

33.3

882

0.31

65

34.2

484

0.35

97

(2012) 932– 936 935

3. Results

The experimental images were captured by a Panasonic cam-era NV-MX300EN/A. Its aperture can be tuned from F = 1.6 to F = 16.Using a video capture card the captured sequences are transformedto a computer. The resolution of the images is 640 × 480. Theparameters for normal exposure are with aperture size F = 1.6, shut-ter time t = 1/50 s. One frame with normal exposure is shown asFig. 1(a). We fix the aperture size and choose different shuttertime from t = 1/8000s, 1/4000s, 1/3000s, 1/2000s, 1/1000s, 1/750s,1/500s, 1/350s, 1/250s and 1/180s. So the exposure under theseshutter time is not adequate. We take F = 1.6, t = 1/1000s as an exam-ple, this condition reduces the exposure to approximately 1/24.5

of normal. Sub part of two experimental pictures are shown asFig. 1(b and c). Their resolution is 256 by 256 pixels. Real displace-ment between them is (−9.19, −6.65) pixels. Fig. 1(b) is selected asthe reference image and Fig. 1(c) is the target image. The waveletdenoised images of Fig. 1(b and c) is shown as Fig. 1(d and e) respec-tively. Here wavelet scale we choose is 6. Phase-encoded of Fig. 1(d)is shown as Fig. 1(f). Total input images including the denoisedtarget and phase-encoded denoised reference image is shown asFig. 1(g). Correlation output is shown as Fig. 1(h). Measured dis-placement is (−9.10, −6.62) pixels. Errors are (0.09, 0.03) pixels.Performance over a video is shown as Fig. 2(a and b). Fig. 2(a) showserrors of displacement on vertical (V) axis using DPEJTC based onwavelet denoising compared to DPEJTC without denoising. Fig. 2(b)shows errors of displacement on horizontal (H) axis using DPEJTCbased on wavelet denoising compared to DPEJTC without denois-ing. Mean square errors (MSE) on H axis and V axis by DPEJTCbased on wavelet denoising is 0.0454, 0.01227 pixels compared to0.0965 and 0.0249 pixels without denoising. The results obviouslyshow that by wavelet denoising the performance of displacementmeasurement based on DPEJTC has improved.

Then we test the performance of DPEJTC based on waveletdenoising over videos captured with other different shutter times.RMSE along vertical (V) and horizontal (H) axis are shown inTable 1. From the table we know when shutter time is longerthan t = 1/1000s, RMSE of displacement measurement using DPEJTCcould be within 0.1 pixels even without denoising the reference andtarget image. But when the shutter time is shorter than 1/2000s,

the noises will bring a serious influence on the detection preci-sion. DPEJTC could not detect the displacement accurately withoutdenoising the input image. After denoising the reference and target

Fig. 3. A possible hybrid optics-electronic set-up.

9 k 123

is

ttdtttatmitbeglriotptpTipis

4

dmmTDnphi

[

[

[

[

[

[15] D.J. Field, Relations between the statistics of natural images and theresponse properties of cortical cells, J. Opt. Soc. Am. A 4 (12) (1987)2379–2394.

36 P. Ge et al. / Opti

mage, RMSE could be within 0.5 pixels even in the condition thathutter time is 1/8000s, that is 1/27.5 of normal exposure.

DPEJTC could be realized by hybrid optic-electronic set-up dueo its high calculation speed based on optical Fourier transforma-ion. A possible set-up is shown as Fig. 3. We accomplish the waveletenoising process to r(x, y), t(x, y) by computer 1 obtaining r′(x, y),

′(x, y). A two dimensional random function ˚(u, v) uniformly dis-ributed between 0 and 2� is generated by Computer1 (COM1). Sohe random phase mask �(u, v) is generated using Eq. (4). SLM1 is

phase and amplitude combination modulator. SLM2 is an ampli-ude modulation and SLM3 is a phase-only modulation. Phase-only

odulation can be realized through displaying a uint8 gray levelmage on SLM3. SLM1 and SLM2 is at front focal plane of Fourierransform lens1 (FTL1) and FTL2 respectively. CCD1, CCD2 is atack focal plane of Fourier transform lens 1 and lens 2. The phasencoded denoised reference image overlaid with the denoised tar-et image is displayed on SLM1 through COM1. Laser generated byaser 1(L1) are collimated by collimator lens 1(CL1) and reflected byeflective mirrors 1(RM1), perpendicularly passing through polar-zed beam splitter 1(PBS1) to SLM1. After modulated and reflectedff SLM1, the reflected light is Fourier transformed by FTL1 andhe JPS is captured by CCD1. The JPS is sent to COM2 and multi-lied by a filter obtaining the HJPS. The HJPS is displayed on SLM2hrough COM2. Laser generated by L2 is collimated by CL2 andhase only modulated by SLM3. It passes through PBS2 to SLM2.he reflected light modulated by SLM2 at front focal plane of FTL2s Fourier transform by FTL2 and correlation output at the back focallane of FTL2 is captured by CCD2. Finally digital image processing

s performed on the correlation output to get the displacement inub-pixel precision.

. Conclusion

Log-Gabor complex wavelet transformation is applied toenoise images with inadequate illumination. The phase infor-ation is preserved and the threshold is self-adaptive. Then theotion vector can be determined through the DPEJTC algorithm.

he results data show that this image displacement detection usingPEJTC based on wavelet denoising can work well in low illumi-

ation and low contrast condition. RMSE could remain within 0.5ixels even the exposure is about 1/27.5 of normal exposure. Itas great promising application in image registration under low

llumination and low contrast condition.

[

(2012) 932– 936

Acknowledgments

This research was supported by the National Basic Research Pro-gram (973) of China (Grant No. 2009CB724002).

References

[1] K. Janschek, V. Tchernykh, S. Dyblenko, Integrated camera motion compen-sation by real-time image motion tracking and image deconvolution [C], in:Proceedings, 2005 IEEE/ASME International Conference on Advanced Intelli-gent Mechatronics, 2005, pp. 1437–1444.

[2] F. Vella, A. Castorina, M. Mancuso, G. Messina, Digital image stabilization byadaptive block motion vectors filtering, IEEE Trans. Consum. Electr. (2002)796–801.

[3] S.J. Ko, S.H. Lee, K.H. Lee, Digital image stabilizing algorithms based on bit-planematching, IEEE Trans. Consum. Electr. (1998) 617–622.

[4] C. Garcia, G. Tziritas, Optimal projection of 2-D displacements for 3-D transla-tional motion estimation, Image Vision Comput. 20 (11) (2002) 793–804.

[5] C. Morimoto, R. Chellappa, Fast electronic digital image stabilization, in: IEEEProceedings of ICPR’96, 1996, pp. 284–288.

[6] J.Y. Chang, W.F. Hu, M.H. Cheng, B.S. Chang, Digital image translational and rota-tional motion stabilization using optical flow technique, IEEE Trans. Consum.Electr. (2002) 108–115.

[7] H. Foroosh, J.B. Zerubia, M. Berthod, Extension of phase correlationto sub-pixel registration, IEEE Trans. Image Process. 11 (3) (2002)188–194.

[8] K. Janschek, V. Tchernykh, S. Dyblenko, et al., Compensation of the attitudeinstability effect on the imaging payload performance with optical correlators,Acta Astronaut. 52 (9–12) (2003) 965–974.

[9] K. Janschek, V. Tchernykh, S. Dyblenko, et al., Compensation of focal planeimage motion perturbations with optical correlator in feedback loop, sen-sors systems and next-generation satellites VIII, Proc. SPIE 5570 (2004)280–288.

10] K. Janschek, V. Tchernykh, S. Dyblenko, Performance analysis of opto-mechatronic image stabilization for a compact space camera, Control Eng. Pract.15 (3) (2007) 333–347.

11] P. Ge, Y.T. Chen, Q. Li, et al., Robustness study of motion detection based onjoint transform correlator without enough illumination, Acta Optica Sinica 29(7) (2009) 1796–1800 (in Chinese).

12] P. Refregier, B. Javidi, Optical image encryption based on input plane and Fourierplane random encoding, Opt. Lett. 20 (1995) 767–769.

13] M.S. Alam, M.A. Karim, Fringe-adjusted joint transform correlation, Appl. Opt.32 (23) (1993) 4344–4350.

14] N. Delprat, B. Escudié, P. Guillemain, et al., Asymptotic wavelet and Gabor anal-ysis: extraction of instantaneous frequencies, IEEE Trans. Inf. Theory 38 (1992)644–664.

16] P. Kovesi, Phase preserving denoising of images, in: Proceeding of DICTA 1999Conference, 1999, pp. 212–217.