blind deconvolution and structured matrix computations

1

Blind Deconvolution and Structured MatrixComputations with Applications to ArrayImaging

Michael K. Ng and Bob PlemmonsHong Kong Baptist University and Wake Forest University

CONTENTS1.1 Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.2 One Dimensional Deconvolution Formulation. . . . . . . . . . . . . . . . . . . . . . . . . . . . 21.3 Regularized and Constrained TLS Formulation. . . . . . . . . . . . . . . . . . . . . . . . . . . 61.4 Numerical Algorithms. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111.5 Two Dimensional Deconvolution Problems. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181.6 Numerical Examples. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 211.7 Application: High-resolution Image Reconstruction. . . . . . . . . . . . . . . . . . . . . . 231.8 Concluding Remarks and Current Work. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37

Acknowledgments. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39References. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39

In this chapter, we study using total least squares (TLS) methods for solving blinddeconvolution problems arising in image recovery. Here, the true image is to be es-timated using only partial information about the blurring operator, or point spreadfunction, which is also subject to error and noise. Iterative, regularized, and con-strained TLS methods are discussed and analyzed. As an application, we study TLSmethods for the reconstruction of high-resolution images from multiple under sam-pled images of a scene that is obtained by using a charge-coupled device (CCD) or aCMOS detector array of sensors which are shifted relative to each other by subpixeldisplacements. The objective is improving the performance of the signal-processingalgorithms in the presence of the ubiquitous perturbations of displacements aroundthe ideal subpixel locations because of imperfections in fabrication, etc., or becauseof shifts designed to enable superresolution reconstructions in array imaging. Thetarget architecture consists of a regular array of identical lenslets whose images aregrouped, combined, and then digitally processed. Such a system will have the reso-lution, field of view, and sensitivity of a camera with an effective aperture that wouldbe considerably larger than the single-lenslet aperture, yet with a short focal lengthtypical of each lenslet. As a means for solving the resulting blind deconvolutionproblems, the errors-in-variables (or the TLS) method is applied.

0-8493-0052-5/00/$0.00+$.50CRC Press LLC 1

2 Blind image deconvolution: theory and applications

1.1 Introduction

The fundamental issue in image enhancement or restoration is blur removal in thepresence of observation noise. Recorded images almost always represent a degradedversion of the original scene. A primary example is images taken by an optical in-strument recording light that has passed through a turbulent medium, such as theatmosphere. Here, changes in the refractive index at different positions in the at-mosphere result in a non-planar wavefront [43]. In general, the degradation by noiseand blur is caused by fluctuations in both the imaging system and the environment.

In the important case where the blurring operation isspatially invariant, the ba-sic restoration computation involved is a deconvolution process that faces the usualdifficulties associated with ill-conditioning in the presence of noise [9, Chapter 2].The image observed from a shift invariant linear blurring process, such as an opticalsystem, is described by how the system blurs a point source of light into a largerimage. The image of a point source is called thepoint spread functionPSF, whichwe denote byh. The observed imageg is then the result of convolving the PSFhwith the “true” imagef, and with noise present ing.

Thestandard deconvolution problemis to recover the imagef given the observedimageg and the point spread functionh. See, e.g., the survey paper on standarddeconvolution written by Banham and Katsaggelos [3]. If the PSFh is not known,then the problem becomes one ofblind deconvolution, sometimes called myopicdeconvolution ifh is partially known, see, e.g., the survey paper on blind or myopicdeconvolution written by Kundar and Hatzinakos [28].

We also mention the recent work of Bardsley, Jefferies, Nagy and Plemmons [4]on the blind restoration of images with an unknown spatially-varying PSF. Their al-gorithm uses a combination of techniques. First, they section the image, and thentreat the sections as a sequence of frames whose unknown PSFs are correlated andapproximately spatially-invariant. To estimate the PSFs in each section phase di-versity, see, e.g. [49], is used. With these PSF estimates in hand, they then use aninterpolation technique to restore the image globally.

In the sections to follow, we concentrate primarily on blind deconvolution. Em-phasis is given to the use of TLS methods where the blurring operator has a natural,structured form determined by boundary conditions. Applications to the construc-tion of high-resolution images using low-resolution images from thin array imagingcameras with millimeter diameter lenslets and focal lengths are provided to illustratethe techniques discussed in this chapter.

Blind Deconvolution and Structured Matrix Computations with Applications to Array Imaging3

1.2 One Dimensional Deconvolution Formulation

For simplicity of notation we begin with the 1-dimensional deblurring problem. Con-sider the original signal

f = (· · · , f−n, f−n+1, · · · , f0, f1, · · · , fn, fn+1, · · · , f2n, f2n+1, · · ·)t

and thediscrete point spread functiongiven by

h = (· · · ,0,0,h−n+1, · · · ,h0, · · · ,hn−1,0,0, · · ·)t .

Here “t” denotes transposition. The blurred signal is the convolution ofh andf, i.e.,the i-th entrygi of the blurred signal is given by

gi =∞

∑j=−∞

hi− j f j . (1.1)

Therefore, the blurred signal vector is given by

g = [g1, · · · , gn]t .

For a detailed discussion of digitizing images, see Castleman [9, Chapter 2].From (1.1), we have

g1

g2...

gn−1

gn

=

hn−1 · · · h1 h0 · · · h−n+1 0...

...... h−n+1

hn−1...

.... ..

0 hn−1 · · · h0 h−1 · · · h−n+1

f−n+2...f0

−−−f1...fn

−−−fn+1

...f2n−1

= [Hl |Hc|Hr ]

f l

−−fc

−−fr

= [Hl |Hc|Hr ]f. (1.2)

Here[A|B] denotes anm-by-(n1+n2) matrix whereA andB arem-by-n1 andm-by-n2

matrices respectively.


For a givenn, the deconvolution problem is to recover the vector[ f1, · · · , fn]t giventhe point spread functionh and a blurred signalg = [g1, · · · , gn]t of finite lengthn.Notice that the blurred signalg is determined not only byfc = [ f1, · · · , fn]t , but byf = [f l fc fr ]t .

The linear system (1.2) is underdetermined. To recover the vectorfc, we assumethe data outsidefc are reflections of the data insidefc, i.e.,

f0 = f1...

......

f−n+2 = fn−1

and

fn+1 = fn...

......

f2n−1 = f2.

(1.3)

In [36], it has been shown that the use of this (Neumann) boundary condition canreduce the boundary artifacts and that solving the resulting systems is much betterand faster than using zero and periodic boundary conditions. For example, revertingto the 2-dimensional case for illustration purposes, see Figures 1.1–1.4. The detaileddiscussion of using the Neumann boundary condition can be found in [36].

In classical image restoration, the point spread function is assumed to be knownor adequately sampled [1]. However, in practice, one is often faced with impreciseknowledge of the PSF. For instance, in two dimensional deconvolution problems aris-ing in ground-based atmospheric imaging, the problem consists of an image receivedby a ground-based imaging system, together with an image of a guide star PSF ob-served under the effects of atmospheric turbulence. Empirical estimates of the PSFcan sometimes be obtained by imaging a relatively bright, isolated point source. Thepoint source might be a natural guide star or a guide star artificially generated usingrange-gated laser backscatter, e.g., [6, 22, 33]. Notice here that the PSF as well asthe image are degraded by blur and noise.

Because the spatial invariance ofh translates into the spatial invariance of thenoise in the blurring matrix. We assume that the “true” PSF can be represented bythe following formula:

h = h+δh, (1.4)

whereh = [h−n+1, . . . ,hn−1]t

is the estimated (or measured) point spread function and

δh = [δh−n+1,δh−n+2, · · · ,δh0, · · · ,δhn−2,δhn−1]t ,

is the error component of the PSF. Eachδhi is modeled as independent uniformlydistributed noise, with zero-mean and varianceσ2

h , see for instance [34]. The blurredsignalg is also subject to errors. We assume that the observed signalg= [g1, . . . ,gn]t

can be represented byg = g+δg, (1.5)

whereδg = [δg1,δg2, · · · ,δgn]t


0

5

10

15

20

0

5

10

15

200

0.005

0.01

0.015

0.02

0.025

0.03

0.035

05

1015

2025

0

5

10

15

20

250

0.005

0.01

0.015

0.02

0.025

0.03

FIGURE 1.1Gaussian (atmospheric turbulence) blur (left) and out-of-focus blur (right).

FIGURE 1.2Original image (left), Noisy and blurred satellite image by Gaussian (middle)and out-of-focus blur (right).

rel. error =1.24×10−1 rel. error =1.15×10−1 rel. error =6.59×10−2

FIGURE 1.3Restoring Gaussian blur with zero boundary (left), periodic boundary (middle)and Neumann boundary (right) conditions.

rel. error =1.20×10−1 rel. error =1.09×10−1 rel. error =4.00×10−2

FIGURE 1.4Restoring out-of-focus blur with zero boundary (left), periodic boundary (mid-dle) and Neumann boundary (right) conditions.


andδgi is independent uniformly distributed noise with zero-mean and varianceσ2g .

Here the noise in the PSF and in the observed signal are assumed to be uncorrelated.Thus our image restoration problem is to recover the vectorf from the given inexactpoint spread functionh and a blurred and noisy signalg.

1.3 Regularized and Constrained TLS Formulation

In the literature, blind deconvolution methods, see e.g. [12, 25, 28, 39, 40, 50], havebeen developed to estimate both the true imagef and the point spread functionhfrom the degraded imageg. In order to obtain a reasonable restored image, thesemethods require one to impose suitable constraints on the PSF and the image. In ourimage restoration applications, the PSF isnot known exactly (e.g., it is corrupted byerrors resulting from blur and/or noise). A review of optimization models for blinddeconvolution can be found in the survey paper by Kundar and Hatzinakos [28].

Recently there has been growing interest and progress in using total least squares(TLS) methods for solving these blind deconvolution problems arising in imagerestoration and reconstruction, see e.g., [16, 26, 34, 38, 51, 52]. It is well-knownthat the total least squares (TLS) is an effective technique for solving a set of errorcontaminated equations [17, 23]. The TLS method is an appropriate method for con-sideration in astro-imaging applications. In [26], Kamm and Nagy have proposedthe use of the TLS method for solving Toeplitz systems arising from image restora-tion problems. They applied Newton and Rayleigh quotient iterations to solve theToeplitz total least squares problems. A possible drawback of their approach is thatthe point spread function in the TLS formulation is not constrained to be spatially-invariant. Mesarovic, Galatsanos and Katsaggelos [34] have shown that formulatingthe TLS problem for image restoration with the spatially-invariant constraint im-proves the restored image greatly, see the numerical results in [34].

The determination off given the recorded datag and knowledge of the PSFh isan inverse problem [15]. Deconvolution algorithms can be extremely sensitive tonoise. It is necessary to incorporate regularization into deconvolution to stabilize thecomputations. Regarding the regularization, Golub, Hansen and O’Leary [16] haveshown how Tikhonov regularization methods, for regularized least squares computa-tions, can be recast in a total least squares framework, suited for problems in whichboth the coefficient matrix and the right-hand side are known only approximately.However, their results do not hold for the constrained total least squares formulation[34]. Therefore, we cannot use the algorithm in [16].

Here we develop constrained TLS approach to solving the image restoration prob-lem, using the one dimensional case for simplicity of presentation.

By using (1.35) and (1.36), the convolution equation (1.2) can be reformulated asfollows:

Hf−g+δHf−δg = 0, (1.6)


where

H =

hn−1 · · · h1 h0 · · · h−n+1 0...

...... h−n+1

hn−1...

..... .

0 hn−1 · · · h0 h−1 · · · h−n+1

= [Hl |Hc|Hr ] (1.7)

and

δH =

δhn−1 · · · δh1 δh0 · · · δh−n+1 0...

...... δh−n+1

δhn−1...

.... . .

0 δhn−1 · · · δh0 δh−1 · · · δh−n+1

= [δHl |δHc|δHr ].

(1.8)Correspondingly, we can define the Toeplitz matricesHl , Hc andHr , andδHl , δHc

andδHr similar to Hl , Hc andHr in (1.2), respectively. The constrained total leastsquares formulation amounts to determining the necessary “minimum” quantitiesδH andδg such that (1.6) is satisfied.

Mathematically, the constrained total least squares formulation can be expressedas

minfc‖[δH|δg]‖2

F subject to Hf−g+δHf−δg = 0,

wheref satisfies (1.3).Recall that image restoration problems are in general ill-conditioned inverse prob-

lems and restoration algorithms can be extremely sensitive to noise [18, p.282]. Reg-ularization can be used to achieve stability. Using classical Tikhonov regularization[15, p.117], stability is attained by introducing a regularization operatorD and a reg-ularization parameterµ to restrict the set of admissible solutions. More specifically,the regularized solutionfc is computed as the solution to

minfc{‖[δH|δg]‖2

F + µ‖Dfc‖22} (1.9)

subject toHf−g+δHf−δg = 0, (1.10)

andf satisfies (1.3). The term‖Dfc‖22 is added in order to regularize the solution. The

regularization parameterµ controls the degree of regularity (i.e., degree of bias) ofthe solution. In many applications [11, 18, 24],‖Dfc‖2 is chosen to be theL2 norm‖fc‖2 or theH1 norm‖Lfc‖2 whereL is a first order difference operator matrix. Inthis paper, we only consider theL2 andH1 regularization functionals.

In [34], the authors addressed the problem of restoring images from noisy mea-surements in both the PSF and the observed data as a regularized and constrainedtotal least squares problem. It was shown in [34] that the regularized minimizationproblem obtained is nonlinear and nonconvex. Thus fast algorithms for solving this


nonlinear optimization problem are required. In [34], circulant, or periodic, approx-imations are used to replace the convolution matrices in subsequent computations.In the Fourier domain, the system of nonlinear equations is decoupled into a setof simplified equations and therefore the computational cost can be reduced signif-icantly. However, practical signals and images often do not satisfy these periodicassumptions and ringing effects will appear on the boundary [32]. In the imageprocessing literature, various methods have been proposed to assign boundary val-ues, see Lagendijk and Biemond [29, p.22] and the references therein. For instance,the boundary values may be fixed at a local image mean, or they can be obtained by amodel-based extrapolation. In this paper, we consider the image formulation modelfor the regularized constrained TLS problem using theNeumann boundary conditionfor the image, i.e., we assume that the scene immediately outside is a reflection ofthe original scene near the boundary. This Neumann boundary condition has beenstudied in image restoration [29, 32, 36] and in image compression [31, 45]. Resultsin [36] show that the boundary artifacts resulting from the deconvolution computa-tions are much less prominent than that under the assumption of zero [26] or periodic[34] boundary conditions.

The theorem below characterizes the constrained, regularized TLS formulation ofthe one dimensional deconvolution problem.

THEOREM 1.1

Under the Neumann boundary condition (1.3), the regularized constrainedtotal least squares solution can be obtained as the fc that minimizes the func-tional:

P(fc) = (Afc−g)tQ(fc)(Afc−g)+ µftcDtDfc, (1.11)

where A is an n-by-n Toeplitz-plus-Hankel matrix

A = Hc +[0|Hl ]J+[Hr |0]J, (1.12)

J is the n-by-n reversal matrix,

Q(fc) = ([T(fc)|I ][T(fc)|I ]t)−1 ≡ [T(fc)T(fc)t + I ]−1,

T(fc) is an n-by-(2n−1) Toeplitz matrix

T(fc) =1√n

fn fn−1 · · · f2 f1 f1 · · · fn−2 fn−1

fn fn. . . . . . f2 f1

. . . fn−3 fn−2...

. . . . . . . . . . . . . . . . . . . . ....

f3 f4. . . fn fn−1

. . . . . . f1 f1f2 f3 · · · fn fn fn−1 · · · f2 f1

, (1.13)

and I is the n-by-n identity matrix.


PROOF From (1.10), we have

[T(fc)|I ][√

nδh−δg

]=√

nT(fc)δh−δg = δHf−δg = g−Hf = g−Afc. (1.14)

We note that

‖[δH|δg]‖2F = n‖δh‖2

2 +‖δg‖22 =

∣∣∣∣∣∣∣∣[√

nδh−δg

]∣∣∣∣∣∣∣∣2

2.

Therefore, we obtain the minimum 2-norm solution of the underdeterminedsystem in (1.14), see for instance [17]. Since the rank of the matrix [T(fc)|I ]is n, we have [√

nδh−δg

]= [T(fc)|I ]tQ(fc)(g−Afc)

or ∣∣∣∣∣∣∣∣[√

nδh−δg

]∣∣∣∣∣∣∣∣2

2= (Afc−g)tQ(fc)(Afc−g). (1.15)

By inserting (1.15) into (1.9), we obtain (1.11).

1.3.1 Symmetric Point Spread Functions

The estimates of the discrete blurring function may not be unique, in the absence ofany additional constraints, mainly because blurs may have any kind of Fourier phase,see [29]. Nonuniqueness of the discrete blurring function can in general be avoidedby enforcing a set of constraints. In many papers dealing with blur identification[12, 19, 30], the point spread function is assumed to be symmetric, i.e.,

hk = h−k, k = 1,2, · · · ,n−1.

We remark that point spread functions are often symmetric, see [24, p.269], for in-stance, the Gaussian point spread function arising in atmospheric turbulence inducedblur is symmetric with respect to the origin. For guide star images [19], this is usuallynot the case. However, they often appear to be fairly symmetric, which can be ob-served by measuring their distance to a nearest symmetric point spread function. In[19], Hanke and Nagy use the symmetric part of the measured point spread functionto restore atmospherically blurred images.

Similarly, we thus incorporate the following symmetry constraints into the totalleast squares formulation of the problem:

hk = h−k and δhk = δh−k, k = 1,2, · · · ,n−1. (1.16)

Then using Neumann boundary conditions (1.3), the convolution equation (1.6) be-comes

Afc−g+δAfc−δg = 0.


whereA is defined in (1.12) andδA is defined similarly. It was shown in [36] thatthese Toeplitz-plus-Hankel matricesA and δA can be diagonalized by ann-by-ndiscrete cosine transform matrixC with entries

Ci j =

√2−δi1

ncos

((i−1)(2 j−1)π

2n

), 1≤ i, j,≤ n,

whereδi j is the Kronecker delta, see [24, p.150]. We note thatC is orthogonal, i.e.,CtC = I . Also, for anyn-vectorv, the matrix-vector multiplicationsCv andCtv canbe computed inO(nlogn) real operations by fast cosine transforms (FCTs); see [42,pp.59–60].

In the following discussion, we write

A = Ct diag(w) C and δA = Ct diag(δw) C. (1.17)

Here for a general vectorv, diag(v) is a diagonal matrix with its diagonal entriesgiven by

[diag(v)]i,i = vi , i = 1,2, · · · ,n.

Using (1.17), we can give a new regularized constrained total least squares formula-tion to this symmetric case as follows:

THEOREM 1.2Under the Neumann boundary condition (1.3) and the symmetry constraint

(1.16), the regularized constrained total least squares solution can be obtainedas the fc that minimizes the functional

P(fc) = [diag(w)fc− g]t{[diag(fc)|I ][diag(fc)|I ]t}−1[diag(w)fc− g]+ µ ftcΛfc,(1.18)

wheref = Cf, g = Cg,

and Λ is an n-by-n diagonal matrix given by

Λ = CDtDCt ,

and D is the regularization operator.

PROOF In this regularized total least squares formulation, we minimize‖[δA|δg]‖2

F + µ‖Dfc‖22 subject to Afc−g+δAfc−δg = 0. Since A and δA can

be diagonalized by C, the constraint now becomes

diag(w)fc− g+diag(δw)fc− δg = 0.

Let us define y = [diag(fc)]−1diag(δw)fc. It’s easy to show that

[diag(fc)|I ][

y−δ g

]= diag(fc)y−δ g = diag(δw)fc− δg = g−diag(w)fc.


Hence we have[

y−δ g

]= [diag(fc)|I ]t{[diag(fc)|I ][diag(fc)|I ]t}−1(g−diag(w)fc) (1.19)

The diagonalization of A by C implies that

‖[δA|δg]‖2F

= ‖Ctdiag(δw)C‖2F +‖Ctδ g‖2

2 = ‖diag(δw)‖2F +‖δ g‖2

2 =∣∣∣∣∣∣∣∣[

y−δ g

]∣∣∣∣∣∣∣∣2

2

Now, by using (1.19), it’s easy to verify that ‖[δA|δg]‖2F + µ‖Dfc‖2

2 is equal tothe second member of (1.18).

We recall that when theL2 and H1 regularization functionals are used in therestoration process, the main diagonal entries ofΛ are just given by

Λii = 1 and Λii = 4cos2(

(i−1)π2n

), 1≤ i ≤ n,

respectively.

1.4 Numerical Algorithms

In this section, we introduce an approach to minimizing (1.11). For simplicity, welet

∂Q(fc)∂ fi

=

∂Q11(fc)∂ fi

· · · ∂Q1n(fc)∂ fi

......

∂Qn1(fc)∂ fi

· · · ∂Qnn(fc)∂ fi

.

Here∂Q jk(fc)/∂ fi is the derivative of the( j,k)-th entry ofQ(fc) with respect tofi .By applying the product rule to the matrix equality

Q(fc)[T(fc)T(fc)t + I ] = I ,

we obtain∂Q(fc)

∂ fi=−Q(fc)

∂{T(fc)T(fc)t + I}∂ fi

Q(fc)

or equivalently

∂Q(fc)∂ fi

=−Q(fc){

∂T(fc)∂ fi

T(fc)t +T(fc)∂T(fc)t

∂ fi

}Q(fc).


The gradientG(fc) (derivative with respect tofc) of the functional (1.11) is given by

G(fc) = 2AtQ(fc)(Afc−g)+2µDtDfc +u(fc),

where

u(fc) =

(Afc−g)t ∂Q(fc)∂ f1

(Afc−g)

(Afc−g)t ∂Q(fc)∂ f2

(Afc−g)...

(Afc−g)t ∂Q(fc)∂ fn

(Afc−g)

.

The gradient descent scheme, yields

f(k+1)c = f(k)c − τkG(f(k)c ), k = 0,1, · · · .

A line search can be added to select the step sizeτk in a manner which gives suf-ficient decrease in the objective functional in (1.11) to guarantee convergence to aminimizer. This gives the method of steepest descent, see [14, 27, 35]. While nu-merical implementation is straightforward, steepest descent has rather undesirableasymptotic convergence properties which can make it very inefficient. Obviously,one can apply other standard unconstrained optimization methods with better conver-gence properties, like the nonlinear conjugate gradient method or Newton’s method.These methods converge rapidly near a minimizer provided the objective functionaldepends on smoothly onfc. Since the objective function in (1.11) is nonconvex, thisresults in a loss of robustness and efficiency for higher order methods like Newton’smethod. Moreover, implementing Newton’s method requires the inversion of ann-by-n unconstructed matrix, clearly an overwhelming task for any reasonable-sizedimage, for instance,n = 65,536for a256×256image. Thus, these approaches mayall be unsuitable for the image restoration problem.

Here we develop an alternative approach to minimizing (1.11). At a minimizer,we know thatG(fc) = 0, or equivalently,

2AtQ(fc)(Afc−g)+2µDtDfc−u(fc) = 0.

The iteration can be expressed as

[AtQ(f(k)c )A+ µDtD

]f(k+1)c =

u(f(k)c )2

+AtQ(f(k)c )g. (1.20)

Note that at each iteration, one must solve a linear system depending on the previous

iteratef(k)c , to obtain the new iteratef(k+1)c . We also find that

d(k) = f(k+1)c − f(k)c =−1

2


]−1G(f(k)c ).

Hence the iteration is of quasi-Newton form, and existing convergence theory can

be applied, see for instance [14, 27]. Since the matrixQ(f(k)c ) is symmetric positive


definite, and therefore(AtQ(f(k)c )A+ µDtD) is symmetric positive definite with itseigenvalues bounded away from zero (because of the regularization), each step com-putes the descent directiond(k), and global convergence can be guaranteed by usingthe appropriate step size, i.e.,

f(k+1)c = f(k)c − τk

2


]−1G(f(k)c ),

whereτk = argminτk>0P(f(k)c + τkd(k)).

With this proposed iterative scheme, one must solve a symmetric positive definitelinear system [

AtQ(f(k)c )A+ µDtD]

x = b (1.21)

for someb at each iteration. Of course these systems are dense in general, but havestructures that can be utilized. We apply a preconditioned conjugate gradient methodto solving these linear systems.

1.4.1 The Preconditioned Ccnjugate Gradient Method

In this subsection, we introduce the conjugate gradient method for solving linearsystems of equations. For a more in-depth treatment of other Krylov space methodssee [44]. The CG method was invented in the 1950s [21] (Hestenes and Steifel, 1952)as a direct method for solving Hermitian positive definite systems. It has come intowide use over the last 20 years as an iterative method.

Let us considerAx = b whereA∈ Cn×n is a nonsingular Hermitian positive def-inite matrix andb ∈ Cn. Given an initial guessx0 and the corresponding initialresidualr0 = b−Ax0, thekth iteratexk of CG minimizes the functional

φ(x)≡ 12

x∗Ax−x∗b

overx0 +Kk, whereKk is thekth Krylov subspace

Kk ≡ span(r0,Ar0, . . . ,Ak−1r0), k = 1,2, . . . .

Note that ifx minimizesφ(x), then5φ(x) = Ax−b = 0 and hencex is the solution.Denotext , the true solution of the system, and define the norm

‖x‖A ≡√

x∗Ax.

One can show that minimizingφ(x) over x0 +Kk is the same as minimizing‖x−xt‖A overx0 +Kk, i.e.

‖xt −xk‖A = miny∈x0+Kk

‖xt −y‖A.


Since anyy ∈ x0 +Kk can be written as

y = x0 +k−1

∑i=0

αiAir0

for some coefficients{αi}k−1i=0 , we can expressxt −y as

xt −y = xt −x0−k−1

∑i=0

αiAir0.

As r0 = b−Ax0 = A(xt −x0), we have

xt −y = xt −x0−k−1

∑i=0

αiAi+1(xt −x0) = p(A)(xt −x0),

where the polynomial

p(z) = 1−k−1

∑i=0

αizi+1

has degreek and satisfiesp(0) = 1. Hence

‖xt −xk‖A = minp∈Pk,p(0)=1

‖p(A)(xt −x0)‖A, (1.22)

wherePk is the set of polynomials of degreek.Hermitian positive definite matrices asserts thatA = UΛU∗, whereU is a unitary

matrix whose columns are the eigenvectors ofA andΛ is the diagonal matrix withthe positive eigenvalues ofA on the diagonal. SinceUU∗ = U∗U = I , we haveAk = UΛkU∗. Hencep(A) = U p(Λ)U∗. DefiningA

12 = UΛ

12U∗, we have

‖p(A)x‖A = ‖A12 p(A)x‖2 ≤ ‖p(A)‖2‖x‖A.

Together with (1.22), this implies that

‖xt −xk‖A ≤ ‖xt −x0‖A minp∈Pk,p(0)=1

maxλ∈σ(A)

|p(λ )|. (1.23)

Clearly, ifk= n, we can choosep to be thenth-degree polynomial that passes throughall the eigenvaluesλ ∈ σ(A) with p(0) = 1. Then the maximum in the right-handside of (1.23) is zero and we have the following theorem in Axelsson and Barker [2,p.24].

THEOREM 1.3Let A be a Hermitian positive definite matrix of size n. Then the CG algorithmfinds the solution of Ax = b within n iterations in the absence of roundofferrors.


In most applications, the number of unknownsn is very large. It is better to con-sider CG as an iterative method and terminate the iteration when some specified errortolerance is reached. The usual implementation of the CG method is to find, for agiven ε, a vectorx such that‖b−Ax‖2 ≤ ε‖b‖2. Algorithm CG is a typical im-plementation of the method. Its inputs are the right-hand sideb, a routine whichcomputes the action ofA on a vector, and the initial guessx0 which will be over-written by the subsequent iteratesxk. We limit the number of iterations tokmax andreturn the solutionxk and the residual normρk.

Algorithm CG(x,b,A,ε,kmax)

1. r = b−Ax, ρ0 = ‖r‖22, k = 1

2. Do while√ρk−1 > ε‖b‖2 and k < kmax

if k = 1 then p = relse

β = ρk−1/ρk−2 and p = r +βp

w = Ap

α = ρk−1/p∗w

x = x+αp

r = r −αw

ρk = ‖r‖22

k = k+1

3. End Do

Note that the matrixA itself need not be formed or stored — only a routine formatrix–vector productsAp is required.

Next we consider the cost. We need to store only four vectors:x,w,p, andr . Eachiteration requires a single matrix–vector product to computew = Ap, two scalarproducts (one forp∗w and one to computeρk = ‖r‖2

2), and three operations of theform αx+y, wherex andy are vectors andα is a scalar. Thus, besides the matrix–vector multiplication, each iteration of AlgorithmCG requiresO(n) operations∗,wheren is the size of the matrixA.

The convergence rate of the conjugate gradient method can be determined by thecondition numberκ(A) of the matrixA, where

κ(A)≡ ‖A‖2 · ‖A−1‖2 =λmax(A)λmin(A)

.

∗Let f andg be two functions defined on the set of integers. We say thatf (n) = O(g(n)) asn→∞ if thereexists a constantK such that| f (n)/g(n)| ≤ K asn→ ∞.


In fact, by choosingp in (1.23) to be akth degree Chebyshev polynomial , one canderive the following theorem in Axelsson and Barker [2, p.26].

THEOREM 1.4Let A be a Hermitian positive definite matrix with condition number κ(A).

Then the kth iterate xk of the conjugate gradient method satisfies

‖xt −xk‖A

‖xt −x0‖A≤ 2

(√κ(A)−1√κ(A)+1

)k

. (1.24)

In particular, for any given tolerance τ > 0, ‖xt −xk‖A/‖xt −x0‖A ≤ τ if

k≥ 12

√κ(A) log

(2τ

)+1 = O(

√κ(A)). (1.25)

This shows that the convergence rate is linear:

‖xt −xk‖A

‖xt −x0‖A≤ 2rk

wherer < 1. However, if we have more information about the spectrum ofA, thenwe can have a better bound of the error as the following theorem asserts (cf. alsoAxelsson (1994)).

When the condition number or the distribution of the eigenvalues of a matrix isnot good, one can improve the performance of the CG iteration by preconditioning.In effect, one tries to replace the given systemAx = b by another Hermitian positivedefinite system with the same solutionx, but with the new coefficient matrix havinga more favorable spectrum.

Suppose thatP is a Hermitian positive definite matrix such that either the conditionnumber ofP−1A is close to 1 or the eigenvalues ofP−1A are clustered around 1.Then, the CG method, when applied to the preconditioned system

P−1Ax = P−1b,

will converge very fast. We will callP the preconditioner of the systemAx =b or of the matrixA. The following algorithm is a typical implementation of thepreconditioned conjugate gradient (PCG) method. The input of this algorithm is thesame as that for AlgorithmCG, except that we now have an extra routine to computethe action of the inverse of the preconditioner on a vector.

Algorithm PCG(x,b,A,P,ε,kmax)

1. r = b−Ax, ρ0 = ‖r‖22, k = 1

2. Do while√ρk−1 > ε‖b‖2 and k < kmax

z = P−1r (or solve Pz = r )


τk−1 = z∗r

if k = 1 then β = 0 and p = zelse

β = τk−1/τk−2 and p = z+βp

w = Ap

α = τk−1/p∗w

x = x+αp

r = r −αw

ρk = ‖r‖22

k = k+1

3. End Do

Note that the cost of AlgorithmPCG is identical to that of AlgorithmCG with theaddition of the solution of the preconditioner systemPz = r and the inner product tocomputeτk−1 in Step 2. Thus the criteria for choosing a good preconditionerP are:

• The systemPz = r for any givenr should be solved efficiently.

• The spectrum ofP−1A should be clustered and/or its condition number shouldbe close to 1.

1.4.2 Cosine Transform Based Preconditioners

In this application, we need to compute a matrix-vector product

(AtQ(f(k)c )A+ µDtD)v

for some vectorv in each CG iteration. Since the matricesA andAt are Toeplitz-plus-Hankel matrices, their matrix-vector multiplications can be done inO(nlogn)operations for anyn-vector, see for instance [10]. However, for the matrix-vectorproduct

Q(f(k)c )v≡ {[T(f(k)c )|I ][T(f(k)c )|I ]t}−1v,

we need to solve another linear system{

[T(f(k)c )|I ][T(f(k)c )|I ]t}

z = v. (1.26)

Notice that the matrix-vector multiplicationsT(f(k)c )y andT(f(k)c )tv can also be com-putedO(nlogn) operations for anyn-vectory. A preconditioned conjugate gradientmethod will also be used for solving this symmetric positive definite linear system.

We remark that all matrices that can be diagonalized by the discrete cosine trans-form matrixC must be symmetric [36], soC above can only diagonalize matrices


with symmetric point spread functions for this problem. On the other hand, fornonsymmetric point spread functions, we can construct cosine transform based pre-conditioners to speed up the convergence of the conjugate gradient method.

Given a matrixX, we define the optimal cosine transform preconditionerc(X) tobe the minimizer of‖X−Q‖2

F over all Q that can diagonalized byC, see [36]. Inthis case, the cosine transform preconditionerc(A) of A in (1.12) is defined to be thematrixCtΛC such thatΛ minimizes

‖CtΛC−A‖2F .

HereΛ is any nonsingular diagonal matrix. Clearly, the cost of computingc(A)−1yfor anyn-vectory is O(nlogn) operations. In [36], Ng et al. gave a simple approachfor findingc(A). The cosine transform preconditionerc(A) is just the blurring matrix(cf. (1.12)) corresponding to the symmetric point spread functionsi ≡ (hi +h−i)/2with the Neumann boundary condition imposed. This approach allows us to precon-dition the symmetric positive definite linear system (1.21).

Next we construct the cosine transform preconditioner for{[T(f(k)c )|I ][T(f(k)c )|I ]t}which exploits the Toeplitz structure of the matrix. We approximateT(fc) by

T(fc) =1

2n−1

fn fn−1 · · · f2 f1 0 · · · · · · 0

0 fn. . .

. . . f2 f1. .. 0

......

. . .. . .

.. .. . .

. .... .

...

0... fn fn−1

. . .. .. f 1 0

0 · · · · · · 0 fn fn−1 · · · f2 f1

In [37], Ng and Plemmons have proved that iffc is a stationary stochastic process,then the expected value ofT(fc)T(fc)t−T(fc)T(fc)t is close to zero. SinceT(fc)T(fc)t

is a Toeplitz matrix,c(T(fc)T(fc)t) can be found inO(n) operations, see [10]. How-ever, the original matrixT(fc)T(fc)t is much more complicated and thus the con-struction cost ofc(T(fc)T(fc)t) is cheaper than that ofc(T(fc)T(fc)t). It is clear thatthe cost of computingc(T(fc))−1y for anyn-vectory is againO(nlogn) operations.It follows that the cost per each iteration in solving the linear systems (1.21) and(1.26) areO(nlogn) operations.

Finally, we remark that the objective function is simplified in the cosine transformdomain when the symmetry constraints are incorporated into the total least squaresformulation. In accordance with the Theorem 2, the minimization ofP(fc) in (1.18)is equivalent to

minf i

[(wi f i − gi)2

f2i +1

+ µΛii f2i

], 1≤ i ≤ n.

We note that the objective function is decoupled inton equations, each to be mini-mized independently with respect to one DCT coefficient offc. It follows that eachminimizer can be determined by an one-dimensional search method, see [35].


1.5 Two Dimensional Deconvolution Problems

The results of the previous section extend in a natural way to two dimensional imagedeconvolution. The main interest concerns optical image enhancement. Applica-tions of image deconvolution in optics can be found in many areas of science andengineering, e.g., see the book by Roggemann and Welsh [43]. For example, workto enhance the quality of optical images has important applications in astronomi-cal imaging [22]. Only partial priori knowledge about the degradation phenomenaor point spread function in aero-optics is generally known, so here the use of con-strained total least squares method is appropriate. In addition, the estimated pointspread function is generally degraded in a manner similar to that of the observedimage [43].

Let f (x,y) and g(x,y) be the functions of the original and the blurred imagesrespectively. The image restoration problem can be expressed as a linear integralequation

g(x,y) =∫ ∫

h(x−y,u−v) f (y,v)dydv. (1.27)

The convolution operation, as is often the case in optical imaging, acts uniformly(i.e., in a spatially invariant manner) onf . We consider numerical methods for ap-proximating the solution to the linear restoration problem in discretized (matrix)form obtained from (1.27). For notation purposes we assume that the image isn-by-n, and thus containsn2 pixels. Typically,n is chosen to be a power of 2, such as 256or larger. Then the number of unknowns grows to at least 65,536. The vectorsf andg represent the “true” and observed image pixel values, respectively, unstacked byrows. After discretization of (1.27), the blurring matrixH defined byh is given by

H =

H(n−1) · · · H(1) H(0) · · · H(−n+1) 0...

...... H(−n+1)

H(n−1) ......

. ..0 H(n−1) · · · H(0) H(−1) · · · H(−n+1)

(1.28)

with each subblockH( j) being an-by-(2n− 1) matrix of the form given by (1.7).The dimensions of the discrete point spread functionh are2n−1 and2n−1 in thex-direction and y-direction, respectively.

Applying Neumann boundary conditions, the resulting matrixA is a block-Toeplitz-


plus-Hankel matrix with Toeplitz-plus-Hankel blocks. More precisely,

A =

A(0) A(−1) · · · · · · A(−n+1)

A(1) A(0) . . .. ..

......

. . .. . .

. .....

.... . .

. . . A(0) A(−1)

A(n−1) · · · · · · A(1) A(0)

+

A(1) A(2) · · · A(n−1) 0A(2) ..

...

.A(−n+1)

... . ..

...

... ...

A(n−1) . ..

...

A(−2)

0 A(−n+1) · · · A(−2) A(−1)

(1.29)with each blockA( j) being ann-by-n matrix of the form given in (1.12). We note thattheA( j) in (1.29) and theH( j) in (1.28) are related by (1.12). A detailed discussion ofusing the Neumann boundary conditions for two dimensional problems can be foundin [36].

Using a similar argument, we can formulate the regularized constrained total leastsquares problems under the Neumann boundary conditions.

THEOREM 1.5Under the Neumann boundary condition (1.3), the regularized constrained

total least squares solution can be obtained as the fc that minimizes the func-tional:

P(fc) = (Afc−g)t([T|I ][T|I ]t)−1(Afc−g)+ µftcDtDfc,

where T is an n2-by-(2n−1)2 block-Toeplitz-Toeplitz-block matrix

T =1n

Tn Tn−1 · · · T2 T1 T1 · · · Tn−2 Tn−1

Tn Tn. . . . . . T2 T1

. . . Tn−3 Tn−2...

. . . . . . . . . . . . . . . . . . . . ....

T3 T4. . . Tn Tn−1

. . . . . . T1 T1

T2 T3 · · · Tn Tn Tn−1 · · · T2 T1

and each subblock Tj is a n-by-(2n−1) matrix of the form given by (1.7).

For a symmetric point spread function, we have the following theorem.

THEOREM 1.6Under the Neumann boundary condition (1.3) and the symmetry constraint

hi, j = hi,− j = h−i, j = h−i,− j ,

the regularized constrained total least squares solution can be obtained as thefc that minimizes the functional

P(fc) = [diag(w)fc− g]t{[diag(fc)|I ][diag(fc)|I ]t}−1[diag(w)fc− g]+ µ ftcΛfc,


FIGURE 1.5“Gatlinburg Conference” test image.

where f =Cf, g=Cg, and Λ is an n-by-n diagonal matrix given by Λ =CDtDCt

and D is the regularization operator.

1.6 Numerical Examples

In this section, we illustrate that the quality of restored image given by using the reg-ularized, constrained TLS method with the Neumann boundary conditions is gen-erally superior to that obtained by the least squares method. The data source is aphoto from the 1964 Gatlinburg Conference on Numerical Algebra taken from Mat-lab. From (1.2), we see that to construct the right hand side vectorg correctly, weneed the vectorsf l andfr , i.e., we need to know the image outside the given domain.Thus we start with the 480-by-640 image of the photo and cut out a 256-by-256portion from the image. Figure 1.5 gives the 256-by-256 image of this picture.

We consider restoring the “Gatlinburg Conference” image blurred by a truncated(band limited) Gaussian point spread function,

hi, j ={

ce−0.1(i2+ j2), if |i− j| ≤ 8,0, otherwise,

see [24, p.269]., wherehi, j is the jth entry of the first column ofA(i) in (1.29) andc is the normalization constant such that∑i, j hi, j = 1. We remark that the Gaussianpoint spread function is symmetric, and is often used in the literature to simulatethe blurring effects of imaging through the atmosphere [3, 43]. Gaussian noise withsignal-to-noise ratios of 40dB, 30dB and 20dB is then added to the blurred imagesand the point spread functions to produce test images. Noisy, blurred images are


Blurred image PSF Exact Noisy NoisyNoise added Noise added PSF PSF PSFSNR (dB) SNR (dB) RLS method RLS method RCTLS method

40 20 7.78×10−2 1.07×10−1 8.94×10−2

40 30 7.78×10−2 8.72×10−2 8.48×10−2

40 40 7.78×10−2 8.07×10−2 8.03×10−2

30 20 8.66×10−2 1.07×10−1 9.98×10−2

30 30 8.66×10−2 9.17×10−2 8.88×10−2

30 40 8.66×10−2 8.75×10−2 8.71×10−2

20 20 9.68×10−2 1.13×10−1 1.09×10−1

20 30 9.68×10−2 1.00×10−1 9.99×10−2

20 40 9.68×10−2 9.76×10−2 9.77×10−2

Table 1.1: The relative errors for different methods.

shown in Figures 1.6(a). We note that after the blurring, the cigarette held by Prof.Householder (the rightmost person) is not clearly shown.

In Table 1.1, we present results for the regularized constrained total least squaresmethod with PSF symmetry constraints. We denote this method by RCTLS in the ta-ble. As a comparison, the results in solving the regularized least squares (RLS) prob-lems with the exact and noisy point spread functions are also listed. For all methodstested, the Neumann boundary conditions are employed and the corresponding blur-ring matrices can be diagonalized by discrete cosine transform matrix. Therefore, theimage restoration can be done efficiently in the transform domain. We remark thatTikhonov regularization of the least squares method can be recast as in a total leastsquares framework, see [16]. In the tests, we used theL2 norm as the regularizationfunctional. The corresponding regularization parametersµ are chosen to minimizethe relative error of the restored image which is defined as

‖fc− fc(µ)‖2

‖fc‖2, (1.30)

wherefc is the original image. In the tests, the regularization parameters are obtainedby trial and error.

We see from Table 1.1 that the relative errors in using the RCTLS method areless than that of using RLS method with the noisy PSF, except the case where theSNR of noises added to the blurred image and PSF are 20dB and 40dB respectively.However, for some cases, the improvement of using RCTLS method is not significantwhen the SNR ratio is low, that is, the noise level to the blurred image is very high.In Figure 1.6, we present the restored images for different methods. We see fromFigure 1.6 that the cigarette is better restored by using the RCTLS method thanthat by using the RLS method. We remark that the corresponding relative error isalso significantly smaller than that obtained by using the RLS method. When noisewith low SNR is added to the blurred image and point spread function, visually, therestored images look similar. In Figure 1.6(e), we present the restored images for the


periodic boundary condition using the RCTLS method. We see from all the figuresthat by using the RCTLS method and imposing the Neumann boundary condition,the relative errors and the ringing effects in the restorations are significantly reduced.

1.7 Application: High-resolution Image Reconstruction

Processing methods for single images often provide unacceptable results because ofthe ill-conditioned nature of associated inverse problems and the lack of diversity inthe data. Therefore, image processing using a sequence of images, as well as im-ages captured simultaneously using a camera with multiple lenslets, has developedinto an active research area because multiple deconvolution operators can be usedto make the problem better posed. Rapid progress in computer and semiconductortechnology is making it possible to implement such image processing tasks reason-ably quickly, but the need for processing in real time requires attention to design ofefficient and robust algorithms for implementation on current and future generationsof computational architectures.

Image sequences may be produced from several snapshots of an object or a scene(LANDSAT images). Using sequential estimation theory in the wave number do-main, an efficient method was developed [8] for interpolating and recursively up-dating to provide filtering provided the displacements of the frames with respect toa reference frame were either known or estimated. It was observed that the perfor-mance deteriorated when the blur produced zeros in the wave number domain andtheoretical justification for this can be provided. The problem of reconstruction inthe wave number domain with errors present both in the observation and data wastackled by the total least squares method [48].

The spatial resolution of an image is often determined by imaging sensors. Ina CCD or a CMOS camera, the image resolution is determined by the size of itsphoto-detector. An ensemble of several shifted images could be collected by a pre-fabricated planar array of CCD or CMOS sensors and one may reconstruct withhigher resolution which is equivalent to an effective increase of the sampling rateby interpolation. Fabrication limitations are known to cause subpixel displacementerrors, which coupled with observation noise limit the deployment of least squarestechniques in this scenario. TLS is an effective technique for solving a set of sucherror contaminated equations [48] and therefore is an appropriate method for con-sideration in high-resolution image reconstruction applications. However, a possibledrawback of using the conventional TLS approach is that the formulation is not con-strained to handle point spread functions obtained from multisensors. In this section,an image processing technique that leads to the deployment of constrained total leastsquares (CTLS) theory is described. Then a computational procedure is advancedand the role of regularization is considered.


(a) (b)

(c) (d)

(e)

FIGURE 1.6(a) Noisy and blurred images with SNR of 40dB, the restored images by using(b) the exact PSF (rel. err. =7.78× 10−2), (c) the RLS method for the noisyPSF with SNR of 20dB (rel. err. =1.07×10−1), (d) the RCTLS method for anoisy PSF with SNR of 20dB under Neumann boundary conditions (rel. err.= 8.94×10−2) and (e) the RCTLS method for a noisy PSF with SNR of 20dBunder periodic boundary conditions (rel. err. = 1.14×10−1).


FIGURE 1.7A depiction of the basic components of an array imaging camera system [46].

lens partially silveredmirrors

relaylens

CCD sensorarray

#1

#2

#3

#4

Referenceframe

Verticallydisplaced

Horizontallydisplaced

Diagonallydisplaced

FIGURE 1.8Example of low-resolution image formation.


1.7.1 Mathematical Model

A brief introduction to a mathematical model in high-resolution image reconstructionis provided first, see Figure 1.9. Details can be found in [7]. Consider a sensor arraywith L1×L2 sensors in which each sensor hasN1×N2 sensing elements (pixels) andthe size of each sensing element isT1×T2. The goal is to reconstruct an image ofresolutionM1×M2, whereM1 = L1N1 andM2 = L2N2. To maintain the aspect ratioof the reconstructed image the case whereL1 = L2 = L is considered. For simplicity,L is assumed to be an even positive integer in the following discussion.

To generate enough information to resolve the high-resolution image, subpixel dis-placements between sensors are necessary. In the ideal case, the sensors are shiftedfrom each other by a value proportional toT1/L×T2/L. However, in practice therecan be small perturbations around these ideal subpixel locations due to imperfectionsof the mechanical imaging system during fabrication. Thus, forl1, l2 = 0,1, · · · ,L−1with (l1, l2) 6= (0,0), the horizontal and vertical displacementsdx

l1l2anddy

l1l2, respec-

tively, of the [l1, l2]-th sensor with respect to the[0,0]-th reference sensor are givenby

dxl1l2

=T1

L(l1 + εx

l1l2) and dy

l1l2=

T2

L(l2 + εy

l1l2),

whereεxl1l2

andεyl1l2

denote, respectively, the actual normalized horizontal and verti-cal displacement errors. The estimates,εx

l1l2andεy

l1l2, of these parameters,εx

l1l2and

εyl1l2

, can be obtained by manufacturers during camera calibration.It is reasonable to assume that

|εxl1l2|< 1

2and |εy

l1l2|< 1

2,

because if that is not the case, then the low-resolution images acquired from twodifferent sensors may have more than the desirable overlapping information for re-constructing satisfactorily the high-resolution image [7].

Let f (x1,x2) denote the original bandlimited high-resolution scene, as a functionof the continuous spatial variables,x1,x2. Then the observed low-resolution digi-tal imagegl1l2 acquired from the(l1, l2)-th sensor, characterized by a point-spreadfunction, is modeled by

gl1l2[n1,n2] =∫ T2(n2+ 1

2)+dyl1l2

T2(n2− 12 )+dy

l1l2

∫ T1(n1+ 12)+dx

l1l2

T1(n1− 12 )+dx

l1l2

f (x1,x2)dx1dx2, (1.31)

for n1 = 1, . . . ,N1 andn2 = 1, . . . ,N2. These low-resolution images are combined toyield theM1×M2 high-resolution imageg by assigning its pixel values according to

g[L(n1−1)+ l1,L(n2−1)+ l2] = gl1l2[n1,n2], (1.32)

for l1, l2 = 0,1, · · · ,(L−1), n1 = 1, . . . ,N1 andn2 = 1, . . . ,N2.The continuous image modelf (x1,x2) in (1.31) can be discretized by the rectangu-

lar rule and approximated by a discrete image model. Letg andf be, respectively, the


vectors formed from discretization ofg(x1,x2) and f (x1,x2) using a column order-ing. The Neumann boundary condition is applied on the images. This assumes thatthe scene immediately outside is a reflection of the original scene at the boundary,i.e.,

f (i, j) = f (k, l) where

k = 1− i, i < 1,k = 2M1 +1− i, i > M1,l = 1− j, j < 1,l = 2M2 +1− j, j > M2.

Under the Neumann boundary condition, the blurring matrices are banded matriceswith bandwidthL + 1, but the entries at the upper left part and the lower right partof the matrices are changed. The resulting matrices, denoted byHx

l1l2(εx

l1,l2) and

Hyl1l2

(εyl1,l2

), have a Toeplitz-plus-Hankel structure.

L/2 ones︷︸︸︷

Hxl1l2

(εxl1,l2

) =1L

1 · · · 1 12− εx

l1l20

......

.. .. . .

.. .

1...

.... . .

.. . 12− εx

l1l2

12 + εx

l1l2

. ... . .

. . ... . 1

......

. . ... .

...0 1

2 + εxl1l2

1 · · · 1

+

︸︷︷︸L/2 ones

L/2−1 ones︷︸︸︷

1L

1 · · · 1 12 + εx

l1l20

... ...

...

1 ... 1

2− εxl1l2

12 + εx

l1l2..

.1

...

... ...

0 12− εx

l1l21 · · · 1

. (1.33)

︸︷︷︸L/2−1 ones

The matrixHyl1l2

(εyl1,l2

) is defined similarly. The blurring matrix corresponding to the(l1, l2)-th sensor under the Neumann boundary condition is given by the Kroneckerproduct,

Hl1l2(εl1,l2) = Hxl1l2

(εxl1,l2

)⊗Hyl1l2

(εyl1,l2

),

where the2×1 vectorεl1,l2 is denoted by(εxl1,l2

εyl1,l2

)t . The blurring matrix for the


whole sensor array is made up of blurring matrices from each sensor:

HL(ε) =L−1

∑l1=0

L−1

∑l2=0

Dl1l2Hl1l2(εl1,l2), (1.34)

where the2L2×1 vectorε is defined as

ε = [εx00 εy

00 εx01 εy

01 · · · εxL−1L−2 εy

L−1L−2 εxL−1L−1 εy

L−1L−1]t .

HereDl1l2 are diagonal matrices with diagonal elements equal to 1 if the correspond-ing component ofg comes from the(l1, l2)-th sensor and zero otherwise, see [7] formore details.

1.7.2 Image Reconstruction Formulation

In this subsection, the displacement errors are not known exactly. The spatial invari-ance of the blurring function translates into the spatial invariance of the displacementerror in the blurring matrix. The “true” blur function is represented as follows. Foreachl1, l2 ∈ {0,1, · · · ,L−1},

hzl1l2

=1L

there areL+1 entries︷︸︸︷[12− εz

l1l21 · · · 1 · · · 1

12

+ εzl1l2

]t = hzl1l2

+δhzl1l2

, ∀z∈ {x,y}, (1.35)

where the(L+1)×1 component vectors are

hzl1l2

=1L

[12− εz

l1l21 · · · 1 · · · 1

12

+ εzl1l2

]t , z∈ {x,y}

and

δhzl1l2

=1L

[−δεzl1l2

0 · · · 0 · · · 0 δεzl1l2

]t , z∈ {x,y}.Herehz

l1l2is the estimated (or measured) point spread function andδhz

l1l2is the error

component of the point spread function at the(l1, l2)th sensor in thez-direction (z∈{x,y}).

The observed signal vectorg is also subject to errors. It is assumed that thisobservedM1M2×1 signal vectorg = [g1 . . . gM1M2]

t can be represented by

g = g+δg, (1.36)

whereδg = [δg1 δg2 · · · δgM1M2]

t

and{δgi} is independent and identically distributed Gaussian noise with zero meanand varianceσ2

g . Then the image reconstruction problem requires the recovery of thevectorf from the given inexact point spread function functionhz

l1l2(l1 = 0,1, · · · ,L1−

1, l2 = 0,1, · · · ,L2−1, z∈ {x,y}) and the observed noisy signal vectorg.


The constrained TLS formulation for multisensors is considered. Using

HL(ε)f =L−1

∑l1=0

L−1

∑l2=0

Dl1l2Hl1l2(εl1,l2)f = g = g+δg

(from (1.34)) and (1.35) and (1.36), the preceding equation can be reformulated asfollows:

[L−1

∑l1=0

L−1

∑l2=0

Dl1l2

(Hx

l1l2(εl1,l2)⊗Hy

l1l2(εl1,l2)

)]f−g+

[L−1

∑l1=0

L−1

∑l2=0

δεxl1l2

(Dl1l2E⊗Hy

l1l2(εl1,l2)

)]f

[L−1

∑l1=0

L−1

∑l2=0

δεyl1l2

(Dl1l2Hx

l1l2(εl1,l2)⊗E

)]

f+

[L−1

∑l1=0

L−1

∑l2=0

δεxl1l2

δεyl1l2

(Dl1l2E⊗E

)]

f−δg = 0

or[

L−1

∑l1=0

L−1

∑l2=0

Dl1l2

(Hx

l1l2(εl1,l2)⊗Hy

l1l2(εl1,l2)

)]f−g+

L−1

∑l1=0

L−1

∑l2=0

δεxl1l2

fyl1l2

L−1

∑l1=0

L−1

∑l2=0

δεyl1l2

fxl1l2

+

[L−1

∑l1=0

L−1

∑l2=0

δεxl1l2

δεyl1l2

(Dl1l2E⊗E

)]

f−δg = 0, (1.37)

where

L/2 zeros︷︸︸︷ L/2−1 zeros︷︸︸︷

E =1L

0 · · · 0 −1 0...

......

. ... . .

0...

.... ..

. . . −1

1...

......

. . . 0...

.... ..

. . ....

0 1 0 · · · 0

+1L

0 · · · 0 1 0... ..

...

...

.

0 ...

... −1

1 ...

...

0

...

...

... ...

0 −1 0 · · · 0

︸︷︷︸L/2 zeros

︸︷︷︸L/2−1 zeros

and

fxl1l2

= Dl1l2

(Hx

l1l2(εl1,l2)⊗E

)f, fy

l1l2= Dl1l2

(E⊗Hy

l1l2(εl1,l2)

)f.


The constrained TLS formulation amounts to minimizing the norms of vectors asso-ciated with

{δεxl1l2

,δεyl1l2}L−1

l1,l2=0 and δg

as explained below such that (1.37) is satisfied.However, it is first noted that becauseδεz

l1l2(l1, l2 = 0,1, · · · ,L−1) should be very

small, the quantity|δεxl1l2

δεyl1l2| can be assumed to be negligible, and, therefore, the

nonlinear term [L−1

∑l1=0

L−1

∑l2=0

δεxl1l2

δεyl1l2

(Dl1l2E⊗E

)]

f (1.38)

can be ignored in (1.37). In this case, (1.37) reduces to a linear system involving{δεx

l1l2,δεy

l1l2}L−1

l1,l2=0 andδg.

For simplicity, denote the2L2×1 vector

4= [δεx00 δεy

00 δεx01 δεy

01 · · · δεxL−1L−2 δεy

L−1L−2 δεxL−1L−1 δεy

L−1L−1]t .

Mathematically, the constrained TLS formulation can be expressed as

minf

∥∥∥∥[ 4

δg

]∥∥∥∥2

2

subject to

HL(ε)f−g+L−1

∑l1=0

L−1

∑l2=0

δεxl1l2· fy

l1l2+

L−1

∑l1=0

L−1

∑l2=0

δεyl1l2· fx

l1l2−δg = 0 (1.39)

Image reconstruction problems are in general ill-conditioned inverse problems andreconstruction algorithms can be extremely sensitive to noise [18]. Regularizationcan be used to achieve stability. Using classical Tikhonov regularization [15], stabil-ity is attained by introducing a regularization operatorP and a regularization para-meterµ to restrict the set of admissible solutions. More specifically, the regularizedsolutionf is computed as the solution to

minf

∥∥∥∥[ 4

δg

]∥∥∥∥2

2+ µ‖Pf‖2

2 (1.40)

subject to (1.39). The termµ‖Pf‖22 is added in order to regularize the solution. The

regularization parameterµ controls the degree of regularity (i.e., degree of bias) ofthe solution. In many applications [18, 24],‖Pf‖2 is chosen to be‖f‖2 or ‖Rf‖2

whereR is a first order difference operator matrix.The theorem below characterizes the regularized, constrained total least squares

(RCTLS) formulation of the high-resolution reconstruction problem with multisen-sors.


THEOREM 1.7The regularized constrained total least squares solution can be obtained as the

f that minimizes the functional:

J(f) = (HL(ε)f−g)t{[Q(f)|−IM1M2][Q(f)|−IM1M2]t}−1(HL(ε)f−g)+ µftPtPf,

(1.41)Q(f) = [ fy

00 | fx00 | fy

01 | fx01 | · · · | fy

L−1L−2 | fxL−1L−2 | fy

L−1L−1 | fxL−1L−1 ].

and IM1M2 is the M1M2-by-M1M2 identity matrix.

The proof of the above theorem is similar to Theorems 1.1 and 1.5. The result-ing objective function (1.41) to be minimized is nonconvex and nonlinear. In thenext section, an iterative algorithm that takes advantage, computationally, of the fastsolvers for image reconstruction problems with known displacement errors [7, 41] isdeveloped.

By (1.39) and (1.40),

minf,4

J(f,4)≡minf,4

{‖4‖22+

∥∥∥∥∥HL(ε)f−g+L−1

∑l1=0

L−1

∑l2=0

δεxl1l2· fy

l1l2+

L−1

∑l1=0

L−1

∑l2=0

δεyl1l2· fx

l1l2

∥∥∥∥∥2

2

+ µ‖Pf‖22

(1.42)

It is noted that the above objective function is equivalent to (1.41).Before solving forf in the RCTLS formulation, it is first noted that for a given

f, the functionJ(f, ·) is convex with respect to4, and for a given4, the functionJ(·,4) is also convex with respect tof. Therefore, with an initial guess40, one canminimize (1.42) by first solving

J(f1,40)≡minf

J(·,40)

and thenJ(f1,41)≡min

4J(f1, ·).

Therefore, an alternating minimization algorithm is developed in which the functionvalueJ(fn,4n) always decreases asn increases. More precisely, the algorithm isstated as follows:

Assume that4n−1 is available:

• Determinefn by solving the following least squares problem

minfn

∥∥∥∥∥HL(ε)fn−g+L−1

∑l1=0

L−1

∑l2=0

δεxn−1 l1l2

fyn l1l2

+L−1

∑l1=0

L−1

∑l2=0

δεyn−1 l1l2

fxn l1l2

∥∥∥∥∥2

2

+

µ ‖Pfn‖2} (1.43)


Here,

fxn l1l2

= Dl1l2

(Hx

l1l2(εl1,l2)⊗E

)fn and fy

n l1l2= Dl1l2

(E⊗Hy

l1l2(εl1,l2)

)fn.

(1.44)As we have noted that the nonlinear term (1.38) can be assumed to be negligi-ble, we can add this term in the objective function (1.43). Therefore, the leastsquares solutionfn can be found by solving the following linear system:

[HL(ε +4n−1)tHL(ε +4n−1)+ µPtP

]fn = HL(ε +4n−1)tg. (1.45)

• Determine4n by solving the following least squares problem

min4n

‖4n‖2

2 +

∥∥∥∥∥HL(ε)fn−g+L−1

∑l1=0

L−1

∑l2=0

δεxl1l2

fyn l1l2

+L−1

∑l1=0

L−1

∑l2=0

δεyl1l2

fxn l1l2

∥∥∥∥∥2

2

.

(1.46)

Using (1.44) and (1.41), the above equation can be rewritten as

min4n

{‖4n‖2

2 +‖HL(ε)fn−g+Q(fn)4n‖22

}, (1.47)

whereQn(f) is defined as in Theorem 1.7. The cost function in (1.47) becomes

J( fn,4n) = 4tn4n +(HL(ε)fn−g+Q(fn)4n)t(HL(ε)fn−g+Q(fn)4n)

= 4tn4n +(HL(ε)fn−g)t(HL(ε)fn−g)+(HL(ε)fn−g)tQ(fn)4n +

(Q(fn)4n)t(HL(ε)fn−g)+(Q(fn)4n)t(Q(fn)4n).

Herefn is fixed. The gradient of this cost function with respect to4n is equalto

24n +2Q(fn)t(HL(ε)fn−g)+2Q(fn)tQ(fn)4n,

and therefore the minimum 2-norm least squares solution4n is given by

4n = [I2L2 +Q(fn)tQ(fn)]−1Q(fn)t(g−HL(ε)fn). (1.48)

The costly stage in the algorithm is the inversion of the matrix

HL(ε +4n−1)tHL(ε +4n−1)+ µPtP.

The preconditioned conjugate gradient method can be applied to solve the linearsystem

[c(HL(ε +4n−1))tc(HL(ε +4n−1))+ µc(PtP)]−1

[HL(ε +4n−1)tHL(ε +4n−1)+ µPtP] fn

= [c(HL(ε +4n−1))tc(HL(ε +4n−1))+ µc(PtP)]−1HL(ε +4n−1)tg. (1.49)


It is noted that when the regularization matrixP is equal toR, the first order differ-ence operator matrix,c(RtR) = RtR and the matrixRtR can be diagonalized by thediscrete cosine transform matrix. In [41], it has been shown that the singular valuesof the preconditioned matrices in (1.49) are clustered around 1 for sufficiently smallsubpixel displacement errors. Hence when the conjugate gradient method is appliedto solve the preconditioned system (1.49), fast convergence is expected. Numeri-cal results in [41] have shown that the cosine transform preconditioners can indeedspeed up the convergence of the method.

For each PCG iteration, one needs to compute a matrix-vector product

[HL(ε +4n)tHL(ε +4n−1)+ µRtR

]v

for some vectorv. SinceHL(ε +4n) has only(L + 1)2 non-zero diagonals andRtR has at most five nonzero entries in each row, the computational complexity ofthis matrix-vector product isO(L2M1M2). Thus the cost per each PCG iterationis O(M1M2 logM1M2 + L2M1M2) operations. Hence the total cost for finding thesolution in (1.45) isO(M1M2 logM1M2 +L2M1M2) operations.

Besides solving the linear system in (1.45), one also needs to solve the leastsquares problem (1.46) or determine the solution from (1.48). From (1.48), it isnoted that one needs to compute

Q(fn)= [ fyn 00 | fx

n 00 | fyn 01 | fx

n 01 | · · · | fyn L−1L−2 | fx

n L−1L−2 | fyn L−1L−1 | fx

n L−1L−1 ]

and then solve a2L2-by-2L2 linear system. Since the matrix-vector productHL(ε +4n)v for any vectorv can be computed inO(L2M1M2) operations, the cost of com-putingQ(fn) is O(L4M1M2). The cost of solving the corresponding2L2-by-2L2 lin-ear system isO(L6) by using Gaussian elimination. Hence the total cost for findingthe solution in (1.46) isO(L6 +L2M1M2) operations.

Finally, it is remarked thatL (for instance, the size of the sensor array is equalto 2 or 4) is usually significantly less thanM1 or M2 (for instance, the size of theimage is larger than 256), Hence the total cost for each iteration in the alternatingminimization algorithm is approximatelyO(M1M2 logM1M2+L2M1M2) operations.

1.7.3 Simulation Results

In the computer simulation, a256×256image (Figure 1.10(left)) is taken to be theoriginal high-resolution image. A(2×2) sensor array with sub-pixel displacementerrors retrieves four128× 128 blurred and undersampled images, which are cor-rupted by white Gaussian noise with a SNR of 30 dB. The image interpolated fromthese low-resolution images is shown in (Figure 1.10(right)). The parametersεx

l1l2andεy

l1l2are random values chosen between1

2 and−12.

In the first simulation, the estimated sub-pixel displacement errors,εxl1l2

andεyl1l2

are set to85 percent of the real sub-pixel displacement errors. In the proposedRCTLS algorithm, the choice of “proper” regularization parameter,µ is very im-portant. Initially, the proposed RCTLS algorithm was implemented with various


lens partially silveredmirrors

relaylens

CCD sensorarray

#1

#2

#3

#4

Referenceframe

Verticallydisplaced

Horizontallydisplaced

Diagonallydisplaced

FIGURE 1.9Example of low-resolution image formation.


FIGURE 1.10Original image of size256× 256 (left) and observed blurred and noisy image256×256; PSNR=24.20dB (right).

FIGURE 1.11Reconstructed image by RLS; PSNR = 25.64 dB. (left) and Reconstructed imageby RCTLS; PSNR = 25.88 dB (right).


0 5 10 15 20 25 30 35 400.034

0.036

0.038

0.04

0.042

0.044

0.046

0.048

0.05

iteration

norm

of e

rror

0 10 20 30 40 50 60 70 80 90 1000.06

0.08

0.1

0.12

0.14

0.16

0.18

iteration

norm

of e

rror

FIGURE 1.12Norm of error with µ = 10−4 (left) and norm of error with µ = 1 (right).

0 5 10 15 20 25 30 35 400.015

0.02

0.025

0.03

0.035

0.04

0.045

0.05

iteration

norm

of e

rror

500 1000 1500 2000 2500 3000 35000

0.5

1

1.5

2

2.5

3x 10

4

||Hf−g||

||Lf||

FIGURE 1.13Norm of error with µ = µopt (left) and L-curve (right).


values ofµ . Define

the norm of error at thenth iteration = ‖ε− (ε +4n)‖2

and plot it for different values ofµ (Figures 1.12 and 1.13). The speed of conver-gence decreases asµ increases. In the case whenµ = 10−4 andµ = 1, the norm oferror after convergence is greater than that in the first iteration (Figure 1.12). The“inappropriate” value ofµ makes the RCTLS solution fall into a local minimum.The L-curve method proposed in [20] may be used to get the optimum value of theregularization parameter. The L-curve method to estimate the “proper”µ for RCTLSwas used here. The L-curve plot is shown in Figure 1.13(right). Withµopt retrievedby L-curve, the RCTLS converges to a better minimum point (the norm of error issignificantly smaller than those obtained by choosingµ = 10−4 andµ = 1). Table1.2 shows the estimated subpixel displacement errors by RCTLS with different val-ues ofµ. It is seen that the updated subpixel displacement errors withµopt are closerto the real subpixel displacement errors than the updated subpixel displacement er-rors with arbitrarily chosen values for regularization parametersµ. It implies betteruse of the more precise blurring matrix for image reconstruction when the “proper”regularization parameter is chosen.

To show the advantage of the proposed RCTLS algorithm over the conventionalRLS (regularized least squares) algorithm, the two methods are compared. The re-constructed image by RLS and the reconstructed image by RCTLS are shown in Fig-ure 1.11(right), respectively. The reconstructed high-resolution image using RCTLSshows improvement both in image quality and PSNR. It is remarked that the optimalregularization parameter for RLS is also determined by L-curve for use in the RLSalgorithm.

1.8 Concluding Remarks and Current Work

In summary, we have presented a new approach image restoration by using regular-ized, constrained total least squares image methods, with Neumann boundary condi-tions. Numerical results indicate the effectiveness of the method. An application forthe approach proposed here is regularized constrained total least squares(RCTLS)reconstruction from a image sequence captured by multisensors with subpixel dis-placement errors that produces a high-resolution output, whose quality is enhancedby a proper choice of the regularization parameter. The novelty of this applicationlies in the treatment of the nonsymmetric PSF estimation problem arising from themultisensor image acquisition system.

Finally, we mention some ongoing work on the development of high-resolution al-gorithms and software for array imaging systems. Compact, multi-lens cameras canprovide a number of significant advantages over standard single-lens camera sys-tems, but the presence of more than one lens poses a number of difficult problems.


Errors (x-direction) `1 = 0, `2 = 0 `1 = 1, `2 = 0 `1 = 0, `2 = 1 `1 = 1, `2 = 1εx

l1l20.2 -0.2 0.1 0.1

εxl1l2

-0.17 -0.17 0.085 0.085εl1l2 +δεx

l1l20.1871 -0.1806 0.0915 0.0856

with µ = 10−4

εl1l2 +δεxl1l2

0.2343 -0.2218 0.1290 0.1262with µ = 1εl1l2 +δεx

l1l20.1987 -0.1946 0.0967 0.0957

with µopt = 0.005092

Errors (x-direction) `1 = 0, `2 = 0 `1 = 1, `2 = 0 `1 = 0, `2 = 1 `1 = 1, `2 = 1εy

l1l20.1 0 -0.3 -0.2

εyl1l2

0.085 0 -0.255 -0.17εl1l2 +δεy

l1l20.0950 -0.0067 -0.2720 -0.1863

with µ = 10−4

εl1l2 +δεyl1l2

0.1034 -0.0235 -0.3373 -0.2361with µ = 1εl1l2 +δεy

l1l20.1038 -0.0011 -0.2857 -0.1952

with µopt = 0.005092

Table 1.2: Estimation errors.

Modern work on the design and testing of array imaging systems began with seminalpapers on the TOMBO (Thin Observational Module for Bound Optics) system de-scribed by Tanida et al., in [46, 47]. High-resolution reconstruction methods basedon interpolation and projection of pixel values were developed by the authors.

In [13], Chan, Lam, Ng, and Mak describe a new super-resolution algorithm andapply it to the reconstruction of a high-resolution image from low-resolution imagesfrom a simulated compound eye (array) imaging system. They explore several vari-ations of the imaging system, such as the incorporation of phase masks to extendthe depth of focus. Software simulations with a virtual compound-eye camera areused to verify the feasibility of the proposed architecture and its variations. Theyalso report on the tolerance of the virtural camera system to variations of physicalparameters, such as optical aberrations.

In [5], the team consisting of Barnard, Chung, van der Gracht, Nagy, Pauca,Plemmons, Prasad, Torgersen, Mathews, Mirotznik and Behrmann investigate thefabrication and use of novel multi-lens imaging systems for iris recognition as partof the PERIODIC (Practical Enhanced-Resolution Integrated Optical-Digital Imag-ing Camera) project. The authors of [?] use established, conventional techniquesto compute registration parameters and leverage appropriate reconstruction of high-resolution images from low-resolution imagery. To verify that the generated high-resolution images have comparable fidelity to those captured by single-lens camerasystems they apply the reconstruction methods to iris recognition biometric problems


FIGURE 1.14Lenslet array imaging for biometric identification.

for personnel identification and verification. Figure 1.14 illustrates the collection ofan eye image for this biometric application with an array imaging camera.

In [5] the authors also report on image registration and reconstruction results usingboth simulated and real laboratory multi-lens image data from a prototype array cam-era fabricated by the PERIODIC research team. Current work on super-resolutionconstruction methods includes not only the use of subpixel displacement methods,but also individual lenslet image diversities such as dynamic range, wavelength andpolarization. Array cameras based on novel diffractive lenslets are also being inves-tigated.

Acknowledgments

Research by the first author was supported in part by Hong Kong Research GrantsCouncil Numbers 7046/03P, 7035/04P, 7035/05P and Hong Kong Baptist UniversityFRGs. Research by the second author was supported in part by the Air Force Officeof Scientific Research under grant F49620-02-1-0107, by the Army Research Officeunder grants DAAD19-00-1-0540 and W911NF-05-1-0402, and by ARDA underContract No. 2364-AR03-A1 through the ORNL.


References

[1] H. Andrews and B. Hunt,Digital Image Restoration, Prentice-Hall, Engle-wood Cliffs, NJ, 1977.

[2] O. Axelsson and V. Barker,Finite Element Solution of Boundary Value Prob-lems, Theory and Computation, Academic Press, Orlando, FL, 1984.

[3] M. Banham and A. Katsaggelos,Digital Image Restoration, IEEE SignalProcessing Magazine, March 1997, pp. 24–41.

[4] J. Bardsley, S. Jefferies, J. Nagy and R. Plemmons,A Computational Methodfor the Restoration of Images with an Unknown, Spatially-varying Blur, OpticsExpress, 14, no. 5 (2006), pp. 1767–1782.

[5] R. Barnard, J. Chung, J. van der Gracht, J. Nagy, P. Pauca, R. Plemmons,S. Prasad, T. Torgersen S. Mathews, M. Mirotznik and G. Behrmann,High-Resolution Iris Image Reconstruction From Low-Resolution Imagery, Proceed-ings of SPIE Annual Conference, Symposium on Advanced Signal ProcessingAlgorithms, Architectures, and Implementations XVI, Vol. 6313, to appear(2006).

[6] T. Bell, Electronics and the Stars, IEEE Spectrum 32, no. 8 (1995), pp. 16–24.

[7] N. K. Bose and K. J. Boo,High-resolution Image Reconstruction with Multi-sensors, International Journal of Imaging Systems and Technology, 9 (1998),pp. 294–304.

[8] N. K. Bose, H. C. Kim and H. M. Valenzuela,Recursive Total Least SquaresAlgorithm for Image Reconstruction from Noisy, Undersampled Frames, Mul-tidimensional Systems and Signal Processing, 4 (1993), pp. 253–268.

[9] K. Castleman,Digital Image Processing, Prentice–Hall, NJ, 1996.

[10] R. Chan and M. Ng,Conjugate Gradient Methods for Toeplitz Systems, SIAMReview, 38 (1996), pp. 427–482.

[11] R. Chan, M. Ng and R. Plemmons,Generalization of Strang’s Preconditionerwith Applications to Toeplitz Least Squares Problems, J. Numer. Linear Alge-bra Appls., 3 (1996), pp. 45–64.

[12] T. Chan and C. Wong,Total Variation Blind Deconvolution, IEEE Transactionson Image Processing, 7, no. 3 (1998), pp. 370–375.

[13] W. Chan, E. Lam, M. Ng, G. Mak,Super-resolution reconstruction in a Com-putational Compound-eye Imaging System, Multidimensional Systems andSignal Processing (2006), to appear.

[14] J. Dennis and R. Schnabel,Numerical Methods for Unconstrained Optimal-ization and Nonlinear Equation, Prentice-Hall, 1983.


[15] H. Engl, M. Hanke and A. Neubauer,Regularization of Inverse Problems,Kluwer Academic Publishers, The Netherlands, 1996.

[16] G. Golub, P. Hansen and D. O’Leary,Tikhonov Regularization and Total LeastSquares, SCCM Research Report, Stanford, 1998.

[17] G. Golub and C. Van Loan,An Analysis of Total Least Squares Problems,SIAM J. Numer. Anal., 17 (1980), pp. 883–893.

[18] R. Gonzalez and R. Woods,Digital Image Processing, Addison Wesley, NewYork, 1992.

[19] M. Hanke and J. Nagy,Restoration of Atmospherically Blurred Images by Sym-metric Indefinite Conjugate Gradient Techniques, Inverse Problems, 12 (1996),pp. 157–173.

[20] P. Hansen and D. O’Leary,The Use of the L-curve in the Regularzation ofDiscrete Ill-posed Problems, SIAM J. Sci. Comput., 14 (1993), pp. 1487–1503.

[21] M. Hestenes and E. Steifel,Methods of Conjugate Gradient for Solving LinearSystems, J. of Res. Nat. Bureau Standards, 49 (1952), pp.409–436.

[22] J. Hardy,Adaptive Optics for Astronomical Telescopes, Oxford Press, NewYork, 1998.

[23] S. Van Huffel and J. Vandewalle,The Total Least Squares Problem, SIAMPress, 1991.

[24] A. Jain, Fundamentals of Digital Image Processing, Prentice-Hall, EnglewoodCliffs, NJ, 1989.

[25] S. Jefferies and J. Christou,Restoration of Astronomical Images by IterativeBlind Deconvolution, Astrophysics J., 415 (1993), pp. 862–874.

[26] J. Kamm and J. Nagy,A Total Least Squares Method for Toeplitz Systems ofEquations, BIT, 38 (1998), pp. 560–582.

[27] T. Kelley, Iterative Methods for Optimization, SIAM Press, 1999.

[28] D. Kundur and D. Hatzinakos,Blind Image Deconvolution, IEEE SignalProcessing Magazine, May, 1996, pp. 43–64.

[29] R. Lagendijk and J. Biemond,Iterative Identification and Restoration of Im-ages, Kluwer Academic Publishers, 1991.

[30] R. Lagendijk, A. Tekalp and J. Biemond,Maximum Likehood Image and BlurIdentification: a Unifying Approach, Optical Engineering, 29 (1990), pp. 422–435.

[31] J. Lim, Two-dimensional Signal and Image Processing, Englewood Cliffs,N.J., Prentice Hall, 1990.


[32] F. Luk and D. Vandevoorde,Reducing boundary distortion in image restora-tion, Proc. SPIE 2296, Advanced Signal Processing Algorithms, Architecturesand Implementations VI, 1994.

[33] J. Nelson,Reinventing the Telescope, Popular Science, 85 (1995), pp. 57–59.

[34] V. Mesarovic, N. Galatsanos and A. Katsaggelos,Regularized Constrained To-tal Least Squares Image Restoration, IEEE Transactions on Image Processing,4 (1995), pp. 1096–1108.

[35] S. Nash and A. Sofer,Linear and Nonlinear Programming, The McGraw-HillCompanies, Inc., New York, 1996.

[36] M. Ng, R. Chan and T. Wang,A Fast Algorithm for Deblurring Models withNeumann Boundary Conditions, Res. Rept. 99-04, Dept. Math., The ChineseUniversity of Hong Kong, to appear in SIAM J. Sci. Comput. (1999).

[37] M. Ng and R. Plemmons,Fast Recursive Least Squares Using FFT-based Con-jugate Gradient Iterations, SIAM J. Sci. Comput., 17 (1996), pp. 920–941.

[38] M. Ng, R. Plemmons and F. Pimentel,A New Approach to Constrained TotalLeast Squares Image Restoration, Lin. Alg. and Applic., 316 (2000), pp. 237–258.

[39] M. K. Ng, R. J. Plemmons and S. Qiao.Regularized blind deconvolution usingrecursive inverse filtering. In G. H. Golub, S. H. Lui, F. T. Luk, and R. J.Plemmons, editors, Proceedings of the Hong Kong Workshop on ScientificComputing, (1997), pp. 110–132.

[40] M. Ng, R. Plemmons and S. Qiao,Regularization of RIF Blind Image De-convolution, IEEE Transactions on Image Processing, 9, no. 6, (2000), pp.1130–1134.

[41] M. Ng and A. Yip,A Fast MAP Algorithm for High-Resolution Image Recon-struction with Multisensors, Multidimensional Systems and Signal Processing,12, 2 (2001), pp. 143–164.

[42] K. Rao and P. Yip,Discrete Cosine Transform: Algorithms, Advantages, Ap-plications, Academic Press, Boston, 1990.

[43] M. Roggemann and B. Welsh,Imaging Through Turbulence, CRC Press, BocaRaton, FL, 1996.

[44] Y. Saad,Iterative methods for sparse linear systems, PWS, Boston, 1996.

[45] G. Strang,The Discrete Cosine Transform, SIAM Review, 41 (1999), pp. 135–147.

[46] J. Tanida, T. Kumagai, K. Yamada, S. Miyatake, K. Ishida, T. Morimoto, N.Kondou, D. Miyazaki and Y. Ichioka,Thin Observation Module by BoundOptics (TOMBO): Concept and Experimental Verification, Appl. Optics, 40(2001), pp. 1806–1813.


[47] J. Tanida, R. Shogenji, Y. Kitamura, K. Yamada, M. Miyamoto and S. Miy-atake,Color Imaging with an Integrated Compound Imaging System, OpticsExpress, 11 (2003), pp. 2109–2117.

[48] S. Van Huffel and J. Vandewalle,The Total Least Squares Problem: Computa-tional Aspects and Analysis, SIAM Publications, Philadelphia, 1991.

[49] C. Vogel, T. Chan and R. Plemmons,Fast Algorithms for Phase Diversity-based Blind Deconvolution, in Adaptive Optical System Technologies, Pro-ceedings of SPIE, Edited by D. Bonaccini and R. Tyson, 3353 (1998) pp. 994–105.

[50] Y. You and M. Kaveh,A Regularization Approach to Joint Blur Identificationand Image Restoration, IEEE Trans. on Image Proc., 5 (1996), pp. 416–427.

[51] W. Zhu, Y. Wang, Y. Yao, J. Chang, H. Graber and R. Barbour,Iterative TotalLeast Squares Image Reconstruction Algorithm for Optical Tomography by theConjugate Gradient Algorithm, J. Opt. Soc. Amer. A, 14 (1997), pp. 799–807.

[52] W. Zhu, Y. Wang, and J. Zhang,Total Least Squares Reconstruction withWavelets for Optical Tomography, J. Opt. Soc. Amer. A, 15 (1997), pp. 2639–2650.

blind deconvolution and structured matrix computations

Documents