Image restoration: linear solutions

— Inverse filtering and Wiener filtering —

Jean-Francois Giovannelli

Groupe Signal – ImageLaboratoire de l’Integration du Materiau au Systeme

Univ. Bordeaux – CNRS – BINP

1 / 39

Topics

Image restoration, deconvolutionMotivating examples: medical, astrophysical, industrial, vision,. . .Various problems: deconvolution, Fourier synthesis, denoising. . .Missing information: ill-posed character and regularisation

Three types of regularised inversion

1 Quadratic penalties and linear solutionsClosed-form expressionComputation through FFTOptimisation (e.g., gradient), system solvers (e.g., splitting)

2 Non-quadratic penalties and edge preservationHalf-quadratic approaches, including computation through FFTOptimisation (e.g., gradient), system solvers (e.g., splitting)

3 Constraints: positivity and supportAugmented Lagrangian and ADMM, including computation by FFTOptimisation (e.g., gradient), system solvers (e.g., splitting)

Bayesian strategy: a few incursionsTuning hyperparameters, instrument parameters,. . .Hidden / latent parameters, segmentation, detection,. . .

2 / 39

Inversion: standard question

y = H(x) + ε = Hx+ ε = h ? x+ ε

xH +

y

ε

x = X (y)

Restoration, deconvolution-denoising

General problem: ill-posed inverse problems, i.e., lack of information

Methodology: regularisation, i.e., information compensation

Specificity of the inversion / reconstruction / restoration methodsTrade off and tuning parameters

Limited quality results

3 / 39

Example due to Hunt (“square” response) [1970]

Convolutive model: y = h ? x+ ε

Samples averaging

−50 0 50 100 150 200

0

0.5

1

1.5

2

True x

−20 0 20

0

0.1

Response h

−50 0 50 100 150 200

0

0.5

1

1.5

2

Observation y4 / 39

Example: brain (“square” response)

Convolutive model: y = h ? x+ ε

Pixels averaging

Think also about the Fourier domain

True x Spatial response h Observation y

5 / 39

Example: brain (“motion blur”)

Convolutive model: y = h ? x+ ε

Pixels averaging

Think also about the Fourier domain

True x Spatial response h Observation y

6 / 39

Convolution

Examples of response

Square Motion Gaussian Diffraction Point

Convolutive model (1D, 2D and also 3D,. . . )

z(n) =

+P∑p=−P

h(p)x(n− p)

z(n,m) =

+P∑p=−P

+Q∑q=−Q

h(p, q)x(n− p,m− q)

Response: h(p) or h(p, q)

impulse response, convolution kernel,. . .. . . point spread function, stain image

7 / 39

Matrix form: 1D convolution

Linear matricial relation: z = Hx

Shift invariance Tœplitz structure

Short response band structure

.

.

.

zn−1 =

zn = hP xn−P + · · ·+ h1xn−1 + h0xn + h−1xn+1 + · · ·+ h−P xn+P

zn+1 =

.

.

.

H =

.

.

....

.

.

....

.

.

....

hP . . . h0 . . . h−P 0 0 0 0 0. . . 0 hP . . . h0 . . . h−P 0 0 0 0 . . .. . . 0 0 hP . . . h0 . . . h−P 0 0 0 . . .. . . 0 0 0 hP . . . h0 . . . h−P 0 0 . . .. . . 0 0 0 0 hP . . . h0 . . . h−P 0 . . .

0 0 0 0 0 hP . . . h0 . . . h−P

.

.

....

.

.

....

.

.

....

8 / 39

Dealing with side effects

9 / 39

Circulant case / Circulant approximation

Extend convolution matrix into a circulant matrix

Approximation “periodic objects”

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.. . . 0 0 . . . 0 . . . 0 0 0 0 hP hP−1 hP−2. . . 0 0 . . . 0 . . . 0 0 0 0 0 hP hP−1. . . 0 0 . . . 0 . . . 0 0 0 0 0 . . . hP

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.. . . 0 hP . . . h0 . . . h−P 0 0 0 0 . . .

. . . 0 0 hP . . . h0 . . . h−P 0 0 0 . . .

. . . 0 0 0 hP . . . h0 . . . h−P 0 0 . . .

. . . 0 0 0 0 hP . . . h0 . . . h−P 0 . . .

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.h−P 0 0 0 . . .

h−P+1 h−P 0 0 . . .

h−P+2 h−P+1 h−P 0 0 . . .

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

Never compute that. Never.

10 / 39

Circulant case: diagonalization

Circulant matrices diagonalization. . .

H = F †ΛhF

. . . in the Fourier basis

F = N−1/2[e−j2π(k−1)(l−1)/N

]k,l∈1,...,N

Reminder of properties

F t = F

F †F = FF † = IN

F−1 = F † = F ∗

Fx = FFT(x)

F †x = IFFT(x)

11 / 39

Circulant case: eigenvalues

Eigenvalues ∼ frequency response

◦h =

◦h0◦h1

.

.

.◦hN−2◦hN−1

=√N F

·0h−P

·h0

·hP

0·

= fft(h,N)

Eigenvalues “read” on the frequency response graph

−0.5 −0.4 −0.3 −0.2 −0.1 0 0.1 0.2 0.3 0.4 0.5

0

0.5

1

Comment: conditioning of H and low pass character

12 / 39

Circulant case: convolution through FFT

Matricial form of the convolution through FFT

z = Hx = F †ΛhFx

Fz = Λh Fx◦z = Λh

◦x

◦z =

◦h .∗ ◦

x

◦zn =

◦hn

◦xn pour n = 1, . . . , N

−0.5 −0.4 −0.3 −0.2 −0.1 0 0.1 0.2 0.3 0.4 0.5

0

0.5

1

Frequency attenuation, lowpass character, ill-conditioned character

Possibly non-invertible system

Remark: exact convolution calculus always possible by FFT but. . .13 / 39

Example: photographed photographer (“square” response)

Convolutive model (pixels averaging): y = h ? x+ ε

Fourier domain:◦y(ν) =

◦h(ν) .

◦x(ν) +

◦ε(ν)

◦y =

◦h .∗ ◦

x +◦ε

−0.5 −0.4 −0.3 −0.2 −0.1 0 0.1 0.2 0.3 0.4 0.5

0

0.2

0.4

0.6

0.8

1

−0.5 −0.4 −0.3 −0.2 −0.1 0 0.1 0.2 0.3 0.4 0.5

−0.5

−0.4

−0.3

−0.2

−0.1

0

0.1

0.2

0.3

0.4

0.5

Spatial response Frequency response

True (x) Observation (y)

14 / 39

Example: photographed photographer (“motion blur”)

Convolutive model (pixels averaging): y = h ? x+ ε

Fourier domain:◦y(ν) =

◦h(ν) .

◦x(ν) +

◦ε(ν)

◦y =

◦h .∗ ◦

x +◦ε

−0.5 −0.4 −0.3 −0.2 −0.1 0 0.1 0.2 0.3 0.4 0.5

−0.5

−0.4

−0.3

−0.2

−0.1

0

0.1

0.2

0.3

0.4

0.5

Spatial response Frequency response

True (x) Observation (y)

15 / 39

First approach for restoration

16 / 39

First restoration: least squares

Compare observations y and model output HxUnknown: xKnown: H and y

Comparison experiment – model

xH

Hx

y

Quadratic criterion: distance observation – model output

JLS(x) = ‖y −Hx‖2

Least squares solution

xLS = arg minx

JLS(x)

Solution xLS: the best one to reproduce dataFaut que “ca colle”, it must fit, it must match

17 / 39

Calculational aspects

Linear model + least squares Quadratic criterion

JLS(x) = ‖y −Hx‖2 = (y −Hx)t(y −Hx)

= xtHtHx− 2ytHx+ yty

Gradient calculus linear

g(x) =∂JLS

∂x= 2HtHx− 2Hty = −2Ht(y −Hx)

And Hessian calculus constant

Q =∂2JLS

∂x2= 2HtH (> 0 or > 0)

Gradient nullification linear system matrix inversion

(HtH) xLS = Hty

xLS = (HtH)−1Hty

18 / 39

Computational aspects and implementation

Various options and many relationships. . .

Direct calculus, compact (closed) form, matrix inversion

Algorithms for linear system

Gauss, Gauss-JordanSubstitutionTriangularisation,. . .

Numerical optimisation

gradient descent . . . and various modificationsPixel wise, pixel by pixel

Diagonalization

Circulant approximation and diagonalization by FFT

Special algorithms, especially for 1D case

Recursive least squaresKalman smoother or filter (and fast versions)

19 / 39

Computations through FFT

x = (HtH)−1Hty

' (HtH)−1Hty

= (F †Λ†hFF†ΛhF )−1F †Λ†hFy

= F †Λ−1h Fy

F x = Λ−1h Fy

◦x = Λ−1

h◦y

◦xn =

◦yn◦hn

pour n = 1, . . . , N

It is just inverse filtering !!

Matlab Pseudo-code

ObjetEstLS = IFFT( FFT(Data) ./ fft(IR) )

20 / 39

Least squares solution (Hunt and Brain)

−50 0 50 100 150 200

0

0.5

1

1.5

2

−50 0 50 100 150 200

0

0.5

1

1.5

2

0 50 100 150 200 250−10

−5

0

5

10

True Observation LS solution

Advantages / Disadvantages

Very general, hyper-fast, no parameter to tune

. . . but does not work . . . (except if . . . )

21 / 39

Frequency analysis

−0.5 −0.4 −0.3 −0.2 −0.1 0 0.1 0.2 0.3 0.4 0.5

0

0.5

1

1.5

2

2.5

Observation y

−0.5 −0.4 −0.3 −0.2 −0.1 0 0.1 0.2 0.3 0.4 0.50

200

400

600

800

1000

Inverse filter◦h

−0.5 −0.4 −0.3 −0.2 −0.1 0 0.1 0.2 0.3 0.4 0.50

5

10

15

LS solution x

22 / 39

Default of the least squares solution

Unacceptable solution, explosive, noise dominated

Three analyses: signal-image, numerical, statistical

Bandwidth, non-observed or badly-observed frequenciesBadly-scaled matrix, eigenvalues, numerical instabilitiesStrong variance (even if minimal variance and unbiased)

Ill-posed problem

Missing informationRegularisation

23 / 39

Least squares solution: well conditioned problem

Frequency

-0.5 0 0.5

0

0.2

0.4

0.6

0.8

1

True Observation LS solution

24 / 39

Regularisation: generalities

Data insufficiently informative Account for prior information

1 Question of compromise, of competition2 Specificity of methods

Here: smoothness of images, ideally edge preserving Explicitation of prior information

Regularisation

Through penalty, constraint, re-parametrisation. . . or other ideas . . . but regularisation anyhow

Regularisation by penalty

JPLS(x) = ‖y −Hx‖2 + µP(x)

Restored imagexPLS = arg min

xJPLS(x)

25 / 39

Quadratic penalty (1)

Differences, higher order derivatives, generalizations,. . .

P(x) =∑n

(xn+1 − xn)2

P(x) =∑n

(xn+1 − 2xn + xn−1)2

P(x) =∑n

(α xn+1 − xn + α′ xn−1)2

P(x) =∑n

(αtn x)2

Linear combinations (wavelet, other-truc-en-et,. . . dictionaries,. . . )

P(x) =∑n

(wtnx)2 =

∑n

(∑m

wnmxm

)2

Redundant or notLink with Haar wavelet and other

26 / 39

Quadratic penalty (2)

2D aspects: derivative, finite differences, gradient approximations

P(x) =∑p∼q

(xp − xq)2

=∑n,m

(xn+1,m − xn,m)2 +∑n,m

(xn,m+1 − xn,m)2

Norms of filtered imagesThrough rows and columns 1

0−1

+[

1 0 −1]

or

1 2 10 0 0−1 −2 −1

+

1 0 −12 0 −21 0 −1

Jointly both of them 0 −1 0

−1 4 −10 −1 0

Remark

Notion of neighborhood and Markov fieldAny highpass filter, contour detector (Prewitt, Sobel,. . . )Linear combinations: wavelet, contourlet and othertruc-en-et,. . . dictionaries,. . .

27 / 39

Quadratic penalty (3)

Other possibilities (slightly different)

Enforcement towards a known shape x0

P(x) = (x− x0)t(x− x0)

Usual Euclidean norm (separable terms)

P(x) = xtx

More general form

P(x) = (x− x0)tM(x− x0)

In the following development

P1(x) =∑p∼q

(xp − xq)2 = ‖Dx‖2 = xtDtDx

und D = . . .

28 / 39

Penalised least squares restoration

Remind the criterion. . .

JPLS(x) = ‖y −Hx‖2 + µ ‖Dx‖2

. . . and its gradient. . .

g(x) =∂JPLS

∂x= −2Ht(y −Hx) + 2µDtDx

. . . and its Hessian. . .

Q =∂2JPLS

∂x2= 2HtH + 2µDtD

. . . the normal system of equation. . .

(HtH + µDtD) xPLS = Hty

. . . and the minimiser . . .

xPLS = (HtH + µDtD)−1Hty

29 / 39

Computational aspects and implementation

Various options and many relationships. . .

Direct calculus, compact (closed) form, matrix inversion

Algorithms for linear system

Gauss, Gauss-JordanSubstitutionTriangularisation,. . .

Numerical optimisation

gradient descent. . . and various modificationsPixel wise, pixel by pixel

Diagonalization

Circulant approximation and diagonalization by FFT

Special algorithms, especially for 1D case

Recursive least squaresKalman smoother or filter (and fast versions)

30 / 39

Circulant form

Circulant approximation

for H as previouslyfor D in addition

One extra-term of interaction

D =

1 −1 0 0 . . . . . . 00 1 −1 0 . . . . . . 0

.

.

.. . .

. . ....

.

.

.. . .

. . ....

0 . . . . . . 0 1 −1 00 . . . . . . 0 0 1 −1−1 . . . . . . 0 0 0 1

(N ×N)

Diagonalization of DD = F †ΛdF

Eigenvalues by FFT◦d = fft([-1,1])

31 / 39

Computations by FFT

x = (HtH + µDtD)−1Hty

= (F †Λ†hFF†ΛhF + F †Λ†dFF

†ΛdFµ)−1F †ΛhFy

= F †(Λ†hΛh + µΛ†dΛd)−1Λ†hFy

◦x = (Λ†hΛh + µΛ†dΛd)

−1Λ†h◦y

◦xn =

◦h∗n

|◦hn|2 + µ|

◦dn|2

◦yn for n = 1, . . . N

This is just Wiener filtering !!

Matlab pseudo-code

Gain = fft(IR)∗ ./ (|fft(IR)|2 + mu * |fft([-1,1])|2)

ObjetEstPLS = IFFT( FFT(Data) .* Gain )

32 / 39

Solutions by quadratic penalties (Hunt)

Evolution with µ

0 50 100 150 200 250−10

−5

0

5

10

0 50 100 150 200 250−1

0

1

2

3

0 50 100 150 200 250−1

0

1

2

3

µ = 0 µ = 10−3 µ = 10−2

0 50 100 150 200 250−1

0

1

2

3

0 50 100 150 200 250−1

0

1

2

3

0 50 100 150 200 250−1

0

1

2

3

µ = 10−1 µ = 1 µ = 10+1

33 / 39

Brain example

True Observation Quadratic penalty

34 / 39

Frequency analysis: equalisation

0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.50

0.2

0.4

0.6

0.8

1

0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.50

200

400

600

800

1000

0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.51

1

1

1

1

1

1

1

0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.50

0.2

0.4

0.6

0.8

1

0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.50

5

10

15

20

25

30

0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.50

0.2

0.4

0.6

0.8

1

0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.50

0.2

0.4

0.6

0.8

1

0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.50

0.5

1

1.5

2

2.5

3

3.5

0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.50

0.2

0.4

0.6

0.8

1

0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.50

0.2

0.4

0.6

0.8

1

0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.50

0.2

0.4

0.6

0.8

1

0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.50

0.2

0.4

0.6

0.8

1

µ = 0 µ = 10−3 µ = 10−1 µ = 101

Depending on the considered frequency

equalisation in the correctly observed bandwidthnullification in the uncorrectly observed bandwidth

35 / 39

Bias / Variance (1)

Bias (normalized) Bias

0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.50

0.5

1

1.5

0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.50

0.01

0.02

0.03

Frequency Frequency

Variance

0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.50

0.005

0.01

0.015

Frequency

36 / 39

Bias / Variance (2)

Bias

−8 −6 −4 −2 0 2 4 60

0.05

0.1

Variance

−8 −6 −4 −2 0 2 4 60

2

4

6

8

Mean Square Error

−8 −6 −4 −2 0 2 4 6−3

−2

−1

0

1

2

µ

37 / 39

Photographed photographer: circulant or not

True Observation PLS PLS

(circulant) (non-circulant)

38 / 39

Conclusions

Synthesis

Image deconvolution

Quadratic penalties and smoothness of solution

Gradient of gray level and extensions (and other transforms)

Closed-form expression and linear w.r.t. the data

Numerical computations

Circulant case (diagonalization) FFT onlyNumerical optimisation, system solvers,. . .

Extensions (next lectures)

Also available for

non-invariant linear direct modelcolour images, multispectral and hyperspectralalso signal, 3D and more, video, 3D+t. . .

Extension to non-quadratic better image resolution

Including constraints better image resolution

Hyperparameters estimation, instrument parameter estimation,. . .

39 / 39