200081003 friday food@ibbt

Ghent University

Image Processing and Interpretation Group

Aleksandra Pizurica

Advances and challenges in image and video restoration

2Image and video restorationIBBT Friday food talk, October 3, 2008

Median filter: Reduction of impulse noise

median over 3x3

Median filter removes isolated noise peaks, without blurring the image

impulse noise


Median filter and reduction of white noise

median over 3x3

For not-isolated noise peaks (e.g., white Gaussian noise) median filter is not very efficient.

original


Why is denoising important

Not only visual enhancement, but also: automatic processing is facilitated!

original denoised

Example: edge detection


Image restoration example

State of the art image restoration methods iterate wavelet domain denoising and

Fourier domain deconvolution. From now on we focus on the denoising step

©Max Planck Institute for Biophysical Chemistry

[F. Rooms et al; Journal of Microscopy 2005]


Overview

• Wavelet domain image restoration

• Gain from using other wavelet-like representations

• Medical applications: MRI, CT, OCT

• On noise and blur estimation

• Video denoising and advances in 3D video


highpass

lowpass

wavelet coefficients

scaling coefficients

s j

2

2

w j+1

h

g

s j+1

2w j+2

h

g

s j+1

2

2

2

w j+3

h

g

s j+3

DWT algorithm: a filter bank iterated on the lowpass output

Discrete Wavelet Transform (DWT)


Choosing a wavelet: Nv, support size K, symmetry

Nv - number of vanishing moments: ,0)( =ψ∫∞

∞−

k10 −≤≤ vNkdttt

Daubechies wavelets dbNv:

K 2Nv-1≥A tradeoff:

0 5 10 15-0.5

0

0.5

1

1.5

-5 0 5-1

-0.5

0

0.5

1

1.5Symmlets (Daubechies)

ϕ ψsym8

1.5 2

-0.5 0 0.5 1 1.50

0.5

1

1.5

-1.5 -1 -0.5 0 0.5-1.5

-1

-0.5

0

0.5

1

1.5

db1

ϕ ψ

0 1 2 3-0.5

0

0.5

1

-1 0 1 2-2

-1

0

1

db2

ϕ ψ

0 5 10 15-0.5

0

0.5

1

1.5

-5 0 5

-1

-0.5

0

0.5

1

db8

ϕ ψ-4 -2 0 2 4

-1

-0.5

0

0.5

1

-5 00

0.2

0.4

0.6

0.8

5

-5 0 5

-0.5

0

0.5

1

1.5

2

-4 -2 0 2 4

-2

-1

0

1

2

Biorthogonal wavelets

ϕ ψ

ϕ~ ψ~K=15

K=3

K=1


DETAIL IMAGESwavelet coefficients

APPROXIMATIONscaling coefficients

Wavelet coefficient values

0

Peaks indicate image edges

Two dimensional DWT


Noise in the wavelet domain

Wavelet coefficient values

Peaks indicate image edges

0

Noise-free reference


Generalized Laplacian (generalized Gaussian) distribution

noise-free histogram f(y)=Aexp(- |y /s |ν )

s: scale parameter

ν: shape parameter

)10( ≤≤ν

�Parameters accurately estimated from a signal corrupted by additive white Gaussian noise

noisy

Marginal priors: Generalized Laplacian

Often yields complicated expressions

Extension to higher dimensions (joint histograms) difficult


Gaussian Scale Mixture (GSM) models

z: mixture variable, random multiplier

= uzy

wavelet coefficient Gaussian random variable

u

f(u)f(y)

y

Efficient for modelling joint histograms of the neighboring wavelet coefficients

A state-of the art denoiser for many years BLS-GSM: Bayesian Least Squares estimator using GSM prior [Portilla et al, IEEE TIP’03]

∫∞

∞−= dzzfzzEE )(),|()|( xyxy

xCC

Cxy

nu

u

+=

z

zzE ),|(

nunyx +=+= z

noise:, nu CC signal and noise

covariances


Locally adaptive denoising ProbShrink

yl

LSAI zl

LSAI

OBSERVATION

ESTIMATE

yyzyHPηξµ

ηξµβ

+==

1),|(ˆ

1

)|(

)|(

0

1

Hyf

Hyf=η

)|(

)|(

0

1

Hzf

Hzf=ξ )(

)(

0

1

HP

HP=µ

zy

f(y|H1)

f(y|H0)f(z|H0)

f(z|H1) P(H1)

P(H0)

LSAInoisy coefficient subband statistics

,ny += β

[Pizurica&Philips, IEEE TIP 2006]LSAI – Local Spatial Activity Indicator

H1 signal of interest present

H0 signal of interest absent


Locally adaptive denoising: ProbShrink…

[Pizurica&Philips, IEEE TIP 2006]


ProbShrink for correlated noise…

50 100 150

20

40

60

80

100

120

140

160

180

local window

23

22

X

X

vector of coefficients

X22 X23

X22

X23

X22

H1

H0

H1

[B. Goosens, A. Pizurica, W. Philips; IEEE TIP 2008, in press]


… ProbShrink for correlated noise

[B. Goosens, A. Pizurica, W. Philips; IEEE TIP 2008, in press]


Denoising by singularity detection

w1 wavelet coefficients

w2

w3

w1

Input signal Rate of increase of the

modulus of the wavelet

transform across scales is

proportional to thelocal Lipschitz regularity

[Mallat&Zhong, IEEE IT 1992]


Statistics: magnitude and the rate of increase

ACR-4 -2 0 2 4 6

0

0.2

0.4

0.6

0.8

1

scale

cone of influence

edgesnoise

-50 0 50 100 150 200 250 3000

0.01

0.02

0.03

0.04

0.05

noisy magnitude

edges

noise

ACR

Noise standard deviation=25.5

Magnitude

-3 -2 -1 0 1 2 30

50

100

150

xl= 1

xl= 0noise

edges

Average Cone Ratio – an estimate

of the local Lipschitz exponent –measures the rate of increase of

the coefficients across the scales

[A. Pizurica et al; IEEE TIP 2002]


Inter- and intrascale dependencies

• Bivariate models• Hidden Markov Tree models• Markov Random Field models


Statistical modeling: MRF models

positive potential

negative potential

Example: penalize isolated peaks

−= ∑

∈ςCCV

ZP )(exp

1)( xx

clique potentials

cliquesneighborhoodx0

Prior model

P(x)

xMAP


Statistical modeling: MRF models

cliquesneighborhoodx0

Prior model

P(x)

xMAP

Initial edges Iteration 3Iteration 2Iteration 1


OriginalGamma MAP filter

MRF based wavelet denoising

wavelet filter

[A. Pizurica et al; ICIP 2001]


GSM in non-overlapping blocks • Ignores non-local correlations

• Block size?

[Guerrero-Colon et al,

IEEE TIP,08]

Spatially variant GSM

(SVGSM)

Computationally expensive

[Portilla et al, Spie2008]

Mixture of GSM

(MGSM)

MRF models

Fields of GSM[Liu and Simoncelli,

PAMI’08]

Field of Experts (FoE)

[Roth and Black, CVPR’05]

[Tappen, Adelson, Freeman,

CVPR’07, CVPR’08]

Mixture of projected GSM

(MPGSM)

[Goossens, in review TIP 2008]Dimension reduction

in MGSM

Current trends in Bayesian wavelet denoising

?

Gaussian Scale Mixture (GSM) model

[Portilla et al, IEEE TIP 2003]

Shortcoming: assumes the same but scaled covariance for the whole subband


Overview







Why other multiresolution representations

Classical wavelets are well suited for point-like singularities,

but not for curvilinear singularities in images

• Poor orientation selectivity; no difference between 45 and -45o

• Checkerboard pattern � appears also as an artifact in denoising

Many wavelet-like representations with a better orientation selectivity: complex wavelets [Kingsbury, Selesnick] , steerable pyramids [Freeman, Adelson], curvelets [Donoho, Candes], contourlets [Do, Vetterli], …

An example of wavelet base functions


Curvelets: specific tiling of the frequency plane:

localized + directional

Curvelet-domain image denoising…


Curvelet Hard Thresholding

PSNR=29.02dBPSNR=22.16dB

Noisy ImageWavelet ProbShrink

PSNR=29.50dB

…Curvelet domain image denoising…

PSNR=30.43dB

Curvelet ProbShrink

[L. Tessens, A. Pizurica, W. Philips, J Electr Imag 2008 in press]


…Curvelet domain image denoising…

Results

Wavelet ProbShrink Curvelet ProbShrink


Wavelet ProbShrink

…Curvelet domain image denoisingCurvelet ProbShrink


Overview







MRI denoising: signal dependent noise

high SNR ( f =f1 )

f1

low SNR (f=0)

m

p(m)

noise-free f

noisy m

ma

gn

itud

e

co

ntr

ast

SNR


Step 1: Bias removal

Square magnitude MRI image – after squaring constant bias, proportional

to noise standard deviation.

For better results: square root the result before denoising!

MRI denosing: algorithm

[Pizurica et al IEEE TMI 2003]

T?

Coarser, processed detail

A noisy detailMask

p(z|H1)

p(z|H0) histograms( )log

p(z|H0)

p(z|H1)

Step 2: Denoising (coarse-to-fine, empirical density estimation)


Noisy image Denoised image Ground truth

MRI denosing: some results


3D MRI volume denoising

using 3D dual-tree complex wavelet transform

[J. Aelterman et al, EUSIPCO 2008]


Denoising low-dose CT images

• Reducing radiation dose increases noise level

• Can we use denoising on low dose CT to obtain the same diagnostic quality as in a higher dose CT image? [IBBT Ica4dt project]

• Difficulties:

- non-stationary correlated noise

- Streak artefacts

- How to estimate noise


Denoising algorithm

wavelet

transform

(WT)

Vector

ProbShrink

Inverse wavelet

transform

(IWT)

(Dual-tree

complex)

segmentation

H1

H0

H1

[B. Goossens et al, EMBS 2007]


Watershed segmentation

…Results

denoised


Qualitative Validation CT: psycho-visual experiment

…

+++

--

++

++

+

++

+

+

+

+++

++

---

+

+

--

+

-

-

-

+++

++ --+ Versatile Probshrink

+++

++

+

+++

Structure

---+++ -Wavelet GSM Filter

--++ --Curvelet Filter

-+ --Versatile Probshrink2

+++---+++ Low-dose CT

QualityNoiseBlurAbdomen/Lung

The abdomen image judged better than the original by radiologists!


Optical Coherence Tomography (OCT) images

2D signal

[IBBT Ica4dt project, with AGFA Healthcare]

OCT – “echography with light”

Noise: speckle similar to that in radar and ultrasound images

3D OCT data


Denoising OCT images

A developed 3D OCT denoiser combines• wavelet domain speckle filter• motion compensated video denoising method

3D OCT data

Video denoising

coarse-to-fine

processing

Locally adaptive denoising

0 50 100 150 200 2500

0.005

0.01

0.015

0.02

0.025

0.03

0.035

Image:g120406breastseconformolnogelLR0050 Detail:Dx1 Parameter:b=10.3752

likelih

ood p

(m|1

)

magnitude m

Gamma, b=10.3752

0 50 100 1500

0.05

0.1

0.15

0.2

Image:g120406breastseconformolnogelLR0050 Detail:Dx2 Parameter:a=4.8348

likelih

ood p

(m|0

)

magnitude m

Laplace, a=4.8348

Signal and noise statistics

[IBBT Ica4dt project]


LeeRKTOur methodNoisy Image SATGTF

Results and evaluation of OCT denoising



Noisy LeeSATOur method

[Pizurica et al; CMIR 2008 in press]



input Our method (2D version)BLS-GSM

[Pizurica et al; CMIR 2008 in press]


Overview







Block-based

Search for blocks of nearly uniform intensity

Wavelet basedGradient distribution based

HH1

HL1

LH1usethis part

Noise variance estimation

or compensate forthe peak shift

noise (Rayleigh distr.)

σσσσ

Smoothing based

signal+noise

smooth

estimateσ

Median{|HH1|}

0.6745=σ̂


Blur estimation using wavelet coefficients

Original image Blurred image-4 -2 0 2 4 6 8

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

original

blurred

ACR 1-2

PDF

ACR - Average Cone Ratio – an estimate of the local Lipschitz exponent –

measures the rate of increase of the coefficients across the scales

• A well known approach: kurtosis of the wavelet coefficient histogram

• An alternative: examine the propagation of the wavelet coefficients across

the scales


blur

-1 0 1 2 3 4 50

1000

2000

3000

4000

5000

6000

ACR 2-4

reference

blur 3

blur 5

blur 7

blur 0 + noise

blur 3 + noise

blur 5 + noise

blur 7 + noise

Different colors -

different levels of blur

0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 23000

3500

4000

4500

5000

5500

6000

ACR 2-4

referenceblur 3blur 5blur 7blur 0 + noiseblur 3 + noiseblur 5 + noiseblur 7 + noise

solid – noise free

dashed – noisy

Blur estimation using wavelet coefficients


Overview







[V. Zlokolica, A. Pizurica, W. Philips; IEEE TCSVT 2006]

Noise Estimation

Time delay

Adaptive SpatialFiltering

2D WaveletTransform

Inverse 2D WaveletTransform

Input NoisyFrame

DenoisedFrame

Adaptive SpatialFiltering

Motion Estimation

Recursive Temporal Filtering

Time delay

Video denoising


center ofthe motion block

motion direction(smaller amplitude)

motion direction(larger amplitude)

Video denoising: motion estimation…

Accurate motion estimation is essential for video denoising.Also important: reliability of the estimated motion at each point


…Motion compensated video denoising

[V. Zlokolica, A. Pizurica, W. Philips; IEEE TCSVT 2006]

Further development currently within IBBT project ISYSS


Reusing motion estimator from video codecs

• Motion estimators from video codecs tolerate errors � cannot be directly used in denoising

• Can we still use them with some postprocessing? The core of our approach:

• Motion field refinement step

• Reliability to motion estimates controls the recursive filter

• Competitive with state-of-the art video denoisers

[LJ. Jovanov et al; IEEE TCSVT 2008, in press]


[Balster; TCSVT 2006] [Jovanov; TCSVT 2008]

Reusing motion estimator from video codecs

noise-free input


Denoising and outlier removal in 3D video

Time-of-flight camera �records simultaneously luminance and depth information

The biggest errors in the depth measurement are induced by strong ambient light �� The measured distance is much smaller than the true distance)

Degradations in the depth image: noise, and outliers (similar to impulse noise but in bursts)

3D reconstructions using “surf” in Matlab


Noisy 3D video sequence (luminance and depth)

Luminance image contains much less noise

Luminance and depth images are correlated

� Use the luminance information for denoising depth data


Denoised luminance and depth


Acknowledgements

Thanks to my colleagues for their contributions

• Vladimir Zlokolica (video denoising)

• Bart Goossens (removal of correlated noise)

• Ljubomir Jovanov (video, 3D video, OCT)

• Linda Tessens (curvelets)

• Jan Aelterman (MRI denoising)

• Filip Rooms (deblurring)

• Ewout Vansteenkiste (quality evaluation CT)

Related material available at: http://telin.ugent.be/~sanja

200081003 friday food@ibbt

Documents