Influence of Signal-to-Noise Ratio and Point Spread Function on

Limits of Super-Resolution

Tuan Pham

Quantitative Imaging GroupDelft University of Technology

The Netherlands

Conf. 5672: Image ProcessingAlgorithms and Systems IV

© 2004 Tuan Pham 2

Super-Resolution: an example

4x super-resolution128x128x100 infra-red sequence

© 2004 Tuan Pham 3

Super-Resolution: an example

4x super-resolutionLow resolution

© 2004 Tuan Pham 4

Overview and Goal

Positioninglimit

SNRlimit

Resolvinglimit

Limits of Super-ResolutionGOAL: Derive the given system inputs

No. of inputs SNR PSF

System inputs

© 2004 Tuan Pham 5

Limit of registration

2nσ

• Cramer-Rao Lower Bound for 2D shift: I2(x, y) = I1(x+vx, y+vy) :

2x x y

S S2 2

x y yS S

I I I1( )

I I Inσ

⎡ ⎤⎢ ⎥= ⎢ ⎥⎢ ⎥⎣ ⎦

∑ ∑∑ ∑

F v

where , is noise variance, and F is the Fisher Information Matrix:

1 2 211

1 2 222

var( ) I ( )

var( ) I ( )

x n yS

y n xS

v Det

v Det

σ

σ

−

−

≥ =

≥ =

∑

∑

F F

F F

• Optimal registration is achievable by iterative optimization

• CRLB also exists for more complicated motion models:- 2D projective - optic flow

/ , /x yI I x I I y= ∂ ∂ = ∂ ∂

© 2004 Tuan Pham 6

Noise of HR image after fusion

• Total noise = Intensity noise + Noise due to registration error

222 2 2

n I regINμσ σ σ= + ∇

I re gIx

σ σ∂=

∂

x

I

σreg

σI

position error distribution

localsignal

Intensity error distribution

Blurred & mis-registered5x5 box blur, pixel

Noise due to mis-registration0.2regσ = mis-registration → noise

: zoom factor

: # of LR images

: gradient energy

μN

2I∇

© 2004 Tuan Pham 7

• After fusion, the High-Resolution image is still blurry due to:– Sensor integration blur (severe if high fill-factor)– Optical blur (severe if high sampling factor)

The need for deconvolution

On-chip microlens of Sony Super HAD CCD

© 2004 Tuan Pham 8

• Spectrum is cut off beyond fc due to optics → data forever lost

The necessity of aliasing

0 0.5 1 1.5−0.4

−0.2

0

0.2

0.4

0.6

0.8

1

frequency in unit of sampling frequency (f/fs)

freq

uenc

y sp

ectr

a / t

rans

fer

func

tions

OTF (sampling factor = 0.25)STF (fill factor = 1)Original scene spectrumBand−limited spectrumAliased image spectrum

0 0.5 1 1.5 2−0.4

−0.2

0

0.2

0.4

0.6

0.8

1

frequency in unit of sampling frequency (f/fs)

freq

uenc

y sp

ectr

a / t

rans

fer

func

tions

OTF (sampling factor = 1)STF (fill factor = 1)Original scene spectrumBand−limited spectrumSampled image spectrum

Aliasing due to under-sampling (fs < 2fc)

No aliasing at critical sampling (fs = 2fc)

© 2004 Tuan Pham 9

Limit of deconvolution

• Blur = attenuation of HF spectrum

• Deconvolution = amplify HF spectrum:– noise is also amplified → limit the deconvolution

• Deconvolution can only recover:– Spectrum whose signal power > noise power

fusion result after deconvolution simulated at resolution = 0.44

resolution factor = 0.44

recoverable

Not recoverable

PS>PN

© 2004 Tuan Pham 10

SR reconstruction experiment

64x64 LR inputsampling=1/4, fill = 100%

4xHR after fusionBSNR = 20 dB

4xSR after deconvolutionSR factor = 3.4

• Aim: show that the attainable SR factor agrees with the prediction• Experiment:

– Inputs: sufficient shifted LR images of the Pentagon– Output: SR image and a measure of SR factor from edge width

© 2004 Tuan Pham 11

SR reconstruction experiment

64x64 LR inputsampling=1/4, fill = 100%

4xHR after fusionBSNR = 20 dB

4xSR after deconvolutionSR factor = 3.4

• Aim: show that the attainable SR factor agrees with the prediction• Experiment:

– Inputs: sufficient shifted LR images of the Pentagon– Output: SR image and a measure of SR factor from edge width

© 2004 Tuan Pham 12

SR factor at BSNR=20dB

0

1

2

0

0.5

1

0

2

4

6

sampling factor (f

s/2f

c) fill factor

SR

fact

or

0.6

1.0

1.9

3.4

3.2

0

1

2

0

0.5

1

0

2

4

6

sampling factor (f

s/2f

c) fill factor

SR

lim

it

0.6

1.0

1.7

2.5

3.0

Measured SR factor Predicted SR factor

• Consistent results between prediction and measurement:– Attainable SR factor depends mainly on sampling factor (i.e. level of aliasing)

© 2004 Tuan Pham 13

Summary

• Limit of Super-Resolution depends on:– input Signal-to-Noise Ratio

– System’s Point Spread Function and how well it can be estimated

• Procedure for estimating SR factor directly from inputs:– Measure noise variance from LR images

– Derive registration error

– Determine SR factor from the Power Spectrum Density (PS > PN)

2Iσ

2regσ