Massey University

Image Resolution Improvement from Multiple Images

Donald Bailey

Institute of Information Sciences and Technology

Massey University

Palmerston North

NEW ZEALAND

Overview

Describe the resolution improvement process Describe the results of my investigations Investigations in 1 dimensions

– Super-resolution of bar codes

Registration in 2 dimensions– Comparison of registration methods– Detailed description of predictive interpolation

Description of the Problem

Given a set of related independent low resolution images, combine these together to construct a single high resolution image– output has more detail than any of the input images

Resolution Improvement

Resolution limited by number of pixels– Resolution depends on sampling density

An ensemble of images:– Each image provides separate samples– Potentially higher sampling density

Reconstruction steps:– Register images– Resample ensemble– Inverse filter

Sampling Requirements

Images must be sub-sampled If sample rate is greater than Nyquist rate

– Can reconstruct the image at any desired resolution– A single image contains all information– Can only improve the signal to noise ratio

If sample rate is less than Nyquist rate– Each individual image is aliased– Cannot obtain higher real resolution from single image– Resampling the ensemble untangles the aliased

information

One Dimension Example

“Super-Resolution of Bar Codes”,D.G. Bailey, Journal of Electronic Imaging, 10 (1), pp 213-221

(January 2001).

Problem: How can we read this bar code?

Information content of UPC Bar Codes

12 digits, each 7 units wide

Guard bands Total width is 95 units Each bar or space is 1-4 units wide Broadband frequency spectrum Centre of main lobe contains required data

6 digits 6 digitsguard bands

Super-resolution procedure

A tilted 2D bar code image provides an ensemble of independently sampled 1D images

Register low-resolution images– gives relationship between individual images

Resample the ensemble at a higher rate– creates a high resolution image

Remove system effects– reduces the sampling blur and effect of camera electronics

Registration

Determines the offset between rows Phase shift in frequency domain proportional to linear

offset and spatial frequency

Procedure:– Fourier Transform each row, keep phase– Subtract phase of first row from each row– Unwrap phase image– Discard higher frequencies– Least squares fit to calculate offset per row

ajeFaxf )()(

Input image

Phase image

Resampling

By interleaving samples from different rows increase the sample rate.

Originalimage

samples

Newimage

samples

Selectedsample

rows

Resampling

By interleaving samples from different rows increase the sample rate.

Originalimage

samples

Newimage

samples

Selectedsample

rows

Resampling

Increase sample rate by an integer multiple of original sample rate

Select rows with offsets nearest the desired sample positions

4 x sample rate

Practical limitations

From synthetic image

From actual image

0Spatial frequency

Am

plitu

de

0/2 0 0 0/2

0Spatial frequency

Am

plitu

de

0/2 0 0 0/2

Practical limitations

Real bar code limitations– Ink smearing means bar and space widths not exact– Smears the envelope in the frequency domain

Image capture degradations

Object

Cameraangle

Lenssystem Sensor

DigitalImage

Videosignal Video

framegrabber

Cameraelectronics

Practical limitations Image distortions

– Perspective distortion from camera angle

– Lens distortion

Lens point spread function– Spatially variant low pass filter

Image sensor– area sampling - low pass filter

– aliasing

Camera electronics– low pass filter, perhaps with high frequency emphasis

Frame grabber– sampling (more aliasing)

Removing system effects

Aliasing is not a problem– it is actually necessary for higher resolution reconstruction– resampling the ensemble untangles the aliased information

Main effects are the low pass filter characteristics– lens point spread function– area sampling in sensor– smoothing filter camera electronics– anti-alias filter in frame grabber

Removing system effects

0Spatial frequency

Am

plitu

de

0/2 0 0 0/2

Assume no distortion, and no spatial variation in the low pass filter characteristic

Estimate the system response by comparing synthetic and actual reconstructed images

Remove using an inverse linear filter

Results

Resampled ensemble

System response removed

Straightened and averaged

Thresholded

Original image

Bar Code Conclusions

A two-dimensional image of a bar code tilted slightly provides an ensemble of related one-dimensional images

The low resolution images must be aliased It is necessary to compensate for limitations in the

image capture system Analogue video cameras make more complex

– Image sampled twice– Additional analogue filters

Modest gains in resolution are achievable

Extending to 2 Dimensions

Problem is considerably more complex Require multiple 2 dimensional images

– Captured at different times– Motion is a limitation

Registration more complex– Bar code images all had constant offset per row– In 2D every image is independent, with 2D offset

Resampling more complex– Need more images for same improvement

• 4 images to improve resolution by 2

Registration in 2 Dimensions

Requirements– Accurate sub-pixel offset between images– Work directly on low resolution images– Insensitive to aliasing– Tolerates a low level of noise– Does not rely on particular objects in the image– For practical use, must be fast

Conventional approach to sub-pixel registration– Interpolate images to chosen high resolution

• Increased data volume slows this method down

– Perform pixel accuracy registration• Requires a search

Sub-Pixel Registration Methods

Phase based methods– Similar to 1D case, but extended to 2D

Determine fit surface on integer grid– Interpolate this to find optimum fit– Correlation methods– Difference methods

Predictive interpolation– A new method that turns problem around

Other approaches– Rely on locating objects or edges within the image

Phase Methods

An offset in the image domain corresponds to a phase shift in the frequency domain

Procedure– Window the image and reference– FFT and keep the phases– Unwrap the phase difference– Weighted least squares fit of a plane to the phase

vyuxvuGvuF N 002),(),(

)(00

002

),(),( vyuxjNevuFyyxxf

Correlation - Pixel Accuracy

Multiply an offset image by a reference Accumulate product in the overlap region

Normalise by the average pixel value– Prevents bias if the image has a gradient

Frequency domain correlation

x y

jyixgyxfjic ),(),(),(

),(),(),( * vuGvuFvuC

Correlation - Sub-pixel Accuracy

i

Correlation peakc(i)

ipk

i0

i1

i 1

c 1

c1

c0

)),min((2 110

110pk

ccccc

ii

Perform pixel level correlation first Interpolate to find peak to sub-pixel accuracy Expect correlation peak to be a pyramid

– Only strictly true with regions of uniform value– Approximately true if there are step edges

Width of pyramid is twice smallest feature width– Only local information should be used

Can be shown to be related to correlation Subtract an offset image from a reference Accumulate difference in the overlap region

Minimum gives offset to nearest pixel For sub-pixel accuracy

– Expect minimum to be an inverted pyramid (locally)– Interpolate to find minimum using previous method

Difference Methods

x y

jyixgyxfjid ),(),(),(

Other Registration Methods

Centre of gravity– Segment objects from background– Centre of gravity of objects to sub-pixel accuracy

Line fitting– Detect lines or edges– Fit a line or curve to detected points

Requires knowledge of contents of image Accuracy limited by size of object / edge and

accuracy of segmentation / detection

Predictive Interpolation

Turns the problem around– Predicts the pixel values as a function of those in a

reference image– Uses the bilinear interpolation equation as a linear predictor

– Subject to the constraint:

– Then offset is:

),( yxf

)1,( yxf )1,1( yxf

),1( yxf

),( yxg

)1,1(),1(

)1,(),(),(

1110

0100

yxfAyxfA

yxfAyxfAyxg

111100100 AAAA

),(),( 1101111000 AAAAyx

Predictive Interpolation

Procedure– Requires the image to be pre-registered to nearest pixel

• Can use a search, or hierarchical registration to do this

– Determine coefficients Axx that minimise the error• Uses weighted least squares• Weight each point with standard deviation of its 4 references

– From coefficients, get offset directly

Properties– Very fast - single pass if registered to nearest pixel– Very good accuracy - typically better than 5% of a pixel

Evaluating the Registration

Requires a set of images with known offsets– Start with a single high resolution image– Simulate capturing with a lower resolution camera by

filtering and subsambling– Add random Gaussian noise to each low resolution image

Evaluation Procedure

Measure offset between each pair of images– 36 offset measurements for 9 images

Enforce consistency between measurements– Only 8 independent offsets– Will reduce errors up to

Error is difference between expected offset and measured offset– Average to give the RMS registration error

N2

Results

Structured image No noise:

– All worked well

Noise sensitivity:– Phase method

only few frequencies– Predictive method

weighting gives onlyfew measurement

A

0.00

0.05

0.10

0.15

0 5 10 15 20 25 30 35 40Noise SD

CorrelationDifferencePhasePredictive

RM

S E

rror

Results

Low detail image No noise

– Phase is best– Others adequate

Noise sensitivity:– Predictive method

sensitive to noise

B

0.00

0.05

0.10

0.15

0 5 10 15 20 25 30 35 40Noise SD

CorrelationDifferencePhasePredictive

RM

S E

rror

Results

Medium detail image No noise

– Phase and Predictivesignificantly better

Noise sensitivity:– Correlation

noise insensitive– Difference

noise insensitive

C

0.00

0.05

0.10

0.15

0 5 10 15 20 25 30 35 40Noise SD

CorrelationDifferencePhasePredictive

RM

S E

rror

Results

Low detail text No noise

– Phase and predictivesignificantly better

Noise sensitivity:– Predictive method

improves with lownoise

D

0.00

0.05

0.10

0.15

0 5 10 15 20 25 30 35 40Noise SD

CorrelationDifferencePhasePredictive

RM

S E

rror

Results

High detail text– Significant aliasing

No noise– Correlation and

Difference aresignificantly poorer

– Phase and Predictivegive excellentresults

E

0.00

0.10

0.20

0.30

0.40

0.50

0 5 10 15 20 25 30 35 40Noise SD

CorrelationDifferencePhasePredictive

RM

S E

rror

Summary of Results

Correlation and Difference methods– Similar results– Performance deteriorates with increasing detail– Relatively insensitive to noise

Phase and Predictive methods– Best overall with 1 - 2 % pixel accuracy

• Includes enforcing consistency

Predictive method– Results almost independent of image– Improved with addition of small amounts of noise

Reconstruction

Low resolution

Resampled

Inverse filtered

Conclusions of Comparison

Correlation and difference methods– Insensitive to noise– Perform poorly in presence of high detail– Pyramidal interpolation method breaks down

Phase and predictive methods– More sensitive to noise– Suitable for registration for resolution improvement– Predictive method less expensive than phase

Detailed Properties of Predictive Method

Two components to the errors– Systematic component– Random component

Random error

0.00

0.01

0.02

0.03

0.04

0.05

0.06

0.00 0.20 0.40 0.60 0.80 1.00

Offset

Err

or

sta

nd

ard

de

via

tio

n

Systematic bias

-0.08

-0.06

-0.04

-0.02

0.00

0.02

0.04

0.06

0.08

0.00 0.20 0.40 0.60 0.80 1.00

Offset

Err

or

me

an

A

B

C

D

E

A

B

C

D

E

Effect of Match Window Size

Systematic bias– No significant change above 20 x 20 pixels– Bias is therefore inherent in the method used

Random component– Depends significantly on window size for small windows– When random component has approximately same

magnitude as the bias, it stops improving– No further improvement above about 100 x 100 pixels– Above this, the error has distinct double peak– Random component is therefore limited by systematic bias

Summary of Predictive Registration

Very fast– Requires only a single pass through the image– (2 passes if not already registered to nearest pixel)

Accuracy is 4 - 5% of pixel– This is between pairs of images– May be improved by enforcing consistency

Optimum match window size is about 100 x 100 pixels Robust to moderate levels of noise Relatively insensitive to aliasing Suitable for images captured in identical conditions

– Sensitive to contrast and brightness

Overall Summary

Described resolution improvement– Reconstruction steps: registration, resampling, inverse filtering – Examined some of the preconditions: aliasing– Discussed some of the limitations: lens and camera blurring

Presented results from 1D– Super-resolution of bar code images

Presented results from 2D registration– Comparison of registration methods– Description of a new fast method

• Predictive Interpolation method

References

“Super-resolution of bar-codes”, D.G. Bailey, Journal of Electronic Imaging, vol 10 (1), pp 213-221 (2001)

“Predictive Interpolation for Registration”, D.G. Bailey, Proceedings of Image and Vision Computing Conference NZ, pp 240-245 (November 2000)

“Image Registration Methods for Resolution Improvement”, D.G. Bailey and T.H. Lill, Proceedings of Image and Vision Computing NZ, pp 91-96 (August 1999)

“Super-Resolution of Bar Codes”, D.G. Bailey, SPIE Proceedings, Vol 3521 Machine Vision Systems for Inspection and Metrology VII, Boston, pp 204-213 (November 1998)