IMAGE MOSAICING

Summer School on Document Image Processing

IMAGE MOSAICING

Summer School on Document Image Processing

Thotreingam Kasar

MEDICAL INTELLIGENCE AND LANGUAGE ENGINEERING LAB,DEPARTMENT OF ELECTRICAL ENGINEERING,

INDIAN INSTITUTE OF SCIENCE, BANGALORE, INDIA – 560 012

Image Mosaicing Image Mosaicing Given multiple images of a scene, with some

degree of overlap, how do we seamlessly blend them into a single composite image?

The basic ingredients Registration Warping Blending

Methods for Image MosaicingMethods for Image Mosaicing

Feature based Method

Generally faster than direct methods More robust against scene movement Robust estimation algorithms

available Suitable for fully automatic mosaicing

The respective features might be hard to detect and/or unstable in time

Direct Method

Preferred for images without many prominent details

Windows of pre-defined size or even the entire images are used for the correspondence estimation

Can handle only translation and small rotation

Sensitive to intensity changes, due to noise, varying illumination, and/or by using different sensors

Projective GeometryProjective Geometry

Set of rays in 3-D space with each point representing a projective point

Planes through the origin are interpreted as lines

Homogeneous Co-ordinates A line in a plane (ax + by + c = 0) can be represented as (a, b, c)T

ax + by + c = 0 (ka)x + (kb)y + kc = 0 Thus, (a, b, c)T and (ka, kb, kc)T are homogeneous vectors.

A point X=(x,y)T lies on the line (a, b, c)T if ax + by + c = 0 or, (x, y,1)T · (a, b, c)T = 0

Thus, the point (x,y)T in R2 is represented as a 3-vector by adding a final co-ordinate of 1.

Any arbitrary homogeneous point P=(x1, x2, x3)T represents the point (x1/x3, x2/x3) in R2

} are the same lines for any k ≠ 0

2-D Transformations2-D Transformations

Translation

Rotation

Scaling

y

x

y

x

cossin

sincos

'

'

y

x

s

s

y

x

y

x

0

0

'

'

We can combine all the multiplicative and translation terms for 2-D geometric transformations into a single 3x3 matrix using Homogeneous Coordinates

y

x

t

t

y

x

y

x

'

'

Geometric TransformationsGeometric Transformations

100

cossin

sincos

y

x

t

t

RIGID

AFFINE PERSPECTIVE

100232221

131211

mmm

mmm

333231

232221

131211

hhh

hhh

hhh

IMAGING GEOMETRYIMAGING GEOMETRY

For a pair of matching points (x, y) and (x’, y’) in the world and image plane respectively, the projective transformation in inhomogeneous form is

4 point correspondences lead to 8 such linear equations which can be solved up to an insignificant multiplicative factor.

HXX'

,

3

2

1

333231

232221

131211

'3

'2

'1

or

x

x

x

hhh

hhh

hhh

x

x

x

232221333231

131211333231

333231

232221'3

'2

333231

131211'3

'1

)('

)(',

''

hyhxhhyhxhy

hyhxhhyhxhxor

hyhxh

hyhxh

x

xy

hyhxh

hyhxh

x

xx

The general projective transformation of one projective plane to another is represented as

yO

X’ X

xy’ x’

Image 1 Image2

Feature DetectionFeature Detection

varianceoffunctionGaussianais

),(),(),(

),(),(),(

,))(()det(),( 2

2

2

G

y

yxI

y

yxI

x

yxI

y

yxI

x

yxI

x

yxI

GwhereTracekyx

MMMR

Harris corner is widely used to localize interest points

Local maxima of the Response function R gives the corners

Small intensity change

Large intensity change in one direction

Large intensity change in both directions

Corner DetectionCorner Detection

Feature MatchingFeature Matching

RANdom SAmple Concensus (M. A. Fischler and R. C. Bolles,1981)RANdom SAmple Concensus (M. A. Fischler and R. C. Bolles,1981)

RANSACRANSAC

sw

p

)1log(

)1log(N

1. Randomly select 4 point matches

2. Estimate homography Hi

3. Count the consensus set Si

4. If |Si|> T, return Hi

5. Repeat N times and return the model with max|Si|

p = P(at least 1 of the random samples is free from outliers)

w = P(any selected data point is an inlier)

e = (1-w) is the probability that it is an outlier

At least N selections are required where (1-w4)N = 1-p

RANSACRANSAC

%age of outliers #Trials required

5 3

10 5

20 9

30 17

40 34

50 72

For a sample size of 4, the values of N required to ensure with a probability p =0.99 that at least 1 of the sample is free from outliers

Feature CorrespondenceFeature Correspondence

Target Image 1 Target Image 2Reference Image

Registered Image 1 Registered Image 2

BLENDINGBLENDING Simple Averaging

Feathering - A weighting function is associated with each image decaying from a maximum at the centre to zero at the image boundary

NjMiy

yj

x

xiji

1,1)(

1)(

1),(2

0

0

2

0

0B

Where M and N represents the dimensions of the image and (x0, y0) is the image center

Target image 1 Reference Image Target image 2

Mosaic output

Mosaic ConstructionMosaic Construction

Why Mosaicing?Why Mosaicing?

Satellite image Analysis

Medical image analysis

3-D Scene reconstruction and Robotic Navigation

Creating Super-resolution images

Video representation and indexing

To enhance the limited field of view of camera

Some Applications

DiscussionsDiscussions

Stable Feature Localization

Invariant Feature Extraction

Sub-pixel registration

Blending

THANK YOUTHANK YOU