introduction to image processing and computer vision...

Ivo Ihrke / Winter 2013

Introduction to Image Processing and Computer Vision

-- Panoramas and Blending --

Winter 2013/14

Ivo Ihrke


Panoramas


Mosaics and Panoramas

- Outline

- Perspective Panoramas

- Hardware-Based

- Software-Based (Multiple Photographs)

- Image registration

- Image blending


Why Mosaic?

Are you getting the whole picture?

– Compact Camera FOV = 50 x 35°

Slide from Brown & Lowe


Why Mosaic?



– Human FOV = 200 x 135°



Why Mosaic?



– Human FOV = 200 x 135°

– Panoramic Mosaic = 360 x 180°



Single vs. Multiple Viewpoint

Single-viewpoint

– Necessary for creating pure perspective images.

– Many vision algorithms assume pinhole cameras.

– Images that aren’t perspective images look distorted.


In the old days of film photography

• Single-viewpoint

• Single exposure

• Standard 35mm film


Omnidirectional Catadioptric Cameras

O-360 EyeSee360

catadioptric = mirror + lens system


images: CAVE lab

Images of an Omnidirectional Camera


Catadioptric System – Full Texture

K

[Kuthirummal 2006]


Cata-Fisheye Camera

[Krishnan and Nayhar 2008]


Catadioptric System – Stereo

objectcenter

object’


Multi-camera, Single-viewpoint ?

PointGrey LadybugImmersive Media “Dodeca2000”


Lens image circle “scanning”

Medium format lens

(6 cm x 6 cm

image circle)

Manual Scan plate

APS-C

(2.4 cm x 1.6 cm)

camera


Perspective Panoramas

Registration


Single Center of Projection

– Take a sequence of images from the same position

─ Rotate the camera about its optical center

– Compute transformation between second image and first

– Transform the second image to overlap with the first

– Blend the two together to create a mosaic

– If there are more images, repeat

…why don’t we need the 3D geometry?

Ivo Ihrke / Winter 2013mosaic PP

Image Reprojection

– The images are reprojected onto a common plane

– The mosaic is formed on this plane

– Mosaic is a synthetic wide-angle camera


A pencil of rays contains all views

real

camerasynthetic

camera

Can generate any synthetic camera view

as long as it has the same center of projection!

scene


No-parallax point

Same center of projection can be ensured by rotating

camera-lens setup around the entrance pupil (and NOT the nodal point!).


Image reprojection

How to relate two images from the samecamera center?

Images contain the same informationalong the same ray.

Use 2D image warp


Taxonomy of Projective Transformations


Perspective Transformation

3D to 2D projection

– Point in world coordinates P(xe ,ye ,ze)

– Distance center of projection – image plane D(=f)

– Image coordinates (xs ,ys)


Homogeneous Coordinates: Point Representation

(x’,y’)

x=(x,y,w)

w=1

y' =y

wx' =

x

w


Homogeneous Coordinates: Point Representation


Hom. Coord.: Line Representation

ax+by+c=0

l=(a,b,c)


Hom. Coordinates: Point on Line

0 lx

l=(a,b,c)

x=(x,y,w)


Hom. Coordinates: Intersection of Lines

xll '

ll’

x


Hom. Coordinates: Line through 2 Points

l=(a,b,c)

x=(x,y,w)

x’=(x’,y’,w’)

lxx '


Embedding of R2 into P2

– For the time being

– Representation of transformations by 3x3 matrices

– Mathematical trick

─ convenient representation to express rotations and translationsas matrix multiplications

─ Easy to find line through points, point-line/line-line intersections

– Easy representation of projective transformation (homography)

Homogeneous Coordinates for 2D

WY

WX

W

Y

X

y

x

y

xR

/

/ and ,P

1

22


Projective Transformations

Projecting one plane onto another using one projection center


Examples of Projective Transformations

• Projection between 2 images via a world plane

Concatenating two projective transforms gives another projective transform

• Projection between 2 images with the same camera center

Rotating camera or camera with varying focal length

• Shadow projection of a plane onto another plane


Taxonomy of Projective Transformations


Distortions under Central Projection

• Similarity: circle remains circle, square remains square

line orientation is preserved

• Affine: circle becomes ellipse, square becomes rhombus

parallel lines remain parallel

• Projective: imaged object size depends on distance from camera

parallel lines converge


Removing Projective Distortion

Projective transformation in inhomogeneous form

4 general point correspondences (x,y ->x’,y’) on the planar facade

lead to eight linear equations of the type

Sufficient to solve for H up to multiplicative factor


The Direct Linear Transform (DLT) Algorithm

Given: 4 2D point correspondences

Objective: estimate the projective transform matrix H

3

2

1

x

x

x

ix

'

'

'

'

3

2

1

x

x

x

ix


The DLT Algorithm II

Re-phrasing H

Re-ording into h vector

gives

0

Estimating matrix H from point correspondences is equivalent to

i

i

i

w

y

x

ix

'

'

'

'

i

i

i

w

y

x

ix

i

i

i

w

y

x

hhh

hhh

hhh

333231

232221

131211


The DLT Algorithm III

Only rows 1 and 2 are linearly independent omit row 3

Inhomogeneous solution: set one matrix entry equal to 1 (e.g. h33)

Solve by Gaussian elimination or least-squares techniques

i

i

i

w

y

x

ix

'

'

'

'

i

i

i

w

y

x

ix


Estimating Homographies


Homography or not ?

• Coincidences between 3D points

at different depths are preserved

• Pure camera rotation

about camera center

2D Homography

• Different depths are imaged to

different image positions

• Camera rotates and translates

Motion Parallax, no Homography


Panoramic Mosaicing

Rotation about camera center: homography

• choose one image as reference

• compute homography to map neighboring

image to reference image plane

• projectively warp image,

add to reference plane

• repeat for all images

bow tie shape


Alternative Panoramas

Project images onto different

surfaces:

Images www.panoguide.com

Cylindrical

Spherical

Cubic (think of cube map)


Example – My former office

Register left and right image to the middle one using two homographies



all images registered to the central one (2 homographies)



seams

ghost


Image Blending


« Moyenne » entre deux images

Pas la moyenne de l’image des objets…

…mais une image de la moyenne des objets

et une moyenne évoluant au cours du temps.

Comment savoir ce qu’est la bonne moyenne ?

On n’en sait rien !

Mais les artistes peuvent nous aider

49

Linear Blending


Interpolation de l’image complète

It = (1-t) * I1 + t * I2

Mais que se passe-t-il si les images ne sont pas alignées ?

50

Fondu « cross-dissolve »


Aligner puis faire le fondu


Image Blending

slides from Alexei Efros


Feathering

01

01

+

Blending

=


Effect of Window Size

0

1 left

right

0

1


Effect of Window Size

0

1

0

1


Good Window Size

0

1

“Optimal” Window: smooth but not ghosted


What is the Optimal Window?

To avoid seams– window >= size of largest prominent feature

To avoid ghosting– window <= 2*size of smallest prominent feature

Natural to cast this in the Fourier domain– largest frequency <= 2*size of smallest frequency

– do blending in different frequency bands


Bandpass Computations


Bandpass Computations

filtered images

Fourier space filter shape

low-pass 1st octave 2nd octave 3rd octave

Octave = doubling frequency


Lowpass


First Octave


Second Octave


Third Octave


Zoom-In 3rd Octave - Jpeg-Artifacts


Reconstruction

filtered images

Fourier space filter shape

low-pass 1st octave 2nd octave 3rd octave

+ + + =

+ + + =

full freq. range

originallow-pass 1st octave 2nd octave 3rd octave


Spatial Domain Interpretation /Implementation

original


What does blurring take away?

smoothed (5x5 Gaussian)


High-Pass Filter

smoothed minus original


Image Pyramids

mipmap or precursor of wavelets – Gaussian Pyramid


Create by Image Sub-sampling

Throw away every other row and

column to create a 1/2 size image

1/4

1/8


Improper Image Sub-sampling

1/4 (2x zoom) 1/8 (4x zoom)

Why does this look so bad?

1/2

Aliasing!


Proper Sub-Sampling

First, band-limit, then sub-sample !

Repeat

– Filter

– Subsample

Until minimum resolution reached

Whole pyramid is only 4/3 the size of the original image!

filter mask


Implementation by Gaussian pre-filtering

G 1/4

G 1/8

Gaussian 1/2

Filter size should double for each ½ size reduction.


Subsampling with Gaussian pre-filtering

G 1/4 G 1/8Gaussian 1/2Solution: filter the image, then subsample

Filter size should double for each ½ size reduction.


Compare with...

1/4 (2x zoom) 1/8 (4x zoom)1/2


Band-pass filtering

Laplacian Pyramid (subband images) Created from Gaussian pyramid by subtraction

Gaussian Pyramid (low-pass images)


Laplacian Pyramid

How can we reconstruct (collapse) this pyramid into the original image?

Need this!Original image


Laplacian Pyramid

Need this!


Pyramid Blending

0

1

0

1

0

1

Left pyramid Right pyramidblend


Blending Apples and Oranges

original apple original orange

blend scale 1 blend scale 2 blend scale 3 pyramid blending



blend scale 1 pyramid blending


Different Frequency Bands


Simplification: Two-band Blending

Brown & Lowe, 2003

– Only use two bands: high freq. and low freq.

– Blends low freq. smoothly

– Blend high freq. with no smoothing: use binary mask


Low frequency (l > 2 pixels)

High frequency (l < 2 pixels)

2-band Blending


Linear Blending


2-band Blending


Still Some Artifacts Left…

Ghosting—objects move in the scene.

Differing exposures between images.

– Pyramid blending does not solve this.


De-Ghosting

In regions with differences don’tblend - crop.

[Uyttendaele et al. 2001]


Gradient Domain Blending

In Pyramid Blending, we decomposed our image into 2nd derivatives (Laplacian) and a low-res image

Let us now look at 1st derivatives (gradients):

No need for low-res image

– captures everything (up to a constant)

– easy to deal with low-frequency differences

Idea:

– Differentiate

– Blend

– Reintegrate


Poisson Image Editing

original mask

Poisson Inpainting result


Poisson Image Editing

original

copy and paste

original to paste

Poisson Image Editing result


Gradient Domain Blending (2D)

– Take partial derivatives dx and dy (the gradient field)

– Fiddle around with them (copy, smooth, blend, feather, etc)

– Reintegrate

─ But now integral(dx) might not equal integral(dy)

– Find the most agreeable solution

─ Equivalent to solving Poisson equation


Gradient Domain Blending (2D)

- But now integral(dx) might not equal integral(dy):

INCONSISTENCY

- There is no UNIQUE SOLUTON!

- Poisson-solver (most widely used) can produce artifacts.

This is how it looks like when we directly integrate an inconsistent gradient field (row-by-row in this case)

+10 ¹ +70


Comparisons [Levin et al 2004]


End


Acknowledgements

Many slides by Steve Seitz, Rick Szeliski

– http://szeliski.org/book

Histogram slides by Samir H. Abdul-Jauwad

Some histogram-matching results by Paul Bourke

More slides by Pierre Bénard, HendrikLensch

http://szeliski.org/book


The End

introduction to image processing and computer vision...

Documents