introduction to image processing and computer vision...

Ivo Ihrke / Winter 2013

Introduction to Image Processing and Computer Vision

-- Panoramas and Blending --

Winter 2013/14

Ivo Ihrke

Panoramas

Mosaics and Panoramas

- Outline

- Perspective Panoramas

- Hardware-Based

- Software-Based (Multiple Photographs)

- Image registration

- Image blending

Why Mosaic?

Are you getting the whole picture?

– Compact Camera FOV = 50 x 35°

Slide from Brown & Lowe

Why Mosaic?

– Human FOV = 200 x 135°

Why Mosaic?

– Human FOV = 200 x 135°

– Panoramic Mosaic = 360 x 180°

Single vs. Multiple Viewpoint

Single-viewpoint

– Necessary for creating pure perspective images.

– Many vision algorithms assume pinhole cameras.

– Images that aren’t perspective images look distorted.

In the old days of film photography

• Single-viewpoint

• Single exposure

• Standard 35mm film

Omnidirectional Catadioptric Cameras

O-360 EyeSee360

catadioptric = mirror + lens system

images: CAVE lab

Images of an Omnidirectional Camera

Catadioptric System – Full Texture

[Kuthirummal 2006]

Cata-Fisheye Camera

[Krishnan and Nayhar 2008]

Catadioptric System – Stereo

objectcenter

object’

Multi-camera, Single-viewpoint ?

PointGrey LadybugImmersive Media “Dodeca2000”

Lens image circle “scanning”

Medium format lens

(6 cm x 6 cm

image circle)

Manual Scan plate

(2.4 cm x 1.6 cm)

camera

Perspective Panoramas

Registration

Single Center of Projection

– Take a sequence of images from the same position

─ Rotate the camera about its optical center

– Compute transformation between second image and first

– Transform the second image to overlap with the first

– Blend the two together to create a mosaic

– If there are more images, repeat

…why don’t we need the 3D geometry?

Ivo Ihrke / Winter 2013mosaic PP

Image Reprojection

– The images are reprojected onto a common plane

– The mosaic is formed on this plane

– Mosaic is a synthetic wide-angle camera

A pencil of rays contains all views

camerasynthetic

camera

Can generate any synthetic camera view

as long as it has the same center of projection!

No-parallax point

Same center of projection can be ensured by rotating

camera-lens setup around the entrance pupil (and NOT the nodal point!).

Image reprojection

How to relate two images from the samecamera center?

Images contain the same informationalong the same ray.

Use 2D image warp

Taxonomy of Projective Transformations

Perspective Transformation

3D to 2D projection

– Point in world coordinates P(xe ,ye ,ze)

– Distance center of projection – image plane D(=f)

– Image coordinates (xs ,ys)

Homogeneous Coordinates: Point Representation

(x’,y’)

x=(x,y,w)

Homogeneous Coordinates: Point Representation

Hom. Coord.: Line Representation

ax+by+c=0

l=(a,b,c)

Hom. Coordinates: Point on Line

l=(a,b,c)

x=(x,y,w)

Hom. Coordinates: Intersection of Lines

Hom. Coordinates: Line through 2 Points

l=(a,b,c)

x=(x,y,w)

x’=(x’,y’,w’)

Embedding of R2 into P2

– For the time being

– Representation of transformations by 3x3 matrices

– Mathematical trick

─ convenient representation to express rotations and translationsas matrix multiplications

─ Easy to find line through points, point-line/line-line intersections

– Easy representation of projective transformation (homography)

Homogeneous Coordinates for 2D

/ and ,P

Projective Transformations

Projecting one plane onto another using one projection center

Examples of Projective Transformations

• Projection between 2 images via a world plane

Concatenating two projective transforms gives another projective transform

• Projection between 2 images with the same camera center

Rotating camera or camera with varying focal length

• Shadow projection of a plane onto another plane

Distortions under Central Projection

• Similarity: circle remains circle, square remains square

line orientation is preserved

• Affine: circle becomes ellipse, square becomes rhombus

parallel lines remain parallel

• Projective: imaged object size depends on distance from camera

parallel lines converge

Removing Projective Distortion

Projective transformation in inhomogeneous form

4 general point correspondences (x,y ->x’,y’) on the planar facade

lead to eight linear equations of the type

Sufficient to solve for H up to multiplicative factor

The Direct Linear Transform (DLT) Algorithm

Given: 4 2D point correspondences

Objective: estimate the projective transform matrix H

The DLT Algorithm II

Re-phrasing H

Re-ording into h vector

Estimating matrix H from point correspondences is equivalent to

333231

232221

131211

The DLT Algorithm III

Only rows 1 and 2 are linearly independent omit row 3

Inhomogeneous solution: set one matrix entry equal to 1 (e.g. h33)

Solve by Gaussian elimination or least-squares techniques

Estimating Homographies

Homography or not ?

• Coincidences between 3D points

at different depths are preserved

• Pure camera rotation

about camera center

2D Homography

• Different depths are imaged to

different image positions

• Camera rotates and translates

Motion Parallax, no Homography

Panoramic Mosaicing

Rotation about camera center: homography

• choose one image as reference

• compute homography to map neighboring

image to reference image plane

• projectively warp image,

add to reference plane

• repeat for all images

bow tie shape

Alternative Panoramas

Project images onto different

surfaces:

Images www.panoguide.com

Cylindrical

Spherical

Cubic (think of cube map)

Example – My former office

Register left and right image to the middle one using two homographies

all images registered to the central one (2 homographies)

Image Blending

« Moyenne » entre deux images

Pas la moyenne de l’image des objets…

…mais une image de la moyenne des objets

et une moyenne évoluant au cours du temps.

Comment savoir ce qu’est la bonne moyenne ?

On n’en sait rien !

Mais les artistes peuvent nous aider

Linear Blending

Interpolation de l’image complète

It = (1-t) * I1 + t * I2

Mais que se passe-t-il si les images ne sont pas alignées ?

Fondu « cross-dissolve »

Aligner puis faire le fondu

Image Blending

slides from Alexei Efros

Feathering

Blending

Effect of Window Size

1 left

Effect of Window Size

Good Window Size

“Optimal” Window: smooth but not ghosted

What is the Optimal Window?

To avoid seams– window >= size of largest prominent feature

To avoid ghosting– window <= 2*size of smallest prominent feature

Natural to cast this in the Fourier domain– largest frequency <= 2*size of smallest frequency

– do blending in different frequency bands

Bandpass Computations

filtered images

Fourier space filter shape

low-pass 1st octave 2nd octave 3rd octave

Octave = doubling frequency

Lowpass

First Octave

Second Octave

Third Octave

Zoom-In 3rd Octave - Jpeg-Artifacts

Reconstruction

filtered images

Fourier space filter shape

low-pass 1st octave 2nd octave 3rd octave

+ + + =

full freq. range

originallow-pass 1st octave 2nd octave 3rd octave

Spatial Domain Interpretation /Implementation

original

What does blurring take away?

smoothed (5x5 Gaussian)

High-Pass Filter

smoothed minus original

Image Pyramids

mipmap or precursor of wavelets – Gaussian Pyramid

Create by Image Sub-sampling

Throw away every other row and

column to create a 1/2 size image

Improper Image Sub-sampling

1/4 (2x zoom) 1/8 (4x zoom)

Why does this look so bad?

Aliasing!

Proper Sub-Sampling

First, band-limit, then sub-sample !

Repeat

– Filter

– Subsample

Until minimum resolution reached

Whole pyramid is only 4/3 the size of the original image!

filter mask

Implementation by Gaussian pre-filtering

Gaussian 1/2

Filter size should double for each ½ size reduction.

Subsampling with Gaussian pre-filtering

G 1/4 G 1/8Gaussian 1/2Solution: filter the image, then subsample

Filter size should double for each ½ size reduction.

Compare with...

1/4 (2x zoom) 1/8 (4x zoom)1/2

Band-pass filtering

Laplacian Pyramid (subband images) Created from Gaussian pyramid by subtraction

Gaussian Pyramid (low-pass images)

Laplacian Pyramid

How can we reconstruct (collapse) this pyramid into the original image?

Need this!Original image

Laplacian Pyramid

Need this!

Pyramid Blending

Left pyramid Right pyramidblend

Blending Apples and Oranges

original apple original orange

blend scale 1 blend scale 2 blend scale 3 pyramid blending

blend scale 1 pyramid blending

Different Frequency Bands

Simplification: Two-band Blending

Brown & Lowe, 2003

– Only use two bands: high freq. and low freq.

– Blends low freq. smoothly

– Blend high freq. with no smoothing: use binary mask

Low frequency (l > 2 pixels)

High frequency (l < 2 pixels)

2-band Blending

Linear Blending

2-band Blending

Still Some Artifacts Left…

Ghosting—objects move in the scene.

Differing exposures between images.

– Pyramid blending does not solve this.

De-Ghosting

In regions with differences don’tblend - crop.

[Uyttendaele et al. 2001]

Gradient Domain Blending

In Pyramid Blending, we decomposed our image into 2nd derivatives (Laplacian) and a low-res image

Let us now look at 1st derivatives (gradients):

No need for low-res image

– captures everything (up to a constant)

– easy to deal with low-frequency differences

– Differentiate

– Blend

– Reintegrate

Poisson Image Editing

original mask

Poisson Inpainting result

Poisson Image Editing

original

copy and paste

original to paste

Poisson Image Editing result

Gradient Domain Blending (2D)

– Take partial derivatives dx and dy (the gradient field)

– Fiddle around with them (copy, smooth, blend, feather, etc)

– Reintegrate

─ But now integral(dx) might not equal integral(dy)

– Find the most agreeable solution

─ Equivalent to solving Poisson equation

Gradient Domain Blending (2D)

- But now integral(dx) might not equal integral(dy):

INCONSISTENCY

- There is no UNIQUE SOLUTON!

- Poisson-solver (most widely used) can produce artifacts.

This is how it looks like when we directly integrate an inconsistent gradient field (row-by-row in this case)

+10 ¹ +70

Comparisons [Levin et al 2004]

Acknowledgements

Many slides by Steve Seitz, Rick Szeliski

– http://szeliski.org/book

Histogram slides by Samir H. Abdul-Jauwad

Some histogram-matching results by Paul Bourke

More slides by Pierre Bénard, HendrikLensch

The End

introduction to image processing and computer vision...

Documents

image #1 - okgrocers.com · image #1. image #2. image #3....

image mosaicing based on neural networks · image...

computational optical imaging - optique numerique...

computational photography and videography christian theobalt...

computational optical imaging - optique numerique...

mirrors in computer graphics, computer vision and time-of...

fundamentals of image processing -...

computational optical imaging - optique numerique...

lexaria - investor presentation website 2016-02 ·...

eg 2011 | computational plenoptic imaging star | i....

computational optical imaging - optique numerique -- image...

empl tenure years name ft / pt fte type date seniority...

principles of light field imaging: briefly revisiting 25...

eg 2011 | computational plenoptic imaging star | vi. high...

patrick suppes and constance ihrke - stanford university ·...

high tech dr. barbara ihrke may 29, 2003

idesirenoonebutyou.files.wordpress.com · web viewimage...

{image} {image} {image} {image} {image}

computational optical imaging - optique numerique --...

blurtags: spatially varying psf estimation with out-of...