applications of image motion estimation i mosaicing
DESCRIPTION
Applications of Image Motion Estimation I Mosaicing. Princeton University COS 429 Lecture Feb. 26, 2004. Harpreet S. Sawhney [email protected]. Visual Motion Estimation : Recapitulation. Plan. Explain optical flow equations Show inclusion of multiple constraints for solution - PowerPoint PPT PresentationTRANSCRIPT
Applications of Image Motion Estimation I
Mosaicing
Harpreet S. [email protected]
Princeton UniversityCOS 429 Lecture
Feb. 26, 2004
Visual Motion Estimation : Recapitulation
Plan
• Explain optical flow equations
• Show inclusion of multiple constraints for solution
• Another way to solve is to use global parametric models
Brightness Constancy Assumption
)p(I1 )p(I2
p)p(up
Model image transformation as :
)p(I))p(up(I)p(I 112 ′== - BrightnessConstancy
ReferenceCoordinate
CorrespondingCoordinate
OpticalFlow
How do we solve for the flow ?
)p(I))p(up(I)p(I 112 ′== -
Use Taylor Series Expansion
)2(O)p(uI)p(I)p(I T112 +∇= -
Image Gradient
Convert constraint into an objective function
∑∈
∇Rp
2T1SSD ))p(I)p(uI()u(E δ+=
)p(I)p(I 12 -
Optical Flow Constraint Equation
)2(O)p(uI)p(I)p(I T112 +∇= -
0)p(I)p(uIT1 ≈+ δ∇
At a Single Pixel
Leads to
0IuIuI ty
yx
x =++
Normal FlowI
I- t
∇
Multiple Constraints in a Region
∑∈
∇Rp
2T1SSD ))p(I)p(uI()u(E δ+=
X
Solution
0u
)u(ESSD =∂
∑∈
∇Rp
2T1SSD ))p(I)p(uI()u(E δ+=
∑∈
∇∇Rp
T1 0))p(I)p(uI(I =+ δ
∑ ∑∈ ∈
∇-∇∇Rp Rp
T1 IIu]II[ δ=
bAu =
Solution
bAu =
]III
III[A
R
2y
Ryx
Ryx
R
2x
= ∑∑
∑∑
δ
δ= ]
II
II[b
Ry
Rx
∑
∑
-
-
Observations:
• A is a sum of outer products of the gradient vector• A is positive semi-definite• A is non-singular if two or more linearly independent gradients are available• Singular value decomposition of A can be used to compute a solution for u
Another way to provide unique solution
Global Parametric Models
∑∈
∇Rp
2T1SSD ))p(I)p(uI()u(E δ+=
• u(p) is described using an affine transformation valid within the whole region R
tp)p(u +Η= ]hh
hh[H
2221
1211
= = ]t
t[t
2
1
T2122211211 ]tthhhh][
10yx00
0100yx[)p(u = βΒ= )p()p(u
∑∈
∇Rp
2T1SSD ))p(I)p(BI()u(E δ+β=
0u
)u(ESSD =∂ ∑ ∑
∈ ∈
∇-∇∇Rp Rp
TT1
T II)p(B)]p(BII)p(B[ δ=β
bA =β
Affine Motion
Good approximation for :– Small motions– Small Camera rotations– Narrow field of view camera– When depth variation in the scene is small
compared to the average depth and small motion
– Affine camera images a planar scene
Affine Motion
• Affine camera: ]Y
X[sp = ]
Y
X[sp
′
′′=′ • 3D Motion: TRPP +=′
xy23
13T22
yT2
xT1
TsZ]r
r[spRs]
TPr
TPr[sp ′+′+′=
+
+′=′
• A 3D Plane: η+βα=η+β+α= p][s
1YXZ
xy23
13T22 Tsp][
s
1]
r
r[spRsp ′+η+βα′+′=′
tHpp-p)p(u +=′=
Two More Ingredients for Success
• Iterative solution through image warping– Linearization of the BCE is valid only when u(p) is small– Warping brings the second image “closer” to the
reference
• Coarse-to-fine motion estimation for estimating a wider range of image displacements– Coarse levels provide a convex function with unique local
minima– Finer levels track the minima for a globally optimum
solution
Image Warping
))p(up(I)p(I 12 -=
• Express u(p) as: )p(u)p(u)p(u )k( δ+=
)p(I))p(up(I w1
)k(1 ≈-
u(p))-p(I)u(p)-)p(up(I)p(I w1
)k(12 δδ= ≈-
Target Image : EmptySource Image : Filled
p)p(up )k(-
Coarse-to-fine Image Alignment : A Primer
p
2
ΘΘ))u(p;(pI(p)I( 21min )
{ R, T, d(p) }{ H, e, k(p) }
{ dx(p), dy(p) }
d(p)
Warper -
Mosaics In Art
…combine individual chips to create a big picture...
Part of the Byzantine mosaic floor that has been preserved in the Church of Multiplication in Tabkha (near the Sea of Galilee).
www.rtlsoft.com/mmmosaic
Image Mosaics
• Chips are images.
• May or may not be captured from known locations of the camera.
OUTPUT IS A SEAMLESS MOSAIC
VIDEOBRUSH IN ACTION
WHAT MAKES MOSAICING POSSIBBLE…the simplest geometry...
Single Center of Projection for all Images
Images
Projection Surfaces
COP
Planar Mosaic Construction
• Align Pairwise: 1:2, 2:3, 3:4, ...• Select a Reference Frame• Align all Images to the Reference Frame• Combine into a Single Mosaic
1 2 3 4
Virtual Camera (Pan)Image Surface - PlaneProjection - Perspective
Key Problem What Is the Mapping From Image Rays to the Mosaic
Coordinates ?
ImagesCOP
P,P'
Rotations/HomographiesPlane Projective Transformations
RPP' cc Rpp '
RKppK ''
RKpKp 1''
pHp'
IMAGE ALIGNMENT IS A BASIC REQUIREMENT
PYRAMID BASED COARSE-TO-FINE ALIGNMENT… a core technology ...
• Coarse levels reduce search.
• Models of image motion reduce modeling complexity.
• Image warping allows model estimation without discrete feature extraction.
• Model parameters are estimated using iterative non-linear optimization.
• Coarse level parameters guide optimization at finer levels.
Assume that at the kth iteration, kΡ, is available
)p(ΡI)p(I)(pI wkww
model the residual transformation between the coordinate systems, wpand p, as:
D]p[Ipw
I(p)D|D
pI(p;0))(pID))(p;(pI 0D
wwwwww T
0D
w
|D
p
1ydxd
d)yd(1xd1ydxd
dyd)xd(1
p
3231
232221
3231
131211
w
2
2
0D
w
yxy1yx000
xyx0001yx|
D
p
D][IΡΡ (k)1)(k
ITERATIVE SOLUTION OF THE ALIGNMENT MODEL
ITERATIVE REWEIGHTED SUM OF SQUARES
• Given a solution )m(Θ at the mth iteration, find Θδ by solving :
k
ii
i
i
l il
l
i
k
i
i
i
θ
rr
r
)r(ρθ
θ
r
θ
r
r
)r(ρ
k
iw
• iw acts as a soft outlier rejecter :
2SS
σ
1
r
)r(ρ
222
2GM
)rσ(
σ2
r
)r(ρ
PROGRESSIVE MODEL COMPLEXITY…combining real-time capture with accurate alignment...
• Provide user feedback by coarsely aligning incoming frames with a low order model– robust matching that covers a wide search range– achieve about 6-8 frames a sec. on a Pentium 200
• Use the coarse alignment parameters to seed the fine alignment– increase model complexity from similarity, to affine, to
projective parameters– coarse-to-fine alignment for wide range of motions and
managing computational complexity
delay
estimateglobal shift
warp
)(tI
)1(ˆ tI
)(td
)(td
I (t 1)
COARSE-TO-FINE ALIGNMENT
VIDEO MOSAIC EXAMPLE
Princeton Chapel Video Sequence54 frames
UNBLENDED CHAPEL MOSAIC
Image Merging withLaplacian Pyramids
Image 1 Image 2
1 2
Combined Seamless Image
VORONOI TESSELATIONS W/ L1 NORM
BLENDED CHAPEL MOSAIC
1D vs. 2D SCANNING
• 1D : The topology of frames is a ribbon or a string.
Frames overlap only with their temporal neighbors.
• 2D : The topology of frames is a 2D graph Frames overlap with neighbors on many sides
(A 300x332 mosaic captured by mosaicing a 1D sequence of 6 frames)
1D vs. 2D SCANNING
The 1D scan scaled by 2 to 600x692 A 2D scanned mosaic of size 600x692
CHOICE OF 1D/2D MANIFOLD
Plane Cylinder Cone
Sphere Arbitrary
THE “DESMILEY” ALGORITHM
• Compute 2D rotation and translation between successive frames
• Compute L by intersecting central lines of each frame• Fill each pixel [l y] in the rectified planar mosaic by
mapping it to the appropriate video frame
2D MOSAICING THROUGH TOPOLOGY INFERENCE&
LOCAL TO GLOBAL ALIGNMENT
… automatic solution to two key problems ...
• Inference of 2D neighborhood relations (topology) between frames
– Input video just provides a temporal 1D ordering of frames
– Need to infer 2D neighborhood relations so that local constraints may be setup between pairs of frames
• Globally consistent alignment and mosaic creation
– Choose appropriate alignment model
– Local constraints incorporated in a global optimization
PROBLEM FORMULATION
Given an arbitrary scan of a scene
Create a globally aligned mosaic by minimizing
Gij i
iij EEE mosaic) theofArea (min 2
}{
iP
Like an MDL measure : Create a compact appearance while being geometrically consistent
ERROR MEASURE
Gij i
iij EEE mosaic) theofArea (min 2
}{
iP
where
ation transformdistortionleast like criterion prioria
for allow toerror term reference to Frame:
relations odneighborho therepresents that Graph :
and neighbors betweenerror alignment of measureAny :
mapping, image-to- Reference:
i
ij
E
G
jiE
XPuP iii
ALGORITHMIC APPROACH
From a 1D ordered collection of framesto
A Globally consistent set of alignment parameters
Iterate through
1. Graph Topology Determination
Given: pose of all framesEstablish neighborhood relations min(Area of Mosaic)
Graph G2. Local Pairwise Alignment
Given: GQuality measure validates hypothesized arcsProvides pairwise constraints
3. Globally Consistent Alignment
Given: pairwise constraintsCompute reference-to-frame pose parameters min ijE
GRAPH TOPOLOGY DETERMINATION
• Given: Current estimate of pose*
• Lay out each frame on the 2D manifold (plane, sphere, etc.)
• Hypothesize new neighbors based on– proximity
– predictability of relative pose
– non redundancy w.r.t. current G
• Specifically, try arc (i,j) ifNormalized Euclidean dist dij << Path distance Dij
• Validate hypothesis by local registration
• Add arc to G if good quality registration
* Initialize using low order frame-to-frame mosaic algorithm on a plane
dij
Dij
LOCAL COARSE & FINE ALIGNMENT
• Given: a frame pair to be registered
• Coarse alignment– Low order parametric model e.g. shift, or 2D R & T
– Majority consensus among subimage estimates
• Fine alignment [Bergen, ECCV 92]– Coarse to fine over Laplacian pyramid
– Progressive model complexity, up to projective
– Incrementally adjust motion parameters to minimize SSD
• Quality measure– Normalized correlation helps reject invalid registrations
GLOBALLY CONSISTENT ALIGNMENT
• Given: arcs ij in graph G of neighbors
• The local alignment parameters, Qij, help establish feature correspondence between i and j
• If uil and ujl are corresponding
points in frames i,j, then
211 |)()(| jljili PP uuEij
• Incrementally adjust poses Pi to minimize
ui uj
Eij
Gij i
iij EEE}{
miniP
SPECIFIC EXAMPLES : 1. PLANAR MOSAICS
• Mosaic to frame transformation model:
• Local Registration– Coarse 2D translation & fine 2D projective alignment
• Topology : Neighborhood graph defined over a plane– Initial graph topology computed with the 2D T estimates– Iterative refinement using arcs based on projective alignment
• Global Alignment
k
ijE2|)()(| jkjiki uAuA
2
1
2|)())()((|k
iE kkkjki AA
Pair Wise Alignment Error
Minimum Distortion Error
XPu i
PLANAR TOPOLOGY EVOLUTION
Whiteboard Video Sequence75 frames
PLANAR TOPOLOGY EVOLUTION
FINAL MOSAIC
SPECIFIC EXAMPLES : 2. SPHERICAL MOSAICS
• Frame to mosaic transformation model:
• Local Registration– Coarse 2D translation & fine 2D projective alignment
• Parameter Initialization– Compute F and R’s from the 2D projective matrices
• Topology : – Initial graph topology computed with the 2D R & T estimates on a plane– Subsequently the topology defined on a sphere– Iterative refinement using arcs based on alignment with F and R’s
• Global Alignment
k
ijE2|| jk
1jik
1i uFRuFR
XFRu iT
SPHERICAL MOSAICS
Sarnoff Library VideoCaptures almost the complete sphere
with 380 frames
SPHERICAL TOPOLOGY EVOLUTION
SPHERICAL MOSAICSarnoff Library
SPHERICAL MOSAICSarnoff Library
NEW SYNTHESIZED VIEWS
FINAL MOSAICPrinceton University Courtyard
Mosaicing from Strips
Mosaicfrom center strips
Image 1 Image 2 Image 3 Image 4 Image 5
Problem: Forward Translation
General Camera Motion
Parallel Flow: Straight Strip
Radial Flow (FOE): Circular Strip
• Strip Perpendicular to Optical Flow• Cut/Paste Strip (warp to make Optical Flow parallel)
Mosaic Construction
Mosaic
Simple Cases
0Max
0)(2
22 Myxb
0
a
v
u
by
bx
v
u
Horizontal Translation
Zoom
(M determines displacement)
(M determines radius)
Manifold for Forward Motion
• Stationary (but rotating) Camera – Viewing Sphere
• Translating Camera– Sphere carves a “Pipe” in space
Pipe Projection
One Image Sequence
Concatenation of Pipes
Forward Motion Mosaicing
Example: Forward Motion
Side View of Mosaic
Forward Mosaicing II
Mosaic Construction
OmniStereo: Stereo in Full 360o Two Panoramas: One for Each Eye
Each panorama can be mapped on a cylinder
Paradigm: A Rotating Stereo Pair of Slit Cameras
•Rays are tangent to viewing circle (Gives 360o stereo)
•Image planes are
radial
(Makes mosaicing difficult)
Panoramic Projections of Slit Cameras
Image Surface
Left Eye panorama (viewing circle)Projection rays tilted right
Right Eye panorama (viewing circle)Projection rays tilted left
Directional Viewpoints
Regular panorama (single viewpoint):Projection rays orthogonal to cylinder
Image Plane
PinholeAperture
(a) (b)Center Strip
(c)Side Strip
Vertical Slit
Slit Camera Model
•Center Strip: Rays perpendicular to image plane
•Side Strip: Rays tilted from image plane
Stereo Panorama from Strips
Left-eye-Panoramafrom right strips
Right-eye-Panoramafrom left strips
Image 1 Image 2 Image 3 Image 4 Image 5
MultiView Panoramas
0
180
360
Regular Panorama
Left-EyePanorama
Right-Eye Panorama
Stereo Panorama from VideoStereo Panorama from Video
Stereo viewing with Red/Blue Glasses
Viewing Panoramic Stereo Printed Cylindrical Surfaces
• Print panorama on a cylinder
• No computation needed!!!