optical flow digital photography cse558, spring 2003 richard szeliski (notes cribbed from p....
Post on 22-Dec-2015
219 Views
Preview:
TRANSCRIPT
Optical Flow
Digital PhotographyCSE558, Spring 2003
Richard Szeliski
(notes cribbed from P. Anandan)
3/9/2003 Optical Flow 2
Classes of Techniques
Feature-based methods• Extract salient visual features (corners, textured areas) and track
them over multiple frames• Analyze the global pattern of motion vectors of these features• Sparse motion fields, but possibly robust tracking• Suitable especially when image motion is large (10-s of pixels)
Direct-methods• Directly recover image motion from spatio-temporal image
brightness variations• Global motion parameters directly recovered without an
intermediate feature motion calculation• Dense motion fields, but more sensitive to appearance variations• Suitable for video and when image motion is small (< 10 pixels)
3/9/2003 Optical Flow 3
Brightness Constancy Equation:
The Brightness Constraint
),(),( ),(),( yxyx vyuxIyxJ
Or, better still, Minimize :2)),(),((),( vyuxIyxJvuE
),(),(),(),(),(),( yxvyxIyxuyxIyxIyxJ yx Linearizing (assuming small (u,v)):
3/9/2003 Optical Flow 6
Patch Translation [Lucas-Kanade]
yx
tyx IvyxIuyxIvuE,
2),(),(),(
Minimizing
Assume a single velocity for all pixels within an image patch
ty
tx
yyx
yxx
II
II
v
u
III
III2
2
tT IIUII
LHS: sum of the 2x2 outer product tensor of the gradient vector
3/9/2003 Optical Flow 7
The Aperture Problem
TIIMLet
• Algorithm: At each pixel compute by solving
• M is singular if all gradient vectors point in the same direction• e.g., along an edge• of course, trivially singular if the summation is over a single pixel or there is no texture• i.e., only normal flow is available (aperture problem)
• Corners and textured areas are OK
and
ty
tx
II
IIb
U bMU
3/9/2003 Optical Flow 10
Iterative Refinement
Estimate velocity at each pixel using one iteration of Lucas and Kanade estimation
Warp one image toward the other using the estimated flow field(easier said than done)
Refine estimate by repeating the process
3/9/2003 Optical Flow 12
Limits of the gradient method
Fails when intensity structure in window is poor
Fails when the displacement is large (typical operating range is motion of 1 pixel)
Linearization of brightness is suitable only for small displacements
Also, brightness is not strictly constant in imagesactually less problematic than it appears, since we can
pre-filter images to make them look similar
3/9/2003 Optical Flow 13
Pyramids
Pyramids were introduced as a multi-resolution image computation paradigm in the early 80s.
The most popular pyramid is the Burt pyramid, which foreshadows wavelets
Two kinds of pyramids:
Low pass or “Gaussian pyramid”
Band-pass or “Laplacian pyramid”
3/9/2003 Optical Flow 14
Gaussian Pyramid
Convolve image with a small Gaussian kernel• Typically 5x5
Subsample (decimate by 2) to get lower resolution imageRepeat for more levelsA sequence of low-pass filtered images
)2,2(),(
),(),(),(
),(),(
^
1
^
0
yxGyxG
yxgyxGyxG
yxIyxG
ll
ll
3/9/2003 Optical Flow 15
Laplacian pyramid
Laplacian as Difference of Gaussian
Band-pass filtered images
Highlights edges at different spatial scales
For matching, this is less sensitive to image illumination changes
But more noisy than using Gaussians
),(),(),(^
yxGyxGyxL lll
3/9/2003 Optical Flow 17
image Iimage J
aJwwarp refine
a
aΔ+
Pyramid of image J Pyramid of image I
image Iimage J
Coarse-to-Fine Estimation
u=10 pixels
u=5 pixels
u=2.5 pixels
u=1.25 pixels
3/9/2003 Optical Flow 18
J Jw Iwarp refine
ina
a
+
J Jw Iwarp refine
a
a+
J
pyramid construction
J Jw Iwarp refine
a+
I
pyramid construction
outa
Coarse-to-Fine Estimation
3/9/2003 Optical Flow 19
Global Motion Models
2D Models:AffineQuadraticPlanar projective transform (Homography)
3D Models:Instantaneous camera motion models Homography+epipolePlane+Parallax
3/9/2003 Optical Flow 20
0)()( 654321 tyx IyaxaaIyaxaaI
Example: Affine Motion
Substituting into the B.C. Equation:yaxaayxv
yaxaayxu
654
321
),(
),(
Each pixel provides 1 linear constraint in 6 global unknowns
0 tyx IvIuI
2 tyx IyaxaaIyaxaaIaErr )()()( 654321
Least Square Minimization (over all pixels):
3/9/2003 Optical Flow 21
Quadratic – instantaneous approximation to planar motion
Other 2D Motion Models
287654
82
7321
yqxyqyqxqqv
xyqxqyqxqqu
yyvxxu
yhxhh
yhxhhy
yhxhh
yhxhhx
','
and
'
'
987
654
987
321
Projective – exact planar motion
3/9/2003 Optical Flow 23
Correlation and SSD
For larger displacements, do template matching• Define a small area around a pixel as the template• Match the template against each pixel within a
search area in next image.• Use a match measure such as correlation,
normalized correlation, or sum-of-squares difference
• Choose the maximum (or minimum) as the match• Sub-pixel interpolation also possible
3/9/2003 Optical Flow 27
Discrete Search vs. Gradient Based Estimation
Consider image I translated by
21
,00
2
,1
)),(),(),((
)),(),((),(
yxvvyuuxIyxI
vyuxIyxIvuE
yx
yx
00 ,vu
),(),(),(
),(),(
1001
0
yxyxIvyuxI
yxIyxI
The discrete search method simply searches for the best estimate.The gradient method linearizes the intensity function and solves for the estimate
3/9/2003 Optical Flow 33
Correlation Window Size
Small windows lead to more false matchesLarge windows are better this way, but…
• Neighboring flow vectors will be more correlated (since the template windows have more in common)
• Flow resolution also lower (same reason)• More expensive to compute
Another way to look at this:Small windows are good for local search but more precise
and less smoothLarge windows good for global search but less precise and
more smooth method
3/9/2003 Optical Flow 34
Robust Estimation
Standard Least Squares Estimation allows too much influence for outlying points
)()
)()(
)()(
2
mxx
x
mxx
xmE
i
ii
ii
( Influence
3/9/2003 Optical Flow 35
Robust Estimation
tsysxssd IvIuIvuE ),( Robust gradient constraint
),(),(),( ssssd vyuxJyxIvuE Robust SSD
3/9/2003 Optical Flow 36
References
J. R. Bergen, P. Anandan, K. J. Hanna, and R. Hingorani. Hierarchical model-based motion estimation. In ECCV’92, pp. 237–252, Italy, May 1992.
M. J. Black and P. Anandan. The robust estimation of multiple motions: Parametric and piecewise-smooth flow fields. Comp. Vis. Image Understanding, 63(1):75–104, 1996.
H. S. Sawhney and S. Ayer. Compact representation of videos through dominant multiple motion estimation. IEEE Trans. Patt. Anal. Mach. Intel., 18(8):814–830, Aug. 1996.
Y. Weiss. Smoothness in layers: Motion segmentation using nonparametric mixture estimation. In CVPR’97, pp. 520–526, June 1997.
top related