tkitracking • optical flow motion estimationcyy/courses/vfx/10... · a scanner darkly (2006) •...
Post on 12-Aug-2020
4 Views
Preview:
TRANSCRIPT
Motion estimation
Digital Visual EffectsgYung-Yu Chuang
with slides by Michael Black and P. Anandan
Motion estimation
• Parametric motion (image alignment)T ki• Tracking
• Optical flow
Parametric motion
direct method for image stitching
Tracking
Optical flow Three assumptions
• Brightness consistencyS i l h• Spatial coherence
• Temporal persistence
Brightness consistency
Image measurement (e g brightness) in a small region Image measurement (e.g. brightness) in a small region remain the same although their location may change.
Spatial coherence
• Neighboring points in the scene typically belong to the • Neighboring points in the scene typically belong to the same surface and hence typically have similar motions.
• Since they also project to nearby pixels in the image, Since they also project to nearby pixels in the image, we expect spatial coherence in image flow.
Temporal persistence
The image motion of a surface patch changes gradually over time.
Image registration
Goal: register a template image T(x) and an input image I(x) where x (x y)T (warp I so that it image I(x), where x=(x,y)T. (warp I so that it matches T)
Image alignment: I(x) and T(x) are two imagesTracking: T(x) is a small patch around a point p in Tracking: T(x) is a small patch around a point p in
the image at t. I(x) is the image at time t+1. O ti l fl T( ) d I( ) t h f i Optical flow: T(x) and I(x) are patches of images
at t and t+1.warp
T Iwarp
fixed
Simple approach (for translation)
• Minimize brightness difference
2 yx
yxTvyuxIvuE,
2),(),(),(
Simple SSD algorithm
For each offset (u, v) ( )compute E(u,v);
Choose (u, v) which minimizes E(u,v);
Problems:Problems:• Not efficient
N b i l • No sub-pixel accuracy
Lucas-Kanade algorithmLucas Kanade algorithm
Newton’s method
• Root finding for f(x)=0March x and test signs• March x and test signs
• Determine Δx (small→slow; large→ miss)
Newton’s method
• Root finding for f(x)=0
Newton’s method
• Root finding for f(x)=0T l ’ iTaylor’s expansion:
2)(''1)(')()( ffff 20000 )(''
2)(')()( xfxfxfxf
)(')()( xfxfxf )()()( 000 xfxfxf
)(xf)(')(
n
nn xf
xf
)(')(
1n
nn xfxfxx )( nxf
Newton’s method
• Root finding for f(x)=0)(f)(')(
n
nn xf
xf
)( nf
x0x1x2
Newton’s method
pick up x=x0
iiterate
)(xfcompute
)(')(
xfxfx
update x by x+Δxtil until converge
Finding root is useful for optimization because Minimize g(x) → find root for f(x)=g’(x)=0Minimize g(x) find root for f(x) g (x) 0
Lucas-Kanade algorithm
yxTvyuxIvuE 2),(),(),( yx
yy,
)()()(
yx vIuIyxIvyuxI ),(),( yxyy ),(),(
yx vIuIyxTyxI 2),(),( yx
yxyy,
),(),(
yx
yxx vIuIyxTyxIIuE
,),(),(20
yx
yxy vIuIyxTyxIIvE ),(),(20 yxv ,
Lucas-Kanade algorithm
yxx vIuIyxTyxIIE ),(),(20 yxyxx yy
u ,),(),(
vIuIyxTyxIIE )()(20
yx
yxy vIuIyxTyxIIv ,
),(),(20
yx yxxyx
yxx
yxIyxTIvIuII
yxIyxTIvIIuI
2, ,,
2
)()(
),(),(
yx
yyx
yyx
yx yxIyxTIvIuII,,,
),(),(
2
yxx
yxyx
yxx
yxIyxTI
yxIyxTI
vu
III
III,
2,,
2
)()(
),(),(
yx
yyx
yyx
yx yxIyxTIvIII,,,
),(),(
Lucas-Kanade algorithm
iterateshift I(x y) with (u v)shift I(x,y) with (u,v)compute gradient image Ix, Iycompute error image T(x y) I(x y)compute error image T(x,y)-I(x,y)compute Hessian matrixsolve the linear systemsolve the linear system(u,v)=(u,v)+(∆u,∆v)til until converge
xyxx yxIyxTIIII 2 ),(),(
y
yxx
yyx
yxyx
yxx
yxIyxTI
yy
vu
III,
2,,
),(),(
),(),(
yxyxyx ,,,
Parametric model
yxTvyuxIvuE 2),(),(),( yx
yy,
)()()(
xp)W(x;p 2)()()( TIEOur goal is to find p to minimize E(p)
xxp)W(x;p )()()( TIE
dx
p to minimize E(p)for all x in T’s domain
Tyx
y
x ddpdydx
),(,
p)W(x;translation
xxyxx yx
ddd1
dAxp)W(x;affine
T
yyyyxy
ddd,
11
dAxp)W(x;affine
Tyxyyyxxyxx ddddddp ),,,,,(
Parametric model
x
xΔp)pW(x; 2)()( TIminimizex
with respect to Δp
W ΔppWp)W(x;Δp)pW(x;
)()( ΔppWp)W(x;Δp)pW(x;
II
ΔppW
xp)W(x;
II )(
px
xΔpWp)W(x;2
)()( TIIminimize
x
pp
p)( ; )()(
Parametric model
image gradient
warped image target image
W2
image gradient
x
xΔppWp)W(x; )()( TII
Jacobian of the warp
xxxx
pW
pW
pW
pW
W
n
yyy
n
y
pW
pW
pW
ppp
pWp
pW
21
21
npppp 21
Jacobian matrix
• The Jacobian matrix is the matrix of all first-order partial derivatives of a vector-valued functionpartial derivatives of a vector-valued function.
mnF RR :),,( 21 nxxxF
)),,(),,,(),,,(( 21212211 nmnn xxxfxxxfxxxf
ff
or),,( 21 nF xxxJ
nxf
xf
1
1
1
),,( or
21 mfff
mm
n
ff
1
),,( 21 nxxx
n
mm
xx
1
ΔxxxΔxx )()()( FJFF
Jacobian matrix
32,0,0: RR F cossinrt
cossinsin
rvru
),,(),,( vutrF
ttt cosrv
uuu
ttrt
),,(
vvv
uururJ F
vv
rv
cossinsincossinsinsinsincoscoscossin
rrrr
0sincos
cossscosss
r
Parametric model
image gradient
warped image target image
W2
image gradient
x
xΔppWp)W(x; )()( TII
Jacobian of the warp
xxxx
pW
pW
pW
pW
W
n
yyy
n
y
pW
pW
pW
ppp
pWp
pW
21
21
npppp 21
Jacobian of the warp
xxxx WWWW
yyy
n
y
x
WWWppp
Wp
pW
21
For example for affine
npppp 21
For example, for affine
xxyxxxxyxx dydxd
yx
ddd )1(1p)W(x;
yyyyxyyyyx dydxd
yddd )1(
11
p)W(x;
10000100
yxyx
pW
ypdxx dyx dxy dyy dx dy
Parametric model
xΔpWp)W(x;2
)()( TIIminarg
x
xΔpp
p)W(x; )()( TII
ΔW)W(W )()(0 TIII
TΔp
minarg
x
xΔppWp)W(x;
pW )()(0 TIII
T
xp)W(x;x
pWHΔp )()(1 ITI
T
WWT
x p
WpWH II(Approximated) Hessian
Lucas-Kanade algorithm
iterate1) warp I with W(x;p)1) warp I with W(x;p)2) compute error image T(x,y)-I(W(x,p))3) compute gradient image with W(x p)I3) compute gradient image with W(x,p)4) evaluate Jacobian at (x;p)5) compute
pW
WI
I
5) compute 6) compute Hessian7) t
pI
WT
7) compute 8) solve 9) d t b
x
p)W(x;xpW )()( ITI
ΔpΔ9) update p by p+
until converge
Δp
p)W(x;xWHΔp )()(1 ITI
T
xp)W(x;x
pHΔp )()( ITI
xp)W(x;x
pWHΔp )()(1 ITI
T
Coarse-to-fine strategy
J Jw Ifi
inaJ IJ Jw Iwarp refine
a+
J Jw Iwarp refine
a
pyramid construction
pyramid construction
a+
J Jw Iwarp refine
a+
outa
Application of image alignment Direct vs feature-based
• Direct methods use all information and can be very accurate but they depend on the fragile very accurate, but they depend on the fragile “brightness constancy” assumption.
• Iterative approaches require initialization• Iterative approaches require initialization.• Not robust to illumination change and noise
imagesimages.• In early days, direct method is better.
• Feature based methods are now more robust and potentially fasterand potentially faster.
• Even better, it can recognize panorama without initializationinitialization.
TrackingTracking
Tracking
(u, v)I(x,y,t) I(x+u,y+v,t+1)
(u, v)
Tracking
0),,()1,,( tyxItvyuxIbrightness constancy
0),,(),,(),,(),,(),,( tyxItyxItyxvItyxuItyxI tyx
0),,(),,(),,( tyxItyxvItyxuI tyx y
0 IvIuI optical flow constraint equation0 tyx IvIuI optical flow constraint equation
Optical flow constraint equation
Multiple constraints Area-based method• Assume spatial smoothness
Area-based method
• Assume spatial smoothness
yx
tyx IvIuIvuE 2),(yx,
Area-based method
must be invertiblemust be invertible
Area-based method• The eigenvalues tell us about the local image
structurestructure.• They also tell us how well we can estimate the
fl i b th di tiflow in both directions.• Link to Harris corner detector.
Textured area
Edge Homogenous area
KLT tracking
• Select features by M i f b i di i il i
),( 21min• Monitor features by measuring dissimilarity
Aperture problem
Aperture problem Aperture problem
Demo for aperture problem
• http://www.sandlotscience.com/Distortions/Breathing Square htmeathing_Square.htm
• http://www.sandlotscience.com/Ambiguous/Bab l Ill i htrberpole_Illusion.htm
Aperture problem
• Larger window reduces ambiguity, but easily violates spatial smoothness assumptionviolates spatial smoothness assumption
KLT tracking
http://www ces clemson edu/~stb/klt/http://www.ces.clemson.edu/ stb/klt/
KLT tracking
http://www ces clemson edu/~stb/klt/http://www.ces.clemson.edu/ stb/klt/
SIFT tracking (matching actually)
Frame 0 Frame 10
SIFT tracking
Frame 0 Frame 100
SIFT tracking
Frame 0 Frame 200
KLT vs SIFT tracking
• KLT has larger accumulating error; partly because our KLT implementation doesn’t have because our KLT implementation doesn t have affine transformation?SIFT i i i l b t• SIFT is surprisingly robust
• Combination of SIFT and KLT (example)http://www.frc.ri.cmu.edu/projects/buzzard/smalls/
Rotoscoping (Max Fleischer 1914)
1937
Tracking for rotoscoping
Tracking for rotoscoping Waking life (2001)
A Scanner Darkly (2006)
• Rotoshop, a proprietary software. Each minute of animation required 500 hours of workof animation required 500 hours of work.
Optical flowOptical flow
Single-motion assumption
Violated byM i di i i• Motion discontinuity
• Shadows• Transparency• Specular reflection• Specular reflection• …
Multiple motion
Multiple motion Simple problem: fit a line
Least-square fit Least-square fit
Robust statistics
• Recover the best fit for the majority of the datadata
• Detect and reject outliers
Approach
Robust weighting
T t d d tiTruncated quadratic
Robust weighting
G & M ClGeman & McClure
Robust estimation Fragmented occlusion
Regularization and dense optical flow
• Neighboring points in the scene typically belong to the • Neighboring points in the scene typically belong to the same surface and hence typically have similar motions.
• Since they also project to nearby pixels in the image, Since they also project to nearby pixels in the image, we expect spatial coherence in image flow.
Input image Horizontal ti
Vertical motionmotion motion
Application of optical flow
videovideomatching
Input for the NPR algorithm Brushes
Edge clipping Gradient
Smooth gradient Textured brush
Edge clipping Temporal artifacts
Frame-by-frame application of the NPR algorithm
Temporal coherence References• B.D. Lucas and T. Kanade, An Iterative Image Registration Technique with
an Application to Stereo Vision, Proceedings of the 1981 DARPA Image U d di W k h 1981 121 130 Understanding Workshop, 1981, pp121-130.
• Bergen, J. R. and Anandan, P. and Hanna, K. J. and Hingorani, R., Hierarchical Model-Based Motion Estimation, ECCV 1992, pp237-252.
• J. Shi and C. Tomasi, Good Features to Track, CVPR 1994, pp593-600. • Michael Black and P. Anandan, The Robust Estimation of Multiple Motions:
Parametric and Piecewise-Smooth Flow Fields, Computer Vision and Image , p gUnderstanding 1996, pp75-104.
• S. Baker and I. Matthews, Lucas-Kanade 20 Years On: A Unifying Framework, International Journal of Computer Vision, 56(3), 2004, pp221 , p , ( ), , pp- 255.
• Peter Litwinowicz, Processing Images and Video for An Impressionist
Eff t SIGGRAPH 1997Effects, SIGGRAPH 1997.
• Aseem Agarwala, Aaron Hertzman, David Salesin and Steven Seitz, Keyframe-Based Tracking for Rotoscoping and Animation, SIGGRAPH 2004, pp584-591.
top related