Motion estimation

Digital Visual EffectsgYung-Yu Chuang

with slides by Michael Black and P. Anandan

Motion estimation

• Parametric motion (image alignment)T ki• Tracking

• Optical flow

Parametric motion

direct method for image stitching

Tracking

Optical flow Three assumptions

• Brightness consistencyS i l h• Spatial coherence

• Temporal persistence

Brightness consistency

Image measurement (e g brightness) in a small region Image measurement (e.g. brightness) in a small region remain the same although their location may change.

Spatial coherence

• Neighboring points in the scene typically belong to the • Neighboring points in the scene typically belong to the same surface and hence typically have similar motions.

• Since they also project to nearby pixels in the image, Since they also project to nearby pixels in the image, we expect spatial coherence in image flow.

Temporal persistence

The image motion of a surface patch changes gradually over time.

Image registration

Goal: register a template image T(x) and an input image I(x) where x (x y)T (warp I so that it image I(x), where x=(x,y)T. (warp I so that it matches T)

Image alignment: I(x) and T(x) are two imagesTracking: T(x) is a small patch around a point p in Tracking: T(x) is a small patch around a point p in

the image at t. I(x) is the image at time t+1. O ti l fl T( ) d I( ) t h f i Optical flow: T(x) and I(x) are patches of images

at t and t+1.warp

T Iwarp

fixed

Simple approach (for translation)

• Minimize brightness difference

2 yx

yxTvyuxIvuE,

2),(),(),(

Simple SSD algorithm

For each offset (u, v) ( )compute E(u,v);

Choose (u, v) which minimizes E(u,v);

Problems:Problems:• Not efficient

N b i l • No sub-pixel accuracy

Lucas-Kanade algorithmLucas Kanade algorithm

Newton’s method

• Root finding for f(x)=0March x and test signs• March x and test signs

• Determine Δx (small→slow; large→ miss)

Newton’s method

• Root finding for f(x)=0

Newton’s method

• Root finding for f(x)=0T l ’ iTaylor’s expansion:

2)(''1)(')()( ffff 20000 )(''

2)(')()( xfxfxfxf

)(')()( xfxfxf )()()( 000 xfxfxf

)(xf)(')(

n

nn xf

xf

)(')(

1n

nn xfxfxx )( nxf

Newton’s method

• Root finding for f(x)=0)(f)(')(

n

nn xf

xf

)( nf

x0x1x2

Newton’s method

pick up x=x0

iiterate

)(xfcompute

)(')(

xfxfx

update x by x+Δxtil until converge

Finding root is useful for optimization because Minimize g(x) → find root for f(x)=g’(x)=0Minimize g(x) find root for f(x) g (x) 0

Lucas-Kanade algorithm

yxTvyuxIvuE 2),(),(),( yx

yy,

)()()(

yx vIuIyxIvyuxI ),(),( yxyy ),(),(

yx vIuIyxTyxI 2),(),( yx

yxyy,

),(),(

yx

yxx vIuIyxTyxIIuE

,),(),(20

yx

yxy vIuIyxTyxIIvE ),(),(20 yxv ,

Lucas-Kanade algorithm

yxx vIuIyxTyxIIE ),(),(20 yxyxx yy

u ,),(),(

vIuIyxTyxIIE )()(20

yx

yxy vIuIyxTyxIIv ,

),(),(20

yx yxxyx

yxx

yxIyxTIvIuII

yxIyxTIvIIuI

2, ,,

2

)()(

),(),(

yx

yyx

yyx

yx yxIyxTIvIuII,,,

),(),(

2

yxx

yxyx

yxx

yxIyxTI

yxIyxTI

vu

III

III,

2,,

2

)()(

),(),(

yx

yyx

yyx

yx yxIyxTIvIII,,,

),(),(

Lucas-Kanade algorithm

iterateshift I(x y) with (u v)shift I(x,y) with (u,v)compute gradient image Ix, Iycompute error image T(x y) I(x y)compute error image T(x,y)-I(x,y)compute Hessian matrixsolve the linear systemsolve the linear system(u,v)=(u,v)+(∆u,∆v)til until converge

xyxx yxIyxTIIII 2 ),(),(

y

yxx

yyx

yxyx

yxx

yxIyxTI

yy

vu

III,

2,,

),(),(

),(),(

yxyxyx ,,,

Parametric model

yxTvyuxIvuE 2),(),(),( yx

yy,

)()()(

xp)W(x;p 2)()()( TIEOur goal is to find p to minimize E(p)

xxp)W(x;p )()()( TIE

dx

p to minimize E(p)for all x in T’s domain

Tyx

y

x ddpdydx

),(,

p)W(x;translation

xxyxx yx

ddd1

dAxp)W(x;affine

T

yyyyxy

ddd,

11

dAxp)W(x;affine

Tyxyyyxxyxx ddddddp ),,,,,(

Parametric model

x

xΔp)pW(x; 2)()( TIminimizex

with respect to Δp

W ΔppWp)W(x;Δp)pW(x;

)()( ΔppWp)W(x;Δp)pW(x;

II

ΔppW

xp)W(x;

II )(

px

xΔpWp)W(x;2

)()( TIIminimize

x

pp

p)( ; )()(

Parametric model

image gradient

warped image target image

W2

image gradient

x

xΔppWp)W(x; )()( TII

Jacobian of the warp

xxxx

pW

pW

pW

pW

W

n

yyy

n

y

pW

pW

pW

ppp

pWp

pW

21

21

npppp 21

Jacobian matrix

• The Jacobian matrix is the matrix of all first-order partial derivatives of a vector-valued functionpartial derivatives of a vector-valued function.

mnF RR :),,( 21 nxxxF

)),,(),,,(),,,(( 21212211 nmnn xxxfxxxfxxxf

ff

or),,( 21 nF xxxJ

nxf

xf

1

1

1

),,( or

21 mfff

mm

n

ff

1

),,( 21 nxxx

n

mm

xx

1

ΔxxxΔxx )()()( FJFF

Jacobian matrix

32,0,0: RR F cossinrt

cossinsin

rvru

),,(),,( vutrF

ttt cosrv

uuu

ttrt

),,(

vvv

uururJ F

vv

rv

cossinsincossinsinsinsincoscoscossin

rrrr

0sincos

cossscosss

r

Parametric model

image gradient

warped image target image

W2

image gradient

x

xΔppWp)W(x; )()( TII

Jacobian of the warp

xxxx

pW

pW

pW

pW

W

n

yyy

n

y

pW

pW

pW

ppp

pWp

pW

21

21

npppp 21

Jacobian of the warp

xxxx WWWW

yyy

n

y

x

WWWppp

Wp

pW

21

For example for affine

npppp 21

For example, for affine

xxyxxxxyxx dydxd

yx

ddd )1(1p)W(x;

yyyyxyyyyx dydxd

yddd )1(

11

p)W(x;

10000100

yxyx

pW

ypdxx dyx dxy dyy dx dy

Parametric model

xΔpWp)W(x;2

)()( TIIminarg

x

xΔpp

p)W(x; )()( TII

ΔW)W(W )()(0 TIII

TΔp

minarg

x

xΔppWp)W(x;

pW )()(0 TIII

T

xp)W(x;x

pWHΔp )()(1 ITI

T

WWT

x p

WpWH II(Approximated) Hessian

Lucas-Kanade algorithm

iterate1) warp I with W(x;p)1) warp I with W(x;p)2) compute error image T(x,y)-I(W(x,p))3) compute gradient image with W(x p)I3) compute gradient image with W(x,p)4) evaluate Jacobian at (x;p)5) compute

pW

WI

I

5) compute 6) compute Hessian7) t

pI

WT

7) compute 8) solve 9) d t b

x

p)W(x;xpW )()( ITI

ΔpΔ9) update p by p+

until converge

Δp

p)W(x;xWHΔp )()(1 ITI

T

xp)W(x;x

pHΔp )()( ITI

xp)W(x;x

pWHΔp )()(1 ITI

T

Coarse-to-fine strategy

J Jw Ifi

inaJ IJ Jw Iwarp refine

a+

J Jw Iwarp refine

a

pyramid construction

pyramid construction

a+

J Jw Iwarp refine

a+

outa

Application of image alignment Direct vs feature-based

• Direct methods use all information and can be very accurate but they depend on the fragile very accurate, but they depend on the fragile “brightness constancy” assumption.

• Iterative approaches require initialization• Iterative approaches require initialization.• Not robust to illumination change and noise

imagesimages.• In early days, direct method is better.

• Feature based methods are now more robust and potentially fasterand potentially faster.

• Even better, it can recognize panorama without initializationinitialization.

TrackingTracking

Tracking

(u, v)I(x,y,t) I(x+u,y+v,t+1)

(u, v)

Tracking

0),,()1,,( tyxItvyuxIbrightness constancy

0),,(),,(),,(),,(),,( tyxItyxItyxvItyxuItyxI tyx

0),,(),,(),,( tyxItyxvItyxuI tyx y

0 IvIuI optical flow constraint equation0 tyx IvIuI optical flow constraint equation

Optical flow constraint equation

Multiple constraints Area-based method• Assume spatial smoothness

Area-based method

• Assume spatial smoothness

yx

tyx IvIuIvuE 2),(yx,

Area-based method

must be invertiblemust be invertible

Area-based method• The eigenvalues tell us about the local image

structurestructure.• They also tell us how well we can estimate the

fl i b th di tiflow in both directions.• Link to Harris corner detector.

Textured area

Edge Homogenous area

KLT tracking

• Select features by M i f b i di i il i

),( 21min• Monitor features by measuring dissimilarity

Aperture problem

Aperture problem Aperture problem

Demo for aperture problem

• http://www.sandlotscience.com/Distortions/Breathing Square htmeathing_Square.htm

• http://www.sandlotscience.com/Ambiguous/Bab l Ill i htrberpole_Illusion.htm

Aperture problem

• Larger window reduces ambiguity, but easily violates spatial smoothness assumptionviolates spatial smoothness assumption

KLT tracking

http://www ces clemson edu/~stb/klt/http://www.ces.clemson.edu/ stb/klt/

KLT tracking

http://www ces clemson edu/~stb/klt/http://www.ces.clemson.edu/ stb/klt/

SIFT tracking (matching actually)

Frame 0 Frame 10

SIFT tracking

Frame 0 Frame 100

SIFT tracking

Frame 0 Frame 200

KLT vs SIFT tracking

• KLT has larger accumulating error; partly because our KLT implementation doesn’t have because our KLT implementation doesn t have affine transformation?SIFT i i i l b t• SIFT is surprisingly robust

• Combination of SIFT and KLT (example)http://www.frc.ri.cmu.edu/projects/buzzard/smalls/

Rotoscoping (Max Fleischer 1914)

1937

Tracking for rotoscoping

Tracking for rotoscoping Waking life (2001)

A Scanner Darkly (2006)

• Rotoshop, a proprietary software. Each minute of animation required 500 hours of workof animation required 500 hours of work.

Optical flowOptical flow

Single-motion assumption

Violated byM i di i i• Motion discontinuity

• Shadows• Transparency• Specular reflection• Specular reflection• …

Multiple motion

Multiple motion Simple problem: fit a line

Least-square fit Least-square fit

Robust statistics

• Recover the best fit for the majority of the datadata

• Detect and reject outliers

Approach

Robust weighting

T t d d tiTruncated quadratic

Robust weighting

G & M ClGeman & McClure

Robust estimation Fragmented occlusion

Regularization and dense optical flow

• Neighboring points in the scene typically belong to the • Neighboring points in the scene typically belong to the same surface and hence typically have similar motions.

• Since they also project to nearby pixels in the image, Since they also project to nearby pixels in the image, we expect spatial coherence in image flow.

Input image Horizontal ti

Vertical motionmotion motion

Application of optical flow

videovideomatching

Input for the NPR algorithm Brushes

Edge clipping Gradient

Smooth gradient Textured brush

Edge clipping Temporal artifacts

Frame-by-frame application of the NPR algorithm

Temporal coherence References• B.D. Lucas and T. Kanade, An Iterative Image Registration Technique with

an Application to Stereo Vision, Proceedings of the 1981 DARPA Image U d di W k h 1981 121 130 Understanding Workshop, 1981, pp121-130.

• Bergen, J. R. and Anandan, P. and Hanna, K. J. and Hingorani, R., Hierarchical Model-Based Motion Estimation, ECCV 1992, pp237-252.

• J. Shi and C. Tomasi, Good Features to Track, CVPR 1994, pp593-600. • Michael Black and P. Anandan, The Robust Estimation of Multiple Motions:

Parametric and Piecewise-Smooth Flow Fields, Computer Vision and Image , p gUnderstanding 1996, pp75-104.

• S. Baker and I. Matthews, Lucas-Kanade 20 Years On: A Unifying Framework, International Journal of Computer Vision, 56(3), 2004, pp221 , p , ( ), , pp- 255.

• Peter Litwinowicz, Processing Images and Video for An Impressionist

Eff t SIGGRAPH 1997Effects, SIGGRAPH 1997.

• Aseem Agarwala, Aaron Hertzman, David Salesin and Steven Seitz, Keyframe-Based Tracking for Rotoscoping and Animation, SIGGRAPH 2004, pp584-591.