full-frame video stabilization with motion...

Full-Frame Video Stabilization with Motion Inpainting

Yasuyuki Matsushita,E l Of kEyal OfekWeina Ge

Xiaoou TangHeung-Yeung Shumg g

IEEE Trans on PAMI, July 2006

OutlineOutline

I t d ti• Introduction• Proposed Method• Experimental results• Quantitative EvaluationQuantitative Evaluation• Computation Cost

C l i• Conclusion

2

IntroductionIntroduction

St bili ti• Stabilization：– Remove undesirable motion caused by

i t ti l h k f h h dunintentional shake of a human hand.• remove high frequency camera motion vs.

completely remove camera motioncompletely remove camera motion.• full frame vs. trimming• motion inpainting vs. mosaicingp g g

3

Prior Work vs NowPrior Work vs. Now

4

Input Video

Global Motion Estimation

Motion Smoothing

Local Motion Estimation

Motion Inpainting Image Deblurring

Completion

5Output Video

Global Motion EstimationGlobal Motion Estimation

GM i ti t d b li i i i• GM is estimated by aligning pair-wise adjacent frames.

min( ( ) ( ))I Tp I p–• Hierarchical motion estimation

– construct an image pyramid

'min( ( ) ( ))t t t tTI Tp I p−

– construct an image pyramid– start from the coarsest level

• By applying the parameter estimation forBy applying the parameter estimation for every pair of adjacent frames, a global transformation chain is obtained.j

iT6

iH.-Y. Shum and R. Szeliski, “Construction of Panoramic Mosaics with Global and Local Alignment,” Int’l J. Computer Vision, vol. 36, no. 2, pp. 101-130, 2000.

Input Video


Motion Smoothing



Completion

7Output Video

Image DeblurringImage Deblurring

T f i h i i l• Transferring sharper image pixels from neighboring frames.– evaluates the “relative blurriness”

1b = 2 2y{(( )( )) (( )( )) }

t

tx t t t t

p

bf I p f I p

=⊗ + ⊗∑

– evaluates the “alignment error”'

' '( ) | ( ) ( ) |t t tt t t t t tE p I T p I p= −

8

( ) | ( ) ( ) |t t t t t tp p p


Bl i l l d b i t l ti• Blurry pixel are replaced by interpolating shaper pixels. '

' '( ) ( ) ( )t tt t t t t t tI p w p I T p+ ∑

w is the eight factor hich consists of

'

''

( )1 ( )

t Nt t t

t tt N

I pw p

∈

∈

=+ ∑

• w is the weight factor which consists of the pixel-wise alignment error and relative blurrinessblurriness

'

'

'( )

0 1( )

t

tt

tt

bbt

bt tb E p

ifw p

otherwiseα

α+

⎧ <⎪= ⎨⎪⎩

9

' ' ( )t t tb E p otherwiseα+⎪⎩


10

Input Video


Motion Smoothing



Completion

11Output Video

Motion SmoothingMotion Smoothing( )iS T G k= ⊗∑

2 2

t

/ 212

( )

{ : }, ( ) ,t

ti N

kt

S T G k

N j t k j t k G k e kσπσ

σ

∈

−

⊗

= − ≤ ≤ + = =

∑

12

Motion SmoothingMotion Smoothing

13

Input Video


Motion Smoothing



Completion

14Output Video

Local Motion EstimationLocal Motion Estimation

A id l i f L K d• A pyramidal version of Lucas-Kanade optical flow computation is applied to bt i th l l ti fi ldobtain the local motion field.

15J. Bouguet, “Pyramidal Implementation of the Lucas Kanade Feature Tracker: Description of the Algorithm,” OpenCV Document, Intel, Microprocessor Research Labs, 2000.

Input Video


Motion Smoothing



Completion

16Output Video

Motion InpaintingMotion Inpainting

M i i ith i t t i t• Mosaicing with consistency constraint.'

' ' ( )( ( ))( )

tt tt t t t if v p Tmidian I T p

I p<⎧

= ⎨

where

( )t tI potherwisekeep it as missing

⎨⎩

2' '' '

'

1( ) ( ( ) ( ))11

t

t tt t t t t t t t

t Mv p I T p I T p

n ∈

= −− ∑

' '' '

'

1( ) ( )t

t tt t t t t t

t MI T p I T p

n ∈

= ∑

17


18A. Telea, “An Image Inpainting Technique Based on the Fast Marching Method,” J. Graphics Tools, vol. 9, no. 1, pp. 23-34, 2004.


Th ti l f i l i• The motion value for pixel pt is generated by a weighted average of th ti t f th i l H( )the motion vectors of the pixels H(pt)

( )( , ) ( | )

( ) t t

t t t tq H p

w p q F p qF ∈

∑

where

( )

( )

( )( , )

t t

t t

q H pt

t tq H p

F pw p q

∈

∈

=∑

( ) ( )

( ) ( )( | ) ( ) ( )( ) ( )

x t x t

y t y t

F q F qx y

t t t t t t t F q F qx y

xF p q F q F q p q F q

y

∂ ∂∂ ∂

∂ ∂∂ ∂

⎡ ⎤ Δ⎡ ⎤⎢ ⎥= + ∇ − = + ⎢ ⎥Δ⎢ ⎥ ⎣ ⎦⎣ ⎦

19' ' ' '

1 1( , )|| || || ( ) ( ) ||t t

t t t t t t t t

w p qp q I q p q I q ε

=− + − − +

Input Video


Motion Smoothing



Completion

20Output Video

Summary of the AlgorithmSummary of the Algorithm

21

Experimental ResultsExperimental Results

30 id li ( b t 80 i t )• 30 video clips (about 80 minutes) with different types of scenes

• k = 6 for motion smoothing• 5x5 filter for motion inpaintingp g

22

Experimental Results (1)Experimental Results (1)

23


24


25

Failure Cases –incorrect estimation of motion

26

Failure Cases –abrupt changes of motion

27

Quantitative EvaluationQuantitative Evaluation

D i ti f th G d T th• Deviation from the Ground Truth.• MAD of intensity

28

Quantitative EvaluationQuantitative Evaluation

E l ti f S ti T l S th• Evaluation of Spatio-Temporal Smoothness.– The normalized discontinuity measure D is

defined asdefined as1 1|| ||

n n

i i ii i

I

D I I In n

∂

= ∇ = ∇ ⋅∇

⎡ ⎤ ⎡ ⎤

∑ ∑( 1, , ) ( 1, , )( , 1, ) ( , 1, )( 1) ( 1)

IxIy

I

I x y t I x y tI I x y t I x y t

I x y t I x y t

∂∂∂∂

∂

⎡ ⎤ + − −⎡ ⎤⎢ ⎥ ⎢ ⎥∇ = ≈ + − −⎢ ⎥ ⎢ ⎥⎢ ⎥ ⎢ ⎥+⎣ ⎦⎣ ⎦

– The relative smoothness is evaluated by(D D )/(D D )

( , , 1) ( , , 1)It

I x y t I x y t∂∂⎢ ⎥ ⎢ ⎥+ − −⎣ ⎦⎣ ⎦

29

(DM-DO)/(DM-DA)– 5.9%~23.5% smoother than mosaicing

Computation CostComputation Cost

2 2 f / f 720 486 id ith• 2.2 frames/s for 720x486 video withP4 2.8GHz CPU

30

ConclusionConclusion

Motion inpainting instead of cropping• Motion inpainting instead of cropping.• Deblurring without estimating PSFs.• Spatial smoothness is indirectly

guaranteed by the smoothness of the extrapolated motionextrapolated motion.

• Temporal consistency on both static and dynamic areas is given by opticaland dynamic areas is given by optical flow from the neighboring frames.

31

Thank You

presented by蕭志傑蕭志傑

Hsiao, Chih-Chieh

full-frame video stabilization with motion...

Documents