ICCV 2007

Optimization & Learning for Registration of Moving Dynamic Textures

Junzhou Huang1, Xiaolei Huang2, Dimitris Metaxas1

Rutgers University1, Lehigh University2

ICCV 2007

Outline

• Background• Goals & Problems• Related Works• Proposed Method• Experiment Results• Discussion & Conclusion

ICCV 2007

Background

• Dynamic textures (DT)– static camera, exhibits a certain stationary

• Moving Dynamic textures (MDT)– dynamic textures captured by a moving camera

DT, [Kwatra et al. SIGGRAPH’03] MDT, [Fitzgibbon ICCV’01]

ICCV 2007

Background

• Video registration– Required by many video analysis applications

• Traditional assumption– Static, rigid, brightness constancy – Bergen et al. ECCV’92, Black et al. ICCV’93

• Relaxing rigid assumption– Dynamic textures– Doretto et al. IJCV’03, Yuan at al. ECCV’04, Chan

et al. NIPS’05, Lin et al. PAMI’07, Rav-Acha at al. Workshop at ICCV’05

ICCV 2007

Our Goals

• Registration of MDT– Recover the camera motion and register the image

sequences including moving dynamic textures

Left Translation Right Translation

ICCV 2007

Complex Optimization Problems

• Complex optimization– Camera motion, dynamic texture model– Chicken-and-Egg Problems

• Challenges– About the mean images– About LDS model– About the camera motion?

ICCV 2007

Related Works

• Fitzgibbon, ICCV’01– Pioneering attempt– Stochastic rigidity– Non-linear optimization

• Vidal et al. CVPR’05– Time varying LDS model– Static assumption in small time window– Simple and general framework but under estimation

ICCV 2007

Formulation

• Registration of MDT– I(t), the video frame– camera motion parameters– y0 , the desired average image of the video – y(t), related with appearance of DT– x(t), related with dynamics of DT

)(t

ICCV 2007

Generative Model

x(t-1) x(t) x(t+1)

y(t-1) y(t) y(t+1)

I (t-1) I (t) I ( t+1)

y0

W(t-1) W(t) W(t+1)

Generative image model for a MDT

ICCV 2007

First Observation

• Good registration– a good registration according to the accurate camera

motion should simplify the dynamic texture model while preserving all useful information

– Used by Fitzgibbon, ICCV’01, Minimizing the entropy function of an auto regressive process

– Used by Vidal, CVPR’05, optimizing time varying LDS model by optimizing piecewise LDS model

ICCV 2007

Second Observation

• Good registration– A good registration according to the

accurate camera motion should lead to a sharp average image whose statistics of derivative filters are similar to those of the input image frames.

• Image statistics – Student-t distribution / heavy tailed image priors– Huang et al. CVPR’99, Roth et al. CVPR’05

ICCV 2007

Prior Models

• The Average image priors• The motion priors• The dynamic priors

ICCV 2007

Average Image Priors

• Student-t distribution– Model parameters / contrastive divergence method

(a) Before registration, (b) in the middle of registration (c) after registration

ICCV 2007

Motion / Dynamic Priors

• Gaussian Perturbation (Motion)– Uncertainty in the motion modeled by a

Gaussian perturbation about the mean estimation M0 / the covariance matrix S ( a diagonal matrix.)

– Motivated by the work [Pickup et al. NIPS’06]• GPDM / MAR model (Dynamic)

– Marginalizing over all possible mappings between appearance and dynamics

– Motivated by the work [Wang et al. NIPS’05] [Moon et al. CVPR’06]

ICCV 2007

Joint Optimization

• Generative image model

• Optimization– Final marginal likelihood

– Scaled conjugate gradients algorithm (SCG)

ICCV 2007

Procedures

• Obtaining image derivative prior model• Dividing the long sequence into many short

image sequences• Initialization for video registration• Performing model optimization with the

proposed prior models until model convergence.

• With estimated y0, Y and X, the camera motion is then obtained

ICCV 2007

Obtaining Data

• Three DT video sequences– DT data, [Kwatra et al. SIGGRAPH’03]

• Synthesized MDT video sequence– 60 frames each, no motion from 1st to 20th frame and

from 41st to 60th – Camera motions with speed [1, 0] from 21st to 40th

ICCV 2007

Grass MDT Video

• The average image

(a) One frame, (b) the average image after registration, (c) before registration

ICCV 2007

Grass MDT Video

• The statistics of derivative filter responses

-60 -40 -20 0 20 40 600

0.05

0.1

0.15

Gradient

Pro

babi

lity

dist

ribut

ion

Input ImagesAfter RegistrationBefore Registration

ICCV 2007

Evaluation / Comparison

• False Estimation Fraction

• Comparison with two classical methods– Hybrid method, [Bergen et al. ECCV’92] [Black et

al. ICCV’93]– Vidal’method, [Vidal et al. CVPR’05]

ICCV 2007

Waterfall MDT Video

• Motion estimation

(a) Ground truth, (b) by hybrid method, (c) by Vidal’s, (d) proposed

ICCV 2007

Waterfall MDT Video

• The average Image and its statistics

The average image and related distribution after registration by (a) proposed method, (b) Vidal’s method, (c) hybrid method

ICCV 2007

FEF Comparisons

• On three synthesized MDT video

ICCV 2007

Real MDT Video

• Moving flower bed video• Ours

– 554 frames totally– Ground truth 110 pixels– Estimation 104.52 pixels

( FEF 4.98%)• Vidal’s

– 250 frames– Ground truth 85 pixels – Estimation 60 pixels ( FEF 29.41%)

ICCV 2007

Conclusions• What proposing:

– Powerful priors for MDT registration• What getting out:

– Camera motions– Average image – Dynamic texture model

• What learning?– Registration simplify DT model while preserving

useful information– Better registration lead to sharper average image

ICCV 2007

Thanks !

ICCV 2007

Thanks !

ICCV 2007

Future Works• More complex camera motions• Different Metric functions for evaluation• Multiple dynamic texture segmentation