precise multi-frame motion estimation and its...

Milanfar et al. EE Dept, UCSC

Precise Multi-Frame Motion Estimation and Its Applications

Peyman MilanfarEE Department

University of California, Santa [email protected]

Joint work with Dirk Robinson, Michael Elad, Sina Farsiu


Motivating Application: Resolution Enhancement from Video

The Idea: “Diversity” + Aliasing• Given multiple low-resolution movingmoving images of a

scene (a video), generate a high resolution image (or video).


Motivation Application: (Hollywood Version!)


Practical Motivation: (Real Video Enhancement)

MotionEstimation

ImageReconstruction

Resolution enhancement x4 from video frames captured by a commercial webcam(3COM Model No.3719)


Problem: Given a pair of frames,

we want to estimate the translation

Typical Assumptions:–Sampled on a finite grid.–Sampled above Nyquist rate

– Will discuss aliased case later.–Additive white Gaussian Noise

Translational Motion Estimation

),(),(),(),(),(),(

2122211212

21121211

xxevxvxfxxfxxexxfxxf+−−=

+=

=

2

1

vv

v


Translational Motion Estimation

Optimum Statistical Estimator: Max. Likelihood

– Correlation Methods• Direct Maximization• Phase-Correlation

– Nonlinear Least Squares• Gradient-Based algorithms• Pyramid-Gradient-Based algorithms• Direct Minimization

– Improving to subpixel accuracy• Fits a quadratic about the peak of the

correlation surface.• Gauss-Newton methods, iterated improvement • Iterating over scale: pyramid-based methods

( )∑ −−−2121 ,

221222111

,),(),(min

xxvvxxfvxvxf

∑ −−2121 ,

21222111,

),(),(maxxxvv

xxfvxvxf


Performance Limits in Image Registration

How close to the limit are typical methods?Image used

At 5 dB

BiasIndependent of underlylingvelocity vector


Effect of Aliasing

How does aliasing affect the ability to estimate translation between sets of images?

Little aliasing Lot of aliasing

Note “false” motions.


Performance of Aliased Image Registration

Observations:

• Very little work addressing registration of aliased image.

• Performance bound depends on the motion parameters (not true for non-aliased registration)

• Traditional algorithms designed for non-aliased scenario will fail.


• Consider a sequence of noisy, translating images over time.

Data and Formulation

{ }kf

noiseSample += )],(*),,([ yxhtyxf kkf

error Translate += −− ),( ,11 kkkk vff

Nf

1f2,1v

2f3,2v

• Image formation model:

Point-spread function

Frame-to-frame motion vectors

Aliasing




• Motion Problem: Given the frames, estimate vectors

– Implicit problem: Estimate underlying high resolution image

Registration of Multiple Video Frames

Desiredunknowns

{ }kkv ,1−

Nuisance Parameter




• Reconstruction Problem: Given the frames, estimate the high resolution image . (Superresolution)

– Implicit problem: Estimate the motion vectors

Fusion of Multiple Video Frames

NuisanceParameters

Desired unknowns

),,( tyxf


Registration Information Information

“Correlation”

Reconstruction Information

vvJ - Depends on the set of motions (sampling offsets) and the amount of texture energy in the signal

- Depends only on the set of motionsffJ

( )

=−

ffTfv

fvvvkk fv

JJJJ

J },{ ,1

Fisher Information Matrix (FIM) is partitioned on the motion parameters and the high-res (alias-free) “atlas” image .

{ }kkv ,1−

f

How well can the problem be solved?


CRB for Aliased Image Registration

Using Schur decomposition, the CRB for aliased image registration is:

With just a pair of aliased images, the FIM becomessingular, hence pairwise registration of aliased images is essentially impossible

{ }( ) ( ) 11,

−−−≥ Tfvfffvvvkjv JJJJCov

Registration Information

Information Loss due to uncertainty about the high resolution image.


Registering Sets of Images

0 5 10 15 20 25 30 35 40 45 500.004

0.006

0.008

0.01

0.012

0.014

0.016

0.018

0.02

0.022

Number of Frames (K+1)

Reg

istr

atio

n C

RB

(pi

x/fr

ame)

M = 1M = 2M = 3M = 4M = 5+

CR Bound (per frame) for multi-frame image registration.

No aliasing

More aliasing


What to do? • Almost all motion estimation algorithms

today deal with the case of only two (consecutive) frames at a time.

• These methods are far from optimal.• Proposal: Use multiple frames

simultaneously, with care!

Pairwise estimation(“Progressive”)

Fixed reference estimation(“Anchored”)


Constraints on Translational Motion Vectors Across Time

Frame i Frame j Frame k

kjv ,

kiv ,

jiv ,

0,

,,

,,,

=

−=

+=

ii

kjjk

kjjiki

v

vv

vvvLinear set of constraints imply that themotion vectors live in a subspace.


An Algorithm Thus Motivated

∑ −kj

p

pkjjkvv

jk ,,}{

),(ff Translate minimize

kjjk

kjjiki

vv

vvv

,,

,,,

−=

+=

to subject

Can be any penalty function

• With p=2, and linear constraints, we have a quadratic programming problem.

• Computationally simpler (but suboptimal) is to project any estimated parameters onto the constraint subspace.

• With p=1, we can have a more robust solution.


Performance

0 10 20 30 40 50 60 7010

-3

10-2

10-1

100

101

SingleProjected

p=2, Constrainedp=1, Constrained

RM

SE

(Pix

els)

SNR (dB)


Application to Simultaneous Demosaicingand Resolution EnhancementBayer Filtered Motion Sequence

. . .

Single-FrameDemosaicing

OLD

Image fusion Hi-resolutionDemosaicing

NEW


Example with Real Data

27 Raw CFA Images

• Registering the color-filtered data is non-standard and difficult in practice.


Example with Real Data• Standard Single-frame demosaicing


Example with Real Data• Multi-frame Progressive Registration


Example with Real Data• Multi-frame Anchored Registration


Example with Real Data• Multi-frame Projected Registration Vectors


Example with Real Data• Multi-frame Constrained Registration


Nice Algebraic Structure

• S is closed under +• S is associative• S has an identity element : 0 • Every element of S has an inverse

{ }kjvS ,= Satisfies

Set of all pairwise motions S between frames is a GROUP

0,

,,

,,,

=

−=

+=

ii

kjjk

kjjiki

v

vv

vvv


A Few Words on Affine Motion

Frame i Frame j Frame k

),( ,, kiki TM

),( ,, jiji TM ),( ,, kjkj TM

Constraints


General Comments • The constraints for the affine case are nonlinear.

• But the algebraic structure persists.– Group operation is no longer simple vector addition

• Algebraic structure also for dense optical flow– Here the elements of the algebra (motions) are

defined by nonlinear transformations applied to images.

– Lie Algebra defined by the composition of operators.– If operators are differentiable, then a Lie Group.


Conclusions

• Accurate motion estimation is a (very) hard problem.

• Registering a pair of aliased frames is an ill-posed problem.

• Using simultaneous image registration and reconstruction is one possible solution.

• Motion estimation with constraints (hard or soft) is another alternative.

• There are many applications.

precise multi-frame motion estimation and its...

Documents