nonlinear slides (1)

8/3/2019 Nonlinear Slides (1)

http://slidepdf.com/reader/full/nonlinear-slides-1 1/38

Regularized Least-Squares

Non-linear Least-Squares




Why non-linear?

• The model is non-linear (e.g. joints, position, ..)

• The error function is non-linear




Setup

• Let f be a function such that

where x is a vector of parameters

( ) R f Rn ∈→∈ x a,a




Setup



• Let {ak ,bk } be a set of measurements/constraints.

We fit f to the data by solving:

( ) R f Rn ∈→∈ x a,a

( )( )2

2

1min ∑ −

k

k k f b x ,a x




Setup



• Let {ak ,bk } be a set of measurements/constraints.

We fit f to the data by solving:

( )( )

( ) x ,a

x ,a x x

k k k

k

k

k

k k

f br

r f b

−=

− ∑∑ with

or22

min2

1min

( ) R f Rn ∈→∈ x a,a




Example


http://slidepdf.com/reader/full/nonlinear-slides-1 7/38Regularized Least-Squares

Overview

• Existence and uniqueness of minimum

• Steepest-descent

• Newton’s method

• Gauss-Newton’s method• Levenberg-Marquardt method



A non-linear function:the Rosenbrock function

2222 )(100)1(),( x y x y x f z −+−==

Global minimum at (1,1)



Existence of minimum

A local minima is characterized by:

1.

2.

( ) 0=∇ min x E

( )

( ) definite)-semipositiveis (e.g.

enoughsmallallfor

min

min ,0

x H

hh x Hh

E

T

E

T ≥



Existence of minimum

min x

( ) x E ( ) ( ) ( ) h x Hh x h x

T

E

T E E 2

1+≈+ minmin



Descent algorithm

• Start at an initial position x 0

• Until convergence

– Find minimizing step d x k

– x k+1= x k+ d x k

Produce a sequence x 0, x 1, …, x n such that

f( x ) > f( x ) > …> f( x n)



Descent algorithm

• Start at an initial position x 0

• Until convergence

– Find minimizing step d x k

using a local approximation of f

– x k+1= x k+ d x k

Produce a sequence x 0, x 1, …, x n such that

f( x 0) > f( x 1) > …> f( x n)



Approximation using Taylor series

( ) ( ) ( ) ( ) ( )2

2

1hOh x Hhh x x h x ++∇+=+ T

E

T T E E E




( ) ( ) ( ) ( ) ( )2

2


E

T T E E E

( )

∂∂=

∂∂== ∑

= i

j

r

i

jm

j

j xr

xr r E

j

2

1

2 ,21 H J x andwith




( ) ( ) ( ) ( ) ( )2

2


E

T T E E E

( )

∑∑∑∑

∑

===

=

=

+=+∇∇=

=∇=∇

∂∂=

∂∂==

m

j

T

r j

T m

j

T

r j

m

j

T

j j E

T m

j

j j

i

j

r

i

jm

j

j

j j

j

r r r r

r r E

xr

xr r E

111

1

2

1

2 ,21

H J JHH

r J

H J x andwith



Steepest descent

Step

where is chosen such that:

using a line search algorithm:

( )k k k E x x x ∇−=+ α 1

α ( ) ( )k k x f x f <+1

( )( )k k k E f x x ∇−α α

min



1+k x

k x

1 x

2 xmin x

( )k E x ∇



1+k x k x

In the plane of thesteepest descent direction




Rosenbrock function (1000 iterations)




Newton’s method

• Step: second order approximation

At the minimum of N

( ) ( ) ( ) ( )T k E

T T

k k k E E N E x Hhh x x hh x 2

1)( +∇+=≈+

( ) ( )

( ) ( )k k E

k E k

E

E N

x x Hh

0h x H x h

∇−=⇒

=+∇⇒=∇−1

)( 0

( ) ( )k k E k k E x x H x x ∇−= −+

1

1




1+k x

k x

1 x

2 xmin x

( ) ( )k k E E x x H- ∇−1

( ) ( ) ( ) h x Hhh x x T

k E

T T

k k E E 2

1+∇+




Problem

• If is not positive semi-definite, then

is not a descent direction: the step increases the error

function

• Uses positive semi-definite approximation of Hessianbased on the jacobian

(quasi-Newton methods)

( ) ( )k k E E x x H- ∇−1( )k E x H




Gauss-Newton method

Step: use

with the approximate hessian

Advantages:

• No second order derivatives

• is positive semi-definite

( ) ( )k k E k k E x x H x x ∇−= −+

1

1

J JH J JH T m

j

T

r j

T

E jr ≈+= ∑

=1

J J T




Rosenbrock function (48 evaluations)




Levenberg-Marquardt algorithm

• Blends Steepest descent and Gauss-Newton

• At each step solve, for the descent direction

( ) ( )k T E λ x hI J J −∇=+

h

( )

( ) ( ) Newton)-(Gauss

smallif descent)(steepest

largeif

1

k

T

k

E

λ E

λ

x J Jh

x h

∇−∝

−∇∝

−




Managing the damping parameter

• General approach:

– If step fails, increase damping until step is successful

– If step succeeds, decrease damping to take larger step

• Improved damping

λ

( ) ( )k

T T E λ x h J J J J −∇=+ )(diag




Rosenbrock function (90 evaluations)




“Resynthesizing Facial Animation through3D Model-Based Tracking”, Pighin etal.,

Video frame

α1

α2

α3

α4

α5

+

+

+

+

Model




Model fitting

Video frame

2α1

α2

α3

α4

α5

+

+

+

+

-

Model

{ }iα min




Model fitting

Video frame Fitted model

2α1

α2

α3

α4

α5

+

+

+

+

-

Model

{ }iα min




“Resynthesizing Facial Animation through 3DModel-Based Tracking”, Pighin etal.,

• Facial tracking by fitting face model






• Problem

– Given an image I

– A 3D face model






• Problem

– Given an image I

– A 3D face model

Minimize

Where is the image produced by the face model

And is a regularization term.

+−∑ )(),)((ˆ),(

2

1min p p p D y x I y x I

i

j j j j

)(ˆ p I

( ) p D




Problem setup

( )

( ) ( ) 221,0max,0min)(

}{

−+=∑ k k

k

k

D α α

α

p

pt R |positionscameratheand

parametersfacialincludingparametersof setais

0 1

)( p D




Solving

• Using a gaussian pyramid

of the input image

• For level in [coarserLevel : finerLever] – Initialize using solution at level-1

– Minimize at level using Levenberg-Marquardt

Solution is solution at finerLevel




Tracking results



Conclusion

• Get a good initial guess

Prediction (temporal/spatial coherency)

Partial solve

Prior knowledge

• Levenberg-Marquardt is your friend

nonlinear slides (1)

Documents