1 numerical geometry of non-rigid shapes multidimensional scaling multidimensional scaling ©...

1Numerical Geometry of Non-Rigid Shapes Multidimensional scaling

Multidimensional scaling

© Alexander & Michael Bronstein, 2006-2009© Michael Bronstein, 2010tosca.cs.technion.ac.il/book

048921 Advanced topics in visionProcessing and Analysis of Geometric Shapes

EE Technion, Spring 2010

2Numerical Geometry of Non-Rigid Shapes Multidimensional scalingIf it doesn’t fit, you must acquit!If it doesn’t fit, you must acquit!

Image: Associated Press

3Numerical Geometry of Non-Rigid Shapes Multidimensional scaling

Metric model

Euclidean metric

Invariant to rigid

motion

EXTRINSIC GEOMETRY

Similarity = isometry w.r.t.

Shape = metric space

Extrinsic geometry is not invariant

under inelastic deformations

4Numerical Geometry of Non-Rigid Shapes Multidimensional scaling

Metric model

Geodesic metric

Invariant to inelastic

deformation

INTRINSIC GEOMETRY

Euclidean metric

Invariant to rigid

motion

EXTRINSIC GEOMETRY

Similarity = isometry w.r.t.

Shape = metric space

5Numerical Geometry of Non-Rigid Shapes Multidimensional scaling

Intrinsic vs. extrinsic similarity

INTRINSIC SIMILARITYEXTRINSIC SIMILARITY

Part of the same metric space Two different metric spaces

SOLUTION: Find a representation of and

in a common metric space

6Numerical Geometry of Non-Rigid Shapes Multidimensional scaling

Canonical forms

Isometric embedding

7Numerical Geometry of Non-Rigid Shapes Multidimensional scaling

Canonical form distance

Compute canonical formsEXTRINSIC SIMILARITY OF CANONICAL FORMS

INTRINSIC SIMILARITY

= INTRINSIC SIMILARITY

8Numerical Geometry of Non-Rigid Shapes Multidimensional scaling

Mapmaker’s problem

9Numerical Geometry of Non-Rigid Shapes Multidimensional scaling

Mapmaker’s problem

A sphere has non-zero curvature, therefore, it

is not isometric to the plane

(a consequence of Theorema egregium)

Karl Friedrich Gauss (1777-1855)

10Numerical Geometry of Non-Rigid Shapes Multidimensional scaling

Linial’s exampleA

C D

1 1

2

AC D

1 1

1 1

2C D

B

A

B

C

D

B

A

C D

1 1

B

1

Conclusion: generally, isometric embedding does not exist!

No contradiction to Nash embedding theorem: Nash guarantees that any

Riemannian structure can be realized as a length metric induced by

Euclidean metric. We try to realize it using restricted Euclidean metric

11Numerical Geometry of Non-Rigid Shapes Multidimensional scaling

Minimum distortion embedding

Minimum distortion embedding

12Numerical Geometry of Non-Rigid Shapes Multidimensional scaling

Multidimensional scaling

The problem is called multidimensional scaling (MDS)

Find an embedding into that distorts the distances the least by

solving the optimization problem

The function measuring the distortion of distances is called stress

where are the coordinates of the canonical form

13Numerical Geometry of Non-Rigid Shapes Multidimensional scaling

Stress

L2-stress

The stress is a function of the canonical form coordinates and

the distances

Lp-stress

L-stress (distortion)

14Numerical Geometry of Non-Rigid Shapes Multidimensional scaling

- an matrix of canonical form coordinates (each row

corresponds to a point)

Matrix expression of L2-stress

Some notation:

Shorthand notation for Euclidean distances

Write the stress as

1 2

15Numerical Geometry of Non-Rigid Shapes Multidimensional scaling

Term 1

16Numerical Geometry of Non-Rigid Shapes Multidimensional scaling

Term 1

where is and matrix with elements

Matrices commute

under trace operator

17Numerical Geometry of Non-Rigid Shapes Multidimensional scaling

Term 2

For and

zero otherwise

18Numerical Geometry of Non-Rigid Shapes Multidimensional scaling

Term 2

where is and matrix-valued function with elements

19Numerical Geometry of Non-Rigid Shapes Multidimensional scaling

variables

Non-linear

Non-convex (thus liable to local convergence)

Optimum defined up to Euclidean transformation

LS-MDS

Minimization of L2-stress is called least squares MDS (LS-MDS)

20Numerical Geometry of Non-Rigid Shapes Multidimensional scaling

Gradient of L2-stress

Recall exercise

in optimization

Quadratic in Nonlinear in Constant in

21Numerical Geometry of Non-Rigid Shapes Multidimensional scaling

Stress computation complexity:

Complexity

Steepest descent

with line search may be prohibitively expensive (requires multiple

evaluations of the stress and the gradient)

Stress gradient computation complexity:

22Numerical Geometry of Non-Rigid Shapes Multidimensional scaling

Observation

for all and

By Cauchy-Schwarz inequality

Equality is achieved for

Write it differently:

23Numerical Geometry of Non-Rigid Shapes Multidimensional scaling

Observation

Consider the nonlinear term in the stress

24Numerical Geometry of Non-Rigid Shapes Multidimensional scaling

Majorizing inequality

[de Leeuw, 1977]

Jan de Leeuw

Equality is achieved for

We have a quadratic majorizing function to

use in iterative majorization algorithm!

25Numerical Geometry of Non-Rigid Shapes Multidimensional scaling

Iterative majorization

Construct a majorizing function satisfying

.

Majorizing inequality: for all

is convex

26Numerical Geometry of Non-Rigid Shapes Multidimensional scaling

Iterative majorization

Start with some

Find such that

Update iterate

Increment iteration counter

Solution

Until

convergence

27Numerical Geometry of Non-Rigid Shapes Multidimensional scaling

Iterative majorization

The majorizing function

is quadratic!

Analytic expression for the minimim

Analytic expression for

pseudoinverse:

28Numerical Geometry of Non-Rigid Shapes Multidimensional scaling

SMACOF algorithm

Start with some

Update iterate

Increment iteration counter

Solution

Until

convergence

The algorithm is called SMACOF (Scaling by Minimizing A COnvex

Function)

29Numerical Geometry of Non-Rigid Shapes Multidimensional scaling

SMACOF is a constant-step gradient descent!

SMACOF algorithm vs steepest descent

Majorization guarantees monotonically decreasing sequence of

stress

values (untypical behavior for constant-step steepest descent)

No guarantee of global convergence

Iteration cost:

30Numerical Geometry of Non-Rigid Shapes Multidimensional scaling

MATLAB® intermezzoSMACOF algorithm

Canonical forms

31Numerical Geometry of Non-Rigid Shapes Multidimensional scaling

Examples of canonical forms

Canonical forms

Near-isometric deformations of a shape

32Numerical Geometry of Non-Rigid Shapes Multidimensional scaling

Examples of canonical forms

Visualization of intrinsic similarity

33Numerical Geometry of Non-Rigid Shapes Multidimensional scaling

Weighted stress

where are some fixed weights

where is and matrix with elements

Matrix expression

SMACOF algorithm can be used to minimize weighted stress!

34Numerical Geometry of Non-Rigid Shapes Multidimensional scaling

Variations on the stress theme

Generic form of the stress

where is some norm

For example, gives the Lp-stress

The necessary condition for to be the minimizer of is

35Numerical Geometry of Non-Rigid Shapes Multidimensional scaling

Variations on the stress theme

Idea: minimize weighted stress instead of the generic stress and choose

the weights such that the two are equivalent

If the weights are selected as

the minimizers of and will coincide

Since is unknown in advance, iterate!

36Numerical Geometry of Non-Rigid Shapes Multidimensional scaling

Iteratively reweighted LS

Start with some and

Find

using as an initialization

Update weights

Increment iteration counter

Until

convergence

The algorithm is called Iteratively Reweighted Least Squares (IRLS)

37Numerical Geometry of Non-Rigid Shapes Multidimensional scaling

L-stress

Optimization problem

can be written equivalently as a constrained problem

is called an artificial variable

38Numerical Geometry of Non-Rigid Shapes Multidimensional scaling

Complexity, bis

N=1000 points

MDS

N=200 points

MDS

x5 less pointsx25 lower complexity

39Numerical Geometry of Non-Rigid Shapes Multidimensional scaling

Multiresolution MDS

Bottom-up approach: solve coarse level MDS problem to initialize fine

level problem

Reduce complexity (less fine-level iterations)

Reduce the risk of local convergence (good initialization)

Can be performed on multiple resolution levels

40Numerical Geometry of Non-Rigid Shapes Multidimensional scaling

Solve coarse level problem

Interpolate

Two-resolution MDS

Solve finelevel problem

Grid

Data

Interpolation operator to transfer solution

from

coarse level to fine level

Can be represented as an sparse matrix

41Numerical Geometry of Non-Rigid Shapes Multidimensional scaling

Solve coarsest level problem

Interpolate

Multiresolution MDS

Solve -st level problem

Interpolate

Solve finestlevel problem

42Numerical Geometry of Non-Rigid Shapes Multidimensional scaling

Multiresolution MDS algorithm

Hierarchy of grids

Hierarchy of data

Interpolation operators

Start with coarsest-level initialization

Solve the l-th level MDS problem

using as initialization

Interpolate to next level

For

Solve finest level problem

using as initialization

43Numerical Geometry of Non-Rigid Shapes Multidimensional scaling

Top-down approach

Start with a fine-level initialization

Decimate fine-level initialization to the coarse grid

Solve the coarse-level MDS problem

using as initialization

Improve the fine-level solution propagating the coarse-level error

Typically

44Numerical Geometry of Non-Rigid Shapes Multidimensional scaling

A problem

Fine level

Coarse level

45Numerical Geometry of Non-Rigid Shapes Multidimensional scaling

Correction

The fine- and the coarse-level minimizers do not coincide!

is called a residual

Force consistency of the coarse- and fine-level problems by introducing a

correction term into the stress

which gives the gradient

46Numerical Geometry of Non-Rigid Shapes Multidimensional scaling

Correction

Fine level

Coarse level

47Numerical Geometry of Non-Rigid Shapes Multidimensional scaling

Correction

In order to guarantee consistency, must satisfy

at

which gives the correction term

Verify: is the minimizer of coarse-level problem with correction,

Chain rule

48Numerical Geometry of Non-Rigid Shapes Multidimensional scaling

Modified stress

Another problem: the coarse grid problem

is unbounded for

Fix the translation ambiguity by adding a quadratic penalty to the stress

Can grow arbitrarily

is called the modified stress [Bronstein et al., 05]

The modified coarse grid problem is bounded

49Numerical Geometry of Non-Rigid Shapes Multidimensional scaling

Solve corrected coarse level problem

Two-grid MDS

Correct fine level solution

Decimate Interpolate

Iteratively improve the fine-level solution using coarse grid residual

External iteration is called cycle

50Numerical Geometry of Non-Rigid Shapes Multidimensional scaling

Solve corrected coarse level problem

Two-grid MDS

Correct fine level solution

Decimate Interpolate

Perform a few optimization iterations at fine level between cycles

(relaxation)

Relax

51Numerical Geometry of Non-Rigid Shapes Multidimensional scaling

Two-grid MDS algorithm

Start with fine-level initialization

Produce an improved fine-level solution by

making a few iterations on initialized with

Decimate fine-level solution

Correction:

Solve the coarse-level corrected MDS problem

using as initialization

Transfer residual:

Update iterate

Solution

Until

convergence

52Numerical Geometry of Non-Rigid Shapes Multidimensional scaling

Solve coarsest level problem

Interpolatecorrect

Multigrid MDS

Relax

Decimate

Relax

Decimate

Interpolatecorrect

Relax

Relax

Apply the two-grid algorithm recursively

Different cycles possible, e.g. V-cycle

53Numerical Geometry of Non-Rigid Shapes Multidimensional scaling

MATLAB® intermezzoMultigrid MDS

54Numerical Geometry of Non-Rigid Shapes Multidimensional scaling

Convergence example

Time (sec)

Str

ess

Convergence of SMACOF and Multigrid MDS (N=2145)

55Numerical Geometry of Non-Rigid Shapes Multidimensional scaling

How to accelerate convergence?

Let be a sequence of iterates produced by some

optimization algorithms converging to

Denote by the remainder

Construct a new sequence converging faster to

in the sense

56Numerical Geometry of Non-Rigid Shapes Multidimensional scaling

Vector extrapolation

Construct the new sequence as a transformation of iterates

Particular choice: linear combination

Question: how to choose the coefficients ?

Ideally, hence

57Numerical Geometry of Non-Rigid Shapes Multidimensional scaling

Reduced rank extrapolation (RRE)

Problem: depends on unknown

Eliminate this dependence by considering the difference

Result: solve the constrained linear system

which ideally should vanish.

of equations in variables

Typically hence the system is over-determined and must be

solved approximately using least-squares

58Numerical Geometry of Non-Rigid Shapes Multidimensional scaling

Start with some and

Generate iterates

using optimization on initialized by

Extrapolate from

Update

Increment iteration counter

Until

convergence

Optimization with vector extrapolation

59Numerical Geometry of Non-Rigid Shapes Multidimensional scaling

Start with some and

Generate iterates

using optimization on initialized by

Extrapolate from

If

else

Increment iteration counter

Until

convergence

Safeguarded optimization with vector extrapolation

60Numerical Geometry of Non-Rigid Shapes Multidimensional scaling

MATLAB® intermezzoRRE MDS