iterative reweighted least squares · 2014-09-06 · 9/7/2014 1 iterative reweighted least squares...

42
9/7/2014 1 Iterative Reweighted Least Squares ECCV, Sept 7, 2014 120 120 120 What point minimizes the distance to the three points of a triangle? Exercise: find this point using ruler and compass construction. Toricelli

Upload: others

Post on 03-Aug-2020

14 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Iterative Reweighted Least Squares · 2014-09-06 · 9/7/2014 1 Iterative Reweighted Least Squares ECCV, Sept 7, 2014 120 120 120 What point minimizes the distance to the three points

9/7/2014

1

Iterative Reweighted Least Squares

ECCV, Sept 7, 2014

120

120

120

What point minimizes the distance to the three points of a triangle?

Exercise:  find this point using ruler and compass construction.

Toricelli

Page 2: Iterative Reweighted Least Squares · 2014-09-06 · 9/7/2014 1 Iterative Reweighted Least Squares ECCV, Sept 7, 2014 120 120 120 What point minimizes the distance to the three points

9/7/2014

2

Ruler and Compass Constructionto find Fermat point

Alfred Weber

Page 3: Iterative Reweighted Least Squares · 2014-09-06 · 9/7/2014 1 Iterative Reweighted Least Squares ECCV, Sept 7, 2014 120 120 120 What point minimizes the distance to the three points

9/7/2014

3

Weber’s Solution (1906)

Gustav Weler

Gustav Weler was a political decoy (doppelgänger or Body‐double) of Adolf Hitler. At the end of the Second World War, he was executed by a gunshot to the forehead in an attempt to confuse the Allied troops when Berlin was taken.[citation needed] He was also used "as a decoy for security reasons".[2] When his corpse was discovered in the Reich Chancellery garden by Soviet troops, it was mistakenly believed to be that of Hitler because of his identical moustache and haircut. The corpse was also photographed and filmed by the Soviets.

One servant from the bunker declared that the dead man was one of Hitler's cooks. He also believed this man "had been assassinated because of his startling likeness to Hitler, while the latter had escaped from the ruins of Berlin".[3]

Weler's body was brought to Moscow for investigations and buried in the yard at Lefortovo prison.[4]

Page 4: Iterative Reweighted Least Squares · 2014-09-06 · 9/7/2014 1 Iterative Reweighted Least Squares ECCV, Sept 7, 2014 120 120 120 What point minimizes the distance to the three points

9/7/2014

4

Andrew Vázsonyi (1916–2003), also known as Endre Weiszfeld and Zepartzatt Gozinto) was a mathematician and operations researcher. He is known for Weiszfeld's algorithm for minimizing the sum of distances to a set of points, and for founding The Institute of Management Sciences.[1][2][3]

Weiszfeld

E. Weiszfeld, Sur le point pour lequel la somme des distances de n points donnés est minimum, Tôhoku Mathematics Journal 43 (1937), 355 - 386.

Page 5: Iterative Reweighted Least Squares · 2014-09-06 · 9/7/2014 1 Iterative Reweighted Least Squares ECCV, Sept 7, 2014 120 120 120 What point minimizes the distance to the three points

9/7/2014

5

Weighted L2 cost function

Robust (L1) cost function

Page 6: Iterative Reweighted Least Squares · 2014-09-06 · 9/7/2014 1 Iterative Reweighted Least Squares ECCV, Sept 7, 2014 120 120 120 What point minimizes the distance to the three points

9/7/2014

6

Page 7: Iterative Reweighted Least Squares · 2014-09-06 · 9/7/2014 1 Iterative Reweighted Least Squares ECCV, Sept 7, 2014 120 120 120 What point minimizes the distance to the three points

9/7/2014

7

Gradient of L2 distance Gradient of L1 distance

Page 8: Iterative Reweighted Least Squares · 2014-09-06 · 9/7/2014 1 Iterative Reweighted Least Squares ECCV, Sept 7, 2014 120 120 120 What point minimizes the distance to the three points

9/7/2014

8

Generalizing IRLS

Functions for which IRLS works

Page 9: Iterative Reweighted Least Squares · 2014-09-06 · 9/7/2014 1 Iterative Reweighted Least Squares ECCV, Sept 7, 2014 120 120 120 What point minimizes the distance to the three points

9/7/2014

9

Resistance to Outliers

Robustness to systematic outliers (e.g. ghost edges)

Page 10: Iterative Reweighted Least Squares · 2014-09-06 · 9/7/2014 1 Iterative Reweighted Least Squares ECCV, Sept 7, 2014 120 120 120 What point minimizes the distance to the three points

9/7/2014

10

Written without the squares

Define weights

Minimize weighted cost

Start with initial value x(0)

Define weights w(t) from x(t)

Solve WLS problemx(t+1)= argmin C(x, w(t))

t=0

t = t+1

IRLS Algorithm

Page 11: Iterative Reweighted Least Squares · 2014-09-06 · 9/7/2014 1 Iterative Reweighted Least Squares ECCV, Sept 7, 2014 120 120 120 What point minimizes the distance to the three points

9/7/2014

11

Start with initial value x(0)

Define weights w(t) from x(t)

Solve WLS problemx(t+1)= argmin C(x, w(t))

t=0

t = t+1

IRLS Algorithm

Start with initial value x(0)

Define weights w(t) from x(t)

Solve WLS problemx(t+1)= argmin C(x, w(t))

t=0

t = t+1

IRLS Algorithm

Page 12: Iterative Reweighted Least Squares · 2014-09-06 · 9/7/2014 1 Iterative Reweighted Least Squares ECCV, Sept 7, 2014 120 120 120 What point minimizes the distance to the three points

9/7/2014

12

Start with initial value x(0)

Define weights w(t) from x(t)

Solve WLS problemx(t+1)= argmin C(x, w(t))

t=0

t = t+1

IRLS Algorithm

Start with initial value x(0)

Define weights w(t) from x(t)

Solve WLS problemx(t+1)= argmin C(x, w(t))

t=0

t = t+1

IRLS Algorithm

Page 13: Iterative Reweighted Least Squares · 2014-09-06 · 9/7/2014 1 Iterative Reweighted Least Squares ECCV, Sept 7, 2014 120 120 120 What point minimizes the distance to the three points

9/7/2014

13

No square !!

Robust cost function

Required weights

Sum of distances

Page 14: Iterative Reweighted Least Squares · 2014-09-06 · 9/7/2014 1 Iterative Reweighted Least Squares ECCV, Sept 7, 2014 120 120 120 What point minimizes the distance to the three points

9/7/2014

14

Weighted residual sum decreases

Robust residual sum decreases

Page 15: Iterative Reweighted Least Squares · 2014-09-06 · 9/7/2014 1 Iterative Reweighted Least Squares ECCV, Sept 7, 2014 120 120 120 What point minimizes the distance to the three points

9/7/2014

15

SupergradientA concave function always has a supergradient

Definition of supergradient

Weighted squared residual decreases

Robust costdecreases

Page 16: Iterative Reweighted Least Squares · 2014-09-06 · 9/7/2014 1 Iterative Reweighted Least Squares ECCV, Sept 7, 2014 120 120 120 What point minimizes the distance to the three points

9/7/2014

16

Where gradient descent does not converge to a minimum

Page 17: Iterative Reweighted Least Squares · 2014-09-06 · 9/7/2014 1 Iterative Reweighted Least Squares ECCV, Sept 7, 2014 120 120 120 What point minimizes the distance to the three points

9/7/2014

17

L1

0.2 0.4 0.6 0.8 1.0

0.2

0.4

0.6

0.8

1.0

0.2 0.4 0.6 0.8 1.0

0.2

0.4

0.6

0.8

1.0

0.2 0.4 0.6 0.8 1.0

2

4

6

8

10

12Advantage: Robust

Disadvantage: • Function not differentiable• Weights not defined at 0• Can stop at non‐minimum.

Page 18: Iterative Reweighted Least Squares · 2014-09-06 · 9/7/2014 1 Iterative Reweighted Least Squares ECCV, Sept 7, 2014 120 120 120 What point minimizes the distance to the three points

9/7/2014

18

Lq

0.2 0.4 0.6 0.8 1.0

0.2

0.4

0.6

0.8

1.0

0.2 0.4 0.6 0.8 1.0

0.2

0.4

0.6

0.8

1.0

0.2 0.4 0.6 0.8 1.0

2

3

4

5

6

7

8

Advantage: • Robust• Cost function differentiable everywhere

Disadvantage: • Weights not defined at 0• Can stop at non‐minimum.

Huber

0.2 0.4 0.6 0.8 1.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.2 0.4 0.6 0.8 1.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.0 0.5 1.0 1.5 2.0 2.5 3.0

0.5

1.0

1.5

2.0

Advantage: • Robust• Cost function differentiable• Weights defined at zero• Convex• Guaranteed to converge (at least to local 

minimum)

Page 19: Iterative Reweighted Least Squares · 2014-09-06 · 9/7/2014 1 Iterative Reweighted Least Squares ECCV, Sept 7, 2014 120 120 120 What point minimizes the distance to the three points

9/7/2014

19

Pseudo Huber

0.2 0.4 0.6 0.8 1.0

0.1

0.2

0.3

0.4

0.5

0.6

0.2 0.4 0.6 0.8 1.0

0.1

0.2

0.3

0.4

0.5

0.6

0.0 0.5 1.0 1.5 2.0 2.5 3.0

0.5

1.0

1.5

2.0 Advantage: • Robust• Cost function differentiable• Weights defined at zero• Convex• Guaranteed to converge (at least to local 

minimum)

Cauchy

0.5 1.0 1.5 2.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.2 0.4 0.6 0.8 1.0

0.1

0.2

0.3

0.4

0.0 0.2 0.4 0.6 0.8 1.0

0.5

1.0

1.5

2.0 Advantage: • Robust• Cost function differentiable• Weights defined at zero• Guaranteed to converge (at least to local 

minimum)

Disadvantage:• Non‐convex• Increased number of local minima

Page 20: Iterative Reweighted Least Squares · 2014-09-06 · 9/7/2014 1 Iterative Reweighted Least Squares ECCV, Sept 7, 2014 120 120 120 What point minimizes the distance to the three points

9/7/2014

20

Blake Zisserman

0.5 1.0 1.5 2.0 2.5 3.0

0.2

0.4

0.6

0.8

0.5 1.0 1.5 2.0 2.5 3.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.5 1.0 1.5 2.0 2.5 3.0

0.2

0.4

0.6

0.8

1.0

Advantage: • Very robust to outliers• Cost function differentiable• Weights defined at zero• Guaranteed to converge (at least to local 

minimum)

Disadvantage:• Non‐convex• Increased number of local minima

1 2 3 4 5

0.2

0.4

0.6

0.8

1.0

1.2

1.4

1 2 3 4 5

0.2

0.4

0.6

0.8

1.0

1.2

1 2 3 4 5

0.2

0.4

0.6

0.8

1.0

1.2

1.4

Corrupted Gaussian

Advantage: • Robust• Cost function differentiable• Weights defined at zero• Guaranteed to converge (at least to local 

minimum)

Disadvantage:• Non‐convex• Increased number of local minima

Page 21: Iterative Reweighted Least Squares · 2014-09-06 · 9/7/2014 1 Iterative Reweighted Least Squares ECCV, Sept 7, 2014 120 120 120 What point minimizes the distance to the three points

9/7/2014

21

Example.  L1 Regression

Page 22: Iterative Reweighted Least Squares · 2014-09-06 · 9/7/2014 1 Iterative Reweighted Least Squares ECCV, Sept 7, 2014 120 120 120 What point minimizes the distance to the three points

9/7/2014

22

Page 23: Iterative Reweighted Least Squares · 2014-09-06 · 9/7/2014 1 Iterative Reweighted Least Squares ECCV, Sept 7, 2014 120 120 120 What point minimizes the distance to the three points

9/7/2014

23

Page 24: Iterative Reweighted Least Squares · 2014-09-06 · 9/7/2014 1 Iterative Reweighted Least Squares ECCV, Sept 7, 2014 120 120 120 What point minimizes the distance to the three points

9/7/2014

24

Page 25: Iterative Reweighted Least Squares · 2014-09-06 · 9/7/2014 1 Iterative Reweighted Least Squares ECCV, Sept 7, 2014 120 120 120 What point minimizes the distance to the three points

9/7/2014

25

Page 26: Iterative Reweighted Least Squares · 2014-09-06 · 9/7/2014 1 Iterative Reweighted Least Squares ECCV, Sept 7, 2014 120 120 120 What point minimizes the distance to the three points

9/7/2014

26

Page 27: Iterative Reweighted Least Squares · 2014-09-06 · 9/7/2014 1 Iterative Reweighted Least Squares ECCV, Sept 7, 2014 120 120 120 What point minimizes the distance to the three points

9/7/2014

27

Page 28: Iterative Reweighted Least Squares · 2014-09-06 · 9/7/2014 1 Iterative Reweighted Least Squares ECCV, Sept 7, 2014 120 120 120 What point minimizes the distance to the three points

9/7/2014

28

Possible application

Page 29: Iterative Reweighted Least Squares · 2014-09-06 · 9/7/2014 1 Iterative Reweighted Least Squares ECCV, Sept 7, 2014 120 120 120 What point minimizes the distance to the three points

9/7/2014

29

Example.  Averaging Rotations

Page 30: Iterative Reweighted Least Squares · 2014-09-06 · 9/7/2014 1 Iterative Reweighted Least Squares ECCV, Sept 7, 2014 120 120 120 What point minimizes the distance to the three points

9/7/2014

30

Page 31: Iterative Reweighted Least Squares · 2014-09-06 · 9/7/2014 1 Iterative Reweighted Least Squares ECCV, Sept 7, 2014 120 120 120 What point minimizes the distance to the three points

9/7/2014

31

Angle between two quaternions is half the angle between the corresponding rotations, defined by

All rotations within a delta‐neighbourhood of a reference rotation form a circle on the quaternion sphere.

Isometry of Rotations and Quaternions

Page 32: Iterative Reweighted Least Squares · 2014-09-06 · 9/7/2014 1 Iterative Reweighted Least Squares ECCV, Sept 7, 2014 120 120 120 What point minimizes the distance to the three points

9/7/2014

32

Flatten out the meridians (longitude lines)

Azimuthal Equidistant Projection

Angle‐axis representation of Rotations

Rotations are represented by a ball of radius pi in 3‐Dimensional space.

Page 33: Iterative Reweighted Least Squares · 2014-09-06 · 9/7/2014 1 Iterative Reweighted Least Squares ECCV, Sept 7, 2014 120 120 120 What point minimizes the distance to the three points

9/7/2014

33

Relative rotations are computed between some nodes in the graph.

Initialization:  Propagate rotations estimates across a tree.

Page 34: Iterative Reweighted Least Squares · 2014-09-06 · 9/7/2014 1 Iterative Reweighted Least Squares ECCV, Sept 7, 2014 120 120 120 What point minimizes the distance to the three points

9/7/2014

34

Orientation of each node in turn is recomputed, given the known orientation of its neigbours, and the relative orientation.

This involves “single‐rotation averaging”.

Page 35: Iterative Reweighted Least Squares · 2014-09-06 · 9/7/2014 1 Iterative Reweighted Least Squares ECCV, Sept 7, 2014 120 120 120 What point minimizes the distance to the three points

9/7/2014

35

o 569 Imageso 280,000 pointso 42000 pairs of overlapping images (more    than 30 points in common)

Task:  Find the orientations of all cameras.

Extension: Optimization on Riemannian Manifolds

Page 36: Iterative Reweighted Least Squares · 2014-09-06 · 9/7/2014 1 Iterative Reweighted Least Squares ECCV, Sept 7, 2014 120 120 120 What point minimizes the distance to the three points

9/7/2014

36

Page 37: Iterative Reweighted Least Squares · 2014-09-06 · 9/7/2014 1 Iterative Reweighted Least Squares ECCV, Sept 7, 2014 120 120 120 What point minimizes the distance to the three points

9/7/2014

37

Page 38: Iterative Reweighted Least Squares · 2014-09-06 · 9/7/2014 1 Iterative Reweighted Least Squares ECCV, Sept 7, 2014 120 120 120 What point minimizes the distance to the three points

9/7/2014

38

Image from http://en.wikipedia.org/wiki/File:Tangentialvektor.svg

Page 39: Iterative Reweighted Least Squares · 2014-09-06 · 9/7/2014 1 Iterative Reweighted Least Squares ECCV, Sept 7, 2014 120 120 120 What point minimizes the distance to the three points

9/7/2014

39

Page 40: Iterative Reweighted Least Squares · 2014-09-06 · 9/7/2014 1 Iterative Reweighted Least Squares ECCV, Sept 7, 2014 120 120 120 What point minimizes the distance to the three points

9/7/2014

40

Page 41: Iterative Reweighted Least Squares · 2014-09-06 · 9/7/2014 1 Iterative Reweighted Least Squares ECCV, Sept 7, 2014 120 120 120 What point minimizes the distance to the three points

9/7/2014

41

Page 42: Iterative Reweighted Least Squares · 2014-09-06 · 9/7/2014 1 Iterative Reweighted Least Squares ECCV, Sept 7, 2014 120 120 120 What point minimizes the distance to the three points

9/7/2014

42

Hyperbolic (negative curvature) manifold