Ordinary Least-Squares

Emmanuel IarussiInria

2

Many graphics problems can be seen as finding the best set of parameters for a model, given some data

Surface reconstruction

3

Color diffusion

Many graphics problems can be seen as finding the best set of parameters for a model, given some data

4

Shape registration

Many graphics problems can be seen as finding the best set of parameters for a model, given some data

5

Temperature (a) Gas pressure (b)

Example: estimation of gas pressure for a given temperature sample

6

Assuming linear relationship

7

Assuming linear relationship

8

Assuming linear relationship

Temperature measurements (m)

Pressure measurements (m)

9

Assuming linear relationship

Temperature measurements (m)

Pressure measurements (m)

=

10

Assuming linear relationship

The relationship might not be exact

Temperature measurements (m)

Pressure measurements (m)

=

11

Assuming linear relationship

The relationship might not be exact

Temperature measurements (m)

Pressure measurements (m)

…

=

Match “as best as possible” the observations

12

Assuming linear relationship

The relationship might not be exact Match “as best as possible” the observations

Temperature measurements (m)

Pressure measurements (m)

=

13

Many of CG problems can be formulated as minimizing the sum of squares of the residuals

between some features in the model and the data.

or

14

Matrix notation (observations)

2

2

)(

)()(

)()(

ab

abab

xxe

xx

xabxe

T

iii

We can rewrite the residual function using linear algebra as:

15

Temperature

Gas pressure

Example: estimation of gas pressure for a given temperature and altitude samples

Altitude

16

Multidimensional linear regression, using a model with n parameters

17

Multidimensional linear regression, using a model with n parameters

before+ +…

18

Multidimensional linear regression, using a model with n parameters

Temperature measurements (m)

Pressure measurements (m)

Altitudemeasurements (m)

19

Multidimensional linear regression, using a model with n parameters

Temperature measurements (m)

Pressure measurements (m)

Altitudemeasurements (m)

=

20

=

21

=

22

=

23

Objective: find x subject to minimize:

24

Objective: find x subject to minimize:

25

Objective: find x subject to minimize:

Convex bowl function has minima when:

26

Objective: find x subject to minimize:

Convex bowl function has minima when:

gradient

27

Expanding square term

28

Expanding square term

29

Differentiating with respect to x (gradient)

Expanding square term

30

Expanding square term

Differentiating with respect to x (gradient)

31

Expanding square term

Normal equation

x = (A.transpose()*A).solve(A.transpose()*b);

Differentiating with respect to x (gradient)

32

Expanding square term

Normal equation

x = (A.transpose()*A).solve(A.transpose()*b);

Issue: matrix multiplication

Differentiating with respect to x (gradient)

33

Differentiating with respect to x

Expanding square term

Normal equation

x = (A.transpose()*A).solve(A.transpose()*b);

Issue: matrix multiplication Solution: find expression

34

Let's go back to gradient = 0

35

Let's go back to gradient = 0

Differentiating again with respect to x

36

Let's go back to gradient = 0

Differentiating again with respect to x

37

Let's go back to gradient = 0

Differentiating again with respect to x

hessian

38

Let's go back to gradient = 0

Differentiating again with respect to x

hessian

39

Let's go back to gradient = 0

Differentiating again with respect to x

x = (A.transpose()*A).solve(A.transpose()*b);

Normal equation:

40

Let's go back to gradient = 0

Differentiating again with respect to x

x = (A.transpose()*A).solve(A.transpose()*b);

Normal equation:

41

Let's go back to gradient = 0

Differentiating again with respect to x

x = (A.transpose()*A).solve(A.transpose()*b);

x = (H).solve(c);

Two alternatives for solving:

Constant terms in gradient

42

Example: Diffusion

43

Example: Diffusion

Minimize difference between neighbors

…

44

Example: Diffusion

Minimize difference between neighbors

…

45

Example: Diffusion

Minimize difference between neighbors

…

46

Example: Diffusion

Minimize difference between neighbors

…

47

Solving with normal equation:

…

1. Build A

48

Solving with normal equation:

…

1. Build A

49

Solving with normal equation:

…

1. Build A

50

Solving with normal equation:

…

1. Build A

51

Solving with normal equation:

…

1. Build A

52

Solving with normal equation:

…

1. Build A

53

Solving with normal equation:

2. Compute and

54

Solving with normal equation:

3. Solve using:• Cholesky decomposition• Conjugate Gradient• …

x = (A.transpose()*A).solve(A.transpose()*b);

in Eigen (i.e):

55

Solving with normal equation:

3. Solve using:• Cholesky decomposition• Conjugate Gradient• …

x = (A.transpose()*A).solve(A.transpose()*b);

in Eigen (i.e):

x wil have:

56

Solving with Hessian:

1. Gradient vector: first-order partial derivatives of energy terms

57

Solving with Hessian:

1. Gradient vector: first-order partial derivatives of energy terms

58

Solving with Hessian:

1. Gradient vector: first-order partial derivatives of energy terms

59

Solving with Hessian:

1. Gradient vector: first-order partial derivatives of energy terms

60

Solving with Hessian:

1. Gradient vector: first-order partial derivatives of energy terms

61

Solving with Hessian:

2. Hessian matrix: second-order partial derivatives of energy terms

62

Solving with Hessian:

2. Hessian matrix: second-order partial derivatives of energy terms

63

Solving with Hessian:

2. Hessian matrix: second-order partial derivatives of energy terms

64

Solving with Hessian:

2. Hessian matrix: second-order partial derivatives of energy terms

65

Solving with Hessian:

66

Solving with Hessian:

1. Gradient vector: first-order partial derivatives of energy terms

67

Solving with Hessian:

1. Gradient vector: first-order partial derivatives of energy terms

68

Solving with Hessian:

1. Gradient vector: first-order partial derivatives of energy terms

69

Solving with Hessian:

1. Gradient vector: first-order partial derivatives of energy terms

70

Solving with Hessian:

2. Hessian matrix: second-order partial derivatives of energy terms

71

Solving with Hessian:

2. Hessian matrix: second-order partial derivatives of energy terms

72

Solving with Hessian:

2. Hessian matrix: second-order partial derivatives of energy terms

73

Solving with Hessian:

2. Hessian matrix: second-order partial derivatives of energy terms

74

Solving with Hessian:

75

Solving with Hessian:

To recap:

76

Solving with Hessian:

3. Build system

…

77

Solving with Hessian:

3. Build system

…

78

Solving with Hessian:

3. Build system: add constraint

…

79

Solving with Hessian:

…

3. Build system: add constraint

80

Solving with Hessian:

…

3. Build system: add constraint

81

Solving with Hessian:

…

3. Build system: add constraint

82

Solving with Hessian:

…

3. Build system: add constraint

83

Solving with Hessian:

…

3. Build system: add constraint

84

Solving with Hessian:

…

3. Build system: add constraint

85

Solving with Hessian:

…

3. Build system: add constraint

86

Solving with Hessian:

…

3. Build system: add constraint

87

Solving with Hessian:Solving with Hessian:

4. Solve using:• Cholesky decomposition• Conjugate Gradient• …

88

Solving with Hessian:Solving with Hessian:

x = (H).solve(b`);

in Eigen (i.e):

x will have:

4. Solve using:• Cholesky decomposition• Conjugate Gradient• …

89

References:

• Solving Least Squares Problems. C. L. Lawson and R. J. Hanson. 1974.

• Practical Least Squares for Computer Graphics. Fred Pighin and J.P. Lewis. Course notes 2007.