v. linear & nonlinear least-squareskaroger/numci/2016/files/chap05/...3 v.0. examples,...
Post on 16-Mar-2020
8 Views
Preview:
TRANSCRIPT
1
V. Linear & Nonlinear Least-Squares
V.0.Examples, linear/nonlinear least-squares
V.1.Linear least-squares
V.1.1.Normal equations
V.1.2.The orthogonal transformation method
V.2.Nonlinear least-squares
V.2.1.Newton method
V.2.2.Gauss-Newton method
V. Linear & Nonlinear Least-Squares
2
V.0. Examples, linear/nonlinear least-squares
In practice, one has often to determine unknown parameters of agiven function (from natural laws or model assumptions) through aseries of measurements.
Usually the number of measurements m is much bigger than thenumber of parameters n, i.e.
Measurement
Time
Quantity
Model: Goal: Find1
2
V. Linear & Nonlinear Least-Squares
3
V.0. Examples, linear/nonlinear least-squares
In practice, one has often to determine unknown parameters of agiven function (from natural laws or model assumptions) through aseries of measurements.
Usually the number of measurements m is much bigger than thenumber of parameters n, i.e.
Measurement
Time
Quantity
Model: Goal: Find1
2
Problem: more equations than unknowns! … overdetermined
V. Linear & Nonlinear Least-Squares
4
V.0. Examples, linear/nonlinear least-squares
Goal: Find1
Data:
Linear in model parameters!
Model:
V. Linear & Nonlinear Least-Squares
5
V.0. Examples, linear/nonlinear least-squares
1
Data:
JUSTNOTATION!
Goal: Find
Linear in model parameters!
Model:
V. Linear & Nonlinear Least-Squares
6
V.0. Examples, linear/nonlinear least-squares
● Idea: choose the parameters such that thedistance between the data and the curve isminimal, i.e. the curve that fits best.
V. Linear & Nonlinear Least-Squares
7
V.0. Examples, linear/nonlinear least-squares
● Idea: choose the parameters such that thedistance between the data and the curve isminimal, i.e. the curve that fits best.
Model: 1
V. Linear & Nonlinear Least-Squares
8
V.0. Examples, linear/nonlinear least-squares
● Idea: choose the parameters such that thedistance between the data and the curve isminimal, i.e. the curve that fits best.
MINIMIZE!
V. Linear & Nonlinear Least-Squares
9
V.0. Examples, linear/nonlinear least-squares
● Idea: choose the parameters such that thedistance between the data and the curve isminimal, i.e. the curve that fits best.
● Least squares solution:
Euclidean distance,a.k.a. 2-normDomain of valid parameters
(from application!)
V. Linear & Nonlinear Least-Squares
10
V.0. Examples, linear/nonlinear least-squares
Model: Goal: Find2
Data:
Nonlinear in model parameters!
JUSTNOTATION!
V. Linear & Nonlinear Least-Squares
11
V.0. Examples, linear/nonlinear least-squares
● Idea: choose the parameters such that thedistance between the data and the curve isminimal, i.e. the curve that fits best.
V. Linear & Nonlinear Least-Squares
12
V.0. Examples, linear/nonlinear least-squares
● Idea: choose the parameters such that thedistance between the data and the curve isminimal, i.e. the curve that fits best.
MINIMIZE!
V. Linear & Nonlinear Least-Squares
13
V.0. Examples, linear/nonlinear least-squares
● General problem:
Lin
ear
No
nlin
ear
In parameters!
measurements parameters
Overdetermined!
usually...
V. Linear & Nonlinear Least-Squares
14
V.0. Examples, linear/nonlinear least-squares
● Least squares solution:
● Define scalar-valued function
● Least squares solution:
Linear
Nonlinear
Linear
Nonlinear
V. Linear & Nonlinear Least-Squares
15
V.0. Examples, linear/nonlinear least-squares
● Quiz: linear or nonlinear model?
1)
2) Model parameters:
Model parameters:
V. Linear & Nonlinear Least-Squares
16
V.0. Examples, linear/nonlinear least-squares
● Quiz: linear or nonlinear model?
1)
2) Model parameters:
Model parameters:
V. Linear & Nonlinear Least-Squares
17
V.0. Examples, linear/nonlinear least-squares
● General problem: measurements parameters
Overdetermined!
Least squares solution:
Assumption: has full column rank (linearly indep.)
V. Linear & Nonlinear Least-Squares
18
● Least squares solution:
● Rewrite: Transpose!
Scalar!
~
~
V.1.1. Normal equations
V. Linear & Nonlinear Least-Squares
19
● Gradient of
must vanish at extremum (min/max):
Gradient
(Gauss') Normal equations
Nothing difficult…
~
V.1.1. Normal equations
V. Linear & Nonlinear Least-Squares
20
● Necessary condition:
● Is it a minimum?We have to make sure that the matrix ispositive definite
(Gauss') Normal equations
1
2
Norm (length!)
Unless Because of our assumptionof rank nColumns linearly independent!
Not sufficient!
~
V.1.1. Normal equations
V. Linear & Nonlinear Least-Squares
21
● Geometric interpretation of the normalequations
● It turns out that they lead to a worserconditioned problem, i.e. difficult to solve thenormal equations numerically…Therefore the next method is preferred
.
V.1.1. Normal equations
V. Linear & Nonlinear Least-Squares
22
V.1.2. The orthogonal decomposition method
● Definition:
● Key property:
An orthogonal matrix is a real squarematrix whose columns and rows are orthogonalunit vectors
This means:
Orthogonal matrices leave the Euclidean lengthinvariant
V. Linear & Nonlinear Least-Squares
23
Every matrix with full column rank(the columns are linearly independent) has a so-calledQR-decomposition
where is an orthogonal matrix and an uppertriangular matrix
!Non zero diagonal!
Matlab: [Q,R]=qr(A)
Fact:
V.1.2. The orthogonal decomposition method
V. Linear & Nonlinear Least-Squares
24
Every matrix with full column rank(the columns are linearly independent) has a so-calledQR-decomposition
where is an orthogonal matrix and an uppertriangular matrix
Matlab: [Q,R]=qr(A)
Fact:
Upper triangular
V.1.2. The orthogonal decomposition method
V. Linear & Nonlinear Least-Squares
25
Minimize residuum:
Orthogonal!We still solve the sameminimization problem!!!
V.1.2. The orthogonal decomposition method
V. Linear & Nonlinear Least-Squares
26
Minimize residuum:
Orthogonal!We still solve the sameminimization problem!!!
V.1.2. The orthogonal decomposition method
V. Linear & Nonlinear Least-Squares
27
V.1. Example: linear least-squares
Goal: Find1
Data:
Linear in model parameters!
Model:
V. Linear & Nonlinear Least-Squares
28
V.1. Example: linear least-squares
Goal: Find1
Data:
Model:
Normal equations:
Orthogonal transformation: analogue…
V. Linear & Nonlinear Least-Squares
29
V.2. Nonlinear least-squares
usually...
● General problem: measurements parameters
Overdetermined!
Least squares solution:
V. Linear & Nonlinear Least-Squares
30
● Gradient of
must vanish at extremum (min/max):
Gradient
V.2.1 Newton method
equations
unknowns!
V. Linear & Nonlinear Least-Squares
31
V.2.1 Newton method
Hessian
Gradient
Apply Newton's method to solve
NOT
V. Linear & Nonlinear Least-Squares
32
Linearize the residuum/error equations
V.2.2 Gauss-Newton method
Linearize at some
Linear least squares problem!
V. Linear & Nonlinear Least-Squares
33
V.2.2 Gauss-Newton method
Given an initial guess:
Solve a sequence of linear least squares problems
Until convergence
Small enough
Problem:
...
V. Linear & Nonlinear Least-Squares
34
V.2. Example: nonlinear least-squares
Model: Goal: Find2
Data:
Nonlinear in model parameters!
V. Linear & Nonlinear Least-Squares
35
V.2. Example: nonlinear least-squares
Model: Goal: Find2
Data:
Newton method:
V. Linear & Nonlinear Least-Squares
36
V.2. Example: nonlinear least-squares
Model: Goal: Find2
Data:
Newton method:
Gradient
Two equations in two unknowns!
V. Linear & Nonlinear Least-Squares
37
V.2. Example: nonlinear least-squares
Model: Goal: Find2
Data:
Newton method:
Hessian
V. Linear & Nonlinear Least-Squares
38
V.2. Example: nonlinear least-squares
Model: Goal: Find2
Data:
Newton method:
Iterate!
Initial guess:
V. Linear & Nonlinear Least-Squares
39
V.2. Example: nonlinear least-squares
Model: Goal: Find2
Data:
Gauss-Newton method:
Linearize residuum/error equations:
Linear in parameters now!
V. Linear & Nonlinear Least-Squares
40
V. Summary
● Linear/Nonlinear in parameters
● Overdetermined system of equations!Determine parameters in the least-squares sense...
● Linear:
– Normal equations
– Orthogonal transformation method
– Example
● Nonlinear:
– Newton method
– Gauss-Newton method
– Example
V. Linear & Nonlinear Least-Squares
top related