nonlinear systems - computer graphics · cs 205a: mathematical methods nonlinear systems 8 / 24...
TRANSCRIPT
Nonlinearity Root-finding Bisection Fixed Point Iteration Newton’s Method Secant Method Conclusion
Nonlinear Systems
CS 205A:Mathematical Methods for Robotics, Vision, and Graphics
Justin Solomon
CS 205A: Mathematical Methods Nonlinear Systems 1 / 24
Nonlinearity Root-finding Bisection Fixed Point Iteration Newton’s Method Secant Method Conclusion
Part III: Nonlinear Problems
Not all numerical problemscan be solved with \ in
Matlab.
CS 205A: Mathematical Methods Nonlinear Systems 2 / 24
Nonlinearity Root-finding Bisection Fixed Point Iteration Newton’s Method Secant Method Conclusion
Question
Have we already seen anonlinear problem?
minimize ‖A~x‖2such that ‖~x‖2 = 1←− nonlinear!
CS 205A: Mathematical Methods Nonlinear Systems 3 / 24
Nonlinearity Root-finding Bisection Fixed Point Iteration Newton’s Method Secant Method Conclusion
Question
Have we already seen anonlinear problem?
minimize ‖A~x‖2such that ‖~x‖2 = 1←− nonlinear!
CS 205A: Mathematical Methods Nonlinear Systems 3 / 24
Nonlinearity Root-finding Bisection Fixed Point Iteration Newton’s Method Secant Method Conclusion
Root-Finding Problem
Given: f : Rn→ Rm
Find: ~x∗ with f (~x∗) = ~0
CS 205A: Mathematical Methods Nonlinear Systems 4 / 24
Nonlinearity Root-finding Bisection Fixed Point Iteration Newton’s Method Secant Method Conclusion
Issue: Regularizing Assumptions
f (x) =
{−1 x ≤ 01 x > 0
f (x) =
{−1 x ∈ Q1 otherwise
CS 205A: Mathematical Methods Nonlinear Systems 5 / 24
Nonlinearity Root-finding Bisection Fixed Point Iteration Newton’s Method Secant Method Conclusion
Issue: Regularizing Assumptions
f (x) =
{−1 x ≤ 01 x > 0
f (x) =
{−1 x ∈ Q1 otherwise
CS 205A: Mathematical Methods Nonlinear Systems 5 / 24
Nonlinearity Root-finding Bisection Fixed Point Iteration Newton’s Method Secant Method Conclusion
Typical Regularizing AssumptionsContinuous
f(~x)→ f(~y) as ~x→ ~y
Lipschitz
‖f(~x)− f(~y)‖ ≤ C‖~x− ~y‖
Differentiable
Df(~x) exists for all ~x
Ck
k derivatives exist and are continuous
CS 205A: Mathematical Methods Nonlinear Systems 6 / 24
Nonlinearity Root-finding Bisection Fixed Point Iteration Newton’s Method Secant Method Conclusion
Typical Regularizing AssumptionsContinuous
f(~x)→ f(~y) as ~x→ ~y
Lipschitz
‖f(~x)− f(~y)‖ ≤ C‖~x− ~y‖
Differentiable
Df(~x) exists for all ~x
Ck
k derivatives exist and are continuous
CS 205A: Mathematical Methods Nonlinear Systems 6 / 24
Nonlinearity Root-finding Bisection Fixed Point Iteration Newton’s Method Secant Method Conclusion
Typical Regularizing AssumptionsContinuous
f(~x)→ f(~y) as ~x→ ~y
Lipschitz
‖f(~x)− f(~y)‖ ≤ C‖~x− ~y‖
Differentiable
Df(~x) exists for all ~x
Ck
k derivatives exist and are continuous
CS 205A: Mathematical Methods Nonlinear Systems 6 / 24
Nonlinearity Root-finding Bisection Fixed Point Iteration Newton’s Method Secant Method Conclusion
Typical Regularizing AssumptionsContinuous
f(~x)→ f(~y) as ~x→ ~y
Lipschitz
‖f(~x)− f(~y)‖ ≤ C‖~x− ~y‖
Differentiable
Df(~x) exists for all ~x
Ck
k derivatives exist and are continuous
CS 205A: Mathematical Methods Nonlinear Systems 6 / 24
Nonlinearity Root-finding Bisection Fixed Point Iteration Newton’s Method Secant Method Conclusion
Today
f : R→ RCS 205A: Mathematical Methods Nonlinear Systems 7 / 24
Nonlinearity Root-finding Bisection Fixed Point Iteration Newton’s Method Secant Method Conclusion
Property of Continuous Functions
Intermediate Value TheoremSuppose f : [a, b]→ R is continuous. Suppose
f (x) < u < f (y). Then, there exists z between
x and y such that f (z) = u.
Used in Homework 1!
CS 205A: Mathematical Methods Nonlinear Systems 8 / 24
Nonlinearity Root-finding Bisection Fixed Point Iteration Newton’s Method Secant Method Conclusion
Property of Continuous Functions
Intermediate Value TheoremSuppose f : [a, b]→ R is continuous. Suppose
f (x) < u < f (y). Then, there exists z between
x and y such that f (z) = u.
Used in Homework 1!
CS 205A: Mathematical Methods Nonlinear Systems 8 / 24
Nonlinearity Root-finding Bisection Fixed Point Iteration Newton’s Method Secant Method Conclusion
Reasonable Starting Point
f (`) · f (r) < 0
CS 205A: Mathematical Methods Nonlinear Systems 9 / 24
Nonlinearity Root-finding Bisection Fixed Point Iteration Newton’s Method Secant Method Conclusion
Bisection Algorithm
1. Compute c = `+r/2.
2. If f (c) = 0, return x∗ = c.
3. If f (`) · f (c) < 0, take r ← c. Otherwise
take `← c.
4. Return to step ?? until |r − `| < ε; then
return c.
CS 205A: Mathematical Methods Nonlinear Systems 10 / 24
Nonlinearity Root-finding Bisection Fixed Point Iteration Newton’s Method Secant Method Conclusion
Two Important Questions
1. Does it converge?
Yes! Unconditionally.
2. How quickly?
CS 205A: Mathematical Methods Nonlinear Systems 11 / 24
Nonlinearity Root-finding Bisection Fixed Point Iteration Newton’s Method Secant Method Conclusion
Two Important Questions
1. Does it converge?Yes! Unconditionally.
2. How quickly?
CS 205A: Mathematical Methods Nonlinear Systems 11 / 24
Nonlinearity Root-finding Bisection Fixed Point Iteration Newton’s Method Secant Method Conclusion
Two Important Questions
1. Does it converge?Yes! Unconditionally.
2. How quickly?
CS 205A: Mathematical Methods Nonlinear Systems 11 / 24
Nonlinearity Root-finding Bisection Fixed Point Iteration Newton’s Method Secant Method Conclusion
Convergence Analysis
Examine Ek with|xk − x∗| < Ek.
CS 205A: Mathematical Methods Nonlinear Systems 12 / 24
Nonlinearity Root-finding Bisection Fixed Point Iteration Newton’s Method Secant Method Conclusion
Bisection: Linear Convergence
Ek+1 ≤ 12Ek
for Ek ≡ |rk − `k|
CS 205A: Mathematical Methods Nonlinear Systems 13 / 24
Nonlinearity Root-finding Bisection Fixed Point Iteration Newton’s Method Secant Method Conclusion
Fixed Points
g(x∗) = x∗
Question:Same as root-finding?
CS 205A: Mathematical Methods Nonlinear Systems 14 / 24
Nonlinearity Root-finding Bisection Fixed Point Iteration Newton’s Method Secant Method Conclusion
Fixed Points
g(x∗) = x∗
Question:Same as root-finding?
CS 205A: Mathematical Methods Nonlinear Systems 14 / 24
Nonlinearity Root-finding Bisection Fixed Point Iteration Newton’s Method Secant Method Conclusion
Simple Strategy
xk+1 = g(xk)
CS 205A: Mathematical Methods Nonlinear Systems 15 / 24
Nonlinearity Root-finding Bisection Fixed Point Iteration Newton’s Method Secant Method Conclusion
Convergence Criterion
Ek ≡ |xk − x∗|= |g(xk−1)− g(x∗)|
≤ C|xk−1 − x∗|if g is Lipschitz
= CEk−1
CS 205A: Mathematical Methods Nonlinear Systems 16 / 24
Nonlinearity Root-finding Bisection Fixed Point Iteration Newton’s Method Secant Method Conclusion
Convergence Criterion
Ek ≡ |xk − x∗|= |g(xk−1)− g(x∗)|≤ C|xk−1 − x∗|
if g is Lipschitz
= CEk−1CS 205A: Mathematical Methods Nonlinear Systems 16 / 24
Nonlinearity Root-finding Bisection Fixed Point Iteration Newton’s Method Secant Method Conclusion
Alternative Criterion
Lipschitz near x∗ with goodstarting point.
e.g. C1 with |g′(x∗)| < 1
CS 205A: Mathematical Methods Nonlinear Systems 17 / 24
Nonlinearity Root-finding Bisection Fixed Point Iteration Newton’s Method Secant Method Conclusion
Alternative Criterion
Lipschitz near x∗ with goodstarting point.
e.g. C1 with |g′(x∗)| < 1
CS 205A: Mathematical Methods Nonlinear Systems 17 / 24
Nonlinearity Root-finding Bisection Fixed Point Iteration Newton’s Method Secant Method Conclusion
Convergence Rate of Fixed Point
When it converges...Always linear (why?)
Often quadratic!
CS 205A: Mathematical Methods Nonlinear Systems 18 / 24
Nonlinearity Root-finding Bisection Fixed Point Iteration Newton’s Method Secant Method Conclusion
Convergence Rate of Fixed Point
When it converges...Always linear (why?)
Often quadratic!
CS 205A: Mathematical Methods Nonlinear Systems 18 / 24
Nonlinearity Root-finding Bisection Fixed Point Iteration Newton’s Method Secant Method Conclusion
Newton’s Method
xk+1 = xk −f (xk)
f ′(xk)
Fixed point iteration on
g(x) ≡ x− f (x)
f ′(x)
CS 205A: Mathematical Methods Nonlinear Systems 19 / 24
Nonlinearity Root-finding Bisection Fixed Point Iteration Newton’s Method Secant Method Conclusion
Newton’s Method
xk+1 = xk −f (xk)
f ′(xk)
Fixed point iteration on
g(x) ≡ x− f (x)
f ′(x)CS 205A: Mathematical Methods Nonlinear Systems 19 / 24
Nonlinearity Root-finding Bisection Fixed Point Iteration Newton’s Method Secant Method Conclusion
Convergence of Newton
Simple Root
A root x∗ with f ′(x∗) 6= 0.
Quadratic convergence in this case!
Higher-order approximations?
CS 205A: Mathematical Methods Nonlinear Systems 20 / 24
Nonlinearity Root-finding Bisection Fixed Point Iteration Newton’s Method Secant Method Conclusion
Convergence of Newton
Simple Root
A root x∗ with f ′(x∗) 6= 0.
Quadratic convergence in this case!
Higher-order approximations?
CS 205A: Mathematical Methods Nonlinear Systems 20 / 24
Nonlinearity Root-finding Bisection Fixed Point Iteration Newton’s Method Secant Method Conclusion
Convergence of Newton
Simple Root
A root x∗ with f ′(x∗) 6= 0.
Quadratic convergence in this case!
Higher-order approximations?
CS 205A: Mathematical Methods Nonlinear Systems 20 / 24
Nonlinearity Root-finding Bisection Fixed Point Iteration Newton’s Method Secant Method Conclusion
Issue
Differentiation is hard!
CS 205A: Mathematical Methods Nonlinear Systems 21 / 24
Nonlinearity Root-finding Bisection Fixed Point Iteration Newton’s Method Secant Method Conclusion
Secant Method
xk+1 = xk −f (xk)(xk − xk−1)
f (xk)− f (xk−1)
Trivia:Converges at rate 1+
√5
2 ≈ 1.6180339887 . . .
(“Golden Ratio”)
CS 205A: Mathematical Methods Nonlinear Systems 22 / 24
Nonlinearity Root-finding Bisection Fixed Point Iteration Newton’s Method Secant Method Conclusion
Secant Method
xk+1 = xk −f (xk)(xk − xk−1)
f (xk)− f (xk−1)
Trivia:Converges at rate 1+
√5
2 ≈ 1.6180339887 . . .
(“Golden Ratio”)
CS 205A: Mathematical Methods Nonlinear Systems 22 / 24
Nonlinearity Root-finding Bisection Fixed Point Iteration Newton’s Method Secant Method Conclusion
Secant Method
xk+1 = xk −f (xk)(xk − xk−1)
f (xk)− f (xk−1)
Trivia:Converges at rate 1+
√5
2 ≈ 1.6180339887 . . .
(“Golden Ratio”)
CS 205A: Mathematical Methods Nonlinear Systems 22 / 24
Nonlinearity Root-finding Bisection Fixed Point Iteration Newton’s Method Secant Method Conclusion
Hybrid Methods
Want: Convergence rate of secant/Newton with
convergence guarantees of bisection
e.g. Dekker’s Method: Take secant step if it is
in the bracket, bisection step otherwise
CS 205A: Mathematical Methods Nonlinear Systems 23 / 24
Nonlinearity Root-finding Bisection Fixed Point Iteration Newton’s Method Secant Method Conclusion
Hybrid Methods
Want: Convergence rate of secant/Newton with
convergence guarantees of bisection
e.g. Dekker’s Method: Take secant step if it is
in the bracket, bisection step otherwise
CS 205A: Mathematical Methods Nonlinear Systems 23 / 24
Nonlinearity Root-finding Bisection Fixed Point Iteration Newton’s Method Secant Method Conclusion
Single-Variable Conclusion
I Unlikely to solve exactly, so we settle for
iterative methodsI Must check that method converges at allI Convergence rates:
I Linear: Ek+1 ≤ CEk for some 0 ≤ C < 1I Superlinear: Ek+1 ≤ CEr
k for some r > 1I Quadratic: r = 2I Cubic: r = 3
I Time per iteration also important
Next
CS 205A: Mathematical Methods Nonlinear Systems 24 / 24