root finding cos 323. 1-d root finding given some function, find location where f ( x )=0given some...
TRANSCRIPT
1-D Root Finding1-D Root Finding
• Given some function, find location Given some function, find location where where ff((xx)=0)=0
• Need:Need:– Starting position Starting position xx00, hopefully close to , hopefully close to
solutionsolution
– Ideally, points that Ideally, points that bracketbracket the root the rootff((xx++) > 0) > 0
ff((xx––) < 0) < 0
1-D Root Finding1-D Root Finding
• Given some function, find location Given some function, find location where where ff((xx)=0)=0
• Need:Need:– Starting position Starting position xx00, hopefully close to , hopefully close to
solutionsolution
– Ideally, points that Ideally, points that bracketbracket the root the root
– Well-behaved functionWell-behaved function
What Goes Wrong?What Goes Wrong?
Tangent point:Tangent point:very difficultvery difficult
to findto find
Singularity:Singularity:brackets don’tbrackets don’tsurround rootsurround root
Pathological case:Pathological case:infinite number ofinfinite number of
roots – e.g. sin(1/x)roots – e.g. sin(1/x)
Example: Press et al., Example: Press et al., Numerical Numerical Recipes in CRecipes in C
EquationEquation (3.0.1), p. 105(3.0.1), p. 105 22
2 ln((Pi - x) )2 ln((Pi - x) ) f := x -> 3 x + ------------- + 1f := x -> 3 x + ------------- + 1
44 PiPi
> evalf(f(Pi-1e-10));> evalf(f(Pi-1e-10));30.1360472472915692...30.1360472472915692...
> evalf(f(Pi-1e-100));> evalf(f(Pi-1e-100));25.8811536623525653...25.8811536623525653...
> evalf(f(Pi-1e-600));> evalf(f(Pi-1e-600));2.24285595777501258...2.24285595777501258...
> evalf(f(Pi-1e-700));> evalf(f(Pi-1e-700));-2.4848035831404979...-2.4848035831404979...
> evalf(f(Pi+1e-700));> evalf(f(Pi+1e-700));-2.4848035831404979...-2.4848035831404979...
> evalf(f(Pi+1e-600));> evalf(f(Pi+1e-600));2.24285595777501258...2.24285595777501258...
> evalf(f(Pi+1e-100));> evalf(f(Pi+1e-100));25.8811536623525653...25.8811536623525653...
> evalf(f(Pi+1e-10));> evalf(f(Pi+1e-10));30.1360472510614803...30.1360472510614803...
Bisection MethodBisection Method
• Given points Given points xx++ and and xx–– that bracket a root, that bracket a root,
findfindxxhalfhalf = ½ ( = ½ (xx+++ + xx––))
and evaluate and evaluate ff((xxhalfhalf))
• If positive, If positive, xx+ + xxhalfhalf else else xx– – xxhalfhalf
• Stop when Stop when xx+ + and and xx–– close enough close enough
• If function is continuous, this If function is continuous, this willwill succeed succeedin finding in finding somesome root root
BisectionBisection
• Very robust methodVery robust method
• Convergence rate:Convergence rate:– Error bounded by size of [Error bounded by size of [xx++… … xx––] interval] interval
– Interval shrinks in half at each iterationInterval shrinks in half at each iteration
– Therefore, error cut in half at each iteration:Therefore, error cut in half at each iteration:||nn+1+1|| = = ½½ ||nn||
– This is called “linear convergence”This is called “linear convergence”
– One extra bit of accuracy in One extra bit of accuracy in xx at each at each iterationiteration
Faster Root-FindingFaster Root-Finding
• Fancier methods get super-linear Fancier methods get super-linear convergenceconvergence– Typical approach: model function locally by Typical approach: model function locally by
something whose root you can find exactlysomething whose root you can find exactly
– Model didn’t match function exactly, so Model didn’t match function exactly, so iterateiterate
– In many cases, these are less safe than In many cases, these are less safe than bisectionbisection
Secant MethodSecant Method
• Simple extension to bisection: Simple extension to bisection: interpolate or extrapolate through two interpolate or extrapolate through two most recent pointsmost recent points
11
2233
44
Secant MethodSecant Method
• Faster than bisection:Faster than bisection:||nn+1+1|| = = const.const. ||nn||1.61.6
• Faster than linear: number of correct bits Faster than linear: number of correct bits multiplied by 1.6multiplied by 1.6
• Drawback: the above only true if Drawback: the above only true if sufficiently closesufficiently close to a root of a to a root of a sufficiently sufficiently smoothsmooth function function– Does not guarantee that root remains Does not guarantee that root remains
bracketedbracketed
False Position MethodFalse Position Method
• Similar to secant, but guarantee Similar to secant, but guarantee bracketingbracketing
• Stable, but linear in bad casesStable, but linear in bad cases
11
2233
44
Other Interpolation StrategiesOther Interpolation Strategies
• Ridders’s method: fit exponential toRidders’s method: fit exponential toff((xx++), ), ff((xx––), and ), and ff((xxhalfhalf))
• Van Wijngaarden-Dekker-Brent method:Van Wijngaarden-Dekker-Brent method:inverse quadratic fit to 3 most recent inverse quadratic fit to 3 most recent pointspointsif within bracket, else bisectionif within bracket, else bisection
• Both of these Both of these safesafe if function is nasty, but if function is nasty, butfastfast (super-linear) if function is nice (super-linear) if function is nice
Newton-RaphsonNewton-Raphson
• Best-known algorithm for getting Best-known algorithm for getting quadraticquadraticconvergence when derivative is easy to convergence when derivative is easy to evaluateevaluate
• Another variant on the extrapolation Another variant on the extrapolation themetheme
)(
)(1
n
nnn xf
xfxx
)(
)(1
n
nnn xf
xfxx
1122
33
44
Slope = derivative at 1Slope = derivative at 1
Newton-RaphsonNewton-Raphson
• Begin with Taylor seriesBegin with Taylor series
• Divide by derivative (can’t be zero!)Divide by derivative (can’t be zero!)
0...2
)()()()( 2
wantn
nnn
xfxfxfxf
0...
2
)()()()( 2
wantn
nnn
xfxfxfxf
21
2
2
2
~)(2
)(
0)(2
)(
0)(2
)(
)(
)(
nnn
nNewton
n
nNewton
n
n
n
n
xf
xf
xf
xf
xf
xf
xf
xf
21
2
2
2
~)(2
)(
0)(2
)(
0)(2
)(
)(
)(
nnn
nNewton
n
nNewton
n
n
n
n
xf
xf
xf
xf
xf
xf
xf
xf
Newton-RaphsonNewton-Raphson
• Method fragile: can easily get confusedMethod fragile: can easily get confused
• Good starting point criticalGood starting point critical– Newton popular for “polishing off” a root Newton popular for “polishing off” a root
found approximately using a more robust found approximately using a more robust methodmethod
Newton-Raphson ConvergenceNewton-Raphson Convergence
• Can talk about “basin of convergence”:Can talk about “basin of convergence”:range of range of xx00 for which method finds a for which method finds a
rootroot
• Can be extremely complex:Can be extremely complex:here’s an examplehere’s an examplein 2-D with 4 rootsin 2-D with 4 roots
Yale siteYale site
D.W. HyattD.W. Hyatt
Popular Example of Newton: Popular Example of Newton: Square RootSquare Root
• Let Let ff((xx) = ) = xx22 – – aa: zero of this is square : zero of this is square root of root of aa
• ff’(’(xx) = 2) = 2xx, so Newton iteration is, so Newton iteration is
• ““Divide and average” methodDivide and average” method
nxa
nn
nnn x
x
axxx
2
1
2
1 2
nxa
nn
nnn x
x
axxx
2
1
2
1 2
Reciprocal via NewtonReciprocal via Newton
• Division is slowest of basic operationsDivision is slowest of basic operations
• On some computers, hardware divide On some computers, hardware divide not available (!): simulate in softwarenot available (!): simulate in software
• Need only subtract and multiplyNeed only subtract and multiply
nn
x
xnn
x
x
bba
bxxb
xx
xf
bxf
a
2
)(
0)(
*
2
2
1
1
1
1
1
1
nn
x
xnn
x
x
bba
bxxb
xx
xf
bxf
a
2
)(
0)(
*
2
2
1
1
1
1
1
1
Rootfinding in >1DRootfinding in >1D
• Behavior can be complex: e.g. in 2DBehavior can be complex: e.g. in 2D
0),(want
yxg 0),(want
yxg
0),(want
yxf 0),(want
yxf
Rootfinding in >1DRootfinding in >1D
• Can’t bracket and bisectCan’t bracket and bisect
• Result: few general methodsResult: few general methods
Newton in Higher DimensionsNewton in Higher Dimensions
• Start withStart with
• Write as vector-valued functionWrite as vector-valued function
0),(
0),(want
want
yxg
yxf
0),(
0),(want
want
yxg
yxf
),(
),()(
yxg
yxfnxf
),(
),()(
yxg
yxfnxf
Newton in Higher DimensionsNewton in Higher Dimensions
• Expand in terms of Taylor seriesExpand in terms of Taylor series
• ff’’ is a Jacobian is a Jacobian
0...)()()(want
nnn δxfxfδxf 0...)()()(want
nnn δxfxfδxf
yxn
ffJxf )( yxn
ffJxf )(
Newton in Higher DimensionsNewton in Higher Dimensions
• Solve for Solve for
• Requires matrix inversion (we’ll see this Requires matrix inversion (we’ll see this later)later)
• Often fragile, must be carefulOften fragile, must be careful– Keep track of whether error decreasesKeep track of whether error decreases
– If not, try a smaller step in direction If not, try a smaller step in direction
)()(1nn xfxJδ )()(1nn xfxJδ