1

Numerical Methods

Solving Non Linear 1-Dimensional Equations

Root Finding

Given a real valued function f of one variable (say x), the idea is to find an x such that:

f(x) = 0

Root Finding Examples

Find real x such that:

Find real x such that:

Find roots of quadratic eq. use formula

For many other fns not easy to determine their roots For example cannot be solved analytically.

Use approximate solutions techniques.

2 4 3 0x x+ + =( )c o s 2 0x − =

2

1,2

4

2

b b acx

a

− ± −=

2 0ax bx c+ + =

xexf x −= −)(

Graphical methods• If we plot the function f(x)=0, and

observe where it crosses the x axis, we can have a rough estimate of the root.

• Limited by the fact that the results are not precise, but useful to get an initial estimate which can be further refined by other method.

• The graphical methods are also helpful to explore function behaviour.

Trial and Error

Use trial and error: • Guess a value of x and evaluate whether f(x) is

zero!!• If not (as it almost always the case) make

another guess, evaluate f(x) again and determine whether the new value provides a better estimate for the root.

• Repeat process until a guess is obtained that results in f(x) being close to zero

An Algorithmic Approach

• Idea: find a sequence of x1,x2,x3,x4…. so that for some N, xN is “close” toa root.

• that is, |f(xN)| < tolerance

• What do we need?

2

Requirements for this to work

• Initial guess: x1

• Relationship between xn+1 and xn and possibly xn-1, xn-2 , xn-3, …

• When to stop the successive guesses?

Methods we will study

• Fixed Point Iteration

• Bracketing Methods -- Bisection Method-- Regular Falsi Method

• Newton’s Method

• Secand Method (avoiding using derivatives)

FIXED POINT ITERATIONThe The equation f(x) = 0 is rearranged into the

form x = g(x).

Roots of the equation x = g(x) are therefore

roots of the equation f(x) = 0.

The sequence x0, x1, x2, x3, … will converge to

a root α of the equation x = g(x) provided a

suitable starting value x0 is chosen and

−1 < g’(α) < 1.

This gives rise to the iterative formula

xn+1 = g(xn), n = 0, 1, 2, 3, …

with initial approximation x0.

y = f(x)

-6

-4

-2

0

2

4

6

8

10

12

-3 -2 -1 0 1 2 3

x

f(x)

y = g(x) and y = x

-4

-3

-2

-1

0

1

2

3

4

5

-3 -2 -1 0 1 2 3

x

y

Convergence of Fixed-point Iteration

• If the interval contains a root and if

• If , the iterations oscillate while converging. If , the iterations converge monotonically.

bxa ≤≤

}{ bxaxxg ≤≤∈<′ ,1)(

0)(1 <′<− xg1)(0 <′< xg

Analysis of fixed-point iteration reveals that the iterations are convergent on the interval

The The equation f(x) = 0, where f(x) = x3 − 7x + 3, may be re-arranged to give

x = (x3 + 3)/7.

-4

-3

-2

-1

0

1

2

3

4

-5 -4 -3 -2 -1 0 1 2 3 4 5

x

y

y = x

y = (x3 + 3)/7

Intersection of the graphs of y = x and y = (x3 + 3)/7 represent roots of the original equation x3 − 7x + 3 = 0.

Fixed Point Iteration: Example 1The rearrangement x = (x3 + 3)/7 leads to the iteration

To find the middle root α, let initial approximation x0 = 2.

The iteration slowly converges to giveα = 0.441 (to 3 s.f.)

...,3,2,1,0,7

33

1 =+=+ nx

x nn

57143.17

32

7

3 330

1 =+=+= xx

etc.etc.

98292.07

357143.1

7

3 331

2 =+=+= xx

56423.07

398292.0

7

3 332

3 =+=+= xx

45423.07

356423.0

7

3 333

4 =+=+= xx

Fixed Point Iteration: Example 1

3

The rearrangement x = (x3 + 3)/7 leads to the iteration

For x0 = 2 the iteration will converge on the middle root α, since g’(α) < 1.

0

0.5

1

1.5

2

0 0.5 1 1.5 2x

y

n x0 21 1.571432 0.982923 0.564234 0.454235 0.441966 0.44097 0.440828 0.44081

α = 0.441 (to 3 s.f.)

α x0x2 x1

...,3,2,1,0,7

33

1 =+=+ nx

x nn

x3

y = (x3 + 3)/7

y = x

Fixed Point Iteration: Example 1…The rearrangement x = (x3 + 3)/7 leads to the iteration

For x0 = 3 the iteration will diverge from the upper root α.

n x0 31 4.285712 11.67393 227.7024 16865595 6.9E+17

0

2

4

6

8

10

0 2 4 6 8 10

x

y

The iteration diverges because g’(α) > 1.

α x0 x1

...,3,2,1,0,7

33

1 =+=+ nx

x nn

Fixed Point Iteration: Example 1…

Fixed-point Iteration: Example 2

-

Fixed-point Iteration: Example 2…

-

g1(x) converges, g2(x) diverges, g3(x) converges very quickly

Note (see later): g3(x) is the Newton-Raphson iteration function for 2)( 31 −−= xxxf

Bracketing Methods• This class of methods exploit the fact that

functions changes sign in the vicinity of a root. Recall the following:– The Intermediate Value Theorem tells us that if a

continuous function is positive at one end of an interval and is negative at the other end of the interval then there is a root somewhere in the interval.

• Called bracketing methods because two initial guesses for the root are required.

• The two guesses must “bracket” or be on either sides of the root.

4

Bracketing Bisection Method: Steps

1) Notice that if f(a)*f(b) <=0 then there is a root somewhere between a and b

2) Suppose we are lucky enough to be given a and b so that f(a)*f(b) <= 0

3) Divide the interval into two and test to see which part of the interval contains the root

4) Repeat

Step 1

Even though the left hand side could have a root it in we are going to drop it from our search.

The right hand side mustcontain a root!!!. So we are going to focus on it.

Step 2

Winning interval

Step 3

Which way?

Step 4

Find so that

For

if then

else

Now the solution

Bisection Method

00,ba 0)()( 00 <bfaf

...2,1,0=k

2/)( kk bam +=

0)()( ≤mfaf kmbaa kkk == ++ 11 ,

kkk bbma == ++ 11 ,

],[ 11*

++∈ kk bax

5

Convergence Rate Analysis Bisection Convergence Rate…

• Every time we split the interval we reduce the search interval by a factor of two.

• So the error in the value of the root of the function after n iterations is given by

−≤−≤

nnn

baba

200ε

nεε =

Bisection Method …• From this relationship we can also determine

the number of iterations needed to satisfy some given error criterion:

Regula Falsi Method• The regula falsi method start with two points,

(a, f(a)) and (b,f(b)), satisfying the condition that f(a)f(b)<0.

• The straight line through the points (a, f(a)), (b, f(b)) is

• The next approximation to the zero is the value of x where the straight line through the initial points crosses the x-axis.

)()()(

)( axab

afbfafy −

−−+=

)()(

)()()(

)()( afbf

abfbafaf

afbf

abax

−−=

−−−=

Regula Falsi Method (cont.)

• If there is a zero in the interval [a, c], we leave the value of a unchanged and set b = c.

• On the other hand, if there is no zero in [a, c], the zero must be in the interval [c, b]; so we set a = c and leave b unchanged.

• The stopping condition may test the size of y, the amount by which the approximate solution x has changed on the last iteration, or whether the process has continued too long.

• Typically, a combination of these conditions is used.

Regula Falsi Method: ExampleFinding the Cube Root of 2 Using Regula Falsi• Since f(1)= -1, f(2)=6, we take as

our starting bounds on the zero a=1 and b=2.

• Our first approximation to the zero is

• We then find the value of the function:

• Since f(a) and f(8/7) are both negative, but f(8/7) and f(b) have opposite signs, we set new a=8/7, and b=2 remains the same.

1429.17/87/62

)6(16

122))((

)()(

≈=−=+−−=

−−−= bf

afbf

abbx

5073.02)7/8()( 3 −≈−== xfy

2)( 3 −== xxfy

6

Bracketing Method: Advantages/Disadvantages

• Two advantages are:

1. the procedure always converges; that is, a root will always be found, provided that at least one root exists.

2. there is good information available about the error associated with the result.

• Two disadvantage are:

1. the method converges relatively slowly.

2. many iterations may be required.

Newton-Raphson Method

• Use the value and first derivative to extrapolate linearly.

Newton-Raphson Method cont. Newton-Raphson Method cont

• Suppose we know the function f and its derivative f’ at any point.

• The tangent at point (xk-1,f(xk-1)) defined by

• where

can be thought of as a linear model of the function f at xk-1

)(1 xLy k−=

)()()()( 1111 −−−− ′−+= kkkk xfxxxfxL

Newton’s Method

m = Slope of tangent line

)( 1−′= kxfm

)( 1−′= kxfm

1

1)(0

−

−

−−=

kk

k

xx

xfm

)(

)(

1

11

−

−− ′

−=k

kkk xf

xfxx

))(,( 11 −− kk xfxat point

))(,( 11 −− kk xfx

Tangent line y=Lk-1(x)

Newton-Raphson Method cont.

• The zero of the linear model Lk-1 is given by:

• If Lk-1 is a good approximation to f over a wide interval then should be a good approximation to a root of f

kx

[ Think of xk-1 as the current approximation to the root and xk as the next one ]

)(

)(

1

11

−

−− ′

−=k

kkk xf

xfxx

7

Deriving Newton Iteration using Taylors Theorem

• Let the current value be xn and zero is approximately located at xn+1, using Taylor expansion

• Solve for xn+1, we get

21 10 ( ) ( ) ( )( ) ( )n n n n nf x f x f x x x O x+ +′≈ = + − + ∆

1

( )

( )n

n nn

f xx x

f x+ = −′

The function g(x) defined by the formula

)(

)()(

xf

xfxxg

′−=

is called the Newton-Raphson iteration function

Let p be a root of the function f(x), that is, f (p) = 0.

Easy to see that g(p) = p.

Thus the Newton-Raphson iteration for finding the root of the equation f (x) = 0 is accomplished by finding a fixed point of the function g(x).

)(

)(

1

11

−

−− ′

−=k

kkk xf

xfxx

Starting from a guess x0 , use Newton-Raphson formula to compute better approximations

Finding a root of function f(x)

(x1,f(x1))

(x2=x1-f(x1)/f’(x1))

First stage:

(x2,f(x2))

(x3=x2-f(x2)/f’(x2))

Notice: we are getting closer

Zoom in for second stage

Newton’s Iteration: Example 1To solve the equation f(x) = 0, where f(x) = x3 − 7x + 3, use the iteration :

To find the upper root α, let initial approximation x0 = 3.

55.2733

33733

73

372

3

20

03

001 =

−×+×−−=

−+−−=

x

xxxx

...,3,2,1,0,73

372

3

1 =−

+−−=+ nx

xxxx

n

nnnn

411573.2755.23

355.2755.255.2

73

372

3

21

13

112 =

−×+×−−=

−+−−=

x

xxxx

397795.27..41.23

3..41.27..41.2..41.2

73

372

3

22

23

223 =

−×+×−−=

−+−−=

x

xxxx

The iteration quickly converges, giving αααα = 2.40 (to 3.s.f.)

etc.etc.

To solve the equation f(x) = 0, where f(x) = x3 − 7x + 3, use the iteration :

To find the upper root α, let initial approximation x0 = 3.

...,2,1,0,73

372

3

1 =−

+−−=+ nx

xxxx

n

nnnn

The iteration quickly converges, giving αααα = 2.40 (to 3.s.f.)

-5

0

5

10

15

2 2.2 2.4 2.6 2.8 3 3.2 3.4

x

f(x)

x0x1

n x

0 3

1 2.55

2 2.411573

3 2.397795

4 2.397662

5 2.397662 x2

y = x3 − 7x + 3

Newton’s Iteration: Example 1…

8

To solve the equation f(x) = 0, where f(x) = x3 − 7x + 3, use the iteration :

Choice of intial approximation x0 will determine which root is found.

...,2,1,0,73

372

3

1 =−

+−−=+ nx

xxxx

n

nnnn

n x(n) x(n) x(n)0 -2 1 31 -3.8 0.25 2.552 -3.10419 0.43578 2.4115733 -2.86763 0.440803 2.3977954 -2.83888 0.440808 2.3976625 -2.83847 0.440808 2.397662

Initial approximations x0 = −2, x0 = 1 and x0 = 3 .

Iterations converge to −2.84, 0.441 and 2.40 respectively (to 3 s.f.)

Newton’s Iteration: Example 1 …

Assume that A > 0 is a real number and let > 0 be an initial approximation to√A. Define the sequence using the recursive rule

Then this sequence converges to √A; that is,

Outline of Proof. Start with function f (x) = x2 − A, and notice that the roots of f (x) = 0 are ±√A. Now use f (x) and the derivative f’(x) to write down the Newton-Raphson iteration formula using

where

It can be proved that the generated sequence will converge for anystarting value generated > 0.

0p∞

=0}{ kkp

Apkk =∞→lim

Newton’s Iteration for Finding Square Roots

1( )k kp g p −=

0p

Convergence of Newton’s Method

• We will show that the rate of convergence is much faster than the bisection method.

• However – as always, there is a catch. The method uses a local linear approximation, which clearly breaks down near a turning point.

• Small makes the linear model very flat and will send the search far away …

)( nxf ′

(x1,f(x1))

(x2=x1-f(x1)/f’(x1))

Say we chose an initial x1 near a turning point. Then the linear fit shoots off into the distance!.

9

Newton’s Method: Advantages/Disadvantages

• The Newton-Raphson method generally provides rapid convergence to the root, provided the initial value x0 is sufficiently close to the root.

• How close is ‘sufficiently close’? That dependson the characteristics of the function itself. As illustrated by the following examples, certain initial values of x may cause the solution to diverge or not produce a valid result.

To solve the equation f(x) = 0, where f(x) = 1/x + 3, use the iteration :

To find the only root α, let initial approximation x0 = −1.

1)1/(1

3)1/(11

/1

3/122

0

001 =

−−+−−−=

−+−=x

xxx

...,3,2,1,0,/1

3/121 =

−+−=+ nx

xxx

n

nnn

The iteration quickly diverges, failing to give the rootα.

etc.etc.

51/1

31/11

/1

3/122

1

112 =

−+−=

−+−=x

xxx

855/1

35/15

/1

3/122

2

223 =

−+−=

−+−=x

xxx

Failure of Newton’s iteration: Example 1

-2

-1

0

1

2

3

4

5

6

-2 -1 0 1 2 3 4 5 6

x

f(x)

n x(n)0 -11 12 53 854 218455 1.43E+09

To solve the equation f(x) = 0, where f(x) = 1/x + 3, use the iteration :

To find the only root α, let initial approximation x0 = −1.

...,3,2,1,0,/1

3/121 =

−+−=+ nx

xxx

n

nnn

The iteration quickly diverges, failing to give the root α = -1/3.

x0 x1 x2α

y = 1/x + 3

Failure of Newton’s iteration: Example 1...

Newton’s iteration for f (x) = xe−x can produce a divergent sequence.

Failure of Newton’s iteration: Example 2

Newton’s iteration for f (x) = x3 − x − 3 can produce a cyclic sequence.

Failure of Newton’s iteration: Example 3

Newton’s iteration can produce a cyclic sequence

Failure of Newton’s iteration: Example 4

10

Newton’s iteration for f (x) = arctan(x) can produce a divergent oscillating sequence.

Failure of Newton’s iteration: Example 5 Newton’s Method: algorithm so far

• Choose initial guess

• Repeat

Until Failure / Convergence

0x

)(

)(1

k

kkk xf

xfxx

′−=+

How do we determine this?

Convergence Criteria

• Convergence checking will avoid searching to unnecessaryaccuracy

• Convergence checking can consider whether two successiveapproximations to the root are close enough to be consideredequal

• Convergence checking can examine whether f(x) is sufficiently close to zero at the current guess

A root-finding procedure needs to�monitor progress towards the root and�stop when current guess is close enough to the desired root

Convergence Criteria …

On the values of x:

On the values of f(x):

BOTH OF THESE CRITERIA NEED TO BE SATISFIED FOR THE ALGORITHM TO BE COMPLETE

Failure to converge• Check against before dividing by this

• Put a maximum on the number of iterations so as to prevent the sequence ‘wandering’ (use for loop instead of repeat-until)

• Check to see if or are growing rather than decreasing

• Check if = NaN or infinity and whether largest representable no. < < smallest

• Check if = infinity, then

0)( ≈′ kxf

)( 1+kxf )()( 1 kk xfxf −+

)( kxf

)()( 1 kk xfxf =+)( kxf ′

kx

11

Software requirements

• Function and its derivative

• An initial iterate

• Convergence tolerance tol (=tol1=tol2) – a small number

• A maximum number of possible iterations

User of such software should provide:

)( kxf )(xf ′0x

NOTE: if there are numerous solutions, different choices of might lead to different correct answers0x

Newton in Matlab

Speed of Convergence

then the sequence is said to converge to p with order of convergence R.

The number A is called the asymptotic error constant.

Assume that converges to p and set En = p − pn for n ≥ 0.If two positive constants A 0 and R > 0 exist, and≠

If R=1, the convergence of is called linear( new error proportional to old error)

If R=2, the convergence of is called quadratic ( new error proportional to old error squared)

Example Start with p0 = −2.4 and use Newton-Raphson iteration to find the root p = −2 of the polynomial f (x) = x3 − 3x + 2. The iteration formula for computing {pk } is

Newton’s Method: Quadratic Convergence at a Simple Root

Checking for quadratic convergence (R = 2 ), we get the values in the following table.

Taking a closer look: where A ≈ 2/3.

Convergence Rate for Newton-Raphson Iteration

Assume that Newton-Raphson iteration produces a sequence that converges to the root p of the function f (x).

If p is a multiple root of order M, convergence is linear and

If p is a simple root, convergence is quadratic and

REMINDER: Assume that and . We say that f (x) = 0 has a root of order M at x = p if and only if

],[ baCf M∈ ),( bap∈

A root of order M = 1 called a simple root, if M > 1 it is called a multiple root.

Convergence in Newton Iteration

• Let x be the exact root, xi the value in i-thiteration, and εi =xi-x the error, then

• Rate of convergence:

2 3

2

1( ) ( ) ( ) ( ) ( ),

2( ) ( ) ( ) ( )

f x f x f x f x O

f x f x f x O

ε ε ε ε

ε ε ε

′ ′′+ = + + +

′ ′ ′′+ = + +

1

2

1

2

( ),

( )

( ) ( ) ( ) / 2

( ) ( ) ( )

( )

2 ( )

ii i

i

i i ii i i

i i

i

f xx x

f x

f x f x f x

f x f x f x

f x

f x

ε εε ε εε

ε

+

+

= −′

′ ′′+= − ≈ −′ ′ ′′+

′′=

′ (Quadratic convergence for simple roots)

12

Secand Method

= slope of tangent line at is approximated by slope of the secant line L passing through points

The iterate xk+1 is the root of this secant line L:

)()(

)()(

1

111

−

−−+ −

−=kk

kkkkk xfxf

xfxxfxx

LSlope of L=

(i.e. to find xk+1 set y=0)

Finding the Square Root of 3 by Secand Method– To find a numerical approximation to , we seek the zero of

.– Since f(1)=-2 and f(2)=1, we take as our starting bounds on

the zero and .– Our first approximation to the zero is

– Calculation of using secant method.

33)( 2 −== xxfy

10 =x 21 =x

667.13

5)1(

)2(1

1221

01

0112 ≈=

−−−−=

−−−= y

yy

xxxx

3

Secand Method: Example 1

Consider the secant method used on f(x)=x3+x2-x-1 with x0=2, x1=0.5. Note this fn is continuous with roots +1,-1.

Secant method: Example 2 Secant Method: Advantages and Disadvantages

• The Secant method generally provides fairly rapid convergence to the root but not as rapidly as the Newton-Raphson method. However, except for the starting iteration, the secant method requires the evaluation of only one function per iteration while the Newton-Raphson method requires that two functions (the function and its derivative) be evaluated for each iteration.

• Similar to the Newton’s method, the secant method may also encounter runaway, flat spot, and cyclical non-convergence characteristics.

1-D Root-Finding Summary• Bisection method : (global convergence method)

Guaranteed to work provided a bracketing pair is given - never diverges, but slow (linear rate of convergence R=1)

• Newton’s method : (local convergence method)risky but fast! (quadratic convergence rate, R=2)

• Secant method : (local convergence) less risky, mid-speed (convergence rate R 1.6180334), similar to Newton’s method but avoids calculation of derivatives

Global convergence methods: converge starting from anywhere

Local convergence methods: converge if x is sufficiently close to root

≈