numerical methodsprofesores.dcb.unam.mx/users/gustavorb/mn/glyn james.pdfemail: j.b.grimbleby...

J. B. Grimbleby Department of Electronic Engineering

The University of Reading Unit EG1C2: 1

Numerical MethodsModule: EG1C2 Engineering Mathematics

Lecturer: James B. GrimblebyURL: http://www.elec.rdg.ac.uk/jbg.htmlemail: j.b.grimbleby reading.ac.uk

Number of Lectures: 10

Recommended text book:Modern Engineering Mathematics (3rd edition) Glyn JamesPrentice-Hall 2001ISBN 0-13-018319-9



Numerical Methods

Modern Engineering Mathematics (3rd edition) Glyn JamesPrentice-Hall 2001ISBN 0-13-018319-9

Approx. price: £25



Numerical precision, absolute and relative errors, binary codes, integer and floating-point codes, IEEE floating-point representations, rounding and truncation errors in arithmetic operations, underflow and overflow

Real zeros of non-linear equations by repeated bisection, secant method, iteration method, Newton-Raphson method, factorisation of polynomials, synthetic division

Curve fitting, linear and quadratic interpolation, Lagrange polynomials, exact polynomial fit through equally-spaced points, numerical differentiation, polynomial least-squares fit

Syllabus



Numerical integration, the trapezium approximation, improper integrals, midpoint approximation, changes of variable to deal with infinite integration limits

Function minimisation, single-variable approximation methods, single-variable search methods, Golden-Section search

Numerical solution of first-order non-linear differential equations; Euler's method, 2nd and 4th order Runge-Kuttamethods

Numerical solution of higher-order differential equations, reduction to a set of 1st-order differential equations

Syllabus



Engineering calculations often involve quantities whose values can vary over a wide range

Frequencies: 1 Hz → 10 GHz

Mass: µg → many tonnes

Distance: µm → thousands of km

Power: nW → MW

Capacitance: 1pF → 1F

The range is further increased when such quantities are multiplied or divided by one another

Numerical Methods



Relative precision: the precision of a number relative to its magnitude

100±2, 1±0.02 and 0.001±0.00002 all have a relative precision of 0.02 or 2%

Absolute precision: the absolute precisions of these values are respectively 2, 0.02 and 0.00002.

In most cases it is the relative precision, rather than the absolute precision, that is of importance

Precision



Numbers are represented in a computer by means of binary codes

Each number is stored and processed in the form of a group of binary digits (bits) known as a word

A binary word consisting of n bits can represent 2n distinct values

Natural binary code: n-bit word represents the integer values 0 to 2n-1

Signed binary code: n-bit word represents the integer values -2n-1 to 2n-1-1

Binary Codes



Binary floating-point codes can represent very large and very small values

Floating-point numbers consist of two fields, the mantissa M and the exponent E

X = M × 2E

The relative precision of a floating-point number is determined by the number of bits in the mantissa M

The range is determined by the number of bits in the exponent E

Floating-Point Codes



The IEEE standard defines the format of single (32 bit), double (64 bit) and extended (80 bit) precision floating-point numbers

Double precision (64 bit):

IEEE Floating-Point Formats

52 bit mantissa11 bit exponent1 bit sign

6362 ……….52 51……………………………………………….0

Mantissa is 53-bit unsigned fixed-point binary

Exponent is 11 bit excess 1023 binary



IEEE Floating-Point FormatsMantissa 52-bit + hidden bit:

1.00000 .. 00000 = 1 (decimal)1.11111 .. 11111 ≈ 2 (decimal)

Hidden bit 52 bits of(always 1) IEEE format

A change in the least-significant bit is equal to:

2-52 = 2.2×10-16

so that IEEE 64-bit numbers have a relative precision of 15-16 decimal digits



IEEE Floating-Point Formats

Exponent 11-bit excess 1023:

1 - 1023 = -1022 2046 - 1023 = +1023

Floating-point range:

1 × 2-1022 = 2.3 × 10-308

2 × 2+1023 = 1.7 × 10+308

The special case of 11-bit exponent all 0s is used to represent the floating-point value 0.0



Truncation and Rounding ErrorsA 32-bit word can represent 4294967296 values

Real quantities cannot be represented exactly and the nearest discrete value of a binary word is used

For example the decimal value 1.2 has a binary representation:

1.00110011001100110011001100110011 ...

Since real values must be stored in a binary word the number of bits must be limited and there are two ways of doing this: Truncation or Rounding

[G James p 42]



TruncationTruncation is the act of simply discarding unwanted digits from a number

Decimal value 1.2 has a binary representation:

1.00110011001100110011001100110011 ...

This exact binary representation could be truncated to 8 bits:

1.0011001

This has decimal value 1.1953125, an absolute error of 0.0046875 or relative error of 0.39%



RoundingRounding involves determining the nearest discrete value of the appropriate word

Decimal value 1.2 has a binary representation:

1.00110011001100110011001100110011 ...

This exact binary representation could be rounded to 8 bits:

1.0011010

This has decimal value 1.203125 , an absolute error of 0.003125 or relative error of 0.26%



Errors Generated by AdditionWhen a small number is added to a larger number a rounding error related to the precision of the larger number occurs

If this procedure is repeated a large number of times then the effects of the rounding errors can become quite significant

When a value of 1.0 × 10-15 is added to 1.0 a million times using IEEE double precision numbers:

1.00000000111022

In spite of the fact that the operations are performed using 15 digit precision, the result is accurate only to 9 digits.



Errors Generated by SubtractionWhen numbers are subtracted the absolute precision of the result is equal to the absolute precision of the subtracted quantities

If the result is smaller in magnitude than the subtracted quantities then it will have a reduced relative precision

When 1000000.0 is subtracted from 1000000.1 using IEEE double precision numbers:

0.0999999999767169

Six digits of accuracy have been lost.



Errors Generated by SubtractionConsider evaluation of the simple formula:

Obviously the answer should be A

If B is larger in magnitude than A then A will suffer a loss in relative precision.

Evaluating the formula with A = 0.1 and B =1.0 × 1015 using IEEE double precision real numbers:

0.125000000000000

BBAX −+= )(



Errors Generated by SubtractionSubtractions of nearly equal quantities are implicit in many engineering calculations

For example if A is small then:

Adding 1 to A limits the information about A in the result X

If A = 1.0×10-12 then the value of ln(X) = A evaluated using IEEE double precision numbers:

1.000088900581841×10-12

AeX A +≈= 1



Simultaneous Linear Equations

11 bxay +=

22 bxay +=

11 bxay +=

22 bxay +=

2112

1221

2211)(

aabbx

bbaaxbxabxay

−−

=

−=−

+=+=Solution:



Roots of a Quadratic Equation

As is well known the roots of this equation are given by the formula:

Quadratic equation with roots at 100.0 and 0.01:

Using the quadratic formula gives the lower root:

aacbbx

242 −±−

=

02 =++ cbxax

01.00.2

99.9901.1000.2

0.40001.1000201.100 =−=−−=x

00.101.1002 =+− xx



Roots of a Quadratic EquationSubtraction of nearly-equal quantities only happens when the roots are real, that is b2≥4ac

For real roots an alternative formula should be used:

It is safe to use the normal quadratic formula if the roots are complex, that is b2<4ac

qcx

aqx

acbbbq

22

4).sgn(

2

1

2

−=

−=

−+=



Roots of a Quadratic EquationQuadratic equation:

For this equation b2≥4ac so that the roots are:

At no point were nearly-equal quantities subtracted

01.00.200

0.2

0.1000.2

2000.20099.9901.1000.40001.1000201.100

2

1

=−−=

==

−=−−=−−−=

x

x

q

00.101.1002 =+− xx



Unstable AlgorithmsOften several algorithms are available to perform some operation

Examples: solving simultaneous linear equations, curve fitting and solving differential equations

All of the algorithms will be mathematically correct and will give the same results if executed on a computer that does not use rounding or truncation

On a real computer they will not necessarily produce results of the same precision

Some algorithms are notoriously unstable



Unstable AlgorithmsConsider raising φ to an integer power, where:

φ is related to the Golden Mean or Golden Section k:

The n+1 th power of φ can be obtained from the nth and n-1 th:

215 −

=ϕ

618034.011 ==−=k

kϕ

( )

nn

nnnnn

ϕϕ

ϕϕϕϕϕ

−=

−−=

+−=

−=

−

−−−−+

1

11112

1

215

41525

415



Unstable Algorithmsn φn (by multiplication) φn (by algorithm)

1 6.1803398874989488×10-1 6.1803398874989488×10-12 3.8196601125010520×10-1 3.8196601125010512×10-13 2.3606797749978976×10-1 2.3606797749978980×10-14 1.4589803375031552×10-1 1.4589803375031530×10-15 9.0169943749474288×10-2 9.0169943749474512×10-26 5.5728090000841248×10-2 5.5728090000840776×10-2. . . . . . . . . . . . . . . . . . . . . . . . . . .32 2.0530310231465800×10-7 2.0518477406028524×10-733 1.2688429522625586×10-7 1.2707575436365914×10-734 7.8418807088402160×10-8 7.8109019696626096×10-835 4.8465488137853720×10-8 4.8966734667033048×10-836 2.9953318950548452×10-8 2.9142285029593040×10-837 1.8512169187305276×10-8 1.9824449637440012×10-838 1.1441149763243180×10-8 9.3178353921530288×10-939 7.0710194240620984×10-9 1.0506614245286982×10-840 4.3701303391810832×10-9 -1.1887788531339538×10-9



IEEE double precision floating point numbers cover the range of magnitudes from 2.3×10-308 to 1.7×10+308

IEEE single precision floating point numbers cover the range of magnitudes from 1.2×10-38 to 3.4×10+38

Both single and double precision numbers also allow a value of zero

Overflow occurs when a floating point number exceeds in magnitude the largest value that can be represented

Underflow occurs when a floating point number is smaller in magnitude than the smallest value that can be represented

Overflow and Underflow



Overflow and UnderflowA complex number x+jy has a modulus m:

Evaluating this the obvious way limits the values of x and y to ½ of the available floating-point range

To avoid this the modulus can be evaluated in an alternative way:

22 yxm +=

yxyxym

yxxyxm

<⎟⎠⎞

⎜⎝⎛+=

≥⎟⎠⎞

⎜⎝⎛+=

2

2

1

1



There are many problems in engineering that require solution of an equation of the form:

In most cases the equation cannot be solved algebraically and numerical methods must be used

For example consider the simple electronic circuit:

Solution of Non-Linear Equations

2.0V

i

1000Ω

v1

0)( =xf



Ohm's law:

Shockley equation for the diode:

Eliminating the current i:

This equation cannot be solved algebraically

0.10000.2 1vi −

=

026.0exp100.1 114 vi −×=

00.1000

0.2026.0

exp100.1)( 11141 =

−−×= − vvvf

Solution of Non-Linear Equations

2.0V

i

1000Ω

v1



0.10000.2 1vi −

=

026.0exp100.1 114 vi −×=

Graphical Solution of Non-Linear Equationsi

2mA

vi2V



Repeated Bisection

If f(a) and f(b) have opposite signs then a root exists between aand b

Bisect the interval a to b, to get a new point c

f(x)

xa cb

2bac +=

Also known as Interval Halving or Binary Search

[G James p 469]



Repeated Bisection algorithm:

If the initial interval is a0 to b0, with midpoint c0, and the interval for the next iteration is a1 to b1, then :

c0 = (a0+b0)/2

If f(a0)×f(c0) < 0 then: a1 = a0, b1 = c0otherwise: a1 = c0, b1 = b0

A new midpoint c1 is obtained by bisecting the interval a1 to b1, the function evaluated at this midpoint, and the process repeated until bn-an < required accuracy

Repeated Bisection



Repeated Bisectiona c b fa fc fb

0.000000 1.000000 2.000000 -2.0e-3 5.1e+2 2.6e+9 0.000000 0.500000 1.000000 -2.0e-3 -1.5e-3 5.1e+20.500000 0.750000 1.000000 -1.5e-3 3.2e-2 5.1e+20.500000 0.625000 0.750000 -1.5e-3 -1.1e-3 3.2e-20.625000 0.687500 0.750000 -1.1e-3 1.7e-3 3.2e-20.625000 0.656250 0.687500 -1.1e-3 -4.3e-4 1.7e-30.656250 0.671875 0.687500 -4.3e-4 3.4e-4 1.7e-3 0.656250 0.664063 0.671875 -4.3e-4 -9.9e-5 3.4e-40.664063 0.667969 0.671875 -9.9e-5 1.1e-4 3.4e-4 0.664063 0.666016 0.667969 -9.9e-5 -8.3e-7 1.1e-4 0.666016 0.666992 0.667969 -8.3e-7 5.1e-5 1.1e-40.666016 0.666504 0.666992 -8.3e-7 2.5e-5 5.1e-50.666016 0.666260 0.666504 -8.3e-7 1.2e-5 2.5e-5 . . . . . . . . . . . . . . . . . . . . . . . . . . 0.666031 0.666033 0.666035 -3.0e-8 7.0e-8 1.7e-70.666031 0.666032 0.666033 -3.0e-8 2.0e-8 7.0e-8



Repeated BisectionThe repeated bisection method is very robust and its rate of convergence (a factor of 2 per iteration) is fixed

After n iterations the size of the interval (bn-an) is given by:

If the required accuracy is ε then the number of iterations nrequired is given by:

nnnabab

200 −

=−

2log

loglog

10

001000

2ε

ab

εabn

−

=−

=



Repeated Bisectionf(x)

x

f(x)

x

21)(−

=x

xf Close or coincident roots



Secant Method

f(a)x

ca

b )()()()(

)()()()(

)(0)()(slope

afbfabafac

afbfabafac

acaf

abafbf

−−

×−=

−−

×−=−

−−=

−−=

Two points a and b on the function f(x) are used to estimate the position c of the root by linear interpolation

f(x)

f(b)

[G James p 470]

Also known as Method of False Position



Secant Method

f(x)

xc

a b)()(

)()(

)()()()(

)()()(slope

afbfabafac

afbfabafca

caaf

abafbf

−−

×−=

−−

×=−

−=

−−=

It is not necessary for the root to lie between the points aand b:

f(a)

f(b)



Secant Method

f(a)

xc

a

b

f(x)

f(b)

The process is now repeated with a1=b0 and b1=c0 to get an even better approximation c1:



Secant Method

Close to root the number of significant figures increases by a factor of 1.618 per iteration

Secant method may be slow and unreliable when operating far from a root

a b f(a) f(b) c0.700000 1.000000 3.6265×10-3 5.0539×10+2 0.6999981.000000 0.699998 5.0539×10+2 3.6261×10-3 0.699996 0.699998 0.699996 3.6261×10-3 3.6257×10-3 0.6809590.699996 0.680959 3.6257×10-3 1.0495×10-3 0.6732030.680959 0.673203 1.0495×10-3 4.3089×10-4 0.667802 0.673203 0.667802 4.3089×10-4 9.5748×10-5 0.6662580.667802 0.666258 9.5748×10-5 1.1914×10-5 0.666039 0.666258 0.666039 1.1914×10-5 3.9138×10-7 0.666031 0.666039 0.666031 3.9138×10-7 1.6718×10-9 0.666031



Iteration MethodThe Iteration method is based on manipulating the equation into the form:

Given an approximation xn to the root, a better approximation xn+1 is found by using the formula:

The iteration method can often converge to a root very quickly

Unfortunately the iteration method does not guarantee convergence

Nevertheless, because of its simplicity it is a valuable method

)(xgx =

)(1 nn xgx =+

[G James p 470]



Convergence of the Iteration Method

)(xgy =

y

x

xy =

x0 x1x2

)(xgy =

y

x

xy =

x0x1 x2

Convergent Divergent)(xgx =



The condition for the iteration method to converge is that the magnitude of the slope of g(x) must be less than unity:

Diode circuit:

which can be written:

or in a form suitable for the iteration method:

1<′ )(xg

00.1000

0.2026.0

exp100.1 1114 =−

−× − vv

026.0exp100.10.2 111

1vv −×−=

026.0exp100.10.2)( 11 xxg −×−=




For a starting value of x0=1.0:

Clearly this does not satisfy the condition for convergence


7100 1091

0260011093 ×−=×−=′ − .

..exp.)(xg

x g(x)1.000000×10+00 -5.053965×10+05 -5.053965×10+05 2.000000×10+002.000000×10+00 -2.554276×10+22



Convergence of the Iteration MethodThe equation can be rearranged:

so that:

or in a form suitable for the iteration method:

026.0100.10.2ln:or

026.0exp

100.10.2 1

1111

111 vvvv

=⎟⎠

⎞⎜⎝

⎛

×

−=

×

−−−

⎟⎠

⎞⎜⎝

⎛

×

−=

−111

1100.1

0.2ln026.0 vv

( ) ( )11

11

100.1ln026.00.2ln026.0100.1

0.2ln026.0)(

−

−

×−−=

⎟⎠

⎞⎜⎝

⎛

×

−=

x

xxg



From a starting point of x0=1.0:

This satisfies the condition for convergence

Iteration Method

0260102

0260020260

0 ...

..)( −=

−−=

−−=′

xxg

x g(x)1.000000 0.6585390.658539 0.6661770.666177 0.6660290.666029 0.6660320.666032 0.6660310.666031 0.666031



Newton-Raphson Method[G James p 619]

The Newton-Raphson method can in principle be applied to any equation of the form:

In practice it is only really suitable for functions whose derivatives can be obtained analytically

All polynomial equations:

have this property and Newton-Raphson method is the preferred technique for solving such equations

0)( =xf

nnxaxaxaxaaxf +++++= ...)( 3

32

210



Suppose that x0 is initial estimate of root

Expand function as a Taylor series around this point:

If x0+h is a root of the equation:

Provided that x0 is close to the root, then h will be small, and:

Newton-Raphson Method

... )("!2

)(')()( 02

000 +++=+ xfhxhfxfhxf

0)( 0 =+ hxf

)(')(:or0)(')(

0

000 xf

xfhxhfxf −==+



This formula will not give distance h to root exactly because of approximation involved

However x1 should be a better approximation to root where:

Newton-Raphson iteration formula:


)(')(

0

0xfxfh −=

)(')(

0

001 xf

xfxx −=

)(')(

1n

nnn xf

xfxx −=+




x0x1x2x

f(x0)

)(')(:or)(slope)('

0

001

10

00 xf

xfxxxx

xfxf −=−

==

f(x)



Newton-Raphson MethodThe Newton-Raphson method requires the derivative of the function to be evaluated

In the case of the diode circuit the derivative can be obtained in algebraic form:

The Newton-Raphson method can now be used with a starting point of 1.0

0.10000.2

026.0exp100.1)( 14 xxxf −−×= −

0.10000.1

026.0exp

026.0100.1)('

14+

×=

− xxf



Newton-Raphson Methodxn f(xn) f'(xn) xn+1

1.000000 5.05397×10+2 1.94384×10+4 0.9740000.974000 1.85925×10+2 7.15100×10+3 0.948000 0.948000 6.83977×10+1 2.63072×10+3 0.922001 0.922001 2.51618×10+1 9.67805×10+2 0.896002 0.896002 9.25622×10+0 3.56052×10+2 0.870005 0.870005 3.40486×10+0 1.31001×10+2 0.844014 0.844014 1.25226×10+0 4.82095×10+1 0.818038 0.818038 4.60361×10-1 1.77526×10+1 0.792106 0.792106 1.69029×10-1 6.54858×10+0 0.766295 0.766295 6.18485×10-2 2.42724×10+0 0.740814 0.740814 2.24153×10-2 9.11558×10-1 0.716224 0.716224 7.91091×10-3 3.54642×10-1 0.693917 0.693917 2.59273×10-3 1.50954×10-1 0.676741 0.676741 6.90641×10-4 7.84577×10-2 0.667939 0.667939 1.03429×10-4 5.62112×10-2 0.666099 0.666099 3.51141×10-6 5.24390×10-2 0.666032 0.666032 4.43172×10-9 5.23066×10-2 0.666031



Newton-Raphson MethodNewton-Raphson method may fail where two or more of the roots are coincident

f(x)

xZero derivative

Zero function



Newton-Raphson MethodAlthough Newton-Raphson works well in most cases there is no guarantee that it will converge:

f(x)

xx0

x1x2

x3



Polynomial EquationsThe Newton-Raphson method is particularly suitable for polynomial equations because their derivatives can always be obtained analytically:

In general polynomial equations have complex roots but he discussion here will be limited to polynomials with real roots

Polynomials up to, and including quartics (n=4) can be solved algebraically

∑=++++==

=

ni

i

ii

nn xaxaxaxaaxf

0

2210 ...)(

∑=++++==

=

−− ni

i

ii

nn xiaxnaxaxaaxf

1

112321 ...32)('



Evaluation of PolynomialsThe most efficient way of evaluating a polynomial is to use Horner's rule

This can be derived by rearranging the polynomial into nested form:

Only one multiplication and one addition is required for each power of x

This method can be used for the polynomial and its derivative.

)..))(...((

...)(

1210

11

2210

xaaxaxaxa

xaxaxaxaaxf

nn

nn

nn

+++++=

+++++=

−

−−



Polynomial EquationsPolynomial with roots: 10, 1, 1 and 0.1:

Estimate of smallest root (or simply use x0=1.0):

Newton-Raphson:

432 1.122.221.121)( xxxxxf +−+−=

0826446.01.12

11

00 =

−==

aax

xn f(xn) f'(xn) xn+1

8.26446×10-2 1.44846×10-1 -8.67625×10+0 9.93391×10-2

9.93391×10-2 5.30788×10-3 -8.04364×10+0 9.99990×10-2

9.99990×10-2 8.11916×10-6 -8.01904×10+0 1.00000×10-1

1.00000×10-1 1.90980×10-11 -8.01900×10+0 1.00000×10-1



Polynomial DeflationRoots are removed from a polynomial by a procedure known as deflation

Any polynomial can be written in the form of the product of nfactors:

Clearly the polynomial becomes zero when x equals any of the zi so that zi are the roots of the polynomial equation

If zi is a root of a polynomial equation then (x-zi) must be a factor of the polynomial

))...()()((...)(

321

11

2210

nn

nn

nn

zxzxzxzxaxaxaxaxaaxf

−−−−=

+++++= −−



Polynomial Deflation[G James p 88]

x3-12.0x2+21.0x -10

x-0.1 x4-12.1x3+22.2x2-12.1x+1x4- 0.1x3-12.0x3+22.2x2-12.0x3+ 1.2x2

+21.0x2-12.1x+21.0x2- 2.1x

-10.0x+1-10.0x+1

The remainder from this polynomial division should always be zero



Polynomial EquationsDeflated polynomial:

Estimate of smallest root:

Newton-Raphson:

102112)( 23 −+−= xxxxf

476190.02110

1

00 ===

aax

xn f(xn) f'(xn) xn+10.476190 -2.613109 10.251701 0.7310860.731086 -0.670281 5.057404 0.863621. . . . . . . . . . . . . . . . . . . . . . . . 0.997837 -0.000042 0.038940 0.9989190.998919 -0.000011 0.019469 0.9994590.999459 -0.000003 0.009734 0.9997300.999730 -0.000001 0.004867 0.999865



Polynomial Deflation[G James p 88]

x2-11x +10

x-1 x3-12x2+21x-10x3 -x2-11x2+21x-11x2+11x

10x-1010x-10

The remaining roots can be found using the quadratic formula:

10and12

9112

40121112

42

==

±=

−±=

−±−=

xxa

acbbx



Cubic Equations

f(x)

x

f(x)

x

33

2210)( xaxaxaaxf +++=

3 real roots 1 real root

Roots of a polynomial equation (with real coefficients) are either real or occur in complex conjugate pairs



Cubic EquationsExample:

Estimate of smallest root:

Newton-Raphson:

20136)( 23 −+−= xxxxf

53.11320

1

00 ===

aax

xn f(xn) f'(xn) xn+11.538462 -10.559854 1.639053 7.9811167.981116 209.948095 108.321259 6.0429186.042918 60.125160 50.035556 4.8412694.841269 15.778298 25.218432 4.2156044.215604 3.091784 15.726702 4.0190094.019009 0.249297 13.229197 4.0001654.000165 0.002144 13.001979 4.0000004.000000 0.000000 13.000000 4.000000



Cubic Equations[G James p 88]

x2-2x +5

x-4 x3-6x2+13x-20x3-4x2-2x2+13x-2x2 +8x

5x-205x-20

The remaining roots can be found using the quadratic formula:

j212

1622

20422

42

±=

−±=

−±=

−±−=

xa

acbbx



Curve FittingCurve fitting consists of obtaining a function f(x) which fits experimental data points

This may be an exact fit if the number of experimental points is small

Alternatively if the number of experimental points is large then the function will normally be an approximate fit

It will be assumed in this course that the function is of polynomial form and is fitted by adjusting the coefficients ai

nn

nn xaxaxaxaaxf +++++= −−

11

2210 ...)(



Uses of Curve FittingOnce a function has been fitted to data it can be used to predict values away from the data points:

Interpolation Extrapolation

y

x1x

x3x2

y3y2

y1

yi

xi

y

x1x

x3x2

y3y2

y1

ye

xe



Interpolation and Extrapolation

5th-order Polynomial

Actual function

Actual function

Data points

y

x



Function Differentiation

Cubic Polynomial

Frequency

Inverter Efficiency Differentiate cubic and set to zero to obtain maximum

Data points



Function Integration7th-Order polynomialfit to data

Data points

Integral = area under curve

y

xa b



Straight Line Through 2 PointsThe simplest type of curve fit is a straight line:

Suppose that the data points are (x1, y1) and (x2, y2):

baxxf +=)(

x

ybaxxf +=)(),( 11 yx

),( 22 yx

[G James p 76]



Straight Line Through 2 PointsData points (x1, y1) and (x2, y2) lie on the straight line:

These simultaneous linear equations in a and b can be solved algebraically to give:

Interpolation or extrapolation using a straight-line fit is termed linear

2211

ybaxybax

=+=+

122112

1212

xxyxyxb

xxyya

−−

=

−−

=



Straight Line Through 2 PointsExample: the resistance R of a wire-wound resistor as a function of temperature T:

T(°C) R(Ω)20 511.440 529.1

Thus:

so that:

7.4932040

1.529204.51140

885.02040

4.5111.529

=−

×−×=

=−−=

b

a

7.493885.0 += TR



Straight Line Through 2 Points7.493885.0 += TR

The data can be linearly interpolated using this function to give the resistance at 27°C:

and can be linearly extrapolated using this function to give the resistance at 51°C:

The temperature coefficient of the resistor is 0.885 Ω/°C

Ω6.5177.49327885.0 =+×=R

Ω8.5387.49351885.0 =+×=R



Quadratic Through 3 PointsA quadratic can exactly fit 3 data points:

Suppose that the data points are (x1, y1), (x2, y2) and (x3, y3):

cbxaxxf ++= 2)(

x

y

),( 11 yx

),( 22 yx

),( 33 yx

cbxaxxf ++= 2)(



Data points (x1, y1), (x2, y2) and (x3, y3) lie on the quadratic:

These simultaneous linear equations can be solved to give a, b and c

The solutions for quadratic and higher-order polynomial fits are not normally obtained in algebraic form; instead the coefficients are determined numerically

3323

2222

1121

ycbxax

ycbxax

ycbxax

=++

=++

=++

Quadratic Through 3 Points



00243.140160000107.13090000040.120400

=++=++=++

cbacbacba

Quadratic Through 3 PointsExample: the frequency F of a crystal oscillator as a function of the ambient temperature T:

T (°C) F (MHz)20 1.0004030 1.0010740 1.00243

These values can be substituted into the quadratic polynomial to give:



Subtract the 1st equation from the 2nd and 3rd equations:

Now multiply the 1st equation by 2, and subtract from the 2nd:

Back substitute a:

Back substitute a and b:

00203.020120000067.010500

=+=+

baba

Quadratic Through 3 Points

610450.3or00069.0200 −×== aa

410055.1or00067.010001725.0 −×−==+ bb

00113.1or00040.100211.000138.0 ==+− cc



Quadratic Through 3 Points00113.110055.110450.3 426 +×−×= −− TTF

The data can now be interpolated using this function to give the oscillator frequency at 32 °C:

The temperature coefficient at 20 °C

MHz00129.100113.13210055.1102410450.3 46

=+××−××= −−F

Hz/C5.32MHz/C10055.12010900.6

10055.110900.6dd

46

46

=×−××=

×−×=

−−

−− TTF



A polynomial of order n-1 can be fitted exactly to n data points:

Such a polynomial has n coefficients and there will be nsimultaneous equations to be solved in order to obtain the coefficients a, b .. p, q, r:

Polynomial Through n Points

rqxpxbxaxxf nn +++++= −− 221 ...)(

nnnnn

nn

nn

nn

yrqxpxbxax

yrqxpxbxax

yrqxpxbxax

=+++++

=+++++

=+++++

−−

−−

−−

221

2222

22

12

1121

21

11

...........................................

...

...



The Lagrange polynomial which fits n data points (x1, y1), (x2, y2) .. (xn, yn) is of n-1th order and has the form:

Lagrange Polynomials

nnnnnn

n

n

n

n

n

yxxxxxxxx

xxxxxxxx

yxxxxxxxx

xxxxxxxx

yxxxxxxxx

xxxxxxxxxf

×−−−−

−−−−+

+

×−−−−

−−−−+

×−−−−

−−−−=

−

−)(..))()((

)(..))()((...

)(..))()(()(..))()(()(..))()((

)(..))()(()(

1321

1321

22423212

431

11413121

432

[G James p 87]



The Lagrange formula exactly fits each of the data points

Consider the first data point y1=f(x1)

All terms in the formula except the first contain a numerator factor (x-x1) which is zero at the first data point x=x1

Consequently all terms except the first are zero at the first data point and the formula simplifies to:


1

11413121

14131211 )(..))()((

)(..))()(()(

y

yxxxxxxxxxxxxxxxxxf

n

n

=

×−−−−−−−−

=



Example: the hull drag F of a racing yacht as a function of the hull speed V:

V (m/s) F (N)0.0 0.00.5 20.411.0 92.751.5 181.002.0 421.452.5 1265.23

A 5th-order Lagrange polynomial can fit this data exactly

Extrapolating to a hull speed of 3 m/s:





23.1265)0.25.2)(5.15.2)(0.15.2)(5.05.2)(0.05.2(

)0.2)(5.1)(0.1)(5.0)(0.0(

45.421)5.20.2)(5.10.2)(0.10.2)(5.00.2)(0.00.2(

)5.2)(5.1)(0.1)(5.0)(0.0(

00.181)5.25.1)(0.25.1)(0.15.1)(5.05.1)(0.05.1(

)5.2)(0.2)(0.1)(5.0)(0.0(

75.92)5.20.1)(0.20.1)(5.10.1)(5.00.1)(0.00.1(

)5.2)(0.2)(5.1)(5.0)(0.0(

41.20)5.25.0)(0.25.0)(5.15.0)(0.15.0)(0.05.0(

)5.2)(0.2)(5.1)(0.1)(0.0(

0.0)5.20.0)(0.20.0)(5.10.0)(0.10.0)(5.00.0(

)5.2)(0.2)(5.1)(0.1)(5.0()(

×−−−−−

−−−−−+

×−−−−−

−−−−−+

×−−−−−

−−−−−+

×−−−−−

−−−−−+

×−−−−−

−−−−−+

×−−−−−

−−−−−=

VVVVV

VVVVV

VVVVV

VVVVV

VVVVV

VVVVVVf



Lagrange PolynomialsLagrange polynomials can be converted to explicit polynomial form

T (°C) F (MHz)20 1.0004030 1.0010740 1.00243

Lagrange polynomial:

00243.1)3040)(2040(

)30)(20(

00107.1)4030)(2030(

)40)(20(

00040.1)4020)(3020(

)40)(30()(

×−−−−+

×−−−−

+

×−−−−

=

TT

TT

TTTf




00243.1200

60050

00107.1100

8006000040.1200

120070)(

2

22

×+−+

×−

+−+×

+−=

TT

TTTTTf

00501215.0)60050(

01001070.0)80060(

00500200.0)120070()(

2

2

2

×+−+

×+−−

×+−=

TT

TT

TTTf

00113.10001055.000000345.0007290.3008560.8002400.6

)2506075.06006420.03501400.0()00501215.001001070.000500200.0()(

2

2

+−=+−+

−+−++−=

TT

TTTf



Least-Squares Curve FittingAny measurements of real physical quantities are subject to errors

It is unrealistic, therefore, to insist that a curve exactly fits the measured data points

Suppose that a set of n data points (x1,y1), (x2,y2) .. (xn,yn) is available, and a function f(x) is to be fitted to this data

The error εi between the function f(x) and the data point (xi,yi) is given by:

ii yxfε −= )(i

[G James p 78]



Least-Squares Curve FittingEach of the data points has an error and the function is chosen in such a way as to minimise these errors, so providing an optimum fit to the data

In order to determine the overall error between the function and the data the individual errors must be combined in some way

In the least-squares method the overall error E is taken to be the sum of the squares of the individual errors:

∑ −=∑===

n

iii

n

ii yxfεE

1

2

1

2 )(



Least-Squares Straight Line FitEquation representing a straight line:

and the overall error E is therefore given by:

Differentiating w.r.t. a:

Differentiating w.r.t. b:

∑−∑+∑=∑ −+=====

n

iii

n

ii

n

ii

n

iiii yxxbxaybaxx

aE

111

2

12222

dd

baxxf +=)(

2

1∑ −+=

=

n

iii ybaxE

∑−∑+∑=∑ −+=====

n

ii

n

i

n

ii

n

iii ybxaybax

bE

111121222

dd



Least-Squares Straight Line FitMinimum of the overall error E:

Not that the summations evaluate to numbers when the data is inserted

These simultaneous equation in a and b are solved to obtain the straight line least-squares fit

∑=∑+∑

∑=∑+∑

===

===n

ii

n

i

n

ii

n

iii

n

ii

n

ii

ybxa

yxxbxa

111

111

2

1



Least-Squares Straight Line FitExample: power output P of a semiconductor laser diode, operating at 35C, as a function of the drive current I:

I (mA) P (mW)70 1.3372 2.0874 2.8876 3.3178 3.8380 4.67

It is known that, above a certain threshold current, the laser power increases linearly with drive current so that it is appropriate to fit a straight line to the data



Least-Squares Straight Line FitThe first stage in obtaining the straight line fit is to evaluate the summations:

i 1 xi yi xi2 xiyi

1 1 70 1.33 4900 93.102 1 72 2.08 5184 149.763 1 74 2.88 5476 213.12 4 1 76 3.31 5776 251.565 1 78 3.83 6084 298.746 1 80 4.67 6400 373.60

Σ 6 450 18.10 33820 1379.88

These sums are used to set up the simultaneous equations



Least-Squares Straight Line Fit

88.1379

33820

1.18

450

61

1

1

2

1

1

1

=∑

=∑

=∑

=∑

=∑

=

=

=

=

=

in

ii

n

ii

n

ii

n

ii

n

i

yx

x

y

x

1.18645088.137945033820

=+=+

baba

Solve equations to obtain coefficients:

a=0.319 b=-21.0

so least-squares fit:

P = 0.319 I - 21.0



Least-Squares Straight Line Fit

10

12

14

16

18

20

0 200 400 600Height above sea level h (m)

Tem

pera

ture

T(°

C)

Least-squares fit:

T = 19 - 0.01h



Least-Squares Quadratic FitA quadratic has three coefficients a, b and c, which must be chosen to obtain the best fit to the data:

The overall error E is given by:

Differentiating with respect to a gives:

cbxaxxf ++= 2)(

2

1

2 ∑ −++==

n

iiii ycbxaxE

∑−∑+∑+∑=

∑ −++=

====

=n

iii

n

ii

n

ii

n

ii

n

iiiii

yxxcxbxa

ycbxaxxaE

1

2

1

2

1

3

1

4

1

22

2222

2dd



Least-Squares Quadratic Fit

∑=∑+∑+∑

∑=∑+∑+∑

∑=∑+∑+∑

====

====

====

n

ii

n

i

n

ii

n

ii

n

iii

n

ii

n

ii

n

ii

n

iii

n

ii

n

ii

n

ii

ycxbxa

yxxcxbxa

yxxcxbxa

1111

2

111

2

1

3

1

2

1

2

1

3

1

4

1

Setting derivatives to zero gives simultaneous linear equations:

These simultaneous equations can be solved to obtain the coefficients a, b and c.



Least-Squares Quadratic FitExample: voltage V of a copper/constantan thermocouple output as a function of temperature T.

T (°C) V (mV)0 0.00010 0.39120 0.78930 1.19640 1.61150 2.03560 2.46770 2.90880 3.35790 3.813

100 4.277



1 xi yi xi2 xi

3 xi4 xiyi xi

2yi1 1 0 0.000 0 0.00×100 0.000×100 0.00 0.0 2 1 10 0.391 100 1.00×103 1.000×104 3.91 39.13 1 20 0.789 400 8.00×103 1.600×105 15.78 315.64 1 30 1.196 900 2.70×104 8.100×105 35.88 1076.45 1 40 1.611 1600 6.40×104 2.560×106 64.44 2577.66 1 50 2.035 2500 1.25×105 6.250×106 101.75 5087.57 1 60 2.467 3600 2.16×105 1.296×107 148.02 8881.28 1 70 2.908 4900 3.43×105 2.401×107 203.56 14249.29 1 80 3.357 6400 5.12×105 4.096×107 268.56 21484.810 1 90 3.813 8100 7.29×105 6.561×107 343.17 30885.311 1 100 4.277 10000 1.00×106 1.000×108 427.70 42770.0

Σ 11 550 22.844 38500 3.03×106 2.533×108 1612.77 127366.7


1124

3246

5468

102844.2101000.1105000.5108500.3

106128.1105000.5108500.3100250.3

102737.1108500.3100250.3105333.2

×=×+×+×

×=×+×+×

×=×+×+×

cba

cba

cba




525

10196.410863.310145.4

−

−

−

×−=×=×=

cba

Solving these equations gives:

and the least-square quadratic fit is:

Alternatively, x values are associated with the voltage V, and the y values associated with the temperature T:

5225 10196.410863.310145.4 −−− ×−×+×= TTV

2121 10193.710561.210287.5 −− ×+×+×−= VVT




V T quadratic error0.000 0.000 0.072 0.0720.391 10.000 10.003 0.0030.789 20.000 19.946 -0.0541.196 30.000 29.940 -0.0601.611 40.000 39.951 -0.0492.035 50.000 49.990 -0.0102.467 60.000 60.024 0.0242.908 70.000 70.063 0.0633.357 80.000 80.073 0.0733.813 90.000 90.021 0.0214.277 100.000 99.917 -0.083

2121 10193.710561.210287.5 −− ×+×+×−= VVT



Numerical IntegrationNumerical integration is the process of evaluating definite integrals by numerical methods:

This is equivalent to solving Z = y(b) for the differential equation:

with the initial condition y(a) = 0

Another way of looking at the definite integral is that it is the area below the curve representing f(x)

∫=b

axxfZ d)(

)(xfdxdy =

[G James p 569]



Numerical Integration

∫=b

axxf d)(area

f(x)

xx=a x=b



Numerical IntegrationAnalytical integration should be used where possible because it will be more efficient and more accurate

Many functions cannot be integrated analytically, for example:

In some cases it is necessary to integrate numerical data: this cannot be done analytically

∫=a

xZ0

x- de2

∫−

=2/

0 22 sin1

dπ

φk

φZ

Gaussiandistribution

Ellipticintegrals

Equivalent noise bandwidth

∫=∞

0

2 d)(21 ωωjHπ

Z



Polar Planimeter



Trapezium Methodf(x)

xx0 x1 x2 x3 x4 x5 x6

f0 f1

Approximate area under curve by a trapezium

[G James p 570]



The area under the curve in each interval is approximated by a trapezium:

Normally the n intervals will be of equal width h:

so that:

Trapezium Method

)(2

)(21area

10

010

ffh

ffhhf

+=

−+=

h=x1-x0

f0 f1

nabh −

=

nkkhaxk ,..,2,1,0where =+=



Total area:

This formula requires the function f(x) to be evaluated twice at each x value (except at x0 and xn)

Trapezium Method

∑ +=

∑ +=

=−

=−

n

kkk

n

kkk

xfxfh

xfxfhZ

11

11

)()(2

)()(2

⎭⎬⎫

⎩⎨⎧

∑ +++=

⎭⎬⎫

⎩⎨⎧

∑ ++=

−

=

−

=1

1

1

10

)(21)()(

21

)(21)()(

21

n

k

n

knk

bfkhafafh

xfxfxfhZ



Example:

Splitting the integration range into 8 equal intervals:k xk f(xk)

0 0.00 0.0000001 0.25 0.0036702 0.50 0.0486753 0.75 0.1802834 1.00 0.3678795 1.25 0.5117476 1.50 0.5335847 1.75 0.4386578 2.00 0.293050

Σ* 2.231020 (Weighted sum)

Trapezium Method

xxxZ dexp2

0

24∫ −=

557755.0231020.225.0Z

=×=



This result is not very accurate because of the small number of intervals used (8)

To obtain a higher accuracy the integration range must be split into much larger number of intervals:

n Z100 0.560805200 0.560820500 0.560824

1000 0.560824

What value of n should be used to achieve some required degree of accuracy ?

Trapezium Method



Use the trapezium rule with successively larger values of nuntil the results agree to within the required accuracy:

Trapezium Method

x0 x4 x6 x8x1 x2 x3 x5 x7

n=8

x0 x4x1 x2 x3

n=4

x0 x1 x2

n=2

x0 x4x1 x2 x3

n=4



Example: increase number of intervals to 16:

k xk f(xk) 1 0.125 0.0002403 0.375 0.0171815 0.625 0.103246 7 0.875 0.2726009 1.125 0.45181011 1.375 0.53966313 1.625 0.49728415 1.875 0.367442Σ 2.249467

Trapezium Method

560061.0)249467.2231020.2(125.0

=+×=Z

Even values of khave already been evaluated for n=8

Sum for n=8



Integration of Tabulated DataThe trapezium approximation can be used to integrate tabulated data

Example: vertical acceleration a of a rocket measured at 0.5 s intervals:

k t(s) a(m/s2) 0 0.0 24.1 1 0.5 27.5 2 1.0 31.2 3 1.5 34.4 4 2.0 37.9 5 2.5 41.7 6 3.0 45.1 7 3.5 48.4 8 4.0 51.9



Integration of Tabulated Data

k t(s) a(m/s2) v(m/s) 0 0.0 24.1 0.000 1 0.5 27.5 12.900 2 1.0 31.2 27.575 3 1.5 34.4 43.9754 2.0 37.9 62.050 5 2.5 41.7 81.950 6 3.0 45.1 103.650 7 3.5 48.4 127.025 8 4.0 51.9 152.100

))((21

d

10010

011

0

aattv

tavvt

t

+−+≈

∫+=



Integration of Tabulated Data

k t(s) a(m/s2) v(m/s) h(m)0 0.0 24.1 0.000 0.0001 0.5 27.5 12.900 3.2252 1.0 31.2 27.575 13.3443 1.5 34.4 43.975 31.2314 2.0 37.9 62.050 57.7385 2.5 41.7 81.950 93.7386 3.0 45.1 103.650 140.1387 3.5 48.4 127.025 197.8068 4.0 51.9 152.100 267.588

))((21

d

10010

011

0

vvtth

tvhht

t

+−+≈

∫+=



Improper IntegralsIn integral is termed improper if:

1. It has an integrable singularity between its upper and lower limits, for example:

2. It has an integrable singularity at its upper or lower limits, for example:

3. Its upper limit is ∞ or its lower limit is -∞

xxZ d1

1

3/1∫=−

−

xxZ d1

0

3/1∫= −

[G James p 595]



Improper IntegralsType 1 improper integrals can always be converted to type 2 integrals by splitting the integration range

Suppose that f(x) has a singularity at c:

The resulting integrals have singularities at their integration limits, and no singularity between their integration limits

The resulting integrals are therefore type 2 improper integrals

xxfxxfxxfZb

c

c

a

b

ad)(d)(d)( ∫+∫=∫=



Improper IntegralsThe trapezium method evaluates the integrand f(x) at each of the integration limits

It cannot therefore be used with type 2 improper integrals

It is also unsuitable for integrating functions such as:

Here the integrand is perfectly well-behaved at x=0

Nevertheless, any attempt to evaluate the integrand at x=0 on a computer will lead to a divide-by-zero error

xx

xZ dsin1

0∫=



Midpoint Methodf(x)

xx1 x2 x3 x4 x5 x6

Approximate area under curve by a rectangle

[G James p 570]

f1



Normally the n intervals will be of equal width h:

and:

Total area:

Midpoint Method

nabh −

=

nkhkaxk ,..,2,1where2

)12(=

−+=

∑ ⎟⎠⎞

⎜⎝⎛ −+=

∑=

=

=n

k

n

kk

hkafh

xhfZ

1

1

2)12(

)(



Example:

Splitting the integration range into 8 equal intervals:

Midpoint Method

xx

xZπ

dsin2

0∫=

k xk f(xk)

1 0.392699 0.9744962 1.178097 0.7842133 1.963496 0.4705284 2.748894 0.1392145 3.534292 -0.1082776 4.319690 -0.2138767 5.105089 -0.1809728 5.890487 -0.064966

Σ 1.800360

413999.1800360.1785398.0

800360.18/2Z

=×=

×= π



This result is not very accurate because of the small number of intervals used (8)

To obtain a higher accuracy the integration range must be split into much larger number of intervals:

n Z100 1.418125200 1.418145500 1.418150

1000 1.418151

What value of n should be used to achieve some required degree of accuracy ?

Midpoint Method



Use the midpoint rule with successively larger values of n until the results agree to within the required accuracy:

Midpoint Method

x9x4 x6 x8x1x2x3 x5 x7

n=9

x1 x2 x3

n=3

x1 x2 x3

n=3

x1

n=1



Infinite Integration LimitsBoth trapezium and midpoint methods will fail if either of the limits are infinite:

It may be possible to replace infinite limits by large but finite values, but this is often impractical

Fortunately the problem of infinite integration limits can easily be overcome by using a variable transformation

The transformation used must convert infinite limits to finite values, thus allowing the definite integral to be evaluated numerically

∫=∞

0d)( xxfZ



Infinite Integration Limits

Let x=tan y; the limits x=0 and x=∞ become y=0 and y=π/2

so:

and the integral becomes:

∫=∞

0d)( xxfZ

yyy

yy

yy

yyx 2

2

2tan1

cossin

coscos

cossin

dd

dd

+=+==

∫ +=∫=∞ 2/

0

2

0dytan1)tan(d)(

πyyfxxfZ

yyx dtan1d 2+=



Infinite Integration LimitsExample: equivalent noise bandwidth B:

where:

to evaluate:

84/2/1680/28/31

042/1342

2

xxdxxd

nxn

ir

ir

−=+−=

=−=

∫=∞

0

2 d)(21 ωωjHπ

B

1680/)(84/)(28/)(32/)(142/)(1)( 432

2

ωjωjωjωjωjωjH

++++

+=

22

222)()(

ir

ir

dd

nnjxHxf

+

+==



Infinite Integration Limits

k yk xk=tan(yk) f(xk) (1+xk2)f(xk)

1 0.098175 0.098491 0.999192 1.0088852 0.294524 0.303347 0.992355 1.0836713 0.490874 0.534511 0.976417 1.2553824 0.687224 0.820679 0.945119 1.5816705 0.883573 1.218504 0.882234 2.1921336 1.079923 1.870869 0.740300 3.3314607 1.276272 3.296560 0.365908 4.3423438 1.472622 10.153170 0.030949 3.221357

Σ 18.016891

Splitting the integration range of π/2 into 8 intervals of π/16:

563028.02

537608.3016891.18196349.0

==

=×=

πZB

Z



OptimisationOptimisation involves adjusting the parameters of a system to obtain the best performance

Most applications of computers in engineering are concerned with analysis

Optimisation differs from these techniques in that it is a true design method

Essentially optimisation closes the loop around analysis; if a system can be analysed then it can also be optimised

Optimisation is usually formulated as a function minimisation problem



Function MinimisationMany designs can be optimised by minimising the difference between their behaviour and some target behaviour

If some performance item is to be maximised (for example efficiency) then function minimisation can still be used:

The function to be minimised is known as the objective function

It is possible to optimise functions of several variables, but the discussion here will be limited to single-variable optimisation

Two methods will be considered: approximation methods and golden-section search

)(minimum)(maximum ff −=



Thermistor Thermometer

TBth AR /e= TBref

threfout

AeRRV

RRRVV /

0

0

0

0+

=+

=

Typical values: A = 4.3×10-3, B = 4000.0

Vout

Vref

Rth

R0



Thermistor Thermometer

Actual voltage

Linearapproximation

Temperatureerror

Temperature0C 50C

V(50)

V(0)

Vout



Thermistor ThermometerTe

mpe

ratu

re E

rror

10ºC

0ºC100 Ω 1 k Ω 10 k Ω

R0



Approximation Methods

x

Objective function f(x)

Cubicpolynomial

Cubic exactly fitsf(x) at 4 x values

Fit a cubic polynomial to the objective function at 4 points:



Coefficients a, b, c, d of cubic polynomial are chosen so that it fits f(x) at four x values:

Differentiating:

which has a solution:

Select minimum using second derivative:


dcxbxaxxp +++= 23)(

023dd 2 =++= cbxax

xp

aacbbx

332

min−±−

=

026dd

2

2>+= bax

xp



The minimum xmin of the polynomial is only an approximation to the minimum of the objective function

The process must be repeated, using xmin in place of the worst of the original points, until a satisfactory accuracy has been obtained

Approximation methods are useful in certain applications but they require the objective function to be well-behaved

The objective function for the thermistor thermometer has a discontinuous gradient at the minimum, and cannot be accurately approximated around this point




Given an initial interval containing a minimum split the interval into two or more sections and determine which of these contains the minimum

Repeat the process until the minimum has been located to a sufficient degree of accuracy

Search methods do not require the objective function to be well-behaved and will find the minimum even if the objective function is discontinuous.

Since the interval is reduced by a fixed ratio at each iteration, search methods have an entirely predictable convergence

Search Methods



Binary SearchO

bjec

tive

func

tion

f(x)

xa bcc-h c+h



Golden-Section SearchThe binary search method has the disadvantage that it is difficult to choose a suitable value for h

If h is large then the convergence rate is reduced

If h is small then errors can occur because of rounding

These problems are overcome in the Golden-Section method, where h is large but only one objective evaluation is required at each iteration

The name Golden-Section derives from the convergence rate which is 1.618



Golden-Section Search

a p q ba’ p’ q’ b’

a” p” q” b”

Obj

ectiv

e fu

nctio

n f(x

)

x



Golden Section SearchRange is reduced by k at each iteration:

kabpbaq −=−=−

Second iteration:

qpbbpa === ///

Applying same reduction factor:

kabpbaq

////// −=−=−

kpbqb −=−

or:



Golden Section SearchCombining equations and using the fact that:

gives:)()( aqabqb −−−=−

012 =−− kk 618.12

51 =+=k

kk111

2 −=

or:k

ababk

abk

pb −−−=−=− )(2

thus:



Golden-Section Search AlgorithmSearch interval a to b contains at least one minimum

While (b-a) is greater than required absolute precision

Let c = (b-a)/k; then p = b-c, q = a+c

Evaluate the objective function at p and q

If f(p) < f(q) then minimum lies between a and q: let a'=a, b'=q

else minimum lies between p and b: let a'=p, b'=b

end while



Golden-Section SearchThe search is continued until the search interval b-a is less than the required precision

Convergence rate is k = 1.618

If required precision is ε and the initial search interval is a to b, then number of iterations n will be:

Golden-Section search will only work if the initial search interval brackets a minimum

)log()log()log(

kεabn −−=



Golden-Section SearchThermistor thermometer: choose initial range a=1kΩ, b=4kΩ

so that:Ωk854.1

618.1Ωk3 ==−=−=−

kabpbaq

Ω=Ω+Ω=Ω+=Ω=Ω−Ω=Ω−=

k854.2k854.1k1k854.1k146.2k854.1k4k854.1

aqbp

a=1kΩ b=4kΩp=2.146kΩ q=2.854kΩ

f(p)=1.171 f(q)=2.061

1.854kΩ1.854kΩ



Golden-Section SearchNext iteration: a'=1kΩ, b'=2.854kΩ

so that:Ωk146.1

618.1Ωk854.1'''''' ==−=−=−

kabpbaq

Ωk146.2Ωk146.1Ωk1Ωk146.1''Ωk708.1Ωk146.1Ωk854.2Ωk146.1''

=+=+==−=−=

aqbp

a'=1kΩ b'=2.854kΩp'=1.708kΩ q'=2.146kΩ

f(p')=2.179 f(q')=1.171

1.146kΩ1.146kΩ



a p q b f(p) f(q)1.000×10+3 2.146×10+3 2.854×10+3 4.000×10+3 1.171×10+0 2.061×10+01.000×10+3 1.708×10+3 2.146×10+3 2.854×10+3 2.179×10+0 1.171×10+01.708×10+3 2.146×10+3 2.416×10+3 2.854×10+3 1.171×10+0 1.257×10+01.708×10+3 1.979×10+3 2.146×10+3 2.416×10+3 1.513×10+0 1.171×10+01.979×10+3 2.146×10+3 2.249×10+3 2.416×10+3 1.171×10+0 9.810×10-12.146×10+3 2.249×10+3 2.313×10+3 2.416×10+3 9.810×10-1 1.063×10+02.146×10+3 2.210×10+3 2.249×10+3 2.313×10+3 1.052×10+0 9.810×10-12.210×10+3 2.249×10+3 2.274×10+3 2.313×10+3 9.810×10-1 9.921×10-12.210×10+3 2.234×10+3 2.249×10+3 2.274×10+3 1.008×10+0 9.810×10-12.234×10+3 2.249×10+3 2.259×10+3 2.274×10+3 9.810×10-1 9.657×10-12.249×10+3 2.259×10+3 2.264×10+3 2.274×10+3 9.657×10-1 9.753×10-12.249×10+3 2.255×10+3 2.259×10+3 2.264×10+3 9.715×10-1 9.657×10-12.255×10+3 2.259×10+3 2.261×10+3 2.264×10+3 9.657×10-1 9.689×10-12.255×10+3 2.257×10+3 2.259×10+3 2.261×10+3 9.679×10-1 9.657×10-12.257×10+3 2.259×10+3 2.259×10+3 2.261×10+3 9.657×10-1 9.664×10-12.257×10+3 2.258×10+3 2.259×10+3 2.259×10+3 9.665×10-1 9.657×10-1

Golden-Section Search

Minimum at R0 = 2.259 kΩ where peak error is 0.97 C.



Bracketing a Minimum

a b ca’ b’ c’

a” b” c”

Obj

ectiv

e fu

nctio

n f(x

)

x



Differential EquationsThe time-domain response of electronic systems and the dynamical behaviour of mechanical systems can be described by differential equations

Many differential equations can be solved analytically, particularly those describing idealised systems

The more realistic the mathematical model of a physical system, the less likely it is that an analytical solution can befound

In cases where an analytical solution is not available the differential equations must be solved by numerical methods and as a consequence the solution will not be exact



Differential EquationsWe shall first consider the problem of solving first-order non-linear differential equations of the form:

Most physical systems cannot be described by a single first-order differential equation

Fortunately any nth-order differential equation can be reduced to n coupled first-order differential equations

The solution of nth-order differential equations is then simply a generalisation of the method used for solving a single equation.

),(dd txf

tx =



Differential Equations

Suppose that prior to t=0 the switch is open and the capacitor discharged: vC=0

Then at t=0 the switch is closed and the capacitor starts to charge towards the input voltage vi

C10nF

R100kΩ

vi4Vdc

vC(t)i(t)



Differential Equations

C10nF

R100kΩ

vi4Vdc

vC(t)i(t)

ic

c

vtvtRittvCti

=+

=

)()(d

)(d)(

icc vtvttvRC =+ )(

d)(d

This differential equation has an analytical solution:

1)( RCt

ic evtv−

−=



A first-order differential equation can be written in the form:

Provided that ∆x and ∆t are sufficiently small:

If x is known at time t, then its value at time t+∆t is:

This is the Euler integration formula

Euler Integration

),(dd txf

tx=

),(∆dd∆∆or

dd

∆∆ txft

txtx

tx

tx

=≈≈

),(∆)(dd∆)(

∆)()∆(

txfttxtxttx

xtxttx

+≈

+≈

+=+



Equation for the capacitor voltage vc can be rewritten:

Using Euler integration formula:

Time step ∆t should be between 1/10 and 1/1000 of shortest time constant T0:

Choose time step:

Euler Integration

)(1d

)(d tvvRCt

tvci

c −=

)(∆)()∆( tvvRC

ttvttv cicc −+=+

s101010 3850

−− =×== RCT

s10∆ 4−=t



t x(t) f(x,t) ∆x=∆t.f(x,t) x(exact)

0.0000 0.000000 4000.00 0.400000 0.0000000.0001 0.400000 3600.00 0.360000 0.3806500.0002 0.760000 3240.00 0.324000 0.7250770.0003 1.084000 2916.00 0.291600 1.0367270.0004 1.375600 2624.40 0.262440 1.3187200.0005 1.638040 2361.96 0.236196 1.5738770.0006 1.874236 2125.76 0.212576 1.8047530.0007 2.086812 1913.19 0.191319 2.0136590.0008 2.278131 1721.87 0.172187 2.2026840.0009 2.450318 1549.68 0.154968 2.3737210.0010 2.605286 1394.71 0.139471 2.528482

Euler Integration

At a time t=1 ms the error in the numerical result is 0.077 V corresponding to a relative precision of about 3%



Euler IntegrationEuler integration using smaller time steps:

∆t Iterations vc error

10-4 10 2.605286 0.07680410-5 100 2.535871 0.00738910-6 1000 2.529218 0.000736

A 10-fold reduction in ∆t reduces the error by a factor of 10

Repeat Euler integration with successively smaller time steps until two results agree to within the required precision

In practice, because of its poor error performance, Euler integration is rarely used



Euler Integration

t

x

x(t)

t+∆t

x(t+∆t)

t

),(ddslope txf

tx == ),(∆∆ txftx =

t∆



Heun IntegrationThe Heun method obtains a better estimate of the slope by averaging the derivative at the start and end of the interval

An approximation to the slope k2 at the end of the interval

The average of k1 and k2 is now used to extrapolate over the full interval from the initial point:

11

1.)(

),(kttxx

txfk∆+=

=

( )ttxfk ∆+= ,12

212)()( kkttxttx +∆+=∆+



Heun Integration

t

x

x(t)

t+∆tt

),(slope 12 ttxfk ∆+==

212kktx +∆=∆

2slope 21 kk +

=

x1

x(t+∆t)

),(slope 1 txfk ==

∆t



Heun Integrationt x(t) k1=f(x,t) x1(t) k2 ∆x exact

0.0000 0.000000 4000.00 0.400000 3600.00 0.380000 0.0000000.0001 0.380000 3620.00 0.742000 3258.00 0.343900 0.3806500.0002 0.723900 3276.10 1.051510 2948.49 0.311230 0.7250770.0003 1.035130 2964.87 1.331617 2668.38 0.281663 1.0367270.0004 1.316792 2683.21 1.585113 2414.89 0.254905 1.3187200.0005 1.571697 2428.30 1.814527 2185.47 0.230689 1.5738770.0006 1.802386 2197.61 2.022147 1977.85 0.208773 1.8047530.0007 2.011159 1988.84 2.210043 1789.96 0.188940 2.0136590.0008 2.200099 1799.90 2.380089 1619.91 0.170991 2.2026840.0009 2.371090 1628.91 2.533981 1466.02 0.154746 2.3737210.0010 2.525836 1474.16 2.673252 1326.75 0.140046 2.528482

The error at time t=1 ms of 0.002646 compares favourably with that obtained using the Euler method; however the Heun method involved twice as many evaluations of f(x,t).



Heun Integration

Heun integration using smaller time steps:


10-4 10 2.525836 0.00264610-5 100 2.528458 0.00002510-6 1000 2.528482 0.000000


The Heun method is one of a family of predictor-corrector integration algorithms



2nd-Order Runge-Kutta IntegrationThe Euler method can be improved by extrapolating to the midpoint of the interval t+∆t/2:

and evaluating the slope k2 at the midpoint:

This slope, which is a good approximation to the average slope over the interval, is used to extrapolate over the full interval from the initial point:

11

1

2∆)(

),(

kttxx

txfk

+=

=

( )2/∆,12 ttxfk +=

2∆)()∆( kttxttx +=+



2nd-Order Runge-Kutta Integration

t

x

x(t)

t+∆tt

)2/∆,(slope 12 ttxfk +==

t+∆t/2

2/∆t

2∆∆ ktx =2slope k=

2/∆t

x1

x(t+∆t)

),(slope 1 txfk ==



2nd-Order Runge-Kutta Integrationt x(t) k1=f(x,t) ∆t/2.f(x,t) x1 k2 ∆t.k2 x(exact)

0.0000 0.000000 4000.00 0.200000 0.200000 3800.00 0.380000 0.000000

0.0001 0.380000 3620.00 0.181000 0.561000 3439.00 0.343900 0.380650

0.0002 0.723900 3276.10 0.163805 0.887705 3112.30 0.311230 0.725077

0.0003 1.035129 2964.87 0.148244 1.183373 2816.63 0.281663 1.036727

0.0004 1.316792 2683.21 0.134160 1.450953 2549.05 0.254905 1.318720

0.0005 1.571697 2428.30 0.121415 1.693112 2306.89 0.230689 1.573877

0.0006 1.802386 2197.61 0.109881 1.912266 2087.73 0.208773 1.804753

0.0007 2.011159 1988.84 0.099442 2.110601 1889.40 0.188940 2.013659

0.0008 2.200099 1799.90 0.089995 2.290094 1709.91 0.170991 2.202684

0.0009 2.371090 1628.91 0.081446 2.452535 1547.47 0.154747 2.373721

0.0010 2.525836 1474.16 0.073708 2.599544 1400.46 0.140046 2.528482

The error at time t=1 ms of 0.002646 compares favourably with that obtained using the Euler method; however the Runge-Kuttamethod involved twice as many evaluations of f(x,t).



2nd-Order Runge-Kutta Integration2nd-order Runge-Kutta integration using smaller time steps:


10-4 10 2.525836 0.00264610-5 100 2.528458 0.00002410-6 1000 2.528482 0.000000


The 2nd-order Runge-Kutta method is one of a family of integration algorithms, of which the 4th-order Runge-Kutta, or RK-4, algorithm is the most generally useful.



The slopes k1, k2, k3, k4 are evaluate at 4 points:

The the weighted sum of these slopes is used to extrapolate over the complete interval:

4th-Order Runge-Kutta Integration

( )tttkxfk

tttkxfk

tttkxfk

txfk

∆,∆2∆,

2∆

2∆,

2∆

),(

34

23

12

1

++=

⎟⎠⎞

⎜⎝⎛ ++=

⎟⎠⎞

⎜⎝⎛ ++=

=

( )4321 226∆)()∆( kkkkttxttx ++++=+



4th-Order Runge-Kutta Integration

t

x

x(t)

t+∆tt

2k

t+∆t/2

3k

x(t+∆t)

1k

2k

3k4k

)22(61

4321 kkkkk +++=



4th-Order Runge-Kutta IntegrationBecause of the complexity of the fourth-order Runge-Kuttaalgorithm it is normally performed on a computer

∆t Iterations vc error10-4 10 2.528481 0.00000110-5 100 2.528482 0.00000010-6 1000 2.528482 0.000000

10 iterations of the Runge-Kutta algorithm provide much better accuracy than 1000 iterations of the Euler algorithm

1000000 Euler iterations are required to achieve the same accuracy as 10 4th-order Runge-Kutta iterations!



nth-Order Differential EquationsThe Euler and Runge-Kutta methods developed so far can be used to solve first-order differential equations:

Most real systems are described by higher-order differential equations

A general nth-order differential equation can be written in the form:

),(dd txf

tx =

⎟⎟⎠

⎞⎜⎜⎝

⎛=

−

−t

tx

tx

txxf

tx

n

n

n

n,

dd...,

dd,

dd,

dd

1

1

2

2



nth-Order Differential EquationsTo reduce this nth-order equation to first-order equations a set of n-1 new variables are introduced:

tx

txx

tx

txx

tx

txx

txx

nn

nn d

ddd

.............d

ddd

dd

dddd

21

11

23

33

12

22

1

−−

−

− ==

==

==

=

( )txxxxf

tt

xtx

txxf

tx

tx

n

n

n

n

nn

,...,,,

,dd...,

dd,

dd,

dd

dd

121

1

1

2

2

1

−

−

−

−

=

⎟⎟⎠

⎞⎜⎜⎝

⎛=

=

so that:



nth-Order Differential EquationsThe n first-order differential equations:

( )txxxxft

x

xt

x

xt

x

xt

x

xtx

nn

nn

,,...,,,d

dd

d.............

ddd

ddd

1211

12

32

21

1

−−

−−

=

=

=

=

=The set of n first-order differential equations can then be solved by the Euler or Runge-Kutta method

Note that the equations are not independent



nth-Order Differential EquationsExample:

Create 2 new variables:

58dd2

dd2

dd

2

2

3

3=+++ xt

tx

tx

tx

5822dd

ddd

ddd

123

32

21

1

+−−−==

=

=

xtxxtx

tx

xtx

xtx

tx

txx

txx

dd

dddd

12

22

1

==

=



n-Variable Euler MethodAn iteration consists of updating each of the n variables using the Euler formula

It is important that all the n derivatives f0 .. fn are calculated before any of the n variables are updated.

5822d

dd

ddd

1222

211

10

+−−−==

==

==

xtxxft

x

xftx

xftx



n-Variable Euler MethodSolution at t=2 s using a time step ∆t=0.01 s and starting from an initial condition of x=0, x1=0 and x2=0 at t=0:

t x x1 x2 ∆x=∆t.f0 ∆x1=∆t.f1 ∆x2=∆t.f2

0.00 0.00000 0.00000 0.00000 0.00000 0.00000 0.050000.01 0.00000 0.00000 0.05000 0.00000 0.00050 0.049000.02 0.00000 0.00050 0.09900 0.00000 0.00099 0.048010.03 0.00000 0.00149 0.14701 0.00001 0.00147 0.047030.04 0.00002 0.00296 0.19404 0.00003 0.00194 0.046060.05 0.00005 0.00490 0.24010 0.00004 0.00240 0.04510

1.99 1.28996 -0.52439 -4.87962 -0.00524 -0.04879 -0.047282.00 1.28471 -0.57319 -4.92690 -0.00573 -0.04926 -0.04555

The Runge-Kutta method can also be generalised to deal with sets of n coupled first-order differential equations



Simulation of Electronic SystemsLinear electronic systems can be characterised by a frequency response function H(jω):

where X(jω) is the Fourier transform of the input x(t), and Y(jω) is the Fourier transform of the output y(t)

Introduce an auxiliary variable z(t) with Fourier transform Z(jω):

so that:

nn

nn

nn

nn

ωbωbωbωbbωaωaωaωaa

ωXωYωH

)j()j(..)j()j()j()j(..)j()j(

)j()j()j( 1

12

210

11

2210

+++++

+++++==

−−

−−

)j()j()j(..)j()j()j( 11

2210 ωZωaωaωaωaaωY n

nn

n +++++= −−

)j()j()j()j(..)j()j( 11

2210 ωXωZωbωbωbωbb n

nn

n =+++++ −−



Simulation of Electronic SystemsThe equivalent of jω in the frequency domain is d/dt in the time domain:

The differential equation involving z(t) and x(t) is therefore:

This differential equation in z can now be solved numerically in the usual way

First the nth-order differential equation is converted to n first-order differential equations

n

nn

tω

tω

tω

dd)j(

dd)j(

dd)j( 2

22 →→→

)()(dd

dd..

dd

dd

1

112

2210 txtz

tb

tb

tb

tbb n

nnn

nn =

⎪⎭

⎪⎬⎫

⎪⎩

⎪⎨⎧

+++++−

−

−



Simulation of Electronic SystemsA set of n-1 new variables are introduced to replace the first n-1 derivatives of z:

)(

..1d

dd

d.............

dddd

11

1101

12

21

1

txzb

zbzbbt

z

zt

z

ztz

ztz

nn

n

n

nn

+−

−−−=

=

=

=

−−

−

−−

tz

tzz

tz

tzz

tzz

nn

nn d

ddd

.............dd

dddd

21

11

12

22

1

−−

−

− ==

==

=



Simulation of Electronic SystemsConverting the defining equation for Z(jω) into the time domain:

The output y can now be obtained by substituting z, z1, ..zn-1

)(..

)(....

dd..

)(dd

dd..

dd)(

11

111

10

0

1111011110

111110

1

1110

txbaz

bbaaz

bbaaz

bbaa

txzbzbzbbazazaza

tzazazaza

tzt

at

at

aaty

n

nn

n

nnn

n

n

n

n

nnnn

nn

nnnn

n

nnn

nn

+⎟⎠

⎞⎜⎝

⎛−++⎟

⎠

⎞⎜⎝

⎛−+⎟

⎠

⎞⎜⎝

⎛−=

+−−−−++++=

++++=

⎪⎭

⎪⎬⎫

⎪⎩

⎪⎨⎧

++++=

−−

−

−−−−

−−−

−

−

−



Simulation of Electronic SystemsExample:

This circuit has a frequency-response function:

0.01µF

0.05µF1.67H

10kΩ

Input Output

212274

28

)j(1033.8)j(1000.1)j(1000.51)j(1067.11)j(

ωωωωωH

−−−

−

×+×+×+

×+=



Simulation of Electronic SystemsThis response is 3rd-order so that their are 3 first-order differential equations in the coupled set:

and the output y(t) is given by:

122

71

42

2

211

10

1033.8)(1000.11000.5

dddddd

−

−−

×

+×−×−−==

==

==

txzzzft

z

zftz

zftz

281067.1)( zzty −×+=



Simulation of Electronic Systems

23

23

)105.0(2)100.2(exp)(

−

−

×

×−= ttx

Outputy(t)

0.0

1.0

t0.0ms 5.0ms

numerical methodsprofesores.dcb.unam.mx/users/gustavorb/mn/glyn james.pdfemail: j.b.grimbleby...

Documents