1 chapter 5 one dimensional search. 2 chapter 5 unidimensional search (1) if have a search...
TRANSCRIPT
1
Ch
apte
r 5
Chapter 5
One Dimensional Search
2
Ch
apte
r 5
Unidimensional Search
(1) If have a search direction, want to minimize in that direction by numerical methods
(2) Search Methods in General2.1. Non Sequential – Simultaneous evaluation of f at n points – no good (unless on parallel computer).2.2. Sequential – One evaluation follows the other.
3
Ch
apte
r 5
(3) Types of search that are better or best is often problem dependent. Some of the types are:
a. Newton, Quasi-Newton, and Secant methods.b. Region Elimination Methods (Fibonacci, Golden
Section, etc.).c. Polynomial Approximation (Quadratic Interpolation,
etc.).d. Random Search
(4) Most methods assume (a) a unimodal function, (b) that the min is bracketed at the start and (c) also you start in a direction that reduces f.
4
Ch
apte
r 5
To Bracket the Minimum
)()( untilx doubling Continue
)()( Compute .2
2let ),()( If
let ),()( If
)( and )( Compute 1.
)1()0()()0(
)0()1(
)0()0(
)0()0(
0)0(
kk
NEW
OLDNEW
OLDNEW
xxfxxf
xxfxf
xxxfxxf
xxxfxxf
xxfxf
5
Ch
apte
r 5
points)closest
the(using f(x) minimum thegivingpoint on the
bracket a keep you to enables point that theDiscard
.,,, points spacedequally 4 have nowYou
)( Compute 3.
)1()2()2
12()3(
)2()1(
xxxx
xxf kk
6
Ch
apte
r 5
1. Newton’s Method
Newton’s method for an equation is
)(
)(
)(
)()(
0))(()()(
0
00
0
00
000
xf
xfxxor
xf
xfxx
xxxfxfxf
Application to Minimization
The necessary condition for f(x) to have a local minimumis f′(x) = 0. Apply Newton’s method.
)(
)()(
)()()1(
k
kkk
xf
xfxx
7
Ch
apte
r 5
ExamplesMinimize
2
1)0(
2
1)0(
2
)0(21)0()1(
2
21
2210
222
2
2)(
2)(
)(
a
ax
a
ax
a
xaaxx
axf
xaaxf
xaxaaxf
Minimize
Continue
x
x
xxxx
xxf
xxxf
xxxf
100.0212
231,1at xStart
212
23
212)(
24)(
1)(
)1((0)
2
4)0()1(
2
3
24
8
Ch
apte
r 5
Advantages of Newton’s Method
(1) Locally quadratically convergent (as long as f′(x) is positive – for a minimum).
(2) For a quadratic function, get min in one step.
Disadvantages
(1) Need to calculate both f′(x) and f″(x)(2) If f″(x)→0, method converges slowly(3) If function has multiple extrema, may not converge
to global optimum.
9
Ch
apte
r 5
2. Finite-Difference Newton Method
Replace derivatives with finite differences
2
)()1(
)()(2)(2
)()(
hhxfxfhxf
hhxfhxf
xx kk
DisadvantageNow need additional function evals (3 here vs. 2 for Newton)
10
Ch
apte
r 5
3. Secant(Quasi-Newton) Method
Analogous equation to (A) is
)(0)()( )()( Bxxmxf kk
The secant approximates f″(x) as a straight line
)()(
)()(
)()()1(
)()(
)()(
)()(
)(
)()(
pq
pq
kkk
pq
pq
xx
xfxf
xfxx
xx
xfxfm
11
Ch
apte
r 5
Start the Secant method by using 2 points spanning x at which first derivatives are of opposite sign.
For next stage, retain either x(q) or x(p) so that the pair of derivatives still have opposite sign.
12
Ch
apte
r 5
Order of Convergence
Can be expressed in various ways. Want to consider how
10*)(
*)1(
*)(
ccxx
xx
Linear
kasxx
k
k
k
usually slow in practice
13
Ch
apte
r 5
10*)(
*)1(
pccxx
xx
POrder
pk
k
Fastest in practiceIf p = 2, quadratic convergencep = 1.32 ?
)0(0lim*)(
*)1(
kascandcr
xx
xx
rSuperlinea
kkk
k
k
Usually fast in practice
Some methods can show theoretically whatthe order is.
14
Ch
apte
r 5
15
Ch
apte
r 5
16
Ch
apte
r 5
17
Ch
apte
r 5
18
Ch
apte
r 5
19
Ch
apte
r 5
Quadratic Interpolation
Approximate f(x) by a quadratic function.Use 3 points
2333
2222
2111
321
2
)(
)(
)(
)(),(,2
*20)(:
)(
cxbxaxf
cxbxaxf
cxbxaxf
xfxf)f(xc
bxsocxbxfMinimize
cxbxaxf
c b,a, for equations ussimultaneo 3 Solve
points3theatf(x)Evaluate
20
Ch
apte
r 5
21 1 1 1
22 2 2 2
23 3 3 3
2 21 1 1 1
2 22 2 2 2
2 23 3 3 3
1 ( ) 1 ( )1 ( ) 1 ( )1 ( ) 1 ( )
1 1
1 1
1 1
f x x x f xf x x x f xf x x x f x
b cx x x x
x x x x
x x x x
(or use Gaussian elimination)
21
Ch
apte
r 5
2 2 2 22 3 3 2 1 3 3 1
2 21 2 2 1
2 2 2 2 2 21 3 2 2 3 1 3 2 1
2 2 2 2 21 2 3 2 3 1 3 2
1 ( ) ( ) 1 ( ) ( )
1 ( ) ( ) : Numerator
( ) ( ) ( ) ( )
( )( ) ( )( ) ( )( ) :
Denominator
b f x x f x x f x x f x x
f x x f x x
c f x x x f x x x f x x x
f x x x f x x x f x x x
c
bx
2*
22
Ch
apte
r 5
23
Ch
apte
r 5
24
Ch
apte
r 5
25
Ch
apte
r 5
26
Ch
apte
r 5
27
Ch
apte
r 5
28
Ch
apte
r 5