CS B553: ALGORITHMS FOR OPTIMIZATION AND LEARNINGUnivariate optimization

x

f(x)

KEY IDEAS

Critical points Direct methods

Exhaustive search Golden section search

Root finding algorithms Bisection [More next time]

Local vs. global optimization Analyzing errors, convergence rates

x

f(x)Local maxima

Local minimaInflection point

Figure 1

x

f(x)

a b

Figure 2a

x

f(x)

a b

Find critical points, apply 2nd derivative test

Figure 2b

x

f(x)

a b

Figure 2b

x

f(x)

a b

Global minimum must be one of these points

Figure 2c

x

f(x)

a b

Exhaustive grid search

Figure 3

x

f(x)

a b

Exhaustive grid search

x

f(x)

Two types of errors

x* xt

f(xt)

f(x*)

Geometric error

Analy

tica

l err

or

Figure 4

x

f(x)

a b

Does exhaustive grid search achieve e/2 geometric error?

e

x*

x

f(x)

a b

Does exhaustive grid searchachieve e/2 geometric error?

Not necessarily for multi-modal objective functions

Error

x*

LIPSCHITZ CONTINUITYSlope +K

Slope -K

|f(x)-f(y)| K|x-y|

Figure 5

x

f(x)

a b

Exhaustive grid search achieves Ke/2 analytical error in worst case

e

Figure 6

x

f(x)

a b

Golden section search

m

Bracket [a,b]Intermediate point m with f(m) < f(a),f(b)

Figure 7a

x

f(x)

a b

Golden section search

m

Candidate bracket 1 [a,m]

c

Candidate bracket 2 [c,b]

Figure 7b

x

f(x)

a b

Golden section search

m

Figure 7b

x

f(x)

a b

Golden section search

mc

Figure 7b

x

f(x)

a b

Golden section search

m

Figure 7b

x

f(x)

a b

Optimal choice: based on golden ratio

m

Choose c so that (c-a)/(m-c) = , where is the golden ratio=> Bracket reduced by a factor of -1 at each step

c

NOTES

Exhaustive search is a global optimization: error bound is for finding the true optimum

GSS is a local optimization: error bound holds only for finding a local minimum

Convergence rate is linear: with xn = sequence of bracket midpoints

x

f(x)

Root finding: find x-value where f’(x) crosses 0

f’(x)

Figure 8

Bisection

g(x)

a b

Bracket [a,b]Invariant: sign(f(a)) != sign(f(b))

Figure 9a

Bisection

g(x)

a b

Bracket [a,b]Invariant: sign(f(a)) != sign(f(b))

m

Figure 9

Bisection

g(x)

a b

Bracket [a,b]Invariant: sign(f(a)) != sign(f(b))

Figure 9

Bisection

g(x)

a b

Bracket [a,b]Invariant: sign(f(a)) != sign(f(b))

m

Figure 9

Bisection

g(x)

a b

Bracket [a,b]Invariant: sign(f(a)) != sign(f(b))

Figure 9

Bisection

g(x)

a b

Bracket [a,b]Invariant: sign(f(a)) != sign(f(b))

m

Figure 9

Bisection

g(x)

a b

Bracket [a,b]Invariant: sign(f(a)) != sign(f(b))

Linear convergence: Bracket size is reduced by factor of 0.5 at each iteration

Figure 9

NEXT TIME

Root finding methods with superlinear convergence

Practical issues