nonlinear optimization: algorithms and modelsrvdb/542/lectures/lec23.pdf · home page title page...
TRANSCRIPT
-
Home Page
Title Page
Contents
JJ II
J I
Page 1 of 33
Go Back
Full Screen
Close
Quit
Nonlinear Optimization:
Algorithms and Models
Robert J. Vanderbei
December 12, 2005ORF 522
Operations Research and Financial Engineering, Princeton University
http://www.princeton.edu/rvdb
http://www.princeton.edu/~rvdbhttp://www.princeton.edu/~rvdb
-
Home Page
Title Page
Contents
JJ II
J I
Page 2 of 33
Go Back
Full Screen
Close
Quit
1. Outline
Algorithm Basic Paradigm
Step-Length Control
Diagonal Perturbation
Convex Problems Minimal Surfaces
Digital Audio Filters
Nonconvex Problems Celestial Mechanics
Putting on an Uneven Green
Goddard Rocket Problem
http://www.princeton.edu/~rvdb
-
Home Page
Title Page
Contents
JJ II
J I
Page 3 of 33
Go Back
Full Screen
Close
Quit
The Interior-Point Algorithm
http://www.princeton.edu/~rvdb
-
Home Page
Title Page
Contents
JJ II
J I
Page 4 of 33
Go Back
Full Screen
Close
Quit
2. Introduce Slack Variables
Start with an optimization problemfor now, the simplest NLP:
minimize f (x)subject to hi(x) 0, i = 1, . . . ,m
Introduce slack variables to make all inequality constraints intononnegativities:
minimize f (x)subject to h(x) w = 0,
w 0
http://www.princeton.edu/~rvdb
-
Home Page
Title Page
Contents
JJ II
J I
Page 5 of 33
Go Back
Full Screen
Close
Quit
3. Associated Log-Barrier Problem
Replace nonnegativity constraints with logarithmic barrier termsin the objective:
minimize f (x) m
i=1
log(wi)
subject to h(x) w = 0
http://www.princeton.edu/~rvdb
-
Home Page
Title Page
Contents
JJ II
J I
Page 6 of 33
Go Back
Full Screen
Close
Quit
4. First-Order Optimality Conditions
Incorporate the equality constraints into the objective using La-grange multipliers:
L(x, w, y) = f (x) m
i=1
log(wi) yT (h(x) w)
Set all derivatives to zero:
f (x)h(x)Ty = 0W1e + y = 0
h(x) w = 0
http://www.princeton.edu/~rvdb
-
Home Page
Title Page
Contents
JJ II
J I
Page 7 of 33
Go Back
Full Screen
Close
Quit
5. Symmetrize Complementarity Condi-tions
Rewrite system:
f (x)h(x)Ty = 0WY e = e
h(x) w = 0
http://www.princeton.edu/~rvdb
-
Home Page
Title Page
Contents
JJ II
J I
Page 8 of 33
Go Back
Full Screen
Close
Quit
6. Apply Newtons Method
Apply Newtons method to compute search directions, x, w,y:
H(x, y) 0 A(x)T0 Y WA(x) I 0
xwy
= f (x) + A(x)TyeWY e
h(x) + w
.Here,
H(x, y) = 2f (x)m
i=1
yi2hi(x)
andA(x) = h(x)
Note: H(x, y) is positive semidefinite if f is convex, each hi isconcave, and each yi 0.
http://www.princeton.edu/~rvdb
-
Home Page
Title Page
Contents
JJ II
J I
Page 9 of 33
Go Back
Full Screen
Close
Quit
7. Reduced KKT System
Use second equation to solve for w. Result is the reducedKKT system:
[H(x, y) AT (x)
A(x) WY 1
] [xy
]=
[f (x) AT (x)yh(x) + Y 1e
]
Iterate:x(k+1) = x(k) + (k)x(k)
w(k+1) = w(k) + (k)w(k)
y(k+1) = y(k) + (k)y(k)
http://www.princeton.edu/~rvdb
-
Home Page
Title Page
Contents
JJ II
J I
Page 10 of 33
Go Back
Full Screen
Close
Quit
8. Convex vs. Nonconvex OptimizationProbs
Nonlinear Programming (NLP)
minimize f (x)subject to hi(x) = 0, i E ,
hi(x) 0, i I.NLP is convex if
his in equality constraints are affine; his in inequality constraints are concave; f is convex;
NLP is smooth if
All are twice continuously differentiable.
http://www.princeton.edu/~rvdb
-
Home Page
Title Page
Contents
JJ II
J I
Page 11 of 33
Go Back
Full Screen
Close
Quit
9. Modifications for Convex Optimiza-tion
For convex nonquadratic optimization, it does not suffice to choosethe steplength simply to maintain positivity of nonnegative vari-ables.
Consider, e.g., minimizing
f (x) = (1 + x2)1/2.
The iterates can be computed explicitly:
x(k+1) = (x(k))3
Converges if and only if |x| 1. Reason: away from 0, function is too linear.
http://www.princeton.edu/~rvdb
-
Home Page
Title Page
Contents
JJ II
J I
Page 12 of 33
Go Back
Full Screen
Close
Quit
10. Step-Length Control
A filter-type method is used to guide the choice of steplength .Define the dual normal matrix:
N(x, y, w) = H(x, y) + AT (x)W1Y A(x).
Theorem Suppose that N(x, y, w) is positive definite.
1. If current solution is primal infeasible, then (x, w) is a de-scent direction for the infeasibility h(x) w.
2. If current solution is primal feasible, then (x, w) is a descentdirection for the barrier function.
Shorten until (x, w) is a descent direction for either the infea-sibility or the barrier function.
http://www.princeton.edu/~rvdb
-
Home Page
Title Page
Contents
JJ II
J I
Page 13 of 33
Go Back
Full Screen
Close
Quit
11. Nonconvex Optimization: DiagonalPerturbation
If H(x, y) is not positive semidefinite then N(x, y, w) might failto be positive definite.
In such a case, we lose the descent properties given in previoustheorem.
To regain those properties, we perturb the Hessian: H(x, y) =H(x, y) + I.
And compute search directions using H instead of H.
Notation: let N denote the dual normal matrix associated with H.
Theorem If N is positive definite, then (x, w, y) is a descentdirection for
1. the primal infeasibility, h(x) w;2. the noncomplementarity, wTy.
http://www.princeton.edu/~rvdb
-
Home Page
Title Page
Contents
JJ II
J I
Page 14 of 33
Go Back
Full Screen
Close
Quit
12. Notes:
Not necessarily a descent direction for dual infeasibility.
A line search is performed to find a value of within a factor of2 of the smallest permissible value.
http://www.princeton.edu/~rvdb
-
Home Page
Title Page
Contents
JJ II
J I
Page 15 of 33
Go Back
Full Screen
Close
Quit
13. Nonconvex Optimization: Jamming
Theorem If the problem is convex and and the current solutionis not optimal and ..., then for any slack variable, say wi, we havewi = 0 implies wi 0.
To paraphrase: for convex problems, as slack variables get smallthey tend to get large again. This is an antijamming theorem.
A recent example of Wachter and Biegler shows that for non-convex problems, jamming really can occur.
Recent modification: if a slack variable gets small and
its component of the step direction contributes to making avery short step,
then increase this slack variable to the average size of thevariables the mainstream slack variables.
This modification corrects all examples of jamming that we knowabout.
http://www.princeton.edu/~rvdb
-
Home Page
Title Page
Contents
JJ II
J I
Page 16 of 33
Go Back
Full Screen
Close
Quit
14. Modifications for General ProblemFormulations
Bounds, ranges, and free variables are all treated implicitly asdescribed in Linear Programming: Foundations and Extensions(LP:F&E).
Net result is following reduced KKT system:[(H(x, y) + D) AT (x)
A(x) E
] [xy
]=
[12
] Here, D and E are positive definite diagonal matrices. Note that D helps reduce frequency of diagonal perturbation. Choice of barrier parameter and initial solution, if none is
provided, is described in the paper.
Stopping rules, matrix reordering heuristics, etc. are as describedin LP:F&E.
http://www.princeton.edu/~rvdb
-
Home Page
Title Page
Contents
JJ II
J I
Page 17 of 33
Go Back
Full Screen
Close
Quit
Examples: Convex Optimization Models
http://www.princeton.edu/~rvdb
-
Home Page
Title Page
Contents
JJ II
J I
Page 18 of 33
Go Back
Full Screen
Close
Quit
15. Minimal Surfaces
Given: a domain D in R2 and an embedding x = (x1, x2, x3) ofits boundary D in R3;
Find: an embedding of the entire domain into R3 that is con-sistent with the boundary embedding and has minimal surfacearea:
minimize
D
xs xt dsdt
subject to x(s, t) fixed for (s, t) Dx1(s, t) fixed for (s, t) Dx2(s, t) fixed for (s, t) D
The specific problems coded below take D to be either a square oran annulus.
Page 1 of 1
6/6/2001http://www.princeton.edu/~rvdb/minsurf/helicoid.gif
Page 1 of 1
6/6/2001http://www.princeton.edu/~rvdb/minsurf/catenoid.gif
Page 1 of 1
6/6/2001http://www.princeton.edu/~rvdb/minsurf/scherk.gif
Page 1 of 1
6/6/2001http://www.princeton.edu/~rvdb/minsurf/twist.gif
http://www.princeton.edu/~rvdbhttp://www.princeton.edu/~rvdb/minsurf/helicoid.wrlhttp://www.princeton.edu/~rvdb/minsurf/catenoid.wrlhttp://www.princeton.edu/~rvdb/minsurf/scherk.wrlhttp://www.princeton.edu/~rvdb/minsurf/twist.wrl
-
Home Page
Title Page
Contents
JJ II
J I
Page 19 of 33
Go Back
Full Screen
Close
Quit
16. Specific Example
Scherk.mod with D discretized into a 6464 grid gives the followingresults:
constraints 0variables 3844time (secs)
loqo 5.1lancelot 4.0snopt *
http://www.princeton.edu/~rvdb
-
Home Page
Title Page
Contents
JJ II
J I
Page 20 of 33
Go Back
Full Screen
Close
Quit
17. Finite Impulse Response (FIR) FilterDesign
Audio is stored digitally in a computer as a stream of shortintegers: uk, k Z.
When the music is played, these integers are used to drive thedisplacement of the speaker from its resting position.
For CD quality sound, 44100 short integers get played per secondper channel.
0 -327681 -327682 -327683 -307534 -288655 -291056 -292017 -26513
8 -236819 -18449
10 -1102511 -691312 -433713 -132914 174315 6223
16 1211117 1731118 2131119 2305520 2351921 2524722 2753523 29471
http://www.princeton.edu/~rvdb
-
Home Page
Title Page
Contents
JJ II
J I
Page 21 of 33
Go Back
Full Screen
Close
Quit
18. FIR Filter DesignContinued
A finite impulse response (FIR) filter takes as input a digitalsignal and convolves this signal with a finite set of fixed numbershn, . . . , hn to produce a filtered output signal:
yk =
ni=n
hiuki.
Sparing the details, the output power at frequency is given by
|H()|
where
H() =
nk=n
hke2ik,
Similarly, the mean squared deviation from a flat frequency re-sponse over a frequency range, say L [0, 1], is given by
1
|L|
L|H() 1|2 d
http://www.princeton.edu/~rvdb
-
Home Page
Title Page
Contents
JJ II
J I
Page 22 of 33
Go Back
Full Screen
Close
Quit
19. Filter Design: Woofer, Midrange,Tweeter
minimize
subject to
10
(Hw() + Hm() + Ht() 1
)2d
(1
|W |
W
H2w()d
)1/2 W = [.2, .8]
(1
|M |
M
H2m()d
)1/2 M = [.4, .6] [.9, .1]
(1
|T |
T
H2t ()d
)1/2 T = [.7, .3]
where
Hi() = hi(0) + 2n1k=1
hi(k) cos(2k), i = W, M, T
hi(k) = filter coefficients, i.e., decision variables
http://www.princeton.edu/~rvdb
-
Home Page
Title Page
Contents
JJ II
J I
Page 23 of 33
Go Back
Full Screen
Close
Quit
20. Specific Example
filter length: n = 14
integral discretization: N = 1000
constraints 4variables 43time (secs)
loqo 79minos 164lancelot 3401snopt 35
Ref: J.O. Coleman, U.S. Naval Research Laboratory,
CISS98 paper available: engr.umbc.edu/jeffc/pubs/abstracts/ciss98.htmlClick here for demo
http://www.princeton.edu/~rvdbhttp://engr.umbc.edu/~jeffc/pubs/abstracts/ciss98.htmlhttp://www.princeton.edu/~rvdb/ampl/nlmodels/jeffc/index.html
-
Home Page
Title Page
Contents
JJ II
J I
Page 24 of 33
Go Back
Full Screen
Close
Quit
Examples: Nonconvex OptimizationModels
http://www.princeton.edu/~rvdb
-
Home Page
Title Page
Contents
JJ II
J I
Page 25 of 33
Go Back
Full Screen
Close
Quit
21. Celestial MechanicsPeriodic Orbits
Find periodic orbits for the planar gravitational n-body problem. Minimize action: 2
0(K(t) P (t))dt,
where K(t) is kinetic energy,
K(t) =1
2
i
(x2i (t) + y
2i (t)
),
and P (t) is potential energy,
P (t) = i
-
Home Page
Title Page
Contents
JJ II
J I
Page 26 of 33
Go Back
Full Screen
Close
Quit
22. Specific Example
Orbits.mod with n = 3 and (0, 2) discretized into a 160 piecesgives the following results:
constraints 0variables 960time (secs)
loqo 1.1lancelot 8.7snopt 287 (no change for last 80% of iterations)
-1.5
-1
-0.5
0
0.5
1
1.5
-1.5 -1 -0.5 0 0.5 1 1.5
"after.out"
http://www.princeton.edu/~rvdbhttp://www.princeton.edu/~rvdb/JAVA/astro/galaxy/Galaxy.html
-
Home Page
Title Page
Contents
JJ II
J I
Page 27 of 33
Go Back
Full Screen
Close
Quit
23. Putting on an Uneven Green
Given:
z(x, y) elevation of the green. Starting position of the ball (x0, y0). Position of hole (xf , yf). Coefficient of friction .
Find: initial velocity vector so that ball will roll to the hole and arrivewith minimal speed.Variables:
u(t) = (x(t), y(t), z(t))position as a function of time t. v(t) = (vx(t), vy(t), vz(t))velocity. a(t) = (ax(t), ay(t), az(t))acceleration. Ttime at which ball arrives at hole.
http://www.princeton.edu/~rvdb
-
Home Page
Title Page
Contents
JJ II
J I
Page 28 of 33
Go Back
Full Screen
Close
Quit
24. PuttingTwo Approaches
Problem can be formulated with two decision variables:
vx(0) and vy(0)
and two constraints:
x(T ) = xf and y(T ) = yf .
In this case, x(T ), y(T ), and the objective function are compli-cated functions of the two variables that can only be computedby integrating the appropriate differential equation.
A discretization of the complete trajectory (including position,velocity, and acceleration) can be taken as variables and thephysical laws encoded in the differential equation can be writtenas constraints.
To implement the first approach, one would need an ode integratorthat provides, in addition to the quantities being sought, first andpossibly second derivatives of those quantities with respect to thedecision variables.The modern trend is to follow the second approach.
http://www.princeton.edu/~rvdb
-
Home Page
Title Page
Contents
JJ II
J I
Page 29 of 33
Go Back
Full Screen
Close
Quit
25. PuttingContinued
Objective:minimize vx(T )
2 + vy(T )2.
Constraints:
v = u
a = v
ma = N + F mgezu(0) = u0 u(T ) = uf ,
where
m is the mass of the golf ball. g is the acceleration due to gravity. ez is a unit vector in the positive z direction.
and ...
http://www.princeton.edu/~rvdb
-
Home Page
Title Page
Contents
JJ II
J I
Page 30 of 33
Go Back
Full Screen
Close
Quit
26. PuttingContinued
N = (Nx, Ny, Nz) is the normal force:
Nz = mg ax(t)zx ay(t)
zy
+ az(t)
(zx
)2 + (zy
)2 + 1
Nx = z
xNz
Ny = z
yNz.
F is the force due to friction:
F = N vv
.
http://www.princeton.edu/~rvdb
-
Home Page
Title Page
Contents
JJ II
J I
Page 31 of 33
Go Back
Full Screen
Close
Quit
27. PuttingSpecific Example
Discretize continuous time into n = 200 discrete time points. Use finite differences to approximate the derivatives.
constraints 597variables 399time (secs)
loqo 14.1lancelot > 600.0snopt 4.1
http://www.princeton.edu/~rvdbhttp://www.princeton.edu/~rvdb/putt_atan.wrl
-
Home Page
Title Page
Contents
JJ II
J I
Page 32 of 33
Go Back
Full Screen
Close
Quit
28. Goddard Rocket Problem
Objective:maximize h(T );
Constraints:
v = h
a = v
= cmma = ( v2eh/h0) gm
0 maxm mmin
h(0) = 0 v(0) = 0 m(0) = 3
where
= Thrust , m = mass max, g, , c, and h0 are given constants h, v, a, Th, and m are functions of time 0 t T .
http://www.princeton.edu/~rvdb
-
Home Page
Title Page
Contents
JJ II
J I
Page 33 of 33
Go Back
Full Screen
Close
Quit
29. Goddard Rocket ProblemSolution
constraints 399variables 599time (secs)
loqo 5.2lancelot (IL)snopt (IL)
http://www.princeton.edu/~rvdbhttp://www.princeton.edu/~rvdb/goddard.html
orange Outline orange Introduce Slack Variables orange Associated Log-Barrier Problem orange First-Order Optimality Conditions orange Symmetrize Complementarity Conditions orange Apply Newton's Method orange Reduced KKT System orange Convex vs. Nonconvex Optimization Probs orange Modifications for Convex Optimization orange Step-Length Control orange Nonconvex Optimization: Diagonal Perturbation orange Notes: orange Nonconvex Optimization: Jamming orange Modifications for General Problem Formulations orange Minimal Surfaces orange Specific Example orange Finite Impulse Response (FIR) Filter Design orange FIR Filter Design---Continued orange Filter Design: Woofer, Midrange, Tweeter orange Specific Example orange Celestial Mechanics---Periodic Orbits orange Specific Example orange Putting on an Uneven Green orange Putting---Two Approaches orange Putting---Continued orange Putting---Continued orange Putting---Specific Example orange Goddard Rocket Problem orange Goddard Rocket Problem---Solution