click to edit master title style tier i: mathematical methods of optimization section 3: nonlinear...
TRANSCRIPT
![Page 1: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/1.jpg)
Process Optimization
![Page 2: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/2.jpg)
Click to edit Master title style
Tier I: Mathematical Methods of Optimization
Section 3:
Nonlinear Programming
![Page 3: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/3.jpg)
Click to edit Master title style
Introduction to Nonlinear Programming
• We already talked about a basic aspect of nonlinear programming (NLP) in the Introduction Chapter when we considered unconstrained optimization.
![Page 4: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/4.jpg)
Click to edit Master title style
Introduction to Nonlinear Programming
• We optimized one-variable nonlinear functions using the 1st and 2nd derivatives.
• We will use the same concept here extended to functions with more than one variable.
![Page 5: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/5.jpg)
Click to edit Master title style
Multivariable Unconstrained Optimization
• For functions with one variable, we use the 1st and 2nd derivatives.
• For functions with multiple variables, we use identical information that is the gradient and the Hessian.
• The gradient is the first derivative with respect to all variables whereas the Hessian is the equivalent of the second derivative
![Page 6: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/6.jpg)
Click to edit Master title style
The Gradient
• Review of the gradient ():
For a function “f ”, of variables x1, x2, …, xn:
nx
f
x
f
x
ff
21
Example: 231
321 )(3)(215 xxxxf
312
22
3 6)(6)(315 xxxxf
![Page 7: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/7.jpg)
Click to edit Master title style
The Hessian
• The Hessian (2) of f(x1, x2, …, xn) is:
2
2
2
2
1
2
2
2
22
2
12
21
2
21
2
21
2
2
nnn
n
n
x
f
xx
f
xx
f
xx
f
x
f
xx
fxx
f
xx
f
x
f
f
![Page 8: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/8.jpg)
Click to edit Master title style
Hessian Example
• Example (from previously):2
313
21 )(3)(215 xxxxf
13
2
32
606
0120
600
xx
x
x
f
312
22
3 6)(6)(315 xxxxf
![Page 9: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/9.jpg)
Click to edit Master title style
Unconstrained Optimization
The optimization procedure for multivariable functions is:
1. Solve for the gradient of the function equal to zero to obtain candidate points.
2. Obtain the Hessian of the function and evaluate it at each of the candidate points
• If the result is “positive definite” (defined later) then the point is a local minimum.
• If the result is “negative definite” (defined later) then the point is a local maximum.
![Page 10: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/10.jpg)
Click to edit Master title style
Positive/Negative Definite
• A matrix is “positive definite” if all of the eigenvalues of the matrix are positive (> 0)
• A matrix is “negative definite” if all of the eigenvalues of the matrix are negative (< 0)
![Page 11: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/11.jpg)
Click to edit Master title style
Positive/Negative Semi-definite
• A matrix is “positive semi-definite” if all of the eigenvalues are non-negative (≥ 0)
• A matrix is “negative semi-definite” if all of the eigenvalues are non-positive (≤ 0)
![Page 12: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/12.jpg)
Click to edit Master title style
Example Matrix
Given the matrix A:
211
175
542
A
The eigenvalues of A are:22 702.31 702.23
This matrix is negative definite
![Page 13: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/13.jpg)
Click to edit Master title style
Unconstrained NLP Example
Consider the problem:
Minimize f(x1, x2, x3) = (x1)2 + x1(1 – x2) + (x2)2 – x2x3 + (x3)2 + x3
First, we find the gradient with respect to xi:
1 2
1 2 3
2 3
2 1
2
2 1
x x
f x x x
x x
![Page 14: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/14.jpg)
Click to edit Master title style
Unconstrained NLP Example
Next, we set the gradient equal to zero:
0f
0
0
0
12
2
12
32
321
21
xx
xx
xx
So, we have a system of 3 equations and 3 unknowns. When we solve, we get:
1
1
1
3
2
1
x
x
x
x
![Page 15: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/15.jpg)
Click to edit Master title style
Unconstrained NLP Example
So we have only one candidate point to check.
Find the Hessian:
210
121
0122 f
![Page 16: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/16.jpg)
Click to edit Master title style
Unconstrained NLP Example
The eigenvalues of this matrix are:
414.31 586.02 23
All of the eigenvalues are > 0, so the Hessian is positive definite.
So, the point is a minimum
1
1
1
3
2
1
x
x
x
x
![Page 17: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/17.jpg)
Click to edit Master title style
Unconstrained NLP Example
Unlike in Linear Programming, unless we know the shape of the function being minimized or can determine whether it is convex, we cannot tell whether this point is the global minimum or if there are function values smaller than it.
![Page 18: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/18.jpg)
Click to edit Master title style
Method of Solution
• In the previous example, when we set the gradient equal to zero, we had a system of 3 linear equations & 3 unknowns.
• For other problems, these equations could be nonlinear.
• Thus, the problem can become trying to solve a system of nonlinear equations, which can be very difficult.
![Page 19: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/19.jpg)
Click to edit Master title style
Method of Solution
• To avoid this difficulty, NLP problems are usually solved numerically.
• We will now look at examples of numerical methods used to find the optimum point for single-variable NLP problems. These and other methods may be found in any numerical methods reference.
![Page 20: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/20.jpg)
Click to edit Master title style
Newton’s Method
When solving the equation f (x) = 0 to find a minimum or maximum, one can use the iteration step:
)(
)(''
'1
k
kkk
xf
xfxx
where k is the current iteration.
Iteration is continued until |xk+1 – xk| < where is some specified tolerance.
![Page 21: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/21.jpg)
Click to edit Master title style
Newton’s Method Diagram
Newton’s Method approximates f (x) as a straight line at xk and obtains a new point (xk+1), which is used to approximate the function at the next iteration. This is carried on until the new point is sufficiently close to x*.
xx* xk+1 xk
Tangent of f (x) at xk
f (x)
![Page 22: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/22.jpg)
Click to edit Master title style
Newton’s Method Comments
• One must ensure that f (xk+1) < f (xk) for finding a minimum and f (xk+1) > f (xk) for finding a maximum.
• Disadvantages:– Both the first and second derivatives must be
calculated– The initial guess is very important – if it is not
close enough to the solution, the method may not converge
![Page 23: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/23.jpg)
Click to edit Master title style
Regula-Falsi Method
This method requires two points, xa & xb that bracket the solution to the equation f (x) = 0.
)()(
)()(''
'
ab
abbbc
xfxf
xxxfxx
where xc will be between xa & xb. The next interval will be xc and either xa or xb, whichever has the sign opposite of xc.
![Page 24: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/24.jpg)
Click to edit Master title style
Regula-Falsi Diagram
The Regula-Falsi method approximates the function f (x) as a straight line and interpolates to find the root.
xxa
xb
xc
x*
f (x)
![Page 25: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/25.jpg)
Click to edit Master title style
Regula-Falsi Comments
• This method requires initial knowledge of two points bounding the solution
• However, it does not require the calculation of the second derivative
• The Regula-Falsi Method requires slightly more iterations to converge than the Newton’s Method
![Page 26: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/26.jpg)
Click to edit Master title style
Multivariable Optimization
• Now we will consider unconstrained multivariable optimization
• Nearly all multivariable optimization methods do the following:
1. Choose a search direction dk
2. Minimize along that direction to find a new point:
where k is the current iteration number and k is a positive scalar called the step size.
kkkk dxx 1
![Page 27: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/27.jpg)
Click to edit Master title style
The Step Size
• The step size, k, is calculated in the following way:
• We want to minimize the function f(xk+1) = f(xk +kdk) where the only variable is k
because xk & dk are known.
• We set and solve for k
using a single-variable solution method
such as the ones shown previously.
0
d
d
k
kkkf
dx
![Page 28: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/28.jpg)
Click to edit Master title style
Steepest Descent Method
• This method is very simple – it uses the gradient (for maximization) or the negative gradient (for minimization) as the search direction:
)( kk f xd
for
min
max
So, )(1 kkkk f xxx
![Page 29: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/29.jpg)
Click to edit Master title style
Steepest Descent Method
• Because the gradient is the rate of change of the function at that point, using the gradient (or negative gradient) as the search direction helps reduce the number of iterations needed
x1
x2 f(x) = 5
f(x) = 20
f(x) = 25
xk
f(xk)
-f(xk)
![Page 30: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/30.jpg)
Click to edit Master title style
Steepest Descent Method Steps
So the steps of the Steepest Descent Method are:
1. Choose an initial point x0
2. Calculate the gradient f(xk) where k is the iteration number
3. Calculate the search vector:
4. Calculate the next x:Use a single-variable optimization method to determine k.
)( kk f xd kkkk dxx 1
![Page 31: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/31.jpg)
Click to edit Master title style
Steepest Descent Method Steps
5. To determine convergence, either use some given tolerance 1 and evaluate:
for convergence
Or, use another tolerance 2 and evaluate:
for convergence
11 )()( kk ff xx
2)( kf x
![Page 32: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/32.jpg)
Click to edit Master title style
Convergence
• These two criteria can be used for any of the multivariable optimization methods discussed here
Recall: The norm of a vector, ||x|| is given by:22
22
1 )()()( nT xxx xxx
![Page 33: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/33.jpg)
Click to edit Master title style
Steepest Descent Example
Let’s solve the earlier problem with the Steepest Descent Method:
Minimize f(x1, x2, x3) = (x1)2 + x1(1 – x2) + (x2)2 – x2x3 + (x3)2 + x3
Let’s pick
0
0
00x
![Page 34: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/34.jpg)
Click to edit Master title style
Steepest Descent Example
10000001)0(2)( 00 xd f
122)1(2)( 3232121 xxxxxxxf x
101101
101000 01 x
Now, we need to determine 0
![Page 35: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/35.jpg)
Click to edit Master title style
Steepest Descent Example
2)(4)( 0
0
1
d
df x
)()(00)1)(()()( 0200201 xf
)(2)(2 020
Now, set equal to zero and solve:
21
420 2)(4 0
![Page 36: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/36.jpg)
Click to edit Master title style
Steepest Descent Example
So,
101000 01 x
2
10
2
1000
2
10
2
11x
![Page 37: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/37.jpg)
Click to edit Master title style
Steepest Descent Example
110
2
10
2
1011)( 11 xd f
0101 d
Take the negative gradient to find the next search direction:
![Page 38: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/38.jpg)
Click to edit Master title style
Steepest Descent Example
0102
10
2
1 12
x
2
1
2
1 1
Update the iteration formula:
![Page 39: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/39.jpg)
Click to edit Master title style
Steepest Descent Example
1)(2)( 1
1
1
d
df x
2
1
4
1
2
1)()(1
2
1
4
1)( 12112
xf
21)( 121
Insert into the original function & take the derivative so that we can find 1:
![Page 40: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/40.jpg)
Click to edit Master title style
Steepest Descent Example
Now we can set the derivative equal to zero and solve for 1:
211 1)(2 1
![Page 41: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/41.jpg)
Click to edit Master title style
Steepest Descent Example
Now, calculate x2:
0102
10
2
1 12
x
0
2
10
2
10
2
1
2
1
2
1
2
12x
![Page 42: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/42.jpg)
Click to edit Master title style
Steepest Descent Example
So,
11
2
1
2
11
2
1
2
111)( 22 xd f
2
10
2
12d
2
10
2
1
2
1
2
1
2
1 23 x
)1(
2
1
2
1)1(
2
1 22
![Page 43: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/43.jpg)
Click to edit Master title style
Steepest Descent Example
2
3)1(
)( 22
3
d
df x
4
1)1(
2
3)1(
2
1)( 2223 xf
Set the derivative equal to zero and solve:
212
2
3)1( 2
Find 2:
![Page 44: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/44.jpg)
Click to edit Master title style
Steepest Descent Example
Calculate x3:
2
10
2
1
2
1
2
1
2
1 23 x
4
10
4
1
2
1
2
1
2
1
4
3
2
1
4
33x
![Page 45: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/45.jpg)
Click to edit Master title style
Steepest Descent Example
0
2
100
2
10)( 33 xd f
0
2
10
4
3
2
1
4
3 34 x
4
3)1(
2
1
4
3 3
Find the next search direction:
![Page 46: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/46.jpg)
Click to edit Master title style
Steepest Descent Example
08
9)1(
2
1)( 33
4
d
df x
2
3)(
2
3)1(
4
1)( 3234 xfFind 3:
4
53
![Page 47: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/47.jpg)
Click to edit Master title style
Steepest Descent Example
So, x4 becomes:
0
8
50
4
3
2
1
4
34x
4
3
8
9
4
34x
![Page 48: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/48.jpg)
Click to edit Master title style
Steepest Descent Example
8
5
4
3
8
5
8
5
4
3
8
5)( 44 xd f
8
5
4
3
8
5
4
3
8
9
4
3 45 x
)
2
53(
4
1)
2
3(
4
3)
2
53(
4
1 444
The next search direction:
![Page 49: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/49.jpg)
Click to edit Master title style
Steepest Descent Example
032
43
16
73)( 44
5
d
df x
64
51
32
43)(
32
73)( 4245 xf
146434
Find 4:
![Page 50: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/50.jpg)
Click to edit Master title style
Steepest Descent Example
Update x5:
8
5
4
3
8
5
146
43
4
3
8
9
4
35x
1168
1091
73
66
1168
10915x
![Page 51: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/51.jpg)
Click to edit Master title style
Steepest Descent Example
Let’s check to see if the convergence criteria is satisfied
Evaluate ||f(x5)||:
584
21
584
35
584
21)( 5xf
0786.058421
58435
58421)(
2225 xf
![Page 52: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/52.jpg)
Click to edit Master title style
Steepest Descent Example
So, ||f(x5)|| = 0.0786, which is very small and we can take it to be close enough to zero for our example
Notice that the answer of
1168
1091
73
66
1168
1091x
is very close to the value of that we obtained analytically
111* x
![Page 53: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/53.jpg)
Click to edit Master title style
Quadratic Functions
• Quadratic functions are important for the next method we will look at
• A quadratic function can be written in the form: xTQx where x is the vector of variables and Q is a matrix of coefficients
Example: )2(x x2x )(x2
210
121
0122
2212
1
3
2
1
321
x
x
x
xxx
)2(x x2x– 2332
![Page 54: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/54.jpg)
Click to edit Master title style
Conjugate Gradient Method
• The Conjugate Gradient Method has the property that if f(x) is quadratic, it will take exactly n iterations to converge, where n is the number of variables in the x vector
• Although it works especially well with quadratic functions, this method will work with non-quadratic functions also
![Page 55: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/55.jpg)
Click to edit Master title style
Conjugate Gradient Steps
1. Choose a starting point x0 and calculate f(x0). Let d0 = -f(x0)
2. Calculate x1 using:Find 0 by performing a single-variable optimization on f(x0 +0d0) using the methods discussed earlier. (See illustration after algorithm explanation)
0001 dxx
![Page 56: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/56.jpg)
Click to edit Master title style
Conjugate Gradient Steps
3. Calculate f(x1) and f(x1). The new search direction is calculated using the equation:
)()(
)()()(
00
11011
xx
xxdxd
ff
fff
T
T
This can be generalized for the kth iteration:
)()(
)()()(
1111
kkT
kkTkkk
ff
fff
xx
xxdxd
![Page 57: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/57.jpg)
Click to edit Master title style
Conjugate Gradient Steps
4. Use either of the two methods discussed earlier to determine tolerance:
11 )()( kk ff xx
2)( kf x
Or,
![Page 58: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/58.jpg)
Click to edit Master title style
Number of Iterations
• For quadratic functions, this method will converge in n iterations (k = n)
• For non-quadratic functions, after n iterations, the algorithm cycles again with dn+1 becoming d0.
![Page 59: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/59.jpg)
Click to edit Master title style
Step Size for Quadratic Functions
• When optimizing the step size, we can approximate the function to be optimized in the following manner:
)()()()()()( 2
2
1 kkTkkkTkkk ffff dxddxxdx
• For a quadratic function, this is not an approximation – it is exact
![Page 60: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/60.jpg)
Click to edit Master title style
Step Size for Quadratic Functions
We take the derivative of that function with respect to and set it equal to zero:
0)()()()( 2
kkTkkkTkk
ffd
dfdxddx
dx
The solution to this equation is:
kkTk
kkToptk
f
f
dxd
dx
)()(
)(2
,
![Page 61: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/61.jpg)
Click to edit Master title style
Step Size for Quadratic Functions
• So, for the problem of optimizing a quadratic function,
kkTk
kkToptk
f
f
dxd
dx
)()(
)(2
,
is the optimum step size.
• For a non-quadratic function, this is an approximation of the optimum step size.
![Page 62: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/62.jpg)
Click to edit Master title style
Multivariable Newton’s Method
We can approximate the gradient of f at a point x0 by:
)()()()( 0020 xxxxx fff
We can set the right-hand side equal to zero and rearrange to give:
)()( 01020 xxxx ff
![Page 63: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/63.jpg)
Click to edit Master title style
Multivariable Newton’s Method
We can generalize this equation to give an iterative expression for the Newton’s Method:
)()(121 kkkk ff xxxx
where k is the iteration number
![Page 64: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/64.jpg)
Click to edit Master title style
Newton’s Method Steps
1. Choose a starting point, x0
2. Calculate f(xk) and 2f(xk)
3. Calculate the next x using the equation
4. Use either of the convergence criteria discussed earlier to determine convergence. If it hasn’t converged, return to step 2.
)()(121 kkkk ff xxxx
![Page 65: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/65.jpg)
Click to edit Master title style
Comments on Newton’s Method
• We can see that unlike the previous two methods, Newton’s Method uses both the gradient and the Hessian
• This usually reduces the number of iterations needed, but increases the computation needed for each iteration
• So, for very complex functions, a simpler method is usually faster
![Page 66: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/66.jpg)
Click to edit Master title style
Newton’s Method Example
For an example, we will use the same problem as before:
Minimize f(x1, x2, x3) = (x1)2 + x1(1 – x2) + (x2)2 – x2x3 + (x3)2 + x3
12212)( 3232121 xxxxxxxf x
![Page 67: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/67.jpg)
Click to edit Master title style
Newton’s Method Example
The Hessian is:
210
121
012
)(2 xf
And we will need the inverse of the Hessian:
43
21
41
2112
14
12
14
3
210
121
012
)(
1
12 xf
![Page 68: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/68.jpg)
Click to edit Master title style
Newton’s Method Example
So, pick
0
0
00x
Calculate the gradient for the 1st iteration:
100000100)( 0 xf
101)( 0 xf
![Page 69: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/69.jpg)
Click to edit Master title style
Newton’s Method ExampleSo, the new x is:
11 0 2 0 0( ) ( )
3 1 14 2 40 1
1 1 0 1 02 20 131 1
4 2 4
f f
x x x x
1
1
11x
![Page 70: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/70.jpg)
Click to edit Master title style
Newton’s Method Example
Now calculate the new gradient:
000121121112)( 1 xf
Since the gradient is zero, the method has converged
![Page 71: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/71.jpg)
Click to edit Master title style
Comments on Example
• Because it uses the 2nd derivative, Newton’s Method models quadratic functions exactly and can find the optimum point in one iteration.
• If the function had been a higher order, the Hessian would not have been constant and it would have been much more work to calculate the Hessian and take the inverse for each iteration.
![Page 72: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/72.jpg)
Click to edit Master title style
Constrained Nonlinear Optimization
• Previously in this chapter, we solved NLP problems that only had objective functions, with no constraints.
• Now we will look at methods on how to solve problems that include constraints.
![Page 73: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/73.jpg)
Click to edit Master title style
NLP with Equality Constraints
• First, we will look at problems that only contain equality constraints:
Minimize f(x) x = [x1 x2 … xn]
Subject to: hi(x) = bi i = 1, 2, …, m
![Page 74: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/74.jpg)
Click to edit Master title style
Illustration
Consider the problem:
Minimize x1 + x2
Subject to: (x1)2 + (x2)2 – 1 = 0
The feasible region is a circle with a radius of one. The possible objective function curves are lines with a slope of -1. The minimum will be the point where the lowest line still touches the circle.
![Page 75: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/75.jpg)
Click to edit Master title style
Graph of Illustration
f(x) = 1
f(x) = 0
f(x) = -1.414
Feasible region
707.0
707.0*x
)(xf
The gradient of f points in the direction of
increasing f
![Page 76: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/76.jpg)
Click to edit Master title style
More on the Graph
• Since the objective function lines are straight parallel lines, the gradient of f is a straight line pointing toward the direction of increasing f, which is to the upper right
• The gradient of h will be pointing out from the circle and so its direction will depend on the point at which the gradient is evaluated.
![Page 77: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/77.jpg)
Click to edit Master title style
Further Details
f(x) = 1
f(x) = 0
f(x) = -1.414
Feasible region
707.0
707.0*x
)( *xh
)( *xf
0
11x
)( 1xf
Tangent Plane
)( 1xhx1
x2
![Page 78: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/78.jpg)
Click to edit Master title style
Conclusions
• At the optimum point, f(x) is perpendicular to h(x)
• As we can see at point x1, f(x) is not perpendicular to h(x) and we can move (down) to improve the objective function
• We can say that at a max or min, f(x) must be perpendicular to h(x) – Otherwise, we could improve the objective
function by changing position
![Page 79: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/79.jpg)
Click to edit Master title style
First Order Necessary Conditions
So, in order for a point to be a minimum (or maximum), it must satisfy the following equation:
This equation means that f(x*) and h(x*) must be in exactly opposite directions at a minimum or maximum point
0)()( *** xx hf
![Page 80: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/80.jpg)
Click to edit Master title style
The Lagrangian Function
To help in using this fact, we introduce the Lagrangian Function, L(x,):
)()(),( xxx hfL
Review: The notation x f(x,y) means the gradient of f with respect to x.
So, )()(),( xxx xxx hfL
![Page 81: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/81.jpg)
Click to edit Master title style
First Order Necessary Conditions
• So, using the new notation to express the First Order Necessary Conditions (FONC), if x* is a minimum (or maximum) then
0),( ** ,
xx xL
and to ensure feasibility.0)( * xh
![Page 82: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/82.jpg)
Click to edit Master title style
First Order Necessary Conditions
• Another way to think about it is that the one Lagrangian function includes all information about our problem
• So, we can treat the Lagrangian as an unconstrained optimization problem with variables x1, x2, …, xn and 1, 2, …, m.
We can solve it by solving the equations
0x
L
& 0λ
L
![Page 83: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/83.jpg)
Click to edit Master title style
Using the FONC
Using the FONC for the previous example,
)()(),( xxx hfL 12
22
121 xxxx
0
0),(
2
1
xL
xL
Lx x
And the first FONC equation is:
![Page 84: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/84.jpg)
Click to edit Master title style
FONC Example
This becomes:
&
021 11
xx
L
021 22
xx
L
The feasibility equation is:
or,
0122
21 xx
0122
21
xxL
![Page 85: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/85.jpg)
Click to edit Master title style
FONC Example
So, we have three equations and three unknowns.
When they are solved simultaneously, we obtain
707.021 xx & 707.0
We can see from the graph that positive x1 & x2 corresponds to a maximum while negative x1 & x2 corresponds to the minimum.
![Page 86: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/86.jpg)
Click to edit Master title style
FONC Observations
• If you go back to the LP Chapter and look at the mathematical definition of the KKT conditions, you may notice that they look just like our FONC that we just used
• This is because it is the same concept
• We simply used a slightly different derivation this time but obtained the same result
![Page 87: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/87.jpg)
Click to edit Master title style
Limitations of FONC
• The FONC do not guarantee that the solution(s) will be minimums/maximums.
• As in the case of unconstrained optimization, they only provide us with candidate points that need to be verified by the second order conditions.
• Only if the problem is convex do the FONC guarantee the solutions will be extreme points.
![Page 88: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/88.jpg)
Click to edit Master title style
Second Order Necessary Conditions (SONC)
For where
and for y where
)()(),( 222 xxx hfLx ),(2 xLx
0y
x
xyxJ
x
)(
1
*
*
)(m
hh
h
If x* is a local minimum, then 0),( *2 yxy LxT
![Page 89: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/89.jpg)
Click to edit Master title style
Second Order Sufficient Conditions (SOSC)
• y can be thought of as being a tangent plane as in the graphical example shown previously– Jh is just the gradients of each h(x) equation
and we saw in the example that the tangent plane must be perpendicular to h(x) and that is why
0),( *2 yxy LxT
0yJ )(xh
![Page 90: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/90.jpg)
Click to edit Master title style
The y Vector
x1
x3
x2
Tangent Plane(all possible y vectors))(xh
0)( xh
*x
The tangent plane is the location of all y vectors and intersects with x*
It must be orthogonal (perpendicular) to h(x)
![Page 91: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/91.jpg)
Click to edit Master title style
Maximization Problems
• The previous definitions of the SONC & SOSC were for minimization problems
• For maximization problems, the sense of the inequality sign will be reversed
For maximization problems:
SONC:
SOSC:
0),( *2 yxy LxT
0),( *2 yxy LxT
![Page 92: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/92.jpg)
Click to edit Master title style
Necessary & Sufficient
• The necessary conditions are required for a point to be an extremum but even if they are satisfied, they do not guarantee that the point is an extremum.
• If the sufficient conditions are true, then the point is guaranteed to be an extremum. But if they are not satisfied, this does not mean that the point is not an extremum.
![Page 93: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/93.jpg)
Click to edit Master title style
Procedure
1. Solve the FONC to obtain candidate points.
2. Test the candidate points with the SONC– Eliminate any points that do not satisfy the
SONC
3. Test the remaining points with the SOSC– The points that satisfy them are min/max’s – For the points that do not satisfy, we cannot
say whether they are extreme points or not
![Page 94: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/94.jpg)
Click to edit Master title style
Problems with Inequality Constraints
We will consider problems such as:
Minimize f(x)
Subject to: hi(x) = 0 i = 1, …, m
& gj(x) ≤ 0 j = 1, …, p
An inequality constraint, gj(x) ≤ 0 is called “active” at x* if gj(x*) = 0.Let the set I(x*) contain all the indices of the active constraints at x*:
0)( * xjg for all j in set I(x*)
![Page 95: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/95.jpg)
Click to edit Master title style
Lagrangian for Equality & Inequality Constraint
ProblemsThe Lagrangian is written:
We use ’s for the equalities & ’s for the inequalities.
p
jjji
m
ii ghfL
11
)()()(),,( xxxμλx
![Page 96: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/96.jpg)
Click to edit Master title style
FONC for Equality & Inequality Constraints
For the general Lagrangian, the FONC become
p
jjji
m
ii ghfL
1
*
1
* *)(*)(*)(*)*,*,( 0xxxμλx
and the complementary slackness condition:
,0)( ** xjj g,0* j pj ,1
![Page 97: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/97.jpg)
Click to edit Master title style
SONC for Equality & Inequality Constraints
The SONC (for a minimization problem) are:
where as before.
0yxy x ),,( ***2 LT
0yxJ )( *
This time, J(x*) is the matrix of the gradients of all the equality constraints and only the inequality constraints that are active at x*.
![Page 98: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/98.jpg)
Click to edit Master title style
SOSC for Equality & Inequality Constraints
• The SOSC for a minimization problem with equality & inequality constraints are:
0yxy ),,( ***2 LxT
![Page 99: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/99.jpg)
Click to edit Master title style
Generalized Lagrangian Example
• Solve the problem:
Minimize f(x) = (x1 – 1)2 + (x2)2
Subject to: h(x) = (x1)2 + (x2)2 + x1 + x2 = 0
g(x) = x1 – (x2)2 ≤ 0
The Lagrangian for this problem is: 21
221
22
21 2)1(),,( xxxxxxL x
221 xx
![Page 100: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/100.jpg)
Click to edit Master title style
Generalized Lagrangian Example
• The first order necessary conditions:
0212 111
xxx
L
0222 2222
xxxx
L
0212
22
1
xxxxL
0221 xx
![Page 101: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/101.jpg)
Click to edit Master title style
Generalized Lagrangian Example
Solving the 4 FONC equations, we get 2 solutions:
4534.0
2056.0)1(x 9537.045.0 &
and
0
0)2(x 20 &2)
1)
![Page 102: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/102.jpg)
Click to edit Master title style
Generalized Lagrangian Example
Now try the SONC at the 1st solution:
Both h(x) & g(x) are active at this point (they both equal zero). So, the Jacobian is the gradient of both functions evaluated at x(1):
9068.01
0932.0411.1
21
1212
)1(2
211
x
xJx
xx
![Page 103: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/103.jpg)
Click to edit Master title style
Generalized Lagrangian Example
The only solution to the equation:
0yxJ )( )1(
is:
00
00y
And the Hessian of the Lagrangian is:
993.00
09.2
2220
022
)1(
2
x
x
L
![Page 104: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/104.jpg)
Click to edit Master title style
Generalized Lagrangian Example
So, the SONC equation is:
0
00
00
00
00
993.00
09.2
00
00
This inequality is true, so the SONC is satisfied for x(1) and it is still a candidate point.
![Page 105: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/105.jpg)
Click to edit Master title style
Generalized Lagrangian Example
The SOSC equation is:
And we just calculated the left-hand side of the equation to be the zero matrix. So, in our case for x2:
0yxy x ),,( ***2 LT
0yyxx
00
00)1(
2LT
So, the SOSC are not satisfied.
![Page 106: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/106.jpg)
Click to edit Master title style
Generalized Lagrangian Example
For the second solution:
Again, both h(x) & g(x) are active at this point. So, the Jacobian is:
01
11
21
1212
)2(2
21)2(
x
xJx
xx
![Page 107: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/107.jpg)
Click to edit Master title style
Generalized Lagrangian Example
The only solution to the equation:
0yxJ )( )2(
is:
00
00y
And the Hessian of the Lagrangian is:
20
02
2220
022
)2(
2
x
x
L
![Page 108: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/108.jpg)
Click to edit Master title style
Generalized Lagrangian Example
So, the SONC equation is:
0
00
00
00
00
20
02
00
00
This inequality is true, so the SONC is satisfied for x(2) and it is still a candidate point
![Page 109: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/109.jpg)
Click to edit Master title style
Generalized Lagrangian Example
The SOSC equation is:
And we just calculated the left-hand side of the equation to be the zero matrix. So, in our case for x2:
0yxy x ),,( ***2 LT
0yyxx
00
00)2(
2LT
So, the SOSC are not satisfied.
![Page 110: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/110.jpg)
Click to edit Master title style
Example Conclusions
• So, we can say that both x(1) & x(2) may be local minimums, but we cannot be sure because the SOSC are not satisfied for either point.
![Page 111: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/111.jpg)
Click to edit Master title style
Numerical Methods
• As you can see from this example, the most difficult step is to solve a system of nonlinear equations to obtain the candidate points.
• Instead of taking gradients of functions, automated NLP solvers use various methods to change a general NLP into an easier optimization problem.
![Page 112: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/112.jpg)
Click to edit Master title style
Excel Example
Let’s solve the previous example with Excel:
Minimize f(x) = (x1 – 1)2 + (x2)2
Subject to: h(x) = (x1)2 + (x2)2 + x1 + x2 = 0
g(x) = x1 – (x2)2 ≤ 0
![Page 113: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/113.jpg)
Click to edit Master title style
Excel Example
We enter the objective function and constraint equations into the spreadsheet:
Variables:x1 x20 0
Value LimitObjective Function: =(A3-1) 2̂+B3 2̂
Constraint 1: =A3 2̂+B3 2̂+A3+B3 0Constraint 2: =A3-B3 2̂ 0
![Page 114: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/114.jpg)
Click to edit Master title style
Excel Example
Now, open the solver dialog box under the Tools menu and specify the objective function value as the target cell and choose the Min option. As it is written, A3 & B3 are the variable cells. And the constraints should be added – the equality constraint and the ≤ constraint.
![Page 115: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/115.jpg)
Click to edit Master title style
Excel Example
The solver box should look like the following:
![Page 116: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/116.jpg)
Click to edit Master title style
Excel Example
• This is a nonlinear model, so unlike the examples in the last chapter, we won’t choose the “Assume Linear Model” in the options menu
• Also, x1 & x2 are not specified to be positive, so we don’t check the “Assume Non-negative” box
• If desired, the tolerance may be decreased to 0.1%
![Page 117: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/117.jpg)
Click to edit Master title style
Excel Example
• When we solve the problem, the spreadsheet doesn’t change because our initial guess of x1 = 0 & x2 = 0 is an optimum solution, as we found when we solved the problem analytically.
![Page 118: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/118.jpg)
Click to edit Master title style
Excel Example
However, if we choose initial values of both x1 & x2 as -1, we get the following solution
![Page 119: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/119.jpg)
Click to edit Master title style
Conclusions
• So, by varying the initial values, we can get both of the candidate points we found previously
• However, the NLP solver tells us that they are both local minimum points
![Page 120: Click to edit Master title style Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d8b5503460f94a7220d/html5/thumbnails/120.jpg)
Click to edit Master title style
References
Material for this chapter has been taken from:
• Optimization of Chemical Processes 2nd Ed.; Edgar, Thomas; David Himmelblau; & Leon Lasdon.