(iii) lagrange multipliers and kuhn-tucker conditions d nagesh kumar, iisc introduction to...
TRANSCRIPT
![Page 1: (iii) Lagrange Multipliers and Kuhn-tucker Conditions D Nagesh Kumar, IISc Introduction to Optimization Water Resources Systems Planning and Management:](https://reader036.vdocuments.us/reader036/viewer/2022081504/5697c0101a28abf838ccb322/html5/thumbnails/1.jpg)
(iii) Lagrange Multipliers and Kuhn-tucker Conditions
D Nagesh Kumar, IISc
Introduction to Optimization
Water Resources Systems Planning and Management: M2L3
![Page 2: (iii) Lagrange Multipliers and Kuhn-tucker Conditions D Nagesh Kumar, IISc Introduction to Optimization Water Resources Systems Planning and Management:](https://reader036.vdocuments.us/reader036/viewer/2022081504/5697c0101a28abf838ccb322/html5/thumbnails/2.jpg)
Objectives
D Nagesh Kumar, IISc2
To study the optimization with multiple decision variables
and equality constraint : Lagrange Multipliers.
To study the optimization with multiple decision variables
and inequality constraint : Kuhn-Tucker (KT) conditions
Water Resources Systems Planning and Management: M2L3
![Page 3: (iii) Lagrange Multipliers and Kuhn-tucker Conditions D Nagesh Kumar, IISc Introduction to Optimization Water Resources Systems Planning and Management:](https://reader036.vdocuments.us/reader036/viewer/2022081504/5697c0101a28abf838ccb322/html5/thumbnails/3.jpg)
Constrained optimization with equality constraints
D Nagesh Kumar, IISc3
A function of multiple variables, f(x), is to be optimized subject to one or more
equality constraints of many variables. These equality constraints, gj(x), may or may
not be linear. The problem statement is as follows:
Maximize (or minimize) f(X), subject to gj(X) = 0, j = 1, 2, … , m
where
(1)
1
2
n
x
x
x
X
Water Resources Systems Planning and Management: M2L3
![Page 4: (iii) Lagrange Multipliers and Kuhn-tucker Conditions D Nagesh Kumar, IISc Introduction to Optimization Water Resources Systems Planning and Management:](https://reader036.vdocuments.us/reader036/viewer/2022081504/5697c0101a28abf838ccb322/html5/thumbnails/4.jpg)
Constrained optimization…
D Nagesh Kumar, IISc4
With the condition that ; or else if m > n then the problem
becomes an over defined one and there will be no solution. Of the
many available methods, the method of constrained variation and the
method of using Lagrange multipliers are discussed.
m n
Water Resources Systems Planning and Management: M2L3
![Page 5: (iii) Lagrange Multipliers and Kuhn-tucker Conditions D Nagesh Kumar, IISc Introduction to Optimization Water Resources Systems Planning and Management:](https://reader036.vdocuments.us/reader036/viewer/2022081504/5697c0101a28abf838ccb322/html5/thumbnails/5.jpg)
Solution by method of Lagrange multipliers
D Nagesh Kumar, IISc5
Continuing with the same specific case of the optimization problem with
n = 2 and m = 1 we define a quantity λ, called the Lagrange multiplier as
(2)
Using this in the constrained variation of f [ given in the previous lecture]
And (2) written as (3)
(4)
* *1 2
2
2 (x , x )
/
/
f x
g x
* *1 2
1 1 (x , x )
0f g
x x
* *1 2
2 2 (x , x )
0f g
x x
1 2
11
1 2 2 (x *, x *)
/0
/
g xf fdf dx
x g x x
Water Resources Systems Planning and Management: M2L3
![Page 6: (iii) Lagrange Multipliers and Kuhn-tucker Conditions D Nagesh Kumar, IISc Introduction to Optimization Water Resources Systems Planning and Management:](https://reader036.vdocuments.us/reader036/viewer/2022081504/5697c0101a28abf838ccb322/html5/thumbnails/6.jpg)
Solution by method of Lagrange multipliers…
D Nagesh Kumar, IISc6
Also, the constraint equation has to be satisfied at the extreme point
(5)
Hence equations (2) to (5) represent the necessary conditions for the point
[x1*, x2*] to be an extreme point.
λ could be expressed in terms of as well has to be non-
zero.
These necessary conditions require that at least one of the partial
derivatives of g(x1 , x2) be non-zero at an extreme point.
* *1 2
1 2 ( , )( , ) 0
x xg x x
1/g x 1/g x
Water Resources Systems Planning and Management: M2L3
![Page 7: (iii) Lagrange Multipliers and Kuhn-tucker Conditions D Nagesh Kumar, IISc Introduction to Optimization Water Resources Systems Planning and Management:](https://reader036.vdocuments.us/reader036/viewer/2022081504/5697c0101a28abf838ccb322/html5/thumbnails/7.jpg)
D Nagesh Kumar, IISc7
The conditions given by equations (2) to (5) can also be generated by
constructing a functions L, known as the Lagrangian function, as
(6)
Alternatively, treating L as a function of x1,x2 and , the necessary
conditions for its extremum are given by
(7)
1 2 1 2 1 2( , , ) ( , ) ( , )L x x f x x g x x
1 2 1 2 1 21 1 1
1 2 1 2 1 22 2 2
1 2 1 2
( , , ) ( , ) ( , ) 0
( , , ) ( , ) ( , ) 0
( , , ) ( , ) 0
L f gx x x x x x
x x x
L f gx x x x x x
x x x
Lx x g x x
Solution by method of Lagrange multipliers…
Water Resources Systems Planning and Management: M2L3
![Page 8: (iii) Lagrange Multipliers and Kuhn-tucker Conditions D Nagesh Kumar, IISc Introduction to Optimization Water Resources Systems Planning and Management:](https://reader036.vdocuments.us/reader036/viewer/2022081504/5697c0101a28abf838ccb322/html5/thumbnails/8.jpg)
Necessary conditions for a general problem
D Nagesh Kumar, IISc8
For a general problem with n variables and m equality constraints the
problem is defined as shown earlier
Maximize (or minimize) f(X), subject to gj(X) = 0, j = 1, 2, … , m
where
In this case the Lagrange function, L, will have one Lagrange multiplier j
for each constraint as
(8)1 2 , 1 2 1 1 2 2( , ,..., , ,..., ) ( ) ( ) ( ) ... ( )n m m mL x x x f g g g X X X X
1
2
n
x
x
x
X
Water Resources Systems Planning and Management: M2L3
![Page 9: (iii) Lagrange Multipliers and Kuhn-tucker Conditions D Nagesh Kumar, IISc Introduction to Optimization Water Resources Systems Planning and Management:](https://reader036.vdocuments.us/reader036/viewer/2022081504/5697c0101a28abf838ccb322/html5/thumbnails/9.jpg)
D Nagesh Kumar, IISc9
L is now a function of n + m unknowns, , and the
necessary conditions for the problem defined above are given by
(9)
which represent n + m equations in terms of the n + m unknowns, xi and j.
The solution to this set of equations gives us
and (10)
1 2 , 1 2, ,..., , ,...,n mx x x
1
( ) ( ) 0, 1, 2,..., ; 1, 2,...,
( ) 0, 1, 2,...,
mj
jji i i
jj
gL fi n j m
x x x
Lg j m
X X
X
*1
*2
*n
x
x
x
X
*1
** 2
*m
Necessary conditions for a general problem…
Water Resources Systems Planning and Management: M2L3
![Page 10: (iii) Lagrange Multipliers and Kuhn-tucker Conditions D Nagesh Kumar, IISc Introduction to Optimization Water Resources Systems Planning and Management:](https://reader036.vdocuments.us/reader036/viewer/2022081504/5697c0101a28abf838ccb322/html5/thumbnails/10.jpg)
Sufficient conditions for a general problem
D Nagesh Kumar, IISc10
A sufficient condition for f(X) to have a relative minimum at X* is that each
root of the polynomial in Є, defined by the following determinant equation
be positive.
(11)
11 12 1 11 21 1
21 22 2 12 22 2
1 2 1 2
11 12 1
21 22 2
1 2
0
0 0
0 0
n m
n m
n n nn n n mn
n
n
m m mn
L L L g g g
L L L g g g
L L L g g g
g g g
g g g
g g g
Water Resources Systems Planning and Management: M2L3
![Page 11: (iii) Lagrange Multipliers and Kuhn-tucker Conditions D Nagesh Kumar, IISc Introduction to Optimization Water Resources Systems Planning and Management:](https://reader036.vdocuments.us/reader036/viewer/2022081504/5697c0101a28abf838ccb322/html5/thumbnails/11.jpg)
D Nagesh Kumar, IISc11
where
(12)
Similarly, a sufficient condition for f(X) to have a relative maximum at X*
is that each root of the polynomial in Є, defined by equation (11) be
negative.
If equation (11), on solving yields roots, some of which are positive and
others negative, then the point X* is neither a maximum nor a minimum.
2* *
*
( , ), for 1,2,..., 1,2,...,
( ), where 1,2,..., and 1,2,...,
iji j
ppq
q
LL i n and j m
x x
gg p m q n
x
X
X
Sufficient conditions for a general problem…
Water Resources Systems Planning and Management: M2L3
![Page 12: (iii) Lagrange Multipliers and Kuhn-tucker Conditions D Nagesh Kumar, IISc Introduction to Optimization Water Resources Systems Planning and Management:](https://reader036.vdocuments.us/reader036/viewer/2022081504/5697c0101a28abf838ccb322/html5/thumbnails/12.jpg)
Example
D Nagesh Kumar, IISc12
Minimize ,
Subject to
Solution
with n = 2 and m = 1
L =
or
2 21 1 2 2 1 2( ) 3 6 5 7 5f x x x x x x X
1 2 5x x
1 1 2( ) 5 0g x x X
1 2 , 1 2 1 1 2 2( , ,..., , ,..., ) ( ) ( ) ( ) ... ( )n m m mx x x f g g g L X X X X
2 21 1 2 2 1 2 1 1 23 6 5 7 5 ( 5)x x x x x x x x
1 2 11
1 2 1
1
6 6 7 0
1(7 )
61
5 (7 )6
x xx
x x
L
1 23
Water Resources Systems Planning and Management: M2L3
![Page 13: (iii) Lagrange Multipliers and Kuhn-tucker Conditions D Nagesh Kumar, IISc Introduction to Optimization Water Resources Systems Planning and Management:](https://reader036.vdocuments.us/reader036/viewer/2022081504/5697c0101a28abf838ccb322/html5/thumbnails/13.jpg)
Example…
D Nagesh Kumar, IISc13
1 2 12
1 2 1
1 2 2 1
6 10 5 0
13 5 (5 )
21
3( ) 2 (5 )2
x xx
x x
x x x
L
2
1
2x
1
11
2x
11 1* , ; * 23
2 2
X λ
11 12 11
21 22 21
11 12
0
0
L L g
L L g
g g
Hence,
Water Resources Systems Planning and Management: M2L3
![Page 14: (iii) Lagrange Multipliers and Kuhn-tucker Conditions D Nagesh Kumar, IISc Introduction to Optimization Water Resources Systems Planning and Management:](https://reader036.vdocuments.us/reader036/viewer/2022081504/5697c0101a28abf838ccb322/html5/thumbnails/14.jpg)
Example…
D Nagesh Kumar, IISc14
2
11 21 ( )
6Lx
X*,λ*
L 2
12 211 2 ( )
6L Lx x
X*,λ*
L 2
22 22 ( )
10Lx
X*,λ*
L
111
1 ( )
112 21
2 ( )
1
1
gg
x
gg g
x
X*,λ*
X*,λ*
6 6 1
6 10 1 0
1 1 0
( 6 )[ 1] ( 6)[ 1] 1[ 6 10 ] 0
2
The determinant becomes
or
*λSince is negative, X*, correspond to a maximum
Water Resources Systems Planning and Management: M2L3
![Page 15: (iii) Lagrange Multipliers and Kuhn-tucker Conditions D Nagesh Kumar, IISc Introduction to Optimization Water Resources Systems Planning and Management:](https://reader036.vdocuments.us/reader036/viewer/2022081504/5697c0101a28abf838ccb322/html5/thumbnails/15.jpg)
Kuhn – Tucker Conditions
D Nagesh Kumar, IISc15
KT condition: Both necessary and sufficient if the objective function
is concave and each constraint is linear or each constraint function is
concave, i.e., the problems belongs to a class called the convex
programming problems.
Water Resources Systems Planning and Management: M2L3
![Page 16: (iii) Lagrange Multipliers and Kuhn-tucker Conditions D Nagesh Kumar, IISc Introduction to Optimization Water Resources Systems Planning and Management:](https://reader036.vdocuments.us/reader036/viewer/2022081504/5697c0101a28abf838ccb322/html5/thumbnails/16.jpg)
Kuhn-Tucker Conditions: Optimization Model
Consider the following optimization problem
Minimize f(X)
subject to
gj(X) ≤ 0 for j=1,2,…,p
where the decision variable vector
X=[x1,x2,…,xn]
D Nagesh Kumar, IIScWater Resources Systems Planning and Management: M2L316
![Page 17: (iii) Lagrange Multipliers and Kuhn-tucker Conditions D Nagesh Kumar, IISc Introduction to Optimization Water Resources Systems Planning and Management:](https://reader036.vdocuments.us/reader036/viewer/2022081504/5697c0101a28abf838ccb322/html5/thumbnails/17.jpg)
Kuhn-Tucker Conditions
1
0 1, 2,...,
0 1, 2,...,
0 1, 2,...,
0 1, 2,...,
m
jji i
j j
j
j
f gi n
x x
g j m
g j m
j m
Kuhn-Tucker conditions for X* = [x1* , x2
* , . . . xn*] to be a local minimum are
D Nagesh Kumar, IIScWater Resources Systems Planning and Management: M2L317
![Page 18: (iii) Lagrange Multipliers and Kuhn-tucker Conditions D Nagesh Kumar, IISc Introduction to Optimization Water Resources Systems Planning and Management:](https://reader036.vdocuments.us/reader036/viewer/2022081504/5697c0101a28abf838ccb322/html5/thumbnails/18.jpg)
Kuhn Tucker Conditions …
In case of minimization problems, if the constraints are of
the form gj(X) ≥ 0, then λj have to be non-positive
If the problem is one of maximization with the constraints
in the form gj(X) ≥ 0, then λj have to be nonnegative.
D Nagesh Kumar, IIScWater Resources Systems Planning and Management: M2L318
![Page 19: (iii) Lagrange Multipliers and Kuhn-tucker Conditions D Nagesh Kumar, IISc Introduction to Optimization Water Resources Systems Planning and Management:](https://reader036.vdocuments.us/reader036/viewer/2022081504/5697c0101a28abf838ccb322/html5/thumbnails/19.jpg)
Example 1
D Nagesh Kumar, IISc19
Minimize
subject to
2 2 21 2 32 3f x x x
1 1 2 3
2 1 2 3
2 12
2 3 8
g x x x
g x x x
Water Resources Systems Planning and Management: M2L3
![Page 20: (iii) Lagrange Multipliers and Kuhn-tucker Conditions D Nagesh Kumar, IISc Introduction to Optimization Water Resources Systems Planning and Management:](https://reader036.vdocuments.us/reader036/viewer/2022081504/5697c0101a28abf838ccb322/html5/thumbnails/20.jpg)
D Nagesh Kumar, IISc20
1 21 2 0
i i i
g gf
x x x
0 j jg
0 jg
0j
Kuhn – Tucker Conditions
Example 1…
1 1 2
2 1 2
3 1 2
2 0 (14)
4 2 0 (15)
6 2 3 0
x
x
x
(16)
1 1 2 3
2 1 2 3
( 2 12) 0 (17)
( 2 3 8) 0 (18)
x x x
x x x
1 2 3
1 2 3
2 12 0 (19)
2 3 8 0 (20)
x x x
x x x
1
2
0 (21)
0 (22)
Water Resources Systems Planning and Management: M2L3
![Page 21: (iii) Lagrange Multipliers and Kuhn-tucker Conditions D Nagesh Kumar, IISc Introduction to Optimization Water Resources Systems Planning and Management:](https://reader036.vdocuments.us/reader036/viewer/2022081504/5697c0101a28abf838ccb322/html5/thumbnails/21.jpg)
From (17) either = 0 or ,
Case 1: = 0
From (14), (15) and (16) we have x1 = x2 = and x3 =
Using these in (18) we get
From (22), , therefore, =0,
Therefore, X* = [ 0, 0, 0 ]
This solution set satisfies all of (18) to (21)
1 1 2 32 12 0x x x
2 / 2 2 / 222 2 28 0, 0 8or
2 0 2
Example 1…
1
D Nagesh Kumar, IISc21 Water Resources Systems Planning and Management: M2L3
![Page 22: (iii) Lagrange Multipliers and Kuhn-tucker Conditions D Nagesh Kumar, IISc Introduction to Optimization Water Resources Systems Planning and Management:](https://reader036.vdocuments.us/reader036/viewer/2022081504/5697c0101a28abf838ccb322/html5/thumbnails/22.jpg)
Case 2:
Using (14), (15) and (16), we have
or
But conditions (21) and (22) give us and
simultaneously, which cannot be
possible with
Hence the solution set for this optimization problem is
X* = [ 0 0 0 ]
1 2 32 12 0x x x
1 2 1 2 1 22 2 312 0
2 4 3
1 217 12 144
1 0 2 0
1 217 12 144
Example 1…
D Nagesh Kumar, IISc22 Water Resources Systems Planning and Management: M2L3
![Page 23: (iii) Lagrange Multipliers and Kuhn-tucker Conditions D Nagesh Kumar, IISc Introduction to Optimization Water Resources Systems Planning and Management:](https://reader036.vdocuments.us/reader036/viewer/2022081504/5697c0101a28abf838ccb322/html5/thumbnails/23.jpg)
Minimize
subject to
2 21 2 160f x x x
1 1
2 1 2
80 0
120 0
g x
g x x
Example 2
D Nagesh Kumar, IISc23 Water Resources Systems Planning and Management: M2L3
![Page 24: (iii) Lagrange Multipliers and Kuhn-tucker Conditions D Nagesh Kumar, IISc Introduction to Optimization Water Resources Systems Planning and Management:](https://reader036.vdocuments.us/reader036/viewer/2022081504/5697c0101a28abf838ccb322/html5/thumbnails/24.jpg)
1 21 2 0
i i i
g gf
x x x
0 j jg
0 jg
0j
Kuhn – Tucker Conditions
Example 2…
1 1 2
2 2
2 60 0 (23)
2 0 (24)
x
x
1 1
2 1 2
( 80) 0 (25)
( 120) 0 (26)
x
x x
1
1 2
80 0 (27)
120 0 (28)
x
x x
1
2
0 (29)
0 (30)
D Nagesh Kumar, IISc24 Water Resources Systems Planning and Management: M2L3
![Page 25: (iii) Lagrange Multipliers and Kuhn-tucker Conditions D Nagesh Kumar, IISc Introduction to Optimization Water Resources Systems Planning and Management:](https://reader036.vdocuments.us/reader036/viewer/2022081504/5697c0101a28abf838ccb322/html5/thumbnails/25.jpg)
From (25) either = 0 or ,
Case 1
From (23) and (24) we have and
Using these in (26) we get
Considering , X* = [ 30, 0]. But this solution set violates (27)
and (28)
For , X* = [ 45, 75]. But this solution set violates (27)
1 1( 80) 0x
21 302x 2
2 2x
2 2 150 0
2 0 150or
2 0
2 150
Example 2…
D Nagesh Kumar, IISc25 Water Resources Systems Planning and Management: M2L3
![Page 26: (iii) Lagrange Multipliers and Kuhn-tucker Conditions D Nagesh Kumar, IISc Introduction to Optimization Water Resources Systems Planning and Management:](https://reader036.vdocuments.us/reader036/viewer/2022081504/5697c0101a28abf838ccb322/html5/thumbnails/26.jpg)
Case 2:
Using in (23) and (24), we have
Substitute (31) in (26), we have
For this to be true, either
1( 80) 0x
1 80x
2 22 40 0x x
2 20 40 0x or x
Example 2…
2 2
1 2
2
2 220 (31)
x
x
D Nagesh Kumar, IISc26 Water Resources Systems Planning and Management: M2L3
![Page 27: (iii) Lagrange Multipliers and Kuhn-tucker Conditions D Nagesh Kumar, IISc Introduction to Optimization Water Resources Systems Planning and Management:](https://reader036.vdocuments.us/reader036/viewer/2022081504/5697c0101a28abf838ccb322/html5/thumbnails/27.jpg)
For ,
This solution set violates (27) and (28)
For ,
This solution set is satisfying all equations from (27) to (31) and hence
the desired
Thus, the solution set for this optimization problem is
X* = [ 80, 40 ]
2 0x 1 220
2 40 0x 1 2140 80and
Example 2…
D Nagesh Kumar, IISc27 Water Resources Systems Planning and Management: M2L3
![Page 28: (iii) Lagrange Multipliers and Kuhn-tucker Conditions D Nagesh Kumar, IISc Introduction to Optimization Water Resources Systems Planning and Management:](https://reader036.vdocuments.us/reader036/viewer/2022081504/5697c0101a28abf838ccb322/html5/thumbnails/28.jpg)
BIBLIOGRAPHY / FURTHER READING
1. Rao S.S., Engineering Optimization – Theory and Practice, Fourth
Edition, John Wiley and Sons, 2009.
2. Ravindran A., D.T. Phillips and J.J. Solberg, Operations Research –
Principles and Practice, John Wiley & Sons, New York, 2001.
3. Taha H.A., Operations Research – An Introduction, 8th edition, Pearson
Education India, 2008.
4. Vedula S., and P.P. Mujumdar, Water Resources Systems: Modelling
Techniques and Analysis, Tata McGraw Hill, New Delhi, 2005.
D Nagesh Kumar, IISc28 Water Resources Systems Planning and Management: M2L3
![Page 29: (iii) Lagrange Multipliers and Kuhn-tucker Conditions D Nagesh Kumar, IISc Introduction to Optimization Water Resources Systems Planning and Management:](https://reader036.vdocuments.us/reader036/viewer/2022081504/5697c0101a28abf838ccb322/html5/thumbnails/29.jpg)
Thank You
D Nagesh Kumar, IIScWater Resources Systems Planning and Management: M2L3