![Page 1: Constrained optimization: direct methodsusers.jyu.fi/~jhaka/opt/TIES483_constrained_direct1.pdfDirect methods Also known as methods of feasible directions Idea βin a point π₯β,](https://reader033.vdocuments.us/reader033/viewer/2022041908/5e64ff1bfbbda607750871ec/html5/thumbnails/1.jpg)
Constrained optimization:
direct methods
Jussi Hakanen
Post-doctoral researcher [email protected]
spring 2014 TIES483 Nonlinear optimization
![Page 2: Constrained optimization: direct methodsusers.jyu.fi/~jhaka/opt/TIES483_constrained_direct1.pdfDirect methods Also known as methods of feasible directions Idea βin a point π₯β,](https://reader033.vdocuments.us/reader033/viewer/2022041908/5e64ff1bfbbda607750871ec/html5/thumbnails/2.jpg)
Classification of the methods
Indirect methods: the constrained problem is
converted into a sequence of unconstrained
problems whose solutions will approach to the
solution of the constrained problem, the
intermediate solutions need not to be feasible
Direct methods: the constraints are taking into
account explicitly, intermediate solutions are
feasible
spring 2014 TIES483 Nonlinear optimization
![Page 3: Constrained optimization: direct methodsusers.jyu.fi/~jhaka/opt/TIES483_constrained_direct1.pdfDirect methods Also known as methods of feasible directions Idea βin a point π₯β,](https://reader033.vdocuments.us/reader033/viewer/2022041908/5e64ff1bfbbda607750871ec/html5/thumbnails/3.jpg)
Direct methods
Also known as methods of feasible directions
Idea β in a point π₯β, generate a feasible search direction where
objection function value can be improved
β use line search to get π₯β+1
Reminder: A direction π β π π is a feasible descent direction in π₯β β π if there exists πΌβ > 0 such that π π₯β + πΌπ < π π₯β and π₯β + πΌπ β π for all πΌ β (0, πΌβ]. Methods differ in β how to choose a feasible direction and
β what is assumed from the constraints (linear/nonlinear, equality/inequality)
spring 2014 TIES483 Nonlinear optimization
![Page 4: Constrained optimization: direct methodsusers.jyu.fi/~jhaka/opt/TIES483_constrained_direct1.pdfDirect methods Also known as methods of feasible directions Idea βin a point π₯β,](https://reader033.vdocuments.us/reader033/viewer/2022041908/5e64ff1bfbbda607750871ec/html5/thumbnails/4.jpg)
Algorithm
1) Choose a starting point π₯1 and set β = 1.
2) Determine a feasible direction πβ such that
for some (small enough) πΌ > 0 it holds that
π₯β + πΌπβ β π and π π₯β + πΌπβ < π(π₯β).
3) Use line search to find an optimal step length
πΌβ > 0 such that π₯β + πΌβπβ β π.
4) Test convergence. If not converged, set
π₯β+1 = π₯β + πΌβπβ , β = β + 1 and go to 2).
spring 2014 TIES483 Nonlinear optimization
![Page 5: Constrained optimization: direct methodsusers.jyu.fi/~jhaka/opt/TIES483_constrained_direct1.pdfDirect methods Also known as methods of feasible directions Idea βin a point π₯β,](https://reader033.vdocuments.us/reader033/viewer/2022041908/5e64ff1bfbbda607750871ec/html5/thumbnails/5.jpg)
Examples of direct methods
Projected gradient method
Active set method
Sequential Quadratic Programming (SQP)
method
spring 2014 TIES483 Nonlinear optimization
![Page 6: Constrained optimization: direct methodsusers.jyu.fi/~jhaka/opt/TIES483_constrained_direct1.pdfDirect methods Also known as methods of feasible directions Idea βin a point π₯β,](https://reader033.vdocuments.us/reader033/viewer/2022041908/5e64ff1bfbbda607750871ec/html5/thumbnails/6.jpg)
Projected gradient method
Consider a problem min π π₯ π . π‘. π΄π₯ = π,
where π΄ is an π Γ π matrix (π β€ π) and π β π π
β π₯β β π if and only if π΄π₯β = π
Direction of the steepest descent is βπ»π(π₯)
β May not be feasible
Idea: project the steepest descent direction
into the feasible region π
β A projection matrix is needed
spring 2014 TIES483 Nonlinear optimization
![Page 7: Constrained optimization: direct methodsusers.jyu.fi/~jhaka/opt/TIES483_constrained_direct1.pdfDirect methods Also known as methods of feasible directions Idea βin a point π₯β,](https://reader033.vdocuments.us/reader033/viewer/2022041908/5e64ff1bfbbda607750871ec/html5/thumbnails/7.jpg)
Projection
Definition: If for an π Γ π matrix π it holds that
ππ = π and ππ = π, then π is a projection
matrix
Vector ππ»π(π₯β) is a projected gradient in π₯β
Matrix ππ» π₯β π is a projected Hessian in π₯β
spring 2014 TIES483 Nonlinear optimization
![Page 8: Constrained optimization: direct methodsusers.jyu.fi/~jhaka/opt/TIES483_constrained_direct1.pdfDirect methods Also known as methods of feasible directions Idea βin a point π₯β,](https://reader033.vdocuments.us/reader033/viewer/2022041908/5e64ff1bfbbda607750871ec/html5/thumbnails/8.jpg)
Idea
If π₯β β π, then π is a feasible direction in π₯β if
and only if π΄π = 0
π₯β is a local minimizer if and only if for all
feasible directions π, π = 1, there exists
πΌ > 0 such that π π₯β + πΌπ β₯ π(π₯β), when
0 < πΌ < πΌ
Find a feasible π where π improves the most
βΉ minπβπ π
π»π π₯β ππ π . π‘. π΄π = 0, π = 1
spring 2014 TIES483 Nonlinear optimization
![Page 9: Constrained optimization: direct methodsusers.jyu.fi/~jhaka/opt/TIES483_constrained_direct1.pdfDirect methods Also known as methods of feasible directions Idea βin a point π₯β,](https://reader033.vdocuments.us/reader033/viewer/2022041908/5e64ff1bfbbda607750871ec/html5/thumbnails/9.jpg)
π΄π = 0
Letβs consider the constraints π΄π = 0: form a subspace in π π
Denote by π the matrix whose columns span π π΄π = 0} β π π β Each feasible direction can be represented as a
linear combination of the columns of π, that is, π = ππ for some π β π πβπ
π can be formed e.g. by using 1) elimination method or 2) LQ decomposition (check some book for numerical methods
spring 2014 TIES483 Nonlinear optimization
![Page 10: Constrained optimization: direct methodsusers.jyu.fi/~jhaka/opt/TIES483_constrained_direct1.pdfDirect methods Also known as methods of feasible directions Idea βin a point π₯β,](https://reader033.vdocuments.us/reader033/viewer/2022041908/5e64ff1bfbbda607750871ec/html5/thumbnails/10.jpg)
Elimination method
minπ π₯ π . π‘. π΄π₯ = π has π linear equality constraints β Can be transformed into an unconstrained problem by eliminating π
variables by using the constraints
Assume that the rows of π΄ are linearly independent (otherwise some of the constraints is not needed or there are no feasible solutions) β π΄ has π linearly independent columns β π΄ = (π΅ π), where π΅ β π πΓπ has linearly independent columns
β π₯ = π₯π΅, π₯ππ , π₯π΅ β π π , π₯π β π πβπ
Then π΄π₯ = π΅ ππ₯π΅
π₯π= π΅π₯π΅ + ππ₯π = π and further π₯π΅ =
π΅β1(π β ππ₯π) since π΅ is nonsingular
Therefore, by choosing π₯π the resulting π₯ β π
Now we can set π΄π = π΅ π π = 0 so that π = βπ΅β1ππΌ
spring 2014 TIES483 Nonlinear optimization
![Page 11: Constrained optimization: direct methodsusers.jyu.fi/~jhaka/opt/TIES483_constrained_direct1.pdfDirect methods Also known as methods of feasible directions Idea βin a point π₯β,](https://reader033.vdocuments.us/reader033/viewer/2022041908/5e64ff1bfbbda607750871ec/html5/thumbnails/11.jpg)
Optimality conditions
Necessary: If π₯β (π΄π₯β = π) is a local
minimizer, then
1) πππ»π π₯β = 0 and
2) matrix πππ» πβ π is positive semidefinite
Sufficient: If in π₯β (π΄π₯β = π)
1) πππ»π π₯β = 0 and
2) matrix πππ» π₯β π is positive definite,
then π₯β is a local minimizer
spring 2014 TIES483 Nonlinear optimization
![Page 12: Constrained optimization: direct methodsusers.jyu.fi/~jhaka/opt/TIES483_constrained_direct1.pdfDirect methods Also known as methods of feasible directions Idea βin a point π₯β,](https://reader033.vdocuments.us/reader033/viewer/2022041908/5e64ff1bfbbda607750871ec/html5/thumbnails/12.jpg)
Using projected gradient
Result: It holds that πππ»π π₯β = 0 if and only if
there exists Lagrange multiplier vector πβ β π π
such that π»π π₯β + π΄π πβ = 0.
So, if πππ»π π₯β = 0, then π₯β is a local
minimizer. Otherwise, βπππ»π π₯β is a descent
direction.
spring 2014 TIES483 Nonlinear optimization
![Page 13: Constrained optimization: direct methodsusers.jyu.fi/~jhaka/opt/TIES483_constrained_direct1.pdfDirect methods Also known as methods of feasible directions Idea βin a point π₯β,](https://reader033.vdocuments.us/reader033/viewer/2022041908/5e64ff1bfbbda607750871ec/html5/thumbnails/13.jpg)
Rosenβs projected gradient method
Feasible direction: πβ = ππβ for some πβ β π πβπ
Maximum decrease:
minπππ»π π₯β = min πππππ»π(π₯β)
If it is required that π = 1, then minπ
πππππ»π(π₯β)
ππ
The solution is πβ = β πππ β1πππ»π(π₯β) and we
have πβ = ππβ = βπ πππ β1πππ»π(π₯β)
π = π πππ β1ππ
spring 2014 TIES483 Nonlinear optimization
![Page 14: Constrained optimization: direct methodsusers.jyu.fi/~jhaka/opt/TIES483_constrained_direct1.pdfDirect methods Also known as methods of feasible directions Idea βin a point π₯β,](https://reader033.vdocuments.us/reader033/viewer/2022041908/5e64ff1bfbbda607750871ec/html5/thumbnails/14.jpg)
Algorithm
1) Choose a starting point π₯1. Determine π and set
β = 1. Compute π = π πππ β1ππ.
2) Compute direction πβ = βππ»π π₯β π.
3) If πβ = 0, stop. Otherwise, find
πΌπππ₯ = max πΌ | π₯β + πΌπβ β π and solve
minπ π₯β + πΌπβ π . π‘. 0 β€ πΌ β€ πΌπππ₯ .
Let the solution be πΌβ. Set π₯β+1 = π₯β + πΌβπβ, β = β +
1 and go to 2).
spring 2014 TIES483 Nonlinear optimization
![Page 15: Constrained optimization: direct methodsusers.jyu.fi/~jhaka/opt/TIES483_constrained_direct1.pdfDirect methods Also known as methods of feasible directions Idea βin a point π₯β,](https://reader033.vdocuments.us/reader033/viewer/2022041908/5e64ff1bfbbda607750871ec/html5/thumbnails/15.jpg)
Note
If there are both linear equality and inequality constraints, the projection matrix does not remain the same β At each iteration, it includes only the equality and active
inequality constraints
π = π πππ β1ππ = πΌ β π΄π π΄π΄π β1π΄ β π is not necessarily needed
β In bigger problems, π is more convenient since it is smaller than π΄
If minπ
πππππ»π(π₯β)
π then πβ = βππππ»π(π₯β), π = πππ
Convergence is quite low since the second derivatives are not taken into account
spring 2014 TIES483 Nonlinear optimization