The New Faces of Nonlinear Programming

Jorge Nocedal Optimization Technology Center

Argonne-Northwestern

n

i

i

x

x

IixgEixh

xf

,0)( ,0)( s.t.

)( min

04/22/23 McMaster 2

New Problems, New Algorithms, New Software

• Traditional Applications: solve larger problems, more robustness• New classes of applications• Advances in modeling languages: AMPL, …• Automatic differentiation• Interior Methods• Test problems (CUTE, COPS)• New packages: LOQO, KNITRO,.• Internet Optimization: NEOS server

04/22/23 McMaster 3

NEOS Server

• Argonne • Northwestern

MINOS. SNOPT, FILTER, LANCELOTLOQO, KNITRO

USER

04/22/23 McMaster 4

Semi-infinite Optimization Mixed Integer Nonlinearly Constrained Optimization Mixed Integer Linear Programming Nonlinearly Constrained Optimization Semidefinite & Second Order Cone Programming Linear Programming Unconstrained Optimization Linear Network Optimization Complementarity Problems Nondifferentiable Optimization Stochastic Linear Programming Global Optimization Application-specific Optimization

04/22/23 McMaster 5

Part I:New Classes of Problems

Instead of new algorithms/software/ adapt existing techniques

• Equilibrium constraints: (T. Luo)o Bi-level programmingo Complementarity constraints

• Semi-definite programming (??)• PDE-constrained optimization

o Differential algebraic systems

04/22/23 McMaster 6

Nonlinear Optimization Formulation

• Theme: constraints involve a difficult computation/simulation.

• Limitations of this formulation?• Logic constraints

n

i

i

x

x

IixgEixh

xf

,0)( ,0)( s.t.

)( min

04/22/23 McMaster 7

Equilibrium Constraints

• Structurally difficult• No strictly feasible direction• Algorithmically…

0 0 ,0

ii yxyx

xy

10

01

x

y

f

c

0ii yx

04/22/23 McMaster 8

• Optimization problem with equilibrium constraints not stable cannot apply NLP algorithms to it

• Confirmed by experimental evidence (??)

• Reality: software not capable of dealing with degeneracy, not sufficiently robust

• Theoretical mistake: lack of stability does not imply practical problems. Structural degeneracy.

• Active Set SQP (Leyffer et al)

• Interior Methods: solve perturbed problem

04/22/23 McMaster 9

origin

destination

For each origin-destination pair (o,d) we have:

• qod: demand (in terms of flow) between o and d

• K : index set of paths from o to d

• fk : flow along path k, for each k in K

•ck(f): cost of travel along path k (usually time), for each path k in K

• λ = λ(qod) : minimum possible travel cost between o and d

Traffic Assignment

• Vector x of link flows, •Efficiency parameters (capacity, speed limit) given at link level• The path flows and costs are aggregated (based on x) through adjacency matrix A• Need constraints for demand satisfaction and conservation of flow• Many origin-destination pairs may exists in the network

04/22/23 McMaster 10

origin

destination

•Improvements to network: Traffic Network Design•Discrete: add lanes, links•Continuous: link capacity expansion•Boyce (1979) ed, •Continuous capacity?

Network Design (Continuous Equilibrium)

•Complex interaction between System Optimal and User Equilibrium is recognized -> bilevel programming (e.g. Abdulaal-LeBlanc)

04/22/23 McMaster 11

origin

destination

•Given network: G=(N,A)•Find additions yi to capacities ci of links i in A•So that:Cost of improvement and efficiency of network is minimized

Example of Continuous Equilibirium Network Design

iii

iiiii

iiii

yygitxyxT

xyxt

oft improvemencapacity ofcost )(link in time travel total ),(

flowfor ilink in time travel),(

xyts

ygyxT iiiii

0 ..

)](),([min

An equlibrium flow

04/22/23 McMaster 12

Together with demand satisfaction and conservation of flow, we need to demand EQUILIBRIUM, which in this case looks like:

λ = λ(qod) : minimum possible travel cost between o and d

• If there is flow on path k (fk > 0): path k is a minimum cost path (ck = λ)

• If path k is relatively expensive (ck > λ): no one uses this path (fk = 0)

0)( and 0 0 ecf ecf

KKT Conditions

04/22/23 McMaster 13

Partial Differential Equations and Optimization (Tsai, Byrd,N)

Mems flaps Desiredflow

Navier-Stokes equations

Determine position of mems flaps to optimizeQuality of exhaust flow

Phase II: boundary control

04/22/23 McMaster 14

Partial Differential Equations (PDEs)

• Systems that evolve in space (several dimensions) and time are described by PDEs

• Solution: function u(x,t) – infinite-dimen prob• More space dimen.: great computational and

storage cost

c)(hyperboli equation wave)(parabolic equation heat

xxtt

xxt

cuucuu

04/22/23 McMaster 15

Success of PDE simulations

3D Large Eddy Simulation around an airfoil

04/22/23 McMaster 16

• Fluid flow described by Navier-Stokes

on 0in 0

in 0)()(1

uu

puuuur

T

•Solution of nonlinear PDEs•Newton-Krylov

•Sequence of meshes, Krylov (FGMRES)-(full)

multigrid (Krylov smoother).

Parallel computing to obtain high resolution

)()(' kkk xcdxc

04/22/23 McMaster 17

Now optimize!

• Robustness of PDE solvers: millions of variables, hundreds of processors, multiple physical interactions

• Introduce free parameters• Finite-dimensional formulation

system of PDEs

avulvucts

vuf

),( 0),( ..),( min

04/22/23 McMaster 18

State-of-the-art algorithms

• KKT system Newton-Lagrange

• Active set: SNOPT, FilterSQP factor subset of A, reduced Hessian

• Interior: LOQO, KNITRO factor

• Algorithms must accept iterative solution of constraint linearization. Av A’v

)(

)()()(xc

xcxfxFT

0

or

0

TT AAI

AAW

c

flp

AAW T

0

04/22/23 McMaster 19

Unconstrained reformulation

• Linearize constraints• Eliminate state variables xs (basic)

• Minimize w.r.t. controls xd (non-bas) • New problem

Modern optimization SQP:

0),( . min

ds

ds

xxcts),xf(x

cdAdtsddq

cTcs

cs

TsA ..

),( min

cdAd cTcs T

sA

)(min dxf

04/22/23 McMaster 20

Weather Forecasting - Oceanography

• = state of atmosphere,

• Observations: Time windows i: length = a few time steps

• Short integration: from initial condition Problem: unknown

2

0

)(min i

M

ii xy

ix 1ix

iy

0x

610nMxxx ...,, ,10

x

04/22/23 McMaster 21

• Background field • Observations • Background covar• Obs covar• Time

Constraints eliminated, no bounds, inequalities3 Spaces: grid point, spectral, observation

Nonlinear Least Squares Problem

))(())((

)()()(min

1

0

01

00

iiiT

ii

M

ii

bb

xHyRxHy

xxBxxxJ

i

i

b

tRByx

04/22/23 McMaster 22

Part III:New Algorithms

• New applications• New methods• New software• New tools (modeling languages, automatic

differentaition)

04/22/23 McMaster 23

Part IIIAdvances in NLP Algorithms:Active Set SQP

Before 1998:• Active Set SQP software: highly complex• Many dense, substandard versions• Quasi-Newton (SNOPT, MINOS)

Present:• Filter, Second derivatives (FilterSQP)• SNOPT second derivatives in progress• Can SQP compete with Interior Methods?Future:• Linear Programming Based (Dundee, Northwestern.)

04/22/23 McMaster 24

Interior Methods Terlaky

Newton’s method to KKT conditions of equal-problem:

Reformulate to avoid rational functions: primal-dualBacktrack (difficulties!)Update barrier parameterInitial point strategy-failures

0. 0)( s.t.

)ln()( min

sg(x)xh

sxf i

0. 0)( s.t.

)( min

g(x)xh

xf

),(),(' sxFdsxF

0, s

)(

)()()(xc

xcxfxFT

04/22/23 McMaster 25

Nonlinear Interior Methods

Approach I: LOQO,OPINEL,BOEING,IPOPT,…• Modify W, • Merit Function/Filter

Approach II: (KNITRO)

0. 0)( s.t.

)ln()( min

sg(x)xh

sxf i

sdsAdxc

q

s

||d|| 0)( s.t.

(d) min

c

flp

AAW T

0

Eliminate constraints-null space approachApproximate solutionIterative solution by conjugate gradientsMerit function

enforcement

04/22/23 McMaster 26

KNITRO 3.0

Interior Active

Iterative Direct

The Future (summer 2003)

04/22/23 McMaster 27

LP-EQP based on a Penalty Approach

Equality constrained quadratic program

||d|| 00s.t

min

dggdhh

df

T

T

T

|||| 0

0 ..

dWidg

dhtsT

T

)( min dq L-1 penalty

LP

EQP

Working set W

04/22/23 McMaster 28

Step computation

dc: min quadratic

model of merit function: Cauchy point

Dogleg approach

EQP by projected CG

dLP

dc

deqpd

xk

04/22/23 McMaster 29

Rationale for Integration

• Active set approach: LP + EQP (Fletcher) shares EQP solution with Interior Algor.• preconditioning• Final active set identification• Warm starts• Share interfaces, stop tests, testing, object c.• Both trust region methods• New algorithms…

04/22/23 McMaster 30

Remarks on:New Software—Interior Methods

– Change of barrier parametero Guiding principles (increase/decrease)o Scale invariance, initial pointo Global convergence

– When to attempt superlinear convergence– Seemingly superior to active set SQP codes

o Only choice for very large (reduced space) Versatile

o 1st derivs only; do/not factor Hessiano Iterative vs direct solverso Feasible/infeasible modes

LOQO, KNITRO

04/22/23 McMaster 31

Tests Sets: CUTE (850 problems) COPS

           

On many practical applications…

04/22/23 McMaster 33

Final Remarks

• Extension of core NLP algorithms/sofware instead of special-purpose methods

• Investigation of limitations: degeneracies, multilevels

• Robustness (fundamental algorithmic)• More iterative options• How far can NLP methods scale up?

04/22/23 McMaster 34

L1 Merit Function

L1 linear program and penalty parameter selection

• Try to find minimum s.t. residuals = zero• W= constraints with zero residuals• Can achieve robustness and efficiency

},0max{||)( ii ghxf

dtdgg

rqdhhtstrqdf

T

T

T

. .)(min

04/22/23 McMaster 35

Interior Method- Direct

• Solve primal-dual system• Retain trust region• Revert to Interior Iterative

Step is too longInertia is not correctStep is rejected by merit functionAdaptive barrier parameter (Morales, Orban)

0AAW T

dn

Slack bound

04/22/23 McMaster 36

Interior Method (Iterative, CG)

Projected CG:

0. 0)( s.t.

)ln()( min

sg(x)xh

sxf i

sdsdSd.r Adsg(x)

rAdxhWddd

ssx

g

h

TT

|||| ||||

)( s.t.min

1

Infeasibleslacks

= barrier function

Primal method, no multipliers