Easy Advanced Global

Optimization (EAGO)

An Open-Source Platform for Robust

and Global Optimization in Julia

Matthew Wilhelm, PhD StudentMatthew Stuber, Assistant Professor (stuber@uconn.edu)

Process Systems and Operations Research LaboratoryDepartment of Chemical and Biomolecular Engineering andUTC Institute for Advanced Systems EngineeringUniversity of Connecticut

AIChE Annual Meeting 2017, Minneapolis, MN, October 31st, 20171

Motivation

Advanced optimization formulations emerge naturally from common system design problems

Implementations for recent theoretical advances aren’t readily accessible.• No publicly available library for differentiable McCormick relaxations2

• No publicly available library for implicit McCormick relaxations3,4

• Prior implementations have an high barrier to usage.

Worst-case constraint

satisfaction1

Semi-infinite Program4

[1] Worst-case design of subsea production facilities using semi-infinite programming. Stuber, M.D. et al. (2014) AIChE Journal, 60, 2513-2524[2] Differentiable McCormick relaxations. Khan, K. et al. (2017) Journal Global Optimization, 67(4), 687-729[3] Convex and concave relaxations of implicit functions. Stuber, M.D. et al. (2015) Optimization Methods and Software, 30, 424-460[4] Semi-Infinite Optimization with Implicit Functions. Stuber, M.D., Barton, P.I. (2015) Industrial & Engineering Chemistry Research, 54, 307-317

Logical Statement4Design Specification

, ,( ) 0

( , )

, :

,

g

P Up

x p

x 0h p

u

u

u

2

*

s.t.

min ( )

( ) 0,

(

, ,

, ) ,,

X

f f

g P

P

x

x p p

h p px 0

x

u

u

Progression of the Talk

Julia Capabilities

Developing Robust Optimization for Julia

Illustrative Examples

Future Directions

3

Why Julia?Overview

Ease of Use• Simplistic Syntax• Package Management• Cloud Computing• Notebook Support

Performance• Speed of C/Fortran• Parallelism• JIT Optimization

[4] DifferentialEquations.jl – A Performant and Feature-Rich Ecosystem for Solving Differential Equations in Julia. Rackauckas, C. & Nie, Q. (2017). Journal of Open Research Software, 5(1), p.15[5] Julia: A Fresh Approach to Numerical Computing. Jeff Bezanson, Alan Edelman, Stefan Karpinski and Viral B. Shah (2017) SIAM Review, 59: 65–98.

Run-time benchmark comparison5Example Julia Code of Solving ODE4

4

Why Julia?Abstract Syntax Tree

• Syntactic macros5

– Work on Abstract Syntax Tree

– Preserve lexical structure reliably

– Increased capability relative to text substitution macros in C++

• Multiple Dispatch Capable

• AST representation allows for graph level optimization and modification as alternative to operator overloading

• Allows for “overdubbing” instead of overloading to generate context unaware code with context dependent behavior

Code Generating Smooth McCormick Objects

Compiler Interaction via Macros

[5] Julia: A Fresh Approach to Numerical Computing. Jeff Bezanson, Alan Edelman, Stefan Karpinski and Viral B. Shah (2017) SIAM Review, 59: 65–98. 5

Why Julia?Design Objectives

Create a toolbox of modules that end-users can modify

Provide pre-built routines for that can be used without advanced subject area knowledge

Objective: Open-Source Objective: Intuitive User InterfaceCurrent Capabilities:o Native use of LAPACK, SuiteSparseo JuMP/JuMPeR6 (Julia AML)

o Low syntax, User Defined Functionso Automatic Derivative Calculationso Callback Support

o ValidatedNumerics.jl (Interval Library)o ForwardDiff.jl8 (Automatic Differentiation)

Future Capabilities:o Future Compatibility for Overdubbing

Approaches to Source-Code Transformation

Objective: Easily Accessible Routines

Provide a graphical interface and interface to AMPL

[6] JuMP: A Modeling Language for Mathematical Optimization Iain Dunning and Joey Huchette and Miles Lubin (2017) SIAM Review, 59: 295–320.[7] ValidatedNumerics.jl [Computer Software]. Retrieved from https://github.com/JuliaIntervals/ValidatedNumerics.jl.[8] Forward-Mode Automatic Differentiation in Julia. (2016) Revels, J., Lubin, M., Paramarkou, T. arXiv:1607.07892

6

Progression of the Talk

Julia CapabilitiesDeveloping Robust Optimization for Julia

Illustrative ExamplesFuture Directions

➢ Nonconvex Semi-Infinite Programming

➢ Branch-and-Bound Library➢ Smooth McCormick

Relaxation Library➢ Implicit Function Bounding

Routine➢ Constraint Propagation via

DAG➢ Global Solver

➢ Directed Graph Toolbox➢ Parametric Interval Methods➢ Flow-sheeting Interface➢ Solution Visualization➢ Data Import/Export/Storage➢ AML Integration

Build global solver*

Analytic Tools

User Interface

* Support for validated calculations* Special routines for imbedded implicit functions

7

SIP Solvers

[9] Global optimization of semi-infinite programs via restriction of the right-hand side. Mitsos, A. (2011). Optimization 60:10-11, 1291-1308[4] Semi-Infinite Optimization with Implicit Functions. Stuber, M.D., Barton, P.I. (2015) Industrial & Engineering Chemistry Research, 54, 307-317[10] Evaluation of process systems operating envelopes. Stuber, M.D., (2013) Ph.D Thesis.

*min ( )

( ) 0, (ˆs.t. , , , )

,ˆ( , ) , ( ),

X

f f

g Y P

Y P

x

y x p y p

h p yy x p0

x

o EAGOSemiInfinite implements implicit and explicit meta-algorithms

Explicit SIP

*min ( )

(s.t. ( , ), ) 0,,

X

f

g P

fx

y x p x p p

x

Implicit SIP

o Solved generally via restriction of right-hand side method9,4

• Discretization of uncertainty set for upper/lower bounds.• Finite convergence (if Slater point arbitrarily near minimizer).

o Problem size and complexity dramatically increased with equality constraints4,9,10

Flowchart for restriction of RHS method4

Lower Bounding Problem (explicit) for restriction of RHS method4

min (

ˆ( , , ) 0 ( , )

ˆ( , , ) (

)

s.t.

, )

LBD

LBD

SIP

X

LBD

LBD

LBD

f

g Y P

Y P

f

x

y x p y p

x y p

x

h y p 08

Branch & Bound Library

Fathom

Nodes

o Configuration• BNB type (option storage)• BNBObject type (node storage)

o Fully customizableo Natively for common schemes

• Best-first, depth-first, breadth-first search• Absolute/relative midpoint bisection

Check

Termination

Select

Node

Initialize

Solve Lower

Bounding

Problem (LBP)

Check

Convergence

Solve Upper

Bounding

Problem (UBP)

Bisect

Node

Store

NodesLBD

Feasible?

End

Solution Routine

Install Packages

Initialize B&B Objects

Define and Set UBP/LBP

Solve ProblemYesNo

True

False

9

McCormick RelaxationsImplementation

Smooth McCormick Object11

Gradients of Relaxations:

• Absent from SMC to allow for flexibility

• Can be readily calculated via automatic differentiation

• Existing packages (JuMP6, ForwardDiff8) outperform

naïve implementation

Expressions Supported:

+, -, /, *, ^, exp, exp2, exp10, log, log2, log10, sqrt, pow, sin, cos, tan, asin, acos, atan, sinh, cosh, tanh, asinh, acosh, atanh, min, max, abs, step, sign

Performance:

Comparable in speed to MC++ library11 +/-30% runtime in benchmarks

[2] Differentiable McCormick relaxations. Khan, K. et al. (2017) Journal Global Optimization, 67(4), 687-729[6] JuMP: A Modeling Language for Mathematical Optimization. Dunning, I. Huchette, J. and Lubin, M. (2017) SIAM Review, 59, 295–320.[8] Forward-Mode Automatic Differentiation in Julia. (2016) Revels, J., Lubin, M., Paramarkou, T. arXiv:1607.07892[11 MC++: A versatile library for McCormick relaxations and Taylor models. B. Chachuat. http://www3.imperial.ac.uk/people/b.chachuat/research[12] McCormick-based relaxations of algorithms. Mitsos et al. (2009) SIAM Journal on Optimization, SIAM, 2009, 20, 73-601

( )f x

x

( ) exp( /2) sin(2 )f x x x x

10

Abstract Syntax TreeDirected Graphs & Interval Constraint Propagation

Expressions Supported:

+, -, /, *, ^, exp, exp2, exp10, log, log2, log10, sqrt, pow, sin, cos, tan, asin, acos, atan, sinh, cosh, tanh, asinh, acosh, atanh, min, max, abs, step, sign

EAGODAGContractor Package

o Methods for interval constraint propagation via natural interval extensions and interval contractors1

o Supports visualization of directed acyclic graphs (DAG) via Gladfly integration

o Integrates with common-subexpression reduction routines (@cse) in Julia

[1] Worst-case design of subsea production facilities using semi-infinite programming. Stuber, M.D. et al. (2014) AIChE Journal, 60, 2513-2524[7] ValidatedNumerics.jl [Computer Software]. Retrieved from https://github.com/JuliaIntervals/ValidatedNumerics.jl.

DAG Visualization

11

Global Solver2,3

IPOPT is standard solver for LBP/UBP• Open-source• Compiled version via Package manager• Sparsity Pattern Detected Automatically• Forward-Mode AD Calculation of

Hessian/Jacobian/Gradients Used

Alternative LBD Solver: • SNOPT• Proximal-Bundle

Alternative UBP Solver: • Any JuMP NLP solver

Affine Relaxations Also Available • JuMP LP solvers

Lower Bounding Problem* Upper Bounding Problem

[2] Differentiable McCormick relaxations. Khan, K. et al. (2017) Journal Global Optimization, 67(4), 687-729[3] Convex and concave relaxations of implicit functions. Stuber, M.D. et al. (2015) Optimization Methods and Software, 30, 424-460

Preprocessing Step• Forward-Reverse Interval Constraint Propagation• (Optional): Interval Parametric Tests

Default settings: • Best-first search• Relative-width bisection• Validated interval calculations

Global Solution Scheme

*Bounds calculated via implicit-function based fixed point iteration3

using smooth McCormick relaxations2.

Solver Options

12

min ( ),

s.t ( )

,

0

(

.

)

,

UBDf f

g

z ,p

h

z p

p

z 0

z

p

Z P

s.t. ( ),

min ( )

( ) 0

( ),LBD cv

cv

f f

g

x

xPp

p p

p p

Progression of the Talk

Julia Capabilities

Developing Robust Optimization for Julia

Illustrative Examples

Future Directions

13

Case Study I Kinetic Parameter Estimation3,12,13,14

[3] Convex and concave relaxations of implicit functions. Stuber, M.D. et al. (2015) Optimization Methods and Software, 30, 424-460[12] McCormick-based relaxations of algorithms. Mitsos et al. (2009) SIAM Journal on Optimization, SIAM, 2009, 20, 73-601[13] Direct measurement of the fast, reversible addition of oxygen to cyclohexadienyl radicals in nonpolar solvents, J. W. Taylor, et al. Phys. Chem. A, 108 (2004), pp. 7193–7203.[14] Global dynamic optimization for parameter estimation in chemical kinetics A. B. Singer et al., J. Phys. Chem. A, 110 (2006), pp. 971–976.

Problem Statement• Determine the forward rate constants (k2f, k3f, k4) of oxygen addition to

cyclohexadienyl radicals from transient flash photolysis data.• Mass-balance ODEs system discretized via implicit Euler.• Minimize residual sum-of-squares for intensity.

2 3

2 3

3

3

2

2

1 2

1 2 3 5

1

3 4

1

2

1

1

1

1

2

2

2

( ) 0

0

( , ) 0

0

0

f f

f

f

k ki i i i i i i i

f f K K

ki i i i

f K

ki i i i

f K

i i i i

s

i i i i

o

o

o

t k k k k

t k k

t k

t k

t k

c

c

c

A A Y Z A D B A

B B A B

D D A D

Y Y Y Z

Z Z Y Z

c c c c c c c c

c c c c

h z p c c c c

c c c c

c c c c

2

1

2 221 21

min

. .

ni i

Pi

i i i i

A B Ds t

datap

I I

I c c c

( , , , , , , )

. . 1

i i i i i

s t i n

A B D Y Zz c c c c c

1 1 5 2 3 2, , ,, , , ,s O t nk k k K K c

Parameters

Objective Function

Uncertain VariablesEquality Constraints

State Variables

2 3 4( , , )f fk k kp

14

Case Study IIKinetic Parameter Estimation

Read Data in Julia

Call EAGO to Solve Problem

Finishes with certificate of global optimality

o Optimal point: (789.02, 423.85,12.969)

o Optimal value: 16796.04

o Solution consistent with the literature

Define objective

Define equality constraints

Setup Box Constraints for Problem

15

Case Study II Flash Separation of Benzene and Toluene14

Problem StatementWe wish to ensure a cut-fraction, α, of less than or equal to 0.7 in the flash separation of a benzene-toluene mixture subject to uncertain temperature, T, by controlling the pressure, P.

,max

. . ( , ) 0.7,

[80,110]

[90,100]

[0,1]

T p P

s t p p P

T

p P

A

SIP Formulation

Process Model2

1

( ( ) 1) ( ), ( ) , log( ( ))

( ( ) 1)

,( , , ) ,

, 1

satsati i i i

i i i

i i i

K p BK p A

K C

z ph p p

p p

[14] Robust simulation and design using semi-infinite programs with implicit functions. Stuber, M. D. & Barton, P. I. (2011)International Journal of Reliability and Safety, 5, 378-397

JuMP/Julia Code for Formulating/Solving Model

Solving the SIP, we find that no pressure setting exists which is robust to the full range of temperature variation.

Flash Separator

16

Recap

Julia Capabilities

Developing Robust Optimization for Julia

Illustrative Examples

Future Directions

17

Case Study III Worst-Case Subsea Separator Design1

1( ) 0.35,0.5G p

Control Settings (Valve Position)

1 2( , ) 0,1 0,1u u uUncertain Parameter (Inlet Gas Fraction)

Problem Statement (Original)For any inlet gas fraction (stream 1) within typical bounds, is there a control setting that will prevent the effluent gas fraction of the liquid-liquid separator (stream 7) from exceeding the specification and damaging the pump.

Problem Statement (Semi-infinite Program)

*

,max

( ( , ), , ),

P U

g U

p u

x u p u p u

( , )( , , ) xh z u p 0 z u p

Equality constraints specified by mass balances, energy balances, and unit operation relations from previous flowsheet.

Semi-infinite constraint:

7 ( , ) 0.05( ( , ), , ) Gg x x u p u p u p

State Variables

74 4 4 7 7 3 4 6 8, , , , , , , , , ,G W O G O GLSH m m m m m z

[1] Worst-case design of subsea production facilities using semi-infinite programming. Stuber, M.D. et al. (2014) AIChE Journal, 60, 2513-2524

18

Case Study III Worst-Case Subsea Separator Design1

1

2

1 2

1 2

1 2 1 1

well

GLS

v

mix

P P

P P

P Pm m u C

SG

Valve 1

5

4 5

4 54 5 2 2

4 /

LLS

v o

W

P P

P Pm m u C

Valve 2

4 5

9 3 8

9 3 8min ,

m m m

P P P

Gas Mixer

1

1 1 1

1 1 1

141.5

131.5

( )

1

O

mix G G W W O O

G W O

SGAPI

SG SG SG SG

Inlet Specification

6 8 7 7

7 5

7 7

8

5 6 7 8 5 5 8 8 7 7 5 5 6 6

1

7 7 7

7 5

7 5

(0,1, 0), (1, 0, 0), 1

exp/

, ,

( / / ) ,

2

G O

oil

G G LLS

LLS

G G G W W

o

W G G O O

oil LLS

LLS LLS LLS LLS LLS LLS L

P P

Vk

m

m m m m m m m m m

SG SG

mV V

m

V L H R R H H

2 2 1cos 1

LLS

LS LLS

LLS

HR

R

Gas-Liquid Separator

Liquid-Liquid Separator

3 4 4 4

4 2

4 4

3 2 4 3

2 3 4 2 2 3 3 4 4 2 2 4 4

1

4 4 4 4

2 2 1

4

(1, 0, 0), 1

exp/

, ,

( / / / ) , ,

2 cos 1

G O W

GLS

G G GLS

G G G W W

o

W G G W W O O

GLS GLS GLS GLS GLS GLS GLS GLS

a GLSP P P P

Vk

m

m m m m m m m m

SG SG SG

V L H R R H H R

g H

2 2&

GLS

GLS

H

R

m wellhead specification

19

Case Study III Worst-Case Subsea Separator Design1

Valve 2

• Each of the four design case in the literature were run for Subsea Separator Design

• Good agreement with published results in all cases.

5

4 5

4 54 5 2 2

4 /

LLS

v o

W

P P

P Pm m u C

20

Accessing EAGO & Future Plans

An intuitive graphical user interface - EAGOFlowsheet

• Equation-Oriented Flow-sheeting• Graphical Manipulation of Directed Graphs• Template Registration of New Relaxations• AMPL Compatibility for Benchmarking

Explore and Contribute to the EAGO Package Library at

https://github.com/MatthewStuber/EAGO

21

Acknowledgements

PSOR lab at UCONN (https://psor.uconn.edu/)

• Robert Ernst

Julia Development Community • Mike Innes• David Saunders

University of Connecticut for funding this research

22

23

The End

Julia MC vs. MC++

• Using 64bit, Intel Xeon E3-1270 v5 @ 3.60 GHz, 32GB RAM• Non-validated interval calculations• Ubuntu/Linux 16.04 LTS and Julia v0.6• Profiling 10,000 evaluations with recommend settings to limit clock noise

24