Easy Advanced Global
Optimization (EAGO)
An Open-Source Platform for Robust
and Global Optimization in Julia
Matthew Wilhelm, PhD StudentMatthew Stuber, Assistant Professor ([email protected])
Process Systems and Operations Research LaboratoryDepartment of Chemical and Biomolecular Engineering andUTC Institute for Advanced Systems EngineeringUniversity of Connecticut
AIChE Annual Meeting 2017, Minneapolis, MN, October 31st, 20171
Motivation
Advanced optimization formulations emerge naturally from common system design problems
Implementations for recent theoretical advances aren’t readily accessible.• No publicly available library for differentiable McCormick relaxations2
• No publicly available library for implicit McCormick relaxations3,4
• Prior implementations have an high barrier to usage.
Worst-case constraint
satisfaction1
Semi-infinite Program4
[1] Worst-case design of subsea production facilities using semi-infinite programming. Stuber, M.D. et al. (2014) AIChE Journal, 60, 2513-2524[2] Differentiable McCormick relaxations. Khan, K. et al. (2017) Journal Global Optimization, 67(4), 687-729[3] Convex and concave relaxations of implicit functions. Stuber, M.D. et al. (2015) Optimization Methods and Software, 30, 424-460[4] Semi-Infinite Optimization with Implicit Functions. Stuber, M.D., Barton, P.I. (2015) Industrial & Engineering Chemistry Research, 54, 307-317
Logical Statement4Design Specification
, ,( ) 0
( , )
, :
,
g
P Up
x p
x 0h p
u
u
u
2
*
s.t.
min ( )
( ) 0,
(
, ,
, ) ,,
X
f f
g P
P
x
x p p
h p px 0
x
u
u
Progression of the Talk
Julia Capabilities
Developing Robust Optimization for Julia
Illustrative Examples
Future Directions
3
Why Julia?Overview
Ease of Use• Simplistic Syntax• Package Management• Cloud Computing• Notebook Support
Performance• Speed of C/Fortran• Parallelism• JIT Optimization
[4] DifferentialEquations.jl – A Performant and Feature-Rich Ecosystem for Solving Differential Equations in Julia. Rackauckas, C. & Nie, Q. (2017). Journal of Open Research Software, 5(1), p.15[5] Julia: A Fresh Approach to Numerical Computing. Jeff Bezanson, Alan Edelman, Stefan Karpinski and Viral B. Shah (2017) SIAM Review, 59: 65–98.
Run-time benchmark comparison5Example Julia Code of Solving ODE4
4
Why Julia?Abstract Syntax Tree
• Syntactic macros5
– Work on Abstract Syntax Tree
– Preserve lexical structure reliably
– Increased capability relative to text substitution macros in C++
• Multiple Dispatch Capable
• AST representation allows for graph level optimization and modification as alternative to operator overloading
• Allows for “overdubbing” instead of overloading to generate context unaware code with context dependent behavior
Code Generating Smooth McCormick Objects
Compiler Interaction via Macros
[5] Julia: A Fresh Approach to Numerical Computing. Jeff Bezanson, Alan Edelman, Stefan Karpinski and Viral B. Shah (2017) SIAM Review, 59: 65–98. 5
Why Julia?Design Objectives
Create a toolbox of modules that end-users can modify
Provide pre-built routines for that can be used without advanced subject area knowledge
Objective: Open-Source Objective: Intuitive User InterfaceCurrent Capabilities:o Native use of LAPACK, SuiteSparseo JuMP/JuMPeR6 (Julia AML)
o Low syntax, User Defined Functionso Automatic Derivative Calculationso Callback Support
o ValidatedNumerics.jl (Interval Library)o ForwardDiff.jl8 (Automatic Differentiation)
Future Capabilities:o Future Compatibility for Overdubbing
Approaches to Source-Code Transformation
Objective: Easily Accessible Routines
Provide a graphical interface and interface to AMPL
[6] JuMP: A Modeling Language for Mathematical Optimization Iain Dunning and Joey Huchette and Miles Lubin (2017) SIAM Review, 59: 295–320.[7] ValidatedNumerics.jl [Computer Software]. Retrieved from https://github.com/JuliaIntervals/ValidatedNumerics.jl.[8] Forward-Mode Automatic Differentiation in Julia. (2016) Revels, J., Lubin, M., Paramarkou, T. arXiv:1607.07892
6
Progression of the Talk
Julia CapabilitiesDeveloping Robust Optimization for Julia
Illustrative ExamplesFuture Directions
➢ Nonconvex Semi-Infinite Programming
➢ Branch-and-Bound Library➢ Smooth McCormick
Relaxation Library➢ Implicit Function Bounding
Routine➢ Constraint Propagation via
DAG➢ Global Solver
➢ Directed Graph Toolbox➢ Parametric Interval Methods➢ Flow-sheeting Interface➢ Solution Visualization➢ Data Import/Export/Storage➢ AML Integration
Build global solver*
Analytic Tools
User Interface
* Support for validated calculations* Special routines for imbedded implicit functions
7
SIP Solvers
[9] Global optimization of semi-infinite programs via restriction of the right-hand side. Mitsos, A. (2011). Optimization 60:10-11, 1291-1308[4] Semi-Infinite Optimization with Implicit Functions. Stuber, M.D., Barton, P.I. (2015) Industrial & Engineering Chemistry Research, 54, 307-317[10] Evaluation of process systems operating envelopes. Stuber, M.D., (2013) Ph.D Thesis.
*min ( )
( ) 0, (ˆs.t. , , , )
,ˆ( , ) , ( ),
X
f f
g Y P
Y P
x
y x p y p
h p yy x p0
x
o EAGOSemiInfinite implements implicit and explicit meta-algorithms
Explicit SIP
*min ( )
(s.t. ( , ), ) 0,,
X
f
g P
fx
y x p x p p
x
Implicit SIP
o Solved generally via restriction of right-hand side method9,4
• Discretization of uncertainty set for upper/lower bounds.• Finite convergence (if Slater point arbitrarily near minimizer).
o Problem size and complexity dramatically increased with equality constraints4,9,10
Flowchart for restriction of RHS method4
Lower Bounding Problem (explicit) for restriction of RHS method4
min (
ˆ( , , ) 0 ( , )
ˆ( , , ) (
)
s.t.
, )
LBD
LBD
SIP
X
LBD
LBD
LBD
f
g Y P
Y P
f
x
y x p y p
x y p
x
h y p 08
Branch & Bound Library
Fathom
Nodes
o Configuration• BNB type (option storage)• BNBObject type (node storage)
o Fully customizableo Natively for common schemes
• Best-first, depth-first, breadth-first search• Absolute/relative midpoint bisection
Check
Termination
Select
Node
Initialize
Solve Lower
Bounding
Problem (LBP)
Check
Convergence
Solve Upper
Bounding
Problem (UBP)
Bisect
Node
Store
NodesLBD
Feasible?
End
Solution Routine
Install Packages
Initialize B&B Objects
Define and Set UBP/LBP
Solve ProblemYesNo
True
False
9
McCormick RelaxationsImplementation
Smooth McCormick Object11
Gradients of Relaxations:
• Absent from SMC to allow for flexibility
• Can be readily calculated via automatic differentiation
• Existing packages (JuMP6, ForwardDiff8) outperform
naïve implementation
Expressions Supported:
+, -, /, *, ^, exp, exp2, exp10, log, log2, log10, sqrt, pow, sin, cos, tan, asin, acos, atan, sinh, cosh, tanh, asinh, acosh, atanh, min, max, abs, step, sign
Performance:
Comparable in speed to MC++ library11 +/-30% runtime in benchmarks
[2] Differentiable McCormick relaxations. Khan, K. et al. (2017) Journal Global Optimization, 67(4), 687-729[6] JuMP: A Modeling Language for Mathematical Optimization. Dunning, I. Huchette, J. and Lubin, M. (2017) SIAM Review, 59, 295–320.[8] Forward-Mode Automatic Differentiation in Julia. (2016) Revels, J., Lubin, M., Paramarkou, T. arXiv:1607.07892[11 MC++: A versatile library for McCormick relaxations and Taylor models. B. Chachuat. http://www3.imperial.ac.uk/people/b.chachuat/research[12] McCormick-based relaxations of algorithms. Mitsos et al. (2009) SIAM Journal on Optimization, SIAM, 2009, 20, 73-601
( )f x
x
( ) exp( /2) sin(2 )f x x x x
10
Abstract Syntax TreeDirected Graphs & Interval Constraint Propagation
Expressions Supported:
+, -, /, *, ^, exp, exp2, exp10, log, log2, log10, sqrt, pow, sin, cos, tan, asin, acos, atan, sinh, cosh, tanh, asinh, acosh, atanh, min, max, abs, step, sign
EAGODAGContractor Package
o Methods for interval constraint propagation via natural interval extensions and interval contractors1
o Supports visualization of directed acyclic graphs (DAG) via Gladfly integration
o Integrates with common-subexpression reduction routines (@cse) in Julia
[1] Worst-case design of subsea production facilities using semi-infinite programming. Stuber, M.D. et al. (2014) AIChE Journal, 60, 2513-2524[7] ValidatedNumerics.jl [Computer Software]. Retrieved from https://github.com/JuliaIntervals/ValidatedNumerics.jl.
DAG Visualization
11
Global Solver2,3
IPOPT is standard solver for LBP/UBP• Open-source• Compiled version via Package manager• Sparsity Pattern Detected Automatically• Forward-Mode AD Calculation of
Hessian/Jacobian/Gradients Used
Alternative LBD Solver: • SNOPT• Proximal-Bundle
Alternative UBP Solver: • Any JuMP NLP solver
Affine Relaxations Also Available • JuMP LP solvers
Lower Bounding Problem* Upper Bounding Problem
[2] Differentiable McCormick relaxations. Khan, K. et al. (2017) Journal Global Optimization, 67(4), 687-729[3] Convex and concave relaxations of implicit functions. Stuber, M.D. et al. (2015) Optimization Methods and Software, 30, 424-460
Preprocessing Step• Forward-Reverse Interval Constraint Propagation• (Optional): Interval Parametric Tests
Default settings: • Best-first search• Relative-width bisection• Validated interval calculations
Global Solution Scheme
*Bounds calculated via implicit-function based fixed point iteration3
using smooth McCormick relaxations2.
Solver Options
12
min ( ),
s.t ( )
,
0
(
.
)
,
UBDf f
g
z ,p
h
z p
p
z 0
z
p
Z P
s.t. ( ),
min ( )
( ) 0
( ),LBD cv
cv
f f
g
x
xPp
p p
p p
Progression of the Talk
Julia Capabilities
Developing Robust Optimization for Julia
Illustrative Examples
Future Directions
13
Case Study I Kinetic Parameter Estimation3,12,13,14
[3] Convex and concave relaxations of implicit functions. Stuber, M.D. et al. (2015) Optimization Methods and Software, 30, 424-460[12] McCormick-based relaxations of algorithms. Mitsos et al. (2009) SIAM Journal on Optimization, SIAM, 2009, 20, 73-601[13] Direct measurement of the fast, reversible addition of oxygen to cyclohexadienyl radicals in nonpolar solvents, J. W. Taylor, et al. Phys. Chem. A, 108 (2004), pp. 7193–7203.[14] Global dynamic optimization for parameter estimation in chemical kinetics A. B. Singer et al., J. Phys. Chem. A, 110 (2006), pp. 971–976.
Problem Statement• Determine the forward rate constants (k2f, k3f, k4) of oxygen addition to
cyclohexadienyl radicals from transient flash photolysis data.• Mass-balance ODEs system discretized via implicit Euler.• Minimize residual sum-of-squares for intensity.
2 3
2 3
3
3
2
2
1 2
1 2 3 5
1
3 4
1
2
1
1
1
1
2
2
2
( ) 0
0
( , ) 0
0
0
f f
f
f
k ki i i i i i i i
f f K K
ki i i i
f K
ki i i i
f K
i i i i
s
i i i i
o
o
o
t k k k k
t k k
t k
t k
t k
c
c
c
A A Y Z A D B A
B B A B
D D A D
Y Y Y Z
Z Z Y Z
c c c c c c c c
c c c c
h z p c c c c
c c c c
c c c c
2
1
2 221 21
min
. .
ni i
Pi
i i i i
A B Ds t
datap
I I
I c c c
( , , , , , , )
. . 1
i i i i i
s t i n
A B D Y Zz c c c c c
1 1 5 2 3 2, , ,, , , ,s O t nk k k K K c
Parameters
Objective Function
Uncertain VariablesEquality Constraints
State Variables
2 3 4( , , )f fk k kp
14
Case Study IIKinetic Parameter Estimation
Read Data in Julia
Call EAGO to Solve Problem
Finishes with certificate of global optimality
o Optimal point: (789.02, 423.85,12.969)
o Optimal value: 16796.04
o Solution consistent with the literature
Define objective
Define equality constraints
Setup Box Constraints for Problem
15
Case Study II Flash Separation of Benzene and Toluene14
Problem StatementWe wish to ensure a cut-fraction, α, of less than or equal to 0.7 in the flash separation of a benzene-toluene mixture subject to uncertain temperature, T, by controlling the pressure, P.
,max
. . ( , ) 0.7,
[80,110]
[90,100]
[0,1]
T p P
s t p p P
T
p P
A
SIP Formulation
Process Model2
1
( ( ) 1) ( ), ( ) , log( ( ))
( ( ) 1)
,( , , ) ,
, 1
satsati i i i
i i i
i i i
K p BK p A
K C
z ph p p
p p
[14] Robust simulation and design using semi-infinite programs with implicit functions. Stuber, M. D. & Barton, P. I. (2011)International Journal of Reliability and Safety, 5, 378-397
JuMP/Julia Code for Formulating/Solving Model
Solving the SIP, we find that no pressure setting exists which is robust to the full range of temperature variation.
Flash Separator
16
Recap
Julia Capabilities
Developing Robust Optimization for Julia
Illustrative Examples
Future Directions
17
Case Study III Worst-Case Subsea Separator Design1
1( ) 0.35,0.5G p
Control Settings (Valve Position)
1 2( , ) 0,1 0,1u u uUncertain Parameter (Inlet Gas Fraction)
Problem Statement (Original)For any inlet gas fraction (stream 1) within typical bounds, is there a control setting that will prevent the effluent gas fraction of the liquid-liquid separator (stream 7) from exceeding the specification and damaging the pump.
Problem Statement (Semi-infinite Program)
*
,max
( ( , ), , ),
P U
g U
p u
x u p u p u
( , )( , , ) xh z u p 0 z u p
Equality constraints specified by mass balances, energy balances, and unit operation relations from previous flowsheet.
Semi-infinite constraint:
7 ( , ) 0.05( ( , ), , ) Gg x x u p u p u p
State Variables
74 4 4 7 7 3 4 6 8, , , , , , , , , ,G W O G O GLSH m m m m m z
[1] Worst-case design of subsea production facilities using semi-infinite programming. Stuber, M.D. et al. (2014) AIChE Journal, 60, 2513-2524
18
Case Study III Worst-Case Subsea Separator Design1
1
2
1 2
1 2
1 2 1 1
well
GLS
v
mix
P P
P P
P Pm m u C
SG
Valve 1
5
4 5
4 54 5 2 2
4 /
LLS
v o
W
P P
P Pm m u C
Valve 2
4 5
9 3 8
9 3 8min ,
m m m
P P P
Gas Mixer
1
1 1 1
1 1 1
141.5
131.5
( )
1
O
mix G G W W O O
G W O
SGAPI
SG SG SG SG
Inlet Specification
6 8 7 7
7 5
7 7
8
5 6 7 8 5 5 8 8 7 7 5 5 6 6
1
7 7 7
7 5
7 5
(0,1, 0), (1, 0, 0), 1
exp/
, ,
( / / ) ,
2
G O
oil
G G LLS
LLS
G G G W W
o
W G G O O
oil LLS
LLS LLS LLS LLS LLS LLS L
P P
Vk
m
m m m m m m m m m
SG SG
mV V
m
V L H R R H H
2 2 1cos 1
LLS
LS LLS
LLS
HR
R
Gas-Liquid Separator
Liquid-Liquid Separator
3 4 4 4
4 2
4 4
3 2 4 3
2 3 4 2 2 3 3 4 4 2 2 4 4
1
4 4 4 4
2 2 1
4
(1, 0, 0), 1
exp/
, ,
( / / / ) , ,
2 cos 1
G O W
GLS
G G GLS
G G G W W
o
W G G W W O O
GLS GLS GLS GLS GLS GLS GLS GLS
a GLSP P P P
Vk
m
m m m m m m m m
SG SG SG
V L H R R H H R
g H
2 2&
GLS
GLS
H
R
m wellhead specification
19
Case Study III Worst-Case Subsea Separator Design1
Valve 2
• Each of the four design case in the literature were run for Subsea Separator Design
• Good agreement with published results in all cases.
5
4 5
4 54 5 2 2
4 /
LLS
v o
W
P P
P Pm m u C
20
Accessing EAGO & Future Plans
An intuitive graphical user interface - EAGOFlowsheet
• Equation-Oriented Flow-sheeting• Graphical Manipulation of Directed Graphs• Template Registration of New Relaxations• AMPL Compatibility for Benchmarking
Explore and Contribute to the EAGO Package Library at
https://github.com/MatthewStuber/EAGO
21
Acknowledgements
PSOR lab at UCONN (https://psor.uconn.edu/)
• Robert Ernst
Julia Development Community • Mike Innes• David Saunders
University of Connecticut for funding this research
22
23
The End
Julia MC vs. MC++
• Using 64bit, Intel Xeon E3-1270 v5 @ 3.60 GHz, 32GB RAM• Non-validated interval calculations• Ubuntu/Linux 16.04 LTS and Julia v0.6• Profiling 10,000 evaluations with recommend settings to limit clock noise
24