design and planning tools - gridopticsgridoptics.org/fpgws16/files/mihai_slides_part_1_and_2.pdf ·...

Design and Planning Tools

Topic 1.4.18 - Computational Science for Grid Management

Mihai AnitescuArgonne National LaboratoryNovember 2016

Grid Modernization Initiative

The Energy Department’s Grid Modernization Initiative (GMI) represents a comprehensive effort to help shape the future of our nation’s grid and solve the challenges of integrating conventional and renewable sources with energy storage and smart buildings, while ensuring that the grid is resilient and secure to withstand growing cybersecurity and climate challenges.

Through the Grid Modernization Multi-Year Program Plan (MYPP), the U.S. Department of Energy (DOE) will coordinate a portfolio of activities to help set the nation on a cost-effective path to an resilient, secure, sustainable, and reliable grid that is flexible enough to provide an array of emerging services while remaining affordable to consumers.

11/22/2016

Argonne National Laboratory ---- Mathematics and Computer Science Division

2

http://energy.gov/under-secretary-science-and-energy/grid-modernization-initiative

Design and Planning Tools (from MYPP)

Design and Planning Tools Incorporate uncertainty and system dynamics into reliability planning tools to

accurately capture effects of renewable generation Coupling grid transmission, distribution, and communications models to

understand cross-domain effects Advanced production cost modeling used to understand cost-benefit tradeoff of

the mix and location of storage and solar

11/22/2016


3

MYPP supported areas

Task 4.3.3: Develop efficient linear, mixed-integer, and nonlinear mixed-integer optimization solution techniques customized for stochastic power system models, novel bounding schemes to use in branch and bound, and structure exploiting algorithms. Demonstrate the cost-benefit achieved by these techniques relative to existing ones. [FY18-20]

Task 4.3.4: Demonstrate the application of parallel and distributed computing algorithms on existing (multiprocessor, multi-core, and GPU) and emerging (many-core and co-processors) computational platforms to significantly improve the performance of applications and methods proposed in Tasks 4.3.1 – 4.3.3 at speeds an order of magnitude faster than is possible presently. [FY19-20]

Task 5.3.5: Develop and distribute advanced libraries of algorithms, solvers, uncertainty quantification, and stochastic optimization modules. [FY16-FY20]

Task 5.3.6: Develop computing frameworks that enable the coupling of advanced computation tools, data, and visualization technologies with easy workflow management. [FY16-FY20]

After proposal round, decision to make this activity a primarily frameworks definition activity; supporting optimization+uncertainty+dynamics, with solvers to be adjusted as needed, but not major significant research component.

11/22/2016


4

The need/Project vision

Drivers: – Increased renewable penetration results in variability that is best captured by stochastic

representations. – GMLC’s goal of a 33 percent decrease in cost of reserve margins, while maintaining

reliability, requires more accuracy, and thus more computation. – An example is the drive to move when possible off-line analyses online in order to

contribute to the margin reduction.

Vision: – Allowing for uncertainty in the analyses increases the problem size in proportion to the

number of scenarios; while aiming for more accuracy and online performance (where appropriate) requires high performance computing, and in particular massive parallelism.

– Create the environment where computational analyses involving optimization, uncertainty and dynamics are expressed to take full advantage of solver features (e.gdecomposition)

– Increase the accuracy and decrease the time to solution of such analyses to fulfill the objectives above.

11/22/2016


5

Project structure/team

Advanced Modeling, Integration, and Computation Frameworks (AMICF) Henry Huang (PNNL) and Wesley Jones (data, NREL)

Optimization (Mihai Anitescu ANL, Cosmin Petra LLNL) Dynamics (Slaven Peles, LLNL) Uncertainty (Jean Paul Watson, SNL; Russell Bent LANL)

Concept: Define use cases spanning the space of the tasks 4.3.3, 4.3.4, 5.3.5.,and 5.3.6

Employ these use cases to define the framework and adjust the solvers to solve large problems efficiently.

Framework/Solvers should be: – Scalable– Portable (Multicore—smart), Open– Extensible, Expressive – Fast– Compact

11/22/2016


6

Lab Capabilities

National Labs Have Excellent Computational Capabilities Numerical Linear Algebra Solvers: PETSc, Trilinos Time Stepping Solvers: SUNDIALS, PETSc, Stochastic Optimization PySP, PIPS, DSP (When solving integer programming, primarily calling a commercial solver, but for

nonlinear continuous optimization fully contained solution).

We aim to make them usable in this context and provide value to GMLC Usable on many architectures Able to scale to many processors to provide timely solutions. Open, Free.

11/22/2016


7

Dynamics Solvers: Overview of SUNDIALSSUite of Nonlinear and DIfferential/ALgebraic Equation Solvers

Developed and supported by Lawrence Livermore National Laboratory (LLNL) Freely available, released under BSD license; Over 4,500 downloads per year. Forward looking, extensible object oriented design with clean linear solver and

vector interfaces. Designed to be incorporated into existing codes with modular structure that allows users to supply their own data structures.

Scales well in simulations on over 500,000 cores. Modules and functionality:

– ODE integrators: (CVODE) variable order and step stiff BDF and non-stiff Adams, (ARKode) variable step implicit, explicit, and additive Runge-Kutta for IMEX approaches.

– Differential-algebraic equation integrator: (IDA) variable order and step stiff BDF.– CVODES and IDAS include forward and adjoint sensitivity capabilities.– KINSOL nonlinear solver: Newton-Krylov and accelerated fixed point and Picard

methods.

Used in industrial and academic applications worldwide. Examples include:– Power grid modeling (RTE France, ISU)– Electrical and heat generation within battery cells (CD-adapco)– Power transmission modeling and simulations (Griddyn, LLNL)– Empirically based simulations of neuron networks (Neuron, Yale University)

11/22/2016


8

Dynamics Solvers: PETSc-TS: Scalability results for 5-sec simulation of a 20000 node power grid:

9

0.00

2.00

4.00

6.00

8.00

1 2 4 8 16 24

Exec

utio

n tim

e (s

ec)

# Cores

1.00

4.00

16.00

64.00

1 2 4 8 16 24

Exec

utio

n tim

e (s

ec)

# Cores

Overlapping Schwarz preconditioningAdaptive time-stepping

Faster than real-time

Slower than real-time

Faster than real-time

Slower than real-time

Stable Scenario

Unstable Scenario

Optimization Solvers: Stochastic programming –computational patterns

Computational pattern

10

Optimization

Deterministic model Effects of uncertainty

OptimizationRecourse problem outcome 1

OptimizationRecourse problem outcome 2

OptimizationRecourse problem outcome N

1st stage computations

2nd stage computations

Mihai Anitescu - Optimization under uncertainty

PIPS-IPM on power grid models

11

Not used on high-end supercomputer for this app, but nice to know we can. Strong scaling == same efficiency on 100-1000 cores and new multicore architectures

4 billion decisions variables4.1 billion constraints(32,768 scenarios)

Relaxation of 24-hour unit commitmentSolution time: < 40 mins

Parallel efficiency > 90%

FLOPS peak performance 12%

How do we tap performance? – A good framework

Library designer’s view of an application Example MATLAB

– SYSTEM=SIMULINK– MATH MODEL FRAMEWORK=“f” encoding

dynamics, provides derivatives for implicit integration, encodes sparsity

– SOLVERS=ODESUITE/LINPACK/LAPACK

To achieve best performance, the model/framework block should allow algebraic access if possible.

It should provide the functionality required for performance

– Gradient– Sparsity – Annotating Decomposition Patterns– …..

Once a good model framework exists, customization to GUI is ”easy”

11/22/2016


12

SYSTEM (DATA INPUT/VIZ)

MATH MODEL FRAMEWORK

LIBRARY

HARDWARE

Framework

Desktop, Cluster, Cloud, GPU, ….

Other Frameworks Examples

In operations/research optimization, we tend to express problems directly in the MODEL, without the SYSTEM block.

– SYSTEM == None (hookups to Matlab for data/viz and solver exist)– MODEL==E.g AMPL, GAMS, AIMMS, (commercial), PyOMO, JuMP”/Julia (Open Source);

all support algebraic expression representation. – LIBRARY=CPLEX,IPOPT, etc

PLEXOS – SYSTEM = PLEXOS GUI– MODEL == AMMO (proprietary, roughly functionally equivalent to AMPL, etc), – LIBRARY=ExpressMP, MOSEK, GUROBI, …– Deane JP, et al., Soft-linking of a power systems model to an energy systems model,

Energy (2012), doi:10.1016/j.energy.2012.03.052

Many (most?) focused computational solutions architectures have separate SYSTEM and MODEL layers

11/22/2016


13

from coopr.pyomo import *

m = ConcreteModel()

m.x1 = Var()m.x2 = Var(bounds=(-1,1))m.x3 = Var(bounds=(1,2))

m.obj = Objective( sense = minimize,expr = m.x1**2 + (m.x2*m.x3)**4 +

m.x1*m.x3 + m.x2 +m.x2*sin(m.x1+m.x3) )

model = m

Pyomo Overview

Formulating optimization models natively within Python– Provides a natural syntax to describe mathematical models– Enables data import and export in commonly used formats– Supports Natively ODEs as constraints (e.g gas/electricity codispatch) – Annotates Decomposition for solvers (for usage with PySP).

Highlights:– Clean syntax– Python scripts provide

a flexible context forexploring the structureof Pyomo models

– Leverage high-quality third-party Python libraries, e.g., SciPy, NumPy, MatPlotLib

Supports ODEs and PDEs Gas PDE-Network Model

# General Declarations for Modelfrom coopr.pyomo import *from coopr.dae import *

m = ConcreteModel()

m.lp = Set(initialize=[0,1,2])m.la = Set(initialize=[2,3,4])m.l = m.lp | m.la # Union of the sets

m.A = Param(m.l, initialize=10)m.rho = Param(m.l, initialize=10)m.lda = Param(m.l, initialize=10)m.D = Param(m.l, initialize=10)

# General Declarations Continuedm.fin = Param(m.l, initialize=10)m.fout = Param(m.l, initialize=10)m.theta_rec = Param(m.l,initialize=10)m.theta_snd = Param(m.l,initialize=10)m.del_theta = Param(m.la,initialize=10)m.Lend = Param(initialize=20)

m.t = ContinuousSet(bounds=(0,10))m.x = ContinuousSet(bounds=(0,value(m.Lend)))

m.p = StateVar(m.l,m.t,m.x)m.f = StateVar(m.l,m.t,m.x)

Gas PDE-Network Model

# PDE Equation 1def _pde1(m,i,j,k):

return m.p[i,j,k].derivative(wrt=m.t) + 1/m.A[i]*m.p[i,j,k]/m.rho[i] * m.f[i,j,k].derivative(wrt=m.x) == 0

m.pde1 = Constraint(m.l,m.t,m.x,rule=_pde1)

# PDE Equation 2def _pde2(m,i,j,k):

return 1/m.A[i]*m.f[i,j,k].derivative(wrt=m.t) +m.p[i,j,k].derivative(wrt=m.x) +(8*m.lda[i])/(3.1415**2*m.D[i]**5)*(m.f[i,j,k]*abs(m.f[i,j,k]))/m.rho[i] == 0

m.pde2 = Constraint(m.l,m.t,m.x,rule=_pde2)

Gas PDE-Network Model

def _flowbound(m,i,j):return m.f[i,j,0] == m.fin[i]

m.flowbound = Constraint(m.l,m.t,rule=_flowbound)

def _fupbound(m,i,j):return m.f[i,j,value(m.Lend)] == m.fout[i]

m.fupbound = Constraint(m.l,m.t,rule=_fupbound)

def _lupbound(m,i,j):return m.p[i,j,value(m.Lend)] == m.theta_rec[i]

m.lupbound = Constraint(m.l,m.t,rule=_lupbound)

def _llowboundp(m,i,j):return m.p[i,j,0] == m.theta_snd[i]

m.llowboundp = Constraint(m.lp,m.t,rule=_llowboundp)

def _llowbounda(m,i,j):return m.p[i,j,0] == m.theta_snd[i] + m.del_theta[i]

m.llowboundp = Constraint(m.la,m.t,rule=_llowbounda)

A Julia Open Representation for the Modeling Layer

– Julia – a fresh approach to scientific and technical computing high-level, high-performance, open-source dynamic language for technical computing keeps productivity of dynamic languages without giving up speed (2x of C/C++/Fortran) “writes like Matlab runs like C”, parallel support via MPI is a breeze

Very fast to create AML – e.g JuMP for optimization

Macros instead of operator overloading (known to have poor performance) Also

18

StructJuMP modeling and solving scenario-like nonlinear optimization problems in parallel

19

Structured extreme-size nolinear optimization problems

JuMPAutoDiffForwardDiffReverseDiffSparse

parallel optimization solver(PIPS, etc.)

parallel evaluationStructJuMP

Algebraic modeling front-end Easy-to-express, compact

mathematical syntax Parallel model manipulation

and function/derivatives evaluation using MPI

Does AD with a revisited edge pushing

A novel data structure to deal with irregular memory access with multiple lookups for each nonzero in the Hessian

Julia/JuMP--almost-C performance

Yet, solver agnostic

Parallel computing back-end Parallelism is present through

the entire stack: instantiation, AD, and optimization solver

No knowledge on parallel computing is required from the user.

Runs on “Blues” @ ANL ….

20

• NLP SC-ACOPF model: 300 buses, 411 lines, 69 generators

• With new (and faster) derivative computation

#procs

Model initiation (seconds)

Structure building

(seconds)

Function & derivative

evaluation (seconds)

Total time (seconds)

1 6.42 2.09 20.56 390.342 4.70 1.59 10.19 279.154 4.21 1.58 5.96 230.168 4.10 1.49 3.5 208.48

16 4.14 1.46 1.86 192.8524 4.09 1.42 1.47 179.9648 3.96 1.31 0.72 191.75

Stochastic LP economic dispatch model for Illinois

.. and on the cloud too; (with parallel support)

21

So this is an example of extensible, expressive, compact, scalable and fast integrative framework

How about Modelica? Modelica is a popular open-source algebraic framework mostly for dynamical

simulations (sort of like simulink); car industry and power plant simulation. Factor of 10 in development time reduction reporter for Building apps (Wetter and

Haugstetter) However, in 2016 Hilding Elmqvist (Modelica lead designer), advocated move to julia

(the Modia project)… “The function part of Modelica is, however, not rich enough. There are no advanced data structures such as union types, no matching construct, no type inference, etc.” .. Need to move to a LANGUAGE.

He gives example of Julia compactness v. modelica (due to type and size inference)

11/22/2016


22

Elmqvist, Hilding, Toivo Henningsson, and Martin Otter. "Systems modeling and programming in a unified environment based on Julia." In International Symposium on Leveraging Applications of Formal Methods, pp. 198-217. Springer International Publishing, 2016.

Summary

We aim to drive the advances and demonstrate value of HPC in grid with use cases.

Such use cases drive the framework design that enables maximum performance in conjunction with advanced solvers.

Framework (with solvers) aims to be – Scalable– Portable (Multicore—smart), Open– Extensible, Expressive – Fast– Compact

May run on cloud or cluster?

There is the reality of limited time and resources. Do we have the right use cases to drive frameworks for HPC in grid?

11/22/2016


23

USE CASES

GMLC Computation – Project 1.4.18

11/22/2016


24

Objectives of use cases

Span a broad space of optimization+dynamics+uncertainty challenges in grid applications; but ideally should not be too many.

Have a focus on problems that need high performance computing and large parallel computing potential.

Can demonstrate intrinsic value of the analysis. Have solvers that can be tailored to solve them efficiently with moderate effort.

After the analysis of these requirements, we settled on 3 uses cases: Use Case 1: Security-Constrained OPF under uncertainty Use Case 2: Transient Security Assessment under uncertainty Use Case 3: Optimization under Uncertainty with Transient Security Constraints.

11/22/2016


25

Use Case 1:Security-Constrained OPF under uncertainty

Need In current practice, power is dispatched from generators based on economic

considerations and assumes everything is deterministic. Many elements, however, are unknown, like load, renewable energy outputs,

equipment failure, etc. This solution may result in inappropriate margins (possibly even too low under

some circumstances; but too high in most) when increasing renewable penetration.

There is a need to (re)frame the OPF problem so that it explicitly models the stochasticity and then develop approaches that can tractably solve those models

11/22/2016


26

Use Case 1: The Problem

Minimize objective (power generation cost)– while fulfilling the demand – satisfying the power flow physics and generation– transmission technological constraints – for a given set of prescribed contingencies (line, bus, or generation failures) – and uncertainty realizations (wind and solar power real value, equipment state).

Mathematical Expression:

11/22/2016


27

Use Case 1 SCOPF under uncertainty

Applicability Preventive control considering voltage stability and penetration of renewable

energy (extension of [4]). Computation of reactive reserve margins under renewable energy uncertainty

(extension of [5]). Effect of load voltage dependence under DER deployment on reliability and

cost/pricing;

Data Needs: power system data, cost coefficients, list of contingencies, intertie or generation/load uncertainty.

Challenges: Nonlinearity of the Equations; Derivative Computation; Large Problem Size;

11/22/2016


28

Use Case 2: Dynamic Security Assessment with Uncertainties

Need: With the penetration of variable energy resources the impacts of the resulting

uncertainties cannot be ignored in dynamic security assessment for both operation and planning applications.

Errors in generation and load forecasts can have a random impact on the grid. Therefore, the traditional deterministic-based applications need to be upgraded to

consider the forecast errors and be performed in a stochastics manner

11/22/2016


29

Use Case 2: The problem

The impact of uncertainties introduced by forecasts of energy and loads needs to be considered in dynamic security assessment (DSA) within the operation and planning study timeframes. Such uncertainties come from both transmission and distribution levels, particularly with the increasing penetration of intermittent generation and distributed energy resources (DERs).

Components: 1. Differential Equation:

2. Network Algebraic Connectivity

3. Contingency Modeling – Contingencies in the power network changes the Y matrix because of the change in

connectivity and impedance.

4. Uncertainty Modeling– Bulk generation uncertainties affect mechanical power– Load uncertainties affect the algebraic equations – Distributed generation uncertainties affect net load the system needs to support.

11/22/2016


30

−=

−−−=

sii

siieimii

s

i

dtd

DPPdt

dH

ωωδ

ωωωω

)(2

EIEY =*

Use Case 2: Dynamic Security Assessment with Uncertainties

Applicability: (1) Effect of increasing penetration of renewable generation on system stability; (2) Effect of DERs and consumer participation on bulk system operation and

planning; and (3) Optimization of DERs for multiple value streams for the power system.

Data Needs: Historical actual data, forecast data, power system model, generator parameters, and contingency list.

Challenges: computationally expensive due to a large number of DAE simulations and uncertainty scenarios. Traditional Monte Carlo simulation has scalability issues; large-scale statistical data analysis to quantify the sources of uncertainties; seamless data communication; user-friendly visualization and information presentation.

11/22/2016


31

Use Case 3: Optimization under Uncertainty with Transient Security Constraints.

Need: Renewable Energy Sources will change several features of current systems. Uncertainty in production will become an element that must be addressed Loss of traditional rotating inertia will require both its consideration in the

optimization and its inclusion in contingency analysis. Emerging need:

– Consider uncertainty, dynamics, and contingency and AC physics together in the real time

– Enable large penetrations of renewable energy without affecting grid stability or higher cost via larger reserves.

This use case contains Use Case 1 and 2. However, it has a higher development risk, so we aim to define and provide solvers for 1,2 first.

11/22/2016


32

Use Case 3: The Problem

11/22/2016


33

Minimize objective (power generation cost)– while fulfilling the demand – satisfying the power flow physics and generation w/o contingency– Satisfying the transient contingency limits– for a given set of prescribed contingencies (line, bus, or generation failures) – and uncertainty realizations (wind and solar power real value, equipment state).

Mathematical Expression:

Use Case 3: Optimization under Uncertainty with Transient Security Constraints

11/22/2016


34

Applicability Preventive rescheduling of power systems subject to stability constraints for

contingencies and uncertainty in load/generation (extension of [1]). Stability constrained interchange limits under renewable energy uncertainty

(extension of [4]). Determination of optimal reserve margins on the renewable energy uncertainty in

operations while considering transient security. Data Needs: uncertainty scenario data ; cost function h() parameters/forecasts ; system steady-state and transient constraint and evolution functions, power flow and transients : (including generator parameters for e.g swing equation) ; post-contingency limit requirement functions (post-contingency voltage and frequency limit functions). Challenges: Scalability and Accuracy of the Framework/Evaluation and Computation. Details: Efficient Expression/Computation of the Model with/for Nonlinearity (Gradients/Hessians); Fast Transient Simulation Requirements; Memory Limitations of Direct Transcription; Cost of computation of adjoint capability for transient constraints.

Questions

Are the use cases appropriate/optimal for demonstrating value of HPC for power grid applications?

Are there better use cases that combine optimization+uncertainty+dynamics?

Are there better areas of applicability for the use cases (i.e that would more clearly demonstrate value for the community/industry?)

Do these use cases have the right features or should they be modified in some way to contain other elements for example?

11/22/2016


35

design and planning tools - gridopticsgridoptics.org/fpgws16/files/mihai_slides_part_1_and_2.pdf ·...

Documents