faculty: dave bakken, carl hauser, anjan bose students: h. gjermundrød, i. dionysiou, r. johnston...

Faculty: Dave Bakken, Carl Hauser, Anjan BoseStudents: H. Gjermundrød, I. Dionysiou, R. Johnston

P. Jiang, S. Sheshadi, K. Swenson

School of Electrical Engineering and Computer ScienceWashington State UniversityPullman, Washington USAhttp://gridstat.eecs.wsu.edu

November, 2003

GridStatMiddleware for More Extensible and

Resilient Status Disseminationfor the Electric Power Grid

Copyright © 2003 Washington State University Dave Bakken GridStat–2

Overview of Presentation

• Background and Motivation

• GridStat Architecture

• Gridstat Implementation and Deployment Issues


Power Grid Today

Generation

Transmission Substation

SubtransmissionSubstations

DistributionSubstations

Residential Commercial Industrial

Distribution

Customers(Create load)

Transmission

Figure credit: NSTAC

Overview• 3 fundamental roles

• Historically one vert-integr. utility

• IT/control based on this fixed hierarchy

Hierarchy• Substation

• Control Area/utility

• Grid


Protection and Control Today are Local• Remedial Action Schemes (RAS): hardwired remote link to trigger a

protective relay

• Otherwise almost exclusively local monitoring (status) & local control– Power dynamics are grid wide, and anomalies can affect a wide geographic area

UTILITY A

Generation

Customer Customer

Distribution

UTILITY B

Generation

Customer

Distribution Distribution

UTILITY C

Generation

Customer


The Changing Landscape

• Higher demand for power transmission – miles x megawatts– More power and longer distances with little new transmission

capacity• Installed transmission capacity constrained by minimum of

– Thermal limit– Stability limit

• More participants whose actions affect grid stability• Technology in recent years is adding

– Many more devices “intelligent” devices– Much more heterogeneity

• Lack of central authority• Terrorist concerns

– Labor disputes, Environmental Liberation Front, everybody is an insider (or can be with a little effort), …


Power Companies Are More Interrelated

• Traditional generation + transmission + distribution is no longer inside a single utility– Utilities can be affected by

many things they cannot sense/detect/measure with today’s communication infrastructure

– Interactions between power dynamics and grid communicaiton dynamics completely unknown

UTILITY A

Generation

Customer Customer

Distribution

UTILITY B

Generation

Customer

Distribution

Distribution

UTILITY C

Generation

Customer


ISO and Grid Security• Independent System Operator (ISO): layer above the control

area layer added in the last few years– A small number of ISOs for bigger grids – Real-time balancing of supply and demand

• ISO is responsible for grid security– Means no actions being considered, or any probable

contingency, can lead to a blackout or brownout– Roughly translates to what computer scientists would consider

stability and reliability• Grid security is an on-line, real-time activity

– ISO monitors status from all control areas– Receives status info from control areas and substations in its

jurisdiction• ISO’s functionality previously performed by the vertically-

integrated utilities– Now too much power flowing across and around them


Status Information & the Power Grid• Changing requirements

– More general topology and connectivity including multicast– Existing hardwired, hierarchical structure does not suffice

• Status items may be needed at multiple locations– New services require more quantity, timeliness, …

• Improved real-time controls to push stability limits nearer thermal limits• Situation awareness: phone calls not adequate!• 4-second SCADA cycle moving to ½ to 4 times per 60 Hz cycle

• Opportunities for new kinds of automatic real-time control– Local control of devices based on remotely sensed status– Closed loop controls with relatively long delays– Stochastic control


GridStat in a Nutshell

• SCADA is a distributed computing problem– Convey status data in a reliable, timely and secure manner (QoS)– Above the network layer

• Implementing these QoS properties requires considerable sophistication• Exploit application-level semantics and QoS requirements• Want a re-usable architecture

– Compare to Web:TCP as SDM:network layer

• GridStat: status dissemination middleware tailored for the power grid– Common service platform for disseminating power grid status

information within and between power utilities, marketers, etc.– Timely, robust, and secure delivery to multiple participants– Collaborative project with CS and EE at WSU, others– Also applicable to status dissemination needs of other

infrastructures: transportation, water, gas, …


GridStat is Publish-Subscribe Middleware• Publish-subscribe architecture

– Publish: periodically announce status values– Subscribe: periodically receive status values– Simple, CORBA-compliant APIs for both publishers and

subscribers, management/control infrastructure, etc.– Subscribers have transparent cache of latest status value– Network of internal servers managed for QoS– Optimized for semantics of status items

• Not just arbitrary event delivery like generic publish-subscribe

Publisher

Subscriber #1

Subscriber #2

Subscriber #N

…


What is Middleware?• Middleware == A layer of software above the operating

system but below the application program that provides common programming abstractions for distributed systems

• Middleware exists to help manage the complexity and heterogeneity inherent in distributed systems

• Middleware provides higher-level building blocks for programmers than the OS provides– Makes code more portable– Makes programmers more productive– Final product is of higher quality– Analogy — MW:sockets ≈ HOL:assembler


MWRouter

MWRouter

MWRouter

GridStat Middleware (MW) in Context

Wide-Area Network

Middle-ware

Distributed Application

OS Comm. CPU Storage

Host 1

Oper. System API

Middleware API

Client

Middle-ware


OS Storage CPU Comm.

Host 2

Oper. System API

Middleware API

Client

MW QoSManagement

controlcontrol


Fundamental GridStat Research Issues

• Status dissemination middleware is a new specialization of publish-subscribe MW– Recognize specialized requirements of status dissemination

– Take advantage of status semantics in order to meet those requirements

– What are the APIs? What promises are made by the middleware to the application regarding functionality and performance?

– What architecture can deliver on these promises?

– How can we validate the correctness, timeliness, quality, etc, of a concrete embodiment of the architecture (the framework)?

– What are the trust issues between grid participants. What policies are required? How can they be implemented?

– How can the architecture be made economically scalable and manageable

• Goal of GridStat resarch is to begin answering some of these questions and embody those answers in the GridStat middleware framework



• Preliminary Information

• GridStat Architecture

• Gridstat Implementation and Deployment Issues


Publication and Subscription• Status variable – located at a publishers

– Periodic sequence of time-stamped values or– Sporadic sequence of time-stamped alerts– Types supported (initial, illustrative set):

• Basic values: scalars of type bool or int or float• Derived values: moving average, change rate, moving average of change

rate, max or min over an interval

– Derived values are first-class: subscribe to them just as to a basic value

• Subscription – requested by a subscriber– A promise (by the MW) to deliver values from a particular

status variable at a given rate within the requested delay (timeliness)

– Subscriber’s rate must not exceed publisher’s rate– Delay is constrained by the network

3.75time3

3.79time2

3.76time1

27.93time3

27.56time2

24.33time1

alerttime1


Basic GridStat Functionality

Publishers Subscribers

Grid AreaController

GridStat Management

Area Controller

Load Following

…

Generator

ISO

…

Wide Area Computer Network

ControlQoS Requirements

Department ofHomeland Security

QoS Requirements


Overview of GridStat’s Architecture

QoS Requirements QoS Requirements

Pub1

PubN

Sub1

SubN

… …

QoS Broker

…

QoS Broker

QoS Broker

R

R R

R R

RR

R… …

Control Control

GridStat delivers status events from publishers to subcribers


Detailed Architecture

QoSbroker 1

QoSbroker 2

QoSbroker 3

leaf QoSbroker 4

leaf QoSbroker 5

leaf QoSbroker 6

R

N

O

P

QA

B

D

C

E

F G

H

I

J

K

L

M

Publisher 1

Publisher 3

BorderStatusRouter

(Edge)StatusRouter

StatusRouter

Key:

S

Publisher 2


Route Allocation to Subscriber1QoS

broker 1

QoSbroker 2

QoSbroker 3

leaf QoSbroker 4

leaf QoSbroker 5

leaf QoSbroker 6

R

N

O

P

QA

B

D

C

E

F G

H

I

J

K

L

M

Subscriber 1

Publisher 1

BorderStatusRouter

(Edge)StatusRouter

StatusRouter

Key:

S

Publisher 3Publisher 2


Route Allocation to Subscriber2QoS

broker 1

QoSbroker 2

QoSbroker 3

leaf QoSbroker 4

leaf QoSbroker 5

leaf QoSbroker 6

R

N

O

P

QA

B

D

C

E

F G

H

I

J

K

L

M

Subscriber 1

Publisher 1

BorderStatusRouter

(Edge)StatusRouter

StatusRouter

Key:

S

Subscriber 2

Note: Sub2 may have a different rate or latency than Sub1

Publisher 3Publisher 2


GridStat & UCA v2 & SCADA Co-Existence

UCAv2Substation

UCAv2Substation

Utility #1ControlCenter

ISO

Utility #2ControlCenter

ISOOther CriticalInfrastructures

HomelandDefense

SCADASubstation

SCADASubstation

KeyGridStat Status Data(phone calls today)

GridStat QoS Control(later status aggregation)

Note: UCA v2 hasno wide-area

network mgmt,though it discussesexploiting “futurecommuncations

services”


Programming Model: GridStat Subscriber Caches

Subscriber1

Subscriber2

Status1

…

…

cache1

cache2

Status2

Status2

Status3

Status Router

Status Router

Status1

Status2

Status3

…

…

…


Programming Model: Condensation Functions

• If desired derivation is not built in, condensation functions allow applications to define new derived status variables– Sometimes subscribers just read a large set of status items

once to calculate a derived variable

– Supported by allowing user-defined condensation functions to be loaded in status routers

Status1

StatusN

StatusJ

StatusRouter

… Condense



• Preliminary Information• GridStat Architecture• Gridstat Implementation and Deployment Issues


Pragmatic Deployment Feasibility Notes

• GridStat not completely deployable using best-effort internet technology– But much of the traffic might be best effort, especially with

bandwidth reservation

• Most likely deployment path: grid-wide intranet– Washington Post: “In a book-length Electricity Infrastructure Security

Assessment, the industry concluded on Jan. 7 that "it may not be possible to provide sufficient security when using the Internet for power system control." Power companies, it said, will probably have to build a parallel private network for themselves.” (emphasis mine)

• Not only would security be lacking, but predictable timeliness and resilience too!!!!! The Internet is arguably as complex as the grid! GridStat Team Opinion

– Some portions likely co-located with tel-cos and national ISPs’ facilities; others built using private facilities of electric utilities

– In addition to short-term application, GridStat should be viewed as a platform for exploring what services to provide in status dissemination middleware


GridStat Capabilities Today

• Static routing of status variables to meet subscriber’s timeliness and redundancy requirements

• Recovery from data link and management link failure• Hierarchical QoS brokers• Graphic visualization of status items (strip charts etc) and of

the internals of leaf QoS brokers (queues, etc)

• Note: GridStat could deliver remote control commands in addition to status data– Just another kind of data to deliver….


GridStat Prototype

• Finished: 2nd-Generation Distributed Prototype with

– Hierarchy of QoS Managers performing the allocations

– Publisher delivery rate & redundancy QoS requirements satisfied

– Optional exception callback to subscriber of QoS violated


Future GridStat Capabilities (Funding Pending)

• Fault Tolerance at many more levels• Broader QoS Routing with runtime feedback• Trust management system to allow secure runtime

subscriptions• Pre-allocated subscription “packages” for rapid deployment

in contingencies• Validation framework (SW quality, QoS delivery)• Hardware support• Modeling and control theory for communication dynamic

and power dynamics interacting


Ongoing & Future Research Issues• Investigating a range of optimizations

– Periodic status items only delivered if enough change• percentage

• fixed delta

– Throttle back lower priority status flows when overload, attack, accidents, etc. using subscription info (max timeliness, min redundancy)

– Subscription aggregation of different kinds of flows and sub-flows

• Resilience– Subscriber cache extrapolation

– Adaptive path management

• Push the data path into hardware or embedded processors– 10% of code perhaps

– Status routers, HW registers for publishers and subscribers


Collaborators, Funding, and Colleagues• Faculty: David E. Bakken, Carl Hauser, Anjan Bose,

• Students: Ioanna Dionysiou, Kjell “Harald” Gjermundrød, Thomas Evje, Ryan Johnston, Supreeth Sheshadri, Ping Jiang

• Funding: – US Dept. of Commerce, National Institute of Standards and Technology

(NIST), Critical Infrastructure Protection Program, Grant #60NANB1D0116 (Dr. Tim Grance, PM)

– The National Science Foundation, Grant CCR-0326006 (Dr. Helen Gill, PM)– Pending: DHS, soon NSF ITR (Feb 03)

• Collaborators: – Prof. Kevin Tomsovic, WSU, realtime grid control with varying feedback loops

and varying time horizons– Prof. Sandip Roy, WSU, control theory with stochastic delays– Prof. Deborah Frincke, U of Idaho, security and trust– Other WSU professors with interests in temporal queries, hardware

implementation, graph theory, software engineering– CMU/CERT: Easel Simulation System and GridStat; IT modeling for power

grids, …


GridStat & Avista

• Now working with Avista Utilities to experiment with distribution status data dissemination– Utility for WSU’s area, with presence in 5 western states

• Technology demonstration deployment underway• Avista has donated $2.4M in dark fiber around

– Spokane area

– Pullman (WSU)

– Moscow, ID (U. Idaho; 8 miles from WSU)

to support GridStat and similar research and distributed evaluation at WSU, U. Idaho

– Funds for “access points” and Avista engineering labor also provided


Related Work

• Computer Science (networking, distributed computing):– PASS (BBN/Gatech ICDCS ’99 Zinky/O’Brien/Bakken/…)

– Sienna (U. Colorado): content-based publish-subscribe

– InfoPipes (GaTech): fresh delivery of status info

– SpinGlass/Astrolabe (Cornell): scaleable multicast

• Electrical Engineering (power)– A few research papers pointing out the wide-area

communication deficiencies– UCA version 2: nice “wrapping” of substation devices,

but no QoS management across WANs.


Conclusions• Existing power grid SCADA/DCS infrastructure is

not adequate– Deregulation and restructuring

– Efficient use of transmission resources

• GridStat: status dissemination middleware tailored for the power grid– Publish-subscribe architecture with simple, CORBA-compliant

APIs for both publishers and subscribers

– Subscribers have transparent cache of latest status value

– Network of internal servers managed for QoS• Timeliness

• Redundancy

• Security


Questions?


Background Slides

• Power Grid 101• Washington Post quotations on power grid cyber attacks• Middleware 101• More GridStat Details


Context“The ultimate challenge in creating the power delivery system of the 21st

century is in the development of a communications infrastructure that allows for universal connectivity.”

“In order to create this new power delivery system, what is needed is a national electricity-communications superhighway that links generation, transmission, substations, consumers, and distribution and delivery controllers.”

Clark Gellings, EPRI Vice President for Power Delivery and Markets, in “Smart Power Delivery ― A vision for the Future,” EPRI Journal Online, Electric Power Research Institute, June 9, 2003

http://www.epri.com/journal/details.asp?doctype=features&id=618.

GridStat is researching, implementing, and evaluating this “national electricity-communications superhighway”, and not just for the power grid but for other critical infrastructures as well.


Power Grid Today• Three fundamental roles in the power grid:

1. Generation 2. Transmission3. Distribution

• Traditionally owned by a single, vertically-integrated company– Based largely on geography– Hierarchical infrastructure– Communications network is

• Hardwired• Dedicated• Slow

• Everything is hard-coded based on this fixed hierarchy– Application programs– Status information– Control decisions


Components of the Power Grid• Generator: generates power, based on requirements given it

• Substation: point of monitoring and control in the grid

– Can service many generators, and/or other functions• Distribution point to customers

• Voltage boosting

• Control functions

– Generally only services one fundamental role

– Always involved in control based on status of a lot of devices

• Control area: a set of substations

– Geographic area ranging from a county to a few states

– Services all three fundamental roles

– Roughly corresponds to one or a few utility companies (most 1:1)

– Collects status info from all substations for control decisions

• Grid: a set of control areas which are synchronously controlled

– AKA “regional reliability council” or “region”


Grids in Canada and the US


ISO and Grid Security• Independent System Operator (ISO): new layer above the

control area layer currently being added

– A small number of ISOs for bigger grids

• ISO is responsible for grid security

– Means no actions being considered, or any probable contingency, can lead to a blackout or brownout

– Roughly translates to what computer scientists would consider stability and reliability

• Grid security is an online, real-time activity

– ISO monitors status from all control areas

– Receives all status info from any control area or substation in its jurisdiction

• ISO’s functionality used to be performed by the vertically- integrated utilities

– Now too much power flowing across them or around them


Pragmatic GridStat Deployment Feasibility Notes

• Allocation algorithms & frequency of subscriptions– In practice nearly all are likely to be pre-allocated and static

– Number of new subscriptions (allocation algorithm runs) per hour small

– Could be batched for offline (weekend/night) computation unless critical

– Even brute-force solutions to NP-hard problems may be practical in many cases


Background Slides



Background Slides

Washington Post quotations on cyber attacks “Cyber-Attacks by El Qaeda Feared” (27 Jun 02, A01))

• Note: emphasis mine and [comments mine] in all cases….• “The event I fear most is a physical attack in conjunction

with a successful cyber-attack on the responders' 911 system or on the power grid,” Ronald Dick, director of the FBI's National Infrastructure Protection Center


Other Related Quotations from Post Article

• The devices are called distributed control systems, or DCS, and supervisory control and data acquisition, or SCADA, systems. ….

What is new and dangerous is that most of these devices are now being connected to the Internet -- some of them, according to classified “Red Team” intrusion exercises, in ways that their owners do not suspect.

Because the digital controls were not designed with public access in mind, they typically lack even rudimentary security, having fewer safeguards than the purchase of flowers online. Much of the technical information required to penetrate these systems is widely discussed in the public forums of the affected industries, and specialists said the security flaws are well known to potential attackers.


Post Quotations (cont.)

• Digital controls are so pervasive, he said, that terrorists might use them to cause damage on a scale that otherwise would “not be available except through a very systematic and comprehensive physical attack.” [He is Director John Tritak of the Commerce Department's Critical Infrastructure Assurance Office]

• To destroy a dam physically would require “tons of explosives,” Assistant Attorney General Michael Chertoff said a year ago. To breach it from cyberspace is not out of the question. In 1998, a 12-year-old hacker, exploring on a lark, broke into the computer system that runs Arizona's Roosevelt Dam. He did not know or care, but federal authorities said he had complete command of the SCADA system controlling the dam's massive floodgates.


Post Quotations (cont.)• Massoud Amin, a mathematician directing new security efforts in the

industry, described the North American power grid as “the most complex machine ever built.” At an April 2 conference hosted by the Commerce Department, participants said, government and industry scientists agreed that they have no idea how the grid would respond to a cyber-attack.

What they do know is that "Red Teams" of mock intruders from the Energy Department's four national laboratories have devised what one government document listed as "eight scenarios for SCADA attack on an electrical power grid" -- and all of them work. Eighteen such exercises have been conducted to date against large regional utilities, and Richard A. Clarke, Bush's cyber-security adviser, said " the intruders “have always, always succeeded.”


Background Slides



Why Middleware?• Middleware == “A layer of software above the operating

system but below the application program that provides a common programming abstraction across a distributed system”

• Middleware exists to help manage the complexity and heterogeneity inherent in distributed systems

• Middleware provides higher-level building blocks (“abstractions”) for programmers than the OS provides– Can make code much more portable– Can make them much more productive– Can make the resulting code have fewer errors– Analogy — MW:sockets ≈ HOL:assembler

• Middleware sometimes is informally called “plumbing”– Connects parts of a distributed application with “data pipes” and

passes data between them


Middleware

Middleware in Context


OS Comm. Processing Storage


Network

Host 1 Host 2

Middleware

Operating System API

OS Comm. Processing Storage

Operating System API

Middleware API Middleware API

Client Server


Middleware Benefit: Masking Heterogeneity

• Middleware’s programming building blocks mask heterogeneity– Makes programmer’s life much easier!!

• Kinds of heterogeneity masked by middleware (MW) frameworks– All MW masks heterogeneity in network technology– All MW masks heterogeneity in host CPU– Almost all MW masks heterogeneity in operating system (or family

thereof)• Notable exception: Microsoft middleware (de facto; not de jure or de fiat)

– Almost all MW masks heterogeneity in programming language• Noteable exception: Java RMI

– Some MW masks heterogeneity in vendor implementations• CORBA best here


Background Slides



Path Determination to Provide Fault Tolerance and Timeliness

• Initial QoS Specification for Subscriptions– Desired latency

– Number of redundant paths desired

• Timeliness and fault tolerance– Choose multiple, disjoint paths that meet the requested delay

constraint while respecting capacity constraints of links and routers

– Many variants of multi-constrained QoS routing are NP-hard (of course!), including most of the ones that we think are relevant for GridStat

– Choose paths using heuristics; if a (set of) paths is found the system should guarantee its performance


GridStat Mapping Capabilities (cont.)

Note: there are not GridStat nodes deployed as above; for illustration purposes only


GridStat Status Patterns• Pragmatic goal: give building blocks that non-

distributed-systems-specialists can effectively use in grid monitoring and control applications– Try to capture status semantics + some QoS info

• Initial derived value examples– Periodic: Bandwidth use downstream limited by downstream

subscriptions. Can be boolean, floating point, integer.

– Alert: Potentially catastrophic situation• Propagate to subscribers immediately

• Deliver using callback/interrupt (not just cache update)

• Note: it is very useful to distinguish between anomaly domains: is it a power grid problem or IT infrastructure problem

faculty: dave bakken, carl hauser, anjan bose students: h. gjermundrød, i. dionysiou, r. johnston...

Documents

electric power grid

power flowing

unknown slide

grid communicaiton dynamics

reliability grid security

grid stability technology

control areas

control today