entity risk assessment - nerc.com assurance initiative... · april 3, 2013. clarity assurance ......

C L A R I T Y ▪ A S S U R A N C E ▪ R E S U LT S

MIDWESTRELIABIL ITYORGANIZATION

Improving RELIABILITY and mitigating RISKS to the Bulk Power System

Entity Risk Assessment

NERC WorkshopReliability Assurance Initiative

Dan Skaar, President and CEOApril 3, 2013

CLARITY ▪ ASSURANCE ▪ RESULTS

Madden - Stabler

2


Hey Wait a MinuteMoment 1 (2008)

If Registered Entities proactively fix and self report violations, why fine them? …. aren’t they just doing the right thing?

Result – don’t fine Registered Entities for doing the right things…Zero Dollar Penalty, ACP/FFT, Compliance Exceptions (RAI)…?

3

“Anybody in there? Think McFly, Think”


Our Philosophy:Risk and Reliability

The hallmark of reliability in complex, real-time, interdependent systems is not that errors won’t

occur...

it’s that errors and operating anomalies won’t create an uncontrolled cascading event outside of the design

criteria - High Reliability Organizational Theory.

4


Using Highly Reliable Organization Theory to Manage RISK

1. Preoccupation with failure • Attention on close calls and near misses (“being lucky vs.

being good”); focus more on failures rather than successes

2. Reluctance to simplify interpretations• Solid “root cause” analysis practices

3. Sensitivity to operations• Situational awareness and carefully designed change

management processes

4. Commitment to resilience• Resources are continually devoted to corrective action plans

and training

5. Deference to expertise• Listen to your experts on the front lines (ex. authorities

follows expertise)

5

Five characteristics of highly reliable organizations


BTW, Small Stuff Matters

6

The small stuff matters.

Small stuff should be detected, reported, corrected, and prevented under normal management practices – its good utility practice.

If Registered Entities are proactively taking care of the small stuff before it escalates, it’s not prosecuted as a federal case. They are simply doing their job and what the public expects!


Hey Wait a MinuteMoment 2 (2009/2010)

Region had a lot of violations with PRC-005/PRC-008…• …there seemed to be wide

variations in the technical application of these standards….

• …it’s costing everyone a lot of money…let’s have a dialogue with the industry…they are the experts...they can solve problems…

Result – Engaged industry producing Application Guides

7

“Houston, we have a problem”


05

10152025303540

2007/2008 2009 2010 2011 2012

PRC Application Guide Roll Out

Increase in self reported violations resulted

from strengthening protection programs

Stabilizing trend

Note: A violation includes multiple elements

Positive Impacts from Engagement with IndustryImproving Protection Systems Maintenance and Testing

Num

ber o

f Vio

latio

ns

8

PRC‐005/008 Violation Trends

2.42.1

1.8 1.8

0.00

0.5

1

1.5

2

2.5

3

2007/2008 2009 2010 2011 2012

Decline in VSLs due to stronger programs

Aver

age

Viol

atio

n Se

verit

y Le

vel

(VSL

)


Hey Wait a MinuteMoment 3 (2010/2011)

…appeared to MRO staff that Registered Entities were only assessing compliance with Reliability Standards prior to an audit…”point in time compliance”……are Registered Entities systematically managing risk with Reliability Standards?…lets open a dialogue with industry on our concern…Result – Model Controls around Reliability Standards

9

You get what you inspect, not what you expect.


Example of Internal Control Program Scheme

ProgramDocuments (Procedures)

StandardWork Order

Supervisory Review

Management Oversight

Checklist followed and completed, exceptions noted, follow‐up notes signed

Review for completeness and accuracy, follow‐up actions closed or scheduled to be completed, signed

Periodic sampling of work orders to determine program is being completed and properly reviewed

Procedure/Process Control Control Activity Control Type

Primary Control

Secondary Control

Tertiary Control

10


Capability Maturity Model

Ad Hoc

Repeatable

Defined

Managed

Optimized

Emer

gent

Matu

re

Internal Control Design

Reliability Stan

dards R

egulation

Less More

HIGH

LOW

Risk

$$$

$

Internal Con

trols arou

nd Reliability

Stan

dards

AD‐HOC

OPTIMIZED

Relationship Between Size/Function and Internal Controls

Pareto Principle80% of inherent risk lies with 20% of Registered Entities

Large

Small


Risk Relationship with Internal Controls

12

Internal controls should be

commensurate with the level of inherent

risk.


What’s the Risk to Reliability?

Uncontrolled, cascading event outside the design criteria (Section 215 of FPA)Causes identified in blackout reports“Patterns” • Series of smaller matters• Documentation doesn’t match performance

Others?

13


Framework for Considering Risk

Inherent Risks

Control Risks

Detection Risks

Entity Specific

System Wide

Quality and rigor of internal controls to address risks to

BES reliability

Evaluated in terms of industry adopted framework or criteria

(ex. corrective action programs, elements of

operational excellence, HRO principles, etc.)

Quality and rigor of oversight, depth of procedures, etc.

• Size and interconnections• Geography/Topology• Technology• Past history, events, other

• Assessments/Studies• Past/Current Performance• Emerging Threats• Cyber (Connectivity)

Applicable Standards

“Hard”

“Soft”

Two Dimensions

Procedures, systems, training, etc.

Governance, culture, etc.

Procedures and Risk Controls

around Applicable Standards

Criteria or Principles

IDENTIFY UNDERSTAND/ADDRESS MITIGATE/PREVENT

• Governance/Training• Procedures/Checklists• Systems/”Flags”


Our Collective Job:Managing RISK

Strategic and Tactical

Identify Risk

UnderstandRisk

Address Risk

Prevent Risk

15

Successful organizations … have learned that the higher the risk, the more necessary it is to engage everyone's commitment and intelligence.

Margaret J. Wheatley


Thank you!

16

“[E]ven when fully developed, risk analyses cannot be used as a substitute for sound professional judgment of engineers, contractors, or review boards.” FERC