Notice of Compliance Audit

Phil O’DonnellManager, Compliance, Operations and Planning

Audits and Investigations

2

Audit Frequency

3 Year CycleBalancing Authority Transmission Operator Reliability Coordinator

All other registered functionsSubject to flexibility in the future as part of NERC’s Reliability Assurance Initiative

3

Compliance Audit (on-site vs. off-site)

• On-Site– Documentation sent to WECC before audit for preliminary

review

– The audit team reviews evidence during off-site week or the first week of the audit and completes its review during the second week or on-site week

– Data Requests or DRs– Tours to observe facilities– In-person interviews for clarification

4

Compliance Audit (on-site vs. off-site)

• Off-Site– Documentation sent to WECC before audit for

preliminary review– Data Requests or DRs– Entity may be present at audit if desired– Telephone interviews for clarification

5

Compliance Audit (on-site vs. off-site)

• Primary difference is:– Location of audit conduct

• Scope is typically smaller for off site• On Site – Required for RC, BA, TOP functions

• Per NERC Rules of Procedure 403.11.2

6

Audit Timeline

145 days 90 days 60 days 30 days 15 days AUDIT

Pre-Audit SurveyDue

Evidence Due

Objections to Team Members

Noticeof

Audit

CIP v5 Request for Information

7

Notice of Audit Packet

Notice of Audit Letter

ATT A: Compliance Monitoring Authority Letter

ATT B: Audit Team Biographies

ATT C: Confidentiality Agreements

ATT D: Audit Scope and WECC RSAWs

ATT E: Certification Letter

ATT F: Pre-Audit Survey

ATT G: Pre-Audit Data Requests

ATT H: Post Audit Feedback Form

8

Notice of Audit Letter

90-Day Notice of Audit Letter Details of your specific audit• Audit Engagement Dates• Audit Period• Registered Functions within Audit Scope• Audit Team Composition Observers (if applicable)

– May include FERC/NERC

• Date/time of proposed Pre-Audit Conference Call• Links to reference documents

9

Attachments A, B and C

Attachment AExplanation of Compliance Monitoring Authority

Attachment BShort Biographies of the WECC Audit Staff

Attachment CSigned Confidentiality Agreements of the WECC Audit Staff

10

Attachments D and E

Attachment DAudit ScopeReliability Standard Audit Worksheets (RSAWs)

Attachment ECertification Letter• Must be printed on your company letterhead

and signed by an Authorized Officer

11

Attachment F

Attachment FPre-Audit Survey• Verify Registered Functions• Audit Logistics • Signed by Authorized Officer• Please complete all applicable fields

12

Attachment G

Attachment GPre-Audit Data Requests – Clarifications for Data Submittal• One Line Diagram• Delegation agreements (if applicable)

• CCA and non-CCA lists• Public Key Encryption

13

Attachment H

Attachment HAudit Feedback• Sent with initial package

Feedback is encouraged for all phases of audit!

14

Evidence Submittal

WECC Enhanced File Transfer (EFT)https://fileupload.wecc.biz

Any questions regarding log in or user credentials please contact

weccsupport@wecc.biz or call 1-877-937-9722

15

Evidence Submittal

File Folder

COM

COM-001-1

16

Evidence Submittal

Adobe Portfolios

COM

Audit Approaches

•We audit to the Requirements of the Standards.•General Approaches included in RSAW•RSAW may ask specific questions•Always includes the section:

“Describe, in narrative form, how you meet compliance with this requirement.”

18

Audit Approaches

“Describe, in narrative form, how you meet compliance with this requirement.”• Describe here how your company knows it is compliant with

this requirement and how you know you have been compliant for the entire period of the audit

• Your place to describe your internal controls• Your evidence should support your narrative

19

Audit Approaches

• List the evidence provided in the RSAW• This road map is important

• Compliance Assessment Approach in RSAW is used as a checklist• Data Request (DR) for gaps or samples

• Document and records review are primary• Interviews and observations are usually for

corroborating

20

Sufficient Audit Evidence

Sufficiency of Evidence• The measure of the quantity of evidence• Quantity of evidence is dependent on the scope of

the audit• Extra quantity does not make up for poor quality• Ensure you provide enough evidence to demonstrate

compliance for the entire audit period

21

Sufficient Audit Evidence

Sampling is used to limit the amount of detailed evidence provided• Normally used in conjunction with summary of a full

set of data• Sampling used to assess details• Reduces the burden on the Audit Team but not really

on the Entity• Audit Team must select the samples

22

Appropriate Audit Evidence

AppropriatenessThe measure of the quality of evidence• Relevance• Validity• Reliability

23

Appropriate Audit Evidence

Quality of Evidence• Good Internal Controls point to reliable evidence• Direct observation is more reliable than indirect

observation• Examination of original documents is more reliable

than examination of copies• Testimonial evidence from system experts is more

reliable than from personnel with indirect or partial knowledge

24

Types of Evidence

• Physical Evidence• Documentary Evidence• Testimonial Evidence

Compliance Audits may use all three types but Documentary Evidence is by far the most frequent type of evidence assessed and relied on.

25

Testimonial Evidence

• Attestations of Compliance or Statements of Compliance are generally not accepted as the only available evidence.

• Attestations may be used to explain minor gaps in documentation or to state if no conditions occurred which are subject to a requirement.

• Attestor must be knowledgeable and qualified.

26

Evidence for Procedural Documents

The characteristics of a valid procedural or policy document include: – Document title – Definition or Purpose– Revision level – Effective dates – Authorizing signatures

27

Non Applicable Requirements

Three instances are acceptable for use of term “Not Applicable”1) Entity is not registered for the applicable function

(only TOP responsible for TOP requirements)

2) Entity does not own, operate or maintain the equipment addressed by the requirement (UVLS, UFLS, SPS etc.)

3) Entity does not use the program or process specified by the requirement (and is not required to… ATC, CBM, etc.)

28

Evidence for Tasks Performed

• When the standard calls for a task to be performed it must be documented.– Records– Logs– Reports– Work Orders– Phone recordings– Transcripts of phone recordings– Shift Schedules

• Dates & Times are critical

29

Evidence of “Coordination” with other entities

• Typical evidence provided initially is a single email.“…If you have any comments please contact ______”

This alone is neither sufficient or appropriate to demonstrate coordination between two or more parties.

• If emails or correspondence are used– Two way communications are needed

• Better are:– Meeting Agendas– Meeting Minutes– Attendance Lists

30

Evidence of “Distribution” of information

• Typical evidence provided initially is a single email with a large distribution list.“…please see attached”

This alone is typically neither sufficient or appropriate to demonstrate distribution to others.

• If emails or correspondence are used– Need clear identification of the personnel on the

distribution list.

• Even better is corroboration by receipt acknowledgement