epri y2k program

2513.1

EPRI Y2K ProgramEPRI Y2K Program

Jim Fortune

Operations Manager - Y2K Program

EPRI

hfortune@epri.com

650-855-2500

The Electric Power Industry and Y2K Embedded Systems

2513.2

EPRIEPRI

This is This is a Y2K a Y2K

Readiness Disclosure.Readiness Disclosure.

2513.3

Presentation OutlinePresentation Outline

• EPRI’s Y2K Program

• Embedded System Issues

• Potential Utility Impact

• How Are We Doing?

• Contingency Planning Issues

• Mitigation Strategies

• Industry Timeline & Sample Scenarios

• Summary

2513.4

EPRI Y2K Program Objectives

• Facilitate information exchange & collaboration via:

– industry-wide web site and knowledge base

– workshops, seminars, and conferences

• Provide technical leadership to the electric industry

– Communicate & develop an awareness and understanding of Y2K

embedded systems issues

– Coordinate activities with other intra and inter industry organizations

– Coordinate field testing and vendor participation

– Assist in developing contingency planning

– Facilitate Y2K readiness of essential support services such as natural

gas and telecommunications

2513.5

EPRI Y2K Program Status: TodayEPRI Y2K Program Status: Today

• 114 Members

– North American Utilities 90 (over 80% of sales)

– International Utilities 14

– Other Industries 10

• EPRI Knowledge Base w/ over 20,000 data sources

• 5 Quarterly Workshops held - over 2500 attendees

• 10 Specialty Workshops held - over 700 attendees

• 5 Regional Workshops for Distribution Co’s held

2513.6

EPRI Y2K Program: TomorrowEPRI Y2K Program: Tomorrow

• 6th Quarterly Workshop - Chicago, August 2-6

• Natural Gas Workshop - Houston, June 22-23

• Asian Workshop - Hong Kong, June 28-30

• 7th Quarterly Workshop - October

• Attorney Workshop - Chicago, June 22

• Caribbean Workshop - Puerto Rico, September

• TransAlta Roll-over Test, July 7, 8 & 9

• Early Warning System

• And More

2513.7

Embedded Systems Issues

• Date-related problems with microprocessor-based devices

• Impacted by several high risk dates, not just 12/31/1999 roll-over (eg, 8/22/99, 9/9/99, 2/29/00)

• Problems can exist at the chip or firmware level

• There are over 25 billion chips in service worldwide

2513.8

Embedded Systems IssuesEmbedded Systems Issues

• Vendor “compliance” statements are not always reliable

• Finding and correcting potential problems is difficult, time consuming, and very expensive

• New devices are not automatically compliant

• Non-compliant devices can impact system performance until they are replaced

2513.9

Potential Utility ImpactPotential Utility Impact

• Most information is generated, transmitted and/or analyzed via microprocessor-based equipment

• A moderate-sized utility may have over 30,000 devices in its generation and T&D systems

2513.10

Identified Embedded Systems Devices within the Electric System

Identified Embedded Systems Devices within the Electric System

2513.11

Potential Utility Impact (cont.)Potential Utility Impact (cont.)

• Prudency, regulatory, and legal issues surround and further complicate the overall effort

• All elements of the utility enterprise can be impacted by each other (including customers and suppliers)

• There is not 100% assurance that all problems will be found and fixed

• Contingency planning at all levels is a MUST!

2513.12

Reducing Y2K RisksReducing Y2K Risks

Probability of Occurrence

Imp

actTesting & Remediation

Con

tin

gen

cy P

lan

nin

g

High

High

Low

2513.13

Y2k is Real - Types of Anomalies FoundY2k is Real - Types of Anomalies Found

Server hardware and software

PC hardware and software

PC operating systems

Computer BIOS operating systems

DCS operator interface date display

Data loggers incorrect date display

Digital electric meters

Digital fault recorders

Digital relays

Digital controllers - substations

Network management software

PBXs and router support software

Continuous emissions monitor data recording software

or analyzer software

Annunciator (alarm) systems sequence of events

recorder

Precipitator controls

Miscellaneous support systems – analyzers, recorders,

etc.

Main frame computers

Software applications

Heating, ventilation, and air conditioning (HVAC)

systems

Security access systems

Fax machines

Source: NERC

2513.14

How Are We Doing ?How Are We Doing ?

Testing results continue to indicate minimal impact of Y2k on ability to operate

Based on 75% of testing and remediation completed as of April 30, 1999

• Contingency planning initiated

• Initial industry drill completed April 9, 1999

2513.15

How Are We Doing?

With 82% of testing and remediation done, results indicate minimal impact of Y2k on ability to operate electric systems of North America

< 3% failure rates inunremediated systems

Primary functions not affected

Source: NERC

2513.16

How Are We Doing? Industry Average Readiness Progress Status

How Are We Doing? Industry Average Readiness Progress Status

Source: NERC

2513.17

How Are We Doing ?How Are We Doing ?

On average, industry is close to meeting readiness targets:

Remediation and Testing – 5/31/1999

Y2k Ready – 6/30/1999

Few items (typically 1-5) scheduled later

Some programs reporting non-essential items

Could operate with facilities ready by 6/30/1999

2513.18

How Are We Doing ? - Generation How Are We Doing ? - Generation

Over 60 remediated fossil generation plants have been rolled-over in North America.

Many companies plan to leave their base-load fossil generation in post 2000 time frames.

U. S. nuclear units cannot be rolled-over due to NRC rules.

One Canadian nuclear unit will roll-over.

At least four hydro-systems including SCADAs have been rolled-over in North America.

2513.19

How Are We Doing?Resources Reporting They Will be Ready for Y2k

How Are We Doing?Resources Reporting They Will be Ready for Y2k

Generation Reporting

Generation

Fossil/Hydro Nuclear

The sum of the non-nuclear and nuclear generation reporting to NERC is 798,309 MW. Based on historical data, demand estimates for the transition period into the Year 2000 are expected in a range between 514,034 MW (70%) and 367,167 MW (50%) of system peak demand. A conservative estimate is that the generation that will be available and Y2k Ready will exceed the highest electrical demand during the transition to the Year 2000 by at least 55%.

Source: NERC

2513.20

• T&D systems have few embedded problems other than :– Substation controls and automation

– SCADA and EMS

• Grid reliability issues revolve around telecom reliability:– Utilities’ private networks can be verified

– Public networks, regional and local companies cannot be verified by testing.

How Are We Doing ? - T&DHow Are We Doing ? - T&D

2513.21

Good Results BUT Nagging ConcernsGood Results BUT Nagging Concerns

• On-line generating unit/station testing– How much is appropriate ?

• Verifying the Roll-Over Test Process– Hidden Real Time Clocks– Can we do a conclusive industry test ?

• Catalog testing verses inventory testing– Catalog testing leaves concerns– Inventory testing not usually feasible

• Telecom network inter-operability issues

2513.22

Contingency Planning: Key Operating RisksContingency Planning: Key Operating Risks

• Voice and data communications– Partial loss communications, excess voice traffic

• Unusual load patterns and minimum generation conditions– Very dependent on weather and location

• Loss of generation• Transmission trips• Distribution trips• Fuel supplies• Severe storm or severe cold

2513.23

Contingency Planning : Key Mitigation StrategiesContingency Planning : Key Mitigation Strategies

• Backup voice systems• Semi-manual operations (field personnel)

– Personnel at key facilities with backup communications• Operate interconnected• Excess generation on-line or standby

– Standby units running & serving house loads• Reduced transfers• Fuel reserves and mix options

– Oil, gas, hydro (reservoirs/pumped storage), nuclear• Staffing, training, drills

2513.24

Industry TimelineIndustry Timeline

• April 9 drill - Walk through of operations under back-up communications

• May 4-5 mid-term contingency plan review

• July 8-9 review of contingency plans– Goal for plans to be “as ready for use as they

can be” given remaining operating uncertainties

• September 8-9 drill - Full industry alert– Backup communications systems operational

2513.25

Sample ScenariosSample Scenarios

• Loss of primary voice systems

• RTU failures

• Partial loss of EMS/SCADA

• Key control centers, substations, power plants

• Some included

– Gas systems

– Emergency notifications

– Black start drill

– Pre-prepared failure scenarios

2513.26

SummarySummary

• Test results are revealing less problems than originally thought (only 5-6% remediation required to be “Y2K Ready”)

• Major utilities are “on schedule”

• Intra/inter industry and customer communication and coordination still required

• Need customers to operate “normally”

2513.27

SummarySummary

• Remaining questions:

– Have we found everything? Is the testing complete and valid?

– Adequate contingency plans?

– Readiness of small distribution companies

– Readiness of telecommunication systems

– Readiness of key industrial customers and suppliers