fep diversity project
DESCRIPTION
FEP Diversity Project. RTS System Admin Group RTUs, Communications. Substation – EMS Communications Overview Traditional 4-wire communications FEP Diversity Project Motivators Pilot Project - POC Technical Solutions Communications Architecture AREVA Database Modifications/Changes - PowerPoint PPT PresentationTRANSCRIPT
FEP Diversity Project
RTS System Admin Group
RTUs, Communications
Real Time Systems2
Agenda
1. Substation – EMS Communications Overview
2. Traditional 4-wire communications
3. FEP Diversity Projecta. Motivators
b. Pilot Project - POC
c. Technical Solutionsi. Communications Architecture
ii. AREVA Database Modifications/Changes
d. Additional Benefits
4. Future Direction – Expansion to other Substation needs
Real Time Systems3
Substation – EMS Communications Overview
Real Time Systems4
Substation to EMS data flow
Operating SystemUnix/ Windows 2000
Real-Time Control System Environment
Energy Management System Applications
HABITAT NETIO Web-FGUtilities: Configuration Manager MRS Alarm
SCADASubsystem
GenerationSubsystem
NetworkSubsystem
Analogs & Statuses
External unit MW schedules,pump unit limits,
incremental cost curves
Frequency, time,plant & tie-line
measurements,unit controls,
ACE, system load
Unit security limits,loss sensitivities,AGC participation
factors
Estimated Values
Unit Controls
FEP
Analogs
StatusesControls
RTUs
Accumulators
SMP – SCADA ManagementPlatform
Substation
Real Time Systems5
SCADA protocols
Protocol is a way of organizing and presenting data in a consistent, repeatable format
Bit and byte oriented protocols– Byte - Harris, DNP– Bit - Conitel, CDC2, TRW S9000, TRW S9550– Special interfaces - Intrac and MOSCAD radio systems
All master station SCADA protocols do the same basic functions
– analog data conversion– status point monitoring– support “select before operate” function for controls– provide analog output capability– provide different types of scans to capture transient events between scans
and minimize bandwidth usage– provide some form of “cyclic redundancy check” CRC so that if the received
message has been corrupted during transport it will be rejected.
Real Time Systems6
FE RTU numbers and types
Approximately 2000 RTUs on FE system
650 in west, 1350 in the east and growing
Diverse hardware platforms
Diverse protocols – including:– DNP 3.0– Harris 5000– CDC II– TRW S9000 / 9550– Conitel 300/3000– Modbus– Intrac/Moscad
Real Time Systems7
RTU communications
Primarily four wire leased circuit – 1200 baud modem
Radio used extensively in east ~ 650 radio equipped RTUs– Primarily found at line switch controllers
Dialup - Approximately 100 dialup RTUs in west– Primarily found at line switch controllers
Real Time Systems8
Substation to EMS data flow
Operating SystemUnix/ Windows 2000
Real-Time Control System Environment
Energy Management System Applications
HABITAT NETIO Web-FGUtilities: Configuration Manager MRS Alarm
SCADASubsystem
GenerationSubsystem
NetworkSubsystem
Analogs & Statuses
External unit MW schedules,pump unit limits,
incremental cost curves
Frequency, time,plant & tie-line
measurements,unit controls,
ACE, system load
Unit security limits,loss sensitivities,AGC participation
factors
Estimated Values
Unit Controls
FEP
Analogs
StatusesControls
RTUs
Accumulators
SMP – SCADA ManagementPlatform
Substation
Real Time Systems9
What is a FEP?
Windows 2k3 server OS
Standard Infrastructure server Hardware– One exception, the FEP has additional Digi PCI cards so that
server has 128 COM ports
Runs the AREVA e_terraControl suite of applications– Process Starter– CFEReader – Polls RTU’s– SCADA – Contains the database– ISD – Connects the FEP to the EMS– Ancillary processes:
– AlarmPager (provides method to pass FEP alarms to EMP)– SpaceSaver– DBUpdate
Key: No FEP’s = No Data entering the EMS
Real Time Systems10
Real Time Systems11
Traditional 4-wire communications
Real Time Systems12
Traditional 4-Wire Communications
Real Time Systems13
FEP Diversity Project
Real Time Systems14
Project Motivators
Issue identified by NERC as a disaster recovery limitation
Loss of DataCenter, either from a physical or communications perspective, essentially eliminates the EMS system.
Upgrade the method of communications to critical transmission substation RTU’s.
Network change without causing system software changes– Keep serial connected FEP ports and serial connected
RTUs
Improve Communications Network Reliability
Real Time Systems15
Proposal to operate selected critical transmission RTU’s from physically diverse FEP’s
3rd/4th Quarter 2005 performed R&D proof of concept testing
Production deployment began 1st quarter of 2006
Initial sites include interconnection and critical transmission sites
By the end of 2006 we successfully cut over 54 production sites
By the end of 2007 we successfully cut over 104 production sites – All 345 and 138kV Tx sites in FE’s MISO area
111 Transmission sites in PJM are scheduled for completion though 2012
All new FE RTU sites are being installed on this technology
Project Overview
Real Time Systems16
Pilot Objectives - What we wanted to prove? Demonstrate that the AREVA system can support data
acquisition using diversified FEP’s at the same level of reliability as currently observed with co-located FEP pairs.
Verify the proposed network communications architecture will support geographically distributed FEP’s.
Demonstrate that in using diversified FEP’s there is no degradation in AREVA system level performance (FEP and EMP).
Demonstrate that alarm processing performance is equivalent to that found with legacy designs - throughput.
Verify system responds properly to the loss of inter-FEP heartbeat.
Real Time Systems17
Required work activities in R&D Pilot period
Build new servers (Windows and Unix)
Develop a point lists for each RTU
Develop software configuration for each RTU
Develop a strategy for modeler configurations
Develop a standard network hardware configuration
Configure AREVA Systems
Configure, debug, and test pilot system
Develop Cut-over strategies for production
Real Time Systems18
Measure of success in all pilot phases? Burn in test using all RTU’s for a one week period of local
continuous polling is successful – No FEP process shutdowns
– Minimal missed scans
Alarm propagation and data integrity - 24 hour test– All status points change state at least once per scan interval
– All analog points change each scan interval
– No missed scans or missed alarms
System stability is maintained for following failover scenarios:– Loss of one FEP
– Loss of inter FEP heartbeat
– Loss of one EMS server
Real Time Systems19
FEP Diversity Pilot Phases
Phase 1 - 10/3/05-10/13/05– Performed local proof of concept testing using new FEP’s and 55
logical RTU’s. Using DYMEC supplied network hardware, failure analysis was be conducted and logical RTU’s polling statistics were gathered. FRAME network is all mocked up locally at DataCenter B.
– End Result- Satisfactory test results based on operational performance and failure analysis. All testing performed locally at DataCenter B.
Real Time Systems20
FEP Diversity Pilot Phases -CONT.
Phase 2 - 10/14/05-10/25/05– DYMEC Equipment was extended to DataCenter A to allow
controlled diversified FEP testing. Required relocating a FEP server to DataCenter A, and the associated FRAME network connectivity.
– End Result- Satisfactory test results based on operational performance and failure analysis.
Real Time Systems21
FEP Diversity Pilot Phases - CONT.
Phase 3 -10/26/05-11/16/05– Selected substations (6 site’s) will be dual ported and connected to
digital circuits. Digital FRAME circuits will appear at DataCenter A and DataCenter B where the diversified FEP’s and EMS servers reside.
– Includes site’s on the FE, and the Telco (leased) FRAME network’s
– End Result- Satisfactory test results based on operational performance and failure analysis.
Real Time Systems22
Results of Pilot
Project team successfully installed and configured the new versions of the SMP software on FEP pair.
– Latest Windows O/S
– Latest AREVA SMP Software
– New hardware components
Verified successful RTU communications using current serial communications.
All three phases of RTU testing completed successfully, including phase III’s real-life field tests
Developed Cutover Plan
Real Time Systems23
Production Implementation Project
Order communication circuits as required– A majority of circuits require High Voltage Protection (Positron)
Make necessary RTU configuration changes– Baud Rate / Modem settings
Complete AREVA modeler work to allow cutover to occur– Run ManageSiteAssignments.pl script to use Primary/Secondary SITE ID– Make PATH changes for new communication circuit
Cutover RTU’s to Diversified FEP– Perform full station check-out on new circuit
Real Time Systems24
Project Timeline
Real Time Systems25
FEP Diversity Architecture
Real Time Systems26
FILENAME
TITLE
FEP DIVERSIFICATION PILOT PHASE #1
Network Services EngineeringConfidential
PAGE
1 OF 1
REVISED
9/14/2006
FULL FILENAME
DRAWN BY
REX DUDAS
FEPDIVERSIFICATION
DYMEC(Physical)
DYMEC
- DSU/CSU (Digital Circuit Interface)(T1-Fractional, 64k)
- ROUTER
-FRAD (Frame-Relay Access Device)
-SUBSTATION HARDENED(-40C -+75C) no fans
5 PORT ETHERNET SWITCH(IP ONLY DEVICES i.e.: PC,
IP CAMERA...etc)
8 PORT TERMINAL SERVER(SERIEL CONNECTED
DEVICES i.e: RTU, DFR,LINE SWITHCES...etc)
FRAME RELAYNETWORK
- BUILT-IN DATA ANALYZER
- FRAME/SERIAL ENCAPSULATION RTU PROTOCOL
10/100
max.: 38400bps
Real Time Systems27
FILENAME
TITLE
FEP DIVERSIFICATION PILOT PHASE #1
Network Services EngineeringConfidential
PAGE
1 OF 1
REVISED
9/14/2006
FULL FILENAME
DRAWN BY
REX DUDAS
FEPDIVERSIFICATION
DYMEC(LOGICAL)
(ROUTER)
5 PORT ETHERNET SWITCH(IP ONLY DEVICES i.e.: PC,
IP CAMERA...etc)
8 PORT TERMINAL SERVER(SERIEL CONNECTED
DEVICES i.e: RTU, DFR,LINE SWITHCES...etc)
- FRAME/SERIAL ENCAPSULATION RTU PROTOCOL(NON-ROUTABLE, ENCRYPTED)
RTU
DYMEC
FRAME RELAYNETWORK PVC 1
PVC 2
PVC 3
CORP. TRAFFIC(FIREWALL, ENCRYPTED)
DataCenter A FEPA
DataCenter B FEPB
RTU MAINTENANCE PORT
IP ROUTABLE PROTOCOL REMOVED OVER WANLINK
Real Time Systems28
Cut-over Strategy (part1)
Need to have both existing and diversified RTU modeled for cutover in AREVA EMS– Model an additional TFE structure for new diversified FEP
– Changes to BAUD, PTMO1, PTMO2 and NoReplyTimeout– Additional RTUC Record pointing to same RTU record as 4-wire circuit
in production– Utilize SITE_* and SITE2_* fields to facilitate the cutover
– Each RTU has potentially hundreds of POINT/ANALOG/COUNT records associated with it.
– These all need their SITE_* field set to the ID_SITE of the new diverse FEP.
– SITE2_* field set to the ID_SITE of the current 4-wire FEP in production– Manage those individual data items that do NOT allow automatic
SITE/SITE2 selection. – Selection is done automatically by SCANNER based on quality. Items
with no status, (CTRL only) do not have a good quality code to follow (TAP Changers)
Areva TFE tree changes made, No RTU tree changes made.– Limit the risk of the change – No production affects
Real Time Systems29
Cut-over Strategy (part2)
RTU Changes– New FRAME T1 Circuit can be installed at any time on
spare pairs– Make RTU configuration changes and load onto spare D20
processor board– Swap out board for cutover, if there are issues original board
can be immediately put back
– No Point mapping changes made
Real Time Systems30
POST DESIGN BENIFITS DR Environment
– Remote RTU can be pulled from either DataCenter Eliminates costly – Out-of-date Analog 4-wire circuits Increase RTU speed/poll interval
– 1200bps to 9600bps (max.: 38400 bps)
Ability to pull byte and bit-oriented RTUs Remote Support
– Allow for mobile workforce support/configuration/data analysis – Secured IP Based
Security– Serial Connectivity – Isolation of corporate substation traffic
HP/OV Support– Complete Network end-to-end alarming
“Smart” Substation platform– Ability to add additional PVC’s for other needs
– IP enabled Security Cameras for monitoring site– Swipe card entry systems
CIP Cyber Security full compliance– Non-Routable Transport
Real Time Systems31
FEP Diversification Overview
Real Time Systems32
FE
Cor
p N
etw
ork
RP
R B
ackb
one
FE Isolated WANSONET/RPR
EMS SONET/RPRBackbone
T1T1
Substation RTU NetworkRPR Exchange WAN
BG #TBD
GMS SONET/RPRBackbone
RTS FWDC-A
RTS FWDC-B
EMS FEPsDC-A
Sub-RTRDC-A
Sub-RTRDC-B
Public CarrierFR/MPLS
TCP/IPNetwork
AggregateRTU Router
Site #1
SubstationRTU Router
Site #5
AggregateRTU Router
Site #4
AggregateRTU Router
Site #3
AggregateRTU Router
Site #2
SubstationRTU
Site #4
SubstationRTU
Site #5
SubstationRTU
Site #3Substation
RTUSite #2
SubstationRTU
Site #1
DynamicFailover
FirstEnergy FEP Diversity: Proposed IP Based Network Architecture A
DynamicFailover
Sub-RTRDC-C
RTS FWDC-C
DynamicFailover
DynamicFailover
EMS FEPsDC-B
EMS FEPsDC-C
GMS DACsDC-A
GMS DACsDC-B
T1
Fa
stE to
ML C
ard
FastE to
ML Card
FastE to
ML Card
Consolidate RTU Communication over anIP Based FE RPR Wan and/or Public
Carrier Frame/MPLS Networks.
Corp FWDC-A
Corp FWDC-D
Electronic Security Perimeter
SubstationRTU
Site #1a
SubstationRTU
Site #1b
Fractional T1
Fa
stE to
ML C
ard
Fast
E t
oM
L C
ard
Ele
ctro
nic
Se
curit
y Pe
rime
ter
Real Time Systems33
EMS SONET/RPRBackbone
FE
Cor
p N
etw
ork
RP
R B
ackb
one
FE Isolated WANSONET/RPR
T1T1
Substation RTU NetworkRPR Exchange WAN
BG #TBD
GMS SONET/RPRBackbone
RTS FWDC-A
RTS FWDC-B
Sub-RTRDC-A
Sub-RTRDC-B
Public CarrierFR/MPLS
TCP/IPNetwork
AggregateRTU Router
Site #1
SubstationRTU Router
Site #5
AggregateRTU Router
Site #4
AggregateRTU Router
Site #3
AggregateRTU Router
Site #2
SubstationRTU
Site #4
SubstationRTU
Site #5
SubstationRTU
Site #3Substation
RTUSite #2
SubstationRTU
Site #1
DynamicFailover
FirstEnergy FEP Diversity: Proposed IP Based Network Architecture B
DynamicFailover
Sub-RTRDC-C
RTS FWDC-C
DynamicFailover
DynamicFailover
GMS DACsDC-A
GMS DACsDC-B
T1
Fa
stE to
ML C
ard
FastE to
ML Card
FastE to
ML Card
Consolidate RTU Communication over anIP Based FE RPR Wan and/or Public
Carrier Frame/MPLS Networks.
Corp FWDC-A
Corp FWDC-D
Electronic Security Perimeter
SubstationRTU
Site #1a
SubstationRTU
Site #1b
Fractional T1
Fa
stE to
ML C
ard
Fast
E t
oM
L C
ard
Ele
ctro
nic
Se
curit
y Pe
rime
ter
DYMECDYMECDYMEC
EMS FEPsDC-A
EMS FEPsDC-B
EMS FEPsDC-C
Real Time Systems34
Questions?
??