teleprotection with mpls ethernet communications...
TRANSCRIPT
![Page 1: Teleprotection with MPLS Ethernet Communications ...prorelay.tamu.edu/wp-content/uploads/sites/3/2018/04/3-SDGE_Tele... · TDM or serial data communications circuits ... Voice Corporate](https://reader031.vdocuments.us/reader031/viewer/2022021823/5b3488997f8b9aa0238e1c8f/html5/thumbnails/1.jpg)
Slide 1
Teleprotection with MPLS Ethernet Communications - Development and Testing of
Practical Installations
Tariq Rahman and James Moralez, San Diego Gas & Electric CompanySolveig Ward and Eric A. Udren, Quanta Technology, LLC
Michael Bryson and Kamal Garg, Schweitzer Engineering Laboratories, Inc.
Presented at Texas A&M Conference for Protective Relay Engineers
College Station, TXMarch 28, 2018
![Page 2: Teleprotection with MPLS Ethernet Communications ...prorelay.tamu.edu/wp-content/uploads/sites/3/2018/04/3-SDGE_Tele... · TDM or serial data communications circuits ... Voice Corporate](https://reader031.vdocuments.us/reader031/viewer/2022021823/5b3488997f8b9aa0238e1c8f/html5/thumbnails/2.jpg)
Slide 2
Background
![Page 3: Teleprotection with MPLS Ethernet Communications ...prorelay.tamu.edu/wp-content/uploads/sites/3/2018/04/3-SDGE_Tele... · TDM or serial data communications circuits ... Voice Corporate](https://reader031.vdocuments.us/reader031/viewer/2022021823/5b3488997f8b9aa0238e1c8f/html5/thumbnails/3.jpg)
Slide 3 © Copyright 2018 SDG&E, SEL, Quanta Technology
San Diego Gas & Electric System■ Provides natural gas and electricity
to San Diego County and southern Orange County in southwestern California to 3.6 million consumers
■ 1.4 million electric meters and 873,000 natural gas meters in a service area that spans 4,100 square miles
■ Currently, SDG&E uses TDM network for teleprotection and SCADA
■ The TDM network consists of a mix of direct fiber, T1 multiplexers on TDM SONET, microwave radio, leased-line, and channel bankequipment
San Francisco
Los Angeles
SDG&E
San Diego
SoCal Gas
![Page 4: Teleprotection with MPLS Ethernet Communications ...prorelay.tamu.edu/wp-content/uploads/sites/3/2018/04/3-SDGE_Tele... · TDM or serial data communications circuits ... Voice Corporate](https://reader031.vdocuments.us/reader031/viewer/2022021823/5b3488997f8b9aa0238e1c8f/html5/thumbnails/4.jpg)
Slide 4 © Copyright 2018 SDG&E, SEL, Quanta Technology
Introduction
■ Technology evolution is driving towards Ethernet communications - converged utility communications network
■ Typically, packet based IP routing in an Ethernet WAN had been fundamentally less predictable than the deterministic point to point TDM or serial data communications circuits
■ Teleprotection is migrating from SONET to MPLS Ethernet■ To validate the design and in preparation for substation field
installations within the SDG&E system, laboratory testing was performed using a Real Time Digital Simulator or RTDS® system model
■ Test MPLS routers & network configurations were applied to protective relays at the SDG&E Integrated Test Facility (ITF)
![Page 5: Teleprotection with MPLS Ethernet Communications ...prorelay.tamu.edu/wp-content/uploads/sites/3/2018/04/3-SDGE_Tele... · TDM or serial data communications circuits ... Voice Corporate](https://reader031.vdocuments.us/reader031/viewer/2022021823/5b3488997f8b9aa0238e1c8f/html5/thumbnails/5.jpg)
Slide 5 © Copyright 2018 SDG&E, SEL, Quanta Technology
Utility Communications Services
System Critical
System Priority
System Administration and Support
0 1 10 102 103 104 105 106 107
Seconds Minutes Hours Days Weeks
Private, dedicated circuits Public Networks, shared circuits
Performance Cost of Service
Telep
rote
ction
Tap
chan
ger c
ontro
lAG
CTi
e Lin
e Co
ntro
lSC
ADA
Alar
mEM
S
Voice
Corp
orat
e co
mpu
ter l
inks
File
trans
fers
Back
ups
Powe
r sys
tem
mar
ketin
gBi
lling
Resid
entia
l met
ering
Adm
inistr
ation
Disp
atch
pho
nes
Powe
r poo
l sch
eduli
ngM
ainte
nanc
eM
eter
ingDi
stribu
tion
auto
mat
ion
![Page 6: Teleprotection with MPLS Ethernet Communications ...prorelay.tamu.edu/wp-content/uploads/sites/3/2018/04/3-SDGE_Tele... · TDM or serial data communications circuits ... Voice Corporate](https://reader031.vdocuments.us/reader031/viewer/2022021823/5b3488997f8b9aa0238e1c8f/html5/thumbnails/6.jpg)
Slide 7 © Copyright 2018 SDG&E, SEL, Quanta Technology
SONET Characteristics
■ Point-to-point connection■ Deterministic and low latency (1 – 3 ms)■ Equal transmit and receive delay (no asymmetry)■ Ring redundancy■ Substation multiplexer fail-over as low as 2 - 3 ms
Teleprotection
Teleprotection
![Page 7: Teleprotection with MPLS Ethernet Communications ...prorelay.tamu.edu/wp-content/uploads/sites/3/2018/04/3-SDGE_Tele... · TDM or serial data communications circuits ... Voice Corporate](https://reader031.vdocuments.us/reader031/viewer/2022021823/5b3488997f8b9aa0238e1c8f/html5/thumbnails/7.jpg)
Slide 8 © Copyright 2018 SDG&E, SEL, Quanta Technology
Ethernet IP Characteristics
■ Ethernet is based on IEEE 802.3 standard with various versions supporting higher data rates and lower latency
■ Widely adopted packet-based technology■ Non-deterministic latency■ Basis for IEC 61850 P&C
Teleprotection
Teleprotection
![Page 8: Teleprotection with MPLS Ethernet Communications ...prorelay.tamu.edu/wp-content/uploads/sites/3/2018/04/3-SDGE_Tele... · TDM or serial data communications circuits ... Voice Corporate](https://reader031.vdocuments.us/reader031/viewer/2022021823/5b3488997f8b9aa0238e1c8f/html5/thumbnails/8.jpg)
Slide 9 © Copyright 2018 SDG&E, SEL, Quanta Technology
MPLS Characteristics■ MultiProtocol Label Switching – packet label field routes Ethernet
packets among MPLS routers■ Dynamic and static routing available■ Predictable latency■ Pseudowire services to support TDM/Serial communications■ Low latency enabled by using a static pre-defined path, and the use of
small jitter (data) buffers for teleprotection traffic ■ High priority provisioning through the use of Quality of Service (QoS)
configuration■ MPLS ensures minimal asymmetry by routing transmit and receive
packets over static paths via the same network nodes■ Path fail-over times 50 - 300 ms
• Mitigated by using redundant teleprotection channels in the relay with 0 – 2 ms fail-over time
![Page 9: Teleprotection with MPLS Ethernet Communications ...prorelay.tamu.edu/wp-content/uploads/sites/3/2018/04/3-SDGE_Tele... · TDM or serial data communications circuits ... Voice Corporate](https://reader031.vdocuments.us/reader031/viewer/2022021823/5b3488997f8b9aa0238e1c8f/html5/thumbnails/9.jpg)
Slide 10 © Copyright 2018 SDG&E, SEL, Quanta Technology
SDG&E MPLS Project Drivers
■ MPLS is the current communications transport standard being widely adopted in other Industrial Control Systems (ICS) environments such as water, public safety networks, land mobile radio backhaul, etc.
■ As MPLS is adopted into substation communications -replacing instead of upgrading older technology - it is expected to deliver significant benefits to overall utility communications, with higher service availability
■ Provides a reduction in maintenance costs (O&M) as utility operates a single communications system
■ Provides comprehensive network monitoring and network diagnostics
![Page 10: Teleprotection with MPLS Ethernet Communications ...prorelay.tamu.edu/wp-content/uploads/sites/3/2018/04/3-SDGE_Tele... · TDM or serial data communications circuits ... Voice Corporate](https://reader031.vdocuments.us/reader031/viewer/2022021823/5b3488997f8b9aa0238e1c8f/html5/thumbnails/10.jpg)
Slide 11
Project Development
![Page 11: Teleprotection with MPLS Ethernet Communications ...prorelay.tamu.edu/wp-content/uploads/sites/3/2018/04/3-SDGE_Tele... · TDM or serial data communications circuits ... Voice Corporate](https://reader031.vdocuments.us/reader031/viewer/2022021823/5b3488997f8b9aa0238e1c8f/html5/thumbnails/11.jpg)
Slide 12 © Copyright 2018 SDG&E, SEL, Quanta Technology
SDG&E Methodology
■ Development of business requirements based on internal and external drivers
■ Development of in-depth technical requirements, and requirements traceability matrix
■ Assuming a successful field trial testing period, the migration of teleprotection will commence as MPLS network service is migrated to substations
■ Creation of an MPLS network lab testing environment■ Implementation and testing of channel monitoring functions■ Installation of transmission line field test relays and monitoring
for a period of 12 months■ RTDS lab testing of teleprotection over MPLS
![Page 12: Teleprotection with MPLS Ethernet Communications ...prorelay.tamu.edu/wp-content/uploads/sites/3/2018/04/3-SDGE_Tele... · TDM or serial data communications circuits ... Voice Corporate](https://reader031.vdocuments.us/reader031/viewer/2022021823/5b3488997f8b9aa0238e1c8f/html5/thumbnails/12.jpg)
Slide 13
RTDS Model
SS
SS
Tap
S
S
S
S
S
L
L
L
C
C
500 kV
230 kV
500 kV
230 kV
500 kV230 kV
500 kV69 kV 500 kV
16.634 Ω
27.45 Ω
j33.7
S
Transfer
Transfer
S
Y Y Δ
34.5 kV
34.5 kV
500 kV
230 kV
1
2
Fault Location
Sliding Fault Location
Circuit Breaker
Static Source
Line Shunt Reactor
Shunt Capacitor
Transfer Impedance Branches
Series Capacitor
PV – Solar Generation
WTG – Wind Generation
S
L
C
C138 kV
L
C
C
L
S
3 124
6
5
8
9
7
1 to 99%
25.6 Ω
WTG
WTG
PVPV
WTG
EQ
![Page 13: Teleprotection with MPLS Ethernet Communications ...prorelay.tamu.edu/wp-content/uploads/sites/3/2018/04/3-SDGE_Tele... · TDM or serial data communications circuits ... Voice Corporate](https://reader031.vdocuments.us/reader031/viewer/2022021823/5b3488997f8b9aa0238e1c8f/html5/thumbnails/13.jpg)
Slide 14 © Copyright 2018 SDG&E, SEL, Quanta Technology
Test Requirements and Test Setup
1. Latency < 5 ms2. Asymmetry < 2 ms3. Failover < 3 ms4. Availability > 99.95%
MPLS Typical Testing Network
Relay 187L CH X87L CH Y
Relay 287L CH 187L CH 2
Relay 387L CH 1
87L CH 2
MPLS Router A MPLS Router B Relay 1
Relay 2
Relay 3MPLS Router C
87L CH X87L CH Y
87L CH 187L CH 2
87L CH 2
87L CH 1
Primary Path
Secondary Path
Router Failover Path
Router Failover Path
MPLS Router D
MPLS Network
MBAMBB
MBAMBB
MBA
MBA
MBB
MBB
![Page 14: Teleprotection with MPLS Ethernet Communications ...prorelay.tamu.edu/wp-content/uploads/sites/3/2018/04/3-SDGE_Tele... · TDM or serial data communications circuits ... Voice Corporate](https://reader031.vdocuments.us/reader031/viewer/2022021823/5b3488997f8b9aa0238e1c8f/html5/thumbnails/14.jpg)
Slide 15 © Copyright 2018 SDG&E, SEL, Quanta Technology
Asymmetry < 2 ms■ 87L with channel based synchronization uses the loop
delay divided by 2 for alignment
ChannelDelay
Local current
90 deg. error
Correct compensation Incorrect compensation
Differential current
Local current memorized for comparison
Current received from remote end
![Page 15: Teleprotection with MPLS Ethernet Communications ...prorelay.tamu.edu/wp-content/uploads/sites/3/2018/04/3-SDGE_Tele... · TDM or serial data communications circuits ... Voice Corporate](https://reader031.vdocuments.us/reader031/viewer/2022021823/5b3488997f8b9aa0238e1c8f/html5/thumbnails/15.jpg)
Slide 16 © Copyright 2018 SDG&E, SEL, Quanta Technology
Asymmetry <2 ms - Test Setup
■ 2 ms asymmetry introduced
87L Channel Asymmetry Test Setup
Relay 187L CH X
Relay 287L CH 1
Relay 387L CH 1
MPLS Router MPLS Router Relay 1
Relay 2
Relay 3Asymmetry Delay(Linux Desktop)
87L CH X
87L CH 1
87L CH 1
87L Forward Path
87L Return Path
![Page 16: Teleprotection with MPLS Ethernet Communications ...prorelay.tamu.edu/wp-content/uploads/sites/3/2018/04/3-SDGE_Tele... · TDM or serial data communications circuits ... Voice Corporate](https://reader031.vdocuments.us/reader031/viewer/2022021823/5b3488997f8b9aa0238e1c8f/html5/thumbnails/16.jpg)
Slide 17
Fail-over <3 ms – Test Setup87L Channel Link Break
Test Setup
Relay 187L CH X
87L CH Y
Relay 287L CH 1
87L CH 2
Relay 387L CH 1
87L CH 2
MPLS Router MPLS Router Relay 1
Relay 2
Relay 3Ethernet Radio Ethernet Radio
87L CH X
87L CH Y
87L CH 1
87L CH 2
87L CH 2
87L CH 1
Primary Path
Secondary Path
Ethernet Link Breaker
Router Failover Path
Router Failover Path
![Page 17: Teleprotection with MPLS Ethernet Communications ...prorelay.tamu.edu/wp-content/uploads/sites/3/2018/04/3-SDGE_Tele... · TDM or serial data communications circuits ... Voice Corporate](https://reader031.vdocuments.us/reader031/viewer/2022021823/5b3488997f8b9aa0238e1c8f/html5/thumbnails/17.jpg)
Slide 18
Latency <5 ms
■ Relays measure latency from (a)-(a) or (b)-(b) depending on the relay type
■ 5 ms specification is for (b)-(b)
![Page 18: Teleprotection with MPLS Ethernet Communications ...prorelay.tamu.edu/wp-content/uploads/sites/3/2018/04/3-SDGE_Tele... · TDM or serial data communications circuits ... Voice Corporate](https://reader031.vdocuments.us/reader031/viewer/2022021823/5b3488997f8b9aa0238e1c8f/html5/thumbnails/18.jpg)
Slide 19
Latency <5ms - Test Results
![Page 19: Teleprotection with MPLS Ethernet Communications ...prorelay.tamu.edu/wp-content/uploads/sites/3/2018/04/3-SDGE_Tele... · TDM or serial data communications circuits ... Voice Corporate](https://reader031.vdocuments.us/reader031/viewer/2022021823/5b3488997f8b9aa0238e1c8f/html5/thumbnails/19.jpg)
Slide 20 © Copyright 2018 SDG&E, SEL, Quanta Technology
Summary of Test Results
Communication Requirement
Specification Results
Latency < 5 ms Pass1
Asymmetry < 2 ms Pass2
Failover < 3 ms Pass3
Availability > 99.95% N/A
1 Latency < 5 ms achieved with specific Jitter Buffer and Payload MPLS router settings.2Asymmetry < 2 ms achievable with specific network design. Laboratory tests show protection operates correctly at 2 ms asymmetry specification limit.3Failover < 3 ms achievable with 2 of 3 relays meeting specification. Protection system with designed failover paths and protective relay failover meets failover specification. MPLS routers do not meet failover specification by failing over to backup Ethernet router path.
![Page 20: Teleprotection with MPLS Ethernet Communications ...prorelay.tamu.edu/wp-content/uploads/sites/3/2018/04/3-SDGE_Tele... · TDM or serial data communications circuits ... Voice Corporate](https://reader031.vdocuments.us/reader031/viewer/2022021823/5b3488997f8b9aa0238e1c8f/html5/thumbnails/20.jpg)
Slide 21 © Copyright 2018 SDG&E, SEL, Quanta Technology
Conclusions
■ The schemes and relay settings are thoroughly tested in the RTDS lab on accurate protected-circuit and system models, and with lab MPLS network routers and connections.
■ It is not possible to emulate all of the in-service MPLS network conditions in the lab, but lab tests with thousands of fault simulations produced extensive baseline reference performance results.
■ With baseline results, root-cause analysis of any protection misoperations during field testing will not require extensive retesting of proven protection schemes.
![Page 21: Teleprotection with MPLS Ethernet Communications ...prorelay.tamu.edu/wp-content/uploads/sites/3/2018/04/3-SDGE_Tele... · TDM or serial data communications circuits ... Voice Corporate](https://reader031.vdocuments.us/reader031/viewer/2022021823/5b3488997f8b9aa0238e1c8f/html5/thumbnails/21.jpg)
Slide 22 © Copyright 2018 SDG&E, SEL, Quanta Technology
Conclusions (continued)
■ Laboratory and field relay testing are validating the new MPLS application and are promoting learning about the new communications system for SDG&E engineers, technicians and operations personnel.
■ The long failover times of 50 to 300 ms for MPLS Ethernet channels are overcome with a redundant live MPLS path scheme enabled by high MPLS data capacity – relays connect directly to redundant paths and achieve failover time of 0 to 2 ms.
■ Direct fiber paths do not need to be converted to MPLS Ethernet.
![Page 22: Teleprotection with MPLS Ethernet Communications ...prorelay.tamu.edu/wp-content/uploads/sites/3/2018/04/3-SDGE_Tele... · TDM or serial data communications circuits ... Voice Corporate](https://reader031.vdocuments.us/reader031/viewer/2022021823/5b3488997f8b9aa0238e1c8f/html5/thumbnails/22.jpg)
Slide 23 © Copyright 2018 SDG&E, SEL, Quanta Technology
Conclusions (continued)
■ Laboratory testing has shown that MPLS networks are a viable communications medium for protective relay telecommunication traffic if designed to account for latency, asymmetry, failover and availability.
■ RTDS tests validated settings for routers and switches of the field MPLS network, as well as for the relays.
■ RTDS testing has allowed SDG&E to specify and set the channel/communications monitoring parameters in the relays to support MPLS Ethernet performance monitoring – had not been implemented or needed with TDM.
![Page 23: Teleprotection with MPLS Ethernet Communications ...prorelay.tamu.edu/wp-content/uploads/sites/3/2018/04/3-SDGE_Tele... · TDM or serial data communications circuits ... Voice Corporate](https://reader031.vdocuments.us/reader031/viewer/2022021823/5b3488997f8b9aa0238e1c8f/html5/thumbnails/23.jpg)
Slide 24 © Copyright 2018 SDG&E, SEL, Quanta Technology
Acknowledgements
A special “Thank You” to those who have contributed to the creation and completion of this paper and presentation:
• Mike Mahoney – Burns & McDonnell• Clint Struth and Cory Struth – SCI Networks• Terry Wright – GDC Consulting