iec 61850 network architectures

5/21/2018 IEC 61850 Network Architectures

1/41

1

IEC 61850 Network Architectures

July, 2010

Maciej Goraj

[email protected]


2/41

2Copyright RuggedCom Inc

Agenda

1. Requirements for substation communications network

2. Types of protocols and traffic patterns in IEC 61850 standard

3. Typical network architectures4. Problem of Multicast and Physical vs. Logical separation of

Process Bus and Station Bus


3/41

3

Requirements for Substation

Hardened Networking Equipment


4/41


Substation Environment

Electric and Magnetic Fields

Electrostatic Discharge

Conducted High Frequency Electrical Transients

High Energy Power Surges Ground Potential Rise during ground faults

Climactic Variation: Temperature & Humidity

Seismic / Vibration

Pollution: Dust, Metallic Particles, Corrosive Chemical Particles,

Condensation, Solar Radiation, Salt, Bird Guano, etc.

EMI & Environmental Phenomena Typical of Substation Environments

Generation Plant HV/MV Substation Wind Farm


5/41


ContinuousPhenomena

Radiated RFI

Induced RFI

Power freq. Magnetic

Field

Slow Voltage Variations Harmonics,

Interharmonics

Ripple on d.c. power

supply

Power Frequency Voltage

Transient Phenomena

(High Occurrence)

Electrostatic Discharge

Voltage Dips

Lightning Ground

Potential Rise (GPR)

HV Switching by Isolators Reactive Load Switching

Transient Phenomena

(Low Occurrence)

Power Frequency

Variation

Power System Faults

Short Duration Power

Freq. Magnetic Fields

EMI Phenomenon

Devicesinsubstations mustdealwithacombinationofEMI

phenomenawhicharebothcontinuousandtransient.


6/41


Requirements for IEDs According to IEC 61850-3

Must operate properly under the influence of a variety of EMI

phenomena commonly found in the substation

IEC 61850-3 specifies a variety of type withstands tests designedto simulate EMI phenomena such as:

Inductive load switching

Lightening strikes

Electrostatic discharges from human contact Radio frequency interference due to personnel using portable radio

handsets

Ground potential rise resulting from high current fault conditions within

the substation

Ethernetswitches,routers,deviceservers,mediaconverters

shall

meet

EMI

requirements

to

the

same

extent

as

IEDs


7/41


Standard for Environmental and Testing Requirements for

Communications Networking Devices in Electric Power

Substations It goes one step further by defining Class 2 operation which

requires that, during the application of the type tests, the switch

must experiment:

No communications errors

No communications delays

No communication interruptions

RuggedizedEthernetswitchshallbeseenasyetanotherIED

Requirements for IEDs According to IEEE 1613


8/41


Fiber Optics Overview

Future proof

Theoretically infinite bandwidth

Up to 100 km distance possible

Immune to EMI

Supported by all current IEDs

Lightweight

Costs continue to drop

Multi-mode for short distances

Single-mode for long distances


9/41


Common Fiber Optic Connectors

ST Stick and Twist and SC Stick and Click historically popular

LC becoming prevalent especially for Gigabit because small form

factor (SFF) allows greater port density GBIC are pluggable SC transceivers using SC connectors

SFP are Small Form Factor Pluggable

SCST

LC MTRJ


10/41

10

Types of Protocols and Traffic

Patterns in IEC 61850 Standard


11/41


Typically 2040IEDs per substation

Large substations mayhave 80

120

IEDs

Power Plants,Oil&Gasinstallation 150500IEDs

Large installations with LVIEDs 10001500IEDs

Large Wind Farms canhave +200IEDs

LargeSolargenerationsitescanhave600 1500IEDs

Number of Devices in Electrical Substations


12/41


IEC 61850 Ed. I Profiles and Protocols Stack

Will be moved to an Annex in Edition II of IEC 61850


13/41


IEC 61850 Ed. II Profiles and Protocols Stack

TimeSync

(SNTP)

TCP/IP

T-Profile

UDP/IP

GOOSESV MMS Protocol

Suite

ISO/IEC 8802-3

Core

ACSI

Services

Time

Sync

Generic

Object

Oriented

Substation

Event

Sampled

Values

(Multicast)

(Type 4) (Type 1, 1A) (Type 6) (Type 2, 3, 5)

ISO/IEC 8802-3 Ethertype

HSR (O)

SMV GOOSE

802.1Q 802.1Q 802.1Q (O)802.1Q (O)

IP (O)


14/41


Types of traffic

Client-server MMS services:

Polling

Reporting (Unsolicited and/or periodic)GOOSE

Asynchronous and unsolicited

Less often synchronous (for heartbeat and for analogue values)

Sampled Values (Process Bus) Synchronous unsolicited transmission

IEC

61850

network

is

a

combination

of

Raw

Ethernet,

MMS/TCP,

SNTP,

IEEE

1588,

TFTP,

FTP,

RSTP,

SNMP,

and

otherEthernetbasedprotocols


15/41


Encapsulated directly in Ethernet layer

High priority, critical, asynchronous and unsolicited

Less often synchronous (for heartbeat and for analogue values) MAC Multicast, uses VLAN for priority and traffic segregation

Frame size approx. 92 250 bytes

Periodic heartbeat messages of 1-60 seconds interval if no events occur

99% of time just the heartbeat message

In case of event an avalanche can occur as many IEDs detect state changes

Typically used for fast transmission of digital events

Less often for transmission of analogue data, e.g. sent every 250ms

Non-IP traffic in IEC 61850 - GOOSE


16/41


GOOSE is connectionless

No confirmation from receivers

Retransmission to increase the probablity of sucessful reception

A burst of 5-6 messages sent in case of event (critical information)

Example of implementation:

1st message: on event

2nd message: 4ms after event

3rd message: 16ms after event

4th message: 80ms after event

5th message: 500ms after event

Retransmission Scheme in GOOSE

Burst of GOOSEs sent on event occurence

Time

Heartbeat GOOSEs

Event occurs, GOOSE with incremented

stNum sent immediately


17/41


GOOSE and Network Performance

GOOSE messages shall be priority tagged

Configuration needed in IEDs and in Ethernet switches

GOOSE frames with the priority tag in VLAN field configured are

placed in the front of the store and forward queue

Frames already being sent are not interrupted

Delay of frames introduced by network is almost zero

Worst case of total network delay is

100 s at 100MBps links speeds

10 s at 1Gbps


18/41


GOOSE and Network Performance

IEC 61850-5 Type 1A Trip total transfer time defined at 4ms

Transfer time = Application to Application and includes:

GOOSE encoding at sender + network delays + GOOSE decoding at receiver

It is difficult to measure as defined in IEC 61850-5

Because the timestamp is added in IED after the internal function

execution time (one scan period)

Typical measured GOOSE total transfer time including functionexecution time in IED is in the range of 6-12ms


19/41


Encapsulated directly in Ethernet layer

High priority, critical, synchronous and unsolicited

MAC Multicast, uses VLAN for priority and traffic segregation

Currently dedicated wiring (IRIG-B or 1PPS) used for time synch of

devices, future will be IEEE 1588

A Merging Unit (sensor) sends 80 or 256 samples/power cycle. At 50Hz it

is 4000 and 12800 samples per second respectively.

A sample is a set of 8 analog values, 4 voltages + 4 currents

@80 samples 4000 packets/sec

A single Merging Unit uses approx. 4.4 5.2Mbit/s of bandwidth at 80 Smp

The bandwidth used depends of sampling rate and if Data Set is according

to IEC61850-9-2LE implementation or other Data Set

1Gbit Ethernet highly recommended for Process Bus in switched Ethernet

Non-IP traffic in IEC 61850 Sampled Values


20/41


Client-Server services

MMS protocol over TCP, port 102

measurements, events, status indications 100-500ms delay accepted Traffic generated by a single IED rarely exceeds 10kbps

Reports save bandwidth. Digitals via Buffered, Analogs via Unbuffred.

Time synchronization SNTP or IEEE 1588

For redundancy mutiple time masters used

File transfer MMS over TCP, FTP, TFTP, other protocols e.g. Modbus/TCP

Typically Oscillography, sequence of events, data logs. Ocassionallyconfiguration, settings, firmware upgrades, etc. File size typically 4 200 kbytes,

IP based traffic in IEC 61850


21/41

21

Typical Network Architectures


22/41


HMI Gateway

Protection and Control IEDs

Star Topology

Not protected against single point of failure

Simplicity


23/41


HMI Gateway


Redundant Star Topology

Blue LAN A

Red LAN B

The entire network is duplicated

Configuration and application complexity, cost issues

Each device has 2 IP addresses, 2 application instances

PRP will be the alternative


24/41


Fiber Optic Ethernet Ring

100/1000 Mbps

HMI Gateway

Dashed Redundant Connections


Single Ring Topology

IEDs can be dual homed and connected via

redundant links

Redundancy with RSTP

PRP or HSR will be the alternative

Blue Electrical 100Mpbs

Red Fiber Optic 100Mbps


25/41


Secondary Ring 1

HMI Gateway


Multiple Rings Topology

Secondary Ring n

Primary Ring

Limited number of switches in each ring

Minimize recovery time

Division criteria by voltage levels or by several bays



26/41


Fiber Optic Ethernet

Ring 100 Mbps

Ring of IEDs


Dashed Lines Redundant LAN

Connections

IEDs with Embedded Switch functionality

Multiple rings may be needed


HSR will be the alternative

HMI Gateway


27/41

27

Problem of Multicast and Physical vs.

Logical separation of Process Bus and

Station Bus


28/41


Problem of Multicast

Multicast is one-to-many communication scheme

Multicast MAC traffic is by default propagated

through the whole LAN

Consumes link bandwidth and increases latency at

switches

Introduces significant overhead at receiving IEDs ifmulticast addresses not allocated properly


29/41


Impact of Multicast Red MU (Merging Unit) multicasts Sampled Values to small group of IEDs

It is dictated by the protection application

In a large substation there can be dozens of IEDs sending multicast

GOOSE and dozens of Merging Units sending multicast Sampled Values

P

C

MU

C

NTP

P

MU

P

P P

C

MU

IEDIED

Primary Ring

Secondary Rings

P

C

MU

P

C


30/41


Impact of Multicast All nodes get the traffic red area

Repeat for every IED/MU in network

Critical messages delayed or maybe dropped

Steady state traffic load can exceed 100Mbps for many MUs Excessive MU traffic can cause IEDs and PCs can mis-operate or crash

P

C

MU

C

NTP

P

MU

P

P P

C

MU

IEDIED

Primary Ring

Secondary Rings

P

C

MU

P

C

Multicastmustbefiltered


31/41


Multicast Addresses and Traffic Management Efficient layer 2 multicast application

Proper allocation of multicast addresses

Filtering of multicast traffic Allocation of multicast addresses

improves processing times at receiving devices by discardingunwanted multicast traffic at hardware level

required for multicast filtering

Multicast filtering saves bandwidth and decreases latency at network switches by

limiting the traffic only to restricted areas of the network

Multicast filtering solves the primary problem of filtering unwanted

GOOSE and SV traffic Use VLAN or MAC address filtering ?

Static or dynamic filtering methods ?


32/41


Where we are today ?

In todays substations often no multicast

management used at all

Lack of knowledge at integrators and utilities Many users just tend to minimize configuration

efforts and rely on default settings

Until now the dominant method for restrictingmulticast traffic was the use of VLANs

Static configuration: manual process for all IEDs and

all network devices


33/41


Example of MisconfigurationCase Study 50 IEDs in the same network all sending GOOSE

No multicast filtering used Wrong!

All IEDs send multicast with the same destination multicast MAC address Wrong!

In case of event there is an avalanche of GOOSEs in the network and approx 20ms

additional processing delay observed at the receiver Improper functioning!

Implementation internals of an IED Network controller at IEDs has hash table that maps all possible multicast MACs to a

small group of addresses

Hash table permits discard unwanted multicast MACs at hardware level

If all IEDs send with the same multicast destination MAC then at receiving IED these are

mapped to the same hash and need to be discarded by software

In some IED implementations decoding of GOOSE message takes up to 1.5ms

Software decoding of 20 unwanted GOOSE messages can take up to 30ms!


34/41


VLAN (IEEE 802.1Q) Virtual LAN: an independent Ethernet network that shares

cabling infrastructure with other networks

Each VLAN has a separate broadcast domain VLANs permit:

Priority tagging

Logical separation of the network into various domains

Standard

FrameDest. Src. Length / Type Data

6 bytes 6 bytes 2 bytes Variable

Dest. Src. Length / Type Data

6 bytes 6 bytes 2 bytes Variable

TPID TCI

Priority CFI VID

2 bytes

3 bits 1 bit 12 bits

2 bytes

Tagged

Frame


35/41


Use of VLANs

VLAN is suitable mechanism for isolation of unrelated

traffic, eg. surveillance video from SCADA traffic

VLANs configuration can be: Static

Dynamic (GVRP)

Today static configuration is a manual process

Static configuration can be semi-automatic with future

enhanced configuration tools

Can use MAC address filtering instead of VLAN

VLANs for priority tagging in order to increasing

performance


36/41


Traffic Segregation with VLANs

Traffic separated with VLANs:

Substation LAN management

SCADA/Engineering Access

GOOSE Messages

Process Bus (Sampled Values)

Synchrophasors

Protection A vs. Protection B

Differenttrafficflowsinasubstationnetworkmerit

segregatingintoseparateVLANs


37/41


GMRP/MMRP for Dynamic Multicast Filtering Generic Multicast Registration

Protocol

Publisher / subscriber model like

IGMP Multicast filtered by default

must subscribe to get it

Adapts dynamically to anynetwork topology andaccommodates any application of

9-2 or GOOSE edge only pruning results in no

traffic delay after topology change

Allows process and station bus toco-exist on same physicalnetwork

P

C

MU

C

P

MU

P

P

Primary Ring

Secondary Rings

SV producer

simply multicasts no change

SV consumer sends a

subscribe message tonetwork periodically

Switches prune the trafficautomatically. Eitheroptimally or edge ony

FirstIEC61850110kVsubstationwithIEEE1588v2and

dynamicGMRPmulticastfilteringcommissionedin2010


38/41


Problem of Time Synchronization and Data Sharing

Process Bus requires that Sampled Values coming from different

sensors and received by an IED have to be synchronized

Synchronization islands are possible, each island spans a

protection zone

Problem of Line Differential protection with one line end using

Process Bus and the other line end using conventional wiring


39/41


Process Bus and Station Bus Separation

Process Bus and Station Bus are logically different

Multicast traffic from Merging Units flooding the network

A single Merging Unit consumes approx 5Mbps of bandwidth The problem of busbar protection based on Process Bus

In a topology with 60 feeders a process bus based busbar protection would

have multicast traffic of > 400Mbps!


40/41


Physically or Logically Separate Networks?

Physically separate LANs are more costly network switches are

duplicated

Physically separate LANs are perceived as more secure Logically separate LANs are more flexible as Merging Units can be

accessed from SCADA (remote maintenance, management, etc.)

Logically separate LANs require network engineering or more

sophisticated dynamic methods (GMRP, GVRP, etc.)

Station Bus could also be connected to Process Bus via router


41/41

41

Questions?

iec 61850 network architectures

Documents

substation iec

mediaconverters iec

network architecturesjuly

station bus5212018 iec

long distances5212018

typical network architectures4

copyright ruggedcom

electrostatic discharges