ieee 1588 time synchronisation and data flow …

POWER TRANSMISSION SUBSTATIONS
A thesis submitted to The University of Manchester for the degree of
Doctor of Philosophy
2020
CONTENTS
2
CONTENTS
1.1.1 Standardised substation secondary system based on IEC 61850 ................... 18
1.1.2 Obtaining time information for substations ................................................... 21
Research objectives ............................................................................................... 22
Introduction ........................................................................................................... 29
2.2.1 Use of time synchronisation in power systems .............................................. 30
2.2.2 Timing methods and challenges in delivering accurate time information ..... 31
Overview of IEEE 1588 ........................................................................................ 38
2.3.1 Clock types .................................................................................................... 39
2.3.4 Synchronisation principle .............................................................................. 44
2.3.5 PTP profiles ................................................................................................... 49
Summary ............................................................................................................... 61
Introduction ........................................................................................................... 63
Data flow in substation automation networks ....................................................... 69
3.3.1 Main traffic composition and characteristics ................................................. 69
3.3.2 Network architecture ..................................................................................... 72
Traffic management methods ................................................................................ 79
Literature survey of substation communication network performance studies ..... 82
3.7.1 Quantitative assessment of network performance using simulation tools ..... 83
3.7.2 Evaluation of maximum delay using analytical approaches .......................... 84
CONTENTS
3
3.7.4 Traffic management strategies to improve communication performance ..... 87
3.7.5 Issues of message delay / loss and proposed solutions .................................. 88
3.7.6 Other communication methods for data transfer within the same substation 90
Summary ............................................................................................................... 91
Testing Philosophy ........................................................................................................ 93
Testing methods .................................................................................................. 102
4.3.2 Data latency measurement ........................................................................... 107
4.3.3 Function and configuration of network impairment emulator ..................... 109
4.3.4 Network traffic injection.............................................................................. 111
Summary ............................................................................................................. 114
Assessment of IEEE 1588 for Precision Timing in IEC 61850
Substations 116
Introduction ......................................................................................................... 116
5.2.1 Long-term satellite visibility and synchronisation accuracy based on different
constellation selections ............................................................................................... 118
5.2.2 Transient characteristics of clocks during loss / restoration of time reference
123
Use of PTP in various sampled value based substation networks ....................... 127
5.3.1 Accuracy of PTP under various process bus architectures .......................... 127
5.3.2 Scalability of PTP in a chain connection ..................................................... 130
5.3.3 Synchronisation performance of PTP in complete substation networks ..... 133
5.3.4 Impact of PTP traffic from multiple time domains ..................................... 138
5.3.5 Slave clock characteristics during Follow_Up message delay .................... 140
Transient response analysis of PTP synchronisation systems ............................. 142
5.4.1 Effect of BMCA switchover ........................................................................ 142
5.4.2 Fault tolerance assessment of highly redundant system architecture due to
communication link loss ............................................................................................. 145
Protection Applications .............................................................................................. 149
6.2.1 Historical cyber-attack events ...................................................................... 150
6.2.2 Typical classification of attacks on IEC 61850 substation automation systems
151
6.2.3 Threats of time synchronisation attacks on protection and control applications
153
CONTENTS
4
6.3.1 Theoretical analysis of introduced time errors on PTP synchronisation
accuracy 154
6.3.3 Attack scenarios and results......................................................................... 159
Impact of time synchronisation errors on transformer differential protection .... 165
6.4.1 Mathematical derivation of time errors on transformer differential protection
scheme 167
6.4.2 Experimental validation using a multi-vendor IEC 61850 testbed .............. 173
Summary ............................................................................................................. 178
180
Challenges in evaluating network latency ........................................................... 184
Experimental evaluation of Ethernet switching behaviours ................................ 186
7.4.1 Laboratory testbed set up ............................................................................. 187
7.4.2 Characteristics of SV and GOOSE residence time in PRP / HSR enabled
switches 187
7.4.3 Transfer Latency of GOOSE in PRP and HSR process bus networks with
multi-vendor switches ................................................................................................. 191
7.4.4 Comparison of PTP message handling in one-step and two-step switches . 194
Visualisation of SV latency in a redundant substation network .......................... 198
7.5.1 Descriptions of test network ........................................................................ 198
7.5.2 Test on seamless transition of SV latency upon communication link failures
200
Conclusions ......................................................................................................... 209
8.2.1 Time synchronisation related studies ........................................................... 214
8.2.2 Network interactions related studies ............................................................ 216
References .................................................................................................................... 217
APPENDIX B Devices Category ............................................................................ 235
APPENDIX C Data For Transformer Differential Protection Operating
Characterises with Time Errors ................................................................................ 236
Word count: 57,450
LIST OF FIGURES
LIST OF FIGURES
Figure 1-1: Roadmap of power system protection development from early 1900s to present
day [5]. .................................................................................................................................. 19 Figure 1-2: Structure of IEC 61850 substation secondary systems [11]. ............................. 21 Figure 2-1: An example of dedicated timing methods used for synchronising various IEDs in
a substation automation network [36]. ................................................................................. 33 Figure 2-2: An example of networked timing methods (NTP or PTP) used for synchronising
various IEDs in a substation automation network [36]. ....................................................... 34 Figure 2-3: Basic working principle of time synchronisation in a packet-switched network
[43]. ...................................................................................................................................... 36 Figure 2-4: Block diagram illustrating how the residence delay is measured and appended to
the PTP message content in both one-step (a) and two-step (b) TCs [49], [50]. .................. 40 Figure 2-5: Block diagram illustrating the architecture of boundary clocks [50]. ............... 41 Figure 2-6: State transition of best master clock algorithm [20]. ......................................... 43 Figure 2-7: Delay request-response mechanism. .................................................................. 46 Figure 2-8: Peer delay request-response mechanism............................................................ 47 Figure 2-9: Designs of test platforms for measuring 1-PPS offset, (a) Oscilloscope [69], (b)
Measurement server fitted with a PCIe NIC card [20] ......................................................... 51 Figure 2-10: 24 hours 1-PPS pulse offset measurement with various mask angle settings [20].
.............................................................................................................................................. 52 Figure 2-11: Performance of time synchronisation using PTP under various topologies and
traffic levels [66]. ................................................................................................................. 57 Figure 3-1: Interface model representation according to IEC 61850-1:2013 [91]. PROT:
protection, CONTR: control and FCT: function. ................................................................. 65 Figure 3-2: Overall transfer time defined in IEC 61850-5 [12]. ........................................... 66 Figure 3-3: GOOSE transmission characteristic defined in IEC 61850-8-1 [33]. ................ 71 Figure 3-4: Ring structure in substation automation systems [100]. C: control, P: protection.
.............................................................................................................................................. 73 Figure 3-5: GOOSE retransmission coexistant with RSTP network reconfiguration [101]. 73 Figure 3-6: PRP network architecture [56]. .......................................................................... 75 Figure 3-7: PRP Frame compared with normal Ethernet Frame [100]. ............................... 75 Figure 3-8: HSR network architecture with an illustration of duplication discard mechanism
of multicast messages [56]. .................................................................................................. 76 Figure 3-9: HSR frame structure with or without VLAN tagging [56] ................................ 77 Figure 3-10: Network interactions at process bus containing SV, GOOSE and PTP traffic [95].
.............................................................................................................................................. 79 Figure 3-11: VLAN frame format compared with standard frame format [102] ................. 80 Figure 3-12: Arrival and service curves to determine delay bound (τMAX) and backlog bound
(QMAX) according to Network Calculus theory [137]. .......................................................... 85 Figure 3-13: Testbed arrangement for investigation SV latency in a PRP process bus network
[144]. .................................................................................................................................... 87 Figure 3-14: Concept of SV residence delay measurement [159] ........................................ 90 Figure 4-1: Schematic representation of the testbed architecture. ........................................ 94 Figure 4-2: Photo of the test system, (a) time synchronisation rack and monitoring system, (b)
protection system rack. ......................................................................................................... 96 Figure 4-3: Time servers used in the test systems. ............................................................... 97 Figure 4-4: Slave X and Slave Y used in the test systems. ................................................... 98 Figure 4-5: Ethernet switches used in the test systems. ........................................................ 98 Figure 4-6: MiCOM relay P643 used for the test systems [175]. ......................................... 99
LIST OF FIGURES
6
Figure 4-7: Merging units used in the test systems. ........................................................... 100 Figure 4-8: Traffic generator/ network emulator [178]. ..................................................... 100 Figure 4-9: Measurement server [179]. .............................................................................. 101 Figure 4-10: Real-time monitoring interface of the measurement server, showing the 1-PPS
deviation of different clocks compared with the reference signal. ..................................... 101 Figure 4-11: 10/100/1000BASE-T Tap used in the test system [180]. .............................. 101 Figure 4-12: Network Capture Card with hardware timestamping used in the test system [181].
............................................................................................................................................ 102 Figure 4-13: System set up for validating signal delay caused by attenuators. ................. 103 Figure 4-14: Signal delay comparison between normal 1-PPS with 1-PPS after attenuation.
Div: division. ...................................................................................................................... 104 Figure 4-15: Test setup for measurement server calibration; (a) GM A connected to the
reference and input channel 1; (b) GM A connected to the reference, and GM B connected to
input channels 1 and 2. ....................................................................................................... 105 Figure 4-16: 1-PPS delay variation, (a) between reference and input channel 1 (b) between
channel 1 and 2. .................................................................................................................. 105 Figure 4-17: Schematic diagram showing the PTP test network arrangement. .................. 106 Figure 4-18: Test setup for precisely measuring transfer latency. ...................................... 108 Figure 4-19: Pin assignment for making the cable to synchronise the Card (only pins 3 and 7
are used for carrying the pulse signals). ............................................................................. 108 Figure 4-20: High precision timestamp recorded by the capture card. ............................... 108 Figure 4-21: Experimental Setup for investigating PTP messages impairment on slave clocks.
............................................................................................................................................ 110 Figure 4-22: Configurations of emulator software interface to achieve message modification.
............................................................................................................................................ 110 Figure 4-23: Wireshark network capture showing the Sync message impairment by inserting
additional values in the correctionField. ............................................................................ 111 Figure 4-24: SV format with VLAN settings. .................................................................... 112 Figure 4-25: Captured SV traffic from Wireshark. ............................................................ 112 Figure 4-26: Traffic pattern of 79.8% at 100 Mb/s using a single SV stream. ................... 113 Figure 4-27: Traffic pattern of 79.5% at 100 Mb/s using multiple SV streams. ................ 113 Figure 4-28: Synthetic SV streams captured by Wireshark................................................ 114 Figure 5-1 Experimental setup to evaluate the effect of constellation selection on satellite
visibility and timing accuracy. ............................................................................................ 119 Figure 5-2 Long term performance of satellite receivers of satellite visibility. ................ 120 Figure 5-3 Satellite trajectory plot on different constellation selections (a) GPS, (b)
GLONASS, (c) GPS+GLONASS. ..................................................................................... 120 Figure 5-4 3D diagram showing antenna locations with tall building to the northeast. .... 121 Figure 5-5 Long term performance of synchronisation accuracy based on different
constellation selections (with reference to GM A) ............................................................. 122 Figure 5-6 1-PPS offset of Grandmaster B and Slave Y (synchronised to Grandmaster B via
PTP) compared to the reference 1-PPS. ............................................................................. 123 Figure 5-7: Experimental setup for investigating holdover capability and transient
characteristic of clocks. ...................................................................................................... 124 Figure 5-8: Clock holdover capability over a 24 hour monitoring period with reference to GM
A; (a) all clocks, (b) zoomed view for GM D and Slave Y. ............................................... 125 Figure 5-9: Resynchronisation of clocks under different time steps with (a) 1µs, (b) 5µs, (c)
10µs. ................................................................................................................................... 126 Figure 5-10: Experimental setup for investigating synchronisation accuracy under process
bus networks, (a) Star, (b) RSTP, (c) PRP, (d) HSR. ......................................................... 129 Figure 5-11: Sample distribution to compare the accuracy of PTP in different process bus
architectures. ....................................................................................................................... 130
7
Figure 5-12: System set-up for measuring the inaccuracy of transparent clocks. .............. 131 Figure 5-13: Pulse difference distribution for timing accuracy in a Star-connected network.
............................................................................................................................................ 132 Figure 5-14: Experimental setup for investigating synchronisation accuracy across substation
networks: (a) PRP over HSR, (b) RSTP over HSR and (c) HSR over HSR. ..................... 134 Figure 5-15: Synchronisation performance of Slave X and Y using highly redundant
topologies............................................................................................................................ 136 Figure 5-16: Synchronisation performance of Slaves X and Y in substation networks in the
presence of GOOSE and SV traffic. ................................................................................... 137 Figure 5-17: Synchronisation performance under multi-domain PTP traffic ..................... 139 Figure 5-18: Impact of multi-domain PTP traffic on synchronisation performance (with
reference to GM A). ............................................................................................................ 139 Figure 5-19: System arrangement to investigate the impact of Follow_Up Delay on PTP
synchronisation performance. ............................................................................................. 141 Figure 5-20: Synchronisation performance of (a) Slave X and (b) Slave Y under Follow_Up
delay impairment (with reference to GM A). ..................................................................... 142 Figure 5-21: Slave clock transient characteristics during BMCA switchover.................... 144 Figure 5-22: Fault tolerance of different highly redundant network topologies due to
communication link loss (with reference to GM B). .......................................................... 146 Figure 6-1: Time synchronisation attacks on a PTP system using E2E delay measurement
mechanism, (a) Delay attack performed on Sync messages, (b) Packet modification attack
performed on Sync messages. ............................................................................................. 155 Figure 6-2: Time synchronisation attacks on a PTP system using P2P delay measurement
mechanism, (a) Delay attack performed on Sync messages, (b) Packet modification attack
performed on Sync messages. ............................................................................................. 156 Figure 6-3: Experimental setup for performing delay attacks and packet modification attacks.
............................................................................................................................................ 157 Figure 6-4: System setup for excessive traffic inject attack. .............................................. 158 Figure 6-5: Testbed setup for PTP spoofing attack. ........................................................... 158 Figure 6-6: PTP delay attack in one-step mode (with reference to GM A). ....................... 159 Figure 6-7: Delay attack for P2P in two-step mode (with reference to GM A). ................ 160 Figure 6-8: Packet modification attack on both one-step and two-step (with reference to
GM A)................................................................................................................................. 161 Figure 6-9: Synchronisation performance of slave clocks under excessive traffic attack (with
reference to GM A). ............................................................................................................ 162 Figure 6-10: PTP spoofing attack with the aid of network impairment emulator (with
reference to GM A) ............................................................................................................. 163 Figure 6-11: Schematic diagram comparing conventional and IEC 61850 based transformer
differential protection scheme. ........................................................................................... 166 Figure 6-12: (a) Typical Bias Characteristic for transformer differential protection, (b) circuit
diagram to illustrate the measured current. 87T: ANSI code for transformer differential
protection ............................................................................................................................ 168 Figure 6-13: Theoretical analysis of differential protection under external faults with time
synchronisation errors. ........................................................................................................ 170 Figure 6-14: Theoretical analysis of the actual restraint curve with synchronisation errors.
............................................................................................................................................ 173 Figure 6-15: Experimental setup for transformer differential protection scheme that uses
sampled value process bus. ................................................................................................. 174 Figure 6-16: Working principle of the Emulator to control the time offset. ...................... 175 Figure 6-17: Performance of differential protection under external faults with added time
errors, (a) full range, (b) zoomed view. .............................................................................. 176
LIST OF FIGURES
8
Figure 6-18: Effect of time error on the operating characteristics compared with the restraint
curve. .................................................................................................................................. 178 Figure 7-1: Transfer latency and Network latency requirement. ........................................ 182 Figure 7-2: Overview of composition of transfer latency. ................................................. 184 Figure 7-3: Challenges in evaluating network latency due to increasing network sizes, various
ports speeds, introductions of various redundancy protocols and complex traffic interactions.
............................................................................................................................................ 186 Figure 7-4: Test bed set up for measuring the network latency. ........................................ 187 Figure 7-5: Illustration of test scenarios for investigating switching behaviours. .............. 188 Figure 7-6: illustration of relationship between packet size and combined delay, (a) Switch H,
(b) Switch R. ....................................................................................................................... 190 Figure 7-7: Analysing average network latency for GOOSE messages (409 bytes) under PRP
or HSR architectures. The type of port used for connecting the switches is shown in the
brackets. .............................................................................................................................. 192 Figure 7-8: Data flow path and latency measurement of SV message stream in HSR-HSR
topology under various communication loss conditions. ................................................... 200 Figure 7-9: Website based configuration interface of Redboxes ........................................ 201 Figure 7-10: Purpose-made serial cable for setting up data connection between switches and
workstation. ........................................................................................................................ 202 Figure 7-11: Command Line Interface console window for switch configuration via serial
connection. .......................................................................................................................... 202 Figure 7-12: Preliminary test results comparing different methods that simulate
communication link failures. .............................................................................................. 203 Figure 7-13: Comparison of network traffic between CLI console and website based interface.
............................................................................................................................................ 204 Figure 7-14: Visualisation of real-time SV data latency measurement in HSR-HSR network
during various communication link losses and traffic conditions. ..................................... 205 Figure 8-1: Test setup by Endace to examine the PTP synchronisation accuracy in the WAN,
considering asymmetrical link delays, heavy loading conditions and contention (collision)
[222]. .................................................................................................................................. 214
LIST OF TABLES
Table 2-1: Time synchronisation classes specified in IEC 61850-5 [12]. ............................ 32 Table 3-1: Transfer time classes and corresponding requirement according to IEC 61850-5
[12]. ...................................................................................................................................... 67 Table 3-2: Examples of recovery time requirement [12]...................................................... 69 Table 3-3: Summary of main process bus traffic and transmission time requirement, modified
based on the published data in [96] and IEC 61850-5 and IEC 61850-90-4 [12], [97]. ....... 72 Table 4-1: Voltage level after attenuation. The colour coding means the voltage level of
attenuated 1-PPS signal is acceptable. ................................................................................ 104 Table 4-2: Statistics of measurement error of 1-PPS measurement server......................... 105 Table 4-3: Statistical of measures of error between two identical Sync messages ............. 109 Table 5-1 24-hour synchronisation accuracy of GM B under different constellations (with
reference to GM A) ............................................................................................................. 122 Table 5-2 Summary of synchronisation accuracy in process bus networks. ...................... 130 Table 5-3 Summary of synchronisation accuracy in a Star-connected network. ............... 133 Table 5-4 Summary of traffic type and bandwidth ............................................................. 135 Table 5-5 Summary of synchronisation accuracy in a highly redundant substation
architectures. ....................................................................................................................... 137 Table 7-1: Summary of residence time of for SV and GOOSE in different test scenarios. 189 Table 7-2: Summary of the combined delay in different test scenarios. ............................ 189 Table 7-3: Relationship between residence time and packet size for both switches and
estimate residence time for 409 bytes GOOSE. ................................................................. 193 Table 7-4: Comparison between test and estimated residence time for 409 byte GOOSE
across PRP and HSR process bus networks. ...................................................................... 194 Table 7-5: Summary of residence time of Sync messages in one-step TC (Switch R) under
various traffic conditions. ................................................................................................... 196 Table 7-6: Summary of residence time of Follow_Up messages in one-step TC (Switch R)
under various traffic conditions. ......................................................................................... 196 Table 7-7: Summary of residence time of Sync messages in two-step TC (Switch M) under
various traffic conditions. ................................................................................................... 197 Table 7-8: Summary of residence time of Follow_Up messages in two-step TC (Switch M)
under various traffic conditions. ......................................................................................... 197 Table 7-9: Summary of SV data latency measurement in HSR-HSR network during various
communication link losses and traffic conditions. ............................................................. 206 Table A-1: Comparison Summary of main differences between PTP profiles that are used for
substation automation systems ........................................................................................... 233 Table B-1: List of equipment in the laboratory testbed ...................................................... 235 Table C-1: Mathematical calculation of transformer differential protection operating
characteristics under 250 µs time synchronisation error. ................................................... 236 Table C-2: Mathematical calculation of transformer differential protection operating
characteristics under 1000 µs time synchronisation error. ................................................. 242
LIST OF ABBREVIATIONS
9-2 LE IEC 61850-9-2 Light Edition
A/D Analogue-to-Digital
b/s Bit Per Second
Base Profile IEC/IEEE 61850-9-3 Precision Time Protocol Profile For Power
Utility Automation
CLI Command Line Interface
DFR Digital Fault Recorders
DoS Denial of Service
GM Grandmaster
GPS Global Positioning System
HMI Human Machine Interface
HSR High-Availability Seamless Redundancy
IEC International Electrotechnical Commission
IED Intelligent Electronic Device
IFG Inter-Frame Gap
IP Internet Protocol
ISG Inter-Stream Gap
IT Information Technology
NIC Network Interface Controller
NIST National Institute of Standards And Technology
NTP Network Time Protocol
OMNeT++ Objective Modular Network Testbed In C++
OPNET Optimized Network Engineering Tool
OSI Open System Interconnection
OTN Optical Transport Network
P2P Peer-to-Peer
PDC Phasor Data Concentrators
PDV Packet Delay Variation
PLC Programmable Logic Controllers
PMU Phasor Measurement Unit
Power Profile v1 IEEE C37.238-2011 Standard Profile For Use Of IEEE 1588
Precision Time Protocol In Power System Applications
Power Profile v2 IEEE C37.238-2017 Standard Profile For Use Of IEEE 1588
Precision Time Protocol In Power System Applications
PRP Parallel Redundancy Protocol
PTP Precise Time Protocol
Synchronization Protocol For Networked Measurement And
Control Systems
PTP v2
Control Systems
PTP v2.1
Control Systems
RX Receive
SDH Synchronous Digital Hierarchy
SDN Software Defined Networking
SFD Start Frame Delimiter
SONET Synchronous Optical Networking
TDoS Telephony Denial-of-Service
UDP User Datagram Protocol
UPS Uninterruptible Power Supply
UTC Coordinated Universal Time
VT Voltage Transformer
WAN Wide Area Network
ABSTRACT
13
ABSTRACT
IEEE 1588 Time Synchronisation and Data Flow Assessment for IEC 61850 based Power
Transmission Substations
Degree: Doctor of Philosophy Date: Dec 2020
Substation automation systems (SAS) in accordance with IEC 61850 are gaining popularity
in the power industry and is expected to dominate the design of substations worldwide in the
near future. The standard offers a ‘future-proof’ solution for the protection, automation and
control (PAC) systems with cost reduction in design, operation and maintenance. IEEE 1588
Precision Time Protocol (PTP) is widely considered for use in IEC 61850 based substations
due to its achievable sub-microsecond accuracy over Ethernet, which satisfies the accuracy
required by sampled value (SV) process bus and other advanced PAC applications. However,
PTP is still not considered by many utilities to be a fully-mature technology for reliable time
distribution in substations. The propagation delay variations of PTP messages caused by
network traffic are sometimes suspected of resulting in unacceptable timing errors. Substation
engineers also require sufficient understanding of the PTP timing principle and the
characteristics of network devices to operate and maintain the system. Additional
investigations are needed urgently to study the characteristics of communication network
behaviours in the presence of all principle IEC 61850 traffic. Confidence must be given to
the system operators that the intrinsic data flow interactions in the well-designed substation
automation system does not degrade the performance of either time synchronisation or other
PAC applications.
This research focuses on the use of PTP for precision timing, and interactions between multi-
protocol substation traffic. A hardware test platform consisting of commercially-available
products (e.g. clocks, Ethernet switches, merging units and protection relays) was designed
and built to explore the performance both at the individual component level and at the
complete substation automation system level. Testing tools including a traffic generator /
network impairment emulator, a time synchronisation measurement server, an OMICRON
test set, a network capture card and an Ethernet tap were integrated into the test platform to
simulate several network conditions in both electrical and data communication systems.
These network conditions were critical to fully understand the network characteristics and
identify performance boundaries.
The results systematically demonstrate that it is feasible to implement PTP as an alternative
for conventional dedicated timing systems. PTP can be used in conjunction with seamless
redundancy protocols as defined in IEC 62439-3 to enhance the system reliability while still
maintaining a ±1 µs accuracy requirement. Meanwhile, care must be taken with system
vulnerabilities discovered under time synchronisation attacks. Delay, modification, denial of
service and systematic spoofing attacks can result in deterioration of synchronisation
accuracy. A transformer differential protection scheme was selected as an example to
evaluate, theoretically and experimentally, the effect of inadequate time synchronisation on
operating characteristics. The transfer latency of various process bus data including PTP, SV
and Generic Object Oriented Substation Event (GOOSE) is studied using in-service Ethernet
switches supporting seamless redundancy protocols. The obtained data transfer
characteristics can be used to enhance the accuracy of latency estimation.
Using the hardware testbed with “live” substation equipment discovers factors that often
struggle to be found in simulation and analytical studies. The findings of this research can be
used as a performance reference which is valuable for various stakeholders including the
standard board, power utilities, manufacturers and other academic researchers.
DECLARATION
14
DECLARATION
No portion of the work referred to in the thesis has been submitted in support of an application
for another degree or qualification of this or any other university or other institute of learning.
COPYRIGHT STATEMENT
COPYRIGHT STATEMENT
i. The author of this thesis (including any appendices and/or schedules to this thesis) owns
certain copyright or related rights in it (the “Copyright”) and s/he has given The University
of Manchester certain rights to use such Copyright, including for administrative purposes.
ii. Copies of this thesis, either in full or in extracts and whether in hard or electronic copy,
may be made only in accordance with the Copyright, Designs and Patents Act 1988 (as
amended) and regulations issued under it or, where appropriate, in accordance Presentation
of Theses Policy You are required to submit your thesis electronically Page 11 of 25 with
licensing agreements which the University has from time to time. This page must form part
of any such copies made.
iii. The ownership of certain Copyright, patents, designs, trademarks and other intellectual
property (the “Intellectual Property”) and any reproductions of copyright works in the thesis,
for example graphs and tables (“Reproductions”), which may be described in this thesis, may
not be owned by the author and may be owned by third parties. Such Intellectual Property
and Reproductions cannot and must not be made available for use without the prior written
permission of the owner(s) of the relevant Intellectual Property and/or Reproductions.
iv. Further information on the conditions under which disclosure, publication and
commercialisation of this thesis, the Copyright and any Intellectual Property and/or
Reproductions described in it may take place is available in the University IP Policy (see
http://documents.manchester.ac.uk/DocuInfo.aspx?DocID=2442 0), in any relevant Thesis
restriction declarations deposited in the University Library, The University Library’s
regulations (see http://www.library.manchester.ac.uk/) and in The University’s policy on
Presentation of Theses.
ACKNOWLEDGEMENT
16
ACKNOWLEDGEMENT
First and foremost, I am immensely grateful to my supervisors, Prof. Peter Crossley and Prof.
Zhongdong Wang, and advisor, Dr. Qiang Liu, for their technical guidance, invaluable advice,
insightful comments, and continuous support during my PhD study. I would not be able to
come this far without their encouragement and inspiration.
I would like to acknowledge EPSRC Centre for Doctoral Training in Power Networks and
The University of Manchester for sponsoring my PhD study. I would like to thank CIGRE
UK B5 committee for accepting me as a UK technical member to contribute to CIGRE
International Working Group B5/D2.67.
I would like to express my special thanks to Dr. Hao Guo, Dr. Yucong Zhao, Dr. Wentao Zhu,
Dr. James Hill in Ferranti Building and Mr. Sajjid Salyani from Siemens plc for co-authoring
the papers. My sincere gratitude also goes to Mr. Stephen Potts, Dr. Nan Liu, Prof. Xiaolong
Chen, Dr. Hasan Uppal and many other colleagues and friends in Ferranti for their advice,
encouragement and support to help me ride through many stressful moments.
I would like to thank Siemens plc, UK for kindly providing Ethernet switches for this project,
and Hirschmann, Tekron, Oregano Systems, and Meinberg for their prompt and valuable
technical discussions.
As one of my proudest achievements alongside my PhD study, I am very grateful to former
Chairmen Dr. James Hill and Mr. Trevor David for their leadership and foundation in the
IEEE PES Student Branch Chapter, and to Dr. Angeliki Loukatou and Ms. Siwei Liu for their
immense contribution during my chairmanship of the committee. Those prestigious awards
we have won will never be forgotten.
Last but not least, I would like to express my deepest gratitude to my family, especially my
parents and my wife, Mrs. Tong Yu. Their understanding, patience and unconditional love
are my greatest driving force to the perfection. The sacrifices Tong made for travelling
thousands of miles to Manchester after six years of long distance relationship to accompany
me to complete my PhD study will always be appreciated from the bottom of my heart. I hope
my accomplishment will make my family proud.
LIST OF PUBLICATIONS
LIST OF PUBLICATIONS
[1] M. Han, H. Guo, and P. Crossley. "IEEE 1588 time synchronisation performance for IEC
61850 transmission substations." International Journal of Electrical Power & Energy
Systems 107 (2019): 264-272.
[2] W. Zhu, M. Han, J. V. Milanovic, and P. Crossley. "Methodology for Reliability
Assessment of Smart Grid Considering Risk of Failure of Communication Architecture."
IEEE Transactions on Smart Grids. vol. 11, no. 5, pp. 4358-4365, Sept. 2020, doi:
10.1109/TSG.2020.2982176.
[3] M. Han, P. Crossley, S. Salyani, “Evaluation of Process Bus Data Latency with
PRP/HSR Enabled Ethernet Switches,” in The 15th International Conference on
Developments in Power System Protection (DPSP) 2020.
[4] M. Han and P. Crossley, "Vulnerability of IEEE 1588 under Time Synchronization
Attacks," 2019 IEEE Power & Energy Society General Meeting (PESGM), Atlanta, GA,
USA, 2019, pp. 1-5, doi: 10.1109/PESGM40551.2019.8973494.
[5] M. Han, Y. Zhao, and P. Crossley, “Impact of Time Synchronisation Errors on
Transformer Differential Protection that uses the IEC 61850 Process Bus,” in CIGRE
Chengdu Symposium, 2019.
[6] M. Han, and P. Crossley, “Performance Evaluation of IEEE 1588 for Precision Timing
in IEC 61850 Substations,” in CIGRE Colloquium 2019, Tromso Norway.
[7] M. Han, J. Hill, Z. Wang, and P. Crossley, “Thermal Evaluation of Railway Transformer
Used in Autotransformer Feeding Systems,” in IEEE International Energy Conference
(ENERGYCON), 2018.
1.1.1 Standardised substation secondary system based on IEC 61850
Substations play a vital role in the electrical power system as they not only convert voltage
levels but also provide switching and protection functions. Inside conventional substations,
protection and control devices are connected point-to-point using hardwired systems, and
signals such as measurements, signals and commands traverse over copper wires to the
control rooms [1]. With growing electricity demand and increasing voltage levels for
transmission, the size of substations increases to accommodate bulkier primary equipment
and the associated secondary system devices [2]. This implies substations which opt for
conventional schemes will require larger cabling systems, raising costs and creating
difficulties in installation and maintenance.
Meanwhile, communication used for substation protection, automation and control (PAC)
system has seen technical advances over the past century; and the world has seen a dramatic
evolution in how the electricity grid is operated [3]. At the beginning of 1900s, protection
was realised using electro-mechanical based relays. During this time, information such as line
loading and other operational commands can only be communicated via telephone [4]. Later
in 1930s, some remote control devices that could communicate via telephone-switching
network became available to read grid status and realise control operations [4]. With the
application of electronic and computer technologies after 1960s, processing capability and
communication bandwidth is no longer a limiting factor for PAC systems. Hence, protection
CHAPTER 1: INTRODUCTION
relays have enjoyed a quick development, from electro-mechanical to integrated circuit and
micro-computer based technologies [5].
This blooming of communication capability was accompanied by various theoretical
breakthroughs in protection and control algorithms. A roadmap of the development of
protection systems is illustrated in Figure 1-1. With the integration of communication services,
modern protection schemes are not restrained to overcurrent, distance or differential
protection. Centralised, adaptive, or even artificial-intelligent based protection algorithms
have been introduced since the 1970s. Furthermore, conveying information beyond the
substation domain using wired or wireless technologies unlocks more opportunities for
advanced wide-area PAC in future power systems [6].
Power system operation is in constant evolution, which creates new challenges for PAC
systems. Countries, such as the UK, are committed to reduce carbon emissions, and the UK
National Grid aims to achieve a net-zero emission electricity system by 2025 [7]. The
increased penetration of renewable energy resources reduces system fault levels and poses
challenges to conventional protection relays [8]. One of the key solutions is to have a better
system status awareness with faster fault identification and clearance, which cannot be
achieved without fast, secure and reliable communication.
Figure 1-1: Roadmap of power system protection development from early 1900s to present
day [5].
Traditionally, many communication infrastructures in substations remained proprietary.
However, the life-time of secondary systems is significantly shorter than their associated
primary plants. For example, micro-processor based numerical relays are expected to have
in-service lifetime of around 15 years, compared to 40-70 years for the primary plant.
Moreover, due to the rapid development of information and communication technology (ICT),
future renewal of substation secondary infrastructure becomes a huge issue as the newly
added devices may not be compatible with existing ones [9]. Consequently, the power
industry required a standardised infrastructure that could provide cheap, reliable,
interoperable and sustainable communication for delivering and managing information [10].
IEC 61850 was introduced by International Electrotechnical Commission (IEC) Technical
Committee 57 as a guideline to standardise the engineering process in order to enable
intelligent electronic devices (IEDs) from various vendors to communicate with each other.
The standard proposes a local area network (LAN) using Ethernet to replace the hardwired
cabling system. IEC 61850 defines three levels in the substation automation system (SAS)
architecture, namely process level, bay level, and station level as shown in Figure 1-2. The
data exchange between different levels is realised using so called station bus and process bus.
In an IEC 61850 based substation that uses the process bus, the analogue voltage and current
measurement signals are digitised as sampled values (SVs). These digital measurement values
are transmitted via optic fibres in the substation secondary Ethernet communication networks,
which can be subscribed to by the IEDs for various protection and control applications.
Process buses increase the flexibility and functionality of the substation secondary systems
and make future expansion and retrofitting easier to implement [9]. Ethernet communication
offers immediate reduction of cabling cost, and removes the safety hazards associated with
having open circuits connected to instrument transformers in the relay control room [1]. In
addition, the standardised system allows utilities to buy products from different vendors with
“plug and play” features. Furthermore, availability of voltage / current measurements is
greatly enhanced by multicasting over Ethernet compared to point-to-point connection in the
conventional substations. This is more suitable for implementing advanced PAC functions.
Figure 1-2: Structure of IEC 61850 substation secondary systems [11].
1.1.2 Obtaining time information for substations
Although IEC 61850 provides standardised information exchange between PAC devices,
without having a common notion of time, a subscribing device cannot correctly coordinate
the sequence of events to reproduce the actual power system status. Many applications require
time synchronisation in order to achieve the desired performance, e.g. data acquisition units
such as merging units (MUs), transmission of Generic Object Oriented Substation Events
(GOOSE) messages, protection relays, and fault recorders. Losing time synchronisation may
affect the accuracy of fault diagnosis, decision making, and automatic PAC devices
operations. The level of required accuracy depends on applications which are defined
comprehensively in IEC 61850-5 [12].
The methods of obtaining time information by substations have been adapted to the
development of the available technology. As digital technology progressed, it was recognised
that information recorded at different sites would be more valuable if they were synchronised
to a common time [13]. Historically, many countries used territorial wireless radio
transmitters to synchronise multiple substations with broadcasting time codes. For example,
22
Rugby-MSF, maintained by the National Physical Laboratory, was used to cover the entire
UK with achievable accuracy of around ±1 ms [14]. The signal is transmitted continuously at
60 kHz. However, such low frequency signals suffers from interference, particularly close to
high voltage plant [13].
Since the early 1990s, distributed Global Positioning System (GPS) receivers have been
deployed in the power industry for more accurate time synchronisation [15]. Clocks with
built-in GPS receiver chips can obtain timing accuracy better than ± 100 ns [16]. There are
also other types of Global Navigation Satellite Systems (GNSS) available such as GLONASS,
Galileo and BeiDou. It is possible to have a signal receiver that is synchronised to multiple
GNSS signals to enhance satellite visibility. More recently, the Optical Transport Network
(OTN) is considered to transmit highly accurate time information across a large geographic
area to provide additional time reference sources at substations [17]. Integrating such wide-
area time references helps detect intentional or unintentional GNSS signal interference and
improves overall reliability [18].
High accuracy time distribution inside substations is traditionally achieved via dedicated
cabling systems carrying one-pulse-per-second (1-PPS) or inter-range instrumentation -
Group B (IRIG-B) time code. Only very few devices equipped with GNSS receivers may be
able to get directly time infeed from roof-mounted antennae. However, the recently
introduced IEEE 1588 Precision Time Protocol (PTP) offers a much simpler solution by
transmitting time over Ethernet with an accuracy better than ± 1 µs [19]. Benefits of PTP can
be found in both technical and economic aspects and these will be extensively reviewed in
Chapter 2. Using PTP as the primary time synchronisation method in conjunction with a small
number of 1-PPS / IRIG-B systems for legacy devices has become a trend for future
implementation of IEC 61850 based substations [20].
Research objectives
The power system of today is becoming more complex and dynamic, requiring more
advanced PAC applications to ensure reliable operation of the power grid. This requirement,
together with computer and communication technology development, foster the birth of
IEC 61850 [21]. Standardised PAC systems based on IEC 61850 help to maintain flexibility
and functionality in the long run [9]. Transmitting time information across the substation
using dedicated timing systems based on legacy 1-PPS / IRIG-B is no longer an economical
and future-proof solution. PTP entered the time synchronisation arena in 2002 and the power
23
industry has shown interest in adopting these technologies to take the advantage of Ethernet
communication systems while achieving sub-microsecond accuracy.
However, the deployment of PTP remains slow. Currently in the UK, PTP is only being
applied at the Wishaw 275 kV substation in Scotland [22]. Transmitting time information
over Ethernet is still not considered to be fully reliable and secure by some utilities; the fear
of not having tangible point-to-point connections as with the conventional dedicated timing
methods makes the power industry hesitant in moving to full Ethernet communication. More
importantly, the in-service performance of PTP in substation Ethernet systems is still largely
unknown. The achievable accuracy with various substation communication system structures
and traffic conditions remains to be answered. In particular, seamless redundancy protocols
based on IEC 62439-3, namely the Parallel Redundancy Protocol (PRP) and the High-
availability Seamless Redundancy (HSR), are widely considered in substation secondary
system design as they offer zero-failover time. The performance assessment of PTP in
complex substation network architectures formed from PRP and HSR are rarely seen in
literature.
Additionally, interruption of satellite signals or PTP message transmission is likely to cause
degradation of synchronisation accuracy. More investigations need to be carried out under
network contingency situations to reveal the transient performance of commercially-available
PTP products. A comprehensive understanding of the system dynamic behaviours is essential
to better design the substation automation system architecture and select the suitable products
to optimise the timing performance.
Furthermore, cyber threats to the PTP system are emerging as the interconnected Ethernet
system inevitably provides more access points to malicious attackers. This results in security
concerns over time synchronisation systems based on PTP. To identify the vulnerability, the
real-time response of individual devices and the entire system during various forms of attacks
must be assessed in detail. Further again, recognition of the impact of resultant time errors on
PAC applications will be crucial for the IEC 61850 substations to defend against potential
time synchronisation attacks.
On the other hand, the packet transmission latency over Ethernet is non-deterministic, and is
mainly governed by several factors such as communication bandwidth, network architecture
and queuing delays induced by background traffic. Some time-critical messages, e.g. SV and
GOOSE, require fast transmission time to meet the performance requirement specified in IEC
61850-5. However, the latency requirement might not be achieved in a large substation
congested by multi-protocol messages. Delays or even failures in delivering time-critical
messages may lead to degradation of PAC performance. Integration of PTP into the
communication systems brings additional network traffic, making the traffic interactions
much more complex. End-to-End (E2E) transfer latency is not a new research topic but most
previous studies were performed using simulations and analytical models. The results are
normally as good as the parameters and network conditions that was assumed. The lack of
realistic data often leads to large errors between simulation and testing. A real-time
assessment of transfer latency of various substation traffic is urgently required to fully explore
the traffic interactions in IEC 61850 substations.
The ultimate goal of this project is to help the power industry gain insights into PTP time
synchronisation and data flow characteristics in multi-function substation communication
networks. Results presented in this thesis can be used to provide technical recommendations
for designing a more reliable and secure timing architecture for IEC 61850 based substations.
Specific objectives are formed as follows:
To design and build a multi-vendor hardware testbed for investigating the PTP
synchronisation accuracy in IEC 61850 substations with various loading conditions
caused by background traffic.
To evaluate multiple time synchronisation architectures for a reliable time
distribution across process bus and complete substation networks.
To understand the PTP time synchronisation performance under steady-state
conditions and investigate dynamic behaviours of the timing system under
contingency situations.
To identify the vulnerability and dynamic responses of PTP system under malicious
time synchronisation attacks.
To analyse the dependency of time synchronisation on PAC applications, e.g.
transformer differential protection.
To characterise traffic interactions between PTP and other message protocols in the
same network, and analyse the transfer latency of various process bus traffic (e.g. SV,
GOOSE and PTP).
25
Contributions
The main contributions of this research are summarised in the following bullet points. The
research outputs have also been published in various journals and international conferences.
Design and build a multi-vendor hardware testbed based on IEC 61850 and PTP
with various testing methods
The innovative testbed described in Chapter 4 simulates a small section of an IEC
61850 transmission substation with PTP being the primary time synchronisation
method. This testbed exceeds the capability of most conventional laboratory facilities
by combining PTP time synchronisation with IEC 61850 process bus functions in a
signal platform. Integrating multi-vendor devices (e.g. clocks, Ethernet switches,
protection IEDs and MUs) into the testbed also provide a rare opportunity to evaluate
the system interoperability and cross-compare the performance of individual devices.
In addition, various testing methods, e.g. 1-PPS offset comparison, network traffic
injection, network impairment emulation, and precise transfer latency measurement
and secondary voltage / current injection from Omicron test set, have been designed
and utilised for the research. These methods allow the experiment to be conducted in
a controlled and repeatable way so that the in-service behaviours of individual
component or the entire systems can be studied thoroughly.
Comprehensive real-time performance evaluation of PTP time synchronisation
Substantial experiment has been carried out to evaluate the real-time PTP time
synchronisation in the substation Ethernet networks in Chapter 5. The uniqueness of
this contribution compared to the existing literature is the use of GNSS in conjunction
with IEC 62439-3 seamless redundancy protocols to enhance the overall PTP system
reliability. Meanwhile, the achievable synchronisation accuracy of PTP in digital
substations from station level (time receivers) to process level (slave clocks) is
investigated in detail, covering both steady-state and transient scenarios. The level of
comprehensiveness and technical depth are very rare to be found in the published
literature.
Identification of vulnerability of PTP system under time synchronisation attacks
The vulnerability analysis described in Chapter 6 is one of the pioneering studies of
cyber-security threats existed in PTP systems under time synchronisation attacks. The
impact of attacks is firstly theoretically analysed with mathematical derivations. After
that, several attack scenarios have been performed on commercial PTP devices from
different manufacturers. Therefore, the real-time response of PTP to the time
26
synchronisation attacks can be revealed. This study also takes into account the PTP
system configurations, which are applicable to the digital substations so that the
results can be directly beneficial to the power industry and fill the research gaps.
Some dynamic characteristics of PTP system during real-time attacks are valuable to
many stakeholders (utilities, manufacturers, standard boards), which cannot be easily
foreseen using simulation or mathematical analysis.
Investigations of the impact of inadequate time synchronisation on transformer
differential protection
The dependency of transformer differential protection on synchronisation accuracy is
investigated both theoretically and experimentally in the second half of Chapter 6. In
actual substations, inadequate time synchronisation may result from communication
network contingencies or even malicious cyber-attacks. The theoretical analysis
presented in this research reveals how the actual bias operating characteristics will
deviate from the relay setting due to introduced phase errors. Moreover, a novel
method is proposed to precisely control the time deviation between MUs in the
testbed. Hence, the impact of time errors on the protection operation can be quantified.
This method is also applicable to other studies that require the creation of time
synchronisation errors on PAC devices.
Assessment of transfer latency of process bus traffic in multi-protocol substation
automation networks
This research carries out a unique analysis of the characteristics of PRP/HSR enabled,
PTP-aware switches handling SV, GOOSE and PTP data latency. Almost all existing
simulation and analytical methods for latency estimation lack the representation of
actual devices, making them less accurate to be used in the real substation studies.
The experimental results (presented in Chapter 7) obtained based on the
commercially-available switches from different manufacturers greatly enhance the
accuracy of latency estimation. In addition, the real-time measurement of SV latency
in a state-in-the-art highly redundant substation network provides insights of data
interactions in a multi-functional substation communication network, which has not
been seen previously in other literature.
Thesis outline
This thesis contains eight chapters and the linkage is showed as follows.
Chapter 1 introduces the background, discusses the objectives and summarises the main
contributions of this research.
Chapter 2 provides an overview of high accuracy time synchronisation in substation
automation systems. Time distribution methods including dedicated timing (1-PPS and IRIG-
B time code) and networked timing (Network Timing Protocol and PTP) are reviewed and
compared. A comprehensive review of PTP is also presented covering clock types,
grandmaster clock election, delay measurement mechanism and various PTP profiles. A
literature survey is produced on PTP performance evaluations to help identify the current
research gaps.
Chapter 3 reviews substation communication networks based on IEC 61850. The chapter
first discusses the functions, message types, and communication requirements of IEC 61850.
This is followed by an introduction of traffic composition and data flow characteristics in the
substation communication networks. This chapter also addresses the increasing demand of
system redundancy by going through some commonly used redundant structures applicable
to IEC 61850 substations. In addition, the challenges of analysing sophisticated traffic
interactions are discussed, highlighting the importance of having a well-designed network
with appropriate traffic management methods. A literature survey is provided on the previous
performance studies of substation communication systems to conclude this chapter.
Chapter 4 focuses on the design and implementation of the laboratory test systems with
various testing philosophies. The architecture of the testbed is presented and individual
devices are described in detail. The experimental methods applied in this research, including
timing accuracy measurement, network impairment and traffic injection, are also discussed.
Chapter 5 presents the design and performance evaluation for using GNSS and PTP in
transmission substations. The content of this chapter forms two published items, [1] and [6]
in the List of Publications. The results presented in this chapter provides a comprehensive
overview of the synchronisation accuracy achievable with commercially available devices in
various network topologies and network conditions. This chapter also examines the transient
behaviours of grandmaster clocks, and investigates the feasibility of using multi-GNSS
constellation grandmaster clocks to deliver a more reliable time reference.
28
Chapter 6 considers the vulnerability of PTP networks and the impact of inadequate time
synchronisation accuracy on transformer differential protection. The content of this chapter
forms two published items, [4] and [5] in the List of Publications. This chapter first illustrates
the potential threats that exist in PTP timing systems using mathematical analysis; the impact
of delay attacks and modification attacks are quantified. A test system is designed and
deployed to perform various “mimic” time synchronisation attacks, which identifies the
characteristics of PTP devices under attack. After this, a theoretical analysis and experimental
validation of the impact of time synchronisation deviations on transformer differential
protection are discussed. The experiment is performed using a hardware testbed which
integrates PTP timing functions into the testing of transformer differential protection.
Chapter 7 addresses the process bus data latency in a multi-protocol traffic environment. The
content of this chapter forms two published items, [2] and [3] in the List of Publications. The
chapter starts with an overview of the composition of transfer latency and current challenges
in the latency evaluation studies. An assessment of the network latency of SV, GOOSE and
PTP is carried out using in-service Ethernet switches from different manufacturers. A real-
time latency measurement of SV messages across a HSR-HSR network is presented to
visualise the seamless transitions of SV transfer during network contingencies.
Chapter 8 concludes the findings of this research and provides an outlook of future research
directions on both time synchronisation and network related topics.
CHAPTER 2: TIME SYNCHRONISED SUBSTATION AUTOMATION NETWORKS USING IEEE 1588
29
Introduction
It is a growing trend to roll out IEC 61850 and PTP in future substation communication
networks. Time synchronisation, data transmission, protection and control can all be achieved
in the same Ethernet network. The IEC 61850 family standardises the engineering process
such that devices in substations can communicate with each other, regardless of their make,
model or manufacturer. IEEE 1588 has become part of IEC 61850 standard series as
IEC/IEEE 61850-9-3 for power utility automation applications. The “plug and play” features
that come with IEC 61850 standard series also provide confidence to the power industry that
the automation systems of today can cope with changing requirements and technology
evolutions of the future.
This chapter provides a comprehensive overview of highly accurate time-synchronised
substation networks based on PTP. Use of timing synchronisation methods with a particular
emphasis of implementation challenges are given in Section 2.2. An overview of PTP is
presented in Section 2.3, including clock types, one- or two-step operation modes, delay
measurement mechanisms and the configuration / performance guidance based on various
PTP profiles. After that, Section 2.3.6 provides a review of the literature on experimental
evaluation on PTP performance conducted by various researchers. Factors that can contribute
to synchronisation inaccuracies are categorised into both steady and transient state and are
discussed in detail. This helps identify the research gaps, and laid the foundation of the results
presented in Chapter 5-7.
30
2.2.1 Use of time synchronisation in power systems
Precision time synchronisation in power systems is challenging, as it not only requires
accurate timing across a large geographical area, but also secure and reliable time information
distribution must be maintained at all times. A common reference time is critically important
for grid monitoring, real-time situation awareness, coordinating system operation and
protecting power systems and their assets. However, issues can be found due to bad antenna
mounting, bugs in the product firmware, interoperability, and reliability of synchronisation
architecture. Utilities, academia, independent laboratories, and government are working on
improving the timing accuracy with advanced technologies and achieving an interoperable
feature across commercial devices [23]. Efforts are also underway to increase the security
level, detect the anomalous behaviours and form a best practice to build a resilient and reliable
timing system.
Most PAC applications need timing accuracy in different levels from sub-microsecond to
seconds. Currently, various industry standards published by the IEEE Power Systems Relay
Committee (PSRC), the IEC Technical Committee 57 - Power Systems Management and
Associated Information Exchange Committee specifies requirements and provides guidance
to achieve the desired level of time synchronisation [24]–[26]. A brief summary of key grid
timing requirements is provided as follows.
2.2.1.1 1 µs or less
A travelling wave fault locator normally synchronises two wave detectors at each end of the
line. When a fault occurs, an electrical wave is initiated at the point of fault and travels in
opposite directions at approximately the speed of light. Each detector identifies the fault
instigated wave and timestamps the arrival time. Comparing the time information from both
terminals, the fault location can be determined accordingly. Precise timing ensures an
accurate fault location detection [27].
Synchrophasors contain high precision timestamps for voltage and current measurements. A
time error of 1 µs would lead to 0.022° phase error at 50Hz according to IEEE C37.118 [24].
These synchrophasors, typically with a sampling rate between 30 and 120 samples per second,
are sent from local phasor measurement units (PMUs) to the phasor data concentrators
(PDCs). They are then compared to provide system operators with grid conditions or to be
used for wide area monitoring, protection and control (WAMPAC). A time synchronisation
error will result in angle shifts in phasor measurement, which raises issues with phase angle
31
monitoring, and other protection and control applications such as anti-islanding protection,
droop control and wide-area oscillation damping [28], [29].
Another application that requires precision timing in sub-microsecond region is the
IEC 61850 process bus that uses SV measurements [30]. Current and voltage measured by
instrument transformers are digitalised by standalone merging units (MUs) into sampled
values with the rate of 80 samples per cycle. SV streams carry a field called sample counter,
which is an incremental value from 0 to 3999, at 50 Hz. The counter resets back to zero at the
start of each second. A common time reference ensures these SV streams can be time aligned
with each other.
Line differential protection or transformer differential protection have a slightly less strict
requirement for time synchronisation; normally an accuracy level of 10 to 20 µs is
recommended [16]. Protection operations are based on the bias characteristics in which bias
and differential currents are compared with the restraint curve configured in the relay. A
synchronisation deviation can cause incorrect differential current measurement, increasing
the risk of relay maloperation [31].
2.2.1.3 1 ms or less
The synchronisation with 1 ms is required by supervisory control and data acquisition
(SCADA) and digital fault recorders (DFRs). SCADA scans the network every a few seconds
to have an overview of the grid conditions. Information exchange contains breaker status,
transformer temperature, voltage and current measurement; these all need time information
to mark when the data is obtained [23]. Similarly, DFRs timestamp when the fault happens
and record the current and voltage waveforms before, during, and after the fault [32].
2.2.2 Timing methods and challenges in delivering accurate time
information
Time synchronisation is necessary for substations to establish a coherent time frame among
various devices. It is particularly important for network devices to have a reasonably accurate
sense of time in terms of analysing the sequence of events which occurred in a power system
and to provide quick response to the network contingencies. IEC 61850-5 specifies various
time synchronisation classes from low (>10 ms) to the most stringent accuracy (≤1 µs) [12]
as shown in Table 2-1. In general practice, for synchronising different events, 1 ms accuracy
is sufficient. However, for aligning voltage and current values from various locations, an
32
accuracy down to 1 µs is required. In other words, for IEC 61850 substations, synchronising
GOOSE messages based on IEC 61850-8-1 [33], 1 ms accuracy is reasonable, Whereas, time
accuracy must meet 1 µs for synchronisation of SV, according to IEC 61850-9-2 Light
Edition (9-2 LE) [30].
Table 2-1: Time synchronisation classes specified in IEC 61850-5 [12].
Time synchronisation class Synchronisation error
TL >10 ms
T0 10 ms
T1 1 ms
T2 100 µs
T3 25 µs
T4 4 µs
T5 1 µs
There are two ways to distribute the time signals from the time sources to various intelligent
electronic devices (IEDs) in substations, the dedicated timing approach and the networked
timing approach. The former uses a standalone cabling system, e.g. coaxial, twisted pair and
fibre optic cables, to covey time information. The latter transmits the time signal via Ethernet
cabling systems shared with other substation data traffic.
2.2.2.1 Dedicated timing systems
Dissemination of time signals based on dedicated timing method can be implemented using
either 1-PPS or IRIG-B time codes via a separated infrastructure (Figure 2-1). 1-PPS provides
highly accurate synchronisation by transmitting one pulse every second. The rising and
falling edge of pulses must be kept sharp with both rising and falling time (10% to 90% and
vice-versa of the pulse level) less than 200 ns. However, 1-PPS has a significant drawback in
that the pulses do not contain time-of-day (absolute time) information. Hence, it must coexist
with other timing methods such as IRIG-B or Network Time Protocol (NTP) if applications
demand absolute time [34]. 1-PPS became the recommended timing solution in 9-2 LE for
process bus applications since MUs do not require absolute time information for
synchronisation. In the recent published standard IEC 61869-9, the synchronisation method
based on IEEE 1588 becomes the primary timing method for digital interface
implementations of instrument transformers, with an option of 1-PPS [35].
33
Figure 2-1: An example of dedicated timing methods used for synchronising various IEDs
in a substation automation network [36].
The history of IRIG-B time code can be traced back to 1952 when the U.S. guided missile
test ranges established the Inter-Range Instrumentation Group with an aim to standardise
different time codes [37]. IRIG contains a set of time code formats starting with one letter
followed by three digits. These designations reflect different attributes of the corresponding
time code. The current IRIG standard 200-04 contains alphabetic designations of A, B, D, E,
G, and H [38]. IRIG-B is the most commonly used dedicated time synchronisation method in
substations; it is encoded as binary coded decimal in one second window to convey absolute
time information. IRIG-B can be transmitted either as raw pulses (B003 and B004) via fibre
optic cables or copper cables using coaxial or twisted pair, or as amplitude modulated (B124)
with 1 kHz carrier signals over coaxial cables. Unmodulated IRIG-B is able to achieve timing
accuracy up to sub-microsecond level. Over the years, IRIG-B has introduced various
extensions, providing more information such as year, day light saving time, offset to
Coordinated Universal Time (UTC), leap second information and time quality to meet the
requirement for new applications.
However, one of the challenges faced by the substation designer is that the client devices in
the same network generally are not supplied by the same vendor. These devices may not be
able to accept all IRIG-B codes since they demand different synchronisation accuracy levels.
This leads to a mixed-use of both modulated and unmodulated IRIG-B time code with
different distribution methods (copper or fibre), complicating the substation timing network
design significantly.
34
In addition, although 1-PPS and IRIG-B are well-established techniques, they require
dedicated cables to distribute the time signals. A substation control room is often a few
hundred meters away from outdoor cubicles in a large open-air substation. A large number of
copper cables or optic fibres therefore need to be deployed. In addition, both 1-PPS and IRIG-
B are strictly one way methods (time information is communicated only from time source to
the clients), meaning they cannot automatically compensate the propagation delay. The
manual delay compensation demands considerable effort during installation and maintenance,
especially in large substations. As the distributed systems are moving quickly towards using
Ethernet as a sole communication medium, the dedicated timing methods are no longer the
preferable option and are being gradually superseded by network based timing methods [39].
2.2.2.2 Networked timing systems
NTP and PTP, generally referred to as the networked timing system, distribute time
information through packet-switched networks to synchronise various devices (Figure 2-2).
Inside a substation, the transmission medium is predominately Ethernet, whilst time domain
multiplexing (TDM) based synchronous digital hierarchy (SDH) or synchronous optical
networking (SONET) and are used more often in wide-area networks (WAN). To achieve a
highly accurate synchronisation over packet-switched Ethernet networks, a two-way time
transfer approach is adopted. Periodical dedicated packet exchange is established between a
reference node (server or grandmaster) and the rest of the nodes (clients or slaves) in a system.
Figure 2-2: An example of networked timing methods (NTP or PTP) used for synchronising
various IEDs in a substation automation network [36].
35
NTP is one of the oldest and most established communication protocols for networked timing.
The protocol was introduced by David L. Mills from The University of Delaware in the 1980s
[40]. NTP is widely used in WAN to synchronise computers and servers with the highest
achievable accuracy in millisecond range and has been adopted in recent years for substation
use [41]. NTP transmits Coordinated Universal Time (UTC) time scale in a server-client
manner; this meets the requirement of industrial standards such as IEC 61850 (without the
process bus) and IEEE 1815 (Distributed Network Protocol 3) that all timestamps shall be
based on an existing time standard [12], [42].
IEEE 1588-2008, known as “PTPv2”, is the second edition of PTP after the first version was
published in 2002. PTPv2 provides significant improvement in terms of synchronisation
accuracy up to sub-microsecond with the aid of hardware time stamping techniques. It forms
the timing hierarchy in a master-slave framework so only one clock can be elected as a
grandmaster clock to synchronise all slave clocks in the network. In contrast to NTP, PTP is
based on International Atomic Time (TAI) which solves the inherited issue of UTC as a
discontinuous timescale. UTC uses leap seconds added or subtracted from the current time
once every a few years to account for the variation in the rate of rotation of the earth. This
means all devices synchronised to UTC have to act accurately at the point of discontinuity
otherwise the timing accuracy will be violated significantly. Using TAI as a single time
reference avoids dealing with leap second issues. The offset to UTC is embedded into
synchronisation messages so UTC time can be easily derived at the slave clocks.
One of the benefits of networked timing is that NTP and PTP shares the same communication
channel with other substation automation protocols such as IEC 61850, Distributed Network
Protocol 3 (DNP3), IEC 60870-5-104, and Modbus so a great saving can be achieved by
eliminating dedicated timing systems. Also, both protocols use two-way clock
synchronisation method, so propagation delay can be computed and compensated at the end
nodes.
The basic working principle of time synchronisation in a packet-switched networks, including
NTP and PTP, is generalised in Figure 2-3. In the downstream direction, the sending time T1
is encoded in the timing message and the recipient time of the same message at the end node
is denoted as T2. T2-T1 equals to the transmission delay plus the offset of the local clocks
between the source and end node as shown in (2-1). Similarly, two timestamps T3 and T4 can
be obtained using upstream message transfer. In this case, the upstream transmission delay
can be calculated as T4-T3 subtracting the clock offset, shown in (2-2). These two timestamps
are eventually communicated to the end node for its local clock adjustment.
36
Figure 2-3: Basic working principle of time synchronisation in a packet-switched network
[43].
2 = 1 + delay−downstream + offset (2-1)
4 = 3 + delay−upstream − offset (2-2)
Assuming the upstream and downstream transmission delay are identical, the end nodes can
utilise the timestamp information to derive both transmission delay and clock offset as
indicated in (2-3) and (2-4).
delay−downstream = (2 − 1) + (4 − 3)
2 (2-3)
2 (2-4)
In the above calculations, an underlying assumption is the time it takes for a PTP message to
transfer from the master to the slave (downstream delay) is the same as the time for a PTP
message to transfer from the slave to the master (upstream delay). However, in a packet-
switched network this is not always the case and, in fact, nearly impossible with various nodes
between transmitter and receiver. Asymmetrical delay increases inaccuracies of clock offset
calculation, which jeopardises the overall synchronisation performance.
There are two major factors that can result in delay asymmetry. The first is related to the
accuracy of timestamping, i.e. how precise the timestamp reflects the moment of the message
either appearing on or arriving from the network. NTP uses a central processing unit (CPU)
to draw timestamps; this method is called software timestamping. When a node is sending a
37
timing message, a timestamp is taken while the CPU is assembling the packet. After drawing
the timestamp, the packet need to be moved from memory to the egress port buffer to be
transmitted. This process can be affected by various factors (e.g. CPU scheduling), so the
delay is not constant. Depending on the occupancy of the egress buffer, a further delay may
be experienced if other packets are being transmitted. Hence, the time span between when the
timestamp is drawn and the moment the packet appears at the egress port is not deterministic
and is difficult to predict precisely. Likewise, the receiving packet suffers from the same
inaccuracy due to the time variation in moving the packet from the ingress port to the point
where the timestamp is taken, normally located at higher layers (e.g. application level)
according to the open system interconnection (OSI) model.
One solution to increase the

ieee 1588 time synchronisation and data flow …

Documents