breakout session 4 - scada -...
TRANSCRIPT
Breakout Session 4 - SCADA
SCD5200 Usage & Project Implementation at SP AusNet
Andrew Ferenac & Lucas Ellis
ICT - Remote Systems Group
6th May 2013
Safety Topic
• Did you bring a suitcase?
• When packing up to go home try and remember safe manual handling
2
• Lucas Ellis
– Software Developer
• WinRadio
– Protection and Control Tester
• SP AusNet
Field Tests
Project Delivery
– Real Time Systems Analyst
• SP AusNet
Who are we?
3
• Andrew Ferenac
– Electrical Technician (Field Tester)
• SECV
– SCADA Technician
• SECV / VPX
– SCADA Consultant
• Distribution Businesses
– Technical Specialist / Project Manager
• Logica
– SCADA Engineer
• AGL, Jemena, Tenix Aliance
– Senior Real Time Systems Analyst
• SP AusNet
Session Agenda
• Introduction
• SP AusNet Business Overview
• Case Study: Network Wide SCD5200/P3 Firmware Upgrade Project
• Project Implementation Using the SCD5200
• Automation Tools Developed In-house
• SP AusNet SCD5200/P3 Security Implementation
• Questions
4
Business Overview
• A diversified energy infrastructure business
• A workforce of around 1600 office & field based employees
• Owns & operates $6.3billion worth of assets
• 51% owned by Singapore Power and listed on both the Singapore &
Australian Stock Exchanges
5
About SP
6
Electricity
Transmission
Electricity
Distribution
Gas
Distribution
Gas
Distribution
Business Overview: Area of Operation
9
Remote System Group (RSG) Overview
• Transmission Electricity Systems – All aspects of station automation
• new projects
• system augmentation
• maintenance
• Distribution Electricity Systems – All aspects of substation automation
• new projects
• system augmentation
• maintenance since 2009
10
RSG Overview: Activities
11
New projects 60%
- New stations
- Rebuilds
- Augmentations
- Internal projects
Incidents 30%
- Faults
- Hotline support
Others 10%
- Meetings
- Training
RSG Overview: SCD5200 RTUs
• SCD5200 implemented in Transmission & Distribution stations
• Optonet Systems range from 3 nodes to 30+ Nodes
• IO counts range from 100 to 1000s
• Multiple Masters:
– Station HMI (Human Machine Interface)
– Transmission & Distribution Control Rooms
– Other Distribution Businesses
– Generation Businesses
– Major Customer, i.e. Victorian Desalination Plant
• Multiple IEDs using various protocols (DNP, Modbus, IEC103, ASCII)
12
RSG Overview: Keilor Terminal Station (KTS)
13
RSG Overview: KTS Nodes 19 to 29 (500kV Relay House)
14
RSG Overview: Remote Connections
• WAN
– IP network gigabit speeds
– Fibre to most stations (OPGW)
• NEXTG / 3G
– Coverage to areas without WAN
– Slow but quicker than driving
• Media Converter
– Enables SCD5200 connection
15
Case Study: Network wide SCD5200/P3
Firmware Upgrade Project
Upgrading firmware at 55 sites
Lucas Ellis
Remote Systems Group
6th of May 2013
Upgrade Drivers – Why Upgrade?
17
Upgrade Drivers – Why Upgrade?
Improvements to system configurator
Improvements to RTV
Updated firmware
Enable network security features
18
Upgrade Overview – What was involved?
19
Upgrade Overview – What was involved?
Updating configurator
Updating configurations
Deploy the configurations
& new firmware
20
10 workstation installations
55+ RTU configurations:
Restore, update, fix, generate, check, backup, check
300+ nodes to upgrade
Upgrade Roadblocks
21
Updating configurator
Updating configurations
Deploy the configurations
& new firmware
Upgrade Roadblocks
22
No Windows 7 Support
Risk introducing errors
Many manual tasks
Data Interruption
Remote locations
Updating configurator
Updating configurations
Deploy the configurations
& new firmware
Upgrade Roadblocks – Solutions
23
No Windows 7 Support
Risk introducing errors
Many manual tasks
Data Interruption
Remote locations
Virtual Machine
Automation Tools
Procedures
Remote Connectivity
Upgrade Process – Planning Stage
Define scope and requirements
Investigate process – find issues & resolve issues
Opportunity: Develop automation tools to reduce labour and error
Develop a procedure and instructions
-Optimised to cut down on manual tasks
-Provides a consistent approach
24
CREATE
Upgrade Process – Testing stage
Test the procedure and firmware – Real equipment
Peer review – Another set of eyes
Confirm RTU works as expected – Sanity check
25
TEST
Upgrade Process – Testing Again!
Deploy to a pilot site
Choose a complex site (33 nodes)
Use experienced staff
26
CHECK
Upgrade Process – Deploy
Divide and conquer – Allocate resources
Deadline (motivation)
Keep track of progress
27
DELIVER
Lessons Learnt & Project Outcome
28
Lessons Learnt & Project Outcome
Successfully completed project!
What went wrong?
- Testing revealed a problem with configurator (IEC103 Di SOE Di)
- Feature creep (extra checking not part of upgrade)
- One case where an old configuration was left in service (1 node only)
What is next?
– Smooth the way for future upgrades
– Implement security
More automation tools are required
– Saves time
– Reduce errors
29
Questions?
Next Up: Project Implementation using SCD5200 RTU
30
Project implementation using the SCD5200 RTU
Andrew Ferenac
ICT - Remote Systems Group
6th May 2013
Project Design
32
• Project Design is generally contracted out to 3rd party Design
Service Providers (DSP).
• DSP is responsible for documenting project SCADA changes
via SPA supplied Mapping Table, use available templates and
documented standards.
• SPA documentation is available to assist DSPs to implement
SCADA changes into Mapping Table correctly.
• Utilizing standards provides consistent and accurate
information for project implementation = TIME SAVINGS during
configuration.
• Poorly implemented standards creates more work for RSG –
mistakes take time to identify and correct.
SCADA Standards - Mapping Table
33
Pros:
• Consistent layout of SCADA information using MS Excel Mapping
Table.
• Details ALL SCADA IO, Rack Layout, Deleted IO.
• Mapping Table templates are created for each standard IED
implementation to be used during project design.
Cons:
• DSPs often take considerable time to grasp Mapping Table concept.
• Updated Mapping Tables often require considerable time to review
and correct.
Mapping Table
34
Source
Device
of point
Protocol
to
Master
Masters
To RTU
Unique TagName
Point Title =
Name + DevType + Function
Master
Protocol
Index
Source
Connection
Type
Source
Connection Index
Or Termination Analog
Ranges
RSG Delivery Stages
35
• Review and Check supplied information (Mapping Table)
• Configuration of RTU, HMI & Master Station
• FAT of RTU & HMI configurations
• Site Delivery of RTU & HMI configurations
• Commissioning Support
• Correct problems detected during commissioning
Review & Configuration
36
• Review & Configuration is automated as much as possible using In-
House developed Tools – assists to ensure consistent
configurations.
• Mapping Table Tool – Excel VBA
• Checks Mapping Table for common errors and deviations from
Mapping Table standards.
• CiBuilder – VB Application
• Further checks Mapping Table against standards, particularly
TagNames, and builds HMI configuration data & graphical objects.
• FoxBuilder – Excel VBA
• Further checks Mapping Table against standards and creates Foxboro
CSV formatted sheets ready for pasting into SysConfigurator.
Configuration
37
• RTU configurations are stored as Single Site SysConfigurator Backups zipped
along with .CFG, .RPT, .SAL & .ELF files for each Node.
• Version control of configurations changes are mandatory.
• This is a manual process for RSG.
• New Master & Slave communications ports must be created manually within
SysConfigurator before pasting to ensure that correct port parameters are setup.
• IEDs & IO are added to the configuration using the SysConfigurator Paste From
Spreadsheet function, pasting the prepared data from FoxBuilder.
• The FoxBuilder CSV sheets can also be saved (manually) to CSV files and
imported into SysConfigurator using the Import CSV Files … function.
• Pasting or Importing needs to be done in a logical order.
FAT – Test set or IEDs
38
• Where possible FAT is completed using IEDs configured with actual settings being
implemented during the project.
• In some cases new panels are
delivered for FAT with the RTU & HMI.
• RSG have a Test Bed facility which
contains RTUs, IEDs, Communications
devices, etc.
• Everything we need for
FAT (generally!)
• Not always possible to cover each
IED implementation as there are
varying IO module configurations
from site to site.
FAT
39
• Where we can’t test to a real IED we use Software Test Sets such as IO Server.
• Supports many protocols via Serial or Ethernet.
• Allows different Test configurations to be saved, i.e. DNP Master, DNP Slave, etc.
• Provides a PLC function for ramping AIs and turning DIs on / off automatically.
• Integrates with MS Office (Excel) using DDE.
• We have developed an Excel workbook with Graphical interface (Buttons) to drive
Binary & Analog Inputs via IO Server. Reduces testing time dramatically.
• We aim to test every IO point to every Master before delivering changes to site.
• Reduces our involvement during commissioning and provides confidence in what
is delivered.
Site Delivery
40
Before performing Site Delivery consider / determine:
• Impact to TOC & DOC, AEMO!
• Can all changes be delivered at once?
OR
Do we have to schedule the changes in-line with Secondary changes onsite.
• Remotely OR On-Site.
• Perform a compare of Before and After configuration & report files.
• Create a Test Schedule for commissioning staff based on Mapping Table.
• Effective and Proactive communication with all stakeholders is essential.
Pre-Commissioning & Commissioning Support
41
• Field staff pre-commission through to Master Station SCADA staff & HMI.
• They often require assistance getting IEDs communicating.
• Cabling Issues and Communications Parameters
• Issues List is generated for RSG to action.
• Unfortunately pre-commissioning is often scheduled close to in-service date, doesn't
leave much time!
All going well RSG will not have much involvement past Site Delivery.
( Wishful Thinking?)
Conclusion
42
• Documented Standards allow for consistent Design information.
• Correctly implemented Standards improve configuration efficiencies &
development time through the use of automated Tools.
• Implemented standards in SCADA should line up with IED configuration
standards.
• Aligned IED & SCADA standards allow for efficient / smooth FAT.
• Complete FAT provides confidence for Site Delivery & Commissioning.
Questions?
Next Up: In-house Automation Tools
43
Automation tools developed In-house
Improve efficiency and reduce errors
Andrew Ferenac & Lucas Ellis
ICT - Remote Systems Group
6th May 2013
SCD5200 Tools: DeleteCSVFiles.bat
45
• Simple to use
• Keeps backups clean
• Wouldn’t get done otherwise (36 directories to check by hand?)
SCD5200 Tools: GenerateReportFiles.bat
46
• Simple to use
• Creates reports used to compare each new RTU configuration
• One tool, not one batch file per site
• External to system configurator
SCD5200 Tools: ToshSALFix.bat
47
• Nested comments
/*** comment
/*** Nested comment ***/
end of comment ***/
• All our Toshiba PLC calculations had this problem
• Problem solved
SCD5200 Tools: CheckReportFiles.bat
48
• Saves a lot of manual checking
• Easy to add new checks
SCD5200 Tools: Future Tools
49
• Interact with the RTU from command line
• Roadblocks:
– No security support
– Found a bug in the download tool
• Wish list:
– Download all files to RTU
– Backup RTU files to PC
– Audit all RTU configurations (wrong f/w or cfg in service?)
– Restart all nodes on an RTU
– Switch all nodes of an RTU to second configuration
– Many many more
Mapping Table Check Tool
50
Simple Excel VBA app to check Mapping Tables for common
errors and against agreed SCADA standards.
• A simple dialog allows the user to select an already
opened mapping table to check.
• Can check each item one at a time, or all at once.
• Can check a 500 row Mapping Table in 10 seconds.
• This might take hours, even days manually.
• Uses Excel Ranges to specify accepted values, which
can be edited at any time, not hard coded!
• Each stage provides a “Completed” dialog with the # of
errors found. A column is inserted into the Mapping
Table to detail errors for each cell.
Cibuilder
51
A Multi-Pass process to build HMI configurations.
• The “Create Device List” function
checks the Mapping Table IO
list against Standards.
• An Error column is inserted into
the Mapping table to provide detailed
error comments.
• Agreed standards are hard-coded into CiBuilder.
• Even though CiBuilder’s primary function is to build HMI configurations, it is a
valuable tool for checking Mapping Table data.
• Once the “Create Device List” passes without error the “Create Database File”
and “Create / Place Devices” functions are used to complete the HMI configuration
changes.
FoxBuilder
52
A Multi-Pass process for building the various parts of a SCD5200 RTU configuration.
• It is imperative that the Mapping Table passes Mapping Table Tool & CiBuilder
checking before running FoxBuilder.
• A stand-alone Excel spread sheet VBA application.
• Has individual sheets representing each SysConfig CSV file.
• Is run from Custom menu items,
Run Builder and Clean Data.
• The Mapping Table data is copied into a FoxBuilder MapAll worksheet.
FoxBuilder
53
• Run Builder provides the Build Options Interface.
• Can process all rows in the MapAll sheet or selected
rows.
• Can be used for Transmission or Distribution
configurations.
• Laid out in a logical order of stages.
• Provides progress indication, status and # of records
processed.
• Allows CSVs for a complete RTU configuration to
be prepared in a matter of minutes.
• In this screen shot 625 rows were processed and 4 IEDs added
to the DNPM sheet in 5 seconds.
• When processing DNP Master IO (3.1) 625 rows were
processed and 276 records added to DNP Polled in 7 seconds.
FoxBuilder
54
• Dialogs are displayed at the end of processing each stage if errors are present.
• Detailed error information is provided in each CSV sheet.
• Control parameters for each IED are defined in a separate sheet “Control Types”.
• Allows for new IEDs to be added without the need for code changes.
• Health Point Names, Polling Parameters, etc. are currently hard-coded.
• Future changes include defining these parameters in a work sheet range
allowing for changes by users at any time.
• FoxBuilder allows CSVs to be prepared in approx. 15 minutes instead of 4 hours
or more preparing them manually.
Questions?
Next Up: SCD5200/P3 Security Implementation
55
SP AusNet SCD5200/P3 Security
Implementation
Creating site specific security files with RTV
Lucas Ellis
ICT - Remote Systems Group
6th May 2013
Overview
• SCD5200 RTU’s connected to WAN need security
• When applying passwords have to consider
– Human error
– Remote locations
– Consequences of error
• Our approach
– Create site specific security files
– Test the file locally
– Deploy to the field
• Security Configurator
– Create default security file
– Previous presentations
57
Process: Creating the password file.
58
Download
“Default.SCF” file to
RTU
Reset
RTU
Enter Default
Password
Copy & Paste SiteX
Password from Excel
Upload
“Security.SCF” as
SiteX.SCF
SiteX.SCF
created.
Repeat as
needed for
other sites
Process: Testing the password file.
59
Download “SiteX.SCF”
file to test RTU
Reset
Test RTU
Copy & Paste SiteX
Password from Excel
Confirm entry to
test RTU
SiteX.SCF
tested.
Repeat as
needed for
other sites
Process: Deploy the password file.
60
Call control room
Reset RTU
Confirm access to RTU
Call control room
RTU password
updated.
Repeat as
needed for
other sites
Update records
Download “SiteX.SCF” file to RTU
Process Issues
• Time consuming
• Risk of human error
– Easy to do, hard to notice
– A wrongly copied password could potentially lock out the RTU, requiring a site visit
• Creating and testing 50+ SCV files can take upwards of eight hours. This
is not a job that can be rushed.
• This is over a full day of work wasted, without even starting to roll the
passwords out to a live site.
• Repeat the process every 6 months.
61
Example: Creating a password file
1: Download and reset
62
Example: Creating a password file
1: Download and reset
2: Wait for reset
63
Example: Creating a password file
1: Download and reset
2: Wait for reset
3: Connect and enter default password
64
Example: Creating a password file
1: Download and reset
2: Wait for reset
3: Connect and enter default password
4: Enter site specific password
- From excel
- Twice!
65
Example: Creating a password file
1: Download and reset
2: Wait for reset
3: Connect and enter password
4: Enter site specific password
- From excel
- Twice!
5: Backup new password file
66
Example: Creating a password file
1: Download and reset
2: Wait for reset
3: Connect and enter password
4: Enter site specific password
- From excel
- Twice!
5: Backup new password file
6: Save with site name
67
A better way?
C:\gen_SCF.exe –user USER –password PASSWORD siteX.SCF
siteX.SCF generated.
C:\
68
• Command line tool
• Saves time
• Reduces errors
• Automation – Can create files for all sites at once
• Doesn’t actually exist (yet)
69
Questions?