kevin manderson hydro tasmania: implementation of cyber security in a scada environment
DESCRIPTION
Kevin Manderson, SCADA Systems and Security, Hydro Tasmania delivered this presentation at the 2013 Corporate Cyber Security Summit. The event examined cyber threats to Australia’s private sector and focussed on solutions and counter cyber-attacks. For more information about the event, please visit the conference website http://www.informa.com.au/cybersecurityconferenceTRANSCRIPT
Cri$cal Infrastructure Power U$lity and SCADA Security
Kevin Manderson Hydro Tasmania
Hydro Tasmania
• Tasmania had the first hydro electric power sta$on in southern hemisphere at Duck Reach, Launceston;
• Australia's largest renewables generator and water manager;
• 30 power genera$ng sta$ons (hydro, gas and wind); • Dams, tunnels, weirs, flumes; and • Substan$al monitoring of infrastructure and environment.
Safety Moment
• October 7th 2013 – A TEPCO employee carelessly pressed a buUon shuVng off cooling pumps that serve the spent fuel pool in reactor #4 — thankfully a backup kicked in before any cri$cal consequences resulted.
• Process vulnerability analysed, managed and the outcome indicates it was appropriately controlled.
From my 1884 edi$on of ‘The Colonists Guide and Cyclopaedia’ The only men$on of ‘secure’ is securing a horse to a slippery pole – op$on, clove hitch
Another Dic$onary Defini$on
se·∙cu·∙ri·∙ty • noun \si-‐ˈkyu̇r-‐ə-‐tē\: the state of being protected or safe from harm
• : things done to make people or places safe
Summary
• Security could be described as ‘things that keep my systems from causing harm’,
• ‘Safe from harm’ depends on the nature of the system, • Jus$fying security is simply an NPV analysis of the consequence – but the likelihood moves,
• Implemen$ng good controls helps by having experience with opera$onal systems being aUacked, and
• My view is security is ‘mechanisms to stop changing the control I have over my systems and data to keep the systems from causing harm, without my knowledge’.
A security `something’ is what
• Any change which causes the overall process to be less safe or in an unknown state or safe,
• The specifica$on or expecta$on is not met, • So what can have an impact? – Malicious hacker, – A ‘Snowden’ or admin event, – Physical interven$on, – Negligence/Accident, – Equipment/sooware malfunc$on, and – Loss of services.
So What am I Keeping Secure
• The dispatch process: – Several sliding windows of $me with $ghtly coupled processes occurring across a group of closely and loosely coupled systems • Four seconds – data exchange with power sta$ons, • Eight seconds – control data exchange with AEMO, • Five minutes – dispatch interval, • 30 minutes – market interval, • Daily – repor$ng and other processes, • A number of ancillary services, and • Other contracted and mandated services.
• But am just the custodian of data in part of the chain of the complete process.
What Is Involved
• The core dispatch process is a con$nuum of: – receiving targets from AEMO, processing them, – sending targets to power sta$ons , and – receiving remote data and sending data to AEMO.
• The secondary dispatch processes are a con$nuum of receiving situa$onal events, local alarm processing/logging, managing water and other local risks
• Cut, rinse and repeat every four seconds, ad infinitum,
What Adds to the Mix
• A requirement for redundant systems,
• A requirement of 99.995% up$me,
• The need to get some data from corporate and other sources, and
• The need to pass data to corporate and others.
99.9% ("three nines")
8.76 hours
43.8 minutes
10.1 minutes
99.95% 4.38 hours
21.56 minutes
5.04 minutes
99.99% ("four nines")
52.56 minutes
4.32 minutes
1.01 minutes
99.999% ("five nines")
5.26 minutes
25.9 seconds
6.05 seconds
Year Month Day
Situa$on – What Bad End Game
• What could be the target – Aurora class event, – System black, – Dam failure, and – Or Just less safe…
What Touches my Systems
• The systems sit in locked racks, rela$vely inert, • What can have an impact? – People, – Data, – Services, and – Physical environment.
Considera$ons
• Cri$cal infrastructure SCADA model: – What core process is required, – Secondary processes/redundancy, and – Support and non essen$al systems.
• Tools: – Defence in depth, and – Vulnerability analysis and early security design.
• Start from the End Game: – The final last gasp must work, – Analyse processes, interac$ons and impact on security, and
– Can process layers be bypassed?
What Op$ons
• Air gap to corporate/internet. Doesn’t work, Nantanz/Iran – Stuxnet via USB key,
• Tightly lock down every point. Fails by social engineering, and
• Design in security from the start. Ongoing vulnerability analyses rela$ve to actual events, conferences and research -‐ then update security posture as needed. Best op$on.
Process Analysis
• Analyse to find the paths/vectors across the vulnerability layers,
• I think of each process/layer as a bubble inside or touching another bubble, and
• Consider controls for each layer or bubble: – If this bubble is popped will the system be resilient and the next bubble remain,
– Can the final ‘must keep intact’ bubble survive?
– Can any vulnerabili$es or layers be bypassed.
SCADA Good Prac$ce Guide (GPG)
Layered on GPG
Process Analysis
• Process 1 – Environment to support the core processes/must remain processes: -‐ Preserve ‘data chain’ for dispatch system -‐ AEMO to online generators and return.
• Process 2 – Redundancy and cri$cal support: -‐ Primary redundancy, communica$ons, compliance ‘func$onality’.
• Process 3 – Suppor$ng systems: -‐ Test, development, other support systems.
• Process 4 – Non or less essen$al services and systems: -‐ Historian, repor$ng, non opera$onal systems.
Hydro Architecture
• The GPG is a baseline, • Addi$onal $ers of access control, some as processes/layers, others as diversity in the boundary traversal and monitoring and aler$ng,
• Hydro’s produc$on environment has about 40 servers/systems integra$ng the produc$on process. Redundant over mul$ple sites and communica$ons paths. Builds on the GPG, and
• Logging, monitoring and aler$ng.
Process Approach
• Vulnerability analysis for each process flow, • Start with an assump$on that all external interfaces are or likely to be compromised (‘there be dragons’), and
• Brainstorm possibili$es: – People, – Services, – Black swans, and – All hazards approach – that is, all or none?
Deciding on Security Barriers
• Aoer analysis iden$fy the process change, ownership transfer and different security groups. These are highly likely to be the vulnerability points,
• What user group • What is the vulnerability: From AIC (or CIA):
– Availability: DOS/interrup$on, – Integrity: data or control injec$on/corrup$on, and – Confiden$ality: data access.
• What consequences, and • What cost to control (inputs or consequence).
Example -‐ Part of Cri$cal Path
• Basic func$onal requirements: – Cri$cal to process, data burst every 8 seconds, – ICCP (IEC 60870) traffic, – Bidirec$onal, high availability, and – To/from external provider.
Data to and from AEMO.
SCADA and Dispatch processing
Power sta$ons
Process -‐ Cri$cal Path
Data to and from AEMO
SCADA and Dispatch processing
Power sta$ons
• Func$onal requirements: – Cri$cal to process, data
burst every 8 seconds, – ICCP traffic, – Bidirec$onal, high
availability, and – To/from external provider.
• Controls implemented: – High availability cross
connected ‘enhanced firewalls’,
– Mul$ple firewalls/vendors, – Monitor link status, – Only allow ICCP traffic, and – Redundant paths.
Monitoring
What Else for Hydro
• Diversity of security barriers, • Golden images for switches,
• Scripted builds for install and then security, • ‘Whitelis$ng’. Hash lists of known and known bad and regularly compare, check updates,
• Monitoring (IDS, Full data packet capture, SMS alarms, environment), and
• Data valida$on.
Now Layer the ASD/DSD 35 List on Top
1. Applica$on whitelis$ng 2. Patch applica$ons 3. Patch opera$ng systems 4. Least privilege 5. Mul$ factor authen$ca$on 6. Firewalls 7. Segmented systems 8. IDS/IPS and monitoring 9. <down to 35>
ASD List
• Targeted at securing the IT components, • Notes: – In many SCADA or process control environments patching is poorly done,
– I know of one current opera$onal SCADA system running under Windows3.11, and
– I know many cri$cal infrastructure systems where patching is years old.
• But in SCADA environments generally everything else is far beUer than average.
ASD List
• Excellent for keeping individual systems secure,
• Many secure systems contribute to keeping an overall complex process secure, par$cularly at the less process intensive boundaries, and
• Consider where the IT component is important in the process chain.
Our Observed/Heard of Events
• Nil IT security • Hydro: – May 2005, CBD power fail caused transport/access issues, – Had a significant power failure event in April, and – Air-‐condi$oning event in June.
• Others: – Data from the field, range failures:
• SCADA input of 10^38 to another SCADA system , received ok but process failed with a squaring func$on.
– Peoples ac$ons, par$cularly when fa$gued: • October 2013, Tepco – operators pushed big red buUon, and • October 2013, loosened pipe and drenched in radioac$ve water.
Root Causes
• CBD power: Substa$on trip, alternate travel paths not considered,
• Power failure: Occurred during regular UPS tes$ng, system deigned to survive
• Air-‐condi$oning: Occurred during regular generator tes$ng, unexpected but survived (now monitored/controlled), and
• Data 10^38: no limit checking,
How to build a secure system
• Use the AG Good Prac$ce Guide as a guideline for the separa$on of form/func$on and privileges,
• Use the ASD 35 point security guide to help with ongoing hardening and maintenance procedures,
• Build for mul$ple concurrent failures. Expect mul$ple unrelated failures to occur simultaneously, and
• Think all hazards during analysis.
Think Alternate Security
• Who went to Ruxcon? • Other white/grey/black hat
conferences, • Use a range of tools and
test your systems, • Do pen tes$ng, • Think touch==own,
physical security is cri$cal, • Simple can be good, • Be aware of what's
happening, and • xkcd is good…
Comments or Ques$ons