On the Horns of a DilemmaOn the Horns of a Dilemma

Successful Patch Management Strategies in SCADA and Industrial Control Systems

Eric Byres, P.Eng.CTO VP of EngineeringCTO, VP of Engineering Tofino Security A Belden Brand

Eric Byres VP Engineering Tofino SecurityEric Byres, VP Engineering, Tofino Security • Founder of the BCIT Critical Infrastructure

Security Centre a leading academicSecurity Centre, a leading academic facility for SCADA cyber-security research

• Canadian representative for IEC TC65/WG10 standards effort for the protection of industrial facilities from cyber attack

• Chair of ISA99 Security Technologies WG• Chair of ISA99 Security Technologies WG• Chair of ISA99 Stuxnet Gap Analysis TG• 2006 SANS Institute Security Leadership Award2006 SANS Institute Security Leadership Award• Six ISA and IEEE awards for security research• Testified to the US Congress on SCADA Security

Caught on the Horns of a Dilemma

• Why Patching is Critical• Why Patching is Difficult

Zotob Worm Security IncidentZotob Worm Security Incident

• August 18, 2005: 13 US auto plants were h t d b i lshut down by a simple worm. • Despite professionally installed firewalls between

the Internet the company network and thethe Internet, the company network, and the control network, the Zotob worm had made its way into the control system (probability via a laptop)laptop).

• Once in the control system, it was able to travel from plant to plant in seconds.p p

• 50,000 assembly line workers ceased work.• Estimated $14 million loss.Estimated $14 million loss.

Hiding Behind the Corporate Firewall Doesn't Work • The Slammer Worm infiltrated:

• A nuclear plant via a contractor’s T1 line;• A power utility SCADA system via a VPN;• A power utility SCADA system via a VPN;• A petroleum control system via laptop;• A paper machine HMI via dial-up modem.

• Corporate firewalls existed in at least three of these cases.

SCADA/ICS Incidents Since 2002SCADA/ICS Incidents Since 2002

• Malware accounts for 2/3 of the external b i id t t l tcyber incidents on control systems.

• Appears to match IT trends.

DoS6%

Sabotage

System Penetration

13%

Sabotage13%

Malware68%

A Perimeter Defense is Not EnoughA Perimeter Defense is Not Enough• We can’t just install a firewall in front of the whole

SCADA system and forget about securitySCADA system and forget about security. • The bugs eventually get in.• So we must harden the plant floor.p• We need Defense in Depth.

Crunchy on the Crunchy on the Outside - Soft in the Middle

Defense in Depth StrategyDefense-in-Depth Strategy

• “By defense-in-depth strategy, we mean the t ti d f thprotection measures composed of more than

one security control to protect the property.”• “By use of these kinds of multi layer• By use of these kinds of multi-layer

measures, another layer will protect the property even if one layer is destroyed, soproperty even if one layer is destroyed, so the property is protected more firmly.”

Yokogawa Security g yStandard of SystemTI 33Y01B30-01E

But Patching SCADA and ICS is DifficultBut Patching SCADA and ICS is Difficult• Patches can affect the reliability of SCADA/ICS

softwaresoftware.• Patches may require staff with special skills to be

present.• Patches may not be approved by SCADA/ICS

vendor.• P t h i t b t• Patches may require a system reboot.• Patches may not be available.

What No Patch for That?What – No Patch for That?• Many SCADA/ICS vendors have a poor history of

releasing security patchesreleasing security patches.• According to Sean McBride of Critical Intelligence,

less than half the disclosed SCADA/ICS vulnerabilities ever have released patches.

What No Patch for That?What – No Patch for That?• Many systems still used in process control or

SCADA are no longer supported with securitySCADA are no longer supported with security patches:

“Beginning January 1, 2005, Pay-per-incident and Premier t ill l b il bl f Wi d NT Ssupport will no longer be available for Windows NT Server

4.0. This includes security hotfixes.

“On July 13, 2010, Extended Support for Windows 2000 Server will end.”

“Support of FactoryLink will cease December 31, 2012”

11

The Typical ICS/SCADA Patch SituationThe Typical ICS/SCADA Patch Situation

• “No central Patch Management process in l f SCADA t ”place for process or SCADA systems”

• “Patching is up to each system owner to decide if and when to do”decide if and when to do

• “IS/IT department is trying to force their overall patching process onto automation systems”patching process onto automation systems

Joakim Moby, ISA Expo 2006

Solutions for Patching in ProcessSolutions for Patching in Process and SCADA Systems

• Patching by Criticality• Patching in Waves• Using Compensating Controls

Patch Management is Not an OptionPatch Management is Not an Option• “A procedure for patch management shall be

established documented and followed ”established, documented, and followed.Requirement 4.3.4.3.7

ANSI/ISA-99.02.01

Patch Management is Change ManagementPatch Management is Change Management• “Patch management is about managing the risk of

change”changeRichard Brown,

Dow Chemical

Patch Management in Control SystemsPatch Management in Control Systems

Classify SCADAInventory All

SCADA SCADA Devices By

Risk

SCADA Devices

Feed Results Back on D i

Create Patch/ Vulnerability

Tracking Devices gSystem

Create Response

Plan

Execute Response

Stages

Inventory SCADA and Control DevicesInventory SCADA and Control Devices• All process control and SCADA devices are

inventoried in terms of:inventoried in terms of:• Device type• Criticality to process• Core software/firmware components• Patching requirements

• Should include both computer and non-computerShould include both computer and non computer devices like PLCS, RTUs and SIS.

Categorizing SCADA and Control DevicesCategorizing SCADA and Control Devices• All devices are categorized into groups that define

when and how they are to be patched Example:when and how they are to be patched. Example: • “Trial Adopters” receive patches as soon as available and

act as Test/Quality Assurance machines. “N T h” hi i l i t ti d/• “No Touch” machines require manual intervention and/or detailed vendor consultation.

Create Patch/Vulnerability Tracking SystemCreate Patch/Vulnerability Tracking System• Develop procedure for keeping track of new patches

and vulnerabilities:and vulnerabilities:• Level of importance to control operations.• Devices affected • Severity level

• For each category, the severity level provides a time frame during which a specific patch must beframe during which a specific patch must be installed to minimize risk.

Create Response PlanCreate Response Plan• Create a system where patch implementation levels

are preset and tied to Response Plans for a givenare preset and tied to Response Plans for a given device class.

Response Plan

Aggressiveness Implementation Window

Level of Testing

Al h Mi i Q t l Hi hAlpha Minimum Quarterly High

Bravo Moderate By end of following week

Best Effort

Zebra Maximum Within 48 hours Minimal

Patch DeploymentPatch Deployment• Each patch deployment is divided into several

waveswaves.• One wave installs the patch/patches onto a number

of machines.• The purpose of waves is to find issues before the

patch is installed on the systems with highest complexity or regulatory impactcomplexity or regulatory impact.

Typical “Active Patching” CycleTypical Active Patching Cycles

1. Central Test Platforms 4 days

• Rapid deployment

ng d

ays

2. Critical & Control GroupCritical

infrastructureSample clients &

servers (0.5%)

4 days

• Rapid deployment• Formal responses gathered

4 w

orki

n

3. Early Adopters 6 days• Devices of low regulatory impact• Allows time for change control and

scheduling

cle

= 34 4. Mainstream 10 days

• Relies on success of previous waves, not on testing

• Responses by exception

1 cy

c

5. Late AdoptersValidated servers

High risk devices

10 days• Mainly devices of high regulatory or

safety impact• Otherwise as for waves 3 & 4

22Source: Joakim Moby, Astra-Zeneca, ISA Expo 2006

FeedbackFeedback• When cycle has ended, a report is produced that

can be used as a guide for the systems still notcan be used as a guide for the systems still not following the process (such as new systems).

• Feed patching results back into inventory database and patch/vulnerability database, including issues.

Exceptions to the Patch Management SystemExceptions to the Patch Management System• Patch Management System catches between 50%

and 70% of the critical patchesand 70% of the critical patches.• Some critical devices/systems will not fit into this

“Microsoft/PC” patch model:• The patch may not be available or approved by the SCADA

vendor in a timely manner.• Servers with old operating systems (like NT4 or Windows p g y (

2000) cannot be patched.• PLCs and RTUs may not be patchable.

Compensating ControlsCompensating Controls• If patching isn’t possible, a compensating control is

needed:needed:• Turn system off.• Improve system isolation.• Reduce system exposure to other systems.

Protection for Unpatchable SystemsProtection for Unpatchable Systems• Example: Industrial firewall for NT Servers restricts

possible infection vectorspossible infection vectors.Internet

Office LAN

NT4 Based

Plant Network

Based Server

Firewall restricts

FCS

Control LAN

Firewall restricts communicating

devices and filters bad traffic

Protection for Embedded SystemsProtection for Embedded Systems• Example: MODBUS/TCP industrial firewall sanitizes

and limits MODBUS commandsand limits MODBUS commands.

Office LAN

Internet

Office LAN

Plant Network MODBUS/TCP FW only allows MODBUS

Read commands

Control LANSafety System

ConclusionsConclusions

• Patching is a change management process ( t h )(you must a have a process).

• Patching does NOT end with the Microsoft patchespatches.

• Compensating controls are important to catch vulnerabilities that can’t be patchedcatch vulnerabilities that can t be patched using standard methods.

• Patching is not a silver bulletPatching is not a silver bullet.• Patching must be done together with other

activities to secure SCADA systemsactivities to secure SCADA systems.

ReferencesReferencesTofino Security White Papers and Application Notes• Using IEC/ISA-62443 Standards for SCADA Security g y

https://www.tofinosecurity.com/blog/using-ansiisa-99-standards-scada-security-plus-white-paper

• Shamoon Malware and SCADA Security – What are the Impacts?https://www.tofinosecurity.com/blog/shamoon-malware-and-scada-security-%E2%80%93-what-are-impacts