patching lessons learned: an interesting perspective · patching lessons learned: an interesting...

Cyber Security | Compliance | Industrial Computing

PATCHING LESSONS LEARNED: An Interesting Perspective

WHO WE ARE-

MISSION We ensure the success of our customers by developing innovative technology solutions that optimize and protect critical infrastructure

HISTORY 4 Company was started in 1981. Happy 36th

birthday to us! 4 25+ years of experience in the design, manufacture,

and lifecycle management of industrial HMIs andSCADA devices

4 11+ years of experience working with cyber securityin the energy industry

4 ISO 9001:2008 certified 4 In 2013, we were awarded a $4.3M cooperative

agreement by the Department of Energy in theCybersecurity for Energy Delivery Systems(CEDS) Program

4 In 2014, we merged the computing and cybersecurity businesses under the name of FoxGuardSolutions

4 Total employees = ~110

Cyber Security | Compliance | Industrial Computing 2www.foxguardsolutions.com

WHAT WE DO-

FoxGuard serves as a solution provider for mission-critical applications in Critical Infrastructure and Key Resource (CIKR) sectors.

Our Solutions Fall Into Three Primary Categories: 1. Cyber Security 2. Compliance 3. Industrial Computing Hardware


PATCH & UPDATE MANAGEMENT PROGRAM-

Patch & Update Data Aggregator/Web Portal 4 Provides users a with single location to find information about patches and updates applicable to

energy delivery industrial control system devices

4 The portal serves as a repository for Hash Authentication information, patch discovery evidence anddevice End of Support (EOS) documentation

Patch & Update Authentication 4 Our aggregated Hash Files from vendors provide users a central location to help verify the integrity of

downloaded patches and updates prior to deployment

4 The Hash Authentication Tool allows customers receiving aggregated patch data via customized reportsto authenticate that reports have not been compromised

Validation Techniques 4 Provides users with proven methodologies to validate patches and updates before deployment

4 Users may self-perform, set up their own validation lab, contract validation services or take a combined approach

Query Engine 4 The Query Engine will support multiple device types and across various energy delivery ICS vendors

4 Enables users to query IT and OT equipment to determine relevant baseline information such as Make,Model, Firmware Version and Serial Number – this information is critical for accurate patch discovery

4 The Query Engine will offer an easy to use user interface supporting a Patch Gap Analysis Dashboardsimplifying the process of determining your current patch status and gaps


PATCH & UPDATE MANAGEMENT PROGRAM-


FOXGUARD SECURITY SOLUTIONS AROUND THE WORLD

4 FoxGuard Solutions builds customized programs that function as an extension of the customer’s organization, enabling them to provide easy to use, easy to manage cyber security to their controls while ensuring their reliability.

4 FoxGuard is currently managing&

114 Companies

181 Sites

36 States

30 Countries


COMPANIES USING FOXGUARD-


PATCHING.-How Hard Can It Be?-

Harder than you think. Trust us.&


WHAT IS A PATCH?-

P A T C H

UPDATE

UPGRADE

FIRMWARE ENHANCEMENT

SERVICE BULLETIN

– Feature Enhancements And / Or Security Patches

– Focus Is On The Security Patches, As These Address Vulnerabilities To Their Company (Not To Mention Compliance Requirements)


WHAT NEEDS PATCHES & UPDATES-

OPERATING 3RD PARTY SYSTEMS APPLICATIONS

NETWORK DEVICES

FIELD DEVICES

SUPPORTED ASSETS


WHY?-

Why is Patch Management important?-

4 Energy Utilities are high-risk targets

4 Patches are crucial 4 189 known vulnerabilities in ICS in 2015* 4 26 had exploits available* 4 170 had patches available*

4 NERC CIP says so 4 CIP-007-6 R2.1, R2.2, R2.3, R2.4 4 Large fines for failure to comply

*Kaspersky Labs Industrial Control Systems Vulnerabilities Statistics


THE STRUGGLE-

Current State 4 Existing solutions:

4 Are fragmented with limited coverage

4 Do not provide standardized actionable output

4 Have widely varying capability sets

4 Changing/increasing compliance scope


COMPLIANCE VS. SECURITY-

Compliance-



Compliance-



Security

4 Installing patches mitigates risks from vulnerabilities

4 Increased reliability of services may occur as a result of patching

4 “Air-gapped” is not enough

4 Compliance only mandates “security patches” be installed, but non-security patches may offer functionality that leads to security features.

4 Near real-time application of patches and/or mitigation

4 Zero Day is a real thing. (Forever Day is a real thing too. Security holes that remain unpatched. – Unfixed vulnerabilities – usually with legacy applications.)


LESSON #1-

Understanding IT vs. OT is Critical

4 OT devices are not and SHOULD NOT be treated as IT devices – Need expertise to understand and manage the differences&

4 Not all vendors report current patch status every month so you must contact them directly. (Keep track of these vendors.)

4 Regular patch notification for some OT vendors is a new&concept&


LESSON #2-

Public AND Private Patches – Which are Which?

4 Approximately half of all EDS have “Private” patches 4 Not readily available on the Internet 4 Must securely track credentials 4 Understand requirements for obtaining this information

4 Private Portal, Current Support Contract, Call, Newsletter, etc.


LESSON #3-

Patch Analysis Accuracy is Difficult-

4 Documenting the process (where to go, how to mine, etc.) is an on-going effort

4 The mining procedure changes with each vendor

4 Vendors are known to change their process

4 Some products can be intricate and time consuming to find details 4 Ex. Cisco –

4 Security bugs are listed individually 4 Security ratings are buried within Release Notes 4 Multiple CVE’s may be listed


LESSON #3-


LESSON #4-

Asset Analysis is Complicated

4 On-Going effort – Keeping it “current” is hard

4 Sub-components 4 Numerous distinct software packages on HMI/SCADA devices

4 Vendor Ownership of Asset (may have many vendors that distribute a&product)&

4 Support Provider (may be different from initial Vendor)

4 Obtaining sufficient information in order to patch properly

4 IT Contractors may help but are not bullet proof

4 Aggregate lists may not be sufficient – Each asset may be different, even if it seems the same (i.e., serial number, patch status)


LESSON #4-


LESSON #5-

Security vs. Non-Security Determination takes Knowledge-

4 Not all vendors provide security rating

4 Need the resources and expertise to determine, if a vendor doesn’t provide

4 CVE Information and CVSS Scores are helpful, but are not alwaysprovided&

4 Vulnerabilities are not always updated by the vendor


LESSON #6-

A Patch is not a Patch is not a Patch

4 Several types of patches: 4 Cumulative – One installation includes previous installation (Microsoft’s

new model)

4 Independent – Each patch can be installed independent from others (Current patch status may be difficult to track)

4 Primary/Dependent – Must have a previous installation before installing the latest (Keep track of what installed)

4 Patches can be changed/rereleased/retracted

4 Back Dating patches is possible (Vendor releases today but is dated three months ago.)


LESSON #6-

October 24, 2016

Three Patches

December 2, 2016


LESSON #6-

October 24, 2016

Four Patches

December 28, 2016


LESSON #7-

Maintaining Evidence is Hard

4 Types of Documentation 4 Patch Documentation

4 Proof you checked 4 Record status at the time you checked

4 EOS Documentation 4 Not every product lives forever 4 Audit-ready documentation must be created 4 Indicate change in status over the course of product lifetime

4 Time Consuming

4 What is appropriate – “Audit-worthy”

4 Maintaining – Keeping up to date, as well as storage of documents

4 Audit trail


LESSON #8-

Your Calendar with Many Sources is Timely to Manage-


LESSON #9-

Time & Resource Intensive

4 Patch Discovery takes MANY hours every month 4 Contact vendors 4 Who to call? 4 Email? 4 Website? 4 Newsletter? 4 Automatically provided? 4 Maintain timeline for each vendor – 35 days? 4 Documentation, documentation, documentation


LESSON #10-

Validation is Intricate 4 Scope

4 What to test, What level of detail (registry change applied to whichpatch, file versions, specific changes in a shared XML file, etc.

4 Resources 4 People -Need the right people with the right training to understand this

process 4 Equipment – Needs to have access to a representative system

(production/lab, physical/virtual

4 Test Equipment – Need equipment different from what you need in&production in order to test the system&


LESSON #10-

Validation Standard


WHY YOU SHOULD CARE?-

The Risks:

4 Safety: 4 Improperly Patched - “Brick” a device or Create a false sense of

security 4 Temporal Vulnerability with a missed patch – The longer you go

without patching, the greater your vulnerability/risk

4 Reliability: 4 Impact on The Grid? Reliability is critical.

4 Efficiency: 4 Stay focused on the job at hand and leave patching to the experts 4 Compliance Risks – Fines for non-compliance


ONE MORE TIME-

The Lessons:

4 Understanding IT vs. OT is Critical 4 Public AND Private Patches – Which are Which? 4 Patch Analysis Accuracy is Difficult 4 Asset Analysis is Complicated 4 Security vs. Non-Security Determination takes Knowledge 4 A Patch is not a Patch is not a Patch 4 Maintaining Evidence is Hard 4 Your Calendar with Many Sources is Timely to Manage 4 Time & Resource Intensive 4 Validation is Intricate

IS YOUR PROGRAM ORGANIZED? ARE YOU READY?


THE BENEFITS OF A HEALTHY PATCHING PROGRAM

End User Benefits 4 Centralizes patch and update information 4 Supports programmatic equipment querying using automation and a common toolset 4 Simplifies association between software and available patches / updates 4 Lower cost with scalability of the program 4 More accurate information 4 Recurring delivery of information

Cyber Security Advancements 4 Promotes end user awareness around patching, presence of vulnerabilities and

change management processes&

4 Provides common security classification in absence of vendor classification 4 Considers named sub-components and libraries to provide more comprehensive

security assessment&4 Reduces likelihood of incorrect patch application 4 Standardizes presentation of patch information to end user


QUESTIONS?-

QUESTIONS? FoxGuard Solutions Offers: 4 Patch Management

4 Security Services

4 Integrated Security Solutions

4 Industrial Computers

4 And Many Other Custom Solutions

For more information please contact:

Michele Wright Lindsey Hale Product Manager Program Manager (540) 382 – 4234 x244 (540) 382 – 4234 x108 [email protected] [email protected]


mailto:[email protected]

mailto:[email protected]

NEED MORE HELP-


WANT TO LEARN MORE?

patching lessons learned: an interesting perspective · patching lessons learned: an interesting...

Documents