6 tools for improving it operations in ics environments
DESCRIPTION
Presented by: Jacob Kitchel, Industrial Defender Abstract: This presentation will review useful concepts and tools that can be applied by DevOps team with “Controlled Remediation”. We’ll demonstrate the application of non-security, system administration, deployment, monitoring and change tracking using tools to achieve controlled remediation. This will build a foundation through which security, compliance, and change management goals can be achieved in an automated fashion within control system environments. DevOps is a juxtaposition of the words “development” and “operations” and is meant to portray a tight relationship between the two traditionally separate roles which build and operate complex computer systems and software applications. DevOps groups work with a unified goal to rapidly and reliably deploy and manage the underlying systems which organizations rely upon to make a profit while balancing resource constraints. “Controlled Remediation” is a concept used to describe the use of automation to maintain acceptable configuration and settings on industrial cyber assets. Additionally, this presentation will discuss the variations of “Automated Remediation” and “Manual Remediation”.TRANSCRIPT
6 Tools for Improving IT Operations in ICS
Jacob Kitchel Sr. Manager, Security & Compliance
9/24/13 2
Before we begin, a little about me …
§ Serve as the internal expert on various regulatory compliance requirements and frequently speaks on ICS security related topics.
§ Past experience includes: performed >100 risk assessments, pen testing, vulnerability assessment, gap analysis, architecture review, etc.
§ Participated in Project Basecamp
§ Also has a background in security operations and monitoring.
§ Endorsed for many hilarious skills on a well known business social network
9/24/13 3
“Amateurs practice until they get it right. Professionals practice until they can’t get it wrong.”
9/24/13 4
What’s this really about?
Reducing the Chance and Impact of Failure, Increasing Reliability, and Improving System
Awareness though:
• Continuous Delivery • DevOps: Development & Operations working together • How you can use these principles and tools to improve your operations
and gain confidence in your environments
9/24/13 5
Why is Continuous Delivery Important to ME (YOU)?
• What “it” is: – Small, frequent changes to production – Actively testing every change across development and test before push to
production – Lowers risk of change – Helps to plan change better
• That ‘thing’ everyone says is the ‘right’ way to do things but it’s really hard gosh darnit!
9/24/13 6
Let’s back up: Present Day
§ Develop, Test (QA), and Production in LARGE chunks § This is called the “waterfall” model § OR “throw it over the wall” § Like it or not, you are pushing CODE whether you develop it or
not § Push changes and wait around to see if anything breaks
What you do now
Where you are now
§ Failure means § HIGH cost § SLOW recovery time § DIFFICULT to recover from
§ Great deal of UNCERTAINTY when recovering from failure
9/24/13 7
Where you want to be
Failure has a low cost
Failure has a quick recovery time
Failure is easy to recover from
You are agile when recovering from failure
You are confident when recovering from failure
Code updates, testing, and deployment are automated
Automation enables you to do more things
9/24/13 8
Continuous Delivery means…
Every change to your environment is proven to be
deployable to production with predictable results
9/24/13 9
Let’s talk about tools
9/24/13 10
In your toolbox…
Continuous
Delivery & DevOps
Version Control & Change Review
Metrics
Configuration Management
Orchestration
Dashboards
Virtualization
9/24/13 11
q Takeaway: Every change must go through version control and also be attributable to a person
q Version Control § Track versions of every change
q Change Review § Allows you to step through every change
q Available tools q Git: http://git-scm.com/
Version Control & Change Review
Metrics
Configuration Change Management
Orchestration
Virtualization
Dashboards
9/24/13 12
• A Holy Grail of Enterprise IT • Enterprise: slow, tedious, high overhead, rarely
‘correct’ electronic paper shuffling exercise • Now: Automation with an audit trail and reporting • Important: Use the same configuration across
Dev, Test, and Production • Free tools to use for practical application:
– Puppet – Chef – Ansible – Salt Stack
Version Control & Change Review
Metrics
Configuration Change Management
Orchestration
Virtualization
Dashboards
9/24/13 13
Configuration Management – 2 Approaches
Passive q Always watching q Never changing production q “Oh, we see a change. Is it ok?
Click ‘Yes’ or ‘No’” q Baseline gets updated after the
fact if ‘Yes’ q Production asset gets
manually reverted if ‘No’
Active q Always watching q Never changing production q “Oh, we see a change.
Revert that change back to the approved configuration automatically.”
q No permanent changes to production until approved configuration change
q Baseline gets updated to enable change
9/24/13 14
q “to arrange or manipulate, especially by means of clever or thorough planning or maneuvering”
q Rolling out applications and configuration changes in a specific order
q Leverage automation to reduce human error and scale
q Free Tools to enable Orchestration: § Puppet § Chef § Mcollective § Ansible § Capistrano § Fabric § WinRM § (Any automated, remote administration tool) § Your own home grown scripts
Version Control & Change Review
Metrics
Configuration Change Management
Orchestration
Virtualization
Dashboards
9/24/13 15
q Vendor specific, but they probably use VMWare § Important to have Dev, Test, and Production
environments mirrored § Use configuration management and orchestration
tools to do this! § Bonus: “backup”/redundant assets
o Example: Server2 and Workstation3 go down? You can spin up virtual instances until hardware instances recover
o Everyone wants a “do over” or “What if?” button. Get one.
Version Control & Change Review
Metrics
Configuration Change Management
Orchestration
Virtualization
Dashboards
9/24/13 16
Version Control & Change Review
Metrics
Configuration Change Management
Orchestration
Virtualization
Dashboards
q Metrics are performance ‘things’ that are measured
q Important because they help you understand how you are performing
q Continually monitor your environment so you can determine how to improve it
q Free Tools to use for Metrics: § Graphite § Logstash § Nagios
q NOTE: you must have a way to consume and evaluate metrics like…
9/24/13 17
q The *other* Enterprise IT Holy Grail q Visual representation of your operating state q Quick ‘hit’, good/bad, green/yellow/red, trending,
etc. q What do you *really* need to know?
§ Development, testing, production roll-outs § Metrics § State § Performance § Some examples:
o Assets (groups, rules, policies, etc.) o Events (all sorts of events in various metric categories,
security, compliance, changes, etc.) o Configuration o Workflow (newly discovered, promotion state)
Version Control & Change Review
Metrics
Configuration Change Management
Orchestration
Virtualization
Dashboards
9/24/13 18
q Free Tools to enable Dashboard use: § Graphite § Logstash § Bamboo § Jenkins § Cactii § Nagios
Version Control & Change Review
Metrics
Configuration Change Management
Orchestration
Virtualization
Dashboards
9/24/13 19
… and the not-so-free kind Version Control & Change Review
Metrics
Configuration Change Management
Orchestration
Virtualization
Dashboards
9/24/13 20
How does it all fit together?
Unit Tests
Platform Tests
Deliver to Staging
Application Acceptance
Tests
Deploy to Production
Post Deploy Tests
9/24/13 21
How We Can Help?
Version Control & Change Review
Metrics
Configuration Management
Orchestration
Dashboards
Virtualization
9/24/13 22
Summary
Leverage tools which can help you improve your operations and reliability
Use automation to glue the tools together
Have confidence in deployments to production
Know and See what is happening in your environment across your systems and assets
9/24/13 23
DevOps Novel
• Head over to the Industrial Defender booth – we have 25 copies to give away!
9/24/13 24
web industrialdefender.com
blog blog.industrialdefender.com
twitter @i_defender
More Information: