beyond the change - using tripwire to promote consistency and roi
DESCRIPTION
TRANSCRIPT
Beyond the Change: Mastering Configuration Controls
Daniel J Blander, CISM, CISSP
Agenda
[ The Need ][ The Approach ][ Configuration Standards ][ Change Control – SDLC Validation ][ The Results ]
Presenter
Presentation Notes
Today I’m going to talk about two fairly recent experiences with using Tripwire – how two companies I worked with looked beyond just checking for changes and used Tripwire to: Improve effectiveness, efficiency and service levels through the ability to track compliance with configuration standards Verify the change control process to improve consistency of delivery I’m going to start by talking about the needs we faced – first what I will call the perceived risks, and then what turned out to be real (and sellable) needs. Then I’ll talk about the process we used – the approach – to address those needs. I’m going to talk about the actual solutions we built. Lastly I’m going to talk about the results we achieved. I’m going to steer this talk towards how you can apply these examples to your potential customers. Take note - these are real examples. While I have some information I have to hold back for confidentiality, the approaches are real, and are in operation today.
The Need
[ Customer Needs ] Hawaiian Airlines - $1B International Airline
• Sarbanes-Oxley• PCI
National Retailer - $1.45B, ~900 Stores• PCI• Sarbanes-Oxley
Presenter
Presentation Notes
The two companies I worked with had very similar issues. The first is Hawaiian Airlines – their primary focus in 2006 was on Sarbanes-Oxley. They had just emerged from bankruptcy and suddenly found themselves being subject to Sarbanes-Oxley (a bit late to the game as it were). They also knew that PCI compliance was something they wanted to achieve, but it was secondary. I was contracted in February of 2006 to guide them through the process of designing and building the controls to achieve these objectives. The second company is a shy National Retailer. I was asked to be the Acting Head (Sr. Dir) of their Information Security Department in 2007. Their stated primary goal was to ensure PCI compliance, and secondarily improve controls that would better support Sarbanes-Oxley. Compliance was their perceived need at both companies. The execs and Board of Directors viewed compliance as “What will keep me out of Jail and off the front page”. They wanted solid controls to meet these needs.
The Need
[ PCI ] Required Standards
[ IT General Controls (SOX 404) ] Verification of Change
[ Universal Goals ] Improving Quality of Production Implementation Consistency of Production Systems Reduce Time to Test + Audit
Presenter
Presentation Notes
We are all familiar with how PCI and SOX are drivers that beg for an implementation of Tripwire. PCI’s requires “file integrity monitoring software”, as well as maintaining configuration standards for all systems. SOX and PCI both require solid Software Development Life Cycle that includes testing and promotion controls. While compliance is often the perceived big driver, we also found an undercurrent of needs that were outside of compliance. Here is what I mean: Example 1: The retailer’s IT organization had a need to improve the consistency of its delivery. The VP had a mantra that Consistent Standards equaled Consistent Support. Imagine the ROI of having a consistent and predictable environment (patched to the same level, configured the same way) Example 2: Having an ability to measure the accuracy of implementations of systems and installation of newly developed software. Holding accountable those who were either doing the implementations, or those who were writing the instructions on how to implement. Example 3: Reducing the time it took to perform mandatory audits and testing of changes and configuration standards, while also improving the accuracy, and allowing for greater frequency of testing. These examples spoke to Return on Investment – because they were about quality of delivery, supportability, time spent repairing, and time distracted by audits and assessments. Why is this important? As security professionals we can always talk about compliance from a perspective of Fear, Uncertainty and Doubt, but based on recent conversations I’ve had with many CSO’s, executives are becoming very callous to FUD. Their response has become “Well it hasn’t happened to us, so I don’t see the concern.” I would point out that not very long after TJX had its breach, its stock price was actually up!
The Approach
[ Focus on Maturity ] Effectiveness of Controls Efficiency of Testing + Audit Repeatability + Service Levels
[ Develop the Process First – Tool Later ]
Presenter
Presentation Notes
So at both Hawaiian Airlines, and the retailer, we took the same approach. Our goal was building maturity: Effectiveness of the controls – repeatable and accurate Efficiency – remove time as a constraint for performing audits/tests. And most important for the retailer: Focus on improving service levels – through repeatable processes and building a culture of consistency and measurable success. Key to our approach was working on the process first before settling on a product. Nearly all the things we do with computers and applications can (and has been done) with paper, pencil (or stone and chisel). The process is the same – the tool just makes it more efficient. A bad process is just bad faster with a tool. So we first laid out our needs, designed a process that worked within the organization, and then looked at the tool as a way to make the process more effective or more efficient.
The Approach
[ Rule 1: “Trust is not a Control.” ]- KPMG Sr. Mgr.
[ Rule 2: “Always Give Something Back.” ]- Daniel Blander
Presenter
Presentation Notes
My favorite quote. I had heard this before, but this became our running joke at Hawaiian Airlines. We had great people at both companies. We had incredible talent, but it came down to: Good Intentions do not offset all Bad Intentions Human Error is always a factor Checks and Balances are the only way to prevent against both Then there is one of my key mantra’s (right after “Compliancy is not a word”): Whatever we implemented better give something back to the company (since too often security was seen as “taking away”). My team was required to always look at how we could use every one of our initiatives as a way to make things better for the users. I usually use this as my ROI measure rather than mitigation of risk because the users and the business can associate with them and see them (“What have we done for them lately” syndrome). We made sure the benefit message went out and that we were vocal about the benefits the users would appreciate and recognize. It was enough for just us (security) to recognize the security benefits. We got that. We wanted the users to hear and see the other benefits. (Too often security tries to justify itself through security. How about how it helps justify everything else?)
Best Practice I: Configuration Standards
[ Step 1: Develop + Document Standards ]
[ Step 2: Configure Your Systems ]
[ Step 3: Tripwire Configuration Assessment ]
[ Execution Details ] Eight Hours of Work per System Type Make it Your Own – Customize and Test
Presenter
Presentation Notes
At the retail company, because of PCI, we started with Configuration Standards. We had done audits, and there was a vague notion of what “Standard” was floated around, but there was nothing written down. Nothing we could point at and say “This Is It!”. Problem 1 identified. Keeping in mind that the process is step 1, we started by researching what regulatory standards demanded. Then we looked at leading practices – CISecurity, NSA.gov, NIST, Microsoft, Cisco, other vendors. We combined these requirements and did a first draft. We tested it. We broke things, we revised, we broke less, we revised, etc., in a cycle. We measured against our own preferences, what the administrators needed for decent functionality, what the end users wanted for functionality, and what was manageable. A key element is having experts who understand the systems. Document the results. Without a tangible document, no one will ever have anything to point at. Next the admins actually pushed the configs out. Meanwhile, back in the Information Security office, we worked feverishly to program the standards into Tripwire Policies. We chose the CIS policies as our starting baseline since they were what we had started with, and we knew we wouldn’t have very much work to adjust them to our standard. We took about 8 hours per system type to customize the configuration standards to our liking. The Cisco devices were mostly just groupings, adding a few, and removing a few. Windows was more detailed since the amount of information is voluminous. We found that if you didn’t know REGularEXpression syntax, it was painful. A lot of trial and error. Thanks to Dan Hogland for testing many of my goofy ideas. We also made the standards our own – how they were labeled, how they were organized, and how they were reported. The flexibility was very helpful and allowed us to wrap our heads around it the way that we thought about the configuration standards.
Best Practice I: The Real Results
1 2 3 4 5 6 7 8 9 10 11 12
[ Red Line – Old Audit + Old Compliance ]
[ Green Line – New Standards + New Compliance ]
Audit Defined Standards + Tripwire
Ad-Hoc Configuration
Changes
Presenter
Presentation Notes
This slide will look a little familiar – from your marketing material, except it is my own real life example. I did a little ad-hoc analysis of configuration changes. I gathered the results from a pre-Tripwire configuration assessment that we ran at the end of 2007. There were findings and the results got some attention, and most things were fixed. At least the things that could be agreed were a standard. Just before we started the Configuration Standards project we re-evaluated the systems, and they had, for the most part, slid back to their old ways. Some things stayed fixed, but new things were broken. Then we built our standards, and implemented Tripwire. We got compliance in much the same manner as before but now we had solid standards that were thorough and everyone had agreed on. Note that the number of “compliance items” is higher. We went from around a dozen items to test, to over 30 tests on a Cisco device alone. We could also evaluate them much quicker. We were able to hold people accountable to timelines for remediation by being able to see when the did get around to fixing it. Administrators hated us because we held them to what they said they were going to do. Then the interesting part happened. They went about their usual pattern of changing things – but this time we were able to test frequently, ad-hoc, without their help or time. We caught the changes almost immediately. They still slipped, but then we got them back on the beaten path sooner. We kept doing this until they realized they couldn’t slide. It also drove them to change control more often – no ad-hoc configuration changes any more. We managed to enforce better Change Control, and awareness of what was going on in our environment went up dramatically. No unexpected changes, and a significant reduction in unexpected outages.
Best Practice I: The Real Results
[ Give Back + ROI ] Visibility to Change for All of IT Reduced Variances Reduced Testing Time - 150 Hours to 2
Presenter
Presentation Notes
What did we end up with? Remember my mantra of always giving something back to the business? The Give Back: The admins at both Hawaiian Airlines and the retail company made many a call to the Information Security team, and looked at Tripwire to find out why suddenly a system didn’t work. One famous example was the fail-over of a Windows Cluster that didn’t work right. We were able to not only identify what had been removed and who did it (all by accident), but also that there were around a dozen key differences between the two systems. Some shock, then some fixes, and the availability of the cluster went up tremendously. This led us to establish a standard for the cluster, and also do a monthly verification between the two! Easy simple tracking. It was so nice to push a button and get a result without having to mess with hours of audit time. My Analyst struggled with the simplicity. I loved being able to check ad-hoc on compliance. For me, the testing time was a key. My Information Security Analyst at the retailer hated doing Configuration Standard Audits. We had about 85 network devices alone at the corporate headquarters and our distribution center that took him about 40-60 man hours to assess (with human errors). That did not include the 1200 at the stores. With Tripwire we could check compliance in less than an hour, report on it, demand remediation (in the name of PCI!) and hold teams accountable. That was a simple savings of around $8000 per audit alone with added benefits of testing in the stores (which we could never tackle before) that probably saved us another $16,000. Imagine $24,000 in savings per audit. I wish I could measure the difference in Service Levels from day one to the present, but I unfortunately the maturity in measuring SLA’s was not there. I do know that from an audit perspective results improved, and we noticed improvements in reliability of devices when the standards stayed in line.
Best Practice II: Change Control Verification
Program Change Management - ApplicationsB
usin
ess
Uni
tIT
Sec
urit
yB
usin
ess
Ow
ner
Dev
elop
erC
onfig
urat
ion
Man
ager
Cha
nge
Con
trol
B
oard
DEVELOPMENT TESTING APPROVAL POST-REVIEW
Input User Change Request
Request is Reviewed
and Approved
Developer checks out code, makes changes,
checks code back in.
Code moved to
TEST environment
Testing Performed
Submitted to CCB for Approval
Changes moved to
Production
Post Implementation
Review
Reviews and approves
testing
INITIATION
Review Changes in Tripwire Promote by Compare
IMPLEMENTATION
Code reviewed by Team
Presenter
Presentation Notes
So on to Challenge #2 – Here is Hawaiian Airlines and the retailer’s change control process. A request is made, it is developed, it is put into a Test environment, it is tested, and if it is okay, and approved, it is put into production. Here is where the challenge comes. Changes could be made during implementation – not just developer programming changes, but mis-configuration or mis-installation by the person doing the install. These are critical risks to SOX – giving assurance the changes are properly controlled. We also found there was a need to improve the quality of implementations – reduced time diagnosing and fixing an installation, and improving the consistency of our implementations. So the question: How can you verify that a change made in production is really what was intended. Does the change match the test environment? Who can verify this information? (Remember “Trust is not a Control”) How can the process be performed without delaying implementation? Hawaiian Airlines was challenged on this issue by their auditors, and when I went to the retail company, I posed the same question to IT, and our auditors. In both situations it was a question everyone struggled with, and no one had a good answer. Fortunately by the time I got to the retailer, I already had an answer. When we built the Change Control process at Hawaiian Airlines, we looked for options. We had the good fortune of talking to Tripwire and the solution of “Promote by Compare”. Promote by compare allowed us to look at an environment, take a Tripwire “snapshot”, and then use it as a baseline to compare against for another system. Hmmm…..
Best Practice II: Change Control Verification
[ Requirements ] Segregated Environments – Dev | Test | Prod Prior to Changes Test Must Match Prod Deployment in Test is Deployment in Prod
[ Implement Tripwire – “Promote by Compare” ]
Presenter
Presentation Notes
There are some pre-requisites and a level of maturity to make this process work. Here’s what I mean: There must be fully segregated environments. Segregation includes access controls, and environments that can be at least logically isolated. If developers and production administrators can venture into each other’s environments, then you have lost necessary controls, period. The Test environment must be a spitting image of Prod. This is a fundamental concept for good QA. You’d be surprised how often this is not the case. If you test in an environment that doesn’t match prod – when you go to prod, you will likely have many issues. I can tell you many stories about this issue. The funny part is that Tripwire actually allows you to do a reverse verification. Take your prod snapshot, and then compare against test when the snapshot is made. A bit esoteric, but possible none-the-less. Lastly, however test is deployed (installed) is how Prod must be installed. You can not have two different install procedures, configuration settings, and the like. It becomes really hard to match what is completely different. Oh, I forgot one – you need Tripwire.
Best Practice II: Change Control Verification
[ The Control – Verification ] Change is Implemented in Test
• Testing Is Conducted + Approved• Snapshot of Test Environment Captured by Tripwire
Approved Change is Implemented in Production• End Users Verify Functionality• Information Security Verifies Change with Tripwire
• Promote by Reference
Presenter
Presentation Notes
Knowing what we could do with Tripwire, and that we had good, segregated environments, and that we had segregation of duties, we now built the process: A change is put into the Test environment. We took a snapshot of the system with Tripwire and made it a baseline. Once the testing process was complete, the change was approved, and it was installed in Production. Now we used the snapshot to compare the change in production against the change in the test environment. If they matched, great. If they didn’t there were questions. When there were differences, we had to investigate them and determine why they existed. In the beginning there were a lot of those differences. These settled down as we moved along and people understood that even minor differences were a big issue to us. We of course found that we still had differences. Some were do to specific configuration differences that were mandatory (an IP address, a license key, an encryption key) We still had work to do. But it was far less than what we might have had to do, and our auditors were fine with the differences and the manual checks.
Best Practice II:Change Control
[ ROI ] Trust of Auditors in Change Validity Improved SLA – Quality and Accuracy Quality of Implementation Improves
[ Musings ] Use of VMware
Presenter
Presentation Notes
The ROI for this portion was much more difficult to quantify, but was noticeable. Since neither company did this level of verification before we had no benchmark to compare against. At the retailer, we were able to see when installations were not going according to plan. It was rather amusing because we had someone without consistency, and this highlighted this issue. The control makes for a *very* mature Change Control process. Our auditors in both instances were tickled, and very interested in the process. They had never really seen this before. We felt the benefit of the increased assurance that changes were verified with certainty would give us great benefit and our external auditors certainly thought so (thank you E&Y). But the real difference was in the ability to improve the SLA’s – the quality of the implementations. To measure the quality, the accuracy, and hold people accountable should give strength to those who hold to ITIL that this is a useful process. We found at Hawaiian Airlines that one of the coolest things we could do was image the source system onto a VMWare (we had a really cool VMWare implementation and great talent). We could take snapshots from Tripwire at any time, monitor the virtual server, and do the compare. Then when we were done, we just tore the whole thing down. We had several servers which rarely saw change (usually vendor supplied systems with little patching available). These were perfect. Patch cycles and major version upgrades worked perfectly the same way. We created the VM, did the upgrade, took the snapshot, and then tested. When it was all done, we just tore down the instance, project over.
Take This Home With You
[ Process First – Then Tools ]
[ Make Sure the Effort Gives Something Back ] Real ROI – not FUD Culture of Consistency Improved Delivery Efficiency Through Automation ($24k Savings)
Presenter
Presentation Notes
So in closing I’d like to give you a few pointers to talk how with you: Sell the customer on good fundamental processes first. Don’t sell a tool for change control if they have no idea what change control is, or if they don’t understand how to go about setting up configuration standards. You will only make a bad situation a bad expensive situation, and a failure in your customer’s eyes – even though the failure is their inability to build a fundamental process. Make sure the solution gives something tangible back to as many parts of the business BESIDES IT!! Happy customers in non-IT areas makes it easy to support the expenditure. Think of how a business will think of a solution when you talk about ROI rather than FUD. Service delivery rather than risk and failures of compliance. Think of the reaction when you improve system uptime through a culture of consistency better delivery of new programs and systems efficiency and savings through the automation Your customer is really the whole business. Knowing the bigger picture of their needs will be your golden key. And you can throw in a little compliance for good measure….
Questions
Daniel Blander
(714) 815-3653
Presenter
Presentation Notes
So there I am. If you have any questions, I am open to them now, or you can contact me directly. I am open for business….