![Page 1: Disaster Recovery Planningjschauma/615A/... · 2012. 4. 26. · A Presentation by Vincent Lipoma Stevens Institute of Technology CS 615 April 2012. ... Load Balancer 678910 Model](https://reader034.vdocuments.us/reader034/viewer/2022051822/5fec16c4aacf4c7ab358b131/html5/thumbnails/1.jpg)
Disaster Recovery Planning
A Presentation by Vincent Lipoma
Stevens Institute of Technology
CS 615 April 2012
![Page 2: Disaster Recovery Planningjschauma/615A/... · 2012. 4. 26. · A Presentation by Vincent Lipoma Stevens Institute of Technology CS 615 April 2012. ... Load Balancer 678910 Model](https://reader034.vdocuments.us/reader034/viewer/2022051822/5fec16c4aacf4c7ab358b131/html5/thumbnails/2.jpg)
Agenda
1. What is and When Should You Have a DRP?
2. How is a DRP Made?
3. Real Life Disaster Stories
4. DR Tools – To the Cloud!
![Page 3: Disaster Recovery Planningjschauma/615A/... · 2012. 4. 26. · A Presentation by Vincent Lipoma Stevens Institute of Technology CS 615 April 2012. ... Load Balancer 678910 Model](https://reader034.vdocuments.us/reader034/viewer/2022051822/5fec16c4aacf4c7ab358b131/html5/thumbnails/3.jpg)
A Disaster Recover Plan is…
• A plan to follow in the event of an emergency
• A list of contact information
• Documentation - Instructions on how to keep the system running and bring them back.
• Normally a physical and well distributed document
![Page 4: Disaster Recovery Planningjschauma/615A/... · 2012. 4. 26. · A Presentation by Vincent Lipoma Stevens Institute of Technology CS 615 April 2012. ... Load Balancer 678910 Model](https://reader034.vdocuments.us/reader034/viewer/2022051822/5fec16c4aacf4c7ab358b131/html5/thumbnails/4.jpg)
When should a DRP be made?
• It should always exist, even in an extremely small project or business.
• The DRP should be taken seriously when the business starts to grow, or when there are legal obligations to protect certain data (HIPAA)
• There should always be a DRP when the resources exit to make a backup.
![Page 5: Disaster Recovery Planningjschauma/615A/... · 2012. 4. 26. · A Presentation by Vincent Lipoma Stevens Institute of Technology CS 615 April 2012. ... Load Balancer 678910 Model](https://reader034.vdocuments.us/reader034/viewer/2022051822/5fec16c4aacf4c7ab358b131/html5/thumbnails/5.jpg)
How is a Disaster Recovery Plan Created?
1. What is and When Should You Have a DRP?
2. How is a DRP Made?
3. DR Tools – To the Cloud!
4. Real Life Disaster Stories
![Page 6: Disaster Recovery Planningjschauma/615A/... · 2012. 4. 26. · A Presentation by Vincent Lipoma Stevens Institute of Technology CS 615 April 2012. ... Load Balancer 678910 Model](https://reader034.vdocuments.us/reader034/viewer/2022051822/5fec16c4aacf4c7ab358b131/html5/thumbnails/6.jpg)
Understand the Scope
• A DRP is a top level document.– Upper management needs a plan
before you do
• The System Administrator’s DRP does not exist if the entire company has no DRP.
• A DRP is not just for terrorist attacks and hurricanes – it’s for any ‘big problem’ you could face.
• When management has a plan, you can go ahead with your DRP.
• Company Size and Budget affect your DRP
![Page 7: Disaster Recovery Planningjschauma/615A/... · 2012. 4. 26. · A Presentation by Vincent Lipoma Stevens Institute of Technology CS 615 April 2012. ... Load Balancer 678910 Model](https://reader034.vdocuments.us/reader034/viewer/2022051822/5fec16c4aacf4c7ab358b131/html5/thumbnails/7.jpg)
The DRP Document
• The document generally has this outline:
– Contact Information
– System Assessment
– Risk Assessment
• Vulnerabilities
• Probabilities
– Risk Mitigation
![Page 8: Disaster Recovery Planningjschauma/615A/... · 2012. 4. 26. · A Presentation by Vincent Lipoma Stevens Institute of Technology CS 615 April 2012. ... Load Balancer 678910 Model](https://reader034.vdocuments.us/reader034/viewer/2022051822/5fec16c4aacf4c7ab358b131/html5/thumbnails/8.jpg)
Contact Information
• More than a Big List of Important People
• A chain of command – Who’s in charge of what, and who’s their backup?
• “If the data center blew up, who do I call first? Who’s second?”
![Page 9: Disaster Recovery Planningjschauma/615A/... · 2012. 4. 26. · A Presentation by Vincent Lipoma Stevens Institute of Technology CS 615 April 2012. ... Load Balancer 678910 Model](https://reader034.vdocuments.us/reader034/viewer/2022051822/5fec16c4aacf4c7ab358b131/html5/thumbnails/9.jpg)
The Important People
• The CEO or Senior Management
– Your Boss’ boss
• Head of IT
• Chief of IT Security
– (If you have it)
• The DRP Author
• Whoever is needed to make the DRP work!
![Page 10: Disaster Recovery Planningjschauma/615A/... · 2012. 4. 26. · A Presentation by Vincent Lipoma Stevens Institute of Technology CS 615 April 2012. ... Load Balancer 678910 Model](https://reader034.vdocuments.us/reader034/viewer/2022051822/5fec16c4aacf4c7ab358b131/html5/thumbnails/10.jpg)
System Assessment
• Objective: “What constitutes my system?”
– Hardware, Software, Data
![Page 11: Disaster Recovery Planningjschauma/615A/... · 2012. 4. 26. · A Presentation by Vincent Lipoma Stevens Institute of Technology CS 615 April 2012. ... Load Balancer 678910 Model](https://reader034.vdocuments.us/reader034/viewer/2022051822/5fec16c4aacf4c7ab358b131/html5/thumbnails/11.jpg)
System Assessment
• Aside from components, there is data
• Biggest challenge is in understanding what data is important
• …then understanding what data is *most* important
• Finally, what else keeps the system running?
![Page 12: Disaster Recovery Planningjschauma/615A/... · 2012. 4. 26. · A Presentation by Vincent Lipoma Stevens Institute of Technology CS 615 April 2012. ... Load Balancer 678910 Model](https://reader034.vdocuments.us/reader034/viewer/2022051822/5fec16c4aacf4c7ab358b131/html5/thumbnails/12.jpg)
List of System Components
Item Serial # Description Manufacturer Responsibility
1 TB Hard Drive 012345 1 TB RAID 0 HD Western Digital Bob Williams
Load Balancer 678910 Model 240 Barracuda Shaun Jones
… … … … …
Acer Inspire 246810 Net book Acer Vincent Lipoma
Item Serial Key Description Distributer Responsibility
Apache 2.2 None Webserver Apache Bob Williams
MS SQL 2000 678910 Model 240 Microsoft Shaun Jones
… … … … …
Company website
None Version 2.0 of the website
Company Vincent Lipoma
![Page 13: Disaster Recovery Planningjschauma/615A/... · 2012. 4. 26. · A Presentation by Vincent Lipoma Stevens Institute of Technology CS 615 April 2012. ... Load Balancer 678910 Model](https://reader034.vdocuments.us/reader034/viewer/2022051822/5fec16c4aacf4c7ab358b131/html5/thumbnails/13.jpg)
Risk Assessment
• The fun part – thinking of every possible thing that can break the system
• Risk assessment encompasses all risks – fire, natural disaster, malicious intent, accident, etc
![Page 14: Disaster Recovery Planningjschauma/615A/... · 2012. 4. 26. · A Presentation by Vincent Lipoma Stevens Institute of Technology CS 615 April 2012. ... Load Balancer 678910 Model](https://reader034.vdocuments.us/reader034/viewer/2022051822/5fec16c4aacf4c7ab358b131/html5/thumbnails/14.jpg)
Terminology
• Risk: A threat to an asset
• Likelihood: The probability of a risk affecting an asset
• Impact: The damage a risk may have
• Control: Preventative measure to minimize risk
![Page 15: Disaster Recovery Planningjschauma/615A/... · 2012. 4. 26. · A Presentation by Vincent Lipoma Stevens Institute of Technology CS 615 April 2012. ... Load Balancer 678910 Model](https://reader034.vdocuments.us/reader034/viewer/2022051822/5fec16c4aacf4c7ab358b131/html5/thumbnails/15.jpg)
Risk Assessment Chart
Threat Description Actions Probability Impact
Hackers Wants to break the system for a
challenge
Social EngineeringSystem Intrusion
5 High
Terrorist Wants money ordestruction
Bombing/PhysicalSystem Attack
System Penetration
3 High
Disgruntled Employee
Wants revenge System IntegrityAbuse
6 Medium
Fire Partial or complete property
destruction
Natural DisasterAccident
8 High
Zombies [Bot] Cyber Crime Denial of Service 4 Medium
Fraud Crime Theft of MoneyTheft of Product
6 Low
Zombies [Real] Eat Brains Physical Attack 1 High
![Page 16: Disaster Recovery Planningjschauma/615A/... · 2012. 4. 26. · A Presentation by Vincent Lipoma Stevens Institute of Technology CS 615 April 2012. ... Load Balancer 678910 Model](https://reader034.vdocuments.us/reader034/viewer/2022051822/5fec16c4aacf4c7ab358b131/html5/thumbnails/16.jpg)
External Risks
• Risks can also come from external factors such as:
– Number of Data Centers
– Data Center Geography
– Vendor Risks
– ISP
![Page 17: Disaster Recovery Planningjschauma/615A/... · 2012. 4. 26. · A Presentation by Vincent Lipoma Stevens Institute of Technology CS 615 April 2012. ... Load Balancer 678910 Model](https://reader034.vdocuments.us/reader034/viewer/2022051822/5fec16c4aacf4c7ab358b131/html5/thumbnails/17.jpg)
Risk Mitigation
• How you handle a disaster
• Assumption – Ignore it
• Avoidance – Try not to let it happen
• Limitation – Confine the damage
• Planning – Establish a procedure to follow
• Research – Understand the threat
• Transference…
![Page 18: Disaster Recovery Planningjschauma/615A/... · 2012. 4. 26. · A Presentation by Vincent Lipoma Stevens Institute of Technology CS 615 April 2012. ... Load Balancer 678910 Model](https://reader034.vdocuments.us/reader034/viewer/2022051822/5fec16c4aacf4c7ab358b131/html5/thumbnails/18.jpg)
Risk Mitigation Cont’d
• Transference - “Failure is an option”
• Cost-Benefit analysis
![Page 19: Disaster Recovery Planningjschauma/615A/... · 2012. 4. 26. · A Presentation by Vincent Lipoma Stevens Institute of Technology CS 615 April 2012. ... Load Balancer 678910 Model](https://reader034.vdocuments.us/reader034/viewer/2022051822/5fec16c4aacf4c7ab358b131/html5/thumbnails/19.jpg)
DRP Maintenance
• Ensure everyone has access to the DRP
• Keep physical Copies
• Know where it is.
– “I can’t seem to find it at the moment…” CWIE Employee
• As the company grows, maintain the DRP.
![Page 20: Disaster Recovery Planningjschauma/615A/... · 2012. 4. 26. · A Presentation by Vincent Lipoma Stevens Institute of Technology CS 615 April 2012. ... Load Balancer 678910 Model](https://reader034.vdocuments.us/reader034/viewer/2022051822/5fec16c4aacf4c7ab358b131/html5/thumbnails/20.jpg)
Sample DRP: Contact Info
![Page 21: Disaster Recovery Planningjschauma/615A/... · 2012. 4. 26. · A Presentation by Vincent Lipoma Stevens Institute of Technology CS 615 April 2012. ... Load Balancer 678910 Model](https://reader034.vdocuments.us/reader034/viewer/2022051822/5fec16c4aacf4c7ab358b131/html5/thumbnails/21.jpg)
Sample DRP: External Contacts
![Page 22: Disaster Recovery Planningjschauma/615A/... · 2012. 4. 26. · A Presentation by Vincent Lipoma Stevens Institute of Technology CS 615 April 2012. ... Load Balancer 678910 Model](https://reader034.vdocuments.us/reader034/viewer/2022051822/5fec16c4aacf4c7ab358b131/html5/thumbnails/22.jpg)
Sample DRP: Define Important Data
![Page 23: Disaster Recovery Planningjschauma/615A/... · 2012. 4. 26. · A Presentation by Vincent Lipoma Stevens Institute of Technology CS 615 April 2012. ... Load Balancer 678910 Model](https://reader034.vdocuments.us/reader034/viewer/2022051822/5fec16c4aacf4c7ab358b131/html5/thumbnails/23.jpg)
Sample DRP: Risk Assessment
![Page 24: Disaster Recovery Planningjschauma/615A/... · 2012. 4. 26. · A Presentation by Vincent Lipoma Stevens Institute of Technology CS 615 April 2012. ... Load Balancer 678910 Model](https://reader034.vdocuments.us/reader034/viewer/2022051822/5fec16c4aacf4c7ab358b131/html5/thumbnails/24.jpg)
DRP in Action
• Hurricane Irene – Veeam Technology
![Page 25: Disaster Recovery Planningjschauma/615A/... · 2012. 4. 26. · A Presentation by Vincent Lipoma Stevens Institute of Technology CS 615 April 2012. ... Load Balancer 678910 Model](https://reader034.vdocuments.us/reader034/viewer/2022051822/5fec16c4aacf4c7ab358b131/html5/thumbnails/25.jpg)
DR Stories: Veeam
• Hurricane Irene vs Veeam Main Office (and 4 smaller offices)
• Small offices contain VOIP services that route to the main office (data center)
• …and the data center was right next to a levee.
![Page 26: Disaster Recovery Planningjschauma/615A/... · 2012. 4. 26. · A Presentation by Vincent Lipoma Stevens Institute of Technology CS 615 April 2012. ... Load Balancer 678910 Model](https://reader034.vdocuments.us/reader034/viewer/2022051822/5fec16c4aacf4c7ab358b131/html5/thumbnails/26.jpg)
DR Stories: Veeam
• “I loaded up a few core servers into my Jeep that I didn’t have located at DR site … Having this hardware with me was one of the best decisions I made during the entire ordeal. The desktop guys got all of our disaster workstations loaded up in a van and prepped for the ride to DR site which thankfully is only 25 minutes away, on top of a mountain.”
![Page 27: Disaster Recovery Planningjschauma/615A/... · 2012. 4. 26. · A Presentation by Vincent Lipoma Stevens Institute of Technology CS 615 April 2012. ... Load Balancer 678910 Model](https://reader034.vdocuments.us/reader034/viewer/2022051822/5fec16c4aacf4c7ab358b131/html5/thumbnails/27.jpg)
DR Stories: Veeam
• “I learned very quickly where our weaknesses were in our disaster plan, it was an area that is often overlooked, communication”
• “Not a single transaction was lost and the company continued to function normally, a true testament to a successful disaster recovery operation.”
![Page 28: Disaster Recovery Planningjschauma/615A/... · 2012. 4. 26. · A Presentation by Vincent Lipoma Stevens Institute of Technology CS 615 April 2012. ... Load Balancer 678910 Model](https://reader034.vdocuments.us/reader034/viewer/2022051822/5fec16c4aacf4c7ab358b131/html5/thumbnails/28.jpg)
DR Stories: Melinda Martin
• Hurricane Ike – Melinda Martin of TFI Resources
• “They had a few laptops and a tower that were to be used in deployment but there was nothing on paper.”
![Page 29: Disaster Recovery Planningjschauma/615A/... · 2012. 4. 26. · A Presentation by Vincent Lipoma Stevens Institute of Technology CS 615 April 2012. ... Load Balancer 678910 Model](https://reader034.vdocuments.us/reader034/viewer/2022051822/5fec16c4aacf4c7ab358b131/html5/thumbnails/29.jpg)
DR Stories
• After some DR maintenance…
• “TFI had leased a small office in Austin to deploy to.”
• “We were able to connect to databases and files at the colo but we had no email. Our email replication solution had failed. Plan B was we did have a website TFIEmergency.com that we broadcast to so we posted updates for mass information…”
![Page 30: Disaster Recovery Planningjschauma/615A/... · 2012. 4. 26. · A Presentation by Vincent Lipoma Stevens Institute of Technology CS 615 April 2012. ... Load Balancer 678910 Model](https://reader034.vdocuments.us/reader034/viewer/2022051822/5fec16c4aacf4c7ab358b131/html5/thumbnails/30.jpg)
DR Stories
• It was also a learning experience:
– “We learned a lot. The first thing was to lease a bigger space. TFI now has two colo facilities”
• Prepare to make friends too:
– “…the team bonded and those of us who deployed for IKE have a special respect for each other.”
![Page 31: Disaster Recovery Planningjschauma/615A/... · 2012. 4. 26. · A Presentation by Vincent Lipoma Stevens Institute of Technology CS 615 April 2012. ... Load Balancer 678910 Model](https://reader034.vdocuments.us/reader034/viewer/2022051822/5fec16c4aacf4c7ab358b131/html5/thumbnails/31.jpg)
When the DRP fails…
“We had to go into New Orleans under armed guard to regain access to documents and email that had not yet been captured by the tape backup system prior to Katrina’s landfall.”
- Yehuda Cagen from Xvand Technology Corporation
![Page 32: Disaster Recovery Planningjschauma/615A/... · 2012. 4. 26. · A Presentation by Vincent Lipoma Stevens Institute of Technology CS 615 April 2012. ... Load Balancer 678910 Model](https://reader034.vdocuments.us/reader034/viewer/2022051822/5fec16c4aacf4c7ab358b131/html5/thumbnails/32.jpg)
When there is no DRP…
• 1996 Docklands Bombing
![Page 33: Disaster Recovery Planningjschauma/615A/... · 2012. 4. 26. · A Presentation by Vincent Lipoma Stevens Institute of Technology CS 615 April 2012. ... Load Balancer 678910 Model](https://reader034.vdocuments.us/reader034/viewer/2022051822/5fec16c4aacf4c7ab358b131/html5/thumbnails/33.jpg)
Tools
1. What is and When Should You Have a DRP?
2. How is a DRP Made?
3. Real Life Disaster Stories
4. Disaster Recovery – To the Cloud!
![Page 34: Disaster Recovery Planningjschauma/615A/... · 2012. 4. 26. · A Presentation by Vincent Lipoma Stevens Institute of Technology CS 615 April 2012. ... Load Balancer 678910 Model](https://reader034.vdocuments.us/reader034/viewer/2022051822/5fec16c4aacf4c7ab358b131/html5/thumbnails/34.jpg)
The Cloud
• Remember that every DR is unique – therefore the tools used will be unique
• The cloud is attractive to small business –offsets costs of Disaster Recovery Planning
– No need to buy a datacenter, backup servers, desktop… just purchase a service.
![Page 35: Disaster Recovery Planningjschauma/615A/... · 2012. 4. 26. · A Presentation by Vincent Lipoma Stevens Institute of Technology CS 615 April 2012. ... Load Balancer 678910 Model](https://reader034.vdocuments.us/reader034/viewer/2022051822/5fec16c4aacf4c7ab358b131/html5/thumbnails/35.jpg)
The Cloud’s Big Advantages
• Provides different recovery options:
– Send data to and retrieve from the cloud
– Go straight to and use cloud instances
• Very fast recovery
![Page 36: Disaster Recovery Planningjschauma/615A/... · 2012. 4. 26. · A Presentation by Vincent Lipoma Stevens Institute of Technology CS 615 April 2012. ... Load Balancer 678910 Model](https://reader034.vdocuments.us/reader034/viewer/2022051822/5fec16c4aacf4c7ab358b131/html5/thumbnails/36.jpg)
Cloud Shortcomings
• Where is my data?– “According to continual updates from Japan’s Ministry of Internal Affairs
and Communication and the Japanese office of news outlet ZDNet, about one dozen major data centers and cloud facilities had reported back with varying degrees of problems, though no loss of life.” – Kern
– (On a related note, all major data centers from Yahoo and Amazon were back online within a matter of hours after the Japanese Earthquake)
• Cloud Reliability– Major EBS Outages of 2011
• Backing up from the cloud can require time (and bandwidth!)
![Page 37: Disaster Recovery Planningjschauma/615A/... · 2012. 4. 26. · A Presentation by Vincent Lipoma Stevens Institute of Technology CS 615 April 2012. ... Load Balancer 678910 Model](https://reader034.vdocuments.us/reader034/viewer/2022051822/5fec16c4aacf4c7ab358b131/html5/thumbnails/37.jpg)
Verdict
• Just like any tool, know how to use it
• It’s not a magic pill, but it does provide flexibility and possible cost savings
• It’s only useful if you know how to use it
![Page 38: Disaster Recovery Planningjschauma/615A/... · 2012. 4. 26. · A Presentation by Vincent Lipoma Stevens Institute of Technology CS 615 April 2012. ... Load Balancer 678910 Model](https://reader034.vdocuments.us/reader034/viewer/2022051822/5fec16c4aacf4c7ab358b131/html5/thumbnails/38.jpg)
3 Kinds of Backup Sites
• A backup site will boil down to either the cloud, a data center, or a private server. All three however can have the following states:
• Cold – built but not up to date
• Warm – up to date but idle
• Hot – up to date files and already serving traffic normally
![Page 39: Disaster Recovery Planningjschauma/615A/... · 2012. 4. 26. · A Presentation by Vincent Lipoma Stevens Institute of Technology CS 615 April 2012. ... Load Balancer 678910 Model](https://reader034.vdocuments.us/reader034/viewer/2022051822/5fec16c4aacf4c7ab358b131/html5/thumbnails/39.jpg)
Backing Up and Synching Files
• Drop Box
• VM Ware
• Subversion
• Sometimes tools have built in backups
– Windows System Restore
– SQL Dumps
• Snapshots
![Page 40: Disaster Recovery Planningjschauma/615A/... · 2012. 4. 26. · A Presentation by Vincent Lipoma Stevens Institute of Technology CS 615 April 2012. ... Load Balancer 678910 Model](https://reader034.vdocuments.us/reader034/viewer/2022051822/5fec16c4aacf4c7ab358b131/html5/thumbnails/40.jpg)
Backing Up Files
• The Point is that tools exist for every organization of every size – look for them! Options are not limited.
![Page 41: Disaster Recovery Planningjschauma/615A/... · 2012. 4. 26. · A Presentation by Vincent Lipoma Stevens Institute of Technology CS 615 April 2012. ... Load Balancer 678910 Model](https://reader034.vdocuments.us/reader034/viewer/2022051822/5fec16c4aacf4c7ab358b131/html5/thumbnails/41.jpg)
Summary
• Have a Plan– Trouble Making One? Look up National Institute of Standards document
sp800-30
• Make Everyone Aware of the Plan
• Practice Your Plan
• Know Your Tools
![Page 42: Disaster Recovery Planningjschauma/615A/... · 2012. 4. 26. · A Presentation by Vincent Lipoma Stevens Institute of Technology CS 615 April 2012. ... Load Balancer 678910 Model](https://reader034.vdocuments.us/reader034/viewer/2022051822/5fec16c4aacf4c7ab358b131/html5/thumbnails/42.jpg)
Links and Sources
http://www.linuxtopia.org/online_books/redhat_linux_sysadmin_intro/s1-disaster-recovery.html
DR Story: http://www.virtualizationimpact.com/?p=1854
DR story 2: http://enterprisefeatures.com/2011/11/real-life-disaster-recovery-stories/
DR Story 3: http://ezinearticles.com/?The-Day-an-IRA-Bomb-Took-Out-the-Data-Center-of-a-Major-Japanese-Bank-and-the-Turmoil-That-Followed&id=2327438
NIST: http://csrc.nist.gov/publications/nistpubs/800-12/800-12-html/index.html
Cool Info:http://www.information-management.com/news/Japan_earthquake_tsunami_data_center_cloud-10019922-1.html
Some Information from Gene Super, former IT VP of “Totsy” child clothes/parental items distributor
Some information from Cave Creek Webhosting (CWIE of Tempe Arizona)
More Links in PPT Annotations