survivor - disaster recovery edition

© 2013 Velocity Technology Solutions, Inc.

EXPERIENCE. INNOVATION.

Survivor – Disaster Recovery Edition

©2013 Velocity Technology Solutions, Inc.

Velocity Webcast

EXPERIENCE. INNOVATION. Velocity Proprietary and Confidential

Your Host

2

• Velocity Managed Disaster Recovery Service expert

• Participated in over 250 DR tests including IBM i, Windows, and Linux platforms

• Been involved in 3 full data center recoveries

Jim Knight


Definition of a Disaster

• A sudden, unplanned, event that causes significant damage or loss to an organization.

• ANYTHING… - That stops your business from functioning

and cannot be corrected within an acceptable amount of time.

- The time factor determines whether an interruption in service is an inconvenience or a disaster and varies from organization to organization.

3


What is Disaster Recovery?

• Reaction to a sudden, unplanned, event that enables an organization to continue critical business functions until normal business operations resume.

• “…It is not enough to arrange for hardware replacement;… planning must address continuation of business operations, or business continuation.”

4


Companies Regularly Deal with Natural and Man-Made Disasters

Risk is increasing Disasters happen

Forrester Research

5


Disasters Have a Major, Adverse Impact

93% of businesses that suffer a significant loss of data are out of business within 5 years. (Bureau of Labor)

2011, Ponemon Institute© Research Report, Sponsored by Emerson Network Power

Risk is increasing Disasters happen Loss is significant

6


The Products of a DR Plan

• Who will execute the recovery actions • What is needed to continue, resume, recover or restore

business functions • Where to go to resume corporate, business &

operational functions • When business functions and operations must resume • How detailed are the procedures for continuity,

resumption, recovery or restoration

CLASSIC: WHO-WHAT-WHERE-WHEN-HOW

7


Common Issues with DR Planning

• Optimistic attitude in good times-complacency • Not organizationally prepared • Unrealistic DR plans • DR plan not aligned with recent IT integrations? • Qualified staff to execute the plan? • DR not integrated into the IT organization change

control process?

8


DR Planning is an Ongoing Process

Plan

Implement Test

Business Requirements

…to be prepared, you must regularly go through the cycle

9


Case Study: Disaster Declarations

The Storyline: • Real-world disaster and how the events unfolded • Manufacturing company with corporate data center that

supports several: - sales offices - distribution facilities - and manufacturing plants throughout the U.S

• We will refer to the organization as Smith Corporation

10


Profile: Smith Corporation

• Last DR test: 1 year ago • 100% successful - 14 hours • Tests annually with 4 IT Staff • Trained in recovery of the system

- Tapes sent offsite daily (VTL is on-site) - RTO & RPO - 24 hours - Business Signed off

12


The Disaster: Smith Corporation

• Water damage under computer room floor…on the 32nd Floor

• Severe power failure – Wednesday night at 7:00 PM • Site expected to be down well past

the weekend • Disaster Declared at 11:00 PM

4 Hours

13


The Disaster: Smith Corporation

• Person declaring disaster “Not authorized“ to declare a disaster and unfamiliar with password activation procedures.

• Hot-site begins preparation of the environment, but requests an authorized individual to call within the next hour

4 Hours

14


Startup Issues: Smith Corporation

• Hot-site equipment is available 4 hours after initial call • Smith Corp still trying to locate authorized staff

- alternates not available or no longer employed • Disaster declaration fee of US$7,500.00.

What’s this?

8 Hours

15


Recovery startup: Smith Corporation

• Utilizes hot-site infrastructure for servers and network • IBM Power: full system save available and Wednesday

night backup which was successful • X86: last full and incremental backups were successful • Tapes are requested for emergency

service delivery • All required tape media arrives:

90 minutes

9.5 Hours

16



• X86 backup policies includes: - VMDK saved to local VTL - Tape backups include:

• All file system volumes • All databases

• Issue: tapes arrive in locked box!!!

17



• IBM Power backups: system state saved on older tape media format LTO-2

• Firmware and operating system cannot be restored from contracted tape drive

• Daily procedures saved on current, new tape format: LTO-5

• Hot-site staff switched tape drives to accommodate system state save from older tape technology 12 Hours

19



Electronic backup solution (VTL) physically removed from data center and transported to hot-site

20


Off to the Races: Smith Corporation

• Procedural execution for IBM Power recovery starts • Procedural execution for Wintel x86 data layer recovery

starts • Notice goes out to the business of an

estimated restoration completion of 14 hours from NOW!

16 Hours

21


Update: Smith Corporation

• Tape restore halted due to a message • Step omitted “*HOLD System message queue”:

approximately 45 minutes lost • Called all 3rd party vendors for keys • New notice is sent to business

for restoration completion: 14 hours from NOW!

17 Hours

22


System Ready: Smith Corporation

• System is restored in 8 hours. Available early!! • Systems are checked and all production data is

restored • Non-critical data was omitted

from initial restore • All 3rd party keys applied except for

one as procedure is missing and waiting for vendor to call back

25 Hours

23


Report Card: Smith Corporation

1. Need to keep “Authorized to Declare” list up to date 2. Know your offsite vendors limits 3. Locked offsite storage boxes 4. Keep hot-site “in the know” about hardware changes 5. Utilize one tape format for backups: enterprise 6. Electronic Backup Solution: VTL Onsite 7. Save virtual machine images to tape and store offsite 8. Missed step: human error 9. The entire staff was very well trained 10. 2 Staffers were on holidays, while the other 2 came through 11. Third party procedures included key retrieval and

activation

24


What about Your Email?

• Mission critical • Preferred or standard communicate channel • Key component of your communications strategy • Is Email listed as a critical application for your

organization and included in your disaster recovery plan?

26


What Does Testing Provide?

• Recovering from a disaster can be overwhelming • People become comfortable in familiar situations • Visually see the recovery goal and efforts • Confidence already in place • Will staff be available when you need them most? • Audit your backup policies….NOW!!!

27


How to Stay on the Island Permanently

• Research a managed disaster recovery provider - DR plan is kept current and well maintained - Emotions are removed - Understands your backups and business requirements - Recovery site must be FEMA diverse - Recovery reports must be off-site - Can act as your agent for 3rd party applications and off-site

storage - Can provide tape, vaulting, or HA solutions - You can focus on your business and your family - Not just a hardware supplier

28


Superstorm Sandy and Hurricane Irene: A Worthwhile Look Back at History

Advance warning… DO NOT WAIT for the disaster to take you down. If you know the storm is coming, begin preparations to declare

29


Solutions to Modernize Recovery

30


Business Requirements Are Changing Managed DR Solutions

Recovery Time Objective (RTO)

Reco

very

Poi

nt O

bjec

tive

(RPO

)

Days Minutes

Days

Seconds

Tape

Electronic Backup

Data Vaulting: Transactions

High Availability

46% Growth

31


Multi-Site DR in the Cloud: Adds Another Layer of Service

Primary

DR Provider

DR Provider

FEMA Region Separation

Last Transaction RPO

“No impact” planned outages

32


For More Information

Jim Knight Managed Disaster Recovery Service expert [email protected] 866.638.2779

33

VELOCITY.CC

mailto:[email protected]

survivor - disaster recovery edition

Education