best practices in disaster recovery planning and testing

CONFIDENTIAL – DO NOT DISTRIBUTE. Copyright © 2014 Axcient, Inc. All Rights Reserved.

Best Practices in

DR Planning and Testing

Paul F Kirvan, CISA, FBCI Independent BC/DR Consultant

Member of the Board and Secretary

The Business Continuity Institute USA Chapter


Agenda

1. Introduction

2. Plan Components

3. Mistakes and Pitfalls to Avoid

4. DR Technology Options

5. Tips for Planning DR Tests

6. Summary

7. Q&A


Why is DR Important?

• Accepted way to ensure that critical data, IT systems and

networks can be recovered in an emergency

• Ensures that corporate business objectives can be

achieved, despite a disruption

• Increasingly accepted by management as a strategy for

keeping the business operational


Quick Poll

Do you currently have a Disaster Recovery plan in place?

a. Yes, I have a comprehensive DR plan at my company

b. Yes, but needs more work

c. No, but would like to get one ready

d. No, and have no plans to create one


Quick Poll

Do you currently have a Disaster Recovery plan in place?


What Do You Need?

A good disaster recovery plan needs:

• Support from senior management

• Funding approved by management

• Structured plan framework

• Access to qualified staff

• Access to relevant information

• Documentation and testing


What’s Your Goal with the Plan?

Build disaster recovery plans and associated

documentation based on a structured framework that is

consistent with good practices and standards.


DR Plan Activities

• Data gathering, interviews, analysis

• DR standards and good practice, emergency response

procedures, data backup and recovery procedures,

system recovery and restart processes, plan templates

• Tests to ensure that plan procedures and processes work

as designed

• Maintenance activities to keep plans up to date and

accurate


Standards and Good Practice

• Standards – NFPA 1600:2010; ISO 24762:2008; ISO

27031:2011; NIST 800-34

• Regulations – NASD 2510/3520; NYSE 446

• Good Practice – BCI Good Practice Guidelines, FFIEC

Handbook

• Corporate DR policies


What You Need to Identify

• DR objectives of the systems, networks or other IT assets

(e.g., uninterrupted operation, max downtime 4.0 hrs)

• Risks and/or threats to the achievement of the DR

objectives

• Define and document the processes and procedures

needed to recover and reactivate the IT assets

• Identify preventive measures to mitigate DR risks to an

acceptable level


Plan Components

The following pages list the typical components found in an IT

disaster recovery plan. There may be some variations based on

your organization’s requirements, but generally the following items

should be included.


Plan Components

A good DR plan usually includes the following

components: • Company DR policies

• DR plan documents

• Business impact analysis reports

• Risk assessment reports

• Exercise results

• IT DR procedures (in the plan)

• Supporting documents (e.g., data backup process, off-site storage

process, vendor contracts, diagrams, maintenance contracts, training

plans)


Plan Components

If there’s an existing plan, use it as a starting point

Define plan scope, purpose, authority

Define a policy statement

Define management approval and funding

Identify planning and response teams

Identify critical IT resources

Identify risks and their impact on IT assets

Determine recovery time objectives (RTOs)

Determine recovery point objectives (RPOs)


Plan Components

Preventive controls (e.g., backup power)

Response and recovery strategies

Data backup and recovery methods, compared to existing data

storage and retrieval procedures

Potential use of alternate IT sites, e.g., a backup data center,

collocated data center, the cloud

Potential use of hot sites, cold sites

Potential use of alternate work (e.g., office) sites, and the technology

needs for those sites


Plan Components

Process for equipment replacement

Process for obtaining spare parts

Staff roles and responsibilities in a disaster

Event notification procedures

Damage assessment procedures

Process and criteria for plan activation

Identify who is authorized to declare a disaster

Recovery / failover procedures

System restart / failback procedures

Resumption of business procedures


Plan Components

Step-by-step procedures for recovery of

IT operations

Desktop systems

Data

Hardware

Operating systems

Applications

Databases

LANs and WANs

Voice and VoIP systems

Servers


Step-by-step procedures for recovery of

Web sites

Mainframes

Distributed systems

Wireless technology

Specialized systems

Information security

User access

Physical security

Vital records

Plan Components


Plan Components

Step-by-step procedures for

Alerting first responder organizations

Alerting family members

Alerting primary/alternate vendors

Alerting staff, senior management

Alerting clients, stakeholders

Escalating recovery efforts

Help desk support

Using call trees

Activating automated notification systems

Activating conference bridges


Plan Components

Links to emergency management and incident response plans,

business continuity plans

Process for exercising DR plans

Process for creating a DR awareness program

Process for DR team training

Process for DR training of employees

Process for communicating with the media

Designated company spokesperson


The next set of slides provides a sample DR plan

outline. While most plans will be different, this outline

includes the most common plan components and is

consistent with standards and good practice.

Plan Components


Plan Components

DR Plan Outline - 1

• Revision History

• Table of Contents

• Emergency Response Actions

‐ Assembly Points

‐ Emergency Call-in Number

‐ Key Personnel Contact Info

‐ Notification Calling Tree

‐ External Contacts

‐ External Contacts Calling Tree


Plan Components

DR Plan Outline - 2

• Policy Statement

• Objectives

• Plan Overview

• Plan Updating

• Plan Documentation Storage

• Backup Strategies

• Emergency Response

‐ Plan Triggering Events

‐ Assembly Points


Plan Components

DR Plan Outline - 3

• Activation of Emergency Management Team

• Technology Services Team

• Emergency Alert, Escalation and DRP Activation

• DR Procedures and Actions

‐ Contact with Employees

‐ Backup Staff

‐ Recorded Messages / Updates

‐ Alternate Recovery Facilities / Hot Site


Plan Components

DR Plan Outline - 4

• Personnel and Family Notification

• Communications with Media, Key Stakeholders

• Media and Key Stakeholders Contact

• Media and Key Stakeholders Team

• Rules for Dealing with Media, Key Stakeholders

• Insurance Requirements

• Financial and Legal Issues

‐ Financial Assessment

‐ Financial Requirements


Plan Components

DR Plan Outline - 5

• Legal Actions

• DR Plan Exercising

• Appendix A – Technology DR Plans

‐ Production Environment

‐ Private Cloud Environment

‐ Internal IT Environment at HQ

‐ Local Area Network (LAN)

‐ Voice over IP (VoIP) System

‐ Remote Connectivity / VPN


Plan Components

DR Plan Outline - 6

• Appendix B – Forms and Reports

‐ Management of DR Activities Forms

‐ Communications and Reporting Form

‐ Disaster Recovery Incident Recording Form

‐ Disaster Recovery Activity Report Form

‐ Mobilizing the Disaster Recovery Team Form

‐ Mobilizing the Business Recovery Team Form

‐ Monitoring Business Recovery Progress Form

‐ Business Process/Function Recovery Form


Mistakes and Pitfalls to Avoid (the not-so-obvious things)


Mistakes and Pitfalls to Avoid

Failure to obtain senior management support

No budget (i.e., no plan)

Lack of upfront research (e.g., risks, RTO/RPO)

Lack of documentation (e.g., assume native knowledge will be

available)

No step-by-step procedures (assume you know what to do first,

second, who to call, etc.)

No plan testing (e.g., rolling the dice)

No regular plan reviews and updates


Mistakes and Pitfalls to Avoid

No DR team training (nobody knows what to do)

Assume that IT staff knows what to do

Assume that IT staff will be available in an emergency

Assume that backup and recovery procedures will work when needed

Assume that systems and networks will work properly when in backup

or recovery mode

Assume that backed-up data will be available when needed


DR Technology Options


Quick Poll

What technologies are you currently using for Disaster Recovery?

a. Local backup to disk or tape

b. Cloud backup

c. Server replication (either locally or to off-site facility)

d. Hybrid technology with local and cloud protection

e. Collocation of data center

f. Other


Quick Poll

What technologies are you currently using for Disaster Recovery?



Data backup and recovery to an alternate site, e.g., backup data

center

Application backup and recovery to an alternate site, e.g., backup

data center

Off-site data storage using a third-party firm

Redundant components, e.g., servers, storage devices, network

components

Diversely run networks, e.g., alternate service using a different carrier

and different paths

System failover / failback technologies to rapidly recover and restart

disrupted systems


Current Process New Cloud Options

Application backup and recovery to an alternate site or data center

Application backup and recovery to the cloud

File/data/database backup and recovery to an alternate site / data center

File/data/database backup and recovery to the cloud

Server backup and recovery via failover to an alternate site / data center

Server backup and recovery via failover to the cloud

Cloud-based solutions have become very popular as

primary and alternate backup and recovery strategies.



Current Process New Cloud Options

Recover the minimum configuration of servers, applications, network resources if it’s necessary to relocate to an alternate office site

“Office virtualization”, which has server failover, access to IP addresses and Active Directory in the cloud; this means rapid office recovery and minimum downtime

Conduct DR plan tests using a local, on-site environment or alternate backup data center resource

Streamline DR tests using a cloud-based and automated DR testing environment

Traditional DR activities can be automated and streamlined

to encourage more testing and reduce risks from disruptions.



Tips for Planning DR Tests


Quick Poll

How often do you test your DR plan and/or the ability to recover from a

disaster?

a. I don’t test

b. Once a year

c. Two to four times a year

d. Every month

e. Not as often as I should


Quick Poll

How often do you test your DR plan and/or the ability to recover from a

disaster?



1. Decide what you want to test, e.g., data recovery, system failover to

a backup site

2. Determine if production systems will be negatively affected during

the test

3. Conduct the test in a non-production environment, e.g., R&D

4. Select test participants and alternates

5. Document step-by-step procedures for performing the test

6. Secure a conference room or suitably equipped work area for the

test

7. Schedule the test so as not to interfere with production activities



8. Notify all IT teams and groups of the test at least two weeks in

advance

9. Include a scribe / timekeeper

10. (If possible) Conduct a dry run to validate that the test procedures

will/should work

11. Complete the test, keeping notes of all actions performed, time

needed for each activity

12. Prepare an after-action report summarizing what worked, what didn’t

work and lessons learned

13. Update the DR plan based on test results


Summary

Develop and document a plan .. follow it

Senior management supports the plan

Policies, procedures, metrics

Document, document, document

Test, test, test

Maintenance and regular review


About Axcient Leader in Recovery-as-a-Service

One SaaS Platform

Backup Disaster

Recovery

Business

Continuity WAN

Optimization Dedupe

vs.

Rapid Recovery Physical & Virtual Application

Continuity

Cloud

Virtualization

True Cloud

Platform


For more information, visit axcient.com or call 800 715.2339

@Axcient linkedin.com/company/axcient axcient.com/facebook

Paul Kirvan, CISA, FBCI

Phone (908) 902-2586

Email [email protected]

mailto:[email protected]