s3 - disaster recovery & business continuity (armour ... · disaster recovery & business...

1

Disaster Recovery & Business Continuity

Dell IT Executive Learning Series

Presented byRich Armour, Debi Higdon &

Mitchell McGovern

THIS PRESENTATION SUMMARY IS FOR INFORMATIONAL PURPOSES ONLY AND MAY CONTAIN TYPOGRAPHICAL ERRORS AND TECHNICAL INACCURACIES. STATEMENTS ARE THE OPINIONS OF THE PRESENTER(S) AND DO NOT NECESSARILY REFLECT THE VIEWS OF DELL. THE CONTENT IS PROVIDED AS IS, WITHOUT EXPRESS OR IMPLIED WARRANTIES OF ANY KIND.

© 2006 Dell Inc. All rights reserved.

Trademarks used in this text: Dell and the DELL logo are trademarks of Dell Inc. Other trademarks and trade names may be used in this document to refer to either the entities claiming the marks and names or their products. Dell Inc. disclaims any proprietary interest in trademarks and trade names other than its own.

2

3 of 23Copyright © 2006 Dell Inc. All Rights Reserved

Disaster Recovery & Business Continuity

Leadership

Do I need Continuous Operations?

What’s the Cost?

Where is My Risk?

What’s most important?Is everyone in the Loop?

What won’t work?

Hardened Data Centers?

Invi

sibl

e Fa

ilove

r?

Keep

ing

it Ev

ergr

een

Build it in, Don’t bolt it onWhat about that “extra” capacity?

Will My Systems Failover?


Integrated Business Continuity Program

BusinessContinuityBusinessContinuity

DynamicCall

Routing

DynamicCall

Routing

RedundantNetworkingRedundantNetworking

RecoverableApplicationsRecoverableApplications

BackupData

BackupData

ContingencyFacilities

ContingencyFacilitiesMultiple

DataCenters

MultipleData

Centers

TemporaryStaff

TemporaryStaff

Outsourcers& Vendors


= Components of I/T Disaster Recovery

3


Integrated Business Continuity Program

BusinessContinuityBusinessContinuity

DynamicCall

Routing

DynamicCall

Routing

RedundantNetworkingRedundantNetworking

RecoverableApplicationsRecoverableApplications

BackupData

BackupData

ContingencyFacilities

ContingencyFacilitiesMultiple

DataCenters

MultipleData

Centers

TemporaryStaff

TemporaryStaff



= Components of I/T Disaster Recovery

Business Continuity Planning• Build Detailed Recovery Plans for Key

Business Facilities & Capabilities– Call Centers– Factories & Distribution Centers– E-Commerce Sites– Service Centers

• Consider both routine & major disruptions• Balance risk & cost of disruptions against

cost of recovery capabilities• Communicate Plans to all Players• Build Business & I/T Disaster Recovery

capabilities• Run periodic exercises


Sep OctNov Feb MarApri

lMay

12

$...

$...

$...

$...

$...

$...

$...

$...

$...

$...

FinancialExposure

(Lost Profit or other metric)

Recovery Time

1

June

2

3

5

11

10

6

8

9

7

4

Dec

< $X Million

Jan

July

Aug Oct Nov Dec

Sep

Without DR Plan

$...

$X Million

Aug

Financial exposure reduced

DR

Implem

entedD

R Im

plemented

Recovery Time vs. Exposure

Link to Cost & Impact of Potential Disaster

4


Sep OctNov Feb MarApri

lMay

12

$...

$...

$...

$...

$...

$...

$...

$...

$...

$...

FinancialExposure

(Lost Profit or other metric)

Recovery Time

1

June

2

3

5

11

10

6

8

9

7

4

Dec

< $X Million

Jan

July

Aug Oct Nov Dec

Sep

Without DR Plan

$...

$X Million

Aug

Financial exposure reduced

DR

Implem

entedD

R Im

plemented

Recovery Time vs. Exposure

Link to Cost & Impact of Potential Disaster

Highlight Financial Benefits• Estimate financial exposure over the

recovery time for loss of each key facility• Determine best effort recovery time given

level of DR capability• Assess impact of program milestones…

– Additional Data Center is operational– Hardware is ordered– Hardware is installed– Applications are distributed & Data is

replicated• DR implementation starts delivering value

long before all capabilities are fully installed & tested


Where’s Your Risk?

5



Top 10 Risks1. Fire2. Thunderstorms3. Utility Interruptions4. Telecommunications Interruptions5. Flood or Surface Water6. Virus7. Heavy Rain or Snow8. Extreme Cold or Freezing Precip9. Water or Liquid Spill or Release10.Denial of Service Attack




Leverage Your Insurance Carrier• Expert at risk analysis• Maintains extensive loss databases• Motivated to reduce your risk• Can drive premium reductions




6



Other Risk Considerations• Tornado• Earthquake• Hurricane/Typhoon• Terrorist Attack• Human Error• Tsunami• Etc. . . .

Each Business & Locationwill have a unique Risk Profile






Determine Criticality of I/T Resources

DR ClassDR Class RTO*RTO*

IBusinessCritical

IBusinessCritical

IIBusinessEssential

IIBusinessEssential

IIIBusinessSupport

IIIBusinessSupport

1-4 Hrs1-4 Hrs

48 Hrs48 Hrs

Best EffortBest Effort

DR Class DescriptionDR Class Description

Any outage immediately impacts the businessAny outage immediately impacts the business

An extended outage (i.e., > 48 hours) impacts the businessAn extended outage (i.e., > 48 hours) impacts the business

An extended outage does not impact the businessAn extended outage does not impact the business

* Recovery Time Objective

What “impacts the business”?• Degrades business operations• Impacts customer experience• Results in financial loss• Erodes brand value• Creates unacceptable liabilities

Business Impact Analysis

1. Define Business Impact in terms relative to your business

2. Relate I/T Resources to Impact of their loss or degradation

3. Assess Applications, Data & Infrastructure

4. Ensure Single Points of Failure are identified & mitigated

5. Include components which support critical resources

6. Link to financial exposure7. Involve & validate with business

owners8. Make DR Class part of the

culture9. Periodically update

Business Impact Analysis

1. Define Business Impact in terms relative to your business

2. Relate I/T Resources to Impact of their loss or degradation

3. Assess Applications, Data & Infrastructure

4. Ensure Single Points of Failure are identified & mitigated

5. Include components which support critical resources

6. Link to financial exposure7. Involve & validate with business

owners8. Make DR Class part of the

culture9. Periodically update

7


Four Levels of Continuity

Site

Platform

Redundant Systems Continuous server, storage, network access

Application Failover/ Load Balancing Continuous application access via clustering

SAN, NAS & DAS Continuous data access

Backup and Restore Tape backup, off-site storage

Site/Datacenter Failover Re-route users and data to replicated sites

Site Recovery Remote or commercial recovery facilities

Data

Application

High Availability System Features Hot-swappable, redundant components with mission-critical support

Rapid Equipment Replacement Vendor services and financing programs


Lay the Foundation:Tier and Protect Your Data

8


I/T Resources to be AnalyzedI/T Resources to be AnalyzedI/T Resources to be Analyzed

Single Point of Failure (SPOF) Analysis

Single Point of Failure (SPOF)• A point in a system where if a failure

occurs, there is no redundancy to compensate for it

• Systems can have more than one “single point of failure”

• One SPOF in a critical component can overshadow well engineered redundancies & resilience in the rest of the system

• Removal of all SPOFs usually not cost effective

• Leverage Failure Mode & Effect Analysis(FMEA) & other tools

Single Point of Failure (SPOF)• A point in a system where if a failure

occurs, there is no redundancy to compensate for it

• Systems can have more than one “single point of failure”

• One SPOF in a critical component can overshadow well engineered redundancies & resilience in the rest of the system

• Removal of all SPOFs usually not cost effective

• Leverage Failure Mode & Effect Analysis(FMEA) & other tools

Conduct analysis of Single Points of Failure (SPOF) in Class I (Business Critical) Systems to ensure Business Continuity

Conduct analysis of Single Points of Failure (SPOF) in Class I (Conduct analysis of Single Points of Failure (SPOF) in Class I (Business Business Critical) Systems to ensure Business ContinuityCritical) Systems to ensure Business Continuity

Data Center FacilityData Center FacilityPowerPower

EnvironmentalEnvironmentalStorageStorageServersServers

Network – CampusNetwork – CampusNetwork – Long-HaulNetwork – Long-Haul

Network – FacilityNetwork – Facility

DataData

MiddlewareMiddlewareApplication SoftwareApplication Software

Client SystemsClient Systems

Shared ServicesShared Services

Up Stream SystemsUp Stream Systems


Build the Team . . . Manage the Complexity

9


Build the Team . . . Manage the Complexity

• Application teams• Data Management• I/T Operations & Engineering• Networking & Telecom• Facilities• Business Customers• Business Continuity Program• Our Program Approach…

– 20,000+ Tasks– 2 years– Dedicated PMO driving

extended virtual teams– Phased Program– Earned Value & Scorecarding


Don’t Assume Infrastructure Will Work When Disaster Strikes

EventEventEvent Potential ImpactPotential ImpactPotential Impact Planning ConsiderationsPlanning ConsiderationsPlanning Considerations

Ice StormIce StormIce Storm Land-based Phones DownLandLand--based Phones Downbased Phones Down Assure wireless connectivityAssure wireless connectivityAssure wireless connectivity

Area EvacuationArea EvacuationArea Evacuation Highways jammedHighways jammedHighways jammed Provide remote access capabilitiesProvide remote access capabilitiesProvide remote access capabilities

Highways impassableHighways impassableHighways impassable Provide remote access capabilitiesProvide remote access capabilitiesProvide remote access capabilities

Security eventSecurity eventSecurity event Network downNetwork downNetwork down Ensure DR plans available off-lineEnsure DR plans available offEnsure DR plans available off--lineline

Internet WormInternet WormInternet Worm Internet downInternet downInternet down Maintain local copies of critical filesMaintain local copies of critical filesMaintain local copies of critical files

Hazmat EventHazmat EventHazmat Event Data Centers inaccessibleData Centers inaccessibleData Centers inaccessible Ensure robust remote access & system management as well as DC management capabilities

Ensure robust remote access & Ensure robust remote access & system management as well as DC system management as well as DC management capabilitiesmanagement capabilities

Building unusable – no HVACBuilding unusable Building unusable –– no HVACno HVAC Relocation of critical usersRelocation of critical usersRelocation of critical users

Area Power OutageArea Power OutageArea Power Outage Cell & land phones downCell & land phones downCell & land phones down Designate sites which will have power & can support e-mail & IM, provide radios to key personnel

Designate sites which will have Designate sites which will have power & can support epower & can support e--mail & IM, mail & IM, provide radios to key personnelprovide radios to key personnel

HurricanesHurricanesHurricanes All of the above – total threatAll of the above All of the above –– total threattotal threat Tabletop dry run of DR plansTabletop dry run of DR plansTabletop dry run of DR plans

Data Center damageData Center damageData Center damage

10


Consolidate, Lock Down &Bullet Proof Your Data Centers

• UPS & Generator Backup

• Diverse Power Feeds• Fire Protection• Physical Security &

Access Control• Alarms & Sensors• Dual Path Network

Routing• Dual Path Power

Routing• N+1 Redundancy• Environmental Controls• Expandable• Manual EPO• Facility Hardening

Data Center Considerations• Plan for the future• Identify & Mitigate Single Points of Failure• Use Threat Assessment & Business Impact

Analysis to drive design & capabilities• Balance trade-offs between DC hardening &

failover capabilities

Data Center Considerations• Plan for the future• Identify & Mitigate Single Points of Failure• Use Threat Assessment & Business Impact

Analysis to drive design & capabilities• Balance trade-offs between DC hardening &

failover capabilities


Leverage Always On, Active-Active,75/75 Architecture

Fully Redundant Fiber NetworkFully Redundant Fiber Network

WorkloadBalanced Across

Multiple Data Centers

Active/Active

• Redundant, load balanced infrastructure runs as fully distributed production environment

• Each Data Center runs 75% of peak capacity/100% of normal• Redundant Fiber Network for scalability & reliability• Failover is automatic & instantaneous: loss of capacity, not data or

function• Floor plan allows immediate expansion for extended recovery

•• Redundant, load balanced infrastructure runs as fully distributeRedundant, load balanced infrastructure runs as fully distributed d production environmentproduction environment

•• Each Data Center runs 75% of peak capacity/100% of normalEach Data Center runs 75% of peak capacity/100% of normal•• Redundant Fiber Network for scalability & reliabilityRedundant Fiber Network for scalability & reliability•• Failover is automatic & instantaneousFailover is automatic & instantaneous: loss of capacity, not data or : loss of capacity, not data or

functionfunction•• Floor plan allows immediate expansion for extended recoveryFloor plan allows immediate expansion for extended recovery

Real-Time Data ReplicationReal-Time Data Replication

11


Recovery StrategyRecovery Strategy

Database Recoverability – Still a Challenge

ActiveActiveActive

• Low Application Design Impact• Recovery time: 1-4 Hours• Failback Complex & Time Consuming• Supported by EMC SRDF, Oracle Data

Guard & other replication technologies

• Low Application Design Impact• Recovery time: 1-4 Hours• Failback Complex & Time Consuming• Supported by EMC SRDF, Oracle Data

Guard & other replication technologies

• High Application Design Impact• Recovery time: Instantaneous• Failback Less Complex & Time

Consuming• Requires application logic to support

data partitioning by instance

• High Application Design Impact• Recovery time: Instantaneous• Failback Less Complex & Time

Consuming• Requires application logic to support

data partitioning by instance

• Low Application Design Impact• Recovery time: Instantaneous• Failback Much Less Complex & Time

Consuming• Not well supported by existing

database products & technologies

• Low Application Design Impact• Recovery time: Instantaneous• Failback Much Less Complex & Time

Consuming• Not well supported by existing

database products & technologies

Features/ToolsFeatures/Tools

App(2)App(2)App(2)

ActiveActiveActive ActiveActiveActive

StandbyStandbyStandby

App(1)App(1)App(1) App(2)App(2)App(2)

HotHotHot

App(1)App(1)App(1) App(2)App(2)App(2)

ActiveActiveActive

App(1)App(1)App(1)

Hot/Standby

Hot/Standby

Partitioned Active/Active

Partitioned Active/Active

Database Active/Active

Database Active/Active

ModelModel


Make Testing a Regular Cycle…•Test some Apps each Quarterly•Test all Business Critical App Annually•Each Test requires planning, staging, execution & analysis


Disaster Recovery Testing

Q1Q1 Q2Q2 Q3Q3 Q4Q4

Objectives…• Validate recovery procedures• Demonstrate recoverability SLA• Identify recoverability problems• Test end-to-end failover integration


Scope…• Business Critical Systems• Key systems of critical vendors• Paper review on some


Strategy…• Cross-Functional Planning• Limited Business Disruption• Not a one-time event


Metrics…• Recovery time• Procedural Errors• Data Integrity• Functional Availability


12




Disaster Recovery Testing

Q1Q1 Q2Q2 Q3Q3 Q4Q4









If it hasn’t been tested, it doesn’t work…


Where is your

Disaster Recovery Program?

Do you have one?

Where is your

Disaster Recovery Program?

Do you have one?

How Mature is Your DR Capability?

MATURE DR PROGRAM• Business continuity risks are

quantified and validated by Audit• Mission critical Apps are flagged

for special handling• Failover Tests are conducted

regularly• DR is part of every new solution• Business Continuity Plans are

fully supported by I/T contingency plans & capabilities

• Multiple Hardened Data Centers linked with redundant networking

• Mission Critical Applications are hosted in I/T Data Centers

• Applications are architected with transparent “always on” Active-Active DR

• Key Vendors’ Business Continuity Programs are regularly reviewed

• Key Vendors involved in DR tests

MATURE DR PROGRAMMATURE DR PROGRAM•• Business continuity risks are Business continuity risks are

quantified and validated by Auditquantified and validated by Audit•• Mission critical Apps are flagged Mission critical Apps are flagged

for special handlingfor special handling•• Failover Tests are conducted Failover Tests are conducted

regularlyregularly•• DR is part of every new solutionDR is part of every new solution•• Business Continuity Plans are Business Continuity Plans are

fully supported by I/T contingency fully supported by I/T contingency plans & capabilitiesplans & capabilities

•• Multiple Hardened Data Centers Multiple Hardened Data Centers linked with redundant networkinglinked with redundant networking

•• Mission Critical Applications are Mission Critical Applications are hosted in I/T Data Centershosted in I/T Data Centers

•• Applications are architected with Applications are architected with transparent transparent ““always onalways on”” ActiveActive--Active DRActive DR

•• Key VendorsKey Vendors’’ Business Continuity Business Continuity Programs are regularly reviewedPrograms are regularly reviewed

•• Key Vendors involved in DR testsKey Vendors involved in DR tests

HIGH RISK DR PROGRAM• Business continuity risks have not

been quantified• Mission critical Apps are not

formally identified• Apps are rarely tested for failover• DR capability is an after thought,

not part of the development process

• Business Continuity Plans are independent of any I/T DR programs

• Data Centers are not supported by adequate power & environmental redundant systems

• Mission Critical Applications are hosted in a variety of locations in & out of Data Centers

• Applications must be manually failed over to backup systems or sites

HIGH RISK DR PROGRAMHIGH RISK DR PROGRAM•• Business continuity risks have not Business continuity risks have not

been quantifiedbeen quantified•• Mission critical Apps are not Mission critical Apps are not

formally identifiedformally identified•• Apps are rarely tested for failoverApps are rarely tested for failover•• DR capability is an after thought, DR capability is an after thought,

not part of the development not part of the development processprocess

•• Business Continuity Plans are Business Continuity Plans are independent of any I/T DR independent of any I/T DR programsprograms

•• Data Centers are not supported by Data Centers are not supported by adequate power & environmental adequate power & environmental redundant systemsredundant systems

•• Mission Critical Applications are Mission Critical Applications are hosted in a variety of locations in hosted in a variety of locations in & out of Data Centers& out of Data Centers

•• Applications must be manually Applications must be manually failed over to backup systems or failed over to backup systems or sitessites

13


Key Takeaways . . .

Getting It Done . . .• Start with Leadership• Integrate with the Business• Assess the Risks• Identify the Critical

Resources• Balance Cost & Benefit• Involve the Entire Team• Harden your Data Centers• Go Active-Active• Assume Nothing Will Work• Do a Full Failover

Getting It Done . . .Getting It Done . . .•• Start with LeadershipStart with Leadership•• Integrate with the BusinessIntegrate with the Business•• Assess the RisksAssess the Risks•• Identify the Critical Identify the Critical

ResourcesResources•• Balance Cost & BenefitBalance Cost & Benefit•• Involve the Entire TeamInvolve the Entire Team•• Harden your Data CentersHarden your Data Centers•• Go ActiveGo Active--ActiveActive•• Assume Nothing Will WorkAssume Nothing Will Work•• Do a Full FailoverDo a Full Failover


Key Takeaways . . .

Getting It Done…• Start with Leadership• Integrate with the Business• Assess the Risks• Identify the Critical Resources• Balance Cost & Benefit• Involve the Entire Team• Harden your Data Centers• Go Active-Active• Assume Nothing Will Work• Do a Full Failover

Getting It DoneGetting It Done……•• Start with LeadershipStart with Leadership•• Integrate with the BusinessIntegrate with the Business•• Assess the RisksAssess the Risks•• Identify the Critical ResourcesIdentify the Critical Resources•• Balance Cost & BenefitBalance Cost & Benefit•• Involve the Entire TeamInvolve the Entire Team•• Harden your Data CentersHarden your Data Centers•• Go ActiveGo Active--ActiveActive•• Assume Nothing Will WorkAssume Nothing Will Work•• Do a Full FailoverDo a Full Failover

Keeping It Evergreen…

• Built DR into every Solution• Don’t Break into the “Piggy Bank”• Test it . . . Regularly!

Make DR part of your CultureEvery Project, Every App, Everyday

Keeping It EvergreenKeeping It Evergreen……

•• Built DR into every SolutionBuilt DR into every Solution•• DonDon’’t Break into the t Break into the ““Piggy BankPiggy Bank””•• Test it . . . Regularly!Test it . . . Regularly!

Make DR part of your CultureMake DR part of your CultureEvery Project, Every App, EverydayEvery Project, Every App, Everyday

14

Questions & Answers

s3 - disaster recovery & business continuity (armour ... · disaster recovery & business...

Documents