what’s in a technical recovery plan?...recovery in same data center run-book type events governed...

Break Out Session – Track 2

Monday - 2:45pm -3:45pm

March 18, 2013

Presenter: Bruce Blank

The content of this presentation represents my own views and

not the views of the North Shore LIJ Health System.

The information herein represents my own views, studies,

experiences, opinions and industry best practices I learned,

changed and used over the last 30 years.

WHAT’S IN A TECHNICAL RECOVERY PLAN?

Presentation Disclaimer, Bruce Blank

Session Goals

This session studies aspects of technical recovery plans

supporting a Health Care type application. Particular attention is

paid to different recovery models; Technology Platforms, Interface

Operations, SAN storage, Data Replication, Virtualization, IO

mappings supporting a recovery.

Those in attendance will get a chance to compare there own

technical recovery plans to what is discussed.

Bruce Blank is a CISM and has more than 30 years of IT and Risk Management experiences and built Disaster

Recovery and Corporate Security and Compliance Programs for the largest companies in the world. Industry

Verticals include; Insurance – Mortgage Finance – Banking – Manufacturing and Health Care.


Assumptions

• Hundreds of Applications Supporting Business and Clinical Care

• One to One relationship between Application and Technical Recovery Plan

• Thousands of Servers across Multiple Data Centers over Multiple Networks

• Multiple Technology Platforms / Solutions supporting Businesses

• Application Criticality Process

• Multiple Recovery Models

• Disparate Support Organizations and Staff Reporting Lines

(Internal, Consultancy, Vendor, Hybrid)

And yes, a model for creating these Technical Recovery Plans


Question: What is important in any Hospital Environment?

Answer: Providing Optimal Clinical Care and Outcomes.

Question: How many systems are there in a large Hospital Environment?

Answer: Many, many, many and did I say many.


Some Informational Sources

http://nyp.org/services/index.html

http://www.swedish.org/About/Overview/Facts---Figures#axzz2JaudvKTk

http://www.orlandohealthdocs.com/

http://www.northshorelij.com/NSLIJ/NSLIJ+HomePage

What is clinically important in any Hospital environment?


Ambulance Tracking &

Communication SystemsRegistration, ADT Blood & Blood Bank Systems

Emergency Dep’t Systems

Tracking Board Systems

Billing Systems

Integration Engines

Orders/ Results Systems

Radiology Information Systems

Surgical Systems

Bed Tracking Systems

Pharmacy Systems

Cardio Systems

Nurse Scheduling Systems

Physician Order

Entry Systems

Discharge Instruction

Systems

Clinical Documentation Systems

Food Tracking Systems

Picture Archiving and Communication Systems

Labor & Delivery Systems

ICU, PICU, NICU Systems


Clinical Care

Cost Containment

Business Functions

Hospital Models – Continuously Competing, Changing Priorities

HIPPADR Mandate

Legal Issues

Other Regulatory

Technology Change

(internal, external)

Operating

Risks

Question ?

How critical is it if any one of the

systems in the previous slide goes

down at your Hospital?


Question ?

How critical is it if many of the systems

in the previous slide goes down at your

Hospital ?


Disaster Recovery Operational Recovery

• Localized situation in same Data Center.

• Requires Management Awareness and Departmental Escalations

• May impact one or more Applications.

• Could be hardware, software or operational type events.

• Examples� Database Cluster and Failover

� Redundant Technology failure and or recovery in same Data Center

� Run-book type events

� Governed by SLAs

• Catastrophic Event.

• Impacts one or more Data Centers.

• Requires Executive Management Declaration and Communication.

• Requires failover to resilient applications in the other Data Center and or requires re-build of the application and its data components

• Examples� Tornado wipes out Data Center

� Fire/Flood impacts Data Center forcing Declaration

� Redundant Technology recovery in other Data Center

� Governed first by BIA and RTO invocation, then SLAs


Disaster Recovery vs. Operational Recovery

Challenges and issues defining a Technical Recovery plan?

1. Which applications require a Technical Recovery plan?

2. Do you have appropriate Technical Infrastructure staffing resources

supporting the Technical Recovery plan? (Quantitative and Qualitative)

3. Does the Business Requirements match Technical implementation

capabilities?

4. How does change impact the Technical Recovery plan?

5. What is generally NOT in a Technical Recovery Plan?

6. Sample Contents of a Technical Recovery Plan.


Criticality Analysis – DR Program Driven

• Business Impact Analysis

• Recovery Point Target

• Data Access Requirement

• Downtime Clinical Processes


1. Which applications require a Technical Recovery plan?


Application “Tier” Levels Define Criticality


App

TierRTO RPO DAR Clinical/Business DT

0 <= 3 No Loss Not yet required Not yet invoked

1 <3 hours, not > than 4 Hours <=0 Mins, not > 10 Mins

Clinical Reports or Business Data

enough to sustain a 4 hour

recovery time and < =10 min data

loss

Downtime Procedures Invoked for

Clinical Applications

2 4 Hours, not > 8 Hours 10 Mins, not > 2 Hours


enough to sustain an 8 hour

recovery time and <= 2 hour

maximum data loss


Clinical and Business Applications

3 8 Hours, not > 1 Day 2 Hours, not > 12 Hours


enough to sustain a 1 Day

recovery time and <=12 hour

maximum data loss



where possible

4 1 Day, not > 7 Days 12 Hours, not > 24 Hours




maximum data loss



where possible

5 Good Luck 12 Hours, not > 24 Hours




maximum data loss



where possible up to point of not

being able to provide Clinical care


2. Do you have appropriate Technical Infrastructure staffing resources

supporting the Technical Recovery plan? (Quantitative and Qualitative)

Does a targeted application in your Health System make use of 1 or more of these technologies?

http://hhnmag.com

Mainframe

Midrange

Windows

Unix Based

DataBase

Client Server

Web Based

SAN Storage

NAS Storage

Citrix

Backup Technologies

Data Replication Methods

Virtualized Based Server

Recovery

Authentication Services

Network Services

DNS Services

Others…..


3. Does the Business Requirements match Technical

implementation capabilities?

Example

• Low Tier Application (Tier 1) is Required.

• Cost Prohibits Redundant Technology.

• Application DR Capability only provides for

a recovery from tape, build it from scratch.

• What do you do?

� Risk Acceptance / Sign-off ?

� Management Reporting ?

� Cost Delta for next years budgeting?

� Keep it in front of Sr. Leadership ?

� No one likes surprises!


4. How does change impact the Technical Recovery plan?

• Does your Change Process accommodate

Technical Recovery Plans? If not why?

� Integrate changes via normal Change

Management Processes

� Documentation of those changes should be

required as part of the change

� If you make use of a change ticketing system,

spawn a change ticket based on the application

selected to back to the originator/implementer of

the change.

� DR staff should review/QA the DR Document

change and close the ticket upon acceptance.


5. *What is generally NOT in a Technical Recovery Plan?

• Your Project Management Process.

• Crisis Management Processes.

• Plant/Business Location Recovery.

• Business Continuity Plan.

• Tier Strategy. The Tier Strategy (and Program)

Delivers the approved Tier - where the budget,

design/architecture and hardware solution is based.

• Hospital Emergency Incident Codes

*Although important elements identified / addressed on this slide are part of more; Corporate, Location, Situational or Global DR Plan component(s).

* DRII and or ITIL practices may help you further develop supporting your plan.

WHAT’S IN A TECHNICAL RECOVERY PLAN?5. Sample Contents of a Technical Recovery Plan.

a) Application Overview

b) Support Model

c) Visio of Application

d) Hardware/Software Section

e) Interfaces

f) Vendor Information and Support

g) Dependencies

h) Backup Requirements and Verification

i) Technical Recovery Scripts

i. Pre Recovery Steps

ii. Recovery

a) Failover Scripts/Steps

b) Failback Scripts/Steps

c) Build from Scratch, Recover from Tape Steps

iii. Validation of Recovery Sequences

j) Business Contact and Support

k) Documentation Signoff

l) Change Management Appendences/Changes Documented

Electronic Health Record System Functional Model Normative Standard (ANSI-approved)

http://www.hl7.org/ehr/downloads/index_2007.asp

Guide to Healthcare IT Disaster Recovery / VMWare

http://www.vmware.com/files/pdf/VMW_Guide-to-Healthcare-IT-Disaster-Recovery.pdf

Healthcare Disaster Recovery

http://www.sans.org/reading_room/whitepapers/hipaa/disaster-recovery-healthcare-organizations-impact-hipaa-security_1336

Best Practices in Healthcare Disaster Recovery Planning

http://www.healthmgttech.com/articles/201205/best-practices-in-healthcare-disaster-recovery-planning.php

Most Healthcare Organization's Lack Robust disaster recovery plans for patient data

http://www.infosecurity-magazine.com/view/26877/most-healthcare-organizations-lack-robust-disaster-recovery-plans-for-patient-data/

Gartner Reports include a wonderful mix of varied information regarding this topic.

contact: [email protected]

WHAT’S IN A TECHNICAL RECOVERY PLAN?Other Interesting Public Reference Sources:

what’s in a technical recovery plan?...recovery in same data center run-book type events governed...

Documents