disasters - a need for three-dimensional thinking a nee… · rpo rto. maximum time in order to...

© 2016 Denovo. Confidential Trade Secret.

Richard Dolewski


Send an Invitation & connect on LinkedIn

Receive Tip & Techniques Monthly

Industry Best Practices to improve your

Business Continuity so you:

Don’t Fall with the Fallen!

www.linkedin.com/in/richarddolewski

[email protected]

Richard Dolewski

Disasters call atthe worst times.

And they don’t leave voicemails.


Noah and The Flood

Companies Regularly Deal with NATURAL and MAN-MADE

DISASTERS

0% 5% 10% 15% 20% 25% 30% 35% 40% 45%

Never

Once

Twice

Greater than five

Three times

Four times

Five times

"How many times in the last 5 years have you had to invoke your DR Plan "


Disaster Recovery: The Need


Definition of a Disaster

A sudden, unplanned event, that causes great damage

or loss to an organization

The time factor determines whether an interruption in

service is an inconvenience or a disaster.


Reaction to a sudden, unplanned, event that enables an organization to continue

critical business functions until normal business operations resume.

“…It is not enough to arrange for hardware

replacement … planning must address continuation

of business operations, or business continuation.”

What is Disaster Recovery?

What is a Disaster?

ANYTHING!!!!

…that stops your business from functioning &

that cannot be corrected within an acceptable

amount of time….


http://images.google.com/imgres?imgurl=http://www.goldcoast.qld.gov.au/image_content/dm_lightning_2.jpg&imgrefurl=http://www.goldcoast.qld.gov.au/t_standard.aspx?pid=5759&h=475&w=704&sz=39&hl=en&start=16&tbnid=MMGg0bEJZehH8M:&tbnh=94&tbnw=140&prev=/images?q=disaster&gbv=2&svnum=10&hl=en&sa=G

http://www.boxer-enterprises.co.uk/images/Disaster_Prevention.jpg

http://www.benfieldhrc.org/disaster_studies/rea/img/turkey3521_large.jpg


Common Misconceptions

• It will never happen to me!

• Business as usual after a disaster

• We have special requirements

• Too many other priorities

Murphy’s law: Disaster strikes when, where and because you are not prepared

The Most Dangerous phrase

in the English Language:

“We’ve always done it this

way before”



What If….

CEO:“What happens if we put all of this effort intoDisaster Planning and a disaster never happens?”

CTO:“What happens if we don’t, and a Disaster Strikes?”


BusinessContinuity


Business Preparedness – No DR Preparedness

▪ In October 2005, a survey by

The Advertising Council found:


Business Continuity Planning (BCP)

A complete and comprehensive strategy to recover, resume, and maintain all critical business functions in the event of a major business disruption

Addressing business risks for:

• Personnel

• Facilities

• Process


Questions for the Business

Include: Key Functional Business Areas

Manufacturing, Finance, Purchasing, Sales, Warehousing.

What about IT ?

• What do they need to keep productsmoving ?

• What services do they provide supplychain ?

Manufacturing

Finance

Logistics

Sales

Purchasing

Internal

IT


Understand The Business:

• Review and understand your business operating model

• Map the key business functions and processes into a “Value Chain”

to identify ALL value add activities

• Focuses on what MUST BE recovered to keep your products and services available

• Identifies what activities and tasks can be undertaken manually

• What MANUAL workarounds are present? If Any ?

Business Impact Analysis


View the supporting technology as a COMPOSITE of business applications that enable your business:

• What can be recovered or MUST always be available ?

• Key-critical Application Services including the internal and external dependencies

• Identify Recovery Time and Recovery Point Objectives (RTO & RPO)

• Where they will be recovered?

• When they must be recovered?

• How they will be recovered?

Develop an organized documented set of agreed upon procedures that align with the BIA findings.

Business Impact Analysis

Analysis

Map critical IT applications to BIA

– Application name

– Application priority

– Special Requirements

– Maximum outage (hours, days)

These applications to be included in the BIA presentation and report.



Mapping Applications To Environments - “M.A.T.E.”

Application Perspective

Recovering Databases, Servers, Network, Middleware, Software, and Integration Items for a specific Software Application

Environment Perspective

Recovering IBM Power, Windows, Linux, Network, Storage, etc.

The Data Center !

Application Recovery

Environment Recovery

What Constitutes a Disaster?

RPO:

Recovery Point Objective

RTO:

Recovery Time Objective

Disaster Recovery deliverables are based on:



Factors to Consider in Continuity

How much data can you afford to lose? How long can you afford to be down?

The maximum tolerable period in which data might be lost.

Typically equal to the time interval between backups.

RPO RTO

Maximum time in order to avoid unacceptable consequences associated with a break in business continuity.

The duration of time and SLA within the business.

Downtime vs. Availability

Downtime Cost Variable is $/Hour

• Understanding downtime $/Hour is the most important key to understanding your availability requirements.

• Labor costs, loss productivity & revenue

• The cost of downtime continues to rise.

• The cost of computing is falling.



How much money would your company lose if a major outage occurred?


• Organizations rely on technology to run their businesses.

• Downtime is a Business issue, not just a technology issue.

Importance of Business Resiliency

http://images.google.com/imgres?q=people+using+home+diesel+generator+in+hurricane+sandy&start=91&hl=en&tbo=d&biw=1600&bih=699&tbm=isch&tbnid=N4gbos43sW5ScM:&imgrefurl=http://gas2.org/2012/11/08/after-hurricane-sandy-bicycles-prove-invaluable-to-locals/&docid=RsdUSSV1JJ0SZM&imgurl=http://gas2.org/files/2012/11/flood-taxis.jpg&w=500&h=331&ei=iqnEUK2oOdDriQKb_4AQ&zoom=1&iact=hc&vpx=580&vpy=102&dur=974&hovh=183&hovw=276&tx=142&ty=125&sig=109179595583375503785&page=4&tbnh=136&tbnw=202&ndsp=35&ved=1t:429,r:1,s:100,i:7


The IT Score Card

Application Availability that Delivers(Continuity, Repeatable , Economical)

• DR Solution with Applications Residency in Mind

• Deliver On:

• Increased levels of Availability Management

• Best Practices in Data Protection

• Innovation – Pro-Active DR

• Measurable & Quantifiable Metrics

Business Agility

Depends on IT Agility

Are your

Backups

Reliable?

CA Commercial


93% of businesses that

suffer a significant loss of

data are out of business

within 5 years.

Sobering Statistic

The Bureau of Labor

Sobering Statistic

According to a Gartner:

Human error is responsible for 40% of all data loss, as compared to just 29% for hardware or system failures.

An IBM study determined:

“Data loss due to human error was as high as 60%”


(From: Business continuity and disaster recovery planning for IT professionals)


Disaster Recovery Background - Understanding The BC/DR Spectrum

Business Continuity

Business Continuity concerns the entire company, Disaster Recovery focuses on IT.

Business Continuity

• A set of procedures organizations can adopt in an effort to minimize the impact an outage for business operations

• Incorporates Communications Plans and Crisis Management

• The business is responsible for Business Continuity

Disaster Recovery

• A subset of BC that addresses IT elements for availability, restoration, of infrastructure, applications and data recovery

• Set of detailed technical actions and activities to be performed when a disaster strikes to restore critical IT Services

• IT is responsible for DR

Disaster Recovery


DisasterRecoveryPlanning


When it rains…

IBM Commercial


Disaster Recovery Planning (DRP)

Your planned IT actionable response to a sudden, unplanned event the enables an organization to keep its business going.

Planning includes the restoration/failover of pre-defined Mission Business Critical:

•Applications

•Servers

• Infrastructure

• Networks

Disaster Recovery Planning


Disaster Recovery Planning represents

the expectations set by your business

to successfully recover time-critical

business functions.

A fully developed DR Plan will enable

the successful and timely recovery of

your data center.Planning is an organizational

imperative that will reduce risk



Cannot be approached casually.

The Plan must be ....

• Well organized

• Action Oriented

• Comprehensive

Objective: Total restoration in a timely manner



Who Will execute recovery actions

What Is needed to continue, resume, recover or restore business functions

When Business functions and operations must resume

Where To go to resume corporate, business and operational functions

How Detailed procedures for continuity, resumption, recovery or restoration

CLASSIC: WHO - WHAT - WHERE - WHEN - HOW

The Products of a DR Plan


Business

Requirements Plan

ImplementTest

…to be prepared, you must regularly

go through the cycle

DR Planning is an Ongoing Process


ImplementationAssessment

BestPractices

Solution

Critical Servers

Data Capture

Document Systems

Planning & Implementation

Test &Optimization

Build Recovery Strategy

Availability Improvements

Audit

ClientFacing

Disaster Recovery Methodology


The DR plan has been developed to recover critical computer based

applications and services within 2 days of a full scale disaster.

CarrierServices

WAN

Relocate Operations

Hotsite

Mission Statement


• The DR plan will provide immediate response and subsequent recovery from any unplanned computing service interruption.

• Provide an organized and consolidated approach to managing response and recovery activities.

• Recover essential operations in a timely manner.

Scope of The Disaster Recovery Plan


+ Only the primary site has been disabled by the disruption, all other facilities are unaffected.

+ The offsite storage location for critical backup media is accessible

+ Qualified personnel identified in the DR plan are available to perform DR responsibilities

+ The hotsite is accessible and available

DR Planning Assumptions


• Identify to the business which computing services have been identified as mission critical and what will be covered within the Mission Statement.

• These servers have been identified as A list recovery servers

• The non-critical business systems will have a low priority and will be referred to as the B list of servers

Critical Servers


Combination of Tiers and CriticalityTier 1 - Availability RTO: 2 RPO: LST

Platform Server Hardware Criticality Core Application

IBMi CUSDB100 Physical A JDE EnterpriseOne

Windows CUSWE1WB100 Virtual A JDE Web Server

Unix CUSUE1AP100 Virtual A JDE Logic Server

Windows CUSWIFDC100 Virtual A Domain Controller

Tier 2 - Protect RTO: 12 RPO: 1Platform Server Hardware Criticality Core Application

Windows CUSWE1DB100 Virtual A DSI Database

Windows CUSWE1DP100 Virtual A Deployment Server

Windows CUXWE1AP101 Virtual B DSI Server

Windows CUSWE1AP102 Virtual B Print Server

Best Effort RTO: BE RPO: 24/BEPlatform Server Hardware Criticality Core Application

Windows CUSWIFAP100 Virtual C Radius Server

Windows CUSWE1AP100 Virtual C 3rd Party Application

Windows CUSWE1AP103 Virtual C CreateForms

Not In Scope RTO: None RPO: NonePlatform Server Hardware Criticality Core Application

Windows CUSWIFEX100 Virtual None Exchange Server

Windows CUSWIFLS100 Virtual None Skype Server

Tier 3 - Recover RTO: 48 RPO: 24Platform Server Hardware Criticality Core Application

Windows CUSWIFFS100 Virtual A File Server

Windows CUSWE1AP100 Virtual A 3rd Party Application

Windows CUSWE1AP103 Virtual B CreateForms


• People assignments, responsibilities & training

• Site: Selection and environment preparation

• Vital records: Inventory and Backup

• Software Systems: Inventory and Backup

• Application Systems: Inventory and Backup

Plan Elements


Appendix

+ Vendor agreements and contracts

+ Telephone calling tree

+ Team notification procedures

+ Recovery and resumption sites, addresses, telephone numbers

Plan Elements

Call Tree Information

• Name

• Title

• Address

(Street address, not PO box number)

• Office telephone number

• Home telephone number

• Pager number, if still applicable

• Cellular telephone number, if available

• Personal Email

• Alternate telephone number



Initial Assessment

First Alert

Event

Repair Server Room

Declare?No

Alert the Hot-SiteAssemble Team Leaders at Command Center

Damage Assessment

If YES, call Hot-Site and make Formal DeclarationIf NO, get estimate on repair

Potential Disaster Notification

Disaster? Or Cancel Alert?

Declare DisasterYes

Plan Activation Matrix


Team Mobilization Authorize travel to Hot-Site facility

Hot-Site Restoration Provide essential services from Hot-Site

RebuildRebuild all Critical Services at Hot-Site

Plan Return Develop “Action Plan” for returning to repaired or new facility

Plan ActivationActivate Recovery TeamRetrieve offsite Storage DR Tapes

Declare DisasterYes

Plan Activation Matrix


Crises communications:

• How will you notify other employees?

+ 1-800 Communications

+ Email via phone / text messaging

• Dealing with the media

+ The media will be there before you

+ Assign a Spokesperson

Key Steps To Disaster Recovery Planning


Home DR Plan


79% of Americans do not have

a family emergency plan.

Bring peace of mind through

preparedness.


• Provide immediate response from any unplanned FEMA related event

• Provide an organized response in an family home emergency

• Provide safe passage in a timely manner

• Family Communications

Family Emergency Planning


Sources


Make a


PLAN to Survive• Know the potential emergencies

• Protect your family -- your most important inner circle

• Involve your entire family in the creation and execution of your emergency plan

• Think first about the basics ( safety, water, shelter, warmth)

• Know what kinds of emergencies might affect your family

• Talk to utility service providers

• Promote family and individual preparedness

• Provide for sheltering-in-place or evacuation measures

• Fire is the most common family emergency

• Prepare for medical emergencies, encourage basic first aid and CPR training


Emergency Preparedness

Starts with:

• Make a Kit• Example: First aid kit; flashlight; emergency numbers; multi-purpose tool; etc.

• Make a Plan

• Have a family meeting• Discuss the emergencies that are most likely to happen where you live, learn, & work

and how to prepare and respond to them.• Memorize family member’s phone numbers

• Practice Your Plan

• At a minimum, talk through the scenarios – “If the fire detector goes off and you see smoke, what will you do?”


Family Emergency Planning

1) Be informed about the different types of emergencies that may impact your family• Plan appropriate protective actions;

2) Make a family emergency plan• Including information on how to reconnect and reunite;

3) Build an emergency supply kit – Have the supplies you will need to survive• Home, at work or in the car;

4) Get involved in your neighborhood• Support community preparedness

See more at: http://www.adcouncil.org/Our-Campaigns/Safety/Emergency-Preparedness#sthash.3jDjLkbZ.pYkMAA5H.dpuf

http://www.adcouncil.org/Our-Campaigns/Safety/Emergency-Preparedness#sthash.3jDjLkbZ.pYkMAA5H.dpuf


Don’t Wait – Communicate


Tornados – Shelby County


Don’t Wait – Communicate


Where & How


Keep your contacts updated across all of your channels, including phone, email and social media.

This will make it easy to reach out to the right people quickly to get information and supply updates. Consider creating a group list serve of your top contacts.

• Know how to send text from your smart phone & Via Internet to your contacts and social channels ✓ text messages and the internet often will work in the event of a phone service disruption

• Keep extra batteries for your phone in a safe/ dry place

• Solar-powered or hand crank charger. • These chargers are good emergency tools to keep your laptop and

other small electronics working in the event of a power outage• Purchase a car phone charger - if you lose power in your home

Where & How


Where & How4 WAYS TO CHARGE:

This Compact & Light weight Radio, Flash Light & Power Bank can be charged by Solar, Hand Cranked, USB cable or with 3 AAA batteries

provides up to 15 hours of light or 6 hours of radio time

• RADIO & TORCH: Emergency NOAA Weather, Radio AM/FM


Emergency Supplies

• Everyone should have some basic supplies on hand in order to survive for at least 3 days in an emergency

• Consider where you live and the unique needs of your family in order to create an emergency supply kit that will meet your needs

• Consider having at least two emergency supply kits, one full kit at home and smaller portable kits in their workplace, vehicle or other places they spend time.


Home Emergency Plan

Danger Phrase

House Logistics PlanPractice what to do when home emergencies happen.

Tornado, Flood, Fire, Hurricane, Mudslide, Home Invasion, Chemical Spill, Animal\Snake attack

Having a secret phrase we say when there is stranger danger, home invasion, or suspicious behavior by non family members.Selecting a word or phrase to say when we sense danger and need to get away. “Great Grandfather's Birthday is tomorrow.”

Three Day Duffel Bag

Prepare a duffel bag to take along that will sustain family for three days.Solar Flashlight, Water Bottles, Peanut Butter, Signal Flairs, Pepper Spray, Socks, Sandals, Mirror, Matches, Rations, Mylar Blankets, Pocket Knives, Duct Tape, Screwdriver, Hatchet, Paper, Pencil, Rain Ponchos, Plastic Bags, Bathroom Tissue, Bug Spray, Sling Shot, Fishing Line, Fish

Hook, Pliers, Gloves, Satellite Phone, Solar Electrical Charger for Phone

Connection PointWhere will you meet up if you get separated?

Choose a safe location that is understood by everyone, but not necessarily publicized that everyone can walk to in 1 day’s time.

Master SpreadsheetCreate a Spreadsheet with all Family Account Numbers, URLs, Logins, Passwords, PINs, and Balances.

Create a spreadsheet for all bills, investments, insurance policies, loans, credit cards, debit cards. Store in Password encrypted file and email to family members.

Emergency FundCreate a $1000 Family Emergency Fund

Put $1,000 in a savings account in the bank. It will come in handy for minor emergencies like the washer or car breaking.

Travel BackupsMake electronic copies of all Important Items and email them to yourself.

Take pictures of all your family passports, driver licenses, car insurance proof, debit cards, credit cards and email them to yourself in case they are lost or stolen…especially when you travel.

Remote VideoSet up cameras in your home that can be remotely activated.

New home systems are accessible by cell phone and can be activated by motion sensors as well. Heating and Cooling sensors can be connected to the system to alert of possible problems. Video can be recorded of home activity when you are away.

Roadside AssistanceObtain Roadside Assistance Cards for All Family Members

Get Roadside Assistance cards like AAA Plus with 100 mile towing for all family members. It will pay for itself and provide safety for travelers.


Home DR

Practical Skills:

• Know basic first aid

• Know basic CPR technique

• Know how to maintain fire/smoke detectors

• Know how to boil water for safe drinking

• Know how to use a fire extinguisher

• Road safety during emergencies

(don’t drive in high water)


Doing the Little Things


Emergency Planning


Earthquake Tools


In Case of Emergency Emergency Response is trained to ask you if you have Program

"In Case of Emergency" (ICE) contacts into your cell phone

• Lets your ICE contacts know that they are programmed into your phone • Emergency personnel can contact those people for you if you are unable

to use your phone • Inform EMS of any medical issues or other special needs you may have


In Case of Emergency

Here are some Advanced ICE apps features:

� One-touch activation � Sends four automatic text messages � Picture for identification � Name , Age � Blood type , Allergies , Medications � Medical condition � Insurance information � (Click to call) Link to primary contact � (Click to call) Link to insurance company � (Click to call) Link to primary physician


In Case of Emergency • If you are evacuated, turn on your call-forwarding on your home phone forward your home phone number to your out of state contact cell phone number

• Keep a prepaid phone card to use if needed during or after a disaster

• Immediately following a disaster, resist using your mobile device to watch streaming videos, download music or videos, or play online video games, all of which can add to network congestion.

• Limiting use of these services can help potentially life-saving emergency calls get through to 9-1-1.

• For non-emergency communications, use text messaging, e-mail, or social media instead of making voice calls on your cell phone.

• You can also use social media to post your status to let family and friends know you are okay. • In addition to Facebook and Twitter, you can use resources such as the American Red Cross's Safe and Well

program.


Safe and Well If you have been affected by a disaster use the Safe and Well Website

post "safe and well messages" that your loved ones can view.

How:1. Registering yourself on the Safe and Well Web site is completely

voluntary and you can update your entry at any time. 1. Family can search on this site for your information need to:

1. enter your name, along with your address or phone number.

The search result will show only your first name, last name, the date and time of registration, and the messages you selected to tell your story. Tell your family you are safe

Registration information may be provided to other organizations to locate missing persons, help reunite loved ones, or provide other disaster relief services.


Emergency Information – Go- to Sites


Human Elements – Ready.gov

• Local Emergency Management Alerts

• Early weather warning Systems

• Evacuation Routes

• FEMA Updates

• Red Cross Updates


Until the basic personal needs are met:

• Family comes first. Always!!

• Staff members will not focus on the Enterprise recovery

• Staff members may choose not to be available

Regional Disaster Effects on Staff

http://images.google.com/imgres?q=family&start=200&num=10&hl=en&tbo=d&biw=1600&bih=699&tbm=isch&tbnid=tO3D1ElputaFhM:&imgrefurl=http://news.travel.aol.com/2010/10/05/phoenix-with-kids/&docid=qervpGQIGBw7WM&imgurl=http://o.aolcdn.com/os/travel/editorial/1-family-vacation-1-320lvg063010&w=320&h=180&ei=V6fEUIOlGIbiigKVn4G4BQ&zoom=1&iact=hc&vpx=904&vpy=146&dur=31&hovh=144&hovw=256&tx=127&ty=69&sig=109179595583375503785&page=6&tbnh=141&tbnw=228&ndsp=43&ved=1t:429,r:13,s:200,i:43


The Ad Council

Will You Be

Prepared?


Send an Invitation & connect on LinkedIn

Receive Tip & Techniques Monthly

Industry Best Practices to improve your Business

Continuity so you:

Don’t Fall with the Fallen!

www.linkedin.com/in/richarddolewski

[email protected]

Richard Dolewski


disasters - a need for three-dimensional thinking a nee… · rpo rto. maximum time in order to...

Documents