disaster recovery: lessons from katrina gail ann mccreary, crm mississippi power company

83
Disaster Recovery: Disaster Recovery: Lessons from Katrina Lessons from Katrina Gail Ann McCreary, CRM Gail Ann McCreary, CRM Mississippi Power Company Mississippi Power Company

Upload: clifton-thompson

Post on 20-Jan-2016

215 views

Category:

Documents


1 download

TRANSCRIPT

Page 1: Disaster Recovery: Lessons from Katrina Gail Ann McCreary, CRM Mississippi Power Company

Disaster Recovery:Disaster Recovery:Lessons from KatrinaLessons from Katrina

Gail Ann McCreary, CRMGail Ann McCreary, CRMMississippi Power CompanyMississippi Power Company

Page 2: Disaster Recovery: Lessons from Katrina Gail Ann McCreary, CRM Mississippi Power Company
Page 3: Disaster Recovery: Lessons from Katrina Gail Ann McCreary, CRM Mississippi Power Company

Hurricane Katrina – August 29, Hurricane Katrina – August 29, 20052005

• Worst natural disaster in U.S. historyWorst natural disaster in U.S. history

• Affected Alabama – Mississippi – Affected Alabama – Mississippi – LouisianaLouisiana

• 250 miles wide250 miles wide

• 140 mph winds140 mph winds

• 30 ft tidal surge along coast line30 ft tidal surge along coast line

Page 4: Disaster Recovery: Lessons from Katrina Gail Ann McCreary, CRM Mississippi Power Company
Page 5: Disaster Recovery: Lessons from Katrina Gail Ann McCreary, CRM Mississippi Power Company
Page 6: Disaster Recovery: Lessons from Katrina Gail Ann McCreary, CRM Mississippi Power Company
Page 7: Disaster Recovery: Lessons from Katrina Gail Ann McCreary, CRM Mississippi Power Company
Page 8: Disaster Recovery: Lessons from Katrina Gail Ann McCreary, CRM Mississippi Power Company
Page 9: Disaster Recovery: Lessons from Katrina Gail Ann McCreary, CRM Mississippi Power Company

Were Dealing With…Were Dealing With…

• Rain and wind damageRain and wind damage• TornadoesTornadoes• FloodingFlooding• ““Coal water”Coal water”• Submerged recordsSubmerged records• Bio-hazardBio-hazard

Page 10: Disaster Recovery: Lessons from Katrina Gail Ann McCreary, CRM Mississippi Power Company
Page 11: Disaster Recovery: Lessons from Katrina Gail Ann McCreary, CRM Mississippi Power Company
Page 12: Disaster Recovery: Lessons from Katrina Gail Ann McCreary, CRM Mississippi Power Company
Page 13: Disaster Recovery: Lessons from Katrina Gail Ann McCreary, CRM Mississippi Power Company
Page 14: Disaster Recovery: Lessons from Katrina Gail Ann McCreary, CRM Mississippi Power Company
Page 15: Disaster Recovery: Lessons from Katrina Gail Ann McCreary, CRM Mississippi Power Company
Page 16: Disaster Recovery: Lessons from Katrina Gail Ann McCreary, CRM Mississippi Power Company
Page 17: Disaster Recovery: Lessons from Katrina Gail Ann McCreary, CRM Mississippi Power Company
Page 18: Disaster Recovery: Lessons from Katrina Gail Ann McCreary, CRM Mississippi Power Company

Determined PrioritiesDetermined Priorities

• Damage incurredDamage incurred

• Business unit and functionBusiness unit and function

• Criticality to restoration effortCriticality to restoration effort

• Building access issuesBuilding access issues

Page 19: Disaster Recovery: Lessons from Katrina Gail Ann McCreary, CRM Mississippi Power Company
Page 20: Disaster Recovery: Lessons from Katrina Gail Ann McCreary, CRM Mississippi Power Company

The Plan….The Plan….

• Identify records needing to be driedIdentify records needing to be dried

• Pack into boxesPack into boxes

• Stage on palletsStage on pallets

• Shrink wrapShrink wrap

• Load into freezer truckLoad into freezer truck

Page 21: Disaster Recovery: Lessons from Katrina Gail Ann McCreary, CRM Mississippi Power Company

First PrioritiesFirst Priorities

• Substation ConstructionSubstation Construction

• PowercoPowerco

• Plant WatsonPlant Watson

Page 22: Disaster Recovery: Lessons from Katrina Gail Ann McCreary, CRM Mississippi Power Company
Page 23: Disaster Recovery: Lessons from Katrina Gail Ann McCreary, CRM Mississippi Power Company
Page 24: Disaster Recovery: Lessons from Katrina Gail Ann McCreary, CRM Mississippi Power Company
Page 25: Disaster Recovery: Lessons from Katrina Gail Ann McCreary, CRM Mississippi Power Company
Page 26: Disaster Recovery: Lessons from Katrina Gail Ann McCreary, CRM Mississippi Power Company
Page 27: Disaster Recovery: Lessons from Katrina Gail Ann McCreary, CRM Mississippi Power Company
Page 28: Disaster Recovery: Lessons from Katrina Gail Ann McCreary, CRM Mississippi Power Company
Page 29: Disaster Recovery: Lessons from Katrina Gail Ann McCreary, CRM Mississippi Power Company

Plan ChangedPlan Changed

• Downtown buildings had to be vacatedDowntown buildings had to be vacated

• All dry recordsAll dry records

• All PCs, printers, phonesAll PCs, printers, phones

• Employee personal effectsEmployee personal effects

• Undamaged furniture and modularUndamaged furniture and modular

• Office suppliesOffice supplies

• 400 employees400 employees

Page 30: Disaster Recovery: Lessons from Katrina Gail Ann McCreary, CRM Mississippi Power Company

New IssuesNew Issues

• Needed more boxes!!!!Needed more boxes!!!!

• Needed temporary office space for Needed temporary office space for 400 employees400 employees

• What to do with all the “stuff”What to do with all the “stuff”

• Employees still on storm dutyEmployees still on storm duty

• 2-3 weeks2-3 weeks

Page 31: Disaster Recovery: Lessons from Katrina Gail Ann McCreary, CRM Mississippi Power Company

Plan BPlan B

DR would stay and pack dry records DR would stay and pack dry records too too

• I did not have the manpowerI did not have the manpower

• DR was experienced DR was experienced

• Had supply and manpower resources Had supply and manpower resources in placein place

Page 32: Disaster Recovery: Lessons from Katrina Gail Ann McCreary, CRM Mississippi Power Company

Determined Numbering Determined Numbering SchemeScheme• Building IDBuilding ID

• Floor numberFloor number

• Location numberLocation number

202-101 = General Office 2202-101 = General Office 2ndnd floor cubicle #202 floor cubicle #202

SI221-106 = G&SI 2SI221-106 = G&SI 2ndnd floor location #221 floor location #221

842-102 = Hancock Bank 8842-102 = Hancock Bank 8thth floor location #842 floor location #842

Page 33: Disaster Recovery: Lessons from Katrina Gail Ann McCreary, CRM Mississippi Power Company
Page 34: Disaster Recovery: Lessons from Katrina Gail Ann McCreary, CRM Mississippi Power Company

Challenges FacedChallenges Faced

• Locked doorsLocked doors

• On-goingOn-going building clean-up & restoration building clean-up & restoration

• No elevatorsNo elevators

• No electricityNo electricity

• Storage for 7700+ boxesStorage for 7700+ boxes

Page 35: Disaster Recovery: Lessons from Katrina Gail Ann McCreary, CRM Mississippi Power Company

Back to “Normal”Back to “Normal”

Page 36: Disaster Recovery: Lessons from Katrina Gail Ann McCreary, CRM Mississippi Power Company
Page 37: Disaster Recovery: Lessons from Katrina Gail Ann McCreary, CRM Mississippi Power Company
Page 38: Disaster Recovery: Lessons from Katrina Gail Ann McCreary, CRM Mississippi Power Company

Phase II Phase II

• Multiple temporary officesMultiple temporary offices

• No storage in temporary officesNo storage in temporary offices

• No file cabinetsNo file cabinets

• Delivery issuesDelivery issues

• Palletized boxes not in order by Palletized boxes not in order by locationlocation

Page 39: Disaster Recovery: Lessons from Katrina Gail Ann McCreary, CRM Mississippi Power Company

Phase IIPhase II

• Employees would have to reviewEmployees would have to review

• Coordinate with occupancy scheduleCoordinate with occupancy schedule

• Named coordinators for each siteNamed coordinators for each site

• Small volumes delivered where Small volumes delivered where possiblepossible

Page 40: Disaster Recovery: Lessons from Katrina Gail Ann McCreary, CRM Mississippi Power Company

Location No. ______818 _________Name _Lateral file __

Skid Box No. Destroy Took Delivered Interim R/C

1006 101

104

105

109

105

1007 102

103

106

Page 41: Disaster Recovery: Lessons from Katrina Gail Ann McCreary, CRM Mississippi Power Company

OptionsOptions

• TakeTake

• DestroyDestroy

• InterimInterim

• Records center storageRecords center storage

Page 42: Disaster Recovery: Lessons from Katrina Gail Ann McCreary, CRM Mississippi Power Company
Page 43: Disaster Recovery: Lessons from Katrina Gail Ann McCreary, CRM Mississippi Power Company
Page 44: Disaster Recovery: Lessons from Katrina Gail Ann McCreary, CRM Mississippi Power Company

Lessons Learned - Lessons Learned - Disaster Recovery ProcessDisaster Recovery Process

• Utilize document recovery contractorUtilize document recovery contractor

Page 45: Disaster Recovery: Lessons from Katrina Gail Ann McCreary, CRM Mississippi Power Company

Lessons Learned -Lessons Learned -Disaster Recovery ProcessDisaster Recovery Process

• Utilize document recovery contractorUtilize document recovery contractor

• Understand the process / clarify costsUnderstand the process / clarify costs

Page 46: Disaster Recovery: Lessons from Katrina Gail Ann McCreary, CRM Mississippi Power Company

Lessons Learned - Lessons Learned - Disaster Recovery ProcessDisaster Recovery Process

• Utilize document recovery contractorUtilize document recovery contractor

• Understand the process / clarify costsUnderstand the process / clarify costs

• Utilize multiple peopleUtilize multiple people

Page 47: Disaster Recovery: Lessons from Katrina Gail Ann McCreary, CRM Mississippi Power Company

Lessons Learned - Lessons Learned - Disaster Recovery ProcessDisaster Recovery Process

• Utilize document recovery contractorUtilize document recovery contractor

• Understand the process / clarify costsUnderstand the process / clarify costs

• Utilize multiple peopleUtilize multiple people

• Work closely with contractorWork closely with contractor

Page 48: Disaster Recovery: Lessons from Katrina Gail Ann McCreary, CRM Mississippi Power Company

Lessons Learned - Lessons Learned - Disaster Recovery ProcessDisaster Recovery Process

• Utilize document recovery contractorUtilize document recovery contractor

• Understand the process / clarify costsUnderstand the process / clarify costs

• Utilize multiple peopleUtilize multiple people

• Work closely with contractorWork closely with contractor

• Walk-thru after packing completedWalk-thru after packing completed

Page 49: Disaster Recovery: Lessons from Katrina Gail Ann McCreary, CRM Mississippi Power Company

Lessons Learned - Lessons Learned - Disaster Recovery ProcessDisaster Recovery Process

• Utilize document recovery contractorUtilize document recovery contractor

• Understand the process / clarify costsUnderstand the process / clarify costs

• Utilize multiple peopleUtilize multiple people

• Work closely with contractorWork closely with contractor

• Walk-thru after packing completedWalk-thru after packing completed

• Have floor plans for all buildingsHave floor plans for all buildings

Page 50: Disaster Recovery: Lessons from Katrina Gail Ann McCreary, CRM Mississippi Power Company

Lessons Learned - Lessons Learned - Disaster Recovery ProcessDisaster Recovery Process

• Utilize document recovery contractorUtilize document recovery contractor

• Understand the process / clarify costsUnderstand the process / clarify costs

• Utilize multiple peopleUtilize multiple people

• Work closely with contractorWork closely with contractor

• Walk-thru after packing completedWalk-thru after packing completed

• Have floor plans for all buildingsHave floor plans for all buildings

• Document! Document! Document!Document! Document! Document!

Page 51: Disaster Recovery: Lessons from Katrina Gail Ann McCreary, CRM Mississippi Power Company

Lessons Learned - Lessons Learned - Disaster Recovery ProcessDisaster Recovery Process

• Need business continuity planNeed business continuity plan

Page 52: Disaster Recovery: Lessons from Katrina Gail Ann McCreary, CRM Mississippi Power Company

Lessons Learned - Lessons Learned - Disaster Recovery ProcessDisaster Recovery Process

• Need business continuity planNeed business continuity plan

• Definition of vital records has Definition of vital records has changedchanged

Page 53: Disaster Recovery: Lessons from Katrina Gail Ann McCreary, CRM Mississippi Power Company

Lessons Learned - Lessons Learned - Disaster Recovery ProcessDisaster Recovery Process

• Need business continuity planNeed business continuity plan

• Definition of vital records has Definition of vital records has changedchanged

• Have emergency contact numbersHave emergency contact numbers

Page 54: Disaster Recovery: Lessons from Katrina Gail Ann McCreary, CRM Mississippi Power Company

Lessons Learned - Lessons Learned - Disaster Recovery ProcessDisaster Recovery Process

• Need business continuity planNeed business continuity plan

• Definition of vital records has Definition of vital records has changedchanged

• Have emergency contact numbersHave emergency contact numbers

• Post-critiquePost-critique

Page 55: Disaster Recovery: Lessons from Katrina Gail Ann McCreary, CRM Mississippi Power Company

Lessons Learned - FacilitiesLessons Learned - Facilities

• Get to know your Facilities Get to know your Facilities departmentdepartment

Page 56: Disaster Recovery: Lessons from Katrina Gail Ann McCreary, CRM Mississippi Power Company

Lessons Learned - FacilitiesLessons Learned - Facilities

• Get to know your Facilities Get to know your Facilities departmentdepartment

• Establish liaisons with field/branch Establish liaisons with field/branch officesoffices

Page 57: Disaster Recovery: Lessons from Katrina Gail Ann McCreary, CRM Mississippi Power Company

Lessons Learned - FacilitiesLessons Learned - Facilities

• Get to know your Facilities Get to know your Facilities departmentdepartment

• Establish liaisons with field/branch Establish liaisons with field/branch officesoffices

• Limit employees’ personal itemsLimit employees’ personal items

Page 58: Disaster Recovery: Lessons from Katrina Gail Ann McCreary, CRM Mississippi Power Company

Lessons Learned - FacilitiesLessons Learned - Facilities

• Get to know your Facilities Get to know your Facilities departmentdepartment

• Establish liaisons with field/branch Establish liaisons with field/branch officesoffices

• Limit employees’ personal itemsLimit employees’ personal items

• No locked file cabinets or overheadsNo locked file cabinets or overheads

Page 59: Disaster Recovery: Lessons from Katrina Gail Ann McCreary, CRM Mississippi Power Company

Lessons Learned - FacilitiesLessons Learned - Facilities

• Get to know your Facilities Get to know your Facilities departmentdepartment

• Establish liaisons with field/branch Establish liaisons with field/branch officesoffices

• Limit employees’ personal itemsLimit employees’ personal items

• No locked file cabinets or overheadsNo locked file cabinets or overheads

• Take office keys with youTake office keys with you

Page 60: Disaster Recovery: Lessons from Katrina Gail Ann McCreary, CRM Mississippi Power Company

Lessons Learned - FacilitiesLessons Learned - Facilities

• Get to know your Facilities departmentGet to know your Facilities department

• Establish liaisons with field/branch Establish liaisons with field/branch officesoffices

• Limit employees’ personal itemsLimit employees’ personal items

• No locked file cabinets or overheadsNo locked file cabinets or overheads

• Take office keys with youTake office keys with you

• No critical or only copy records in No critical or only copy records in disaster susceptible areasdisaster susceptible areas

Page 61: Disaster Recovery: Lessons from Katrina Gail Ann McCreary, CRM Mississippi Power Company

Lessons Learned - PersonnelLessons Learned - Personnel

• Storm procedures not heededStorm procedures not heeded

Page 62: Disaster Recovery: Lessons from Katrina Gail Ann McCreary, CRM Mississippi Power Company

Lessons Learned - PersonnelLessons Learned - Personnel

• Storm procedures not heededStorm procedures not heeded

• Records preparation should be part Records preparation should be part of storm preparation and of storm preparation and communicationscommunications

Page 63: Disaster Recovery: Lessons from Katrina Gail Ann McCreary, CRM Mississippi Power Company

Lessons Learned - PersonnelLessons Learned - Personnel

• Storm procedures not heededStorm procedures not heeded

• Records preparation should be part Records preparation should be part of storm preparation and of storm preparation and communicationscommunications

• Files should be labeled / identifiedFiles should be labeled / identified

Page 64: Disaster Recovery: Lessons from Katrina Gail Ann McCreary, CRM Mississippi Power Company

Lessons Learned - PersonnelLessons Learned - Personnel

• Storm procedures not heededStorm procedures not heeded

• Records preparation should be part Records preparation should be part of storm preparation and of storm preparation and communicationscommunications

• Files should be labeled / identifiedFiles should be labeled / identified

• Enforce corporate policiesEnforce corporate policies

Page 65: Disaster Recovery: Lessons from Katrina Gail Ann McCreary, CRM Mississippi Power Company

Lessons Learned - PersonnelLessons Learned - Personnel

• Storm procedures not heededStorm procedures not heeded

• Records preparation should be part Records preparation should be part of storm preparation and of storm preparation and communicationscommunications

• Files should be labeled / identifiedFiles should be labeled / identified

• Enforce corporate policiesEnforce corporate policies

• People are stressedPeople are stressed

Page 66: Disaster Recovery: Lessons from Katrina Gail Ann McCreary, CRM Mississippi Power Company

Lessons Learned - PersonnelLessons Learned - Personnel

• Storm procedures not heededStorm procedures not heeded

• Records preparation should be part of Records preparation should be part of storm preparation and communicationsstorm preparation and communications

• Files should be labeled / identifiedFiles should be labeled / identified

• Enforce corporate policiesEnforce corporate policies

• People are stressedPeople are stressed

• Promote disaster preparedness and vital Promote disaster preparedness and vital records protectionrecords protection

Page 67: Disaster Recovery: Lessons from Katrina Gail Ann McCreary, CRM Mississippi Power Company

Silver LiningsSilver Linings

• Files were cleaned outFiles were cleaned out– Inactive records sent to storageInactive records sent to storage– Destruction of expired and unnecessary Destruction of expired and unnecessary

recordsrecords– Reduction in file volumeReduction in file volume

Page 68: Disaster Recovery: Lessons from Katrina Gail Ann McCreary, CRM Mississippi Power Company

Silver LiningsSilver Linings

• Files were cleaned outFiles were cleaned out– Inactive records sent to storageInactive records sent to storage– Destruction of expired and unnecessary Destruction of expired and unnecessary

recordsrecords– Reduction in file volumeReduction in file volume

• New contacts with business unitsNew contacts with business units

Page 69: Disaster Recovery: Lessons from Katrina Gail Ann McCreary, CRM Mississippi Power Company

Silver LiningsSilver Linings

• Files were cleaned outFiles were cleaned out– Inactive records sent to storageInactive records sent to storage– Destruction of expired and unnecessary Destruction of expired and unnecessary

recordsrecords– Reduction in file volumeReduction in file volume

• New contacts with business unitsNew contacts with business units

• Multiple opportunities to promote Multiple opportunities to promote RIMRIM

Page 70: Disaster Recovery: Lessons from Katrina Gail Ann McCreary, CRM Mississippi Power Company

Silver LiningsSilver Linings

• Files were cleaned outFiles were cleaned out– Inactive records sent to storageInactive records sent to storage– Destruction of expired and unnecessary Destruction of expired and unnecessary

recordsrecords– Reduction in file volumeReduction in file volume

• New contacts with business unitsNew contacts with business units• Multiple opportunities to promote RIMMultiple opportunities to promote RIM• Department credibility enhancedDepartment credibility enhanced

Page 71: Disaster Recovery: Lessons from Katrina Gail Ann McCreary, CRM Mississippi Power Company

Silver LiningsSilver Linings

• Disaster planning for records now Disaster planning for records now prioritypriority

Page 72: Disaster Recovery: Lessons from Katrina Gail Ann McCreary, CRM Mississippi Power Company

Silver LiningsSilver Linings

• Disaster planning for records now Disaster planning for records now prioritypriority

• Records protection procedures part Records protection procedures part of storm manualof storm manual

Page 73: Disaster Recovery: Lessons from Katrina Gail Ann McCreary, CRM Mississippi Power Company

Silver LiningsSilver Linings

• Disaster planning for records now Disaster planning for records now prioritypriority

• Records protection procedures part Records protection procedures part of storm manualof storm manual

• No people – or records – in Lower No people – or records – in Lower LevelLevel

Page 74: Disaster Recovery: Lessons from Katrina Gail Ann McCreary, CRM Mississippi Power Company

Silver LiningsSilver Linings

• Disaster planning for records now Disaster planning for records now prioritypriority

• Records protection procedures part of Records protection procedures part of storm manualstorm manual

• No people – or records – in Lower No people – or records – in Lower LevelLevel

• EOC and critical functions moved from EOC and critical functions moved from coastal zonecoastal zone

Page 75: Disaster Recovery: Lessons from Katrina Gail Ann McCreary, CRM Mississippi Power Company

Silver LiningsSilver Linings

• RIM storm assignment changedRIM storm assignment changed

Page 76: Disaster Recovery: Lessons from Katrina Gail Ann McCreary, CRM Mississippi Power Company

Silver LiningsSilver Linings

• RIM storm assignment changedRIM storm assignment changed

• Employees will not be complacent Employees will not be complacent about next stormabout next storm

Page 77: Disaster Recovery: Lessons from Katrina Gail Ann McCreary, CRM Mississippi Power Company

ConclusionsConclusions

• Be proactive vs. reactiveBe proactive vs. reactive

Page 78: Disaster Recovery: Lessons from Katrina Gail Ann McCreary, CRM Mississippi Power Company

ConclusionsConclusions

• Be proactive vs. reactiveBe proactive vs. reactive

• Good sense, sound judgment and Good sense, sound judgment and creativity are necessarycreativity are necessary

Page 79: Disaster Recovery: Lessons from Katrina Gail Ann McCreary, CRM Mississippi Power Company

ConclusionsConclusions

• Be proactive vs. reactiveBe proactive vs. reactive

• Good sense, sound judgment and Good sense, sound judgment and creativity are necessarycreativity are necessary

• Don’t be complacentDon’t be complacent

Page 80: Disaster Recovery: Lessons from Katrina Gail Ann McCreary, CRM Mississippi Power Company

ConclusionsConclusions

• Be proactive vs. reactiveBe proactive vs. reactive

• Good sense, sound judgment and Good sense, sound judgment and creativity are necessarycreativity are necessary

• Don’t be complacentDon’t be complacent

• Expect the unexpectedExpect the unexpected

Page 81: Disaster Recovery: Lessons from Katrina Gail Ann McCreary, CRM Mississippi Power Company

ConclusionsConclusions

• Be proactive vs. reactiveBe proactive vs. reactive

• Good sense, sound judgment and Good sense, sound judgment and creativity are necessarycreativity are necessary

• Don’t be complacentDon’t be complacent

• Expect the unexpectedExpect the unexpected

• Evaluate suppliers and servicesEvaluate suppliers and services

Page 82: Disaster Recovery: Lessons from Katrina Gail Ann McCreary, CRM Mississippi Power Company

ConclusionsConclusions

• Be proactive vs. reactiveBe proactive vs. reactive

• Good sense, sound judgment and Good sense, sound judgment and creativity are necessarycreativity are necessary

• Don’t be complacentDon’t be complacent

• Expect the unexpectedExpect the unexpected

• Evaluate suppliers and servicesEvaluate suppliers and services

• Empower peopleEmpower people

Page 83: Disaster Recovery: Lessons from Katrina Gail Ann McCreary, CRM Mississippi Power Company

Questions?Questions?

Gail Ann McCreary, CRMGail Ann McCreary, CRM228-865-5467 228-865-5467 [email protected]@southernco.com