disaster recovery: lessons from katrina gail ann mccreary, crm mississippi power company

Disaster Recovery:Disaster Recovery:Lessons from KatrinaLessons from Katrina

Gail Ann McCreary, CRMGail Ann McCreary, CRMMississippi Power CompanyMississippi Power Company

Hurricane Katrina – August 29, Hurricane Katrina – August 29, 20052005

• Worst natural disaster in U.S. historyWorst natural disaster in U.S. history

• Affected Alabama – Mississippi – Affected Alabama – Mississippi – LouisianaLouisiana

• 250 miles wide250 miles wide

• 140 mph winds140 mph winds

• 30 ft tidal surge along coast line30 ft tidal surge along coast line

Were Dealing With…Were Dealing With…

• Rain and wind damageRain and wind damage• TornadoesTornadoes• FloodingFlooding• ““Coal water”Coal water”• Submerged recordsSubmerged records• Bio-hazardBio-hazard

Determined PrioritiesDetermined Priorities

• Damage incurredDamage incurred

• Business unit and functionBusiness unit and function

• Criticality to restoration effortCriticality to restoration effort

• Building access issuesBuilding access issues

The Plan….The Plan….

• Identify records needing to be driedIdentify records needing to be dried

• Pack into boxesPack into boxes

• Stage on palletsStage on pallets

• Shrink wrapShrink wrap

• Load into freezer truckLoad into freezer truck

First PrioritiesFirst Priorities

• Substation ConstructionSubstation Construction

• PowercoPowerco

• Plant WatsonPlant Watson

Plan ChangedPlan Changed

• Downtown buildings had to be vacatedDowntown buildings had to be vacated

• All dry recordsAll dry records

• All PCs, printers, phonesAll PCs, printers, phones

• Employee personal effectsEmployee personal effects

• Undamaged furniture and modularUndamaged furniture and modular

• Office suppliesOffice supplies

• 400 employees400 employees

New IssuesNew Issues

• Needed more boxes!!!!Needed more boxes!!!!

• Needed temporary office space for Needed temporary office space for 400 employees400 employees

• What to do with all the “stuff”What to do with all the “stuff”

• Employees still on storm dutyEmployees still on storm duty

• 2-3 weeks2-3 weeks

Plan BPlan B

DR would stay and pack dry records DR would stay and pack dry records too too

• I did not have the manpowerI did not have the manpower

• DR was experienced DR was experienced

• Had supply and manpower resources Had supply and manpower resources in placein place

Determined Numbering Determined Numbering SchemeScheme• Building IDBuilding ID

• Floor numberFloor number

• Location numberLocation number

202-101 = General Office 2202-101 = General Office 2ndnd floor cubicle #202 floor cubicle #202

SI221-106 = G&SI 2SI221-106 = G&SI 2ndnd floor location #221 floor location #221

842-102 = Hancock Bank 8842-102 = Hancock Bank 8thth floor location #842 floor location #842

Challenges FacedChallenges Faced

• Locked doorsLocked doors

• On-goingOn-going building clean-up & restoration building clean-up & restoration

• No elevatorsNo elevators

• No electricityNo electricity

• Storage for 7700+ boxesStorage for 7700+ boxes

Back to “Normal”Back to “Normal”

Phase II Phase II

• Multiple temporary officesMultiple temporary offices

• No storage in temporary officesNo storage in temporary offices

• No file cabinetsNo file cabinets

• Delivery issuesDelivery issues

• Palletized boxes not in order by Palletized boxes not in order by locationlocation

Phase IIPhase II

• Employees would have to reviewEmployees would have to review

• Coordinate with occupancy scheduleCoordinate with occupancy schedule

• Named coordinators for each siteNamed coordinators for each site

• Small volumes delivered where Small volumes delivered where possiblepossible

Location No. ______818 _________Name _Lateral file __

Skid Box No. Destroy Took Delivered Interim R/C

1006 101

104

105

109

105

1007 102

103

106

OptionsOptions

• TakeTake

• DestroyDestroy

• InterimInterim

• Records center storageRecords center storage

Lessons Learned - Lessons Learned - Disaster Recovery ProcessDisaster Recovery Process

• Utilize document recovery contractorUtilize document recovery contractor

Lessons Learned -Lessons Learned -Disaster Recovery ProcessDisaster Recovery Process

• Utilize document recovery contractorUtilize document recovery contractor

• Understand the process / clarify costsUnderstand the process / clarify costs

Lessons Learned - Lessons Learned - Disaster Recovery ProcessDisaster Recovery Process

• Utilize document recovery contractorUtilize document recovery contractor

• Understand the process / clarify costsUnderstand the process / clarify costs

• Utilize multiple peopleUtilize multiple people

Lessons Learned - Lessons Learned - Disaster Recovery ProcessDisaster Recovery Process

• Utilize document recovery contractorUtilize document recovery contractor

• Understand the process / clarify costsUnderstand the process / clarify costs

• Utilize multiple peopleUtilize multiple people

• Work closely with contractorWork closely with contractor

Lessons Learned - Lessons Learned - Disaster Recovery ProcessDisaster Recovery Process

• Utilize document recovery contractorUtilize document recovery contractor

• Understand the process / clarify costsUnderstand the process / clarify costs

• Utilize multiple peopleUtilize multiple people

• Work closely with contractorWork closely with contractor

• Walk-thru after packing completedWalk-thru after packing completed

Lessons Learned - Lessons Learned - Disaster Recovery ProcessDisaster Recovery Process

• Utilize document recovery contractorUtilize document recovery contractor

• Understand the process / clarify costsUnderstand the process / clarify costs

• Utilize multiple peopleUtilize multiple people

• Work closely with contractorWork closely with contractor

• Walk-thru after packing completedWalk-thru after packing completed

• Have floor plans for all buildingsHave floor plans for all buildings

Lessons Learned - Lessons Learned - Disaster Recovery ProcessDisaster Recovery Process

• Utilize document recovery contractorUtilize document recovery contractor

• Understand the process / clarify costsUnderstand the process / clarify costs

• Utilize multiple peopleUtilize multiple people

• Work closely with contractorWork closely with contractor

• Walk-thru after packing completedWalk-thru after packing completed

• Have floor plans for all buildingsHave floor plans for all buildings

• Document! Document! Document!Document! Document! Document!

Lessons Learned - Lessons Learned - Disaster Recovery ProcessDisaster Recovery Process

• Need business continuity planNeed business continuity plan

Lessons Learned - Lessons Learned - Disaster Recovery ProcessDisaster Recovery Process

• Need business continuity planNeed business continuity plan

• Definition of vital records has Definition of vital records has changedchanged

Lessons Learned - Lessons Learned - Disaster Recovery ProcessDisaster Recovery Process

• Need business continuity planNeed business continuity plan

• Definition of vital records has Definition of vital records has changedchanged

• Have emergency contact numbersHave emergency contact numbers

Lessons Learned - Lessons Learned - Disaster Recovery ProcessDisaster Recovery Process

• Need business continuity planNeed business continuity plan

• Definition of vital records has Definition of vital records has changedchanged

• Have emergency contact numbersHave emergency contact numbers

• Post-critiquePost-critique

Lessons Learned - FacilitiesLessons Learned - Facilities

• Get to know your Facilities Get to know your Facilities departmentdepartment

Lessons Learned - FacilitiesLessons Learned - Facilities

• Get to know your Facilities Get to know your Facilities departmentdepartment

• Establish liaisons with field/branch Establish liaisons with field/branch officesoffices

Lessons Learned - FacilitiesLessons Learned - Facilities

• Get to know your Facilities Get to know your Facilities departmentdepartment

• Establish liaisons with field/branch Establish liaisons with field/branch officesoffices

• Limit employees’ personal itemsLimit employees’ personal items

Lessons Learned - FacilitiesLessons Learned - Facilities

• Get to know your Facilities Get to know your Facilities departmentdepartment

• Establish liaisons with field/branch Establish liaisons with field/branch officesoffices

• Limit employees’ personal itemsLimit employees’ personal items

• No locked file cabinets or overheadsNo locked file cabinets or overheads

Lessons Learned - FacilitiesLessons Learned - Facilities

• Get to know your Facilities Get to know your Facilities departmentdepartment

• Establish liaisons with field/branch Establish liaisons with field/branch officesoffices

• Limit employees’ personal itemsLimit employees’ personal items

• No locked file cabinets or overheadsNo locked file cabinets or overheads

• Take office keys with youTake office keys with you

Lessons Learned - FacilitiesLessons Learned - Facilities

• Get to know your Facilities departmentGet to know your Facilities department

• Establish liaisons with field/branch Establish liaisons with field/branch officesoffices

• Limit employees’ personal itemsLimit employees’ personal items

• No locked file cabinets or overheadsNo locked file cabinets or overheads

• Take office keys with youTake office keys with you

• No critical or only copy records in No critical or only copy records in disaster susceptible areasdisaster susceptible areas

Lessons Learned - PersonnelLessons Learned - Personnel

• Storm procedures not heededStorm procedures not heeded

Lessons Learned - PersonnelLessons Learned - Personnel

• Storm procedures not heededStorm procedures not heeded

• Records preparation should be part Records preparation should be part of storm preparation and of storm preparation and communicationscommunications

Lessons Learned - PersonnelLessons Learned - Personnel

• Storm procedures not heededStorm procedures not heeded

• Records preparation should be part Records preparation should be part of storm preparation and of storm preparation and communicationscommunications

• Files should be labeled / identifiedFiles should be labeled / identified

Lessons Learned - PersonnelLessons Learned - Personnel

• Storm procedures not heededStorm procedures not heeded

• Records preparation should be part Records preparation should be part of storm preparation and of storm preparation and communicationscommunications

• Files should be labeled / identifiedFiles should be labeled / identified

• Enforce corporate policiesEnforce corporate policies

Lessons Learned - PersonnelLessons Learned - Personnel

• Storm procedures not heededStorm procedures not heeded

• Records preparation should be part Records preparation should be part of storm preparation and of storm preparation and communicationscommunications

• Files should be labeled / identifiedFiles should be labeled / identified

• Enforce corporate policiesEnforce corporate policies

• People are stressedPeople are stressed

Lessons Learned - PersonnelLessons Learned - Personnel

• Storm procedures not heededStorm procedures not heeded

• Records preparation should be part of Records preparation should be part of storm preparation and communicationsstorm preparation and communications

• Files should be labeled / identifiedFiles should be labeled / identified

• Enforce corporate policiesEnforce corporate policies

• People are stressedPeople are stressed

• Promote disaster preparedness and vital Promote disaster preparedness and vital records protectionrecords protection

Silver LiningsSilver Linings

• Files were cleaned outFiles were cleaned out– Inactive records sent to storageInactive records sent to storage– Destruction of expired and unnecessary Destruction of expired and unnecessary

recordsrecords– Reduction in file volumeReduction in file volume

Silver LiningsSilver Linings

• Files were cleaned outFiles were cleaned out– Inactive records sent to storageInactive records sent to storage– Destruction of expired and unnecessary Destruction of expired and unnecessary

recordsrecords– Reduction in file volumeReduction in file volume

• New contacts with business unitsNew contacts with business units

Silver LiningsSilver Linings

• Files were cleaned outFiles were cleaned out– Inactive records sent to storageInactive records sent to storage– Destruction of expired and unnecessary Destruction of expired and unnecessary

recordsrecords– Reduction in file volumeReduction in file volume

• New contacts with business unitsNew contacts with business units

• Multiple opportunities to promote Multiple opportunities to promote RIMRIM

Silver LiningsSilver Linings

• Files were cleaned outFiles were cleaned out– Inactive records sent to storageInactive records sent to storage– Destruction of expired and unnecessary Destruction of expired and unnecessary

recordsrecords– Reduction in file volumeReduction in file volume

• New contacts with business unitsNew contacts with business units• Multiple opportunities to promote RIMMultiple opportunities to promote RIM• Department credibility enhancedDepartment credibility enhanced

Silver LiningsSilver Linings

• Disaster planning for records now Disaster planning for records now prioritypriority

Silver LiningsSilver Linings

• Disaster planning for records now Disaster planning for records now prioritypriority

• Records protection procedures part Records protection procedures part of storm manualof storm manual

Silver LiningsSilver Linings

• Disaster planning for records now Disaster planning for records now prioritypriority

• Records protection procedures part Records protection procedures part of storm manualof storm manual

• No people – or records – in Lower No people – or records – in Lower LevelLevel

Silver LiningsSilver Linings

• Disaster planning for records now Disaster planning for records now prioritypriority

• Records protection procedures part of Records protection procedures part of storm manualstorm manual

• No people – or records – in Lower No people – or records – in Lower LevelLevel

• EOC and critical functions moved from EOC and critical functions moved from coastal zonecoastal zone

Silver LiningsSilver Linings

• RIM storm assignment changedRIM storm assignment changed

Silver LiningsSilver Linings

• RIM storm assignment changedRIM storm assignment changed

• Employees will not be complacent Employees will not be complacent about next stormabout next storm

ConclusionsConclusions

• Be proactive vs. reactiveBe proactive vs. reactive

ConclusionsConclusions

• Be proactive vs. reactiveBe proactive vs. reactive

• Good sense, sound judgment and Good sense, sound judgment and creativity are necessarycreativity are necessary

ConclusionsConclusions

• Be proactive vs. reactiveBe proactive vs. reactive

• Good sense, sound judgment and Good sense, sound judgment and creativity are necessarycreativity are necessary

• Don’t be complacentDon’t be complacent

ConclusionsConclusions

• Be proactive vs. reactiveBe proactive vs. reactive

• Good sense, sound judgment and Good sense, sound judgment and creativity are necessarycreativity are necessary

• Don’t be complacentDon’t be complacent

• Expect the unexpectedExpect the unexpected

ConclusionsConclusions

• Be proactive vs. reactiveBe proactive vs. reactive

• Good sense, sound judgment and Good sense, sound judgment and creativity are necessarycreativity are necessary

• Don’t be complacentDon’t be complacent

• Expect the unexpectedExpect the unexpected

• Evaluate suppliers and servicesEvaluate suppliers and services

ConclusionsConclusions

• Be proactive vs. reactiveBe proactive vs. reactive

• Good sense, sound judgment and Good sense, sound judgment and creativity are necessarycreativity are necessary

• Don’t be complacentDon’t be complacent

• Expect the unexpectedExpect the unexpected

• Evaluate suppliers and servicesEvaluate suppliers and services

• Empower peopleEmpower people

Questions?Questions?

Gail Ann McCreary, CRMGail Ann McCreary, CRM228-865-5467 228-865-5467 gamccrea@southernco.comgamccrea@southernco.com