CONTENTS
Background
Objectives and Scope
Approach and Methodology
Findings and Recommendations
Data Processing Item Processing Data Communications
Summary of Recommendations
Background (cont’d)
BIA recommended that:
• Deposit processing be normalized within 3 days.
• Loan and other key operations be normalized within
8-10 days.
• Other production operations be normalized within 14 days.
Background
Business impact Analysis (BIA) concluded that without Mainframe Data Processing —
• Depositor Base would start to erode at Day 3 after disaster because of customer frustration caused by: — lengthened service queues — degraded service and Information availability
• Reduced deposit levels would increase cost of funds by 200 to 250 basis points for the eroded deposit amounts. — cost= $700 million over 10 years
• The difficulty of manual operations/controls and lack of current balances would: — paralyze deposit operations before two weeks — accelerate rate of depositor erosion.
Background (cont’d)
BIA Conclusions (cont’d) • Loan imitation activity could continue for a month at reduced
levels of efficiency. Where current balance information is required, loan operations would break down more rapidly: — portfolio packaging — loan payoffs — servicing
• All accounting operations would stop, except for manual processing in problem areas.
• After two weeks, dunning activities would stop. • Estimated losses from an outage are estimated at:
— 3-day outage—nominal — 2-week outage — $25,000,000 — 2-month outage—Association Is no longer a viable economic
entity
Objectives and Scope of Strategy
Objectives of Disaster Recovery Strategy*
• Determine alternative emergency compute operations scenarios that would permit bank to fulfill Business Impact Analysis (BIA) recommendations.
• Assist bank management in selecting the alternative emergency computer operations approach that best permits rapid and effective Implementation.
• Prepare an Implementation Plan for the selected emergency computer operations approach.
* Disaster Recovery Project Task Ill. A deliverable.
Approach and Methodology
• Developed jointly with bank team feasible alternative scenarios that meet BIA recommendations for emergency computer operations and telecommunications after a disaster.
• Assisted bank team in selecting a strategy that effectively and efficiently permits rapid implementation of contingency processing capability.
• Prepared Draft Disaster Recovery Strategy Report and reviewed with Project Team.
Findings: Data Processing Alternatives
Available Data Processina backup resource alternatives include:
Meets BIA Requirements A. Dual Data Centers
- Capable of resuming processing within 1-2 shifts
B. Vendor provided hot/cold site
- Capable of resuming deposit processing within 3-4 days
Does Not Meet BIA Requirements
C. Vendor provided cold site
- Capable of resuming processing within 3-4 weeks
D. Identified/Unprepared site
- Capable of resuming processing within 6-8 weeks
Findings: Data Processing Alternatives
Alternatives that Meet BIA Requirements I. Dual Data Centers (California and Florida)
Pro: - Permit resumption of Deposit and Loan processing within 1 day and other production processing within 2 weeks
- Complete protection against local and area-wide disaster
Con: - Additional costs can run In the area of millions of dollars per year
- Requires 3-5 years to Implement
II. Vendor Hot/Cold Site
Pro: - Permit resumption of Deposit processing within 3 days, loan processing within 7 days and other processing within 2 weeks
- Complete protection against local and partial protection against area-wide disaster
Con: - Additional costs can run in the area of millions of dollars per year
- Significant exposure In event of major area-wide disaster* (e.g. 8.0 on San Andreas near LA)
*Sungard has 3 and Comdisco has 5 feasible sites
Findings: Data Processing Alternatives (cont’d)
Data Processing Alternatives that meet BIA Requirements A. Dual Data Center (California & Florida)
• Capable of restoring critical deposit and loan processing within a shift
• Capable of restoring other production processing within two weeks.
Dual Data Centers with Electronic Archiving
Findings: Data Processing Alternatives (cont’d)
Dual Data Center (continued) • Normally cost justified because of growing East Coast business
base and requirement for complete protection from area-wide disaster*
• Processing architecture typically involves: — West Coast production and development — East Coast production and cold site
• A long-term alternative — 2-3 years to Implement — Additional millions in annual cost In redundant operations
*An 8.0 earthquake on San Andreas near Los Angeles
Findings: Data Processing Alternatives (cont’d)
B. Vendor Provided Hot/Cold Site • Capable of restoring critical processing within 3-4 days after a
localized disaster (assumes fully tested recovery plan, and access to backup data files and personnel)
Hot Site Restoration Schedule- Most Likely
Findings: Data Processing Alternatives (cont’d)
Vendor Hot/Cold Site (cont’d)• Capable of moving all production processing to the Cold Site
from the Hot Site within 2 weeks after a localized disaster
Cold Site ActivatIon Schedule
Vendor Hot/Cold Site (cont’d)
• Complete protection for localized disaster (fire, bomb, sabotage, small/close earthquake*)
• Partial protection from area -wide disaster - multiple vendor clients in northern LA basin- vendor sites (3 for Sunguard and 5 for Comdisco) are
allocated on a first come/first served basis with rapid response, would probably get a 3090-200, but might not get the second 3090-200 needed to catch up loan system
• A short-term and/or long-term alternative - 90-120 days to Implement - annual cost of $
* A 6.0 earthquake on Verdugo Fault
Findings: Data Processing Alternatives (cont’d)
Alternatives That Do Not Meet BIA Requirements
C. Vendor Provided Cold Site • Capable of resuming processing within 3 weeks • Small annual cost but no capability to test
D. Identified/Unprepared Site • Capable of resuming processing within 6-8 weeks.
• Little initial or annual costs.
Findings: Cost of Data Processing Alternatives
Annual Outlays(preliminary)
Data Center Facility Data Center OperationsComputer: 3090-200
3090-400 Extensions Remote
PrintingOperations Staff Testing *
TOTAL
Owned Dual Vender
Centers Hot/Cold Site
$ 0
$ 0
$ 0
$
$
$ 0
0 $
$ $
Recommendations: Data processing
Utilization of Vendor Provided Hot/Cold Site is Recommended
• Only feasible short term alternative
• Start negotiations with Comdisco with intent to sign contract by April 1, 1988
• Start preparation of Disaster Recovery Plan and Initial Testing immediately to gain protection from disaster within 90-120 days
• Level of Protection provided to operations outside area affected by disaster
Localized Disaster Good
Area Wide Disaster Fair
(may have limited access to backup data and personnel)
Findings: Item Processing Alternatives
Available Item Processing Backup Approaches:
• Local Mutual Aid Arrangement with another Los Angeles area Financial Institution
— Typical 12 hour delay in capture processing
— Good protection from localized disasters
• Dual Bank Capture Centers Backup Arrangement (Florida backs up Los Angeles and Los Angeles Backs up Florida)
— Typical 24 hour delay In capture processing
— Good protection from Area-Wide disaster
Findings: Item Processing Alternatives
Local Mutual Aid Arrangement
• Several financial institutions have available time on compatible systems
• Utilize time during off-shift and weekends
• Protection level: Localized Disaster Good
Area-Wide Disaster Poor
• Minimal annual costs for administration & testing
Findings: Item Processing Alternatives
California & Florida Capture Centers Backup Each Other
• Both centers have capacity to backup each other
• Vendor Hot/Cold Site has 1 channel extender which will support 1 sorter
• Fed would forward all items to alternate capture center causing a 24 hour delay
• Protection Level:
Localized Disaster Good
Area-Wide Disaster Good
• Cost Estimates:Full Service Level (2 channel extenders) $ initial cost
plus $ per year
Reduced Service Level (1 channel extender) $ per year
Recommendations: Item Processing
Utilize both alternatives: • Mutual Aid
- For a Local disaster- Mutual Aid provides full service levels with 12 hour delay
• Dual Capture Centers with Reduced Service Levels- For an Area-Wide disaster- Dual Centers providing partial
service levels with a 2448 hour delay
- Annual Fee $
Findings: Data Communication Alternatives
Available data Communication Alternatives Include:
I. Line switching (passive links) from Los Angeles PacBell office to Cypress Recovery Center and from Northern California to San Ramon Center. Switch 56 connection of Florida to hot site.
• Poor protection from Local Area Disaster, since LA PacBell office 12 miles from Burbank Data Center
• Significant monthly cost for contingency capability
• 90-120 days to Implement
Il. Line switching (passive links) from AT&T LATA offices to San Ramon Recovery Center. Switch 56 connection of Florida to hot site
• Major monthly cost for contingency capability
• Individual lines subject to outage from damage to any TELCO facility utilized by the Line
• 90-120 days to implement
Findings: Data Communication Alternatives(cont’d)
I. Line Switching to Cypress and Sam Ramon Recovery Centers
• Switch Southern California lines to Cypress Backup Center from LA TELCO offices and Northern California Lines to the San Ramon Backup Center
• Significant monthly cost for Contingency Capability
• Protection of Operations Outside Area Affected by Disaster
Localized Outage Good
Area-Wide Outage Poor (since LA Pac Bell Office only 12 miles from Burbank Data
Center)
• Implementation Requirements 90-120 days
Findings: Data Communication Alternatives(cont’d)
II. Line Switching to San Ramon Recovery center Only
• Switch Northern and Southern California lines to San Ramon backup center
• Major monthly cost for contingency capability
• Protection of Operations Outside Area Affected by Disaster
Localized Outage Good Area-Wide Outage Fair
(may have limited access to backup data and to personnel)
• Implementation Requirements 90-120 days
Findings: Data Communication Service Levels
Available Data Communication Service Levels:
A.Full Service
• ability to activate all multidrop branch support lines through line switching within 24 hours
• Ability to activate administrative point4o-point lines with critical application requirements through Dial Backup within 24 hours.
• Ability to activate ATM lines and non-critical administrative lines through emergency ordering within approximately 10-15 days
B. Partial Service
• Ability to activate approximately 50% of branch support and critical administrative support lines within 24 hours
• Ability to activate remaining lines through emergency ordering within approximately 10-15 days
FINDINGS: DATA COMMUNICATION ALTERNATIVES COSTS
Estimated Annualized Costs (5 Year Write-off)
A: Full B: Partial
Service Service
I. Line switching of
NC to San Ramon,
SC to Cypress, and
Florida to Hot Site $ $
II. Line switching of NC 8 SC to San Ramon, and
Florida to Hot Sit. $ $
RECOMMENDATIONS: DATA COMMUNICATIONS
As a Short Term Solution:
• Partial Service Level and line switching to both Cypress and San Ramon Is a cost/effective temporary strategy
• On a permanent basis a revised network architecture can offer improved service levels and Improved level of protection from area-wide disasters.
For A Long Term Solution:
• Evaluate as a separate business case alternatives such as:
- redesigned network using concentrators
- reconfigured network to better distribute immediately supported locations
Summary of Recommendations
Data Processing
• Implement vendor provided Hot/Cold site with Initial test of processing capability during May, 1988
Item Processing
• Implement Both Mutual Aid and Dual Data Capture Center approaches with initial test of alternate capture capability during May, 1988
Data Communications
• Implement, over the short term, Partial Service Level and Local Disaster Protection approach with initial test during July, 1988
Impact of Recommendations
Protection Level of Recommended Strategies
Localized Area-Wide Disaster DisasterData Processing Good Fair Item Processing Good Good Data Communications Good Fair Overall Good Fair
Service Level of Recommended Strategies to Operations Outside Area of Disaster
Localized Area-Wide Disaster Disaster Data Processing Good Fair Item Processing Good Fair Data Communications Fair* Fair Overall Fair* Fair
* Long Term Network redesign of configuration can raise this level