comdisco confidential
DESCRIPTION
WORLD TRADE CENTER And Related Recoveries Len Guddemi Vice President Northeast Operations. Comdisco Confidential. SunGard Worldwide Disaster Support Experience. Recoveries per Year. Environmental 8/2%. Software 10/2%. Telecomm 14/3%. Terrorist Attack 100/19%. Bomb 16/3%. - PowerPoint PPT PresentationTRANSCRIPT
WORLD TRADE CENTERAnd Related Recoveries
Len GuddemiVice PresidentNortheast Operations
Comdisco Confidential
0
20
40
60
80
100
'81 '82 '83 '84 '85 '86 '87 '88 '89 '90 '91 '92 '93 '94 '95 '96 '97 '98 '99 '00 '01
SunGard Worldwide Disaster Support Experience
Recoveries per Year
Terrorist Attack 100/19%
Hardware 97/18%
Hurricane 84/16%Power Outage 76/14%
Flood 57/11%
Fire/Explosion 27/5%
Miscellaneous 20/4%
Earthquake 19/4%
Bomb 16/3%
Telecomm 14/3%
Software 10/2%
Environmental 8/2%
Data Center
7/1%
Civil Unrest 2/0%
Disruption Cause: Number/Percentage
SunGard Worldwide Disaster Support Experience
DEC 43/8%
AS/400 56/10%
Sun 16/3%
Item Processing 5/1%
NT Server 4/1%
Trading Floor 19/4%
Advanced Recovery 13/2%
Miscellaneous 26/5%
HP 22/4%
Workarea Recovery 130/24%
Mainframe 138/26%
RS/6000 10/2%
Unisys 2/0%
Mobile Cluster 3/1%
Systems On-site Mobile 39/7%
Tandem 3/1%
Shell 7/1%
Stratus 1/0%
SunGard Worldwide Disaster Support Experience
Recovery Platforms Used: Quantity/Percentage
Terrorist Attack 1154/20%
Flood 691/12%
Hardware Problem 666/12%
Power Outage 370/7%
Miscellaneous 858/15%
Fire/Explosion 359/6%
Hurricane 318/6%
Data Center Move 269/5%
Earthquake 136/2%
Bomb 146/3%Environmental 573/10%
Civil Unrest 3/0%
Telecomm 30/1%
SunGard Worldwide Disaster Support Experience
Days of Occupancy: Days/Percentage
Declaration Information
Number of Events
Overall - 93 Schedules, 47 Customers
Simultaneous High Point - 76 Declarations, 16 Facilities, 14 Platforms
Comdisco Confidential
Reason
Facilities Unavailable
Loss of Power, telecommunications, denied access to buildings,
concerns over building structural safety
Atlanta: Work Area Seats, Remote Sun,
AS400, RS6000
Carlstadt: Voice/Work Area, Business
Recovery/Quick Ship, Trading Floor,
Dedicated Seats, SUN, AS400,
DEC/VAX, RS6000, HP9000, Intel
Based Servers; Dedicated HP9000,
RS6000, NT Servers
Boston: SUN, Work Area
Cypress: Work Area, Remote Tandem
Mobile Recovery: 13 Units/5,000sq.ft., 10
Units/3,000sq.ft. for Work Area
Minnetonka: Work Area
Comdisco Confidential
North Bergen: Voice/Work Area,
Main-Frame, ARS & Networking
Phoenix: Work Area
Queens: Work Area, Trading Floor,
Dedicated, Trading PIT
Rosemont: Work Area
San Ramon: Work Area, Trading Floor
Wood Dale: DEC/VAX, DEC/Alpha,
VAX3100, VAX7620, Filenet, AS400,
Tandem
Wood Cliff Lake: Intel Servers, Work Area
Jersey City, workgroup, AS/400
Philadelphia: Work area, RS6000,
AS/400,Mainframe, Voice
Herndon: Work Area
Warminster: Work Area,Unisys
SunGard Sites & Platforms
Declaration Information - Continued
Under-subscription or additional needs assessment remains an issue Many businesses requested additional needs ATOD
Responded to over 500 special requests for service. Built out with best efforts
This puts recovery & business continuity at risk due to extended recovery timeline.
Personnel & Intellectual Property Keys to Recovery Methodologies, ideas and documented plans utilized. Assigned backup roles and succession planning implemented. Intellectual capital and personnel in alternate locations. Use of professional organizations to augment staff.
Comdisco Confidential
Items of Note Implement Your Plan & Associated Vendor Services with Urgency
14 Declarations in 30 minutes; 24 within 1 hour; 19 in 2nd hour You will be in competition with other business for services & manufacturers
Quickly establish damage assessment & need for services, and implement
Declaration Information - Continued
Air Travel Unavailable Delayed delivery of offsite stored media to recovery location
Identify alternate means of transportation Assess need for electronic journaling of data to alternate location
Expert staff unable to travel Identify options for local access by staff to remote alternate systems (and
possibly home systems) Assess value of ‘turnkey’ type services with vendors
Offsite Storage Critical Evaluate for frequency, completeness Paper records and transactions may be totally lost Evaluate if your offsite storage is far enough away Evaluate need/value of electronic journaling
Comdisco Confidential
Items of Note - Continued
Reluctance to Travel End Users generally won’t travel far from home.
Ensure adequate end user facilities available nearby (about 50 miles)
Declaration Information - Continued
Active Testers More successful at meeting RTO/RPO More realistic understanding of application performance Fewer problems
Very High Demand for Voice Recovery Realistically evaluate needs: line trunking, extensions, call recording, voice
mail, call forwarding Avoid Fast Busy
Comdisco Confidential
Items of Note - Continued High Availability Solutions Extremely Successful
Electronic Journaling, Data Base Mirroring, SRDF, etc. 45 Terabytes ready very quickly Data requiring physical transport delayed
SunGard Challenges & Tactics Disaster Hotline & Notification
Increased Staff handling incoming Disaster calls
Security Immediate increase in security staff on all shifts. Contacted local authorities to assist in access control
Additional Staff Request for additional support staff Mobil Recovery Team
Additional Resource Contacted third party vendors Arranged additional parking with neighbors
SunGard Challenges & Tactics
Implemented Swat Support Teams Established 24/7 Command Center at Corporate HQ with Open Bridge SunGard Canadian employees, Illinois, Minneapolis, and Texas employees
drove to Carlstadt & North Bergen to help SunGard’s Professional Services Consultants, Sales, Sales Support,
Administration and SunGard’s Operations staff worked side by side. Dispatched relief support from other SunGard facilities once air travel restored Activated facilities, equipment procurement, shipping and solutions teams to
respond to special requests Received offers of help from customers
Gratefully declined Network Alternate Routing Critical
Significant damage due to the event Fiber cut after event by construction/demolition crews SunGard alternate routings worked without issue Vendors terrific with T1s, DS3 installations in days
Comdisco Confidential
SunGard Facilities and Staff not directly affected
Customer Challenges & Tactics
Comdisco Confidential
Inaccurate Needs Assessment
Special requests for PCs, & Servers IP addressing Storage Tape Drives Electricity Processing power LAN bridging Televisions & Printers Network capability, Voice services & Routers Faxes Software and more
Customer Challenges & Tactics Logistics
Travel Restriction an Impact - used ground travel for many shipments Shipped over 4,200 PCs along with much other equipment Customer parking, bussing, and catering
Comdisco Confidential
Getting Restoration Media to Alternate Site Transportation challenge Complete set available and labeled Missing or unavailable documentation
Start-up scriptsRun instructions
Learned / Re-Affirmed
Important to Have a Team Leader for Each Shift
Integration of recovery requirements into change control Number of PCs, Positions Network Capabilities Processors, disk, tape capabilities Printers, fax machines PBX features & functionality Platform Technology upgrades, changes, maintenance
Importance of Quick Assessment & Implementation of Plan when appropriate
Comdisco Confidential
Value in Pre-Planning Staff Placement within Rooms End User priorities & usage End User equipment requirements – avoids setting up more than once
Learned / Re-Affirmed - Continued Value of Solid Backup Records Program
Tape, electronic, & paper records foundation of recovery Evaluate right balance for your company
Documented & labeled so vendor(s) can assist Have scripts shipped with restore media or stored at alternate location Frequency of backups Offsite location & transportation dependencies Examine benefit of electronic transmission of offsite storage data
Can significantly improve recovery time and reduce data loss
Comdisco Confidential
Evaluate Strategy of Shared versus Dedicated Resources Guarantees resource access for some number of most critical requirements
Complement with shared resources
PC Image Restoration Tool Effective Ghost, or similar
Learned / Re-Affirmed - Continued Basics are Fundamental
Establish personnel roles & assign backup
Document methodologies, ideas, business plans & recovery plans
Review, update & follow established procedures
Test the way you would recover, recover the way you test
Worthwhile to review how home procedures apply in alternate site production
mode
Offsite Storage, Disk mirroring, Miscellaneous other
Comdisco Confidential
Revisit your Plan & Recovery Requirements Often – Technology & Business
In Times of Need, Help Comes From Many Sources
Customer Comments
Comdisco Confidential
“Although operating in our primary location is preferred, the equipment, professional and courteous staff, and outstanding overall accommodations provided for an effective and successful business recovery. I am glad that we have contracted with your organization!”
“I have in the past worked with a number of outstanding individuals, but never before have I worked with such a highly functional, customer service oriented team. Individually each team member I have had the pleasure to deal with was exceptional; as a team there is none better.”
Customer Comments
“Thanks for all the support and assistance. We’re glad you are here.”
Comdisco Confidential
“I would like you to know of the "more than wonderful" service we are receiving at the SunGard site. ... have gone out of their way to assist us with some special needs. ... have answered numerous questions and assisted --she is great. ... worked with ENS last night, assisted me with set up so I can stay on line (due to the situation in New York and the weather closing down our site) and has been a great help to the IT staff.”
“I would like you to know it is people like this that make SunGard a vendor we choose to depend on in the event of an emergency. Please pass this message along to their managers, it would be greatly appreciated.”
Questions and Answer
Comdisco Confidential