oakville hydro business continuity edist 2016 · disaster recovery plan (drp) vs business...
TRANSCRIPT
![Page 1: Oakville Hydro Business Continuity Edist 2016 · Disaster Recovery Plan (DRP) vs Business Continuity Plan (BCP) Information Technology: Ice Storm Response](https://reader035.vdocuments.us/reader035/viewer/2022070612/5b3669c97f8b9a7e4b8e58e6/html5/thumbnails/1.jpg)
Oakville Hydro January 14, 2016
BUSINESS CONTINUITY
![Page 2: Oakville Hydro Business Continuity Edist 2016 · Disaster Recovery Plan (DRP) vs Business Continuity Plan (BCP) Information Technology: Ice Storm Response](https://reader035.vdocuments.us/reader035/viewer/2022070612/5b3669c97f8b9a7e4b8e58e6/html5/thumbnails/2.jpg)
Oakville Enterprises Corporation
![Page 3: Oakville Hydro Business Continuity Edist 2016 · Disaster Recovery Plan (DRP) vs Business Continuity Plan (BCP) Information Technology: Ice Storm Response](https://reader035.vdocuments.us/reader035/viewer/2022070612/5b3669c97f8b9a7e4b8e58e6/html5/thumbnails/3.jpg)
Oakville Hydro Electricity Distribution
13th largest electricity distributor in Ontario
OUR ASSETS Total asset base of $230 million
Distribution system consisting of
o Underground cable: 1,348 km
o Overhead circuit wires: 486 km
OUR CUSTOMERS 67,821
OUR SERVICE AREA 143 square km
![Page 4: Oakville Hydro Business Continuity Edist 2016 · Disaster Recovery Plan (DRP) vs Business Continuity Plan (BCP) Information Technology: Ice Storm Response](https://reader035.vdocuments.us/reader035/viewer/2022070612/5b3669c97f8b9a7e4b8e58e6/html5/thumbnails/4.jpg)
Business Continuity vs Emergency Operation Plan 2013 Ice Storm Oakville Hydro’s Initiatives “Mock Disaster” of the Emergency Operations Plan “Casper Test” of the Business Continuity Plan Test
Lessons Learned Conclusions
Outline
![Page 5: Oakville Hydro Business Continuity Edist 2016 · Disaster Recovery Plan (DRP) vs Business Continuity Plan (BCP) Information Technology: Ice Storm Response](https://reader035.vdocuments.us/reader035/viewer/2022070612/5b3669c97f8b9a7e4b8e58e6/html5/thumbnails/5.jpg)
Business Continuity Plans at OEC
Emergency Operation
Plan
IT DRP
Pandemic Plan
Emergency Communication
Plan Customer Service
Plan Affiliate
Emergency Plans
![Page 6: Oakville Hydro Business Continuity Edist 2016 · Disaster Recovery Plan (DRP) vs Business Continuity Plan (BCP) Information Technology: Ice Storm Response](https://reader035.vdocuments.us/reader035/viewer/2022070612/5b3669c97f8b9a7e4b8e58e6/html5/thumbnails/6.jpg)
Background
![Page 7: Oakville Hydro Business Continuity Edist 2016 · Disaster Recovery Plan (DRP) vs Business Continuity Plan (BCP) Information Technology: Ice Storm Response](https://reader035.vdocuments.us/reader035/viewer/2022070612/5b3669c97f8b9a7e4b8e58e6/html5/thumbnails/7.jpg)
Timeline of Events
2015
EOP Review
Q4
Impact Analysis & Mitigation
Casper Test
2014 2013 Q1 Q3 Q1 Q2 Q3 Q4 Q1 Q2
Ongoing Business Continuity Program
2016
Ice Storm
Q2 Q4
Mock Disaster
![Page 8: Oakville Hydro Business Continuity Edist 2016 · Disaster Recovery Plan (DRP) vs Business Continuity Plan (BCP) Information Technology: Ice Storm Response](https://reader035.vdocuments.us/reader035/viewer/2022070612/5b3669c97f8b9a7e4b8e58e6/html5/thumbnails/8.jpg)
Potential Business Interruptions
Enterprise Critical Functions
Vulnerability? Interruption Tolerance?
Impacts
Mitigate Probability Mitigate Impact
Provide for Interruption
Emergency Incident
Response & Recovery Emergency Command
Communications
Continuous Enterprise
Risk Management
Risks
Tracking
BCP Framework Implementation
![Page 9: Oakville Hydro Business Continuity Edist 2016 · Disaster Recovery Plan (DRP) vs Business Continuity Plan (BCP) Information Technology: Ice Storm Response](https://reader035.vdocuments.us/reader035/viewer/2022070612/5b3669c97f8b9a7e4b8e58e6/html5/thumbnails/9.jpg)
First Phase: The Mock Disaster
Scope: major ice storm increased customer calls loss of power from the Grid loss of feeder to office and control room
Mitigations to be tested – prepared simulations Departments’ preparations including I.T. support Test performed June 2015
![Page 10: Oakville Hydro Business Continuity Edist 2016 · Disaster Recovery Plan (DRP) vs Business Continuity Plan (BCP) Information Technology: Ice Storm Response](https://reader035.vdocuments.us/reader035/viewer/2022070612/5b3669c97f8b9a7e4b8e58e6/html5/thumbnails/10.jpg)
Second Phase: The Casper Event
Why a Second test?
Major Interruption: Building unavailable
What did we want to achieve out of the test? Test Performed October 2015
![Page 11: Oakville Hydro Business Continuity Edist 2016 · Disaster Recovery Plan (DRP) vs Business Continuity Plan (BCP) Information Technology: Ice Storm Response](https://reader035.vdocuments.us/reader035/viewer/2022070612/5b3669c97f8b9a7e4b8e58e6/html5/thumbnails/11.jpg)
Casper Event
![Page 12: Oakville Hydro Business Continuity Edist 2016 · Disaster Recovery Plan (DRP) vs Business Continuity Plan (BCP) Information Technology: Ice Storm Response](https://reader035.vdocuments.us/reader035/viewer/2022070612/5b3669c97f8b9a7e4b8e58e6/html5/thumbnails/12.jpg)
Distribution Operations
Consists of: Line Operations
Maintenance
Construction
Warehouse
![Page 13: Oakville Hydro Business Continuity Edist 2016 · Disaster Recovery Plan (DRP) vs Business Continuity Plan (BCP) Information Technology: Ice Storm Response](https://reader035.vdocuments.us/reader035/viewer/2022070612/5b3669c97f8b9a7e4b8e58e6/html5/thumbnails/13.jpg)
Distribution Operations
![Page 14: Oakville Hydro Business Continuity Edist 2016 · Disaster Recovery Plan (DRP) vs Business Continuity Plan (BCP) Information Technology: Ice Storm Response](https://reader035.vdocuments.us/reader035/viewer/2022070612/5b3669c97f8b9a7e4b8e58e6/html5/thumbnails/14.jpg)
Operations: Ice Storm Response
Our Experience of the Ice Storm Customers Feedback was lost High volume of customer calls Met customer needs but took a large concerted effort
Used scouts to locate the worst hit areas
Constantly prioritized work – in a reactive nature
Paper based planning and reporting
![Page 15: Oakville Hydro Business Continuity Edist 2016 · Disaster Recovery Plan (DRP) vs Business Continuity Plan (BCP) Information Technology: Ice Storm Response](https://reader035.vdocuments.us/reader035/viewer/2022070612/5b3669c97f8b9a7e4b8e58e6/html5/thumbnails/15.jpg)
Operations: Mock Disaster
The Strategy Leveraged the work order system
Moved away from paper
Trained staff in new process
Provide more accurate Estimated Time of Restoration
(ETR)
![Page 16: Oakville Hydro Business Continuity Edist 2016 · Disaster Recovery Plan (DRP) vs Business Continuity Plan (BCP) Information Technology: Ice Storm Response](https://reader035.vdocuments.us/reader035/viewer/2022070612/5b3669c97f8b9a7e4b8e58e6/html5/thumbnails/16.jpg)
![Page 17: Oakville Hydro Business Continuity Edist 2016 · Disaster Recovery Plan (DRP) vs Business Continuity Plan (BCP) Information Technology: Ice Storm Response](https://reader035.vdocuments.us/reader035/viewer/2022070612/5b3669c97f8b9a7e4b8e58e6/html5/thumbnails/17.jpg)
Operations: Casper Event
An event with significant preparation Challenges Find locations for: vehicles equipment material
Relocation of critical staff to “Hotel”
![Page 18: Oakville Hydro Business Continuity Edist 2016 · Disaster Recovery Plan (DRP) vs Business Continuity Plan (BCP) Information Technology: Ice Storm Response](https://reader035.vdocuments.us/reader035/viewer/2022070612/5b3669c97f8b9a7e4b8e58e6/html5/thumbnails/18.jpg)
![Page 19: Oakville Hydro Business Continuity Edist 2016 · Disaster Recovery Plan (DRP) vs Business Continuity Plan (BCP) Information Technology: Ice Storm Response](https://reader035.vdocuments.us/reader035/viewer/2022070612/5b3669c97f8b9a7e4b8e58e6/html5/thumbnails/19.jpg)
Operations: Casper Event
Lessons Learned Long preparation time Resource availability
Next Steps How can we shorten the lag Continually assess your tolerances for business
interruptions
![Page 20: Oakville Hydro Business Continuity Edist 2016 · Disaster Recovery Plan (DRP) vs Business Continuity Plan (BCP) Information Technology: Ice Storm Response](https://reader035.vdocuments.us/reader035/viewer/2022070612/5b3669c97f8b9a7e4b8e58e6/html5/thumbnails/20.jpg)
Engineering & Construction
Consists of: Planning, Design & Standards
Construction Management
GIS & Records Management
New Customer Connections
![Page 21: Oakville Hydro Business Continuity Edist 2016 · Disaster Recovery Plan (DRP) vs Business Continuity Plan (BCP) Information Technology: Ice Storm Response](https://reader035.vdocuments.us/reader035/viewer/2022070612/5b3669c97f8b9a7e4b8e58e6/html5/thumbnails/21.jpg)
Engineering: Ice Storm Response
All Staff Re-deployed to Emergency Operations
Prioritized Customer Calls
Managed Scout Deployment
Prepared Paper “Work Packages” for Operations team
Entered Trouble Calls into GIS
![Page 22: Oakville Hydro Business Continuity Edist 2016 · Disaster Recovery Plan (DRP) vs Business Continuity Plan (BCP) Information Technology: Ice Storm Response](https://reader035.vdocuments.us/reader035/viewer/2022070612/5b3669c97f8b9a7e4b8e58e6/html5/thumbnails/22.jpg)
Engineering: Mock Disaster
Customer Calls Logged in Service Manager Dispatched Electronically to Scouts Service Calls Logged Manually into GIS Scouts: Assigned To Geographic Zones Redeploy as Required Updated Calls on Laptop
![Page 23: Oakville Hydro Business Continuity Edist 2016 · Disaster Recovery Plan (DRP) vs Business Continuity Plan (BCP) Information Technology: Ice Storm Response](https://reader035.vdocuments.us/reader035/viewer/2022070612/5b3669c97f8b9a7e4b8e58e6/html5/thumbnails/23.jpg)
Service Manager Dispatch Board
![Page 24: Oakville Hydro Business Continuity Edist 2016 · Disaster Recovery Plan (DRP) vs Business Continuity Plan (BCP) Information Technology: Ice Storm Response](https://reader035.vdocuments.us/reader035/viewer/2022070612/5b3669c97f8b9a7e4b8e58e6/html5/thumbnails/24.jpg)
GIS Overall Showing Trouble Calls
![Page 25: Oakville Hydro Business Continuity Edist 2016 · Disaster Recovery Plan (DRP) vs Business Continuity Plan (BCP) Information Technology: Ice Storm Response](https://reader035.vdocuments.us/reader035/viewer/2022070612/5b3669c97f8b9a7e4b8e58e6/html5/thumbnails/25.jpg)
Home Office set up for Technical Staff
Mobile Office set up for Field Staff
Hoteling organized for critical Administrative Staff What About All of that Paper in the Office?
GIS/CAD – “The Need For Speed”
Engineering: Casper Event
![Page 26: Oakville Hydro Business Continuity Edist 2016 · Disaster Recovery Plan (DRP) vs Business Continuity Plan (BCP) Information Technology: Ice Storm Response](https://reader035.vdocuments.us/reader035/viewer/2022070612/5b3669c97f8b9a7e4b8e58e6/html5/thumbnails/26.jpg)
Engineering: Future Considerations
Automate Interface Between Service Manager & GIS
Accelerate current project – Paperless Filing System Enhance Mobile & Off Site Access Capabilities Cloud Solutions For Access to Information Secure Offsite Facilities for “Heavy Users” - GIS
![Page 27: Oakville Hydro Business Continuity Edist 2016 · Disaster Recovery Plan (DRP) vs Business Continuity Plan (BCP) Information Technology: Ice Storm Response](https://reader035.vdocuments.us/reader035/viewer/2022070612/5b3669c97f8b9a7e4b8e58e6/html5/thumbnails/27.jpg)
Customer Service
Customer Service: Providing our customers with a positive experience
when they interact with Oakville Hydro
Billing
Electricity and Water
Bi-monthly
![Page 28: Oakville Hydro Business Continuity Edist 2016 · Disaster Recovery Plan (DRP) vs Business Continuity Plan (BCP) Information Technology: Ice Storm Response](https://reader035.vdocuments.us/reader035/viewer/2022070612/5b3669c97f8b9a7e4b8e58e6/html5/thumbnails/28.jpg)
Customer Service: Ice Storm Response
Issues Identified Limited phone capacity – simultaneous calls Limitations to Call Centre capabilities – few
options Limitations with after hours service Logging calls into excel – need for a common
database
![Page 29: Oakville Hydro Business Continuity Edist 2016 · Disaster Recovery Plan (DRP) vs Business Continuity Plan (BCP) Information Technology: Ice Storm Response](https://reader035.vdocuments.us/reader035/viewer/2022070612/5b3669c97f8b9a7e4b8e58e6/html5/thumbnails/29.jpg)
Customer Service: Casper Event
Onsite – Staff available to serve customers
Offsite – Vendor Facility
Field Services
Home setup with Jabber software for phones
![Page 30: Oakville Hydro Business Continuity Edist 2016 · Disaster Recovery Plan (DRP) vs Business Continuity Plan (BCP) Information Technology: Ice Storm Response](https://reader035.vdocuments.us/reader035/viewer/2022070612/5b3669c97f8b9a7e4b8e58e6/html5/thumbnails/30.jpg)
Customer Service: Enhancements and Future Considerations
1. Call Centre - Simultaneous calls and flow
2. After Hours Service – Power Assist
3. Data base – Outage Management System, Great Plains
4. Social Media – Twitter and Facebook
5. Interactive Voice Response (IVR)
![Page 31: Oakville Hydro Business Continuity Edist 2016 · Disaster Recovery Plan (DRP) vs Business Continuity Plan (BCP) Information Technology: Ice Storm Response](https://reader035.vdocuments.us/reader035/viewer/2022070612/5b3669c97f8b9a7e4b8e58e6/html5/thumbnails/31.jpg)
Information Technology
IT Staff servicing Oakville Enterprises Corporation and responsible for: Infrastructure Core Business Applications Client services
Operation technology (OT)
Disaster Recovery Plan (DRP) vs Business Continuity Plan (BCP)
![Page 32: Oakville Hydro Business Continuity Edist 2016 · Disaster Recovery Plan (DRP) vs Business Continuity Plan (BCP) Information Technology: Ice Storm Response](https://reader035.vdocuments.us/reader035/viewer/2022070612/5b3669c97f8b9a7e4b8e58e6/html5/thumbnails/32.jpg)
Information Technology: Ice Storm Response
Building availability, systems intact
Providing access to data externally, VPN, Citrix was available
Challenges Web site, phone lines capacity Communications to the customers Field Mobility
![Page 33: Oakville Hydro Business Continuity Edist 2016 · Disaster Recovery Plan (DRP) vs Business Continuity Plan (BCP) Information Technology: Ice Storm Response](https://reader035.vdocuments.us/reader035/viewer/2022070612/5b3669c97f8b9a7e4b8e58e6/html5/thumbnails/33.jpg)
Information Technology: Mock Disaster
Mock Disaster Emergency Operating Plan (EOP) IT department fully engaged Innovation for new ways to provide for the business
needs Short term planning for next test
Facilities – Generator power requirements for IT
Infrastructure
![Page 34: Oakville Hydro Business Continuity Edist 2016 · Disaster Recovery Plan (DRP) vs Business Continuity Plan (BCP) Information Technology: Ice Storm Response](https://reader035.vdocuments.us/reader035/viewer/2022070612/5b3669c97f8b9a7e4b8e58e6/html5/thumbnails/34.jpg)
Information Technology: Casper Event New services Jabber – VPN access to the CS queues Support of OMS initiatives New Citrix environment New business applications for field work order management Upgrade of mobile devices Casper IT enhances its understanding of the expectations of the
business Business’s understanding of IT team support
![Page 35: Oakville Hydro Business Continuity Edist 2016 · Disaster Recovery Plan (DRP) vs Business Continuity Plan (BCP) Information Technology: Ice Storm Response](https://reader035.vdocuments.us/reader035/viewer/2022070612/5b3669c97f8b9a7e4b8e58e6/html5/thumbnails/35.jpg)
Information Technology: Future Considerations
Ongoing technical and system improvements
Building on Collaboration
Long term capital planning
![Page 36: Oakville Hydro Business Continuity Edist 2016 · Disaster Recovery Plan (DRP) vs Business Continuity Plan (BCP) Information Technology: Ice Storm Response](https://reader035.vdocuments.us/reader035/viewer/2022070612/5b3669c97f8b9a7e4b8e58e6/html5/thumbnails/36.jpg)
Lessons Learned
Modular Testing Formal Preparation Clear Objectives & Goals Proper Planning Debrief and learn
![Page 37: Oakville Hydro Business Continuity Edist 2016 · Disaster Recovery Plan (DRP) vs Business Continuity Plan (BCP) Information Technology: Ice Storm Response](https://reader035.vdocuments.us/reader035/viewer/2022070612/5b3669c97f8b9a7e4b8e58e6/html5/thumbnails/37.jpg)
Conclusions
Support from the leaders of the organization Significant commitment and effort required Goals and targets Reaching elevated levels of engagement and
collaboration Conducting “Test without warning” Journey of continual improvement
![Page 38: Oakville Hydro Business Continuity Edist 2016 · Disaster Recovery Plan (DRP) vs Business Continuity Plan (BCP) Information Technology: Ice Storm Response](https://reader035.vdocuments.us/reader035/viewer/2022070612/5b3669c97f8b9a7e4b8e58e6/html5/thumbnails/38.jpg)