information technology update ercot board of directors meeting january 17th, 2005
TRANSCRIPT
Information Technology Update
ERCOT Board of Directors MeetingJanuary 17th, 2005
page 2
System Outage Time line
Outage Time line
12/25
9:29 pm
First drive fails
Replacement scheduled by
10:37pm
12/26
5:34 am
Second drive fails
12/26
1 pm
NAESB Recovery
begins
Assessment, recovery planning
12/26
10:30 pm
1st market notice sent
12/27
11:00 pm
TCH Recovery
begins
12/28
2:00 am
NAESB transactions started
12/30
12:00 am
TCH Online
1/1/06
11:15 pm
Retail Systems
online
1/4/06
Siebel Daily Extracts Posted
Replicated Lodestar Current
1/5/06
Siebel Full extracts posted
Recovery completed
(extracts)
1/15/06
page 3
System Outage What Happened?
RAID – Redundant Array of Independent disk
RAID 5 - Block Interleaved Distributed Parity: Provides data striping at the byte level and also stripe error correction information. This results in excellent performance and good fault tolerance. Level 5 is one of the most popular implementations of RAID (Webophedia 2006)
Data
Data
Data
Parity
Spindle
(ERCOT runs more than 2000 spindles in SAN environment)
RAID 5 Array
Track
Each spindle contains usable data and parity data
for other drives in the array.
If one drive fails, the parity data on the other drives
rebuilds the missing drive from the parity tracks of the
working drives.
First drive failed 12/25 at 9:29
Second drive failed 12/26 at
5:34
Note: 1 additional drive for parity indexing not depicted
page 4
System Outage Review Steps
Data Storage Management Processes
—EMC (Data Storage Vendor) engagement• Staff augmentation and consulting• Administer storage environment• Review ERCOT storage processes and procedures• Recommend changes per “Best Practices” if appropriate
Technology—“Hot Swap” implementation—Third layer of protection
People—Review of skills in Storage area—Provide additional training if needed—Review of Management structure
5© Property of ERCOT 2005
Market Redesign Time line and Critical Path
•For effort estimation assumed 1/1/2009 for Go Live•With 3 months for Pilot and 6 months for Mock Market, 27 months remain to complete all Nodal projects•A number of Nodal projects are now on the critical path (require ~27 months to deliver)
•NMMS, NSA Enhancements, Settlements and Billing, EDW framework, and Nodal training
6© Property of ERCOT 2005
Market Redesign Project Cost Estimate
Ground Up Approach• ~735,000 estimated hours• 86 projects/initiatives anticipated• ~50 ERCOT employees require backfill• ~40 external contractors
Project cost• $95 - $130 million• Previous CBA $59.4 – 76.2 million
• Key differences • Hardware estimated around $40 million
• Several minor differences in labor and assumptions
• CA-ISO publicly disclosed estimate $139.2 million
7© Property of ERCOT 2005
IT Update Accomplishments
Improved Change and Release Management Capacity Plan development Key upgrades/implementations
• Release 3 and Release 4• LodeStar 3.7• Siebel 7.7• Texas Set 2.1
Service Oriented Architecture Market Redesign efforts
• Protocol reviews• Readiness planning
Outstanding group of people across all of ERCOT!!!