April 14, 2010

RMS

ERCOT 3/27/10 Production Issue Overview and Lessons Learned

Karen FarleyManager, Retail Customer Choice

2

Outline for RMS

• Project History• Troubleshooting Timeline• Market Communications• Market Impacts• Lessons Learned

3

Project History

PR90024 – Expedited Switch – implemented weekend of 8/16/09• Switch transactions submitted to ERCOT will be processed using the new timelines

as indicated in the PUCT expedited switch rule.

PR90024 – POLR A – implemented weekend of 1/23/10• Increase number of Non-Volunteer POLRs (henceforth to be referred to as Large

Service Providers).• Notification to defaulted customers via postcard, email and phone.• The cancellation (814_08) and notification (814_06) windows for all switches will be

shortened from five to two business days prior to the Scheduled Meter Read Date (SMRD).

• The reject code used on the 814_09s to communicate that the cancellation is within the two business day window prior to the SMRD is being changed to reflect this from the code CW5 to CW2.

PR90024 – POLR B – implemented weekend of 3/27/10• Monitor transitioned ESIIDs for 60 days: Identify Mass Transitioned ESI IDs to ensure

that the first switch following the Mass Transition, if received within 60 calendar days of the effectuating date of the Mass Transition, is forwarded to the TDSP with a requested date equal to FASD. The identification shall terminate at the first completed switch or at the end of the 60-day period, whichever occurs first.

• POLR Switch Monitoring Report: Internal ERCOT report.

4

Troubleshooting Timeline

• 03/30 – ERCOT identified a problem with 814_04s failing internally and traced this back to the 814_03 for Self Selected Switches. Emergency SIR was submitted– ERCOT was contacted by a Market Participant concerning the

issue with the LIN segment of 814_03s on self-selected switched.

– ERCOT reprocessed impacted 814_04 transactions that failed for Texas Set validations to schedule the switch orders.

• 03/31 – Emergency SIR implemented to correct 814_03 transaction.

– Continued reprocessing impacted 814_04 transactions– Identified an additional set of impacted orders that had been

Cancelled/Rejected by TDSP.– MarkeTrak issues were submitted to CRs on 4/1 and 4/2 to work

through potential out of synch conditions.• 04/01 – Continued reprocessing impacted 814_04 transactions

5

Troubleshooting Timeline Continued

• 4/1 – A REP notified ERCOT that a subset of the self-selected switches were scheduled or completed for a different date than what was requested on the initiating switch transaction. – 4/1 – 4/2 – ERCOT worked with the TDSP#4 to identify impacted

list. – 4/5 – ERCOT submitted MarkeTrak issues to the REPs for the

impacted switches.– 4/6 – ERCOT provided a list to the TDSP#4 of the MarkeTrak

issues submitted to REPs. – 4/7 – 4/9 – Discussions between TDSP #4 and ERCOT

concerning resolution method. Finalized by TDSP #4 on Friday, 4/9.

• ERCOT proposed solutions in the MarkeTrak issues when originally submitted. After discussions with TDSP #4 a different solution was implemented.

6

Troubleshooting Timeline Continued

• 4/9 – 4/12 – ERCOT completed data repair on affected orders to update scheduled meter read dates and updated MarkeTrak issues with the TDSP #4 solution

Calls with individual Market Participants

TDSP #4 – 04/08/10 – TDSP requested conference call

TDSP #4 – 04/09/10 – New plan was provided by TDSP

7

Issue Identification / Market Communication

Tues 03/30/10 - Market Notice R-A0330-01 INITIAL - Retail Processing

LONG DESCRIPTION:  As result of Project 90024 which was implemented in this past weekend’s Retail Release, 814_04s on Self Selected Switches are failing mapping status at ERCOT due to incorrect LIN segment in the 814_03 sent to the TDSPs.

Wed 03/31/10 - Market Notice R-A0330-021 UPDATE- Retail Processing

LONG DESCRIPTION:  ERCOT will be implementing an emergency SIR for the invalid mapping status failures related to the weekend outage and self-selected switches. There will be no outage necessary and no market impact.

All 321 self-selected switches impacted by the invalid mapping status failure and received prior to 8:00 AM this morning, March 31, 2010 have been successfully reprocessed and are in either a Scheduled or Complete state. ERCOT is working through the reprocessing of the self-selected switches received after 8:00 AM today, March 31, 2010 and will have them reprocessed before noon (12:00 PM) tomorrow, April 1, 2010.

ADDITIONAL INFORMATION:  There were 25 self-selected switches in a Cancelled state prior to the ERCOT reprocessing effort. ERCOT will reach out to the submitting CRs of these individually to request resubmission and/or manual intervention.

8

Issue Identification / Market Communication

Thurs 04/01/10- Market Notice R-A0330-03 UPDATE- Retail Processing

LONG DESCRIPTION:  ERCOT successfully implemented an emergency SIR for the invalid mapping status failures related to the weekend outage and self-selected switches on March 31, 2010.

ADDITIONAL INFORMATION:  ERCOT is continuing to identify impacted transactions related to this incident. Affected MPs have been sent initial e-mails with MarkeTrak issues opened for them. Additional e-mails will be sent on April 2, 2010 detailing further impacts.

Fri 04/02/10- Market Notice R-A0330-04 FINAL Retail Processing

LONG DESCRIPTION:  On Wednesday, March 31, 2010, ERCOT successfully implemented an emergency SIR for the invalid mapping status failures related to the weekend outage and self-selected switches.

The volume of impacted switches is 682. ERCOT completed reprocessing of affected 814_04 transactions on Thursday, April 1, 2010. 

ERCOT is continuing to work with affected market participants to identify transactions which could potentially cause out-of-synch conditions. ERCOT Initiated MarkeTrak issues are being opened with affected Market Participants.

ADDITIONAL INFORMATION:  Affected Market Participants received lists of reprocessed transactions on April 1, 2010. MarkeTrak issues are continuing to be opened as out of synch situations are identified. Updated lists will be sent to affected Market Participants as issues are identified.

9

Market Impacts

• Customer impacts – switches occurring on date different than requested

• 31 REPs and 5 TDSPs impacted – delay of transactions and manual efforts

• REPs were emailed lists and were requested to submit 814_26, ad-hoc historical usage, requests where ERCOT failed to convey this request in the 814_03

• Subset of Switches were processed by TDSP #4 as standard switches resulting in completion of orders earlier than the date requested by the customer. This also resulted in 814_06s sent with the incorrect dates. REPs will receive cancel/rebill 867s. MarkeTrak issues with lists of ESI IDs affected were logged to the submitting Switch REPs.

• Self Selected Switches were rejected by TDSP #6. Once SIR was implemented, REPs could resubmit) (they had received 814_05 reject.

• Out of synch conditions between TDSP, REPs and ERCOT

10

Volumes

Self-selected that required re-work

TDSP #1 10

TDSP #2 68

TDSP #3 16

TDSP #4 315

TDSP #5 3

TDSP #6 258  670

Total Switches submitted between 3/27 and 3/31

Switch 10561

Self - Selected 787*

Standard 9774

Grand Total 10561

*Difference – there were 2 scenarios that did not have an issue, had to do with the order in the LIN segment of the HU/HI and SW

Issues Self-selected that required re-work

1-24 22 CRs

25-49 4 CRs

50-74 4 CRs

75-100 1 CR

Total 31 CRs

11

MarkeTrak issues

Impacts MarkeTrak Issues

Reprocessed 814_04s 412 orders reprocessed. Lists were provided on 4/5 via email to affected CRs and TDSPs.

Scheduled / Completed for incorrect date

23 MarkeTrak issues submitted on 236 orders: 4 – Complete 10 – Pending Complete 3 – In Progress (Assignee) 6 – New (back with CR for 2nd time)

*Status as of 4/13

This was a subset of the 412 reprocessed orders.

Cancelled/ Rejected by TDSP 45 MarkeTrak issues submitted on 270 orders: 45 – Complete

12

Lessons Learned

Communication

• Breakdown of communications at ERCOT affected notifications to the market

– Delay in list of the MarkeTrak issues to impacted TDSPs

– Inconsistent information was provided by ERCOT to Market Participants on resolution activities

• ERCOT Action Item – host a Market call to discuss impacts• ERCOT Action Item – confirm course of action before

notifying Market Participants• ERCOT Action Item – when time sensitive ERCOT Initiated

MarkeTrak issues are submitted where a response is needed, ERCOT should follow-up with email and phone call to indicate importance and facilitate solution. A majority of ERCOT initiated issues are informational in nature.

13

Lessons Learned

TrainingChallenge faced – Impacted TDSPs processed the 814_03 differently.

REPs internal processes differ. This resulted in out of synch and numerous scenarios.

– ERCOT Action item – RCC team members will have additional training on system issues and data analysis

14

Questions?