go no-go criteria v24

Version1.02.0

3.04.0

5.06.07.0

8.0

9.010.011.012.013.014.015.0

16.017.018.019.020.021.022.023.024.0

This document contains information about the Go/No-Go Criteria for the Integrated Marketplace program. Each tab contains the criteria for the indicated phase of the program (i.e. a major phase of Market Trials or Go-Live).

These criteria are the quantitative inputs into the overall qualitative Go/No-Go decision making-process. These are the minimum suggested requirements (per phase); there may be additional risks that are documented.

Assumptions:- The criteria will be provided to SPP Program Leadership, the CWG, and the Go-Live Team for use in their respective subjective discussions and Go/No-Go recommendations or decisions- The criteria will be presented to the CWG, as part of the overall Go/No-Go process, as an input into their recommendation- The Go/No-Go Decision Process will not be used for the start of Market Trials Connectivity Testing or Phase 1 of TCR Market Trials- System Delivery is on-time prior to each phase, as defined by the program (i.e. R3 is delivered with all functionality needed for Structured/Unstructured prior to the phase)-Specific testing phase exit criteria for each work stream is defined in Internal SPP documentation.

Reference Documents:Market Trials Folder- SPP IM Market Trials Structured Unstructured Test Market Participant Guide- SPP Integrated Marketplace Market Trials TCR Market Participant Guide- SPP IM Market Trials Connectivity Test Market Participant Guide Readiness & Metrics Folder- SPP IM Participant Engagement Details- SPP IM Readiness Metrics

Date Author1/10/13 J. Woods/E. Aguilar1/21/13 J. Woods/E. Aguilar

2/12/13 E. Aguilar3/4/13 E. Aguilar

3/8/13 E. Aguilar3/11/13 E. Aguilar3/12/13 E. Aguilar

5/10/13 E. Aguilar

5/24/13 E. Aguilar

7/23/13 A. Miller9/19/2013 A. Miller9/25/2013 A. Miller

10/11/2013 A. Miller10/25/2013 A. Miller11/7/2013 A. Miller

11/21/2013 A.Miller/R.Bing

12/11/2013 A.Miller/R.Bing12/19/2013 A.Miller/R.Bing

1/2/2014 A.Miller/R.Bing1/16/2014 A.Miller/R.Bing2/6/2014 A.Miller/R.Bing

2/14/2014 A.Miller/R.Bing2/21/2014 A.Miller/R.Bing2/28/2014 A.Miller/R.Bing

Change DescriptionInitial Draft for MTRG ReviewPublished to CWG

Updated with status for TCR Phase 2 Market Trials Assessment

Updated status/analysis of TCR-05 & TCR-06; updated analysis of TCR-10Updated analysis of TCR-06 and TCR-10

Updated with status for Structured/Unstructured Market Trials Final Recommendation

Updated with Parallel Operations and TCR Go-Live Criteria and identifiers Updated with TCR Go-Live Criteria Preliminary StatusesUpdated TCR status and Parallel Operations tab based on CWG feedbackUpdated preliminary statuses or Parallel Operations CriteriaUpdated preliminary statuses or Parallel Operations CriteriaUpdated preliminary statuses or Parallel Operations Criteria

Updated statuses of Go-Live and Parallel Operations CriteriaUpdated statuses of Go-Live and Parallel Operations CriteriaUpdated statuses of Go-Live and Parallel Operations CriteriaUpdated statuses of Go-Live and Parallel Operations CriteriaUpdated statuses of Go-Live and Parallel Operations CriteriaUpdated statuses of Go-Live and Parallel Operations CriteriaUpdated statuses of Go-Live and Parallel Operations CriteriaUpdated statuses of Go-Live and Parallel Operations Criteria

Updated based on MTRG/CWG feedback: added TCR Go-Live tab, updated notes for metric-related criteria, updated assumptions

Updated with status for TCR Phase 2 Market Trials Final Recommendation; updated Structured/Unstructured to remove references to "Phase 1"

Updated Structured-Unstructured identifiers, added SMT-19, updated notes for SMT-08, 09, 10, 11, 12, 14, 15, and 16.

Updated with a draft of the Go-Live tab and inserted a draft of the Parallel Ops Exit Criteria, Finalized Parallel Operations Criteria, retired TCR Phase 2, Structured-Unstructured, Parallel Operations, and TCR Go-Live tabs

Identifier Phase Sub-Phase

TCR-01 TCR Phase 2 MP

TCR-02 TCR Phase 2 MP

TCR-03 TCR Phase 2 SPP





Responsible Entity

Retired




Retired

Criteria Status Criteria Met?

Market Participant TCR Market Trials Resources Trained Tracking in progress

Met

TCR Accelerated Connectivity Test Complete Tracking in progress

Met

Tracking complete

Met

Tracking in progressMet

Tracking complete

Met

Tracking complete

Met

Tracking complete

Met

100% of SAT test cases deemed 'required' for TCR Market Trials Phase 2 executed

90% of SAT test cases deemed 'required' for TCR Market Trials Phase 2 passed

100% of FIT test cases deemed 'required' for TCR Market Trials Phase 2 executed

90% of FIT test cases deemed 'required' for TCR Market Trials Phase 2 passed

In-scope components, as documented in the TCR Market Trials Market Participant Guide, installed in Production environment

Retired

Tracking complete

Met

Tracking in progressMet

Tracking in progress

Met

SPP Staff ready to support TCR Market Trials Phase 2 - certified by the manager over the areas in scope for the phase. (e.g. TCR, IT)

0 open critical defects required for TCR Market Trials Phase 2

Fewer than 10 open high defects required for TCR Market Trials Phase 2

Retired


Met

All known issues/risks that could impact TCR Market Trials Phase 2 are published and assessed

Retired

Additional Analysis

100% of 'required' SAT test cases were executed

99% of 'required' SAT test cases passed

As of the February Engagement Reports, the status of this activity is considered 'Green' or tracking on time. 100% of High Priority Market Participants, as defined by the TCR workstream, have trained resources for TCR Market Trials. Six of the Market Participants that registered for the Integrated Marketplace by 6/1/12 have not completed TCR resource training as of February 22nd.

As of the January Readiness Metrics update (and confirmed with recent data from the Market Trials team) this metric is 'Blue' or complete. 100% of High Priority TCR Market Participants, as defined by the TCR workstream, have completed TCR Accelerated connectivity testing and the exit criteria have been met.

100% of FIT test cases that were deemed required for TCR Phase 2 Market Trials have been executed as of March 11, 2013. (Up from 72% as reported in v1.0 of the Final Recommendation Report on Friday, March 8, 2013.)

96% of 'required' FIT test cases have passed as of March 11, 2013. (Up from 72% as reported in v1.0 of the Final Recommendation Report on Friday, March 8, 2013 and 94% reported on March 11.)

The remaining test cases (4%) involve changes to the CROW interface. The changes to that interface originally represented 20% of the remaining test cases described in v1.0 of the Final Recommendation Report. There are minor workarounds to address this functionality.

All in-scope components are installed in the Production environment. The previously reported outstanding testing in Production for one component (Integration Services Architecture: Informatica) has been complete.

There are currently no open critical defects.

The manager responsible for the TCR resources supporting TCR Market Trials Phase 2 has confirmed that SPP Staff in the Congesting Hedging group is ready to support TCR Phase 2 Market Trials, including testing with the Integrated Marketplace Market Participants. Enough staff have demonstrated that they have the knowledge and skills to support the business functions as defined in the applicable Market Trials Guide.

The manager responsible for the IT resources supporting TCR Market Trials Phase 2 has confirmed that SPP IT Staff is ready to support TCR Phase 2 Market Trials, including testing with the Integrated Marketplace Market Participants. Enough staff have demonstrated that they have the knowledge and skills to support the IT functions as defined in the applicable Market Trials Guide.

There is 1 open high defect as of March 12, 2013. The one open high defect has a manual workaround and is documented in the Known Issues Log (#30). TCR Market Trials Known Issue # 30: Unable to delete Candidate ARRs in the UI and the API.

After further analysis, the second high defect was that was discovered on March 11, 2013 was resolved and is no longer a defect.

All known issues are published and regularly updated in the Market Trials Known Issues Log on SPP.org. There is the potential that new issues/risks will be discovered as FIT continues. If new issues/risks are discovered, the Market Participants will be notified with the impacts and mitigations.

The TCR workstream has done an additional assessment of the risks to starting TCR Market Trials Phase 2 on 3/18 and have found only minimal risks.

1) Regarding the FERC Order, there are no known risks associated with compliance in association with Phase 2 of TCR Market Trials:• TSRs with Redispatch: Currently on track to have this in place for PTP TSRs for Phase 2.• Monthly Allocation: This is in scope for Phase 2 and plan to have it installed on April 26th. This is not needed until June 7th.

2) Remaining vendor work that is not associated with the FERC Order and is considered in-scope for TCR Market Trials Phase 2:• Turn off compression for member facing APIs: This request is in progress. These APIs can still currently be used by MPs without the fix, so no risk is assumed.• iHedge Credit Interface Redesign: The current estimate has the interface functioning as needed in time for the 5/17 run in Phase 2 with very little slack. If additional time is needed, there is a risk that the cancellation of the Secondary Market for the month of May would be required. Other workarounds are currently being investigated.

Notes Location of Applicable Documents

This will be measured based on 'high priority' TCR MPs as defined by Engagement Activity ENG.014. Completion date is prior to Phase 1 TCR Market Trials. Green (tracking on time) or Blue (complete) status is required for criteria to be met. Readiness & Metrics Project Folder

This will be measured based on 'high priority' TCR MPs as defined by Readiness Metric CON-01. Green (tracking on time) or Blue (complete) status is required for criteria to be met. Readiness & Metrics Project Folder

The 'required' SAT test cases will be identified prior to the start of Phase 2 TCR Market Trials and will be based on the scenarios defined in the Market Trials TCR Market Participant Guide. This includes system updates due to FERC Order and Unintended Consequences. Market Trials Project Folder

The 'required' FIT test cases will be identified prior to the start of Phase 2 TCR Market Trials and will be based on the scenarios defined in the Market Trials TCR Market Participant Guide. This includes system updates due to FERC Order and Unintended Consequences. Market Trials Project Folder

Market Trials Project Folder

http://www.spp.org/section.asp?group=2184&pageID=27





Defects will be available in the Known Issues Log. Analysis will be done on all of the defects. Market Trials Project Folder


Link to location of document TBD

Risks will be evaluated by the TLT and MLT and others, as needed. The risks assessments would be included in the recommendation to the Go-Live Team.

Identifier Phase

SMT-01 Structured/Unstructured MP







SMT-08 Structured/Unstructured SPP




Responsible Entity

Retired

SMT-12 Structured/Unstructured SPPSMT-13 Structured/Unstructured SPP







Criteria Status

Participant System Testing Complete Tracking in progress

MP Market Trials Structured/Unstructured Resources are Trained Tracking in progress

ICCP Modeling and Validation Complete (Refer to ENG.025) Tracking in progress

MP/TO Testing with the MCST Tool Complete Tracking complete

Connectivity Test Complete Tracking in progress

Data Exchange Certification Complete Tracking in progress

Market Participant Registration Complete Tracking in progress





100% of SAT test cases executed for functionality that will be available at the start of Structured/Unstructured Market Trials on June 3

90% of SAT test cases passed for functionality that will be available at the start of Structured/Unstructured Market Trials on June 3

100% of FIT test cases executed for functionality that will be available at the start of Structured/Unstructured Market Trials on June 3

90% of FIT test cases passed for functionality that will be available at the start of Structured/Unstructured Market Trials on June 3

Retired

Tracking in progressMCST Go-Live Tracking complete



SPP Staff ready to support the Structured/Unstructured Market Trials Tracking complete


Structured/Unstructured Market Trials test data plan complete Tracking complete


100% Performance Test cases complete for functionality that will be available at the start of Structured/Unstructured Market Trials on June 3

0 open critical defects for functionality that will be available at the start of Structured/Unstructured Market Trials on June 3

No more than 2 open high defects per Component for functionality that will be available at the start of Structured/Unstructured Market Trials on June 3

All known issues/risks that could impact Structured/Unstructured Market Trials are published and assessed

Systems components that are in scope for the start of Structured/Unstructured Market Trials on June 3 have been installed in Marketplace Production

Criteria Met? Notes/Assumptions

Tracking on time

Tracking on time

Tracking on time

Met

Met

Met

Met

Not Met

Met

N/A

N/A

This will be measured based on Target Mass, as defined by Engagement Activity ENG.020. Green (tracking on time) or Blue (complete) status is required for criteria to be met.




This will be measured based on Target Mass, as defined by Readiness Metric CON-02. Green (tracking on time) or Blue (complete) status is required for criteria to be met.

This will be measured based on Target Mass, as defined by Readiness Metric CON-03. Green (tracking on time) or Blue (complete) status is required for criteria to be met.

This will be measured based on Target Mass, as defined by Readiness Metric REG-02. Green (tracking on time) or Blue (complete) status is required for criteria to be met.

Based on the SMT Roadmap, the SPP test team will determine which test cases are required for functionality that will be available on June 3. The reminder of the test cases (for functionality that will be phased in throughout SMT) will be tracked and reported on a bi-weekly basis.




Retired

Tracking on timeMet

Not Met

Not Met

Met

Met

Met

Met

The Performance test cases will be identified by the SPP test team based on the SMT Roadmap.



Resources will be confirmed ready by the director over the areas in scope for the phase. Qualitative based on director's assessment of their people and knowledge of their training, testing, etc.

Risks will be evaluated by the TLT and MLT and others, as needed. The risks assessments would be included in the recommendation to the Go-Live Team.

Data requirements will be based on scenarios defined in Structured/Unstructured Test Market Participant Guide.

In scope components are based on the SMT Roadmap

Availability of functionality will be tracked in the SMT Release scorecard

Location of Applicable Documents

Readiness & Metrics Project Folder




















Identifier Responsible Entity

TCR.GL.01 MP

TCR.GL.02 SPP

TCR.GL.03 SPP

TCR.GL.04 SPP Retired

TCR.GL.05 SPP

TCR.GL.06 SPP

TCR.GL.07

TCR.GL.08 SPP

TCR.GL.09 SPP

TCR.GL.10 SPP

TCR.GL.11 SPP

TCR.GL.12 SPP

TCR.GL.13 SPP

TCR.GL.14 SPP

TCR.GL.15 SPP

TCR.GL.16 SPP

TCR.GL.17 SPP

Criteria

Market Participant participation in TCR Market Trials verified

Annual TCR auction timeline validated through TCR Market Trials

Monthly TCR auction timeline validated through TCR Market Trials

SPP posts three consecutive Settlement Statements with TCR daily charge typesRetired

SPP posts one Settlement Statement with TCR daily and monthly charge types

Business owner approval for the exit of TCR SAT (Site Acceptance Testing)

Business owner approval for the exit of TCR SIT (System Integration Testing)

Business owner approval for the exit of TCR FIT (Functional Integration Testing)

Business owner approval for the exit of TCR Performance Test Phase

Business owner approval for the exit of Non-Functional Test Phase

All systems supporting TCR have the latest Commercial/ Network Model

TCR Go-Live plan is documented and approved by SPP Business OwnersRetired

Plan for risk mitigation for potential market delay after TCR awards is documented

SPP Staff ready to support the TCR Go-Live

The EADS TCR data store is operational in production prior to TCR Go-Live

CFTC application for waiver submitted by Legal/Regulatory in July for approval prior to TCR Go-Live

All known issues/risks that could impact TCR Go-Live are published and assessed

Retired

Criteria Notes Tracking Status


Tracking complete



This is based on Readiness Metrics TCR-05 and TCR-06 (new metric pending).

This is based on Readiness Metric TCR-01. This criterion is considered met if the metric has a green status

This is based on Readiness Metric TCR-02. This criterion is considered met if the metric has a green status

TCR Daily Charge Types: DaMccHrlyPrc sink, hDaMccHrlyPrc source, hTcrFundAoAmt a, m, dTcrFundMpAmt m, dTcrUpliftDlyAmt a, dTcrUpliftRatioAoDlyAmt a, d TcrUpliftRatioSppDlyAmt dShortFallDlyAmt dDaMccHrlyPrc s, hDaClrdHrlyQty a, s, h,DaClrdVHrlyQty a, s, h, tDaImpExp5minQty a, s, i, tTcrHrlyQty a, h, tTcrFundHrlyAmt a, h TcrUpliftDlyMpAmt m, dECFDlyAmt d DaMccHrlyPrc s, hDaClrdHrlyQty a, s, hDaClrdVHrlyQty a, s, h, tDaImpExp5minQty a, s, i, tTcrFundHrlyAmt a, h ArrNominationCapQty

Retired


Tracking complete

TCR Monthly Charge Types:TcrPaybackMnthlyAmt a, mnTcrUpliftAoMnthlyAmt a, mnTcrUpliftSppMnthlyAmt mnECFMnthlyAmt mn TcrPaybackMnthlyMpAmt m, mnTcrPaybackMnthlyAmt a, mnECFMnthlyAmt mn

Tracking complete

Tracking complete

Tracking complete

Tracking complete

Applicable Systems: EMS, TCR, CMS, Portal Tracking in progress

Tracking completeRetired

Tracking complete

Applicable Departments: TCR, Credit, IT Apps, IT Ops Tracking complete

Tracking complete


Tracking complete

Retired

Criteria Status Status Notes

Complete

Complete TCR-01 is green.

Complete

Complete Settlement statements are being posted daily.


High Priority MPs have been participating per the defined metric (TCR-05). Participation in the October monthly auction is being closely

monitored; HP MPs were asked to submit their intended level of participation and participation

will be tracked based on that.

Rediness Metrics Dashboard

Rediness Metrics Dashboard

TCR-02 is green. Will continue to monitor for October monthly. Rediness Metrics Dashboard

https://cloudpal.idashboards.com/idashboards/?guestuser=2spp2



Complete

Complete

A monthly settlement statement was produced at the end of Septmeber, but the data was

compromised.

KIL# 695 and 696 Month end ArrPaybackMnthlyAmt and

TcrPaybackMnthlyAmt are incorrect. A fix is complete. A fix for the two defects has been tested and installed in Production on October

2.

The September 31 monthly TCR charges will be published to Participants by October 7 with the

corrected functionality.

Known Issues List

Business owner approval has been obtained for stage exit.

All planned test scripts for the release have been executed and results are documented.

All defects found in SAT have been communicated to the business/development

team.All defects that were addressed in SAT were

verified (as fixed or reopened).0 (zero) priority 1 defects and 0 (zero) priority 2

defects based upon the Defect Priority Scale (detailed above in section 2.8 Defect

Management). The Nexant and SPP Project Managers will evaluate Priority 3 defects to determine those that will be required to be

fixed in order to enter SITIE 8 Patches have been received from Nexant

and have been regressioned tested by SPP. If a significant delay occurs, the TCR Business owner has the authority to defer the IE 8

regression testing to the SIT/FIT test stage.


Complete

Complete


100% of SIT Test Cases have been executed with 80% of SIT Test Cases passed.

0 open Critical Defects associated to the integration services components associated with the test, and fewer than 5 open High

Defects associated to the integration services components associated with the test.


This will include 100% test cases executed and 95% test cases passed, there are 0 critical

defects and all high defects have a documented work around.

Complete

Complete

Complete

Complete


• Average response time for page load should meet response time criteria of 5 seconds.

• Average response time for web service calls should meet response time criteria of 3

seconds.• Application failure rate should be less than

1%Systems

• CPU utilization should be below 80%• Memory Utilization should be below 80%• CPU & Memory resources should NOT be

exhausted for more than 5 Minutes• No memory leaks

• No occurrence of HTTP 500 errorsIn cases where performance requirements or

goals cannot be achieved due to situations outside of the control, the performance effort will be considered complete when any of the

following conditions are met: • The performance team and the stakeholders

agree that the application performs acceptably, although, some performance requirements or

goals may have not been achieved. • The pre-determined engagement end date is

reached. • All bottlenecks preventing the application from achieving the performance criteria are

determined to be outside performance testing team’s control.

Business owner approval has been obtained.

>95% Test cases passed for Tier 1 systems and infrastructure, pending that the percentage

failure of the system is not core functionality or availability requirement, and that any

exceptions are reviewed and approved with ITAWG and IT support teams

An issue was identified on 10/02 that requires TCR, EMS, CMS and portal to consume a model

update. This is a low risk and the updated model will be consumed prior to 10/18.

The TCR Go-Live plan has been approved by SPP Business owners.

Complete

Complete

Complete

Tracking on time

Complete

This plan has been documented and was presented to the CWG during the August

meeting.

Resources are confirmed ready by the Directors over the areas in scope for TCR Go-Live. The

three high impact departments have also gone through a Staff Readiness Certification process.

The TCR to EADS interface has been running in production for several months. The EADS team

concluded validation of all functionality with the last fixes implemented on October 2.

SPP received an order from FERC on 9/30/13 conditionally accepting the CFTC tariff changes subject to SPP making a compliance filing due

in 30 days from the date of the order which will be 10/30/13. There is no impact to TCR Go-Live

if the filing is received later than 10/30/2013.

Issues/Risks were evaluated by the TCR workstream and other resources as needed.

MPs were also submitted and assessed.

Identifier Phase Criteria Criteria Notes Tracking Status Criteria Status Status Notes Tracking Location

PO-01 Parallel Operations SPP/MP Tracking complete Complete

PO-02 Parallel Operations MP Tracking in progress Tracking on time TRL-01 is green. Readiness Metrics

PO-03 Parallel Operations MP Tracking in progress Tracking on time

PO-04 Parallel Operations MP Tracking in progress Tracking at risk

PO-05 Parallel Operations MP Tracking in progress Complete

PO-06 Parallel Operations SPP Tracking complete Complete


PO-08 Parallel Operations SPP Tracking in progress Tracking on time Go-Live Criteria

Responsible Entity


All known issues/risks that could impact Parallel Operations are published and assessed

Issues/Risks will be gathered at the workstream and program-wide level as well as MP submissions via RMS.

Request for submission by 10/17 was sent to CWG and Readiness Liasons on 10/9.

Target mass participates in Structured/Unstructured Market Trials

Measured by Readiness Metric TRL-01 Metrics

100% of Structured/Unstructured testing scenarios are successfully executed and completed or removed from scope and successful completion is validated through Readiness Metric TRL-02.

Based on Readiness Metric TRL-02

The number of failed scenarios has caused a significant amount of retesting to occur. This strains the timeline for completion of all testing scenarios prior to Parallel Operations.Issues causing scenarios to fail include system defects and MP participationDetails: 9 retest scenarios, 4 rescheduled. Of the 9, 4 will not be executed until v1.8, 2 will not be retested until the mitigation logic is changed based on the FERC's response. The remaining 3 are (commitment mismatch, RRS, & EDR).

Parallel Operations Exit Criteria Metrics

SPP and member ICCP Models are up to date and all ICCP and XML inbound and outbound Marketplace data points have been validated with all applicable Market Participants as part of Structured Market Trials

Will be updated with final status. As of the week of Nov. 13:

There are 16 resources that have not been validated with MPs. The remaining MPs are not critical to the entry of Parallel Operations, are manageable for IDT, are outside of Target mass and account for less than 3% of MWs in SPP.

MPs need to complete the setup of these points and checkout data with SPP:-Resource Control Mode Status (16)-Resource MW output (4) - pending checkouts of data values

Metrics

All MP EMS related changes, including AGC (if applicable), have been completed

Between 10/21 - 11/6 MPs must complete a brief survey through KeySurvey to confirm completion of system changes or determine that none are applicable.

All BAs have completed the survey and have indicated that they have already completed changes or will complete

them prior to Parallel Operations.

There are 7 outstanding BAs that have indicated that they had not completed EMS changes by 10/26. An outreach effort is ongoing to work with outstanding BAs that still

have to complete outstanding EMS related changes.

Successfully execute Integrated Marketplace Bid to Bill functionality consistently for 3 consecutive Operating Days according to protocol timelines

SMT Exit Criteria; Bid to Bill internal definition: Market Participant bid submission, Day Ahead Market, Real Time Balancing Market, RUC, EADS, POPS, Settlements

This criterion was first met for Operating Days 10/8, 10/9 and 10/10. It has been achieved several times since.

Execute Integrated Marketplace Bid to Bill functionality consistently for 15 supported Unstructured Operating Days according to protocol timelines.

Unstructured days for this criteria will begin tracking on September 25, 2013 and continue through the end of SMT. These are not consecutive days.

20 unstructured days were executed since 9/26 as of 10/25

SAT (Site Acceptance Testing) Stage Exit Criteria is met by all applicable workstreams and Business Owner approval is obtained (with the exception of Markets)

Applicable workstreams: CMT/MCST, EMS, Load Forecasting, OATI, Portal, RTGEN, Credit, EADS, POPS, Settlements

EADS - No impact to participant experience: SAT/SIT has completed for all EADS areas in scope for Parallel Operations. 96% pass rate achieved as of 11/1, SPP is currently in the process of confirming stage exit with Business Owner POPS - Low impact to participant experience: POPS functionality has been running for several weeks in support of Settlements SMT. Remaining items are largely situational and can be managed by the POPS/Settlements teams. POPS is at 88% complete and is targeting completion by 11/15.

Retired






PO-11 Parallel Operations SPP Tracking in progress Tracking at risk

PO-12 Parallel Operations SPP Tracking in progress Complete

SIT (System Integration Testing) Stage Exit Criteria is met by all applicable workstreams and Business Owner approval is obtained

Applicable workstreams: CMT/MCST, EMS, Load Forecasting, Markets, OATI, Portal, RTGEN, Accounting, Credit, EADS, POPS, Settlements

EADS - No impact to participant experience: SAT/SIT has completed for all EADS areas in scope for Parallel Operations. 96% pass rate achieved as of 11/1, SPP is currently in the process of confirming stage exit with Business Owner POPS - Low impact to participant experience: POPS functionality has been running for several weeks in support of Settlements SMT. Remaining items are largely situational and can be managed by the POPS/Settlements teams. POPS is at 88% complete and is targeting completion by 11/15. OATI - No impact to participant experience: The remaining functionality is specific functionality required to report inadvertent to NERC after Go-Live. Remaining testing is specific to testing the Tie Line Validation Tool (14 test cases) and target completion is 11/15.

FIT (Functional Integration Testing) Stage Exit Criteria is met by all applicable workstreams and Business Owner approval is obtained (with the exception of Settlements)

Applicable workstreams: CMT/MCST, EMS/Load Forecasting, Markets, OATI, RTGEN, Accounting, Credit

OATI - Low impact to participant experience: Remaining OATI FIT is at 75% complete. Issues with Ramp Reservation System have been resolved and successfully tested in SMT. Remaining OATI FIT is currently planned through 1/31 including regresstion test of webOASIS and validation of Inadvertent Reporting to NERC.Markets - Moderate impact to participant experience: 93% complete. Integration with OATI systems and CMS represent moderate impacts to the execution of Parallel Operations. Target test phase completion is 11/15.RTGEN - Low impact to participant experience:87% complete. Tie line modeling is closely being monitored and is on track to be closed out soon. Integration between RTOSS and RTGEN not fully required to start Parallel Operations as EIS schedules and NSI will be used. Target completion is 11/29.CMS - Low/no impact to participant experience: 77% complete. Integration with Markets is working sufficiently to begin Parallel Operations, noting two defects related to Participant Default scenario and passing deficiency dollar amount back through Markets system. CMS integration with Settlements and Accounting is not required for Parallel Operations since Participants will not be posting actual payments. Target testing completion is 11/29.

All ICCP and XML inbound and outbound Marketplace data points have been validated with all applicable Market Participants as part of Structured Market Trials

This criterion will demonstrate the additional Markets functionality required to be SAT tested prior to Parallel Operations

Due to redundancy, this criterion has been combined with PO-04.

All charge types for target mass have been successfully published and verified through Structured Market Trials and Business Owner approval is obtained as demonstrated through the Settlements tracker

This will be demonstrated with minimum defects, and actual upstream test data is used to ensure quality. This criterion will demonstrate the additional Settlements functionality required to be FIT tested prior to Parallel Operations.

Target mass has hit 100% of applicable charge types, but not all charge types are applicable for all participants.

Retired



PO-15 Parallel Operations SPP/MP Tracking complete Complete



PO-18 Parallel Operations SPP Tracking in progress Complete

PO-19 Parallel Operations SPP Protocol Compliance has been validated Tracking in progress Tracking on time Go-Live Criteria

Performance Testing Exit Criteria is met by all applicable workstreams and Business Owner approval is obtained

Applicable workstreams: Markets, MCST, Portal, Settlements

Settlements - No impact to participant experience:The focus of these tests does not apply to Parallel Operations but rather the gradual increase of Settlements data over time.Markets - No impact to participant experience:Remaining performance testing is all specific to internal SPP processes. Participant experience through Parallel Operations should be consistent with SMT regarding Markets system performance.

Both of these have no impact on the ability to operate in Parallel Operations. Performance testing will occur over the life of the system. The focus of the remaining planned testing does not apply to Parallel Operations.

Non Functional Testing should be completed and reviewed by SPP Technical Architects validating the systems demonstrate sufficient stability to commence Parallel Operations

Applicable workstreams: CMT, CMS, ICCP, Markets, Portal, Legacy Applications, EMS, EADS, POPS, Settlements

Settlements - No impact to participant experience:Remaining NFT for all Market Settlement systems is scheduled to complete in December. Participant experience through Parallel Operations should be consistent with SMT regarding Settlements availability.Markets - Low impact to participant experience:Remaining NFT is all specific to internal Markets sub-systems. Participant experience through Parallel Operations should be consistent with SMT regarding markets system availability.

Impacts of Critical/High defects are deemed acceptable by SPP and Market Participants

Defects were published with the Assessment The critical and high defects were discussed at the CWG

conference ccall on October 29.

MCE Certification sign-off obtained from Markets Business Owners

Moderate impact to particpant experience – Not all protocol compliance validations can be performed with MOS 1.7. remaining items will be received in MOS 1.8, and sign-off will occur after release 1.9 when mitigation is available.

Systems components that are in scope for the start of Parallel Operations have been installed and integrated in Marketplace Production

Great Plains, CMS, CMT, CROW, EADS, RSS, RRS, IDC, OATI/RTOSS, EMS, Portal, DBDA-RUC, RRC, PFC, DA-Market, DA-RUC, ID-RUC, RTBM, Settlements

All systems supporting Parallel Operations have the latest Commercial/ Network Model

CMS, CMT, EADS, RSS, OATI/RTOSS, EMS, Portal, Markets (DBDA-RUC, RRC, PFC, DA-Market, DA-RUC, ID-RUC, RTBM), Settlements

Exceptions: RSG, GFA, Marginal losses

Scarcity Conditions: Scenario was largely successful but a few issues are being addressed.Special Resources: BDR/DDR resources work operationally. Settlements is implanting code that allows them to be settled correctly.Emergency Logic: Emergency logic has been tested successfully in RTBM.Mitigation: The mitigation scenarios were largely successful with a defect found that impact resources in certain hours.

SPP is seeking reconsideration / rehearing, requesting authority to implement this plan and represent that SPP will be compliant with the order within a certain period of time after Go-Live. Additionally SPP will propose to resettle upon implementing compliant systems. FERC is under no time obligation to act on a reconsideration request. SPP will request expedited action, but will not expect an order until February.

Retired


PO-21 Parallel Operations SPP The SPP Parallel Operations Go-Live Plan is approved internally Tracking in progress Complete


PO-23 Parallel Operations MP Tracking complete Complete


CPS1 and CPS2 waivers complete (approved by NERC) or an alternative plan is implemented (approved by CBASC)

The CPS1 and CPS2 waivers were approved by SPP RE & MRO on August 20, 2013.

A master Go-Live Internal Implementation Plan is being prepared to guide SPP only deployment activities and will contain the following:

• A high-level visual timeline of the major activities taking place over the Go-Live period, the approximate times during which they will be occurring, and the primary owner for each activity. • Individual tasks within each of those high level activities with the individual responsible for executing the task, their backup, the responsible team, estimated start/end times, duration, as well as a link to specific instructions needed to carry out the task if necessary.

The SPP Parallel Operations Roll-In/Roll-Out Integrated Deployment Plan is approved internally including IDT MP Guide and Communication Plan

EIS BA procedures/cutover plans have been submitted and validated by SPP

The current deadline for EIS BA submission is 8/12/2013. The assumption is that validation is defined as including the primary basic elements of the plan

All 16 EIS Balancing Authorities submitted their cutover plans to SPP in August and all plans were validated by SPP.

Physical setup of the SPP facilities supporting Parallel Operations is complete

Ensure that Operations floor, desks, and screens are ready to support Parallel Operations.Retired

Identifier Phase

GL-01 Go-Live

GL-02 Go-Live

GL-03 Go-Live

GL-04 Go-Live PO-08

GL-05 Go-Live PO-09

GL-06 Go-Live PO-10

GL-07 Go-Live PO-13

GL-08 Go-Live PO-14

GL-09 Go-Live

GL-10 Go-Live

GL-11 Go-Live

GL-12 Go-Live

GL-13 Go-Live

GL-14 Go-Live

Responsible Entity

Rollover from PO?

GL-15 Go-Live

GL-16 Go-Live

GL-17 Go-Live PO-19

GL-18 Go-Live

GL-19 Go-Live

GL-20 Go-Live

GL-21 Go-Live

Criteria

Integrated Deployment Testing is complete

SAT stage exit criteria is met

SIT is complete stage exit criteria is met

FIT is complete stage exit criteria is met

Performance Testing stage exit criteria is met

Non-Functional Testing stage exit criteria is met

0 open critical defects

The SPP Markets Go-Live Plan is approved internally

All systems have the latest Commercial/ Network Model

All known issues/risks that could impact Go-Live are published and assessed

SPP will provide an 30 day system freeze, and Go-Live will not begin until the conclusion of the freeze.

All high defects have acceptable documented workarounds, a mitigation plan, a planned release for fixes, or have been flagged as acceptable for post Go-Live

All systems components have been installed and integrated in Marketplace Production

The physical setup of the SPP facilities supporting Go-live is complete

SPP Staff is ready to support Go-Live

100% of Target Mass completes Readiness Certification

Protocol Compliance has been validated

FERC Approval Order Received

Readiness and Reversion Plan Accepted by FERC

Completion of any approved open Readiness critical metrics

Completion of any approved open Parallel Operations Exit Criteria items

Notes Completion Date Status

Published in the Go-Live Assessment 1/17/2014 Complete

2/28/2014 Complete

Not Met

2/21/2014 Complete

All functional areas in the program have completed SIT. 1/24/2014 Complete

All functional areas in the program have completed FIT. 2/28/2014 Complete

1/30/2014 Complete

Tracking on Time

2/28/2014 Complete

2/21/2014 Complete

Completed with the MCE patch on February 24 2/28/2014 Complete

The plan has been reviewed and approved internally. 2/5/2014 Complete

1/20/2014 Complete

All systems have comsumed the March 1, 2014 model 2/28/2014 Complete

TRL-03

All scheduled Deployment tests were completed successfully. The final IDT is scheduled for February 28, as part of the Markets Go-Live.

Continued defects and development have resulted in the need for patches being installed during the freeze period. The issues and impacts have been communicated during the CWG checkpoint discussions.

All functional areas in the program have completed SAT.

All functional areas in the program have completed Performance Testing.

The functional areas that currently have outstanding Non-Functional Testing are Markets, Settlements and Informatica/Netezza.

Completed and awaiting final exit signoff.

MCE patch on February 24 closed the remaining open critical defects.

High level defects are described in the Go-Live Assessment

The Operations Data Center setup is complete, and the physical configuration has been tested throughout Integrated Deployment Testing.

1/20/2014 Complete

RGL-03 1/14/2014 Complete

2/28/2014 Complete

2/28/2014 Complete

RGL-02 12/19/2013 Complete

See the Parallel Operations Exit tab Not Met

2/21/2014 Complete

SPP Staff have been confirmed to be ready by the director over the areas in scope for the phase. The applicable functional areas include System Operations, Operations Support, Markets Administration, Credit, IT Applications, IT Operations, Settlements, Customer Relations, and Market Monitoring.

The MCE patch on February closed out the open Protocol Compliance issues and MCE certification

FERC response to compliance filing received, SPP compliance filing made on February 26.

All critical readiness metrics are complete, including the CBA certification activities, completed 2/14. Many of the readiness metrics are complete or tracking on time.

Criteria Met? Location of Applicable Documents

Yes

Yes

Yes

Yes

Yes

Yes

Yes

Yes

Yes

Yes

Yes

Yes

Yes

Yes

Yes

Yes

Yes

Identifier Phase

POX-01 Parallel Operations SPP/MP PO-03

POX-02 Parallel Operations

POX-03 Parallel Operations SPP/MP PO-12

POX-04 Parallel Operations SPP





Responsible Entity

Rollover from PO?




POX-20 Parallel Operations MP





Criteria Notes

100% of Structured/Unstructured testing scenarios are successfully executed and completed or removed from scope and successful completion is validated through Readiness Metric TRL-02

TRL-02

2/28 : 75-100% of Structured/Unstructured operational scenarios are complete

Integrated Marketplace Bid to Bill functionality successfully completed for 7 consecutive Operating Days according to protocol timelines

All charge types for target mass have been successfully published and verified through Structured Market Trials and Business Owner approval is obtained as demonstrated through the Settlements tracker

STL-07

100% of Charge Types with calculated results were utilized by Target Mass. Charge Types with calculated results not utilized by Target Mass are explainable.

State Estimator has ninety-seven percent (97%) availability of five (5) minute solutions for 7 consecutive days during Parallel Ops

OPS-02

State Estimator has ninety-seven percent (97%) availability of five (5) minute solutions for 7 consecutive operating days prior to Parallel Operations

State Estimator solves 90% of the time with Mismatch level lower than 50 MW for 7 consecutive days during Parallel Ops

OPS-03

State Estimator solves 90% of the time with Mismatch level lower than 50 MW for 7 consecutive days prior to Parallel Ops

Short term load forecast for the SPP market footprint used for RTBM for conforming loads solves 97% of time with +/- 1% accuracy for seven consecutive days during Parallel Ops

OPS-04

Short term load forecast for the SPP market footprint used for RTBM for conforming loads solves 97% of time with +/- 1% accuracy for seven consecutive days prior to Parallel Ops

4am Mid-term load forecast for conforming loads for the SPP market footprint solves 97% of time with +/- 3% accuracy for seven consecutive days during Parallel Ops

OPS-05a

4am Mid-term load forecast for conforming loads for the SPP market footprint solves 97% of time with +/- 3% accuracy for seven consecutive days prior to Parallel Ops

12pm Mid-term load forecast for conforming loads for the SPP market footprint solves 97% of time with +/- 3% accuracy for seven consecutive days during Parallel Ops

OPS-05b


5pm Mid-term load forecast for conforming loads for the SPP market footprint solves 97% of time with +/- 3% accuracy for seven consecutive days during Parallel Ops

OPS-05c


Wind forecast for SPP Market footprint solves with +/- 15% accuracy for six out of seven consecutive days during Parallel Ops

OPS-06a

Wind forecast for SPP Market footprint solves with +/- 15% accuracy for six out of seven consecutive operating days prior to Parallel Ops


OPS-06b



OPS-06c


Current Hour: Market Flow Calculations successful ninety-seven percent (97%) of 5 minute solutions for seven consecutive days during Parallel Ops

OPS-07

Market Flow Calculations successful ninety-seven percent (97%) of 5 minute solutions for seven consecutive days prior to Parallel Ops

Next Hour: Market Flow Calculations successful ninety-seven percent (97%) of 5 minute solutions for seven consecutive days during Parallel Ops

OPS-08

Market Flow Calculations successful ninety-seven percent (97%) of 5 minute solutions for seven consecutive days prior to Parallel Ops

Day-Ahead Market results are posted every day by 4pm for 5 consecutive days during Parallel Ops

OPS-09

Day-Ahead Market results are posted every day by 4pm for 5 consecutive operating days prior to Parallel Ops

LMPs are made available for each settlement location every day by 4pm for 5 consecutive days during Parallel Ops

OPS-10

LMPs are made available for each settlement location every day by 4pm for 5 consecutive days prior to Parallel Ops

Real-Time Calculations successful ninety-nine percent (99%) of five (5) minute solutions for 7 consecutive days during Parallel Ops

OPS-11

Real-Time Calculations successful ninety-nine percent (99%) of five (5) minute solutions for 7 consecutive operating days prior to Parallel Ops

Operating Reserves are calculated and posted on time for 7 consecutive days during Parallel Ops

OPS-12

Operating Reserves are calculated and posted on time for 7 consecutive days prior to Parallel Ops

The 5 min rolling average delta between the SPP ACE and the SUM of the individual EIS BA ACE values is within +/- 10% of the corresponding Average Sum of the individual SPP BA ACE for 7 days with delta’s of +/- 30 %that are explainable.

OPS-14

The 5 min rolling average delta between the SPP ACE and the SUM of the individual EIS BA ACE values is within a band defined as the greater of +/- 20MW and +/- 10% of the corresponding Average Sum of the individual SPP BA ACE for 7 days

100% of resource-owning MPs reports outages through SCADA and CROW within 30 minutes

OPS-15

2/9-2/15: 12/14 of resource-owning MPs who have reported outages have not reported them through SCADA and CROW within 30 minutes. SPP is providing feedback to MPs on their performance on an individual basis, and metrics have shown improvement. SPP made a push at the January CWG to address this issue and has already seen improvement since the meeting.

100% of resource-owning Market Participants ICCP models are up-to-date and validated

OPS-17

100% of resource-owning MPs are complete

100% of applicable Market Participants correctly submitted control modes

OPS-18

2/11-2/19: 37% of MPs submitted control modes correctly 100% of the time. SPP is providing feedback to MPs on their performance on an individual basis, and metrics have shown improvement.

100% of Target Mass were able to successfully download a Statement

STL-01

100% of Target Mass were able to successfully download a Statement

100% of Target Mass were able to successfully download an Invoice

STL-02

100% of Target Mass were able to successfully download an Invoice

100% of Target Mass successfully submit a dispute

100% of meter agents successfully submit meter data for each of their assets

STL-03

100% of meter agents successfully submit meter data for each of their assets

STL-05

100% of Target Mass successfully submitted a dispute

Settlement Statements are posted by end of day for 7 consecutive calendar days during Market Trials, including Initial and Final

STL-06

Settlement Statements are posted by end of day for 7 consecutive calendar days during Market Trials, including Initial and Final

SPP has zero open Critical severity defects documented

MCE patch on February 24 closed the remaining open critical defects.

Status Criteria Met? Location of Applicable Documents

Complete

From Parallel Ops Go/No Go

12/19/2013Complete

Yes From Parallel Ops Go/No Go

11/6/2013

Complete

Yes Readiness Metrics

12/6/2013

Complete


12/6/2013

Complete


12/6/2013

Complete


12/6/2013

Complete


12/6/2013

Complete


Completion Date

12/6/2013

Complete


12/6/2013

Complete


12/6/2013

Complete


12/6/2013

Complete


12/6/2013

Complete


12/6/2013

Complete


12/6/2013

Complete


12/6/2013

Complete


12/6/2013

Complete


12/6/2013

Complete


12/6/2013

Complete


Not Met

Readiness Metrics

CompleteYes Readiness Metrics

Not Met

Readiness Metrics

10/18/2013

Complete


10/18/2013

Complete


8/31/2013

Complete


8/31/2013

Complete


9/30/2013

Complete


Complete