assessment of nctd program management … implementation of the ptc system is complex and dynamic,...
TRANSCRIPT
0 icfi.com/transportation © 2013
Prepared for:
Brad Hansen, M.S., PMP Director, PMO Capital Projects
Assessment of NCTD Program Management Framework for Positive Train Control Program Subtask 2: Analysis – Gap Analysis
May 2013
1 icfi.com/transportation © 2013
Subtask Objective EXECUTIVE SUMMARY
Identify gaps in the PTC Program project management processes, artifacts, tools and staff resources required for effective management Assess the PTC Program’s Program Management Plan (PMP) compliance with:
– PMI PMBOK – 49 Code of Federal Regulation (CFR) § 633.25 - Contents of a project
management plan – ICF’s experience and lessons learned from analogous programs, tailored to the
NCTD PTC Program environment Review priority components of the PMP, as requested by NCTD:
– Program Management Organization – the organizational structure and associated resources
– Requirements Management – the processes and tools NCTD employs to manage requirements
– Risk Management – the process and tools employed by NCTD to manage programmatic risk
Recommend prioritized improvement areas to NCTD’s program management
approach based on expected benefit to the NCTD organization
2 icfi.com/transportation © 2013
Key Themes EXECUTIVE SUMMARY
The implementation of the PTC System is complex and dynamic, placing stress on NCTD’s organizational capabilities
In many instances, the PTC Program’s “current state” activities are inconsistent with the Program Management Plan’s (PMP) documented processes, tools, and structures
Formally documented program-level controls, integrating all PTC-related activities and components, do not exist or are not being exercised
– The program-level controls that do exist are largely specific to the HTI PTC System contract—which represents only 40% of the total program budget
3 icfi.com/transportation © 2013
Gap Analysis Results EXECUTIVE SUMMARY
PMBOK Knowledge Area NCTD PTC
Ratings Summary PMP Current State
Integration Management Little evidence of integration processes or tools were observed
Scope Management The entirety of the PTC Program scope is not defined and documented
Cost Management No cost management baseline exists and inconsistencies between PTC Program and NCTD cost management processes are present
Schedule Management Lack of a schedule baseline or a single integrated master schedule
Risk Management Undefined risk management process, but recently instituted project-level processes will enhance Program level risk management
Quality Management An organizational quality capability does not exist, nor is there a Quality Management System
Human Resource Management The current organizational structure is not the most effective approach
Procurement Management Mature organizational procurement capabilities with few needed improvements
Stakeholder Management Level of stakeholder identification, analysis and engagement for a Program of this complexity is not evident
Communications Management Strong organizational capability, though ineffectively integrated into the PTC Program
The color scale reflects the degree of compliance with (i.e. all, most, some, few) the processes, artifacts and tools prescribed within the PMBOK knowledge areas
All
Most
Some
Few
How well does the PTC PMP stack up to PMBOK? How well does the Current State of the Program follow PMBOK?
4 icfi.com/transportation © 2013
Recommended Improvements EXECUTIVE SUMMARY
Baseline the Program
Enhance program control
Re-align program organization
Implement a coordinated stakeholder strategy
Dedicate resources to prepare for post-implementation operations and maintenance
2
1
3
4
5
5 icfi.com/transportation © 2013 5
Methodology
6 icfi.com/transportation © 2013
Scope of Gap Analysis METHODOLOGY
Evaluation of PTC Program Management Plan
Assessment of Numerous Program artifacts, Program staff interviews, PTC Program organization charts, and observations – all constitute the “current state”
Standards used :
A Guide to the Project Management Body of Knowledge (PMBOK) – Fifth edition (with emphasis on the ten Knowledge Areas)
The Standard for Program Management – Third Edition
7 icfi.com/transportation © 2013
FRA§ 633.25 PMP Content Requirements are addressed by PMI’s PMBOK Knowledge Areas
METHODOLOGY
PMBOK Knowledge Area § 633.25 PMP Content Requirement
Integration Management
(a) description of adequate recipient staff organization, complete with well-defined reporting relationships, statements of functional responsibilities, job descriptions, and job qualifications
(d) document control procedure and recordkeeping system
(i) plan for internal reporting requirements including cost and schedule control procedures
Scope Management (e) change order procedure which includes a documented, systematic approach to the handling of construction change orders
Cost Management (b) budget covering the project management organization, appropriate consultants, property acquisition, utility
relocation, systems demonstration staff, audits, and such miscellaneous costs as the recipient may be prepared to justify
Schedule Management (c) construction schedule
Risk Management Not required (but recommended, by the assessment team)
Quality Management
(g) quality control and quality assurance programs which define functions, procedures, and responsibilities for construction and for system installation and integration of system components
(h) material testing policies and procedures
(j) criteria and procedures to be used for testing the operational system or its major components
Human Resource Management (f) description of organizational structures, management skills, and staffing levels required throughout the construction phase
Procurement Management Not required (but recommended, by the assessment team)
Stakeholder Management Not required (but recommended, by the assessment team)
Communications Management Not required (but recommended, by the assessment team)
8 icfi.com/transportation © 2013
Rating METHODOLOGY
The color scale reflects the degree of compliance with (i.e. all, most, some, few) the processes, artifacts and tools prescribed within the PMBOK knowledge areas
All
Most
Some
Few
PMP Current State
Assesses the PMP against PMBOK Knowledge Areas
Assesses the Program’s “current state” processes, artifacts and activities against PMBOK Knowledge Areas
Our color coded scale provides a rating for each knowledge area:
Ratings are provided for the PMP and the “current state”
9 icfi.com/transportation © 2013 9
Gap Analysis Results
10 icfi.com/transportation © 2013
“… includes the activities needed to identify, define, combine, unify, and coordinate multiple components within the program.”
Integration Management GAP ANALYSIS RESULTS
Rating: PMP Current State
Analysis Identified Gaps Several key integration-related artifacts exist
– PMP, last revised on March 22, 2012 – Project charter – Organizational charts – Program management information systems
Recent project-level standard processes,
templates, and reporting requirements have been implemented and are contributing to integration management
Inconsistencies between the “current state” organizational structure and the structure defined in the PMP
Documented process and tools that integrate Program-level scope, schedule, schedule and budget (i.e., no Integrated Master Schedule, Program WBS, or time-phased budget)
Many current state activities do not follow processes documented in the PMP
Process for maintaining the PMP
Process for program-level monitoring and control
Program Charter lacks clear articulation of the purpose, scope, budget, schedule and roles/responsibilities
Process for integrated change control (scope, schedule, cost, requirements, system configuration
11 icfi.com/transportation © 2013
“… defines the work required to deliver a benefit (major product, service, or result with specified features and functions) [and] includes all of the activities involved in planning and managing the program’s scope.”
Scope Management GAP ANALYSIS RESULTS
Analysis Identified Gaps System requirements for PTC are provided in
49 Code of Federal Regulation (CFR) § 236.1005 – Requirements for Positive Train Control systems – Additional due diligence is required to ensure
PTC Program requirements are compliant with this CFR
The PMP contains elements of scope
management including, but are delegated to HTI: – Contract WBS – Scope change control process – Requirements management and traceability
A scope statement that defines the entire PTC Program – (note: an “original Scope Statement” was
referenced in the materials reviewed during the Discovery subtask, but not found)
A Program-level Work Breakdown Structure (WBS) to include ALL Program work (vendor-implemented and NCTD-implemented projects and tasks)
Control of Program scope
Program-level requirements management and traceability for all PTC Program projects
Rating: PMP Current State
12 icfi.com/transportation © 2013
“… identifying the program’s financial sources and resources, integrating the budgets of the program components, developing the overall budget for the program, and controlling costs throughout the duration of both the components and the program.”
Cost Management GAP ANALYSIS RESULTS
Analysis Identified Gaps Total program budget of $87.3M is funded
through nine sources
The program’s contingency fund decreased by 70% during the period from July 2011 through January 2013. – It is not clear whether the remaining
contingency funds will be sufficient
There are differences in the budget management practices used by the Program and the NCTD Finance Division, resulting in at least $21.2M (~24% of the total program budget) of funds that cannot be currently reconciled by the Finance Division
Program cost management processes do not reconcile with organizational cost management practices (15 accounts versus 5 accounts)
Post-implementation operations and maintenance cost estimates have not been documented - current estimates appear to be anecdotal and not analytically based
Cost management process
Rating: PMP Current State
13 icfi.com/transportation © 2013
Rating: PMP Current State
“… includes determining the order in which the individual components are to be implemented, the roadmap for the program, and the milestones to be measured to keep the overall program on track and within defined constraints.”
Time Management GAP ANALYSIS RESULTS
Analysis Identified Gaps Activities of individual contracts and certain
Program elements appear in multiple schedules
Project Managers are assigned to monitor contractor progress against project-level schedule baselines
The PMP defines an approach to schedule management using a three-level hierarchy, each with a different level of detail - NCTD is not executing the approach
A Program-level schedule baseline
A single integrated master schedule that encompasses ALL Program activities Dependencies Predecessor/successor relationships Critical Path Schedule hierarchy
Schedule management process
Objective basis for NCTD’s ability to achieve
the December 31, 2015 completion date
14 icfi.com/transportation © 2013
Rating: PMP Current State
“… includes the processes of conducting risk identification, analysis, and response planning in order to reduce the likelihood of occurrence and/or impact an individual, or series of risks, have on a program.”
Risk Management GAP ANALYSIS RESULTS
Analysis Identified Gaps There are currently four versions of risk
registers within the program, some more current than others
Risk management discussion in the PMP and the Program Risk Management Plan have major inconsistencies
Project-level risk management plans and processes have recently been implemented
A documented risk management process to ensure risk management consistency within all Program elements
A single integrated risk register
Risks are currently assessed and prioritized using only qualitative measures
15 icfi.com/transportation © 2013
“… includes the activities of the performing organization that determines program quality policies, objectives, and responsibilities so that the program will be successful.”
Quality Management GAP ANALYSIS RESULTS
Analysis Identified Gaps Current quality management processes in the
PMP are specific to the HTI contract
NCTD is required to document “a complete description of the safety assessment and Verification and Validation processes applied to the PTC system” in an FRA-mandated PTC Safety Plan (PTCSP) prior to FRA-testing of the system – Rail Safety Consulting LLC is developing
NCTD’s PTCSP
Program-level quality management processes as part of a Quality Management System
Clarity as to who is accountable within the PTC Program or NCTD organizations for quality management
In-house staff accountability and capability for program quality management
Rating: PMP Current State
16 icfi.com/transportation © 2013
“… includes the process that organize, manage, and lead the project team—comprised of the people with assigned roles and responsibilities for completing the project.”
Human Resource Management GAP ANALYSIS RESULTS
Analysis Identified Gaps The program manager has authority to assign
priorities, apply resources, and direct the work of individuals assigned to the project
The PTC Program organization is largely geared to oversee the work of contractors and consultants – is this precluding attention to internal work elements?
Five Project Managers oversee 15 projects (contracts)
The organizational chart in the PMP does not match 1) the associated narrative description or 2) NCTD’s organizational chart
The PMP’s listing of contractor and consultant resources is incomplete and does not match the contents of the PTC Vendor List
Job skills, experience, and qualifications of personnel fulfilling positions within the PTC program are not documented – This makes it difficult to assess whether the
PTC Program has been appropriately and adequately staffed
Rating: PMP Current State
17 icfi.com/transportation © 2013
“… addresses specific procurement needs that are unique to managing the overall program and the needs of the constituent projects/components.”
Procurement Management GAP ANALYSIS RESULTS
Analysis Identified Gaps An NCTD Contract and Procurement
representative is ‘matrixed’ to the program and provides contracting/procurement support
Procurement management processes are integrated with, and leverage, NCTD’s organizational procurement capabilities
Processes associated with formally closing out each contract after all deliverables and payments have been made are not defined
The PTC Vendor List does not include “all vendors and suppliers of each PTC Program system, subsystem, component, and associated product, and process in use system-wide.”
Rating: PMP Current State
18 icfi.com/transportation © 2013
“… includes the activities associated with identifying, categorizing, and tracking all stakeholders who will interact with the program or be affected by the implementation of the program.”
Stakeholder Management GAP ANALYSIS RESULTS
Analysis Identified Gaps PMP identifies three stakeholder groups:
– Communities – Media Outlets – Government Entities
Preliminary analyses conducted by NCTD’s
Government Affairs and Communications Division identified 15 stakeholder groups in three categories: – Internal – General Public – Targeted
A dedicated PMP section for stakeholder management
A complete list of stakeholders to manage
Coordination and collaboration between the PTC program and NCTD’s Government Affairs and Communications Division—this does not appear to take full advantage of the agency’s capabilities
Rating: PMP Current State
19 icfi.com/transportation © 2013
Rating: PMP Current State
“… includes the activities for facilitating timely and appropriate generation, collection, distribution, storage, retrieval, and ultimate disposition of program information.”
Communications Management GAP ANALYSIS RESULTS
Analysis Identified Gaps The approach outlined in the PMP is largely
focused towards external communications
NCTD has a robust internal communications capability that has developed key communication-related elements regarding PTC
Stakeholder management and communication is a critical aspect of the PTC Program and its success
The PMP fails to address how communications will be managed and disseminated to internal stakeholders regarding the status of the program
Lack of internal communication, resulting in an insufficient level of understanding and engagement within the PTC Program among project team members
20 icfi.com/transportation © 2013 20
Prioritized based on expected benefit to the NCTD organization
Recommendations for Improvement
21 icfi.com/transportation © 2013
Recommended Improvements RECOMMENDATIONS FOR IMPROVEMENT
Baseline the Program
Enhance program control
Re-align program organization
Implement a coordinated stakeholder strategy
Dedicate resources to prepare for post-implementation operations and maintenance
2
1
3
4
5
22 icfi.com/transportation © 2013
Recommended Improvements RECOMMENDATIONS FOR IMPROVEMENT
Scope: Identify all work that’s required to be accomplished by the PTC Program and develop an associated program-level WBS
Cost: Reconcile current cost estimates and resource requirements with the newly developed WBS; develop risk-based budget contingencies
Schedule: Develop a program-level integrated master schedule that encompasses all currently-identified tasks
23 icfi.com/transportation © 2013
Recommended Improvements RECOMMENDATIONS FOR IMPROVEMENT
Evaluate and implement a program management information system sufficient for the size and complexity of the PTC Program
Review and update the Risk Management plan (e.g. implement an integrated master risk register that includes quantitative analyses)
Revise the PTC’s Quality Management plan so it aligns with the future PTCSP and newly established Quality function within NCTD
Identify and define configuration management requirements specific to the PTC Program
Implement program-level variance analysis and reporting mechanisms
Implement a process for maintaining and updating the PMP for the duration of the program
24 icfi.com/transportation © 2013
Recommended Improvements RECOMMENDATIONS FOR IMPROVEMENT
Strengthen the organizational relationship between the PTC Program and NCTD’s Project Management Oversight Office, Capital Projects
Establish an independent Quality management capability within NCTD
Formalize cross-organizational representation within PTC Program
Establish formal job descriptions that include skills, experience, and qualification requirements for each role within the PTC Program organization
Reissue an updated Program Charter to all NCTD staff that clearly highlights the purpose, objectives, governance structure, and roadmap for the PTC Program
25 icfi.com/transportation © 2013
Recommended Improvements RECOMMENDATIONS FOR IMPROVEMENT
Review and expand upon the stakeholder analysis currently drafted by NCTD’s Government Affairs and Communications Division
Implement a coordinated internal and external communications strategy based on the updated stakeholder analysis
Explore opportunities to link the PTC outreach strategy with Board of Directors and external project champions engagement
26 icfi.com/transportation © 2013
Recommended Improvements RECOMMENDATIONS FOR IMPROVEMENT
Document on-going maintenance requirements for PTC-related components and ensure adequate maintenance plans are developed, implemented, and the appropriate personnel trained
Identify additional organizational training needs, for example: – Locomotive operators: work with Herzog to understand and document initial and
recurrent training plans – Maintenance of Way: determine which PTC-related components require
familiarity with and understanding of by MOW personnel
Plan, document, and integrate newly implemented PTC-related components into NCTD’s configuration management capability
Develop and document one-time, on-going, and incremental operating costs related to PTC and implement processes for reviewing and revising these costs (and assumptions) on a regular basis.
27 icfi.com/transportation © 2013 27
Next Steps
28 icfi.com/transportation © 2013
Plan of Action NEXT STEPS
To include: – Implementation plan for the recommendations identified in this task – Current and proposed project management organization chart
29 icfi.com/transportation © 2013