performance metrics chris hamm operations director (acting) gsa fedsim
DESCRIPTION
Performance Metrics Chris Hamm Operations Director (acting) GSA FEDSIM. Quote PBSA Perspectives Contract Types Definitions Performance Metrics SLAs. Implementation Details Pre-Solicitation Solicitation Pre-Award Kickoff Post-Award Measurements Exceptions & Pitfalls. Topics. - PowerPoint PPT PresentationTRANSCRIPT
Federal Acquisition Service
U.S. General Services Administration
Performance Metrics
Chris HammOperations Director (acting)
GSA FEDSIM
Performance Metrics
Chris HammOperations Director (acting)
GSA FEDSIM
Federal Acquisition Service
2
Topics
Quote PBSA Perspectives Contract Types Definitions
Performance Metrics SLAs
Implementation Details Pre-Solicitation Solicitation Pre-Award Kickoff Post-Award
Measurements Exceptions & Pitfalls
Federal Acquisition Service
3
If you can’t measure it, you can’t manage it
Federal Acquisition Service
4
Performance Based Services Acquisition
FY08 – 50% of all eligible services acquisitions should be PBSA
FAR Subpart 37.6 (b) Performance-based contracts for services shall include—
(1) A performance work statement (PWS); (2) Measurable performance standards (i.e., in terms of quality,
timeliness, quantity, etc.) and the method of assessing contractor performance against performance standards; and
(3) Performance incentives where appropriate. When used, the performance incentives shall correspond to the performance standards set forth in the contract (see 16.402-2).
Federal Acquisition Service
5
Government Perspective
1. Compliance – We have to do it
2. Funding – Less $ to do more work
3. Cost – Best Value for the taxpayer
4. Performance – Use of commercial best practices
5. Schedule – Congressional & Executive mandated dates
But . . . PBSA is hard, especially when we are overwhelmed withwork and are starting with the old SOW
Federal Acquisition Service
6
Contractor Perspective
Profitability – Companies are in business to make money first. Project Success – Delivering success typically results in more
follow-on business Cash Flow – Regular payments and meeting corporate
profitability is essential for publicly traded companies. Missing deliveries and failing SLAs may result in lower stock prices and replacement of personnel
Personal Services – Use of performance metrics reduces the ability for the Government to dictate the solution
Federal Acquisition Service
7
Cost Reimburseable to Fixed Price continuumResearch Development
Government assumes
more cost riskContractor assumes
more cost risk
Production/Sustainment
As you move to the right, the more detailed and meaningful the performance measures become. . . .
Higher risk, less-defined requirements
CPFF CPFF / CPAF CPIF/CPAF
R&D Development
FPAF/FPIF/ FFP FPIF /FFPEnhancements / Production Operations &
Maintenance
Lower Risk, well-defined requirements
Federal Acquisition Service
8
Definitions
Performance Metric A measure collected by the Contractor that may or
may not have financial incentives & disincentives
SLA – Service Level Agreement Part of a contract between the Government and
the Contractor to provide services reliably and at a specified performance level. SLAs have financial incentives and/or disincentives
FYI: There are lots of other names, like performance goals or SLOs – Service Level Objectives.
Federal Acquisition Service
9
Performance Metrics are. . . .
1. A communications tool. The value of an agreement is not just in the final product; the very process of establishing metrics helps to open up communications.
2. A conflict-prevention tool. An agreement helps to avoid or alleviate disputes by providing a shared understanding of needs and priorities. And if conflicts do occur, they tend to be resolved more readily and with less gnashing of teeth.
3. A living document. This is one of its most important benefits. The agreement isn't a dead-end document consigned to the Forget Forever file. On a predetermined frequency, the parties to the review the metric to assess service adequacy and negotiate adjustments.
4. An objective basis for gauging service effectiveness. An metric ensures that all parties use the same criteria to evaluate service quality.
Federal Acquisition Service
10
Definitions
Cost – The fully loaded cost of performing the service (DCAA cost accounting system). This comes into play on CPAF & CPIF, but does not apply to FFP types
Base Fee – Some Incentive & Award Fees have two components. The Base Fee is like percentage of fee that is automatically earned, while the rest is part of the award/incentive fee pool. Contractors like base fee for two reasons: (1) They automatically earn it, and (2) it helps their cash flow. Otherwise, fee isn’t earned until the end of the period
Rollover – If any fee is not earned at the end of a period, it can be lost or “rolled over”.
Federal Acquisition Service
11
How GSA FEDSIM implements Incentives
Primarily, we have SLAs on our larger task orders There are about 30 active projects with SLAs
Most incentives at FEDSIM are part of task orders with hybrid contract types (CPIF & CPAF) with a few notable exceptions (FFP)
SLAs tend to be implemented more on our Operations & Maintenance projects, but also exist on Development projects
Federal Acquisition Service
12
Examples of SLAs
• Network Availability• Server Availability• Circuit Installation• Move / Add / Change
• called a MAC• Trouble Ticket
• Tier 1/2/3/4• Premium / Standard
• Call Center• 1st Call Resolution• Speed of Answer• Abandonment Rate
• Customer Satisfaction• Cost
• CPI from EVMS• Budget / Savings
• Schedule• SPI from EVMS
• Small Business Subcontracting• Time to fill vacancies• Backups• COOP• Invoicing Accuracy
Federal Acquisition Service
13
Examples of Measurements
Percentages See Chart
Hours Indicate normal business
hours for response! Days
Always clarify business vs. calendar days!
Yes / No
Measure Downtime in one year
99% 3.7 days
99.90% 8.8 hours
99.99% 53 minutes
99.999% 5.3 minutes
Most clients do not need Five 9s and could not afford it if they wanted it. The cost is prohibitive!NEW EXCEPTION - CLOUD availability
Federal Acquisition Service
14
Philosophy
Why have incentives? Setting performance standards, measuring performance,
and understanding cost impacts is difficult. Without SLAs, all actions are inherently subjective
Incentives improve project performance only if contract administration and quality assurance functions are implemented. Otherwise, they just add cost.
SLAs should be “end to end” as much as possible. Don’t measure email server availability if you can measure
the time from when an email is sent to when it is received
Federal Acquisition Service
15
Philosophy
SLAs should align with the client and project objective AND motivate good contractor behavior Do not design an SLA that, when missed, motivates the
contractor to give up on that measure for the rest of the period
– Example: Doing an “average” over a six month period. If the contractor fails in the 1st month, there is no incentive to maintain service levels or improve over time.
Just because something can be measured doesn’t mean it should be an SLA Example: It is easy to measure how fast someone picks up a
phone, but a better measure is how long it takes to actually solve the problem.
Federal Acquisition Service
16
Process (Pre-Award)
• Draft Incentive/ Award Fee Plan and SLA Format included in Section J• Evaluation Decision:
• Option 1: Specify SLAs & have all vendors bid the same• Option 2: Let vendors chose SLAs & targets
• Review requirements with the client before release and see if tasks are sufficiently defined to create notional SLAs to include as recommendations
• Remember that the Government’s strongest negotiation position is pre-award.• Fee Plans are unilateral before the period begins, bi-lateral
during the period (if the CO allows any changes at all)
Option # 2 is harder to evaluate but better post award: Better measures, more leverage after award.
Federal Acquisition Service
17
Process (immediate Post-Award)
Ensure Contract Award includes any contractor proposed SLAs and/or changes from Negotiation
Set the percentage of award fee pool that is SLA based Set up separate meeting from Kickoff Meeting with key
stakeholders to review SLAs. Ensure that all stakeholders agree to the SLAs Set the “weight” of each SLA Make sure the SLA is entirely within the control of the
contractor (no mixed responsibility) Add further definition of the SLA and all exclusions
Federal Acquisition Service
18
Process (immediate Post-Award)
Develop the methodology of how to measure performance Periods (monthly, quarterly, bi-annually) Set Levels (Stretch, primary, secondary, tertiary) Set Measurement Scale with Incentive / Disincentive
– For Example: Stretch: 100% + rollover Primary: 100% of available fee Secondary: 90% Tertiary: 75% Failure: 0 % Fee
Not all levels are required. Some only have one target. It depends on the client requirements.
Federal Acquisition Service
19
Process (immediate Post-Award)
Award Fee Plan scale:
– Outstanding (95-100)– Excellent (90-95)– Good (80-90)– Average (70-80)– Poor (0-70)
SLA Measurement Scale:
– Outstanding 99.99+
– Excellent 99.9 – 99.99
– Good 99 – 99.9
– Average 98 – 99
– Poor >98
Federal Acquisition Service
20
Process (a few months post award)
Hold a regular review of the metrics to ensure that the levels are set correctly
Create a dashboard for simplified reporting to executive management
Set up a quality assurance function to check the contractor’s self reporting metrics
Insist on access to the raw data from the system tools used to generate the SLA reports
Review all trends and focus on failures Root cause analysis can identify process issues
Federal Acquisition Service
21
Process (a few months post award)
Set a structure for gradual improvement of SLAs over time
For particularly difficult SLAs to negotiate, consider benchmarking them for a period without making them SLAs to obtain buy-in and set the appropriate target
Gradually increase the overall weight of SLAs in the award fee plan overtime, with the goal of eliminating subjectivity as much as possible
Federal Acquisition Service
22
Exceptions
Scheduled Downtime Government Intervention in Contractor Response Other Contractor Intervention Acts of God / Force Majure
Make sure someone from the Government is aware of and approves all scheduled downtime or deviations from SLAs. No fox in the hen house!
Another benefit of outsourcing your infrastructure to the “cloud” . . . No more scheduled downtime!
Federal Acquisition Service
23
Measurement
• Recommend use of Contractor owned automated tools• If not owned by the contractor and proposed at time of
award, be prepared to pay for the tool• Some clients have their own tools. Make sure the client can
extend the license to contractors• Use an IV&V contractor to sample the data from the contractor• Send the monthly results of the SLAs to stakeholders so there
are no surprises when it comes time to pay the contractor
Federal Acquisition Service
24
Structure & Rollover
Design your SLAs to motivate improved performance on SLAs overtime.
If one SLAs is missed during a period, put some rollover on the table to improve the performance against that measure for the future and earn back the lost fee. That will motivate the contractor to achieve a higher standard for the remaining period and result in better service to your client. Example: If a contractor misses a measure in Period 1, put
some rollover towards that measure (and that measure only) in Period 2 if they meet a stretch goal. You can build this into your award fee plan’s structure to happen automatically
Note: This is not always appropriate. Some failures need to have financial disincentives to change behavior.
Federal Acquisition Service
25
Pitfalls
Having too many measures Example: Too many SLAs, some of which were redundant
and worth less than 1% of the pool Example: Aligning SLAs to Government staff. At least one
SLA (and sometimes more) for each of 25 tasks to appease stakeholders
Targets Just like performance reviews, if you set a target, the
contractor will plan and staff to meet that target. If the target is an average, make sure that they plan to make the average, not to always beat the target.
Federal Acquisition Service
26
Pitfalls
Funding If the SLAs has an incentive, that amount must be fully
funded to prevent any anti-deficiency situation– This is what makes really good incentives difficult level of
oversight
Oversight Clients and Contractor may initially balk at the level of effort
required to stand up a SLA management process
Federal Acquisition Service
27
Pitfalls
Reliance on Industry Standards Benchmarks are great, but that does not mean they are right
for your client’s infrastructure. Just because Gartner says five 9s is normal for a call center doesn’t mean that your requirement needs that.
Customer Satisfaction This one is difficult. It will take time to reach agreement on
what scale to use (1-5, Y/N, etc), what survey to use, how to count the no-response pool.
At the end of the day, customer satisfaction may be the most important measure but the hardest to get right
Federal Acquisition Service
28
Pitfalls Tendency to award very high scores on Award fee (Government wide)
94.6%
93.8%
96.1%
95.7%
92.5%
93.0%
93.5%
94.0%
94.5%
95.0%
95.5%
96.0%
96.5%
FY02 FY03 FY04 FY05
% Awarded on DOD
Federal Acquisition Service
29
Questions