using measurements of contractor performance to make informed decisions regarding services 6th...

Using Measurements of Contractor Performance to make Informed Decisions

Regarding Services

6th Annual Strategic Facilities Management ConferenceConferenz, Auckland March 23, 2009

Garry LawLaw Associates Ltd Consulting Engineers

PO Box 87311 Meadowbank Auckland – 09 520 2152 - 0275 665764

Garry Law: Performance Measurement 2

Road Map

Using measurements of contractor performance to make informed decisions regarding services rendered.

Quantifiably measuring contractor performance is not straight forward; yet doing so yields authoritative data upon which to base decisions on regarding your contractor’s performance.

This session will reveal how best to link KPIs, Service Level Agreement and contracts to quantifiably determine your contractor’s performance.

– Extracting metrics from their performance

– Best processes for interlinking KPIs, SLAs and contracts

– Examples and guidelines for using the metrics yielded to make decisions regarding contractors


Why Measure?

Truism – What you can’t measure you can’t manage

But – not everything is measurable

Relationships are hard to quantify: Tolstoy, Anna Karenina: “Happy families are all alike. Unhappy ones are each miserable in their own way"

Don’t expect the sum of a series of measures to capture the whole relationship


Working on the relationship is not just about reviewing the performance metrics

They can be a good place to start – they should be an important part of a periodic review meeting – but they are not the whole part.

Relationships are two way – not one scoring the other A breech of duty / trust / confidence is better directly

addressed than taken into some subjective performance score

Relationships


Why Use KPIs?

The objective in service contracts is not just cost management

Want reliability / quality / service / information On one-off work paid on time and materials - Want

productivity Want the service aligned to your broad needs Multiple objectives are the norm May link performance to rewards


Setting KPIs

Need to start with your organisation’s objectives (Key Results Areas)

Only some of these will involve the contract You will have some organisational KPIs. Some will

measure broad things including the relevant service. To get organisational objectives into a useful form,

cascade the objectives to the relevant level and set a KPI on that.

If the contractor can’t control most of what influences them they can’t be used.


Cascading ObjectivesLevel Objective KPI

Corporate Increase EVA > +1% for year 2009/10

Manufacturing Increase asset utilisation and free assets for sale

Output / asset value employed + 5% by Dec 2009

Plant xyz Free old widget line for sale

Sold by Jan 2010

Service Contract at Plant xyz

Increase plant availability

New widget line availability > 95%

The objectives are hierarchically linked - the KPIs relate to objectives not each other


Sorts of KPIs “Milestone” Achieve something by a date.e.g. Populate the Computerised Maintenance

Management System by July 30th 2011

“Measure” Meet a target on a defined measurement.

e.g. Measure: Rework rate on class 3 tasks Target: < 5%

A service contract will most likely have both.


Good KPI Characteristics

Output oriented Measured for a long time Have external benchmarks to set good performance Contractor has primary influence over achievement Close to real time (contractor can act to correct) Contractor can measure (but with auditability) Balanced set across the scope of the contract Limited natural variation Not too many!


Output oriented

The client’s need’s are outputs not inputs The contractor’s inputs do not guarantee the

outputs will be achieved

Leave the “how” to the contractor – that’s their job not yours


Measured for a long time

- If Yes - Is it still relevant? –or is it an ingrained habit?

- If No – its not fatal– Look at what other people have been measuring (they may

have benchmarks as well)– If its entirely new – do a slow start – don’t make it central /

vital / all important until you have a history of knowing how to measure – knowing what makes it better or worse and know what a “good” score is.


Have external benchmarks to set good performance

Great if you have these But the compromise will often be you have to

measure them in exactly the same way that others do (sample size / frequency etc.)

The price of sharing BM information may be revealing your own performance (not always a bad thing, but don’t do it unthinkingly)


Close to real time (so the contractor can act to correct) If you undertake surveys on a long period

interval the contractor will be in the dark as to their performance for much of the time.

If your results have some sort of delay from when they were collected the contractor will be likely not to give them much credence


Contractor can measure (but with auditability)

If the client does all the measurement it can lead to problems:

Master / slave relationships / “Gotcha” approach from client Contractor duplicating measurements in their

own defence and disputing the client’s measurements


Limited Natural Variation Common Causes - variations from the contractor’s

systems - manageable by them)

Special Causes - variations external to the contractor’s systems (but will include variations caused by the client)

(W E Deming on Quality)

Using alliance methods may allow client side issues to be worked on as well.


Balanced

Akin to the ‘balanced score card’ within an undertaking

Some examples:


Roading Operations and Maintenance Contract

(Extra work and extension reward)12 KPIs (Monthly to Annual)Groups: Communications and Customer Services and

Emergency Response (20%) Health, Safety, Quality & Environmental (20%) Operations, Physical Works (40%) Technical Support Services (20%)

Hurdle scores for extra work and extensions


Water/ Wastewater / Stormwater Operations and Maintenance Contract

(Extra work and extension reward)15 KPIs (Monthly to Annual)Groups: Communications, Customer Care, Relationship &

Emergency Management (20%) Health & Safety, Quality, Environmental (20%) Operations, Physical Works (40% ,) Reporting, Management of Assets, Innovation (20%)

Hurdle scores for extra work and extensions


Wastewater Treatment Plant Maintenance Contract

(Sum at risk penalty)9 KPIs (Monthly to annual)Groups: Quality 70% Reporting 20% Health and Safety 10%

Sum paid proportional to scores with range: awful / slight stretch


Alliance Design Build Contract, Wastewater Treatment and Recycling Plant

Construction(Quality pool reward / penalty)12 KPIs (Monthly to whole of term)Groups: Acceptance / commissioning 25% Environmental compliance 15% Operating costs 30% Legacy 30%

Sum paid + proportional to scores with range: awful / BAU / stretch


Managing Contract, Water / Wastewater Reticulation Design / Construction

(Quality pool reward / penalty)12 KPIs (Monthly to annual)Groups: Environmental outcomes 30% Stakeholder management 30% Legacy / quality 40%

Sum paid + proportional to scores with range: awful / BAU / stretch


Mathematically:Mathematically there are different sorts of measurement scales:

Non- Parametric: Nominal scales: milestones, pass/fail, yes/no, on/off, true/false, counts by

classes Ordinal scales: ranks, counts by ranks Interval scales: defined units but no natural zero, differences are real but ratios

notParametric: Ratio scales: defined units with a natural zero, ratios are real and unit-

independent

Engineers tend to be happiest with the last – but the others are perfectly respectable (and have their own branch of statistics)


Defining KPIs Units Time window looked at What counts Who measures When reported Exceptions management Who can correct errors

A methodology datasheet per KPI is good practice


Linking KPIs to Rewards / Penalties - 1

Termination Extra work within the contract term Extending the contract term Renewing the contract Performance payment incentives / disincentives


Linking KPIs to Rewards / Penalties - 2

Use all the KPIs? – Probably not Extra work within the contract term? – may not

be an incentive if there are better opportunities elsewhere

Extending the contract term? – ditto, May be a distant reward at the beginning, in the final term extension the incentive is lost

Renewing? – Can only be one factor - a distant reward at the beginning

Performance Incentive

Reward is: Fixed fee for management / overheads Fixed fee for specified scope and / or Variable fee (time and materials) for

unpredictable items “At risk” element variable on performance


KPIs linked to Payment: Sum at Risk Method

The at-risk element may be a fixed sum or a percentage of the base payment, or the profit revealed– If the client sets this too large the contractor will manage risk

by bidding up the base payment The actual payment is the “at risk” element factored by an overall

performance measure The overall performance measure is made up of a “basket” of

performances on KPIs Full reward is at a “Stretch Target” level for the KPIs. Zero reward is an abysmal performance level


KPIs linked to Payment: Quality Pool Method

The at-risk element is a fixed sum set by the client The actual payment is the pool factored by an overall

performance measure, additional if the performance score is favourable, a deduction if unfavourable

The overall performance measure is made up of a “basket” of performances on KPIs

“No payment” level is performance at a “Business as Usual” level

Full positive reward is at a “Stretch Target” level for the KPIs. Full negative penalty is an abysmal performance level


KPIs linked to Payment

Why offer more? (i.e. the quality pool method)

In a competitive environment it in fact makes little difference – the contractors will tailor their bid around the profit they are targeting, where they think they can perform on the KPIs and price accordingly


KPIs within the overall performance score Can use as “all – or – nothing” - appropriate

for measures where failure has an external risk – environmental prosecution or such. Must be achievable. Don’t use this on matters which may be cause for contract termination

Can be proportional. Set the upper and lower limits and pro rata in between

With a proportional linking review if you actually want better than ‘business as usual’ for a particular KPI – don’t do it unless it has value to you


Opportunities with Targets

Set them once measurement has run for a bit (on some - not too many)

Step them up – if way below a benchmark standard, get there progressively rather than pretend you will do it overnight.

Ratchet them – a new performance record becomes the new target (don’t do if there is much variation beyond the control of the Contractor i.e. Deming's special causes dominate )


Risks with KPIsRisk Mitigation Strategies

Disputes on KPI calculation - Use methodology datasheets- Contractor self-production

External measure discontinued Minimise use of external measures

Measure unavailable at start-up Allow full score for start up period

Measure unavailability at payment Incentive: Score zero for Contractor generated, full for Principal generated, no back calculation.

Contractor mis-calculation Audit trail – keeping of records – and post term

Disputes over applicability – unforeseen circumstances

Have disputes procedure apply, with staged escalation if unresolved

Declining relevance of KPIs over time Have a periodic review – substitution by agreement.

Declining relevance of targets over time

Ratchet, re-evaluation against external bench marks with substitution by agreement.

Tenderers sceptical on achievability Have history available

Performance history treated as commercially sensitive

Contract must make clear will be available to tenderers in renewal


END Law Associates Ltd Consulting Engineers

PO Box 87311 Meadowbank Auckland – 09 520 2152 - 0275 665764

using measurements of contractor performance to make informed decisions regarding services 6th...

Documents

contractors performance

good performance contractor

performance measurement12

performance measurement11

garry law law associates

good performance great

performance best processes

contractor thats