aimee rensford march 2013

Outcomes-Focussed Contracting: Accountability,

learning, and new approaches to

integrated contracting

Aimee RensfordMarch 2013

“In the public sector, proposals are created and evaluated with the expectation that every one of

them will be a great success – that anything short of completely solving a problem is a failure.

Creating a civil culture in which people understand that things seldom work as expected,

especially when trying to solve complex, intractable problems, would support a shift

toward learning. This shift is not to avoid accountability but rather to make learning a focus

of meaningful accountability” (Westley, Zimmerman, & Patton, 2006)

Outline

• Key findings from my research into the dynamics of accountability and learning in contracting for health promotion services

• How do we foster ‘meaningful accountability’?– 3 Principles to guide us

• Integrated contracting – the application of ‘Results-Based Accountability’

• Recent developments with Integrated Contracting – Streamlined Contracting with NGOs Pilot

Contracts• Link financial resources to health service outputs

and outcomes

• Specify the nature of the performance required• Create an accountability relationship where

providers are required to explain and take responsibility for their actions (or inaction)

Model of Accountability

Key questions

• Classical or relational contracts?• What was the funder/provider

relationship like?• Do the accountability mechanisms

support providers to learn about and improve what they do?

Classical Contracts

• So called ‘complete’ contracts – specification of promises, obligations, and processes for dispute resolution is highly specified

• Costly to write and enforce (leading to high transaction costs)

• Particularly hard for HP – neatly defined deliverables are inflexible and not suitable for ongoing programmes that may need to adapt over time

Relational Contracts• Provide flexibility and easy flow of information

within a trusting contract relationship• Preferable when some elements of a service are

unverifiable or hard to measure• Strong relationship between funder and

provider is critical (but also leads to high transaction costs)

• Balance between trust and accountability

Contracting relationship theory: Agency theory

• Foundational component of NZ Public Sector reforms of the 1990s

• Assumes self interest and that agents (providers) will act opportunistically and shirk as much as possible

• External monitoring and reporting key to controlling providers and ensuring goal alignment

• Criticised for assumptions made about provider’s intentions

• Threat of sanctions typically comes after the service has been delivered

Stewardship Theory• Proposes that goals are aligned• Relationships built on trust, reputation, &

collective goals• Less emphasis on the validation of

performance measures – proposes that if providers are trusted they will tend to provide more accurate information

• Criticised for painting an overly ‘rosy’ picture of providers intentions

Findings: Contracts

• Service Specifications formed the basis for ongoing reporting, and included measures to assess performance

• Typically focused on outputs • Funders found it hard to develop ‘good’

performance measures (esp. in health promotion context)

• Flexibility to deviate from the service specs necessary

Provision of regular monitoring reports against SS

• So basically yeah, [providers] have service specs, which can be of varying quality, and they're accountable to deliver against those service specs (Funder)

• You report to the measures so it’s basically outputs like have you done this, have you done that, but there is an opportunity in there to write, to tell your story (Provider)

Reporting as a progress monitoring tool

One of the positive aspects about reporting is that it also makes sure that you keep things on track and that you are actually delivering what you agreed to, so it also gives, so it gives you a chance to reflect, if you like, on how things are going, what processes, etcetera, etcetera. So it’s a bit of a sort of a snap shot of where you’re at, so it’s really important (Provider)

If [the programme] veers away from what was originally decided but there’s a rationale for that and good reason then I think that’s fine as long as it’s being transparent and being discussed. So I don’t know that there’s just one way for something to end up working

(Funder)

Flexibility to deviate from SS

Classical or relational?Agency or stewardship?

• Strong focus on achievement of contracted outputs reflects a classical approach to contracting

• Yet the flexibility afforded in the management of contracts suggests a relational approach– Funders and providers showed a preference

towards partnership-based stewardship approaches

Findings: Learning and Evaluation

• Some NGOs incorporated evaluation into their programme planning processes but this was not across the board

• Role of the manager in provider organisations critical to establishing a learning culture

• Barriers to learning:• Difficult to measure behaviour change• Lack of time• Lack of funding for evaluation• Lack of emphasis on discussing failures• Lack of evaluation capacity

I think barriers to learning are often around the fact that we’ve got outputs we have to report and you can get very caught up in the operational side of programming and ‘cause you're so busy and you’ve got to get your reports done, you’ve got to get some outputs happening and then you don’t perhaps think about the bigger picture to learn from what you’re doing

(Provider)

I don’t think that as providers of nutrition and physical activity we’ve really latched on to continually improving. You get something going and then you keep it ticking on would be my impression. I don’t think we’ve quite got there with that. Like I mean for example no one seems to report failures and there must be, I’ve never been to a nutrition conference or anything where someone’s stood up and said actually don’t, this process or programme didn’t work

(Funder)

The relationship between Accountability and Learning

• There was clear consensus that the accountability mechanisms such as reporting, while useful for monitoring progress, were not enough to ensure that learning and improvement is taking place

• Both funders and providers looked outside of the reports to judge whether providers are ‘doing a good job’

• The degree to which learning and evaluation is integrated into service delivery depends heavily on providers and/or funders motivations to include it

• The accountability mechanisms are not set up in a way that requires providers to report on learning and evaluation findings

• This contributes to a de-prioritisation of learning and evaluation

So what? Why do anything about it?

• Both funders and providers identified the need to integrate accountability for learning into the reporting process to ensure effectiveness and efficiency of public health services to achieve health outcomes

• The quality of information used to assess performance is critical given the role it should play in the management of contracts, and in the policy cycle

• A key purpose of accountability is the promotion of learning in pursuit of continuous improvement (Aucoin & Hentzman, 2000)

Where do we start..?

Principle 1

• Work together to include formal expectations for reporting on learning and evaluation in the contract service specifications

• Helps to ‘legitimise’ time spent on learning, and helps to make it a priority from the beginning of the contracting process

Principle 2

• Support evaluation capacity building strategies for both funders and providers• This will help to improve the quality of the measures used to

assess performance

Examples of ECB Strategies:• Training: attending courses, workshops, and seminars on evaluation• Coaching and mentoring: building a relationship with an evaluation expert who

provides individualised technical and professional support • Involvement in evaluation process: participating in all stages • Technical assistance: receiving help from an internal or external evaluator• Appreciative inquiry: using strengths-based, collaborative, narrative approach to

learning that focuses on what ‘gives life’ to a programme

Principle 3

• Integrate learning and evaluation in a way that supports a ‘stewardship’ approach to contract management

• Strong, trust-based relationship is critical• “In situations where trust is present the trusted tend to

disclose more accurate, relevant and complete information, whereas the trusting feels less need to impose social controls in order to gain access to or influence over information” (Swift, 2001)

Further considerations

• Providers have multiple accountability requirements and need to streamline these where possible

• Realities of contracting timeframes, short term contracts

• Increasing pressure on planning and funding teams and reduced FTE to manage contracts

Integrated Contracting across Govt

• Longer term agreement • Common set of terms and conditions across

agencies• Ideally brings together all funding that an NGO is

receiving from any government funder• Provider and all of their funders around the

table • Results Based Accountability approach to

developing the contract and reporting template

RBA Refresher

• 2 key aspects: – Population level accountability

• High-level outcomes e.g. adult smoking rate• Influenced by a plethora of external factors -NGO

can not be held accountable for impacting the outcomes by themselves

– Performance level accountability • What were the actual results achieved by those

who directly received your service? – How much did we do?– How well did we do it?– Was anyone better off?

Benefits of ICs• Ensures that reporting is not just on outputs but

also progress towards outcomes and the quality of service delivery

• Funders no longer working in isolation – better understanding of the other services delivered

• Opportunities for service integration where appropriate to support the achievement of common outcomes

• Opportunity to reduce/streamline compliance costs (e.g. audit)

Streamlined contracting with NGOs pilot

• Part of Better Public Services Demonstration project, lead by Ministry of Business Innovation & Employment

• Aims to create a contracting framework that will:– Reduce compliance burden on NGOs – Reduce duplication – in compliance costs and

contract management processes• Uses an RBA approach to linking the services provided by

an NGO to the Crown purchaser’s desired population and client specific outcomes

• Interested in measuring (among other things) the contribution the NGO makes to achieving outcomes

This may be in the form of high trust, integrated or a standard agreement,

as guided by the decision-support tool

RBA OUTCOMES

This may be in the form of high trust, integrated or a standard agreement,

as guided by the decision-support tool

RBA OUTCOMES

This may be in the form of high trust, integrated or a standard agreement,

as guided by the decision-support tool

RBA OUTCOMES

OUTCOME AGRMT

RBA OUTCOMES

OUTCOME AGREEMENT (OA)

Based on a standard form template that can be used to describe a high trust, integrated or standard

agreement, as guided by the decision-support tool

RBA OUTCOMES

Outcome Agreement

Management Plan (OAMP)

Outcome Agreement

Management Plan (OAMP)

Outcome Agreement

Management Plan (OAMP)

Outcome Agreement

Management Plan (OAMP)

Outcome Agreement

Management Plan (OAMP)

(optional non-contractual

management plan)

Outcome Agreement (5)Outcome Agreement (4)Outcome Agreement (3)Outcome Agreement (2)

Proposed contracting framework

DECISION-SUPPORT TOOL (DST)

Framework Terms and Conditions (FTC)

Standard terms and conditions governing the relationship between government and NGO providers

Framework Agreement Management PlanSpecifies the binding commitments between Government agencies when engaging an NGO Provider (collaboration, coordination & consistency). The provider is not party to this document.

Decision Support Tool (DST)

Assessment Area Score

Provider 5 4 3 2 1 0Historical performance in delivery of services (not necessarily these services) 3

0 0 0

Trust - the provider is accountable and responsible; they demonstrate integrity and can be relied upon to do what they say they will do

2

0 0 0 0

Financial soundness/viability of provider 20 0 0 0

Services Complexity of Services and strategic importance to agency 4

0 0Are the services new or standard and implementation challenges - is this something that has been done before and can be replicated in this environment with little risk?

1

0 0 0 0 0

Score Graph

scores showing are example only

Summary

• There are opportunities make the reporting process a driver of learning and service improvement

• Integrated contracts provide one way of doing this in a coordinated way – but can also be supported in a 1:1 traditional style contract

• MBIE model may be used to inform ICs going forward

Questions?

Aimee_Rensford@moh.govt.nz