benedict wauters 24-10-2012 esf com. twg -cyprus introduction to results based management and to the...

Benedict Wauters 24-10-2012ESF Com. TWG -Cyprus

Introduction to Results based Management and to the Maastricht conference

Overview

1. The COP RBM

2. Results based management: a brief

history

3. So what?

4. Introducing the Maastricht conference.

Roots of the COP

• EQUAL programme: working group (led by the EQUAL unit) on introducing PCM (Project Cycle Management) in the ESFPartnership development Toolkit - September 2005Conference on sound planning - November 2005Sourcebook and website on sound planning

- September 2006 (www.esfsourcebook.eu)

• Independent report “An Agenda for a Reformed Cohesion Policy” by F. Barca – May 2009

Who?• Managing Authority of Czech Human Resources and Employment Operational

Programme• Czech National Coordination Authority of the NSRF• Greek Managing Authority for “Health and welfare” OP and Intermediate Body for

“Development of Human Resources” OP• Lithuanian intermediate body for Operational Programme for the Development of

Human Resources OP and Promotion of Cohesion OP• Lithuanian Managing Authority• Managing Authority for the Italian Region of Molise Regional OP• Slovak ESF Managing Authority• Swedish ESF Managing Authority• Lithuanian ESF Agency• Managing Authority for the Walloon Convergence and Regional Competitiveness and

Employment Ops• Managing Authority of the Flemish obj. 2 OP (coordinator of the COP)• National support structure for the ESF for the Italian Regions (Tecnostruttura)• European Institute for Public Administration• Originally also: MA Italian Region of Piemonte and MA of Romania

Definition of results

• EC guidance document, November 2011*: The intended result is the specific dimension of well-being and

progress for people that motivates policy action, i.e. what is intended to be changed. The notion of change also comprises changes in behaviour, social practices, institutions etc.

Outputs are the direct products of programmes, they are intended to contribute to results.

• COP RBM: the essential difference between a result and an output (both as described above) is a shift of ownership from a producer to a user If it cannot be used, it is just an “output”: eg with a training it is

the knowledge that can be used, not the actual training

*MONITORING AND EVALUATION OF EUROPEAN COHESION POLICY - EUROPEAN REGIONAL DEVELOPMENT FUND AND COHESION FUND –CONCEPTS AND RECOMMENDATIONS.

The journey of the COP

Warsaw

Brussels (x2)

Warsaw

Vilnius

Prague

Rome

Stockholm

Athens

Core of the COP work : 8 interactive seminars (of 2-3 days) with experts from around the world (incl. US, Canada, Australia) in

debate with practitioners from our own programmes

Between seminars:

participating in global networks, going deeper into

issues derived from the meetings

Key outputs: • Definitions of Results• RBM baseline study• Practice descriptions• Huge online

knowledge base• RBM strategic

management and self-assessment system

Some key insights from the seminars-1• not clear what RBM really is, many tools and approaches seem relevant; no clear

and generally accepted definition what the word “results” actually means!• huge variety and selection of tools for mapping and analysing stakeholders when

setting up projects but this tends to obscure the much more important question: how do you involve them and keep them involved?

• how do you know who are stakeholders in a project? Ask the target group how they run into in real life!

• bottom-up vs. top down deployment of an ESF OP is very different• central to RBM is the “results chain”• RBM at organisational level is quite different from RBM at project level; for

organisations it is more like a a specific type of strategic planning and management system

• quality management (ISO) only helps “stabilise” and incrementally improve processes but does not itself provide a results orientation

• we are focused on changes that depend on long chains of cause and effect but tend not to look at what/who is/are in front of our noses (e.g. intermediariy organisations)

• Strategy maps / Balanced scorecards have potential for supporting RBM at organisational level and can in some way be linked to results chains

Some key insights from the seminars-1• not clear what RBM really is, many tools and approaches seem relevant; no clear

and generally accepted definition what the word “results” actually means!• huge variety and selection of tools for mapping and analysing stakeholders when

setting up projects but this tends to obscure the much more important question: how do you involve them and keep them involved?

• how do you know who are stakeholders in a project? Ask the target group how they run into in real life!

• bottom-up vs. top down deployment of an ESF OP is very different• central to RBM is the “results chain”• RBM at organisational level is quite different from RBM at project level; for

organisations it is more like a a specific type of strategic planning and management system

• quality management (ISO) only helps “stabilise” and incrementally improve processes but does not itself provide a results orientation

• we are focused on changes that depend on long chains of cause and effect but tend not to look at what/who is/are in front of our noses (e.g. intermediary organsations)

• Strategy maps / Balanced scorecards have potential for supporting RBM at organisational level and can in some way be linked to results chains

RBM at organisational level is quite

different from RBM at project level;

for organisations it is more like a a

specific type of strategic planning

and management system

We are focused on changes that

depend on long chains of cause and

effect but tend not to look at

what/who is/are in front of our noses

(e.g. intermediary organisations )

central to RBM is the “results chain”

Some key insights from the seminars -2• measurement via indicators and target setting on them tends to lead to perverse effects

when they have direct consequences; they should only be used as flashlights to illuminate an area that raises questions

• to be strategic implies that choices have to be made and these should become visible when an organisation (e.g. a Managing Authority) sets objectives; a BSC/strategy map can help-bottom up logic and top down indicator setting for aggregation clash with each other

• an indicator that is just measuring quantity (amounts) without “quality” (how good) is useless

• accountability should be linked to learning, not to indicators• linking money to outputs directly requires deep insight in the local situation and

sophisticated processes for preparation and follow-up: it is not an easy fix• human systems are all but stable and predictable. They are complex and adaptive . This

has huge implications for planning and evaluation.• counterfactual impact evaluation is to be used with great care as it assumes stable and

sufficient cause and effect relations which rarely applies in practice• longer “results chains” require different actions throughout the chain that assume an earlier

action already had its effect• importance of experimentation and adapation when faced with complexity; innovation is

always complex

Some key insights from the seminars -2• measurement via indicators and target setting on them tends to lead to perverse effects

when they have direct consequences; they should only be used as flashlights to illuminate an area that raises questions

• to be strategic implies that choices have to be made and these should become visible when an organisation (e.g. a Managing Authority) sets objectives; a BSC/strategy map can help-bottom up logic and top down indicator setting for aggregation clash with each other

• an indicator that is just measuring quantity (amounts) without “quality” (how good) is useless

• accountability should be linked to learning, not to indicators• linking money to outputs directly requires deep insight in the local situation and

sophisticated processes for preparation and follow-up: it is not an easy fix• human systems are all but stable and predictable. They are complex and adaptive . This

has huge implications for planning and evaluation.• counterfactual impact evaluation is to be used with great care as it assumes stable and

sufficient cause and effect relations which rarely applies in practice• longer “results chains” require different actions throughout the chain that assume an earlier

action already had its effect• importance of experimentation and adapation when faced with complexity; innovation is

always complex

measurement via indicators and target

setting on them tends to lead to perverse

effects when they have direct

consequences; they should only be used

as flashlights to illuminate an area that

raises questionsaccountability should be linked to

learning, not to indicators

human systems are all but stable and predictable. They are complex and

adaptive . This has huge implications for planning and evaluation.

• The COP RBM, to its surprise, found itself exposed to a huge variety of (sometimes conflicting) views…

• Who came up with this Results Based Management in the first place?

RBM is a child with many parents…

• MBO: management by objectives, launched by Peter Drucker in “the Practice of management” in 1954 It emphasised the learning aspect and discouraged

mechanical models (of “scientific” management) Key is adaptation of empowered employees to changing

circumstances and performance feed-back rather than control and predictability

MBO’s current most popular form is through Kaplan and Norton’s strategy focused organisation and XPP (execution premium proces) which revolve around the Balanced Scorecard.

• LFA: logical framework approach launched by USAID in 1969. This developed into GOPP, OOPP and PCM*.

(Vahamaki 2011)12

*Goal oriented project planning, Objectives oriented project planning, Project cycle management

.. Some more parents

• Planning, programming and budgeting systems in the 60s that

emphasised financial planning and cost accounting. This developed into “Programme management by activity” during the

70s and 80s (eg using Gantt charts, network analysis, etc.),

emphasising further the planning of activities

These streams were the antipode to Druckers’ MBO.

• Corporate performance management and Performance based

budgeting attempted to combine both approaches (trying to cost

results) in the 70-90s but were met with relatively little uptake

• Finally comes the New Public management Philosophy (as a loose

collection of doctrines) in the 80s with…. what is now called RBM as

its intended emanation in development aid

(Vahamaki 2011)13

New public management: a collection of principles

Hood (1991)14

Hood (1991)

New public management: zooming in on two key principles

15

RBM: no singular model exists…

…but there are some key aspects:

This usually includes granting administrators the autonomy to use their resources inthe most efficient and effective way.

This generates learning.

(Vahamaki 2011)16

RBM: no singular model exists…

…but there are some key aspects:

This usually includes granting administrators the autonomy to use their resources inthe most efficient and effective way.

This generates learning.

(Vahamaki 2011)

An analytic and performance oriented approach

17

RBM: no singular model exists…

…but there are some key aspects:

This usually includes granting administrators the autonomy to use their resources inthe most efficient and effective way.

This generates learning.

(Vahamaki 2011)

Centred on end-user level development, only thereafter moving to identify what is necessary to get there

18

RBM: no singular model exists…

…but there are some key aspects:

This usually includes granting administrators the autonomy to use their resources inthe most efficient and effective way.

This generates learning.

(Vahamaki 2011)

Generate information and learn from it

19

RBM: no singular model exists…

…but there are some key aspects:

This usually includes granting administrators the autonomy to use their resources inthe most efficient and effective way.

This generates learning.

(Vahamaki 2011)

Influence decisions based on this information and learning

20

RBM: no singular model exists…

…but there are some key aspects:

This usually includes granting administrators the autonomy to use their resources inthe most efficient and effective way.

Key tool: the “results chain”This generates learning.

(Vahamaki 2011)

21

RBM: no singular model exists…

Action

RBM: no singular model exists…

“Outcome” orientation is a key differentiator from basicNew Public Management

thinking which is more output oriented; it is the heritage of some of the “other” parents

RBM: no singular model exists…

…but there are some key aspects:

This usually includes granting administrators the autonomy to use their resources inthe most efficient and effective way.

This generates learning.

(Vahamaki 2011) 24

It is intended to be an organisationwide approach, not just a tool for planning projects

• So far the history of RBM…• …but how is this relevant for ESF?

Question

• What is the single most important concern, that occupies public servants’ minds, implicitly or explicitly?

Answer?

• Accountability

How was this expressed recently?

• “The most evident weaknesses which indicate the need for reform of cohesion policy are: …Methodological and operational problems that have prevented

both the appropriate use of indicators and targets – for which no comparable information is available - and a satisfactory analysis of “what works” in terms of policy impact.

A remarkable lack of political and policy debate on results in terms of the well-being of people, at both local and EU level, most of the attention being focused on financial absorption and irregularities.”

Independent report “An Agenda for a Reformed Cohesion Policy” delivered at the request of Commissioner for Regional Policy, Ms Hübner.

What does it mean for us tomorrow?• We are faced with:

Ever increasing audit burden essentially to check the materiality and regularity of expenditure (was the money really spent on the project, in compliance with a multitude of rules)

A performance framework that attempts to capture a complex programme in a few abstract indicators (common or specific) and associated milestones and targets with consequences if missed

Broad use of impact evaluation (just advancing is not good enough, it’s relative to NO action!)

That’s

the

RBM,

part

right?

How likely is it that this really leads to more “well-being of people?”

Imagine…

• Structural Funds- the movie, starring…

CEO: EC and national/regional policy-makers

Shareholders: EU taxpayers

Employees: MAs, IBs,…

Clients: target groups

Suppliers: project promotors –the final beneficiaries

EU 2020 = the objectives

Imagine… -2• A new CEO starts and sets out a clear objective for the organisation:

in 7 years time, we will become the global market leader -a clear target! For this, a considerable investment budget was secured from the shareholders! Employees are enthusiastic about this bold goal! Clients and suppliers rejoice at the prospect of new products/services and

business prospects!• Over the next 7 years the CEO ensures the following actions are

taken to reach the target: checking invoices were really paid according to procedures, checking that employees logged their activities in time sheets, checking the budget was spent according to the agreed deadlines, checking the checks to see whether there is less than 2% difference in

finding errors… checking regularly whether the relative market position is improving

according to plan…giving employees hell if it isn’t checking whether the data on market position has been correctly

gathered…

Imagine… -2• A new CEO starts and sets out a clear objective for the organisation:

in 7 years time, we will become the global market leader -a clear target! For this, a considerable investment budget was secured from the shareholders! Employees are enthusiastic about this bold goal! Clients and suppliers rejoice at the prospect of new products/services and

business prospects!• Over the next 7 years the CEO ensures the following actions are

taken to reach the target: checking invoices were really paid according to procedures, checking that employees logged their activities in time sheets, checking the budget was spent according to the agreed deadlines, checking the checks to see whether there is less than 2% difference in

finding errors… checking regularly whether the relative market position is improving

according to plan…giving employees hell if it isn’t checking whether the data on market position has been correctly

gathered…

What are the odds that this company will really be nr 1

after 7 years?

Lehman reported record profits every year from 2005 to 2007. In 2007, the firm reported net income of a record $4.2 billion on revenue of $19.3 billion. In February 2007, the stock reached a record $86.18, giving Lehman a market capitalization of close to $60 billion.

On Monday September 15, 2008, Lehman declared bankruptcy.

Let’s think again….

• Accountability?

Accountability for what?

35

Hood (1991)

36

Accountability for what?

37

Hood (1991)

38

Accountability for what?

39

Hood (1991)40

Hood (1991)

42

Lean and purposeful = sole accountability for NPM

A broader view of RBM is called for

• Complexity theory: Harry Jones (ODI)

• Relationalism vs substantialism: Rosalind Eyben (IDS –Big push)

• Social change theory: Irene Guyt

• Cultural theory: Donald Curtis• Management theory: Owen

Barder (Centre for global development, EU branch)

• …

Keeping in mind new knowledge…• New public management under fire:

• Separation of policy execution and policy development has led to policies that are impossible to execute

• Loosening of standards of operational management has led to cost increases

• Financing of agencies on the basis of output targets has led to loss of service quality and bureaucracy

• Budgeting on the basis of output targets has led to loss of control by Parliament and unreadable budget documentation

• Outsourcing of intermediate production to the market has led to decrease of service quality and higher costs

Keeping in mind new knowledge…• Separation of policy execution and

policy development has led to policies that are impossible to execute

• Loosening of standards of operational management has led to cost increases

• Financing of agencies on the basis of output targets has led to loss of service quality and bureaucracy

• Budgeting on the basis of output targets has led to loss of control by Parliament and unreadable budget documentation

• Outsourcing of intermediate production to the market has led to decrease of service quality and higher costs

“..sanctions if output targets are not achieved

…perverse incentive leads to manipulation of

the data (artificially lowered targets…, choice

of output indicators that are easy to meet but

have nothing to do with the outcomes

politicians are interested in, sometimes

outright fraud with the numbers) and gaming

behaviour (putting emphasis on activities

that improve the data).“

Keeping in mind new knowledge…• Separation of policy execution and

policy development has led to policies that are impossible to execute

• Loosening of standards of operational management has led to cost increases

• Financing of agencies on the basis of output targets has led to loss of service quality and bureaucracy

• Budgeting on the basis of output targets has led to loss of control by Parliament and unreadable budget documentation

• Outsourcing of intermediate production to the market has led to decrease of service quality and higher costs

…No campaign to promote civil service values

or public ethos can compensate for that, the

less so if managers and professionals feel

frustrated in their own right on how their

performance is assessed. Everything we knew

already about central planning in socialist

states but now limited to the public sector of a

market economy.“

Results based management (RBM) = Strategic management for the public sector

• Review and synthesis of tools and approaches from leading organisations in terms of public sector management, Public sector oriented sources

• New Zealand: Getting Better at Managing for Outcomes and Getting Better at Managing for Shared Outcomes

• Treasure Board of Canada: The Managing for Results Self-Assessment Tool• World Bank: CAP-Scan Managing for Development Results Capacity Scan• Asian Development Bank: Readiness Assessment Tool - Implementing a

Results Focus in Organizations• EIPA: CAF• ODI (Overseas development Institute) etc.: integrating elements from

complexity theory, … Mainly private sector but also attention to public sector:

• Palladium: Balanced Scorecard Hall of Fame award for Executing Strategy

• This led to the development of a RBM self-assessment framework and strategic management system

47

RBM originally aimed to return to Drucker’s

Management by Objectives with emphasis on

mind-set, essentially of feeling responsible for

results rather than just compliant; this is the

approach followed by the COP strategic

management and self-assessment system

48

Results based management (RBM) = Strategic management for the public sector

CAUTION:No system can replace a “community of inquirers”

that takes the day to day experience in trying to

make things work as a starting point of ongoing ,

joint reflection, figuring out what to do next!

Systems do not make something work, people do!

But some systems can make it harder to establish

this community and others can support it.

49

RBM assessment framework

• Part 1: core RBM practices 1. Orientation of the PMO (programme management

organisation) is clear 2. The PMO strategy is reflected in a strategic results framework 3. The strategy is translated into operations 4. Performance information is collected and supplied 5. Performance information is used 6. External stakeholders are involved in all aspects

• Part 2: enablers • Part 3: the institutional context• See http://www.coprbm.eu/?q=node/430

/ evaluation

demand plan

50Kaplan and Norton, 2008

2. The PMO strategy is reflected in a strategic results framework

Three levels of detail in the framework!

53

RBM assessment framework and ESF

The ex ante evaluation should assess "the adequacy of human resources and administrative capacity for management of the programme", as

well as "the suitability of the procedures for monitoring the programme and collecting the

data necessary to carry out evaluations"(Article 48 (i) and (j) of the proposed structural funds

regulations).

54

RBM assessment framework

• Part 1: core RBM practices 1. Orientation of the PMO (programme management

organisation) is clear 2. The PMO strategy is reflected in a strategic results framework 3. The strategy is translated into operations 4. Performance information is collected and supplied 5. Performance information is used 6. External stakeholders are involved in all aspects

• Part 2: enablers • Part 3: the institutional context• See http://www.coprbm.eu/?q=node/430

From a Programme Management Organisation results chain…

Starting point: PMO organisational VISION for change incl. how to add value!

57

Position of detailed results chains in the overall PMO strategy

…to a programme action results chain

Adding value as an ESF PMO• 1. Volume effects: ESF action 'adds' to

existing action, either by supporting national action in general ('mirroring') or specific areas of national policy ('boosting');

• 2. Scope effects: ESF action 'broadens' existing action by supporting groups or policy areas that would not otherwise receive support;

• 3. Role effects: ESF action supports local/regional innovations that are taken up at national level or national innovative actions that are then 'mainstreamed';

• 4. Process effects: ESF action influences Member States administrations and organisations involved in the programmes.

Adding value as an ESF PMO• 1. Volume effects: ESF action 'adds' to

existing action, either by supporting national action in general ('mirroring') or specific areas of national policy ('boosting');

• 2. Scope effects: ESF action 'broadens' existing action by supporting groups or policy areas that would not otherwise receive support;

• 3. Role effects: ESF action supports local/regional innovations that are taken up at national level or national innovative actions that are then 'mainstreamed';

• 4. Process effects: ESF action influences Member States administrations and organisations involved in the programmes.

An enhancer supports in a highly efficient way delivery partners to produce (a higher volume) of already existing products/services to satisfy existing needs of constituents. This support is being continuously enhanced as are the supported products and services.

Adding value as an ESF PMO• 1. Volume effects: ESF action 'adds' to

existing action, either by supporting national action in general ('mirroring') or specific areas of national policy ('boosting');

• 2. Scope effects: ESF action 'broadens' existing action by supporting groups or policy areas that would not otherwise receive support;

• 3. Role effects: ESF action supports local/regional innovations that are taken up at national level or national innovative actions that are then 'mainstreamed';

• 4. Process effects: ESF action influences Member States administrations and organisations involved in the programmes.

An innovator supports delivery and other partners to continuously (re)develop new products and services for new constituent needs and deploy them until they become stable and mainstream.

Adding value as an ESF PMO• 1. Volume effects: ESF action 'adds' to

existing action, either by supporting national action in general ('mirroring') or specific areas of national policy ('boosting');

• 2. Scope effects: ESF action 'broadens' existing action by supporting groups or policy areas that would not otherwise receive support;

• 3. Role effects: ESF action supports local/regional innovations that are taken up at national level or national innovative actions that are then 'mainstreamed';

• 4. Process effects: ESF action influences Member States administrations and organisations involved in the programmes.

For a solutions manager the focus is NOT on specific products and services but on developing detailed knowledge concerning specific challenges (a limited number of) delivery partners (the customers) are facing regarding their constituents and work closely with them to solve these challenges.

The journey of the COP: taking stock

Warsaw

BrusselsWarsaw

Vilnius

Prague

Rome

Stockholm

Athens

Maastricht

5-6 November

2014 and beyond: how to ensure delivery of better and more results by

the European Social Fund?

The journey goes on after Maastricht…

• “evidence suggests at least four to five years of consistent effort – and many organizations have been working towards it for much longer. The problem is that key people move on, governance structures change, priorities shift. A lot of relearning has taken place in the history of results-based management. And four to five years is just the timeline to get up and running. Good management requires constant attention. To quote a cliché, integrating performance information into managing and budgeting is a journey, not a destination.”

Source: John Mayne, Challenges and Lessons in Implementing Results-Based Management, 2007, Evaluation

The journey goes on after Maastricht…

• “evidence suggests at least four to five years of consistent effort – and many organizations have been working towards it for much longer. The problem is that key people move on, governance structures change, priorities shift. A lot of relearning has taken place in the history of results-based management. And four to five years is just the timeline to get up and running. Good management requires constant attention. To quote a cliché, integrating performance information into managing and budgeting is a journey, not a destination.”

Source: John Mayne, Challenges and Lessons in Implementing Results-Based Management, 2007, Evaluation

New COP RBM proposal –with a focus on supporting the use of the developed system- has been

submitted to DG EMPL with new partners joining

• Thank you for listening!• Questions?

• More info: www.coprbm.eu or benedict.wauters@esf.vlaanderen.be