benedict wauters 24-10-2012 esf com. twg -cyprus introduction to results based management and to the...
TRANSCRIPT
Benedict Wauters 24-10-2012ESF Com. TWG -Cyprus
Introduction to Results based Management and to the Maastricht conference
Overview
1. The COP RBM
2. Results based management: a brief
history
3. So what?
4. Introducing the Maastricht conference.
Roots of the COP
• EQUAL programme: working group (led by the EQUAL unit) on introducing PCM (Project Cycle Management) in the ESFPartnership development Toolkit - September 2005Conference on sound planning - November 2005Sourcebook and website on sound planning
- September 2006 (www.esfsourcebook.eu)
• Independent report “An Agenda for a Reformed Cohesion Policy” by F. Barca – May 2009
Who?• Managing Authority of Czech Human Resources and Employment Operational
Programme• Czech National Coordination Authority of the NSRF• Greek Managing Authority for “Health and welfare” OP and Intermediate Body for
“Development of Human Resources” OP• Lithuanian intermediate body for Operational Programme for the Development of
Human Resources OP and Promotion of Cohesion OP• Lithuanian Managing Authority• Managing Authority for the Italian Region of Molise Regional OP• Slovak ESF Managing Authority• Swedish ESF Managing Authority• Lithuanian ESF Agency• Managing Authority for the Walloon Convergence and Regional Competitiveness and
Employment Ops• Managing Authority of the Flemish obj. 2 OP (coordinator of the COP)• National support structure for the ESF for the Italian Regions (Tecnostruttura)• European Institute for Public Administration• Originally also: MA Italian Region of Piemonte and MA of Romania
Definition of results
• EC guidance document, November 2011*: The intended result is the specific dimension of well-being and
progress for people that motivates policy action, i.e. what is intended to be changed. The notion of change also comprises changes in behaviour, social practices, institutions etc.
Outputs are the direct products of programmes, they are intended to contribute to results.
• COP RBM: the essential difference between a result and an output (both as described above) is a shift of ownership from a producer to a user If it cannot be used, it is just an “output”: eg with a training it is
the knowledge that can be used, not the actual training
*MONITORING AND EVALUATION OF EUROPEAN COHESION POLICY - EUROPEAN REGIONAL DEVELOPMENT FUND AND COHESION FUND –CONCEPTS AND RECOMMENDATIONS.
The journey of the COP
Warsaw
Brussels (x2)
Warsaw
Vilnius
Prague
Rome
Stockholm
Athens
Core of the COP work : 8 interactive seminars (of 2-3 days) with experts from around the world (incl. US, Canada, Australia) in
debate with practitioners from our own programmes
Between seminars:
participating in global networks, going deeper into
issues derived from the meetings
Key outputs: • Definitions of Results• RBM baseline study• Practice descriptions• Huge online
knowledge base• RBM strategic
management and self-assessment system
Some key insights from the seminars-1• not clear what RBM really is, many tools and approaches seem relevant; no clear
and generally accepted definition what the word “results” actually means!• huge variety and selection of tools for mapping and analysing stakeholders when
setting up projects but this tends to obscure the much more important question: how do you involve them and keep them involved?
• how do you know who are stakeholders in a project? Ask the target group how they run into in real life!
• bottom-up vs. top down deployment of an ESF OP is very different• central to RBM is the “results chain”• RBM at organisational level is quite different from RBM at project level; for
organisations it is more like a a specific type of strategic planning and management system
• quality management (ISO) only helps “stabilise” and incrementally improve processes but does not itself provide a results orientation
• we are focused on changes that depend on long chains of cause and effect but tend not to look at what/who is/are in front of our noses (e.g. intermediariy organisations)
• Strategy maps / Balanced scorecards have potential for supporting RBM at organisational level and can in some way be linked to results chains
Some key insights from the seminars-1• not clear what RBM really is, many tools and approaches seem relevant; no clear
and generally accepted definition what the word “results” actually means!• huge variety and selection of tools for mapping and analysing stakeholders when
setting up projects but this tends to obscure the much more important question: how do you involve them and keep them involved?
• how do you know who are stakeholders in a project? Ask the target group how they run into in real life!
• bottom-up vs. top down deployment of an ESF OP is very different• central to RBM is the “results chain”• RBM at organisational level is quite different from RBM at project level; for
organisations it is more like a a specific type of strategic planning and management system
• quality management (ISO) only helps “stabilise” and incrementally improve processes but does not itself provide a results orientation
• we are focused on changes that depend on long chains of cause and effect but tend not to look at what/who is/are in front of our noses (e.g. intermediary organsations)
• Strategy maps / Balanced scorecards have potential for supporting RBM at organisational level and can in some way be linked to results chains
RBM at organisational level is quite
different from RBM at project level;
for organisations it is more like a a
specific type of strategic planning
and management system
We are focused on changes that
depend on long chains of cause and
effect but tend not to look at
what/who is/are in front of our noses
(e.g. intermediary organisations )
central to RBM is the “results chain”
Some key insights from the seminars -2• measurement via indicators and target setting on them tends to lead to perverse effects
when they have direct consequences; they should only be used as flashlights to illuminate an area that raises questions
• to be strategic implies that choices have to be made and these should become visible when an organisation (e.g. a Managing Authority) sets objectives; a BSC/strategy map can help-bottom up logic and top down indicator setting for aggregation clash with each other
• an indicator that is just measuring quantity (amounts) without “quality” (how good) is useless
• accountability should be linked to learning, not to indicators• linking money to outputs directly requires deep insight in the local situation and
sophisticated processes for preparation and follow-up: it is not an easy fix• human systems are all but stable and predictable. They are complex and adaptive . This
has huge implications for planning and evaluation.• counterfactual impact evaluation is to be used with great care as it assumes stable and
sufficient cause and effect relations which rarely applies in practice• longer “results chains” require different actions throughout the chain that assume an earlier
action already had its effect• importance of experimentation and adapation when faced with complexity; innovation is
always complex
Some key insights from the seminars -2• measurement via indicators and target setting on them tends to lead to perverse effects
when they have direct consequences; they should only be used as flashlights to illuminate an area that raises questions
• to be strategic implies that choices have to be made and these should become visible when an organisation (e.g. a Managing Authority) sets objectives; a BSC/strategy map can help-bottom up logic and top down indicator setting for aggregation clash with each other
• an indicator that is just measuring quantity (amounts) without “quality” (how good) is useless
• accountability should be linked to learning, not to indicators• linking money to outputs directly requires deep insight in the local situation and
sophisticated processes for preparation and follow-up: it is not an easy fix• human systems are all but stable and predictable. They are complex and adaptive . This
has huge implications for planning and evaluation.• counterfactual impact evaluation is to be used with great care as it assumes stable and
sufficient cause and effect relations which rarely applies in practice• longer “results chains” require different actions throughout the chain that assume an earlier
action already had its effect• importance of experimentation and adapation when faced with complexity; innovation is
always complex
measurement via indicators and target
setting on them tends to lead to perverse
effects when they have direct
consequences; they should only be used
as flashlights to illuminate an area that
raises questionsaccountability should be linked to
learning, not to indicators
human systems are all but stable and predictable. They are complex and
adaptive . This has huge implications for planning and evaluation.
• The COP RBM, to its surprise, found itself exposed to a huge variety of (sometimes conflicting) views…
• Who came up with this Results Based Management in the first place?
RBM is a child with many parents…
• MBO: management by objectives, launched by Peter Drucker in “the Practice of management” in 1954 It emphasised the learning aspect and discouraged
mechanical models (of “scientific” management) Key is adaptation of empowered employees to changing
circumstances and performance feed-back rather than control and predictability
MBO’s current most popular form is through Kaplan and Norton’s strategy focused organisation and XPP (execution premium proces) which revolve around the Balanced Scorecard.
• LFA: logical framework approach launched by USAID in 1969. This developed into GOPP, OOPP and PCM*.
(Vahamaki 2011)12
*Goal oriented project planning, Objectives oriented project planning, Project cycle management
.. Some more parents
• Planning, programming and budgeting systems in the 60s that
emphasised financial planning and cost accounting. This developed into “Programme management by activity” during the
70s and 80s (eg using Gantt charts, network analysis, etc.),
emphasising further the planning of activities
These streams were the antipode to Druckers’ MBO.
• Corporate performance management and Performance based
budgeting attempted to combine both approaches (trying to cost
results) in the 70-90s but were met with relatively little uptake
• Finally comes the New Public management Philosophy (as a loose
collection of doctrines) in the 80s with…. what is now called RBM as
its intended emanation in development aid
(Vahamaki 2011)13
New public management: a collection of principles
Hood (1991)14
Hood (1991)
New public management: zooming in on two key principles
15
RBM: no singular model exists…
…but there are some key aspects:
This usually includes granting administrators the autonomy to use their resources inthe most efficient and effective way.
This generates learning.
(Vahamaki 2011)16
RBM: no singular model exists…
…but there are some key aspects:
This usually includes granting administrators the autonomy to use their resources inthe most efficient and effective way.
This generates learning.
(Vahamaki 2011)
An analytic and performance oriented approach
17
RBM: no singular model exists…
…but there are some key aspects:
This usually includes granting administrators the autonomy to use their resources inthe most efficient and effective way.
This generates learning.
(Vahamaki 2011)
Centred on end-user level development, only thereafter moving to identify what is necessary to get there
18
RBM: no singular model exists…
…but there are some key aspects:
This usually includes granting administrators the autonomy to use their resources inthe most efficient and effective way.
This generates learning.
(Vahamaki 2011)
Generate information and learn from it
19
RBM: no singular model exists…
…but there are some key aspects:
This usually includes granting administrators the autonomy to use their resources inthe most efficient and effective way.
This generates learning.
(Vahamaki 2011)
Influence decisions based on this information and learning
20
RBM: no singular model exists…
…but there are some key aspects:
This usually includes granting administrators the autonomy to use their resources inthe most efficient and effective way.
Key tool: the “results chain”This generates learning.
(Vahamaki 2011)
21
RBM: no singular model exists…
Action
RBM: no singular model exists…
“Outcome” orientation is a key differentiator from basicNew Public Management
thinking which is more output oriented; it is the heritage of some of the “other” parents
RBM: no singular model exists…
…but there are some key aspects:
This usually includes granting administrators the autonomy to use their resources inthe most efficient and effective way.
This generates learning.
(Vahamaki 2011) 24
It is intended to be an organisationwide approach, not just a tool for planning projects
• So far the history of RBM…• …but how is this relevant for ESF?
Question
• What is the single most important concern, that occupies public servants’ minds, implicitly or explicitly?
Answer?
• Accountability
How was this expressed recently?
• “The most evident weaknesses which indicate the need for reform of cohesion policy are: …Methodological and operational problems that have prevented
both the appropriate use of indicators and targets – for which no comparable information is available - and a satisfactory analysis of “what works” in terms of policy impact.
A remarkable lack of political and policy debate on results in terms of the well-being of people, at both local and EU level, most of the attention being focused on financial absorption and irregularities.”
Independent report “An Agenda for a Reformed Cohesion Policy” delivered at the request of Commissioner for Regional Policy, Ms Hübner.
What does it mean for us tomorrow?• We are faced with:
Ever increasing audit burden essentially to check the materiality and regularity of expenditure (was the money really spent on the project, in compliance with a multitude of rules)
A performance framework that attempts to capture a complex programme in a few abstract indicators (common or specific) and associated milestones and targets with consequences if missed
Broad use of impact evaluation (just advancing is not good enough, it’s relative to NO action!)
That’s
the
RBM,
part
right?
How likely is it that this really leads to more “well-being of people?”
Imagine…
• Structural Funds- the movie, starring…
CEO: EC and national/regional policy-makers
Shareholders: EU taxpayers
Employees: MAs, IBs,…
Clients: target groups
Suppliers: project promotors –the final beneficiaries
EU 2020 = the objectives
Imagine… -2• A new CEO starts and sets out a clear objective for the organisation:
in 7 years time, we will become the global market leader -a clear target! For this, a considerable investment budget was secured from the shareholders! Employees are enthusiastic about this bold goal! Clients and suppliers rejoice at the prospect of new products/services and
business prospects!• Over the next 7 years the CEO ensures the following actions are
taken to reach the target: checking invoices were really paid according to procedures, checking that employees logged their activities in time sheets, checking the budget was spent according to the agreed deadlines, checking the checks to see whether there is less than 2% difference in
finding errors… checking regularly whether the relative market position is improving
according to plan…giving employees hell if it isn’t checking whether the data on market position has been correctly
gathered…
Imagine… -2• A new CEO starts and sets out a clear objective for the organisation:
in 7 years time, we will become the global market leader -a clear target! For this, a considerable investment budget was secured from the shareholders! Employees are enthusiastic about this bold goal! Clients and suppliers rejoice at the prospect of new products/services and
business prospects!• Over the next 7 years the CEO ensures the following actions are
taken to reach the target: checking invoices were really paid according to procedures, checking that employees logged their activities in time sheets, checking the budget was spent according to the agreed deadlines, checking the checks to see whether there is less than 2% difference in
finding errors… checking regularly whether the relative market position is improving
according to plan…giving employees hell if it isn’t checking whether the data on market position has been correctly
gathered…
What are the odds that this company will really be nr 1
after 7 years?
Lehman reported record profits every year from 2005 to 2007. In 2007, the firm reported net income of a record $4.2 billion on revenue of $19.3 billion. In February 2007, the stock reached a record $86.18, giving Lehman a market capitalization of close to $60 billion.
On Monday September 15, 2008, Lehman declared bankruptcy.
Let’s think again….
• Accountability?
Accountability for what?
35
Hood (1991)
36
Accountability for what?
37
Hood (1991)
38
Accountability for what?
39
Hood (1991)40
Hood (1991)
42
Lean and purposeful = sole accountability for NPM
A broader view of RBM is called for
• Complexity theory: Harry Jones (ODI)
• Relationalism vs substantialism: Rosalind Eyben (IDS –Big push)
• Social change theory: Irene Guyt
• Cultural theory: Donald Curtis• Management theory: Owen
Barder (Centre for global development, EU branch)
• …
Keeping in mind new knowledge…• New public management under fire:
• Separation of policy execution and policy development has led to policies that are impossible to execute
• Loosening of standards of operational management has led to cost increases
• Financing of agencies on the basis of output targets has led to loss of service quality and bureaucracy
• Budgeting on the basis of output targets has led to loss of control by Parliament and unreadable budget documentation
• Outsourcing of intermediate production to the market has led to decrease of service quality and higher costs
Keeping in mind new knowledge…• Separation of policy execution and
policy development has led to policies that are impossible to execute
• Loosening of standards of operational management has led to cost increases
• Financing of agencies on the basis of output targets has led to loss of service quality and bureaucracy
• Budgeting on the basis of output targets has led to loss of control by Parliament and unreadable budget documentation
• Outsourcing of intermediate production to the market has led to decrease of service quality and higher costs
“..sanctions if output targets are not achieved
…perverse incentive leads to manipulation of
the data (artificially lowered targets…, choice
of output indicators that are easy to meet but
have nothing to do with the outcomes
politicians are interested in, sometimes
outright fraud with the numbers) and gaming
behaviour (putting emphasis on activities
that improve the data).“
Keeping in mind new knowledge…• Separation of policy execution and
policy development has led to policies that are impossible to execute
• Loosening of standards of operational management has led to cost increases
• Financing of agencies on the basis of output targets has led to loss of service quality and bureaucracy
• Budgeting on the basis of output targets has led to loss of control by Parliament and unreadable budget documentation
• Outsourcing of intermediate production to the market has led to decrease of service quality and higher costs
…No campaign to promote civil service values
or public ethos can compensate for that, the
less so if managers and professionals feel
frustrated in their own right on how their
performance is assessed. Everything we knew
already about central planning in socialist
states but now limited to the public sector of a
market economy.“
Results based management (RBM) = Strategic management for the public sector
• Review and synthesis of tools and approaches from leading organisations in terms of public sector management, Public sector oriented sources
• New Zealand: Getting Better at Managing for Outcomes and Getting Better at Managing for Shared Outcomes
• Treasure Board of Canada: The Managing for Results Self-Assessment Tool• World Bank: CAP-Scan Managing for Development Results Capacity Scan• Asian Development Bank: Readiness Assessment Tool - Implementing a
Results Focus in Organizations• EIPA: CAF• ODI (Overseas development Institute) etc.: integrating elements from
complexity theory, … Mainly private sector but also attention to public sector:
• Palladium: Balanced Scorecard Hall of Fame award for Executing Strategy
• This led to the development of a RBM self-assessment framework and strategic management system
47
RBM originally aimed to return to Drucker’s
Management by Objectives with emphasis on
mind-set, essentially of feeling responsible for
results rather than just compliant; this is the
approach followed by the COP strategic
management and self-assessment system
48
Results based management (RBM) = Strategic management for the public sector
CAUTION:No system can replace a “community of inquirers”
that takes the day to day experience in trying to
make things work as a starting point of ongoing ,
joint reflection, figuring out what to do next!
Systems do not make something work, people do!
But some systems can make it harder to establish
this community and others can support it.
49
RBM assessment framework
• Part 1: core RBM practices 1. Orientation of the PMO (programme management
organisation) is clear 2. The PMO strategy is reflected in a strategic results framework 3. The strategy is translated into operations 4. Performance information is collected and supplied 5. Performance information is used 6. External stakeholders are involved in all aspects
• Part 2: enablers • Part 3: the institutional context• See http://www.coprbm.eu/?q=node/430
/ evaluation
demand plan
50Kaplan and Norton, 2008
2. The PMO strategy is reflected in a strategic results framework
Three levels of detail in the framework!
53
RBM assessment framework and ESF
The ex ante evaluation should assess "the adequacy of human resources and administrative capacity for management of the programme", as
well as "the suitability of the procedures for monitoring the programme and collecting the
data necessary to carry out evaluations"(Article 48 (i) and (j) of the proposed structural funds
regulations).
54
RBM assessment framework
• Part 1: core RBM practices 1. Orientation of the PMO (programme management
organisation) is clear 2. The PMO strategy is reflected in a strategic results framework 3. The strategy is translated into operations 4. Performance information is collected and supplied 5. Performance information is used 6. External stakeholders are involved in all aspects
• Part 2: enablers • Part 3: the institutional context• See http://www.coprbm.eu/?q=node/430
From a Programme Management Organisation results chain…
Starting point: PMO organisational VISION for change incl. how to add value!
57
Position of detailed results chains in the overall PMO strategy
…to a programme action results chain
Adding value as an ESF PMO• 1. Volume effects: ESF action 'adds' to
existing action, either by supporting national action in general ('mirroring') or specific areas of national policy ('boosting');
• 2. Scope effects: ESF action 'broadens' existing action by supporting groups or policy areas that would not otherwise receive support;
• 3. Role effects: ESF action supports local/regional innovations that are taken up at national level or national innovative actions that are then 'mainstreamed';
• 4. Process effects: ESF action influences Member States administrations and organisations involved in the programmes.
Adding value as an ESF PMO• 1. Volume effects: ESF action 'adds' to
existing action, either by supporting national action in general ('mirroring') or specific areas of national policy ('boosting');
• 2. Scope effects: ESF action 'broadens' existing action by supporting groups or policy areas that would not otherwise receive support;
• 3. Role effects: ESF action supports local/regional innovations that are taken up at national level or national innovative actions that are then 'mainstreamed';
• 4. Process effects: ESF action influences Member States administrations and organisations involved in the programmes.
An enhancer supports in a highly efficient way delivery partners to produce (a higher volume) of already existing products/services to satisfy existing needs of constituents. This support is being continuously enhanced as are the supported products and services.
Adding value as an ESF PMO• 1. Volume effects: ESF action 'adds' to
existing action, either by supporting national action in general ('mirroring') or specific areas of national policy ('boosting');
• 2. Scope effects: ESF action 'broadens' existing action by supporting groups or policy areas that would not otherwise receive support;
• 3. Role effects: ESF action supports local/regional innovations that are taken up at national level or national innovative actions that are then 'mainstreamed';
• 4. Process effects: ESF action influences Member States administrations and organisations involved in the programmes.
An innovator supports delivery and other partners to continuously (re)develop new products and services for new constituent needs and deploy them until they become stable and mainstream.
Adding value as an ESF PMO• 1. Volume effects: ESF action 'adds' to
existing action, either by supporting national action in general ('mirroring') or specific areas of national policy ('boosting');
• 2. Scope effects: ESF action 'broadens' existing action by supporting groups or policy areas that would not otherwise receive support;
• 3. Role effects: ESF action supports local/regional innovations that are taken up at national level or national innovative actions that are then 'mainstreamed';
• 4. Process effects: ESF action influences Member States administrations and organisations involved in the programmes.
For a solutions manager the focus is NOT on specific products and services but on developing detailed knowledge concerning specific challenges (a limited number of) delivery partners (the customers) are facing regarding their constituents and work closely with them to solve these challenges.
The journey of the COP: taking stock
Warsaw
BrusselsWarsaw
Vilnius
Prague
Rome
Stockholm
Athens
Maastricht
5-6 November
2014 and beyond: how to ensure delivery of better and more results by
the European Social Fund?
The journey goes on after Maastricht…
• “evidence suggests at least four to five years of consistent effort – and many organizations have been working towards it for much longer. The problem is that key people move on, governance structures change, priorities shift. A lot of relearning has taken place in the history of results-based management. And four to five years is just the timeline to get up and running. Good management requires constant attention. To quote a cliché, integrating performance information into managing and budgeting is a journey, not a destination.”
Source: John Mayne, Challenges and Lessons in Implementing Results-Based Management, 2007, Evaluation
The journey goes on after Maastricht…
• “evidence suggests at least four to five years of consistent effort – and many organizations have been working towards it for much longer. The problem is that key people move on, governance structures change, priorities shift. A lot of relearning has taken place in the history of results-based management. And four to five years is just the timeline to get up and running. Good management requires constant attention. To quote a cliché, integrating performance information into managing and budgeting is a journey, not a destination.”
Source: John Mayne, Challenges and Lessons in Implementing Results-Based Management, 2007, Evaluation
New COP RBM proposal –with a focus on supporting the use of the developed system- has been
submitted to DG EMPL with new partners joining
