a realist evaluation of performance management for social services lessons from the implementation...
TRANSCRIPT
A Realist Evaluation of Performance Management for
Social Services
Lessons from the implementation of “Outcome Management” in the
voluntary sector in Singapore
Robyn Tan, PhD candidateInstitute of Development Policy and Management
University of ManchesterEmail: [email protected]
“Public sector performance measurement is, in effect, like putting a meter on a black box: we have little knowledge of the mechanism
inside and no theory linking inputs, processes, outputs and outcomes to explain
why a particular result occurred or to prescribe what management or
organizational adjustments are needed to improve performance”
(Grenier 1997)
The core of performance management is for achieving accountability and
improvement
- Yes? No? Not sure?
Research Aim
To examine how and why performance management works (or not) to achieve accountability and improvement among voluntary organisations, through the use
of performance information
Case Example: Outcome Management for “Job Placement and Job Support”
Outcome StatementOut of 55 clients who were engaged in job placement interview, 40 stay in the job for at least 6 months.
Milestones Targets Measurement Tools
1.Clients are engaged in a job placement interview
55 Registration Client Progress Report
2. Clients are engaged in the development of employment plans
55 Programme recordEvaluation Report
3. Clients are matched to suitable jobs and attend job interviews
50 Placement Record
4. Clients accept job offer 50 Placement Record
5. Clients stay in the job for at least 2 weeks
50 Placement Record
6. Clients stay in the job for at least 3 months
42 Feedback form Site visitCase notes
7. Clients stay in the job for at least 6 months
40 Feedback formSite visitCase notes
For practitioners to meet specified targets, in order to maintain accountability
For practitioners,
(1) To learn about client’s
progress in relation
to the milestones;
(2) To improve
the programme
Applying realist evaluation to the study of performance
managementRealist evaluation Primary data sources Secondary data sources
Develop the initial programme theories of how and why Outcome Management is expected to work
NIL Policy papersTraining manualOutcome Management related literaturePerformance management literature
Test the actual theories of how and why Outcome Management works against the initial theories, using 7 identified case studies
Individual & group interviews with Council programme administrators and voluntary organisation practitioners
Social service contractPerformance reports
Develop refined programme theories of how and why Outcome Management would work
Data analysis from primary and secondary data sources
Implementation chain of Outcome Management
Step 1: Identify initial, intermediate & long term outcomes
Step 2: Specify
programme in relation
to outcomes
Step 3: Select performance indicators &
measurement tools
Step 4: Set & meet targets
Step 5: Use performance information
for learning & improvement
How to identify the outcomes for the study?
“Outcome Management is intended to serve as a robust and multifaceted
oversight system for social services and to improve the overall service standards
of service delivery”
– National Council of Social Service
Developing initial progamme theories:
Identifying sub-outcomes & higher-
order outcomes
Step 1: Identify initial, intermediate & long term outcomes
Step 2: Specify
programme in relation
to outcomes
Step 3: Select performance indicators &
measurement tools
Step 4: Set & meet targets
Step 5: Use performance information
for learning & improvement
Oi: Voluntary organisations
formulate initial, intermediate
& desired outcomes
Oii: Voluntary organisations specify
prog activities or intervention strategies in relation to the outcomes
Oiii: Performance information generates
feedback on client progress in relation to
the outcomes
O1: Voluntary organisations meet
targets, in turn achieve accountability
to the Council
O2: Voluntary organisations learn
about clients’ progress
O3: Voluntary organisations make actual changes to
improve the programme
Initial programme theories & actual programme theoriesOutcome
ManagementInitial programme theories Actual programme theories
Step 1: Developing outcomes
C: Institutional requirementC: Equipped with necessary knowledge and skillsM: “Acquiescence”Oi: To formulate well-defined outcomes
C: Limited knowledge and skillsC: Intervention-specific contextM: “Mimicking” tactics M: “Compromise” tacticsO-: Fail to formulate well-defined outcomes
Step 2: Specifying
programme
C: Lack of institutional requirementC: Equipped with necessary knowledge and skillsM: “Tight coupling”Oii: To specify programme in relation to the outcomes
C: Lack of institutional requirement; depends on organisational discretionC: Intervention-specific contextC: Limited knowledge and skills M: “Decoupling” O-: Fail to specify programme in relation to the outcomes
Step 3: Adopting
measurement tools
C: Institutional requirement to measure outcomes quantitatively; choice of measurement tools based on organisational discretionC: Equipped with necessary knowledge and skills M: “Tight coupling”Oiii: To generate feedback on client progress in relation to the outcomes
C: Choice of measurement tools based on organisational discretionC: Limited knowledge and skillsM: “Decoupling”M: “Pragmatism”O: Lack of standardisation in the use of measurement toolsO: Lack of routine monitoringO-: Fail to generate feedback updates on client progress
Step 4: Setting and
meeting targets
C: Institutional requirements to meet targetsM: “Target system” M: “Ranking system”M: “Signal function”O1: To meet targets in order to achieve accountability
M: “Target system”O-: Targets were met BUT they were measured against poorly-defined outcomes
Step 5: Using
performance information
Ci: Well-defined outcomesCii: Specification of programme in relation to outcomes Cii: Generated feedback on client progress in relation to outcomesM: “Feedback” mechanismO2: To learn about clients’ progress in relation to outcomesO3: To make actual changes for improvement
Ci-: Poorly-defined outcomesCii-: Fail to specify programme in relation to outcomesCiii-: Fail to generate feedback on clients’ progressO2-: Fail to learn about clients’ progressO3-: Fail to make actual changes for improvement
Key findings
Of the 7 case study programmes, only 1 achieved accountability and improvement
Lack of institutional requirement for voluntary organisations to articulate and test the underlying theory of change (institutional context)
Outcome Management is applied to the measurement of outcomes of a wide array of social services without considering the programme-specific characteristics (intervention-specific context)
Key findings (II)Programmatic intervention Single practitioner-driven
interventione.g. Job placement and job support; sheltered workshop
e.g. Counselling; case management
Reasonably standardised programme across practitioners
Individualised interventions based on practitioner’s choice of theoretical approach and needs of clients
One overarching theory of change There are as many theories of change as there are practitioners and clients
Practitioners deliver programme according to the articulated theory of change
Practitioner delivers intervention based on his or her own theory of change (whether articulated or not).
Facilitates learning and improvement across practitioners
Learning and improvement (if any) is based on practitioner discretion
Developing refined programme theories
Step 2: Identify initial,
intermediate & long term outcomes
Step 3: Specify programme in
relation to outcomes
Step 4: Select performance indicators &
measurement tools
Step 5: Set & meet targets
Step 6: Use performance
information for learning &
improvement
Step 1: Articulate theory of change
underlying programmatic
intervention
Reflections
Identifying the level of abstraction for data collection and analysis – higher or lower or at what level?
Identify the level of abstraction that can provide relevant theoretical and practical insights required for the study
Reflections (II)
Identifying mechanisms from contexts – which is which? And does it matter?
e.g.
“The Council gives voluntary organisations discretion over choice of measurement tools” (context)
“Voluntary organisations exercising their discretion for their choice of measurement tools” (mechanism)
Identify the context-mechanism dyad responsible for generating the outcome
Reflections (III)
Making sense of the contexts for the development of refined programme theories
Categorising contexts according to their layers of social reality – individual capabilities, organisational context, institutional context, intervention-specific context
Identifying the “order” in which they are likely to work to give the intervention a reasonable chance of success.