better results, stronger communities pathfinder mission: maximise consequential outcomes for new...

Better Results, Stronger Communities

PATHFINDERMission: Maximise consequential

outcomes for New Zealanders


Outcomes for a Vital Nation

• Focus on outcomes; keep output platform

• Clearly specify & measure outcomes, to inform strategy, plans, purchase, ops

• Continuously improve interventions, etc

• Business models for enhancing outcomes (focus: core activities & new initiatives)

Increased benefit to our communities


Current Initiatives

PATHFINDER

CAPValue-for-

Money

TOOLBOX; DECISIONMAKING SYSTEMS

STRATEGY,CAPABILITY,

PERFORMANCE

TRADE-OFFS,TRANSFERS(Within Vote)

(Next FY)


Overall Schedule

• 2 years – like each agency to achieve 3-4 outcome mgmt goals it sets (more later)

• Early milestones (indicative) are by:- Oct ’01: identify / confirm outcome measures- Dec ’01: identify international best practice for managing for improved outcomes in your area- Dec ’01: proposed outcome measurement, analysis and internal reporting frameworks out for discussion

• You set the pace via goals / resources


Sponsors’ Group• Support & guidance for Working Group• Peer review of learning points, etc• Link to Senior Mgmt teams, CE, Minister

Working Group• Identify / document best practice• Review agencies mgmt systems / models• Advice on design & application

Both: Growing capability to improve outcomes


Participation / Lead RepresentativeAgency Sponsors’ Group Working Group

CYFS Brenda Pilott Rose Ryan

Conservation Hugh Logan (Chair) Grant Baker

Corrections Jared Mullin Leon Bakker

Customs Tim Horner Margaret Niven

Health Gillian Durham Mark Booth

Inland Revenue Alan Pinder David Toohey

Labour Geoff Bascand Marg Harvey

LTSA Tony Bliss Martin Small

SSC Tony Hartevelt Greg Claridge

Treasury Andrew Kibblewhite Roger Waite (Chair)


Principles of Pathfinder

• Bottom-up, organic process• Coalition of the willing• Eight Govt Agencies, each with diverse

businesses & business processes• Hot house for sharing / building capacity• DC / Vote Teams remain point of contact• SSC & TSY advise; collate lessons learnt


Ownership from Agencies, backed by Collective Action

• Want ownership / steerage from your CE, senior management and your organisation

• Benefits your agency (& community served)• Want you to own process & mgmt models• Other agencies offer ‘wise counsel’• Positive culture: sharing, helping, learning


Working Group Functions?

• Monthly meeting (2-3 topics, & material for SG)

• Focus teams +/- workshops

• Establish mentoring / bilateral relationships• Brokerage (SSC/TSY/Others putting you in contact)

• E-mail discussion (hot topics needing urgent advice)

• Web Site (password access to outcome mgmt info)

• Collate WG papers into ‘living manual’


Resource Commitment

• Gains will require high calibre staffing

• Low marginal cost (<0.2 FTE?), if committed to outcome-driven mgmt

• For newer entrants, marginal costs are driven by the objectives you set (and orientating business to achieve them)

• Central agencies cover admin. support (for meetings / process - not an offer to do your photocopying)


Agency Team Profile?

• Highly competent analyst-managers• Highly motivated, results-driven• Good communicators, working within

and with strong internal networks• Knowledgable about your business (incl.

purpose, goals and operational realities)• Continuity of staffing critical


Unique Sector, Unique Outcomes,Unique Opportunities

• Primary outcomes differ …• What can be measured will differ …• What applications will work varies by sector …• No one approach will work for all agencies …• For some functions, no approach may work well

Be clear about what you want / need, butBe prepared to innovate & make compromises


Managing for Outcomes

• Scope (measures, maximisation, strategy)

• Decision making, better outcomes (evaluation just a tool)

• Measuring outcomes (measure types, groups)

• Linking measures• Outcome hierarchies• Defining your work agenda (‘straw man’)

• Next Steps


Feedback +/- motivation for staff

Active prioritisation within teams

‘Blind’ managers see hurdles

Business and strategic planning activities informed and focused


3 Families of Outcome Measure ?3 Families of Outcome Measure ?

Family Typical Use (Comparison Group)

State IndicatorState Indicator Situation report; Trend report; Benchmarking; Target setting; Broad targeting of individuals

(Comparison group: None, or other state indicators)

Intervention Intervention Effect SizeEffect Size

Intervention effectiveness with target group; Investment decisions (CBA, total $, etc)

(Comparison group: untreated or ‘status quo’ best treatment - matched pair, random selection, etc)

Decision Making Decision Making (e.g. assessment systems)(e.g. assessment systems)

Risk management; Priority setting; Demand analysis; Investment decisions (CBA, total $)

(Comparison group: None, or adjacent risk groups)


Outcome Management

• Outcomes measured to inform decisions (start by defining the decisions to be informed)

• Rhetorical vs. quantified & measurable • Defining measurement groups critical• Focus on critical measures / groups

(overdesign will result in poor prioritisation)

• Check utility of systems & measures against needs of different stakeholders

Increasing benefit to our communities


Short-Term Wins – Long-term Gains1. Define & measure ‘mission critical’ outcomes

(using ‘state’ or ‘situation’ outcome indicators)

2. Map causal logic linking outcomes to outputs3. Assess impact of interventions / targeting4. Assess cost-effectiveness (vs. outcomes achieved)

5. Managing to maximise outcomes (e.g. core/pilots)

6. Benchmarking with outcomes (business units / nations)

7. Focus strategic / annual plans on improving o/c8. Redesign planning & operations to maximise o/c

(incl. feedback & continuous improvement)

See information pack for details

INTERVENTION FRAMEWORK(Systems Analysis & Design)

SCREENING?

ASSESS?

ASSESSMENT?

TREAT?

ALLOCATETREATMENTS

?PROGRAM

?

NOTREATMENT

(Screened Out)

NOTREATMENT

(Assessed Out)

TREATMENTA

TREATMENTB

TREATMENTEtc.

YY Y Y Y

No

No

CLASSIFICATION ERRORS (Types 1 & 2)

PERFORMANCE OF DECISION MAKING / ALLOCATION SYSTEMS

INTERVENTION EFFECT SIZE;PERFORMANCE OF INTVN. MIX

NOTREATMENT

(Logistics,or Poor Fit)

Use outcomes of assessment system to enhance system and set intervention thresholds to maximise intervention outcomes (go beyond just measuring targeting errors)


When is Targeting Useful?Universal / Defined Access

Mildly Problematic

/ Remedial

Highly Problematic / Remedial

Low Marg-inal Cost

Interventionn.a.

Low cost,

Lo selectiveHi. Selective Hi. Selective (Least cost) (Least cost)

Moderate Cost Intvntn. n.a.

Least cost, Selective

Hi. Selective Hi. Selective (Least cost)(Least cost)

High Cost Intervention Rare



Goal: maximise intervention effect from (targeting + intervention) effort

Target

Intervention

Sector / Department Level Outcome Frameworks

V ic tim isa tion R ates(In d ivid u a ls / E n tit ies )

F ewer V ic tim s(Ju s tice )

F rom C rim e P rosecu tin g C rim e

P erce ived S a fe ty(Ju s tice )

R eso lved

Tim es from C h arg esF iled to P rosecu tion

(C ou rts )

C ou rt H ou rsb y C rim e Typ e

(C ou rts )

P rop ort ion C on vic tion sO vertu rn ed on A p p ea l

(C ou rts )

P rosecu ted C on vic ted

C rim es S o lved(P o lice )

[P ro po rtio n; T im eliness]

R ed u cedR e-o ffen d in g

(C orrec tion s , F G C ,D ivers ion s )

L ess R ep orted C rim e(Ju s tice / P o lice )

C O M M UNIT Y S A F E T Y(F rom C rim e)

(Disaggregation generally required by demographic / service groups)


Cross-Service Outcome Framework

New Initiatives M oney: Interventions w ith Com m on Outcom es

O p en R oad S p eedM an ag em en t

L ig h t V eh ic leS tan d ard s

E n forcem en t

E xp an dC on s tru c tion

E n g in eerin g

Tran sp ort W e lfa re

C on d it ion -Trea tm en tP a irs # 1 to 6 6 6

Trea tm en t P reven ta tive

H ea lth

D ie t S m ok in g

L ife Q u a lityE d u ca tion

E d u cation

Q uali ty A djus ted L ife YearsIn te rven tion s F rom A n y S ec to r

(M easu re o f E ffec tiven ess :C os t p er Q A L Y G a in ed )

Pool all proposals with a common outcome, and allocate new initiatives funding using cost-effectiveness or cost-benefit principles?


Next Steps• Identify learning fora on topics where you

have special skills & experience to share, or have needs (build collective knowledge base)

• Identify your development objectives (3-4 ‘wins’?) & indicative timeline

• SG (you?) to comment on ToR out of session

• Early Sept: second Working Group meeting• Indicative work plan to discuss Sept at SG

Comment on ToR ASAP please; Identify your work agenda


(Slides to be Presented End)

Outcome Performance Measurement & Alignment to Govt Goals

Ex-Ante Intervention Analysis

Ex-Post Policy Evaluation

Governance &

Accountability

Arrangements

Output Measurement

& Delivery

Production & Processes

Input Monitorin

g & Control

Output Pricing & Funding

Arrangements

1

2

3

4 5

6

7

8

Economy

Efficiency

Delivery (current and future = capability)

Dir

ectio

n

Effe

ctiv

enes

sInputsOutcomes

Outputs

Value for Money

Treasury

19 April 2001


Outcome Measurement Traps

• Outcome definition specific to particular service(s)• Selective outcome measures• Fuzzy outcome definitions• Measuring outcomes for few / wrong groups• Slow feedback (decision cycle times critical)

• Measurement error• Poor accountability / attribution / disclosure• Outcomes via other agents (e.g. policy; science?)

• Generating capability only (e.g. Defence)


• Measure(s) too specific (if feasible, identify at least one unifying measure spanning service group(s), preferably in terms linked to risks & costs, e.g. days unemployment [average or avoided], Quality Adjusted Life Years, lives lost / saved, criminal re-offending risks / costs avoided; future costs to the Crown / society avoided)

• Selective measures (define range & hierarchy of potential outcome measures and groups against strategic goals and plans; review & plagiarise outcome definitions used by overseas agencies)

• Fuzzy definitions (specify clear units of measure; ensure measures & measurement groups cover critical interests & expectations of the agency / sector [see ‘Poor Accountability’])

• Slow feedback (reduce selection interval if sample size permits; reduce follow-up period for initial measures; intermediate outcomes)

Measurement Tips (1 of 3)


• Few / wrong groups (report overall outcomes for major service categories; must also identify & report outcomes for key demographic, treatment, geographic, business and/or risk sub-groups [see strategic plans for sector/agency]) (for intervention outcomes, ensure intervention and comparison groups differ ‘only’ vs. intervention(s) being evaluated, or use specialist statistical tools [e.g. Odds Ratio]; can also partition individual intervention groups by demographic variables, risk, service provider, etc for more resolution)

• Measurement error (match outcomes to best data; enhance input and grouping data quality; determine statistical variation; allow for statistical variation in interpreting outcome information) (for intervention outcomes, randomly select or match comparison group) (for risk tools, must validate precision of risk measurements against NZ longitudinal data set other than that used to create the risk tool)



• Poor accountability (ensure groups used in primary care of accountable agency; and outcome is specific to that agency; record external influences as inputs to statistical tests) (for intervention outcomes, must create reference group [e.g. randomly assigned, paired or wait-list controls, alternate treatment(s)]; allow for interactions between interventions via evaluation protocols)

• Outcomes delivered via others (Problem: accountability limited unless agency has control or substantive influence over other agencies; otherwise excuses abound. In the case of sector policy agencies, limited accountability could be expected for polices contributing across the sectors’ hierarchy of outcomes)

• Capability outcome (define clear performance expectation; model likely performance against expectation or assess during unplanned contingencies +/- field exercises +/- deployments)



Costs & Risks of Targeting

• Information distortion (‘cheat strategies’, error)

• Incentive distortion (e.g. reduced motivation)

• Negative effects & stigma (for those targeted)

• Administrative costs & invasive losses• Political sustainability (incl. quality reduction)

After Amartya Sen, in Public Spending & the Poor (World Bank, 1995)

Without recognising it, we are targeting all the time …

The question is how far to push … discrimination


Targeting Efficiency (Error)

CLASSIFICATION ERRORS (Types 1 & 2)

“… greatest source of error is with false-negative diagnoses, which are

detected only rarely by review …” AA Renshaw,

Am.J. Clin. Pathol. 115 (3): 338-341, 2001

Poverty line

INC

OM

E (

$)

PROPORTION OF FAMILIES (%)

B

A

C

D

Effect of transfer = A + B + CEfficiency = A / (A + B + C)

Break-Even

Income before interventionIncome after intervention


Outputs and OutcomesHM Treasury (UK)

Value for Money

Better healthMore treatmentsMore nursesMore money

Other influences

Resources Input Output Outcome

Economy Efficiency Effectiveness



outcomes for the public from Government activity

better results, stronger communities pathfinder mission: maximise consequential outcomes for new...

Documents