les hems ey (ernst & young) 21 september2016 impact bonds - ey...social impact bonds workshop...

Social Impact Bonds Workshop No. 2:Outcome Measures

Les HemsEY (Ernst & young)

21 September2016

Outcomes measures: Agenda

1. Fundamental importance of outcomes and outcomes measurement

2. Identifying and defining outcomes

3. Why client cohort groups are important

4. Which measures to use from the continuum of measures

5. Key learning

6. Questions and Answers

Copyright © 2014 Ernst & Young Australia. All Rights Reserved. Liability limited by a scheme approved under Professional Standards Legislation

Fundamental importance of Outcomes and Outcomes Measurement


Exam question: Potential benefits of Social Impact Bonds

“are suited best to situations where there are a number of complex, often

interrelated, factors that impact negatively on individuals and there are no clear,

effective and existing programs directly targeting the problem.”

Statement of Opportunities Page 3


• for clients through the measurement and collection of outcomes-driven data

Focus on agreed outcomes

• allowing providers to deliver solutions that are specifically tailored to client need

Flexibility in service delivery

• rather than providing crisis services at later stages when interventions may be more complex or costly

Evidence base for early intervention

Clients

Service providers

Government

Identifying and defining outcomes


Tell a compelling story: Identifying and defining outcomes


Theory of Change

• Story of how change is created in an activity/ program

• Output: Tells you an activity has taken place and is usually quantitative (e.g. number of people trained)

• Outcome: The change that occurs as a result of an activity (e.g. improved well-being of training participants)

• What is the evidence that underpins the story? Literature / research and program track record

Inputs Activities Outputs Outcomes

Things aren’t simple: Outcomes hierarchy and different client pathways

Stable housing

Positive family, kinship & community connections

Physical & mental health (including trauma)

Life skills, parenting, financial management, tenancy management

Learning & education

Employment / productive time

Desistance Independence

No program is an island: System & network

Referral (30%)Referral

(25%) Referral (25%)

Referral (10%)

Youth Justice (DHHS)

Child Protection (DHHS)

DOE

Home School Liaison Officer

(DOE)

TOOL

UTURN

TasTAFE

Edzone

Work Skills

Centrelink

Interagency Support Teams Clarence Plains

Red Cross Bridgewater

Youth Magistrate

Legal Aid

Police and Youth

Community Club

Tas Police Early Intervention

Unit

Royal Hobart Hospital

Sorell Youth Centre

Link Youth Health Service

Pulse Youth Centre

GlenorchyTYSS Mission Australia

Supporting Young People on Bail

Linkage into education services

Rome was not built in a day: Short/medium /long term outcomes for young people

Physical health

Independence

Engagement with

Education and Training

Employability

Positive Social Connections

Future Aspirations

· Less smoking, alcohol and drug use· Regular gym visits· Regular attendance to medical

appointments

· Financial independence· Transport without reliance on others · Stable and secure accommodation

· Routinely attending education or training

· Progress towards completing school / obtaining work-related certification

· Sociable, attentive behaviour in class

· Interview-related skills· Feeling comfortable at work · Acceptance of authority

· Associating with new, positive circles· Less time spent with negative cycles· Ability to deal with peer pressure

· New, positive perception of life · Setting goals for the future· Less court-related stress· Less pro-criminal thoughts· Less interactions with court/police· Reduced sentencing / avoided

detention (SYPoB)

· Awareness of the benefits of regular physical activities / healthy eating habits

· Awareness of services to assist with personal needs: health / education / employment / housing)

· Receiving vital personal documentation for ID

· Support for educational / vocational activities

· Motivation to go to school

· Support to find employment · Understanding of appropriate job

opportunities / work environment· Motivation to go to work

· Support for recreational activities and equipment

· Engaging in recreational / social / community activities

· Family support· Awareness of offending behaviours

and triggers· Access to a mentor, positive adult role

model· Assistance to deal with personal issues

through “reflective communication”· Motivation to get up from bed and 'do

things'· Meeting legal obligations

Young person

· Developing a Bail Support Plan with Youth Worker

· Interacting/spending one on one time with Youth Worker

· Attending court with Youth Worker

Save the Children / Youth Worker

· Gateway to services providing support to disengaged young people

· Representing young person in court

· Providing reports to Magistrate and Youth Justice

ACTIVITIES

SHORT-TERM OUTCOMES

MEDIUM-TERM OUTCOMES

LONG-TERM OUTCOMES

· Interpersonal / communication skills· Anger management strategies· Less violent behaviour· Ability to trust

Improved Family

Relations

Exam question +ve return for Government: Outcomes for Government

Professionalism Expertise

Commitment

SYPoB and TRaD program costs

SCA Youth Workers and

Program Manager

Specialised Youth Court (pilot)

Department of Health (grant)

Department of Education (grant)

Tasmanian Community Fund

(grant)

Law Foundation (grant)


Specialised Youth Court

PoliceDepartment of Education

INPUTS ACTIVITIES OUTPUTSSHORT-TERM OUTCOMES


LONG-TERM OUTCOMES

FINAL OUTCOME

Gateway to services

Immediacy of response

Holistic problem-

solvingBuilding

rapport and trust

Consistent Court decisions

Change in Court attitude

Increased number of young people on bail

Reduced number of young people in

detention

Voluntary participation

Effective collaboration and

trust between youth justice agencies

Young people can navigate complex

youth justice system

Appropriate COMMUNITY

support

Appropriate DETENTION support

Gap in service delivery

Reduced arrests

Reduced sentencing costs

Reduced crime in Tasmania

Reduced NUMBER of

arrests

Reduced SEVERITY of

arrests

Community-based

supervision

Detention (Ashley)

Court (admin)

Police

Legal Aid

Early intervention

Customised approach

Youth justice agency

Save the Children

Save the Children

KEY

Why client cohort groups are important


Client segmentation & outcomes matrix

Client segmentation

Identify client segments – do their referral pathways differ – do their outcomes differ?

• Many to many relationships between client segments and outcomes

How will the program/intervention respond to these different client segments and outcomes? What are the differential costs of different client segments?

• High, medium and low vulnerabilities and capabilities –complexity of service

Clients/participants are not all the same:• Demographics• Vulnerabilities• Capabilities• Expectations and aspirations

Defining client cohorts

How do clients get referred in to the program? Eligibility and profiling / assessment tools

Understanding complexity


Inputs Activities Outputs Outcomes

Client segment #1High

Client segment #2Medium

Client segment #3Low

How does the theory of change differ between the key client segments identified in the Stakeholder Segmentation Matrix:

Linking client segmentation to cost savings to government


Where do the benefits flow?

• The benefits (financial, economic and social) should outweigh the costs• There should be substantial evidence demonstrating that the program

activities have the capacity to generate the benefits claimed• The margin of benefit value over cost value needs to be sufficiently

robust to sustain the payment stream from Government

The Nature of the Benefits

The Recipient of the Benefits

The Timing of the Benefits

• Cash savings (current)• Cash savings (future)• Avoided costs• Productivity/capacity

enhancements• Other measureable

benefits

• Single Government Agency

• Multiple Government Agencies

• Commonwealth Government

• Private Individuals• Community

• Short Term• Medium Term• Long Term

Incr

ea

sin

g D

iffi

cult

y in

e

sta

blis

hin

g leve

l o

f B

en

efi

ts

Which measures to use from the continuum of measures


Evaluation frameworks & outcomes measurement


Key considerations

Outcomes are the long term change that occurs as a result of an activity

Inputs Activities OutputsIntermediate

OutcomesFinal

outcomes

Typically increasing in time and complexity to measure, but also in importance to policy makers and investors

Driver of benefits to individuals and governments

► Resources –labour, staff, skills, material, IT systems, facilities, funding

► Activities undertaken to deliver high quality outputs

► Short-medium term objectives that contribute to final outcomes

► End service delivered

► Intended long term impacts

Lots of things can be measured: What is most important for young people?

Physical health

Independence

Engagement with

Education and Training

Employability

Positive Social Connections

Future Aspirations

· Less smoking, alcohol and drug use· Regular gym visits· Regular attendance to medical

appointments

· Financial independence· Transport without reliance on others · Stable and secure accommodation

· Routinely attending education or training

· Progress towards completing school / obtaining work-related certification

· Sociable, attentive behaviour in class

· Interview-related skills· Feeling comfortable at work · Acceptance of authority

· Associating with new, positive circles· Less time spent with negative cycles· Ability to deal with peer pressure

· New, positive perception of life · Setting goals for the future· Less court-related stress· Less pro-criminal thoughts· Less interactions with court/police· Reduced sentencing / avoided

detention (SYPoB)

· Awareness of the benefits of regular physical activities / healthy eating habits

· Awareness of services to assist with personal needs: health / education / employment / housing)

· Receiving vital personal documentation for ID

· Support for educational / vocational activities

· Motivation to go to school

· Support to find employment · Understanding of appropriate job

opportunities / work environment· Motivation to go to work

· Support for recreational activities and equipment

· Engaging in recreational / social / community activities

· Family support· Awareness of offending behaviours

and triggers· Access to a mentor, positive adult role

model· Assistance to deal with personal issues

through “reflective communication”· Motivation to get up from bed and 'do

things'· Meeting legal obligations

Young person

· Developing a Bail Support Plan with Youth Worker

· Interacting/spending one on one time with Youth Worker

· Attending court with Youth Worker

Save the Children / Youth Worker

· Gateway to services providing support to disengaged young people

· Representing young person in court

· Providing reports to Magistrate and Youth Justice

ACTIVITIES

SHORT-TERM OUTCOMES


LONG-TERM OUTCOMES

· Interpersonal / communication skills· Anger management strategies· Less violent behaviour· Ability to trust

Improved Family

Relations

What metrics are directly linked to +vereturns for Government

Professionalism Expertise

Commitment

SYPoB and TRaD program costs

SCA Youth Workers and

Program Manager

Specialised Youth Court (pilot)

Department of Health (grant)

Department of Education (grant)

Tasmanian Community Fund

(grant)

Law Foundation (grant)


Specialised Youth Court

PoliceDepartment of Education

INPUTS ACTIVITIES OUTPUTSSHORT-TERM OUTCOMES


LONG-TERM OUTCOMES

FINAL OUTCOME

Gateway to services

Immediacy of response

Holistic problem-

solvingBuilding

rapport and trust

Consistent Court decisions

Change in Court attitude

Increased number of young people on bail

Reduced number of young people in

detention

Voluntary participation

Effective collaboration and

trust between youth justice agencies

Young people can navigate complex

youth justice system

Appropriate COMMUNITY

support

Appropriate DETENTION support

Gap in service delivery

Reduced arrests

Reduced sentencing costs

Reduced crime in Tasmania

Reduced NUMBER of

arrests

Reduced SEVERITY of

arrests

Community-based

supervision

Detention (Ashley)

Court (admin)

Police

Legal Aid

Early intervention

Customised approach

Youth justice agency

Save the Children

Save the Children

KEY

The holy grail: Robust lead indicators and metrics linked to benefits to Government

► Theory of change and existing evidence may reveal “lead indicators” –

actions that are correlated with desired outcomes e.g.

► Progress on education pathway – course attendance and completion

► Progress on employment pathways – use of jobactive metrics

► Housing stability – maintain tenancy

► Successful completion of drug / alcohol rehabilitation program

► May also reveal metrics that are directly linked to Government

benefits – cost savings / avoidance, productivity improvements

► Reduced incidence and severity of arrests

► Reduced rates of recidivism / incarceration

► Reduced use of emergency housing

► Reduced use of emergency hospital services / hospitalisation

► Reduced incidence of child protection notices


United Kingdom: Payment by Results (PbR) drug and alcohol recovery pilotsOutcome measures:

► Abstinence from all presenting substances: proportion of clients who have

stopped using all their presenting substances

► Successfully completed treatment: proportion of clients who became free

of dependency

► Resolved housing problems: proportion of clients in the pilots who started

with a housing issue (e.g. they were homeless) but can now access and

sustain suitable accommodation

► Stopped injecting

► Improved quality of life: proportion of clients who have achieved a score on

the Treatment Outcome Profile (TOP) at or above the functional range of

those clients who go on to leave treatment successfully and do not return

The TOP measures change and progress in key areas of the lives of people

being treated in your drug and alcohol services. TOP consists of 20 simple

questions focusing on the areas such as substance use, injecting risk behaviour,

crime, health and quality of life.


Building the counterfactual:

Historical baseline: - Comparison between before and after the start of the

pilot:

► Magnitude of change was assessed for each of the five outcome measures

over the pilot duration (April 2012 to February 2013). It was then compared to

the same period 12 months before the pilot.

► Service users were segmented into five complexity groups based on their

likelihood of achieving each of the outcomes. This enabled commissioners to

take into account any changes in the profile of service users.

National comparison: Comparison to average for rest of England:

► Magnitude of change was assessed for each of the five outcome measures. It

was then compared to the aggregated performance for the rest of the country

over the same period (results were also broken down by the five complexity

groups).


Activity – Develop outcomes measurement


How would you measure the outcomes of the GoodHope Society’s program?

EstablishDesign & measure

Analyse Value Assure Use

• What measures?• What data needed?

• What outcomes will you measure?

• How to ensure statistical significance?

• How to improve the program?

• How to foster innovation?

• How would you estimate the monetary benefits?

• Options for independent verification? Pros/cons?

Key learning


Key learning


• a compelling story above how a program achieves outcomes for individuals and benefits to Government

Tell a story

• simplicity may hide costs and barriers to achieving outcomes

Embrace complexity

• referrals in, differentiated program delivery, links to support services, different outcomes mix

Recognise different client pathways

• drive different benefits for governmentClient segments

• that “predict” outcomes but can be measured in real / short time

Find lead indicators

• to validate lead indicator and critical success factors

Continue to measure long term outcomes

• cherry picking and parking difficult clientsAvoid risk of perverse

behaviours

Question and Answers


les hems ey (ernst & young) 21 september2016 impact bonds - ey...social impact bonds workshop...

Documents