#CHYPS2013

Measurement and EvaluationEibhlin Ni Ogain

New Philanthropy CapitalCHYPS presentation.pptx

SHARED APPROACHES TO MEASURING IMPACT

Confederation of Heads of Young People’s Services Summer SeminarManchester, 4th July 2013

NPC: TRANSFORMING THE CHARITY SECTOR

3

NPC works at the nexus between

charities and funders

Charity

SectorFunder

Increasing the impact of charities

eg, impact-focused theories of change

Strengthening the partnership

Eg, collaboration towards shared goals

Increasing the impact of funders

eg, effective commissioning

ConsultancyThink tank

4

Charities and social enterprises under pressure to demonstrate impactNew funding mechanisms, e.g. PbR and SIBS’s require more robust and comparable measurementBut: impact information often of low quality, not embedded, rarely standardised→ impedes learning and improvement

• What should they do to improve?• Driven by funders or for own benefit?• What’s proportionate for them?• Working in isolation, reinventing the wheel

Shared measurement is a potential solution

BACKGROUND

WHAT IS SHARED MEASUREMENT?

5

• Shared measurement is about using common tools to track outcomes across similar organisations and settings.

• Shared measurement is a process: understanding a sector’s shared outcomes

• Shared measurement is also a product: any tool used by more than one organisation to measure impact.

THE JOURNEY TO EMPLOYMENT FRAMEWORK

A shared approach to measuring young people’s employability

6

STEP TO SHARING

7

Understanding your sector

Developing shared outcomes

Using common tools

Using common methods

Sharing and comparing results

ORGANISATIONS CONSULTED

8

• Big Lottery Fund• Birmingham City Council• Careers Development Group / Shaw

Trust• Centre for Economic and Social

Inclusion (CESI)• City Gateway• Department for Work and Pensions• Edge• Employment Pathways• Impetus• London Youth• Participle• Private Equity Foundation• Resurgo• Sheffield Hallam University

• Street League

• Teens and Toddlers• Tomorrow’s People• Young Foundation• Barry Sheerman MP• Clare Coghill, Councillor for Waltham

Forest• Employment Related Services

Association (ERSA)• University of Portsmouth• University of Bath• Chartered Institute of Personnel and

Development (CIPD)• nef (the new economics foundation) • ASDAN• London Employment and Skills Policy

Network

+ +

+

++

+

Intrinsic factors

Extrinsic factors

External factor: The labour market

Employment+ Sustainable

+ Quality

Personal circumstances

Emotional capabilities

Attitudes

Employability skills

Qualifications, education +

training

Experience + involvement

Career management

Journey to EmploymenT framework (JET)

EMOTIONAL CAPABILITIES• Self-esteem• Autonomy and control• Grit and determination• EmpathyATTITUDES• Aspirations• Attitudes to workEMPLOYABILITY SKILLS• Teamwork• Communication• Problem solving • Self-management• Leadership

QUALIFICATIONS, EDUCATION + TRAINING• Basic skills• Achieving qualifications• Attendance and behaviourEXPERIENCES + INVOLVEMENT• Work experience• Perception of value of work experience• Networks• Community involvementCAREER MANAGEMENT• Career direction• Job search skills• Presentation to employers

PERSONAL CIRCUMSTANCES• Access to transport• Access to the internet• Access to childcare• Access to support for young carers• Access to support for disabled people• Reduced substance abuse• Reduced offending/anti-social behaviour

EMPLOYMENT• Entry into employment• Sustaining employment• Quality of employment• Satisfaction with employment

JET—THE RESPONSE

• ‘really good job at capturing the key points that we as a sector need to consider when assessing the journey to employment.’

—Tomorrow’s People

• ‘…my immediate response is that this is an excellent framework for all of us working in EET with young people. The contents resonate with all the aspects of EET work with vulnerable young people here at NHYC. I have shared this with our Youth Work Manager who agrees with my estimation and is going to use it at a review taking place tomorrow of EET provision at the centre.’

—New Horizon Youth Centre

• ‘It would be extremely useful if we could use this framework to reduce duplication in reporting to funders.’

—Resurgo

10

STEP TO SHARING

11

Understanding your sector

Developing shared outcomes

Using common tools

Using common methods

Sharing and comparing results

NPC’S WELL BEING MEASURE

NPC’S WELL-BEING MEASURE

• An online survey-based tool for 11 to 16 year olds

• Measures eight aspects of well-being

• Valid for a group of 10 or more

• Survey includes the ‘best bits’ from academic well-being scales

• Rigorously tested, with a variety of groups of young people

13

INITIAL SURVEY INTERVENTION FOLLOW-UP SURVEY

SUBJECTIVE WELL-BEING IS WHAT WE THINK AND FEEL ABOUT OUR LIVES

14

Aspect of well-being

Description

Self-esteem A child’s feeling of his or her own worth

Emotional health Whether the child experiences worries and feelings of stress

Resilience Whether the child is good at dealing with difficult events and setbacks

Friends Quality of relationships with peers

Family Quality of relationships within the child’s home

School Satisfaction or enjoyment of school

Community Satisfaction or enjoyment with where the child lives

Life satisfaction Overall rating of views on life

These are measures associated with feelings about self

These are measures associated with feelings about experiences

This is a measure of overall life satisfaction

Source: NPC’s Well-being Measure

NPC’S WELL-BEING MEASURE - RESULTS

15

WHERE WE ARE TRYING TO GET

TO?

CAADA: SHARED MEASUREMENT FOR DOMESTIC VIOLENCE

17Embedded shared measurement magnifies the impact of funders and grantees17

SHARED MEASUREMENT SYSTEM

Commissioners and policy

makersBeneficiaries

Practitioners

Service managers

Funders

My experience influences the services I receive

I know how to improve the services we provide, I can evidence our impact

Data gathering doesn’t get in the way of my job, I can see what a difference I make

My grantees have the capacity to deliver better services and become sustainable, I can work more effectively with fellow funders

I know which interventions work, I have evidence of need

Better servicesBetter outcomes

CAADA: SHARED MEASUREMENT FOR DOMESTIC VIOLENCE

18 18

Serv

ice

3

Serv

ice

4

Serv

ice

5

Serv

ice

9

Serv

ice

11

Serv

ice

8

Serv

ice

7

Serv

ice

13

Serv

ice

2

Serv

ice

12

Serv

ice

1

Serv

ice

10

Serv

ice

14

Serv

ice

60%

10%20%30%40%50%

Disclosure of sexual violence

Disclosure of sexual violenceAverage

Serv

ice

4

Serv

ice

9

Serv

ice

13

Serv

ice

8

Serv

ice

7

Serv

ice

1

Serv

ice

11

Serv

ice

10

Serv

ice

2

Serv

ice

14

Serv

ice

12

Serv

ice

5

Serv

ice

3

Serv

ice

60%

10%20%30%40%50%60%

Disclosure of mental health issues

Disclosure Average

18

Impact of training: 50% increase in disclosure

Impact of co-location 50% increase in disclosure

Serv

ice

1

Serv

ice

14

Serv

ice

3

Serv

ice

7

Serv

ice

9

Serv

ice

11

Serv

ice

2

Serv

ice

4

Serv

ice

8

Serv

ice

10

Serv

ice

12

Serv

ice

13

Serv

ice

5

Serv

ice

60%

10%

20%

30%

Disclosure of alcohol misuse

Disclosure of alcohol misuseAverage

TABLE DISCUSSIONS

Reactions to JET?

How do you currently measure impact?

How can you use the JET framework?

How important is it for you to measure well-being?

How would using a tool like this work for you?

How can you get buy-in from partners?

What other challenges with measurement and evaluation do you face?

What do you need support with?19