bond.org.uk

Introduction to the Improve It Framework -

Effectiveness:

Improving and Proving It Conference

Edinburgh November 2012

Ben Jackson Bjackson@bond.org.uk

Chief Executive, Bond

Context

bond.org.uk 3

Consortium Effectiveness Programme

1. Developing agreement and supporting implementation of a sector wide

framework of common outcome indicators and data collection

tools (Improve it Framework); and online organisational self-

assessment and resource portal (Organisational Health Check;

NIDOS Effectiveness Tool); and agreed quality standards regarding

NGO evidence.

2. Building knowledge and skills to support members to measure and

manage effectiveness and value for money

3. Creating an enabling environment that encourages and supports

organisations to deliver improvements in their effectiveness

…supports UK NGOs in improving how they measure, manage and demonstrate their contribution to social change

bond.org.uk 4

Consortium Effectiveness Programme:

Key elements

• NIDOS Effectiveness Tool / Bond Organisational Health Check

• Value for Money

• NGO Evidence Principles

• Improve it Framework

bond.org.uk 5

Contributors to the programme

Consortium partners:

bond.org.uk 6

How the strands of programme fit together

Strong systems, culture, leadership etc

Bond Organisational Health Check/

NIDOS Effectiveness Tool

Robust, practical

approaches to measuring

effectiveness

Appropriate approaches to managing cost efficiencies & effectiveness

Development effectiveness

Improve It Framework

bond.org.uk

The Improve it Framework

What is it?

• A tool for improving how UK NGOs measure,

learn from and demonstrate outcomes and

impact

• A platform for sharing information on

outcomes, indicators and data collection tools

Why are we developing it?

• Developing robust methods for measuring

effectiveness is a collective challenge for UK

NGOs

8

• UK NGOs need to start telling a more robust story at sector level as well as

at organisational level about their added value

bond.org.uk 9

Improve it Framework: what it is and is not

• …about creating a single way of

assessing effectiveness –it’s about

encouraging greater harmonisation

and consistency where appropriate

• …going to offer an off the shelf

answer to measuring effectiveness,

but it will provide a common starting

point

• …going to produce an encyclopaedia

of indicators and tools. There will be

an element of prioritisation.

• …an organisation specific tool;

seeing 60% of your organisation in it

is good enough

• …about providing a collective resource

that UK NGOs can draw on when

developing their own context specific

M&E frameworks

• …about developing shared

approaches where we can

• ….about providing organisations with

the tools to be able to tell a more

robust story of how they are

contributing to change

• …going to be practical & simple

• …going to be dynamic & will continue

to evolve

It is not… It is…

bond.org.uk

1 2 3 4 5 6

Where the Improve it Framework can help in

your work

Focus of Improve it Framework

Identify the problem & underlying

causes

Articulate your vision of change

Identify who you will work

with & influence &

how

Identify what you want to

change

Identify the evidence

that you will collect and

how

Reflect and learn from the data

Communicate results

7 8

Collect, analyze and

draw conclusions for the data

Outcomes Indicators &

tools Principles

In practice

Capacity development

Improve it

Framework

1) All children access quality

education where they learn useful knowledge and

skills

2

Governments and other mandated bodies

ensure all children access quality

education

3

Schools are well managed, safe and well

resourced

4

All children receive their full allocation of high quality teaching

6

Communities actively support the education

of all children

Outcomes

a) Mothers, fathers and other caregivers support all their children to access education

b) Communities actively support and participate in children’s education

c) CSOs support and advocate for improved education for all

Outcomes

a) School management committees exist and manage schools effectively

b) School managers access, manage, and deploy resources in an effective and transparent way

c) Teachers, resources and

infrastructure are in place to

support education

d) Schools are safe and

healthy environments for girls

and boys

Outcomes

a) Teaching is high quality,

child-centred and child-

friendly

b) Teachers deliver a full

allocation of lessons

c) Teachers are trained and

supported to deliver high

quality teaching

d) The curriculum is relevant

and appropriate

e) Teaching supports equality

and empowerment

Domains of Change Framework for Education

Outcomes

a) Children participate in a meaningful way in decisions that affect their education

b) Children participate in their own and others’ learning

c) Children have a positive attitude towards education

Outcomes

a) Laws, policies and practices are in place that support quality education that is accessible for all children

b) Adequate resources are allocated and adequate teachers recruited and trained for the provision of quality education for all

c) Performance in education is monitored at a national and local level

5

All children participate in their education and

in school life

1) All children access quality

education where they learn useful knowledge and

skills

2

Governments and other mandated bodies

ensure all children access quality

education

3

Schools are well managed, safe and well

resourced

4

All children receive their full allocation of high quality teaching

6

Communities actively support the education

of all children

Outcomes

a) Mothers, fathers and other caregivers support all their children to access education

b) Communities actively support and participate in children’s education

c) CSOs support and advocate for improved education for all

Outcomes

a) School management committees exist and manage schools effectively

b) School managers access, manage, and deploy resources in an effective and transparent way

c) Teachers, resources and

infrastructure are in place to

support education

d) Schools are safe and

healthy environments for girls

and boys

Outcomes

a) Teaching is high quality,

child-centred and child-

friendly

b) Teachers deliver a full

allocation of lessons

c) Teachers are trained and

supported to deliver high

quality teaching

d) The curriculum is relevant

and appropriate

e) Teaching supports equality

and empowerment

Domains of Change Framework for Education

Outcomes

a) Children participate in a meaningful way in decisions that affect their education

b) Children participate in their own and others’ learning

c) Children have a positive attitude towards education

Outcomes

a) Laws, policies and practices are in place that support quality education that is accessible for all children

b) Adequate resources are allocated and adequate teachers recruited and trained for the provision of quality education for all

c) Performance in education is monitored at a national and local level

5

All children participate in their education and

in school life

1) All children access quality

education where they learn useful knowledge and

skills

2

Governments and other mandated bodies

ensure all children access quality

education

3

Schools are well managed, safe and well

resourced

4

All children receive their full allocation of high quality teaching

6

Communities actively support the education

of all children

Outcomes

a) Mothers, fathers and other caregivers support all their children to access education

b) Communities actively support and participate in children’s education

c) CSOs support and advocate for improved education for all

Outcomes

a) School management committees exist and manage schools effectively

b) School managers access, manage, and deploy resources in an effective and transparent way

c) Teachers, resources and

infrastructure are in place to

support education

d) Schools are safe and

healthy environments for girls

and boys

Outcomes

a) Teaching is high quality,

child-centred and child-

friendly

b) Teachers deliver a full

allocation of lessons

c) Teachers are trained and

supported to deliver high

quality teaching

d) The curriculum is relevant

and appropriate

e) Teaching supports equality

and empowerment

Domains of Change Framework for Education

Outcomes

a) Children participate in a meaningful way in decisions that affect their education

b) Children participate in their own and others’ learning

c) Children have a positive attitude towards education

Outcomes

a) Laws, policies and practices are in place that support quality education that is accessible for all children

b) Adequate resources are allocated and adequate teachers recruited and trained for the provision of quality education for all

c) Performance in education is monitored at a national and local level

5

All children participate in their education and

in school life

bond.org.uk 16

School management committees exist and manage schools effectively

Indicators Tools

• # and % of schools with a Parent Teacher

Association (PTA)

• Level of activity of the Parent Teacher

Association (PTA)

• # and % of schools with a School

management committee (SMC)

• Level of activity of the School

management committee (SMC)

The Right to Education indicators and form for section 9: right to transparent and accountable schools.

The Plan UK child friendly tool Section 1: School management

The VSO quality scale for education services quality areas on education management and community engagement

Timelines

bond.org.uk 22

What to look forward to

• Organisational Health Check pilot starts (Summer 2012)

• Launch of pilot Evidence Principles (October 2012)

• Beta version of the Improve it Framework (December 2012)

• Comic Relief, Big Lottery Fund, DFID, Bond research into the cost

of measuring effectiveness (Spring 2013)

• Guide to demystifying theories of change (Spring 2013)

bond.org.uk 23

Thank you

Contact:

Ben Jackson

Chief Executive

bjackson@bond.org.uk

Regent’s Wharf • 8 All Saints Street • London • N1 9RL • United Kingdom

+44 (0)20 7837 8344 • bond@bond.org.uk • bond.org.uk

bond.org.uk

What is it?

• An online self-assessment that:

• Identifies strengths and weaknesses in

organisational capacity / functionality

• Facilitates benchmarking with peers

• Sign posts to other initiatives, tools, training and

support

Why are we developing it?

• Strong systems, leadership etc are key building

blocks for delivering good development outcomes

24

Organisational Health Check

• An online self-assessment enables bond to provide more tailored

capacity support

• A common framework can help facilitate structured peer learning and the

sharing of best practice

bond.org.uk

Why are we doing it?

• At the heart of the VfM agenda are important questions about how UK NGOs

use their resources

• VfM is a part of the wider results agenda

• Want to ensure VfM remains appropriate and proportionate to NGO

operations and programmes

Value for Money (VfM)

What are we doing?

• Providing common tools and frameworks

for NGOs to navigate the VfM agenda

• Bringing NGOs together to share emerging

experience

• Ensuring constructive dialogue with DfID

25

bond.org.uk

NGO Evidence Principles (pilot to May 2013)

1. Voice and Inclusion: We listen to beneficiaries’ perspectives on the

effects of the intervention, and can identify who has been affected and

how.

2. Appropriateness: We use methods for data collection and analysis

that are justifiable given the nature of the intervention and purpose of the

assessment

3. Triangulation: We make conclusions about the intervention’s effects by

using a mix of methods, data sources, and perspectives

4. Contribution: We show how change happened, explore a range of

explanations for why it happened, test our theory of how we thought it

would happen, and explain how we contributed

5. Transparency: We are open about the data sources and methods

used, the results achieved, and the strengths and limitations of the

evidence

26

bond.org.uk

NGO Evidence Principles – the checklist

A NGO checklist for assessing the quality of evidence (DRAFT)

Piece of evidence being assessed: Name of assessor:

Principle Criteria 1 2 3 4 NA Comments/evidence

1)VoiceandInclusion

Welistento

beneficiaries’

perspectivesonthe

effectsofthe

intervention,andcan

identifywhohasbeen

affectedandhow

1a.Aretheperspectivesofbeneficiariesincludedinthe

evidence?

1b.Aretheperspectivesofthemostexcludedand

marginalisedgroupsincludedintheevidence?

1c.Arethefindingsdisaggregatedaccordingtosex,

disabilityandotherrelevantsocialdifferences?

1d.Dobeneficiariesplayanactiveroleindesigningthe

assessmentprocessand/oranalysingdata?

Scoreforvoiceandinclusion(averageofcriteriascores):

2)Appropriateness

Weusemethodsthat

arejustifiablegiven

thenatureofthe

interventionand

purposeofthe

assessment

2a.Arethemethodsusedtocollectdataappropriategiven

thenatureoftheinterventionandthepurposeofthe

assessment?

2b.Isthesizeandcompositionofthegroupfromwhich

dataiscollectedappropriategiventhenatureofthe

interventionandpurposeoftheassessment?

2c.Isthecompositionoftheteamcollectingandanalysing

thedataappropriategiventhenatureoftheintervention

andpurposeoftheassessment?

2d.Arethemethodsusedtoanalysethedataappropriate

giventhenatureoftheinterventionandpurposeofthe

assessment?

Scoreforvoiceandinclusion(averageofcriteriascores):

bond.org.uk

NGO Evidence Principles – the results

• A quality assurance

framework

• A kite mark / seal of

approval

• A tool for identifying skill

gaps and building capacity

28

Voice&Inclusion

13/16Appropriateness

7/16

Triangula on

12/16

Contribu on

4/16

Transparency

9/16