www.nominettrust.org.uk

Theory of Change & Measuring Social Impact

www.nominettrust.org.uk

Theory of Change – Background and Use

• EVALUATION • Planning• Engagement• Communication (verbal and visual)

www.nominettrust.org.uk

Evaluation – 3 questions

What? So What? Now What?

Michael Quinn Patton

Exercise 1: What is my project/organisation trying to change?(Team Discussion – 15mins)

www.nominettrust.org.uk

Theory of Change - Definition

“No matter how it is defined, at its heart, a theory of change lays out what specific changes the group wants to see in the world, and how and why a group expects its actions to lead to those changes. “

(Guthrie et al. 2005 – emphasis by Lamb, 2011, p6)

Exercise 2: Theory of Change – Elevator Pitch (Groups – 20mins)

1. What are we (i.e. my project/organisation) trying to change?

2. What are we doing to bring about that change?

3. How and why do we believe our activities will bring about the change we want to see?

Outputs Outcomes

Tangible results of the your work

The changes that result (new behaviours, changes in well being, increased skills etc...

Outputs vs Outcomes

Logic Models

Causality – ‘Black Box’ Evaluations

Programme or Intervention(e.g. parenting course)

Outcomes(e.g. happy, well-adjusted children)

www.nominettrust.org.uk

Theory of Change approach – In a word

REALLY?

www.nominettrust.org.uk

Questioning Causality (aka ‘exploding the arrow’)

i.e. asking whether an organisation’s activities will bring about the changes they intend in the way they describe.

www.nominettrust.org.uk

Causality ‘Exploding the Arrow’ (1)

1) Evidence – what evidence is there the intervention will lead to the desired outcome?

2) Single or multiple steps – will the intervention lead directly to the desired outcome*

*....or are there a number of micro steps along the way? Is there a chain of connected outcomes? (aka: ‘so that’ chains)

(Funnell and Rogers, 2011)

Outcomes Chain – Example (Text)

Causality – Outcomes Chains (blank)

Programme or Intervention(e.g. parenting course)

Outcomes(e.g. happy, well-adjusted children)

? ? ?

???

Exercise 3: Discussion – Outcomes Chain (Team – 10mins)

www.nominettrust.org.uk

Causality ‘Exploding the Arrow’ (2)

3) Linear or multifaceted – is the outcomes chain a simple linear process or is it multifaceted?*

4) Internal and External factors – does the outcomes chain take into account the influence of internal and external factors?

*Do separate outcomes chains need to run in parallel in order to combine towards the final, ultimate outcome? Necessary vs. Sufficient.

(Funnell and Rogers, 2011)

www.thinknpc.org

www.nominettrust.org.uk

Causality ‘Exploding the Arrow’ (3)

5) Plausibility and coherency – ultimately any plan that details a causal link between an intervention and an outcome needs to be scrutinised, ideally by those who have knowledge relevant to the intended change (e.g. practitioners, academics and intended beneficiaries).

(Funnell and Rogers, 2011)

www.nominettrust.org.uk

Making the implicit explicit

“..takes for granted that social programs are based on explicit or implicit theories about how and why the programme will work. The evaluation should surface those theories and lay them out in as fine detail as possible, identifying all the assumptions and sub-assumptions built into the program. … The aims is to examine the extent to which the program theories hold.” (Weiss, 1995, pp66-7)

Assumptions

Programme or Intervention(e.g. parenting course)

Outcomes(e.g. happy, well adjusted children)

? ? ?

? ? ?

?

Exercise 4: Assumptions – Outcomes Chain (Groups – 15mins)

www.nominettrust.org.uk

What makes a good theory of change? (1)1) Plausible -

stakeholders believe the logic of the model is correct: if we do these things, we will get the results we want and expect

2) Doable - human, political and economic resources are seen as sufficient to implement the action strategies in the theory

Source: https://communities.usaidallnet.gov/fa/system/files/Applying+Theory+of+Change+Approach.pdf

www.nominettrust.org.uk

What makes a good theory of change? (2)3) Testable -

stakeholders believe there are credible ways to discover whether the results are as predicted)

4) Meaningful - stakeholders see the outcomes as important and the magnitude of change in these outcomes being pursued as worth the effort

www.nominettrust.org.uk

Theories of Change – Some Examples

From Mayne, J (2008)

One Project – Several theories of changeSome projects use a limited number of activities to serve multiple purposes. For example, a project involving the construction of a water well in a village in Burundi could be linked to the following theories of change:

• By providing access to a closer water source, we will improve safety (less danger of assault or rape, as is the case in many refugee camps) and health (improved sanitation, no contamination in drinking water, ability to cook with water).

• If there is closer access to water, then children can attend school instead of having to travel for hours to provide water for the family.

• If we teach locals how to build water wells, then they can repeat the process in neighbouring villages, thus spreading access to water throughout the region.

Practical Approaches to Theories of Change in Conflict, Security & Justice Programmes. Department for International Development

www.nominettrust.org.uk

Defining success – the ToC approach:

1) Offers guidance on how to get to your ultimate goal, it will not tell you what that goal should be

2) Is dependent on having a clearly articulated ultimate goal, but defining, agreeing and sticking to such a clear can be a challenge

www.nominettrust.org.uk

Theory of Change Toolbox

ToC approach contains different elements and levels of detail that will appeal variously to organisations depending on their interests, values, and size.

Theory of Leverage

“Give me a place to stand and with a lever I will move the whole world.” Archimedes

Lean Start Up and Theories of Change:

Interesting parallels e.g. Testing assumptions (ToC) and validating hypotheses (LSU).

www.theleanstartup.com/

www.nominettrust.org.uk

Criticisms and Challenges

• Limited research available – especially for non-US contexts and smaller charities

• Most documentation produced by ToC ‘champions’

• ToC still an emerging approach (no orthodoxy)

www.nominettrust.org.uk

Criticisms and Challenges

Complexity Science and Complex Adaptive Systems1. Complex systems can be defined as having

various characteristics including a large number of interacting elements and being dynamic compositions, where the whole is larger than sum of parts (Snowden & Boone, 2007, p71).

2. Cause and effect pathways often unclear, and visible only in hindsight

Further reflections

The more complex (and less linear) the social change the more difficult it is to develop a ToC

ToC approach about unearthing assumptions, but also values, beliefs and paradigms

Tension - evaluators like stable theories and models, whilst innovators happier working with still evolving models

Half-life/strategic decay. Also applies to ToC, which need to be refreshed/reviewed. ToC change. The three main reasons are: 1) context changes2) new players/change makers, and 3) new evidence. Complex change = variety (Gladwell’s pasta sauces), i.e. not 'what works', but 'what works for who, in what context and why?'

ToC Approach – better as a thinking tool than a planning tool?

ResourcesPapers & Books• Brest, P. (2010). The Power of Theories of Change. Available:

http://www.kipp.org/news/stanford-social-innovation-review-the-power-of-theories-of-change- • Clark, H and Anderson, A. (November 2004) Theories of Change and Logic Models: Telling Them Apart. Presentation at American

Evaluation Association, Atlanta, Georgia. http://www.theoryofchange.org/wp-content/uploads/toco_library/pdf/TOCs_and_Logic_Models_forAEA.pdf

• Funnell, S. C. and Rogers, P. J. (2011) Purposeful Program Theory: Effective Use of Theories of Change and Logic Models. Jossey-Bass• Keystone Accountability. (2008). Developing a theory of change: A guide to developing a theory of change as a framework for

inclusive dialogue, learning and accountability for social impact. Available: http://www.researchtoaction.org/theory-of-change-useful-resources/

• Lamb, B. (February 2011) Campaigning for Change: Learning from the United States. Campaigning Effectiveness, NCVO. http://www.ncvo-vol.org.uk/sites/default/files/campaigning_for_change_-_learning_from_the_us.pdf orhttp://www.advocacyinitiative.ie/resource/campaigning-change-learning-united-states

• Mayne, J (2208) Contribution Analysis: An approach to exploring cause and effect, ILAC, http://www.cgiar-ilac.org/files/ILAC_Brief16_Contribution_Analysis_0.pdf

• Stachowiak, S. (2009) Pathways for Change: 6 Theories About How Policy Change Happens. Seattle, Organizational Research Services.http://www.organizationalresearch.com/publicationsandresources/pathways_for_change_6_theories_about_how_policy_change_happens.pdf

• Weiss, C. (1995) ‘Nothings as practical as a good theory: Exploring Theory-Based Evaluation for Comprehensive Community Initiatives for Children and Families’. In Connell, J. P. et al. (eds) Approaches to Evaluating Community Initiatives: Concepts, Methods and Contexts. Washington; Aspen Institute. Available: http://www.eric.ed.gov:80/PDFS/ED383817.pdf

Links:http://www.theoryofchange.orghttp://www.thinknpc.org/publications/theory-of-change/ http://www.ces-vol.org.uk/tools-and-resources/Evaluation-methods/making-connections-toolshttp://www.aecf.org/resources/theory-of-change/ www.whatleads.to

For a critical view:• Reusga, A. (2011). Philanthropy’s Albatross: Debunking Theories of Change.

http://postcards.typepad.com/Philanthropy%20s%20Albatross.doc