Impact Measurement and You

Understanding Impact Measurement, Practical Tools, Key Questions

PI’s Paper on IM• Locates IM within the Programme Approach –

therefore impact is measured at programme level.• Focuses on three key elements of impact

measurement:– Are we achieving the high-level goals we set out to

achieve [tracking change]?– Is our theory of change holding true [testing the ToC]?– What’s changing in the context which may influence the

above [on-going contextual analysis]?• The paper seeks to provide a mix of the conceptual

and the practical – but errs on the side of conceptual

Some Key Principles

• IM is reliant on (and can help build) a strong culture and processes for learning and knowledge management.

• Prior to work on IM, COs need to have a clear vision and understanding of the Theory of Change for their long-term programmes.

• IM focuses on contribution analysis at higher levels, drawn from strong M & E at lower levels.

• IM requires strong internal knowledge, but also partnership with researchers, think tanks, collegial organisations, etc.

• Emphasis has been placed on quantitative data and analysis, but we need to build more support for qualitative work as well.

Elements of IM

a bit more detain on key focus areas

Tracking Change

• Understanding – and then quantifying – the changes expressed in the Domains of Change and Impact Goal of each programme.

• High-level changes, which should show contribution to national and international standard indicators – e.g. the MDI plus list.

• This is a process of contribution analysis – seeking to understand our (CARE + Partners) contribution to population-level changes.

Testing the Theory of Change• The programme ToC is a set of hypotheses, which, if they hold

true, should give rise to the kinds of social change articulated in the Domains of Change and Impact Goal.

• Our hypotheses are based on evidence, but also on intuition. Some are foundational, and, if they haven’t already been “proven” will need to be – otherwise the ToC is a house of cards.

• Hypotheses for testing are based on critical assumptions, which, if do not hold true, threaten the overall ToC. They inform programming strategy, rather than being a part of the strategy.

• There are many ways to test key hypotheses: through new or existing initiatives; through targeted research; through literature reviews; etc.

On-Going Context Analysis

• The Theory of Change is located within evolving contexts, which must, on an on-going basis, be reviewed and integrated into the ToC and discussions about change.

• This is an opportunity to integrate the Programme Approach, IM and EPP processes.

• Timing will vary in different COs – e.g. Egypt’s rapidly changing context will require closer monitoring and meaning making than some other countries – the same is true for “emergency-prone” COs.

Knowledge Systems and Sub-Systems (with thanks to Tom Barton and ECARMU)

The importance of “holding systems”

Sub-systems from Tom Barton

Country Office Knowledge

System

Design, Monitoring & evaluation

(DM&E) sub-system

Impact Measurement &

Tracking (IM) sub-system

Knowledge Management sub-

systemLearning sub-system

Portfolio Coordination &

Management sub-system

DM&E System• Design: aims at the formulation (and on-going modification) of

useful, significant efforts toward alleviation of poverty, social injustice and enhancement of human dignity.

• M & E: the coordinated set of interlinked activities for gathering and analysing information, reporting and supporting decision-making and the implementation of improvements.

• Monitoring: the regular collection (plus analysis and use) of information about progress within the programme and its projects.

• Evaluation: periodic reviews and reflective practice toward information from within, as well as about, programmes and projects and their performance. Reflective practice refers to the process of challenging ourselves based on the information we get by asking key questions, e.g., why, so what, and now what?

Knowledge Management System

• Purpose: turning information into knowledge and evidence;

• Making knowledge accessible• Many COs have very weak and disbursed

knowledge management systems – some innovative ideas: sharepoints, M & E and Learning Units, etc.

Learning System

• Purpose: to support on-going improvements in personal & organisational practice; track and reassess what’s “right.”

• Learning Systems are based on:– Reflective Practice;– Communication;– Application.

• Learning happens by doing, and by reading and engaging others.

Synthesis

• COs generate data and knowledge through monitoring systems;

• This knowledge (and data) needs to be aligned to the relevant CO programme – and may form part of a learning agenda, or process toward exploring hypotheses;

• Learning systems need to support the interpretation of data and the use of knowledge – at initiative and programme levels.

Thoughts to Ponder

General Questions• How best to create dialogue about IM and

supporting systems?• What support is there for IM and supporting

systems?• How is the link between IM and PPLA’s current

work on knowledge management articulated?• In times of tight budgets, which are the most

important processes/elements?• Where can a repository of good practice be

housed?

WE-IM Specific Emerging Questions

• WE: Means to an end? End in itself? And is there space/appetite for this discussion?

• Common hypotheses to test in multiple contexts - e.g. contribution of WE to other outcomes? Role of VSLA in building self-efficacy? Others?

• Experimental pilots for measuring WE at impact level (Pathways, perhaps….?)

Other thoughts, questions, ideas, missing points???