MethodsEnabling Constituency Voice

in Impact Evaluation for Improving Development

Andre Proctor, Keystone

March 2009

Rethinking IMPACT

IPAL innovations

What is Constituency Voice Bringing the voices of constituents,

especially those meant to benefit, meaningfully into decision-making.

All constituents play a role in How we define successHow we work towards it (strategy)How we learn (measure impacts and reflect

on their meaning)How we communicate it (reporting and

sharing)

CV addresses the 6 purposes of Impact Planning, Assessment and Learning (1) Improve projects

ensures usefulness for constituents Demonstrate impact

from different perspectives on key enablers of change, insight in causal mechanism, triangulation

Inform strategy real time feedback

CV addresses the 6 purposes of Impact Planning, Assessment and Learning (2) Sustain credibility and legitimacy

Confidence and trust among constituents Build capacity

empowers constituents Educate society

reporting re-imagined as a learning activity both for constituents and wider society

Comparative Constituency Feedback

An alternative approach to measuring performance and driving improvements: “Ask them!” Ask recipients how useful they find an

organisation’s work.

Related to ‘customer satisfaction’. Empowers primary constituents.

An emerging fieldMacro level: Humanitarian Response Index

www.daraint.org All OECD government funding for humanitarian response

Center for Effective Philanthropy

www.effectivephilanthropy.org Grantee perceptions of US foundations

 

Micro level: Measuring Empowerment

quality.bond.org.uk Social movement in Bangladesh Listen First

www.listenfirst.org Concern Worldwide pilot

Comparative feedback

Comparative constituency feedback Structured feedback from primary constituents

on their experience of receiving funding / services / assistance.

Generates quantitative data, for a group of similar actors. Performance can be aggregated and benchmarked.

Provides data from the bottom upwards for learning and improvement.

Case study

East-African grantmakers Cohort of 10 grantmakers. Keystone surveyed all of their grantees.

50% response rate, 336 completed surveysSix key areas:

Nature of funding Application processMonitoring, reporting

& evaluation

Relationship Non-financial supportGrantmaker’s knowledge &

influence

Ethical FrameworkArea Purpose

Primary constituents are involved in designing survey

Constituency voiceRelevance

Independently administered Objectivity

Informed consent of respondents

No one is forced to participate

Anonymous responses & data kept confidential

No discrimination

Report back to constituents Shared learning and open dialogue

Summary findings (extract)

“XXX’s grantees are very satisfied with XXX’s application processes, rating it 15.3 out of 20. … XXX keeps grantees better informed of progress and provides them with clearer information than most other grantmakers.”

“XXX’s grantees rated it 6.4 out of 20 for non-financial support (e.g. capacity building or advice). All grantmakers struggle in this area. Around 40% of XXX’s grantees do not receive assistance in each area; and in some areas over 20% of grantees receive assistance that they do not find helpful.”

Follow up questions… “Could XXX consider:

more site visits, and think about what would be most valuable to do on a site visit?

reviewing its communications – especially web site and its annual report as communication tools?

improving the way that it listens and responds to feedback from Grantees (responding to the feedback from this survey could be a good start).

developing its staff understanding of developmental issues and create opportunities for learning from constituents?

others …”

Conclusion Robust data on grantmaker’s performance – not

just focusing on grantees. Comparison drives insight and learning;

approach fits with other monitoring activities. Drives real change (e.g. CEP). Adaptable to many different situations; the field

is emerging. Puts the rhetoric of empowerment and

accountability into practice. The future … ??

Discussion questions CV How does Constituency Voice contribute

to Impact Evaluation for Improving Development?

How useful could Comparative Constituency Feedback be in what you do?