impact evaluation an overview
TRANSCRIPT
![Page 1: Impact evaluation an overview](https://reader036.vdocuments.us/reader036/viewer/2022081604/58a312581a28ab1d068b50a5/html5/thumbnails/1.jpg)
Research Methodology: An Overview of Impact Evaluation
BY: IBITOLA OMOTAYO23RD SEPTEMBER 2016
![Page 2: Impact evaluation an overview](https://reader036.vdocuments.us/reader036/viewer/2022081604/58a312581a28ab1d068b50a5/html5/thumbnails/2.jpg)
Knowledge Check 1. What is Impact Evaluation? 2. Is there a need for Impact Evaluation? If yes, why? If no, why not? 3. What are the methods for conducting Impact Evaluations?
![Page 3: Impact evaluation an overview](https://reader036.vdocuments.us/reader036/viewer/2022081604/58a312581a28ab1d068b50a5/html5/thumbnails/3.jpg)
ObjectivesTo understand the:key concepts of Impact Evaluation.aspects of Impact Evaluation.3Ws of Impact Evaluation: why, what, and who methods for conducting Impact Evaluation.considerations for ensuring quality in Impact Evaluation
![Page 4: Impact evaluation an overview](https://reader036.vdocuments.us/reader036/viewer/2022081604/58a312581a28ab1d068b50a5/html5/thumbnails/4.jpg)
Impact Evaluation: what is it? “IMPACT” refers to the positive and negative, intended and unintended, direct
and indirect, primary and secondary effects produced by an intervention. (OECD)
•Impact evaluations measure the change in a development outcome that is
attributable to a defined intervention; (USAID)
•Impact evaluation is a systematic and empirical assessment of the effects brought
about by an intervention. (AusAID)
![Page 5: Impact evaluation an overview](https://reader036.vdocuments.us/reader036/viewer/2022081604/58a312581a28ab1d068b50a5/html5/thumbnails/5.jpg)
Aspects of Impact Evaluation
Impact Evaluation
Formative(ex-ante)
Summative
![Page 6: Impact evaluation an overview](https://reader036.vdocuments.us/reader036/viewer/2022081604/58a312581a28ab1d068b50a5/html5/thumbnails/6.jpg)
Formative Aspect of Impact Evaluation
Seeks to inform decisions with regards to making changes to a programme or
policy.
While many formative evaluations (i.e. baseline evaluation) focus on processes,
impact evaluations can be used formatively if an intervention is ongoing. For
example, the findings of an impact evaluation can be used to improve
implementation of a programme for the next intake of participants.
![Page 7: Impact evaluation an overview](https://reader036.vdocuments.us/reader036/viewer/2022081604/58a312581a28ab1d068b50a5/html5/thumbnails/7.jpg)
Summative Aspect of Impact Evaluation
Seeks to inform decisions about whether to continue, discontinue, replicate or
scale up an intervention.
Ideally, summative impact evaluation not only produces findings about ‘what
works’ but also provides information about what is needed to make the
intervention work for different groups in different settings, which can then be
used to inform decisions.
![Page 8: Impact evaluation an overview](https://reader036.vdocuments.us/reader036/viewer/2022081604/58a312581a28ab1d068b50a5/html5/thumbnails/8.jpg)
Impact Evaluation: Why should it be done?
•To decide whether to fund an intervention– “ex-ante evaluation” is conducted before
an intervention is implemented to estimate its likely impacts and inform funding
decisions.
•To decide whether or not to continue or expand an intervention.
•To learn how to replicate scale up a pilot.
•To learn how to successfully adapt a successful intervention to suit another context.
![Page 9: Impact evaluation an overview](https://reader036.vdocuments.us/reader036/viewer/2022081604/58a312581a28ab1d068b50a5/html5/thumbnails/9.jpg)
Impact Evaluation: Why should it be done?•To show accountability:
reassure funders, including donors and taxpayers (upward accountability),
that money is being wisely invested.
inform intended beneficiaries and communities (downward accountability)
about whether or not, and in what ways, a program is benefiting the
community
![Page 10: Impact evaluation an overview](https://reader036.vdocuments.us/reader036/viewer/2022081604/58a312581a28ab1d068b50a5/html5/thumbnails/10.jpg)
What Questions Does Impact Evaluation Seek to Answer?High-level questions directed at the overall purpose of the intervention and should address-:
1. Overall impact
2. Nature of impacts and their distribution
3. Influence of other factors on the impacts
4. Mode of operation
5. Alignment of intended impacts to needs***These are not the specific questions that are to be asked during interviews
![Page 11: Impact evaluation an overview](https://reader036.vdocuments.us/reader036/viewer/2022081604/58a312581a28ab1d068b50a5/html5/thumbnails/11.jpg)
Overall impact
•Did it work? Did the intervention produce the intended impacts in the short,
medium and long term?
•For whom, in what ways and in what circumstances did the intervention work?
•What unintended impacts (positive and negative) did the intervention
produce?
![Page 12: Impact evaluation an overview](https://reader036.vdocuments.us/reader036/viewer/2022081604/58a312581a28ab1d068b50a5/html5/thumbnails/12.jpg)
Nature of impacts and their distribution
•Are the impacts likely to be sustainable? •Did these impacts reach all intended beneficiaries?
impact
![Page 13: Impact evaluation an overview](https://reader036.vdocuments.us/reader036/viewer/2022081604/58a312581a28ab1d068b50a5/html5/thumbnails/13.jpg)
Influence of other factors on the impacts
•How did [the intervention] work in conjunction with other interventions, programs or services to achieve outcomes?
•What helped or hindered [the intervention] to achieve these impacts?
![Page 14: Impact evaluation an overview](https://reader036.vdocuments.us/reader036/viewer/2022081604/58a312581a28ab1d068b50a5/html5/thumbnails/14.jpg)
Mode Of Operation
•How did the intervention contribute to intended impacts?•What were the particular features of the intervention that made a difference?•What variations were there in implementation?•What has been the quality of implementation in different sites?•To what extent are differences in impact explained by variations in implementation?
![Page 15: Impact evaluation an overview](https://reader036.vdocuments.us/reader036/viewer/2022081604/58a312581a28ab1d068b50a5/html5/thumbnails/15.jpg)
Alignment of intended impact to needs•To what extent did the impacts match the needs of the intended beneficiaries?
![Page 16: Impact evaluation an overview](https://reader036.vdocuments.us/reader036/viewer/2022081604/58a312581a28ab1d068b50a5/html5/thumbnails/16.jpg)
Who Should Conduct Impact Evaluation?
Impact evaluation can be undertaken by:
an external evaluator or evaluation team (e.g Preston –ENGINE)
an internal but separate unit of the implementing organization (e.g. M&E dept
within implementing organizations)
those involved in an intervention (including community members);
a combined team of internal and external evaluators.
![Page 17: Impact evaluation an overview](https://reader036.vdocuments.us/reader036/viewer/2022081604/58a312581a28ab1d068b50a5/html5/thumbnails/17.jpg)
Choosing Methods for Impact Evaluation?
Evaluation design
& Estimation methods
purpose of the
evaluation
availability of resources
nature of the
intervention
![Page 18: Impact evaluation an overview](https://reader036.vdocuments.us/reader036/viewer/2022081604/58a312581a28ab1d068b50a5/html5/thumbnails/18.jpg)
Evaluation designs
http://www.socialresearchmethods.net
![Page 19: Impact evaluation an overview](https://reader036.vdocuments.us/reader036/viewer/2022081604/58a312581a28ab1d068b50a5/html5/thumbnails/19.jpg)
Evaluation designsExperimental
• Independent measures
• Repeated Measures• Matched Pairs
Quasi- experimental
• Time series• Separate sample
pre-test/post-test• Separate sample
pre-test/post-test control group etc….
Non-experimental
• single-variable research
• correlational and quasi-experimental research
• qualitative research
www.simplypsychology.org/experimental-designs.html
![Page 20: Impact evaluation an overview](https://reader036.vdocuments.us/reader036/viewer/2022081604/58a312581a28ab1d068b50a5/html5/thumbnails/20.jpg)
Estimation methods The are usually based on the evaluation designs. Different designs require different estimation methods to measure the counterfactual.
The four categories of estimation methods are:◦ Comparison of means◦ Multi-variate regression◦ Instrumental variables methods◦ Double difference or difference-in-difference.
*** Counterfactual: a comparison between what actually happened and what would have happened in the absence of the intervention
![Page 21: Impact evaluation an overview](https://reader036.vdocuments.us/reader036/viewer/2022081604/58a312581a28ab1d068b50a5/html5/thumbnails/21.jpg)
Considerations for quality Impact Evaluation•UTILITY
•ACCURACY
•PROPRIETY (ETHICS)
•PRACTICALITY
•ACCOUNTABILITY
![Page 22: Impact evaluation an overview](https://reader036.vdocuments.us/reader036/viewer/2022081604/58a312581a28ab1d068b50a5/html5/thumbnails/22.jpg)
Considerations for quality Impact Evaluation
•UTILITY: the results of the evaluation must be useful to those who require the information.
•ACCURACY: ensuring that findings are reported fairly, comprehensively and clearly.
•PROPRIETY (ETHICS):ethical issues of confidentiality and anonymity, as well as potential harmful effects of being involved in the evaluation must be adequately addressed.
![Page 23: Impact evaluation an overview](https://reader036.vdocuments.us/reader036/viewer/2022081604/58a312581a28ab1d068b50a5/html5/thumbnails/23.jpg)
Considerations for quality Impact Evaluation?
•PRACTICALITY: taking into account the available resources (time, money, expertise and existing data) and when the results are needed to inform decisions.
•ACCOUNTABILITY: refers to presenting clear evidence and criteria on which conclusions have been drawn, and acknowledging their limitations. Transparency about data sources is important.
![Page 24: Impact evaluation an overview](https://reader036.vdocuments.us/reader036/viewer/2022081604/58a312581a28ab1d068b50a5/html5/thumbnails/24.jpg)
QUESTIONS & COMMENTS
![Page 25: Impact evaluation an overview](https://reader036.vdocuments.us/reader036/viewer/2022081604/58a312581a28ab1d068b50a5/html5/thumbnails/25.jpg)
More Information
www.betterevaluation.orginteractive textbook at http://www.worldbank.org/pdthttps://www.cdc.gov/std/Program/pupestd/Types%20of%20Evaluation.pdfhttp://www.interaction.org/impact-evaluation-notes.https://dfat.gov.au/aid/how-we-measure-performance/ode/Documents/impact-evaluation-discussion-paper.pdf
![Page 26: Impact evaluation an overview](https://reader036.vdocuments.us/reader036/viewer/2022081604/58a312581a28ab1d068b50a5/html5/thumbnails/26.jpg)
![Page 27: Impact evaluation an overview](https://reader036.vdocuments.us/reader036/viewer/2022081604/58a312581a28ab1d068b50a5/html5/thumbnails/27.jpg)
Post-test 1. What is impact evaluation? 2. Is there a need for impact evaluation? If yes, why? If no, why not? 3. What are the methods for conducting Impact evaluations?