poverty action lab: translating research into actionpubdocs.worldbank.org › en ›...
TRANSCRIPT
![Page 1: Poverty Action Lab: Translating Research into Actionpubdocs.worldbank.org › en › 908771463426766162 › Core-2... · Poverty Action Lab: Translating Research into Action . MIT](https://reader036.vdocuments.us/reader036/viewer/2022081407/5f23397d07fa8e71780e498e/html5/thumbnails/1.jpg)
povertyactionlab.org
Poverty Action Lab: Translating Research into Action
MIT
![Page 2: Poverty Action Lab: Translating Research into Actionpubdocs.worldbank.org › en › 908771463426766162 › Core-2... · Poverty Action Lab: Translating Research into Action . MIT](https://reader036.vdocuments.us/reader036/viewer/2022081407/5f23397d07fa8e71780e498e/html5/thumbnails/2.jpg)
povertyactionlab.org
The what, why and how of evaluations
Abhijit Vinayak Banerjee MIT and Poverty Action Lab
![Page 3: Poverty Action Lab: Translating Research into Actionpubdocs.worldbank.org › en › 908771463426766162 › Core-2... · Poverty Action Lab: Translating Research into Action . MIT](https://reader036.vdocuments.us/reader036/viewer/2022081407/5f23397d07fa8e71780e498e/html5/thumbnails/3.jpg)
povertyactionlab.org
What is Evaluation
Start with a Real Example
![Page 4: Poverty Action Lab: Translating Research into Actionpubdocs.worldbank.org › en › 908771463426766162 › Core-2... · Poverty Action Lab: Translating Research into Action . MIT](https://reader036.vdocuments.us/reader036/viewer/2022081407/5f23397d07fa8e71780e498e/html5/thumbnails/4.jpg)
Program • What was the program?
– Flipcharts donated to village primary schools • Where were the primary schools?
– Western Province of Kenya • When did this happen?
– 1997 • Who donated the flipcharts?
– International Child Support Africa (ICS Africa) (a local chapter of a Dutch NGO)
• Who, specifically, were they meant for? – For Students in 6th - 8th Grade
Why flipcharts?
![Page 5: Poverty Action Lab: Translating Research into Actionpubdocs.worldbank.org › en › 908771463426766162 › Core-2... · Poverty Action Lab: Translating Research into Action . MIT](https://reader036.vdocuments.us/reader036/viewer/2022081407/5f23397d07fa8e71780e498e/html5/thumbnails/5.jpg)
Three sets of questions
Needs Assessment
Impact Evaluation
Process Evaluation
![Page 6: Poverty Action Lab: Translating Research into Actionpubdocs.worldbank.org › en › 908771463426766162 › Core-2... · Poverty Action Lab: Translating Research into Action . MIT](https://reader036.vdocuments.us/reader036/viewer/2022081407/5f23397d07fa8e71780e498e/html5/thumbnails/6.jpg)
Needs Evaluation • Why are we doing this project: Laying out the theory • What’s the nature of the problem being solved?
– To compensate for a shortage of textbooks – To compensate for a lack of teaching and learning materials in
general – To help those who cannot read English very well – To help those who find it easier to learn from pictures – How will flipcharts solve it? – Why do we need to answer this question?
• Who is the targeted population? – All children? – ? – Why do we need to answer this question?
![Page 7: Poverty Action Lab: Translating Research into Actionpubdocs.worldbank.org › en › 908771463426766162 › Core-2... · Poverty Action Lab: Translating Research into Action . MIT](https://reader036.vdocuments.us/reader036/viewer/2022081407/5f23397d07fa8e71780e498e/html5/thumbnails/7.jpg)
Needs Evaluation
• Why do we think it will work in this context? – Do teachers feel comfortable using them? – Does it fit with the curriculum?
• What else could we do?
– Adopt a different strategy for addressing the same need
– Do something entirely different with the money – Why do we need to answer this question?
![Page 8: Poverty Action Lab: Translating Research into Actionpubdocs.worldbank.org › en › 908771463426766162 › Core-2... · Poverty Action Lab: Translating Research into Action . MIT](https://reader036.vdocuments.us/reader036/viewer/2022081407/5f23397d07fa8e71780e498e/html5/thumbnails/8.jpg)
Needs Evaluation Should Provide
• Clear sense of target population – Weaker students – Students with poor English – ?
• Clear sense of program design – What are teachers lacking in terms using these charts? – How to deliver? How much? What are potential barriers?
• Clear articulation of program benefits – Is a wrong being righted? Is a right being expanded?
• Clear sense of alternatives – Is this the most effective, efficient, cost-effective method of
meeting teacher/student needs?
![Page 9: Poverty Action Lab: Translating Research into Actionpubdocs.worldbank.org › en › 908771463426766162 › Core-2... · Poverty Action Lab: Translating Research into Action . MIT](https://reader036.vdocuments.us/reader036/viewer/2022081407/5f23397d07fa8e71780e498e/html5/thumbnails/9.jpg)
Process Evaluation • Is the program actually doing what it says? • Are the services being delivered?
– Are flipcharts are reaching the classroom, are teachers are being trained to use them
• Are there ways of improving cost effectiveness? – Are the flipcharts being procured/delivered in the best
possible way? • Are the services reaching the right population?
– Are they being targeted to the classes with lower-performing students?
• Are the clients satisfied with service? – Teachers’, students’ response to new teaching method
![Page 10: Poverty Action Lab: Translating Research into Actionpubdocs.worldbank.org › en › 908771463426766162 › Core-2... · Poverty Action Lab: Translating Research into Action . MIT](https://reader036.vdocuments.us/reader036/viewer/2022081407/5f23397d07fa8e71780e498e/html5/thumbnails/10.jpg)
Impact Evaluation
• What did the program achieve? • This question is only well-defined if we
have determined: • What measures do we expect impacts on? • What populations do we expect impact for? • When should we have answered
these questions?
![Page 11: Poverty Action Lab: Translating Research into Actionpubdocs.worldbank.org › en › 908771463426766162 › Core-2... · Poverty Action Lab: Translating Research into Action . MIT](https://reader036.vdocuments.us/reader036/viewer/2022081407/5f23397d07fa8e71780e498e/html5/thumbnails/11.jpg)
Impact Evaluation
• Key question: Was there an impact? • Auxiliary questions:
– What was the nature of the benefit? • For example: Did flipcharts lead to uniform
improvements in test scores (for all students)? – Did some types of people benefit more
than others? • Students who were doing worse, poorer students,
etc.
![Page 12: Poverty Action Lab: Translating Research into Actionpubdocs.worldbank.org › en › 908771463426766162 › Core-2... · Poverty Action Lab: Translating Research into Action . MIT](https://reader036.vdocuments.us/reader036/viewer/2022081407/5f23397d07fa8e71780e498e/html5/thumbnails/12.jpg)
Cost Benefit Analysis
Cost-Benefit Analysis
Process Evaluation
Impact Evaluation
Needs Assessment
![Page 13: Poverty Action Lab: Translating Research into Actionpubdocs.worldbank.org › en › 908771463426766162 › Core-2... · Poverty Action Lab: Translating Research into Action . MIT](https://reader036.vdocuments.us/reader036/viewer/2022081407/5f23397d07fa8e71780e498e/html5/thumbnails/13.jpg)
Cost Benefit Analysis
• Why are all three elements useful for a cost benefit analysis?
• Impact evaluation gives benefits • Process evaluation allows careful
quantification of costs – May help to lower costs or increase efficiency
• Needs analysis frames opportunity cost of money
![Page 14: Poverty Action Lab: Translating Research into Actionpubdocs.worldbank.org › en › 908771463426766162 › Core-2... · Poverty Action Lab: Translating Research into Action . MIT](https://reader036.vdocuments.us/reader036/viewer/2022081407/5f23397d07fa8e71780e498e/html5/thumbnails/14.jpg)
povertyactionlab.org
This Week • Focus on impact evaluation: Why? • It is what we know how to do well. • Surprisingly little is known about what works:
– Empowerment and Poverty Reduction: a Source-book – World Bank publication listing what works – Reports that the Gyandoot program that put computers in
villages in India was hard hit by lack of power. The computer kiosks lost money and many had shut.
– Then says, without any apparent irony, “Following the success of the program..”
![Page 15: Poverty Action Lab: Translating Research into Actionpubdocs.worldbank.org › en › 908771463426766162 › Core-2... · Poverty Action Lab: Translating Research into Action . MIT](https://reader036.vdocuments.us/reader036/viewer/2022081407/5f23397d07fa8e71780e498e/html5/thumbnails/15.jpg)
povertyactionlab.org
Why impact evaluation?
• Central issues in policy design – Do we know that anything is working? – How do we identify what works? – Pick what really works
![Page 16: Poverty Action Lab: Translating Research into Actionpubdocs.worldbank.org › en › 908771463426766162 › Core-2... · Poverty Action Lab: Translating Research into Action . MIT](https://reader036.vdocuments.us/reader036/viewer/2022081407/5f23397d07fa8e71780e498e/html5/thumbnails/16.jpg)
povertyactionlab.org
How much is the right answer worth?
![Page 17: Poverty Action Lab: Translating Research into Actionpubdocs.worldbank.org › en › 908771463426766162 › Core-2... · Poverty Action Lab: Translating Research into Action . MIT](https://reader036.vdocuments.us/reader036/viewer/2022081407/5f23397d07fa8e71780e498e/html5/thumbnails/17.jpg)
povertyactionlab.org
Why Impact Evaluation?
• The devil, unfortunately, seems to be in the details.
• A great advantage of combining design with evaluation is that we can focus on the details
• This not only allows the possibility of refining them but also of redesigning them.
![Page 18: Poverty Action Lab: Translating Research into Actionpubdocs.worldbank.org › en › 908771463426766162 › Core-2... · Poverty Action Lab: Translating Research into Action . MIT](https://reader036.vdocuments.us/reader036/viewer/2022081407/5f23397d07fa8e71780e498e/html5/thumbnails/18.jpg)
How to assess impact
What is beneficiary’s test score with program compared to without program?
Compare same individual with & without programs at same point in time
Formally, program impact is: α = (Y | P=1) - (Y | P=0)
e.g. How much does an education program improve test scores (learning)?
![Page 19: Poverty Action Lab: Translating Research into Actionpubdocs.worldbank.org › en › 908771463426766162 › Core-2... · Poverty Action Lab: Translating Research into Action . MIT](https://reader036.vdocuments.us/reader036/viewer/2022081407/5f23397d07fa8e71780e498e/html5/thumbnails/19.jpg)
Solving the evaluation problem
Estimated impact is difference between treated observation and counterfactual.
Need to estimate counterfactual.
Never observe same individual with and without program at same point in time.
Counterfactual is key to impact evaluation.
We need a counterfactual
![Page 20: Poverty Action Lab: Translating Research into Actionpubdocs.worldbank.org › en › 908771463426766162 › Core-2... · Poverty Action Lab: Translating Research into Action . MIT](https://reader036.vdocuments.us/reader036/viewer/2022081407/5f23397d07fa8e71780e498e/html5/thumbnails/20.jpg)
Counterfactual Criteria Treated & Counterfactual (1) Have identical characteristics (2) Except for benefitting from the intervention No other reason for differences in outcomes of treated and counterfactual
Only reason for the difference in outcomes is due to the intervention.
![Page 21: Poverty Action Lab: Translating Research into Actionpubdocs.worldbank.org › en › 908771463426766162 › Core-2... · Poverty Action Lab: Translating Research into Action . MIT](https://reader036.vdocuments.us/reader036/viewer/2022081407/5f23397d07fa8e71780e498e/html5/thumbnails/21.jpg)
2 Counterfeit Counterfactuals
Those not enrolled
o Those who choose not to enroll
in the program o Those who were not offered
the program
Before and after
![Page 22: Poverty Action Lab: Translating Research into Actionpubdocs.worldbank.org › en › 908771463426766162 › Core-2... · Poverty Action Lab: Translating Research into Action . MIT](https://reader036.vdocuments.us/reader036/viewer/2022081407/5f23397d07fa8e71780e498e/html5/thumbnails/22.jpg)
1. Before and After: Examples
School scholarship program on enrollment
Agricultural assistance program o Financial assistance to purchase inputs. o Compare rice yields before and after. o Before is normal rainfall, but after is drought. o Find fall in rice yield. o Did the program fail? o Could not separate (identify) effect of financial
assistance program from effect of rainfall.
![Page 23: Poverty Action Lab: Translating Research into Actionpubdocs.worldbank.org › en › 908771463426766162 › Core-2... · Poverty Action Lab: Translating Research into Action . MIT](https://reader036.vdocuments.us/reader036/viewer/2022081407/5f23397d07fa8e71780e498e/html5/thumbnails/23.jpg)
Those not enrolled: Example of job training
Compare employment & earning of those who sign up to those who did not
Who signs up? Those who are most likely to benefit -i.e. those with more ability- would have higher earnings than non-participants without job training
Poor estimate of counterfactual
![Page 24: Poverty Action Lab: Translating Research into Actionpubdocs.worldbank.org › en › 908771463426766162 › Core-2... · Poverty Action Lab: Translating Research into Action . MIT](https://reader036.vdocuments.us/reader036/viewer/2022081407/5f23397d07fa8e71780e498e/html5/thumbnails/24.jpg)
Those not enrolled: Example of those offered health insurance
With no insurance: Those that did not buy, have lower medical costs than that did
Compare health care utilization of those who bought insurance to those who did not o Who buys insurance?: those that expect large medical
expenditures o Who does not?: those who are healthy
Poor estimate of counterfactual
![Page 25: Poverty Action Lab: Translating Research into Actionpubdocs.worldbank.org › en › 908771463426766162 › Core-2... · Poverty Action Lab: Translating Research into Action . MIT](https://reader036.vdocuments.us/reader036/viewer/2022081407/5f23397d07fa8e71780e498e/html5/thumbnails/25.jpg)
Program placement: example
Compare fertility in villages offered program to fertility in other villages
Government offers a family planning program to villages with high fertility
Program targeted based on fertility, so (1) Treatments have high fertility and (2) counterfactuals have low fertility.
Estimated program impact confounded with targeting criteria
![Page 26: Poverty Action Lab: Translating Research into Actionpubdocs.worldbank.org › en › 908771463426766162 › Core-2... · Poverty Action Lab: Translating Research into Action . MIT](https://reader036.vdocuments.us/reader036/viewer/2022081407/5f23397d07fa8e71780e498e/html5/thumbnails/26.jpg)
What's wrong? Selection bias: People choose to participate for specific reasons
1
2
3
o Job Training: ability and earning o Health Insurance: health status and medical
expenditures
Many times reasons are related to the outcome of interest
Cannot separately identify impact of the program from these other factors/reasons
![Page 27: Poverty Action Lab: Translating Research into Actionpubdocs.worldbank.org › en › 908771463426766162 › Core-2... · Poverty Action Lab: Translating Research into Action . MIT](https://reader036.vdocuments.us/reader036/viewer/2022081407/5f23397d07fa8e71780e498e/html5/thumbnails/27.jpg)
Need to know… All the reasons why someone gets the
program and others not.
All the reasons why individuals are in the treatment versus control group.
If reasons correlated w/ outcome cannot identify/separate program impact from other
explanations of differences in outcomes.
![Page 28: Poverty Action Lab: Translating Research into Actionpubdocs.worldbank.org › en › 908771463426766162 › Core-2... · Poverty Action Lab: Translating Research into Action . MIT](https://reader036.vdocuments.us/reader036/viewer/2022081407/5f23397d07fa8e71780e498e/html5/thumbnails/28.jpg)
Possible Solutions Need to guarantee comparability of treatment and control groups.
ONLY remaining difference is intervention.
In this seminar we will consider: o Experimental design/randomization o Quasi-experiments (Regression Discontinuity,
Double differences) o Instrumental Variables.
![Page 29: Poverty Action Lab: Translating Research into Actionpubdocs.worldbank.org › en › 908771463426766162 › Core-2... · Poverty Action Lab: Translating Research into Action . MIT](https://reader036.vdocuments.us/reader036/viewer/2022081407/5f23397d07fa8e71780e498e/html5/thumbnails/29.jpg)
These solutions all involve… Knowing how the data are generated. Randomization – Give all equal chance of being in control or
treatment groups – Guarantees that all factors/characteristics will
be on average equal btw groups – Only difference is the intervention If not, need transparent & observable criteria for who is offered program.
![Page 30: Poverty Action Lab: Translating Research into Actionpubdocs.worldbank.org › en › 908771463426766162 › Core-2... · Poverty Action Lab: Translating Research into Action . MIT](https://reader036.vdocuments.us/reader036/viewer/2022081407/5f23397d07fa8e71780e498e/html5/thumbnails/30.jpg)
THANK YOU