koica executive education course in evaluating social ... · 4. how to randomize (rohit naimpally)...
TRANSCRIPT
![Page 1: KOICA Executive Education Course in Evaluating Social ... · 4. How to Randomize (Rohit Naimpally) 5. Sampling and Sample Size (Rohit Naimpally) 6. Threats and Analysis (Lina Marliani)](https://reader033.vdocuments.us/reader033/viewer/2022060506/5f1f2361952a4c4f081add04/html5/thumbnails/1.jpg)
KOICA Executive Education Course in Evaluating Social Programs22-26 February 2016Welcome
Lina MarlianiExecutive DirectorJ-PAL Southeast Asia
![Page 2: KOICA Executive Education Course in Evaluating Social ... · 4. How to Randomize (Rohit Naimpally) 5. Sampling and Sample Size (Rohit Naimpally) 6. Threats and Analysis (Lina Marliani)](https://reader033.vdocuments.us/reader033/viewer/2022060506/5f1f2361952a4c4f081add04/html5/thumbnails/2.jpg)
Course Introduction
![Page 3: KOICA Executive Education Course in Evaluating Social ... · 4. How to Randomize (Rohit Naimpally) 5. Sampling and Sample Size (Rohit Naimpally) 6. Threats and Analysis (Lina Marliani)](https://reader033.vdocuments.us/reader033/viewer/2022060506/5f1f2361952a4c4f081add04/html5/thumbnails/3.jpg)
• Thomas Chupein
• John Floretta
• Lina Marliani
• Rohit Naimpally
• Ariella Park
• Hira Siddiqui
Course Leaders
J-PAL | INTRODUCING RANDOMIZED IMPACT EVALUATIONS 3
![Page 4: KOICA Executive Education Course in Evaluating Social ... · 4. How to Randomize (Rohit Naimpally) 5. Sampling and Sample Size (Rohit Naimpally) 6. Threats and Analysis (Lina Marliani)](https://reader033.vdocuments.us/reader033/viewer/2022060506/5f1f2361952a4c4f081add04/html5/thumbnails/4.jpg)
1. Introducing Randomized Impact Evaluations (Thomas Chupein)2. Outcomes, Impact, and Indicators (Rohit Naimpally)3. Why Randomize? (Thomas Chupein)4. How to Randomize (Rohit Naimpally)5. Sampling and Sample Size (Rohit Naimpally)6. Threats and Analysis (Lina Marliani)7. Evaluation from Start to Finish (Lina Marliani)8. Evidence from Community-Driven Development, Health, and Education
Programs (Thomas Chupein and John Floretta)9. Using Evidence from Randomized Evaluations for Decision-Making and Policy
Change (John Floretta)10. Discussion: Where do RCTs fit in a Good M&E Strategy? (All)
Course Overview
J-PAL | INTRODUCING RANDOMIZED IMPACT EVALUATIONS 4
![Page 5: KOICA Executive Education Course in Evaluating Social ... · 4. How to Randomize (Rohit Naimpally) 5. Sampling and Sample Size (Rohit Naimpally) 6. Threats and Analysis (Lina Marliani)](https://reader033.vdocuments.us/reader033/viewer/2022060506/5f1f2361952a4c4f081add04/html5/thumbnails/5.jpg)
• Why and when is a rigorous evaluation of social impact needed?
• The common pitfalls of evaluations, and how randomization can help.
• The key components of a good randomized evaluation design
• Alternative techniques for incorporating randomization into project design.
• How do you determine the appropriate sample size, measure outcomes, and manage data?
• Guarding against threats that may undermine the integrity of the results.
• Techniques for the analysis and interpretation of results.
• How to maximise policy impact and test external validity.
Course Objectives
J-PAL | INTRODUCING RANDOMIZED IMPACT EVALUATIONS 5
![Page 6: KOICA Executive Education Course in Evaluating Social ... · 4. How to Randomize (Rohit Naimpally) 5. Sampling and Sample Size (Rohit Naimpally) 6. Threats and Analysis (Lina Marliani)](https://reader033.vdocuments.us/reader033/viewer/2022060506/5f1f2361952a4c4f081add04/html5/thumbnails/6.jpg)
Course Agenda
J-PAL | INTRODUCING RANDOMIZED IMPACT EVALUATIONS 6
Monday
February 22, 2016
Tuesday
February 23, 2016
Wednesday
February 24, 2016
Thursday
February 25, 2016
Friday
February 26, 20168:00 – 9:00 Registration and Pre-Course
AssessmentBreakfast Breakfast Breakfast Breakfast
9:00 – 10:30 Welcoming remarks
Lecture 1:Introducing Randomized Impact
Evaluations
Thomas Chupein
Lecture 3: Why Randomize?
Thomas Chupein
Lecture 5: Sampling and Sample Size
Rohit Naimpally
Lecture 7: Evaluation from Start to Finish:
Raskin in Indonesia
Lina Marliani
Lecture 9: Using Evidence from Randomized Evaluations for Decision-Making
and Policy Change
John Floretta
10:30 – 10:45 Coffee Break Coffee Break Coffee Break Coffee Break Coffee Break10:45 – 12:00 Group work on case study 1: Theory
of Change: Women as Policymakers in India
Decision on group project
Group Exercise A: Random Sampling
Group work on presentation: Indicators
Group work on case study 4: Threats and Analysis: Deworming in
Kenya
Group work on presentation: Power and sample size
Discussion: Where do RCTs fit in a good M&E Strategy
12:00 – 1:00 Lunch Lunch Lunch Lunch Lunch1:00 – 2:30 Lecture 2:
Outcomes, Impact, and Indicators
Rohit Naimpally
Lecture 4: How to Randomize
Rohit Naimpally
Lecture 6: Threats and Analysis
Lina Marliani
Lecture 8: Evidence from Community-Driven Development, Health, and Education Programs
Thomas Chupein and John Floretta
Feedback survey
Group presentations
Closing remarks
2:30 – 3:00 Coffee Break Coffee Break Coffee Break Coffee Break3:00 – 4:00 Group work on presentation: Theory
of change, research questionGroup Exercise B: Randomization
MechanicsGroup work on presentation:
Randomization Design Group work on presentation:
Threats and Analysis
4:00 – 5:00 Group work on case study 2: Why Randomize: Learn to Read India
Group work on case study 3: How to Randomize: Combating
Corruption in Indonesia
Group Exercise C: Sample Size Estimation
Group work on presentation:
Finalize presentation
Course Agenda
![Page 7: KOICA Executive Education Course in Evaluating Social ... · 4. How to Randomize (Rohit Naimpally) 5. Sampling and Sample Size (Rohit Naimpally) 6. Threats and Analysis (Lina Marliani)](https://reader033.vdocuments.us/reader033/viewer/2022060506/5f1f2361952a4c4f081add04/html5/thumbnails/7.jpg)
• Lectures
• Case Studies
• Exercises
Course Structure
![Page 8: KOICA Executive Education Course in Evaluating Social ... · 4. How to Randomize (Rohit Naimpally) 5. Sampling and Sample Size (Rohit Naimpally) 6. Threats and Analysis (Lina Marliani)](https://reader033.vdocuments.us/reader033/viewer/2022060506/5f1f2361952a4c4f081add04/html5/thumbnails/8.jpg)
• Course Schedule.....................................................................................................................1• Biographies of J-PAL Lecturers...............................................................................................5• Course Material• Case Study 1: Women as Policymakers…….........................................................................7• Case Study 2: Learn to Read Evaluations…........................................................................13• Case Study 3: Combating Corruption..................................................................................21• Case Study 4: Deworming in Kenya…..................................................................................27• Exercise A: Random Sampling and Law of Large Numbers...........................................-.35• Exercise B: Mechanics of Randomization ..........................................................................37• Exercise C: Sample Size Estimation……… ..........................................................................47• Group Presentation Guidelines…………………………………………………………………..63• Impact Evaluation Glossary..................................................................................................67
Course Binder
![Page 9: KOICA Executive Education Course in Evaluating Social ... · 4. How to Randomize (Rohit Naimpally) 5. Sampling and Sample Size (Rohit Naimpally) 6. Threats and Analysis (Lina Marliani)](https://reader033.vdocuments.us/reader033/viewer/2022060506/5f1f2361952a4c4f081add04/html5/thumbnails/9.jpg)
Expectation Survey
J-PAL | INTRODUCING RANDOMIZED IMPACT EVALUATIONS 9
![Page 10: KOICA Executive Education Course in Evaluating Social ... · 4. How to Randomize (Rohit Naimpally) 5. Sampling and Sample Size (Rohit Naimpally) 6. Threats and Analysis (Lina Marliani)](https://reader033.vdocuments.us/reader033/viewer/2022060506/5f1f2361952a4c4f081add04/html5/thumbnails/10.jpg)
Pre-course Assessment
J-PAL | INTRODUCING RANDOMIZED IMPACT EVALUATIONS 10
![Page 11: KOICA Executive Education Course in Evaluating Social ... · 4. How to Randomize (Rohit Naimpally) 5. Sampling and Sample Size (Rohit Naimpally) 6. Threats and Analysis (Lina Marliani)](https://reader033.vdocuments.us/reader033/viewer/2022060506/5f1f2361952a4c4f081add04/html5/thumbnails/11.jpg)
• Everyone gets one
• Each clicker has number on back that matches the number assigned to you on the participant list
• Please hold on to them and turn them in at the end of each day
Course “Clickers”
![Page 12: KOICA Executive Education Course in Evaluating Social ... · 4. How to Randomize (Rohit Naimpally) 5. Sampling and Sample Size (Rohit Naimpally) 6. Threats and Analysis (Lina Marliani)](https://reader033.vdocuments.us/reader033/viewer/2022060506/5f1f2361952a4c4f081add04/html5/thumbnails/12.jpg)
Course “Clickers”: Have you used these before?A. YesB. NoC. Something similarD. Something different
A. B. C. D.
0% 0%0%0%
![Page 13: KOICA Executive Education Course in Evaluating Social ... · 4. How to Randomize (Rohit Naimpally) 5. Sampling and Sample Size (Rohit Naimpally) 6. Threats and Analysis (Lina Marliani)](https://reader033.vdocuments.us/reader033/viewer/2022060506/5f1f2361952a4c4f081add04/html5/thumbnails/13.jpg)
Introduction to J-PAL
![Page 14: KOICA Executive Education Course in Evaluating Social ... · 4. How to Randomize (Rohit Naimpally) 5. Sampling and Sample Size (Rohit Naimpally) 6. Threats and Analysis (Lina Marliani)](https://reader033.vdocuments.us/reader033/viewer/2022060506/5f1f2361952a4c4f081add04/html5/thumbnails/14.jpg)
J-PAL’s mission is to ensure that policy is informed by evidence and research is translated into action
J-PAL | INTRODUCING RANDOMIZED IMPACT EVALUATIONS 14
![Page 15: KOICA Executive Education Course in Evaluating Social ... · 4. How to Randomize (Rohit Naimpally) 5. Sampling and Sample Size (Rohit Naimpally) 6. Threats and Analysis (Lina Marliani)](https://reader033.vdocuments.us/reader033/viewer/2022060506/5f1f2361952a4c4f081add04/html5/thumbnails/15.jpg)
J-PAL’s network of 131 professors use randomized evaluations to inform policy
J-PAL | INTRODUCING RANDOMIZED IMPACT EVALUATIONS 15
![Page 16: KOICA Executive Education Course in Evaluating Social ... · 4. How to Randomize (Rohit Naimpally) 5. Sampling and Sample Size (Rohit Naimpally) 6. Threats and Analysis (Lina Marliani)](https://reader033.vdocuments.us/reader033/viewer/2022060506/5f1f2361952a4c4f081add04/html5/thumbnails/16.jpg)
J-PAL has 7 offices and 700 ongoing and completed projects in 64 countries
J-PAL | INTRODUCING RANDOMIZED IMPACT EVALUATIONS 16
![Page 17: KOICA Executive Education Course in Evaluating Social ... · 4. How to Randomize (Rohit Naimpally) 5. Sampling and Sample Size (Rohit Naimpally) 6. Threats and Analysis (Lina Marliani)](https://reader033.vdocuments.us/reader033/viewer/2022060506/5f1f2361952a4c4f081add04/html5/thumbnails/17.jpg)
J-PAL’s activities are organized into eight sectors
J-PAL | INTRODUCING RANDOMIZED IMPACT EVALUATIONS 17
![Page 18: KOICA Executive Education Course in Evaluating Social ... · 4. How to Randomize (Rohit Naimpally) 5. Sampling and Sample Size (Rohit Naimpally) 6. Threats and Analysis (Lina Marliani)](https://reader033.vdocuments.us/reader033/viewer/2022060506/5f1f2361952a4c4f081add04/html5/thumbnails/18.jpg)
Introducing Randomized Impact Evaluations
Thomas ChupeinPolicy ManagerJ-PAL Global at MIT
![Page 19: KOICA Executive Education Course in Evaluating Social ... · 4. How to Randomize (Rohit Naimpally) 5. Sampling and Sample Size (Rohit Naimpally) 6. Threats and Analysis (Lina Marliani)](https://reader033.vdocuments.us/reader033/viewer/2022060506/5f1f2361952a4c4f081add04/html5/thumbnails/19.jpg)
Evaluation
Program Evaluation
Impact Evaluation
What is Evaluation?
J-PAL | INTRODUCING RANDOMIZED IMPACT EVALUATIONS 19
![Page 20: KOICA Executive Education Course in Evaluating Social ... · 4. How to Randomize (Rohit Naimpally) 5. Sampling and Sample Size (Rohit Naimpally) 6. Threats and Analysis (Lina Marliani)](https://reader033.vdocuments.us/reader033/viewer/2022060506/5f1f2361952a4c4f081add04/html5/thumbnails/20.jpg)
Evaluation
Program Evaluation
Impact Evaluation
Program Evaluation
J-PAL | INTRODUCING RANDOMIZED IMPACT EVALUATIONS 20
![Page 21: KOICA Executive Education Course in Evaluating Social ... · 4. How to Randomize (Rohit Naimpally) 5. Sampling and Sample Size (Rohit Naimpally) 6. Threats and Analysis (Lina Marliani)](https://reader033.vdocuments.us/reader033/viewer/2022060506/5f1f2361952a4c4f081add04/html5/thumbnails/21.jpg)
Evaluation
Program Evaluation
Impact Evaluation
Monitoring and Evaluation
J-PAL | INTRODUCING RANDOMIZED IMPACT EVALUATIONS 21
Monitoring
![Page 22: KOICA Executive Education Course in Evaluating Social ... · 4. How to Randomize (Rohit Naimpally) 5. Sampling and Sample Size (Rohit Naimpally) 6. Threats and Analysis (Lina Marliani)](https://reader033.vdocuments.us/reader033/viewer/2022060506/5f1f2361952a4c4f081add04/html5/thumbnails/22.jpg)
What’s the difference between:Monitoring and EvaluationA. Nothing. They are different
words to describe the same activity
B. Monitoring is conducted internally, Evaluation is conducted externally
C. Monitoring is for management, Evaluation is for accountability
D. Don’t knowE. Other
A. B. C. D. E.
0%
28%
0%0%
72%
J-PAL | INTRODUCING RANDOMIZED IMPACT EVALUATIONS 22
![Page 23: KOICA Executive Education Course in Evaluating Social ... · 4. How to Randomize (Rohit Naimpally) 5. Sampling and Sample Size (Rohit Naimpally) 6. Threats and Analysis (Lina Marliani)](https://reader033.vdocuments.us/reader033/viewer/2022060506/5f1f2361952a4c4f081add04/html5/thumbnails/23.jpg)
Program Evaluation
J-PAL | INTRODUCING RANDOMIZED IMPACT EVALUATIONS 23
Evaluation
Program Evaluation
Impact Evaluation
Monitoring
![Page 24: KOICA Executive Education Course in Evaluating Social ... · 4. How to Randomize (Rohit Naimpally) 5. Sampling and Sample Size (Rohit Naimpally) 6. Threats and Analysis (Lina Marliani)](https://reader033.vdocuments.us/reader033/viewer/2022060506/5f1f2361952a4c4f081add04/html5/thumbnails/24.jpg)
5 Components of Program Evaluation
• Needs Assessment
• Program Theory Assessment
• Process Evaluation
• Impact Evaluation
• Cost Effectiveness
• What is the problem?
• How, in theory, does the program fix the problem?
• Does the program work as planned?
• Were its goals achieved?The magnitude?
• Given magnitude and cost, how does it compare to alternatives?
J-PAL | INTRODUCING RANDOMIZED IMPACT EVALUATIONS 24
![Page 25: KOICA Executive Education Course in Evaluating Social ... · 4. How to Randomize (Rohit Naimpally) 5. Sampling and Sample Size (Rohit Naimpally) 6. Threats and Analysis (Lina Marliani)](https://reader033.vdocuments.us/reader033/viewer/2022060506/5f1f2361952a4c4f081add04/html5/thumbnails/25.jpg)
Rossi, Lipsey and Freeman
J-PAL | INTRODUCING RANDOMIZED IMPACT EVALUATIONS 25
![Page 26: KOICA Executive Education Course in Evaluating Social ... · 4. How to Randomize (Rohit Naimpally) 5. Sampling and Sample Size (Rohit Naimpally) 6. Threats and Analysis (Lina Marliani)](https://reader033.vdocuments.us/reader033/viewer/2022060506/5f1f2361952a4c4f081add04/html5/thumbnails/26.jpg)
Evaluation should usually be conducted:
A. Externally and independent from the implementers of the program being evaluated
B. Externally and closely integrated with program implementers
C.InternallyD. Don’t know
J-PAL | INTRODUCING RANDOMIZED IMPACT EVALUATIONS 26A. B. C. D.
32%
0%5%
63%
![Page 27: KOICA Executive Education Course in Evaluating Social ... · 4. How to Randomize (Rohit Naimpally) 5. Sampling and Sample Size (Rohit Naimpally) 6. Threats and Analysis (Lina Marliani)](https://reader033.vdocuments.us/reader033/viewer/2022060506/5f1f2361952a4c4f081add04/html5/thumbnails/27.jpg)
How can Impact Evaluation Help Us?
• Surprisingly little rigorous evidence on what works
• Can do more with given budget with better evidence
• If people knew donor funds are used for effective programs, this could help increase resources for anti-poverty programs
• Instead of asking “do aid/development programs work?” should be asking:
– Which work best, why and when?
– How can we scale up what works?
J-PAL | INTRODUCING RANDOMIZED IMPACT EVALUATIONS 27
![Page 28: KOICA Executive Education Course in Evaluating Social ... · 4. How to Randomize (Rohit Naimpally) 5. Sampling and Sample Size (Rohit Naimpally) 6. Threats and Analysis (Lina Marliani)](https://reader033.vdocuments.us/reader033/viewer/2022060506/5f1f2361952a4c4f081add04/html5/thumbnails/28.jpg)
Programs and their Evaluations: Where do we start?
Intervention• Start with a problem
• Verify that the problem actually exists
• Generate a theory of why the problem exists
• Design the program
• Think about whether the solution is cost effective
Program Evaluation• Start with a question
• Verify the question hasn’t been answered
• State a hypothesis
• Design the evaluation
• Determine whether the value of the answer is worth the cost of the evaluation
J-PAL | INTRODUCING RANDOMIZED IMPACT EVALUATIONS 28
![Page 29: KOICA Executive Education Course in Evaluating Social ... · 4. How to Randomize (Rohit Naimpally) 5. Sampling and Sample Size (Rohit Naimpally) 6. Threats and Analysis (Lina Marliani)](https://reader033.vdocuments.us/reader033/viewer/2022060506/5f1f2361952a4c4f081add04/html5/thumbnails/29.jpg)
What do you think is the most cost-effective way to increase immunization rates?
A. Community mobilization campaign
B. Improve healthcare worker attendance
C. Develop new vaccines, such as pneumococcal
D. Hold special ‘immunization camps’
E. Incentivize parents to immunize their children
J-PAL | INTRODUCING RANDOMIZED IMPACT EVALUATIONS 29Co
mmun
ity m
obiliz
ation
...
Impr
ove he
althc
are wo
rk..
Develo
p ne
w vaccine
s, s...
Hold sp
ecial
‘immun
izati..
Ince
ntivi
ze paren
ts to i..
.
29%24%
41%
0%6%
![Page 30: KOICA Executive Education Course in Evaluating Social ... · 4. How to Randomize (Rohit Naimpally) 5. Sampling and Sample Size (Rohit Naimpally) 6. Threats and Analysis (Lina Marliani)](https://reader033.vdocuments.us/reader033/viewer/2022060506/5f1f2361952a4c4f081add04/html5/thumbnails/30.jpg)
Child Immunization
An Example
![Page 31: KOICA Executive Education Course in Evaluating Social ... · 4. How to Randomize (Rohit Naimpally) 5. Sampling and Sample Size (Rohit Naimpally) 6. Threats and Analysis (Lina Marliani)](https://reader033.vdocuments.us/reader033/viewer/2022060506/5f1f2361952a4c4f081add04/html5/thumbnails/31.jpg)
1. Needs Assessment
Identifying the problem
![Page 32: KOICA Executive Education Course in Evaluating Social ... · 4. How to Randomize (Rohit Naimpally) 5. Sampling and Sample Size (Rohit Naimpally) 6. Threats and Analysis (Lina Marliani)](https://reader033.vdocuments.us/reader033/viewer/2022060506/5f1f2361952a4c4f081add04/html5/thumbnails/32.jpg)
• Every year, between 2 and 3 million people die from vaccine-preventable diseases
• In India, only 54% of 1-2 year olds receive the basic package of immunizations
• In rural Rajasthan, this rate falls to 22%
The Need
J-PAL | INTRODUCING RANDOMIZED IMPACT EVALUATIONS 32
![Page 33: KOICA Executive Education Course in Evaluating Social ... · 4. How to Randomize (Rohit Naimpally) 5. Sampling and Sample Size (Rohit Naimpally) 6. Threats and Analysis (Lina Marliani)](https://reader033.vdocuments.us/reader033/viewer/2022060506/5f1f2361952a4c4f081add04/html5/thumbnails/33.jpg)
• In India, immunizations are offered for free, but the immunization rate remains low
• Average household is within 2 kilometers of the nearest clinic
• High absenteeism at government health facilities – 45% of Auxiliary Nurse Midwives are absent on any given workday
The Problem
J-PAL | INTRODUCING RANDOMIZED IMPACT EVALUATIONS 33
![Page 34: KOICA Executive Education Course in Evaluating Social ... · 4. How to Randomize (Rohit Naimpally) 5. Sampling and Sample Size (Rohit Naimpally) 6. Threats and Analysis (Lina Marliani)](https://reader033.vdocuments.us/reader033/viewer/2022060506/5f1f2361952a4c4f081add04/html5/thumbnails/34.jpg)
• Is there cultural resistance or distrust in public health institutions?
• Parents may not be able to afford to take a day off of work
• People may not value immunizations: short-term cost for long-term (and invisible) benefits
But is Supply the Entirety of the Problem?
J-PAL | INTRODUCING RANDOMIZED IMPACT EVALUATIONS 34
![Page 35: KOICA Executive Education Course in Evaluating Social ... · 4. How to Randomize (Rohit Naimpally) 5. Sampling and Sample Size (Rohit Naimpally) 6. Threats and Analysis (Lina Marliani)](https://reader033.vdocuments.us/reader033/viewer/2022060506/5f1f2361952a4c4f081add04/html5/thumbnails/35.jpg)
One Potential Solution: Solve the Supply Problem
J-PAL | INTRODUCING RANDOMIZED IMPACT EVALUATIONS 35
• Reliable NGO-provided monthly immunization camps at the village level tripled rates of full immunization.
![Page 36: KOICA Executive Education Course in Evaluating Social ... · 4. How to Randomize (Rohit Naimpally) 5. Sampling and Sample Size (Rohit Naimpally) 6. Threats and Analysis (Lina Marliani)](https://reader033.vdocuments.us/reader033/viewer/2022060506/5f1f2361952a4c4f081add04/html5/thumbnails/36.jpg)
Another Potential Solution: Solve the Demand Problem
• Alongside reliable NGO-provided monthly immunization camps, offer mothers a small incentive to bring their child
J-PAL | INTRODUCING RANDOMIZED IMPACT EVALUATIONS 36
• When coupled with in-kind incentives (raw lentils and metal plates for completing immunization), full immunization rates increased six-fold
![Page 37: KOICA Executive Education Course in Evaluating Social ... · 4. How to Randomize (Rohit Naimpally) 5. Sampling and Sample Size (Rohit Naimpally) 6. Threats and Analysis (Lina Marliani)](https://reader033.vdocuments.us/reader033/viewer/2022060506/5f1f2361952a4c4f081add04/html5/thumbnails/37.jpg)
Behavioral theory on use of incentives
J-PAL | INTRODUCING RANDOMIZED IMPACT EVALUATIONS 37
Time
Benefit
Tomorrow
Today
Benefit
Time
![Page 38: KOICA Executive Education Course in Evaluating Social ... · 4. How to Randomize (Rohit Naimpally) 5. Sampling and Sample Size (Rohit Naimpally) 6. Threats and Analysis (Lina Marliani)](https://reader033.vdocuments.us/reader033/viewer/2022060506/5f1f2361952a4c4f081add04/html5/thumbnails/38.jpg)
2. Program Theory Assessment
Blueprint for Change
![Page 39: KOICA Executive Education Course in Evaluating Social ... · 4. How to Randomize (Rohit Naimpally) 5. Sampling and Sample Size (Rohit Naimpally) 6. Threats and Analysis (Lina Marliani)](https://reader033.vdocuments.us/reader033/viewer/2022060506/5f1f2361952a4c4f081add04/html5/thumbnails/39.jpg)
Program Theory Assessment
• Theory of Change
• Logical Framework (LogFrame)
• Results Framework
• Outcome Mapping
• Causal chain• Causal model • Cause map• Impact pathways• Intervention theory • Intervention framework• Intervention logic• Investment logic• Logic model• Outcomes chain• Outcomes hierarchy• Outcome line• Program logic• Program theory• Programme theory• Results chain• Theory-based evaluation• Theory-driven evaluation • Theory-of-action
J-PAL | INTRODUCING RANDOMIZED IMPACT EVALUATIONS 39Source: Patricia Rogers
![Page 40: KOICA Executive Education Course in Evaluating Social ... · 4. How to Randomize (Rohit Naimpally) 5. Sampling and Sample Size (Rohit Naimpally) 6. Threats and Analysis (Lina Marliani)](https://reader033.vdocuments.us/reader033/viewer/2022060506/5f1f2361952a4c4f081add04/html5/thumbnails/40.jpg)
“A theory of change is a road map
of where we are going (results) and
how we are getting there (process)”
What is a Theory of Change?
J-PAL | INTRODUCING RANDOMIZED IMPACT EVALUATIONS 40
![Page 41: KOICA Executive Education Course in Evaluating Social ... · 4. How to Randomize (Rohit Naimpally) 5. Sampling and Sample Size (Rohit Naimpally) 6. Threats and Analysis (Lina Marliani)](https://reader033.vdocuments.us/reader033/viewer/2022060506/5f1f2361952a4c4f081add04/html5/thumbnails/41.jpg)
Q: How do I expect results to be achieved?
A: If [inputs] and [activities] produce [outputs] this should lead to [outcomes] which will ultimately contribute to [goal]
Causal Hypothesis
J-PAL | INTRODUCING RANDOMIZED IMPACT EVALUATIONS 41
![Page 42: KOICA Executive Education Course in Evaluating Social ... · 4. How to Randomize (Rohit Naimpally) 5. Sampling and Sample Size (Rohit Naimpally) 6. Threats and Analysis (Lina Marliani)](https://reader033.vdocuments.us/reader033/viewer/2022060506/5f1f2361952a4c4f081add04/html5/thumbnails/42.jpg)
Immunization Program Theory of Change
Increased immunization rates
Parents bring children to regular
camp
Parents believe camps are regular
Establish regular camps
Supply-side limits on immunization
Parents value incentive
Incentives for full course
Incentives regularly paid
Demand-side constrains on take-up of immunization
Camps provide immunizations
J-PAL | INTRODUCING RANDOMIZED IMPACT EVALUATIONS 42
![Page 43: KOICA Executive Education Course in Evaluating Social ... · 4. How to Randomize (Rohit Naimpally) 5. Sampling and Sample Size (Rohit Naimpally) 6. Threats and Analysis (Lina Marliani)](https://reader033.vdocuments.us/reader033/viewer/2022060506/5f1f2361952a4c4f081add04/html5/thumbnails/43.jpg)
Immunization Program Theory of Change
Parents want to vaccinate
Can access clinic
Provider presence sufficient
Do basic conditions hold locally?
Incentives given to parents
Local logistics critical
Incentives delivered to
clinicParents pro-crastinate
Evidence on behavioral bias
Small incentives offset bias
Impact
Immunization rises
Health improves
![Page 44: KOICA Executive Education Course in Evaluating Social ... · 4. How to Randomize (Rohit Naimpally) 5. Sampling and Sample Size (Rohit Naimpally) 6. Threats and Analysis (Lina Marliani)](https://reader033.vdocuments.us/reader033/viewer/2022060506/5f1f2361952a4c4f081add04/html5/thumbnails/44.jpg)
Necessary and positive external conditions that should be in place for the chain of cause and effect (in an intervention) to go forward
Assumptions
J-PAL | INTRODUCING RANDOMIZED IMPACT EVALUATIONS 44
![Page 45: KOICA Executive Education Course in Evaluating Social ... · 4. How to Randomize (Rohit Naimpally) 5. Sampling and Sample Size (Rohit Naimpally) 6. Threats and Analysis (Lina Marliani)](https://reader033.vdocuments.us/reader033/viewer/2022060506/5f1f2361952a4c4f081add04/html5/thumbnails/45.jpg)
Immunization Program Theory of Change
Increased immunization rates
Parents bring children to regular
camp
Parents believe camps are regular
Establish regular camps
Supply-side limits on immunization
Parents value incentive
Incentives for full course
Incentives regularly paid
Demand-side constrains on take-up of immunization
Camps provide immunizations
J-PAL | INTRODUCING RANDOMIZED IMPACT EVALUATIONS 45
![Page 46: KOICA Executive Education Course in Evaluating Social ... · 4. How to Randomize (Rohit Naimpally) 5. Sampling and Sample Size (Rohit Naimpally) 6. Threats and Analysis (Lina Marliani)](https://reader033.vdocuments.us/reader033/viewer/2022060506/5f1f2361952a4c4f081add04/html5/thumbnails/46.jpg)
Results Levels
Inputs Activities Outputs Outcomes Goal
Resources Actions Products and
services
KASBs Dev. status
J-PAL | INTRODUCING RANDOMIZED IMPACT EVALUATIONS 46
![Page 47: KOICA Executive Education Course in Evaluating Social ... · 4. How to Randomize (Rohit Naimpally) 5. Sampling and Sample Size (Rohit Naimpally) 6. Threats and Analysis (Lina Marliani)](https://reader033.vdocuments.us/reader033/viewer/2022060506/5f1f2361952a4c4f081add04/html5/thumbnails/47.jpg)
Log Frame
J-PAL | INTRODUCING RANDOMIZED IMPACT EVALUATIONS 47
Source: Roduner, Schlappi (2008) Logical Framework Approach and Outcome Mapping, A constructive Attempt of Synthesis
Needs assessment
Process evaluation
Impactevaluation
Objectives Hierarchy
Indicators Sources of Verification
Assumptions / Threats
Impact(Goal/ Overall
objective)
Increased immunization
Immunization rates
Household survey
Adequate vaccine supply, parents do not have second thoughts
Outcome(Project
Objective)
Parents attend the immunization camps repeatedly
Follow-up attendance
Household survey; Immunization card
Parents have the time to come
Outputs Immunizationcamps are reliably open; Incentives are delivered
Number of kgbags delivered; Camp schedules
Random audits;Camp administrative data
Nurses/assistants will show up to camp and give out incentives properly
Inputs(Activities)
Camps + incentives are established
Camps arebuilt, functional
Random auditsof camps
Sufficient materials, funding, manpower
![Page 48: KOICA Executive Education Course in Evaluating Social ... · 4. How to Randomize (Rohit Naimpally) 5. Sampling and Sample Size (Rohit Naimpally) 6. Threats and Analysis (Lina Marliani)](https://reader033.vdocuments.us/reader033/viewer/2022060506/5f1f2361952a4c4f081add04/html5/thumbnails/48.jpg)
“Theory of change thinking is a habit not a product.”
Theory of Change: Product or Process?
J-PAL | INTRODUCING RANDOMIZED IMPACT EVALUATIONS 48
![Page 49: KOICA Executive Education Course in Evaluating Social ... · 4. How to Randomize (Rohit Naimpally) 5. Sampling and Sample Size (Rohit Naimpally) 6. Threats and Analysis (Lina Marliani)](https://reader033.vdocuments.us/reader033/viewer/2022060506/5f1f2361952a4c4f081add04/html5/thumbnails/49.jpg)
3. Process Evaluation
Making the program work
![Page 50: KOICA Executive Education Course in Evaluating Social ... · 4. How to Randomize (Rohit Naimpally) 5. Sampling and Sample Size (Rohit Naimpally) 6. Threats and Analysis (Lina Marliani)](https://reader033.vdocuments.us/reader033/viewer/2022060506/5f1f2361952a4c4f081add04/html5/thumbnails/50.jpg)
Solving the Black Box Problem
Intervention Intervention design/Inputs
Final outcome
Low immunization rates Needs Assessment
Black Box
No increase in full immunization
J-PAL | INTRODUCING RANDOMIZED IMPACT EVALUATIONS 50
![Page 51: KOICA Executive Education Course in Evaluating Social ... · 4. How to Randomize (Rohit Naimpally) 5. Sampling and Sample Size (Rohit Naimpally) 6. Threats and Analysis (Lina Marliani)](https://reader033.vdocuments.us/reader033/viewer/2022060506/5f1f2361952a4c4f081add04/html5/thumbnails/51.jpg)
Inputs Activities Outputs Outcomes Goal
Identifying Implementation Failure vs. Theory Failure
Successful intervention
Inputs Activities Outputs Outcomes Goal
J-PAL | INTRODUCING RANDOMIZED IMPACT EVALUATIONS 51
Inputs Activities Outputs Outcomes Goal
Implementation Failure
Theoretical Failure
![Page 52: KOICA Executive Education Course in Evaluating Social ... · 4. How to Randomize (Rohit Naimpally) 5. Sampling and Sample Size (Rohit Naimpally) 6. Threats and Analysis (Lina Marliani)](https://reader033.vdocuments.us/reader033/viewer/2022060506/5f1f2361952a4c4f081add04/html5/thumbnails/52.jpg)
Process Evaluation
• On the supply side– Logistics
– Management
• On the demand side– Assumption of knowledge,
preferences
– Assumptions of response
J-PAL | INTRODUCING RANDOMIZED IMPACT EVALUATIONS 52
![Page 53: KOICA Executive Education Course in Evaluating Social ... · 4. How to Randomize (Rohit Naimpally) 5. Sampling and Sample Size (Rohit Naimpally) 6. Threats and Analysis (Lina Marliani)](https://reader033.vdocuments.us/reader033/viewer/2022060506/5f1f2361952a4c4f081add04/html5/thumbnails/53.jpg)
• Establish camp
– Hiring nurses and administrators
– Installing temporary camp site
– Procuring vaccines and other medical supplies
• Organize incentive scheme
– Identify viable incentive
– Purchase lentils and dinner plate sets
Process Evaluation: Supply-Side
J-PAL | INTRODUCING RANDOMIZED IMPACT EVALUATIONS 53
![Page 54: KOICA Executive Education Course in Evaluating Social ... · 4. How to Randomize (Rohit Naimpally) 5. Sampling and Sample Size (Rohit Naimpally) 6. Threats and Analysis (Lina Marliani)](https://reader033.vdocuments.us/reader033/viewer/2022060506/5f1f2361952a4c4f081add04/html5/thumbnails/54.jpg)
• Do parents visit the camps?
• Do they come back?
Process Evaluation: Demand-Side
J-PAL | INTRODUCING RANDOMIZED IMPACT EVALUATIONS 54
![Page 55: KOICA Executive Education Course in Evaluating Social ... · 4. How to Randomize (Rohit Naimpally) 5. Sampling and Sample Size (Rohit Naimpally) 6. Threats and Analysis (Lina Marliani)](https://reader033.vdocuments.us/reader033/viewer/2022060506/5f1f2361952a4c4f081add04/html5/thumbnails/55.jpg)
Evaluation
Program Evaluation
Impact Evaluation
Monitoring and Evaluation
J-PAL | INTRODUCING RANDOMIZED IMPACT EVALUATIONS 55
Monitoring
![Page 56: KOICA Executive Education Course in Evaluating Social ... · 4. How to Randomize (Rohit Naimpally) 5. Sampling and Sample Size (Rohit Naimpally) 6. Threats and Analysis (Lina Marliani)](https://reader033.vdocuments.us/reader033/viewer/2022060506/5f1f2361952a4c4f081add04/html5/thumbnails/56.jpg)
Log Frame
J-PAL | INTRODUCING RANDOMIZED IMPACT EVALUATIONS 56
Source: Roduner, Schlappi (2008) Logical Framework Approach and Outcome Mapping, A constructive Attempt of Synthesis
Needs assessment
Process evaluation
Impactevaluation
Objectives Hierarchy
Indicators Sources of Verification
Assumptions / Threats
Impact(Goal/ Overall
objective)
Increased immunization
Immunization rates
Household survey
Adequate vaccine supply, parents do not have second thoughts
Outcome(Project
Objective)
Parents attend the immunization camps repeatedly
Follow-up attendance
Household survey; Immunization card
Parents have the time to come
Outputs Immunizationcamps are reliably open; Incentives are delivered
Number of kgbags delivered; Camp schedules
Random audits;Camp administrative data
Nurses/assistants will show up to camp and give out incentives properly
Inputs(Activities)
Camps + incentives are established
Camps arebuilt, functional
Random auditsof camps
Sufficient materials, funding, manpower
![Page 57: KOICA Executive Education Course in Evaluating Social ... · 4. How to Randomize (Rohit Naimpally) 5. Sampling and Sample Size (Rohit Naimpally) 6. Threats and Analysis (Lina Marliani)](https://reader033.vdocuments.us/reader033/viewer/2022060506/5f1f2361952a4c4f081add04/html5/thumbnails/57.jpg)
• Was the program implemented as planned?
• Did people respond as expected?
With process evaluation, we learn…
J-PAL | INTRODUCING RANDOMIZED IMPACT EVALUATIONS 57
• If so, then what happened to immunization rates?
![Page 58: KOICA Executive Education Course in Evaluating Social ... · 4. How to Randomize (Rohit Naimpally) 5. Sampling and Sample Size (Rohit Naimpally) 6. Threats and Analysis (Lina Marliani)](https://reader033.vdocuments.us/reader033/viewer/2022060506/5f1f2361952a4c4f081add04/html5/thumbnails/58.jpg)
4. Impact Evaluation
Measuring how well the program worked
![Page 59: KOICA Executive Education Course in Evaluating Social ... · 4. How to Randomize (Rohit Naimpally) 5. Sampling and Sample Size (Rohit Naimpally) 6. Threats and Analysis (Lina Marliani)](https://reader033.vdocuments.us/reader033/viewer/2022060506/5f1f2361952a4c4f081add04/html5/thumbnails/59.jpg)
• Primary outcome (impact): did camps (or camps + incentives) raise children’s full immunization rates?
• Also distributional questions: for example, what was the impact of incentives for households by level of income?
Did we Achieve our Goals?
J-PAL | INTRODUCING RANDOMIZED IMPACT EVALUATIONS 59
![Page 60: KOICA Executive Education Course in Evaluating Social ... · 4. How to Randomize (Rohit Naimpally) 5. Sampling and Sample Size (Rohit Naimpally) 6. Threats and Analysis (Lina Marliani)](https://reader033.vdocuments.us/reader033/viewer/2022060506/5f1f2361952a4c4f081add04/html5/thumbnails/60.jpg)
Intervention
Time
Prim
ary
outc
ome
Counterfactual
Impact
What is Impact?
J-PAL | INTRODUCING RANDOMIZED IMPACT EVALUATIONS 60
![Page 61: KOICA Executive Education Course in Evaluating Social ... · 4. How to Randomize (Rohit Naimpally) 5. Sampling and Sample Size (Rohit Naimpally) 6. Threats and Analysis (Lina Marliani)](https://reader033.vdocuments.us/reader033/viewer/2022060506/5f1f2361952a4c4f081add04/html5/thumbnails/61.jpg)
• We need to know the counterfactual (i.e. What would have happened in the absence of the program?)
• Take the difference between – what happened (with the program)and
– what would have happened (without the program)
• This yields the impact of the program
How to measure impact?
J-PAL | INTRODUCING RANDOMIZED IMPACT EVALUATIONS 61
![Page 62: KOICA Executive Education Course in Evaluating Social ... · 4. How to Randomize (Rohit Naimpally) 5. Sampling and Sample Size (Rohit Naimpally) 6. Threats and Analysis (Lina Marliani)](https://reader033.vdocuments.us/reader033/viewer/2022060506/5f1f2361952a4c4f081add04/html5/thumbnails/62.jpg)
• Counterfactual is often constructed by selecting a group not affected by the program
• Randomized method:– Use random assignment of the program to create a control (comparison) group
which mimics the counterfactual.
• Non-randomized method:– Argue that a certain excluded group mimics the counterfactual.
Constructing the Counterfactual
J-PAL | INTRODUCING RANDOMIZED IMPACT EVALUATIONS 62
![Page 63: KOICA Executive Education Course in Evaluating Social ... · 4. How to Randomize (Rohit Naimpally) 5. Sampling and Sample Size (Rohit Naimpally) 6. Threats and Analysis (Lina Marliani)](https://reader033.vdocuments.us/reader033/viewer/2022060506/5f1f2361952a4c4f081add04/html5/thumbnails/63.jpg)
• When we answer a process question, we need to describe what happened.
• When we answer an impact question, we need to compare what happened to what would have happened without the program
How impact differs from process?
J-PAL | INTRODUCING RANDOMIZED IMPACT EVALUATIONS 63
![Page 64: KOICA Executive Education Course in Evaluating Social ... · 4. How to Randomize (Rohit Naimpally) 5. Sampling and Sample Size (Rohit Naimpally) 6. Threats and Analysis (Lina Marliani)](https://reader033.vdocuments.us/reader033/viewer/2022060506/5f1f2361952a4c4f081add04/html5/thumbnails/64.jpg)
Randomized Evaluation
![Page 65: KOICA Executive Education Course in Evaluating Social ... · 4. How to Randomize (Rohit Naimpally) 5. Sampling and Sample Size (Rohit Naimpally) 6. Threats and Analysis (Lina Marliani)](https://reader033.vdocuments.us/reader033/viewer/2022060506/5f1f2361952a4c4f081add04/html5/thumbnails/65.jpg)
Non-random treatment and control groups.
Purposive Assignment
J-PAL | INTRODUCING RANDOMIZED IMPACT EVALUATIONS 65
HQ
![Page 66: KOICA Executive Education Course in Evaluating Social ... · 4. How to Randomize (Rohit Naimpally) 5. Sampling and Sample Size (Rohit Naimpally) 6. Threats and Analysis (Lina Marliani)](https://reader033.vdocuments.us/reader033/viewer/2022060506/5f1f2361952a4c4f081add04/html5/thumbnails/66.jpg)
Randomly samplefrom area of interest
Random Sampling and Random Assignment
J-PAL | INTRODUCING RANDOMIZED IMPACT EVALUATIONS 66
![Page 67: KOICA Executive Education Course in Evaluating Social ... · 4. How to Randomize (Rohit Naimpally) 5. Sampling and Sample Size (Rohit Naimpally) 6. Threats and Analysis (Lina Marliani)](https://reader033.vdocuments.us/reader033/viewer/2022060506/5f1f2361952a4c4f081add04/html5/thumbnails/67.jpg)
Random Sampling and Random Assignment
J-PAL | INTRODUCING RANDOMIZED IMPACT EVALUATIONS 67
Randomly samplefrom area of interest
Randomly assignto treatmentand control
Randomly samplefrom both treatment and control
![Page 68: KOICA Executive Education Course in Evaluating Social ... · 4. How to Randomize (Rohit Naimpally) 5. Sampling and Sample Size (Rohit Naimpally) 6. Threats and Analysis (Lina Marliani)](https://reader033.vdocuments.us/reader033/viewer/2022060506/5f1f2361952a4c4f081add04/html5/thumbnails/68.jpg)
Immunization Example
Target Population
(134)
Not in evaluation
(0)
Evaluation Sample
(134)
TotalPopulation
(700+ villages)
Random Assignment
Camps + Incentives
(30)
Control(74)
Camps(30)
J-PAL | INTRODUCING RANDOMIZED IMPACT EVALUATIONS 68
![Page 69: KOICA Executive Education Course in Evaluating Social ... · 4. How to Randomize (Rohit Naimpally) 5. Sampling and Sample Size (Rohit Naimpally) 6. Threats and Analysis (Lina Marliani)](https://reader033.vdocuments.us/reader033/viewer/2022060506/5f1f2361952a4c4f081add04/html5/thumbnails/69.jpg)
• 6% full immunization in control villages
• 18% full immunization in camps villages
• 39% full immunization in camps + incentives villages
Impact
J-PAL | INTRODUCING RANDOMIZED IMPACT EVALUATIONS 69
![Page 70: KOICA Executive Education Course in Evaluating Social ... · 4. How to Randomize (Rohit Naimpally) 5. Sampling and Sample Size (Rohit Naimpally) 6. Threats and Analysis (Lina Marliani)](https://reader033.vdocuments.us/reader033/viewer/2022060506/5f1f2361952a4c4f081add04/html5/thumbnails/70.jpg)
• Should we scale up this program nationally– How representative is rural Rajasthan? (recall: 22% vs. 44% nationally)
– Do the same barriers to immunization exist in other parts of India?
– What is the cost of this program to implement?
– Do we need to replicate this study with a different version and/or in a different context?
Designing Policy from Evidence
J-PAL | INTRODUCING RANDOMIZED IMPACT EVALUATIONS 70
![Page 71: KOICA Executive Education Course in Evaluating Social ... · 4. How to Randomize (Rohit Naimpally) 5. Sampling and Sample Size (Rohit Naimpally) 6. Threats and Analysis (Lina Marliani)](https://reader033.vdocuments.us/reader033/viewer/2022060506/5f1f2361952a4c4f081add04/html5/thumbnails/71.jpg)
5. Cost-effectiveness Analysis
Evidence-Based Policymaking
![Page 72: KOICA Executive Education Course in Evaluating Social ... · 4. How to Randomize (Rohit Naimpally) 5. Sampling and Sample Size (Rohit Naimpally) 6. Threats and Analysis (Lina Marliani)](https://reader033.vdocuments.us/reader033/viewer/2022060506/5f1f2361952a4c4f081add04/html5/thumbnails/72.jpg)
Incentives and Reliable Supply
J-PAL | INTRODUCING RANDOMIZED IMPACT EVALUATIONS 72
![Page 73: KOICA Executive Education Course in Evaluating Social ... · 4. How to Randomize (Rohit Naimpally) 5. Sampling and Sample Size (Rohit Naimpally) 6. Threats and Analysis (Lina Marliani)](https://reader033.vdocuments.us/reader033/viewer/2022060506/5f1f2361952a4c4f081add04/html5/thumbnails/73.jpg)
Cost-Effectiveness Diagram
J-PAL | INTRODUCING RANDOMIZED IMPACT EVALUATIONS 73
![Page 74: KOICA Executive Education Course in Evaluating Social ... · 4. How to Randomize (Rohit Naimpally) 5. Sampling and Sample Size (Rohit Naimpally) 6. Threats and Analysis (Lina Marliani)](https://reader033.vdocuments.us/reader033/viewer/2022060506/5f1f2361952a4c4f081add04/html5/thumbnails/74.jpg)
When is a good time to do a randomized evaluation?
A. After the program has begun and you are not expanding it elsewhere
B. When a positive impact has been proven using rigorous methodology
C. When you are rolling out a program with the intention of taking it to scale
D. When a program is on a very small scale e.g one village with treatment and one without
J-PAL | INTRODUCING RANDOMIZED IMPACT EVALUATIONS 74A. B. C. D.
11% 11%
67%
11%
![Page 75: KOICA Executive Education Course in Evaluating Social ... · 4. How to Randomize (Rohit Naimpally) 5. Sampling and Sample Size (Rohit Naimpally) 6. Threats and Analysis (Lina Marliani)](https://reader033.vdocuments.us/reader033/viewer/2022060506/5f1f2361952a4c4f081add04/html5/thumbnails/75.jpg)
• When there is an important question for which you want/need to know the precise answer
• In terms of timing, not too early and not too late
• When the program and its implementation is representative and not gold plated (or tests a basic concept)
• You have the time, expertise, and money to do it right
• Develop an evaluation plan to prioritize
When to do a randomized evaluation?
J-PAL | INTRODUCING RANDOMIZED IMPACT EVALUATIONS 75
![Page 76: KOICA Executive Education Course in Evaluating Social ... · 4. How to Randomize (Rohit Naimpally) 5. Sampling and Sample Size (Rohit Naimpally) 6. Threats and Analysis (Lina Marliani)](https://reader033.vdocuments.us/reader033/viewer/2022060506/5f1f2361952a4c4f081add04/html5/thumbnails/76.jpg)
• When the program is premature and still requires considerable design work
• When the project is on too small in scale to randomize into at least two representative groups
• If a positive impact has been proven using rigorous methodology and resources are sufficient to cover everyone
• After the program has already begun and you are not expanding elsewhere
When not to do a randomized evaluation?
J-PAL | INTRODUCING RANDOMIZED IMPACT EVALUATIONS 76
![Page 77: KOICA Executive Education Course in Evaluating Social ... · 4. How to Randomize (Rohit Naimpally) 5. Sampling and Sample Size (Rohit Naimpally) 6. Threats and Analysis (Lina Marliani)](https://reader033.vdocuments.us/reader033/viewer/2022060506/5f1f2361952a4c4f081add04/html5/thumbnails/77.jpg)
• Start with a question• Verify the question hasn’t been answered• State a hypothesis• Design the evaluation• Determine whether the value of the answer is worth the cost of the
evaluation• With key questions answered from impact evaluations, process
evaluation can give your overall impact• A few high quality impact studies are worth more than many poor
quality ones• If you ask the right question, you’re more likely to care
Developing an evaluation strategy
J-PAL | INTRODUCING RANDOMIZED IMPACT EVALUATIONS 77
![Page 78: KOICA Executive Education Course in Evaluating Social ... · 4. How to Randomize (Rohit Naimpally) 5. Sampling and Sample Size (Rohit Naimpally) 6. Threats and Analysis (Lina Marliani)](https://reader033.vdocuments.us/reader033/viewer/2022060506/5f1f2361952a4c4f081add04/html5/thumbnails/78.jpg)
Some further readings
J-PAL | INTRODUCING RANDOMIZED IMPACT EVALUATIONS 78
![Page 79: KOICA Executive Education Course in Evaluating Social ... · 4. How to Randomize (Rohit Naimpally) 5. Sampling and Sample Size (Rohit Naimpally) 6. Threats and Analysis (Lina Marliani)](https://reader033.vdocuments.us/reader033/viewer/2022060506/5f1f2361952a4c4f081add04/html5/thumbnails/79.jpg)
Thank you