nigeria impact evaluation community of practice abuja, nigeria, april 2, 2014 measuring program...
TRANSCRIPT
![Page 1: Nigeria Impact Evaluation Community of Practice Abuja, Nigeria, April 2, 2014 Measuring Program Impacts Through Randomization David Evans (World Bank)](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649f1c5503460f94c32c29/html5/thumbnails/1.jpg)
Nigeria Impact EvaluationCommunity of Practice
Abuja, Nigeria, April 2, 2014
Measuring Program Impacts ThroughRandomization
David Evans (World Bank)
![Page 2: Nigeria Impact Evaluation Community of Practice Abuja, Nigeria, April 2, 2014 Measuring Program Impacts Through Randomization David Evans (World Bank)](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649f1c5503460f94c32c29/html5/thumbnails/2.jpg)
2
Objective
Evaluate the causal impact of a program or an intervention on some outcome
Examples How much did free distribution of
bednets decrease malaria incidence?
Which of two supply chain models was most effective at eliminating drug shortages?
![Page 3: Nigeria Impact Evaluation Community of Practice Abuja, Nigeria, April 2, 2014 Measuring Program Impacts Through Randomization David Evans (World Bank)](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649f1c5503460f94c32c29/html5/thumbnails/3.jpg)
3
Counterfactual Criteria
Treated & comparison groups…Have identical average
characteristics (observed & unobserved)
The only difference is the treatment
Therefore the only reason for any difference in outcomes is the treatment
Key question: What would participant look like if she hadn’t received the program?
![Page 4: Nigeria Impact Evaluation Community of Practice Abuja, Nigeria, April 2, 2014 Measuring Program Impacts Through Randomization David Evans (World Bank)](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649f1c5503460f94c32c29/html5/thumbnails/4.jpg)
4
Perfect Experiment
1. Identify target beneficiaries
2. Clone them!
Identical on the outside (observable)
Identical on the inside (unobservable)
Chief Ahun
We’re both middle-aged
chiefs
We both love to take up new
health interventions!
Chief Batun
![Page 5: Nigeria Impact Evaluation Community of Practice Abuja, Nigeria, April 2, 2014 Measuring Program Impacts Through Randomization David Evans (World Bank)](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649f1c5503460f94c32c29/html5/thumbnails/5.jpg)
5
Perfect Experiment
Give the intervention to one set of clones
Ahun Batun
![Page 6: Nigeria Impact Evaluation Community of Practice Abuja, Nigeria, April 2, 2014 Measuring Program Impacts Through Randomization David Evans (World Bank)](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649f1c5503460f94c32c29/html5/thumbnails/6.jpg)
6
Perfect Experiment
Observe some time later
Because the groups are identical (inside & out), the difference is due to the bednets!
Ahun
Batun
![Page 7: Nigeria Impact Evaluation Community of Practice Abuja, Nigeria, April 2, 2014 Measuring Program Impacts Through Randomization David Evans (World Bank)](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649f1c5503460f94c32c29/html5/thumbnails/7.jpg)
7
Back to Reality
What would Batun look like if he didn’t receive the bednet?
Room For Improvement Control Groups
Before – After Participants – Non Participants
???
![Page 8: Nigeria Impact Evaluation Community of Practice Abuja, Nigeria, April 2, 2014 Measuring Program Impacts Through Randomization David Evans (World Bank)](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649f1c5503460f94c32c29/html5/thumbnails/8.jpg)
RFI: Before-After
BEFORE BEDNETS
6 malaria episodes in 6 months
AFTER BEDNETS
2 malaria episodes in 6 months
What else might be going on besides the bednets?
• Seasonal differences• Rising incomes: Households invest in other measures
Too many other factors!
Impact of bednets = ???
![Page 9: Nigeria Impact Evaluation Community of Practice Abuja, Nigeria, April 2, 2014 Measuring Program Impacts Through Randomization David Evans (World Bank)](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649f1c5503460f94c32c29/html5/thumbnails/9.jpg)
RFI: Before-After
Important to monitor before-after Insufficient to show impact of
programToo many factors changing over time
Example of cash transfers in Nicaragua!
Counterfactual: What would have happened in the absence of the project, with everything else the same
![Page 10: Nigeria Impact Evaluation Community of Practice Abuja, Nigeria, April 2, 2014 Measuring Program Impacts Through Randomization David Evans (World Bank)](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649f1c5503460f94c32c29/html5/thumbnails/10.jpg)
10
RFI: Participants vs Non-Participants
Compare recipients of a program to People who were not eligible for the
program People who chose not to enroll in the
program
Home births Clinic births
Example: Complications in childbirth
Impact of clinic births?
What else might explain the difference?
![Page 11: Nigeria Impact Evaluation Community of Practice Abuja, Nigeria, April 2, 2014 Measuring Program Impacts Through Randomization David Evans (World Bank)](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649f1c5503460f94c32c29/html5/thumbnails/11.jpg)
11
RFI: Participants vs Non-Participants
Observable differences Income Education
Unobservable differences Heard rumor about hospitals Neighbor available to care for other
children
Mercy Patience
![Page 12: Nigeria Impact Evaluation Community of Practice Abuja, Nigeria, April 2, 2014 Measuring Program Impacts Through Randomization David Evans (World Bank)](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649f1c5503460f94c32c29/html5/thumbnails/12.jpg)
12
RFI: Participants vs Non-Participants
How much of difference is because of clinic?
Impact of clinic births = ???
Home births Clinic births
Example: Complications in childbirth
Impact of clinic births
Other factors!
![Page 13: Nigeria Impact Evaluation Community of Practice Abuja, Nigeria, April 2, 2014 Measuring Program Impacts Through Randomization David Evans (World Bank)](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649f1c5503460f94c32c29/html5/thumbnails/13.jpg)
13
Selection bias
People who choose to join the program are different!
If we cannot account completely for those differences in our data…
We never can How do you capture attitudes toward
health systems? Initiative? …then our comparison will not show
the true impact of the program
![Page 14: Nigeria Impact Evaluation Community of Practice Abuja, Nigeria, April 2, 2014 Measuring Program Impacts Through Randomization David Evans (World Bank)](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649f1c5503460f94c32c29/html5/thumbnails/14.jpg)
What should we do?
Gold standard:Randomized experimental design
![Page 15: Nigeria Impact Evaluation Community of Practice Abuja, Nigeria, April 2, 2014 Measuring Program Impacts Through Randomization David Evans (World Bank)](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649f1c5503460f94c32c29/html5/thumbnails/15.jpg)
15
Randomized Experimental Design
Randomly assign potential beneficiaries to be in the treatment or comparison group
Treatment and comparison have the same characteristics (observed and unobserved), on average, so…
Any difference in outcomes is due to treatment
![Page 16: Nigeria Impact Evaluation Community of Practice Abuja, Nigeria, April 2, 2014 Measuring Program Impacts Through Randomization David Evans (World Bank)](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649f1c5503460f94c32c29/html5/thumbnails/16.jpg)
16
Why Randomization Works
Randomization with two doesn’t work!
But differences average out in a big sample
On average, same number of Ahuns and Patiences Observable AND unobservable
Result: Measure true impact of program
ComparisonTreatment
Comparison
Treatment
![Page 17: Nigeria Impact Evaluation Community of Practice Abuja, Nigeria, April 2, 2014 Measuring Program Impacts Through Randomization David Evans (World Bank)](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649f1c5503460f94c32c29/html5/thumbnails/17.jpg)
17
Random Sample or Assignment?
RANDOM SAMPLE
Select randomly who to gather data on
Gives unbiased average of the group NOT of impact
If take random sample of group: Half women, half men – Sample should be about ½ women, ½ men
RANDOM ASSIGNMENT
Randomly assign who gets the program
Gives unbiased estimate of program impact
Why? Treatment & comparison
are IDENTICAL (on average)
T
T
C
C
![Page 18: Nigeria Impact Evaluation Community of Practice Abuja, Nigeria, April 2, 2014 Measuring Program Impacts Through Randomization David Evans (World Bank)](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649f1c5503460f94c32c29/html5/thumbnails/18.jpg)
18
LET’S RANDOMIZE!
1. Identify the eligible participants
What is the impact of
receiving a new car on body-mass
index?
![Page 19: Nigeria Impact Evaluation Community of Practice Abuja, Nigeria, April 2, 2014 Measuring Program Impacts Through Randomization David Evans (World Bank)](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649f1c5503460f94c32c29/html5/thumbnails/19.jpg)
19
LET’S RANDOMIZE!
2. Generate a random number for each one
![Page 20: Nigeria Impact Evaluation Community of Practice Abuja, Nigeria, April 2, 2014 Measuring Program Impacts Through Randomization David Evans (World Bank)](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649f1c5503460f94c32c29/html5/thumbnails/20.jpg)
20
LET’S RANDOMIZE!
2. Generate a random number for each one
![Page 21: Nigeria Impact Evaluation Community of Practice Abuja, Nigeria, April 2, 2014 Measuring Program Impacts Through Randomization David Evans (World Bank)](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649f1c5503460f94c32c29/html5/thumbnails/21.jpg)
21
LET’S RANDOMIZE!
3. Re-order based on the random number
![Page 22: Nigeria Impact Evaluation Community of Practice Abuja, Nigeria, April 2, 2014 Measuring Program Impacts Through Randomization David Evans (World Bank)](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649f1c5503460f94c32c29/html5/thumbnails/22.jpg)
22
LET’S RANDOMIZE!
4. Assign the first ten to receive cars
I really wanted a car!
![Page 23: Nigeria Impact Evaluation Community of Practice Abuja, Nigeria, April 2, 2014 Measuring Program Impacts Through Randomization David Evans (World Bank)](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649f1c5503460f94c32c29/html5/thumbnails/23.jpg)
23
LET’S RANDOMIZE!
5. Check for balance across treatment & control
Treatment# Drs: 4
Control# Drs: 6
Total observations21
Not a very big sample!
How close? 2/3
![Page 24: Nigeria Impact Evaluation Community of Practice Abuja, Nigeria, April 2, 2014 Measuring Program Impacts Through Randomization David Evans (World Bank)](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649f1c5503460f94c32c29/html5/thumbnails/24.jpg)
24
LET’S RANDOMIZE!
What if we had 500 observations?
Treatment# Drs: 114
Control# Drs: 116
How close? 98/100
![Page 25: Nigeria Impact Evaluation Community of Practice Abuja, Nigeria, April 2, 2014 Measuring Program Impacts Through Randomization David Evans (World Bank)](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649f1c5503460f94c32c29/html5/thumbnails/25.jpg)
25
Is there more?
That’s a simple way to randomize Works with BIG samples
You can help randomization by stratifying Randomize within each sub-group Ensure that each group is equally
represented
![Page 26: Nigeria Impact Evaluation Community of Practice Abuja, Nigeria, April 2, 2014 Measuring Program Impacts Through Randomization David Evans (World Bank)](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649f1c5503460f94c32c29/html5/thumbnails/26.jpg)
26
Stratifying
1. Identify the characteristic(s) to stratify on
![Page 27: Nigeria Impact Evaluation Community of Practice Abuja, Nigeria, April 2, 2014 Measuring Program Impacts Through Randomization David Evans (World Bank)](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649f1c5503460f94c32c29/html5/thumbnails/27.jpg)
27
Stratifying
2. Sort on those characteristics
![Page 28: Nigeria Impact Evaluation Community of Practice Abuja, Nigeria, April 2, 2014 Measuring Program Impacts Through Randomization David Evans (World Bank)](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649f1c5503460f94c32c29/html5/thumbnails/28.jpg)
28
Stratifying
3. Generate those random numbers
![Page 29: Nigeria Impact Evaluation Community of Practice Abuja, Nigeria, April 2, 2014 Measuring Program Impacts Through Randomization David Evans (World Bank)](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649f1c5503460f94c32c29/html5/thumbnails/29.jpg)
29
Stratifying
4. Sort on those characteristics
![Page 30: Nigeria Impact Evaluation Community of Practice Abuja, Nigeria, April 2, 2014 Measuring Program Impacts Through Randomization David Evans (World Bank)](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649f1c5503460f94c32c29/html5/thumbnails/30.jpg)
30
Stratifying
5. Assign treatment within each sub-group
Result: Equal doctors in each group
![Page 31: Nigeria Impact Evaluation Community of Practice Abuja, Nigeria, April 2, 2014 Measuring Program Impacts Through Randomization David Evans (World Bank)](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649f1c5503460f94c32c29/html5/thumbnails/31.jpg)
31
Can we really randomize?
Randomization does not mean denying people the benefits of the project
Usually existing constraints in project roll-out allow randomization
Randomization often the fairest way to allocate treatment
![Page 32: Nigeria Impact Evaluation Community of Practice Abuja, Nigeria, April 2, 2014 Measuring Program Impacts Through Randomization David Evans (World Bank)](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649f1c5503460f94c32c29/html5/thumbnails/32.jpg)
32
Use Staggered Roll-out
Roll-out to 200 clinicsRoll-out to 400 more
clinics
Jan 2014
July 2014
Jan 2015
• Randomize the order in which clinics receive program
• Compare Jan 2014 group to Jan 2015 group at end of first year
![Page 33: Nigeria Impact Evaluation Community of Practice Abuja, Nigeria, April 2, 2014 Measuring Program Impacts Through Randomization David Evans (World Bank)](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649f1c5503460f94c32c29/html5/thumbnails/33.jpg)
33
Some groups must get the program
![Page 34: Nigeria Impact Evaluation Community of Practice Abuja, Nigeria, April 2, 2014 Measuring Program Impacts Through Randomization David Evans (World Bank)](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649f1c5503460f94c32c29/html5/thumbnails/34.jpg)
34
Vary treatment
INTENSITY OF TREATMENT
Malaria Information Campaign•100 villagesMalaria Information Campaign + SMS Reminders•100 villages
NATURE OF TREATMENT
Radio campaign•100 villages
Newspaper campaign•100 villages
![Page 35: Nigeria Impact Evaluation Community of Practice Abuja, Nigeria, April 2, 2014 Measuring Program Impacts Through Randomization David Evans (World Bank)](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649f1c5503460f94c32c29/html5/thumbnails/35.jpg)
35
Randomization is often the fairest
Watch the movie![link]
Randomization of an early child development program in Côte d’Ivoire
![Page 36: Nigeria Impact Evaluation Community of Practice Abuja, Nigeria, April 2, 2014 Measuring Program Impacts Through Randomization David Evans (World Bank)](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649f1c5503460f94c32c29/html5/thumbnails/36.jpg)
36
What if randomization is impossible?
Think again: It often is possible on some level, and it’s the best way to get a clear measure of impact Always begin the IE with imagining what the
ideal would look like
With a national policy Use randomization to test implementation
![Page 37: Nigeria Impact Evaluation Community of Practice Abuja, Nigeria, April 2, 2014 Measuring Program Impacts Through Randomization David Evans (World Bank)](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649f1c5503460f94c32c29/html5/thumbnails/37.jpg)
37
Key takeaway #1
The single best way to evaluate the unbiased average impact of an activity is by randomizing treatment.
![Page 38: Nigeria Impact Evaluation Community of Practice Abuja, Nigeria, April 2, 2014 Measuring Program Impacts Through Randomization David Evans (World Bank)](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649f1c5503460f94c32c29/html5/thumbnails/38.jpg)
38
Key takeaway #2
It is more ethical to test programs rigorously before universally implementing them than it is to use scarce public resources to implement a universal program with uncertain benefits.
![Page 39: Nigeria Impact Evaluation Community of Practice Abuja, Nigeria, April 2, 2014 Measuring Program Impacts Through Randomization David Evans (World Bank)](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649f1c5503460f94c32c29/html5/thumbnails/39.jpg)
39
Key takeaway #3
Randomization is more flexible than you think:
It does not require withholding of benefits.
It can take advantage of necessary staggered roll-out.
It can test different reforms or packages of services across groups at the same time (so all receive at least some package).
![Page 40: Nigeria Impact Evaluation Community of Practice Abuja, Nigeria, April 2, 2014 Measuring Program Impacts Through Randomization David Evans (World Bank)](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649f1c5503460f94c32c29/html5/thumbnails/40.jpg)
40
Let’s randomize
![Page 41: Nigeria Impact Evaluation Community of Practice Abuja, Nigeria, April 2, 2014 Measuring Program Impacts Through Randomization David Evans (World Bank)](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649f1c5503460f94c32c29/html5/thumbnails/41.jpg)
41
Thank you!
![Page 42: Nigeria Impact Evaluation Community of Practice Abuja, Nigeria, April 2, 2014 Measuring Program Impacts Through Randomization David Evans (World Bank)](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649f1c5503460f94c32c29/html5/thumbnails/42.jpg)
BONUS SLIDES
![Page 43: Nigeria Impact Evaluation Community of Practice Abuja, Nigeria, April 2, 2014 Measuring Program Impacts Through Randomization David Evans (World Bank)](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649f1c5503460f94c32c29/html5/thumbnails/43.jpg)
43
![Page 44: Nigeria Impact Evaluation Community of Practice Abuja, Nigeria, April 2, 2014 Measuring Program Impacts Through Randomization David Evans (World Bank)](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649f1c5503460f94c32c29/html5/thumbnails/44.jpg)
44
LET’S RANDOMIZE!
What if we had 100 observations?
Treatment# Drs: 26
Control# Drs: 22
Total observations21
Not a very big sample!