using randomized evaluations to improve policy
DESCRIPTION
Nandini Krishnan. Using Randomized Evaluations to Improve Policy. Objective. To Identify the causal effect of an intervention separate the impact of the program from other factors Need to find out what would have happened without the program - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Using Randomized Evaluations to Improve Policy](https://reader035.vdocuments.us/reader035/viewer/2022062521/568166ee550346895ddb41a2/html5/thumbnails/1.jpg)
1
AADAPT Workshop South AsiaGoa, December 17-21, 2009
Using Randomized Evaluations to Improve Policy
Nandini Krishnan
![Page 2: Using Randomized Evaluations to Improve Policy](https://reader035.vdocuments.us/reader035/viewer/2022062521/568166ee550346895ddb41a2/html5/thumbnails/2.jpg)
2
Objective
To Identify the causal effect of an intervention separate the impact of the program from
other factors Need to find out what would have
happened without the program Cannot observe the same person with and
without the program at the same point of time>> Create a valid counterfactual
![Page 3: Using Randomized Evaluations to Improve Policy](https://reader035.vdocuments.us/reader035/viewer/2022062521/568166ee550346895ddb41a2/html5/thumbnails/3.jpg)
Correlation is not causation
High Yield
Fertilizer Use
OR ?
2) ?
1) ?
High Yield
Knowledge of
Technologies Fertilizer
Use
![Page 4: Using Randomized Evaluations to Improve Policy](https://reader035.vdocuments.us/reader035/viewer/2022062521/568166ee550346895ddb41a2/html5/thumbnails/4.jpg)
Before After02468
101214
Treatment GroupTreatment Group
4
(+6) BIASED measure of the program impact
Illustration: Fertilizer Voucher Program (Before-After)
![Page 5: Using Randomized Evaluations to Improve Policy](https://reader035.vdocuments.us/reader035/viewer/2022062521/568166ee550346895ddb41a2/html5/thumbnails/5.jpg)
5
Hard to distinguish causation from correlation from statistical analysis of existing data However complex the statistics can only see that X
moves with Y Hard to correct for unobserved characteristics, like
motivation/ability May be very important- also affect outcomes of interest
Selection bias a major issue for impact evaluation Projects started at specific times and places for particular
reasons Participants may be selected or self-select into programs First farmers to adopt a new technology are likely to be
very different from the average farmer, looking at their yields will give you a misleading impression of the benefits of a new technology
Motivation
![Page 6: Using Randomized Evaluations to Improve Policy](https://reader035.vdocuments.us/reader035/viewer/2022062521/568166ee550346895ddb41a2/html5/thumbnails/6.jpg)
Before After0
2
4
6
8
10
12
14Control GroupTreatment Group
6
(+4) Impact of the program
(+2) Impact of other (external) factors
Illustration: Fertilizer Voucher Program (Valid Counterfactual)
![Page 7: Using Randomized Evaluations to Improve Policy](https://reader035.vdocuments.us/reader035/viewer/2022062521/568166ee550346895ddb41a2/html5/thumbnails/7.jpg)
7
All those in the study have the same chance of being in the treatment or comparison group
By design, treatment and comparison have the same characteristics (observed and unobserved), on average Only difference is treatment
With large sample, all characteristics average out
Unbiased impact estimates
Experimental Design
![Page 8: Using Randomized Evaluations to Improve Policy](https://reader035.vdocuments.us/reader035/viewer/2022062521/568166ee550346895ddb41a2/html5/thumbnails/8.jpg)
8
Lottery (0nly some receive) Lottery to receive information about a new
agricultural technology Random phase-in (everyone gets it eventually)
Train some farmers groups each year Variation in treatment
Some get information on new seed, others get credit, others get demonstration plot on their land etc
Encouragement design One farmers support center per district Some farmers get travel voucher to attend the center
Options for Randomization
![Page 9: Using Randomized Evaluations to Improve Policy](https://reader035.vdocuments.us/reader035/viewer/2022062521/568166ee550346895ddb41a2/html5/thumbnails/9.jpg)
Lottery among the qualified
Must receive the program
Not suitable for the program
Randomize who gets the program
![Page 10: Using Randomized Evaluations to Improve Policy](https://reader035.vdocuments.us/reader035/viewer/2022062521/568166ee550346895ddb41a2/html5/thumbnails/10.jpg)
10
Budget constraint prevents full coverage Random assignment (lottery) is fair and
transparent Limited implementation capacity
Phase-in gives all the same chance to go first
No evidence on which alternative is best Random assignment to alternatives
with equal ex ante chance of success
Opportunities
![Page 11: Using Randomized Evaluations to Improve Policy](https://reader035.vdocuments.us/reader035/viewer/2022062521/568166ee550346895ddb41a2/html5/thumbnails/11.jpg)
11
Opportunities for Randomization Take up of existing program is not
complete Provide information or incentive for some to
sign up- Randomize encouragement Pilot a new program
Good opportunity to test design before scaling up
Operational changes to ongoing programs Good opportunity to test changes before
scaling them up
![Page 12: Using Randomized Evaluations to Improve Policy](https://reader035.vdocuments.us/reader035/viewer/2022062521/568166ee550346895ddb41a2/html5/thumbnails/12.jpg)
12
Individual Farm Farmers’ Association Irrigation block
Village level Women’s
association Youth groups School level
Different levels you can randomize at
![Page 13: Using Randomized Evaluations to Improve Policy](https://reader035.vdocuments.us/reader035/viewer/2022062521/568166ee550346895ddb41a2/html5/thumbnails/13.jpg)
Group or individual randomization? If a program impacts a whole group--
usually randomize whole community to treatment or comparison
Easier to get big enough sample if randomize individuals
Individual randomization Group randomization
![Page 14: Using Randomized Evaluations to Improve Policy](https://reader035.vdocuments.us/reader035/viewer/2022062521/568166ee550346895ddb41a2/html5/thumbnails/14.jpg)
14
Randomizing at higher level sometimes necessary: Political constraints on differential treatment
within community Practical constraints—confusing to implement
different versions Spillover effects may require higher level
randomization
Randomizing at group level requires many groups because of within community correlation
Unit of Randomization
![Page 15: Using Randomized Evaluations to Improve Policy](https://reader035.vdocuments.us/reader035/viewer/2022062521/568166ee550346895ddb41a2/html5/thumbnails/15.jpg)
15
Elements of an experimental design
![Page 16: Using Randomized Evaluations to Improve Policy](https://reader035.vdocuments.us/reader035/viewer/2022062521/568166ee550346895ddb41a2/html5/thumbnails/16.jpg)
16
External validity The evaluation sample is representative of the total population The results in the sample represent the results in the population We can apply the lessons to the whole population
Internal validity The intervention and comparison groups are truly comparable estimated effect of the intervention/program on the evaluated population reflects the real impact on that population
External and Internal Validity (1)
![Page 17: Using Randomized Evaluations to Improve Policy](https://reader035.vdocuments.us/reader035/viewer/2022062521/568166ee550346895ddb41a2/html5/thumbnails/17.jpg)
17
External and Internal Validity (2) An evaluation can have internal validity without external validity
Example: A randomized evaluation of encouraging women to stand for elections in urban areas may not tell us much about impact of a similar program in rural areas
An evaluation without internal validity, can’t have external validity If you don’t know whether a program works in
one place, then you have learnt nothing about whether it works elsewhere.
![Page 18: Using Randomized Evaluations to Improve Policy](https://reader035.vdocuments.us/reader035/viewer/2022062521/568166ee550346895ddb41a2/html5/thumbnails/18.jpg)
18
Internal & external validity
Random Sample- Randomization
Randomization
National Population
Samples National Population
![Page 19: Using Randomized Evaluations to Improve Policy](https://reader035.vdocuments.us/reader035/viewer/2022062521/568166ee550346895ddb41a2/html5/thumbnails/19.jpg)
19
Internal validity
Stratification
Randomization
Population
Population stratumSamples of Population
Stratum
![Page 20: Using Randomized Evaluations to Improve Policy](https://reader035.vdocuments.us/reader035/viewer/2022062521/568166ee550346895ddb41a2/html5/thumbnails/20.jpg)
20
Representative but biased: useless
National Population
Biased assignmentUSELESS!
Randomization
![Page 21: Using Randomized Evaluations to Improve Policy](https://reader035.vdocuments.us/reader035/viewer/2022062521/568166ee550346895ddb41a2/html5/thumbnails/21.jpg)
21
Efficacy & Effectiveness Efficacy
Proof of concept Smaller scale Pilot in ideal conditions
Effectiveness At scale Prevailing implementation arrangements --
“real life”
Higher or lower impact? Higher or lower costs?
![Page 22: Using Randomized Evaluations to Improve Policy](https://reader035.vdocuments.us/reader035/viewer/2022062521/568166ee550346895ddb41a2/html5/thumbnails/22.jpg)
22
Advantages of “experiments” Clear and precise causal impact Relative to other methods
Much easier to analyze- Difference in averages
Cheaper (smaller sample sizes) Easier to explain More convincing to policymakers Methodologically uncontroversial
![Page 23: Using Randomized Evaluations to Improve Policy](https://reader035.vdocuments.us/reader035/viewer/2022062521/568166ee550346895ddb41a2/html5/thumbnails/23.jpg)
Randomly assigning wheat plots to receive regular irrigation water
Wheat plants do NOT Raise ethical or practical concerns about
randomization Fail to comply with Treatment Find a better Treatment Move away—so lost to measurement Refuse to answer questionnaires
Human beings can be a little more challenging!
![Page 24: Using Randomized Evaluations to Improve Policy](https://reader035.vdocuments.us/reader035/viewer/2022062521/568166ee550346895ddb41a2/html5/thumbnails/24.jpg)
What if there are constraints on randomization?
Some interventions can’t be assigned randomly
Partial take up or demand-driven interventions: Randomly promote the program to some Participants make their own choices about
adoption Perhaps there is contamination- for
instance, if some in the control group take-up treatment
24
![Page 25: Using Randomized Evaluations to Improve Policy](https://reader035.vdocuments.us/reader035/viewer/2022062521/568166ee550346895ddb41a2/html5/thumbnails/25.jpg)
Random Promotion(Encouragement Design) Those who get promotion are more likely to
enroll But who got promotion was determined
randomly, so not correlated with other observables/non-observables Compare average outcomes of two groups:
promoted/not promoted Effect of offering the encouragement (Intent-To-
Treat) Effect of the intervention on the complier population
(Local Average Treatment Effect)▪ LATE= ITT/proportion of those who took it up
![Page 26: Using Randomized Evaluations to Improve Policy](https://reader035.vdocuments.us/reader035/viewer/2022062521/568166ee550346895ddb41a2/html5/thumbnails/26.jpg)
RandomizationAssigned to treatment
Assigned to control
Difference Impact: Average treatment effect on the treated
Non-treated
Treated
Proportion treated
100% 0% 100%Impact of assignment
100%
Mean outcome
103 80 23Intent-to-treat estimate
23/100%=23Average treatment on the treated
![Page 27: Using Randomized Evaluations to Improve Policy](https://reader035.vdocuments.us/reader035/viewer/2022062521/568166ee550346895ddb41a2/html5/thumbnails/27.jpg)
Random encouragement
RandomlyEncouraged
Not encouraged
Difference Impact: Average treatment effect on compliers
Non-treated(did not take up program)Treated(did take up program)
Proportion treated
70% 30% 40%Impact of encouragement
100%
Outcome 100 92 8Intent-to-treat estimate
8/40%=20Local average treatment effect
![Page 28: Using Randomized Evaluations to Improve Policy](https://reader035.vdocuments.us/reader035/viewer/2022062521/568166ee550346895ddb41a2/html5/thumbnails/28.jpg)
Common pitfalls to avoid Calculating sample size incorrectly
Randomizing one district to treatment and one district to control and calculating sample size on number of people you interview
Collecting data in treatment and control differently
Counting those assigned to treatment who do not take up program as control—don’t undo your randomization!!
28
![Page 29: Using Randomized Evaluations to Improve Policy](https://reader035.vdocuments.us/reader035/viewer/2022062521/568166ee550346895ddb41a2/html5/thumbnails/29.jpg)
29
When is it really not possible? The treatment already assigned and
announcedand no possibility for expansion of treatment
The program is over (retrospective) Universal take up already Program is national and non
excludable Freedom of the press, exchange rate
policy(sometimes some components can be
randomized) Sample size is too small to make it
worth it
![Page 30: Using Randomized Evaluations to Improve Policy](https://reader035.vdocuments.us/reader035/viewer/2022062521/568166ee550346895ddb41a2/html5/thumbnails/30.jpg)
30
Thank You