quasi-experimental methods
DESCRIPTION
Quasi-Experimental Methods. Florence Kondylis (World Bank). Objective. Find a plausible counterfactual Reality check Every method is associated with an assumption The stronger the assumption the more we need to worry about the causal effect Question your assumptions. Program to evaluate. - PowerPoint PPT PresentationTRANSCRIPT
AIM-CDD
Quasi-Experimental Methods
Florence Kondylis (World Bank)
AIM-CDD
Objective
• Find a plausible counterfactual»Reality check
• Every method is associated with an assumption
• The stronger the assumption the more we need to worry about the causal effect
»Question your assumptions
2
AIM-CDD
3
Program to evaluateFertilizer vouchers Program (2007-08)–Main Objective• Increase maize production
– Intervention: vouchers distribution–Target group:• Maize producers• Farmers owning >1 Ha, <3 Ha land
– Indicator: Yield (Maize)
AIM-CDD
I. Before-after identification strategy
Counterfactual:
Yield before program started
» EFFECT = After minus Before
Counterfactual assumption:
There is no other factor than the vouchers affecting yield from 2007 to 2008
years
4
AIM-CDD
5
AIM-CDD
6
Questioning the counterfactual assumption
Question: what else might have happened in 2007-2008 to affect maize yield ?
AIM-CDD
7
Examine assumption with prior data
Assumption of no change over time not so great ! >> There are external
factors (rainfall, pests…)
AIM-CDD
II. Non-participant identification strategy
Counterfactual:
Rate of pregnancy among non-participants
Counterfactual assumption:
Without vouchers, participants would as
productive as non-participants in a given year
8
AIM-CDD
9
AIM-CDD
Questioning the counterfactual assumption
10
Question: how might participants differ from non-participants?
AIM-CDD
Test assumption with pre-program data
11
REJECT counterfactual hypothesis of same productivity
AIM-CDD
III. Difference-in-Difference identification strategy
Counterfactual:
1.Non-participant maize yield, purging pre-program differences between participants/nonparticipants
2.“Before vouchers” maize yield, purging before-after change for nonparticipants (external factors)
• 1 and 2 are equivalent
12
AIM-CDD
13
AIM-CDD
14
57.50 - 46.37 = 11.13
66.37 – 62.90 = 3.47
Non-participants
Participants
Effect = 3.47 – 11.13 = - 7.66
AIM-CDD
15
After
Before
Effect = 8.87 – 16.53 = - 7.66
66.37 – 57.50 = 8.87
62.90 – 46.37 = 16.53
AIM-CDD
16
Counterfactual assumption:
Without intervention participants and nonparticipants’ pregnancy rates follow same trends
AIM-CDD
17
74.0
16.5
AIM-CDD
18
74.0 -7.6
AIM-CDD
19
Questioning the assumption
• Why might participants’ trends differ from that of nonparticipants?
AIM-CDD
20
Examine assumption with pre-program data
counterfactual hypothesis of same trends doesn’t look so believable
Average rate of teen pregnancy in
2004 2008 Difference (2004-2008)
Participants (P) 54.96 62.90 7.94
Non-participants (NP) 39.96 46.37 6.41
Difference (P=NP) 15.00 16.53 +1.53 ?
AIM-CDD
21
IV. Matching with Difference-in-Difference identification strategy
Counterfactual:
Comparison group is constructed by pairing each program participant with a “similar” nonparticipant using larger dataset – creating a control group from similar (in observable ways) non-participants
AIM-CDD
22
Counterfactual assumption:
Question: how might participants differ from matched nonparticipants?
Unobserved characteristics do not affect outcomes of interest
Unobserved = things we cannot measure (e.g. ability) or things we left out of the dataset
AIM-CDD
23
73.36
66.37Matched
nonparticipant
Participant
Effect = - 7.01
AIM-CDD
24
Can only test assumptionwith experimental data
Apply with care – think very hard about unobservables
Studies that compare both methods (because they have experimental data) find that:
unobservables often matter!
direction of bias is unpredictable!
AIM-CDD
Summary
• Randomization requires minimal assumptions needed and procures intuitive estimates (sample means !)
• Non-experimental requires assumptions that must be carefully assessed
»More data-intensive
25
AIM-CDD
Example: Irrigation for rice producers + Enhanced Market Access• Impact of interest measured by:
– Input use & repayment of irrigation fee– Rice yield– (Cash) income from rice– Non-rice cash income (spillovers to other value chains)
• Data: 500 farmers in project area / 500 random sample farmers– Before & after treatment
»Can’t randomize irrigation so what is the counterfactual?
AIM-CDD
Plausible counterfactuals• Random sample difference in difference
– Are farmers outside the scheme on the same trajectory ?
• Farmers in the vicinity of the scheme but not included in scheme– Selection of project area needs to be carefully documented
(elevation…)
– Proximity implies “just-outside farmers” might also benefit from enhanced market linkages
» What do we want to measure?
• Propensity score matching
– Unobservables determining on-farm productivity ?
27
AIM-CDD
28
Thank You