Download - Making the Most out of Discontinuities
DIME – FRAGILE STATESDUBAI, MAY 31 – JUNE 4
Making the Most out of Discontinuities
Presented byMalte Lierl (Yale University)
How do we measure program impact when random assignment is not possible ? e.g. universal take-up non-excludable intervention treatment already assigned
Solutions Make assumptions about what constitutes a plausible
control group (matching on observables, diff-in-diff) Exploit quasi-random aspects of program
implementation Quasi-experiments
Example: Regression Discontinuity Design (RDD)
Introduction
Discontinuity = Arbitrarily placed cutoff for program eligibility
Regression Discontinuity Design (RDD)
vulnerability index
income
Around the cutoff, beneficiary (‘treated’) and non-beneficiary (‘untreated’) populations are very similar.
For the population around the cutoff, RDD can be as credible as a randomized experiment.
vulnerability index
income PROGRAM
IMPACT
CUTOFF
RDD: Some examples
Example 1: Evaluate reintegration assistance for former child soldiers aged 16 and below.
An ex-combatant aged 16 years and one day would not benefit from the program.
RDD would compare individuals just above and just below 16 years of age.
RDD: Some examplesExample 2:
If you are elected into parliament, will this make you wealthier?
Can’t randomize who gets into parliament. In majoritarian systems such as in the UK, you
get into parliament if you have the majority of votes in a district.
Some districts have very close election results. Between two candidates with 49.5% and 50.5%
of votes it is as good as random who gets into parliament.
RDD: compares winners and losers in very close runoffs.
Another RDD exampleExample 3:
Minimum legal drinking age in the United States is 21
It is illegal to sell alcohol to people younger than 21
People aged 21 and people aged 20, 11 months, 29 days are treated very differently under the drinking age policy
But they are not inherently different (likelihood to go to parties, obedience, propensity to engage in risky behavior, etc.)
What is the effect of alcohol on mortality rates? In effect, the minimum drinking age
assigns people into ‘treatment’ and ‘comparison groups’ Treatment group: People between ages
20 years and 11 months and 20 years 11 months and 29 days cannot drink alcohol.
Comparison group: People just above 21 can drink.
Both groups should be similar in terms of observable and unobservable characteristics that affect outcomes (mortality rates).
If we use the drinking age cutoff as RDD, we can estimate the causal impact of alcohol consumption on mortality rates among young adults.
What is the effect of alcohol on mortality rates?
Source: Carpenter & Dubkin, 2009
RDD
Proportion of days drinking, by age
Increased alcohol consumption causes higher mortality rates around the age of 21
All deaths
All deaths associated with injuries, alcohol or drug use
All other deaths
RDD
What is the effect of alcohol on mortality rates?
Death rates, by age
Source: Carpenter & Dubkin, 2009
Internal Validity If the cutoff is arbitrary:
Individuals directly above and below the cutoff should be very similar in expectation
Systematic differences in outcomes are caused by the policy
Major assumptions: Individuals have no precise control over
assignment variable Nothing else is happening. In absence of the
policy, we would not observe a discontinuity around the cutoff.
Might not be the case if: ▪ Drinking age is 18, and driving also becomes legal at
age 18 ▪ Another program provides reintegration assistance for
ex-combatants over 16 years.
RDD Requirements
Transparency and precise knowledge of the selection process
‘Treatment’ is discontinuous with respect to an assignment variable
Individuals cannot precisely manipulate the assignment variable
All other factors are continuous with respect to the assignment variable (“nothing else is happening”)
Enough data points around the cutoff
Sharp discontinuity Discontinuity precisely determines treatment status
▪ All people 21 and older drink alcohol and no one else does
▪ All ex-combatants younger than 16 receive assistance, nobody else does
Fuzzy discontinuity Percentage of participants changes discontinuously at cut-
off, but not from 0% to 100% (or from 100% to 0%)▪ Some people younger than 21 end up consuming
alcohol and/or some older than 21 don’t consume at all▪ Some youth ex-combatants under 16 don’t participate,
and their slots are given to others who are just over 16.
Sharp and Fuzzy RDDs
Sharp and Fuzzy RDDs
1
0
1
0assignment variable
assignment variable
Probability of being treated
Probability of being treated
SHARP DISCONTINUITY
FUZZY DISCONTINUITY
External validity
Are RDD estimates of program impact generalizable?
Counterfactual/control group in RDD: Individuals marginally excluded from benefits Examples: Ex-combatants over 16, candidates
with 49.5% of votes Causal interpretation is limited to
individuals/households/villages near the cutoff Extrapolation beyond this group needs
additional (often unwarranted assumptions) Or multiple cutoffs!
RDD Implementation
Data collection: Make sure to have enough observations around the cutoff
Analysis: Observations away from the cutoff should have less weight
Why? Only near the cutoff can we assume that people find themselves to the left and to the right of the cut-off by chance.
weight
assignment variable
outcome
RDD Implementation
Carefully justify study design Baseline data will be useful to verify
assumptions
assignment variable
outcome
assignment variable
outcome
BEFORE PROGRAM
AFTER PROGRAM
RDD Implementation
Carefully justify study design Graphical analysis is an important tool
assignment variable
outcome
Summary
Advantages of RDDs: RDD can be applied even when
randomization is not feasible ▪ e.g. to programs with means tests for
eligibility For the population around the cutoff,
RDD is as credible as a randomized experiment▪ Requires fewer assumptions than other non-
experimental methods RDD can be used like a ‘natural
experiment’ to evaluate a program ex-post
Summary
Drawbacks of RDDs: Limited external validity: The estimates
of program effects are informative only for the population around the cutoff.
RDD requires a lot of data around the cutoff
Knowledge about the cutoff may induce behavioral change that can bias your evaluation▪ e.g. ex-combatants misreport their age▪ e.g. candidates become frustrated because
they were ‘so close’ to getting elected
شكرا 20
Thank you!
Further reading:
Lee, David and Thomas Lemieux (2009): Regression Discontinuity Designs in Economics, NBER Working Paper No. 14723. http://www.nber.org/papers/w14723