Global Workshop onDevelopment Impact Evaluation

in Finance and Private SectorRio de Janeiro, June 6-10, 2011

Making the Most out of Discontinuities

Florence Kondylis

Introduction (1)

Context we want to measure the causal impact of an intervention the assignment to this intervention cannot be randomized selection into program participation cannot be exploited to

establish an adequate comparison group

In general: Individuals, households, villages, or other entities, are either

exposed or not exposed to a treatment / policy regime selection into the program makes it impossible to compare

treated / non-treated to establish the impact of the program Example: Individuals who wish to take part in a micro-

finance program and those who don’t – participation is likely driven by key characteristics

2

Introduction (2)

When randomization is not feasible, how to exploit the roll-out of an intervention to measure its causal impact?

Proposal: we can use quasi/non-experimental methods Difference-In-Differences and Matching Regression Discontinuity Design (RDD)

3

Regression Discontinuity Designs

RDD is closer cousin of randomized experiments than other competitors

RDD is based on the selection process When in presence of an official/bureaucratic, clear and

reasonably enforced eligibility rule A simple, quantifiable score

Assignment to treatment is based on this score A threshold is established

▪ Ex: target firms with sales above a certain amount▪ Those above receive, those below do not

▪ compare firms just above the threshold to firms just below the threshold 4

RDD in Practice

Policy: US drinking age, minimum legal age is 21 under 21, alcohol consumption is illegal

Outcomes: alcohol consumption and mortality rate

Observation: The policy implies that individuals aged 20 years, 11 months and 29 days cannot drink

individuals ages 21 years, 0 month and 1 day can drink

however, do we think that these individuals are inherently different? wisdom, preferences for alcohol and driving, party-going behavior, etc

People born “few days apart” are treated differently, because of the arbitrary age cut off established by the law

a few days or a month age difference could is unlikely to yield variations in behavior and attitude towards alcohol

The legal status is the only difference between the treatment group (just above 21) and comparison group (just below 21)

5

RDD in Practice

In practice, making alcohol consumption illegal lowers consumption and, therefore, the incidence of drunk-driving Idea: use the following groups to measure the impact of a minimum drinking age on mortality rate of young adults

Treatment group: individuals 20 years and 11 months to 21 years old

Comparison group: individuals 21 years to 21 years and a month old

Around the threshold, we can safely assume that individuals are randomly assigned to the treatmentWe can then measure the causal impact of the policy on mortality rates around the threshold

6

RDD Example

7

MLDA (Treatment) reduces alcohol consumption

RDD Example

8

Total number of Deaths

Higher alcohol consumption increases death rate around age 21

Total number of accidental deaths related to alcohol and drug consumption

Total number of other deaths

RDD Logic

Assignment to the treatment depends, either completely or partly, on a continuous “score”, ranking (age in the previous case):

potential beneficiaries are ordered by looking at the scorethere is a cut-off point for “eligibility” – clearly defined criterion determined ex-antecut-off determines the assignment to the treatment or no-treatment groups

These de facto assignments often result from administrative decisions

resource constraints limit coveragevery targeted intervention with expected heterogeneous impact transparent rules rather than discretion used

9

Example: matching grants(fuzzy design)

Government gives matching grants to firms Eligibility rule based on annual sales:

• If annual sales < $5,000 then firm receives grants

• If annual sales >= $5,000 then no matching grants

A firm with sales=$5,001 would not be treated (be eligible) but would be very similar to a firm with sales=$5,000 Need to measure annual sales before the scheme is

announced to prevent manipulation of the figure

RDD would compare firms just above and just below the $5,000 threshold

10

Subtle point …

Question: How to address incomplete compliance to the treatment Ex: Low take-up of a matching grant

scheme There are two types of discontinuity

Sharp (near full compliance, e.g. a law) Fuzzy (incomplete compliance, e.g. a

subsidy) Going back to our example …

11

Example: matching grant (fuzzy design)

Now suppose that not all the eligible firms receive the grants. Why? limited knowledge of the program voluntary participation these reasons signal a selection bias into the program:

decision to enter the program is correlated with other firm characteristics

Yet, the percentage of participants still changes discontinuously at cut-off from zero to less than 100% this is called a fuzzy discontinuity (vs. sharp)

12

0.2

5.5

.75

1

tre

atm

en

t pro

bab

ility

assignment variable

Sharp Design for Voucher receipt

0.2

5.5

.75

1

assignment variable

Fuzzy Design for Voucher receipt

Probability of Participation under Alternative Designs

100%

0%

75%

0%

Sharp Design for Grant receipt Fuzzy Design for Grant receipt

13

Variationsabove the threshold

Sharp and Fuzzy Discontinuities (1)

Ideal setting: Sharp discontinuity the discontinuity precisely determines

treatment status ▪ e.g. ONLY people 21 and older drink alcohol, and

ALL drink it!▪ Only small firms receive grants▪ Progressive taxation rate

14

Sharp and Fuzzy Discontinuities (2)

Fuzzy discontinuitythe percentage of participants changes discontinuously at cut-off, but not from zero to 100%

▪ e.g. rules determine eligibility but amongst the small firms there is only partial compliance / take-up

▪ Some people younger than 21 end up consuming alcohol and some older than 21 don’t consume at all

15

Internal Validity

General idea the arbitrary cut off implies that individuals to the immediate left and

right of the cut-off are similar therefore, differences in outcomes can be directly attributed to the

policy.

Assumption Nothing else is happening: in the absence of the policy, we

would not observe a discontinuity in the outcomes around the cut off.

▪ there is nothing else going on around the same cut off that impacts our outcome of interest

would not hold if, for instance:▪ 21 year olds can start drinking however the moment they turn 21 they have

to enroll in a “drinking responsibly” type seminar▪ Grants: there is another policy that gives grants to firms with sales bigger

than $5,000.16

Outcome Profile Before and After the Intervention

outc

om

e

assignment variable

Baseline

assignment variable

Follow-up

17

outc

om

e

assignment variable

Baseline

assignment variable

Follow-up

Different shape

External Validity

How general are the results?

Counterfactual: individuals “marginally excluded from benefits”

just under 21

sales just under $5,000

get results for these neighborhoods only

Causal conclusions are limited to individuals, households, villages and firms at the cut-off

• The effect estimated is for individuals “marginally eligible for benefits”

• extrapolation beyond this point needs additional, often unwarranted, assumptions (or multiple cut-offs)

[Fuzzy designs exacerbate the problem]

18

Graphical Analysiso

utc

om

e

assignment variable

19

The “nuts and bolts” of implementing RDDs

Major advantages of RDD transparency

graphical, intuitive presentation

Major shortcomings requires many observations around cut-off

▪ (down-weight observations away from the cut-off)

Why? only near the cut-off can we assume that people find

themselves by chance to the left and to the right of the cut-off

think about firms with $1M sales vs. firms with $1,000

or compare a 16 vs a 25 year-old

20

Wrap Up

Can be used to design a prospective evaluation when randomization is not feasible The design applies to all means tested

programs Multiple cut-offs to enhance external

validity▪ Menu of subsidies targeting various types of

firmsCan be used to evaluate ex-post

interventions using discontinuities as “natural experiments”.

21

Thank you

Financial support from: Bank Netherlands Partnership Program (BNPP), Bovespa,

CVM, Gender Action Plan (GAP), Belgium & Luxemburg Poverty Reduction

Partnerships (BPRP/LPRP), Knowledge for Change Program (KCP), Russia Financial Literacy and Education Trust Fund (RTF), and the Trust Fund for Environmentally &

Socially Sustainable Development (TFESSD), is gratefully acknowledged.