are disasters any different? challenges and opportunities for post-disaster impact evaluation alison...

Are Disasters Any Different?

Challenges and Opportunities

for Post-Disaster Impact Evaluation

Alison Buttenheim, Princeton UniversityHoward White, 3ie

Rizwana Siddiqui, PIDEKatie Hsih, Princeton University

April 1, 2009Cairo

2

3ie post-disaster impact evaluation (PDIE) study

Motivation:

• Frequency and severity of natural disasters

• Quantity of assistance provided in post-disaster

settings

• Recent interest from humanitarian and development

sectors in more and better impact evaluation

• Opportunity to use Pakistan ERRA experience as

case study

3

3ie post-disaster impact evaluation (PDIE) study

Goals:

• Review existing approaches to PDIE

• Develop a framework for rigorous PDIE

• Apply framework to the 2005 Pakistan

earthquake case

• Identify a set of principles to guide PDIE

4

Disasters

Natural events:414 reported in 2007

(CRED criteria)

5

Disasters

Natural events:414 reported in 2007

(CRED criteria)

Human consequences:211 million affected

16,847 lives lostUSD 100+ billion damages

6

Disasters

Natural events:414 reported in 2007

(CRED criteria)

Human consequences:211 million affected

16,847 lives lostUSD 100+ billion damages

Institutional responses

7

Post-disaster relief and recovery efforts

• USD 5.9 billion (pledged) for 2005 Pakistan

earthquake

• USD 13.5 billion (pledged) for 2004 Indian

Ocean tsunami

• Actors: Diverse mix of governments,

funders, IFIs, aid agencies, humanitarian

agencies, int’l/local NGOs.

8

How does PD assistance get evaluated?

• Extensive process evaluations

• Multiple levels of analysis (project, agency,

sector, disaster)

• Some joint evaluations (e.g. TEC)

• Review of ALNAP database, etc. suggests

few examples of “rigorous” impact evaluation

9

Why so little focus on IE in PD settings?

10

Why so little focus on IE in PD settings?

“Disasters are

different”

“Disasters are

different”

“Disasters are

different”

“Disasters are

different”“Disasters

are different”

“Disasters are

different”

11

Are disasters any different?

1. Unpredictable, rapid-onset event

2. Proven life-saving measures cannot be

randomized or withheld

3. Mismatch between resources and need

(sometimes)

4. Absence of baseline data (usually)

5. Which counterfactual is the right one?

12

Are disasters any different? Maybe not…

1. Nonrandom exposure to disaster event and

consequences

2. Nonrandom assignment of interventions

3. Fragile states/vulnerable populations

4. Multiple concurrent interventions

5. Which counterfactual is the right one?

13

• Bangladesh floods, 1998

• Hurricane Mitch, 1998

• Indian Ocean tsunami, 2004

• Hurricane Katrina, 2005

Lessons learned from other PDIE experiences

14

Disaster-related time periods

14

Pre-disaster Immediate post-disaster

Post-intervention (1)

Post-intervention (2)

Emergency

Relief

Recovery/Reconstruction

t-1 t0 t1 t2D

ISA

ST

ER

15

Disaster-related populations

15

A Disaster-affected households* that

receive assistance or interventions

B Disaster-affected households that do

not receive assistance or

interventions‡

C Non-affected households† that were

similar to A before the disaster

* or communities (or other unit of analysis)‡ or receive them later, or receive different ones† or less-affected households/communities

16

Time Description Disaster-affected households, treated

t0-t-1

t1-t-1

Disaster-related losses

Restoration to baseline

A0-A-1

A1-A-1

t1-t0

t2-t-1

t2-t0

t2-t1

Recovery from disaster

Sustained restoration to baseline

Sustained recovery from disaster

Persistence of recovery

A1-A0

A2-A-1

A2-A0

A2-A1

Within treatment group, single-difference over time

17

Time Description Disaster-affected households, treated

t0-t-1

t1-t-1

Disaster-related losses

Restoration to baseline

A0-A-1

A1-A-1

t1-t0

t2-t-1

t2-t0

t2-t1

Recovery from disaster

Sustained restoration to baseline

Sustained recovery from disaster

Persistence of recovery

A1-A0

A2-A-1

A2-A0

A2-A1

Within treatment group, single-difference over time

ERRA: “Build Back Better”

18

Time Description Disaster-affected households, treated

t0-t-1

t1-t-1

Disaster-related losses

Restoration to baseline

A0-A-1

A1-A-1

t1-t0

t2-t-1

t2-t0

t2-t1

Recovery from disaster

Sustained restoration to baseline

Sustained recovery from disaster

Persistence of recovery

A1-A0

A2-A-1

A2-A0

A2-A1

Within treatment group, single-difference over time

Problems: Recall bias if no baseline; attribution?

19

Time Description Affected treated-Non-affected

t-1 Baseline (pre-disaster) A-1-C-1

t0 Emergency (immediate post-disaster) A0-C0

t1 Relief/Reconstruction (post-intervention #1) A1-C1

t2 Recovery (post-intervention #2) A2-C2

Cross-sectional, single-difference over treatment groups (A vs. C)

20

Time Description Affected treated-Non-affected

t-1 Baseline (pre-disaster) A-1-C-1

t0 Emergency (immediate post-disaster) A0-C0

t1 Relief/Reconstruction (post-intervention #1) A1-C1

t2 Recovery (post-intervention #2) A2-C2

Cross-sectional, single-difference over treatment groups (A vs. C)

21

Time Description Affected treated-Non-affected

t-1 Baseline (pre-disaster) A-1-C-1

t0 Emergency (immediate post-disaster) A0-C0

t1 Relief/Reconstruction (post-intervention #1) A1-C1

t2 Recovery (post-intervention #2) A2-C2

Cross-sectional, single-difference over treatment groups (A vs. C)

Implied counterfactual: What would “A” households look like if there had been no disaster?

22

Time Description Affected treated-Non-affected

t-1 Baseline (pre-disaster) A-1-C-1

t0 Emergency (immediate post-disaster) A0-C0

t1 Relief/Reconstruction (post-intervention #1) A1-C1

t2 Recovery (post-intervention #2) A2-C2

Cross-sectional, single-difference over treatment groups (A vs. C)

Problems: Is there an appropriate “C” group? If so, were they observed? Attribution?

23

Time Description Affected — Unaffected

t0-t-1

t1-t-1

Disaster-related losses

Restoration to baseline

(A0-A-1)

(A1-A-1)

— (C0-C-1)

— (C1-C-1)

t1-t0

t2-t-1

t2-t0

t2-t1

Recovery from disaster

Sustained restoration to baseline

Sustained recovery from disaster

Persistence of recovery

(A1-A0)

(A2-A-1)

(A2-A0)

(A2-A1)

— (C1-C0)

— (C2-C-1)

— (C2-C0)

— (C2-C1)

Difference-in-difference (A vs. C)

24

Time Description Affected — Unaffected

t0-t-1

t1-t-1

Disaster-related losses

Restoration to baseline

(A0-A-1)

(A1-A-1)

— (C0-C-1)

— (C1-C-1)

t1-t0

t2-t-1

t2-t0

t2-t1

Recovery from disaster

Sustained restoration to baseline

Sustained recovery from disaster

Persistence of recovery

(A1-A0)

(A2-A-1)

(A2-A0)

(A2-A1)

— (C1-C0)

— (C2-C-1)

— (C2-C0)

— (C2-C1)

Difference-in-difference (A vs. C)

Controls time-variant factors that are the same between A & C

25

Time Description Affected treated-Affected control

t-1 Baseline (pre-disaster) A-1-B-1

t0 Emergency (immediate post-disaster) A0-B0

t1 Relief/Reconstruction (post-intervention #1) A1-B1

t2 Recovery (post-intervention #2) A2-B2

Cross-sectional, single-difference over treatment groups (A vs. B)

26

Time Description Affected treated-Affected control

t-1 Baseline (pre-disaster) A-1-B-1

t0 Emergency (immediate post-disaster) A0-B0

t1 Relief/Reconstruction (post-intervention #1) A1-B1

t2 Recovery (post-intervention #2) A2-B2

Cross-sectional, single-difference over treatment groups (A vs. B)

27

Time Description Affected treated-Affected control

t-1 Baseline (pre-disaster) A-1-B-1

t0 Emergency (immediate post-disaster) A0-B0

t1 Relief/Reconstruction (post-intervention #1) A1-B1

t2 Recovery (post-intervention #2) A2-B2

Cross-sectional, single-difference over treatment groups (A vs. B)

Implied counterfactual: What would “A” households look like if there had been no intervention?

28

Time Description Affected treated-Affected control

t-1 Baseline (pre-disaster) A-1-B-1

t0 Emergency (immediate post-disaster) A0-B0

t1 Relief/Reconstruction (post-intervention #1) A1-B1

t2 Recovery (post-intervention #2) A2-B2

Cross-sectional, single-difference over treatment groups (A vs. B)

Problems: How were interventions assigned to A but not to B?

29

Time Description Disaster-affected households

“Treated” — “Control”

t0-t-1

t1-t-1

Disaster-related losses

Restoration to baseline

(A0-A-1)

(A1-A-1)

— (B0-B-1)

— (B1-B-1)

t1-t0

t2-t-1

t2-t0

t2-t1

Recovery from disaster

Sustained restoration to baseline

Sustained recovery from disaster

Persistence of recovery

(A1-A0)

(A2-A-1)

(A2-A0)

(A2-A1)

— (B1-B0)

— (B2-B-1)

— (B2-B0)

— (B2-B1)

Difference-in-difference (A vs. B)

30

Time Description Disaster-affected households

“Treated” — “Control”

t0-t-1

t1-t-1

Disaster-related losses

Restoration to baseline

(A0-A-1)

(A1-A-1)

— (B0-B-1)

— (B1-B-1)

t1-t0

t2-t-1

t2-t0

t2-t1

Recovery from disaster

Sustained restoration to baseline

Sustained recovery from disaster

Persistence of recovery

(A1-A0)

(A2-A-1)

(A2-A0)

(A2-A1)

— (B1-B0)

— (B2-B-1)

— (B2-B0)

— (B2-B1)

Difference-in-difference (A vs. B)

Controls time-variant factors that are the same between A & B

31

World Bank impact evaluation of housing and livelihood grants

32

• Instrumental variable approach to disaster impact:

– Villages at same distance from epicenter, at same elevation and slope had comparable pre-disaster SES

– Villages at different distance from fault line experienced different earthquake severity.

World Bank impact evaluation of housing and livelihood grants

33

• Instrumental variable approach to disaster impact:

– Villages at same distance from epicenter, at same elevation and slope had comparable pre-disaster SES

– Villages at different distance from fault line experienced different earthquake severity.

World Bank impact evaluation of housing and livelihood grants

A1-C1

34

• Instrumental variable approach to disaster impact:

– Villages at same distance from epicenter, at same elevation and slope had comparable pre-disaster SES

– Villages at different distance from fault line experienced different earthquake severity.

• Variation in receipt of relief and recovery funds:

– Between-district variation in implementing agency for housing grant

– Threshold eligibility for livelihoods grant of 5 dependents/households: regression continuity design.

World Bank impact evaluation of housing and livelihood grants

A1-C1

35

• Instrumental variable approach to disaster impact:

– Villages at same distance from epicenter, at same elevation and slope had comparable pre-disaster SES

– Villages at different distance from fault line experienced different earthquake severity.

• Variation in receipt of relief and recovery funds:

– Between-district variation in implementing agency for housing grant

– Threshold eligibility for livelihoods grant of 5 dependents/households: regression continuity design.

World Bank impact evaluation of housing and livelihood grants

A1-C1

A1-B1

36

ERRA impact evaluation case study

1. Evaluation opportunities using existing data & HH sample

– Household data collection at t2

– Retrospective household reports of t0

– Use of ongoing government household surveys (e.g., HIES)

as baseline

– Randomization of some interventions from 2009

37

ERRA impact evaluation case study

2. Evaluation opportunities in a future disaster– Maintain surveillance sample in disaster-prone regions

– Household-level data collection at t0

– Randomized interventions, e.g,

• Timing of interventions:

– Group 1: Housing grant first, followed by livelihood cash grant

– Group 2: Livelihood cash grant first, followed by housing grant

• Conditionality of grants

• Types of interventions, e.g, different formats or recipients of livelihoods cash

grant

38

PDIE Guiding Principles

1. PDIE is necessary to ensure that relief and recovery

funds are appropriately targeted, effective, and efficient.

2. Each phase of a disaster (emergency, relief,

recovery/reconstruction) presents distinct evaluation

challenges and therefore may require a different

evaluation approach or methodology.

3. “Evaluation preparedness” is an important part of

disaster preparedness.

39

PDIE Guiding Principles

5. PDIE should incorporate evaluation of (pre-disaster)

investments in disaster mitigation, prevention, and

resilience.

6. Rigorous PDIE requires the tools and perspectives of

multiple disciplines and sectors.

7. Quantitative PDIE can benefit from the qualitative and

mixed-methods approaches.

40

PDIE Guiding Principles

7. Proportionate changes in outcomes over time and

over groups can be as instructive as changes in

levels.

8. Change-over-time impact evaluations should

recognize two distinct baselines: pre-disaster, and

immediately post-disaster.

41

PDIE Guiding Principles (ct’d)

9. PDIE will be most successful when the goals of the

intervention are clearly defined through a logical

framework or similar model; when the interventions

are appropriately targeted, and when the purpose/use

of the evaluation is clear.

10. Experimental and quasi-experimental approaches are

feasible in PDIE if ethical, logistical and “fit” issues are

adequately addressed.