using evidence to inform policy: impact evaluation of human development programs overseas...

USING EVIDENCE TO INFORM POLICY:IMPACT EVALUATION OF HUMAN DEVELOPMENT PROGRAMS

Overseas Development Institute

May 4, 2010

ARIEL FISZBEINCHIEF ECONOMIST, HUMAN DEVELOPMENTWORLD BANK

Motivation

‘Since it is difficult to distinguish the good from the bad prophet, we must be suspicious of all prophets: it is better to avoid revealed truths, even if we feel exalted by their simplicity and splendor, even if we find them comfortable because they come at no cost. It is better to be content with more modest and less inspiring truths that are laboriously conquered, step by step, with no shortcuts, by studying, discussion and reasoning, and that can be verified and demonstrated'’ Primo Levi

2

How can we learn from experience?

Traditional approach: identify best practice cases and inductively derive conclusions.

But the strength of those conclusions is directly related to the internal and external validity of the underlying evaluation.

We could do better if we plan well and take a more systematic approach to learning from experience.

3

The evaluation problem

Impact: difference in outcome with and without program. Cannot observe simultaneously. with and without comparisons deceptive if

participation depends on individual attributes that influence outcomes

before/after comparisons can be misleading if other things happened during the period

Need to estimate counterfactual (control or comparison) Treated & control have same characteristics on

average, Only reason for difference in outcomes is the

intervention

4

Multiple design options

Experimental (often using phase in) Quasi-experiments/non-experimental

Regression Discontinuity (RD) Difference in difference – panel data Other (Instrumental Variables,

matching, etc) In all cases, these will involve

knowing the rule for assigning treatment

Advantages of prospective design

5

Evolution of IE in the World Bank6

...Completed before 2004

…Completed from 2004

…Ongoing in 2010

0

25

50

75

100

125

150

175

200

18

58

175

Number of Ongoing IEs at the World Bank…

Strong focus on HD areas7

Social Protec-tion20%

Health17%

Education13%ECD & Nutr.

7%

Other9%

Gov .4%

Urban Upgr.6%

Agric.6%

Fin. & Priv. Sec.8%

Local Dev.10%

The World Bank Impact Evaluation Program: Thematic Areas

HD thematic ar-eas: 58% of total World Bank IEs

Strategic themes8

Education, Other

Pay For Performance in Health

Malaria

HIV

Early Childhood Development

Active Labor Market Programs

School Accountability

Conditional Cash Transfers

0 5 10 15 20 25

6

9

9

13

14

16

19

20

The World Bank Impact Evaluation program on Human Development

Number of OngoingImpact Evaluations

Thematic areas

… and IEs of non-HD projects often measure impact on HD outcomes:

Agricultural Development Indicators: anthropometrics

Housing Indicators: morbidity, anthropometrics

Rural Electrification Indicators: study time, respiratory illnesses

Rural Roads Indicators: school attendance

Urban Upgrading Indicators: employment

Water Supply and Sanitation Indicators: morbidity, anthropometrics

9

Building a global evidence base on the impact of HD programs

10

Europe & Central Asia

Middle East & North Africa

East Asia & the Pacific

Latin America & Caribbean

South Asia

Africa

0 10 20 30 40 50 60

1

3

10

17

21

54

The World Bank Impact Evaluation program on Human Development

Number of OngoingImpact Evaluations

Regions

Priority thematic areas

1. Conditional Cash Transfers2. Paying for Performance in Health (P4P)3. School Accountability4. Malaria Control5. Active Labor Market Programs / Youth

Employment6. HIV/AIDS Prevention7. Early Childhood Development

11

12

CCTs have become very popular….

13

CCTs have become very popular….

First generation evaluation questions14

Impact on consumption and poverty Side effects?

Impacts on service utilization

Impacts on human development outcomes

Impacts on consumption and poverty

Mexico(1999)

Nicaragua(2002)

Colombia(2006)

Honduras(2002)

Average transfer (% of per capita consumption)

20% 30% 13% 11%

Impact on per capita consumption (%)

8.3%** 20.6%** 10%** 7%*

Impact: headcount index (% points)

1.3** 5.3** 2.9* --

Impact: poverty gap (% points)

3.0** 9.0** 7.0** 2.0*

Impact: sqd. poverty gap (% points)

3.4** 8.6** 2.2** 2.0*

15

CCT impacts on food consumption16

Impacts on school enrollment

Age range Baseline enrollment

Impact (% points)

Size of transfer

Colombia 8-1314-17

91.7%63.2%

2.1**5.6***

17%

Chile 6-15 60.7% 7.5*** 3-7%

Ecuador 6-17 75.2% 10.3*** 10%

Mexico Grade 0-5Grade 6

Grade 7-9

94.0%45.0%42.5%

1.98.7***

0.620%

Nicaragua 7-13 72.0% 12.8*** 30%

Cambodia Grade 7-9 65.0% 31.3*** 2%

Bangladesh 11-18 44.1% 12.0** 1%

Pakistan 10-14 29.0% 11.1*** 3%

17

Impacts on health service utilization

Age range

Baseline level

Impact (% points)

Size of transfer

Colombia <24 months24-48

months

n.a.

n.a.

22.8**

33.2***

17%

Chile 0-6 years 17.6% 2.4 7%

Ecuador 3-7 years n.a. 2.7 10%

Honduras 0-3 years 44.0% 20.2*** 9%

Mexico 0-3 years 69.8% 8.4 20%

Nicaragua 0-3 55.4% 13.1* 27%

18

Impacts on education and health outcomes

Only mixed success in terms of improving final outcomes in education and health:

Education: Increases in school enrollment and years of completed schooling have not come hand-in-hand with improved learning outcomes Health: Some programs, but my no means all, have improved child nutrition (as measured by height-for-age, hemoglobin status)

19

20

Second generation evaluation questions

Role of conditions: Testing CCTs vs. UCTs (Burkina Faso, Malawi, Morocco and Yemen)

Does it matter who receives the cash? Mothers vs. fathers (Burkina Faso, Morocco and Yemen); Girls vs. parents in Malawi

Design of payments: Cambodia; Colombia


1. Conditional Cash Transfers2. Paying For Performance in Health3. School Accountability4. Malaria Control5. Active Labor Market Programs / Youth


21

Results-based financing (RBF) ≈ Pay-for-performance (P4P)

Provision of payment for the

attainment of well-defined results

Transfer of money or material goods conditional on taking a measureable action or achieving a predetermined

performance target (CGD, 2009)

DonorCentral governmentLocal governmentPrivate insurer

$Service providersFacilities / NGOsCentral governmentLocal governments

RBF takes many forms…

Payers Payees

22

Impact evaluation of P4P in Health

Effects on non-contracted services provided?

Impact on equity of services?

Impact on out-of-pocket expenditures?

Impact Evaluations:

Will the introduction of pay for performance for patient services impact the quality and performance of hospitals/health centers?

23

24

P4P in Rwanda

National P4P scheme to supplement input-based budgets with bonus payments based on the quantity and quality of maternal and child healthcare (14 output indicators)

25

“Paying Primary Health Care Centers for Performance in Rwanda” (Basinga et al., 2010)

Impact on proportion of institutional deliveries

36.3

49.7

34.9

55.6

30.0

40.0

50.0

60.0

Baseline (2006) Follow up (2008)

Prop

ortio

n of o

f ins

tituti

onal

deliv

erie

s

Control facilities Treatment (PBF facilities)

7.3 % increasedue to PBF

26

“Paying Primary Health Care Centers for Performance in Rwanda” (Basinga et al., 2010)

Impact on quality of prenatal care

-0.10

0

-0.13

0.15

-0.15

-0.10

-0.05

0.00

0.05

0.10

0.15

0.20

Baseline (2006) Follow up (2008)

Stan

dard

ized

Pren

atal

effo

rt sc

ore

Control facilities Treatment (PBF facilities)

15 % Standard deviation increase due to PBF

27

P4P in Rwanda: Lessons

No effect on the number of prenatal care visits or on immunization rates. Need demand side incentives?

Greatest effect on services that had the highest payment rates and are under the provider control.

Financial performance incentives can improve both use of and quality of health services.

An equal amount of financial resources without the incentives would not have achieved the same gain in outcomes.

28

P4P experiment in UgandaLundberg, Marek and Pariyo (2007) Contracting for Primary Health Care in Uganda

Three arms: (1) performance bonus, (2) freedom to allocate base grant, (3) control.

Performance bonus based on meeting targets (pre-natal care, attended births, immunizations, etc.)

Results after 18 months: bonus facilities perform the same as

or worse than others; freedom-to-allocate facilities perform

significantly better than others.

29

P4P experiment in Uganda

This performance bonus didn’t work.Amounts not large enough? Scheme too complicated? Not enough time? (some evidence of

learning curve)

Facilities may allocate budgets more effectively than the Ministry of Health




30

Strategies for improving learning outcomes

Supply side policies (classrooms, textbooks, extra tutors, teacher training)

Demand side policies (CCTs?) Child endowments/readiness to

learn -- ECD, de-worming Accountability reforms: Change the

environment in which decisions are made

31

WDR 2004 framework of accountability

32

33

Rationale for Education Accountability Schemes

How do we transform this teacher….

…into this teacher?

School Accountability34

35

3 Modalities on School Accountability

1. School-based Management (SBM)

2. Information for accountability projects

3. Teacher contracting and pay for performance projects

36

How can SBM improve outcomes?

Those at the local level have more or better information and different objectivesKey decisions about school personnelKey decisions about spendingChanges in the educational processResource mobilizationChanges in the school climate

37

School Based Management in Mexico

(Apoyo a la Gestión Escolar, AGE)

Grants + training to Parent Associations (PA).

AGEs increase school autonomy through improved mechanisms for participation of directors, teachers and PA in the management of the schools.

Effects on: School failure, repetition, drop out

Use phasing in of program for evaluation

38

School Base Management in Mexico

Gertler P. et al, 2007. “Empowering Parents to Improve Education: Evidence from Rural Mexico”

39



40



41


1. School-based Management

2. Information for accountability (IFA)projects


How could IFA improve outcomes?42

Empower parents and communities vis a vis local & school authorities – Increased importance of education in the political

sphere– Increased attention to results valued by parents

Enable better technocratic decision-making• Help guide allocation decisions

Under “strong accountability” systems:– Threat of sanction will spur increased teaching

effort When there is school choice:

– “Market-pressure” for performance

Newspaper Campaign in Uganda

– In 1995 only 13 % of non-wage recurrent spending reached schools.

– From 1996-2001, amounts and dates of grant to local governments published in local and national newspapers.

43

Impact of information campaign

Source: Reinikka and Svensson (2001), Reinikka and Svensson (2003a)

IMPACT ON LEAKAGE:A school close to a newspaper outlet experienced 25 percentage points less leakage compared to a school one standard-deviation (30 km) further away from a newspaper outlet.

IMPACT ON LEARNING:A one standard deviation reduction (30 km) in distance lead to a 0.09 standard deviation increase in average test scores.

44

45

Information Campaigns in India

Village-level education committees (VECs), representing parents and village leaders, created by state education policy

Presumption that information campaigns are needed to empower parents/communities

Two sets of experiments PRATHAM --Banerjee et al. (2008) “Pitfalls of

participatory programs: Evidence from a Randomized Evaluation in Education in India”

Three states in India –WB + Nike Foundation --Pandey et al. (2008) Community Participation in Public Schools: Impact of Information Campaigns in three Indian States.

46

Impact of Information Campaign (Banerjee et al.)

Program: Advocacy and information about VECs =(1); (1)+teaching how “learning reports cards” are done=(2); (2)+ Teaching: Capacity Building to Directly Improve

Learning (reading classes)=( 3)

Information campaigns ((1) and (2)) had no effect on VEC activism or on learning outcomes

Trained teaching volunteers (3) did have a significant effect on “reading” learning outcomes

47

Impact of Information Campaign (Pandey et al.)

REGION Teacher Attendan

ce

Teacher

Activity

% Receiv

ed Unifor

m

Participation rate of parents in school inspections

Scholarships

Students received

Improvement in

Reading

UP 11 % Unchanged

18 % (girls)

.11 33 %(general caste)

27 %(Grade 3)

MP Unchanged 30% 14 % (general caste)

.02 26 %(SC/ SC)

14 %(Grade 3)

Karnataka

Unchanged Unchanged

N/A Unchanged N/A 15 %(Grade 4)

Pandey, Priyanka et al. Community Participation in Public Schools: Impact of Information Campaigns in three Indian States. South Asia Human Development. Presentation. World Bank. 2008.

47

48


1. School-based Management

2. Information for accountability projects


49

How could TI improve outcomes?

Accreditation and merit-based incentives lead to better teaching professionals joining the system.

Decentralized hiring and monitoring of teacher performance leads to higher attendance rates.

Career advancement incentives stimulate sustained effort, improving student performance.

Pay-for-performance incentivizes improved quality and quantity of teaching improving student test scores.

Teacher Pay Incentives in Andra Pradesh (Muralidharan and Sundararaman, 2008)

50

Three policies Group and individual pay incentives based on

average improvement in test scores of all students Block grants to schools (extra teachers, more inputs) Assessments conducted by an independent NGO

Results (two rounds) Incentives increase student performance by 0.22 SD 1st year both incentives work well. 2nd year individual

incentives work better Additional inputs improve scores only by 0.08 SD Teachers liked the program



Employment6. HIV/AIDS7. Early Childhood Development

51

52

Look for more information….

www.worldbank.org/hdchiefeconomist

..and remember Primo Levi

‘It is better to be content with more modest and less inspiring truths that are laboriously conquered, step by step, with no shortcuts, by studying, discussion and reasoning, and that can be verified and demonstrated'’

53

using evidence to inform policy: impact evaluation of human development programs overseas...

Documents

impact of hd programs

hd outcomes

hd areas

underlying evaluation

evaluation problemimpact

poverty gap

cct impacts

ies of nonhd projects