Cost Effectiveness Analysis and Evidence to Policy

John Floretta J-PAL South Asia at IFMR

Monitoring & Evaluation Training Course for the Indian Economic Service

August 08, 2013

Lecture Overview

Cost Effective Analysis

From Impacts to Policy Decisions

Scale Ups

COST EFFECTIVE ANALYSIS

Most Development Problems (e.g. Attendance) can Have Multiple Solutions

1. Books for Free

2. Building Improvements

3. Cash Grants for Attendance

4. Community Participation

5. Deworming

6. Lunch for Free

7. Remedial Education

8. Information Campaigns

9. School Supplies (Blackboards, Chalk, etc.)

10. Teacher Attendance (Monitoring)

11. Free Uniforms

12. Merit Scholarships

Children in Udaipur, India wait for their absent teacher

• Low Enrollment > High Dropout > Low Attendance > Low Achievement

Randomized Evaluations Provide You Estimates of Impact

CCT for Attendance (Mexico): 1.3 percentage points higher in treatment

Deworming (Kenya): 7.5 pp

Information Campaigns (Madagascar): 3.5 pp

Free Uniforms (Kenya): 6.4 pp

Merit Scholarships (Kenya): 3.2 pp

• But is this enough to compare programs and choose the right one?

Comparing and Choosing Programs

0.075

0.064

0.035 0.032

0.013

0

0.01

0.02

0.03

0.04

0.05

0.06

0.07

0.08

Year of Participation Gained per Child

8

9

7

11

13

0

2

4

6

8

10

12

14

Years of Participation Gained Per $100

Cost Effective Analysis as a starting point for discussions on evidence based policy

Cost-effectiveness Analysis (CEA) and Comparative CEA

Summarizes complex program as simple ratio of costs to effects

• Measure the cost for a given level of effectiveness: e.g. cost to increase school attendance by 1 year

• Or, measure the level of effectiveness for a given cost: years of additional attendance induced by spending $100

Comparative cost-effectiveness then compares this cost-effectiveness ratio for multiple programs

• Must compute costs and benefits using similar methodology for all programs

• If done correctly, it helps policymakers choose between multiple programs addressing the same problem – e.g. what are the most cost-effective ways to increase school attendance?

Cost-Effectiveness (CEA) vs. Cost-Benefit Analysis (CBA)

CBA compares monetary value of benefits against costs:

• As a result CBA can show ratio of all benefits to costs of a program

• CBA can also deliver absolute judgment on whether program is worth the investment, CEA cannot

But this means that CBA requires a number of assumptions about the monetary value of all the different benefits

• This requires value judgments (cost of life, disability, lower crime among school kids)

Cost-Effectiveness (CEA) vs. Cost-Benefit Analysis (CBA)

Advantage of CEA is its simplicity:

• Allows user to choose an objective outcome measure (e.g. cost to induce an additional day of schooling) – no need for making judgments on monetary value of that schooling

• Easier for policymakers to compare programs when they are primarily concerned about one outcome of interest (e.g. increasing school attendance, not child health)

FROM EVIDENCE TO POLICY Using Evidence of Impact to Inform Policy

“Evidenced-based policymaking”

Incorporating evaluation evidence into new policies and programs

Why doesn’t evidence get integrated into new program?

Demand-side constraints: Systemic

Political

Administrative

Technical

Time

But also the “4 Is”:

Ignorance

Inertia

Ideology

Instincts

INCORPORATING EVIDENCE

Finding, synthesizing high-quality, policy relevant evidence

Where to go find good evidence?

• J-PAL

• BREAD

• NBER

Where to go find synthesis of evidence?

• J-PAL

• 3ie

• Health: Campbell, Cochrane

• And us!

J-PAL evaluation summaries and syntheses

J-PAL policy lessons: Cost-effectiveness and lit reviews

J-PAL APPROACH TO SCALING - UP

Evidence to Policy: J-PAL’s Path to Scaling - Up

Policy Relevant

Evaluations Replications Policy Pilots Scale-ups

Haryana Education Urban sanitation Double Fortified Salt

Early Grade Reading Targeting Ultra Poor

Learning camps Immunization incentives Satara deworming

Common Failures in Evaluations for Scale Up

Failure to document the entire set of activities, processes and intangibles that produced the result Unable to reproduce the results at scale

Failure to accurately identify and assess external factors that affected impact

Failure to identify what organizational culture, capabilities and characteristics are necessary to implement the model with the impact achieved

Failure to anticipate total and unit cost budget constraints at scale and assess the cost effectiveness activities and components

Failure to identify what organizations will adopt, fund, implement etc. and generate evidence that meets the needs of decision makers in those organizations

© Richard Kohl – Center for Large Scale Social Change LLC

There are Different Paths from Impact Evaluations to Scale-Ups – Here is One That J-PAL Follows

There are Different Paths from Impact Evaluations to Scale-Ups

Governments evaluate their pilot programs to demonstrate usefulness to

public, gather support for their expansion and learn lessons to make it more

effective (e.g. Progresa)

Leveraging Evidence by implementing organization to expand existing

programs and get more funding (e.g. Pratham)

Independent organizations can use evidence to replicate or scale-up programs

found to be highly cost-effective, and/or simple to implement (e.g. Deworm

the World)

If an evaluation helps provide evidence on a very policy relevant and salient

topic, it gets a huge amount of traction very easily (e.g. Pricing)

Careful study of the new context, collaboration with original evaluator and

implementer and a pilot replication (e.g. TCAI and immunization in Haryana)

Interventions on the Scale-up Menu

How Did They Get on the Menu?

Agriculture Education

Health

Political Economy & Governance

Labour Markets

Environment & Energy

Finance & Microfinance

Building policymaker interest and knowledge

Sharing evaluation lessons with policymakers

Technical assistance to implement the model

Understand key elements of the model

Adapt according to context

Means: Technical guidebooks, training, deputize staff, field visits

Who does this: Implementing partners, J-PAL consultants

Evidence to Scale example: School-based Deworming

THE PROBLEM 600 million school-aged children infected with parasitic worms. As a result, children are often either too sick or too tired to attend school or concentrate in class. .

Policy Outreach J-PAL SA organised the Bihar Development Conference, 2010 in collaboration with the Govt of Bihar, India. It was attended by the Deputy Chief Minister and senior bureaucrats of Bihar. Evidence on deworming was shared with them, and they decided to replicate deworming among all school-age children of Bihar.

Research Research by J-PAL affiliates Michael Kremer and Edward Miguel in Kenya shows that school-based deworming is one of the most cost-effective methods of improving school participation.

Scale-ups First round of deworming has been completed in Bihar, Andhra Pradesh, Delhi and Rajasthan.

Timeline 1998 2010 2011

Results Student worm

infections fell by 50% Absenteeism fell by

25% Anemia fell. Gains in height. Students reported

feeling sick less often. Infection rates also fell

in nearby (non-treatment) schools.

Wage earnings higher by 20% in long run

From Evidence to Scale – School-based Deworming

Results of Kremer & Miguel’s 1998 study reinforced in Delhi pre-school deworming program evaluation in 2001/2002

Deworm the World (DtW) was launched in 2007. • A partnership of Innovations for Poverty Action (IPA) and the Partnership

for Child Development, and sister organization to J-PAL.

The World Food Program (WFP) decided to extend school-based deworming in all of its school feeding programs in locations where parasitic worms are prevalent in 2009.

Government of Kenya announced a commitment to deworm 3 million children

in the most at-risk areas of the country in 2009.

Government of Andhra Pradesh pilots school-based worming in 2009, reaching

2 million children across 21,000 schools.

Kohl Scale-Up Framework: 7 Elements to Scaling-up

1. Model: Intervention + Implementation / Process

2. Getting the Model Ready: Evidence, Simplification, Documentation, Cost

3. Strategy: Where, What, How and Who

4. Advocacy: Convincing organizations to sanction, fund, manage and implement

5. Strengthening/Building Systems and Organizations: Creating the capacity, capabilities and enabling environment

6. Implementation at Scale: Monitoring and Feedback for Quality, Fidelity and Adaptation

7. Sustainability: Political, Financial and Institutional

© Richard Kohl – Center for Large Scale Social Change LLC

Key Questions for a Scaling-up Strategy

Should the model be scaled up?

What is being scaled up?

Where is the scaling up to occur (What are the goals of scaling up)?

How is scaling up to occur (method)?

Who is doing each of the three steps and ten tasks of scaling up?

© Richard Kohl – Center for Large Scale Social Change LLC

A Three Step, Ten Task Scaling Up Management Framework

Step 1: Preparing the Model, Setting Goals and Planning • Task 1: Identifying the Model

• Task 2: Setting Goals

• Task 3: Assessing Scalability & Filling In Gaps

• Task 4: Creating a Scaling Up Strategy

• Task 5: Reconciling Model, Vision, Method and Capability Building

Step 2: Creating the Preconditions • Task 6: Advocacy for the Issue, Sanction, Management, Funding and

Implementation

• Task 7: Modifying and Strengthening Organizations and Systems

• Task 8: Creating Coordination Mechanisms

© Richard Kohl – Center for Large Scale Social Change LLC

Step 3: Implementation At Scale

• Task 9: Monitoring for Quality, Fidelity, & Flexibility

• Task 10: Ensuring Political, Financial and Institutional Sustainability

© Richard Kohl – Center for Large Scale Social Change LLC

A Three Step, Ten Task Scaling Up Management Framework

Conclusion / What we have learned

CEA is a useful first step in comparing alternate programs that are aimed at the same outcome

The journey from impact evaluation to scale-ups is neither automatic, nor easy. Definitely not a linear process. Luck and timing matter more than a little.

Define clear goals for scale-up –where, who, what impact?

Scale-up partner needs more than just technical capabilities

Need to contextualize innovation (tension w/ standardization)

Evidence-based policymaking and you!

Using evidence to when inputting on design of programs

Advocating for pilot—rigorous test—learn—scale approach

Advise on running technical evaluations

Assess quality of evaluations

Implement rigorous monitoring systems to inform policy decisions today

Help lead the generational shift to results based management