Jennifer Bryce

Institute for International Programs

The Johns Hopkins University

FIRST GLOBAL SYMPOSIUM ON HEALTH SYSTEMS RESEARCH

Outline

1. Lessons from the evaluation of the ACCELERATING CHILD SURVIVAL AND

DEVELOPMENT (ACSD) Program

2. Lessons from prospective evaluations of

the CATALYTIC INITIATIVE TO SAVE ONE MIIILION

LIVES (CI) to date

3. Large-scale evaluations: A new business

model

ACSD, 2002-2005

11 countries in Africa

Support from CIDA and other partners

Implemented through UNICEF

Aim: To reduce mortality among children less than 5 years of age

Strategy: Accelerate coverage with three packages of high-impact interventions (EPI+, IMCI+, ANC+), with a special focus on community-based delivery

Mali

ChadNiger

Nigeria

Cameroon

Central AfricanRepublic

Congo - Democratic Republic

Congo

SenegalCape Verde

Gabon

Equatorial Guinea

Sao Tome &Principe

Gambia

Guinea Bissau

Guinea

Sierra Leone

Liberia

Burkina Faso

Ghana

TogoBenin

High Impact Package

EPI + Expansion

Accelerated Child Survival Accelerated Child Survival and and DevelopmentDevelopment

CIDA CIDA funded projectfunded project

Côte d’Ivoire

Mauritania

The retrospective independent

evaluation of ACSD

High-impact districts in

Benin, Ghana, Mali

Standard indicators

Existing DHS/MICS with

oversampling

National comparison areas

Documentation of program

implementation & contextual

factors

Stepwise design

Key: Bars represent districts in the following order: Builsa, Bawku East, Kasena-Nankana, Bolgatanga, Bawku West, Bongo

2001

2002

2003

2004

2005

2006

2007

Activities

Surveys

DHS

DHS ’98-’99IMCIEPI + ITN IPTp

Lmt’d ITNs Available

Facility & Community

Limited

Coverage

Partial

Coverage

Facility Community

Documentation of ACSD program:

implementation – Ghana example

Coverage for IMCI+ interventions

before and after ACSD, in HIDs Benin Ghana Mali

*

*

*

*

*

*

*

* Change was significant at p < 0.05.

No coverage gains, and some significant losses, in sick child care.

Exclusive breastfeeding increased in Ghana, declined in Mali.

Before ACSD

After ACSD

Key

Under-five mortality in the ACSD HIDs

19% (p=0.10) 20% (p=0.10)

Bryce J, Gilroy K, Jones G, Hazel E, Black RE, Victora CG. The Accelerated Child Survival and Development

program in west Africa: a retrospective evaluation. Lancet 2010; 375:572-82.

Under-five mortality in the ACSD HIDs

and national comparison areas Declines in U5M in ACSD focus districts,

but not greater than national comparison areas.

Bryce J, Gilroy K, Jones G, Hazel E, Black RE, Victora CG. The Accelerated Child Survival and Development

program in west Africa: a retrospective evaluation. Lancet 2010; 375:572-82.

Why did ACSD not accelerate mortality declines

relative to the remainder of each country?

1. Missed opportunities in prioritization of interventions No consistent gains in tx for pneumo, diarrhea and malaria

Undernutrition given low priority in program

2. Key policies not in place No community-based treatment for pneumonia

3. Essential commodities not continuously available ACTs recommended in policies, but drugs not available

Global stockouts of ITNs

4. Community component weak No remuneration for CHWs, few incentives

Weak supervision

Too many messages, many unrelated to impact

Contributors & acknowledgements

Contributors

Jennifer Bryce

Kate Gilroy

Elizabeth Hazel

Gareth Jones

Robert Black

Cesar Victora

Acknowledgements

Ministries of Health, National

Statistics Offices, UNICEF

country staff, Collaborators in

documentation

UNICEF regional and global

staff Genevieve Begkoyian,

Mark Young, Sam Bickel

Technical consultants Trevor

Croft, Macro International

UNICEF leadership for their

commitment to learning and

change

EVALUATING THE

CATALYTIC INITIATIVE

TO SAVE A MILLION LIVES

Part 2

Independent Evaluation of the

MNCH Rapid Scale-Up

Overall objective: Provide “proof of concept” that proven interventions can be scaled up rapidly to reduce newborn and child mortality.

Supported by: BMGF

Implementing partners: Governments and UNICEF, WHO, UNFPA

“Real-time”

Mortality Monitoring (RMM)

Overall objective: To monitor changes in under-five mortality in real-time.

Countries: Burkina Faso, Malawi Countries: Ethiopia, Ghana, Malawi, Mali, Mozambique, Niger

Two Linked Evaluations

The Catalytic Initiative to Save a Million Lives

Supported by: CIDA

Implementing partners: Governments and UNICEF

Design process

Assessment visits

Estimation of program impact

Design of RMM/Evaluation

Prepare

Documentation

of program implementation

Coverage

surveys (Consensus

indicators)

Measured (“real-time”

mortality

monitoring)

Quality of care assessments

in facility & community

Modeled

using LiST

Costs and Cost-effectiveness; Equity

Contextual Factors

CI evaluation designs at a glance

Designs include comparable measures and methods

All country designs do not include all components;

challenges of partner coordination and funding

Inputs Outputs Outcomes Impact Process

Challenges similar to those faced by

other new initiatives Results focus demands comparable outcome and impact

indicators, but strategies for delivery and implementation contexts

will differ widely.

Simultaneous implementation of multiple programs with

overlapping objectives.

Health systems strength a critical part of all program strategies

and must be assessed.

Flexible evaluation design required because evaluators do not

control pace or quality of implementation and strategy could/should

change over time.

Cooperation & collaboration needed to implement full evaluation

design

Lesson 1:

Intermediate results welcomed and used

by MOHs and partners

Community case management by trained HSAs in

Malawi, 2009

Correct tx

ARI

Correct tx

fever/

malaria

ORS Rational

use of AB

Correct

referral

Lesson 2:

Implementation takes time

In Mali, the MoH scheduled a “forum” to decide on CCM for childhood pneumonia & malaria.

July

2008 November

2010 November

2008

February

2009

March

2009

Original

date

(cancelled) Planned

(cancelled)

Planned

(cancelled)

Forum held;

agreed

“YES” on

CCM

Months 4 3 1 7 months + + =

Discussions

about how to

implement are

still under way

in a 3-year

CI project

Lesson 3: “Virgin” comparison areas do

not exist; new designs needed

Mozambique

Simultaneous implementation of

multiple programs

Separate, uncoordinated, inefficient

evaluations, if any

Contributors

In-country partners

Agbessi Amouzou, Abdullah

Baqui, Robert Black, Jennifer

Bryce, Kate Gilroy, Elizabeth

Hazel, Gareth Jones, Marjorie

Opuni, Jeremy Schiefen, Cesar

Victora

IIP-JHU

Burkina Faso: ISSP, INSP

Ghana: Noguchi Institute,

University of Ghana

Malawi: National Statistics

Office, Department of

Economics, University of

Malawi

Mali: CREDOS

LARGE-SCALE EVALUATIONS:

A NEW BUSINESS MODEL

Part 3

Large-scale evaluations:

A new business model

“Business as usual” What is needed

Retrospective Prospective

Summative Formative & summative

Divorced from routine

monitoring

Linked to and building upon

routine monitoring

Short time line Informs program design &

continues throughout

External evaluators Evaluators who are

independent but linked

Single, one-off report Regular feedback with

intermediate results

Further details at www.jhsph.edu/iip