Jennifer Bryce
Institute for International Programs
The Johns Hopkins University
FIRST GLOBAL SYMPOSIUM ON HEALTH SYSTEMS RESEARCH
Outline
1. Lessons from the evaluation of the ACCELERATING CHILD SURVIVAL AND
DEVELOPMENT (ACSD) Program
2. Lessons from prospective evaluations of
the CATALYTIC INITIATIVE TO SAVE ONE MIIILION
LIVES (CI) to date
3. Large-scale evaluations: A new business
model
ACSD, 2002-2005
11 countries in Africa
Support from CIDA and other partners
Implemented through UNICEF
Aim: To reduce mortality among children less than 5 years of age
Strategy: Accelerate coverage with three packages of high-impact interventions (EPI+, IMCI+, ANC+), with a special focus on community-based delivery
Mali
ChadNiger
Nigeria
Cameroon
Central AfricanRepublic
Congo - Democratic Republic
Congo
SenegalCape Verde
Gabon
Equatorial Guinea
Sao Tome &Principe
Gambia
Guinea Bissau
Guinea
Sierra Leone
Liberia
Burkina Faso
Ghana
TogoBenin
High Impact Package
EPI + Expansion
Accelerated Child Survival Accelerated Child Survival and and DevelopmentDevelopment
CIDA CIDA funded projectfunded project
Côte d’Ivoire
Mauritania
The retrospective independent
evaluation of ACSD
High-impact districts in
Benin, Ghana, Mali
Standard indicators
Existing DHS/MICS with
oversampling
National comparison areas
Documentation of program
implementation & contextual
factors
Stepwise design
Key: Bars represent districts in the following order: Builsa, Bawku East, Kasena-Nankana, Bolgatanga, Bawku West, Bongo
2001
2002
2003
2004
2005
2006
2007
Activities
Surveys
DHS
DHS ’98-’99IMCIEPI + ITN IPTp
Lmt’d ITNs Available
Facility & Community
Limited
Coverage
Partial
Coverage
Facility Community
Documentation of ACSD program:
implementation – Ghana example
Coverage for IMCI+ interventions
before and after ACSD, in HIDs Benin Ghana Mali
*
*
*
*
*
*
*
* Change was significant at p < 0.05.
No coverage gains, and some significant losses, in sick child care.
Exclusive breastfeeding increased in Ghana, declined in Mali.
Before ACSD
After ACSD
Key
Under-five mortality in the ACSD HIDs
19% (p=0.10) 20% (p=0.10)
Bryce J, Gilroy K, Jones G, Hazel E, Black RE, Victora CG. The Accelerated Child Survival and Development
program in west Africa: a retrospective evaluation. Lancet 2010; 375:572-82.
Under-five mortality in the ACSD HIDs
and national comparison areas Declines in U5M in ACSD focus districts,
but not greater than national comparison areas.
Bryce J, Gilroy K, Jones G, Hazel E, Black RE, Victora CG. The Accelerated Child Survival and Development
program in west Africa: a retrospective evaluation. Lancet 2010; 375:572-82.
Why did ACSD not accelerate mortality declines
relative to the remainder of each country?
1. Missed opportunities in prioritization of interventions No consistent gains in tx for pneumo, diarrhea and malaria
Undernutrition given low priority in program
2. Key policies not in place No community-based treatment for pneumonia
3. Essential commodities not continuously available ACTs recommended in policies, but drugs not available
Global stockouts of ITNs
4. Community component weak No remuneration for CHWs, few incentives
Weak supervision
Too many messages, many unrelated to impact
Contributors & acknowledgements
Contributors
Jennifer Bryce
Kate Gilroy
Elizabeth Hazel
Gareth Jones
Robert Black
Cesar Victora
Acknowledgements
Ministries of Health, National
Statistics Offices, UNICEF
country staff, Collaborators in
documentation
UNICEF regional and global
staff Genevieve Begkoyian,
Mark Young, Sam Bickel
Technical consultants Trevor
Croft, Macro International
UNICEF leadership for their
commitment to learning and
change
EVALUATING THE
CATALYTIC INITIATIVE
TO SAVE A MILLION LIVES
Part 2
Independent Evaluation of the
MNCH Rapid Scale-Up
Overall objective: Provide “proof of concept” that proven interventions can be scaled up rapidly to reduce newborn and child mortality.
Supported by: BMGF
Implementing partners: Governments and UNICEF, WHO, UNFPA
“Real-time”
Mortality Monitoring (RMM)
Overall objective: To monitor changes in under-five mortality in real-time.
Countries: Burkina Faso, Malawi Countries: Ethiopia, Ghana, Malawi, Mali, Mozambique, Niger
Two Linked Evaluations
The Catalytic Initiative to Save a Million Lives
Supported by: CIDA
Implementing partners: Governments and UNICEF
Design process
Assessment visits
Estimation of program impact
Design of RMM/Evaluation
Prepare
Documentation
of program implementation
Coverage
surveys (Consensus
indicators)
Measured (“real-time”
mortality
monitoring)
Quality of care assessments
in facility & community
Modeled
using LiST
Costs and Cost-effectiveness; Equity
Contextual Factors
CI evaluation designs at a glance
Designs include comparable measures and methods
All country designs do not include all components;
challenges of partner coordination and funding
Inputs Outputs Outcomes Impact Process
Challenges similar to those faced by
other new initiatives Results focus demands comparable outcome and impact
indicators, but strategies for delivery and implementation contexts
will differ widely.
Simultaneous implementation of multiple programs with
overlapping objectives.
Health systems strength a critical part of all program strategies
and must be assessed.
Flexible evaluation design required because evaluators do not
control pace or quality of implementation and strategy could/should
change over time.
Cooperation & collaboration needed to implement full evaluation
design
Lesson 1:
Intermediate results welcomed and used
by MOHs and partners
Community case management by trained HSAs in
Malawi, 2009
Correct tx
ARI
Correct tx
fever/
malaria
ORS Rational
use of AB
Correct
referral
Lesson 2:
Implementation takes time
In Mali, the MoH scheduled a “forum” to decide on CCM for childhood pneumonia & malaria.
July
2008 November
2010 November
2008
February
2009
March
2009
Original
date
(cancelled) Planned
(cancelled)
Planned
(cancelled)
Forum held;
agreed
“YES” on
CCM
Months 4 3 1 7 months + + =
Discussions
about how to
implement are
still under way
in a 3-year
CI project
Lesson 3: “Virgin” comparison areas do
not exist; new designs needed
Mozambique
Simultaneous implementation of
multiple programs
Separate, uncoordinated, inefficient
evaluations, if any
Contributors
In-country partners
Agbessi Amouzou, Abdullah
Baqui, Robert Black, Jennifer
Bryce, Kate Gilroy, Elizabeth
Hazel, Gareth Jones, Marjorie
Opuni, Jeremy Schiefen, Cesar
Victora
IIP-JHU
Burkina Faso: ISSP, INSP
Ghana: Noguchi Institute,
University of Ghana
Malawi: National Statistics
Office, Department of
Economics, University of
Malawi
Mali: CREDOS
LARGE-SCALE EVALUATIONS:
A NEW BUSINESS MODEL
Part 3
Large-scale evaluations:
A new business model
“Business as usual” What is needed
Retrospective Prospective
Summative Formative & summative
Divorced from routine
monitoring
Linked to and building upon
routine monitoring
Short time line Informs program design &
continues throughout
External evaluators Evaluators who are
independent but linked
Single, one-off report Regular feedback with
intermediate results
Further details at www.jhsph.edu/iip