National Evaluation Platforms: A Solution that Serves

Governments and their Partners

Robert Black

Institute for International Programs Johns Hopkins Bloomberg School of

Public Health

FIRST GLOBAL SYMPOSIUM ON HEALTH SYSTEMS RESEARCH

Outline

1. Why a new approach is needed

2. National Evaluation Platforms (NEPs):

The basics

3. Country example: Malawi

4. Practicalities and costs

Most current evaluations of large-scale

programs aim to use designs like this

Impact

Coverage

Program

No impact

No coverage

No program

But reality is much more complex

General socioeconomic and other contextual factors

Impact

Coverage

Routine health

services Interventions in

other sectors

Other health

programs

Program

Other health

programs

New evaluation designs are needed

Large-scale programs

Evaluators do not control

timetable or strength of

implementation

Multiple simultaneous

programs with overlapping

interventions and aims

Contextual factors that cannot

be anticipated

Need for country capacity and

local evidence to guide

programming

Lancet, 2007

Bulletin of WHO, 2009

Sources: Victora CG, Bryce JB, Black RE. Learning from new initiatives in maternal and child health. Lancet 2007; 370 (9593): 1113-4.

Victora CG, Black RE, Bryce J. Evaluating child survival programs. Bull World Health Organ 2009; 87: 83.

NATIONAL EVALUATION

PLATFORMS: THE BASICS

Lancet, 2010

Source: Victora CG, Black RE, Boerma JT, Bryce J. Measuring impact in the MDG era and beyond: A new

approach to large-scale effectiveness evaluations. Lancet, published on line 9 July 2010.

Builds on a common evaluation

framework, adapted at country level

Common principles (with IHP+, Countdown, etc.)

Standard indicators

Broad acceptance

Evaluation databases

with districts as the units

District-level databases covering the entire country

Data for standard impact pathway:

Inputs (partners, programs, budget allocations, infrastructure)

Processes/outputs (DHMT plans, ongoing training,

supervision, campaigns, community participation, financing

schemes such as conditional cash transfers)

Outcomes (availability of commodities, quality of care

measures, human resources, coverage)

Impact (mortality, nutritional status)

Contextual factors (demographics, poverty, migration)

Permits national-level evaluations

of multiple simultaneous programs

Types of comparisons

supported by the platform approach

Areas with or without a given program

Traditional before-and-after analysis with a

comparison group

Dose response analyses

Regression analyses of outcome variables according

to dose of implementation

Stepped wedge analyses

In case program is implemented sequentially

Evaluation platform Interim (formative) data analyses

• Are programs being deployed where need is greatest?

– Correlate baseline characteristics (mortality, coverage, SES,

health systems strength, etc) with implementation strength

– Allows assessment of placement bias

• Is implementation strong enough to have an impact?

– Document implementation strength and run simulations for likely

impact (e.g., LiST)

• How best to increase coverage?

– Correlate implementation strength/approaches with achieved

coverage (measured in midline surveys)

• How can programs be improved?

– Disseminate preliminary findings with feedback to government

and partners

(All analyses at district level)

Evaluation platform Summative data analyses

Did programs increase coverage?

– Comparison of areas with and without each program over time

– Dose-response time-series analyses correlating strength of

program implementation to achieved coverage

Was coverage associated with impact?

– Dose-response time-series analyses of coverage and impact

indicators

– Simulation models (e.g. LiST) to corroborate results

Did programs have an impact on mortality and nutritional

status?

– Comparison of areas with and without each program over time

– Dose-response time-series analyses correlating strength of

program implementation with impact measures

COUNTRY EXAMPLE

MALAWI

Simultaneous

implementation of multiple

programs

Separate, uncoordinated,

inefficient evaluations (if

any)

Inability to compare

different programs due to

differences in

methodological

approaches and indicators

Malawi

Community case-management

(CCM) for childhood illness

Region and districts

CCM Partners

PMNCH MSH/

BASICS SAVE PSI UNICEF # CCM

partners NORTHERN REGION Chitipa 1 Karonga 1 Mzimba 1 Nkhata Bay 1 Rumphi 1 Likoma 1

CENTRAL REGION Dedza 1 Dowa 1 Kasungu 2 Lilongwe 1 Mchinji 1 Nkhotakota 2 Ntcheu 1 Ntchisi 1 Salima 1

SOUTHERN REGION Balaka 2 Blantyre 1 Chikwawa 1 Chiradzulu 1 Machinga 1 Mangochi 1 Mulanje 1 Mwanza 1 Nsanje 2 Phalombe 2 Thyolo 1 Zomba 2 Neno 1 Total districts

supported 10 8 6 5 4

Policy = rapid scale-up

of CCM

MOH identified ≥ 1

partner to support in

each district

“Comparison” districts

therefore not available

But implementation is

likely to be uneven,

allowing dose-response

analyses

Advantageous context for NEP

strong network of MNCH partners implementing

CCM

administrative structure decentralized to 28

districts

SWAp

district-level data bases (2006 MICS, 2010 DHS,

Malawi Socio-Economic Database (MASEDA))

DHS includes approx. 1,000 households in each

district

Analysis Plan

“Dose”

CCM implementation

strength (per 1,000 pop):

+ CHWs

+ CHWs trained in CCM

+ CHWs supervised

+ CHWs with essential

commodities available

Financial inputs

“Response”

Change in Tx rates for

childhood illnesses

Change in U5M

PRACTICALITIES

AND LIMITATIONS

Sample sizes must be calculated

on a country-by-country basis

Statistical power (likelihood of detecting an effect) will

depend on:

Number of districts in country (fixed; e.g. 28 in Malawi)

How strongly the program is implemented, and by how much

implementation affects coverage and mortality

How much implementation varies from district to district

Baseline coverage levels

Presence of other programs throughout the districts

How many households are included in surveys in each district

• May require oversampling

Main costs of the platform approach

Building and maintaining database with secondary information

already collected by others

Requires database manager and statistician/epidemiologist for

supervision

May require reanalysis of existing surveys, censuses, etc

Keeping track of implementation of different programs at

district level

Requires hiring local informants, training them and supervising their

work

Adding special assessments (costs, quality of care, etc)

May require substantial investments in facility or CHW surveys

Oversampling household surveys

May require substantial investments

But this will not be required in all countries

Evaluation platform

Advantages – Adapted to current reality of

multiple simultaneous

programs/interventions

– Identification of selection

biases

– Promotes country ownership

and donor coordination

– Evaluation as a continuous

process

– Flexible design allows for

changes in implementation

Limitations

– Observational design (but

no other alternative is

possible)

– High cost particularly due

to large size of surveys

• But may be less than

several standalone surveys

– Requires transparency and

collaboration by multiple

programs and agencies

Platform design overview

Design element Data sources (sample = 28 districts)

Documentation of program implementation and contextual factors

Full documentation every 6 months through systematic engagement of DHMTs

Quality of care survey at 1st-level health facilities

Existing 2009 data to be used for 18 districts; repeat survey in 2011

Quality of care at community level (HSAs)

Desirable to conduct in all 28 districts (Not included in this budget proposal)

Intervention coverage DHS 2010, with samples of 1,000 households representative at district level in all 28 districts

DHS/MICS 2014 with samples representative at district level in all 28 districts

Costs Costing exercises in ≈ 1/3 of districts distributed by region and chosen systematically to reflect differences in implementation strategy or health system context

Impact (under-five mortality and nutritional status)

End-line household survey (MICS or DHS?) in 2014

Modeled estimates of impact based on measured changes in coverage using LiST

Average baseline coverage level (% of children with

suspected pneumonia treated with antibiotics) 30%*

Standard deviation of baseline average coverage 13 pp*

Coefficient of variation of baseline average coverage 0.42*

Average endline coverage (assumed based on target set by

country) 67%

Standard deviation of endline coverage (assuming same

coefficient of variation as in baseline) 28 pp

Standard deviation of change (Y variable) 20 Pp

Assumed 10 percentage point increase in implementation

strength score leads to 7% change in coverage 1 = 0.7

Standard deviation of implementation strength score 20 pp

ASSUMPTIONS

* Based on the latest DHS results

Sample size calculations

(dose-response analysis)

26 districts required if 10 pp increase in

implementation leads to 5 pp coverage increase

Number Districts Slope

(N) (B) 191 0.2

82 0.3 44 0.4

26 0.5 17 0.6 11 0.7

7 0.8 5 0.9

Alpha = 5%; power = 80%

SD(y) = SD(x) = 20 pp

N vs B with SX=20.00 SY=20.00 Alpha=0.05 Power=1.00T-Test

N

B

0

50

100

150

200

0.2 0.4 0.6 0.8 1.0 1.2

Contextual Factors

Categories Indicators ENVIRONMENTAL, DEMOGRAPHIC AND SOCIOECONOMIC

Rainfall patterns Average annual rainfall; seasonal rain patterns

Altitude Height above sea level

Epidemics Qualitative

Humanitarian crises Qualitative

Socio-economic

factors

Women’s education & literacy; household assets;

ethnicity, religion and occupation of head of household

Demographic Population; population density; urbanization; total

fertility rate; family size

HEALTH SYSTEMS AND PROGRAMS

User fees Changes in user fees for IMCI drugs

Other MNCH Health

Programs

The presence of other programs or partners working in

MNCH

Practical arrangements

Platform should be led by national academic

institution (e.g. University or Statistical Office)

Supported by an external academic group if

necessary

Steering committee with MOH, Statistical Office,

international and bilateral organizations, NGOs, etc