USING PROCESS EVALUATION TO STRENGTHEN PROGRAM DELIVERY

GHPC May 17, 2016 – Jennifer Nielsen, PhD, Senior Nutrition Advisor

WHY PROCESS EVALUATION?

• Randomized controlled trials are considered the gold standard in medicine for evidence of impact of a treatment; increasingly used for effectiveness of broader public health interventions

• With a highly complex intervention there is need for more information on the dose and fidelity of delivery to add plausibility to the evidence (intervention - - - > impact)

• HKI’s Enhanced Homestead Food Production (EHFP) includes • Training to improve horticultural and husbandry practices • Behavior change communication to influence nutrition, health, hygiene practices • Multiple partnerships • Cascade training structures with three levels • Gender empowerment/transformation strategies

In short, many opportunities for losses in both dose and fidelity!

Rationale

BCC training on ENA for beneficiaries

Outcomes Impact Inputs Process Outputs

HKI, APRG and governmental

structures (Ministries of

Health, Agriculture,

Animals, Environment, and the Promotion of

Women, local authorities and officials) work

together

Training in plant and

animal production

techniques for master trainers

BCC training on ENA for community

level nutrition trainers

Develop a training strategy in animal; and

plant production techniques

BCC training on ENA for

master trainers

Develop a behavior change communication

(BCC) strategy to promote Essential Nutrition Actions

(ENA)

Training in plant and

animal production

techniques for Village Farm

Leaders (VFL)

Village Model Farms (VMF)

established

Training in plant and

animal production techniques

for beneficiaries

Establishment of individual

farms (40 women per

village)

Improved maternal and child health

and nutrition outcomes

Agriculture and zoological inputs

distributed

Improvements in small

ruminant and poultry

production

Improvements in fruit and vegetable

production

Improvements in household

consumption

Increased Income

Beneficiaries received and understood

BCC training on ENA

Improvements in nutrition and

feeding practices for children,

pregnant women and breastfeeding

mothers

Adoption of agriculture practices

Women’s empowerment

improved

Women’s assets increased

Increased availability of micronutrient rich fruits and

vegetables

Adoption of ENA

practices by beneficiaries

Improvements in care & hygiene

practices for children,

pregnant women and breastfeeding

mothers

Beneficiaries received and understood agriculture

training

Increased availability of

food from animal origin

Nutrition Pathway Production

Pathway Income Pathway

Program Theory - EHFP Burkina

AKA…..

5

• Program funded by USAID – OFDA (2009-2012)

• IFPRI research lead; HKI implementation lead

• First attempt to adapt EHFP model to Africa

• Implemented as RCT with longitudinal design following HH with children 3-12 months at baseline1

• Process evaluation conducted ~ midterm April-May 2011

1Olney et al (2015, 2016) J. Nutr.

PROCESS EVALUATION IN BURKINA

• Mixed methods – SSI both closed and open ended with

random sample of beneficiaries (n=145) and controls (n=75)

– KII with purposively selected master trainers in ag (n=13) and nutrition (n=24)

– KII with community nutrition volunteers: • older women leaders (n=30) • health committee members (n=28)

– KII with village farm leaders (n=58)

• Guided by PIP (supplies delivered; training attended; contents understood; knowledge gained; motivation and activities at all levels)

• Explored informants perceptions of program quality and areas for improvement

METHODOLOGY

• Enumerator training, data collection and analysis led by IFPRI with significant involvement of HKI implementing team

• Findings vetted and debated with implementation team

• Findings also shared with advisory committees at local and national levels

• Recommendations incorporated, as feasible, in subsequent work plan

ROLL OUT

• Water challenges: demand exceeding supply – Repaired and built boreholes – Foot pumps – Drip irrigation for VMF and HH – Watering cans and wheelbarrows – Relocated some VMF – More drought-resistant crops

• High loss of poultry, low production & consumption

– Mixed poultry breeds and more local procurement – Assured vaccine supply

• Low fruit production and consumption – Mangoes not yet bearing fruit (3-4 years from) – Papayas bear for 8 months but ~50% died

FINDINGS: PRODUCTION & INCOME PATHWAYS

• Leakage from Master Trainers to Volunteers to Beneficiaries

– Some topics in particular (rich complementary foods, feeding sick child, anemia control, women’s nutrition)

– Refresher training on key topics – Focus messages (egg consumption) – Training in facilitation skills – Supportive supervision

• Motivation of volunteers mixed – Public recognition – Small gifts

• Home visits less than expected

– Emphasize group discussions

FINDINGS: KNOWLEDGE - CONSUMPTION PATHWAY

• Lag times at multiple stages reduced exposure

– From award to beginning of implementation (6 months)

• Training of Master Trainers and delivery of inputs

• Planning of research – Between baseline and VMF production +

nutrition BCC (6 months) • Training of VMF then HH • Staple (rainy season) production • ENA training • HH gardens year 2

– Horticulture production on off (dry) season

FINDINGS: LOGISTICS

• Seasonality constraint (endline = baseline) allowed only 2 years of implementation

• Tremendous enthusiasm of communities and local partners

• Within intervention communities

– “My husband thinks that my work at the VMF is contributing to improved health in our household”

– “I have a family garden to cultivate vegetables and meet the nutritional needs of my household”

• Spillover doubled population reach; demand from control communities

TESTIMONIALS

• Research findings valuable but come at cost to implementation

– Staff time to support logistics – Time burden on beneficiaries and controls

• Not all problems can be solved – Water and soil constraints – Weaknesses in larger input delivery systems – Intrinsic motivation not universal (also VMF vs. ENA) – Low education level of community volunteers

• Complexity of model – Multisectorality of contents and partners – ENA (and now EHA) dense with messages and behaviors to

change

CHALLENGES IN PROCESS EVALUATION

• Findings informed strengthened program delivery

• Process gave voice to staff and beneficiaries

• Informed changes to model for subsequent phase

– Timing/design of research – Water management plans – Incentive strategies – Reinforced WASH and malaria

control components – Test LNS to fill nutrient gaps

• Effort is high; returns are higher

RETURNS

MERCI

15

“We could never learn to be brave and patient if there were only joy in the world.” – Helen Keller