SHAPING THE LEARNING AGENDAThe way forward

Dinesh Nair and Jake Robbins

Program Implementation provides many Opportunities for Learning

Q u a l i t a t i v e L e a r n i n g

Two Broad Pathways to Learn with opportunities to intertwine them

• Implementation Research and Delivery Science • different forms: operations research, mixed method • helps understanding the context

• Impact Evaluation • Establish causal link between observed results and evaluated program• Usually has a credible counterfactual, in the form of a valid control or

comparison group, is critical

Implementation Research and Delivery Science

The Five I’s of DDS

5

What results will DDS strive to achieve?Support teams, countries and partners to:

6

Make the best possible investment decisions Respond to, generate demand for and deliver services to the best feasible standards,

for the right people,

in the right places, in the right ways,

at the right time

Achieve the best possible health impact

Plan early to ensure that proven approaches are

institutionalized and sustained

India AIDS DDS Example

M

7

Operational Data helps track sub national performance

1 2 3 4 1 2 3 4 1 2 3 4 12012 2013 2014 2015

0

5

10

15

20

25

30

35

40

45

Pilot

Expansion 1

Expansion 2

Expansion 3

%QUALITY OF CARE

Dece

mbe

rM

arch

June

Sept

embe

rDe

cem

ber

Mar

chJu

neSe

ptem

ber

Dece

mbe

rM

arch

June

Sept

embe

rDe

cem

ber

Mar

ch

2011

2012 2013 2014 2015

0.00

10.00

20.00

30.00

40.00

50.00

60.00

70.00

80.00

90.00

100.00

AdamawaNasarawaOndoNational

Axis Title

Axis Title

Dynamic quality measures improve outcomes

Recalibration of quantified

quality checklist

COVERAGE OF INSTITUTIONAL DELIVERY

Coverage increases sustained over phases

Nigeria

Nigeria case study on key determinants

• What differentiate the good and poor performers under the PBF scheme?

• Health center management• Contextual factors• Health systems factors (e.g.,

supervision)

• Interviews, document review, direct observations

• Best and poorest performers• Devise appropriate support to

poor performers

Research question

Areas to look into

Approaches

Potential use

Detailed look at the operational data revealed the large variations in performance across Health Centers

Institutional Delivery in Adamawa, normalized by 100,000 population

December

January

February

March

AprilMay June

July

August

September

-

20

40

60

80

100

120

140 Pariya HC

Chigari HC

Dasin Hausa HC

Farang HC

Ribadu HC

Furore MCH HC

Choli HC

Gurin HC

Malabu HC

Karlahi HC

Wuro Bokki HC

Kabilo HC

Saint Mary's Clinic HC

Mayo-Ine HC

• Before PBF, all health centers were equally at very low levels

• After the PBF, some facilities achieved 100% coverage while others struggle with limited improvement

Case study on determinants suggests the importance of community engagement and OIC management

Identified determinants and non-determinants (preliminary)

Non-Determinants

• Level of staffing (best performers lack staff)

• Remoteness of facilities (best performers are very rural)

• Technical qualifications of OIC (many community health workers manage facilities well)

• Business planning (none use it effectively yet)

Determinants

• Community engagement (e.g., involve and reward community leaders, daily visits, incentivize for use of facility)

• OIC’s management capacity (e.g., full staff involvement, improve staff environment using performance bonus, rigorous performance review)

Impact Evaluation

13

Impact evaluation

Impact Evaluation:• Explores whether there is a causal link between the RBF

intervention and the results.

ProviderReport_________quantity__quality___services__________

Purchaser

Counter Verification

Verification

Contract_________quantity__quality___services__________ Out

puts

Outcomes

IE IE

An Impact Evaluation Answers:

What was the effect of the program on outcomes?

How much better off are the beneficiaries because of the program/policy?

How would outcomes change if one changed the program design?

Traditional M&E cannot answer these.

Cameroon RBF Impact evaluation design

• Research questions1. Does the PBF program increase the coverage and of MCH

services?2. Is it the enhanced monitoring & evaluation and

supervision or the link between payments and results that leads to improvements observed in quality or coverage?

3. What is the contribution of enhanced supervision and monitoring to improving MCH service coverage and quality in the absence of increased autonomy or additional financial resources?

• Public randomization ceremonies in each region• Baseline June 2012, Endline June 2015

• Results expected by December 2015• Midline qualitative study embedded in impact evaluation

Cameroon RBF IE Midline Qualitative Study• Focused on two primary themes:

• Experiences in the piloting of PBF at the central, regional and district level: perspectives of decision-makers and policymakers

• Experiential elements of health service delivery at the operational level: perspectives of patients and providers

• Research questions:• What has been the experience of piloting performance-based financing at

various administrative and operational levels of the health system in Cameroon?• What has been the experience of health service delivery for health workers,

patients and community members during the first two years of Performance-based financing?

Qualitative study: Sampling Methodology

PBF in Cameroun: Mid-line Qualitative Study Interview Format Total IDIs Total FGDs

Proposed Sampling and # of IDIs and FGDs IDIs FGDs Central Level

Ministry of Health 2 x 2 Development Partners 2 x 2

Regional Level (4 TOTAL Regions: NW, SW, Littoral, and East) Regional Health Delegates 1 x 4 District Medical Officers 2* x 8

IE Study Groups (T1, C1, and C2) District Hospitals and Primary Health Centres (Directors and Administrators) 6** x 72

Community Level Community Leaders 2*** x 32 xCommunity Members, Health Workers, etc. x 4/region*** 16* 2 DMOs per region**Stratifed by IE Study Group; 18 total IDIs per region

***Following FGD catchment (2 per village/4 villages total/8 per region) GRAND TOTALS for Midline Qualitative Study****Stratifed by T1/C2 IE Study Groups; Urban/Rural; Female only IDIs 120

FGs 16136 interviews TOTAL

Qualitative study: Results• Positive impact on service delivery

• Facilities are cleaner, more organized and better managed, staff motivated• Increased utilization and quality, price reduction for services• Better availability of drugs (breaking public monopoly)• Reduced under the table payments

• Increased collaboration among the various stakeholders• Regional/district supervision teams and health facilities (in particular private

health facilities)

• Management tools and procedures used in PBF led to enhanced transparency and accountability in resource management

• Obstacles encountered• Initial reluctance and adjustment of health facility staff to program • Obtaining initial buy-in and support of providers and ministry• Lack of confidence in Ministry of Health to follow through with promises• Challenges with increasing autonomy of health facilities 18

Discussion• What opportunities exist to make Learning a key part of the GFF

Agenda• How can “Learning” be mainstreamed through Investment Cases,

implementation plans , projects?• What technical and other support will be required to make this a

reality?• How can we incentivize learning and dissemination to be a major part

of the value proposition?