annual results and impact evaluation workshop for rbf - day four - the science of delivery - impact...
DESCRIPTION
A presentation from the 2014 Annual Results and Impact Evaluation Workshop for RBF, held in Buenos Aires, Argentina.TRANSCRIPT
![Page 1: Annual Results and Impact Evaluation Workshop for RBF - Day Four - The Science of Delivery - Impact Evaluation Results and Lessons from RBF](https://reader033.vdocuments.us/reader033/viewer/2022060109/5551b79fb4c905bb708b4f86/html5/thumbnails/1.jpg)
The Science of Delivery:
Impact Evaluation Results and
Lessons from RBF
Dinesh Nair, Benjamin Loevinsohn and Ifelayo Ojo
Learning from Implementation
![Page 2: Annual Results and Impact Evaluation Workshop for RBF - Day Four - The Science of Delivery - Impact Evaluation Results and Lessons from RBF](https://reader033.vdocuments.us/reader033/viewer/2022060109/5551b79fb4c905bb708b4f86/html5/thumbnails/2.jpg)
The Why and How of
Top Ten Lessons • Review of ongoing PBF operations suggested
some consistent challenges & useful lessons
• Focusing on the most important lessons will
facilitate learning by other teams
• Selection of the Top Ten was done by TTLs
and RBF experts involved in a portfolio
review of eight ongoing RBF schemes
2
![Page 3: Annual Results and Impact Evaluation Workshop for RBF - Day Four - The Science of Delivery - Impact Evaluation Results and Lessons from RBF](https://reader033.vdocuments.us/reader033/viewer/2022060109/5551b79fb4c905bb708b4f86/html5/thumbnails/3.jpg)
Lesson #1
“Show me the money!”
- Make timely payments
• Over 70% of projects have had issues with
making payments on time
3
![Page 4: Annual Results and Impact Evaluation Workshop for RBF - Day Four - The Science of Delivery - Impact Evaluation Results and Lessons from RBF](https://reader033.vdocuments.us/reader033/viewer/2022060109/5551b79fb4c905bb708b4f86/html5/thumbnails/4.jpg)
Outpatient visits in Nigeria declined when
payments were interrupted
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
Dec
Jan
Feb
Mar
Apr
May
Jun
Jul
Aug
Sep
Oct
Nov
Dec
Jan
Feb
Mar
Apr
May
Jun
Jul
Aug
Sep
Adamawa
Nasarawa
Ondo
Payments
interrupted
4
![Page 5: Annual Results and Impact Evaluation Workshop for RBF - Day Four - The Science of Delivery - Impact Evaluation Results and Lessons from RBF](https://reader033.vdocuments.us/reader033/viewer/2022060109/5551b79fb4c905bb708b4f86/html5/thumbnails/5.jpg)
Lesson 1: “Show me the Money”
Make Timely Payments
Recommendations
Track time required for payments to reach facilities
Establish explicit standards for maximum allowable
delay (2 or 3 months)
Provide resources up-front to health facilities: (i)
gains their confidence; (ii) indicates PBF has
started; and (iii) provides investment funds
5
![Page 6: Annual Results and Impact Evaluation Workshop for RBF - Day Four - The Science of Delivery - Impact Evaluation Results and Lessons from RBF](https://reader033.vdocuments.us/reader033/viewer/2022060109/5551b79fb4c905bb708b4f86/html5/thumbnails/6.jpg)
Lesson #2
“Keep moving the goalposts!”
Continuous Quality Improvement
(CQI) implies Changing the
Quality Indicators
• Many facilities make rapid improvements in
quality and then plateau
6
![Page 7: Annual Results and Impact Evaluation Workshop for RBF - Day Four - The Science of Delivery - Impact Evaluation Results and Lessons from RBF](https://reader033.vdocuments.us/reader033/viewer/2022060109/5551b79fb4c905bb708b4f86/html5/thumbnails/7.jpg)
0
20
40
60
80
100
Mwaro
Muramvya
Kirundo
Cibitoke
Buja-Rural
Kayanza
Ngozi
Makamba
Rutana
Bubanza
Bururi
Gitega
Karuzi
Muyinga
Ruyigi
Cankuzo
Buja-Mairie
Quality scores by province
Average quality score plateaued after 3
quarters in Burundi
7
![Page 8: Annual Results and Impact Evaluation Workshop for RBF - Day Four - The Science of Delivery - Impact Evaluation Results and Lessons from RBF](https://reader033.vdocuments.us/reader033/viewer/2022060109/5551b79fb4c905bb708b4f86/html5/thumbnails/8.jpg)
Lesson 2: “Keep moving the goalposts!”
Continuous Quality Improvement (CQI)
implies Changing Quality Indicators
Recommendations
Quantified quality checklists should emphasize
process over structural indicators
Revise checklist every 1–2 years to emphasize
continuous improvement
Include: 1) observation of patient-provider
interaction; 2) records reviews (need standard
records); 3) Vignettes; 4) Exit interviews, etc.
Invest in training supervisors
8
![Page 9: Annual Results and Impact Evaluation Workshop for RBF - Day Four - The Science of Delivery - Impact Evaluation Results and Lessons from RBF](https://reader033.vdocuments.us/reader033/viewer/2022060109/5551b79fb4c905bb708b4f86/html5/thumbnails/9.jpg)
Lesson #3
“To be discerning, We keep
on Learning”
• RBF as a tool allows for experimentation,
learning and course-correction
9
![Page 10: Annual Results and Impact Evaluation Workshop for RBF - Day Four - The Science of Delivery - Impact Evaluation Results and Lessons from RBF](https://reader033.vdocuments.us/reader033/viewer/2022060109/5551b79fb4c905bb708b4f86/html5/thumbnails/10.jpg)
What did Zimbabwe do?
Quantitative analysis
Qualitative analysis
Design demand and supply-side RBF (urban pilot)
Targeted urban maternal voucher (demand-side)
Pay-for-quality RBF
Reviewed pricing
Improved quality checklists
10
![Page 11: Annual Results and Impact Evaluation Workshop for RBF - Day Four - The Science of Delivery - Impact Evaluation Results and Lessons from RBF](https://reader033.vdocuments.us/reader033/viewer/2022060109/5551b79fb4c905bb708b4f86/html5/thumbnails/11.jpg)
Operational research studies should be budgeted
Carry out process evaluations and use results for
implementation dialogue and decision-making.
Participate in RBF portfolio reviews – look for new
ideas
Lesson 3: To be discerning - Keep on
Learning
11
![Page 12: Annual Results and Impact Evaluation Workshop for RBF - Day Four - The Science of Delivery - Impact Evaluation Results and Lessons from RBF](https://reader033.vdocuments.us/reader033/viewer/2022060109/5551b79fb4c905bb708b4f86/html5/thumbnails/12.jpg)
Lesson #4
“Money is NOT the root of all evil, Lack of Money is” – Worry about Financial
Sustainability
• PBF should not be made to substitute
deficiencies in national health financing
12
![Page 13: Annual Results and Impact Evaluation Workshop for RBF - Day Four - The Science of Delivery - Impact Evaluation Results and Lessons from RBF](https://reader033.vdocuments.us/reader033/viewer/2022060109/5551b79fb4c905bb708b4f86/html5/thumbnails/13.jpg)
PBF sustainability requires wider health
system reforms
Burundi PBF is
combined with FHC
Government pays almost
half–1.4% of budget
allocated
Common pool for
development partners– two
decreased contribution
Cumulative deficit of
almost US$ 8M
Rwanda PBF health
reforms were part of
larger reforms CBHI reform
HR reforms
13
![Page 14: Annual Results and Impact Evaluation Workshop for RBF - Day Four - The Science of Delivery - Impact Evaluation Results and Lessons from RBF](https://reader033.vdocuments.us/reader033/viewer/2022060109/5551b79fb4c905bb708b4f86/html5/thumbnails/14.jpg)
Where is the PBF money going?
14
![Page 15: Annual Results and Impact Evaluation Workshop for RBF - Day Four - The Science of Delivery - Impact Evaluation Results and Lessons from RBF](https://reader033.vdocuments.us/reader033/viewer/2022060109/5551b79fb4c905bb708b4f86/html5/thumbnails/15.jpg)
35%
37%
39%
40%
45%
11%
10%
10%
9%
5%
16%
16%
12%
13%
10%
29%
31%
31%
33%
37%
8%
7%
7%
5%
3%
Q2 2012
Q3 2012
Q4 2012
Q1 2013
Q2 2013
Intervention HFs Intervention DMO Hospital
Control 1 HFs Control 1 DMO
Zambia: Aggregate consumption of PBF funds from Q2, 2012 – Q2, 2013
Where is the PBF money going?
15
![Page 16: Annual Results and Impact Evaluation Workshop for RBF - Day Four - The Science of Delivery - Impact Evaluation Results and Lessons from RBF](https://reader033.vdocuments.us/reader033/viewer/2022060109/5551b79fb4c905bb708b4f86/html5/thumbnails/16.jpg)
Recommendations
Track overall PBF expenditures and understand where
most PBF funds are going
It is easier to increase tariffs than decrease them. This
suggests starting with relatively lower tariffs and
increasing them
Insist that Government(s) live within an envelope
budget.
Be careful about increasing expenditure untied to
improved performance.
Lesson 4: “Money is NOT the root of all
evil, Lack of Money is” – Worry about
Financial Sustainability
16
![Page 17: Annual Results and Impact Evaluation Workshop for RBF - Day Four - The Science of Delivery - Impact Evaluation Results and Lessons from RBF](https://reader033.vdocuments.us/reader033/viewer/2022060109/5551b79fb4c905bb708b4f86/html5/thumbnails/17.jpg)
Lesson #5
“Math-Phobes of the World Unite!”
– Use your data
• Data is vital but under-utilized, despite a lot
of effort invested into collecting, verifying and putting payment data on the web
17
![Page 18: Annual Results and Impact Evaluation Workshop for RBF - Day Four - The Science of Delivery - Impact Evaluation Results and Lessons from RBF](https://reader033.vdocuments.us/reader033/viewer/2022060109/5551b79fb4c905bb708b4f86/html5/thumbnails/18.jpg)
Internet applications with public front-
end displaying performance & financial
information
18
Burundi
Benin
Nigeria
![Page 19: Annual Results and Impact Evaluation Workshop for RBF - Day Four - The Science of Delivery - Impact Evaluation Results and Lessons from RBF](https://reader033.vdocuments.us/reader033/viewer/2022060109/5551b79fb4c905bb708b4f86/html5/thumbnails/19.jpg)
How Zambia uses its data
• Quarterly analysis
• Extensive analysis by expert
o Diff-in-diff analysis of trends using HMIS to compare
across the 3 study arms
RBF vs. Additional financing
Coef 0.904 0.815 0.696 12.944 2.220 -9.316 -2.783
p-value 0.045 0.231 0.229 0.002 0.019 0.735 0.024
RBF vs. Control
Coef 1.174 1.954 1.586 7.850 2.243 -39.929 -2.761
p-value 0.005 0.002 0.011 0.055 0.031 0.158 0.011
Attendance
outpatient
total (calc)
Immunised
fully <1 year
new
Antenatal
1st visit
before 20
weeks
IPT 3rd
dose to
pregnant
woman
Postnatal
care within
6 days
Attendance
Family
Planning
total (Calc)
Delivery by
skilled
personnel
19
![Page 20: Annual Results and Impact Evaluation Workshop for RBF - Day Four - The Science of Delivery - Impact Evaluation Results and Lessons from RBF](https://reader033.vdocuments.us/reader033/viewer/2022060109/5551b79fb4c905bb708b4f86/html5/thumbnails/20.jpg)
Lesson 5: Math-Phobes of the World
Unite! – Use your data
Recommendations
Create reliable systems for data collection and
compilation
Ensure consistent flow of PBF data with minimal
reporting burden on health workers
Make it somebody’s job to analyze data
Deploy software to help with data analysis
Crowd-source analysis!
Attempt to triangulate all available data sources to
determine trends in utilization and outcomes
20
![Page 21: Annual Results and Impact Evaluation Workshop for RBF - Day Four - The Science of Delivery - Impact Evaluation Results and Lessons from RBF](https://reader033.vdocuments.us/reader033/viewer/2022060109/5551b79fb4c905bb708b4f86/html5/thumbnails/21.jpg)
Lesson #6
“Understand your Customers”
-Demand side issues may be under-appreciated
• There is need to understand root-causes of
the problem RBF is trying to solve.
21
![Page 22: Annual Results and Impact Evaluation Workshop for RBF - Day Four - The Science of Delivery - Impact Evaluation Results and Lessons from RBF](https://reader033.vdocuments.us/reader033/viewer/2022060109/5551b79fb4c905bb708b4f86/html5/thumbnails/22.jpg)
Even with large improvements in utilization
since PBF was introduced, absolute
coverage levels remain low in Nigeria
Barriers to service utilization in two Nigerian LGAs:
Transportation Challenges
Variable & unpredictable fees for Services and Drugs
Social and cultural Barriers
Proposed solution: A voucher scheme to improve service uptake
Continuous cycle of learning and responding to help households overcome service utilization barriers and improve health outcomes.
22
![Page 23: Annual Results and Impact Evaluation Workshop for RBF - Day Four - The Science of Delivery - Impact Evaluation Results and Lessons from RBF](https://reader033.vdocuments.us/reader033/viewer/2022060109/5551b79fb4c905bb708b4f86/html5/thumbnails/23.jpg)
Lesson 6: Understand your customers –
Demand-side issues may be under-
appreciated
Recommendations
Consider demand-side issues if coverage levels remain
low in spite of PBF (even if PBF has made a big
difference).
Understand barriers to access – cultural, social and
financial barriers through household surveys; focus
groups, key informant interviews, etc.,
Options for addressing demand side include: (i) closer
work with community structures; (ii) CCTs; (iii)
vouchers; (iv) BCC through facility staff or NGOs
23
![Page 24: Annual Results and Impact Evaluation Workshop for RBF - Day Four - The Science of Delivery - Impact Evaluation Results and Lessons from RBF](https://reader033.vdocuments.us/reader033/viewer/2022060109/5551b79fb4c905bb708b4f86/html5/thumbnails/24.jpg)
Lesson #7
“Away with Flat-Liners”
- Identify poorly performing regions & facilities. Do something about them!
• Detailed understanding of predictors of
success and failure is required
24
![Page 25: Annual Results and Impact Evaluation Workshop for RBF - Day Four - The Science of Delivery - Impact Evaluation Results and Lessons from RBF](https://reader033.vdocuments.us/reader033/viewer/2022060109/5551b79fb4c905bb708b4f86/html5/thumbnails/25.jpg)
There are clearly facilities that are NOT making
progress i.e. “flat-liners”
Non-determinants
• Number of staff
• Remoteness
• Qualification of in-
charge
• Business planning
-
20
40
60
80
100
120
140 Positive deviants
• Community engagement
• Management capacity of in-charge
Institutional Delivery in Adamawa health facilities,
normalized by 100,000 population
25
![Page 26: Annual Results and Impact Evaluation Workshop for RBF - Day Four - The Science of Delivery - Impact Evaluation Results and Lessons from RBF](https://reader033.vdocuments.us/reader033/viewer/2022060109/5551b79fb4c905bb708b4f86/html5/thumbnails/26.jpg)
Lesson 7: Away with “Flat-Liners”
Recommendations
Use existing data to identify facilities that have not
improved or are not performing well
Carry out quantitative and qualitative studies (e.g.
Nigeria and Zimbabwe)
Consider the kind of technical support needed for
the poorly performing facilities and re-examine
what is currently available.
26
![Page 27: Annual Results and Impact Evaluation Workshop for RBF - Day Four - The Science of Delivery - Impact Evaluation Results and Lessons from RBF](https://reader033.vdocuments.us/reader033/viewer/2022060109/5551b79fb4c905bb708b4f86/html5/thumbnails/27.jpg)
Lesson #8
“You can run but you can’t hide”
– Worry about the politicaI threats to PBF
• Even successful RBF schemes face political
pressures that threaten their sustainability
27
![Page 28: Annual Results and Impact Evaluation Workshop for RBF - Day Four - The Science of Delivery - Impact Evaluation Results and Lessons from RBF](https://reader033.vdocuments.us/reader033/viewer/2022060109/5551b79fb4c905bb708b4f86/html5/thumbnails/28.jpg)
Zambia RBF Pilot
RBF PIU not mainstreamed into MOH structure
Institutional changes in MOH
Key champions moved
28
Watch for winners and losers
![Page 29: Annual Results and Impact Evaluation Workshop for RBF - Day Four - The Science of Delivery - Impact Evaluation Results and Lessons from RBF](https://reader033.vdocuments.us/reader033/viewer/2022060109/5551b79fb4c905bb708b4f86/html5/thumbnails/29.jpg)
Recommendations
Political economy expertise should be engaged to
study factors important for scale-up and
sustainability of RBF, to figure out what can be
done differently.
Institutionalizing roles such as purchasing and
project implementation within government
structures may help improve buy in
29
Lesson 8: Worry about the political
threats to PBF
![Page 30: Annual Results and Impact Evaluation Workshop for RBF - Day Four - The Science of Delivery - Impact Evaluation Results and Lessons from RBF](https://reader033.vdocuments.us/reader033/viewer/2022060109/5551b79fb4c905bb708b4f86/html5/thumbnails/30.jpg)
Lesson #9
“Even the Best Laid Plans can go wrong”
– Analyze your Project Design to Ensure Assumptions
Remain Valid
• Understand key-determinants of the PBF program and the linkages between them to produce results
30
![Page 31: Annual Results and Impact Evaluation Workshop for RBF - Day Four - The Science of Delivery - Impact Evaluation Results and Lessons from RBF](https://reader033.vdocuments.us/reader033/viewer/2022060109/5551b79fb4c905bb708b4f86/html5/thumbnails/31.jpg)
Benin
Bureaucracy makes it
difficult for facilities to
spend PBF incentives
May reduce health worker
motivation to perform
User-fees easier to spend
High fees may decrease
healthcare utilization
31
Challenges with health facility autonomy
in Benin and nested PBF in Afghanistan
made implementation different
Afghanistan
Unique environment of
performance agreement
with NGO
Ongoing security concerns
PBF overlaid on PPA
![Page 32: Annual Results and Impact Evaluation Workshop for RBF - Day Four - The Science of Delivery - Impact Evaluation Results and Lessons from RBF](https://reader033.vdocuments.us/reader033/viewer/2022060109/5551b79fb4c905bb708b4f86/html5/thumbnails/32.jpg)
Recommendations
Periodically review the progression of elements in your
theory of change, not just what you are trying to
achieve, but how you get there
Useful analyses for every project include:
What is additional percentage of PBF to health workers'
take-home pay? Are they motivated by this amount?
Balance of supervision versus autonomy at the health
facility level
Resources available at the front-line for health facilities
32
Lesson 9: “Even the Best Laid Plans can
go wrong” – analyze project design to
ensure assumptions remain valid
![Page 33: Annual Results and Impact Evaluation Workshop for RBF - Day Four - The Science of Delivery - Impact Evaluation Results and Lessons from RBF](https://reader033.vdocuments.us/reader033/viewer/2022060109/5551b79fb4c905bb708b4f86/html5/thumbnails/33.jpg)
Lesson #10
“KISS – Keep Impact Studies Straightforward”
- Avoid Impact Evaluation Questions that are TOO Subtle
• It is very difficult to assess differential impacts when there is minimal variation between experimental groups
33
![Page 34: Annual Results and Impact Evaluation Workshop for RBF - Day Four - The Science of Delivery - Impact Evaluation Results and Lessons from RBF](https://reader033.vdocuments.us/reader033/viewer/2022060109/5551b79fb4c905bb708b4f86/html5/thumbnails/34.jpg)
0
600
1200
1800
2400
1* 2 3 4 1 2
2012 2013
Children immunized
ControlTreatment
The RBF intervention and additional
financing group in Benin are looking the
same
No clear difference is
emerging
Another control group
with no added finances
has been included for
counterfactual
34
0
2000
4000
6000
8000
1* 2 3 4 1 2
2012 2013
Assisted delivery
Control
Treatment
![Page 35: Annual Results and Impact Evaluation Workshop for RBF - Day Four - The Science of Delivery - Impact Evaluation Results and Lessons from RBF](https://reader033.vdocuments.us/reader033/viewer/2022060109/5551b79fb4c905bb708b4f86/html5/thumbnails/35.jpg)
Experimental & Control groups in Nigeria
– try hard to be explicit on differences !!!!
35
![Page 36: Annual Results and Impact Evaluation Workshop for RBF - Day Four - The Science of Delivery - Impact Evaluation Results and Lessons from RBF](https://reader033.vdocuments.us/reader033/viewer/2022060109/5551b79fb4c905bb708b4f86/html5/thumbnails/36.jpg)
Recommendations
Assess in advance if experimental variations are
likely to make a difference in outcomes.
Work with the IE team from the start
Ensure counterparts fully understand research
questions, and are convinced of utility.
36
Lesson 10: “KISS – Keep Impact Studies
Straightforward” Avoid Impact Evaluation
Questions that are TOO Subtle
![Page 37: Annual Results and Impact Evaluation Workshop for RBF - Day Four - The Science of Delivery - Impact Evaluation Results and Lessons from RBF](https://reader033.vdocuments.us/reader033/viewer/2022060109/5551b79fb4c905bb708b4f86/html5/thumbnails/37.jpg)
A ship in the harbor is safe – but that is not
what ships are built for. -John A Shedd (1928)
Implement, learn and share
lessons!
Annual Results and Impact Evaluation Workshop for RBF - Day One - Verification of Results Group Work
Annual Results and Impact Evaluation Workshop for RBF - Day Eight - Impact Evaluation Note - Armenia