annual results and impact evaluation workshop for rbf - day one - using operational and hmis data...
Post on 07-May-2015
32 Views
Preview:
DESCRIPTION
TRANSCRIPT
Using Operational and HMIS data for Program Monitoring and Impact Evaluation - Zambia
2014 Results & Impact Evaluation Workshop
Zambian Delegation
25th March 2014
Zambia RBF Model
One of the few examples of “contracting in” through the public health sector
Quasi Provider-Purchaser split by different levels of the Zambian Health Care Delivery System
Quantity and Quality data verification Steering Committees (SCs) as Independent Verifiers Periodic External Verification
Performance-Based Payments
“Fee-for-service” on a set of MNCH indicators at Health Centres
Performance Evaluation Framework for District Medical Offices
Managerial and financial autonomy of health facilities
2
Operational Data
3
Trends in Performance Indicators: Q2 2012-Q4 2013
0
2,000
4,000
6,000
8,000
10,000
12,000
14,000
16,000
Q2 2012 Q3 2012 Q4 2012 Q1 2013 Q2 2013 Q3 2013 Q4 2013
Nu
mb
er
Skilled Deliveries
ANC prenatal andfollow up visits
Postnatal visit
Fully vaccinatedchild
Third doseFansidar IPT
Preg. womengiven Niverapine
and AZT
4
Trends in Performance Indicators: Q2 2012-Q4 2013…
0
100,000
200,000
300,000
400,000
500,000
600,000
700,000
800,000
900,000
Q2 2012 Q3 2012 Q4 2012 Q1 2013 Q2 2013 Q3 2013 Q4 2013
Nu
mb
er
CurativeConsultation
FP users ofmodern methods
Preg. womencounseled andtested for HIV
5
Percentage Increase – Quantity Indicators
14%
-1%
23%
-49%
69%
44%
94% 70%
203%
14% 19% 21% 22% 22% 25%
63% 67%
88%
-100%
-50%
0%
50%
100%
150%
200%
250%
CurativeConsultation
HIV Preg.women
givenNiverapine
& AZT
PregnantWomen C&T
for HIV
ANCprenatal and
follow upvisits
SkilledDeliveries
Third doseFansider IPT
Postnatalvisit
FullyImmunized
Children
FP users ofmodern
methods
Per
cen
tag
e C
han
ge
Q2 2012 Vs Q4 2013 (Since Project Start) Q4 2012 Vs Q4 2013
6
Use of HMIS Data
7
Similarities and Differences: OP Vs HMIS Data
• OP and HMIS data are collected from the same health facilities, and same data records
• However, OP data is compiled much quicker than HMIS data given the requirements for verification, and linkages to payments
• OP data is only complied from patient registered as compared to HMIS which is complied from registers and tally sheets
• Consolidation of OP and HMIS data is done by different personnel at district level
• 100% of the OP data is verified on a monthly basis while the HMIS mainly relies on self-reported data which is occasionally verified
• OP and HMIS data MUST show the same trend in indicators, despite differences in magnitude
8
Using HMIS to compare across the 3 study arms of the Impact Evaluation
What can we say before endline?
Diff-in-diff analysis of trends before and after RBF
Analysis period – Jan 2011 to Dec 2012: RBF introduced April 2012
Not definitive analysis since HMIS is self-reported but:
Important check on RBF: Are results from the operational data consistent with HMIS?
What is the impact of RBF on non-incentivized indicators?
9
Impact Evaluation
Explores whether there is a causal link between the RBF project and the results
Baseline – Quantitative and Qualitative
Process Evaluation (interviews, Observations, operational and HMIS data review)
End line – Quantitative and Qualitative
Three (3) study arms:
10 RBF Intervention Districts (RBF)
10 Input-Based Financing Districts (C1)
10 Pure Control Districts (C2)
10
Impact on incentivized indicators
All measures: per service per facility per month
Gains in several targeted services, no change in total utilization, and declines in immunization
No gains from additional financing to districts
RBF vs. Additional financing
Coef 0.904 0.815 0.696 12.944 2.220 -9.316 -2.783
p-value 0.045 0.231 0.229 0.002 0.019 0.735 0.024
RBF vs. Control
Coef 1.174 1.954 1.586 7.850 2.243 -39.929 -2.761
p-value 0.005 0.002 0.011 0.055 0.031 0.158 0.011
Attendance
outpatient
total (calc)
Immunised
fully <1 year
new
Antenatal
1st visit
before 20
weeks
IPT 3rd
dose to
pregnant
woman
Postnatal
care within
6 days
Attendance
Family
Planning
total (Calc)
Delivery by
skilled
personnel
Additional financing vs. Control
Coef 0.035 1.117 0.882 -5.022 0.027 -31.420 0.035
p-value 0.979 0.107 0.114 0.194 0.976 0.247 0.979
11
Trend in Skilled delivery (Jan 2011-Dec 2012)
0
200
400
600
800
1000
1200
1400
1600
1800
2000
Input Based
Pure Control
Data Source: HMIS, MoH
RBF implemented
12
RBF
Impact on non-incentivized indicators
Little spill-over to the non-incentivized
Additional financing hired more staff (but no change in service measures)
RBF vs. Additional financing
Coef -0.945 3.046 0.136 -0.326 -3.578 -1.018
p-value 0.780 0.090 0.203 0.027 0.563 0.773
RBF vs. Control
Coef -3.847 2.147 0.046 0.181 8.057 0.298
p-value 0.306 0.191 0.799 0.289 0.032 0.759
Additional financing vs. Control
Coef -3.302 -0.840 -0.084 0.511 10.542 1.359
p-value 0.327 0.679 0.588 0.001 0.138 0.748
Supportive
supervision
visits this
month
Vitamin A
supplement
to 6-11
months
New TB
patient total
(calc)
TB patient
completed
treatment
Support
staff newly
recruited
Nurse
midwife
workdays
on duty
13
Use of OP and HMIS data has made it possible to: Triangulate OP data with HMIS data i.e. Check on
consistency of OP data with HMIS
Independently verify the OP data
Monitor trends in incentivized and non-incentivized indicators across the three (3) research arms
Monitor the utilization of funds by indicators, and the overall amount allocated to the RBF Project
Make adjustments to the RBF design, as well as to provide more capacity building and technical support
14
Examples of how the Emerging Information has been used
Application of the quality tool changed to reward for quality improvements instead of penalizing for quality deficits
Set investment component at a minimum of 40%, and staff performance incentives at a maximum of 60%
Increase assessment fees for hospitals doing quality audits
Revise TA package to draw on local capacities
Enhanced technical support to underperforming health facilities
Introduction of supervision fees for Provincial RBF Steering Committees
15
Challenges
• Late transmission of HMIS data
• Poor quality of HMIS data due to migration to a web-based DHIS-2
• Inadequate data entry clerks at health facilities particularly in the control districts
• Costly to conduct a process evaluation involving observations at health facilities, and interviews with service providers, patients, and community members
16
Thank You
17
top related