Understanding causal pathways within health systems policy evaluation through mediation
analysis: an application to payment for performance (P4P) in Tanzania
Laura Anselmi, Peter Binyaruka, Masuma Mamdani
Josephine Borghi
Payment for Performance: a health systems perspective
A workshop for scientists and practitioners
Dar es Salaam, 26 November 2015
Rationale
• Programme evaluation has mainly focused on measuring the impact on outcomes, with little attention to the causal pathways
• The relevance of undertaking process evaluation to be integrated with outcome evaluation is increasingly recognised
• Process evaluation is particularly relevant for complex interventions: It increases confidence in the plausibility of outcome effects
It increases the external validity of the evaluation
• Process evaluations have been carried out, but without formal assessment of causal pathways
Causal mediation analysis
• A causal mechanism is a process through which a programme or intervention influences an outcome
• It can be identified by specifying intermediate outcomes or variables(mediators) that are on the causal pathway between intervention and outcome
• Causal mediation analysis has been employed to test change pathways within the evaluation of public health programmes
• Mediators have been limited to individual level indicators, psychological or physical
• Health system mediators which are relevant to the evaluation of health systems or services interventions have not been considered
Program Outcome
Mediator 1 P4P Indirect
effect
P4P Measured effect = P4P Direct Effect + P4P Indirect effect 1 + P4P Indirect Effect 2
Confounder
Mediator 2 P4P indirect
effect
Sequential ignorability b):Given treatment status and pre-treatment confounders the mediators are ignorable (no confounders affecting both mediators and outcome)
Programme Measured effect
Confounder
Confounder
Causal mediation analysisSequential ignorability a):
Given pre-treatment confounders the treatment is assigned
independently of potential outcomes and mediators
P4P direct effect
P4P Programme in Tanzania
• P4P Scheme introduced in 2011 by the MoH in the Pwani region
• Target on Maternal and Child Health Care outcomes
• Outcome evaluation:• 8.2% increase in coverage of institutional deliveries (ID)
• 6.5% increase in delivery in public health facilities
• 10.3% increase in the uptake of two doses of anti-malarial (IPT) during pregnancy
• The effect of P4P on a number of governance, financing and human resources factors has been identified
• Data:• Health facility survey: 150 HFs (75 P4P + 75 control)
• 1-2 interviews with health workers per HF
• 1,500 exit interviews
• 3,000 household survey
• Baseline: Jan-March 2012, Endline: February 2013
P4P theory of change in a Causal Mediation Analyses Framework
P4P Outcome
P4P indirect effect through Governance
P4P indirect effect through Human
Resources
P4P indirect effect through Financing
P4P direct effect
Methods
• Step1: Estimating the impact of P4P on outcomes (DiD)
P4Pt indicator of P4P districtδt time indicatorXijt women socio-economic characteristicsγj HF fixed effects
• Step 2: Identifying effect of P4P on potential mediators (DiD)
• Step 3: Identifying direct and indirect causal effects (DiD)
𝛽13 P4P direct effect
𝛽12 X 𝛽4
3 P4P indirect effect through mediator M
𝑌𝑖𝑗𝑡 = 𝛽03 + 𝛽1
3(𝑃4𝑃𝑗 × 𝛿𝑡) + 𝛽23𝛿𝑡 + 𝛽3
3𝑋𝑖𝑗𝑡 + 𝛽43𝑀𝑖𝑗𝑡 + 𝛾𝑗 + 휀𝑖𝑗𝑡
3
𝑌𝑖𝑗𝑡 = 𝛽01 + 𝛽1
1(𝑃4𝑃𝑗 × 𝛿𝑡) + 𝛽21𝛿𝑡 + 𝛽3
1𝑋𝑖𝑗𝑡 + 𝛾𝑗 + 휀𝑖𝑗𝑡1
𝑀𝑖𝑗𝑡 = 𝛽02 + 𝛽1
2(𝑃4𝑃𝑗 × 𝛿𝑡) + 𝛽22𝛿𝑡 + 𝛽3
2𝑋𝑖𝑗𝑡 + 𝛾𝑗 + 휀𝑖𝑗𝑡2
Results: Potential mediators (Step 2)Potential mediators Effect of P4P (% change)
Financing
Proportion of women who paid for delivery in a HF (1) -8.0**
Proportion of women who paid for delivery in a public HF (1) -7.5***BS
Service delivery disrupted due to broken equipment last 90days -149**
Drug stock-out index-general (0-1 index) -17.2***BS
Medical supplies stock-out index (0-1 index) -14.8***BS
Oxytocin injection stock-out last 90days -36.2***BS
Ergometrine injection stock-out last 90days -26.1**
Drugs at delivery stock-out index (0-1 index) -27.0***BS
Mean all financing indicators(0-1 index) -8.3**
Factor analysis weighted score (2) -60.0***BS
Governance
Max time from external supervision: 90 days ago -18.0**
Dist/Regional supervision provided positive feed-back 23.8**
Dist/Regional supervision provided negative feed-back 28.2**
Dist/Regional supervision delivered supply -19.3**
Dist/Regional checked records 1.5**
Dist/Regional observed consultation 0.8**
Human resources
Mean patient satisfaction with interpersonal care (0-1 scale) (1) 6.7***BS
Mean kindness ranks for HW at delivery (0-1 scale) (1) 10.3***BS
* p<0.10, ** p<0.05, *** p<0.01 , BS: Significant at 5% level with Bonferroni adjusted p-value for multiple outcomes: Bonferroni adjusted p-value
Financing 0.0047, Governance 0.0017, Human resources 0.0414, (1) Out of all women delivering in a HF in same catchment area (2) equipment,
vaccines, drugs, medical supply
Results: P4P direct and indirect effect (Step 3)
• Facility based delivery
P4P total effect: +8.2%
P4P indirect effect through mean of all financing indicators: +1 %
P4P indirect effect through reduction in stock-out of oxytocine: +1.8 %
P4P direct effect: +7.2% or +6.4%
• Delivery in public health facility
P4P total effect: +6.5 %
P4P indirect effect through reduction in stock-out of oxytocine: +1.9 %
P4P direct effect: +4.6%
• Uptake of two doses of IPT during pregnancy
P4P total effect: +10.3 %
P4P indirect effect through reduction in last supervision being 90 days ago: +1.5 %
P4P direct effect: +8.8%
Sensitivity analysis
• Semiparametric mediation analysis to quantify the sensitivity of results to the assumption of no confounders affecting mediator and outcome
• Analysis carried out at the HF level
• Estimate a logit model for binary mediators
• Multiple hypothesis testing adjustment of p-values for families of mediators
• DiD with district fixed effects
Summary: Indirect effects of P4P
• P4P significantly affects a number of financing, governance and human resources factors which could potentially mediate its effect on maternal care outcomes
• The effect of P4P on the reduction of oxytocine injection stock-out mediates the effect of P4P on institutional deliveries (22%) and deliveries in a public health facility (30%)
• The effect of P4P on the frequency of supervisions mediates 15% of the effect of P4P on the uptake of at least two doses of IPT during pregnancy
Some reflections on mediation analysis
• Mediation analysis rarely applied using difference in difference
• Quasi–experimental setting + Difference-in-Difference provide confidence that the assumption of no pre-treatment confounders is satisfied
• But how plausible are the assumptions of no confounders between mediator and outcome?
• Data availability limits testing pre-trends for mediators
• Possible differences in results according to the level of the analysis
• Little analysis of the role of individual level factors or other moderating factors
• Possibly simplified description of the causal chain
Conclusions
• Mediation analysis is helpful to quantify causal direct and indirect effects and the relative relevance of change pathways
• It requires assumptions to identify causality and these can not be tested formally
• Quantify the P4P indirect effects helps in thinking about relative cost-effectiveness compared to alternative interventions
Thank you!
Acknowledgements to the whole “P4P team”