midline evaluation of hospitals selected from a quality ...€¦ · midline evaluation of hospitals...

Midline Evaluation of Hospitals Selected from a Quality-Based Pay for Performance

Scheme in Liberia

Luke Bawo

Kenneth L Leonard

Rianna Mohammed

Manasi Sharma

June 2016

2

Table of Contents

I. Health Systems Strengthening Project in Liberia ................................................................................. 7

A. Background ....................................................................................................................................... 7

B. The Project Development Objective ............................................................................................... 10

C. Theory of Change ........................................................................................................................... 12

II. Data Collection Effort ......................................................................................................................... 13

A. Instruments filled ............................................................................................................................ 17

B. Health workers observed ................................................................................................................. 17

III. Instrument summary ....................................................................................................................... 22

A. Motivation surveys .......................................................................................................................... 22

B. Competence/Capacity Vignettes ..................................................................................................... 29

1. Facility and health worker type averages for vignettes ............................................................... 29

2. Validating the theory of vignettes ............................................................................................... 34

C. Direct observation ........................................................................................................................... 37

D. Exit surveys ..................................................................................................................................... 41

IV. Evaluating the theory of change within HSSP facilities ................................................................. 50

A. The Three Gap Framework ............................................................................................................. 50

1. The Three Gaps in our Sample ................................................................................................... 53

2. Analyzing the three gaps by facility ........................................................................................... 54

B. Changes between 2013 Baseline Data Collection and 2015 Baseline Data Collection .................. 58

V. Conclusion .......................................................................................................................................... 65

VI. Appendices ...................................................................................................................................... 67

A. Acronyms ........................................................................................................................................ 67

B. Data Collection Schedule ................................................................................................................ 68

3

C. Quality Assessment Tool ................................................................................................................ 69

A. Instrument Summaries .................................................................................................................... 71

1. Surgical Vignettes ....................................................................................................................... 71

2. Pediatric Vignettes ...................................................................................................................... 80

3. Obstetric Vignettes ...................................................................................................................... 85

List of Tables

Table 1 Summary of Instruments by Service and Location ........................................................................ 16

Table 2 Number of Instruments Collected by Facility ................................................................................ 19

Table 3 Credentials of Health Workers on Hospital Rosters ...................................................................... 21

Table 4 Motivations Factors and Factor Loadings of Items ....................................................................... 23

Table 5 Motivations Factors across Facilities and wards............................................................................ 26

Table 6 Factor Means by Facility ............................................................................................................... 28

Table 7 Average Vignette Scores by Inputs, Diagnosis and Treatment...................................................... 30

Table 8 Regression coefficients for vignette scores by facility for assessment, diagnosis, and treatment

factors .......................................................................................................................................................... 32

Table 9 Regression Coefficients for Vignettes by Health Worker Type for Assessment, Diagnosis, and

Treatment Factors ....................................................................................................................................... 33

Table 10 Relationship between Inputs, Diagnosis, Detailed Diagnosis, and Treatment ............................ 35

Table 11 Direct Observation of Obstetric Care........................................................................................... 38

Table 12 Direct Observation of Pediatric Care ........................................................................................... 39

Table 13 Direct Observation of Surgical Care ............................................................................................ 40

Table 14 Patient Characteristics .................................................................................................................. 42

Table 15 Satisfaction As Expressed Across All Three Types of Exit Surveys ........................................... 44

4

Table 16 Factor Loading for Patient Exit Interviews .................................................................................. 46

Table 17 Satisfaction Summary Scores ...................................................................................................... 47

Table 18 Satisfaction Summary Scores by Facility and Ward .................................................................... 48

Table 19 Change in Average General Hospital Infrastructure Observation Scores, by Facility ................. 59

Table 20 Change in Average Motivation Factor Scores, by Facility .......................................................... 60

Table 21 Change in Key Indicators for Obstetrics Ward Observation ....................................................... 61

Table 22 Change in Key Indicators for Pediatrics Ward Observation ........................................................ 62

Table 23 Change in Key Indicators for Pediatrics Ward Observation, by Facility ..................................... 62

Table 25 Change in Key Indicators for Surgery Ward Observation ........................................................... 63

Table 26 Change in Vignette Scores, by Facility ........................................................................................ 63

Table 27 Change in Patient Characteristics for All Patient Exit Interviews, by Facility ............................ 64

Table 28 Change in Satisfaction Summary Scores for All Patient Exit Interviews, by Facility ................. 65

Table 29 Data Collection Schedule ............................................................................................................. 68

Table 30 Management and Structural Checklist Overview ........................................................................ 69

Table 31: Quality Checklist Overview ........................................................................................................ 69

Table 32: Quantity of Services.................................................................................................................... 70

Table 32: Surgery Vignette Scores by Facility (S_V_1_A) ........................................................................ 71

Table 2: Surgery Vignette Scores by Facility (S_V_1_B) .......................................................................... 72

Table 3: Surgery Vignette Scores by Facility (S_V_2_A).......................................................................... 74


Table 5: Surgery Vignette Scores by Facility (S_V_3) .............................................................................. 77

Table 6: Surgery Vignette Scores by Facility (S_V_5_A).......................................................................... 78


Table 8: Pediatric Vignette Scores by Facility (P_V_1) ............................................................................. 80

Table 9: Pediatric Vignette Scores by Facility (P_V_2) ............................................................................. 81

Table 10: Pediatric Vignette Scores by Facility (P_V_3) ........................................................................... 82

5

Table 11: Pediatric Vignette Scores by Facility (P_V_4) ........................................................................... 84

Table 12: Obstetrics Vignette Scores by Facility (O_V_1_A) ................................................................... 85

Table 13: Obstetrics Vignette Scores by Facility (O_V_1_B) .................................................................... 86



Table 16: Obstetrics Vignette Scores by Facility (O_V_3) ........................................................................ 90



List of Figures

Figure 1 Theoretical Framework: Theory of Change ................................................................................. 13

Figure 2 Probability of Being Correct Across All Vignettes by Facility .................................................... 32

Figure 3 Probability of Being Correct by Health Worker Type .................................................................. 33

Figure 4 Proportion of Possible Inputs into Correct Diagnosis and Treatment by Health Worker Type ... 34

Figure 5 Diagnoses are closely related to the use of key inputs ................................................................. 35

Figure 6 Correct treatment is related to the correct use of Physical Examination ...................................... 36

Figure 7 The Three-Gap Framework .......................................................................................................... 52

Figure 8 The Three Gap Framework for our sample (pediatrics) ............................................................... 54

Figure 9 Four Gaps across facilities ............................................................................................................ 55

Figure 10 Motivation factors and the four gaps with facility fixed effects ................................................. 57

Figure 11 Motivation factors and the four gaps with facility fixed effects ................................................. 58

6

Abstract:

Background: Improving the quality of care at hospitals is a key next step in rebuilding Liberia’s

health system. In order to improve the efficiency, effectiveness, and quality of care at the secondary

hospital level, the country is developing a system to upgrade health worker skills and competencies,

and shifting towards improved provider-accountability for results through a twin-pronged

approach involving the development of a Graduate Medical Residency Program (GMRP) and the

introduction of performance-based financing (PBF) at the hospital level.

Methods/Design: This document examines the assessment of quality in pediatric, obstetrics,

emergency and surgical wards in five hospitals enrolled in the Health Systems Strengthening

Project (HSSP) and five control hospitals. We examine first the overall level of quality in these

facilities as measured by our teams in intensive four to five days visits that examine quality using

competence and capacity vignettes as well as direct observation of patient care. Second, we

examine the data for evidence of patterns linking competence, capacity and performance as

outlined in the original proposal: the three gap model. As part of this, we also look at the gaps in

each facility and the differences in these gaps across facilities. Finally, four of the facilities

examine in this measurement exercise have been enrolled in the program now for almost two years

and although the implementation of the program was slowed by the Ebola crises, we examine the

data for evidence of improvements in the quality of care provided in these four facilities.

Findings: The quality of care remains relatively low in our sample, with important differences

across facilities. In examining the three gap model, we find that there is a very small competence-

capacity gap but a large can-do gap. Although we find that competence is low, suggesting

possibilities for more training, we find the link between competence (or capacity) and performance

is so weak as to call into question the value of improvements in competence. It is far more useful

to focus on the reasons why health workers do not use their knowledge and capacity in practice.

Fortunately, addressing this weak link was precisely the intention behind the HSSP. Finally, we

look at the changes across the four facilities that have been in the program for almost two years

and find no evidence of improvements in quality. There are some cases in which things have

improved but they fit not overall pattern that would suggest that they were caused by the HSSP.

Rather, many of the changes seen (for example in surgical preparation) are likely to have been

caused by the response to the Ebola crises.

7

I. Health Systems Strengthening Project in

Liberia

A. Background

The Liberian civil wars (1989-1996 and 1999-2003) completely destroyed the country’s

infrastructure and health care system. This included many hospitals and clinics being burned to

the ground; healthcare workers being killed or fleeing the country; equipment and supplies being

stolen or destroyed; a dramatic reduction in the number of functional laboratories; poor quality of

sanitation and basic supplies like water and electricity; and major disruptions in transport and

communication mechanisms.

Consequently, with the presence of the United Nations Mission in Liberia (UNMIL) and several

national and international organizations and countries providing humanitarian aid, the post-

conflict period saw improvements in efforts to rebuild the healthcare system. Despite impressive

gains in overall health systems management and in health services delivery since the end of the

war, Liberia continues to face significant challenges in improving maternal and child health

outcomes, as well as other health-related Millennium Development Goal (MDG) outcomes.

The country’s progress was severely disrupted when its healthcare system suffered a devastating

blow caused by the Ebola Virus Disease (EVD) outbreak between 2014 and 2015. At least 4,800

persons are reported to have died directly due to Ebola in Liberia, with thousands more dead due

to indirect causes such as overburdened healthcare facilities; decreased funding and attention on

other serious conditions like malaria, cholera, and Lassa fever; reduced supplies and infrastructure

in hospitals and clinics; and reduction in health care staff. Liberia was said to be the worst hit by

Ebola among the three affected West African countries (Sierra Leone and Guinea being the other

two), with 300-400 new cases being reported weekly during the peak of the outbreak in August-

September, 2014. On 13 January, 2016, the World Health Organization (WHO) declared Liberia

Ebola-free for the third time, due to the emergence of one or two cases in between. This was after

42 days of no new reported cases, as per WHO guidelines.

8

In terms of the country’s health indicators, although the maternal mortality ratio (MMR) declined

from close to 1000 per 100,000 births in 2007, to an estimated 770 per 100,000 in 2010, current

statistics indicate an increase to 994 per 100,000, possibly due to the Ebola outbreak. Only 46

percent of births are attended by skilled health workers. Approximately 18 percent of children

under five are underweight, and the under-five mortality rate is 114 deaths per 1,000 live births.

Only 59.7 percent one year olds are vaccinated against Measles. Malaria continues to be the

leading cause of mortality for children under age 5 (22 percent in 2013). For all age groups, Lower

Respiratory Infections is the leading cause of death (12 percent), followed by Malaria (8 percent),

Tuberculosis and HIV (6 percent).

Gains, however, remain skewed in favor of urban populations. For example, 63 percent of

deliveries in urban areas are facility-based, compared with 25 percent in rural areas; similarly, 77

percent of urban deliveries are by a skilled service provider compared with only 32 percent of rural

deliveries. While over one in ten children will die before the age of five, infant and under five

mortality rates have almost halved to 71 and 110 per 1,000 births respectively over the last 20

years due to improved access resulting from the Government of Liberia’s (GoL) free health care

policy, and restoration of a number of key child health services like immunizations. Malaria,

however, continues to be a major source of morbidity and mortality; 38 percent of outpatient

attendance and 42 percent of inpatient deaths was attributable to malaria in 2007.

Hospitals in Liberia remain in generally poor physical condition; are understaffed in key areas of

competence; and, have long waiting times and a lack of equipment and drugs. As a consequence,

hospitals in general provide low quality of care. This is reflected in high levels of post-surgery

complications and infection rates; very limited maternal and child death audits; and no systematic

use of clinical guidelines and protocols. Notably, accreditation scores on the quality of services

are worse in secondary vis a vis primary facilities. Poor quality is a particularly critical concern at

the severely resource-constrained hospital-level in Liberia because it can obviate all the implied

benefits of good access and effective treatment, and lead to sub-optimal and wasteful use of

resources.

9

In addition to restoring essential health services disrupted during the EVD crisis, improving the

quality of care at hospitals is a key step in strengthening Liberia’s health system, and ensuring

resilience to future health shocks. Hospitals in Liberia remain in generally poor physical condition;

are understaffed in key areas of competence with insufficient numbers of productive, responsive,

and qualified staff; and, have long waiting times and a lack of equipment and drugs. As a

consequence, hospitals in general provide low quality of care. This is reflected in high levels of

post-surgery complications and infection rates, low quality data on clinical outcomes, very limited

maternal and child death audits, and no systematic use of clinical guidelines and protocols. Poor

quality is a particularly critical concern at the severely resource-constrained hospital-level in

Liberia because it can obviate the implied benefits of good access and effective treatment; frustrate

the positive achievements at the primary health care system by not being able to respond to referral

patients with complications; and lead to sub-optimal and wasteful use of resources.

In order to improve the efficiency, effectiveness, and quality of care at the secondary hospital level,

the country is developing a system to upgrade health worker skills and competencies, and shifting

towards improved provider-accountability for results.

The Post Graduate Medical Council (PGMC) is tasked to develop the Graduate Medical

Residency Programme (GMRP) to facilitate in-country specialization of core MDG-related

hospital-level competencies. Residents will be selected from the existing pool of medical school

graduates based on standardized criteria. The GMRP requires both a critical stream of specialist

faculty to support the program, as well as the upgrading of essential equipment and supplies at

target teaching facilities.

In addition to the development of an MDG-related GMRP, the GoL is also moving towards

provider-accountability for improvements in quality through performance-based financing (PBF)

at target hospitals. The PBF approach is an innovative, results-oriented mechanism to incentivize

providers to achieve agreed-upon, measurable performance targets, with incentives typically

including financial payments, bonuses, other material goods, or public recognition. The shift

towards PBF is influenced by experiences and evidence in a number of high-, middle-, and low-

income countries which show that performance-based approaches can be effectively deployed to:

(i) clearly signal health priorities and ensure that there is adequate focus on corresponding

10

interventions; (ii) ensure that health facilities focus on delivering targeted and cost-effective health

services; (iii) strengthen monitoring and evaluation systems; (iv) empower decision-makers in the

field to set priorities and improve health facilities according to more local needs; (v) motivate staff

to change behavior and improve performance; and, (vi) increase provider autonomy and enhancing

accountability for better results. The latter can stimulate innovations in an effort to overcome

implementation constraints.

A rigorous, randomized controlled evaluation in Rwanda showed a positive impact of PBF at the

primary care level on utilization for institutional deliveries, growth monitoring consultations, and

increased levels of quality of care (Basinga et al., 2011). Rigorous evaluation of PBF programs is

essential for generating solid evidence that can inform the GoL and more broadly other

governments and partners to effectively design and use PBF mechanisms. At this time, solid

evidence on the impact of PBF in a hospital setting is scarce.

A. The Project Development Objective

The project development objective of the Liberia Health Systems Strengthening Project (HSSP) is

to “improve the quality of maternal health, child health, and infectious disease services in selected

secondary-level health facilities.” The project aims to strengthen the institutional capacity needed

to improve Maternal and Child Health (MCH), and infectious disease-related health outcomes at

target facilities through an innovative approach involving systematic and coordinated

improvements to the quality of services delivered at target facilities (through performance-based

financial incentives), and an expansion of health worker skills (through the provision of specialized

training, and the decentralization of specialist availability).

Specifically, the project aims to: (a) improve the quality of care standards (in both diagnosis and

treatment) for services with proven effectiveness; (b) increase the availability of qualified graduate

physicians (pediatricians, obstetricians, general surgeons, and internal medicine specialists, with

cross-cutting focus on anesthesiology); (c) enhance the clinical capabilities and competencies of

mid-level cadres - nurses, midwives, and physician assistants- in emergency obstetrics, surgery,

11

pediatrics, and internal medicine; and, (d) improve provider-accountability mechanisms related to

both the achievement of results, and health-worker performance at selected facilities. These

improvements should provide a thrust towards improved outcomes.

The HSSP is focused on improving: (a) the quality of care (in terms of both diagnosis and

treatment) for key services of proven effectiveness (e.g. maternity, pediatric/ neonatal, surgery,

management hygiene and patient satisfaction, and health worker performance); (b) structural and

management quality (related to, for example, availability of drugs and commodities, and health

facility rehabilitation); and (c) utilization across a limited number of clearly defined services. The

Quality Assessment Tool is outlined in Appendix C (Table 29, Table 30, Table 31). The project

incentivizes an increase in the quantity of services delivered for a only a limited number of services

because, as hospitals, the goal is not to draw patients away from more appropriate locations (e.g.

the primary health care level), but to provide the best care for those patients who need hospital

care. Health facilities will receive 75 percent of incentive payments based on quality

improvements, and twenty-five percent based on improved utilization of incentivized services. The

level of incentives will be adjusted to take into account equity considerations; for example, the

remoteness of a health facility.

Although the PBF aspect of the program focuses on specific services, the levers of the HSSP are

expected to have a much broader impact beyond these targeted services. In particular, the Project

aims to enhance the clinical capabilities and competencies of mid-level cadres - nurses, midwives,

and physician assistants- in emergency obstetrics, surgery, pediatrics, and internal medicine

through the provision of in-service training and direct coaching support; upgrade the equipment

and infrastructure levels at target facilities through both direct investments, and PBF incentives;

increases the motivation of health workers through both improved opportunities for training, and

individual performance-related bonuses; and improve provider-accountability mechanisms related

to the achievement of results, health-worker performance at selected facilities, and improved

access to information. These changes are expected to lead to improvements in the quality of

services and in addition, to increases in utilization for key services.

12

In examining the expected impact of the HSSP, the following set of research questions (RQs) will

be addressed:

RQ1: Did the program work, that is, did the HSSP achieve its goal of improving the quality of

service delivery in target hospitals?

RQ2: How did facilities improve their performance and meet the objective of improved quality?

Did levels of motivation, capacity and competence change as a result of the HSSP?

Which of the intervention levers (management, information, structural improvements, incentives

or training) made an important contribution to the success of the HSS project?

RQ3: What role did existing human resources in the health sector play in the success/failure of the

program? Did existing levels of skills and experience or motivation hinder or facilitate the

interventions?

B. Theory of Change

Figure 1 diagrammatically demonstrates the theory of change linking changes in skills, equipment,

and motivation brought about by the levers of the HSSP. The skills and experience of a health

worker determines their competence. Competence can be increased by training. The equipment

and infrastructure available to a health worker, combined with their competence, determines their

capacity. Capacity can be improved by changes in competence or by structural and supply

improvements to the facility. Finally, health worker effort, combined with capacity, determines the

performance of health workers. Effort is driven by motivation and can be improved through better

management, increased information, and/or improved incentives. Performance improves

measurable outcomes directly and indirectly through improved quality scores across the process

of care and structural and management checklists, and increased utilization of health care services

from patients, who are more likely to trust the services provided.

13

Figure 1 Theoretical Framework: Theory of Change

II. Data Collection Effort

The training and data collection for the baseline evaluation of the HSSP Impact Evaluation took

place from June 2 to July 20, 2015. The objectives of the baseline-evaluation effort were to: (a)

train enumerators on baseline data collection instruments; and (b) conduct baseline data collection

in 10 health facilities (5 intervention hospitals targeted by the performance-based financing (PBF)

program and 5 control hospitals) (c) prepare to begin the continuous data collection phase of the

project in Redemption hospital. The ten hospitals for data collection were: Redemption Hospital,

Liberian Government Hospital, CH Rennie Hospital, St. Francis Hospital, F.J.Grante Hospital,

Performance

(Quality,

Responsiveness,

Skills,

Experience

Equipment,

infrastructure

Motivation

Training

Competence Capacity

Structural and

supply

Improvements

Management,

Information,

Incentives

Baseline level of

health inputs

HSS Levers

Inputs

Measurable

Outcomes

Utilization and

Trust

Effort

Outputs

Outcomes

14

Phebe Hospital, Curren Hospital, Tellewoyan Hospital, G.W.Harley Hospital, and J.F.Doe

Hospital.

In the first week of June, 2015, the mission successfully undertook the training of the 32

enumerators and 2 supervisors mobilized by the MOHSW and World Bank. The training covered

topics including the objectives of the data collection, the ethics of research in the field, the main

guidance rules for quality assurance during data collection, the organization and logistics of the

data collection, and familiarization and practice with the data collection instruments. The

enumerators were recruited and specifically trained to administer certain instruments based on

their area of expertise. This is because a large part of the data collection instruments require

medical training to be successfully administered.

The data-collection was done in two groups for 5 hospitals each, with 16 enumerators and a

supervisor in each group. Each group’s enumerators were divided into 5 teams

(Obstetrics/Gynecology, Outpatient focused on Pediatrics, Surgery, Emergency/Triage, and

Hospital Assessment) to focus on specific data collection instruments based on their medical

training and/or area of expertise. Group 1 visited the following hospitals: F.J. Grante hospital

(Sinoe), St. Francis hospital (Rivercess), C.H.Rennie hospital (Margibi), Liberian Government

Hospital (Bomi), and Redemption hospital (Montserrado). Group 2 visited the following hospitals:

J.F.Doe (Nimba), G.W.Harley (Nimba), Tellewoyan (Lofa), Curren Hospital (Lofa), and Phebe

Hospital (Bong). The data collection schedule for both groups is presented in the appendix as Table

28. At each hospital, the groups spent 5 days for data collection, and often an additional sixth day

to observe staffing on weekends

Each enumerator and supervisor was assigned a unique identifier (ID) and was responsible for

remembering their ID, and recording it on each of the instruments they fill out. A quality check

process was established to ensure that all groups properly filled out all forms properly, to insure

that there was minimal missing data due to non-response or skipped questions. Due to possible

overlap of health worker shifts between the different wards, a protocol was established to ensure

that supervisors and enumerators constantly coordinated and checked to make sure all attendance

sheets, health worker rosters, and consent forms were periodically updated throughout each day.

15

Procurement of several goods and services were completed prior to the baseline survey. In

particular:

Personal Protective Equipment (PPE) was procured for the enumerators. Specifically, scrubs and

boots were procured for the enumerators in the OB and surgical wards, and gloves, masks, and

hand sanitizers were procured for all enumerators and supervisors.

Stationery (pencils, erasers, clipboards) were procured for all enumerators and supervisors.

Transportation to and from hospitals outside of Monrovia for the data collection team was arranged

for both groups. Inside Monrovia, transport was organized to and from Redemption hospital with

the Ministry of Health and Social Welfare (MOHSW) as a pick-up/drop-off point.

Printing and photocopying of all data collection instruments was done at the end of the training,

and then subsequently at the beginning of each hospital data collection where required.

The data collection comprised 33 separate instruments, which were designed to measure the

presence of health workers, their skills, capacity to perform, performance, and motivation. In

addition, the infrastructure and supplies of each facility were also assessed. All baseline survey

instruments were used in an earlier evaluation of the project in the 2013, and have been reviewed

since by the PIs in consultation with the clinical MOHSW staff in order to verify their medical

accuracy and local appropriateness. The instruments were revised accordingly prior to the training

of the enumerators MOHSW. Table 1 shows the breakdown of the services evaluated, and the

types of survey instruments used in each service.

16

Table 1 Summary of Instruments by Service and Location

17

A. Instruments filled

Table 2 shows the number of instruments completed per facility, disaggregated by motivation

surveys, staff registers, infection prevention and control forms, vignettes, direct observation and

exit surveys for each facility, as well as the total number of vignettes administered in each facility.

A total of 3,693 instruments were completed across the ten facilities. Overall, the maximum

number of instruments was collected for FJ Grante and Phebe hospitals. Data collection was lower

at Curran hospital, possibly due to low patient turnout (since the hospital is not free of charge) as

well as limited staff present. Hospital names have been abbreviated in all tables and represent the

following: Jackson F. Doe Hospital (JFD), Redemption Hospital (Redem), Phebe Hospital

(Phebe), St. Francis Hospital (St. F), Tellewoyan Memorial Hospital (Tell), Liberian Government

Hospital (LGH), C.H. Rennie Hospital (CHR), Curran Lutheran Hospital (Curr), F.J. Grante

Hospital (FJG), G.W. Harley Hospital (GWH). The instruments in the first column for Table 2

represent the following: General Observation of the Hospital Infrastructure (F_DO_1), Hospital

Staff Rosters (F_DO_2 and F_DO_3), Staff Motivation Survey (F_V_2), Infection, Prevention

and Control (IPC), Obstetrics Labor and Delivery Observation Checklist (O_DO_1), Obstetrics

Post Natal Care Observation (O_DO_2), Obstetrics Vignettes (O_V_1_A, O_V_1_B, O_V_2_A,

O_V_2_B, O_V_3, O_V_4_A, O_V_4_B), Obstetrics Patient Exit Survey (O_X), Pediatrics

Inpatient and Outpatient Direct Observation (P_DO_1), Emergency and Pediatrics Post Triage

Ward Care Observation (P_DO_2), Pediatrics Vignettes (P_V_1, P_V_2, P_V_3, P_V_4),

Pediatrics Patient Exit Survey (P_X), Surgical Safety Observation Checklist (S_DO_1),

Emergency and Surgery Post Triage Post-Surgical Ward Care Observation (S_DO_2), Surgery

Vignettes (S_V_1_A, S_V_1_B, S_V_2_A, S_V_2_B, S_V_3, S_V_4, S_V_5_A, O_V_5_B),

and Surgery Patient Exit Survey (S_X).

B. Health workers observed

Table 3 shows the total number of health workers (per facility) that participated in the data-

collection effort by being observed, by participating in a vignette or by answering survey questions

about their job. We looked at these numbers according to the credentials of the health workers who

18

were on the rosters of the hospital. We also looked at the proportion present on the first day of the

research at the facility. The rosters were sent by the hospital in advance of the visit and the names

were verified with the director of human resources (or staff responsible for human resources).

Health workers not present on the original rosters were added and names from the roster that were

no longer on the staff were deleted. The proportion present is the response to the question “is this

person present here at the facility today?” and was not independently verified. In general, the

medical officers present in the hospital rosters were not present for many procedures observed in

the four to five days. Physician assistants, midwives, and nurses provided most of the care in the

hospitals. Some of the hospitals had very few specialists and surgeons to perform procedures, often

leading to delays and referrals, for example, GW Harley, FJ Grante, and Liberian Government

Hospital. In addition, many of the staff in the hospitals were performing tasks much higher than

what they were qualified for, indicating an urgent need for more specialized and qualified health

workers.

19

Table 2 Number of Instruments Collected by Facility

JFD Redem Phebe St. F Tell LGH CHR Curr FJG GWH Total

F_DO_1 1 1 1 1 1 1 1 1 1 1 10

F_DO_2 1 1 1 1 1 1 1 1 1 1 10

F_DO_3 1 1 1 1 1 1 1 1 1 1 10

F_V_2 72 106 79 43 53 82 61 41 72 63 672

IPC 10 31 19 19 9 39 35 17 25 25 229

O_DO_1 0 11 8 4 12 6 8 4 11 9 73

O_DO_2 0 18 8 5 10 9 8 4 10 8 80

O_V_1_A 2 6 10 3 4 2 5 4 6 8 50

O_V_1_B 6 4 7 4 7 5 7 5 7 3 55

O_V_2_A 5 6 9 3 5 2 5 4 6 8 53

O_V_2_B 6 4 6 4 6 5 7 5 7 3 53

O_V_3 10 10 8 7 9 7 12 9 13 11 96

O_V_4_A 4 6 7 3 4 2 5 4 6 8 49

O_V_4_B 3 4 6 4 6 5 7 5 7 3 50

O_X 0 32 11 8 16 14 14 9 14 13 131

P_DO_1 72 60 60 65 66 60 61 26 76 40 586

P_DO_2 24 23 20 22 24 20 20 20 21 35 229

P_V_1 11 10 14 7 7 6 7 8 10 9 89

P_V_2 12 10 13 7 7 6 7 8 10 9 89

20

P_V_3 12 10 16 7 7 6 7 8 10 10 93

P_V_4 11 10 16 7 7 6 7 8 10 10 92

P_X 59 60 60 60 55 62 62 12 68 40 538

S_DO_1 12 5 3 2 6 9 6 1 5 13 62

S_DO_2 11 5 3 3 6 10 6 1 5 13 63

S_V_1_A 4 0 3 3 2 0 1 3 3 4 23

S_V_1_B 4 1 2 1 3 3 1 4 2 1 22

S_V_2_A 3 1 2 1 4 3 1 2 2 1 20

S_V_2_B 5 0 2 3 3 0 1 4 3 4 25

S_V_3 7 1 7 4 7 3 2 6 5 3 45

S_V_4 0 1 0 2 1 0 0 1 0 1 6

S_V_5_A 4 0 3 3 3 0 1 3 3 2 22

S_V_5_B 4 1 2 1 4 3 1 2 2 1 21

S_X 7 0 1 3 1 9 14 0 2 10 47

Total 298 299 307 246 292 263 283 170 324 280 3693

21

Table 3 Credentials of Health Workers on Hospital Rosters

JFD Redem Phebe St. F Tell LGH CHR Curr FJG GWH

Cadre N Pres N Pres N Pres N Pres N Pres N Pres N Pres N Pres N Pres N Pres

Certified Midwife 2 50% 27 82% 20 75% 4 100

% 9 22% 7 86%

1

4

100

% 8 63% 4

100

% 5 80%

Registered Nurse 7 43% 47 64% 20 60% 1

0

100

%

2

8 46% 21 76%

1

2 83% 8 38% 20

100

% 8 25%

Registered Nurse /

Certified Midwife 1

100

% 3 33% 2 50% 2 50% 1

100

%

Anesthetist

Registered

Nurse

2 0% 3 100

% 7 29% 1

100

% 2

100

% 1 0% 1

100

% 2 50%

Associate Degree

Nurse 3 67% 12 75% 4 75% 2

100

% 4

100

% 2

100

% 5 20% 1

100

%

Bachelors of Science

Nurse 2

3 39% 38 79% 37 43% 2

100

% 4 75% 4

100

% 4

100

% 5 60% 1

100

%

1

2 50%

Physician Assistant 1 0% 14 86% 3 100

% 4

100

% 8 63% 19 95%

1

5 87% 2 50% 6 83% 8 38%

Medical Doctor 6 67% 4 100

% 55 51% 3

100

% 6 33% 7 71% 4 50% 4

100

% 8 38%

Specialist 6 100

% 1

100

% 2

100

% 6

100

% 3

100

% 2

100

% 2 50% 2

100

% 5

100

%

Other 4

5 44% 69 70%

2

6 73%

3

7 30% 41 76%

3

3 88%

1

8 22% 66 55%

2

6 50%

Total 9

5 47%

21

6 74%

14

9 54%

5

4 87%

9

8 43%

11

0 82%

8

9 87%

4

9 39%

10

4 70%

7

5 51%

22

III. Instrument summary

In this section, we present the summary statistics for each instrument used in the survey by facility and ward type. We also show indices

of inputs, correct treatments and diagnoses, etc. The raw data (i.e. summary statistics for each question) of each instrument is contained

in the appendix.

A. Motivation surveys

Health workers were asked to answer a series of questions about their attitudes and motivations, with Likert options that ranged on a

five-point scale from “Strongly Agree” to “Strongly Disagree”. A factor analysis using varimax rotation was conducted on the full list

of questions showed a number of significant factors in the data.1 Using the cutoff of eigenvalues above 1, we reduced the data to five

factors listed in Table 4, along with the factor loadings for each item. As is always the case with factors, the labels attached are somewhat

arbitrary, however by looking at the items given the largest weights in each of the five factors we can develop simple labels, that,

although not unique, can be used to interpret the results. The factors are described as follows, and in order of the proportion of the data

they explain:

1. I can make a difference.

1 Some of the questions are negative in tone, and the score was inverted so that a higher score always reflected a more positive evaluation. In Table 4, these

questions are indicated with a *.

23

2. This hospital and management provide the right tools.

3. In this organization, I am taken seriously.

4. I am keen to improve.

5. Feedback, teamwork, recognition are important.

Table 4 Motivations Factors and Factor Loadings of Items

Factor 1

I can make

a difference

Factor 2

This

hospital

and

manageme

nt provide

the right

tools

Factor 3

In this

organizatio

n, I am

taken

seriously

Factor 4

I am keen

to improve

Factor 5

Feedback,

teamwork,

recognition

are

important

3.10: Good performance is recognized by our superiors 0.40 0.22 0.082 0.13 -0.095

3.11: This facility has a fair system for rewarding staff 0.19 0.43 0.19 0.081 -0.046

3.12: My performance is appraised regularly 0.30 0.42 0.22 0.082 -0.12

3.13: * Some of the team members work well, yet others do

not and so this facility doesn't perform well overall

0.14 0.21 -0.067 0.20 0.27

3.14: * We do not know how our facility is performing

compared to others in the district

0.18 0.20 -0.055 0.12 0.28

3.15: Our facility has clear goals that we are working towards 0.38 0.26 -0.071 0.33 -0.15

3.16: I understand how my work contributes to the facility's

overall goals

0.42 0.066 -0.13 0.32 -0.24

3.17: I am keen use any new tools to improve my performance 0.31 0.036 -0.24 0.35 -0.22

3.18: This facility has a good reputation in the community 0.35 0.40 -0.011 0.20 -0.051

3.19: This facility provides everything I need to perform well

at work

0.19 0.47 0.17 0.24 -0.035

24

3.20: There are enough health providers to do the work in this

facility

0.18 0.26 -0.019 0.046 -0.14

3.21: * Too often the referral system does not work efficiently 0.092 0.15 -0.057 0.027 0.15

3.22: Maintenance of broken equipment at this facility is

prompt and reliable

0.16 0.50 0.13 0.23 0.097

3.23: * I do not get feedback from my superiors so it is hard to

improve my performance

0.13 0.27 0.028 0.12 0.40

3.24: My job duties and responsibilities are clear and specific 0.34 0.18 -0.073 0.10 -0.042

3.25: Relevant guidelines are easy to access at this facility 0.30 0.33 0.10 0.13 -0.079

3.26: * I often feel left alone when I have to make difficult

decisions about a patient’s care

0.20 0.13 -0.19 0.14 0.21

3.27: I regularly have access to relevant trainings to keep my

skills up to date

0.18 0.23 0.18 0.11 -0.12

3.28: It makes me feel appreciated when patients are grateful 0.42 -0.23 -0.083 0.11 -0.083

3.29: I usually cope well with changes at work 0.25 0.086 -0.013 0.028 -0.12

3.30: * It is difficult for me to speak openly to my superiors

about how things are really going at work

0.21 0.28 0.026 0.17 0.27

3.31: * Suggestions made by health workers on how to

improve the facility are generally ignored

0.16 0.34 0.03 0.18 0.21

3.32: * I intend to leave this facility as soon as I can find

another position

0.20 0.37 0.060 -0.21 0.11

3.33: I would recommend to my children that they choose the

health profession

0.15 0.23 0.17 -0.28 -0.12

3.34: I am willing to put in a great deal of effort to make this

facility successful

0.60 -0.15 -0.15 -0.071 -0.013

3.35: I am proud to be working for this health facility 0.57 0.23 -0.089 -0.23 0.084

3.36: This hospital inspires me to do my best on the job 0.38 0.46 0.10 -0.22 -0.033

3.37: I am proud to tell others that I work in this ward / part of

the hospital

0.59 0.22 -0.094 -0.27 -0.046

3.38: I am glad that I work for this facility rather than other

facilities in the country

0.33 0.38 0.015 -0.36 -0.083

25

3.39: These days I feel motivated to work as hard as I can 0.34 0.43 -0.0016 -0.21 -0.067

3.40: My profession helps me to achieve my goals in life 0.46 0.10 -0.11 -0.21 0.19

3.41: Overall, I am very satisfied with my work in this ward

/part of the hospital

0.53 0.32 0.017 -0.30 -0.034

3.42: I am very satisfied to have a position where I can work

closely with the community

0.45 0.093 -0.16 -0.32 -0.073

3.43: I am satisfied with the opportunity to use my abilities in

my job.

0.55 0.073 -0.26 -0.19 -0.054

3.44:I am punctual about coming to work 0.61 -0.064 -0.14 -0.041 0.056

3.45:I am a hard worker 0.61 -0.18 -0.20 0.035 0.059

3.46:I work hard to make sure that no patient has to wait a long

time before being seen

0.58 -0.17 -0.21 0.13 0.015

3.47: I always complete my tasks efficiently and correctly 0.54 -0.10 -0.09 0.13 0.0027

3.48: When I am not sure how to treat a patient's condition I

look for information or ask for advice

0.61 -0.25 -0.29 -0.037 0.028

3.49: I try to get on well with the other health staff because it

makes the work

0.62 -0.19 -0.32 0.087 -0.040

3.50: I get along well with my superiors at work 0.50 0.13 -0.099 0.0040 0.024

4.10: I feel that I am a person of worth, at least on an equal

plane with others

0.36 -0.071 -0.23 0.079 -0.11

4.11: I feel that I have a number of good qualities 0.53 -0.25 -0.22 -0.013 0.10

4.12: * All in all, I am inclined to feel that I am a failure 0.15 -0.024 -0.011 -0.076 0.21

4.13: I am able to do things as well as most other people 0.47 -0.25 -0.11 0.078 -0.15

4.14: * I feel I do not have much to be proud of 0.079 -0.092 -0.082 -0.0062 0.35

4.15: I take a positive attitude toward myself 0.50 -0.32 0.057 -0.074 -0.028

4.16: On the whole, I am satisfied with myself 0.53 -0.18 0.20 -0.031 0.0025

4.20: In this organization, I am taken seriously 0.41 -0.0041 0.44 0.022 -0.079

4.21: In this organization, I am trusted 0.57 -0.26 0.37 0.063 -0.10

4.22: In this organization, I am important 0.62 -0.33 0.42 -0.047 0.094

4.23: In this organization, I can make a difference 0.65 -0.43 0.35 0.055 0.065

4.24: In this organization, I am valuable 0.62 -0.42 0.37 0.060 0.091

26

4.25: In this organization, I am helpful 0.65 -0.39 0.31 0.021 0.12

Table 5 shows the levels of these scores across the 5 facilities, by ward type. In addition, we show the mean Likert score (with all

negative questions reversed) in Table 6. Overall, the differences between the five facilities are small, but there are differences in the

factors. Curran and Redemption hospitals had the lowest overall factor scores, with especially low scores on Factor 1: “I can make a

difference.” For Curran hospital, one reason for this may be the low patient flow due to hospital fees. At Redemption, it was the opposite

situation with extremely high patient flow and workload, perhaps leading to tiredness and low motivation levels. Staff motivation levels

were highest for JF Doe and St. Francis hospitals. For example, during data collection at St. Francis hospital, the enumerators noted that

although the facility does not have many health workers, the few they have are very hard working and thus need more

motivation/incentives to stay in the hospital. Since JF Doe is a referral hospital mainly for complicated cases, it too has fewer staff, but

mostly specialists who were found to be hardworking, cooperative and motivated.

Table 5 Motivations Factors across Facilities and wards

Ward JFD Redem Phebe St. F Tell LGH CHR Curr FJG GWH Total

Means for Factor 1: I can make a difference

ER 0.25 -2.62 0.15 0.19 0.11 -1.3 -1.67 -7.42 -0.04 -0.3 -1.33

OB 0.64 -1.86 0.6 0.1 0.38 - -3.24 -6.98 0.19 0.56 -0.42

PE 0.49 -2.24 -0.3 -0.16 -0.05 0.56 -2.29 -7.33 -0.22 0.09 -1.03

SU 0.55 -4.87 -0.29 -0.21 0.37 - -0.7 -6.38 -0.33 0.01 -1.39

Total 0.49 -2.84 0.04 -0.08 0.19 0.25 -2.14 -6.96 -0.03 0.13 -1.05


Means for Factor 2: This hospital and management provide the right tools

ER 0.58 1.25 -0.03 0.17 0.35 0.61 0.88 3.14 -0.09 -0.02 0.76

OB 0.15 0.91 0.11 0.06 0.18 - 2.09 3.41 -0.59 0.39 0.47

PE 0.47 1.18 0.2 0.2 0.05 -0.46 1.11 2.88 -0.07 0.23 0.6

27

SU 0.56 2.52 0.1 -0.26 -0.39 - 0.63 3.86 -0.39 0.23 0.86

Total 0.47 1.43 0.11 0.01 0.06 -0.28 1.2 3.38 -0.3 0.23 0.67


Means for Factor 3: In this organization, I am taken seriously

ER 0.07 0.09 -0.52 -0.23 -0.04 0.54 -0.1 1.16 -0.36 -0.02 -0.02

OB 0.11 0.19 -0.08 0.07 -0.03 - 0.61 0.56 -0.06 -0.04 0.08

PE -0.33 0.03 -0.28 0.14 0.02 0.26 -0.02 0.96 -0.2 -0.22 -0.05

SU 0.01 0.53 -0.51 0.26 -0.01 - -0.01 1.19 -0.26 0.35 0.16

Total -0.03 0.18 -0.31 0.12 -0.01 0.31 0.09 1.03 -0.19 0.03 0.04


Means for Factor 4: I am keen to improve

ER 0.72 -0.05 -0.15 -0.27 0.27 0.56 0.16 -0.04 -0.2 0.32 0.09

OB 0.23 0.08 0.05 -0.36 -0.09 - -0.31 0.56 -0.15 -0.03 -0.01

PE 0.64 -0.03 0.07 -0.14 0.34 0.6 0.15 -0.15 -0.06 -0.05 0.11

SU 0.61 0.02 -0.15 -0.57 0.38 - -0.09 0.18 -0.01 0.4 0.1

Total 0.57 -0.01 -0.02 -0.35 0.22 0.6 0.03 0.11 -0.12 0.16 0.08


Means for Factor 5: Feedback, teamwork, recognition

ER 0.14 -0.04 0.44 0.41 0.24 -0.4 0 0.2 -0.11 0.43 0.13

OB 0.04 -0.25 -0.09 0.04 -0.39 - -0.11 -0.34 0.46 -0.14 -0.12

PE -0.04 -0.13 0.01 0 0.26 -0.24 -0.33 0.75 0.24 0.3 0.01

SU 0.22 0.05 -0.41 0.17 -0.1 - 0.05 0.9 0.11 0.04 0.06

Total 0.11 -0.08 -0.04 0.12 0.01 -0.27 -0.14 0.49 0.22 0.13 0.02

28

Table 6 Factor Means by Facility


Factor Avg N Avg N Avg N Avg N Avg N Avg N Avg N Avg N Avg N Avg N

I can make a

difference 0.56 64 -6.8

9

6 -0.15 72 0.61 19 0.19 47 -0.28 80 -2.22 60 -6.9

3

2 -0.03 35 0.13 57

This hospital

and

management

provide the right

tools

0.34 64 3.98 9

6 0.15 72 0.03 19 0.08 47 -0.37 80 1.26 60 3.33

3

2 -0.3 35 0.21 57

In this

organization, I

am taken

seriously

-0.14 64 0.58 9

6 -0.47 72 0.13 19 -0.16 47 -0.42 80 0.1 60 0.99

3

2 -0.19 35 0 57

I am keen to

improve 0.66 64 -0.06

9

6 0.02 72 0.34 19 0.24 47 0.17 80 0.05 60 0.12

3

2 -0.12 35 0.14 57

Feedback,

teamwork,

recognition

0 64 0.02 9

6 -0.18 72 0.21 19 0.09 47 0.08 80 -0.16 60 0.44

3

2 0.22 35 0.08 57

Unweighted

average positive

responses

4.01 71 2.22 9

6 3.72 80 4.01 20 3.9 53 3.62 81 3.26 60 2.1

4

1 3.74 35 3.86 61

29

B. Competence/Capacity Vignettes

To measure competence and capacity, we used vignettes that were designed to test knowledge,

and to simultaneously assess the availability of equipment and materials that would allow health

workers to use that knowledge. Thus, many of the vignettes administered measure competence and

capacity, where capacity is equal to knowledge, after subtracting the things that health workers

knew how to do, but could not do because the required equipment was not available or not

functioning.

In addition, vignettes form an essential part of our identification strategy because they demonstrate

the link between key health inputs (physical examination for example) and important outcomes. If

we can demonstrate that a health worker who does more in trying to get to the correct diagnosis,

is indeed, more likely to obtain the correct diagnosis, then when we see health workers doing more

in practice with their patients, we can know that this will lead to probabilistically better outcomes.

1. Facility and health worker type averages for vignettes

Table 7 shows the average scores on inputs, diagnosis, and treatment, by vignette type (Obstetrics,

Pediatrics, or Surgery). Some of the scores are available only for a subset of vignettes, so the total

number of observations is not constant for all facilities. For many of the vignettes, the health

worker was attempting to obtain the correct diagnosis or treatment, and did so by using history

taking, physical examination, and laboratory tests. Each of these vignettes was associated with a

list of possible history-taking questions, physical-exam procedures and laboratory tests, with

health workers receiving a higher score if they used more of these items. These inputs make more

sense in the pediatric and surgical vignettes where health workers are trying to assess the health

worker’s condition. However, for the obstetric vignettes, the inputs are less standardized and

therefore difficult to summarize.

30

Table 7 Average Vignette Scores by Inputs, Diagnosis and Treatment

History Physical Correct

Assessment

Partial Correct

Diagnosis

Completely

Correct Diagnosis

Correct Treatment

N Mean N Mean N Mean N Mean N Mean N Mean

OB 0 - 0 - 59 0.64 61 0.39 61 0.3 61 0.67

Peds 301 0.33 301 0.39 0 - 212 0.36 212 0.54 176 0.65

Surg 169 0.45 169 0.52 125 0.41 169 0.66 85 0.34 104 0.21

Total 470 0.37 470 0.44 184 0.48 442 0.48 358 0.45 341 0.52

31

For OB, we do not have any input scores, however, OB health workers have higher average scores

for correct assessment and correct treatment but lower correct diagnosis scores, compared to

Pediatrics and Surgery. For Surgery vignettes, the average correct treatment score was lowest

compared to those for health workers completing Pediatrics and OB vignettes. This may in part be

due to the fact that there were very few specialist surgeon staff available to complete vignettes,

and some of the surgery vignettes were completed by GPs or qualified nursing staff, who may have

gotten some of the more complex cases wrong, reflecting the need for more highly skilled and

qualified surgeons in the hospitals.

To better understand the differences across facilities and across health worker types in the outputs

of the vignettes, we show the result of an analysis of each score by both facility and cadre type.

The results are presented in two sets of tables and graphs, but they come from the same regression.

Table 8 presents the probability of the correct assessment, partially correct diagnosis, completely

correct diagnosis, and treatment for each facility with JF Doe as the reference or control hospital.

This table includes the surgical, pediatric, and obstetric simulated patient vignettes. The same

results are shown graphically in Figure 2.

The results indicate that, compared to JF Doe, in terms of ability to do the assessment correctly,

Tellewoyan was the best performing and FJ Grante was the worst performing. In terms of

achieving partially or completely correct diagnoses, almost all the hospitals performed worse than

JF Doe, with the exception of Tellewoyan, Curran, and CH Rennie hospitals, although the slight

differences between them and JF Doe were not statistically significant. Health workers from

Phebe, St. Francis, Redemption, Tellewoyan, and Bomi, all performed statistically significantly

worse than JF Doe hospital, in terms of making correct diagnosis, signaling the need for better

training in screening and diagnosis for healthcare professionals. For doing the correct treatment as

well, almost all the hospitals performed worse than JF Doe, with the exception of CH Rennie.

Health workers from Phebe, St. Francis, Redemption, Tellewoyan, and Bomi, Curran, and GW

Harley, all performed statistically significantly worse than JF Doe hospital, in terms of going about

the correct treatment, revealing a serious gap to be addressed.

32

Table 8 Regression coefficients for vignette scores by facility for assessment, diagnosis, and treatment factors

Assessment Partially Correct

Diagnosis

Completely

Correct Diagnosis

Treatment

b (SE) b (SE) b (SE) b (SE)

JF Doe (ref) (ref) (ref) (ref)

Redemption -0.041 (0.087) -0.025 (0.062) -0.122 (0.061)** -0.202 (0.088)**

Phebe 0.06 (0.088) -0.108 (0.060)* -0.021 (0.060) -0.188 (0.089)**

St. Francis -0.139 (0.142) -0.17 (0.099)* -0.03 (0.102) -0.234 (0.125)*

Tellewoyan 0.126 (0.086) 0.044 (0.057) -0.14 (0.059)** -0.266 (0.082)***

Bomi 0.038 (0.168) -0.046 (0.123) -0.172 (0.103)* -0.207 (0.128)

CH Rennie -0.097 (0.170) -0.151 (0.122) 0.031 (0.121) 0.065 (0.167)

Curran -0.103 (0.141) 0.059 (0.092) -0.054 (0.103) -0.247 (0.124)**

FJ Grante -0.207 (0.112)* -0.036 (0.099) -0.147 (0.092) -0.179 (0.128)

GW Harley 0.084 (0.138) -0.047 (0.101) -0.1 (0.101) -0.371 (0.090)***

Note: *p<0.05, **p<0.01, ***p<0.001

Figure 2 Probability of Being Correct Across All Vignettes by Facility

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Assessment Partial Diag Complete Diag Treatment

33

Error! Not a valid bookmark self-reference. presents the same results but with health worker

category as predictors, with the values also shown in Figure 3. Nurses with bachelors or the

equivalent served as the reference category. The other two categories were 1) Medical Doctors

(MDs) including specialists, general practitioners and medical students and 2) Registered Nurses

(RN), including nurse’s aides, and nursing certificate and diploma holders. We found that MDs

made statistically significantly greater correct diagnoses and treatment calls compared to their

nurse counterpart, as expected due to their greater years in specialized training. However, RNs

also made statistically significantly more partially correct diagnosis compared to nurses, but none

of the other outcomes were significantly different between the two nursing categories.

Table 9 Regression Coefficients for Vignettes by Health Worker Type for Assessment, Diagnosis, and Treatment Factors

Assessment Partially Correct

Diagnosis

Completely

Correct Diagnosis

Treatment


Nurse (ref) (ref) (ref) (ref)

MD 0.071 0.451*** 0.209*** 0.223**

RN -0.09 0.128*** -0.041 0.025

Note: *p<0.05, **p<0.01, ***p<0.001

Figure 3 Probability of Being Correct by Health Worker Type

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Assessment Partial Diag Complete Diag Treatment

MD RN Nurse

34

2. Validating the theory of vignettes

The superior training of MDs (and of RNs) might be the reason that they achieve better outcomes

in these vignettes, but Figure 4 shows that their superior knowledge and training is not the direct

cause of their better outcomes. In particular, MDs provide more history taking and physical

examination than either RNs or other nurses.

Figure 4 Proportion of Possible Inputs into Correct Diagnosis and Treatment by Health Worker Type

Table 10 presents the results of the regression analyses with inputs (history taking, physical, and

lab tests) as predictors, and correct assessment, diagnosis, and treatment as outcomes, across all

vignettes. We see a similar pattern, except now there is no strong relationship between inputs and

the ability to obtain the correct general diagnosis. Overall physical examination is important for

the detailed diagnosis and treatment, whereas history taking is important for the partial diagnosis

of the condition.

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

History Physical Lab Tests

MD RN Nurse

35

Table 10 Relationship between Inputs, Diagnosis, Detailed Diagnosis, and Treatment

Assessment Partial Correct

Diagnosis

Completely

Correct Diagnosis

Treatment


History 0.069 (0.113) 0.217 (0.092)** 0.077 (0.082) 0.143 (0.131)

Physical 0.088 (0.115) 0.241 (0.090)*** 0.305 (0.073)*** 0.244 (0.121)**

Lab tests 0.027 (0.174) -0.194 (0.161) 0.162 (0.134) 0.081 (0.191)

N 324 403 403 265

As we can see in Figure 5 and Figure 6, the increased use of key inputs improves the probability

of obtaining the correct diagnosis and the use of physical examination increases the probability of

the correct treatment. It is not the case that training allows one to shortcut the diagnosis process

and get to the correct diagnosis quicker; training teaches the health workers how to properly use

physical examination, history taking and laboratory tests to obtain the correct diagnoses and to

prescribe the correct treatment.

Figure 5 Association between the Use of Key Inputs and Correct Diagnosis

.4.4

5.5

.55

.6.6

5

Pro

ba

bili

ty o

f co

rrect d

iagn

osis

0 .2 .4 .6 .8 1Proportion of Correct Inputs

History Taking Physical Examination

HT + PE

Relationship of Correct Diagnosis and Inputs

36

Figure 6 Association between the Use of Physical Examination and Correct Treatment

.35

.4.4

5.5

.55

Pro

ba

bili

ty o

f co

rrect tr

ea

tme

nt

0 .2 .4 .6 .8 1Proportion of Correct Inputs

History Taking Physical Examination

HT + PE

Relationship of Correct Treatment and Inputs

37

C. Direct observation

The researchers observed health workers in four different settings, the obstetric ward, the pediatric

ward, the emergency ward and the surgical ward. In these settings, the activities of health workers

were recorded with three different sets of instruments: obstetric, pediatric, and surgical.

Table 11 examines markers of quality in obstetric care. Not all patients were observed in all stages,

and of course, not all patients experienced complications, so the numbers of observations are

relatively low. We do not expect to observe many cases on neonatal asphyxiation, for example,

which is one of the reasons why vignettes were designed to test for health worker responses in

these cases. The variance is quite high for different aspects of care. In general, the quality of care

is much higher when there are complications such as pre-eclampsia, across all hospitals. On the

other hand, the use of the partograph is generally low, with proper use of partograph being

inadequate or inappropriately timed (in some cases, after the delivery). There are not enough

observations to draw strong conclusions about the differences between hospitals, but it seems clear

that there is ample opportunity to improve quality in all hospitals.

Table 12 examines the quality of care in pediatric care, both in the pediatric outpatient ward and

the emergency ward. There are significantly more observations for these instruments allowing us

to make stronger statements about the difference between hospitals. However, average

observation scores on different aspects of pediatric care were fairly similar, with little variation.

Table 13 examines key features of the few surgical procedures observed. Evaluating the quality of

a surgical process is difficult, particularly since most of them are quite different from each other.

The table therefore focuses on certain key markers in the preparation for surgery, including items

such as politeness, the use of a surgical check list and the proper administration of antibiotic

prophylaxis before the incision.

38

Table 11 Direct Observation of Obstetric Care

Facility


Avg N Avg N Avg N Avg N Av

g

N Avg N Avg N Avg N Avg N Avg N

Obstetric Intake:

History Taking

1 2 0.33 6 0.96 6 0 4 0.75 12 0.36 3 0.13 3 0.97 2 0.16 4 0.09 6

Obstetric Intake:

Presence of

Complications

0.06 2 0.33 3 0.16 6 1 3 0.19 10 3.13 3 6 3 0.44 2 1.63 4 5.9 5

Obstetric Intake:

Overall Quality

0.6 2 0.44 6 0.66 6 0.29 4 0.59 12 0.39 3 4.06 3 0.61 2 0.33 4 0.47 6

First Stage of Labor

Observation

Quality

0.7 2 0.64 1

6

0.73 13 1.68 8 0.56 14 1.09 4 7.13 5 0.57 4 0.48 5 0.9 8

Second and third

stage observation

quality

0.75 4 0.83 2

2

0.88 15 0.75 11 0.94 19 1.96 4 1.36 5 1.03 4 0.78 8 0.98 8

Episiotomy:

Usually

unnecessary

0 4 0.09 2

2

0.13 15 0.18 11 0.05 19 0 4 0.4 5 0 4 0.13 8 0 8

Newborn

Resuscitation

Quality

0 0.68 5 0.92 1 0.78 2 0.87 1 0.87 1 0 0 0.29 2 0.8 1

Potentially Harmful

Practices

0 4 0.08 2

5

0 17 0.05 12 0.02 19 0.01 6 0.05 8 0 4 0.09 12 0.01 8

Partograph Quality 0 4 0.49 2

4

0.67 17 0.73 12 0.79 19 0.15 6 1.08 8 0 4 0.42 12 0.71 8

Filling Partograph

After Delivery

0 0.67 1

5

0.83 12 0.67 9 1.88 8 1 1 0.57 7 0 0.33 6 1 6

Obstetric Care PPH

Quality

0 1 1 0 0.88 1 0.88 1 0 0 0 0.86 1

39

Obstetric Care: Pre-

Eclampsia Quality

1 4 0.97 2

7

0.97 17 0.95 13 0.97 21 1 6 1 8 1 4 1 12 1 9

Table 12 Direct Observation of Pediatric Care

Facility


Avg N Av

g

N Av

g

N Avg N Avg N Avg N Avg N Avg N Avg N Avg N

Pediatric

Observatio

n Physical

Examinatio

n Score

0.48 151 0.37 120 0.41 111 0.41 118 0.2 124 0.29 59 0.36 60 0.38 26 0.29 74 0.24 74

Pediatric

Observatio

n History

Taking and

Physical

Exam

0.41 151 0.35 121 0.36 113 0.38 118 0.19 125 0.28 59 0.37 60 0.36 26 0.27 74 0.23 74

Pediatric

Observatio

n Health

Education

Score

0.29 124 0.36 112 0.34 90 0.34 107 0.27 104 0.31 46 0.44 56 0.23 22 0.39 55 0.27 67

Pediatric

Observatio

n General

Effort

Score

0.25 139 0.22 106 0.26 106 0.17 79 0.24 100 0.17 57 0.18 45 0.21 27 0.22 71 0.22 56

Pediatric

Observatio

n Physical

Examinatio

n Score

0.48 151 0.37 120 0.41 111 0.41 118 0.2 124 0.29 59 0.36 60 0.38 26 0.29 74 0.24 74

40

Table 13 Direct Observation of Surgical Care

Facility


avg N avg N avg N avg N avg N avg N avg N avg N avg N avg N

cleanliness of operating

theater 0.92 27 0.9 16 0.92 14 0.56 11 0.83 7 0.8 9 0.77 6 0.56 1 0.5 5 0.8 13

Staff was polite,

respectful 1 24 0.94 18 1 12 0.64 11 1 6 1 9 1 6 1 1 0.8 5 1 13

Preparation of materials

pre-surgery 0.83 25 0.68 18 0.86 12 0.7 12 0.86 6 0.71 9 0.77 6 0.91 1 1.23 5 0.78 13

Preparation of the

primary surgeon 0.87 49 0.59 28 0.82 27 0.77 22 0.76 13 0.59 18 0.67 12 0.81 2 0.65 10 0.82 25

Surgeon was polite,

respectful 1 24 0.94 18 1 12 0.64 11 1 6 1 9 1 6 1 1 0.8 5 1 13

use of surgical check list 0.16 58 0 34 0.04 27 0 25 0.29 14 0 18 0 12 0.5 2 0.1 10 0.04 25

antibiotic prophylaxis

given or confirmed 0.36 25 0.27 15 0.14 14 0 9 0.43 7 0.11 9 0 6 1 1 0.2 5 0 12

antibiotic prophylaxis

known given within 60

minutes of incision

0.08 25 0 15 0 14 0 9 0 7 0.11 9 0 6 0 1 0 5 0 12

General surgical Quality

(before incision) 0.77 27 0.61 15 0.74 14 0.65 9 0.78 7 0.48 9 0.53 6 0.92 1 0.6 5 0.71 13

The Surgeon broke the

surgical field during

surgery

0.1 58 0 34 0.04 27 0.08 25 0 14 0.06 18 0.08 12 0 2 0 10 0 25

Medical Staff Post

Surgery Clean Up 0.61 24 0.45 15 0.56 12 0.41 11 0.7 7 0.5 9 0.66 6 0.69 1 0.71 5 0.74 12

Housekeeping Staff Post

Surgery Clean Up 0.68 25 0.65 13 0.74 12 0.32 11 0.89 7 0.64 9 0.82 6 1 1 0.77 5 0.92 12

41

D. Exit surveys

Patients were also interviewed about their experiences and observations of the hospitals. Table 14

shows the socio-demographic characteristics of the patient sample by facility and ward (pediatrics,

obstetrics, and surgery).

42

Table 14 Patient Characteristics

Facility


Avg N Avg N Avg N Avg N Avg N Avg N Avg N Avg N Avg N Avg N

Age 26 125 27.29 180 25.68 154 29.16 143 26.77 146 27.34 77 27.64 80 17.59 22 29.92 83 21.78 65

Patients without

any schooling

0.33 146 0.24 199 0.37 164 0.39 155 0.42 158 0.34 85 0.34 88 0.27 22 0.3 86 0.18 74

Patients completed

primary schooling

0.32 146 0.22 199 0.32 164 0.25 155 0.22 158 0.22 85 0.32 88 0.5 22 0.26 86 0.32 74

Patients completed

secondary schooling

0.33 146 0.45 199 0.24 164 0.31 155 0.28 158 0.34 85 0.25 88 0.23 22 0.35 86 0.42 74

Patients completed

vocational training

0 146 0.02 199 0 164 0.01 155 0.01 158 0.02 85 0.01 88 0 22 0.05 86 0.04 74

Patients completed

post-secondary schooling

0.02 146 0.06 199 0.04 164 0.02 155 0.07 158 0.02 85 0.02 88 0 22 0.02 86 0.01 74

Patients reporting

being able to read easily

0.26 146 0.4 199 0.18 164 0.26 155 0.23 158 0.22 85 0.22 88 0.32 22 0.31 86 0.36 74

Patients reporting being

able to read with difficulty

0.32 146 0.29 199 0.39 164 0.3 155 0.29 158 0.35 85 0.39 88 0.32 22 0.26 86 0.39 74

Patients reporting not

Being able to read at all

0.42 146 0.29 199 0.43 164 0.44 155 0.44 158 0.4 85 0.4 88 0.36 22 0.42 86 0.23 74

Travel Time: Less than

30 minutes

0.17 146 0.27 199 0.18 164 0.23 155 0.34 158 0.38 85 0.36 88 0.36 22 0.53 86 0.51 74

Travel time:

30 - 60 minutes

0.12 146 0.22 199 0.16 164 0.14 155 0.09 158 0.32 85 0.4 88 0.23 22 0.33 86 0.16 74

Travel Time: More

than 60 Minutes

0.21 146 0.1 199 0.17 164 0.17 155 0.13 158 0.28 85 0.19 88 0.36 22 0.13 86 0.19 74

Index of Assets

Owned

-0.15 145 0.89 197 -0.33 164 -0.17 152 -0.19 158 -0.45 83 -0.04 88 -0.1 22 0.12 85 -0.16 72

43

Table 15 shows the responses given by facility about general satisfaction across the obstetric,

pediatric and surgery/emergency services. Respondents scored a number of characteristics on a

Likert score and were asked, for each characteristic, if it was important to them. Likert options

ranged on a five-point scale from “Strongly Agree” to “Strongly Disagree”.

44

Table 15 Satisfaction As Expressed Across All Three Types of Exit Surveys

Facility



It is convenient to

travel from your house

to the health facility.

2.67 85 3.78 134 2.98 107 3.53 95 3.43 103 3.9 21 3.41 22 3.89 9 3.13 15 3.78 23

The health facility is

clean. 4.24 85 4.02 134 3.93 107 3.86 95 4.17 103 4.24 21 4.41 22 4.11 9 3.2 15 3.91 23

The health staff is

courteous and

respectful.

4.01 85 4.01 134 3.93 107 3.88 95 4.15 103 4.71 21 4.59 22 4.22 9 3.53 15 4.26 23

The health workers did

a good job of

explaining your

condition.

3.82 85 3.6 134 3.64 107 3.65 95 3.81 103 4.19 21 4.64 22 4.22 9 3.13 15 4.17 23

It is easy to get

medicine that health

workers prescribe.

3.47 85 3.84 134 3.31 107 3.56 95 3.32 103 3.95 21 4.05 22 3.22 9 3.53 15 3.61 23

The amount of time

you spent waiting to be

seen by a health

worker was not

3.04 85 3.72 134 3.46 107 3.27 95 3.69 103 3.76 21 4 22 4.22 9 3.07 15 3.52 23

You had enough

privacy during your

visit.

3.94 85 3.46 134 3.82 107 3.59 95 3.93 103 3.95 21 4.5 22 4.22 9 3.07 15 3.57 23

The health worker

spent a sufficient

amount of time with

you.

3.85 85 3.98 134 3.73 107 3.72 95 3.96 103 4.33 21 4.32 22 4.33 9 2.93 15 3.65 23

45

The overall quality of

services provided was

satisfactory.

3.95 146 4.01 195 3.89 164 3.95 154 4.01 158 4.01 83 4.12 84 4.05 22 3.68 84 3.75 73

You felt that the

facility maintained

confidentiality of your

personal info

3.92 146 3.99 195 3.93 164 3.91 154 3.98 158 4.01 83 4.07 84 4 22 3.74 84 3.85 73

You were asked for

consent (permission)

before any procedures.

3.36 145 3.43 195 3.38 164 3.36 154 3.25 158 3.86 83 3.86 84 3.32 22 3.61 84 3.01 73

You felt respected by

your healthcare

provider.

4 144 4.08 195 4.01 164 3.99 154 4.11 158 4.08 83 4.15 84 4.05 22 3.92 84 4.03 73

You had a warm and

compassionate

healthcare provider.

3.91 145 4.1 195 3.99 164 4.03 154 4.04 158 4.1 83 4.15 84 4.09 22 3.94 84 3.92 72

46

Using factor analysis, we reduced this list of responses to two significant factors: quality and

convenience. The factor scores are presented in Table 16. The summary scores for these factors

are reported in Table 17, alongside the average Likert score. Note that the overall Likert score

(which shows little variation) masks the important differences in the two factors of quality and

convenience. For example J.F. Doe is ranked as much higher than St. Francis in quality, but the

facilities are nearly opposite in convenience. Note also that the ranking of facilities by health

workers and patients is not the same. Health workers were the most satisfied at Liberian

Government Hospital, followed by GW Harley, FJ Grante, Tellewoyan, JF Doe, and Redemption

(all attaining positive overall satisfaction scores). These were followed by St. Francis, Curran,

Phebe, and finally CH Rennie, with all these hospitals having negative overall satisfaction scores.

Table 16 Factor Loading for Patient Exit Interviews

Factor1:

Quality

Factor2:

Convenience

It is convenient to travel from your house to the

health facility.

0.77 0.31

The health facility is clean. 0.91 -0.077

The health staff is courteous and respectful. 0.95 -0.14

The health workers did a good job of explaining

your condition.

0.88 0.07

It is easy to get medicine that health workers

prescribe.

0.84 0.24

The amount of time you spent waiting to be seen by

a health worker was not

0.80 0.30

You had enough privacy during your visit. 0.84 0.01

The health worker spent a sufficient amount of time

with you.

0.90 0.01

The overall quality of services provided was

satisfactory.

0.94 -0.13

You felt that the facility maintained confidentiality

of your personal info

0.90 -0.15

You were asked for consent (permission) before

any procedures.

0.77 0.12

You felt respected by your healthcare provider. 0.95 -0.21

You had a warm and compassionate healthcare

provider.

0.95 -0.20

47

Table 17 Satisfaction Summary Scores

Facility



Quality 0.16 85 0.27 134 0.03 107 -0.07 95 0.41 103 1.61 21 1.46 22 0.87 9 -1.61 15 0.52 22

Convenie

nce -0.18 85 0.31 134 -0.15 107 0.07 95 -0.01 103 0.05 21 -0.04 22 -0.01 9 0.14 15 0.11 22

Average

Likert

scale score 3.73 146 3.88 195 3.73 164 3.78 154 3.82 158 3.96 83 4.06 84 3.91 22 3.76 84 3.67 73

Important:

overall

satisfactio

n 0.09 78 0.08 134 -0.14 106 -0.01 92 0.11 102 1.6 12 -0.15 10 -0.13 9 0.75 13 1.49 13

Important:

individual

attention 0.66 78 -0.2 134 -0.25 106 -0.01 92 -0.17 102 -0.49 12 -0.01 10 -0.51 9 -0.31 13 -0.38 13

48

Table 18 Satisfaction Summary Scores by Facility and Ward


Means for Factor 1: Quality

SU 0.11 0.18 -0.1 0.3 0.43 1.87 2.08 - -1.34 -0.2 0.49

OB 1.18 0.7 0.12 0.37 1.02 1.41 0.72 0.87 -1.65 1.12 0.58

PE 0.01 -0.12 0.01 -0.32 0.1 - - - - - -0.06

Total 0.16 0.27 0.03 -0.07 0.41 1.61 1.46 0.87 -1.61 0.52 0.25

Means for Factor 2: Convenience

SU -0.25 -0.34 -0.43 -0.17 -0.27 -0.18 -0.39 - 0.74 -0.17 -0.26

OB 0.36 0.61 0.09 0.15 0.22 0.22 0.38 -0.01 0.05 0.34 0.3

PE -0.24 0.24 -0.22 0.08 -0.08 - - - - - -0.05

Total -0.18 0.31 -0.15 0.07 -0.01 0.05 -0.04 -0.01 0.14 0.11 0.03

Means Total Score

SU 3.76 3.78 3.56 3.83 3.79 4.29 4.46 - 3.31 3.61 3.86

OB 4.09 3.98 3.71 3.87 4.07 4.23 4.02 4.04 3.24 4.12 3.93

PE 3.7 3.85 3.74 3.76 3.76 3.86 3.99 3.82 3.88 3.56 3.78

Total 3.73 3.88 3.73 3.78 3.82 3.96 4.06 3.91 3.76 3.67 3.82

Mean Importance Factor 1: Overall Satisfaction

SU -0.03 0.03 -0.06 -0.01 0.05 - - - - - 0

OB -0.23 0.21 -0.43 -0.29 0.41 1.6 -0.15 -0.13 0.75 1.49 0.24

49

PE 0.17 -0.02 -0.05 0.09 -0.04 - - - - - 0.03

Total 0.09 0.08 -0.14 -0.01 0.11 1.6 -0.15 -0.13 0.75 1.49 0.1

Mean Importance Factor 2: Respect and Warmth

SU -0.05 0.11 -0.16 0.09 0.31 - - - - - 0.08

OB -0.54 -0.46 -0.7 -0.35 -0.58 -0.49 -0.01 -0.51 -0.31 -0.38 -0.47

PE 1.01 -0.04 -0.09 0.11 -0.05 - - - - - 0.17

Total 0.66 -0.2 -0.25 -0.01 -0.17 -0.49 -0.01 -0.51 -0.31 -0.38 -0.07

50

I. Impact Evaluation Strategy

The HSSP will take place in five pre-selected hospitals. Given the small number of hospitals in

Liberia, it was determined at early stages that a full randomized controlled trial was not feasible.

Thus, for the evaluation at the hospital level, the impact evaluation takes the project as a given,

without control over the selection of treated or control hospitals, the timing, or structure of any

interventions. Five control facilities have been chosen to match the treated hospitals. The original

criteria were that hospitals be drawn from urban and rural areas, from different regions of Liberia

and from public and private services. Thus, each control facility is matched to the paired treatment

facility based on these characteristics. Facilities were not selected on the basis of initial quality

levels or on capacity to absorb the Project.

These restrictions on the number of hospitals suggest a simple before-and-after, matched pair

comparison using measurable outcomes directly and indirectly tied to the project. However, with

only five units in treatment and control groups, this strategy has very limited power to detect

statistically significant improvements in outcomes. Although we will report these outcomes, the

evaluation strategy focuses, instead, on the theory of change within treated facilities and on a series

of augmented PBF interventions at the health-worker level within treated facilities. The external

validity of the Project as a whole is limited because the design is specific to Liberia, but the lessons

about how things change within hospitals should have more generalizable findings.

IV. Evaluating the theory of change within

HSSP facilities

In order to better understand how PBF, as defined by the HSSP, is functioning within hospitals,

we propose to collect the data necessary to measure “the three gaps” for health workers in the

treated and control facilities.

A. The Three Gap Framework

51

The three-gap framework focuses on four levels of care: 1) the competence to perform; 2) the

capacity to perform; 3) actual performance; and 4) the target levels of performance. The three gaps

defined by these four levels are the know gap, the know-can gap and the can-do gap. Figure 7

graphically demonstrates the three-gap framework. Performance originates with knowledge,

education, and skills as measured by competence (C), shown on the vertical axis extending below

the origin: increases in competence are shown by points closer to the bottom of the figure (further

from the origin). Capacity (K) comes from competence, taking into account the infrastructure,

equipment, and medicines needed to appropriately use training, education, and skill, shown on the

horizontal axis to the right of the origin. Finally, performance (P) comes from taking capacity and

combining it with effort: health workers must choose to use their knowledge and equipment in

order to perform, shown on the vertical axis above the origin.

An important feature of this model is the fact that performance is limited by capacity, and that

capacity is limited by competence: a health worker cannot do better than what he or she knows

how to do, for example. This is shown in the graph by what we call the capacity and performance

barriers. In the translation from competence to capacity, and again from capacity to performance,

we take into account the capacity barrier and performance barrier, shown as 45-degree lines from

the origin in the lower right and upper right quadrants. These barriers reflect the fact that better

equipment cannot produce capacity beyond the level of competence, and that greater effort cannot

produce performance beyond the level of capacity. In other words, competence limits capacity,

and capacity limits performance.

52

Figure 7 The Three-Gap Framework

Target competence is the level required to perform according to the target level of performance,

marked in Figure 1 as CT. In the ideal world, any health worker with target competence would also

have target capacity KT: if health workers had all the equipment and medicines to work according

to their training, capacity and competence would be the same. In addition, this capacity (KT) would

ideally translate directly into targeted performance, PT.

The idea of the three-gap framework is that, in the real world, not all health workers have target

competence, competence does not always translate into capacity and capacity does not always

translate into performance. In addition to this ideal triplet (CT, KT, PT), Figure 7 shows an example

of another possible triplet (C, K, P). Competence (C) is lower than targeted performance, and

because some equipment or medicine is not available, this particular health worker has capacity

(K), which is lower than his competence. This is shown by the pair (K, C) in the lower right

quadrant and the fact that (K, C) is to the left of the capacity barrier. Furthermore, because effort

is not ideal, the health worker does not fully transform capacity into performance (P), as shown by

the pair (K, P) in the upper right quadrant. Thus, performance is significantly below target

performance. Importantly, we can divide the shortfall into three gaps:

53

1) The know gap (shown as G1), which is the difference between targeted performance and the

competence to perform;

2) The know-do gap (shown as G2), which is the difference between competence and capacity; and

3) The can-do gap (shown as G3), which is the difference between capacity and performance.

Note that these gaps are important for policy reasons. A large know-gap suggests deficiencies in

training. A large know-do gap suggests deficiencies in infrastructure, equipment, or medicines,

and a large can-do gap suggests deficiencies in motivation. However, the size of the gaps by

themselves is not enough to understand the potential gains from changes in policy. In order to do

this, we need to understand the relationship between competence, capacity, and performance.

1. The Three Gaps in our Sample

Figure 8 shows the actual pattern of the three gap framework across all the facilities in the sample

in pediatric care. We focus on pediatric care because the correspondence between competence and

capacity vignettes and performance observations is the most straightforward. In other words, there

are a large number of items for which we can observe whether or not a health worker would

perform the action if presented with a case study vignette (competence), whether or not they have

the equipment present that would allow them to perform the action in practice (capacity) and

whether or not the same health worker does perform the action in practice. In addition, most care

in pediatrics can be linked to one health worker, whereas in obstetrics, in particular, multiple health

workers are responsible for the same patient making it more difficult to link three types of outputs

across one health worker.

Note that Figure 8 shows all health workers measured across all facilities, so we observe the widest

possible variety of abilities. What is readily apparent in this graph is that the competence capacity

gap is not a major factor. Except at the lowest end of competence, health workers are not limited

by the availability of equipment and the line connecting competence and capacity is not

statistically different from the 45 degree line representing a one to one mapping of competence

onto capacity. On the other hand the relationship between capacity and performance is quite

different. Except that it slopes slightly upwards (greater capacity leads to greater performance) the

relationship between these two measures is very week. In fact, health workers with very low levels

54

of capacity actually perform at levels above that expected for the competence, and health workers

with very high levels of capacity perform at levels significantly below their level of capacity.

Figure 8 The Three Gap Framework for our sample (pediatrics)

Overall, this suggests that across all facilities, performance is a matter of habit, not a matter of

deliberate application of knowledge. More importantly, it suggests that improvement in

competence or capacity will not automatically result in any improvements in performance. Even

when competence is low, it is not the limiting factor in performance.

2. Analyzing the three gaps by facility

Figure 9 shows the size of four gaps for each facility in the sample. The four gaps we examined

are the target to performance gap (the target-do gap), the target-know gap, the know-can gap and

the can-do gap. Recall that the overall gap between the targeted behavior and the actual behavior

can be broken up into the three subsequent gaps. In general it is clear that the target-know gap is

the largest gap: health worker knowledge of what they should be doing (as demonstrated on the

55

vignettes) is very low. Recall that MDs do better on vignettes than other cadres, and recall that in

general a very small proportion of all health care in a hospital is directly provided by an MD.

Importantly the know-can gap is very small across all facilities. This shows that health workers

have the capacity to do what they know how to do: they are not obviously lacking in equipment or

medicines. The can-do gap is larger than the know-do gap and demonstrates some interesting

variation. St Francis and Curran hospitals, for example, demonstrate a very low level of the can-

do gap: health workers in these hospitals do what they can do. On the other hand the can-do gap

at Redemption is much larger. In fact, redemption hospital demonstrates the smallest target-

knowledge gap as well as the largest can-do gap: they know much more than other hospitals but

do much less of what they know.

Figure 9 Four Gaps across facilities

Figure 10 examines the links between these four gaps and the five motivation factors looking at

all health workers across all facilities. It helps explain why these gaps exist and what are some of

the factors that help explain their size. For example if the management at one facility is very good,

then health workers are likely to be highly motivated and to do their job well, certainly compared

to health workers at a facility that is less well managed. It is difficult to say that they do their job

because they are motivated, or conversely that they are motivated because they do their job.

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%JF Doe

Redemption

Phebe

St. Francis

Tellewoyan

Bomi

CH Rennie

Curran

FJ Grant

GW Harley

56

Nonetheless the patterns we see can be informative. Figure 10 can also be considered a graphical

representation of the t-statistics of a regression of each of the four gaps on all five of the motivation

factors. The size of the coefficient has no natural interpretation, but the t-statistic shows how

important the factor is. In particular, if the t-statistic is larger than 1.9 (or smaller than -1.9) then

there is only a 5% chance that the factor is not statistically important. Thus, for the overall gap (the

target-performance gap) there are two factors that are significantly related to the size of the gap:

“This hospital and management provide the right tools” and “In this organization, I am taken

seriously.” The other three factors have t-statistics below the 5% cutoff, and are therefore not

shown in the graph. Note that the first factor (“This hospital and management provide the right

tools”) has a negative association with the gap, which means an increase in this factor closes the

gap. On the other hand “In this organization, I am taken seriously” has a positive association,

which means that for health workers and hospitals where this factor is more present, the gap is

larger.

57

Figure 10 Motivation factors and the four gaps with facility fixed effects

Figure 11 shows similar results but each regression includes fixed effects for each facility. As a

result the differences in gaps and motivations across facilities are eliminated and we can interpret

this as comparing health workers within facilities. Here we see that the second factor (“This

hospital and management provide the right tools”) still plays an important role in eliminating the

overall gap and the can-do performance gap. Note that, since we have included facility fixed

effects, this compares, for example, two workers within the same hospital: the one who is more

likely to believe that the hospital provides the right tools is the one more likely to provide better

quality care.

Note that because the gaps are connected to each other, the relationships to the factors look counter

intuitive at first glance. “I am keen to improve,” for example increases the target-know gap and

the know-can do gap, but decreases the can-do gap. This is because, by increasing the two previous

-6

-4

-2

0

2

4

6

target-do target-know know-can can-do

T-s

tati

stic

(ab

ov

e 5

% c

uto

ff)

The four gaps

The role of motivation factors in 4 gaps (without facility effects)

I can make a difference

This hospital and management provide the right tools

In this organization, I am taken seriously

I am keen to improve

Feedback, teamwork, recognition

58

gaps, it results in a low level of capacity for the health worker and with a low level of capacity the

can-do gap is generally smaller.

Figure 11 Motivation factors and the four gaps with facility fixed effects

Overall, we see that two factors play a consistently important role in these gaps. “This hospital and

management provide the right tools” tends to reduce gaps and “In this organization I am taken

seriously” tends to increase the gaps. The interpretation of this finding is less straightforward. It is

interesting that positive perceptions about the role of the individual within the hospital do not play

a stronger role in improving performance. Rather, it is the perceptions that the hospital is

adequately supporting the health worker that improves performance.

B. Changes between 2013 Baseline Data Collection and

2015 Baseline Data Collection

-6

-4

-2

0

2

4

6

target-do target-know know-can can-do

T-s

tati

stic

s (a

bo

ve

5%

cu

toff

)

The four gaps

The role of motivation in 4 gaps (with facility fixed effects)

I can make a difference

This hospital and management provide the right tools

In this organization, I am taken seriously

I am keen to improve

Feedback, teamwork, recognition

59

The initial baseline was collected in four of the facilities (JF Doe, Redemption, Phebe and

Tellowayan) in 2013 and the follow up was collected in a total of 10 facilities in 2015. Initially,

the design was to use this second round of data collection as a mid-line review of results. However,

because of the intervening EVD crisis it is not possible to interpret the results as being caused by

the quality PBF program. Nonetheless is it is instructive to examine the changes that took place in

the four facilities over this gap.

Each table, with two exceptions, shown below examines a t-test of the difference in means for the

four repeated facilities from 2013 to 2015. For the obstetric and surgical direct observation, there

are not enough observations in each of the facilities to allow a test of changes across time and

therefore we combine all four hospitals in a before and after for the combination of treated

facilities. In each table, in addition to the actual difference, we report the p-value of that difference.

Differences below a p-value of 0.1 can be seen as statistically important, though of course the p-

value does not say why something has changed.

Table 19 Change in Average General Hospital Infrastructure Observation Scores, by Facility

Change in overall scores between 2014 and

2015

Indicators for General Observation of the Hospital

Infrastructure

JFD Rede

m

Phebe Tell Total

Average of necessary equipment present 0.15 0.33 0.11 0.04 0.16

Average of necessary equipment working 0.13 0.32 0.09 0.02 0.14

Proportion of infrastructure present that is not working 0.02 0.01 0.03 0.06 0.03

Note: *p<0.10, **p<0.05, ***p<0.001

Table 19 examines the changes in the summary infrastructure scores for each facility. In general

each facility has more equipment present than in 2013, but more of the equipment is not presently

working. However, overall there are small increases in the proportion of equipment that is working.

Table 20 examines the changes in motivation scores (with the factor weights fixed at the 2013

level). Across all facilities there is a significant switch from factor 1 to factor 2. Health workers

are significantly less likely to believe “I can make a difference” and significantly more likely to

believe “this hospital and management provide the right tools.” Although this might be caused by

60

the PBF program, it seems to align well with what we might expect from the experience of the

EVD crises, particularly since this was most acutely felt at Redemption and Phebe hospitals.

Health workers feel somewhat powerless in the face of such a crises, but now have come to believe

that resources will be made available to help them, if necessary.

Table 20 Change in Average Motivation Factor Scores, by Facility

Change in overall scores between 2014 and 2015

Motivation Survey Factors JFD Redem Phebe Tell Total

I can make a difference 0.18 -6.6*** -0.37* 0.05 -2.27***

This hospital and management provide the right tools -0.26 4.23*** 0.07 0.05 1.49***

In this organization, I am taken seriously -0.27 0.66*** -0.3 -0.26 0.05

I am keen to improve 0.19 -0.09 0.07 0.05 0.06

Feedback, teamwork, recognition -0.24 0.16 -0.27* 0.15 -0.01

Total Score -0.01 -1.47*** -0.16** 0.08 -0.47***

Note: *p<0.10, **p<0.05, ***p<0.001

Table 21 shows the changes across all four facilities in key measures of quality in the obstetric

ward. There are a number of significant changes over the two years in question, but no overall

pattern. History taking increases and the quality of observation at the second and third stage also

increases, but the quality at the first stage decreases. There are no significant changes in the use of

harmful practices over this period, although the final category, an increase in no harmful practices

observed (“none of the above”), is almost significant at the 10% level. Overall, it would be

dangerous to try and draw conclusions about the impact of the program (or EVD) from the changes

observed in this table.

61

Table 21 Change in Key Indicators for Obstetrics Ward Observation

Obstetrics Observation Category Change in overall scores between 2014

and 2015

p-value

Obstetric Intake: History Taking 0.29 0.08 Obstetric Intake: Presence of Complications 0.1 0.52 Obstetric Intake: Overall Quality 0.07 0.37 First Stage of Labor Observation Quality -0.1 0.06 Second and third stage observation quality 0.08 0.08 Episiotomy: Usually unnecessary -0.02 0.76 Newborn Resuscitation Quality -0.09 . Potentially Harmful Practices 0.01 0.72 Partograph Quality 0.01 0.98 Filling Partograph After Delivery 0.6 0.24 Obstetric Care PPH Quality 0.13 . Obstetric Care: Pre-Eclampsia Quality 0.02 0.45 Itemized harmful practices Use of enema -0.03 0.35 Pubic shaving 0.03 0.29 Woman routinely forced to push during second stage

of labor -0.03

0.35 Apply fundal pressure to hasten delivery of baby or

placenta -0.03

0.35 Lavage of uterus after delivery 0 0.95 Minor tears stitched when not bleeding -0.03 0.64 Episiotomy / tears not repaired with local anesthesia -0.09 0.1 Vagina swabbed with antiseptics after delivery -0.03 0.35 Disinfectant put on the perineum after delivery -0.06 0.18 Bladder catheterization performed postpartum 0.03 0.3 Cervix checked after delivery -0.03 0.64 Slap newborn 0 0.93 Hold newborn upside down 0.07 0.13 Milking the newborn's chest -0.03 0.35 Excessive stretching of the perineum -0.08 0.22 Shout, insult or threaten the woman during labor or

after 0

. Slap, hit or pinch the woman during labor or after 0 . None of the above 0.17 0.1

62

Table 22 examines the changes under direct observation in the pediatric ward over all four facilities

and Table 23 examines the same scores by facility. Here, the quality of care (as seen in physical

examination and the combination of physical examination and history taking) actually falls over

the period in question.

Table 22 Change in Key Indicators for Pediatrics Ward Observation

Pediatrics Observation Category Change in overall scores between 2014 and 2015

Pediatric Observation Physical Examination Score -0.2***

Pediatric Observation History Taking and Physical Exam -0.14***

Pediatric Observation Health Education Score 0.04

Pediatric Observation General Effort Score -0.02

Note: *p<0.10, **p<0.05, ***p<0.001

When we examine these numbers for each facility we see that the decrease is common to all four

facilities. In fact, no score increases over this period.

Table 23 Change in Key Indicators for Pediatrics Ward Observation, by Facility


Pediatrics Observation Category JFD Redem Phebe Tell Total

Pediatric Observation Physical Examination Score -0.29*** -0.16*** -0.24*** -0.06* -0.2***

Pediatric Observation History Taking and Physical Exam -0.24*** -0.06 -0.2*** 0 -0.14***

Pediatric Observation Health Education Score 0.12*** 0.09 -0.06 -0.03 0.04

Pediatric Observation General Effort Score 0.04* -0.13*** -0.02 0.02 -0.02

Note: *p<0.10, **p<0.05, ***p<0.001

Table 24 examines the overall changes for the surgical direct observation. Here we see a number

of improvements across all four facilities. The preparation of the patient (obtaining charts,

histories, and consent) is improved and the team is much more likely to insure that the proper

protocol is followed for antibiotic prophylaxis. The quality of the surgery prep in general is also

improved and the cleanup both by the surgical team and the hospital staff is also improved. These

improvements are much more likely to be the result of the EVD crises than the PBF intervention.

Although at the time of the survey Liberia was declared Ebola free, the process of surgery is a

particularly sensitive and dangerous place likely to be impressed by recent memory.

63

Table 24 Change in Key Indicators for Surgery Ward Observation

Surgery Observation Category Change in overall scores between 2014 and 2015

Cleanliness of operating theater 0.07

Preparation of patient for surgery 0.13***

Surgeon preparation for surgery -0.05

Antibiotic prophylaxis given or confirmed 0.41***

General surgery quality 0.22***

The surgeon broke the surgical field during surgery 0.05

Medical staff post-surgery clean up 0.3***

Housekeeping staff post-surgery clean up 0.37***

Note: *p<0.10, **p<0.05, ***p<0.001

Table 25 takes advantage of the design of our data collection by examining quality in the vignettes.

These do not suffer when there are not enough patients and should show immediate gains in

competence (or capacity). Looking across all vignettes we slight improvements in the use of

history taking and the use of lab tests, but the gains are very small. Importantly, we see no evidence

that the drop in scores in the pediatric setting is due to a decline in knowledge about how to

properly diagnose cases.

Table 25 Change in Vignette Scores, by Facility


Vignette Scores JFD Redem Phebe Tell Total

vignette history taking score -0.06 0.12* 0.04 0.1* 0.05*

vignette physical examination score -0.15*** -0.02 0 0.05 -0.02

vignette: proportion of lab tests 0.03 -0.21*** 0.16*** 0.05 0.04*

Correct Assessment of Condition -0.21 -0.44* -0.03 -0.04 -0.12

Correct overall diagnosis -0.04 -0.08 -0.01 0.08 -0.01

Correct detailed diagnosis -0.14 0.12 0.12 -0.14 0

Correct Treatment -0.09 0.09 0.07 -0.14 0

Note: *p<0.10, **p<0.05, ***p<0.001

Table 26 examines the changes in characteristics of patients reporting at facilities. We might see

important changes if there is a change in the perception of quality at the facility. Indeed we see

that patients are younger, less likely to have no schooling and less likely to report being unable to

64

read. Patients are not wealthier however, as measured by the index of assets owned. In addition,

patients at Phebe and Tellowayan are less likely to have traveled great distances to seek care

(although there is no overall difference across all facilities).

Table 26 Change in Patient Characteristics for All Patient Exit Interviews, by Facility


Pediatrics Observation Category

JFD Redem Phebe Tell Total

Age -6.47*** 1.58 -2.82 -2.48 -2.1*

Patients without any schooling -0.43*** -0.02 -0.02 0.01 -0.11***

Patients completed primary schooling 0.31*** 0.08 -0.06 -0.01 0.07*

Patients completed secondary

schooling

0.06 -0.06 0.05 0.04 0.03

Patients completed vocational training 0 -0.01 0 -0.01 0

Patients completed post-secondary

schooling

0.04 0.01 0 -0.02 0.01

Patients reporting being able to read

easily

0.11 -0.02 0.12* -0.05 0.04

Patients reporting being able to read

with difficulty

0.31*** -0.01 -0.07 0.17* 0.08*

Patients reporting not being able to

read at all

-0.43*** 0.03 -0.05 -0.05 -0.12***

Travel Time: Less than 30 minutes -0.06 -0.05 0.11 0.25** 0.04

Travel time: 30 - 60 minutes -0.03 -0.02 0.16 -0.02 0.01

Travel Time: More than 60 Minutes 0.03 0.07 -0.29** -0.23** -0.07

Index of Assets Owned 0.36 -0.11 0.23 0.06 0.15

Note: *p<0.10, **p<0.05, ***p<0.001

65

Table 27 examines the changes in the satisfaction of patients by facility. Here we see that, overall

the patients are more pleased with the quality and convenience of facility and have a greater overall

opinion. They are slightly less likely to feel that they have received the right amount of individual

attention.

Table 27 Change in Satisfaction Summary Scores for All Patient Exit Interviews, by Facility


Pediatrics Observation Category JFD Redem Phebe Tell Total

Quality -0.24 1.18*** 1.22*** 1.6*** 1.1***

Convenience -0.25 0.48*** -0.03 0.08 0.27***

Average Likert scale score -0.01 0.28*** 0.14 0.07 0.14***

Important: overall satisfaction -0.38 -0.21 0.06 1.47* 0.28

Important: individual attention -1.98 -0.02 -0.03 -0.32 -0.3**

Note: *p<0.10, **p<0.05, ***p<0.001

V. Conclusion

This report discusses the data collected in the spring of 2015, evaluating the quality of 10 hospitals

providing care in Liberia. Although there are important differences between the hospitals studied,

the report focuses on validating the measures of quality collected and examining the changes that

have been seen in the past two years. Overall, we find that low performance of health workers in

the collection of hospitals is linked to a large can-do gap: the difference between what a health

worker is capable of doing and what he actually does. There exists a significant knowledge gap in

all of the facilities observed, but since there is very little connection between knowledge and

practice, it seems unlikely that improving knowledge will improve quality. Our measures of quality

show that there is much room to improve, but also show that the quality based PBF might find

fertile ground to make progress for exactly this reason.

We also examine the evidence for changes to date in the hospitals that are part of the program.

Here we find little statistically significant evidence of changes. Some things have improved, others

66

have not. If we could associate all of the positive changes observed to the implementation of the

program, the results would still be disappointing. However, due to the intervening EVD crises and

the lack of a proper control, we cannot associate even these modest gains with the implementation

of the program.

Fortunately, we now have a more robust identification strategy through the inclusion of five control

facilities. Although Redemption hospital does stand out in some categories, these five control

facilities span the range of characteristics seen in the other four treatment facilities and will help

us to establish, more precisely the changes we expect to see in the next round of evaluation.

67

VI. Appendices A. Acronyms

EVD Ebola Virus Disease

GMRP Graduate Medical Residency Programme

GoL Government of Liberia

HSSP Health Systems Strengthening Project

MCH Maternal and Child Health

MDG Millennium Development Goal

PGMC Post Graduate Medical Council

PGMC Post Graduate Medical Council

PBF Performance-Based Financing

PPE Personal Protective Equipment

MOHSW Ministry of Health and Social Welfare

WHO World Health Organization

68

B. Data Collection Schedule

Table 28 Data Collection Schedule

Departure Date Arrival Date Hospital for data

collection

Date for data collection at

each Hospital

Group 1

Leave from

Monrovia on 14

June

15 June F.J. Grante (Sinoe)

16 June – 20 June

21 June 21 June St.Francis Hospital

(Rivercess)

22 June –26 June

27 June 28 June C.H.Rennie (Margibi) 29 June – 3 July

5 July 5 July Liberian Government

Hospital (Bomi)

6 – 10 July

Leave for

Monrovia on 11

July

11 July

Redemption

(Montserrado)

13—17 July

Take off from MoH by 7am

daily.

Arrive at Redemption by 8am

daily.

Group II

14 June 14 June J.F.Doe (Nimba)

15—19 June

20 June 20 June G.W.Harley (Nimba)

22—26 June

27 June 27 June Tellewoyan (Lofa) 29 June – 3 July

4 July 4 July Curren Hospital (Lofa)

6 – 10 July

11 July 11 July Phebe Hospital (Bong)

13 – 17 July

Depart from

Bong to

Monrovia

18 -- 19 July

69

C. Quality Assessment Tool

Table 29 Management and Structural Checklist Overview

Category Component

I. Management 1. General Management

2. Human Resources for Health

II. Structural

3. Hygiene and medical waste disposal management

4. Drugs Management

5. Equipment and Supplies

Table 30: Quality Checklist Overview

Service-delivery

Type Priority Clinical Focus Areas

Childbirth:

Maternal-

Newborn

(intra- and post-

partum)

Routine intra- and post-partum high-impact care:

Intra-partum (labor/immediate post-partum)Post-partum (mother &

newborn):

Maternal Complications:

1. Obstructed Labor

2. Hemorrhage

3. Sepsis

4. Eclampsia

Neonatal Complications:

5. Asphyxia,

6. Sepsis,

7. Prematurity

Pediatric

(in-patient care)

8. Maternal Newborn Best Practices Chart review

9. ETAT: Emergency Triage, initial Assessment & Treatment

(Routine triage & initial stabilization all children presenting to hospital)

10. Malaria

11. Pneumonia

12. Acute Diarrhea

13. Severe Acute Malnutrition

Other: Pediatric Outcome and Processes

(Note: Neonatal Sepsis included above)

Surgical Care

14. WHO Surgical Safety Checklist (based on modified WHO content)

-Pre-operative assessment of patient

70

(General Surgery

& Caesarian)

-Anesthesia assessment & safety planning

-Management of intra-operative complications (e.g. bleeding, breathing,

cardiovascular (BP, arrhythmias, etc)

-Post-operative care

-Management of intra-operative complications

Table 31: Quantity of Services

PBF Services Definition

Quality of complicated and assisted

pregnancy and delivery (including C-

section)

Any labor that is made more difficult or complex by a

deviation from the normal procedure. Complicated

delivery is defined as: assisted vaginal deliveries

(vacuum extraction or forceps), C-section, episiotomy

and other procedures.

Quantity of normal deliveries for at

risk referrals

High-risk pregnant women referred by health center to

the hospital but delivered normally. A high-risk

pregnancy is defined as: evidence of edema, mal

presentation, increased BP, multi-parity, etc.

Quantity of counter referral letters

returned to health centers

Hospital returns counter referrals letter with feedback

on the referred patient to the referring health center. The

counter referral letter is completed in triplicate, with

one also given to the patient, and one retained by the

hospital.

Quantity of newborns referred for

emergency neonatal care treatment

Newborns referred for emergency neonatal care due to:

perinatal complications, low birth weight, congenital

malformation, asphyxia, etc.

Quantity of Referred under-fives with

fever

Infants and under-fives with fever who were referred to

the hospital for management of Malaria and Pneumonia.

Quality of Minor surgical intervention Any surgical procedure that does not involve anesthesia

or respiratory assistance.

Quality of major surgery (excluding

CS, including major trauma)

Any surgery in which the patient must be put under

general spinal/anesthesia and given respiratory

assistance. Major surgery in the case of this package of

services is defined as any of the following:

Herniarraphy, Appendectomy, Myomectomy,

Sleenectomy, Salpingectomy, Hysterectomy,

Thyrodectomy, and Mastectomy.

Quantity of patients transported by

ambulance

Patients transferred from a lower-level facility (health

center or health clinic) to the hospital for emergency

treatment.

71

D. Instrument Summaries

In the following sections, we look at the scores for each vignette in turn.

1. Surgical Vignettes

Table 32: Surgery Vignette Scores by Facility (S_V_1_A)

Facility

JFD Redem JFD St. F JFD LGH JFD Curr JFD GWH


Surgical vignette history

taking score

0.47 10 0.44 16 0.45 11 0.32 7 0.31 9 0 0 0.63 2 0.42 3 0.63 4

Surgical vignette

physical examination

score

0.52 10 0.56 16 0.44 11 0.54 7 0.29 9 0 0 3.3 2 0.53 3 0.5 4

Surgical vignette:

proportion of lab tests

0.25 10 0.26 16 0.22 11 0.27 7 0.21 9 0 0 0.61 2 0.37 3 0.33 4

Mean Capacity Vignette

Score (average of items

correct and available)

0.16 6 0.16 16 0.15 8 0.29 4 0.09 7 0 0 0 0 0

Mean Capacity

Competence Gap

(competence-capacity)

0.06 6 0.1 16 0.06 8 0.08 4 0.1 7 0 0 0 0 0

Correct Assessment of

Condition: Stable

0.91 11 0.3 20 0.82 11 0.67 9 0.89 9 0 0 1 3 0.67 3 0.75 4

Correct overall diagnosis:

hernia

0.73 11 0.7 20 0.64 11 0.67 9 0.44 9 0 0 0 3 1 3 0.5 4

72

Correct detailed

diagnosis: Inguinal

Direct or Indirect Hernia

0.64 11 0.3 20 0.64 11 0.78 9 0.44 9 0 0 1 3 1 3 1 4

Correct Treatment:

Elective Herniorrhaypy

0.71 7 0.18 11 0.63 8 0.5 8 1 1 0 0 0 1 1 0 1

Surgical vignette:

proportion of rational lab

tests

0.38 10 0.38 16 0.16 11 0.33 7 0.29 9 0 0 0.8 2 0.53 3 0.4 4

Surgical vignette:

proportion of irrational

lab tests

0.26 11 0.28 20 0.25 11 0.31 9 0.17 9 0 0 1.44 3 0.22 3 0.25 4

Table 33: Surgery Vignette Scores by Facility (S_V_1_B)

Facility




taking score 0.54 7 0.28 8 0.42 10 0.35 5 0.27 12 0.33 3 0 0 0.25 3 0

Surgical vignette


score 0.68 7 0.56 8 0.4 10 0.5 5 0.46 12 0.92 3 0 0 0.5 3 0

Surgical vignette:

proportion of lab tests 0.3 7 0.31 8 0.2 10 0.3 5 0.19 12 0.41 3 0 0 0.15 3 0



correct and available) 0.16 2 0.17 8 0.13 8 0.26 5 0.12 7 0 0 0 0 0

73

Mean Capacity

Competence Gap

(competence-capacity) 0.03 2 0.15 8 0.03 8 0.04 5 0.05 7 0 0 0 0 0


Condition: Unstable 0.63 8 0.7 10 0.4 10 0.67 6 0.67 12 0.33 3 0 0 0 3 0


hernia 0.75 8 0.5 10 0.5 10 0.33 6 0.33 12 1 3 0 0 0.33 3 0

Correct detailed

diagnosis: Femoral

Hernia 0.5 8 0.4 10 0.3 10 0.67 6 0.5 12 0.33 3 0 0 0.33 3 0

Correct Treatment:

Monitor, Emergency

Surgery 0.67 6 0.83 6 0.75 4 0.83 6 1 3 1 3 0 0 1 1 0

Dangerous Treatment:

Discharge 0.25 8 0.5 10 0.7 10 0.33 6 0.75 12 0 3 0 0 0.67 3 0

74


Facility




taking score 0.29 6 0.44 8 0.19 9 0.38 6 0.45 11 0.67 3 0 0.25 2 0.25 2 1 1

Surgical vignette


score 0.43 6 0.48 8 0.42 9 0.6 6 0.4 11 0.67 3 0 0.4 2 0.4 2 1 1

Surgical vignette:

proportion of lab tests 0.17 6 0.16 8 0.13 9 0.32 6 0.17 11 0.4 3 0 0.4 2 0.1 2 0.8 1




Mean Capacity

Competence Gap



Condition: Stable 0.25 8 0.45 11 0.11 9 0.29 7 0.36 11 0.33 3 0 0.5 2 0 2 1 1


hernia 0.88 8 0.91 11 1 9 1 7 0.91 11 1 3 0 1 2 0.5 2 0 1

Correct detailed

diagnosis: Inguinal

Direct Hernia 0.38 8 0.27 11 0 9 0.29 7 0 11 0.33 3 0 0 2 0 2 0 1

Correct Treatment:

Elective Herniorrhaypy 0.29 7 0.09 11 0 5 0.29 7 0 4 0.33 3 0 0 0 1 0 1

75


Facility




taking score 0.41 11 0.52 11 0.42 12 0.25 8 0.28 9 0 0 0.75 4 0.25 3 0.5 4

Surgical vignette


score 0.55 11 0.47 11 0.43 12 0.6 8 0.33 9 0 0 0.5 4 0.27 3 0.45 4

Surgical vignette:

proportion of lab tests 0.21 11 0.25 11 0.21 12 0.3 8 0.11 9 0 0 0.35 4 0 3 0.25 4




Mean Capacity

Competence Gap

(competence-capacity) 0.01 6 0.05 11 0.02 9 0 5 0 6 0 0 0 0 0


Condition: Stable 0.67 12 0.45 11 0.42 12 0.33 9 0.56 9 0 0 0.25 4 0 3 0.5 4

76


hernia 0.92 12 0.82 11 0.75 12 0.89 9 0.78 9 0 0 0.75 4 0.33 3 0.75 4

Correct detailed

diagnosis: Inguinal

Direct Hernia 0.25 12 0.18 11 0 12 0.11 9 0 9 0 0 0 4 0 3 0 4

Correct Treatment:

Elective Herniorrhaypy 0 11 0.1 10 0 8 0.11 9 0 3 0 0 0 2 0 1 0 1

77

Table 36: Surgery Vignette Scores by Facility (S_V_3)

Facility


Avg N Avg N Avg N Avg N Avg N Avg N Av

g

N Avg N Avg N Avg N


taking score 0.53 9 0.42 17 0.42 15 0.45 5 0.4 15 0.67 3

0.7

5 1 0.5 6 0.2 5 0.42 3

Surgical vignette


score 0.49 9 0.4 17 0.35 15 0.6 5 0.36 15 0.6 3 0.8 1 0.37 6 0.44 5 0.33 3

Surgical vignette:

proportion of lab tests 0.08 9 0.04 17 0.11 15 0.18 5 0.05 15 0.07 3 0.2 1 0.2 6 0.04 5 0.15 3



correct and available) 0.3 1 0.03 14 0.05 10 0 0.05 9 0 0 0 0 0

Mean Capacity

Competence Gap

(competence-capacity) 0 1 0.02 14 0.02 10 0 0 9 0 0 0 0 0


hernia 0.61 18 0.43 21 0.53 15 0.4 15 0.53 15 0.33 3 1 1 0.83 6 0.8 5 0.67 3

Correct Treatment:

Elective Herniorrhaypy 0 6 0 8 0 11 0.38 8 0 8 0.5 2 0 0 4 0 4 0 1

78


Facility




taking score 0.34 8 0.35 10 0.4 12 0.42 9 0.35 10 0 0 0 0.25 3 0.5 2

Surgical vignette


score 0.48 8 0.44 10 0.53 12 0.67 9 0.32 10 0 0 0 0.47 3 0.5 2

Surgical vignette:

proportion of lab tests 0.17 8 0.21 10 0.19 12 0.19 9 0.09 10 0 0 0 0.17 3 0.17 2




Mean Capacity

Competence Gap



Condition: Stable 0.33 9 0.73 11 0.42 12 0.36 11 0.4 10 0 0 0 0 3 0.5 2


hernia 0.44 9 0.27 11 0.17 12 0.64 11 0.1 10 0 0 0 0.33 3 0.5 2

Correct Treatment:

Elective Herniorrhaypy 0.17 6 0 10 0 11 0 11 0 7 0 0 0 0 3 0 2

79


Facility




taking score 0.38 6 0.44 9 0.17 9 0.18 7 0.47 9 0.42 3 0.25 1 0.25 2 0.13 2 0.25 1

Surgical vignette


score 0.7 6 0.6 9 0.47 9 0.51 7 0.42 9 0.73 3 0.6 1 0.5 2 0.2 2 0.2 1

Surgical vignette:

proportion of lab tests 0.29 6 0.15 9 0.17 9 0.18 7 0.15 9 0.4 3 0.4 1 0.3 2 0 2 0.2 1




Mean Capacity

Competence Gap

(competence-capacity) 0.07 2 0.06 5 0.03 7 0 6 0 5 0 0 0 0 0


Condition: Stable 0.43 7 0.42 12 0.44 9 0.75 8 0.44 9 0.67 3 0 1 0.5 2 0 2 1 1


hernia 0.71 7 0.75 12 0.56 9 0.88 8 0.89 9 1 3 1 1 1 2 0.5 2 1 1

Correct Treatment:

Elective Herniorrhaypy 0.83 6 0.42 12 0.17 6 0.75 8 0 6 0 3 0 1 0.5 2 1 1 0

80

2. Pediatric Vignettes

Table 39: Pediatric Vignette Scores by Facility (P_V_1)

Facility



Correct treatment: refer,

IV or naso-gastric

feeding plus ORS 0.81 31 0.68 22 0.66 37 0.7 25 0.63 27 0.67 6 0.6 5 0.75 8 0.61 9 0.61 9

Correct and equipment

available 0.74 31 0.64 22 0.51 37 0.62 25 0.56 27 0.58 6 0.5 5 0.5 8 0.28 9 0.56 9

Mean Capacity

Competence Gap

(competence-capacity) 0.06 31 0.05 22 0.15 37 0.08 25 0.07 27 0.08 6 0.1 5 0.25 8 0.33 9 0.06 9

81


Facility



Average proportion of

history taking items 0.41 53 0.4 34 0.31 57 0.33 38 0.28 43 0.35 6 0.41 5 0.36 8 0.28 9 0.29 10



items 0.53 53 0.45 34 0.48 57 0.47 38 0.33 43 0.49 6 0.43 5 0.46 8 0.34 9 0.61 10

Diagnosis correct:

Malaria + anemia 0.43 53 0.21 34 0.46 57 0.5 38 0.23 43 0.33 6 0.2 5 0.13 8 0.22 9 0.1 10

Diagnosis partially

correct: Malaria 0.53 53 0.68 34 0.51 57 0.45 38 0.7 43 0.67 6 0.8 5 0.75 8 0.78 9 0.7 10

Diagnosis wrong:

Meningitis 0.04 53 0.09 34 0.04 57 0.05 38 0.09 43 0 6 0 5 0 8 0 9 0 10

Capacity to perform

physical examination 0.5 53 0.45 34 0.48 57 0.43 38 0.33 43 0.47 6 0.42 5 0.43 8 0.33 9 0.57 8

Physical Examination

Competence Capacity

Gap 0.03 53 0 34 0.01 57 0.03 38 0.01 43 0.01 6 0.02 5 0.03 8 0.01 9 0.02 8

82


Facility







items 0.56 28 0.6 21 0.47 35 0.5 25 0.44 31 0.39 6 0.56 5 0.43 8 0.45 9 0.44 11

Diagnosis correct:

Acute Diarrhea 0.82 28 0.81 21 0.91 35 0.92 25 0.94 31 0.67 6 1 5 0.75 8 0.78 9 0.91 11

Diagnosis partially

correct: Dehydration,

dysentery,

gastroenteritis 0.07 28 0.05 21 0 35 0 25 0.03 32 0 6 0 5 0 8 0 9 0 11

Diagnosis wrong:

malaria 0.57 28 0.48 21 0.4 35 0.24 25 0.48 31 0.5 6 0.2 5 0.75 8 0.44 9 0.64 11

Diagnosis: Capacity to

perform physical

examination 0.52 28 0.6 21 0.46 35 0.48 25 0.41 31 0.39 6 0.56 5 0.38 8 0.43 9 0.4 11

Diagnosis: Physical

Examination

Competence Capacity

Gap 0.04 28 0 21 0.01 35 0.03 25 0.02 31 0 6 0 5 0.06 8 0.02 9 0.03 11

Treatment correct: ORS

or IV fluids 0.93 28 0.76 21 0.73 33 0.72 25 0.68 31 0.67 6 0.6 5 0.63 8 0.89 9 0.5 10

Treatment partially

correct 0.5 28 0.43 21 0.42 33 0.32 25 0.52 31 0.5 6 0.2 5 0.63 8 0.33 9 0.82 11

83

Treatment wrong: only

Anti-malarials and/or

PCM 0.61 28 0.57 21 0.67 33 0.8 25 0.87 31 0.67 6 0.4 5 0.88 8 0.78 9 0.45 11

Treatment dangerous:

antidiarrheal 0.46 28 0.38 21 0.42 33 0.4 25 0.52 31 0.67 6 0.4 5 0.88 8 0.67 9 0.45 11

Treatment increment:

prescribing zinc 0 28 0.05 21 0.09 35 0 25 0.13 32 0 6 0 5 0 8 0 9 0 11

Health Education:

Proportion of good

health education done 0.59 28 0.54 21 0.46 35 0.51 24 0.44 31 0.4 6 0.48 5 0.65 8 0.36 9 0.61 11

84


Facility






physcial examination

items 0.45 23 0.32 21 0.36 36 0.38 25 0.28 27 0.34 6 0.19 5 0.26 8 0.4 9 0.23 10

Capacity to perform

physcial examination 0.43 19 0.3 21 0.34 36 0.35 25 0.26 24 0.32 6 0.15 5 0.18 8 0.37 8 0.23 9

Diagnosis: Physical

Examination

Competence Capacity

Gap 0.01 19 0.02 21 0.03 36 0.03 25 0.03 24 0.02 6 0.04 5 0.08 8 0.02 8 0.01 9

85

3. Obstetric Vignettes

Table 43: Obstetrics Vignette Scores by Facility (O_V_1_A)

Facility



Mean Vignette Score

(average of items

correctly

used/mentioned) 0.5 2 0.52 6 0.71 11 0.58 3 0.55 4 0.41 3 0.51 9 0.71 4 0.52 7 0.67 10



correct and available) 0 0.45 4 0.88 1 0.94 1 0 0 0.62 1 0.68 3 0.51 6 0

Mean Capacity

Competence Gap

(competence-capacity) 0 0.03 4 0 1

-

0.24 1 0 0 0.03 1 0 3 0.02 6 0

86

Table 44: Obstetrics Vignette Scores by Facility (O_V_1_B)

Facility



Partially Correct

Diagnosis (Pre-

eclampsia) 0.67 6 0.5 4 0.33 6 0 4 0.75 8 0.4 5 0.15 13 0.4 5 0.43 7 0.33 3

Correct Diagnosis

(Severe Pre-eclampsia) 0.17 6 0 4 0.67 6 0.25 4 0.13 8 0.4 5 0.46 13 0.4 5 0.14 7 0 3

average of correct

treatment responses

(MgSO4,

antihypertensive,

document) 0.72 6 0.25 4 0.61 6 0.33 4 0.58 8 0.47 5 0.49 13 0.47 5 0.39 6 0.44 3

Correct Treatment

(stabilize with MgSO4) 0.67 6 0.5 4 0.83 6 0.25 4 0.63 8 0.6 5 1 13 0.6 5 0.43 7 0.67 3

Correct response to

convulsions (MS,

oxygen and protect from

injury or lay on si 0.5 6 0 4 0.5 6 0.25 4 0.25 8 0 5 0 13 0.4 5 0.29 7 0.33 3

Wrong response to

convulsions (Diazepam,

antihypertensives or

active restraint) 1 6 1 4 1 6 0.5 4 0.88 8 1 5 0.85 13 1 5 1 7 1 3

Number of essential

items listed (a, b, c, f, g,

h, i) 0.69 6 0.44 4 0.71 6 0.09 4 0.45 8 0.55 5 0.46 13 0.65 5 0.42 6 0.71 3

Correct response to

stabilization (repeat 0.64 6 0.33 4 0.5 6 0.17 2 0.23 8 0.37 5 0.33 13 0.43 5 0.33 6 0.58 3

87

MgSO4, partograph,

record, reflexes)

88


Facility



Mean Vignette Score

(average of items

correctly

used/mentioned) 0.71 9 0.61 8 0.72 18 0.71 9 0.65 12 0.7 3 0.46 7 0.76 4 0.44 9 0.74 10




Essential items: open

airway, suction,

ventilate 0.67 9 0.38 8 0.67 18 0.89 9 0.75 12 0.67 3 0 7 0.75 4 0.44 9 0.7 10

Mean availability

score(average of

equipment available) 2.44 9 1 8 2.33 18 3.33 9 2.25 12 0 3 0 7 0 4 0 9 0 10

Mean Capacity

Competence Gap

(competence-capacity) 0 4 0.07 2 0.03 9 0.01 6 0.03 7 0 0 0 0 0

89


Facility



Health worker follows

correct process for

resuscitation 0.83 10 0.67 6 0.7 11 1.49 8 1.33 13 0.44 4 0.43 7 0.76 5 2.26 7 2.97 3

Health worker is able to

properly ventilate on

first try 0.5 10 0.5 6 0.45 11 0.29 7 1.46 13 0.25 4 0.14 7 0.8 5 0.43 7 0.5 2

Health worker is able to

properly ventilate on

second try 0.33 6 1 4 0.67 6 0.43 7 3.67 3 0.33 3 0 6 1 2 0.25 4 0 1

Health worker cannot

properly ventilate 0.3 10 0.17 6 0.18 11 0.25 8 0 13 0.5 4 0.71 7 0 5 0.43 7 0.33 3

90

Table 47: Obstetrics Vignette Scores by Facility (O_V_3)

Facility



Mean Vignette Score

(average of items

correctly

used/mentioned) 0.69 18 0.61 23 0.72 18 0.53 16 0.96 22 0.62 8 0.53 13 0.69 10 0.48 15 0.67 13

Mean availability

score(average of

equipment available) 0.21 18 0.21 23 0.21 18 0.22 16 0.18 22 0 8 0 13 0 10 0 15 0 13



correct and available) 0.51 18 0.46 23 0.5 18 0.43 15 0.42 19 0.38 8 0.32 13 0.4 8 0.28 15 0.35 10

Mean Capacity

Competence Gap

(competence-capacity) 0.18 18 0.15 23 0.21 18 0.11 15 0.15 19 0.24 8 0.21 13 0.3 8 0.21 15 0.29 10

91


Facility



Mean Vignette Score

(average of partograph

items correct) 0.68 9 0.65 14 0.81 20 0.8 8 0.56 11 0.15 2 0.71 6 0.85 4 0.51 8 0.84 9

92


Facility



Mean Vignette Score

(average of partograph

items correct) 0.53 5 0.45 9 0.78 13 0.58 10 0.65 15 0.41 5 0.62 6 0.89 5 0.33 7 0.68 3

midline evaluation of hospitals selected from a quality ...€¦ · midline evaluation of hospitals...

Documents