Assessing Provincial and District Health System Capacity to Sustain HIV/AIDS Care and Treatment Services—A Literature Review Ilona Varallyay, Jennifer Yourkavitch, and Eric Sarriot. ICF Macro- 2010

IntroductionAs PEPFAR transitions from an emergency response to promoting sustainable country programs,1 there is a need to create a benchmarking process through which one can determine the status of the transition of service delivery from implementing organizations to local entities. ICF is creating a rapid assessment process structured around standard health system domains to monitor provincial and district MOH capacity, and thereby to assess the status of the transition efforts. Issues of local ownership and institutionalization of programs within national structures are now key priorities and critical to these aims is the transition of a wide range of essential functions to national government structures. As this transfer of responsibilities gains ground, the role of peripheral/sub-national management structures, such as those at district or provincial level, will become increasingly important in making this local ownership viable. The question then arises, how can we assess the readiness and/or capacity of these lower level government structures (at the district or province levels) to undertake this transition? To address this question, this literature review aims to inform the development of a new tool and process to assess sub-national level capacity of the health system.

This literature review supports the effort to strengthen analytical approaches to assess provincial and district health system capacities to support HIV services by providing a rationale for the inclusion of selected core domains of assessment, and by identifying certain methodologies and contextual factors to consider when designing the assessment tool. The authors reviewed more than 40 sources on health systems strengthening, specifically for HIV services and other relevant areas, and discussed related issues with health systems experts (Annexes 1 and 2). The findings and recommendations gleaned from this review are summarized in this document. This literature review lays a solid foundation for developing a rapid assessment tool on provincial and district health system capacity to sustain HIV care and treatment services.

Methodology of the review We performed a systematic review of published and unpublished (grey) literature on the assessment of health systems capacity at various levels and also specifically relating to HIV service delivery, as described below.

Search Strategy

We identified sources from a systematic search of computerized databases (Medline, Popline, Dialog –includes Global Health, Federal Research in Progress (FEDRIP), EMCare, The Lancet), a search of the electronic archives of relevant international organizations using the Google search engine (Abt, Health Systems 20/20, MEASURE Evaluation, WHO, the World Bank, FHI, etc.), and by studying the bibliographies and reference lists of identified sources.

Searches used combinations of the following keywords:

1 PEPFAR Strategy, 2009: http://www.pepfar.gov/strategy/document/133244.htm

1

Peripheral health system capacity \ District health team capacity \ Provincial level capacity

HIV/AIDS care and treatment \ HIV integration\ HIV \ HIV infections \ HIV transmission

Health Services Performance \ health planning Management capacity \ Essential functions \ Supervision of Assessment of… \ Review of… \ Assessment tool \ Methodology \ capacity assessment

Temporal limits were set to include sources from the past 20 years; language restrictions were set to include references in Spanish, French, and English.

Article Selection

Documents were eligible for inclusion in the literature review if they addressed health systems strengthening at the sub-national level; if they focused on health systems assessment at a broader level but had lower level applications; if they addressed ART programs specifically; if they addressed capacity assessment along one of the core domain areas2; or if they addressed issues of sustainability of health services.

For HIV-related resources, a temporal limit was set to all documents dated post-2001, as this was the approximate time that international organizations started scaling up the introduction of ARVs in developing countries.

Overall, 43 documents were identified, including 17 assessment tools, 7 of which were HIV-focused.

Key Informant Interviews

In addition to the literature reviewed, we conducted semi-structured key informant interviews with several international health experts, whom we felt could contribute to the discussions around the selection of domain areas for the development of this tool. The selection of these contacts was based on initial recommendations from colleagues involved in health systems strengthening and facility assessment work, and then spanned out as each contact made references to others in a relevant field. Individuals from Abt Associates Inc, AIDSTAR-Two, MCHIP, MSH, USAID, WHO provided information included in this literature review. All contacts with informants took place between April 26th and May 11th, 2010 (See Annex 2 for list of informants).

District Capacity within a Health Systems PerspectiveThe dominant framework for assessing the capacity and performance of the health system as a whole is the WHO Six Building Blocks Model (WHO, 2007). This model breaks the health system functions into 6 broad categories (Figure below):

2 Strategic Planning for Integrated HIV Services; Human Resource Management for Clinical Services; Supervision of Clinical Services; Health Facility Renovation/Maintenance/Equipment; Laboratory Services Support; Pharmacy Supply/Logistics; HMIS, M&E Systems; Quality Management/Improvement Systems; Community Linkages; Stakeholder Management and Coordination; Financial Management of Donor Funding for HIV services

2

1. Service delivery: packages; delivery models; infrastructure; management; safety & quality; demand for care

2. Health workforce: national workforce policies and investment plans; advocacy; norms, standards and data

3. Information: facility and population based information & surveillance systems; global standards, tools

4. Medical products, vaccines & technologies: norms, standards, policies; reliable procurement; equitable access; quality

5. Financing: national health financing policies; tools and data on health expenditures; costing6. Leadership and governance: health sector policies; harmonization and alignment; oversight

and regulation

Source: WHO. Everybody’s business: strengthening health systems to improve health outcomes : WHO’s framework for action. World Health Organization, Geneva (2007).

Most of the more recent health systems literature and tools adhere to this comprehensive conceptualization. We focused our review on the operational definition and measure of capacity at provincial and district levels and did not find reason to challenge the ‘building blocks’ as an overarching model for looking at capacity and performance in the health section. However, given the importance of community-level interventions in HIV prevention and treatment work, particularly around linkages between the community and health facilities, we added a seventh building block, ‘community linkages,’ which looks at various aspects of client participation or engagement in the production and consumption of health-related services (for example, outreach services, referral mechanisms, the role of community health workers, etc.) This inclusion is consistent with WHO’s conceptualization of a health system.3

It is useful at this point to state important premises of our review.

3 According to WHO, a health system consists of all organizations, people and actions whose primary intent is to promote, restore or maintain health. Among other things, it includes a mother caring for a sick child at home. Everybody’s Business: Strengthening Health Systems to Improve Health Outcomes. Geneva:WHO, 2007, p. 2).

3

Capacity is not Performance

Organizational Capacity is the ability of an organization to meet its mandate and achieve its objectives.4 Performance is how effectively the same organization implements its activities and delivers its services. Those two concepts are obviously related, but they are distinct. Performance can be assessed by the delivery of a good or service to clients. This can be translated into a relatively small number of simple metrics. A given organizational performance (i.e. providing a skilled care provider, trained and equipped in a timely fashion to a client of counseling services) requires the expression of different capabilities from a district: sound human resources management, from recruiting to training to supervising; appropriate resourcing of finances, goods and commodities; sound planning and work organization, etc. Capacity, on the other hand, is a far more multidimensional concept, with strong interdependency between its different dimensions. Its development does not necessarily fit linear patterns. It is important to bear in mind that both capacity and performance are influenced by context. The translation of capacity into performance is affected by the context, and possibly a number of unknown variables. Consequently, the same capacity is not always expressed in the same level of performance because it is affected by external circumstances and intervening events

These two characteristics have provided numerous challenges to the measurement of capacity over the years.5 Such difficulties also affect the assessment of capacity of health districts, as a linchpin organization within national health systems. The literature is rich in facility assessment tools, but – as shown in this review – far leaner in assessments conducted at district level (and even more so with an HIV/AIDS care and treatment focus).

The District has a unique and pivotal role in the health system

Brown and LaFond again reviewed the state of the art in assessing and measuring capacity in the health sector. In the figure below6, their model helps visualize how assessing each level has to be analyzed within an overall system.

4 LaFond, A., L. Brown, K. Macintyre. Measuring Capacity in the Health Sector: a Conceptual Framework. Int J Health Plann Mgmt 2002; 17: 3-22. Published online in Wiley InterScience (www.interscience.wiley.com) DOI: 10.1002/hpm.6495 Ibid.6 Ibid.

4

Under this model, the District fits at the level of the Organization.

Depending on size and political administrative structures of each country, districts report directly to a central level or through a regional administrative level. Provinces sometimes play the role of districts, other times act as regional or quasi-central structures. Where health districts as operational units are essential elements of a health system,7 the role of the province should be clearly defined and may include monitoring the performance of district health systems, training district-level staff, and conveying central health policies, among others.8 Our focus here is on structures – district or province – directly overseeing and organizing the work of service delivery units and health workers.

Assessment challengesThe first challenge lies with the understanding and measurement of capacity. The reader is referred again to Brown and LaFond for a thorough treatment of the concepts and measurement challenges. In summary: capacity supports performance, but the relationship between the two is multidimensional, non-linear, and complex.

Then, the unique position of districts within the broader health system creates some challenges in the evaluation of its performance but also its capacity. For example, assessing how a district organizes and implements supervision, or secures drugs and commodities to service providers, is

7 Kloss-Quiroga, B. (Ed.). InWEnt: District Health Management Tools: Facilitator’s Manual. Berlin, 20048 Chatora, Rufaro and Prosper Tumusiime. Health Sector Reform and District Health Systems. District Health Management Team Training Modules, Module 1. Brazzaville, WHO, 2004.

5

certainly better assessed at health facility level than in the district health office itself. While this tension mostly affects measures of performance, it may on occasion be relevant to the assessment of the district’s capacity as well. Similarly, some elements of district capacity are determined by central level policy or resource allocation decisions. For example, a district capacity assessment may identify human resources weaknesses, but these may be the expression of inadequate policies or interventions at a higher level.

The take-home message is that a ‘district capacity assessment’, by which an assessment is conducted at district level exclusively, must be understood with a proper appreciation for the boundaries and limitations it carries-- that is, some capabilities of the districts are better analyzed upstream or downstream. For example, the capacity of district management to fill vacant positions is limited by policy directives “upstream,” at the national level. Similarly, if we want to better understand the district’s capacity to supervise health workers, we need to ask questions of the health workers themselves (“downstream”). A comprehensive and rapid assessment tool has to be flexible enough to capture elements of the district/provincial capacity that are influenced by, and manifested in, other levels of the health system.

Review of existing assessment models

Nature of assessment methodologies and type of indicators

The tools reviewed included qualitative and quantitative assessment methodologies; a few incorporated both methods. Approaches included self-assessments, facilitated self-assessments, and traditional survey methodologies, albeit mainly at facility level. Assessments varied in the level of applicability from national to provincial to district to facility levels; for purposes of this search, we focused on relevant district and provincial level resources. National level tools that could serve as a guide for the development of relevant sub-national assessment criteria were also included. The most common data collection methodologies among the tools reviewed involved using secondary data, document review, and stakeholder interviews. Some are rapid assessments; others are intended for longer-term implementation.

Both qualitative and quantitative indicators are included, in order to provide a measure of capacity and/or performance and to describe factors that may affect capacity or performance . The different tools capture similar domain areas with distinct indicators; these will be assessed for relevance. Many will have to be modified to fit the context of district/province, according to the roles of the health system at this level.9

Domains of assessment at provincial / district levels

Sambo presents a type of framework, which looks specifically at the District level and assesses the functionality of the system according to broader categories, which in sum capture all of the WHO building blocks: 1) the existence of functional district level management structures (village/town health committee, health facility management committee, DHMT, etc) 2) the managerial process (planning, collaboration, guidelines, supervision/ monitoring, drug management, referral mechanisms, HMIS) 3) the health activities/interventions delivered 4) the community health initiatives 5) the availability of locally managed health-related resources

9 Ibid.

6

(facility budget, cost recovery, human resources, infrastructure, equipment, supplies).10 This framework does not look at health system performance.

One health system assessment tool which adheres to the WHO framework was adapted for use at provincial level, adjusting the framework for applicability at sub-national level.11 A close analysis of the specific indicators used at provincial level should inform the development of a district-level assessment. The governance core domain is captured through indicators such as: 1) Responsiveness of government to public needs, 2) Voice of the people 3) Exercising local technical oversight of health service quality 4) Production of services needed by the public 5) Information and reporting 6) Direction, oversight and resource allocation tasks carried by government. The health financing core domain is captured through indicators such as: 1) revenue collection 2) pooling and allocation of financial resources 3) purchasing and provider payments. Service delivery is assessed with indicators such as: 1) availability 2) general access, coverage and utilization 3) service outcomes 4) service delivery access and utilization 5) organization of service delivery 6) quality assurance of care 7) community participation in service delivery. Human resources information such as HR planning, HR policies, performance management, and training and education are assessed. The pharmaceutical management system domain captures elements including relevant budget, policies/laws/regulations, procurement, storage & distribution, appropriate use, access to quality products, and financing. The HIS component includes information on resources/policies/regulation, data collection and quality, data analysis, and use of information for management.

Another assessment framework intended for use at the district level captures the elements of the WHO six building blocks through 7 modules, which looked in turn at 1) District health management and support systems 2) primary health care facility information 3) rural/district hospital information 4) community participation and traditional health systems 5) community and household information 6) patterns of mortality.12 Such an approach combines household level, facility level, and district level data collection. Components which will be of particular relevance to the development of a provincial and district capacity assessment tool include those on district health management and support systems and on community participation/traditional health systems.

The WHO District Health Management Teams Training Modules used in the Democratic Republic of Congo13 have been developed to assist each team member to work as a leader and manager in leading health development in the district. The methodology is of particular relevance for the development of the assessment tool as it provides a useful framework for translating the functions of the health system at national level into equivalent roles at district level, according to four broad health system function domains: stewardship, resource generation, service delivery and financing.  It defines a number of key roles at province or regional level, particularly relevant in a context of decentralization.

10 Sambo et al. Tools Assessing the Operationality of District Health Systems. Brazzaville, WHO, 2003.11 Thi Mai Oanh, et al. Assessing Provincial Health Systems in Vietnam: Lessons from Two Provinces. March 2009. Bethesda, MD: Health Systems 20/20 project, Abt Associates Inc.12 Maier et al. Assessment of the district health system – using qualitative methods. GTZ/WHO/ITHÖG Heidelberg. (1994).13 Chatora, Rufaro and Prosper Tumusiime. Health Sector Reform and District Health Systems. District Health Management Team Training Modules, Module 1. Brazzaville, WHO, 2004.

7

HIV-Specific Resources

A number of tools focused on assessing HIV programs or service delivery specifically, though primarily at either national or facility level. The domains of assessment are nonetheless pertinent to the proposed provincial/district assessment and can be adapted to suit applicability at that level. One tool14 translated the 6 WHO building blocks into ART-specific domains and indicators in the following way:

1) Site Leadership and Model (Leadership; Model of HIV Care; ART-Specific protocols)

2) Services and Clinical Care (Comprehensive HIV care services other than ART; ART delivery; Physical Space; Community Involvement)

3) Health Information Management, Monitoring, Evaluation, and Quality (HMIS; Quality Management; ART Program M&E )

4) Human Resource Capacity (Staffing needs; training and skills development; Management, supervision and staff retention)

5) Lab Capacity (Lab procedures capability; Quality standards)

6) Drug Management and Procurement (Drug Management and Pharmacy Management)

While this assessment is intended for facility/site level implementation, these domain areas and indicators can be modified for use at sub-national level.

Another relevant tool, HIV/AIDS Program Sustainability Analysis Tool –HAPSAT, focused on assessing the sustainability of HIV/AIDS programs (including VCT, PMTCT, ART, care and support services, and prevention) through a computer-based forecasting model.15 This tool calculates resource needs for an HIV/AIDS program with a user-defined set of service level goals. HAPSAT also compares projected resource needs (focusing primarily on human resources and financial resources) to expected resource availability and can be used to identify and report expected gaps and resource imbalances that may lead to inefficient or poor quality service delivery. It captures the following domains: 1) demographic data 2) epidemiological data 3) financial data (trends in donor funding for HIV/AIDS programs and services and outline current and potential funding possibilities 4) labor data 5) service volume data 6) medical data 7) cost data. Examples of specific indicators include: quantity of health workers; average salaries of health workers; ART drug regimens cost; number of patients on each ART regimen; amount of donor funding and internal revenue for HIV/AIDS currently and expected; cost of laboratory tests; etc. This tool is very comprehensive and very in depth; only a significantly modified and simplified version of such a sustainability assessment component could be pertinent for the district/province rapid assessment tool.

Of notable interest were models addressing the chronic nature of HIV/AIDS and the importance of maintaining the client at the center of any assessment framework. While such models were primarily designed for developed-country contexts, we identified such frameworks as highly applicable for the developing country context as well. One model16 looks at three core components with respect to care for chronic conditions: positive policy environment, the

14 Hirschhorn, L. et al. Tool to Assess Site Program Readiness for Initiating Antiretroviral Therapy or Capacity for Existing ART sites. Boston, MA: John Snow, Inc. 2003.15 Health Systems 20/20 project. Using HAPSAT for HIV Program Sustainability Analysis: An Introductory Guide. April 2010. Bethesda, MD: Abt Associates Inc.16 WHO. Innovative Care for Chronic Conditions: Building Blocks for Action. WHO, 2002.

8

community, and the health care organization; of particular interest are the ‘building blocks’ for health care organization which include promoting continuity and coordination; encouraging quality care through leadership and incentives; organizing and equipping health care teams; supporting self-management and prevention; using information systems. While the focus of the proposed assessment is at district/provincial level, this model highlights the importance of bearing in mind the ultimate needs of the patient/client, which is relevant for district level monitoring and supervision. This framework, however, is designed for a developed country context and assumes that certain capabilities are in place ‘as a given’ and that the assessment can move toward those capacity areas which support higher standards of quality care; this would need to be reframed for the developing country context, where many of these assumptions would not be valid.

Domain-specific Resources

A number of domain-specific tools were also reviewed.

One tool focuses on assessing the logistics system for HIV/AIDS programs, using quantitative indicators to assess product availability and inventory management practices.17 Specific indicators address: inventory control systems (ICSs); logistics management information systems (LMISs); and storage, logistics reporting, and ordering and institutional support. Another tool focuses on human resources management (HRM) for HIV/AIDS programs18. It contains 26 HRM components (sub-domains) which are grouped within five broad areas of HRM: HRM capacity (staffing, budget, and planning); personnel policy and practice; performance management; training; HRM data.

We also reviewed several articles and grey literature documents addressing issues around scale up of ART programs which highlight a number of key issues for consideration in the development of this rapid assessment. One report describes an assessment which focused on two areas: logistical requirements for ensuring a reliable and consistent supply of quality antiretroviral drugs (ARVs) and related commodities, and infrastructure and personnel requirements necessary to ensure their safe and effective use by patients.19 It outlines key criteria to assess site readiness for ART initiation which can be adapted into indicators for ART delivery capacity at a higher level. This document also stresses the importance of a robust supply chain to manage commodities—a key area for assessment at district/provincial systemic level—and outlines the key elements for effective LMIS.

Another report on the scale up of ART in Uganda highlights a number of indicators according to the WHO framework, and also includes a domain for ‘communications’ and IEC, exploring issues which influence the degree of community level awareness of HIV services: information on the availability of treatment, care, and support services at community level; information campaigns; communication materials such as booklets on ARV therapy; guides for service providers on dispensing ARV drugs; details of all ARV drugs—dosage, pharmacokinetics/ pharmacodynamics, side effects, drug interactions, etc.20 This element could be incorporated into

17 Nyenwa, Jabulani, et al. Zimbabwe HIV & AIDS Logistics System Assessment. Arlington, Va.: John Snow, Inc./DELIVER, for the U.S. Agency for International Development, 2005.18 MSH, 2003. Human Resource Management Rapid Assessment Tool for HIV/AIDS Environments. Cambridge, MA: MSH.19 Noguera, M., et al. Zimbabwe: Antiretroviral Therapy Program – Issues and Opportunities for Initiation and Expansion. Arlington, VA: DELIVER/John Snow, Inc., 2003.20 Okera, A. et al, May 2003. Scaling Up Antiretroviral Therapy. Ugandan Experience. Draft. Geneva, Switzerland: World Health Organization.

9

a ‘Community Linkages’ domain of assessment. Another article cites program planning guidance for the implementation of expanded ART services and includes valuable information on coordination with other programs that could be useful in developing indicators for a ‘Stakeholder Management and Coordination’ core domain.21.

Leads for the Development of a Rapid Assessment Tool

A couple of other domain areas identified in the literature review which deserve consideration are components pertaining to district/provincial health team role in: 1) positive policy environment, and 2) communications.

With respect to policy environment it may be useful to assess the district’s capacity in: 1) strengthening partnerships; 2) supporting legislative frameworks; 3) integrating policies; 4) providing leadership and advocacy; and 5) developing and allocating human resources adequately. These functions are frequently considered at a higher level, but the case can be made for their relevance at district level in the context of decentralization, and given the increasing requirement for public structures to ensure the viability of their traditionally assigned roles.

With respect to communications/IEC it would be useful to assess the district’s capacity in organizing and providing information (for example, interventions such as information campaigns) to the communities about ART and other HIV services available, be it directly, by outsourcing, or through partnerships.

Because this tool is intended for use at district/province level, it should include both general elements that address the operationality of the broader health system at intermediate level and also elements which address HIV-specific capacity requirements for the delivery of ART services. Drawing from the literature and from key informant interviews, the proposed assessment tool under development would be most relevant if it took a system-wide approach, incorporating other closely related health service areas such as MCH and TB, at least as pertains to HIV service delivery—particularly as it is often difficult to clearly demarcate between HIV and other services.22 One key informant noted that the selection of proposed domains includes all of the core WHO building block elements, but has simply broken some down into multiple domain areas; this simply allows for a more in depth analysis of any given building block and essentially gives strong ‘weight’ to this building block in the assessment.

Regarding the Human Resources domain, the literature and the feedback from key informant interviews suggests that it will be critical to extend the analysis around this domain beyond just clinical services-- equally relevant is HR capacity at program management level (e.g. HR shortages do not only concern clinical staff, but also management staff). With respect to governance issues, the proposed domains capture two key elements: “Strategic Planning for integrated HIV services” and “Stakeholder management and coordination.” Our review suggests the importance of incorporating indicators on accountability mechanisms (both upward and downward) to determine the extent to which there is external accountability at the district level. An important element to consider in relation to “supervision of clinical services” is the question

21 Tawfik, Y., et al. Introducing Antiretroviral Therapy (ART) on a Large Scale: Hope and Caution. Program Planning Guidance Based on Early Experience from Resource-Limited and Middle-Income Countries. Washington DC, USA: Academy for Educational Development Global Health, Population and Nutrition Group. November 2002.22 Negussie; Hodgins; Travis, Phyllida et al. “Subnational Health Systems Performance Assessment: Objectives, Challenges and Strategies.” Chapter 59 in Health systems performance assessment : debates, methods and empiricism / edited by Christopher J.L. Murray, David B. Evans. WHO 2003.

10

around whether managers have the “real authority” to reward or sanction staff based on performance and at what level this authority lies. The domain on stakeholder coordination should look at coordination across different sectors and also across different actors (private providers, NGOs, agencies, CBOs, etc).

Conclusion This literature review has brought to light a number of issues for consideration while developing specific indicators under these domain areas relevant to district/province level application. Most of these considerations are of broader relevance for the development of other similar district/province capacity assessment tools.

Content of Assessment

Based on this review of literature and key informant interviews, we recommend a district/province capacity assessment tool which is structured around the six WHO health systems building blocks, along with two additional domains: one focusing on community linkages and another one focusing on service integration. Given the importance of community-level interventions in HIV prevention and treatment work, particularly around linkages between the community and health facilities, the capacity domain for ‘community linkages,’ looks at various aspects of client participation or engagement in the production and consumption of health-related services23. The services integration domain will be primarily concerned with the HIV care and treatment service integration within the larger provincial or district portfolio, particularly given the degree to which HIV care overlaps with elements of MCH and TB care. All together, these core assessment areas should provide a systematic and comprehensive review of the health system at district/province level (see table below).

Proposed Core Domains for District/Province Capacity Assessment Tool

Strategic Planning for Integrated HIV Services

Human Resource Management for Clinical Services

Supervision of Clinical Services

Health Facility Renovation/Maintenance/Equipment

Laboratory Services Support

Pharmacy Supply/Logistics

HMIS, M&E Systems

Quality Management/Improvement Systems

Stakeholder Management and Coordination

Financial Management of Donor Funding for HIV services

Community Linkages

23 According to WHO, a health system consists of all organizations, people and actions whose primary intent is to promote, restore or maintain health. Among other things, it includes community level actions such as a mother caring for a sick child at home. Everybody’s Business: Strengthening Health Systems to Improve Health Outcomes. Geneva:WHO, 2007, p. 2).

11

Integration of Services

Capturing Capacity and Performance

For each of these domain areas, both the capacity and the performance of the health system will be assessed in order to paint a more complete picture of the district’s or province’s level of competence. It is important to note that assessing capacity is asking a question about the likelihood of performance being achieved through the range of capabilities of an institution (in order to develop areas showing insufficient capacity). Therefore, while capacity does not always lead to performance, one would expect that – as a group – organizations with the sufficient level of capacity will tend to have better performance. Conversely, observed performance suggests that at least some measure of capacity exists. The performance indicators for the proposed rapid assessment tool should be selected with the intention of quantifying, to the extent possible, the corresponding capacity indicator and providing evidence of the capacity being assessed. While we cannot truly “validate” a capacity measure through a performance measure, the combination of appropriate capacity with acceptable performance is a robust indicator of the measured capacity. Similarly, weak capacity associated with no performance strengthens the case for the need to redress capacity. Disconnects between capacity and performance also suggest the need for further questions and analyses.24

Application of Assessment

With this understanding of capacity and performance and the relationship between the two, we propose that a rapid assessment tool could measure capacity through a facilitated self-assessment exercise with DHMT or PHMT members focusing on easily identifiable standards established in the literature reviewed and supplemented by a verification exercise, such as a document review. The overall aim of the tool should be to quickly establish whether a particular district or province has “good” capacity or not. The definition of “good” is based on the purpose of the tool, which in this case is to distinguish provinces or districts with a capacity level likely to allow sustaining performance in HIV/AIDS care and treatment versus those with a capacity level insufficient to expect sustainable contribution to organization and delivery of HIV/AIDS care and treatment. This kind of rapid assessment should allow for the identification of gross variation and differences in capacity in order to serve as a management tool to identify those districts that are doing very well, and those that are doing very poorly.

In addition, it should be noted that some of the capacity and performance indicators are not completely under the control of the province or district. While the assessment exercise should take into account that some capacity areas are strongly influenced by either higher or lower levels of the system, the tool should be able to gauge how the district/province itself is faring, in so far as its control/autonomy over a particular capacity area permits. As it is expected that the strong leadership, direction, and overall capacity of the DHMT should translate into strong capacity at the service delivery level, it would therefore be useful for such an assessment to include a validation component that is conducted at the health facility. Though there are various

24 For example, weak capacity with strong performance suggests that individuals go out of their way to redress the weakness of the organization, or that an actor in the environment has palliated for the weakness of the organization. In both cases, the sustainability of the performance is in question. On the other hand, strong capacity with weak performance suggests a maladjustment with the environment, individual or organizational occult (for our assessment tool) dysfunction. This might even represent a weakness of our assessment methodology.

12

other factors (besides the level of capacity of the district) that influence the capacity of the health facility, it will be beneficial for such an assessment to look at the capacity of a small sample of health facilities (three to six per district) in order to highlight whether there are any glaring discrepancies with the information shared by the DHMT that may require further investigation.

Therefore, in order to gauge overall performance of the district/province health system, quantitative indicators based on data from HMIS records, which reflect actual performance according to clear benchmarks, can be used (with ratings such as ‘good,’ ‘fair’ or ‘poor,’ determined according to clear thresholds of performance at each level).

It is important to note that this kind of rapid assessment tool will not provide a comprehensive assessment of district performance. What it offers is a rapid check on a limited number of performance indicators for different capacity areas. Capacity is required for performance, but is neither sufficient to achieve it, nor linearly related to it in a predictable manner. In this way, the proposed tool serves as a rapid exercise in triangulating sources of data to guide management decisions and to instigate further in-depth investigation.

Other Considerations

Domains and Indicators

Through this literature review, we have identified a number of other items as important or emerging issues critical to the capacity of the health system at sub-national level. The items that we will consider as we proceed with the design of tool options include assessment areas and indicators related to:

Community linkages, including health system communication with communities and health service delivery at community level

HR capacity in management, in addition to technical/clinical capacity Accountability mechanisms—both ‘upstream’ from district/province to national level and

‘downstream’ from district/province level to service delivery level HIV/AIDS as not only an infectious disease, but also a chronic condition which requires

long-term support interventions for the clients and the relevant health system infrastructure to deliver such care.

Patients’ input regarding HIV care and treatment services Policy environment, particularly national standards and protocols that enable the health

system to respond adequately to the local HIV/AIDS epidemic Communications, both within the health system at different levels and between the health

system and the communities

While there are several tools concerned with measuring quality of care at a health facility and others concerned with the performance of the health system, very few assess the health district or province as an institutional and operational unit. This literature review has not only identified key capacity areas for inclusion in the proposed assessment tool, but also raised a number of process considerations for executing such an assessment at sub-national (intermediary) level. Finally, it introduces the fruits of this labor, in the form of a pre-pilot draft assessment.

13

Documenting the development, testing and use of the forthcoming rapid assessment will fill a gap in the literature and assist others concerned with the important issues related to the management transition and sustainability of HIV care and treatment services.

14

Annexes Annex 1 – Annotated Bibliography

Annex 2 – List of Persons Interviewed

Annex 3 – H-CAT 2 Pre-Pilot Tool Description (attached Word document)

15

Annex 1Annotated Bibliography

1. Alva, Soumya, E. Kleinau, A. Pomeroy, K. Rowan. Measuring the Impact of Health Systems Strengthening. A Review of the Literature. USAID, Nov. 2009.

This is a comprehensive Health Systems Strengthening literature review which seeks to summarize current efforts in measuring health system performance and to highlight the indicators and performance benchmarks most frequently used by the global community. Of particular use for the development of this assessment tool is the list of key indicators by building block (WHO model).

2. Cassels, Andrew & Katja Janovsky. Strengthening Health Management in Districts and Provinces. WHO, Geneva, 1995.

This is a manual that provides guidance for a facilitated self-assessment on sub-national problem identification, analysis and action planning, which focuses on the following broad areas: Finance, Personnel, Staff Training, Statistics, Supplies, Transport, Maintenance, Health Education Community Involvement. It is not directly relevant to the purposes of the proposed district/provincial level assessment.

3. Chatora, Rufaro and Prosper Tumusiime. Health Sector Reform and District Health Systems. District Health Management Team Training Modules, Module 1. Brazzaville, WHO, 2004.

This is one of a set of four management training modules aimed at District Health Management Teams in the countries of the African Region. Module 1 has been developed to assist each team member to work as a leader and manager in leading health development in the district. It is of particular relevance for the development of the assessment tool as it provides a useful framework for translating the functions of the health system at national level into equivalent roles at district level, according to 4 broad health system function domains: stewardship, resource generation, service delivery and financing. It also redefines a number of key roles at province or regional level, particularly relevant in a context of decentralization.

4. Family Health International [FHI]; Ethiopia. Addis Ababa. City Administration Health Bureau. Addis Ababa HIV care and support service assessment. Addis Ababa, Ethiopia, FHI, 2002 Aug.

A descriptive cross-sectional study was conducted to assess existing care and support services. This assessment looked at HIV/AIDS care and support services at the management and coordination level and the service provision level, focusing on “how people working at service management and coordination level organize their work, what guidelines they use, training they give to service providers, and the needs they identify to be able to function optimally.” (P.1). At the service provider level, it looks at what they do and what they need to function well. Conclusions of interest: the assessment began with a discussion about role distribution among the partners, including various government levels and NGOs. Although few programmers/providers address stigma, feedback from service providers indicated that stigma reduction interventions should be given overall higher priority.

5. GTZ: Assessment of the District Health System Using Qualitative Methods (1994)

16

(See Maier et al, 1994)

6. Health Metrics Network, 2006b. Strengthening Country Health Information Systems: Assessment and Monitoring Tool. Geneva: WHO.

The Health Metrics Network (HMN) began in 2005 to help countries and other partners improve global health by strengthening health information systems. This tool answers the need for an effective assessment of the existing national HIS in order to establish a baseline and to monitor progress. See Annex 2 for details.

_________________________________________________________________________________________7. Health Systems 20/20: HIV/AIDS Program Sustainability Analysis Tool (HAPSAT). Abt

Associates Inc, 2007.(See below and Annex 2 for details.)

8. Health Systems 20/20. Using HAPSAT for HIV Program Sustainability Analysis: An Introductory Guide. April 2010. Bethesda, MD: Abt Associates Inc.

HAPSAT is a Microsoft Excel-based tool for forecasting and analyzing the sustainability of HIV programs at national (country) level. HAPSAT calculates resource needs for an HIV/AIDS program and also compares projected resource needs to expected resource availability. The resources required to provide these services are broken into three categories: financial resources (donor and government funding), human resources for service delivery (e.g. doctors, nurses, lab technicians, pharmacists, administrative staff), and physical capital (buildings, vehicles, laboratory machines). This tool is directly relevant to the financial management domain; one of the proposed tool designs will incorporate elements from this tool. See Annex 2 for details.

9. Hirschhorn, L., A. Fullem, C. Shaw, W. Prosser, and M. Noguera. Tool to Assess Site Program Readiness for Initiating Antiretroviral Therapy or Capacity for Existing ART sites. Boston, MA: John Snow, Inc., 2003

This tool, which can be used either for site self-assessment or by external reviewers, was designed to provide sites and programs with a set of criteria to assess a site's readiness to implement ART or the current capacity and needs of an existing program, and to identify key areas that need strengthening. The rating system identifies sites along 5 progressive stages of readiness, which can be adapted for the assessment tool. This tool is also particularly useful for its development of domain areas along the 6 WHO building blocks for ART-specific programs. See Annex 2 for details.

10. Islam, M., ed. 2007. Health Systems Assessment Approach: A How-To Manual. Submitted to the U.S. Agency for International Development in collaboration with Health Systems 20/20, Partners for Health Reformplus, Quality Assurance Project, and Rational Pharmaceutical Management Plus. Arlington, VA: Management Sciences for Health.

(See Thi Mai Oahn, 2009)

___________________________________________________________________________

11. Israr, S.M., A. Islam. Good Governance and Stability: a case study from Pakistan. Int J Health Plann Mgmt 2006; 21: 313–325. Published online in Wiley InterScience (www.interscience.wiley.com) DOI: 10.1002/hpm.852

17

This paper argues that good governance, characterized by transparency, accountability and meaningful community participation, is an essential component of sustainability for donor-funded health systems projects in the public health sector. It describes the failure of an effort to decentralize Pakistan’s health system, an effort which also encouraged community participation and democratic decision making in a classically hierarchical management structure. The authors introduce a Sustainable Management Approach “that can be used to ensure the sustainability of health systems projects, particularly those funded by international organizations in developing countries” (p. 313). The approach consists of these parameters: participation and consensus orientation; building institutional capacity; strategic vision and appropriate leadership; effectiveness, efficiency and responsiveness; and accountability, and includes a list of output and outcome indicators. Important conclusion: “Often health sector reforms fail to achieve their intended results mainly due to a failure in addressing the complex dynamics of the existing system. Establishing and sustaining an enabling environment is imperative for introducing and successfully implementing a large-scale project” (p. 324).

12. Kielmann, Arnfried A. Assessing district health needs, services, and systems: protocols for rapid data collection and analysis. African Medical and Research Foundation, Macmillan Education Ltd. (Nairobi, Kenya, 1991).

Reviewed as part of Sambo, L.G. et al. Tools and Methods for Health System Assessment: Inventory and Review. Geneva, WHO, 1998.

13. Kloss-Quiroga, B. (Ed.). InWEnt: District Health Management Tools: Facilitator’s Manual. Berlin, 2004

“Since the mid 1980s, the health district as an operational unit has become the essential element of health policy concepts in low income countries (Harare conference in 1987)” (p.2). This manual is a training guide and reference for trainers, facilitators or presenters who have some experience with District Health Management. The concept and principles of Primary Health Care form this approach to District Health Management. Besides technical knowledge, District Health Managers need leadership ability, communications skills, process-oriented thinking and the capacity to co-operate in local and regional networks. This experiential, participatory training approach promotes practical knowledge and skills, gender/diversity-sensitivity, and the exchange of ideas and personal experiences. See Annex 2 for details.

14. Kolyada, Lena M.Sc. Health Systems Strengthening and HIV/AIDS: Annotated Bibliography and Resources. Partners for Health Reformplus. Funded by U.S. Agency for International Development. Abt Associates Inc. March 2004.

Four resources were selected for review from this Bibliography: Tawfik & Kinoti, 2002; Hirschhorn et al 2003; Noguera 3t al, 2003; and Okera et al, 2003.

15. LaFond, A., L. Brown, K. Macintyre. Measuring Capacity in the Health Sector: a Conceptual Framework. Int J Health Plann Mgmt 2002; 17: 3-22. Published online in Wiley InterScience (www.interscience.wiley.com) DOI: 10.1002/hpm.649

18

and

LaFond, A., L. Brown. A Guide to Monitoring and Evaluation of Capacity-Building Interventions in the Health Sector in Developing Countries. MEASURE Evaluation Manual Series, No. 7. March 2003. MEASURE Evaluation Project.

In this publication (available under a peer-reviewed format and a more detailed report by MEASURE Evaluation), LaFond and Brown propose a conceptual framework for understanding and measuring capacity in the health sector. They review type of indicators to be considered along the input-process-output-outcome continuum by level of analysis, from the Health System to the Organization onto the individual Health Worker. The document reviews assessment and measurement approaches available at the time, but is particularly pertinent for its conceptual argumentation of the definition of capacity and identification of challenges and limitations in its assessment and measurement. Although the Health District Office can be approached as an Organizational Structure within the Health System, the review does not provide enough focus on this specific level to inform the proposed district/provincial level assessment.

16. Maier, B., R. Görgen, A.A. Kielmann, H.J. Diesfeld, R. Korte. Assessment of the district health system – using qualitative methods. GTZ/WHO/ITHÖG Heidelberg. (1994)

This manual aims to help health Professionals arrive at a comprehensive understanding of health needs, Services and Systems in a particular area through a process of rapid qualitative data collection and analysis guided by a set of structured protocols. Data is collected at district, facility and household level; information is obtained by interviewing knowledgeable informants and by reviewing locally available documents. Of particular relevance for this project are the sections on 1) District Health Management & Support Systems and 2) Community participation and traditional health systems. See Annex 2 for details.

17. Makombe SD, M. Hochgesang, A. Jahn, H. Tweya, B. Hedt. Assessing the quality of data aggregated by antiretroviral treatment clinics in Malawi. Bulletin of the World Health Organization. 2008 Apr; 86 (4): 310-314.

This study assessed the quality of quarterly aggregate summary data compiled and reported by ART facilities (“site report”) as compared to the “gold standard” facility summary data compiled independently by the Ministry of Health supervision team (“supervision report”). It explores the various factors influencing data quality and highlights supervision and human resources for record keeping (clerks) as two actionable areas which positively affect data quality. Also outlined in this article are several recommendations to ensure data quality, which can be translated into relevant assessment indicators for the HMIS/M&E domain of the tool.

18. Ministry of Health-Zambia. Zambia HIV / AIDS Service Provision Assessment Survey 2005. Lusaka, Zambia, Ministry of Health, 2006 Jul.

The ZHSPA survey measures the capacity of health facilities to provide preventive services as well as to meet the care and support needs of people living with HIV/AIDS and their families through two data collection instruments: the facility resources audit questionnaire and the health worker interview questionnaire. The core HIV/AIDS health services components assessed include: HIV/AIDS-related outpatient and inpatient care that includes general health system management; Infection prevention and compliance to standard precautions; HIV/AIDS testing and counseling; Antiretroviral therapy; Prevention of mother-to-child transmission services

19

(PMTCT); Health management information systems (record keeping) relevant to HIV/AIDS; Laboratory diagnostics for HIV, tuberculosis, malaria and most common STIs, and; Availability and management of essential medications, supplies, and treatment services related to HIV/AIDS. The indicators herein will be useful for development of the verification piece of the assessment tool. See Annex 2 for details.

19. MSH, 2003. Human Resource Management Rapid Assessment Tool for HIV/AIDS Environments. Cambridge, MA: MSH. http://erc.msh.org/newpages/english/toolkit/hr_hiv_assessment_tool.pdf

This tool outlines a process to help an organization to quickly assess the performance of its human resource management system and develop an action plan for making necessary improvements, with a focus on HIV/AIDS workplace strategies. It contains 26 HRM components (sub-domains) which are grouped within five broad areas of HRM: HRM capacity (staffing, budget, and planning); personnel policy and practice; performance management; training; HRM data. This tool can serve as a reference for the types of HRM issues that must be addressed at every organizational level in order to better plan, staff and implement HIV/AIDS programs.

20. Management Sciences for Health. Health Systems in Action: An eHandbook for Leaders and Managers. Cambridge, MA: Management Sciences for Health, 2010. Available online at http://www.msh.org/resource-center/health-systems-in-action.cfm and as a CD-ROM.

This handbook is a comprehensive, practical guide which includes a set of tools and resources that address common issues in leading and managing health services; it is also available in electronic version to facilitate access to external linkages. The handbook is designed to address various key elements of an effective health system, which include strong leadership and management; good governance; well-crafted plans; professional human resource management; sound financial management; good management of medicines and health products; monitoring and evaluation focused on results; and the delivery of high-quality health services. For purposes of the tool design, this handbook will serve as a good reference material for concepts and approaches regarding each of the 6 WHO building blocks.

21. Nash, D et al. Strategies for More Effective Monitoring and Evaluation Systems in HIV Programmatic Scale-up in Resource-limited settings: Implications for Health Systems Strengthening. Journal of Acquired Immune Deficiency Syndrome, Vol. 52, Supp.1, Nov.1, 2009.

This article discusses common challenges to M&E systems used in the rapid scale-up of HIV services as well as innovations that may have relevance to systems used to monitor, evaluate, and inform health systems strengthening. It is particularly relevant for the HMIS/M&E domain of the assessment tool as it provides guiding principles for sustainable national M&E systems which can be adapted into indicators for sub-national level application.

22. Noguera, M., D. Alt, L. Hirschhorn, C. Maponga, P. Osewe, and A. Sam- Abbenyi. Zimbabwe: Antiretroviral Therapy Program – Issues and Opportunities for Initiation and Expansion. Arlington, VA: DELIVER/John Snow, Inc., 2003

20

This report presents the findings of a four-week assessment of the readiness and capacity of Zimbabwe’s health sector to deliver the range of services and manage the health commodities required for effective antiretroviral (ART) treatment. The assessment looks at all the key functions of the logistics system required for the ART program, which will be relevant for the development of the pharmacy supply/logistics domain of the assessment tool.

23. Nordberg E, H. Oganga, S. Kazibwe, J. Onyango. Rapid assessment of an African district health system. Test of a planning tool. Int J Health Plann Manage. 1993 Jul-Sep;8(3):219-33

The authors report on a rapid health system assessment conducted in rural Kenya in 1991. The assessment included a self-administered questionnaire for each health facility; interviews with officers in charge of each health institutions in the sub-district, and a review of records and reports at each facility. Generally, the focus was on service provision and the R-HFA is considered to be a stronger tool for these purposes; however, we may adapt an open-ended question used in this tool for qualitative information on the greatest challenges faced at the District/province level.

24. Nyenwa, J., D. Alt, A. Karim, T. Kufa, J. Mboyane, Y. Ouedraogo, and T. Simoyi. Zimbabwe HIV & AIDS Logistics System Assessment. Arlington, Va.: John Snow, Inc./DELIVER, for the U.S. Agency for International Development, 2005.

The LIAT (Logistics Indicator Assessment Tool) and LSAT (Logistics System Assessment Tool) tools assess the “health” of the logistics system for HIV/AIDS related commodities and supplies. These are facility-level assessments that include key informant interviews and observation which use quantitative indicators to assess various aspects of logistic systems for HIV/AIDS programs. The aim of the assessment tools is to understand the challenges facing the logistics systems and to obtain a description of the supply chain system for several commodities. The elements from these tools will be adapted for use in the Pharmacy/logistics supply domain of the assessment tool. See Annex 2 for details.

25. Okera, A., J. Serutoke, E. Madraa, and E. Namagala. Scaling Up Antiretroviral Therapy. Ugandan Experience. Draft. Geneva, Switzerland: World Health Organization, May 2003.

This document is a case study on the early years of ARV therapy scale-up in Uganda (2002-2003); it provides an overview of the factors which influenced the success of this scale up and in the process highlights some of the critical elements to ARV programs which could be relevant for inclusion in the assessment tool.

26. Orobaton N, X. Nsabagasani, E. Ekochu, J. Oki, S. Kironde, T. Lippeveld. Promoting unity of purpose in district health service delivery in Uganda through partnerships, trust building and evidence-based decision-making. Educ Health (Abingdon). 2007 Aug;20(2):58. Epub 2007 Aug 20.

This publication presents selected results of the UPHOLD project in Uganda, a $76 million USAID-funded health and education sector project. It does not present a district health assessment methodology, but discusses important concepts in district capacity. The project was structured around two pillars of intervention: trust building and evidence based planning and decision-making. The former is central to the social capital literature but possibly under-appreciated in approaches to assessing district capacity. Also essential, are two elements in the

21

strategic approach: “unity of purpose” within the district management team and among district partners (unity and consistency of purpose are central concepts of quality management in the broader management literature); and the development of “core values”, such as teamwork, empowerment, excellence, innovation, “boundarylessness”, and responsible speed of program implementation. Partnership was another central strategic element, translated into the districts supporting grants to CSO partners. Finally, other elements of capacity referred to in this publication are more traditionally integrated in capacity assessment tools within a health system; this includes use of quality improvement processes, partnership with CSOs, and institutionalization of key practices supported by the UPHOLD project.

Limitation: this paper does not offer how to operationalize or measure the district capabilities directly or indirectly related to these topics. This deserves consideration in view of the development of the assessment tool.

27. Population Council. Horizons; SHARAN: Society for Service to Urban Poverty. The PLHA-friendly achievement checklist. A self-assessment tool for hospitals and other medical institutions caring for people living with HIV / AIDS (PLHA). New Delhi, India, Population Council, Horizons, 2003.

The PLHA-friendly Achievement Checklist is intended as a self-assessment tool for managers to use in gauging how well their facility (hospital, clinic, or department) reaches, serves, and treats HIV-positive patients. This gives managers an opportunity to identify institutional strengths and weaknesses, consider ways to address the weaknesses, and later to assess progress toward PLHA-friendliness. The tool components can serve as a model for the verification piece of the rapid assessment. See Annex 2 for details.

28. Project Concern International, I-STAR (Integrated System for Transformation, Assessment and Results) User Guide (PCI and EDC, 2005)

I-STAR is a comprehensive approach to building effective and sustainable non-governmental organizations (NGOs) and networks that are able to contribute to improving the health and development of the communities they serve. The cornerstone of I-STAR is a capacity self assessment. The tool bears no direct relevance to the development of the rapid assessment tool.

29. Sambo, L.G. et al. Tools and Methods for Health System Assessment: Inventory and Review. Geneva, WHO, 1998.

This document summarizes 26 tools and methods that review all or part of the health system. Overall, most tools review inputs, outputs, and outcomes of specific health services/programs rather than assess insitutional factors influencing implementation. For the most part, quantitative indicators (for determining service outputs and health status) are used. Tools included in this inventory that were reviewed in the literature review include Maier et al, 1994.

30. Sambo et al, Tools Assessing the Operationality of District Health Systems. Brazzaville, WHO, 2003.

Assessment of the operationality of a district health system can be described as the review of the organization and management of a health system in terms of its structures, managerial processes, priority health activities, community participation and the availability and management of resources; it does not include the assessment of its performance. The tools presented here

22

include a health facility and a district questionnaire which are administered through a self-assessment methodology. The domains included in this framework are relevant for the tool, particularly the domains on structures, managerial processes, community health initiatives and availability of resources. See Annex 2 for details.

31. Sarriot, Eric. Issue Paper: Supervision of Health Care in Developing Countries. Quality Assurance Project, University Research Corporation (1999).

This report presents supervision as both a control and a service function of management. It presents several positive and negative supervisory behaviors influencing provider performance; a facilitative supervision description; and a diagram on the management culture of supervision (balancing Concern for People vs. Concern for Task) which will be relevant for the development of indicators for the supervision domain.

32. Sarriot, Eric, J. Ricca, L. Ryan, J. Basnet, and S. Arscott-Mills. Measuring sustainability as a programming tool for health sector investments—report from a pilot sustainability assessment in five Nepalese health districts. Int J Health Plann Mgmt 2009; 24: 326–350. Published online in Wiley InterScience (www.interscience.wiley.com) DOI: 10.1002/hpm.1012

In this report, the authors develop a standardized set of assessment tools to measure 53 indicators along six components of assessments measured for the most part at district level according to 6 main components. Of key interest were the components looking at District level structures’ capacity and viability. The study also identified a range of central functions (e.g., budgeting, planning, policymaking, establishing standards and protocols, human resources allocation) outside the control of the districts and therefore not relevant for direct assessment at this level.

33. Sarriot, Eric, Shamim Jahan, and Sustainability Evaluation Team. Sustainability of the Saidpur and Parbatipur Urban Health Model (Bangladesh) Five Years After the End of Concern’s Child Survival Project. Final Evaluation Report-January 10, 2010.

This report details the systematic assessment of all components of the Concern Worldwide-adapted ‘sustainability framework’ which allowed strategic recommendations to be made to municipalities and stakeholders, in evaluating the sustainability of health outcomes and reviewing changes in the two municipalities studied. There was limited direct relevance to the development of the rapid assessment tool, as Municipal Health Departments share only some of the responsibilities common to a Health District and do not oversee curative services.

34. Shediac-Rizkallah MC, LR Bone. 1998. Planning for the sustainability of community-based health programs: conceptual frameworks and future directions for research, practice and policy. Health Educ Res 13(1): 87–108.

Shediac-Rizkallah offers a definition which synthesizes a diversity of concepts and approaches about sustainability, specifically in health promotion programs. For her, sustainability refers to the general phenomenon of continuation of a health-enhancing program, and she offers three main mechanisms through which this can be observed: 1) The maintenance of health benefits achieved through the initial program; 2) The continuation of program activities within an organizational structure (institutionalization); and 3) The maintenance of health benefits through building the capacity of the recipient community. This definition focuses on the ultimate benefit

23

for the communities, and recognizes a wide array of stakeholders as responsible for the sustainability of health programs.

Factors in the project design (negotiation, effectiveness, duration, financing, type, training), factors in the organizational setting (institutional strength, integration, leadership), and factors in the community environment (socioeconomic-political environment and participation), interact to influence sustainability, which is achieved through a continuation of the initial program, or its institutionalization within a new (local) organizational structure, or through the development of the capacity of the recipient community.

The main relevance to the district/province assessment tool is the recognition of the interconnectedness of different stakeholders. (This suggests that the ability of districts to partner, cooperation, provide guidance to other partners is a capacity area deserving examination.) Beyond this, this paper does not offer enough institutional level focus to provide measurement tools or resources.

35. Stein J., S. Lewin, L. Fairall, P. Mayers, R. English, A. Bheekie, E. Bateman, M. Zwarenstein. Building capacity for antiretroviral delivery in South Africa: a qualitative evaluation of the PALSA PLUS nurse training programme. BMC Health Serv Res. 2008 Nov 18;8:240.

This article describes the evaluation of a training program for PHC nurses in the management of adult lung diseases and HIV/AIDS, including ART. The authors used qualitative methods and analyzed data thematically. Outcome of interest: training all PHC nurses in use of this guideline, as opposed to ART nurses only, was perceived to better facilitate the integration of AIDS care within the clinics.

36. Tawfik, Y., S. Kinoti, and G.C. Blain. Introducing Antiretroviral Therapy (ART) on a Large Scale: Hope and Caution. Program Planning Guidance Based on Early Experience from Resource-Limited and Middle-Income Countries. Washington DC, USA: Academy for Educational Development Global Health, Population and Nutrition Group, November 2002.

This paper provides program planning and management guidance to resource-limited countries that seek to implement expanded ART services. In addition, this paper includes guidance for estimating program costs, human resources and training requirements, and examples of communication messages specific to ART programs, which will be relevant for the development of the design of the rapid assessment tool.

37. Thi Mai Oanh et al, Assessing Provincial Health Systems in Vietnam: Lessons from Two Provinces. March 2009. Bethesda, MD: Health Systems 20/20 project, Abt Associates Inc.

This report presents data from the first two provincial health system assessments conducted in Vietnam. The HSA tool was developed to enable policymakers and program managers to undertake a comprehensive view of six major health systems functions delineated by WHO (governance, finance, human resources, service delivery, pharmaceutical management, and health information). The tool allows users to assess each health system function using a set of performance indicators. It will be a key reference in the development of the rapid assessment tool. See Annex 2 for details.

24

38. Travis, Phyllida et al. “Subnational Health Systems Performance Assessment: Objectives, Challenges and Strategies.” Chapter 59 in Health systems performance assessment : debates, methods and empiricism / edited by Christopher J.L. Murray, David B. Evans. WHO 2003.

This chapter of the publication highlights key considerations regarding the purpose and challenges of health systems performance assessments, which will guide the design of the tool. Of key interest are the discussions around the various types of information proposed for collection at sub-national level; the application of the WHO health systems framework at the local level; specific challenges for data collection and use at sub-national level.

39. USAID Child Survival and Health Grants Program. R-HFA for core Maternal, Neonatal, and Child Health (MNCH) services at the primary level (2007) (DHO Module)

The Rapid Health Facility Assessment (R-HFA) was developed in 2006 by ICF Macro in collaboration with MEASURE Evaluation and a panel of experts from US PVOs, USAID, and other cooperating agencies. Pilot testing has shown that it is suitable for use by District Health Management Teams (DHMTs), as well as NGOs. It is a relatively rapid instrument for measuring a small set of key indicators to give a "balanced scorecard" for MNCH services at the primary health care level (including an optional module for use with CHWs for community outreach services). It identifies key bottlenecks to quality service delivery.

Domains include structure and administration, planning , budget management, and coordination. The tool combines indicator scores for each domain with other indicators of capacity and performance collected from health facilities, to create an overall score for DHMT. Indicators collected at health facility level for this purpose include: guidelines, supervision, training, data for decision making, access or availability of services and quality-related issues like staffing, infrastructure, supplies, drugs, infection control, community/provider relations, HW technical performance, and client satisfaction. See Annex 2 for details.

40. USAID Center for Development Information and Evaluation. MEASURING INSTITUTIONAL CAPACITY. Recent Practices In Monitoring and Evaluation. TIPS. 2000, Number 15.

This grey publication provides information on the measurement of institutional capacity, including some tools that measure the capacity of an entire organization as well as others that look at individual components or functions of an organization. The discussion concentrates on the internal capacities of individual organizations, rather than on the entire institutional context in which organizations function. It is fairly generic, possibly more appropriate to civil society organization and provides no district-level focus. Annex presents links to a number of tools and a full set of assessment questions from the Organizational Capacity Assessment Tool (OCAT).

41. WHO. Innovative Care for Chronic Conditions: Building Blocks for Action. WHO, 2002.

This document presents a framework for chronic care, which emphasizes patient, family, and community involvement. It proposes specific building blocks for health care organization which include promote continuity and coordination; encourage quality care through leadership and incentives; organize and equip health care teams; support self-management and prevention; use information systems. This model has broader relevance to the rapid assessment and will be incorporated into one of the tool designs. See Annex 2 for details.

25

42. WHO. Everybody’s business: strengthening health systems to improve health outcomes : WHO’s framework for action. World Health Organization, Geneva (2007).

This document presents a conceptual framework for health systems which highlights six key building blocks in any health system: Service delivery; Health workforce; Information; Medical products, vaccines & technologies; Financing; Leadership and governance. This framework will serve as a key source of comparison for the proposed domain areas. See Annex 2 for details.

43. WHO. Patient Monitoring Guidelines for HIV Care and Antiretroviral Therapy (ART). MEASURE Evaluation. WHO, 2006.

This document provides guidelines to aid in the development of an effective national HIV care and antiretroviral therapy (ART) patient monitoring system. The focus of these guidelines is the list of essential minimum standard HIV care and ART patient monitoring data elements and how their collection facilitates clinical care and measurement of agreed upon indicators. The list is broken down into four categories: demographic information; HIV care and family status; ART summary; and patient level encounter information. This document provides good detailed information on the specific kinds of data required for patient monitoring, which should be aggregated and collected at district level and could be useful in developing assessment indicators for the HMIS/M&E domain.

26

Annex 2 Individuals Contacted (phone, interview or email) from 4/26/2010 to 5/11/2010

Name Title/Organization

Betizazu, Sisay Sirgu  WHO

Boni, Tony Office of Health, Infectious Diseases, and Nutrition-USAID

Doherty, Julie Abt, Inc

Ford, Sara Sr. Technical Advisor, Capacity Building—AIDSTAR Two

Eichler, Rena Technical Team Leader-Health Systems Strengthening, MCHIP

Emery, Bob Office of Health, Infectious Diseases, and Nutrition-USAID

Heiby, Jim Medical Officer, Global Health Bureau-USAID

Hodgins, Steve Global Leadership Team Leader, MCHIP

Lion, Ann Project Director, Health Systems 20/20 (Abt, Inc)

Negussie , Eyerusalem Kebede

WHO-HIV Department, IMAI Team

Seims, La Rue MSH

Taye, Amy Sr. Analyst, Abt, Inc

27

28