documents.worldbank.orgdocuments.worldbank.org/curated/en/...upland-health-su… · web viewadb....

Document of The World Bank

Report No: ICR00003781

IMPLEMENTATION COMPLETION AND RESULTS REPORT

ON A

CREDIT

IN THE AMOUNT OF SDR 38.3 MILLION

(USD60.0 MILLION EQUIVALENT)

TO THE

SOCIALIST REPUBLIC OF VIETNAM

FOR A

NORTHERN UPLAND HEALTH SUPPORT PROJECT

August 24, 2016

Health, Nutrition and Population Global PracticeEast Asia and Pacific Region

CURRENCY EQUIVALENTS(Exchange Rate Effective February 29, 2016)

Currency Unit = Vietnamese Dong (VND)VND 22,174.64 = USD 1.00

FISCAL YEAR January 1 – December 31

ABBREVIATIONS AND ACRONYMS

ADB Asian Development BankCNHSP Central North Health Support ProjectCPMU Central Project Management UnitCPS Country Partnership StrategyDO Development ObjectiveEC European CommunityFA Financing AgreementFHCIC Free Health Care Insurance for Children under six years HCFP Health Care Funds for the PoorHCWM Health Care Waste ManagementHEMA Health Care Support to the Poor of the Northern Upland and Central

HighlandsHI Health InsuranceHIC Health Insurance CardHIS Health Information SystemHMIS Health Management and Information SystemICR Implementation Completion and Results ReportIEC Information, Education, and CommunicationIMR Infant Mortality RateIOI Intermediate Outcome IndicatorIPF Investment Project FinancingISR Implementation Status and Results ReportKPI Key Performance IndicatorMHI Millennium Development Goal’s Health IndicatorMHSP Mekong Regional Health Support ProjectMDG Millennium Development GoalM&E Monitoring and EvaluationMMR Maternal Mortality RateMoH Ministry of HealthMTR Midterm ReviewNMR Neonatal Mortality RateNORRED North East Red River Delta Region Health System Support ProjectNUP Northern Upland Health Support Project

PAD Project Appraisal DocumentPDO Project Development ObjectivePPMU Provincial Project Management UnitQER Quality Enhancement Review RF Results FrameworkSBA Skilled Birth AttendantSHI Social Health InsuranceSIL Specific Investment LoanTTL Task Team LeaderUHC Universal Health CoverageVHI Voluntary Health InsuranceVHLSS Vietnam Household Living Standard Survey

Senior Global Practice Director: Timothy G. EvansPractice Manager: Toomas PaluProject Team Leader: Anh Thuy NguyenICR Team Leader: Andre MediciICR Primary Author Andre Medici

VIETNAM

Northern Upland Health Support Project

TABLE OF CONTENTS

A. Basic Information…………………………………………………………………....i

B. Key Dates.....................................................................................................................iC. Ratings Summary.........................................................................................................iD. Sector and Theme Codes............................................................................................iiE. Bank Staff...................................................................................................................iiF. Results Framework Analysis.......................................................................................iiG. Ratings of Project Performance in ISRs....................................................................ixH. Restructuring...............................................................................................................xI. Disbursement Profile...................................................................................................xi

1. Project Context, Development Objectives and Design................................................12. Key Factors Affecting Implementation and Outcomes...............................................63. Assessment of Outcomes...........................................................................................154. Assessment of Risk to Development Outcome.........................................................235. Assessment of Bank and Borrower Performance......................................................246. Lessons Learned........................................................................................................277. Comments on Issues Raised by Borrower/Implementing Agencies/Partners...........28

Annex 1. Project Costs and Financing...........................................................................30Annex 2. Project Outputs (According PAD -Annex 4).................................................31Annex 2A. Project Output Map……………………………………………………….43Annex 3. Economic Analysis: Outputs Efficiency, Benefits and Equity Impacts…….48Annex 3A Estimated Unitary Costs of the Project Outputs ……………………..……59Annex 4. Bank Lending and Implementation Support/Supervision Processes………..60Annex 5. Results Framework: Analysis of the PDO Achievement…..………...……..62Annex 5A. Rating of the Indicators According Achievement………………………...66Annex 6. Borrowers ICR………………………………………………………….…...71Annex 6A. Project Risk management (Borrowers View)……………………………..86Annex 6B. Project Results Framework (Borrowers View)……………………………89Annex 6C. Achievements by components (Borrowers View)……………………...…92

Annex 7. Summary of the Project Aide-Memoires……………………………………98Annex 8. List of Supporting Documents……………………………………….…….100MAP…………………………………………………………………………….…….102

A. Basic Information

Country: Vietnam Project Name:Northern Upland Health Support Project

Project ID: P082672 L/C/TF Number(s): IDA-43980ICR Date: 08/31/2016 ICR Type: Core ICR

Lending Instrument: SIL Borrower:SOCIALIST REPUBLIC OF VIETNAM

Original Total Commitment:

XDR 38.30M Disbursed Amount: XDR 37.31M

Revised Amount: XDR 38.30MEnvironmental Category: BImplementing Agencies: Ministry of Health, Vietnam Central Project Management Unit (CPMU)The Seven Northern Upland Provinces (Cao Bang, Bac Kan, Lao Cai, Ha Giang, Son La, Dien Bien and Lai Chau) and their respective Provincial Project Management Units (PPMU)Co-financiers and Other External Partners: No

B. Key Dates

Process Date Process Original Date Revised / Actual Date(s)

Concept Review: 05/31/2006 Effectiveness: 10/08/2008 10/08/2008 Appraisal: 11/15/2007 Restructuring(s): — 08/29/2014 Approval: 03/13/2008 Mid-term Review: 07/16/2012 07/16/2012 Closing: 08/31/2014 02/29/2016

C. Ratings Summary C.1 Performance Rating by ICR Outcomes: Satisfactory Risk to Development Outcome: Moderate Bank Performance: Satisfactory Borrower Performance: Satisfactory

C.2 Detailed Ratings of Bank and Borrower Performance (by ICR)Bank Ratings Borrower Ratings

Quality at Entry: Moderately Satisfactory Central Government - CPMU: Satisfactory

Quality of Supervision: Satisfactory Regional GovernmentsPPMU: Moderately Satisfactory

i

Overall Bank Performance: Satisfactory Overall Borrower

Performance: Satisfactory

C.3 Quality at Entry and Implementation Performance IndicatorsImplementation

Performance Indicators QAG Assessments (if any) Rating

Potential Problem Project at any time (Yes/No):

NoQuality at Entry (QEA):

None

Problem Project at any time (Yes/No):

NoQuality of Supervision (QSA):

None

DO rating before Closing/Inactive status:

Satisfactory

D. Sector and Theme Codes Original Actual

Sector Code (as % of total Bank financing) Central government administration 8 8 Compulsory health finance 15 15 Health 71 71 Sub-national government administration 6 6

Theme Code (as % of total Bank financing) Health system performance 100 100

E. Bank Staff Positions At ICR At Approval

Vice President: Victoria Kwakwa James W. Adams Country Director: Achim Fock (Acting) Ajay Chibber Practice Manager/Manager:

Toomas Palu Fadia M. Saadh

Project Team Leader: Anh Thuy Nguyen Maryam Salim ICR Team Leader: Andre Medici — ICR Primary Author: Andre Medici —

F. Results Framework Analysis

Project Development Objectives (from Project Financial Agreement)

The objective of the Project is to increase the utilization of district health services especially among the poor and ethnic minorities population of the Northern Upland

ii

Provinces through improving the quality of district-level hospitals and reducing financial constraints to access to health services.

Revised Project Development Objectives (as approved by original approving authority) The PDO was not revised. The Project originally had four Key Performance Indicators (KPIs) and 10 Intermediate Outcome Indictors (IOI) in the Project Appraisal Document (PAD). However, most of baselines and targets of the project indicators were set during the first year of implementation according to the result of a Baseline Survey conducted in June 2009 and published in July 2009. The four KPIs were retained and the number of IOIs increased from 10 to 13. No other changes were made to the indicators. All KPIs and most of the IOIs’ targets were achieved by December 31, 2015 (before the project Closing Date of February 29, 2016).

(a) PDO Indicator(s)

Indicator1 Baseline Value

Original Target Values (from

approval documents)

Formally Revised Target Values

Actual Value Achieved at

Completion or Target Years

Indicator 1: Utilization rates of in-patient services in District Hospitals among Decision 139 beneficiaries

Value (Quantitative or Qualitative)

0.027 0.033 — 0.096

Date achieved 06/30/2009 08/31/2014 12/31/2015Comments (including % achievement)

Target surpassed (191%).

Indicator 2: Utilization rates of out-patient health services in district hospitals by Decision 139 beneficiaries


0.067 0.075 — 0.247



Indicator 3: Percentage of households who experience catastrophic healthcare expenditures in the year prior to the survey.


14.27% 13.23% — 2.00%

Date achieved 06/30/2008 08/31/2014 08/31/2014Comments (including %

Target surpassed (561%). Not measured during the 18-month extension period.

1 Percentage of achievement in this table was calculated as a simple percentage increase of the last actual on the proposed target.

iii

achievement)

Indicator 4: Proportion of district hospitals that provide full set of health services according to the national norms (Decision 23/205/QB- BYT) adjusted to the Northern Uplands.


39.1% 70.0% — 80.4%


Target surpassed (15%). For operational reasons, after the MTR, this indicator was adjusted to be read as “average percentage of health services covered by the district hospitals”.

(b) Intermediate Outcome Indicator(s)

Indicator Baseline Value


approval documents)

Formally Revised

Target Values



Indicator 1: Percent of patients satisfied with the health services;Value (Quantitative or Qualitative)

8.5% 10.2% — 84.4%



Indicator 2: Adherence of treatment protocols for selected conditions in impatient settings.Value (Quantitative or Qualitative)

— — — —


Measured by sub-indicators 2.1 to 2.9 (all surpassed). Not measured during the 18-month extension period.

Indicator 2.1: Percent of health workers with knowledge of diagnosing and treating Level A/B/C dehydrated diarrhea


9.7% 14.0% — 95.2%



Indicator 2.2: Percent of health workers with knowledge of diagnosing and treating severe pneumonia


13.2% 18.5% — 86.9%

Date achieved 06/30/2009 08/31/2014 08/31/2014

iv



approval documents)

Formally Revised

Target Values



Comments (including % achievement)


Indicator 2.3: Percent of health workers with knowledge of diagnosing and treating poisoningValue (Quantitative or Qualitative)

26.8% 37.5% — 83.0%



Indicator 2.4: Percent of Reasonable diagnoses of severe pneumoniaValue (Quantitative or Qualitative)

45.5% 63.7% — 71.1%

Date achieved 06/30/2009 08/31/2014 08/31/2014Comments (incl. % achievement)


Indicator 2.5: Percent of Clinical health workers’ reasonable diagnosis of general pneumonia.Value (Quantitative or Qualitative)

19.6% 27.4% — 57.0%



Indicator 2.6: Percent of Reasonable diagnoses of Level A dehydrated diarrheaValue (Quantitative or Qualitative)

37.2% 52.1% — 78.9%



Indicator 2.7: Percent of Reasonable diagnoses of Level B dehydrated diarrhea.Value (Quantitative or Qualitative)

48.9% 68.5% — 85.1%



Indicator 2.8: Percent of Reasonable diagnosis of Level C dehydrated diarrhea

v



approval documents)

Formally Revised

Target Values




41.2% 57.7% — 80.0%



Indicator 2.9: Percent of Reasonable diagnoses of poisoning.Value (Quantitative or Qualitative)

61.1% 85.5% — 86.2%


Target achieved. Not measured during the 18-month extension period.

Indicator 3: Percent of eligible district health staff who have successfully completed training provided by the project


— — — —


Measured by sub-indicators 3.1 to 3.3 (all surpassed).

Indicator 3.1: Percent of doctors and assistant doctors and pharmacists at district hospitals trained by the project


0% 80% — 189%



Indicator 3.2: Percent of health staffs with completed short-term training courses compared to the plan.


0% 80% — 357%



Indicator 3.3: Percent of health staffs completed long-term training courses compared to the plan

vi



approval documents)

Formally Revised

Target Values




0% 80% — 88%



Indicator 4: Percent of eligible district hospitals with acceptable operations and maintenance plans and budget for facility and equipment maintenance


— — — —


Measured by sub-indicators 4.1 and 4.2 (both surpassed)

Indicator 4.1: Percent of district hospitals having schedule and budget for maintenance of infrastructure


49.2% 40.0% — 79.7%

Date achieved 06/30/2008 08/31/2014 12/31/2015


This indicator had a target value of 40% in the PAD. The baseline was measured after the project approval. The target value was not revised during implementation to be compatible with the baseline. Consequently, achievement for this indicator has been calculated over the baseline value of 49.2% instead of its target value of 40%. Baseline value surpassed (62%)

Indicator 4.2: Percent of district hospitals having schedule and budget for maintenance of equipment.


77.1% 40.0% — 89.1%

Date achieved 06/30/2008 08/31/2014 06/30/2014


This indicator had a target value of 40% in the PAD. The baseline was measured after the project approval. The target value was not revised during implementation to be compatible with the baseline. Consequently, achievement for this indicator has been calculated over the baseline value of 77.1% instead of its target value of 40%. Baseline value surpassed (16%).

Indicator 5: Number of Health facilities constructed renovated, and/or equipped.Value (quantitative or Qualitative)

0 61 — 64

Date achieved 06/30/2009 08/31/2014 12/31/2015Comments Target achieved. This core indicator was added during project implementation.

vii



approval documents)

Formally Revised

Target Values



(including % achievement) Indicator 6: Percent of recently discharged patients satisfied with health services.Value (Quantitative or Qualitative)

8.5%10.2% — 84.4%


Target Surpassed (727%). This indicator was added during project implementation.

Indicator 7: Number of People with access to a basic package of health, nutrition & reproductive health services.


20% 70% — —


This indicator was added during project implementation. It cannot be measured because the baseline and the target (expressed in percentage) were not converted in the number of beneficiaries with access to the basic package of health, nutrition & reproductive health services during the project life. Nevertheless, the number of beneficiaries reached was 270,254 by December 2015.

Indicator 8: Percentage of Decision 139 beneficiaries who have received Health Insurance Cards


82.1% 70% — 95.2


Target surpassed (16%). The target of this indicator was established before the baseline survey. For this reason, the baseline value in the PAD was higher than the target value. The project team did not revised the target during project implementation. Consequently, the achievement of this indicator has been calculated over the baseline instead the target.

Indicator 9: Percentage of 139 Beneficiaries with cards who can correctly identify at least three benefits covered under the HCFP program


14.8% 75% — 57.4%

Date achieved 06/30/2009 08/31/2014 08/31/2014


The target was not achieved. Not measured during 18-month project extension period. However, the following alternate indicator was measured during implementation “% the poor knowing at least 1 right of health insurance card holders”. The achievement of this indicator was 95% by December 2015.

Indicator 10: Percentage of households who identify financial barriers as a main cause for not seeking health care.

viii



approval documents)

Formally Revised

Target Values




2.0% 1.8% — 1.2%


The target was surpassed (50%). Not measured during the project extension.

Indicator 11: Project management units at central level and provincial level established; bank accounts opened, staff and consultant recruited, system established


— — — Achieved

Date achieved 06/30/2008 06/30/2009Comments (including % achievement)

CPMU/PPMUs management, staff and consultants fully board by June 2009. Project operational and financial systems have been maintained throughout the project life.

Indicator 12: Project management units prepare adequate plans, meet annual implementation targets and provide timely financial and activity report.



Date achieved 06/30/2009 12/31/2015


The project's annual working plans, progress reports, interim financial reports and audits reports were submitted in time with acceptable quality, except for some reports on project result indicators toward the last year of project implementation.

Indicator 13: Availability of data for project monitoring and evaluationValue (Quantitative or Qualitative)


Date achieved 02/15/2016Comments (including % achievement)

The project collected data from project related surveys (2009, 2012, and 2014) and from administrative records by the CPMU. Final project evaluation was conducted and completed in mid-February 2016.

G. Ratings of Project Performance in ISRs

No. Date ISR Archived DO IP

Actual Disbursements(USD millions)

1 11/21/2008 Satisfactory Satisfactory 0.00 2 01/22/2010 Moderately Satisfactory Moderately Satisfactory 3.03 3 02/21/2011 Moderately Satisfactory Moderately Satisfactory 9.47 4 03/26/2012 Moderately Satisfactory Moderately Satisfactory 18.64

ix

5 04/24/2013 Satisfactory Moderately Satisfactory 28.87 6 08/26/2013 Satisfactory Moderately Satisfactory 35.97 7 05/05/2014 Moderately Satisfactory Moderately Satisfactory 49.80 8 11/24/2014 Moderately Satisfactory Moderately Satisfactory 53.51 9 05/14/2015 Moderately Satisfactory Moderately Satisfactory 55.96

10 07/29/2015 Satisfactory Moderately Satisfactory 55.96 11 02/25/2016 Satisfactory Moderately Satisfactory 57.04

H. Restructuring (if any)

Restructuring Date(s)

Board Approved

PDO Change

ISR Ratings at Restructuring Amount Disbursed

at Restructuring in USD, millions

Reason for Restructuring & Key Changes MadeDO IP

08/29/2014 No MS MS 52.1

Level II restructuring: extension of the closing date from August 31, 2014 to February 29, 2016. The project achieved its KPIs’ targets on time and generated savings. The restructuring did not change the PDOs and KPIs/IOIs, but redirected the remaining funds to: (a) Maximize efficiency and enhance the sustainability of the Project investments in the seven provinces, and (b) Contribute to the achievement of Vietnam health MDGs in related areas, especially toward reducing child mortality and improving maternal health. Many of the IOIs (especially those which required a new client survey) were not monitored because Government decisions to do not conduct a project survey after the 18-month extension period. The end of project survey was conducted in May-June 2014 and published in October 2014.

x

I. Disbursement Profile

xi

1. Project Context, Development Objectives, and Design

1.1 Context at Appraisal

1. In the 20 years prior to 2007, Vietnam had one of the highest rates of economic growth and poverty reduction in the world, despite the persistence of higher inequality. By appraisal (2007), the country had accumulated a remarkable development performance because of important economic, social, and political reforms delivered since 1986. These reforms contributed to an average annual growth rate of 7.4 percent per year between 1990 and 2008, coupled with a fast reduction in poverty from 58.0 percent to 14.5 percent in the same period, based on the national poverty line, and a fall in the proportion of people living under a dollar a day from 63.0 percent to 21.5 percent between 1993 and 2006.

2. Despite the fact that an important dimension of the nature of the Vietnamese economic growth had been its inclusiveness, inequalities and disparities across regions remained a challenge. In 1993, poverty in rural areas was 2.6 times higher than in urban areas. By 2008, it was 5.7 times greater. Poverty was concentrated in the Northeast, North, Central Coast, and Central Highlands—with approximately one-third of the population in those regions living below the poverty line (mostly associated with ethnic minorities) by the time of project appraisal.

3. In 2007, most Vietnam’s health-related Millennium Development Goal (MDG) outcome indicators were on track. The under-five mortality rate decreased by 65 percent between 1990 and 2005 from 53 to 19 per 1,000 births. The maternal mortality ratio fell by two-thirds, from 250 deaths in 1990 to 85 deaths per 100,000 births in 2007. Cases of Malaria were significantly reduced, accounting for only 35 deaths in 2007. Tuberculosis programs, since 1997, had reached and exceeded the global targets for control, detecting 70 percent of new smear-positive pulmonary tuberculosis cases, curing 85 percent of these detected cases, and leading to a 44 percent decrease of the incidence rates over the period 1997–2004. No progress was registered in the reduction of HIV-AIDS cases and the incidence went up from 0.34 percent in 2001 to 0.44 percent in 2005 among the general population.

4. Despite progress in health-related MDGs, the burden of disease in Vietnam, at project appraisal, was already concentrated in non-communicable diseases, injuries, and accidents, accounting for almost three-quarters of reported deaths nationwide. In 2008, hypertension prevalence had almost doubled in less than 20 years and close to one-third of adults had high blood cholesterol. Overweight, obesity, and physical inactivity prevalence increased because of changes in nutrition and the labor market structure and smoking rates for males remained at 56 percent in the 10 years before the Project appraisal.

5. Improvements in health system protection. Since 1992, Vietnam’s Government intended to achieve universal health coverage (UHC) by expanding health insurance (HI) mechanisms. Despite this intention only 49 percent of the Vietnamese population was covered by HI at the Project appraisal. The main impacts of the expansion of HI in the 15 years before 2007 were increased utilization of health facilities, reduction of health out-of-pocket payment, and trimming of catastrophic spending risks for families.

1

6. In 2007, Vietnam offered four types of HI for its citizens: (a) Social Health Insurance (SHI), for those employed in the formal labor markets, retirees, and disabled and meritorious people, representing only 9 percent of the population; (b) Health Care Funds for the Poor (HCFP), directed to the poor ethnic minorities in mountainous areas (such as the Northern Upland Provinces) and inhabitants in disadvantaged communities, enrolling 18 percent of the country’s population2; (c) Free Health Care Insurance for Children under six years (FHCIC) addressing 11 percent of the population, and; (d) Voluntary Health Insurance (VHI), covering self-employed and informal sector workers, dependents of the SHI members, and students and children over six years, covering 11 percent of the entire population. The SHI was financed by payroll contributions while the HCFP and FHCIC were financed by fiscal revenues. The VHI was financed by premiums calculated according to the insureds’ ability to pay.

7. Together with the health reforms from the early 1990s to late 2000s, Vietnam introduced changes in the provision of health services. The most important was the liberalization of the health care and pharmaceuticals markets. This was accompanied by the introduction of user fees at public health facilities and the transfer of health workers’ salary payments from local communities to the central Government. Some of these changes did not improve the health care delivery process, particularly in the disadvantage regions, given the poor access and quality of health services in rural areas and district hospitals. Health services were delivered by public and private hospitals, with the latter mostly providing specialized care in urban areas, while primary and basic care were provided by public hospitals and community health centers. A large share of pharmaceuticals was purchased directly by the population from private vendors with traditional medicine playing a major role, as it is recognized by the Government as part of the health system and offered by public and private providers.

8. Despite progress resulting from health reforms, coverage and health outcomes were not uniform within the country. Poverty reduction, HI coverage, and health improvement remained uneven, with some segments of the population lagging behind the national average and high disparities in health indicators among regions. For example, in the Northern Upland Provinces,3 the infant mortality rate (IMR) was 60 per 1,000 live births compared to the national average of 18 per 1,000 live births. While the majority of births were attended by skilled health workers, less than 20 percent of births by ethnic minority women were attended by qualified personnel. In the Northern Upland provinces, the role of district hospitals was compromised by the lack of adequate and qualified human resources and poor conditions of the physical infrastructure. The poor and ethnic minorities did not use these hospitals because of poor quality of the services and difficulty of access. Consequently, these groups were particularly vulnerable to receiving suboptimal health care and achieving poor health outcomes, particularly mothers and children.

9. Rationale for World Bank assistance. The Project design was based on (a) best practices from World Bank health projects and other initiatives in countries similar to Vietnam and (b) the government health policy to improve equity on achievement of health outcomes at

2 These two insurances—SHI and HCFP—are referred to as Compulsory HI Schemes.3 The extremely disadvantaged population of the Northern Upland Provinces led the Government to issue the Decision 139 of 2002 to cover the following beneficiaries: poor, ethnic minorities living in the mountainous provinces and population living in Government-defined difficult communes. The project initially limited the support to the poor, but then expanded to the ethnic minorities following the Decision 139.

2

the regional level. Both recommended interventions on the supply side (improving infrastructure, training human resources, improving management, and maintenance of health infrastructure) and demand side (providing means to the poor and ethnic minorities to gain access to quality health services). Therefore, this Project focused on district hospitals and complemented another health project in the NUP areas approved in June 2008 (Health Care Support to the Poor of the Northern Upland and Central Highlands - P110251, EC Grant No. TF-091328-VN) and which focused on primary care service delivery at the commune and village levels. The Project was consistent with the second pillar of the Country Partnership Strategy (CPS) for Vietnam (Report No. 38236-VN) issued in January 3, 2007, especially in the area of strengthening social inclusion, assuring economic growth with social equity by improving social services to the poor and marginalized groups.

1.2 Original Project Development Objectives (PDO) and Key Indicators

10. According the Financing Agreement (FA), the objective of the project was to increase the utilization of district health services, especially among the poor and ethnic minorities population of the Northern Upland Provinces through improving the quality of district-level hospitals and reducing financial constraints to access to health services . The PDO in the project appraisal document (PAD) had slightly different phrasing, but its main essence was consistent with the FA. The PDO in the PAD was as follows: Increase utilization of district health services especially among Decision 139 beneficiaries4 in the Northern Upland Provinces through (a) strengthening of district hospitals and (b) reducing financial constraints to accessing health services. For the purpose of this Implementation Completion and Results Report (ICR), this PDO is split into three parts, all especially aimed at the poor and ethnic minorities’ population of the NUP: (a) increase utilization of district hospitals services5; (b) improve the quality of district-level hospitals; and (c) reduce financial constraints to access health services.

11. The four PDO-level indicators (Key Performance Indicators [KPIs]) were defined in the results framework (RF) of the PAD. The KPIs are the following: (a) utilization rates of inpatient services in district hospitals among Decision 139 beneficiaries; (b) utilization rates of outpatient services in district hospitals among Decision 139 beneficiaries; (c) percent of households which experience catastrophic health care expenditures in the year prior to the survey; and (d) proportion of district hospitals that provide full set of health services according to the national norms (Decision 23/2005/QD-BYT) adjusted for the Northern Upland.6 As baseline data were not available for some indicators at appraisal stage (KPI #4 and Intermediate Outcome Indicators [IOIs] #1, #2, and #6), the targets were nevertheless set based on the team’s best guess estimates of the expected percentage of increase/decrease during the project life. Baselines known at appraisal were adjusted to the outcome of the Baseline Survey of 2009 (KPIs#1, #2, and #3 and IOIs #3, #4, #5, and #7). However, these changes were not formally recorded through restructuring.

4 The Decision 139 created the HCFP in 2002 to increase access to health care and reduce the financial burden of health expenditure faced by the poor and ethnic minorities.5 The first part of the PDO is the overarching objective of the Project. The 2nd and 3rd parts are the means to achieve this overall objective throughout supply and demand side interventions.6 According to annex 3 (Project Results Framework) of the PAD, page 32. This indicator was kept but was measured as the percent of the health services that district hospitals can provide in comparison with the national norms (Decision 23/2005/QD-BYT).

3

1.3 Revised PDO (as approved by original approving authority) and Key Indicators, and reasons/justification

12. The PDO was not revised during project implementation. However, the original baselines and targets were revised for KPIs and IOIs during project implementation (without a formal restructuring) to reflect the outcome of the baseline survey conducted in the first semester of 2009 and published in July 2009. Baseline values of few indicators were kept as in the original PAD and, at the end of the project, appeared to be modest if compared with the values achieved after implementation. See section 2.3 on monitoring and evaluation (M&E) for details.

1.4 Main Beneficiaries

13. The main project beneficiaries were meant to be the poor and ethnic minorities of the Seven Northern Upland Provinces: Bac Kan, Cao Bang, Son La, Ha Giang, Lao Cai, Dien Bien, and Lai Chau. These populations were to benefit from project investments by increased access to quality health services provided by the district-level hospitals and by receiving subsidies (meals and transportation) to improve their regular access to the district hospitals. Health professionals and managers of the district hospitals in the NUP provinces were to benefit from training, improved infrastructure, and additional resources from fees received from the HCFP to increase financial sustainability of these hospitals.

1.5 Original Components

14. The project consisted of three components aimed at increasing the utilization of district health services in the NUP provinces (supply-side interventions) and reducing the financial constraints to accessing health services for the poor and ethnic minorities (demand-side interventions).

Component 1: Strengthening District-level Health Services (cost estimated at USD42.9 million).

15. This component aimed to improve the quality of district hospitals through the following three subcomponents: (a) human resources development, to strengthen the health workforce capacity and increase staff retention, expertise, and technical capacity of district hospitals through the provision of long-term training for doctors and specialists at level 17; (b) improving the quality of district hospitals, through the provision of basic medical equipment and selective facility repair and refurbishment; and (c) improving hospital management, to support the investment in district hospitals by creating a management environment that will sustain project outcomes measured by quality of improvement, infrastructure maintenance, and human resources management. This component involves training district hospital management staff in basic principles of management of health care institutions, developing hospital maintenance plans as an integral part of the management improvement program, and creating a management excellence award program for district hospitals that meet good management practices.

7 Level-1 specialists are doctors with the following specializations: internal medicine, obstetrics, intensive care, traditional medicine, and imaging.

4

Component 2: Increasing Financial Access to Healthcare Services for Decision 139 Beneficiaries (cost estimated at USD10.0 million)

16. This component aimed to address the demand-side constraints of access to health services by piloting mechanisms to further reduce financial barriers for the Decision 139 beneficiaries in the project provinces. The component had three subcomponents. They are: (a) support for direct catastrophic and nonmedical expenditures of health care for Decision 139 beneficiaries, aimed to get information on the distribution and pattern of catastrophic expenditures and to support nonmedical expenditures such as primarily transportation and food for the beneficiaries, which had a higher proportion (62 percent) of total inpatient out-of-pocket costs in the NUP areas; (b) strengthening capacity for HCFP, through institutional capacity-building activities to increase the financial capacity of the district hospitals to enroll beneficiaries and recover the medical expenses incurred by them. It involves the identification of beneficiaries according to poverty and ethnicity criteria and issuing and distributing the beneficiaries’ cards among them; and (c) strengthening local access to health services through promoting health seeking behavior, to increase the knowledge and understanding of the rights, entitlements, and benefits covered by the HCFP among the beneficiary population through information, education and communication (IEC) campaigns.

Component 3: Monitoring, Evaluation, and Project Management (Cost estimated at USD13.1 million, of which USD6.0 million from Government counterpart)

17. This component supported the set up and management of the Central Project Management Unit (CPMU) and Provincial Project Management Units (PPMUs) through (a) consulting services to cover technical issues as well as procurement, financial management, and disbursement; (b) training of project management staff; (c) provision of necessary office equipment; (d) financing of incremental operating costs; and (d) M&E activities, including baseline data collection, indicator updates, midterm review (MTR), end-of-project completion report and audits. This component also supported an initiative to streamline the Ministry of Health (MoH) internal procurement review and approval process, which was considered a critical initiative in the broader public procurement reform efforts in the sector and the country.

1.6 Revised Components

18. The project restructuring of August 29, 2014, proposed an 18-month extension of the project closing date and marginally reallocated the project funds among the components. With 82 percent of the project funds disbursed, an end project evaluation was conducted from May to June 2014 in the NUP targeted provinces, revealing that all KPIs and most of the IOIs had already been achieved by the project before restructuring, saving 18 percent of the project funds. The restructuring did not propose any review of the project components.

19. The Government proposed to use the remaining funds to continue activities under the project components. No other significant changes were made to the project. The 18-month extension was proposed to allow adequate time to: (a) improve the sustainability of the project outcomes by continuing and adding training courses on specialized techniques and skills such as the use and maintenance of equipment; (b) ensure that the financing and management of the HCFP had been transferred to the provincial authorities and finance the entitled activities

5

properly; (c) improve the efficiency of the project throughout, providing technical assistance in health service delivery, especially related to outpatient services which are essential to improve maternal and child health and to the achievement of the MDGs in the Project areas until December 2015; (d) ensure proper use of the equipment; and (e) provide the technical support for the development of policies to strengthen the district-level health services and support the achievement of the MDGs.

2. Key Factors Affecting Implementation and Outcomes

2.1 Project Preparation, Design, and Quality at Entry

20. Soundness of the background analysis. The project team accessed extensive and relevant background information about the health conditions, government initiatives, and constraints in the national context and in the project intervention areas. The project benefited from lessons learned in designing and implementing other health projects in Vietnam, such as the Mekong Regional Health Support Project (MHSP) and the National Health Support Project. The project was prepared as a specific investment loan (SIL) inspired by the design of a similar approved project —the MHSP—which provided the project team important lessons to guide the NUP project preparation, which was the first of four similar regional programs8. The main lessons learned that were incorporated in the project design were: (a) the need to focus on the demand-side interventions and underpin quality health coverage to increase utilization of health facilities; (b) the relevance of addressing regional variations to adapt investments to local conditions; (c) the need to improve project implementation skills at the local levels through technical support by the central level and by exchange of experiences among the PPMUs; and (d) the need to build capacity (by training staff) on World Bank fiduciary procedures and guidelines for the local implementation units, to avoid implementation delays, particularly with regard to civil works and procurement of equipment.

21. Assessment of the project design. The PDO, the balance of the activities among the project components, and project implementation arrangements were formulated realistically and in line with project complexity. The project design addressed many aspects of the expected implementation challenges associated with one central and seven local project implementation units, difficulties in accessing mountainous areas, scattered distribution of the beneficiary population, and the social (ethnic minority) issues and environmental (associated mostly with the hospital waste management issues) safeguards.

8 The MHSP (P079663) closed in June 2012 and was the first of a generation of universal coverage projects in Vietnam that aimed at expanding coverage for the poorest population. The project indicators were achieved and even exceeded and the project brought huge benefits to the poor, because the enrollment of the poor became a national policy based on this experience. However, some shortcomings were observed with regard to efficiency, such as the potential supplier-induced demand in a fee-for-service environment, leading the World Bank to open a dialogue with the client to implement policy interventions with the Vietnam Social Security Administration to focus on provider payment reforms. Beside the Mekong and the NUP Project, there are two more similar projects—the Central North Health Support Project (CNHSP) and North East Red River Delta Region Health System Support Project (NORRED)—both still in implementation. All projects provided support to the health sector in provinces that were geographically difficult to access, economically depressed, and had interventions in both demand and supply sides. Despite that, each project had some different features. The MHSP is for provincial hospitals, while the NUP focuses on district-level hospitals and the CNHSP deals with district hospitals and community health centers. The NORRED Project was designed for both provincial and district hospitals with some distinctive interventions.

6

22. Other complementary World Bank health projects and international partner activities. The PDO did not overlap with other institutional partner support and projects, but complemented activities supported by another World Bank project (Health Care Support to the Poor of the Northern Upland and Central Highlands [HEMA] - P110251), which focused on primary care in three (Son la, Lai Chau, and Dien Bien) of the seven provinces addressed by the project. HEMA (approved in June 2008) was designed to improve access to primary health care by the poor at community health centers (not district hospitals). HEMA was trust-funded by the European Commission and managed by the World Bank. During project preparation, the Asian Development Bank (ADB) was also financing activities related to the promotion and prevention in three of the project provinces, complementing other health interventions (such as immunization, promotion, and preventive interventions) in these provinces. All these activities enhanced the perspective of an integral health care approach as promoted by the World Bank’s health strategy in Vietnam and supported the idea to focus the project design on district hospitals.

23. Adequacy of Government’s commitment. The MoH authorities were completely involved in the project preparation and in development of its design, which followed the MHSP model. They expressed their full commitment to guarantee adequate budget for communications, supervision, monitoring, evaluation, and travel during implementation. The provincial health authorities and district hospital managements were also committed to participate in the project development. Field visits were organized during preparation to establish links and networks with the local government health and hospital’s authorities.

24. Project preparation timeline. The project was prepared in 22 months (from concept review in May 2006 to approval in March 2008). This is considerably longer than the average for Health Nutrition and Population Global Practice projects of 18 months, but still under the World Bank’s benchmark for Investment Project Financing of 24 months. The FA was signed four months after approval (July 10, 2008) and declared effective on October 8, 2008. The project was to be implemented in six years in view of the institutional implementation conditions and challenges at the Northern Upland Provinces.

25. Assessment of risks. The overall risk for the project was rated Substantial and mitigation measures were appropriately described in the PAD. The major risk of not achieving the PDO was a possible uneven implementation progress across project components. It could prevent the positive effects of the coordination between the demand side incentives and the supply side investments on improving health access to the poor during the project implementation. Other risks were associated with weak procurement capacity at the provincial level and at district hospitals and the difficulty in retaining trained medical staff at the district hospitals. The CPMU issued an operational manual that was approved before project effectiveness. During implementation, as indicated in the project missions, project risks were well identified and managed. Some measures to mitigate the project risks during implementation are described in annex 6A of this ICR.

26. Quality Enhancement Review (QER) and Decision Review Meeting. The project underwent a QER in May 2007; the decision review meeting was held in September 2007. During both meetings, the element of the project that was mostly appreciated by the reviewers was its good rationale and direction and a solid results framework even though baselines and targets were to be confirmed at a later stage through the baseline survey. Some of the issues and

7

recommendations raised during these meetings not only revealed that it was crucial to confirm quality at entry, but were eventually found to be critical during implementation. The main recommendations to the team were to (a) provide more details on technical aspects of the project interventions (such as incentives for human resources retention, capacity building at district hospitals, governance of district hospitals, health promotion, implementation and financing of the project pilots)9; (b) focus the economic analysis on direct benefits for the target population; (c) improve the section on lessons learned; (d) improve the description of institutional arrangements for data collection related to project M&E; and (e) reassess project risk ratings, which appeared to be overestimated. The PAD addresses all the recommendations, except point ‘d’ on institutional arrangements for data collection for project M&E, which was not entirely addressed.

2.2 Implementation

27. Project implementation timeline. The project was to be implemented in six years (one year more than the usual five-year implementation period for SIL projects in the health sector), because of the difficult institutional implementation conditions in the Project areas. The original Project closing date of August 31, 2014, was extended by 18 months to February 29, 2016.10 Even when the PDOs were substantially achieved, project extension was justified by the Government as a way to use the project savings to ensure sustainability of project interventions and to contribute to the achievement of the health MDGs in the NUP areas.

28. The July 2012 MTR confirmed that the PDO remained relevant with a satisfactory performance. A MTR survey was conducted by a local research institute on behalf of the CPMU from November 2011 to May 2012. Substantial progress was registered in the achievement of the KPI and IOI targets. By the end of 2012, inpatient utilization of the Decision 139 beneficiaries increased more than two times and outpatient utilization increased 28 percent compared with the 2009 baseline. According to the survey results, the percentage of poor households suffering catastrophic health expenditures was reduced from 22.1 percent to 13.3 percent (almost a 9 percent reduction). The proposed targets in these areas expected about 10 percent to 15 percent improvement by project closing. Therefore, the project far exceeded these targets, which were established quite modestly from a low base. The MTR survey also revealed that the district hospitals in the Project areas were able to deliver 10 percent more services, approaching the expected number of services to be provided by district hospitals, according the MoH’s regulations. 9 These pilots of ‘incentive packages’ would consist of a combination of different types of incentives (monetary and non-monetary), which would be designed according to health labor market studies. It would include differentials payments according to hardship areas, subsidies for continuing education or education costs for the health workers’ children, housing allowances schemes, fast tracked promotion through pay grades for health workers in remote areas, increased recruitment of students from the project provinces to medical schools, and so on. Paragraph 25 of the PAD said “Evaluation of the effectiveness of the pilots will be an important part of the project’s overall effectiveness”. However, this activity was not implemented by the project because of its weak political feasibility in the context of health human resources policies in the country.10 The main reasons for project extension were (a) improving the sustainability of the project results by continuing and adding supplemental training courses on specialized techniques and skills, use and maintenance of equipment, and other training activities; (b) ensuring that the financing and management of the HCFP have been transferred to the provincial authorities and are running; (c) improving the project efficiency by strengthening the technical assistance in health service delivery with particular emphasis on delivery and outreach of essential outpatient services to maternal and child health care related to the achievement of the MDGs; and (d) distilling lessons learned and disseminating them.

8

29. At the MTR, delays in procurement and project disbursement led to classify the overall project implementation as Moderately Satisfactory. In 2012, only one-third of the credit funds were disbursed. The reasons attributed to this poor performance were: (a) the issuance of Government Resolution 11 of 2011 limiting some capital investments (vehicles/office equipment) even if externally funded; (b) inability to translate provincial proposals for non-training recruitment/retrenchment options into pilot activities; (c) lack of a plan for the utilization of the technical assistance subcomponent under Component 2 to analyze the barriers to access by the Decision 139’s beneficiaries; (d) no decisions taken at the time of the MTR on investments related to the disposal of solid waste treatment at the district hospitals11; (e) delay in the update of the NUP definition of beneficiary based on the revision of Decision 139 and inclusion of “HEMA” districts; and (f) slow development of a strategy for IEC activities. These issues were appropriately addressed after the MTR, resulting in a small reallocation of the project proceeds to Component 1. As a result, the project improved its performance and procurement was classified as satisfactory in the last project Implementation Status and Results Report (ISR).

30. All four KPIs’ final targets were achieved and surpassed even before the original project closing date of August 2014. However, the development objective (DO) was rated Moderately Satisfactory at the time of the original closing date because of internal evaluations of the World Bank team, which assessed slow progress in the RF, given a reduction of the achieved values of some indicators from 2012 to 201412. However, by August 2014, the project showed achievement toward the KPIs and a significant number of IOIs compared with the targets, including: (a) knowledge of the health workers in diagnosing and treating common illnesses; (b) knowledge of the benefits of HI by the beneficiaries; and (c) availability of facility and equipment maintenance plans.

31. The final ISR, of February 25, 2016, rated the DO Satisfactory and the implementation progress Moderately Satisfactory. All major planned activities were completed by the closing date. Increased utilization of the district-level health services in the NUP region by the beneficiaries was noted. The four KPIs used to measure the achievement of the PDO in the seven provinces were fully achieved, as follows: (a) average inpatient visits per capita in the district hospitals increased 3.9 times over the baseline and was 2.9 times greater than the end project target for 2014; (b) average outpatient visits per capita increased 3.7 times over the baseline and was 28 percent greater than the end of project target for 2014; (c) percentage of households experiencing catastrophic health expenditures was reduced from 14.7 percent in 2008 (baseline) to 2.0 percent (2014), overachieving the target of 13.2 percent for 2014; and (d) average percentage of health package procedures covered by the district hospitals according the national norms increased from 39 percent (2008) to 80.4 percent (2015), surpassing the target of 70 percent (2014).

11 In fact, the MoH/CPMU requested to buy the incinerators for solid waste. However, due to the World Bank’s new regulation not allowing using IDA credit for burning technology, the activities were delayed and then cancelled at the World Bank’s request.12 See the Project ISR Sequence 7, issued in May 2014. According to this ISR, “the outpatient utilization increased to 0.082 from 0.067 visits per capita (a 22 percent increase against the end-of-project target of 10 percent increase); this was a slight (5 percent) decrease from 2012 of 0.086. In August 31, 2014, the district hospitals supported by the project could provide 71 percent of the services required according to the national standards, compared with 52 percent at the baseline.”

9

32. Project contribution was crucial in supporting the PDO progress toward the MDGs on maternal and child mortality. During project implementation, all health workers and traditional birth attendants of the seven provinces were trained and received technical advice and monitoring from the MoH’s Department of Maternal and Child Care. All the provinces and district hospitals were provided with training materials and equipment to improve the performance of neonatal care units. The hospitals also received clean delivery kits to help improve hospitals’ performance and home-based birth delivery. By December 2015, the Project undertook an evaluation of maternal and neonatal mortality in the Northern Upland provinces, which indicated that the Project contributed to the improvement of maternal and child health. The IMR, neonatal mortality rate (NMR), and MMR decreased from 31.1 per 1,000, 11.2 per 1,000, and 178 per 100,000 in 2007–2008 to 29.4, 10.8, and 106 in 2013–2014, respectively. Data for 2015 was not processed at the time of this ICR.

33. CPMU commitment and capacity. The CPMU was established before project effectiveness and was always committed to improve the project performance and achieve its goals. The CPMU built strong and systematic capacity and was essential in establishing the operational framework of PPMUs, necessary to implement the project in each of the seven Northern Upland Provinces. The CPMU was in charge of launching and operating project management systems, including human resources, fiduciary functions, offices, and equipment throughout the implementation of the project. During project implementation, the CPMU and the PPMUs prepared the project’s annual plans, monitored annual implementation targets, and provided timely interim financial and activity reports and audits. Exception should be made to some reports on project result indicators toward the last year of project implementation because of some staff shortness from both CPMU and PPMUs.

2.3 Monitoring and Evaluation (M&E) Design, Implementation and Utilization

34. Design. Despite the extended time to prepare the project, the Government and the World Bank agreed that the baseline survey and the completion of the RF would be completed during the beginning of Project implementation as part of the Project’s Component 3, activity (v)13. Some baselines and targets for the KPIs and IOIs set in the PAD (annex 3) were to be confirmed or changed after the 2009 Baseline Survey. The poor quality of the health information system (HIS) in the country, provinces, and district hospitals at the time of project preparation established barriers to confirm upfront KPI and IOI baselines and therefore targets. For this reason, the Government and the World Bank team opted for a baseline survey to be launched after project effectiveness. The RF would be monitored and evaluated by comparing data from the baseline survey with the results of a midterm evaluation survey and a project-end survey. The baseline survey was part of the project design, as indicated in the PAD.

35. The four KPIs were linked to the three parts of the project PDO14 - KPI #1 (increase utilization rates of inpatient services in district hospitals among Decision 139 beneficiaries) and KPI #2 (increase utilization rates of outpatient services in district hospitals among Decision 139 beneficiaries)15 were linked with the PDO part 1, increase utilization of district health services; 13 See page 5 of the PAD.14 See paragraph 10 of this ICR.15 The baseline inpatient and outpatient rates of district hospitals, apparently low, have to be seen as part of a global inpatient and outpatient rates that include other health services, such as community health centers and community

10

KPI #4 (proportion of district hospitals that provide full set of health services according to the national norms) was linked with the PDO part 2, improving the quality of district-level hospitals, and KPI #3 was linked to PDO part 3, reduce financial constraints to access health services16.

36. The 13 IOIs were fully aligned with the project components. Component 1 (Strengthening District-level Health Services) was evaluated by six IOIs, addressing the following dimensions: patient satisfaction (IOI #1), adherence to treatment protocols (IOI #2), human resources training (IOI #3); maintenance plans and budget for facility and equipment maintenance (IOI #4); health facilities constructed, renovated, and equipped (IOI #5)17; recent discharged patients (IOI #6). Component 2 (Increasing Financial Access to Healthcare Services for Decision 139 Beneficiaries) was monitored by four IOIs addressing: people with access to a basic package of health, nutrition, and reproductive health services (IOI #7)18; percentage of Decision 139 beneficiaries who had received health insurance cards (HICs) (IOI #8); percentage of Decision 139 beneficiaries with cards, who can correctly identify at least three benefits covered under the HCFP program (IOI #9) and households identified with financial barriers for not seeking health care (IOI #10). Component 3 (Monitoring, Evaluation, and Project Management) was monitored by the following tasks: project management units at the central and provincial level established and maintained, bank accounts opened, staff and consultants recruited and systems established (IOI #11); availability of data for project M&E in years three and six (IOI #12); and project management units provided with adequate plans, annual implementation targets met, and timely financial and activity reporting (IOI #13).

37. Implementation. During the Baseline Survey in 2009, the Government and the World Bank team considered that, given operational reasons, KPI #4 was adjusted to measure “the average percentage of health services provided by the district hospitals.” Baseline values were available for all KPIs after project approval. The baselines for KPI #119 and #2 were calculated as of June 2009, while baseline values for KPI #3 (from VHLSS) and #4 (from hospital administrative records) were calculated as of June 2008. The 2009 Baseline Survey conducted during the first semester of 2009 provided baseline values for the NUP and HEMA Projects.

38. Baseline values for some IOI’s received special treatment during project implementation. IOIs #1, #2, and #5 to #10 baselines were calculated as of June 2009 while baselines for IOI # 3 and #4 were calculated as of June 2008. No baselines were established to the three IOIs related to Component 3 (IOIs #11, #12, and #13) because they were process indicators. Given their complexity, IOIs #2, #3, and #4 were split into several sub-indicators to

health stations (for outpatients) and provincial hospitals, national hospitals, and private hospitals for inpatient rates. According to the Vietnamese Household Living Standard Surveys (VHLSS) 2014, in the Northern Upland Provinces, the number of total inpatient visits per inhabitant was 0.116 (0.063 in district hospitals) and the number of outpatients visits was 0.462 per inhabitant (0.074 in district hospitals).16 Baselines and follow up data for this indicator was obtained from household surveys (VHLSS).17 IOIs #5 and #6 did not appear in the PAD and were added during project implementation.18 IOI #7 did not appear in the PAD and was added during project implementation. However, it cannot be measured because the baseline and target (expressed in percentage) were not converted by the number of beneficiaries as referred to in the title of the indicator. This indicator is therefore not a measure for the purpose of this ICR. However, by December 2015, the number of beneficiaries reached was 270,300.19 Baseline for KPIs #1 and #3 and for IOI #9 were initially established in annex 3 of the PAD as based in the VHLSS 2004. However, the baseline of these indicators was revised to June 30, 2009 (KPI #1 and IOI #9, using the baseline survey) and to June 2008 (KPI #3, using the VHLSS 2008), respectively.

11

be appropriately measured. IOI #2 (adherence to treatment protocols) were split into nine sub-indicators of three relevant health conditions affecting population in the NUP areas—diarrhea, pneumonia, and poisoning20. A similar process was used to measure IOI #3 (percent of eligible district health staff who have successfully completed training provided by the project), measured by three sub-indicators21, and IOI #4 (proportion of district hospitals with acceptable operations and maintenance plans and budget for facility and equipment maintenance) measured by two sub-indicators22. All baselines for the IOI #2 to IOI #4 sub-indicators were calculated as of June 2009.

39. Supplemental M&E indicators. To improve the monitoring of project achievements, the CPMU created and monitored additional indicators that were not part of the PAD and legal document (Supplemental Letter #2), but were reported in project ISRs. These indicators were recently discharged patients (IOI #6); number of health facilities constructed, renovated, and/or equipped (IOI #5), and; number of inpatient beneficiaries with access to a basic package of health, nutrition, or reproductive health services (IOI #7). The latter was crucial to support the achievement of the health MDGs in the NUP areas upon project closing. However, no baseline was established for these indicators.

40. MDG’s health indicators (MHIs) monitored after project extension and measured by the Government. The level II restructuring of August 2014 to extend the project by 18 months did not use that opportunity to formally revise the RF and establish the baselines and targets from the 2009 Baseline Survey. Considering the need to strengthen the capacity to improve health MDGs by the district hospitals in the NUP areas, some additional indicators (not followed by the World Bank) were followed by the CPMU and the PPMUs to compare positive variations between June 2009 and December 2015. These indicators were the following: percentage of district hospitals providing caesarean section surgeries (MHI #1); percentage of district hospitals providing blood transfusion (MHI #2); percentage of district hospitals having continuous positive airway pressure systems (MHI #3); percentage of district hospitals having light for jaundice phototherapy treatment (MHI #4); percentage of district hospitals having oxygen breathing systems (MHI #5); and percentage of district hospitals having newborn resuscitator (MHI #6). Several other indicators were part of this M&E block, but without baselines.

41. Final targets for the project RF’s indicators. All KPIs and IOIs (except IOIs #11 to #13, which required yearly monitoring) considered August 31, 2014 as the final target date. By agreement between the Government and the World Bank, the CPMU did not extend the final target indictors as part of the project restructuring of August 2014, but as mentioned before, the project missed the opportunity to formally revise the 2009 baselines and update targets to 2015. Even keeping the original date for the final targets, some of the project indicators were measured

20The description of the 9 sub-indicators could be found in Annex 5. According the provincial health authorities, these three conditions represented the majority of the demand for health care in district hospitals.21 The sub-indicators are (a) percentage of doctors and assistant doctors at district hospitals trained by the project; (b) percentage of health staffs who completed short-term training courses compared to the plan and; (c) percentage of health staffs who completed long-term training courses compared to the plan.22 The sub-indicators are (a) proportion of district hospitals with acceptable operations and maintenance plans and budget for facility maintenance and (b) proportion of district hospitals with acceptable operations and maintenance plans and budget for equipment maintenance.

12

by December 31, 2015 by the independent evaluation of the project, conducted by the CPMU during the project extension period. These indicators are KPIs #1, #2, and #4; IOI #3, #4, #5, and #8.

42. Intermediate targets and M&E processes. Given the hard conditions to obtain health information associated with the NUP areas and district hospitals, the project established its own administrative records at the PPMUs and associated its evaluations to midterm and final surveys with the same methodology used in the 2009 Baseline Survey. The current Government’s surveys are improving, but they still could not be used as the main source of information for the project’s M&E processes because of lack of reliability and availability of the data, especially for project areas where the sample surveys are not representative. During the project life, three surveys associated to the M&E process were done: for the baseline (first semester of 2009); for the MTR (November 2011–May 2012), and at the project original closing date (before the extension, May–June 2014). Intermediate targets are measured only once (May 2012) and final targets in June 2014. District hospitals’ administrative records and eventual regional surveys provided additional information that was recorded in the project’s ISRs in the interim.

43. Utilization of M&E data. Given that district hospitals’ HIS had weak capacity in the first years of the project implementation, the utilization of M&E data by the CPMU and PPMUs was limited during part of the project life. Some difficulties remained in the use of definitions and in the calculation of some project indicators by the PPMUs and therefore building the aggregated data for some indicators was delayed. As a result, many data and information were not recorded timely and accurately, leading to delays in the submission of M&E reports. The MTR evaluation was useful to refocus the project, in the needed areas, with more support, such as providing technical assistance of qualified consultants to support the provinces to improve data reporting and accurate calculation of the project indicators. At the project end, the RF was reported regularly by the consultants, creating the database for the final project evaluation.

2.4 Safeguard and Fiduciary Compliance

44. There were two types of safeguard policies triggered by the project: social and environmental. The World Bank’s safeguard policy on Indigenous People (OP/BP 4.10) was triggered given that the majority of project beneficiaries were ethnic minority groups. Since its design, the project was expected to have positive impacts for these beneficiaries, which represented 82 percent of the NUP population. As a result, the project increased demand and utilization of health services by the poor and ethnic groups, with positive impacts on reducing the financial burden on health care for beneficiary households through increasing coverage and strengthening the implementation of the HCFP. The project execution was considered satisfactory on implementing the indigenous people policy, given that ethnic minorities were part of the project primary target population.

45. The project hospitals achieved remarkable improvements in the waste management process compared with the baseline (2007), when most of them did not comply with health care waste management regulations. The project design included the development of a health care waste management (HCWM) plan to provide district hospitals with proper disposal of the medical wastes. It included the preparation of specific HCWM plans for each district hospital under the project; training and use of IEC materials on HCWM for district hospital staff;

13

procurement of HCWM equipment and supplies; and; the introduction of wastewater treatment facilities at district hospitals. The implementation of the HCWM plans experienced some delays during the project life and was classified in the last ISR as moderately satisfactory, because of difficulties encountered in the procurement of equipment and corresponding staff training and changes in the World Bank’s policy on the procurement of some HCWM that were included in the original procurement plans, such as incinerators.

46. By the end of the project, most of the district hospitals had a well-prepared HCWM plan and monitoring program. Out of the 65 project hospitals, 59 have trained their HCWM officers. All project hospitals fully complied with the regulations on waste segregation and collection. Regarding waste storage, 45 percent of project hospitals fully complied and 51 percent of project hospitals partly complied with the related regulation. By project closing, around 60 percent of project hospitals were treating their hazardous waste and 38 percent of project hospitals were treating wastewater in a proper manner. Because none of the hospitals had relevant experience in waste management processes at project onset, the project was quite successful in that regard and in the CPMU’s ability to make sure that all district hospitals would continue implementing HCWM plans after project closing.

47. The project provided good solutions and implementation of waste management processes at district hospital level. Basic training on regulations related to health care waste management and nosocomial infection control were provided for the district hospital managers and the staff. The CPMU and PPMUs received guidance for planning and implementing measures for health care waste management. Consumables and equipment for health care waste management were procured by the PPMUs and distributed to the project district hospitals. The availability and proper use of waste containers, transportation, and cooling devices resulted in significant improvements in health care waste separation, collection, storage, and final disposal in district hospitals.

48. Financial management. The financial management and counterpart’s fund commitment performance moved from satisfactory, in the first years of project implementation to moderately satisfactory in the last three years of implementation because of some delays in the audit processes. The CPMU and PPMUs were appropriately staffed during the great part of project implementation. Financial reports were delivered with satisfactory quality and audit reports were provided, but with some delays. Despite the fact that project implementation went faster than project disbursement, the 18-month project extension lead to the use of 97 percent of the loan funds (USD58.4 million). About USD1 million equivalent of the project funds were unused and returned to the World Bank.

49. Procurement. During the major part of the project life, procurement performance was rated as moderately satisfactory because of some delays in the delivery of procurement packages, but it was improved during project implementation. The Government and the World Bank were proactive in solving many of the problems and in reviewing and adjusting procurement plans. Most of the packages were implemented on schedule or slightly behind schedule. Post review of procurement found no major deviation or non-compliance. At closing, procurement plans were respected with no complaints and progress was rated as satisfactory.

14

2.5 Post-completion Operation/Next Phase

50. Project sustainability is likely, by the commitment of provincial government’s budgets, to finance transport and meals subsidies for the poor/ethnic minorities to access health care services at district hospitals, maintain the equipment, and retain skilled human resources in district hospitals. The MoH issued national norms to create Provincial Health Funds, establishing incentives to keep health care providers in the mountainous provinces. MoH Decision 38, issued in 2012, allows the use of district hospital revenues to maintain and buy new equipment. In addition, Decision 14, also issued in 2012 by the Prime Minister, allows the Provinces to use their fiscal revenues to support transportation and meals to the poor and near poor for their visit to district hospitals. Both government’s decisions were taken as the positive outcomes of the NUP project23. Four of the seven provinces have established provincial HCFP to finance meals and transportation for the poor and ethnic minorities’ medical visits. The provinces which have not yet established HCFP - Dien Bien, Bac Kan, and Cao Bang - are in the process of doing so with support from the MoH. However, during the project extension phase, between 2014 and 2015, the number of district hospitals with budgets to maintain infrastructure and equipment had a slight reduction by 20 percent and 10 percent, respectively.

3. Assessment of Outcomes

3.1 Relevance of Objectives, Design, and Implementation Rating: Substantial

51. Relevance of Objectives: High. The project was fully aligned with Vietnam’s CPS 2007–2011 (Report 38236-VN) and with Vietnam’s CPS 2012–2016 (Report 85986-VN) in terms of priorities for the health sector and also with the country’s future health strategies. The project was essential to support the country’s health sector along with its transition from low- (IDA) to middle-income country (IBRD), improving quality coverage for essential district hospital services to the poor and ethnic minorities in the Northern Upland Provinces, especially to mothers and children. The project was also a priority of the MoH and provincial government by its relevance to reduce health coverage inequities and strengthen district hospitals’ performance, contributing to attending the health needs of the mountainous areas under the Vietnamese Health Reform priorities. The project was consistent with the country’s Health Sector Development Plan (2011–2015) and the Health Sector Strategy, with a vision to 2020. The PDO remains relevant now and in the coming years. In 2016, the Vietnamese Government and the World Bank launched the Vietnam 2035 Agenda, which is completely in line with the project objectives, especially the objective of increasing quality health services access to the poor24.

23 Center for Environmental and Health Studies. 2014. Report on the Final Evaluation of the Northern Uplands Health Support Project, Hanoi, July 2014. 24 The World Bank and the Ministry of Planning and Investment of Vietnam. 2016. Vietnam 2035: Toward Prosperity, Creativity, Equity and Democracy, Hanoi, February 2016. According this document ‘the major policy challenge facing Vietnam’s health system over the next 20 years will be to achieve universal health coverage that is, to ensure that everyone has access to high quality services without suffering financial hardship. The objective of universal health coverage is closely linked to Vietnam’s overall equity agenda, both in ensuring access to services to promote social inclusion and in reducing poverty due to out-of-pocket payments for health care’.

15

52. Relevance of Design: Substantial. The project design drew from projects in other poor regions of the country (such as MHSP, CNHSP, and NORRED) aimed at ensuring equity in the coverage and access to health services for the poor and ethnic minorities. The project design was partially based from the MHSP design. It also was influential in developing the design of the CNHSP and NORRED projects—both still under implementation. The project design appropriately addressed both demand and supply sides to improve coverage and access to health services provided by district hospitals in the mountainous areas.

3.2 Achievement of Project Development ObjectivesRating: High

53. The overall efficacy rating is the result of the assessment of the achievement of the indicators associated with the three parts of the PDO, which are rated High (part 1), High (part 2), and Substantial (part 3) (See annex 5, Analysis of the PDO Achievement).

54. Achievement of the project PDO part 1: High. The PDO part 1, related to the supply-side interventions, was increased utilization of district health services (in the project case, district hospitals). The utilization level of district hospitals could be measured majorly by two indicators: inpatient utilization rates (KPI #1) and outpatient utilization rates (KPI #2). The targets for these two KPIs were significantly surpassed as the project created effective access to district hospitals services for the poor and ethnic minorities. Targets were exceeded by a set of health services that were broader than those originally defined as the project target.

55. Achievement of the project PDO part 2: High. The PDO part 2, related also to the supply side, was improving the quality of district-level hospitals. This part of the PDO was measured by the proportion of district hospitals that provide the full set of health services according the national norms (KPI #4), which was surpassed, and seven IOIs (#1 to #7, six surpassed and one achieved). These IOIs addressed quality aspects of health care at district hospitals, such as client satisfaction, adherence of treatment protocols for select health conditions, and interventions for inpatients; training of district hospitals staff on maintenance schedule, and budget allocation for hospital infrastructure and equipment; construction/renovation and equipment of health facilities; and patients discharged.

56. Achievement of the project PDO part 3: Substantial. The PDO part 3, related to the demand side, was reducing financial constraints to access health services. This part of the PDO is measured by the percentage of households experiencing catastrophic health care expenditures in the year before the survey (KPI #3), which was surpassed, and three IOIs: #8 and #10 (surpassed) and #9 (not achieved)25. These IOIs addressed aspects such as the possession of HICs by the beneficiaries, their knowledge about how to use/access the HI benefits and the percentage of households identifying financial barriers as the main cause for not seeking health care. Achievement of this part of the PDO is considered substantial according the methodology used to rate the PDO’s efficacy. IOI #9 (percentage of Decision 139’s beneficiaries with cards who can correctly identify at least three benefits covered under the HCFP program) was not achieved

25 The CPMU manifested to the World Bank team, after the MTR that the achievement of the IOI #9 should be ambitious, but did not suggest to change the indicator or its targets during the project implementation.

16

according to the criteria used to evaluate PDOs26. One possible reason, among others, for this failure may be the need for more appropriate IEC activities to increase the understanding of the HIC benefits to the ethnic minorities.

Table 1. Achievement of Project Indicators

Target Achievement Status KPI IOI IOISub-

indicators

Total

Target surpassed 4 9 12 25Target achieved or substantially achieved (>=85 percent met) 0 2 2 4

Target partially achieved (65%–84% met) 0 0 0 0Target not achieved (<65% met) 0 1 0 1Not considered 0 1 0 1Total 4 13 14 31

% surpassed and achieved/substantially achieved 100 80 100 94Source: ICR Datasheet, pages iii–ix. This table does not include progresses in the MHI because they are not part of the PAD or the legal agreement and neither had established targets.

57. An overall analysis of the 31 project indicators shows that 94 percent of the project indicators were surpassed, achieved, or substantially achieved by the end of project implementation. As can be seen in table 1, the project had 4 KPI’s, 13 IOI’s (three of them split into sub-indicators) and 14 sub-indicators used to measure three IOI’s (#2, #3, and #4). Table 1 shows the level of achievements of project indicators. Part of this good performance could be explained by modest targets attributed to the KPIs and IOIs at the design stage of the project RF. The lack of information to foresee how the project investments should affect the district hospital performance made it difficult for the Government and the World Bank to increase the project risk by using challenging targets for the project indicators. During the project restructuring of 201427, both— Government and the World Bank—should have formally agreed on setting more ambitious targets for some of the KPIs and IOIs for the following 18-month extension, based on the accomplishments verified by the 2014 end-of-project survey. However, this was not done because many original project activities were not continued during the extended period and were replaced by new training activities and the provision of equipment/consumables for maternal and child health care for achieving the MDGs.

58. The project showed excellent performance on the achievement of the RF targets. There are two reasons for this: first, the uncertainty about the results of the project investments during the project design and beginning of implementation led the Government and the project team to be cautious about the KPIs and IOIs targets. Second, as it happens in other international contexts (see the efficiency analysis section of this ICR), projects that drive the investments to

26 See annex 5, paragraphs 2 and 3 and annex 5A.27 The project had problems that should lead to require an earlier project restructuring, such as adjustments in the indicators baselines and others. However, the decision process to do a project restructuring in Vietnam is complex. It requires six months of anticipation and needs to be approved by the country’s prime minister. The MoH did not considered adjustment in the project indicators a relevant reason to require a project restructuring. Only subjects considered more relevant, such as a project extension, should be reasons to lead the sectoral ministers to submit to the prime minister a project restructuring in Vietnam.

17

simultaneously solve supply- and demand-side constraints are proven to be more efficient in achieving good performance in outputs and outcomes.

59. Assessment of achievement of the project activities and outputs. Another way to verify the PDO’s achievements is evaluating the completion of the project outputs by activity, subcomponent, and component (see annex 2). According to the Project Output Map (annex 2A), the project was structured in three components, six subcomponents, 19 activities, and 29 outputs. Two outputs were not in the PAD, but were added during project implementation (long-term training for pharmacists and short-term training for health staff on MDG-related activities). The other 27 outputs could be split into two groups: 10 with quantitative results and 17 with qualitative results. For the first group, the same rating categories could be used to classify the KPIs and IOIs in the PDO analysis (surpassed, achieved, substantially achieved, partially achieved, and not achieved)28. Assessment for the second group is based on the ISR records. This exercise allowed the ICR team to confirm that 9 out of 10 outputs in the first group were surpassed and one was substantially achieved. These outputs are related to long-term and short-term training activities, techniques and skills transfers, medical equipment provision and facilities construction/improvement, management training, and hospital maintenance plans. In the second group of 17 outputs, 9 were achieved, 7 were partially achieved, and 1 not achieved and cancelled (the issuance of a management excellence award program), because the MoH created a systematic process to award health services and personnel with excellent performance which was not related to the NUP. This system has not been followed up by the CPMU and the project. The achievements (full or partial) are related to support capacity improvement of the HCFP, implementation of studies on the benefits’ incidence of the program, provision of office equipment to PPMUs, and others.

60. Substantial progress has been verified in KPIs and IOIs during the 18-month project extension. Table 2 shows the achievements during the project extension period: three KPIs, two IOIs, and a few MHI indicators were achieved. At project closing, three of the four KPIs were measured and were found to have shown improvements. Short- and long-term training for health staff at the district hospitals level have also improved with increases from 20 percent to 115 percent. Some problems were identified with regard to the health budgets and the maintenance and operations of district hospitals’ infrastructure and equipment, which could have indicated commitment toward sustainability of project interventions. Substantial progress could be registered in the health MDG indicators, especially in the reduction of maternal mortality, given that it was one of the main reasons used by the Government to request the 18-month extension of the closing date.

Table 2. Indicators Achievements during the Project Extension Period

Indicator Baseline (2009)

End Project Survey (2014)

Extension Phase (2015)

Percent Increase

During the Project

extension period

(2014–2015)KPIs

28 See description of the indicators rating in paragraph 2 of annex 5.

18

Indicator Baseline (2009)

End Project Survey (2014)


Percent Increase

During the Project

extension period

(2014–2015)#1 - Increase utilization rates of inpatient services in district hospitals among Decision 139 beneficiaries (per 100 inhabitant)

0.025 0.085 0.096 13

#2 - Increase utilization rates of outpatient services in district hospitals among Decision 139 beneficiaries (per 100 inhabitant)

0.067 0.206 0.247 20

#4 - Proportion of district hospitals that provide full set of health services according to the national norms (%)

39.1 71,4 80,4 13

IOIs#2.1 - Percentage of health staffs who completed short-term training courses compared to the plan

0 357.3 427.3 20

#2.2 - Percentage of health staffs who completed long-term training courses (specialty level-1 doctors) compared to the plan

0 88.0 189.2 115

#3.3 - Total number of health staffs trained in short-term and long-term courses

0 8,929 11,868 33

#4.1 - Percentage of district hospitals with acceptable operations and maintenance plans and budget for facility maintenance

49.2 99.3 85.9 −13

#4.2 - Percentage of district hospitals with acceptable operations and maintenance plans and budget for equipment maintenance

77.1 99.2 93.8 −5

MHIs#1 - Percentage of district hospitals providing caesarean section surgeries 74.6 81.2 92.1 13

#2 - Percentage of district hospitals providing blood transfusion

56.7 67.2 82.8 23

#3 - Percentage of district hospitals having CPAP devices 25.0

75.0 79.7 6

#4 - Percentage of district hospitals having light for jaundice phototherapy treatment 21.4 82.1 87.5 7#5 - Percentage of district hospitals having oxygen breathing system 42.9 75.0 78.1 4#6 - Percentage of district hospitals having newborn resuscitator

46.4 92.9 92.2 −1

Note: CPAP = Continuous Positive Airway Pressure.

3.3 EfficiencyRating: Substantial

61. A quantitative cost-benefit analysis was not included in the PAD because there was no empirical basis for estimating the project’s impact on health outcomes at that time. However, project investments improved the coverage and quality of health services, especially among the poor and ethnic minorities population of the NUP, increasing the efficiency in both the supply and demand side of health care. Annex 3 presents the project’s economic analysis based on (a) rationale of the government investment in the region; (b) the

19

http://www.webmd.com/sleep-disorders/sleep-apnea/continuous-positive-airway-pressure-cpap-for-obstructive-sleep-apnea

efficiency on achieving quality access for district hospitals by the NUP population (project outputs); (c) the project contribution to improve health benefits (project outcomes) and; (d) the impact of the project interventions in the equity on health access and health spending in benefit of the poor and ethnic minorities. On the supply side, the project effectiveness was associated with improvements in the outputs of district-level health services by providing medical equipment and better trained personnel. On the demand side, effectiveness was associated with increases in the utilization of cost-effective health care by poor and ethnic minorities, whose previous levels of health care utilization were considerably lower than those of other social groups in the country.

62. The Government’s choice to invest in this region was based on the unfavorable socioeconomic conditions and the difficult access and generally poor quality of health services, especially at district health hospitals. The Northern Upland is the region with the highest concentration of ethnic minorities, which is also part of the Decision 139’s beneficiaries targeted by the project. The central Government sought to improve the health status of the population and reduce the health gap between the NUP provinces and the rest of the Vietnamese population. The project achieved this by improving the efficiency on delivering health outputs, reducing maternal mortality in higher proportion than the country’s average, and reducing the equity gap in out-of-pocket health spending between the poorest and the richest income quintiles.

63. The main conclusions of the economic analysis are the following: (a) the project was efficient in delivering its outputs, reducing costs for training, equipment installation, and civil works. This led to the savings being used, during the project extension period, for new activities related to the MDGs in the project area and (b) the project interventions and subsidies to the poor and ethnic minorities were efficient by contributing to reducing maternal, neonatal, and infant mortality and by improving the equity on assessing health care and reducing health spending to poor and ethnic minorities.

64. The project has significantly contributed to the improvement of equitable access of health care in the NUP areas. Project benefits resulted in health improvements of the poor and ethnic minorities as they represent a high share of the Northern Upland populations in addition to the project demand-side subsidies targeted to the poor and ethnic minorities.

Allocative efficiency

65. Improving access to mother and child health interventions at the community-level hospitals is recognized by literature as a highly cost-effective investment. Most of the health interventions provided by district hospitals are focused on reducing maternal, newborn, and child morbidity and mortality. A recent Diseases Control Priorities Program Third Edition publication29 shows high economic return rates for interventions aimed at increasing coverage of services where good evidence exists for demand-side interventions to motivate service uptake. Following this evidence, the project interventions contributed to the reduction of infant and

29 Black, Robert E., Ramanan Laxminarayan, Marleen Temmerman, and Neff Walker, Editors. 2016., Reproductive, Maternal, Newborn, and Child Health, Disease Control Priorities, Third Edition (Volume 2), Ed. Washington (DC): The International Bank for Reconstruction and Development/World Bank; Apr 5, 2016. ISBN-13: 978-1-4648-0348-2ISBN-13: 978-1-4648-0368-0.

20

maternal mortality rates in the NUP regions, which were reduced from 31.1 to 29.4 per 1,000 births alive and from 178 to 106 per 100,000 births alive between 2008 and 2014, respectively.30

66. Improving health infrastructure and skilled professional attendance at district hospitals were identified as the main factors to remove barriers to health service availability for women and children in the poor mountainous areas of the Northern Upland provinces. Therefore, closing the gap in health care access required targeting resources on the identified barriers and on ethnic minority’s populations. The return on infrastructure investments has clear long-term benefits, which facilitate improvements in the provision of quality services. In addition to infrastructure investments, the project funded some important and highly cost-effective demand-side interventions under Component 2.

Efficiency of project preparation and implementation

67. No major delays were registered during project preparation. The project was prepared in 22 months, which is slightly longer than the average for health projects in the World Bank, by 4 months, and longer than the average time for IPF preparation of 18 months for all sectors. However, considering that no previous experience and information were available to inform the project design at the concept stage, several documents were elaborated or assessed during project preparation, which required substantial time from the project team and country staff31.

68. The PDO was achieved within the originally planned period. The project’s implementing efficiency is associated with the fact that, by July 2014, the project had substantially accomplished the KPIs and IOIs using only 82 percent of the credit amount. Given the need to improve MDGs’ outcomes and consolidate the project beneficiary gains, the Government proposed an 18-month extension to use the remaining funds. By its closing in February 2016, the 94% of the project indicators (KPIs and IOIs) were achieved and surpassed and 97 percent of the credit proceeds had been disbursed.

69. The efficiency associated with the project closing date extension period could be considered moderate. The investments in training and equipment to improve mother and child care (see table 2) were appropriate and contributed to further achieve project outcomes. However, the provincial budget’s consolidation to guarantee the maintenance of health

30 The impact on maternal mortality reduction is directly associated with the quality of birth attendance provided by better access and quality birth delivered at the district hospitals. The impact on infant mortality appears to be more modest, because it should be associated to other non-hospital and nutritional interventions at the community levels.31 The main reports produced to feed the project preparation are: Health status in the seven provinces of Northern Upland; Health care service use and accessibility status in the seven provinces of Northern Upland: Son La, Dien Bien, Lai Chau, Cao Bang, Bac Can; Health Human Resource Analysis in the seven provinces of Northern Upland; Health care for the Poor: Identification of the needs and proposal of investment for capacity building and management capacity development; Assessment of the HCFP in the seven provinces of Northern Upland; Health care for the poor: management according to the Decision 139 in seven provinces of Northern Upland (most difficult provinces); Health System Assessment for seven provinces of Northern Upland; Inventory of medical equipment in hospitals of seven provinces of Northern Upland; Output indicators after analysis; Socioeconomic, Demographic, Cultural Geographic and Health Status indicators: Morbidity, Mortality, Crude Death Rate/IMR, Under 5-child nutrition status; List of medical staff to be trained and; Cost estimate and Cost Table for training component.

21

infrastructure was slightly reduced from 99.3 percent in 2014 to 79.7 percent in 2015, indicating that efforts may be needed to ensure proper maintenance of project investments32.

Fiscal impact and sustainability

70. The fiscal impact of the project was considerable at the provincial and district hospitals’ level. The project had significant fiscal impact in the additional recurrent expenditures needed to maintain project investments in infrastructure and equipment of district hospitals, enabling these hospitals to receive funds from the HCFP. For example, from 2008 to 2013, the average revenue of the 28 district hospitals supported by the project increased by 336 percent.33 On the other hand, the MoH supported the creation of the Provincial Health Funds, establishing the financial basis to support the recurrent costs of district hospitals in the future, guaranteeing the sustainability of project investments beyond the project life with a significant fiscal impact. Vietnam’s rapid economic growth, expected over the coming years, will ensure that provincial health budgets and district hospitals sustainability are likely to continue to grow at a high rate.

3.4 Justification of Overall Outcome Rating

71. The overall outcome rating is Satisfactory. This rating takes into account the project remained relevant throughout implementation and beyond and the activities financed by the project being efficient in providing supply- and demand-side interventions to improve quality access to health care at the district hospitals. In addition, the overall achievement against the PDOs is considered substantial.

Table 3. Summary of Outcome Ratings

Project Outcome RatingsRelevance SubstantialEfficacy HighEfficiency SubstantialOverall Outcome Rating Satisfactory

3.5 Overarching Themes, Other Outcomes and Impact

(a) Poverty Impacts, Gender Aspects, and Social Development

72. The project specifically targeted the poorest mountainous regions, home of ethnic minorities and isolated communities, and focused on the most vulnerable—women and children, achieving excellent outcomes in the reduction of maternal mortality and improving the coverage of HI and utilization of district hospitals by poor and ethnic minorities. The equity dimension was important in conceptualizing and implementing the project, which contributed to

32 The CPMU considered that district hospitals did have plans and minimum budgets for infrastructure and medical equipment maintenance and repairing at the project end. These types of plans and budgets are submitted yearly by the provincial Departments of Health to the MoH. However, in 2015 (the last year of the project implementation), the buildings, and equipment supported by project were still in good condition. Because of that, part of these budgets was not spent because it was not necessary. 33 Center for Environment and Health Studies. 2016. Report on the Final Evaluation of the Northern Uplands Health Support Project in the Extension Phase, Hanoi, February 2016.

22

social inclusion and promoted a solid basis to increase access to better quality health services by providing economic subsidies, such as transportation costs and meals to encourage medical visits by poor and ethnic minorities. More information on the equity impact of the project and the district hospitals access improvement for the poor and ethnic minorities is addressed in Annex 3.

73. Strengthening local levels’ facilities and staff based on social consultations. The project showed progress in building capacity at the Provincial Health Services Management in the NUP, benefiting government representatives and different categories of health professionals. The project investments were defined according to the findings of extensive consultations over a period of eight weeks in 2007 by the Government team, involving site visits to 10 district hospitals, focus group discussions with hospital staff at provincial and district levels, 20 in-depth interviews with health workers, and 40 direct interviews with patients at the district hospitals. According to the end line project evaluation results, realized by the end of 2015, more than 90 percent of the interviewed district hospital staff said that they were entirely satisfied with the training received and its appropriateness to job demands.

74. Environmental contribution of the project. The project improved the awareness of environmental issues at district hospitals by training personnel on appropriated treatment of medical and hazardous hospital waste, and contributed to improving the environmental quality and reducing the risk of hospital infection for the community.

3.6 Summary of Findings of Beneficiary Survey and/or Stakeholder Workshops

75. At the project original closing date and at the end of the extension phase, the CPMU contracted a consulting firm to conduct beneficiary surveys and collect administrative data comparable with the 2009 Baseline Survey and the 2012 MTR survey34. The main findings of the beneficiary surveys highlighted that the support and interventions of the project had a strong influence on changing health care services seeking behavior of the poor/ethnic minorities. The beneficiaries perceived a continuous improvement of service delivery capacity of district hospitals, which attracted more people to use the district hospitals’ health facilities. In addition, the project financing support policy to the poor increased the opportunity to use inpatient services at the district hospitals by the poor/ethnic minority patients. Moreover, the coverage of HIC increased and the people’s perception about the district hospitals improved, contributing to create positive changes in the poor/ethnic minorities’ behavior of seeking health care services.

4. Assessment of Risk to Development Outcome Rating: Moderate

76. The project strengthened health policies and institutions at the district level, improved health techniques and technology, provided new managerial tools, and created financial sustainability of district hospitals in the NUP areas. District hospitals that participated in the project strengthened quality of the services, developed qualified health

34 The assessment tools of these surveys included questionnaires for health staffs, households, and in/outpatients at district hospitals, in-depth interviews with leaders/staffs at CPMU/PPMUs, leaders at provincial health departments and social insurance units, leaders/staffs at provincial/district general hospitals, heads of commune health centers, trainers and trainees of the project, and focus group discussions with patients who completed their treatment at district general hospitals in the last three months.

23

workforce and will continue to satisfy the local people’s needs based on sustainable mechanisms for service provision at the district and commune levels. Supported health care policies for the poor and ethnic minority groups are based on better management capacity of the resources received from HCFP. The project contributed significantly to the implementation of policies on strengthening human health resources, especially for health care at the district level. The district hospitals developed plans to maintain and use the health staffs recruited from the local human resources after training and plan to maintain a regular training system to strengthen the capacity of health staff. On the financial side, all project provinces had specific commitments to provide funds for the maintenance of equipment and upgrading and repairing of facilities after the project phases out. Besides the commitments and specific plans, the localities still face a limited budget, creating, despite the government provisions and regulations to assure the future sustainability of the project investments, some uncertainty regarding the adequacy of the resources to guarantee the continuation of project outcomes after closing.

5. Assessment of Bank and Borrower Performance

5.1 Bank Performance

(a) Bank Performance in Ensuring Quality at Entry Rating: Moderately Satisfactory

77. The project preparation time (22 months) was a bit longer than the average for Health Nutrition and Population Global Practice projects, but still under the World Bank’s benchmark for IPFs. The signing of the financial agreement happened 4 months after Board approval (July 10, 2008) and the project effectiveness took 3 additional months (October 8, 2008), totaling 29 months between the issuance of the project concept note and the effectiveness. The project was proposed to be implemented in six years given the institutional implementation conditions at the NUP, but despite achievement of the PDO within its planned time frame, the total implementation time was about eight years after project extension. The World Bank team supported the preparation and transferred to the government team the appropriate technical skills to prepare the project and used the lessons learned for previous World Bank projects in the country. The project design was totally consistent with the CPS Government Strategy, described in paragraph 9 of this ICR. Given the experience in previous projects, the Government and the World Bank, during preparation, concentrated their efforts on preparing the institutional arrangements to implement the PDOs and mitigate the main challenges that could prevent the achievements. The project Operational Manual was prepared and approved by the World Bank before effectiveness, which contributed to guide the project implementation at the CPMU and the PPMUs.

78. Despite the good and focused design, the project could have improved efficiency in its implementation if the baseline survey had been carried out during the project preparation period. The project RF was well designed. However, the baseline survey was designed during implementation as part of the project Component 3, as can be read in the PAD35. During the project’s QER, no recommendation was provided to change this implementation

35 in the paragraph 97(page 50) of the PAD is written the following: “This component will support the set up and management of the Central Project Management Unit (CPMU) and PPMUs through: ….(v) M&E activities, including the baseline data collection, indicator updates, MTRs, end of project completion report, and audits”.

24

arrangement, but the project could have sped up its implementation if the team pushed to set up the baseline survey as part of the project preparation instead of completing it in the first semester of 2009 when the project had been under way for almost one year of implementation. If the baseline survey had been carried out during project preparation, the World Bank team would have had more room to assist the Government in establishing a sound and rigorous project M&E system.

(b) Quality of Supervision Rating: Satisfactory

79. The task team conducted 11 implementation support visits during the project life. Financial management and safeguards missions were performed adequately. In addition to formal missions, the project benefited from having during most of the implementation a field-based task team leader (TTL) who was in constant communication with the CPMU36. The TTL and the team, including the fiduciary technical support were readily available to solve issues and explore the best solutions to address technical and administrative problems, and the borrowers’ report (annex 6) recognizes the capacity of the World Bank team in addressing promptly the demands from the client. Records of ISRs and aide memoires were kept adequately and the project documents were easily made available. The team composition, staff resources, and budget to supervision missions were well used during the project implementation. The quality of the aide memoires was adequate for supervision purposes. Annex 7 summarizes the project aide memoires.

80. The ISRs reported progress on most of the indicators, but a complete view of these indicators was only available after the MTR survey. The team was aware of the outdated baseline and targets during project implementation, but the RF depended on administrative records and surveys that sometimes were not available or delayed. The project team made many efforts to improve data availability of the RF, and many district hospitals implemented IT systems to improve their administrative records as part of the tasks proposed by the project. However, the response was sometimes slow because it depended on the local HIS implemented by the PPMUs at the provincial level and their support to the district hospitals.

81. Environmental and social safeguards were well monitored resulting in considerable improvements in the waste management systems at the district hospitals and a change of perception of the quality of these hospitals by poor and ethnic minorities. The World Bank team’s performance was crucial to guarantee the procurement for the hospital waste management equipment and to help the country design adequate solutions for district hospitals.

36 Along the implementation, the project had four TTLs, but from 2010 to 2016, local TTL’s were in charge of implementation.

25

(c) Justification of Rating for Overall Bank PerformanceRating: Satisfactory

81. The project was implemented in a very proactive manner by the Bank team, in terms of technical assistance, good project reports and follow up of the project implementation results. The client was satisfied with the Bank assistance during implementation and the Bank team try to move the project positive agenda in the best way, recognizing the limitations and constraints of the government health policy in the country. Given all features described in paragraphs 78 to 80, combined with project results in terms of relevance, efficacy and efficiency, the overall Bank performance could be rated as satisfactory.

5.2 Borrower Performance

(a) Government PerformanceCPMU Performance: Satisfactory

82. The CPMU was established with sufficient human resources (including the national consultants) and good capacity to manage and implement the activities and to use the financial resources. As expressed in paragraph 23, the MoH authorities were totally committed to project preparation and to development of the project design. After the project be approved, they worked to guarantee adequate budget for communications, supervision, M&E, and travel during implementation. The provincial health authorities and district hospital managements were also committed to participate in the project development. Field visits were organized during preparation to establish links and networks with the local government health and hospital’s authorities. The CPMU has promoted an effective coordination with the functional departments of the MoH, accelerating the processes for evaluation and approval of the project needs in the higher government levels. The handbook for guiding the implementation of the project was compiled and issued by the decision of the Minister of Health right at the beginning of the project, contributing to speeding up its implementation. The Government established a project steering committee that works satisfactorily with regular meetings to solve issues during the project implementation.

83. The CPMU was able to promote the effort, commitment, and responsibility of the provincial leaders to increase the efficiency of the project on the economic and social development of the provinces. The CPMU also provided capacity building for the PPMUs’ staff to improve project management, including financial, accounting, and procurement issues; assets management; civil works; training on HCFP assessment; and M&E activities. The CPMU performance was essential to implement the project on time and exceed the targets. The CPMU assistance to the provinces and district hospitals was highly effective, providing significant support to the health management of the NUP beneficiaries.

(b) Implementing Agency or Agencies Performance Implementing Agency Performance Rating (PPMUs): Moderately Satisfactory

84. In the provinces with high attention and strong support from the local governments, the PPMUs performed the project more efficiently than in others. The progress on the project implementation should be considered uneven among the PPMUs. Given this situation,

26

some delays in auditing and reporting happened with differences in performance of health indicators among them. Additionally, many PPMUs did not provide stable human resources that could guarantee constant improvements in the project management and results M&E, especially close to the end of the project when staff departure was observed in some PPMUs.

(c) Justification of Rating for Overall Borrower Performance: Satisfactory

85. The overall borrower performance could be classified as satisfactory, given that despite the weakness of part of the PPMUs on M&E, the CPMU provided all elements to fix these problems by contracting local consultants and provide technical assistance to produce data and good reports that fed the final project evaluations. On the other hand, the CPMU direction increased the managerial autonomy of the district hospitals and assist the provincial health departments on improve the general management of the local health systems in the project areas

6. Lessons Learned

86. Investments in hospital infrastructure, equipment, improved skills of health staff, and incentives to retain qualified human resources at grassroots levels (provinces, districts, and communities) are important factors to achieve basic health goals (such as the MDGs) and improve access to health services for achieving universal health coverage. The project assisted localities to develop their health plans, to implement preparation steps for maintaining the activities after the project ended, and to continue the issuance of policies and plans on health support for the northern mountainous provinces. Two aspects could be highlighted: (a) the health workforce development policy and (b) the retention and training of teams of village midwives to serve remote areas where home birth delivery is necessary.

87. The project supported progressive improvements in the financing mechanisms for provincial health services, but financial sustainability at the district hospitals in the NUP areas remains a challenge. National and provincial hospitals have more ability to increase their revenues and sustain and balance their budgets than district hospitals, which face lower levels of funding, contributing to increasing the risk of poor quality services. However, in Vietnam, HI payments and user fees are the dominant source of finance for public (non-district) hospitals. The Government progressively is introducing mechanisms to strengthen the sustainability of the district health services, including the use of government bonds, which, according to a MoH study, reached 91 percent of district hospitals by 2011. However, the Government needs to set up mechanisms that could allow these hospitals to have adequate financing as well as higher autonomy and less dependency on the provincial revenues. Part of this effort should be to increase and adjust the price of the health services provided by the district hospitals (and paid by the HI) according to their real costs, especially considering that the district hospitals received lower fees than those paid to regional and national public hospitals.

88. Balanced interventions and health investments in both the supply and demand sides enable district hospitals to improve coverage, access to health services, and the range of services offered in deprived areas. Although district hospitals in Vietnam have the ability to provide about 75 percent–85 percent of their assigned service list, these hospitals in the Northern Upland provinces at the beginning of the project only provided 35 percent of the list because of lack of medical equipment, lack of health professionals, particularly specialists, and the

27

weakness of provider payment mechanism and service price list, which do not support or encourage district hospitals to provide services. The project increased the range of services from 35 percent to 80 percent addressing investments in medical equipment and human resources, but failed in addressing innovative provider payment mechanisms to increase incentives for adequate health human resources retention.

89. Support for transport and meals are efficient mechanisms to increase the utilization of the health services as these were the key barriers for the poor and ethnic minority population. Once understanding their benefits and availability of health services, the poorest populations living in distant areas need the means to travel and sustain themselves when seeking health care at district hospitals. Further, the support policy must be monitored closely to avoid abuse or overuse by both the service providers (health facilities) and users (patients and their family) such as unnecessary longer stay for inpatients, unnecessary hospitalization of the patients who can be treated as outpatients, and so on.

90. Despite the investments and incentives implemented by the project, district hospitals still have difficulty to retain staff. They have less capacity to generate incentives as well as reward health staff and improve their salaries. Thus, the provider payment mechanism should be changed and revised to encourage hospitals to produce better services. The project was able to solve partially problems of staff, benefiting from rotation of health workers from provincial hospitals and supplemental salaries to skilled doctors and health specialists working at the district hospitals. Some district hospitals in Vietnam are using capitation as an incentive to retain staff and pay for performance, but this kind of arrangement was not used in the district hospitals of the NUP provinces.

91. The institutional capacity of the PPMUs and CPMU plays an important role in implementing the project. Staff in implementation units should be well trained on all aspects of project management and work closely with the technical support from the central project team. Staff turnover should be limited. During original project implementation time, the quality of project implementation by the PPMUs was granted, but during the extension period, the transition of the project staff to the Health Provincial Services was not completed and some of the administrative functions, such as M&E, were missed. Although CPMU had only 20 staff (less than many other projects of the same scale), project activities were well implemented, the schedule and work plan were always on track, and the targets/results were surpassed compared to the planned outputs.

92. M&E systems and plans should be built ahead of the project effectiveness with standard forms for collecting data at the implementing sites. This will allow future projects to have good data and reference sources from the start, improve monitoring of projects achievements and get the accurate information to set more realistic baselines for some project indicators.

93. Strengthened coordination to assure financial sustainability is key to guarantee the project continuity at the provincial and local levels. The PPMUs should closely follow up with Provincial People’s Committees to approve annual financial plans at the earliest, so activities can be implemented at the beginning of each year.

28

7. Comments on Issues Raised by Borrower/Implementing Agencies/Partners

94. The borrower issued a report, annexed to this ICR in full, where the main comments and issues raised are summarized as follows:

(a) The project has successfully achieved its overall development objective. The utilization of district health services (by the poor and vulnerable population defined accordingly with the Decision 139) has sharply increased after IEC campaigns, improving the effectiveness of the Government’s priority to UHC, both by improving the geographical accessibility to quality health services at district hospitals and by reducing the financial burden in accessing health services for the poor and ethnic minorities.

(b) The project has been successful on implementing the health care policies for the poor, increasing their access to quality health services. The number of poor people receiving financial assistance (meals and travel costs) to visit district hospitals from NUP has increased yearly, along with the significant increase in the rate of use of district health services among the general population, especially vulnerable groups such as poor and ethnic minorities. The average number of inpatient and outpatient visits per capita per year of ‘HN’ (Ho Ngheo or Poor Household in English) in the district hospitals increased 390 percent (from 0.0247 to 0.096) and 369 percent (from 0.067 to 0.247) from 2009 to the end of 2014/2015, respectively, exceeding the project’s expectations. The awareness and confidence of the vulnerable population on health care services has improved. The percentage of households who experienced catastrophic health care expenditures at the project end was substantially lower than the baseline.

(c) The project was designed and implemented based on the needs and recommendations of the local authorities and communities. The MoH accumulated experience in implementing projects with similar components and activities. Therefore, the implementation of the project was quite favorable and the risk management process was well conducted and did not menace significantly the project performance.

(d) The project implementation was provided with sufficient human resources (including national consultants) and good capacity to manage and implement the activities and to use the financial resources. The CPMU provided capacity building for the PPMUs’ staff on project management, including financial, procurement, accounting, assets management, civil works, training, and M&E activities.

(e) The success in implementation of the project was also the consequence of good management and positive support from the World Bank team. Close monitoring of the project activities, timely issuance of no objections for procurement and work plan, suggestions for important solutions and recommendation for speeding up implementation progress, and participation in technical missions were some of the valuable contributions of the World Bank team to the success of the project.

29

(f) For the improvement of future projects, the M&E system and M&E plan should be built ahead of effectiveness with standard forms for collecting data from the PPMUs and implementing sites. This will allow future projects to have good data and reference sources from the start and improve monitoring on the projects’ achievements. The PPMUs should closely follow up with Provincial People’s Committees to approve annual financial plans at the earliest, so activities can be implemented at the beginning of each year.

30

Annex 1. Project Costs and Financing

(a) Project Cost by Component (in USD, Million equivalent)

Components Appraisal Estimate (USD Millions)

Actual/Latest Estimate (USD

Millions)

Percentage of Appraisal

1. Strengthening District-level Health Services 42.90 40.40 94.0

2. Increasing Financial Access to Health Care Services for Decision 139 Beneficiaries

10.00 10.03 101.0

3. Monitoring, Evaluation and Project Management 13.10 9.71 61.1

Total Baseline Cost 66.00 60.14 88.5Total Financing 66.00 60.14 91.1

Note: * Exchange rate loss is USD 1.59 million. Total IDA credit is USD58.41 million at project closing.

(b) Financing

Source of FundsAppraisal Estimate

(USD Millions)

Actual/Latest Estimate

(USD millions)

Percentage of Appraisal

Borrower 6.00 1.73* 28.8 International Development Association 60.00 58.41 97.3

Total Financing 66.00 60.14 91.1Note: *Without in kind contribution.

31

Annex 2. Project Outputs

Summary

1. The NUP project implementation presented a relatively good performance. The project had 3 components, 6 subcomponents, 19 identified activities, and 29 identified outputs. Annex 2A provides the project output map before the extension period and during the 18-month extension phase, identifying project achievements compared with planned or expected outputs. Of the 29 identified outputs, 8 outputs were surpassed, 9 were achieved, 9 were partially achieved, 1 was not achieved, and 2 were added during project implementation as supplemental outputs37.

2. Project outputs of Components 1 and 2 are related to the PDOs, while outputs of Component 3 are related with project administrative tasks. Giving that, this annex analyzes the achievements related to Components 1 and 2. Outputs for Component 3 are found in the project output map (Annex 2A).

A: Achievements during Original Project Implementation Period (October 2008–August 2014)

Component 1: Strengthening District-level Health Services

3. Component 1 was organized in three subcomponents (a) human resources development; (b) improving quality of the district hospitals and; (c) improving hospital management. The first subcomponent tackled training activities for health workers; the second included activities related to basic medical equipment minor repairs and upgrading in infrastructure and the last included subjects related to training of district hospitals management staff, developing hospital maintenance plans, and a management excellence award program. This component has 14 outputs: 8 were surpassed, 2 were achieved, 2 are supplemental outputs, 1 was partially achieved, and 1 was not achieved.

(a) Subcomponent 1: Human resources development

4. General conditions. This subcomponent has five activities: (a) long-term training for health workers; (b) short-term training for health workers; (c) technical handover/skills transfer; (c) innovative incentive schemes to retain health workers (Phase 1) and; (d) innovative incentive schemes to retain health workers (Phase 2). All activities were implemented by 10 outputs: 5 were surpassed, 2 achieved, 1 partially achieved, and 2 are supplemental outputs (without targets defined upfront).

5. The project financed long- and short-term trainings, covering the costs of examination and tuition fees, living allowances and per diem, training materials, and transportation. The training activities were based on plans prepared by district hospitals and the PPMUs, which were revised and approved by the CPMU and the World Bank. As can be seen, the number of trained staffs exceeded the original training plans. This subcomponent had 8

37 The achievement of the supplemental outputs is difficult to measure given that no baselines were established at the onset.

32

outputs: 5 were surpassed, 1 was achieved, and 2 were supplemental outputs (without defined targets upfront).

6. Long-term training: level-1 specialists. Table 2.1 shows that under the project funds, 377 level-1 specialist doctors were supported: 246 at the district level and 131 at the provincial level. This represents an increase of 206 percent of the original project plans. Out of those 377 level-1 doctors, 309 graduated before December 2013. The participation of doctors from ethnic minorities was 48 percent. Around 97 percent of the trained doctors returned to work at the original district hospitals, representing a significant percentage of retention of qualified doctors.

Table 2.1. Number of Level-1 Doctors Trained in Project Provinces According to Plan

Project Provinces

Level-1 Doctors

as Planned

Level-1 Doctors Graduated Level-1 Doctors who Returned to Work at Localities Number

Supported by Project

Number of Person

Years under Project Support

Total Female (%)

Ethnic Minority

(%)Total Female

(%)

Ethnic Minority

(%)Cao Bang 23 56 58.9 92.9 54 59.3 92.6 56 (243%) 102 (222%)Bac Kan 31 36 50.0 100.0 34 50.0 100.0 50 (161%) 96 (155%)Ha Giang 38 89 31.5 14.6 86 31.4 14.0 96 (253%) 176 (232%)Lao Cai 22 38 39.5 18.4 35 37.1 28.6 51 (232%) 97 (220%)Lai Chau 22 17 35.3 47.1 17 35.3 47.1 19 (86%) 37 (84%)Dien Bien 19 47 34.0 42.6 47 34.0 42.6 50 (263%) 95 (250%)Son La 28 26 34.6 38.5 26 34.6 38.5 55 (196%) 102 (182%)Total 183 309 40.5 47.2 299 40.1 48.2 377 (206%) 705 (193%)District level 104 — — — — — — 246 (236%) 457 (220%)Province level 79 — — — — — — 131 (166%) 248 (157%)

Source: Center for Environment and Health Studies (2014) Final Report: End-line evaluation of the Northern Uplands Health Support Project, Hanoi, 2014.

7. Long-term training: assistant doctors to become medical doctors. The project supported 1,467 assistant doctors attending the four-year additional training, exceeding 98 percent of the agreed target on the original project plans. Of this number, 54 percent were ethnic minorities. Additionally, the project trained 98 assistant pharmacists, attending the four-year additional training to become pharmacists (32 percent ethnic minorities). This additional outcome was not originally planned under the project.

8. Short-term trainings: curative care. The project sponsored 3,187 doctors to receive short-term training in several specialties (table 2.2), compared with 1,041 originally planned by district hospitals in agreement with the PPMUs (306 percent increase). The CPMU surveyed the opinions of 226 health staffs who participated in short-term trainings. According to the survey, the participants highly appreciated the training methods and contents of the courses. Approximately 91 percent and 93 percent of respondents were satisfied with the teaching methods and contents of the courses and 75 percent reported that the training durations were appropriate.

Table 2.2. Number of Doctors Receiving Short-term Trainings according to the Type of Trainings

Course Plan Total Percentage of PlanAnesthesia 92 135 147Lab test 95 149 156Rehabilitation 105 121 115Surgery 101 143 142

33

Course Plan Total Percentage of PlanPediatrics 103 214 208Internal medicine 111 194 175Image diagnosis 165 404 244Obstetrics 102 191 187Communicable diseases 80 165 206X-ray 87 103 118Pediatrics emergency n.a. 681 —Obstetrics emergency n.a. 687 —Total 1,041 3,187 306


9. Short-term trainings: preventive care. The project trained 335 staffs (156 at the provincial level and 179 at the district level) focusing on the planning, implementation, evaluation, and monitoring of preventive medicine activities at the primary level. This number exceeded the originally planned target of the PPMUs, by 57 percent. The trainings included the preventive and counseling aspects of several diseases including HIV, tuberculosis, and malaria, which are targeted at the MDGs.

10. Other short-term trainings: hospital management, health systems management, information systems, and maintenance of medical equipment. Table 2.3 shows that the number of trained staffs on these specific short-term trainings exceed the original plans agreed with the district hospitals and the PPMUs. The number of personnel trained on medical waste management shows that a huge part of the district hospital staff was involved in the accomplishment of the project’s environment safeguard, given that almost 3,000 staffs were trained in this specific subject.

Table 2.3. Number of Hospital Staffs Trained and

Comparison between the Planed and Achieved

Field of training Number of Staff Training Planed

Staff Trained in 2014

Percentage Plan Achievement

Hospital management 303 477 157 SurpassedMedical waste management 215 2,922 1,300 SurpassedHealth Management and Information Systems (HMIS) 358 397 11 Surpassed

Maintenance of equipment 134 169 126 SurpassedSource: Center for Environment and Health Studies (2014) Final Report: End-line evaluation of the Northern Uplands Health Support Project, Hanoi, 2014.

11. Incentives to retain human resources in district hospitals: The PAD included, as one of the project activities, the use of incentive schemes to attract and retain the newly trained health workers at district hospitals because of a severe shortage of health staff in the NUP areas. To address this issue, the CPMU and the PPMUs discussed and proposed solutions in workshops that were held to seek sustainable solutions to maintain adequate health workforce for disadvantaged areas in the NUP. The workshops discussed national and international experiences on enrollment and training of health staff. However, the solutions and recommendations focused mainly on training contents and less on financial support, sustainability, and budget feasibility of the proposed incentives. To tackle the staff shortage in the NUP areas, during the health workers’

34

training period, the World Bank agreed that the provinces could hire retired doctors to work at the district hospitals and health centers. Some provinces, including Cao Bang, Lao Cai, and Bac Kan had used this hiring mechanism to partially overcome staff shortages with positive results.

12. District hospitals gained expertise on many specialties such as surgery, internal medicine, obstetrics, pediatrics, intensive care, traditional medicine, nursing, and anesthesia, focusing on emergency techniques and surgery. The project financed the technical support to the district hospitals’ doctors by assigning provincial doctors to guide the former on every step of these specific techniques and helping them practice these techniques themselves. In 2014, over 670 provincial doctors participated in the transfer of these techniques to district hospitals exceeding 167 percent of the PPMU project’s plans. This activity was planned as a result of the project MTR in 2012.

(b) Subcomponent 2: Improving quality of district hospitals

General conditions. When compared with the original plans, two outputs of this subcomponent were surpassed.

13. Equipment for district hospitals was procured in two phases (as recommended in the PAD): Phase 1 in 2011–2012, and Phase 2, in 2013–2014. Table 2.4 shows the number of equipment acquired in the two phases in the seven provinces, which reached 4,400 units (109 percent of the original procurement plans). Phase 1 included ambulances to address the transfer of emergency patients in remote districts and four groups of medical equipment: laboratory equipment, treatment equipment, monitoring devices and ventilators, and infection control team equipment. Phase 2 included equipment of high-value technology and trained staffs capable of using this equipment. The project combined investment and infrastructure facilities and human resource staffing and training in a synchronous manner to avoid under use and depreciation of the equipment.

Table 2.4. Number of Project-supported Equipment for the District Hospitals at Seven NUP (2012–2014)

Equipment Son La

Ha Giang

Bac Kan

Cao Bang

Dien Bien

Lai Chau

Lao Cai Total Plan Percentage

ObtainedLab test devices 181 131 128 127 83 74 98 822 664 123.8Intensive care equipment 117 145 126 195 93 78 119 873 1255 69.6Monitoring equipment 21 24 28 23 14 11 21 142 157 90.4Ventilators 22 19 14 25 14 4 14 112 217 51.6Infection control equipment 29 28 22 34 14 10 23 160 173 92.5Incinerators 0 0 0 0 0 0 0 0 37 0.0Ambulance care 8 5 5 14 6 6 8 52 52 100.0Ultrasound machine 24 14 16 16 11 11 17 109 77 141.6Endoscopic machine — — — — — — — — 37 —Emergency care devices 59 69 43 75 37 36 47 366 390 93.8Internal medicine 111 40 59 59 25 18 35 347 209 166.0Surgical room equipment 85 46 55 65 40 36 55 382 225 169.8Surgical equipment 158 82 94 120 62 72 67 655 548 119.5Examining equipment, 69 55 53 68 37 35 48 365 296 123.3X-ray machine 18 18 16 12 14 9 15 102 78 130.8Pediatric/ obstetrics surgery 19 7 3 31 25 68 21 174 0 —Intensive care in obstetrics 15 11 3 31 20 45 35 160 0 —Total 936 694 665 895 495 513 623 4,821 4,415 109.0


35

14. However, a small portion of the equipment provision was delayed. Some of the equipment supplied by the project faced difficulties in the import procedures and by 2014, few bidding processes were ongoing. The delivery of part of the acquired equipment to hospitals in remote areas was challenging at times because of difficult weather conditions.

15. The equipment installed by the project achieved high level of utilization. The 2014 project results survey shows that 95 percent of the equipment was delivered and installed into the district hospitals and was frequently used for diagnosis and treatment. Only 1.4 percent of the installed equipment was broken and 3.9 percent was unused because of lack of technical skills or utility/infrastructure associated problems. Table 2.5 shows that in January 2014, the installed equipment achieved a high level of monthly utilization in the respective district hospitals.

Table 2.5. Number of District Hospitals’ Patients using Equipment in the NUP Areas (January 2014)

Group of Equipment Number of Cases using the Equipment/MonthSurgical room 1,116 Obstetric monitoring 159 Cardiograph 49Semiautomatic biochemical analyzer 1,981Automatic hematological analyzer 600Ultrasound 418X-ray (high voltage) 466


16. The project implemented the upgrading or new construction of 18 district hospitals compared to 10 that were originally planned. All civil works were carried out in 6 provinces between 2012 and 2013. The only province where civil works were delayed was Lao Cai, where two hospitals were expected to have minor repairs. These hospitals were supposed to be upgraded further using public resources.

(c) Subcomponent 3: Improving hospital management

17. General conditions. This subcomponent had three outputs: one was achieved, one partially achieved, and one not achieved.

18. Many hospital managers and provincial authorities were trained in hospital management programs by recognized institutions. The project has signed an agreement with training units such as the School of Public Health, Bach Mai Hospital, and Hanoi Medical University to organize management training for 477 hospital leaders and heads of departments in provincial and district hospitals. The training contents were focused on human resources management, health financial management, health-financing analysis, procurement and bidding processes, and quality control of hospital performance. Considering that the planned goal was the training of 303 hospital managers, the original target was exceeded by 57 percent. The trainees evaluated the quality of the courses very positively. Survey results from in-depth interviews with some hospital leaders showed that the management skills that were learned were effectively used in the preparation of the district hospitals annual plans, plans for human resource development, and other activities of the hospital. Overall, the project achieved its goals and the project funds were used effectively.

36

19. Most of the district hospitals under the project developed and implemented maintenance plans for facilities and equipment and increased the budgets to implement these plans in 2013. The results of the 2014 survey revealed that, when compared with the early project implementation years, the proportion of district hospitals having annual plans for maintenance increased, but with some shortcomings by the end of the project. In December 2013, 99 percent of the hospitals developed maintenance plans. However, at the end of 2014, only 80 percent of the district hospitals kept these plans active. The executed budget for facilities and equipment maintenance at the district hospitals achieved 87 percent of the total estimated costs in 2013, compared to 57 percent in 2008.

20. Excellence award for human resources. The project planned to implement an excellence award for human resources, but this initiative was the only activity that was not implemented, as a project activity, during the project life. The CPMU did not collect information on this activity. However, annually, the provincial health departments follow health units and individuals who performed well. They are rewarded and apprised according to the Emulation and Reward Regulation.

Component 2: Increasing Financial Access to Healthcare Services for Decision 139 Beneficiaries

21. Component 2 was organized around three subcomponents: (a) support for direct catastrophic and nonmedical expenditures of health care for Decision 139 beneficiaries; (b) strengthening capacity for HCFP; and (c) strengthening local access to health services through promoting health seeking behavior. The first subcomponent addressed the support for nonmedical expenditures to poor and ethnic minorities’ beneficiaries. The second supported institutional capacity building for district hospitals to identify target beneficiaries; to provide them HICs and to institute mechanisms to receive medical expenses payments. The third subcomponent developed IEC campaigns to increase awareness and knowledge from ethnic minorities about their rights, entitlements, and benefits from HI. This component had four outputs and, at the project end, three were achieved and one was partially achieved

(a) Subcomponent 1: Support for direct catastrophic and nonmedical expenditures of health care for Decision 139 beneficiaries.

22. The number of project assistance recipients and the value of the monetary allowances increased significantly during the project. In 2009, the project was committed to support 730,183 (only the poor) beneficiaries (or 25 percent of the population in the seven provinces), but in 2013 this number jumped to 3,185,341 beneficiaries, given the inclusion of the ethnic minorities (82 percent of the population in the same provinces). This support covered recurrent costs for travel and meals for the population seeking health services at the district hospitals. From June 2009 to August 2011, the project covered these expenses for the population under the poverty threshold. Since September 2011, the project coverage was expanded for all ethnic minorities (including those above the poverty threshold) because they are quasi-poor and represented the majority of the NUP population. Until May 2012, the value of the allowance was VND 15,000 per day for inpatient meals. After May 2012, the allowance value for inpatient meals increased to VND 25,000 per day. The allowance for travel varied according to the distance (VND 60,000 under 100 km and VND 100,000 for 100 km or more). Table 2.6 shows

37

the expansion of the coverage of the nonmedical expenditures for poor and ethnic minorities from 2009 to 2013 in the seven NUP areas.

Table 2.6. Number of Targeted Beneficiaries of NUP, 2009–2013

Province

Number of

Districts withNUP

Support

Persons in 2010 (below the poverty threshold)

Number of Persons in

2011 (below the poverty threshold)

Number of Persons since August 2011

(poor + ethnic minorities of

communes 30a)

Number of Persons since

June 2012 (added from

HEMA Project)

Number of

Persons in 2013

Dien Bien 9 29,399 50,008 83,426 466,000 411,405Lai Chau 7 50,279 78,916 133,933 300,000 263,390Son La 9 157,653 103,054 339,126 700,000 700,000Cao Bang 13 136,497 204,305 427,453 427,453 425,342Bac Kan 8 73,748 87,949 282,154 282,154 263,390Ha Giang 10 155,522 318,243 607,181 607,181 647,194Lao Cai 8 127,085 313,736 424,145 424,145 471,082Total 64 730,183 1,156,211 2,297,418 3,206,933 3,185,341


23. The number of inpatients’ coverage increased substantially, after the implementation of the policy, extending the allowance for all ethnic minorities. According to table 2.7, in 2009 only 1,054 district hospitals’ inpatients were supported by the project, compared to 244,801 in 2013. The accumulated number of inpatients who benefited by the project from 2009 to 2013 reached 544,000, expending 86 percent of the resources planned for this activity in the project. The household survey results at the project end line (2014) showed that 77 percent of the respondents reported that they were supported on travel cost for inpatient treatment at the district hospitals and 78 percent of those said that the NUP project supported meal costs for the poor/ethnic minority people during their treatment at the district hospitals. The project contributed significantly to promote the increased access to district hospitals for the poor and ethnic minorities in the seven provinces.

Table 2.7. Total District Hospitals’ Inpatients Supported by Provinces, 2009–2014

Provinces 2009 2010 2011 2012 2013 First Three Months of 2014

Dien Bien 411 1,779 3,814 16,889 30,058 7,524Lai Chau — 1,264 4,205 13,605 18,132 4,716Son La — 875 2,617 24,856 44,195 10,106Cao Bang 186 4,853 18,129 44,248 45,044 10,533Bac Can — 3,654 12,769 26,975 27,756 7,088Ha Giang 30 3,457 17,727 38,781 39,702 11,231Lao Cai 427 3,508 20,252 35,345 39,914 8,020Total 1,054 19,390 79,469 200,709 244,801 59,218


38

(b) Subcomponent 2: Strengthening capacity for HCFP

24. Given improvements and gains of scale and efficiency in the management of this subcomponent, 100 percent of the objective was achieved with a substantial cost reduction . Since May 2012, the management functions of the HCFP were extended to the HEMA Project provinces, increasing the total number of HCFP beneficiary districts up to 64 districts compared to the initial 50 districts. Two out of five of the HEMA Project provinces (Dien Bien and Lai Chau) upgraded to manage HCFP were part of the NUP project. Therefore, the planned budget of this subcomponent was reduced by 71 percent compared to the plan in the project document, (from USD 872,282 to USD 255,925). On the other hand, simplifications to transfer the HCFP funds to provinces with poor and ethnic minorities reduced the cost of the procedures that were expected to be created at the provincial HCFP management units.

25. In March 2014, all activities of this subcomponent were completed and high efficiency in strengthening the management capacity of the HCFP was achieved. These activities (including training courses for the fund management unit, support of equipment, and monitoring of the fund) were completed, disbursing USD 191,652.59 (75 percent of the planned budget after adjustment).

(c) Subcomponent 3: Strengthening local access to health services through promoting health seeking behavior

26. IEC-related activities, such as assessment needs, staff training, and provision of IEC equipment paved the way to raise awareness and changes in behavior, thus promoting higher utilization of health care services for the poor and ethnic minority people. From 2010 to 2014, the project implemented several IEC activities such as (a) conducting an IEC needs assessment in the project provinces as a basis to develop the IEC strategy and prepare IEC materials for community and training for educators; (b) training 411 district and 9,070 commune staffs in the seven provinces, for IEC activities; (c) providing essential IEC equipment, such as digital video cameras, non-linear editing systems, loudspeakers, cameras, recorders, portable speakers, image editing kits, and the so on to the provincial IEC centers established by the project; and (d) printing and distributing materials to the provincial centers and district rooms to conduct IEC activities for the community, according to the plan that was proposed by the PPMUs and approved by the CPMU.

27. The local health staffs and community reported that the IEC activities of the project substantially reached out to the poor and ethnic minority people. According to the final report of the CPMU, in 2013, 880 out of 899 IEC project communes in the provinces had IEC activities implemented, reaching 98 percent of the original plan. The total number of local people benefiting from the IEC activities of the project was 25,744.

28. The effectiveness of the IEC activities of the project was highly appreciated based on the results of the household survey at the end line as compared to the end-line target of the project. The end-line evaluation results show that most NUP inhabitants who had HICs could name at least one right of the HICs (IOI #9). The rate of respondents who could name at least three rights of the HICs increased significantly from 14.8 percent to 57.4 percent.

39

29. The implementation of the IEC subcomponent led to savings in the project . As of March 31, 2014, disbursement of the subcomponent reached USD1.13 million (86 percent of adjusted plan). However, by March 31, 2014, IEC activities continued to be implemented until December 31, 2014, for an amount of USD 186,625.

B: Project Achievements during the 18-Month Project Extension Period (August 2014–February 2016)

Component 1: Strengthening District-level Health Services

(a) Subcomponent 1: Human resources development

30. Long-term training: level-1 specialists. During the project extension, 57 level-1 specialists graduated before December 2015, bringing the total number of graduated doctors to 367. Ten additional level-1 specialists were expected to finish graduation after project closing. Thus, by the end of the extension phase, there was no change in the number of level-1 specialists supported by the project and an addition of 57 new graduated doctors. At the late extension phase, nearly 97 percent of the trained level-1 specialists, who graduated to doctors, returned to their former workplace: 57 percent of the level-1 trained doctors are ethnic minorities and 53 percent are female. This reflects increased sustainability of the human resources supply in the provinces.

31. Long-term training: assistant doctors to become medical doctors. During the project’s extended phase, 324 additional assistant doctors graduated as medical doctors using the project funds, totaling 1,058 assistant doctors who transformed to medical doctors along the project. In total, it represents 160 percent of the original plans’ targets. Additionally 40 pharmacists graduated in the extension phase (see table 2.8).

Table 2.8. Number of Four-year Additional Doctors and Pharmacists who Graduated by the Evaluation Time

Health Professionals Trained CaoBang

BacKan

HaGiang

LaoCai

LaiChau

DienBien

SonLa Total

Support for four-year added doctorsTotal doctors graduated (#) 131 53 213 145 142 192 182 1,058Doctors graduated in regular project time (#) 104 42 130 106 91 142 119 734Doctors graduated in project extension phase (#) 27 11 83 39 51 50 63 324Doctors returned to work at their units (#) 122 44 208 96 136 193 179 978Plan achievement (%) 147 144 192 179 128 179 148 160Rate of ethnic minority doctors (%) 99 100 36 36 15 34 44 46Rate of female doctors (%) 99 50 32 41 43 29 40 45

Support for four-year additional pharmacist trainingTotal pharmacists graduated (#) 13 7 15 23 13 16 11 98Pharmacists graduated in regular project time (#) 6 6 9 10 6 13 8 58Pharmacists graduated in project extension phase (#) 7 1 6 13 7 3 3 40Rate of ethnic minority pharmacists (%) 92.3 71.4 20 26.1 7.7 6.0 18.2 30.6Rate of female pharmacists (%) 86.4 71.4 60 82.6 69.2 50 72.7 70.4

Source: Center for Environment and Health Studies (2016) Final Evaluation Of The Northern Uplands Health Support Project In The Extension Phase, Hanoi, 2016.

32. Short-term training: curative care. The extended phase contributed to complete the plan’s achievement for some important short-term trainings (emergency care and x-ray) and to

40

exceed the plan in other short-term trainings that were already accomplished in the regular project time, adjusted according to the needs from district hospitals (anesthesia, and diagnose imaging). During the extension phase, additional training was provided in areas that were not present in the project’s regular time (ear, nose, throat and dental care, and nursing managing). The extension phase added 18 percent additional short-term trainings in this specific activity. Therefore, at project closing, the short-term trainings completed was 2.6 times the original plans’ goals, as seen in table 2.9.

Table 2.9. Number and Rate of Doctors Attending Short-term Training in the Extension Phase

Course PlanRegular Project Time

Extension Phase

Percentage Added in Extension

Phase

Total in Project

Life

Percentage Plan Increase

Anesthesia 117 135 22 19 157 134Teasing 95 149 0 — 149 156Emergency care 140 121 31 22 152 109External medicine 101 143 0 — 143 142Pediatrics 103 214 0 — 214 208Internal medicine 111 194 0 — 194 175Diagnostic imaging 200 404 32 16 436 218Obstetrics 102 191 0 — 191 187Communicable diseases 80 165 0 — 165 206X-ray 107 103 18 17 121 113Ear, nose, and throat, and dental 108 0 70 65 70 65Nursing management 70 0 63 90 63 90Pediatric emergency n.a. 681 0 — 681 —Obstetric emergency n.a. 687 0 — 687 —Total 1,334 3,187 236 18 3,423 257


33. Additional short-term training on MDG related activities in the project extension. Along the regular project time, training for obstetric and pediatric emergencies was important to prepare health staff to tackle maternal and neonatal mortality for ethnic minorities in the NUP mountainous areas. During the project extension, the CPMU organized additional training to certify health staff on appropriate interventions to increase the accessibility and utilization of maternal and child health care services in villages, community health centers, and district hospitals. The CPMU worked with the related stakeholders (Maternal and Child Health Department, Provincial Departments of Health, and so on) to establish training and certification for skilled birth attendants (SBA) (directed to general doctors and other health workers) and for specialists to work on emergency in gynecological and obstetric care. Specific training was delivered to midwives at the village level. As result, between August 2014 and December 2015, the following professionals were trained and certified: (a) 794 health staffs working on obstetric and gynecologic services were certified as SBAs; (b) 307 general doctors working on obstetric and gynecologic services at the district/commune levels were certified as SBAs; (c) 110 medical doctors were trained on emergency care for obstetric and gynecologic services at district level; (d) 252 village midwives graduated; and (e) 79 clinical staff at neonatal units of the district hospitals, and other minor trainings were provided according to the needs.

41

34. Short-term training: preventive care. Given that the training in this area was achieved and exceeded during the project’s original phase, no additional training was provided during the extension phase.

35. Short-term training: hospital management, medical waste management, HMIS, and maintenance of medical equipment. During the regular project time, the goals for these four kinds of short-term trainings were already achieved and surpassed. In the project extension time (table 2.10), these short-term training courses were also delivered with 75 percent more staff over the original plans (761 staff) trained. At the end of the project, the number of professionals trained in hospital management was 2.5 times more than what was originally planned, focusing on areas such as human resources management, health financial management, health financial analysis, procurement management, bidding, and hospital quality management. In the area of medical waste management, no additional training was delivered in the project extension phase, but it already achieved 13.6 more times of staff training than planned. Regarding HMIS, the extension time added 125 percent of trained staff. Regarding the maintenance of equipment, the achievement was 28 percent of additional trained staff. In summary, the extension phase provided additional trained staff to the already achieved goals. The overall achievement of short-term training for health staff was almost five times higher than planned.

Table 2.10. Health Staffs Attending Hospital Management, Health Systems Management, Information Systems, and Maintenance of Medical Equipment Short-term Training in the Extension Phase

Course PlanRegular Project Time

Extension Phase

Percentage Added in Extension

Phase

Total in Project Life Percentage Plan Increase

Hospital management 303 477 276 91 753 249Medical waste management 215 2,922 0 — 2,922 1,359HMIS 358 397 448 125 845 213Maintenance of equipment 134 169 37 28 206 154Total 1,010 3,965 761 75 4,726 468


(b) Subcomponent 2: Improving quality of district hospitals

36. Given that this activity was accomplished in the regular project time, few new equipment was provided during the extension period to the district hospitals. Only new equipment related to improving the quality of care for mother and children undergoing surgeries and intensive care were acquired during that period. Equipment included the model of delivery attendance instruction and neonatal resuscitation and accompanied appendixes; clean birth delivery packages; instrument bags for village midwives; equipment for infant weight and length measurement and kits for newborn resuscitation (including vacuum and suction pipe, metals collection box, heating lamp, mask, and oxygen ventilation). Therefore, during the extension period, the project’s new purchases were mainly focused on equipment for neonatal units and village health teams, including instruments and consumable supplies for neonatal units. In total, the project, with the extension phase, purchased 5,065 medical equipment; 14.7 percent more than what was originally planned.

42

37. By the end of 2015, all equipment acquired by the project achieved high levels of utilization in the district hospitals. Figure 2.1 shows the levels of utilization of almost 100 percent in three of the seven provinces. The only province where the level of utilization was lower than 90 percent was Cao Bang, which was expected due to the small demand for hospital inpatient facilities when compared with other project district hospitals. No new constructions and civil works in the district hospitals were financed by the project during the extension phase.

Figure 2.1. Percentage of the Equipment Acquired by the Project that are in Use in the District Hospitals of the Seven NUP Provinces.

Son La Ha Giang Bac Kan Cao Bang Dien Bien Lai Chau Lao Cai Total0

20

40

60

80

100 93.7 99.6 93.8 87.9100 99.7 97.3 95.1

Source: Center For Environment And Health Studies (2016) Final Evaluation Of The Northern Uplands Health Support Project In The Extension Phase, Hanoi, 2016.

(c) Subcomponent 3: Improving hospital management

38. Short-term trainings on hospital management and maintenance plans for facilities and equipment were kept as project priorities during the project extension phase for sustainability purposes. However, the district hospitals and health provincial authorities at the provincial level are still struggling with the perspectives of increasing budgets to build long-term sustainability of project interventions. One of the lessons learned in this area is that policy development is a process that requires a lot of time and involvement of ministries and provincial people's committees in Vietnam. In this case, the project could be considered successful, given that between 2014 and 2015 (a) the percentage of district hospitals with plans to maintain physical infrastructure increased from 80 percent to 86 percent and the budget for these activities increased by 11 percent and (b) the percentage of district hospitals with plans to maintain medical equipment increased from 91 percent to 94 percent and the budget for these activities increased by 47 percent.

Component 2: Increasing Financial Access to Healthcare Services for Decision 139 Beneficiaries

39. Three out of the four outputs of this component were achieved and one partially achieved before the extension period. However, no additional information about the indicators of this component has been provided during the extension phase, making it difficult to know if the achievements remain sustained. There are indirect evidences (such as budget allocations) confirming that the support to health care nonmedical expenditures for the poor, by providing meals and transportation subsidies in the seven provinces, worked during the project extension phase. However, IEC activities were not carried out during the extended period (only until December 2015).

43

Annex 2A. Project Output Map

Subcomponent Related Activities Planned Outputs

Output Achieved Before Extension (October 2008–August 2014)

Achievement as Percentage of the Planned Output before Extension

Output Achieved during Project

Extension Period (February 2016)

Additional Achievement as

Percentage of the Planned Output

Total Percentage of Achievement of the

Planned Output during the Project

LifeComponent 1: Strengthening District-level Health Services (USD42.9 million)

(a) Human resources development

Long-term training activities for health workers

Train 183 level-1 specialists

309 level-1 specialists trained 169 58 level-1 specialists

finished training*1 32 201 (Surpassed)

741 assistant doctors to become medical doctors

734 assistant doctors became medical doctors

99324 additional assistant doctors became medical doctors*2

44 143 (Surpassed)

No planned outputs for assistant pharmacists

58 assistant pharmacists became pharmacists

—40 additional assistant pharmacists became pharmacists

— Complementary Output

Short -term training activities for health workers.

1,334 medical doctors to receive short-term training in medical techniques on curative care

3187 medical doctors received short term training in medical techniques on curative care

238

236 additional medical doctors received short-term training in medical techniques on curative care

18 256 (Surpassed)

No planned additional short-term training on MDG-related activities

—

—

1542 health staffs received short-term training on MDG-related activities*3

— Complementary Output

213 health staffs to receive short-term training in preventive care

336 health staffs received short-term training in preventive care

157 — — 157 (Surpassed)

1,010 health staffs to receive short-term training on hospital management, HMIS, and maintenance of medical equipment

3,965 health staffs received short-term training on hospital management, HMIS, and maintenance of medical equipment

393

761 additional health staffs received short-term training on hospital management, HMIS, and maintenance of medical equipment

75 468 (Surpassed)

Techniques/skills transfer

Promote knowledge sharing between provincial and district level facilities (402 times of technique transfers at the district hospitals)

Over 670 times of technique transfers were implemented in the district hospitals at regular project time 167

The project gave priority to transfer techniques/services for emergency of obstetrics, pediatrics, that district hospitals have not had ability to do as required by the MoH.

—

167 (Surpassed) Transfers of techniques were achieved establishing sustainable mechanisms to develop capacity in service provision, especially at district and commune levels

Innovative incentive schemes to retain health workers

Study on the characteristics of health workers

In 2009, the CPMU conducted a study on ‘Health Human


44










Life

Phase I - Rapid labor market assessment

currently working in the NUP and job characteristics in the NUP provinces (salaries and opportunities)

Resource Analysis in 7 Provinces of Northern Upland’. The study was used to plan human resources health needs in the NUP and establish training goals and incentives

Innovative incentive schemes to retain health workers

Phase II - Developing and implementing innovative incentive schemes

Develop potential incentiveschemes and study their likely impact on recruitment andretention

Backup arrangement for health staffs when being trained;

Hiring doctors who are retired to work for the hospitals/district health centers.

— — —

This output was partially achieved, The provinces need to continue training and developing mechanisms and policies to support retention of health staffs at district level.

(b) Improving quality of district hospitals

Basic medical equipment

Acquire and install 4,415 medical equipment

4,821 medical equipment were acquired and installed

109244 additional medical equipment were acquired and installed

6 115 (Surpassed)

Minor repairs and upgrading

10 district hospitals to have new construction and upgrades

18 district hospitals had new construction and upgrades 180 — — 180 (Surpassed)

(c) Imp1roving hospital management

Training of district hospital management staff

303 health staffs to attend hospital management training

477 health staffs had attended hospital management training 157

276 additional health staffs had attended hospital management training

91 248 (Surpassed)

Developing hospital maintenance plans

All district hospitals to have infrastructure and equipment maintenance plans

80% of the district hospitals had maintenance plans in December 2014

80

86% of the district hospitals had maintenance plans in December 2016

86 86 (substantially achieved)

Management excellence award program

The manager excellence award program does not appear in the program execution documents*4

— — — — Not achieved

Component 2: Increasing Financial Access to Healthcare Services for Decision 139 Beneficiaries (USD10.0 million)(a) Support for

direct catastrophic and nonmedical

Study of distribution and patterns of catastrophic expenditures in the first year

Develop a survey in the first project year to establish a baseline for the KPI and IOI related to the

The survey was done in the first year of project execution and the baseline for Component 2 KPI

Achieved

Two other similar surveys were developed in the MTR and in the original project closing to follow up the project

Achieved Achieved*5

45










Lifeexpenditures of health care for Decision 139 beneficiaries

project Component 2 and IOI set Component 2 KPI- and IOI-related targets.

730,183 beneficiaries (25% of the population in the seven provinces) were targeted to be supported according to the 2009 survey

3,185,341 beneficiaries (82% of the seven provinces population) were supported in December 2013 according to the end project survey

Achieved — — Achieved*6

(b) Strengthening capacity for HCFP

The HCFP capacity is measured by their capacity, at the provincial level, to provide the support for transportation and food allowance.

All activities were completed with a disbursement rate of 75% of the budget schedule after the adjustment. Achieved

At the time of the extension-phase project evaluation, three provinces (Dien Bien, Bac Kan and Cao Bang) have not yet established the health care for the poor fund because they could not balance the funds for this activity.

— Partially achieved

(c) Strengthening local access to health services through promoting health seeking behavior

IEC activities, such as (a) searching for assessment needs, (b) training staff in IEC and, (c) providing IEC equipment

899 communities in the seven provinces to receive IEC activities

880 communities in the seven provinces received IEC activities addressing 25,744 local people in December 2013.

98

No IEC activities were identified as performed in the project extension phase. — Achieved*7

Component 3: Monitoring, Evaluation, and Project Management (USD13.1 million)No subcomponent has been identified

Strengthen procurement, financial management, and disbursement

The CPMU contributed to increase local capacity (PPMUs) for fiduciary procedures. Some delays in procurement of medical equipment were identified. At the end of the first phase, disbursement was 88% of project loan.

Partially achieved

The project management was effective, focused on promoting the project results, and enhanced the sustainability. The procedures and processes of project implementation were relatively clear and no significant gaps were identified.

— Partially achieved*8

Training of project management staff The CPMU, with World Bank support, provided training for the project

Achieved

In the extension phase the PPMUs reduced staff and increased rotation creating some

Partially achieved Partially achieved

46










Lifemanagement staff at the PPMUs

gaps on trained staff, especially for M&E purposes

Provision of necessary office equipment Office equipment was provided to the CPMU. However, some PPMUs were difficult to be fairly equipped.

Achieved

Office equipment was provided to the CPMU. However, some PPMUs had difficult to be fairly equipped.

Achieved Achieved

Financing of incremental operating costs The project activities were financed properly. Achieved

Some PPMUs had constraints to finance operational costs during the extension phase.

Partially achieved Partially achieved

M&E activities Baseline data collection

As scheduled, this was completed in the first semester of 2009

Achieved — — Achieved

Indicators’ update Indicators’ update was done based on surveys (2009, 2012, and 2014) and project administrative records

Achieved

Some indicators were not updated after the project extension Partially achieved Partially achieved

MTR MTR was done in 2012 Achieved — — Achieved

End-of-project completion reports

The first end-of-project completion report was done in July 2014 (regular project time)

Achieved

The second end of project completion report was done for the extension phase in February 2016 (project extension phase)

Achieved Achieved

Audits Project financial audits were done for procurement and financial management with some delays*9

Partially achieved — — Substantially achieved

Source: Center for Environment and Health Studies (2014) FINAL REPORT: End-line evaluation of the Northern Uplands Health Support Project, Hanoi, 2014, and Center for Environment and Health Studies (2016) FINAL EVALUATION OF THE NORTHERN UPLANDS HEALTH SUPPORT PROJECT IN THE EXTENSION PHASE, Hanoi, 2016.

(*1) After the project closing in February 2016, there were 10 remaining Level 1 medical doctors expecting to be graduated.(*2) After the project closing in February 2016 there were 409 remaining assistant doctors having training to be graduated medical doctors. So, the total assistant doctors supported by the project were 1467.(*3) Training was offered at provincial, district and village level according the specialty.(*4) The Government had their own long standing system and mechanism for awarding well-performing hospitals. They did it every year as a routine activity. However this did not appear as a project related activity and the results of this award mechanism were not reported to the Bank in the project documents.

47

(*5) The last survey was related with the original project closing. During the extension project phase a new survey was not planned given that the extension was not planned to follow-up the Project Component 2.(*6) This achievement is related to the project regular time. The increase of the number of beneficiaries entitled to receive benefits is related to the inclusion of all ethnic minorities as project beneficiaries by May 2011. In the first three months of 2014 the district hospitals under the project provided subsidies to 59,182 beneficiaries. From 2009 to 2013, the number of ethnic minorities’ inpatients receiving subsidies increased from 1054 to 244,181 totalizing more than 545 thousand inpatient along this period.(*7) Despite the 98% of the target achievement compared to the plan in the regular project time, the Project Evaluation Report of the Extension phase recommend strengthening of the IEC activities under the project. (*8) The implementation progress of some training courses and provision of medical equipment were slightly slow according the project registers.(*9) The delays in the project financial audits were registered in the beginning of the project. In the last 3 years of execution and during the project extension, the audit reports were submitted on time.

48

Annex 3. Economic Analysis: Output Efficiency, Benefits, and Equity Impacts

Introduction

1. This annex addresses the project’s efficiency by analyzing (a) the rationale of the Government interventions in the project and in the Northern Upland Provinces; (b) the efficiency in achieving quality access for the district hospitals by the NUP population (project outputs); (c) the project contribution to improve health benefits (project outcomes); and (d) the impact of the project interventions on the equity on health access and health spending in benefit of the poor and ethnic minorities.38 According to the project economic analysis (annex 9 of the PAD), the expected benefits were associated with health outcome improvements for the NUP areas’ population and to narrow the health spending gap between the poor and the average population. The benefits would be achieved by the following main interventions: (a) improve efficiency on the health services delivery by increasing supply of skilled human resources and refurbishing and equipping the district hospitals; (b) improve efficiency for the poor on assessing health services by removing financial barriers to increase health care services utilization; and (c) reduce the risk of impoverishment for the NUP poor and ethnic minorities populations.

Rationale of the Government Interventions

2. As stated in the PAD, the Northern Upland Provinces constitute the most disadvantaged region in Vietnam, with an unusually high concentration of poor and ethnic minorities living in sparsely populated, mountainous localities under difficult circumstances. The Government’s choice to invest in this region was based on the unfavorable socioeconomic conditions and the difficult access and generally poor quality of the health services, especially at the district health hospitals. The Government seeks to improve the health status of the population and reduce the health gap between the NUP and the rest of the Vietnamese population. The project achieved this by improving the efficiency on delivering health outputs, reducing maternal mortality in higher proportion than the country’s average, and reducing the equity gap in out-of-pocket health spending between the poorest income quintile and the richest income quintile.

38 According the PAD, a quantitative economic analysis based on costs was not feasible for the project because there was no empirical basis for estimating the project’s health outcome costs. Neither an economic rate of return nor a net present value of the benefits of the project was calculated/forecasted upfront during the project appraisal. The PAD highlighted the difficulty to estimate costs of the proposed interventions. Given the special conditions to implement project activities in the NUP areas, the economic costs may differ significantly from their financial costs or budget expenditures. For example, under Component 1, the opportunity cost of the personnel sent for training is a significant economic cost of the project that may not be completely reflected in the project budget. It has been the experience of other projects (for example, the ADB-supported Health Care in the Central Highlands Project) that sending large numbers of personnel for training from facilities that are already understaffed imposed a serious budgetary burden on the public health system. Another example is associated with the reimbursement of the travel and food costs of poor hospital inpatients, which is an income transfer, not an economic cost. Economic costs in this case would be limited to the cost of administering the transfers and the cost of any additional health care utilization that might result from this support.

49

Efficiency on Achieving the Project Outputs

3. The project surpassed all relevant planned output targets increasing efficiency by reducing unitary costs. Annex 3A shows the project’s planned and achieved output targets and the correspondent budget allocations. These outputs, related to training human resources and equipping, repairing, and upgrading infrastructure in the district hospitals, had targets that were surpassed during the project execution with simultaneous reductions in the costs, thus increasing savings that allowed the project to execute additional activities during the extension phase. For example, the implicit unitary cost for the level-1 specialists and the transformation of assistant doctors to medical doctors were, at the end of the project, 50 percent and 37 percent lower than the original plans, while the number of individuals trained as level-1 specialists and medical doctors were 101 percent and 50 percent higher than planned.

Analysis of the Contribution to the Project Outcomes

a) Benefits associated with reducing mortality

4. The analysis of the project interventions and PDO achievements (annexes 2 and 5 of this ICR) evidenced improvements in the access and utilization of health services in the NUP areas. This contributed to the reduction of maternal and child mortality rates, which are two major problems in the burden of diseases of the NUP areas. The project interventions were intended to improve the quality and efficiency in delivering prenatal care, birth delivery, and childcare in the first 12 months of life. So, the reduction of maternal and child mortality rates along the project implementation should be relevant benefits reflecting improvements in the health status of the NUP areas’ population.39

5. Current literature evidences that the project used the right interventions to reduce maternal and child mortality rates. Most maternal and newborn deaths can be prevented using existing, proven, cost-effective interventions, such as clean delivery packages, composed of antibiotics, sterile blades for cutting umbilical cords, drugs that prevent and treat postpartum hemorrhage, resuscitation, immediate and exclusive breastfeeding, and education and communication to the mother to keep the newborn warm with skin-to-skin contact and breastfeeding.40 Increasing access to mothers to deliver their babies at first-level facilities and/or by SBAs was the focus of the project, providing an opportunity to expand quality services around the time of birth. All these initiatives were used by the project interventions, providing a solid ground to start a process to reduce maternal, neonatal, and infant mortality in the northern mountainous provinces in Vietnam.

6. As can be seen, between 2007 and 2014, the maternal mortality reduction was remarkable in the NUP areas when compared with the average reduction verified in the country, but it is still very high in the region compared with the national benchmark and

39 However, given that both—maternal and infant mortality—are affected by many factors, such as water and sanitation conditions, nutrition patterns, and others, it is difficult to attribute the results (for good or for bad) only to the health interventions. 40 Bill and Melinda Gates Foundation. 2016. Maternal, Newborn and Child Health Strategy Overview. http://www.gatesfoundation.org/What-We-Do/Global-Development/Maternal-Newborn-and-Child-Health

50

http://www.gatesfoundation.org/What-We-Do/Global-Development/Maternal-Newborn-and-Child-Health

international standards. During the project regular time implementation, MMR in the NUP areas reduced 40 percent compared with 7 percent in the whole country. However, MMR in the seven NUP provinces is still nearly two times higher than the country average, as seen in table 3.1.

Table 3.1. Maternal, Neonatal, and IMR Vietnam and the NUP: 2007–2014

Years MMR(per 100,000 born alive)

NMR(per 1,000 born alive)

IMR(per 1,000 born alive)

Vietnam NUP Vietnam NUP Vietnam NUP2007 58 178 13.0 11.2 21 31.12014 54 106 12.0 10.8 18 29.4 Percent of reduction

−6.9 −40.4 −7.7 −3.6 −14.3 −5.4

Source: Vietnam: World Bank Data (http://data.worldbank.org/indicator/SP.DYN.IMRT.IN?locations=VN; http://data.worldbank.org/indicator/SH.STA.MMRT?locations=VN; http://data.worldbank.org/indicator/SH.DYN.NMRT?locations=VN). Government of Vietnam: NUP Project Surveys 2009 and 2014.

7. The project opened the floor to continue achieving progress on reducing MMR in the region and the relevance to keep it sustainable along the time. A study on maternal and neonatal mortality financed by the CPMU as part of the project closing evaluation activities found that in 2014, the still high MMR in the NUP areas is related to: (a) the high proportion of home birth deliveries, representing 38 percent of the total birth deliveries in the NUP41; (b) delays in detection and decision to seek care by women and their families; (c) delays in reaching care, mainly due to difficult geographic conditions and lack of transportation means; and (d) delays in receiving health care and appropriate treatment at the health facilities, mainly due to shortcomings in compliance with the process of care and monitoring of pregnant mothers, especially during and after delivery. The reduction registered in the MMR between 2007 and 2014 was not uniform among the seven provinces and in one of them (Bac Kan) the MMR increased during the project implementation (figure 3.1).

Figure 3.1. MMR in Six Northern Upland Provinces

Dien Bien Lai Chau Son La Cao Bang Lao Cai Bac Kan Total0

50

100

150

200

250

300

350

400

450409

215

167143 133

46

178

83

150

104 95128

75106

2007-20082013-2014

Source: Ministry of Health of Vietnam: Survey on Maternal and Neonatal Mortality in the Northern Upland Provinces: 2007-2008 and 2013-2014.

41 According to the study, the risk of maternal mortality in a home delivery is 3.7 times higher than when deliveries are performed at health facilities.

51

http://data.worldbank.org/indicator/SH.DYN.NMRT?locations=VN

http://data.worldbank.org/indicator/SH.STA.MMRT?locations=VN

http://data.worldbank.org/indicator/SP.DYN.IMRT.IN?locations=VN

8. Though there is no information related to MMR in the 2015 report, the project extension phase provided training for midwives to reduce maternal mortality at home, thus providing safer birth delivery. The administrative registration of the NUP health services showed that between 2014 and 2015, the proportion of mothers giving birth at home supported by village midwives increased from 6.4 percent to 16.5 percent and the proportion of clean delivery packages provided to the mothers giving birth at home increased from 34 percent to 43.1 percent in the same period respectively. The continued training of village midwives and policies aimed at maintaining a network of village midwives show that the possibilities to increase the project benefits in the extension phase were very high, given that these solutions are crucial for safe motherhood in the mountainous areas.

9. The reduction of neonatal and infant mortality in the NUP areas was lesser than the country average. Table 3.1 shows that IMR and NMR reduced by 5.4 percent and 3.6 percent in the NUP between 2007 and 2014, compared with 14.3 percent and 7.7 percent in the national average, respectively. These results were also influenced by the high proportion of home birth deliveries in the NUP areas. Neonatal mortality among newborns delivered at home was 14.2 per 1,000 live births, almost twice as much as in health facilities, according to the end-of-project survey.

10. Figures 3.2 and 3.3 show that the NMR and IMR increased in some of the provinces along the project execution period, despite the decreasing average. In fact, in the case of NMR, only Bac Kan and Son La presented decreases in NMRs and IMRs. In the case of IMRs, decreases were verified in Ha Giang, Cao Bang, Son La, and Bac Kan. Given that most of the data is from administrative registrations, maybe the reporting system in some provinces, such as Lai Chau and Dien Bien, had improved sharply during (and because) the project implementation, increasing the confidence in the administrative records of NMR and IMR in 2014, compared with 2008, when the mortality data were not captured well.

52

Figure 3.2. NMR in Six Northern Uplands Provinces

Lai Chau Lao Cai Cao Bang Dien Bien Bac Kan Son La Total0

4

8

12

16

20

11.5

13.8

12.3

9.1

7.4

11.7 11.2

17.1

1412.8

12.1

5.6 5.9

10.8

2007-2008

2013-2014

Source: Ministry of Health of Vietnam: Survey on Maternal and Neonatal Mortality in the Northern Upland Provinces: 2007-2008 and 2013-2014.

Figure 3.3. IMR in Six Northern Uplands Provinces

Ha Giang Cao Bang Dien Bien Lai Chau Son La Lao Cai Bac Kan Total0

10

20

30

40

50

40 40

33 33

28

2321

31.134.2

24.3

34.4

42.6

23.2

29.2

17.9

29.4

2008

2014

Source: Ministry of Health of Vietnam: Health Statistical Yearbooks 2008 and 2014.

11. The disadvantaged situation of the MDGs in the NUP areas, revealed during the project implementation by the improvement of the medical records, lead the Government to focus the project extension on reducing the MMR and IMR, striving to achieve the MDGs on health, by providing clean birth delivery packages, supporting basic tools for newborn care, equipment for neonatal units at the district hospitals, and related training. Despite the outputs related to these tasks being positively recorded (see annex 2), their impact on the reduction of IMR and NMR after the project extension was unknown at the time this ICR was prepared.

53

b) Benefits associated to improved supply and quality of health services

12. The project interventions (health staff training, management improvement, and investment in infrastructure, equipment, and maintenance) increased the availability and quality of the essential health services delivery in the district hospitals benefiting the population in the NUP areas. The number of inpatient and outpatient visits per capita delivered in the district hospitals increased by 290 percent and 270 percent, between 2009 and 2015, respectively, according to the project administrative records.

13. However, more important than the number and coverage of the services delivered is the quality of these services. Table 3.2 shows indicators associated with the benefits provided by the project in improving the quality and capacity to deliver health services to mother and children by district hospitals. The number of health services that district hospitals can according to the national norms increased by 105%. Hospitals that have equipment needed for quality procedures, such as positive airway pressure machines, phototherapy for newborn machines, oxygen breathing systems and newborn resuscitators increased remarkably. High improvements were achieved by following the national quality norms, enhancing diagnosis capacity, and having skilled personnel and equipment to implement appropriate techniques for regular and emergency care services.

Table 3.2. Quality Improvements of the Services Delivered by the District Hospitals in the NUP Areas: Project Baseline (2009), Regular Closing Date (2014), and Extended Closing Date (2015)

Health Services Quality IndicatorProject Baseline

(%)

Project Regular Closing

Date(%)

Project Extension

Closing Date(%)

Quality Improvements at Extension Closing Date

(%)Percentage of health services that district hospital can provide according to the national norms

39 71 80 105

Percentage of medical records with appropriate diagnosis of acute respiratory infections 18 39 62 244

Percentage of medical records with appropriate diagnosis of acute diarrhea 41 69 82 200

Percentage of medical records with appropriate diagnosis of acute poisoning 61 86 89 46

Percentage of district hospitals equipped with skilled staff to implement appropriate pediatric techniques

69 75 85 23

Percentage of district hospitals equipped with skilled staff to implement appropriate obstetric techniques

65 84 88 35

Percentage of district hospitals having capacity for caesarean section. 86 89 92 7

Percentage of district hospitals having capacity for blood transfusion

64 75 83 137

Percentage of district hospitals having breathing 25 75 80 220

54

Health Services Quality IndicatorProject Baseline

(%)

Project Regular Closing

Date(%)

Project Extension

Closing Date(%)

Quality Improvements at Extension Closing Date

(%)machines with continuous positive airway pressurePercentage of district hospitals having light for jaundice phototherapy treatment 21 82 88 219

Percentage of district hospitals having oxygen breathing systems 43 75 78 81

Percentage of district hospitals having newborn resuscitators 46 93 92 100

Source: Center for Environment and Health Studies (2016), Final Evaluation of the Northern Uplands Health Support Project in The Extension Phase, Hanoi, 2016.

c) Benefits associated with increased HI coverage and health services demand to prevent catastrophic expenditures for the poor and ethnic minorities

14. The project increased not only the coverage, but also the demand for health services utilization by providing transportation and meals subsidies to the poor and ethnic minorities. Because of adjustments, the number of project beneficiaries increased significantly. From 2009 to 2011, only the poor were covered by the project, but given that the majority of the ethnic minorities were quasi-poor, they were included as beneficiaries of the subsidies since 2012. The number of project recipients, were 730,183 persons in 2009 (25 percent of the NUP population) when compared with 3,185,341 in 2013 (82 percent of the NUP population). Since 2011, the project implemented a communication campaign (IEC) to increase the beneficiaries’ awareness about HI rights and the processes to achieve their subsidies. The number of beneficiaries knowing at least three HI rights increased substantially.

15. The number of beneficiaries receiving direct monetary subsidies for transportation, meals, and health expenditures, increased several times along the project implementation, as can be seen in figure 3.5.42 They were only 1,000 in 2009 but increased to almost 245,000 in 2013. From the project start in 2009 to January 2014, the number of poor/ethnic minority inpatient who received support to use medical services from the project totaled 545,423, with registered expenditures of USD7,398,693.69 (around USD13.56 per inpatient beneficiary). As of January 2014, 86 percent of the budget for this activity has been spent, but the sustainability of these subsidies was granted during the project extension and hopefully beyond, according the CPMU information.

42 The project’s 2009 Baseline Survey showed that, of the inpatients in the NUP areas who visited the district hospitals in 2008 to assess health care, 96 percent paid for food, 88 percent paid for transportation, and 5 percent paid for other costs, such as medication, exams, tests, and so on. The inpatient average daily cost paid by the families was VND 122,224 (USD 7.46 at the 2008 exchange rate) and the average stance was 4.4 days, representing an average cost of USD 32.80 per inpatient. To finance these costs, 33 percent of the families got loans, 22 percent asked for family support, and 5 percent sold assets or means of production.

55

Figure 3.4. Number of Poor/ethnic Minorities Benefited by Inpatient Heath Subsidies (transportation, meals, health expenditures) by the NUP Project 2009–2013

2009 2010 2011 2012 2013105419390

79469

200709

244801

Number of Beneficiaries

16. The subsidies paid to increase coverage and access to health services were crucial to reduce out-of-pocket health-related spending, prevent catastrophic expenditures, and increase the coverage of HI in the NUP areas. According to the CPMU sponsored surveys, between 2009 and 2014, the percentage of households which experienced catastrophic health care expenditures in the NUP had a remarkable reduction: from 15 percent to 2 percent, respectively. The percentage of population living in these areas with HICs increased from 82 percent to 95 percent between June 2009 to December 2015, mostly among the poor and ethnic minorities populations.

Analysis of the Project Impact on Equity Pro-poor and Ethnic Minorities

(a) Methodological considerations

17. The equity analysis of the project interventions will consider the impact of the project, mostly from 2012 to 2014. During 2009 and 2011, most of the efforts of the NUP Project were concentrated on planning and implementing the process to provide training for health staff, procurement for investments in infrastructure and equipment of the district hospitals, and the institutional arrangements to finance incentives to the poor and ethnic minorities to increase the access of health services and utilization of district hospitals. From 2012 to 2015, the project was able to measure the impact of these investments for improving the equity on accessing the project benefits, especially to the poor and ethnic minorities.

18. This equity analysis will consider the progress on equity indicators related to (a) affiliation to HI; (b) household income spent on health; and (c) household income spent on inpatient visits. It is based on the data collected through the VHLSS of the respective years. Equity impact will be calculated on variables such as gender and age of the beneficiaries, ethnicity, residency status (rural or urban), and income (poorest, near poor, middle, near richest, and richest income quintiles).

19. Main hypotheses. The main hypotheses used in this analysis are the following: (a) the project improved the equity on accessing inpatient visits, disproportionally benefiting women and children over five years; (b) the project reduced the relative spending of the poor and ethnic

56

minorities to access inpatient visits; (c) the project increased the number of inpatient visits of the poor in the district hospitals, increasing the equity of the access to hospital services43; (d) the project increased the proportion of the poor and ethnic minorities having HI; and (e) the project reduced the participation of health spending in the poor families’ out-of-pocket expenses, increasing the equity of the health spending.

(b) Equity impact on inpatient medical visits for women, children, and rural population

20. Given that the project was focused on achieving the health MDGs, it was expected that the proportion of inpatient visits related to mother and children would increase faster than for other groups, increasing the gender and age equity on the access to the district hospitals. The equity impact is measured by the equity ratio.44 As shown in table 3.3, between 2010 and 2014, women inpatient visits increased 32 percent as a proportion of men inpatient visits, and children inpatient visits increased 181 percent as a proportion of total inpatient visits at the district hospitals. Rural population inpatient visits also increased 19 percent as a proportion of urban medical visits during the same period. So equity in accessing inpatient services at district hospitals was improved for women (because women at a reproductive age always need more services than men because of reproductive health issues), for children until they are five years old (where the associated mortality risks are higher), and for the rural population, who were previously underserved by the health system.

Table 3.3. Equity Gap on Annual Per Capita Inpatient Visits at the District Hospitals NUP Provinces: 2010–2014

Annual Per Capita inpatient Visits: Equity Ratios for Different

Population Groups2010 2012 2014

Increase 2010–2012

(%)


(%)

Increase (2010-2014)

(%)Gender Equity Ratio

Women visits/men visits 1.14 1.46 1.50 28 3 32Child Visits Equity Ratio

0–5 years old visits/average visits 1.00 1.13 2.81 13 149 181Place of Residency Equity Ratio

Rural visits/urban visits 0.94 0.78 1.12 −17 44 19Source: VHLSS, 2010, 2012, and 2014.

(c)Equity on the impatient visits to the district hospitals benefiting the poor and near poor

21. The project did not collect data for the near poor, because its focus was the poor and ethnic minorities. However, the information of VHLSS use the classification of poor and near poor in two ways: (a) The global poverty line (the World Bank criteria); and (b) by income

43 Considering the nature of the Project investments, the impact on health services utilization in equity is more sensitive to the inpatient than to outpatient visits to the district hospitals. The Project influenced the Provincial Departments of Health to functionalize the health referral process, according the levels of complexity in the health care provision, inducing the population to do not use district hospitals for unnecessary outpatient services for that level of complexity. So, many outpatient visits that in the past went to district hospitals (especially among the center village populations) were redirect to Community Health Centers. So this analysis is pertinent only for inpatient services.44 The equity ratio for inpatient visits is defined as the proportion of the visits per capita in the category where the number of visits per capita is expected to increase disproportionally as a proportion of the complement (for example, women compared to men) or the total visits per capita (poor compared to the total population). If this ratio has positive variations, the equity impact was achieved, but if the ratio has negative variation, the equity was reduced.

57

quintile, which means that the poorest should be considered the first quintile and the near poorest are the second quintile. Table 3.4 shows the impact of the project in the inpatient visits to the district hospitals by poverty status and income quintiles.

Table 3.4. Inpatient Visits to the District Hospitals According Poverty Status and Income Quintiles

Poverty Status and Income Quintiles

Inpatient Visits to the District Hospitals per inhabitant2012 2014 Increase (2012-2014)

(%)Poverty Status (according World Bank Global Poverty Line)

Poor 0.037 0.054 45.9Near-Poor 0.088 0.070 -20.5Non-Poor 0.056 0.075 33.9

Income QuintilesPoorest 0.028 0.044 57.1Near-Poorest 0.034 0.061 79.4Middle 0.048 0.063 31.3Near-Richest 0.083 0.093 12.0Richest 0.047 0.053 12.8Average 0.048 0.063 31.2

Source: VHLSS, 2012 and 2014.

22. Table 3.4 shows that the inpatient visits in the district hospitals, between 2012 and 2014, had a higher increase for the poor than for the average population and the non-poor . This is a strong evidence of the positive impact of the Project in the equity of the health services utilization at the district hospitals level. The inpatient visits per inhabitant increased in average 31% compared with 57% in the poorest quintile and 46% among the poor. Regarding the near poor, the income quintile approach shows an increase of 79% in the near poorest quintile. However, using the global poverty line approach, the data shows a reduction of 20% of the number of inpatient visits between 2012 and 2014, which could be associated with methodological differences in the way that the information associated with each year was captured45.

(d) Equity on the relative spending of the poor and ethnic minorities to access inpatient visits

23. Another way to verify the equity impact is the proportion of the out-of-pocket expenditures of mothers and children and the poor and ethnic minorities on inpatient visits to the district hospitals compared to other groups, along the project implementation. The VHLSS data shows that the spending per inpatient visit at the district hospitals as a share of project target groups was relatively reduced. As can be seen in table 3.5, the equity ratio for the inpatient visit spending improved for women compared to men; for children compared to the average population; for ethnic minorities compared to the Kin/Hoa ethnicity; for rural populations compared to urban, and especially for the poorest quintile compared with the average population. The relative reduction of the out-of-pocket spending with inpatient visits verified for the poor was supported by the project subsidies for meals, transportation, and medical expenses, the last covered by the HI, during the project implementation.

45 The data for 2012 used the international poverty line (in purchasing parity power –PPP) of and income of USD 1.25 a day. However, this line was upgraded to US$ 1.90 in the 2014 data analysis, creating difficulties to compare both years using the World Bank global poverty line criteria.

58

Table 3.5. Equity Gap in Out-of-Pocket Spending for Inpatient Visits to District Hospitals NUP Provinces: 2010–2014

Out-of-Pocket Spending per Inpatient Visit: Equity Ratios for

Different Population Groups2010 2012 2014


(%)


(%)

Increase (2010–2014)

(%)Gender Equity Ratio

Women/men spending 1.01 0.89 0.91 −12 2 −10Child Visits Equity Ratio

0-5 years old/average spending 0.72 0.79 0.53 10 -33 −26Ethnic Minorities Equity Ratio

Ethnic minorities/Kin-Hoa spending 1.19 0.56 1.13 −53 102 −5Place of Residency Equity Ratio

Rural/urban spending 1.34 1.89 1.11 41 −41 −17Income Equity Ratio

Poorest quintile/average spending 1.04 0.69 0.60 −34 −13 −42Source: VHLSS, 2010, 2012, and 2014.

(e) Equity impact of the project in increasing HI coverage

24. The VHLSS data also shows relevant impacts in the coverage of HI during the project implementation time, especially for the poor. The total NUP population without HI, decreased slightly between 2010 and 2014 (from 7.4 percent to 7.2 percent, respectively) and the HI coverage for the poor improved. The percentage of the poorest economic quintile affiliated to the HI increased from 98.6 percent to 99.2 percent between 2010 and 2014 and for the near-poor (second poorest quintile) the HI coverage increased from 93.5 percent to 98.9 percent, according the VHLSS data. As part of the affiliation to the HI mechanisms for the poor, the project spent USD 8.5 million in subsidies for the poor and ethnic minorities, contributing to pay for transportation and meals for 596,700 poor and ethnic minorities’ inhabitants of the NUP areas, with an average expenditure of USD 14.27 per medical visit.

(f) Equity impact on reducing the share of out-of-pocket expense in health for the poor and ethnic minorities

25. During project implementation, the poorest quintile reduced the health spending as a share of the out-of-pocket expense in the NUP areas. From 2010 to 2014, the share of out-of-pocket health spending of the poorest quintile was reduced slightly from 6.7 percent to 6.5 percent, while the average out-of-pocket family spending on health increased from 7.9 percent to 8.3 percent in the NUP areas. Probably health subsidies to the poor and ethnic minorities for visiting district hospitals and improvements in the health care assistance at the villages, sustained partially by the project, had positively affected the family budgets, contributing to avoiding the risk of catastrophic health expenditures for these populations. That is one of the reasons why the proportion of the NUP families with catastrophic health expenditures reduced from 10.4 percent to 2.0 percent during the project execution, as is demonstrated by the project KPI #3.

Final Considerations

26. The present economic analysis intends to demonstrate the following: (a) the project was efficient in delivering its outputs, reducing unitary costs for training, equipment installation, and civil works during implementation, compared with the original implicit costs. This allowed, savings from the original implementation time to be used during the project extension in new

59

activities related to improving the MDGs in the project area and (b) efficient project interventions and subsidies to the poor and ethnic minorities contributed to reducing maternal, neonatal, and infant mortality and improved the equity in assessing health care and reducing health spending for the poor and ethnic minorities.

60

Annex 3A. Estimated Unitary Costs of the Project Outputs According to the Original Plan (2009) and Closing Implementation (2016)

Component and OutputProject Outputs Budget (USD, thousands) Estimated Unitary Cost (USD)

Planned Implemented Variation (%) Planned Implemented Variation

(%) Planned Implemented Variation (%)

Trained Level-1 specialists 183 367 101 691.7 687.8 −1 3,779.78 1,874.11 −50Assistant doctors trained as doctors*1 741 1,156 56 6,236.0 6,085.6 −2 8,415.65 5,264.36 −37

Short-term-doctors trained in curative care techniques*2 1,334 4,965 272 2,409.3 1842.3 −24 1,806.07 371.00 −79

Short-term-doctors trained in preventive care 213 336 58 211.0 200.1 −5 990.61 595.53 −40

Skills techniques transfer to district hospitals*3 402 670 67 365.6 296.9 −19 909.45 443.13 −51

Basic medical equipment (number of units installed) 4415 5165 17 26,698.5 25,648.5 −4 6,047.23 4,965.83 −18

Minor repairs and upgrade of district hospitals (number) 10 18 80 3,465.9 3,465.9 — 346,590.00 192,550.00 −44

Training of district hospital management staff 303 753 248 729.5 493.4 −32 2,407.59 655.25 −73

Note: *1 Includes 98 trained pharmaceutical assistants transformed to pharmacists which was not planned at the beginning of the project.*2 Includes 1,542 health staffs completing short-term training on MDG-related activities during the extension phase.*3 Times of skill transfers. The budget includes expenses to support mobilized staff.

61

Annex 4. Bank Lending and Implementation Support/Supervision Processes

(a) Task Team members

Names Title UnitLendingBukhuti Shengelia Task Team Leader (left the World Bank)Mai Thi Nguyen Team Member GED02Nga Quynh Nguyen Senior Program Assistant EAPDEMaryam Salim Team Leader MDI Hoi Chan Nguyen Country Counselor (retired)Samuel S. Lieberman Task Team Leader (retired)

Kelichi Ohiri Health Specialist/Team Member (already left the World Bank

Lingzhi Xu Senior Operation Officer GHN03Marko Vujicic Economist (left the World Bank)Lan Thi Thu Nguyen Safeguards Specialist GEN2BHung Viet Le Financial Management Specialist EAPCOSupervision/ICRMai Thi Nguyen Team Member GED02Kari L. Hurt Team Leader GHN06Anh Thuy Nguyen Team Leader GHN02Bukhuti Shengelia Task Team Leader (left the World Bank)Andre C. Medici ICR Author GHN04Hoang Xuan Nguyen Procurement Specialist GGO08Mai Thi Phuong Tran Senior Financial Management GGO20Sang Minh Le Environment Safeguards Specialist GHN02Giang Tam Nguyen Social Safeguards Specialist GSU02Nga Thi Anh Hoang Program Assistant EACVFNghi Quy Nguyen Social Development Specialist GSU02Trang Phuong Thi Nguyen Safeguards Specialist EASVSMaryam Salim Team Leader MDI Nguyen Hoang Nguyen Procurement Specialist GGODRQuynh Xuan Thi Phan Financial Management Specialist GEFPOMaya Razat Program Assistant GSPMinh Thi Hoang Trinh Program Assistant AFCNGNga Quynh Nguyen Senior Program Assistant EAPDEDuong Minh Duc Public Health Consultant

62

(b) Staff Time and Cost

Stage of Project CycleStaff Time and Cost (Bank Budget Only)

No. of Staff Weeks US$, thousands (including travel and consultant costs)

Lending FY06 19.89 96.00 FY07 31.35 246.50 FY08 34.80 145.50

Total: 86.04 488.00Supervision/ICR

FY09 24.50 83.70FY10 26.50 96.80FY11 17.00 75.00FY12 14.00 53.50FY13 22.00 70.20FY14 17.30 49.50FY15 14.40 42.00FY16 26.80 100.50

Total: 162.50 571.00

63

Annex 5. Results Framework - Analysis of the PDO Achievement

Introduction

1. The PDO has three parts: (a) PDO 1 - increase the utilization of district hospital services; (ii) PDO 2 - improve the quality of district-level hospitals; and (iii) PDO 3 - reduce financial constraints to access health services. The objective of this annex is to attribute the KPIs and IOIs to the three parts of the PDO and to rate the PDO (and its parts) according of the achievement of the corresponding indicators.46

2. The methodology to define the indicators’ ratings is the following: For quantitative indicators, the achievement at the end of the project is compared with its end target. If the result is above 105 percent the indicator was surpassed; if it is between 95 percent and 104 percent, it was achieved. If it is between 85 percent and 94 percent, it was substantially achieved. If it ranges between 65 percent and 84 percent, it was partially achieved, and if it is lower than 65 percent, it was not achieved. For qualitative indicators, the classification is only achieved (if the qualitative target was accomplished) and not achieved (if it was not accomplished). The IOI #7, with no reliable information about baselines and targets of values of achievement will not be considered as part of the PDO rating.

3. The criteria used to calculate the indicator achievement is the following: (a) if the baseline is not zero, it is calculated on the difference between what was intended (baseline) and actual47 and divided by the difference between the target and the baseline; (b) if the baseline is zero it is calculated on the coefficient between the actual and the target; and (c) if the target is lower than the baseline, it is calculated on the coefficient between the actual and the baseline.

4. Annex 5A presents a table calculating the rating of each indicator according to achievements recorded during project implementation. This table has the following columns: (a) original indicators (according to the PAD); (b) indicators added during project implementation; (c) value and date of the indicator baseline; (d) value and date of the indicator target; (e) value and date of the indicator at the project’s original closing date of August 31, 2014; (f) percentage of target achieved at the project’s original closing date; (g) value of the indicator at the project’s revised closing date of February 29, 2016; (h) percentage of target achieved at the project’s revised closing date; and (i) indicator achievement rate according to the methodology presented in paragraph 2.

5. The rating of the PDO is attributed to the proportion of the indicators’ values that have been surpassed, achieved, or substantially achieved as a share of the total project indicators. It is high, when more than 95 percent of the indicators’ target values have been surpassed, achieved, or substantially achieved; substantial, from 75 percent to 94 percent of achievement; modest, from 50 percent to 74 percent, and negligible when less than 50 percent of the indicators have met their target values.

46 PDOs and IOIs will have the same weight to classify the PDO’s achievement.47 A= (Ia-Ib)/(It-Ib), where A is achievement, Ia is indicator’s actual; Ib is indicator’s baseline, and It is indicator’s target.

64

6. This annex has three tables. Table 5.1 shows the KPIs and IOIs distributed according to the three parts of the PDO. Table 5.2 summarizes the results found in annex 5A, and table 5.3 summarizes the PDOs’ rating to measure the project efficacy.

Project KPIs and IOIs related to the PDO

7. Number of indicators. According to the PAD, the project had 4 KPIs and 10 IOIs. All KPIs and IOIs #1 to #7 were related to the PDO. The other IOIs (#8 to #10) were designed to measure project management performance. Some IOIs were complex to be measured through just one indicator. Then, during the project implementation, the Government and the World Bank agreed on creating subindicators to measure these complex and multidimensional IOIs. Accordingly, IOIs #2, #3, and #4 were measured by nine, three, and two subindicators, respectively. Additionally, other IOIs were included during the project execution without a formal project restructuring. Other administrative indicators were also included using similar processes. The last project ISR,48 issued in February 2016, lists a total of 25 KPIs (4) and IOIs (21). Table 5.1 shows the PDO parts 1, 2, and 3 and the corresponding KPIs and IOIs. PDO part 1 was measured by 2 indicators, PDO part 2 by 19 indicators and subindicators, and PDO part 3 by 4 indicators.

Table 5.1. Distribution of the Project Indicators (KPIs and IOIs) according to the PDO parts.

PDO’S PARTS KPI (*) IOI * IOI Sub Indicators **

PDO #1: Increase utilization of district health services.(2 indicators)

KPI #1: Increase utilization rates of inpatient services in district hospitals among Decision 139 beneficiariesKPI #2: Increase utilization rates of outpatient services in district hospitals among Decision 139 beneficiaries

PDO #2: Improve the quality of district-level hospitals(19 indicators)

KPI #4: Proportion of district hospitals that provide full set of health services according to the national norms (Decision 23/205/QB- BYT) adjusted to the Northern Uplands

IOI #1: Percentage of patients satisfied with the health services IOI #2: Adherence of treatment protocols for selected conditions in inpatient settings (based on six subindicators)

IOI #2.1: Percentage of health workers with knowledge of diagnosing and treating Level A/B/C dehydrated diarrheaIOI #2.2: Percentage of health workers with knowledge of diagnosing and treating severe pneumoniaIOI #2.3: Percentage of health workers with knowledge of diagnosing and treating poisoningIOI #2.4: Percentage of reasonable diagnoses of severe pneumoniaIOI #2.5: Percentage of clinical health workers’ reasonable diagnosis of general pneumoniaIOI #2.6: Percentage of reasonable diagnoses of Level A dehydrated diarrhea

48 The project documentation does not reflect when all these indicators and the corresponding baselines were set and included. They are not in the PAD and start to appear only in ISR #6 (issued in August 2013). From 2008 (project starting) to 2013, the five project ISRs do not have clear information on the project M&E and RF.

65

PDO’S PARTS KPI (*) IOI * IOI Sub Indicators **

IOI #2.7: Percentage of reasonable diagnoses of Level B dehydrated diarrheaIOI #2.8: Percentage of reasonable diagnosis of Level C dehydrated diarrheaIOI #2.9: Percentage of reasonable diagnoses of poisoning

IOI #3: Percentage of eligible district health staffs who have successfully completed training provided by the project (based on three subindicators).

IOI #3.1: Percentage of doctors and assistant doctors at district hospitals trained by the projectIOI #3.2: Percentage of health staffs with completed short-term training courses compared to the planIOI # 3.3: Percentage of health staffs completed long-term training courses compared to the plan

IOI #4: Percentage of eligible district hospitals with acceptable operation and maintenance plans and budgets for facility and equipment maintenance (based on two subindicators).

IOI #4.1: Percentage of district hospitals having schedule and budget for maintenance of infrastructure

IOI #4.2: Percentage of district hospitals having schedule and budget for maintenance of equipment

IOI #5: Number of health facilities constructed, renovated, and/or equippedIOI #6: Percentage of recently discharged patients satisfied with health servicesIOI #7: Number of people with access to a basic package of health, nutrition, and reproductive health services

PDO #3: Reduce financial constraints to access health services.(4 indicators)

KPI #3: Percentage of households who experienced catastrophic health care expenditures in the year prior to the survey

IOI #8: Percentage of Decision 139 beneficiaries who have received HICsIOI #9: Percentage of 139 beneficiaries with cards who can correctly identify at least three benefits covered under the HCFP programIOI #10: Percentage of households who identify financial barriers as a main cause for not seeking health care

Note: * According to page 34–36 of the PAD.** According to project ISRs Sequence #6 to #11 (last).

8. Indicators’ baseline. Despite the fact that annex 3 of the PAD presents the project RF with the project indicators (KPIs and IOIs), the RF was incomplete and most of the baselines and targets were revised in 2008 (VHLSS) and 2009 (Baseline Survey). Different dates for project baselines could be found. Some indicators incorporated during project implementation do not have baselines. Some indicators should report progress twice during the project life (Year 3 and Year 6). This is the case for KPIs #1, #2, and #3 and IOIs # 3, #5, and #6. The survey developed for the MTR captured the results of these indicators at Year 3 of project implementation. All

66

other indicators had to be measured at project end, according to the PAD, but some were followed by the CPMU using the project administrative registries.

9. Results of the PDO’s achievement evaluation: Substantial. Project efficacy could be considered substantial according to the ratings obtained in each one of the parts of the methodology. Efficacy of PDO 1 is High, given that all KPIs surpassed their target values. Efficacy of PDO 2 is considered High, given that the percentage of indicators surpassed or achieved the targets is 100 percent. Efficacy of PDO 3 is considered Substantial, given that 75 percent of the indicators surpassed their target values.

Table 5.2. Summary Table of Indicator’s Achievement

Rating Categories KPI’s IOIs TotalPDO Part 1 - Increase utilization of district health services - High

Surpassed (>105%) 2 — 2Achieved (95%–105%) — — —Substantially achieved (85%–104%) — — —Partially achieved (65%–84%) — — —Not achieved (<65%) — — —Not considered — — —Total PDO Part 1 2 — 2

PDO Part 2 - Improve the quality of district-level hospitals - HighSurpassed (>105%) 1 13 14Achieved (95%–105%) — 4 4Substantially achieved (85%–104%) — — —Partially achieved (65%–84%) — — —Not achieved (<65%) — — —Not considered — — —Total PDO Part 2 1 17 18

PDO Part 3 - Reduce financial constraints to access to health services - SubstantialSurpassed (>105%) 1 2 3Achieved (95%–105%) — — —Substantially achieved (85%–104%) — — —Partially achieved (65%–84%) — — —Not achieved (<65%) — 1 1Not considered — — —Total PDO Part 3 1 3 4Grand Total 4 20 24

Table 5.3. Summarized Rating for Project Efficacy

PDOs Parts Efficacy Rates Based on the Achievement of the IndicatorsPDO Part 1 HighPDO Part 2 HighPDO Part 3 ModestOverall Rating Substantial

67

Annex 5A. Status of the Indicators According to Achievement

Original Indicators (According to the

PAD)

Indicators Added during Project

Implementation

Value and Date of the Indicator Baseline

Value and Date of the Indicator

Target

Value and Date of the Indicator at

Project Original

Closing Date(Aug 31, 2014)

Ratio of Achievement

at Project Original

Closing Date

Value of the Indicator at the End of

Project Extension

Period(February 29, 2016)

Ratio of Achievement at the End of

theProject

ExtensionPeriod (*)

Rating of the

Indicator

Key Performance Indicators (KPIs)KPI #1: Increase utilization rates of inpatient services in district hospitals among Decision 139 beneficiaries

— 0.027(June 2009)

0.033(August 2014)

0.081(December

2013)9.00

0.096(December

2015)11.50 Surpassed

KPI #2: Increase utilization rates of outpatient health services in district hospitals among Decision 139 beneficiaries

—0.067

(December 2009)

0.075(August 2014)

0.082(December

2013)1.87

0.247(December

2015)22.50 Surpassed

KPI #3: Percentage of households who experienced catastrophic healthcare expenditures in the year prior to the survey

— 14.27%(June 2008)

13.23%(August

2014

2.0%(August 2014) 11.80 Not measured — Surpassed

KPI #4: Proportion of district hospitals that provide full set of health services according to the national norms (Decision 23/205/QB- BYT) adjusted to the Northern Uplands

— 39.1%(June 2008)

70%(August 2014)

71.4%(October 2014) 1.05

80.4%(December

2015)1.34 Surpassed

Intermediate Outcome Indicators (IOIs)IOI #1: Percentage of patients satisfied with the health services

— 8.5%(July 2009)

10.2%(August 2014)

84.4%(October 2014) 44.71 Not measured — Surpassed

68


PAD)


Implementation



Target


Project Original



at Project Original

Closing Date


Project Extension



theProject

ExtensionPeriod (*)

Rating of the

Indicator

IOI #2: Adherence of treatment protocols for selected conditions in inpatient settings (based on six subindicators)

IOI #2.1: Percentage of health workers with knowledge of diagnosing and treating Level A/B/C dehydrated diarrhea

9.7(June 2009)

14.0%(August 2014)


IOI #2.2: Percentage of health workers with knowledge of diagnosing and treating severe pneumonia

13.2%(June 2009)

18.5%(August 2014)


IOI #2.3: Percentage of health workers with knowledge of diagnosing and treating poisoning

26.8%(June 2012)

37.5%(August 2014)

83.0(October 2014) 5.25 Not measured — Surpassed

IOI #2.4: Percentage of reasonable diagnoses of severe pneumonia

45.5%(June 2009)

63.7%(August 2014)


IOI #2.5: Percentage of clinical health workers’ reasonable diagnosis of general pneumonia

19.60%(June 2009)

27.44%(August 2014)


IOI #2.6: Percentage of reasonable diagnoses of Level A dehydrated diarrhea

37.2%(June 2009)

52.1%(August 2014)


IOI #2.7: Percentage of reasonable diagnoses of Level B dehydrated diarrhea

48.9%(June 2009)

68.5%(August 2014)


IOI #2.8: Percentage of reasonable diagnosis of Level C dehydrated

41.2%(June 2009)

57.7%(August 2014)

80.0%(October 2014) 2.35% Not measured — Surpassed

69


PAD)


Implementation



Target


Project Original



at Project Original

Closing Date


Project Extension



theProject

ExtensionPeriod (*)

Rating of the

Indicator

diarrheaIOI #2.9: Percentage of reasonable diagnoses of poisoning

61,1%(June 2009)

85.5%(August 2014)

86.2%(October 2014) 1.03 Not measured — Achieved

IOI #3: Percentage of eligible district health staff who have successfully completed training provided by the project

IOI #3.1: Percentage of doctors and assistant doctors and pharmacists at district hospitals trained by the project

0(June 2008)

80(August 2014)

102(December

2013) 1.27189

(December 2015)

2.36 Surpassed

IOI #3.2: Percentage of health staffs completed short-term training courses compared to the plan *1

0(June 2008)

80(August 2014)

357(June 2014) 4.46 — — Surpassed

IOI # 3.3: Percentage of health staffs completed long-term training courses compared to the plan *1

0(June 2008)

80(August 2014)

88(June 2014) 1.10 — — Surpassed

IOI #4: Percentage of eligible district hospitals with acceptable operations and maintenance plans and budget for facility and equipment maintenance

IOI #4.1: Percentage of district hospitals having schedule and budget for maintenance of infrastructure *2

49.2(June 2009)

40.0(August 2014)

99.3(October 2014) 2.02

79.7(December

2015)1.62 Achieved

IOI #4.2: Percentage of district hospitals having schedule and budget for maintenance of equipment. *2

77.1(June 2009)

40(August 2014)

99.2(October 2014) 1.29

89.1(December

2015)1.16 Achieved

IOI #5: Number of health facilities constructed, renovated, and/or equipped

0(June 2008)

61(August 2014)

63( August 2014) 1.03

64(December

2015)1.05 Achieved

IOI #6: Percentage of 8.5 10.2 84.4 44.65 — — Surpassed

70


PAD)


Implementation



Target


Project Original



at Project Original

Closing Date


Project Extension



theProject

ExtensionPeriod (*)

Rating of the

Indicator

recently discharged patients satisfied with health services

(June 2009) (August 2014) (August 2014)

IOI #7: Number of people with access to a basic package of health, nutrition, and reproductive health services *4

20%(June 2009)

70%(August 2014)

244,801(August 2014) —

270,274(December

2015)— Not

considered

IOI #8: Percentage of Decision 139 beneficiaries who have received HICs *5.

— 82.1 (June 2009)

70(June 2013)

94.3(June 2012) 1.15

95.2(December

2015)1.16 Surpassed

IOI #9: Percentage of 139 beneficiaries with cards who can correctly identify at least three benefits covered under the HCFP program. *6

— 14.8(June 2009)

75.0(August 2014)

57.4(August 2014)

0.57 — — .Not Achieved

IOI #10: Percentage of households who identify financial barriers as a main cause for not seeking health care.

— 2.0(June 2009)

1.8(August 2014)

1.2(August 2014) 4.03 — — Surpassed

Note: *1 This indicator was not listed in the PAD. The information was obtained from the Center for Environment and Health Studies (2014) Final Report: End-line evaluation of the Northern Uplands Health Support Project, Hanoi, 2014, and Center for Environment and Health Studies (2016) Final Evaluation of the Northern Uplands Health Support Project in the Extension Phase, Hanoi, 2016.*2 The target of this indicator was established before the baseline survey. For this reason, the baseline value in the PAD was higher than the target value. However, the project did not revise the target during project implementation. For this reason, the achievement for this indicator had been calculated over the baseline instead of the target. Even using these criteria, the actual value for this indicator was twice the baseline value by August 2014, but was only 62 percent over the baseline by the end of the project extension period, which could bring some doubts on the sustainability of infrastructure maintenance. Despite this issue, the ICR team considered the target achieved.*3 According to the PAD, this indicator’s provisory baseline was 20.14 percent in 2014. An actualization of this indicator was expected to update the baseline after start the project implementation.*4 This indicator cannot be measured because the baseline and the target were not converted in the number of beneficiaries with access to the basic package of health, nutrition, and reproductive health services during the project life.

71

*5 The target of this indicator was established before the baseline survey and was based according to the preparation team best guess estimate. However, the project did not revise the target during project implementation. For this reason, the achievement for this indicator had been calculated against the baseline instead of the target. Despite this issue, the ICR team considered the target achieved.*6 The Government additionally followed the indicator ‘Percentage of Decision 139 Beneficiaries with cards who can correctly identify at least three benefits covered under the HCFP program’. The achievement of this indicator was 95 percent in December 2015.

72

Annex 6: Summary of Borrower's ICR

Northern Uplands Health Support Project

1. Context

1. The World Bank supported the Vietnamese Government with a loan to implement the Northern Uplands Health Support Project (hereafter called the NUP). The project was implemented in eight years (2008–2015) in the seven northern provinces of Cao Bang, Bac Kan, Lao Cai, Ha Giang, Son La, Dien Bien, and Lai Chau.

2. The general objective of the project was to improve the health status of the poor ethnical minorities in these provinces, which required strengthening the capacity of the health care system, providing better quality health services, and improving access. The legal background of the project is composed of:

(a) the general policies and decisions of the party and the state during the period 2001–201049 to promote health care and protect people’s health;

(b) the master plan to develop Vietnam’s health care system up to 2010 and vision to 2020;50 and

(c) the creation of the HCFP in 2002 (known as Decision 139) to increase access to health care and reduce the financial burden of health expenditure faced by the poor and ethnic minorities.

3. To achieve this general objective, the Project supported (a) the upgrade of district hospitals in these seven provinces, by training health staff, developing human resources for health care, renovating these hospitals, providing medical equipment to achieve better health care services, creating mechanisms and skills to repair infrastructure and medical devices; and (b) the increase of health care services’ access for the poor and the ethnical minorities by providing economic subsidies for transportation and meals and ensuring equity in protecting, caring, and improving people’s health.

4. The seven NUP provinces constitute the most disadvantaged regions in Vietnam, with high concentrations of poor and ethnic minorities living in sparsely populated, mountainous localities, under difficult circumstances. As a result of these unfavorable socioeconomic conditions, and despite somewhat higher public health expenditure per capita than in other regions, health services in these provinces are difficult to access and generally poor in quality, while the health status of the population is significantly worse than the rest of the Vietnamese population.

49 Declared at Decision No. 35/2001/QD/TTg (dated 03/19/2001) and the Resolution No. 46-NQ/TW (dated 02/19/2005).50 Decision No. 153/2006/QD-TTg of the prime minister.

73

5. The health system in these mountainous provinces was weak with regard to the quantity of qualified human resources, infrastructure, medical equipment, and health financing. The investments provided by the project for these provinces significantly improved the health supply and the health status of local people, contributing toward the achievement of the MDG targets in health, in accordance with the party and the state policies. It also contributed to developing Vietnam’s health sector toward fairness and efficiency.

6. After seven years of project implementation, the health indicators of the Northern Upland Provinces have improved significantly. The inpatient services’ utilization of the poor ethnical minorities (beneficiaries under Decision 139) increased almost four times between 2009 and 2015. The outpatient services’ utilization rates increased two times in the same period and the proportion of households with catastrophic health expenditures reduced from 10 percent to 2 percent, between 2009 and 2014.

7. Health human resources (weak and inadequate in the past) have been strengthened. All health staff in district hospitals (111,800 workers) were trained. Health facilities were repaired and upgraded in parallel with investments in infrastructure. Equipment provided to district hospitals have been upgraded and used effectively.

8. The project design draws on best practice examples and lessons learned from other international and Vietnam health investment projects. The project supported interventions on both the ‘supply’ and ‘demand’ sides. On the supply side, the project supported district hospitals by providing training and developing health human resources and repairing and upgrading district hospitals’ infrastructure and equipment. On the demand side, the contribution was facilitated, by providing economic subsidies to the poor, access to good quality services, and thereby, increasing the probability of success and efficiency of investments to improve the population’s health.

9. The project design is well articulated with other donor-supported activities in the region and therefore does not cause any duplication. It only complements other ongoing initiatives. There are some projects also implemented in the NUP, such as Project 225, financed by the Government, to upgrade the provincial district hospitals. Between 2005 and 2007, Project 225 has invested around US$10 million (VND 169 billion), which only met 20 percent of the total health investment needs for the district level in these seven provinces. HEMA (sponsored by the European Commission) supported health care investments for three of the seven NUP provinces (Dien Bien, Son La, and Lai Chau). The HEMA Project was focused on the community health services. The Global Alliance for Vaccination and Immunization Project (2007–2010) supported training for village health workers in 10 provinces, including 4 of the Northern Upland provinces (Ha Giang, Cao Bang, Bac Kan, and Dien Bien). The ADB financed a project for development of provincial preventive medicine system (by providing equipment and training for laboratories). This project was also financed by several bilateral donors and nongovernmental organizations with limited funds.

10. The PDO risks were well managed and limited at the lowest level with appropriate measures from the project. To avoid the abuse of inpatient services (admitted with mild

74

cases) at district hospitals in the project areas, the project provided trainings for the officers on topics such as management, operation of the fund for the poor, and verifying the admission cases. The audits (in collaboration with HI officers) of using funds for the poor were regularly conducted, to ensure that the admission cases were appropriate, inpatient cases received treatments, and inpatient’s medical records were maintained. The monitoring activities, which focus on support for the poor, from the CPMU and the PPMUs to the hospitals had been made periodically. The results of the supervisions and audits showed that management of the HCFP was done in accordance with the regulations of the Government and the project.

11. The risk of the non-poor group also benefiting from the interventions of the project, leading to rising inequality, was well managed by different interventions. The project developed a good management system to control these risks by coordinating well with Vietnam Social Insurance in the provinces, for review and synthesis of the project beneficiaries annually. The list of the poor and ethnic minorities who were qualified for support from the NUP was provided to district hospitals. To receive the NUP support, the patients should present the insurance card with the code ‘HN’ or the certification of poor from the communal people’s committee

12. Many officers/staff were sent for trainings causing short-term shortage of health workers and affecting the availability and quality of services. However, various activities were implemented to alleviate these difficulties. The implementation of training courses was spread throughout the project duration. District hospitals had plans for replacing the staff who attended trainings. The leaders, physicians, and assistant doctors working at ancillary departments (management board, departments of planning, financial, or infection control) at the hospital also shared the responsibility of treating and caring for patients. Medical staffs are required to work in night shifts more frequently. The project allowed district hospitals to sign contracts/hire retired medical doctors or temporarily transfer staff among hospitals to help each other when their staff were attending the training. The project risk management is described in annex 6A.

2. Achievement of the PDOs

13. The project has successfully achieved its overall DO. The utilization of district health services (by the poor and vulnerable population defined accordingly with Decision 139) has increased sharply after IEC campaigns, improving the effectiveness of the Government’s priority to UHC, both by improving the geographical accessibility of quality basic health services at district hospitals and by reducing the financial burden in accessing health services for the poor and ethnical minorities.

14. The project has been successful in implementing the health care policies for the poor and increasing their access to quality health services. The number of the poor receiving financial assistance (meals and travel costs) to visit district hospitals from the NUP has been increased yearly, along with a significant increase in the rate of using district health services among the general population, especially among vulnerable groups such as the poor and ethnic minorities. The average number of inpatient and outpatient visits per capita per year of

75

‘HN’ (Ho Ngheo or Poor Household in English) in district hospitals, increased 390 percent (from 0.0247 to 0.096) and 369 percent (from 0.067 to 0.247) from 2009 to the end of 2014, respectively, exceeding the project’s expectations. The awareness and confidence of the population vulnerable to health care services has been improved. The percentage of households who experienced catastrophic health care expenditures in the year before the survey reduced 81 percent in comparison to the baseline. Detailed data is presented in annex 6B.

15. The activities related to the project components51 have remarkably improved the supply and quality of the health services offered by district hospitals in the seven provinces. The percentage of health techniques (based on the national list of techniques/protocols defined by the MoH) in the seven provinces’ district hospitals increased from 39.1 percent at the baseline (2009) to 80.4 percent at the end of 2015. The percentage of patients who were satisfied with health care services increased from 8.5 percent at the baseline to 84.4 percent at the end line. Around 90 percent of patients were satisfied with the qualification of medical staff, facility infrastructure, medical equipment, and drugs used for treatment (annex B).

16. District hospitals could provide more complex clinical techniques, especially emergency surgeries and various endoscopic surgeries. The capacity of obstetric and newborn care in district hospitals has improved significantly, with more than 75 percent of the district hospitals providing cesarean section and blood transfusion services. The strengthened hospital capacity and improvement of staff quality, has as a consequence, reduced the transferring of patients to a higher-level hospital, and average length of inpatient stay by a half of day, in comparison to the baseline (6.6 days down to 6.1 days).

17. At the end of the project, the number of health staffs with higher qualifications (general medical doctors and medical doctors-level-1 specialists) was doubled. The knowledge and skills of health workers to manage common health episodes were significantly raised (annexes 6B and 6C). Besides district hospitals’ infrastructure and equipment were upgraded by the project, thereby improving district hospitals’ efficiency by absorbing a higher number of medium to complex cases and by reducing referral to provincial hospitals. These positive effects in the district hospitals’ performance are widely recognized and have long-term effects, increasing the sustainability of the provincial health systems. However, the hospitals still lack specialized doctors (in surgery, trauma, and in specialized departments such as eyes and dental) which requires appropriate measures to attract qualified human resources to fulfill this need at district hospitals.

3. Analysis of the Results Framework (baselines, targets, results)

18. There were no major changes in the Project Development Indicators (PDI’s) and Intermediate Indicators (II’s) with regard to concept or calculation formula during the project execution. The only exception is the indicator of proportion of district hospitals that provide

51 Such as strengthening capacity in provision and quality of health care service at district level; comprehensive interventions of the NUP in supporting human resources; investment in upgrading infrastructure; procurement, purchasing, and installment of medical equipment; and improvements in hospital management have remarkably improved the health services (quality and quantity, diversify the types of services) in the seven provinces.

76

full set of health services according to the national norms (Decision 23/205/QD-BYT), which was adjusted for better measuring the improvements of the district hospitals’ capacity. Some key maternal and child health indicators were presented to reflect the impacts toward the achievement of MDGs targets at the project provinces.

19. The evaluation results showed that 9 out of 10 KPIs were achieved and even exceeded several times the objectives set in the project document. One indicator where the goal was not achieved is the percentage of ‘HN’ beneficiaries knowing at least three beneficiaries’ rights guaranteed by the HI system. This indicator reached 57.4 percent of the expected target at the end of the project. The results also indicated that the trainings and equipment support improved the district hospitals’ capacity of providing obstetric/pediatric emergency services, contributing to improving the MDG performance in the region. In the extension period, in collaboration with Mother and Child Health centers, PPMUs and CPMU focused on the trainings that aimed at improving the capacity for newborn care, maternal health care, traditional birth attendants, and providing clean delivery kits for the traditional birth attendants. The results of the survey in 2015 showed that the maternal mortality ratio, adjusted in seven provinces, is 98 per 100,000 live births, significantly reduced in comparison to the 2008 survey data. The IMR was 10.6 percent, similar to the 2008 survey results in 2015. The MMR and IMR in the seven project provinces are still higher than those of the whole country, which set out the needs to continue communications on safe motherhood to the local people, especially ethnic minority groups, develop the traditional birth attendants’ networks in remote villages, and strengthen the management of pregnancy and antenatal care at the commune health centers, to achieve the MDGs of the country. The project RF is presented in annex B.

4. Achievements by Components

20. The performed activities and interventions are the same as defined in the project document. The outputs of the components have surpassed the targets that were set. See detailed information in annex 6C.

4.1 Strengthening District-level Health Services (disbursement rate=90 percent (USD 38,584,300/USD 42,880,362)

21. The project has supported long-term and short-term trainings for health staff in the NUP in many fields. The training programs fulfilled actual needs of the locality and were implemented in accordance with guidelines, policies, and strategies of the MoH. The training activities improved staff skills and reduced health workforce shortages, providing professionals who were better prepared. It also contributed to reduce the unbalanced health skills in district hospitals and provide a stable workforce to attend their needs in the long term in the seven provinces. The project has supported the development of 377 level-1 doctors, achieving 266 percent of the original target, and 56 percent are ethnic minority doctors. About 1,500 assistant doctors (46 percent with ethnical minority background) were trained and converted to principal doctors. The project has also trained 98 intermediate pharmacists.

77

22. Overall, 95 percent of the staff who graduated through the project training activities returned to work at the host hospitals. Since 2011, the PPMUs decided to enroll assistant doctors from commune health stations as part of the long-term training activities and to include important new specialties in the level-1 training, such as image diagnoses, tuberculosis treatment, clinical pathology, anesthesiology, and others.

23. The short-term training courses on specialized contents have improved knowledge and practical skills on disease treatment to address the shortage of professional capacity and strengthen the decentralized techniques in the district hospital. Besides the clinical areas, trainings on preventive medicine and health management and related areas (hospital management, health information systems, medical equipment repair, and medical waste management) were also provided.

24. To improve maternal and child health status and support the achievement of the health MDGs by 2015, relevant training for nurses and emergency care for maternal and newborn babies were also added in the training programs. The project also conducted short-term training courses on safe motherhood, focusing on maternal and newborn health care.

25. The implementation of non-training human resources activities such as ‘technology transfer; (in collaboration with the Government 1816 program)52 and ‘rotation of doctors’ (temporary placement of a doctor from a higher-level facility at a district hospital) had partially alleviated the shortage of doctors and other staff in the district hospitals, providing capacitation on specific techniques while emphasizing more practical skills. Detailed training results are in tables 6.1–6.6 of annex 6C.

26. The project also performed activities to repair and upgrade 18 district hospitals (100 percent working plan, completed in 2010–2012) providing the installation and effective use of the medical equipment supported by the NUP (table 3.7, annex 6C).

27. The project provided 52 ambulances and 5,065 essential medical equipment for 64 district hospitals (average of 80 devices per hospital achieving 102 percent of the working plan). It included valuable equipment such as high-tech x-ray machines (102), endoscopic systems (37), ultrasound systems (109), ventilators (112), and monitor tracking devices (142) (table 3.8, annex 6C).

28. The equipment provided is strongly based on the hospitals’ demands, possible because of carefully reviewing and assessing the needs before starting the purchasing process. The equipment that was received immediately had a positive effect on providing health care for the people and offering favorable conditions for the health staffs to practice the knowledge and skills that they had gained from the long-term and short-term training programs.

29. The maintenance of this equipment had complied with the requirements. Health staff were fully trained and instructed to adequately use and explore the equipment. The specific 52 1816 Program is a program of the Government of Vietnam on ‘Sending the professional staff from higher-level hospitals to support lower-level hospitals professionally to improve the quality of the health care services’.

78

teams responsible for maintaining and repairing the medical equipment were established in each of the seven provinces.

4.2 Increasing Financial Access to Healthcare Services for Decision 139 Beneficiaries (disbursement rate=100.5 percent (USD 10,048,816/USD 10,000,000)

30. To increase the use of health services for the poor and minorities, the project financed travel expenses, meals, and direct costs of the beneficiaries who were not supported by HI or other health care funds. This support helped the poor/minorities access and use quality basic health care services, ensure fairness in the health financial protection, and improve the health of the beneficiary population. It sustained the effective implementation of the provincial health care policy for the poor/minority, defined in Decision 139. From 2009 to 2014, around 65,000 inpatients’ medical visits received support. (Table 6.9 annex 6C). The percentage of poverty households which experienced catastrophic health care expenditures (according to the World Health Organization criteria) reduced from 10.4 percent to 2.0 percent between 2009 and 2014 surpassing the target of 9.4 percent. (Annex 6B).

31. The final project survey showed that the rate of households which experienced expenditures for health care services decreased five times and three times compared to the baseline and MTR, respectively. In 2015, the HCFP (supported by Decision No. 14/2012 / QD-TTg) was operated in four provinces (Lao Cai, Ha Giang, Son La, and Lai Chau). Total 269,400 inpatient visits were supported in four provinces in the 18 months of the project extension period (table 6.10, annex 6C). Three other provinces—Cao Bang, Bac Kan, and Dien Bien—had not implemented this decision as the provincial budgets were not adequate or allocated to continue these supports. During the extension period, the number of inpatients in these three provinces was slightly reduced in comparison to the previous period. Without support on transport and food allowances, poor households tended not to go to the hospitals as they could not afford travel and food during their hospitalization period. This suggested that the financial barrier is still one of the main reasons that limit access to and use of health care services to households, especially poor households, so the NUP’s support for the poor in accessing health services is very important.

32. The project also promoted health-seeking behavior for the poor by developing IEC activities to increase the target population’s knowledge about their benefits under the HI scheme and the additional support provided by the project. As part of the activities related to this area, IEC material was distributed to the households, IEC messages were announced in the community and newspapers, movies, videos, and comedies were produced and broadcasted on mass media (TV, speakers, and radio) and posters were displayed in the hospital departments/rooms. The project provided some essential communication equipment to be used in the provincial centers for the IEC activities. Nearly 5,000 meetings and campaigns were conducted at crowded places and the IEC teams visited around 8,000 households.

33. The IEC activities increased access and utilization of health care services for the poor/ethnic minority groups in the district hospitals. The final project evaluation showed that 94.7 percent of the poor/ethnic minority people could tell at least one right of HI cardholders,

79

94.3 percent of the poor/ethnic minority people in the project areas had HICs and the number of inpatients at district hospitals sharply increased.

34. To strengthen capacity for the HCFP, the project provided training for the health provincial teams and basic equipment (computers, printers, and photocopiers) to district hospitals. The PPMU implemented systematic supervisions in the HCFP and guaranteed that the management of the fund activities was in compliance with the provided guidance.

4.3 Monitoring, Evaluation, and Project Management (disbursement rate=85.3 percent (USD 6,075,917/ USD 6,744,638)

35. The project maintained an adequate management structure and implementation arrangements. The Project Steering Committee at the MoH was established to directly lead project management. The CPMU and PPMUs were the key players in implementing and managing the project. Monthly meetings with the PPMUs gave district hospital leaders the chance to express their needs, comments, and suggestions for more appropriate and informed implementation. The coordination between the MoH, CPMU, and PPMUs were systematic and effective.

36. The project was implemented faster than previously scheduled, in the original plan. The project outputs have reached and exceeded the outputs agreed in the project document. After the project closing, the rate of disbursement was estimated at 96 percent. The goals and targets are completed and beyond the schedule, at a cost lower than originally expected, suggesting that the project was very cost-effective as well. The MoH authorities are satisfied with the project achievement results.

5. Restructuring: Project Extension for 18 Months

37. After five years of execution, the project was implemented on schedule and the outputs were achieved and exceeded the original plan, with savings in the original budgets (mainly through procurement and tendering activities). However, there were a number of new activities—non-planned in the original project design—that could be implemented with the saved funds to promote efficiency and enhance the future sustainability of the project. The main arguments to explain the rationale for the project extension are

although the district has invested in training and upgrading of facilities and equipment, the needs for medical support at the seven NUP provinces were still huge because the district and commune levels were still facing many difficulties;

the project investment has significantly helped the quality of services, increased the number and type of health services provided, and increased accessibility and the use of health services by the population at the district level. However, the NUP team identified additional needs to be implemented in the short term (12–18 months) to guarantee the project’s long-term efficiency of investments and maximize the effective use of the upgraded health facilities and the acquired equipment; and

80

the health indicators in general, especially those related to the health MDGs in the NUP are the lowest compared to other regions in the country. The prime minister issued Resolution No. 05/NQ-CP, dated 01/13/2014, on accelerating the implementation of the health MDGs. Therefore, the support to accomplish this resolution in the NUP areas (especially the goal of reducing child mortality rate and MMR) was very urgent, especially considering the budget cuts that occurred during this period.

38. Given these arguments, the DO and the scope of the interventions for the extension phase did not change in comparison with the original stated in the project document. In addition, the extension phase of the project looked for ways to

maximize efficiency and enhance the sustainability of the project investments in the seven provinces; and

contribute to the achievement of Vietnam’s health MDGs in the related areas, especially by reducing child mortality and improving maternal health.

39. During the extension phase, the project has achieved the objectives and implemented the planned activities such as training on maternal and childcare and procuring equipment for newborn health units and consumables to ensure safe birth delivery. The project management was effective, focused on achieving the project results and promoting the sustainability of the investment.

40. Special attention was dedicated to improve inpatient and outpatient services utilization and capacity to implement clinical techniques at the district hospitals. The project has contributed significantly to reach the MDGs (especially in reducing maternal and child mortality) through capacity-building activities on obstetric and neonatal emergency at the health facilities, particularly at district hospitals. The project contributed to the development of village midwives’ teams and to increase the percentage of women giving birth at home, with SBAs’ assistance, by providing clean delivery packages for pregnant women. During 2008–2014, the maternal and child mortality rates declined remarkably, as can be seen in the annex 6B: Project Results Framework, MDG indicators.

6. Beneficiaries

41. The main project beneficiaries, as stated in the project document, were the poor and ethnic minorities in the Northern Upland Provinces. These population groups were supported by reducing their financial barriers to access and make better use of quality health services. Project investments focused on the district level, an appropriate level at which the poor and ethnic minorities had the ability to access their health care needs.

42. Other project beneficiaries were the health workers of the district hospitals in the seven NUP areas who participated in trainings in many fields, such as (a) training level-1 doctors for treatment; (b) doctors with four-year added training; and (b) short-term training courses for clinical, preventive medicine, and health management. Since 2011, the project

81

extended the training support to pharmacists and assistant doctors at the community health centers.

43. Last, but not the least, the project benefited the health management agencies at provincial and district levels (Department of Health, Division/District Health Centre) by improving their managerial capacity. The district hospitals’ management teams were also benefited by trainings that ensured better organization and means to provide health care for the poor and ethnic minorities.

7. Evaluation of Other Project Aspects (risks, safeguards, fiduciary)

44. The project was designed based on the needs and recommendations of the local authorities and communities. There were several rounds of referendums, for finalizing project objectives and activities, with different local stakeholders (the provincial people committees, department of health, district hospitals, and specialists), using different methods, including participatory rapid assessment, consultation workshops, and direct consultations. The MoH accumulated experience in implementing projects with similar components and activities. Therefore, the implementation of the Project was quite favorable. The risk management process was well conducted and did not significantly impede the project’s performance.

45. Environment safeguards. The project provided good solutions and implementation of waste management processes at district hospitals. Basic training on regulations related to HCWM and nosocomial infection control was provided for the district hospital managers and staff. The CPMU and PPMUs received guidance for planning and implementing measures for HCWM. Consumables and equipment for HCWM53 were procured by the PPMUs and distributed to the project district hospitals. The project hospitals achieved remarkable improvements in the waste management process compared to the baseline (2007), when most of them did not comply with the HCWM regulations. By 2013, a well-prepared HCWM plan and monitoring program started to be implemented and all district hospitals had strengthened institutional arrangements for that. The availability and proper use of waste containers, transportation, and cooling devices resulted in significant improvements in health care waste separation, collection, storage, and final disposal in district hospitals.

46. Social safeguards. The project beneficiaries included the poor and ethnic minorities living in disadvantaged areas. Given the typical geographical condition of the NUP provinces, the proportion of ethnic minorities is quite high. Data reported from PPMUs and surveys are always disaggregated by Kinh (the majority of the population) and ethnical minorities to indicate that the support from the NUP is considerable to ethnical minorities and the poor. Most of the ethnical minority health staff received long- and short-term trainings, which contributed to the increased ratio of ethnical minority doctors in the seven project provinces. The percentage of beneficiaries who are ethnical minority is higher than 75 percent. In some provinces, ethnical minority staff account for a higher percentage of total participants (such as 91 percent in Cao Bang). The project tailored IEC campaigns with the 53 It includes 23,000 kg of color-coded plastic bags, 25,000 sharp boxes, together with fixation frames, more than 500 waste containers, 12 waste on-site transportation devices, and 64 cooling devices.

82

linguistic and cultural characteristics of ethnic minorities. The IEC materials were translated into the local languages and, to the extent possible, verbal and graphical means of information transfer were used. The key success was the development and implementation of a strategy that reaches the diverse and dispersed beneficiaries in the Northern Upland Provinces, given the ethnical diversity of the region and the difficult geographical terrain.

47. Fiduciary aspects. The project did not register major fiduciary problems (regarding procurement of civil works or medical equipment acquisition). The counterpart funds from the MoH and provincial budgets were allocated/provided on time and met the demand for timely implementation of the management activities of the project.

8. Bank and Borrower Performance

48. The project implementation had sufficient human resources (including national consultants) and good capacity to manage and implement the activities and use the financial resources. The CPMU provided capacity building for PPMUs staff on project management, including financial, procurement, accounting, assets management, civil works, training, HCFP, and M&E activities. In addition, the project had international experts in the related fields. The project was implemented on time and exceeded the set targets. Project interventions were highly effective, providing significant benefits to health care of the NUP beneficiaries.

49. Project management activities were carried out with high demand for quality management. Due to the design (simultaneous interventions on both the ’supply’ and demand’ sides) the project interventions were completed and exceeded the targets.

50. The basic elements for managing the project (Project Manual, job description for each position, and financial management software) were available right from the start of the implementation. The accounting software (installed at the central and provincial levels) met the regulations of the Ministry of Finance and the project requirements. The financial management system provided accurate and timely information on whether the credit proceeds were used for the intended purposes. Cash accounting were conducted monthly. Financial statements were prepared quarterly by the PPMUs and sent to the CPMU for consolidation and submission to the World Bank. The procurement processes and procedures strictly followed the requirements and regulations laid down by the Vietnamese Government and the World Bank.

51. The involvement of stakeholders (CPMU, related departments of the MoH, PPMUs, Department of Health, provincial/district hospitals) in the planning process indicated that the plans were carefully considered before approval. The project plan and adjustments/revisions were made based on actual demands and need of localities, in accordance with the project objectives so as not to impede the progress of the project. The procedures and processes of approval were agreed by both the MoH and World Bank to harmonize both institutional perspectives. Therefore, in general, the progress on implementing the activities satisfied the request of the World Bank and the MoH.

83

52. The operation implementation was sequenced as follows: staff training, hospital repair, upgrade and acquisition of new medical equipment or supply. Communication activities were carried out simultaneously to encourage people to seek medical care and treatment at the health facility. The project activities were carried out in accordance with the approved working and financial plans.

53. The M&E process was implemented according to the project M&E framework. The baseline survey provided the initial and final targets of the project indicators. The MTR and final project evaluation reviewed all the targets. The progress monitoring missions collected outcomes at the localities with the appropriate tools. The data on indicators and performance coming from provinces was aggregated, analyzed, and used to adjust project activities as needed (some adjustments to increase the quotas and additional medical staff training, enhanced communication activities to change behavior, and others were decided and implemented in the middle of the project cycle).

54. The CPMU also provided training on monitoring, evaluation, and reporting and data quality assurance for the PPMUs staff.

55. The success in implementation of the project is also the consequence of good management and positive support from the World Bank team. Closed monitoring of project activities, timely issuance of ‘no objections’ for procurement and work plan, suggestions for important solutions and recommendation for speeding up the progress of the project, and participation in technical missions were some of the valuable contributions of the World Bank team to the success of the project.

56. For the improvement of future projects, the M&E system and M&E plan should be built at the beginning, with the standard forms for collecting data from the PPMUs and implementing sites. This will allow future projects to have good data and reference sources from the beginning, improving the follow-up of the projects’ achievements. The PPMUs should closely follow up with provincial people’s committees for approval of annual financial plans as soon as possible, so the activities can be implemented at the beginning of each year.

9. Arrangements for Sustainability of Results

57. The NUP project investment presents a high level of sustainability and has contributed to strengthening the NUP health systems. The project interventions were in line with the policies and priority of the health sector and consistent with local needs. The seven provinces are committed to continue to perform efficiently and maintain the sustainability of the project. Many interventions were recognized to guarantee long-term sustainability to provide and promote access of health services to the poor and ethnic minorities, such as the following:

Training/developing human resources. The doctors (who received support from NUP) will receive funds (from the provincial, host hospitals) to further continue long-term trainings to become specialized doctors in surgery, traumatic,

84

and subordinate departments. This will certainly help hospitals to use more techniques at district hospitals. For the policies’ implementation to strengthen health workforce, especially for health care at the district level, the localities recruited local staff and organized plans to maintain regular trainings to strengthen the capacity of health staff.

Quality of care. The model of technology transference among different health complexity levels has paved a sustainable mechanism to develop capacity in the provision of health care services and to constantly improve the quality of the health care services at all levels, especially at the district level.

Upgraded medical equipment. All provinces participating in the project provinces have specific commitments to provide funds to maintain equipment and upgrade and repair infrastructure, after the project closing. The infrastructure and equipment will continue to be used and will ensure the quality in providing health care services for the local people at district hospitals.

Funding access to health for the poor. The project’s high sustainability was also demonstrated through the health care policies for the poor/ethnic minorities currently being applied. Infrastructure, equipment, and human resources training supported by the project have worked well in providing quality health care services to beneficiaries. Many provinces have mobilized funds to support the local people to access and use health care services at district hospitals. Four of the seven provinces have established provincial HCFPs. In the remaining three provinces, where the Decision 1454 is not implemented, the Government allocated partial funds (of transportation and meals) for the ethnical minorities and the poor who need to be served by district hospitals.

Other aspects. The IEC materials developed by the project and the increased management capacity of the HCFP have been essential to support the provinces to implement the program effectively and transparently. These are considered as sustainable contributions for health care activities for the poor.

10. Lesson learned

58. The NUP project is designed with specific activities after careful consultations with local stakeholders in the seven provinces. Right after coming in to effect, the project activities are implemented, without any delays. Some projects are designed in the form of a project framework; the project started with need assessments and specific activities are designed after that, consequently, the project is implemented slowly.

59. The project is built bottom-up, based on local needs, and referred to the criteria and standards issued by the MoH and Ministry of Finance (list of medical equipment, contents, and training curricula). The NUP project does not develop its own cost norm or project 54 Decision 14, issued by the prime minister in 2012 on the revisions to Decision 139, issued in 2008 for the health care for the poor.

85

implementation guidance, but applies the existing regulations, so that project implementation is fast, convenient, and highly sustainable.

60. Interventions of the NUP project are designed on both the supply and demand sides, including supporting the provision of services (trainings, hospital renovations, providing equipment) and using services. This design brings high efficiency: the improvement in the quality of health service, interacting with the support for the poor, and increase in the access of the poor to health services.

61. The NUP’s support focused on northern mountainous region, with comprehensive investments for district hospitals, and created equal development for the health sector in the northern mountainous region. Therefore, the project implementation is far more favorable in comparison with the projects where targets are scattered in different regions across the country.

62. The NUP project focused on the district level, at the grassroots level of health of Vietnam (from the district level and below). This is one of the few investments in health that support the district level in Vietnam. The project targeted the districts of the most difficult areas in Vietnam, focused on primary health care, maternal, and child health care, which are very important factors that contribute to pursuing the MDGs and the UHC in Vietnam today and up to 2035. The effectiveness of these investments is significantly observed. The project interventions (the support for long-term training, the IEC activities on health and support for the poor, and so on) should be continued, using regional and local funds to ensure that the project achievements will be strengthened and sustained.

63. The project assisted localities to develop their health plans and implement preparation steps for maintaining the activities after the project ended and to continue the issuance of policies and plans on health support for the northern mountainous provinces. Particularly, two aspects could be highlighted: (a) the health workforce development policy and (b) the retention and training of a village midwives team to serve remote areas where home birth delivery is necessary.

64. A number of activities, for example, procurement of equipment or training of staff, although the work plan has been approved a year before, are still reconfirmed with local needs (quantities and types of training/equipment) before deployment. This ensures that the project investments are appropriate and meet the real needs from district hospitals.

65. The capacity of PPMUs plays an important role in implementing the project. The staff of the PPMUs should be stable, be well trained in project management, and work closely with the technical support and regular supervisions from the central project. Although the CPMU has only 20 staff (much less than many other projects of the same scale), the project activities were well implemented, the schedule and work plan were always on track, and the targets/results were surpassed in comparison to those assigned at the start of the project.

66. The support for the poor to access health care services is very well implemented in the seven NUP provinces. Experience in implementing and critical results gained from the

86

NUP, is evidenced, and driven to replicate this policy nationally, through Decision No. 14 of the Government. The policy of providing HICs to the poor should be continued to achieve universal HI.

67. The very close collaboration and helpful support from the World Bank office in Hanoi, by quickly exchanging information and solving the difficulties, which was the responsibility of the TTL and relevant World Bank staff greatly contributed to the achievements and success of the NUP.

87

Annex 6A. Project Risk Management (Borrower’s View)

Risks Level of Risks Measures to Overcome

The non-poor group could benefit from the intervention of the project, leading to rising inequality

Medium The project developed a good management system to control these risks by coordinating well with Vietnam Social Insurance in the provinces for review and synthesis of the project beneficiaries annually. The list of the poor and ethnic minorities who were qualified for support from the NUP was provided to the district hospitals. To receive the NUP support, the patients should present the insurance card with the code ‘HN’ or the certification of poor from the communal people’s committee

The project implementation in the district hospitals may not be synchronized with the commune health centers, which could misuse the funds of the HI system.

High Because the project investment and health care support for poor people was focused at the district level, a large part of the poor, instead of seeking health checkups at the commune health center, tended to go to the district hospital to access and use the support of the project, compromising the use of the HI fund at the commune level due to the impact from the abovementioned patient flow. The project had controlled this risk well by supporting only inpatients and setting the cap for the length of stay (10 days) to avoid overuse of the support.

Uneven implementation progress across project components undermines the integrated health systems development approach

Potential mismatch between the timing of demand-side interventions that are expected to boost demand for services and investments

The project developed a yearly work plan to ensure that the activities were implemented in an appropriate sequence. Closely monitoring the implementation progress to ensure coordination between components. The completion of civil works in the first two years has created a favorable condition for installing the medical equipment. The trainings for health staff on using and maintaining the different types of equipment were conducted before and immediately after the equipment was provided. A four-year training to upgrade assistants to become medical doctors for commune health stations, was implemented in 2011, when many doctors from the district level graduated and covered for the commune heath center. The IEC support for the poor and training for district hospitals to manage funds for the poor were implemented in parallel with others.

Staff not qualified for entrance exams for level-1 specialists

Medium The project supported the staff preparation for entrance exams.

The staff did not return to the district hospitals to work after graduation

Medium Recruitment of staff training was done in accordance with local needs and selection criteria of the project. Staff signed a terms of responsibility to return to their origin district hospitals’ workplaces before going for training. The certificate was issued only after staff completed the training and returned to work in the district hospital.

Too many officers/staffs were sent for training, affecting the health services delivery at health facilities

Medium The district hospital managed to schedule plans before sending staff for training. Training courses were organized to be spread throughout the project duration, thus avoiding problems that could compromise the

88


service delivery process at the district hospitals. Otherwise, the project supported temporary staff transfers among hospitals to help each other when their staff was attending the training.

Delays in hospital civil works could compromise the investment plans

Medium Civil works investments were selected in district hospitals where the area for construction is ready and where PPMU and CPMU capacity to develop the fiduciary tasks exist.

Project delays due to specialized procurement, tendering, and construction tasks

High Procurement tasks followed the existing bidding laws and instructions. The MoH conducted administrative reforms in procurement, creating favorable conditions for faster implementation. Project staff, and especially the team working with the World Bank are trained and accumulated experience in procurement before the project started.

Medical equipment provided to the district hospital by the project are not used effectively

Medium During the implementation, the team identified investment needs to match the acquired equipment with the social and epidemiological needs and organized training and human resource development to use equipment from the first year of the project. The team also trained staff to carry out minor repairs, creating favorable conditions for the installation and efficient operation of the equipment provided.

Equipment is inadequately maintained throughout the life of the project

The civil works investments in facility repairs and refurbishments were not maintained, leading to rapid degradation of the facilities

Medium The project provides trainings for the hospital leaders on hospital management, including the section of managing the equipment. The hospital had plans and allocated funds for maintaining/repairing equipment and facilities based on the actual hospital needs. These tasks are gradually improved because the hospitals are aware of the important roles of regularly maintaining the equipment.

Unrecognized barriers continue to limit access for the poor, despite improved financial access

Medium— The project regularly collected data on inpatient visits who received support from the NUP. Data showed that the number of beneficiaries gradually increased. The rate of occupied hospital beds has increased to average 6% per year. Another activity is the promotion of health-seeking behavior through information dissemination and other outreach activities regarding benefits and entitlements under the HCFP. The improvement of quality of the services also was a factor to attract the poor who went to the hospitals for treatments

Coordinating capacity strengthening for HCFP management that builds on existing systems and does not create a duplicate administrative system supported by the project

Low The project has guidance for PPMUs and district hospitals, which clearly states the roles and responsibilities of the staff administering HCFP. The management of HCFP has been strengthened during the project life, through various support, including training courses for the fund management unit, and support of equipment to help fund management at the hospitals more conveniently and rapidly. The examination and monitoring of the health care for the poor has been

89


carried out in a systematic way, to ensure that the implementation is in accordance with the project’s regulations.

The beneficiaries, not being adequately informed of their benefits, do not make adequate use of their HICs

Low The project improved the quality of medical services and conducted various types of behavior change communication interventions to include messages geared toward educating beneficiaries on health education, the scheme and their benefits (support indirect costs in addition to support from the state through the HCFP) to poor families who went for the services.

90

Annex 6B. Project Results Framework

No. Outcome IndicatorsBaseline Survey(2009)

MTR(2012)

Final Survey(2014)

Extension Phase(2015)

Disparity2009–

2014/2015

Targeted Year 6

Indicator 1:(original)

Utilization rates of inpatient services in district hospitals among Decision 139 beneficiaries (according to household survey)

0.063 0.054 0.074 n.a. 117%10%

increase

(supplement) Utilization rates of inpatient services in district hospitals among Decision 139 beneficiaries (according to reports of district hospitals)

0.0247 0.049 0.085 0.096 389%10%

increase

Indicator 2: (original)

Utilization rates of outpatient services in district hospitals among Decision 139 beneficiaries (according to household survey)

0.016 0.071 0.032 n.a. 200% >15% increase

(supplement) Utilization rates of outpatient services in district hospitals among Decision 139 beneficiaries (according to reports of district hospitals)

0.067 0.162 0.206 0.247 369% >15% increase

Indicator 3: Percentage of households which experienced catastrophic health care expenditures in the year prior to the survey

10.4 7.1 2.0 n.a. −81% > 10% decrease

Indicator 4: Percentage of health services according to the national norms that district hospitals can implement

39.1 47.5 71.4 80.4 206% 70%

Result Indicators for Each ComponentComponent 1: Strengthening District-level Hospitals

Indicator 5: Percentage of patients satisfied with health care services at district hospitals

8.5 48.8 84.4 n.a. 993%Increased by 20%

Indicator 6: Adherence to treatment protocols in treatment of three common diseases (MOH)

— — — — —Increased by 40%

Percentage of health staffs with correct knowledge of diagnosis and treatment for ARIs

0.4 41.6 83.6 n.a. 20,900% —

Percentage of health staffs with correct knowledge of diagnosis and treatment for severe pneumonia

13.2 58.1 86.9 n.a. 658% —

Percentage of health staffs with correct knowledge of 9.7 65.5 95.2 n.a. 981% —

91


MTR(2012)

Final Survey(2014)


Disparity2009–

2014/2015

Targeted Year 6

diagnosis and treatment for diarrhea level A/B/CPercentage of health staffs with correct knowledge of diagnosis and treatment for poisoning

26.8 53.8 83.0 n.a. 310% —

Percentage of ARIs medical records with appropriate diagnosis

18.5 38.6 62.2 n.a. 336% —

Percentage of severe pneumonia medical records with appropriate diagnosis

45.5 54.5 71.1 n.a. 156% —

Percentage of pneumonia medical records with appropriate diagnosis

19.6 20.9 57.0 n.a. 291% —

Percentage of diarrhea level A/B/C medical records appropriate diagnosis

41.1 69.4 81.6 n.a. 199% —

Percentage of poisoning medical records appropriate diagnosis

61.1 86.2 88.8 n.a. 145% —

Indicator 7: Percentage of district health staffs trained by the project — — — — 145% 80%

Percentage of health staffs completed short-term training courses compared to the plan

0.0 102.0 357.3 427.3 — —

Percentage of health staffs completed long-term training courses (specialty level-1s doctors) compared to the plan

0.0 54.9 88.0 189.2 — —

Total number of health staffs trained in short-term and long-term courses

0 2,664 8,929 11,868 — —

Indicator 8: Proportion of district hospitals with acceptable operations and maintenance plans and budget for facility and equipment maintenance

— — — — — 40%

Proportion of district hospitals with acceptable operations and maintenance plans and budget for facility

49.2 30.8 99.3 85.9 175% —

Proportion of district hospitals with acceptable operations and maintenance plans and budget for equipment maintenance

77.1 61.5 99.2 93.8 122% —

92


MTR(2012)

Final Survey(2014)


Disparity2009–

2014/2015

Targeted Year 6

Component 2: Increasing Financial Access to Healthcare Services for Decision 139 BeneficiariesIndicator 9: Percentage of Decision 139

beneficiaries provided with HI cards

94.3 86.8 94.3 95.2 101% 70%

Indicator 10 Percentage of 139 beneficiaries with cards, who can correctly identify at least 3 rights of the health insurance card holders

14.8 19.8 57.4 n.a. 388% 75%

Indicator 10 (supplemental)

Percentage of 139 beneficiaries with cards, who can correctly identify at least one right of the health insurance card holders

74.0 98.1 94.7 n.a. 128% —

Indicator 11: Percentage of households who identify financial barriers as a main cause for not seeking health care

2.0 0.9 1.2 n.a. −40% >10% decrease

MDG indicators 2009 2014 2015 (%)1 Percentage of district hospitals providing

caesarean section 74.6 82.1 92,2 124

2 Percentage of district hospitals providing blood transfusion 56.7 67.2 82,8 146

3 Percentage of district hospitals having neonatal units meeting standards — — 49,9 —

4 Capacity of district hospitals in obstetric and pediatric emergency care — — — —

5 Having Continuous Positive Airway Pressure devices 25.0 75.0 79.7 319

6 Having light for jaundice phototherapy treatment 21.4 82.1 87.5 4097 Having oxygen breathing system 42.9 75.0 78,1 1828 Having newborn resuscitator 46.4 92.9 92.2 199

MDG indicators 2007–2008 2013–2014 (%)8 IMR 31.1 29.4 −59 NMR 11.2 10.8 −411 MMR 178 106 −40

93

Annex 6C. Achievements by Components

Table 6.1. Status of Health Human Resources at District Hospitals in Seven Project Provinces over the Evaluation Periods

Health Human Resources Baseline (2008)

Final (2013)


DisparityBaseline – Extension

Phase

Number of Health Staffs who Returned to Work after Being

Trained under Support of the Project,

till 2015Number of health staffs 599 1,097 1,286 687 (2.2 times) 1,329Master, MD 1 11 17 16 0Level-2 doctors 0 9 11 11 0Level-1 doctors 148 325 396 248 (2.7 times) 351General doctor 450 752 818 368 (1.8 times) 978Postgraduate or above 27 87 116 89 (4.3 times) 33Pharmacist (college, secondary school) 207 486 554 347 0

Midwife 355 525 612 257 0Nurse 1,124 2,059 2,238 1,114 0Number of district hospitals 61 67 68 — 68

Source: Reports of district hospitals.

Table 6.2. Number of Level-1 Doctors Trained in Project Provinces

Province

Plan

Level-1 Doctors who Graduated

Level-1 doctors who Continue to Work at

Localities

Total of Level-1 Doctors

Supported by the Project

Total of Person-Years55

Late

2013

Late

2015

% Femal

e

% Ethnic Minorit

y

Total

% Femal

e

% Ethnic Minorit

yCao Bang 23 56 56 57 91 56 57 89 56 (243%) 112

(243%)

Bac Kan 31 36 50 50 98 44 57 95 50 (161%) 100 (161%)

Ha Giang 38 89 98 68 59 95 69 57 98 (258%) 196

(258%)

Lao cai 22 38 50 38 20 49 39 24 51 (232%) 101(230%)

Lai Chau 22 17 18 28 50 18 28 50 19 (86%) 37 (84%)

Dien Bien 19 47 50 50 44 50 50 44 50 (263%) 100

(263%)

Son La 28 26 53 42 21 53 42 21 53 (189%) 106 (139%)

Total 183 309 367 53 57 351 55 56 377(206%)

752 (205%)

55

94

Table 6.3. Number of Four-year Added Doctor and Pharmacist Graduated by the Evaluation Time

CaoBang

BacKan

HaGiang

LaoCai

LaiChau

DienBien

SonLa

Total

Support for four-year added doctor trainingTotal 185 82 302 208 182 250 258 1467Number of doctors who graduated by late 2014

104 42 130 106 91 142 119 734

Person years supported by late 2014 505 191 788 544 577 714 690 4009Converted to number of persons with full four-year added training

126 48 197 136 144 178 173 1002

Number of doctors who graduated by late 2015

131 53 213 145 142 192 182 1058

Number of doctors who returned to work at their units

122 44 208 96 136 193 179 978

Person years supported by late 2015 586 231 960 646 668 822 829 4742Converted to number of persons with full four-year added training

147 58 240 162 167 206 207 1186

Plan achieved (%) 147 144 192 179 128 179 148 160Rate of ethnic minority doctors 99.5 100 36.1 36.1 15.4 33.6 44.2 46.1Rate of female doctors 98.9 50 32.1 40.9 43.4 29.2 39.9 45.1

Support for four-year added pharmacist trainingTotal 13 7 15 23 13 16 11 98Rate of ethnic minority pharmacists (%) 93 86 31 17 7 6 18 31Person years supported by late 2014 31 11 28 57 29 29 19 204Converted to number of persons with full four-year added training 7.8 2.8 7.0 14.3 7.3 7.3 4.8 51.0Graduated by 2015 7 1 6 13 7 3 3 40Person years supported by late 2015 44 18 41 82 42 45 30 302Converted to number of persons with full four-year added training 11 4.5 10.25 20.5 10.5 11.25 7.5 75.5Rate of ethnic minority pharmacists 92.3 71.4 20 26.1 7.7 6.0 18.2 30.6Rate of female pharmacists 86.4 71.4 60 82.6 69.2 50 72.7 70.4

Table 6.4. Number and Rate of Clinical Staff Attending Short-term Training Courses on Health Examination and Treatment in the Extension Phase

Course Plan Total Percentage versus Project Document

Anesthesia 117 157 134Testing 95 149 156Emergency care 140 152 109External medicine 101 143 142Pediatrics 103 214 208Internal medicine 111 194 175Diagnostic imaging 200 436 218Obstetrics 102 191 187Communicable diseases 80 165 206X-ray 107 121 113Ear, nose, and throat, and dental 108 70 65Nursing management 70 63 90Pediatric emergency n.a. 681 —

95

Course Plan Total Percentage versus Project Document

Obstetric emergency n.a. 687 —Total 1.334 3.423 257

Table 6.5. Number and Percentage of Doctors/Assistant Doctors Participating in Short-term Training Courses on Mother and Child Health Care in the Extension Phase

Training Course

Planned Quantity (Number of Trainees per

Course)

Actual Quantity

(Number of Trainees per

Course)

%

Training to standardize the health workers who work on obstetrics/gynecology be certified as SBA 35 22 80

Training to standardize the health workers who work on obstetrics/gynecology be certified as SBA 600/30 courses 794/30 courses 113

Training to standardize the general doctors who work on obstetrics/gynecology at district/commune be certified as SBA

420/21 courses 307/12 courses 118

Emergency care for obstetrics//gynecology medical doctors at district level 154/7 courses 110/5 courses 69

Village midwives 240/12 courses 252/12 courses 105Training for clinical staff at neonatal units - district hospitals

70–100/7–10 courses 79/6 94

Training for service providers on Integrated Management of Childhood Illness 350/14 courses 140/6 courses 40

Table 6.6. Number of Doctors Trained on Management, Comparison between the Planned and the Actual Achieved

Type of Training Planned Quantity

Actual Quantity % Achieved

Hospital management 603 753 125 ExceededMedical waste management 215 2,922 1,300 ExceededHealth system management and HMIS 806 845 105 ExceededMaintenance of medical equipment 171 206 120 ExceededEffectively using the medical equipment provided by the project — — — —

Effectively using laboratory equipment 35 35 100 AchievedExploring and using equipment for emergency care, fluid vacuum, aerosol 7 7 100 Achieved

Effectively using disinfection equipment 1 31/1 100 Achieved

96

Table 6.7. Upgrading of 18 District Hospitals

No. Name of Facilities, Location Start Date

Total Investment(VND)

Completed Date

Settlement Date

1 Renovate, minor repair of Pac Nam district hospital September

2010

884,641,819 January 2011 January 2012

2 Renovate, minor repair of Ngan Son district hospital 780,595,945

3Renovate technical house of Bao Lam general district hospital

March 2010

694,900,000

June 2010

June 20124Renovate technical house and toilet - Tra Linh general district hospital

859,000,000

5

Inpatient house for external and internal department -Quang Uyen general district hospital

3,750,000,000 March 2010–March 2011

6

Inpatient house for external, internal, and traditional medicine department - Hoa An general district hospital

July 2011 10,354,000,000 July 2012 July 2013

7 Renovate Muong Cha district hospital

August 2009 2,480,318,000 June 2012 June 2012

8 Renovate Dien Bien Dong district hospital

December 2010 11,469,000,000 December

2011December

2012

9Meo Vac general district hospital: technical house and supporting facilities

April 2010

4,531,446,000

April 2011 April 201210Quan Ba general district hospital: technical house, toilet, and water tank

3,602,837,534

11Hoang Su Phi general hospital: outpatient examination house and administrative

5,020,981,190

12 Minor repair of Phong Tho district hospital

November 2009

163,940,587

December 2009

December 201013 Minor repair of Than Uyen

district hospital 617,135,748

14 Minor repair of Tan Uyen general district hospital 163,194,151

15 Build high-tech house for Than Uyen District Health Centers May 2011 8,525,000,000 May 2012 May 2013

16

Repair and upgrade building for technical, pharmacy department, nutrition department of Moc Chau general district hospital

November 2011 12,937,587,000 November

2012November

2013

17

Renovate consultation department and emergency resuscitation department of Mai Son general district hospital

December 2011 907,320,000 September

2012September

2012

97

No. Name of Facilities, Location Start Date

Total Investment(VND)

Completed Date

Settlement Date

18Renovate technical house of Muong La general district hospital

December 2012 1,249,290,000 December

2012September

2012

Total 68,991,187,974

Table 6.8. Bids Procured by CPMU

No Code Name of Bids Quantity Total Cost (US$) Notes

1 ICB 01 Equipment and technology (6 types) 278 1,839,775.252 ICB 02 Equipment for intensive care (8 types) 873 1,019,763.173 ICB 03 Monitor (3 kinds) 142 845,600.004 ICB 04 Ventilator (2 kinds) 112 1,281,178.005 ICB 05 Sterilization device (3 kinds) 160 1,449,028.386 ICB 06 Furnaces for medical waste treatment 39 Canceled7 ICB 07 Ambulance (52 pieces) 52 3,071,809.168 ICB 08 Working cars (8 pieces) 8 908,794.779 ICB 09 Communication equipment (11 kinds) 140 524,408.0010 ICB 10 Ultrasound (2 types) 109 1,471,900.0011 ICB 11 Endoscopic systems (2 types) 77 Canceled12 ICB 12 Cardiopulmonary resuscitation (2 types) 366 386,805.0013 ICB 13 Medical equipment (4 categories) 337 956,065.0014 ICB 14 Operating room equipment (6 types) 396 4,257,544.4915 ICB 15 Surgical instruments (7 types) 655 532,867.0516 ICB 16 Special equipment (6 types) 365 2,198,896.0017 ICB 17 X-ray (2 types) 102 1,309,275.0018 ICB 18 Laboratory equipment (9 categories) 565 1,095,884.6419 ICB 19 Basic equipment (19 kinds) 115 467,885.4120 ICB 20 Monitor device (5 categories) 94 764,210.0021 NCB 01 Ambulance (7 pieces) 7 393,180.20

22 ICB 21 Pediatric surgical equipment and gynecology (6 types) 174 589,105.60

23 ICB 22 Emergency resuscitation equipment and obstetrics (7 types) 160 529,963.96

24 ICB 01/2015

Model for trainings on obstetric care and clean package for delivery, tool bag for traditional birth attendants s

1.276.708,92

Total: 25,893,939.08

98

Table 6.9. Total Times of Inpatients Supported by NUP, by Provinces, 2009–2014

Province 2009 2010 2011 2012 2013 QI/2014Dien Bien 411 1,779 3,814 16,889 30,058 7,524Lai Chau — 1,264 4,205 13,605 18,132 4,716Son La — 875 2,617 24,856 44,195 10,106Cao Bang 186 4,853 18,129 44,248 45,044 10,533Bac Can — 3,654 12,769 26,975 27,756 7,088Ha Giang 30 3,457 17,727 38,781 39,702 11,231Lao Cai 427 3,508 20,252 35,345 39,914 8,020Total 1,054 19,390 79,469 200,709 244,801 59,218

Table 6.10. Number of Poor/ethnic Minority People being supported from the HCFP in the Project Provinces - Extension phase

No. Contents Son La Ha Giang Lai Chau Lao Cai Total1 Total times

poor/ethnic minority people were supported

52,597 82,676 46,147 87,985 269,405

Supportedtravelling expenses

51,174 82,676 305 57,354 191,509

Supportedmeal expenses

52,597 82,676 36,907 87,985 260,165

Supported direct expenses for health care

3,896 — 8,935 45,683 58,514

2 Total expenses supported by the HCFP (VND)

16,175,884,199 33,397,067,456 30,185,983,270 19,128,174,711 98,887,109,636

Supported travelling expenses

5,007,080,521 — 1,138,446,108 2,213,686,966 8,359,213,595

Supported meal expenses

10,843,276,112 — 4,998,535,291 16,203,882,250 32,045,693,653

Supported direct expenses for health care

325,527,566 — 24,049,001,871 710,605,495 25,085,134,932

99

Annex 7. NUP (P082672) Project Supervision Missions Datasheet

Year

Number of Missions, Dates, and

TTLs

Provinces Visited During the Missions

Main Issues Raised and Solved during the Missions

2009 2 (May and November)TTL: Bakhuti Shengelia

2 provincesHa Giang, Dien Bien

May: Review project implementation progress by components; discuss the baseline survey; discuss on how to accelerate implementation of work plan; review arrangements for the implementation of ‘Health Care for the Poor’ component at the district level; and finalize training arrangements.

June: Review the implementation of 2009 plans; discuss 2010 action plans, related procurement plans, and the measures to improve the effectiveness of implementation; assess the participatory planning process under the HEMA Project and review M&E arrangements; conduct post review of procurement activities in 2009 and the financial management practices; and conduct an implementation assessment in Dien Bien province.

2010 2 (June and December)TTL: Bakhuti Shengelia

2 provinces: Lai Chau, Son La (for financial management review)

June: Review the implementation progress of 2010 annual work plans; review implementation of the recommendations of November 2009 supervision mission; review procurement and financial management processes at the CPMU and PPMUs; update implementation and procurement plans as needed; and assess developments in the health policy field that may have implications for the projects.

December: Review the implementation progress of 2010 annual work plans; review implementation of the recommendations of November 2009 supervision mission; review procurement and financial management processes at the CPMU and PPMUs; update implementation and procurement plans as needed; and assess developments in the health policy field that may have implications for the projects.

2011 2 (May and October)TTLs: Bakhuti Shengelia and Kari HurtNote: An additional safeguard review mission in December 2011

4 provincesBac Can, Cao Bang, Lai Chau, Lao Cai

May: Review the general assessment of the project progress by component and activities; review procurement, financial management, and environmental safeguards; and review social safeguards.

October: Discuss the progress toward achieving the DOs of increased utilization particularly by the poor in the project provinces and strengthen district hospitals; review the implementation progress of 2011 annual work plans and look at the priorities for the 2012 annual work plans; document the progress against the recommendations of December 2010 supervision mission; assess procurement and financial management processes at the CPMU and PPMUs; update implementation and procurement plans as needed; and support progress of the current project implementation issues, particularly concerning the HCWM plan and investments in support of the hospital plans and the implementation of pilots for retention of rural health professionals in the project provinces.

2012 1 (July - MTR mission)TTL: Kari Hurt

3 provincesHa Giang, Son La, and Dien Bien (for financial management

Review the general assessment of the project progress by component and activities; review procurement, financial management and environmental safeguards, review social safeguards. Evaluation of the Project Results Matrix as part of the MTR.

100

Year

Number of Missions, Dates, and

TTLs

Provinces Visited During the Missions

Main Issues Raised and Solved during the Missions

only)2013 2 (May and

December)TTL: Thuy Anh Nguyen*December mission for financial management review only.

4 provincesSon La (May),Son La, Lai Chau, Dien Bien (December for financial management review)

May: Review the general assessment of the project progress by component and activities; time remaining for disbursement, M&E, procurement medical equipment, financial management, and environmental safeguards; and review social safeguards.

December: Update the World Bank’s understanding on the financial management arrangements for the project since last financial management supervision missions (Son La in November 2012, Lai Chau in May 2013, and Dien Bien in April 2012) which covers all areas of financial management, including planning and budgeting, disbursement, funds flows, accounting system and software, reporting, and auditing. The mission will also follow up on all outstanding issues raised during the previous mission and in the 2012 audited financial statements.

2014 2 (January and September**)

3 provinces: Dien Bien (April),Cao Bang, Bac Can (September 2013)

January: Review the general assessment of the project progress by component and activities; time remaining for disbursement, M&E, procurement medical equipment, financial management, and environmental safeguards; and review social safeguards.

September: interim mission. No official announcement was sent.

2015 2 (May and December**)

No site visit May: Key issues for ministry and project management attention, progress and key issues by component and subcomponent; M&E; financial management and disbursement arrangements; procurement, safeguards

December: Review the implementation progress of the project by components and actions agreed and concluded in the aide memoires from mission in May, 2015, to discuss and agree the final actions for project closing in February; to discuss the preparation for the final implementation support mission combined with the ICR mission that is planned in late February/early March 2016.

2016 1 (February) Lai Chau; Lao Cai

Closing Mission - Discussion of the ICR assessment

Notes: *There are also procurement post review missions, at least one every year. ** No aide memoires for September 2014 mission because of the TTL’s sudden sickness and December 2015 mission as it may be combined with the ICR mission which is two months later.

101

Annex 8. List of Supporting Documents

1. Bill and Melinda Gates Foundation.2016. Maternal, Newborn and Child Health Strategy Overview, in http://www.gatesfoundation.org/What-We-Do/Global-Development/Maternal-Newborn-and-Child-Health.

2. Ekman, B; L. Thanh, H. A. Duc, and H. Axelson. 2008. “Health insurance reform in Vietnam: A Review of Recent Developments and Future Challenges.” Health Policy and Planning 23:252–263, doi:10.1093/heapol/czn009.

3. Center for Environmental and Health Studies. 2014. Report on the Final Evaluation of the Northern Uplands Health Support Project, Hanoi, July 2014.

4. Center for Environmental and Health Studies. 2016. Report on the Final Evaluation of the Northern Uplands Health Support Project in the Extension Phase, Hanoi, February 2016.

5. Fosberg, L.T,.2011. The Political Economy of Health Care Reform in Vietnam, Oxford – Princeton Global Leaders Fellow, Woodrow Wilson School of Public and International Affairs, Princeton University, 2011.

6. World Bank. 2011. Country Partnership Strategy for the Republic of Vietnam, IBRD, IDA, Report No. 62500-VN.

7. World Bank. 2006. Mekong Regional Health Support Project. Project Appraisal Document. Washington DC, February 9, 2006.

8. World Bank. 2012. Mekong Regional Health Support Project Implementation Completion Report Results, Washington DC, December 2012.

9. World Bank. 2008. Northern Upland Regional Health Support Project, Project Appraisal Document, The World Bank, Washington DC, February 2008.

10. Government of Vietnam, Ministry of Health. Annual Health Statistics Year Book, Hanoi, Series from 2007–2014.

11. Government of Vietnam. 2012. Ministry of Health. Master Plan on Universal Health Insurance Coverage, draft, Hanoi, June, 2012;

12. Government of Vietnam, Ministry of Health. 2015. Success Factors for Women’s and Children’s Health, Ed. WHO, Geneva, 2015

13. Lieberman, S. and Wagstaff, A. 2009. Health Financing and Delivery on Vietnam – Looking forward. Health, Nutrition and Population Series, The World Bank, Washington DC, 2009.

102



14. Tiberti, L. 2000. Health Sector Reform in Transition Economies: The Case of Vietnam, University of Florence, Florence, 2000.

15. Toan, Tran Khanh .2012. Antenatal and Delivery Care Utilization in Urban and Rural Contexts of Vietnam: A study in two health and demographic surveillance sites, Doctoral thesis at the Nordic School of Public Health NHV, Gothenburg, Sweden, 2012.

16. World Health Organization .1999. Mother-Baby Package Costing Spreadsheet, Geneva, Switzerland, 2012.

REPORTS PRODUCED BY THE GOVERNMENT TO SUBSIDIZE THE PROJECT PREPARATION

1. Health status in the 7 provinces of Northern Upland

2. Health care service use and accessibility status in the 7 provinces of northern upland: Cao Bang, Bac Kan, Lao Cai, Ha Giang, Son La, Dien Bien and Lai Chau)

3. Health Human Resource Analysis in the 7 provinces of northern upland - Health care for the Poor: Identification of the needs and proposal of investment for capacity building and management capacity development

4. Assessment of the Healthcare Fund for the Poor in the 7 provinces of northern upland - Health care for the poor: management according to the Decision 139 in 7 provinces of northern upland (most difficult provinces)

5. Health System Assessment for 7 provinces of northern upland - Inventory of medical equipment in hospitals of 7 provinces of northern upland

6. Output indicators after analysis - Socioeconomic, Demographic, Cultural Geographic and Health Status indicators: Morbidity, Mortality, CDR/IMR, Under 5-child nutrition status

7. List of medical staff to be trained and cost estimate and cost table for training component

103

Map

Source: World Bank Maps

104