final evaluation report - unicef.org€¦ · immunization program in sao tome and principe...

Final Evaluation Report

Summative Evaluation of the Immunization Program in Sao Tome

and Principe (2017-2019) with a focus on the Cold Chain

Commissioned by UNICEF in Sao Tome and Principe

with the support of the Regional Office West and Central Africa

Melissa Andrade Costa, Senior Evaluation Consultant

Joerg Rengel, Thematic Expert

Paulo Jorge de Castro, National Consultant

Summative Evaluation of the Immunization Program in Sao Tome and Principe with a focus on the Cold Chain

2

2

Index LIST OF ABBREVIATIONS ....................................................................................................... 4

LIST OF FIGURES AND TABLES .............................................................................................. 5

Acknowledgments .............................................................................................................. 8

Executive Summary............................................................................................................. 9

1. Evaluation Object ...................................................................................................... 17

2. Overview of the intervention.................................................................................... 19

3. Evaluation Purpose .................................................................................................... 23

4. Evaluation Objectives ............................................................................................... 23

5. Evaluation Scope ....................................................................................................... 24

5.1. Thematic scope .............................................................................................. 24

5.2. Chronological scope ..................................................................................... 24

5.3. Geographical scope ...................................................................................... 24

6. Evaluation Criteria ..................................................................................................... 24

7. Methodology .............................................................................................................. 26

7.1. Methods for data collection ............................................................................. 26

7.2. Data Analysis ....................................................................................................... 28

7.3. Data Sources and Sampling strategy .............................................................. 29

7.4. Limitations of the evaluation process .............................................................. 32

8. Ethical Considerations ............................................................................................... 33

8.1. Key ethical evaluation principles ..................................................................... 33

8.2. Ethical Safeguards .............................................................................................. 33

8.3. Human rights and participation of stakeholders ........................................... 33

9. Findings and Preliminary Conclusions (by criterion) .............................................. 35

9.1. Relevance ........................................................................................................... 36

9.2. Effectiveness ........................................................................................................ 39

9.3. Efficiency ............................................................................................................. 49

9.4. Sustainability ........................................................................................................ 60

9.5. Human Rights, Gender and Equity .................................................................. 64

10. Final Conclusions ........................................................................................................ 69

11. Lessons Learned ......................................................................................................... 73

12. Recommendations .................................................................................................... 74

13. Annexes ....................................................................................................................... 79

Appendix A. UNEG Evaluation norms and standards .............................................. 79

Appendix B. Stakeholder Analysis ............................................................................... 80


3

3

Appendix C. List of people interviewed ..................................................................... 82

Appendix D. Evaluation Matrix .................................................................................... 84

Appendix E. Interview guide ........................................................................................ 92

Appendix F. Interview Guide for Focus Groups ........................................................ 95

Appendix G. Check List for Health Units ..................................................................... 96

Appendix H. Description of the Theory of Change .................................................. 97

Appendix I. Additional Information by Evaluation Criterion .................................... 99

Appendix J. Additional information on Immunization, Health System Strengthening and the Regional Experience .......................................................... 109

Appendix K. Agenda of Field Visit ............................................................................. 112

Appendix L. Main Documents Reviewed ................................................................ 115

Appendix M. Terms of Reference .............................................................................. 117


4

4

LIST OF ABBREVIATIONS

AEFI Adverse Events Following Immunization

CS Centers

CC Cold Chain

CBO Community Based Organization

DAC Development Assistance Committee

DHS2 District Health Information Software 2 (DHIS2)

EQ Evaluation Question

EPI Expanded Program on Immunization

FGD Focus Groups Discussion

GA Gender Analysis

GoSTP Government of Sao Tome and Principe

GVAP Global Vaccine Action Plan

HR Human Rights

HSS Health System Strengthening

HU Health Unit

M&E Monitoring and Evaluation

MoH Ministry of Health

OECD Organization for Economic Cooperation and Development

PAV Programa Alargado de Vacinação

PQS Performance, Quality and Safety

PS Health Units

SDD Solar Direct Drive

STP Sao Tome and Principe


5

5

ToC Theory of Change

ToR Terms of Reference

UNDAF United Nations Development Assistance Framework

VII Vaccine Independence Initiative

WHO World Health Organization

LIST OF FIGURES AND TABLES

Figure 1. Map of Sao Tome and Principe ...................................................................... 17

Figure 2. The Cold Chain .................................................................................................. 21

Figure 3. Theory of Change for the Cold Chain in Sao Tome and Principe ............. 22

Figure 4. Guidance on Vaccine Vial Monitor Source: Gavi Alliance. ....................... 27

Figure 5. Proportion of children completely vaccinated by placel of residence, gender and level of education of mothers in STP in 2017 ....................... 65

Figure 6. Evolution of VAR1 Vaccination Rate 2015-2018 ......................................... 102

Figure 7. Immunization Rates for VAR 1 in STP - Jan-July 2017 and 2019 ................. 103

Figure 8. Vaccination Rates for VAR2 in STP - Jan-July 2017 and 2019 .................... 103

Figure 9. Evolution of DTP3 Vaccination Rate 2015-2018........................................... 104

Figure 10. Evolution of RR1 Vaccination Rates – 2015-2018 ...................................... 104

Figure 11. Coverage of HUV 1st dose – 2018................................................................ 104

Figure 12.Proportion of children 12-23 months completely vaccinated in 2018 .... 105

Table 1. Districts that benefited from new refrigerators in Centers and Health Units ............................................................................................................................................. 18

Table 2. Evaluation Users and Uses ................................................................................. 23

Table 3. Evaluation Criteria and Questions ................................................................... 25

Table 4. Categories of Stakeholders Interviewed ......................................................... 30

Table 5. Number of participants in the mini focus groups (by district) ..................... 32

Table 6. Limitations of the evaluation and mitigation measures ................................ 32

Table 7. Availability of Penta and Measles by Health Units (2017 and 2019)* .......... 40

Table 8. Storage Capacity by District - 2017 and 2018 in temperature of 5oC ........ 41


6

6

Table 9. Results expected and level of achievement under effectiveness ............. 48

Table 10. Immunization Rates 2018 in Gabon, Guinea Equatorial and Sao Tome and Principe for selected vaccines ............................................................. 51

Table 11. Targets and number of CC equipment installed in STP .............................. 54

Table 12. Gavi´s support in co-financing vaccines in STP ........................................... 55

Table 13. Percentage of government contribution for paying for Gavi co-financed vaccination costs in STP (in US$) ..................................................................................... 55

Table 14. Vaccination costs foreseen for STP for the GAVI co-financed vaccines 2020-2024 (in US$) .............................................................................................................. 56

Table 15. Vaccine Costs per child from birth until 2 years old for 2018 when all doses even of the multi-dose vaccines are completely used ................................... 56

Table 16. Multi-dose vials used in STP for Children up to 2 years ................................ 57

Table 17. Rate of vaccine loss between 2008 and 2013 (excluding 2011)* in percentage ........................................................................................................................ 57

Table 18. Vaccine Costs per child from birth until 2 years old from multi-vial vaccines when only one dose is used ........................................................................... 57

Table 19. Compliance to UNEG evaluation norms and standards ............................ 79

Table 20. Stakeholders involved in the CC management and their roles ................ 81

Table 21. List of people interviewed ............................................................................... 82

Table 22. Vaccine Stock Management, STP, year 2017 ............................................ 100

Table 23. Vaccine Stock Management, STP, year 2018 ............................................ 100

Table 24. Vaccine Distribution by District for selected type - Part I - 2017 and 2018* ........................................................................................................................................... 102

Table 25. Vaccine Distribution by District for Selected type - Part II - 2017 and 2018* ........................................................................................................................................... 102

Table 26. Vaccination sessions by Health Unit and status of CC equipment - January-July - 2017 and 2019 ........................................................................................ 105

Info Box 1. Cleaning monthly guidance for CC equipment ..................................... 106

Info Box 2. Basic Information about management of TCW 40 SDD ......................... 107

Picture 1. Map of Health Units in Sao Tome and Principe benefited by the CCE ... 19

Picture 2. Presidential Palace of Sao Tome and Principe ........................................... 36

Picture 3. Health Unit of Lembá....................................................................................... 39


7

7

Picture 4. Airport Storage Room ...................................................................................... 49

Picture 5. Unicef´s representative in STP and Minister of Health at the occasion of UNICEF´s committment to fight Malaria in the country ............................................... 60

Picture 6. Focus Group of Rights holders in R.A.P.* ....................................................... 64

Picture Box 1. Deterioration of CC Equipment .............................................................. 99

Picture Box 2. Maintenance Instructions for CC Equipment and solar panels ....... 101

Picture Box 3. Temperature chart and open lid in CC equipment .......................... 106

Picture Box 4. Pictures of damaged condenser and rusty screw ............................ 106

Textbox 1. Quote from Nurse of LOBATA about the CC ............................................ 101

Textbox 2. Quote from Management Staff on Sustainability .................................... 107

Textbox 3. Quote from Nurse, Distribution Center on Sustainability ......................... 107

Textbox 4. Quote from father, 29 years, District of Cauê on Family Planning ........ 108

Textbox 5. Quote from Father, 36 years, District of Cauê on Family Planning ........ 108


8

8

Acknowledgments

This evaluation would not have been possible without the valuable contributions of the following individuals and organizations:

• Mariavittoria Ballota, Deputy Representative, UNICEF Sao Tome and Principe

• Dr Luis Bonfim, Health specialist, UNICEF Sao Tome and Principe

• Serge Ganivet, Immunization Supply Chain Management Specialist, UNICEF Regional Office for West and Central Africa

• Michele Tarsilla, Regional Evaluation Advisor, UNICEF Regional Office for West and Central Africa

• Edgar Neves, Minister of Health

• Ednilza Solange Barros, Coordinator the Expanded Program on Immunization, Ministry of Health

• Vladimir Sousa, National Cold Chain manager, Ministry of Health

• Thierry Vincent, Focal Point, Gavi Alliance

We also wish to express our gratitude to the health agents and caretakers interviewed in Sao Tome and Principe. Their contribution gave light and guidance to the report as they directly experience the impact of the Cold Chain and live the Expanded Program on Immunization in STP.


9

9

Executive Summary

BACKGROUND

This report presents the results of the evaluation of the Immunization Program in Sao Tome and Principe (STP) with a focus on the Cold Chain. The evaluation was commissioned by the UNICEF STP Country Office (CO) and financed by the CO and the UNICEF Regional Office for West and Central Africa and carried out between September and December 2019. STP has high levels of vaccination rates for the Region (95.4% for DTP3 in 2018) and this evaluation is intended to clarify why that is the case and how immunization links with Health System Strengthening (HSS) and the acquisition of new Cold Chain equipment. Despite the promising vaccination figures nationwide, there are children who remain unprotected and miss a second or third recommended vaccine dose. Why parents and/or caregivers fail to take children for the required shots was explored. The aim of the evaluation is to provide stakeholders the necessary information to make strategic decisions in immunization programming and financing in the future. The evaluation stakeholders include the following: UNICEF STP Health/Immunization Section Staff, other UNICEF Sections Staff, UN and other development partners, Government (Health and other line ministries) and NGOs/Community Based Organizations (CBOs) involved in immunization programs.

KEY INFORMATION

This evaluation pertains to the support that UNICEF and Gavi Alliance provided to the Sao Tome and Principe (GSTP) Government for the acquisition of new Cold Chain (CC) equipment with the aim of increasing storage capacity for vaccines in the country. More specifically, thirty-nine new Solar Direct Drive (SDD) refrigerators were purchased at the cost of US$296,143.62 between September 2017 and April 2018. Seven districts in the country were involved in this program: Água Grande, Cantagalo, Caue, Lembá, Lobata, Me-Zochi and the Autonomous Region of Principe (R.A.P.)

EVALUATION PURPOSE

The purpose of this evaluation is twofold: accountability and learning. First, the evaluation results will provide donors (vertical accountability) and expected beneficiaries (horizontal accountability) with information on the extent to which the program has been evaluated and achieved its objectives. Second, the evaluation aims to foster increased immunization in the country through more evidence-informed programming.

EVALUATION OBJECTIVES

The objectives of the evaluation are:

• To measure the effects of the national Cold Chain operating system in the country´s immunization coverage;


10

10

• To identify the ´vaccine chain´ shortcomings, constraints, and challenges that hinder current progress of the Vaccination Program towards universal coverage;

• To establish recommendations for immediate, medium and long-term actions, to achieve universal coverage and Health System Strengthening (HSS) based on the final conclusions and drawing from the experiences of other countries in the sub-region.

SCOPE

Thematic: the evaluation covered the following thematic areas:

• The expansion of the Immunization Program coverage with a special focus on the Cold Chain (including the installation and use of new solar-run refrigerators) ;

• The effectiveness of the vaccine procurement services and the CC equipment selection process;

• The efficiency of the current transportation system and the feasibility of the set-up of a possible storage mechanism in the African sub-region;

• The frequency of the vaccination services;

• The state of storage and availability of vaccines;

• The level of vaccine distribution and the related equity concerns;

• The modalities to further improve the country’s

immunization coverage and strengthen the national health system;

• The total costs of the supply chain (including the costs of immunizing a child);

Chronological: The evaluation covered the period between September 2017 and April 2018. However, it also considered follow-up activities carried out after this period, up to September 2019.

Geographical: the evaluation focused on the immunization and cold chain activities that took place in 7 of country’s districts, namely Água Grande, Cantagalo, Caué, Lembá, Lobata, Me-Zochi and Príncipe.

METHODOLOGY A Theory of Change (ToC), which was developed at the beginning of the evaluation, underpinned the overall Mixed Methods approach adopted by the evaluation team. Primary and secondary data resources were collected to strengthen the findings. The quantitative analysis of the country’s current immunization data added depth to the evaluation. The qualitative analysis consisted of the content analysis, coding, and data synthesis of the evaluation participants’ responses.

Overall, the evaluation team interviewed 45 individuals (28 men and 17 women. in addition, 4 focus group discussions were organized (total number of participants: 20 of whom 4 men and 16 women). Thanks to the fact that all the 20 participants of the focus groups represented legitimate rights holders located in


11

11

various regions of the country, the evaluation team was able to capture a wide variety of perspectives amongst beneficiaries encountered in the Health Units. Triangulation of data was also used to increase the reliability and validity of the findings. Furthermore, multiple sources were accessed to substantiate the evaluation data, and to increase reliability as themes developed during the analysis process.

KEY FINDINGS AND CONCLUSIONS (BY CRITERION) Relevance: UNICEF various divisions worked closely with the Gavi Alliance and the Government of Sao Tome and Principe to meet the country’s manifested need for new Cold Chain equipment (CCE). Such need was also highlighted in a World Health Organization (WHO) study conducted during the planning of the 2016-2020 Expanded Program on Immunization (EPI). More in particular, the CCE existing in country was outdated and local authorities found particularly challenging to repair the faulty pieces. Unfortunately, given that the grids were not stainless steel resistant, the purchased equipment was not adequate for withstanding the corrosion caused by the seaside air of STP (Paragraphs 3, 4, 5, 6).

Effectiveness: The evaluation demonstrated that the CC investment contributed to increasing vaccine availability in Health Units across the country, except for bVPO, because the new equipment allowed for increased and steady storage capacity. That, in turn, translated into less frequent visits by Health Unit (HU)

staff to the District Centers to collect vaccines (now readily available at their own sites), which then became an unprecedented time-saving opportunity. The secondary data analysis conducted by the evaluation team confirmed that vaccination rates have increased or kept stable between 2017 and 2018. For 2019, data suggested that such trends would persist. However, higher or stable rates ought not be attributed directly or exclusively to the newly introduced CCE, since various other factors affect vaccination rates. In some Health Units, for instance, the new CCE brought an increase in vaccination sessions, and in others it did not. Likewise, the purchase of vehicles for home visits, which was made possible through Gavi funding, seems to have contributed to bring vaccination rates up again.

CC also contributed to the quality of health care, but along with another host of factors. While no conclusive evidence was found that could attest the contribution the link between improvements in health care delivery and the particular type of equipment, installation method, or verification system in use at any particular HU, the evaluation team observed that the new equipment generally enabled health workers to complete tasks more quickly.

The implementation of innovative measures to reach unvaccinated children was also beneficial. These included phone calls to caretakers and relatives, to ensure children are brought in for critical follow-up vaccines. Furthermore, the Programme attained a few positive unexpected outcomes. The CC


12

12

equipment, for instance, improved the work environment of health agents, in that they could attend to their patients on a more continuous basis without having to leave the health centers to retrieve vaccines in case of shortage or to leave the workplace during power blackouts to ensure vaccines were stored at the right temperature.

For the sake of clarity and intellectual honesty, not all the positive effects highlighted above are directly and exclusively attributable to the introduction of new CCE under the egis of the UNICEF- STP Country Programme. While it is widely recognized that STP performs better than other countries in the Region in many areas (vaccination rates, Cold Chain management, and outreach services), it is also true that a plethora of actors contributed to the improvement of immunization nationwide. In 2015-2016, for instance, GAVI made $3 million available for the implementation of the 5-year Health System Strengthening (HSS) project, which used vaccination as the entry point. This project, too, provided for the distribution of new refrigerators to all districts in STP.

However, the added value of CCE in a country like STP, which already features high vaccination coverage rates, could only reach a certain point. What really matters to attain full vaccine coverage in such a context is Reaching the goal of Universal Health Coverage is the combination of four key strategies: (i) to improve the quality of vaccination data, which would also imply the set-up and implementation of an effective information system; (ii) to analyze

national, provincial, and district level data in real-time and on a systematic basis; (iii) to roll-out capacity building programs aimed to improve the quality of health services (this would include a focus on equipping health workers with more knowledge of solar panels care and maintenance, which still appears quite low ; and (iv) to monitor Health staff performance on a more regular basis and to improve HU working conditions.

Based on the analysis of the collected data and a series of follow-up exchanges with a selected number of key informants, the evaluation team also identified three a few good practices and lessons learned. First, the relevance of using of a district-level analysis and coverage plan similar to what is currently implemented in Senegal. Second, the positive contribution that an interactive dialogue with civil society has on the uptake of vaccination, as it is currently the case in Cameroon. Third and last, the criticality of selecting the material of solar power refrigerators (e.g. it should be resistant to the corrosion caused by seaside air).

Efficiency: Transportation costs in STP are considerably high, due to the country’s specific location and the relatively small quantity of the supplies ordered. However, the rather limited availability and reliability of airlines within the Region hinders, to this date, the collective purchase of vaccines and the set-up of a regional warehouse. In addition, the three countries considered eligible to establish a common warehouse (STP, Gabon, and Guinea Equatorial) have very different pharmaceutical


13

13

infrastructures, which pose potential high coordination and transaction costs.

Acquiring the CC is a standard procedure which all involved parties are used to and are generally satisfied with. The fact that UNICEF and the other partners have been part of a larger partnership, where overhead costs are paid in advance, inevitably entailed some savings. The quantity of the human resources employed in the program implementation was also adequate, if compared to the magnitude of the actions planned. However, the quality of the human resources employed is mixed. Several healthcare workers (nurses), for instance, revealed knowledge gaps, especially in the area of CC equipment management.

Similarly, the HU infra-structure was not always of the best quality: it generally did not meet the minimum standards for installation. Delivery, too, was not always timely: while most solar panels and equipment were installed according to schedule, three refrigerators have not been used yet (their packaging is still intact) after two years since their delivery.

Sustainability: Sustainability is key to the continuation of the CC as well as of the whole vaccine management process in STP. The graduation of STP from the category of Least Developed Countries is making traditional partners withdraw their support. Although the Government of Sao Tome and Principe (GoSTP) is actively looking for alternative sources of funding (e.g. new taxes, new partners), vaccination is only one more element of dispute among the various demands of the government.

Whil, the National Plan for Health Development 2017-2021 confirmed the GoSTP commitment to finance vaccines and related consumables according to the GAVI guidelines (GoSTP financial contribution to the purchase of Gavi co-financed vaccines from 6% in 2013 to 18% in 2017), the GoSTP is far from full funding its vaccination programme. That is all the more apparent given that, as of 2023, the country is expected to invest US$155,600.00 every year in the purchase of GAVI co-financed vaccines (Yellow Fever, Pentavalent, PCV, Rota), excluding additional supplies involved. As a result, the lack of full funding represents an area of concern over the sustainability of vaccine supply in the country. As a reminder, a child in STP is generally vaccinated until the age of 2 years 20 times with different vaccines. Twenty vaccines for one child cost between US$ 30.97 and US$57.74. However, for the vaccination of a child, some other costs need to be envisaged: CC management, salaries, transportation to the Health Center, medication, the costs of parent`s time away from work, infrastructure, fuel, vehicles, and loss of vaccines.

Equity: the evaluation team identified Important obstacles caretakers or parents from vaccinating their children. These include, but are not limited to, the following a) difficult access to Health Units; b) excessively long duration of visits to the Health Units; c) little incentives to go to the Health Units; d) adverse events following immunization (AEFI) in which children may cry and disrupt the family routine.


14

14

In addition, important issues related to gender equity were discovered, not in relation to the CC alone, but in relation to a smaller group of males who may pose obstacles to vaccination due to the concern that their spouses may be receiving family planning advice and contraceptive drugs without their consent. Male partners’ perceptions of the interactions between their wives and family planning professionals seem to be more relevant than what was previously thought and, therefore, deserves greater attention in the future.

Other factors that seem to have hindered the successful delivery of immunization services include the lack of integrated information systems, the paucity of proper information that mothers receive from providers and, lastly, the poor quality of treatment, which does not encourage caregivers to return.

CONCLUSIONS

C1: UNICEF, Gavi, and WHO have been critical partners in support of the expansion of immunization in STP through funding, procurement and technical assistance, such as in the case of CCE and beyond. Without such resources, procurement systems and guidance on international good practices and data systems, STP would have progressed at a decreased pace. With respect to the CCE selection, there was close alignment among the three partners. However, a related gap was identified: the lack of proposed stainless-steel resistant materials, which would have proven necessary in a country like STP, which is heavily exposed to the corrosive effects of seaside air.

C2: The CC investment has contributed to increased vaccine availability at a local level and has freed up time for health staff, who conducted fewer visits to the district vaccines collection points. Moreover, the supply chain has improved through vaccine availability. However, this has not impacted directly the quality of health care or the vaccination rates in a context that featured already high vaccination rates.

C3: The programme was efficient in acquiring the CC equipment, but less efficient in the roll-out of the installation process, mainly due to infrastructure issues and inadequate quantity and quality of training provided to nurses that were responsible for CCE maintenance. Maintenance efficiency was a key area of concern. The equipment in one of the country’s District had deteriorated because of the corrosive effects of seaside air and lack of equipment upkeep.

C4: As the transportation costs to STP are high. Discussions were held to foster collective purchase and the set-up of a Regional Warehouse. However, these ideas posed great challenges in terms of airline connections between STP and its “neighboring” countries in the Region and did not resonate with the differences in history, culture, and pharmaceutical structures among the countries.

C5: Although the GoSTP is making a great effort to reach all children via home visits and phone calls to the mothers, the collection and harmonization of data across the country still represent a great challenge to identify unvaccinated


15

15

children. The data are crucial to develop a micro-planning strategy to inform and guide the nurses and other medical staff.

C6: The funding for vaccines and related supplies in STP face potential danger if Gavi withdraws by 2023.

C7: Critical obstacles remain for caretakers to vaccinate their children and increase immunization in the country. Accessibility, quality of information and health care, and adverse associations between vaccination and family planning were identified.

C 8: The Theory of Change (ToC) behind CC is complex, and the Government manages the CC well. However, procurement and maintenance are challenges under construction.

LESSONS LEARNED • The installation of new CC

equipment may facilitate access to readily available vaccines and help improve the quality of the work environment.

• In a country with high vaccination rates such as STP, where availability of vaccines is not at a critical level, the impact of new CC equipment may be limited.

• The linkages between vaccination and family planning could bring undesired effects that deserve further investigation.

RECOMMENDATIONS These recommendations were validated during several exchanges held between the evaluation team and UNICEF Country Office staff as well as other stakeholders. Each

recommendation was rigorously based on findings from the evaluation process and the dialogue with the 65 participants who helped to identify possible opportunities for the right timing for action.

Strategic:

For the GoSTP and Unicef:

R1: Explore alternative funding mechanisms for vaccination in the country in conjunction with advocacy for immunization.

R2: Promote high-level advocacy for immunization in the country and promote STP as a source of know-how for vaccination coverage to other countries of the region (which should also include collaborative learning with other countries in such areas as data analysis, district planning and dialogue with civil society).

R3: Improve quality of data and implement an effective information system with Monitoring and Evaluation mechanisms to help reach the remaining unvaccinated children.

Operational

For the GoSTP and UNICEF:

R4: Reinforce HH staff capacity on data analysis and micro-planning, by District and Health Unit, to reach unimmunized children through the roll-out of training and coaching programs.

R5: Increase communication campaigns and education for health in the area of immunization and develop direct means of communication with families (e.g. exploring new technologies).

R6: Conduct routine maintenance of the new CC equipment to counteract


16

16

the sea air corrosive effects. This would include fraft an appropriate maintenance plan, which may include the hiring of a maintenance company for the solar panels and install the three remaining CC equipment. Future purchases should be limited to stainless steel materials resistant corrosion caused by seaside air.

R7: Develop a CC management protocol which includes forms for daily, weekly and monthly maintenance, location of equipment and frequency of vaccine collection. Explore developing a quality management according to ISO 9001:2015.

R8: Promote training in CCE management, managerial skills, and quality of health care.

1. Evaluation Object

This evaluation refers to the support of UNICEF and Gavi Alliance given to the Government of Sao Tome and Principe for the acquisition of new Cold Chain equipment with the aim of increasing storage capacity for vaccines in the country within the broader context of the Immunization Program and Health System Strengthening. Between September 2017 and April 2018, thirty-nine new Solar Direct Drive (SDD) refrigerators were purchased at the cost of US$ 296,143. 62. Thirty five out of 39 refrigerators with the solar panels were installed in the original planned regions. One out of the 39 refrigerators are now being used in the Maternity Ward at the Central Hospital in Sao Tome. Three refrigerators have not been installed and remain in packages. The equipment of the central remote control of the temperatures for the EPI Distribution center has not reached the country yet. All the country’s 7 districts benefited from the project, namely Água Grande, Cantagalo, Caue, Lembá, Lobata, Me-Zochi and the Autonomous Region of Principe (R.A.P.)

Figure 1. Map of Sao Tome and Principe

The direct beneficiaries of the new equipment were the District Centers and Health Units, as well as caretakers who benefited from having more vaccines available in the Health Units throughout the country. An estimate of 5,448 children (PAV, 2018) were potential beneficiaries from the CC equipment. STP is a country with high levels of vaccination rates for the Region (95% for DTP3 in 2018, 95% for PAV in 2018). The importance of this support for UNICEF is not restricted to the CC, but this evaluation helps to

Source: http://www.megatimes.com.br/2018/04/sao-tome-e-principe.html


18

18

trace how this intervention fits within the broader framework of UNICEF´s contribution to the Expanded Program on Immunization (EPI) and Health System Strengthening (HSS) in the country.

The main stakeholders involved in the intervention were: UNICEF, Gavi Alliance, Ministry of Health (MoH) central officials, Delegates at the District level, Nurses at the Health Units, and caretakers (generally mothers) and children as rights holders. Beyond these key stakeholders, UNICEF works very closely with the World Health Organization, a key partner. The target group of this evaluation was all the children of Sao Tome and Principe, considering the aim of expanding immunization coverage to the greatest possible extent.

When it comes to equity, data from the 2017 National Survey of Vaccine Coverage (Firmin, 2017) show that lower coverage of vaccine vary according to the type of vaccine in each District. The lowest coverage for BCG (for Tuberculosis) and DTP-HepB-Hib – 3rd dose (Diphtheria-Tetanus Toxoids-Pertussis Vaccine, Hepatitis B and Haemophilus influenzae type B Vaccine) were for the District of Água Grande (92.9% and 88.5%, respectively), Cantagalo had the lowest coverage for VPO (84.6% for 3rd dose) while Lembá had the lowest for ROTA (17.3% for the 3rd dose), Lobata for PCV (61.3% for the 3rd dose) and Principe for CIC (79.9%). This demonstrates an uneven progress across regions.

According to the same Survey, the reasons of non-vaccination of children between 12 and 23 months were reported to be lack of awareness (39%), lack of motivation (22%), and the obstacles involved (38%) which may be the location of Health Units. These reasons initially surveyed will be better discussed under the findings sessions, where the Evaluation Questions (EQ) will be answered and discussed in juxtaposition to previous evidence of the reasons of non-vaccination in the country. Table 1: the locations of newly installed equipment:

Table 1. Districts that benefited from new refrigerators in Centers and Health Units

District Centre/Health Post Água grande

Centre, PS Água Arroz, PS Sao Marçal, PS Pantufo, PS Vila Fernanda, PS Madre Deus, PS Praia Gamboa

Mé-Zóchi PS Bombom, PS C.Grande, PS Madalena, CS Trindade, PS Monte Café Lobata PS Desejada, PS S.Amaro, PS Conde, PS Micolo, CS Guadalupe Lembá CS Neves, PS Santa Catarina Cauê CS Angolares, PS R. Peixe, PS Porto Alegre Cantagalo PS Voz d'America, PS R.Santana, PS Pinheira Roça, PS R.Afonso I e II, PS

Santa Cecilia, CS Água Izé R.A.P S.S.R, PS Picão, PS Nova Estrela, PS Aeroporto, PS Sundy, Porto Real


19

19

2. Overview of the intervention Immunization is the key to promoting healthy lives, especially for children who are prone to a number of infectious diseases aggravated by poor water and sanitation conditions associated with developing countries. Vaccines are one of the most effective ways of preventing diseases caused by Polio, Tetanus, Diphtheria, Whooping Cough (Pertussis), HiB, Hepatitis B, Measles, Mumps, Rubella, Chickenpox (Varicella), Pneumococcal and Rotavirus microorganisms. Infections may result in fatal deaths or leave serious health consequences. In a few countries with sufficient advocacy for vaccination programs, regulated vaccinations may seem simple and logical. In fragile economies, which may not have the necessary resources or the capacity to manage the whole vaccination process, these life-saving vaccinations often require additional actions to promote public awareness, to maintain a quality vaccination program.

About six thousand newborn children of various backgrounds are potential beneficiaries of this initiative, as Health Units in STP have the mandate to serve every child in the country, regardless of social stature. The operation benefited all 7 districts of the country: Água Grande, Cantagalo, Caue, Lembá, Lobata, Me-Zochi and the Autonomous Region of Principe (R.A.P.), and more specifically, thirty-six Health Units (3 devices remain uninstalled). See figure below for map of the locations.

Picture 1. Map of Health Units in Sao Tome and Principe benefited by the CCE

Sao Tome and Principe is a country of medium Human Development Index, 0.609 and registers at 137th out of 189 countries and territories. It ranks higher than the average of Sub-Saharan Africa (0.541) and other countries of its own Region (such as Guinea Equatorial with an HDI of 0.588), however, lower than other Island countries such as Cape Verde (0.651). STP has grown in HDI from 0.437 in 1990 to 0.609 in 2018, which is


20

20

a significant growth (UNDP, 2019). In fact, the country is in the process of graduating from the status of Least Developed Country by December 2024.

STP has made significant progress from 1990 to 2019 in different areas. Data from the WHO evidence that in 1990, under-five mortality rate was 108.6 per 1,000.00 live births, while in 2017 it was 32.4. Data from 2014 show that the Proportion of households owning at least one insecticide-treated net (ITN) (%) was 78% and the Non-Communicable Diseases (NDCs) were estimated to account for 55% of all deaths (WHO, 2014), which reveals that contagious diseases are under control with the exception of Malaria. Concurrently, the proportion of people practicing open defecation was 54% (MICS 2014) which demonstrates many challenges remain in the area of public health.

São Tomé and Príncipe experience higher vaccine coverage than other countries of the Region. Through the Expanded Program on Immunization (EPI), vaccination is provided in 39 health facilities in different districts of the country along with a number of other health services. However, if one considers vaccination is high in STP (97% for the first dose of DTP-Hib-HepB and 95% for the first dose of Measles/Rubella – MR vaccine in 2018), the coverage of second and third doses are lower (95% for the third dose of DTP-Hib-HepB and 74% for the second dose of MR). The focus of the acquisition of the Cold Chain equipment was to increase second and third doses of crucial vaccines to ensure higher levels of protection for children, and to enable the introduction of new vaccines in the country.

In May 2012, the Global Vaccine Action Plan (GVAP) – 2011-2020 was endorsed by the World Health Assembly. The Plan was led by a number of organizations, such as the Bill and Melinda Gates Foundation, the World Health Organization (WHO), UNICEF and the United States National Institute of Allergies and Infectious Diseases. Gavi Alliance was founded by the Bill and Melinda Gates Foundation with the mission of ´Saving children’s lives and protecting people’s health by increasing equitable use of vaccines in lower-income countries´. Since then, it has been working in close partnership with governments from the poorest countries1 , UNICEF and WHO.

In 2019, STP hosted the African Vaccination Week and received several countries for the launch of the campaign. However, even though STP does have high vaccination rates for the Central African Region and the entire African Region, little recognition has been received in terms of the merit involved in promoting such rates. There is a tendency to explain STP’s results due to the population size of the country which is rather small in comparison to its peers (206,801 inhabitants in 2019). This evaluation fits into the context of using the CC management as a proxy for examining broader questions of vaccination management in the country.

In Sao Tomé and Príncipe, Gavi Alliance has supported the government since its inception in 2011 with different types of assistance. More recently, between 1 The criteria used by Gavi Alliance is a Gross National Income (GNI) per capita which is less than or equal to US$ 1,580 over the past three years, according to the World Bank.


21

21

September 2017 and April 2018, Gavi Alliance provided funding through UNICEF to purchase and install 39 new Solar Direct Drive (SDD) refrigerators: 7 TCW 2000 SDD and 32 TCW 42 SDD) in the 7 different districts in the country. Before, the country relied only on 27 refrigerators RCW 42 EK and 5 VESTFROST. These devices are crucial for the management of vaccines in the country, which have to be stored in a temperature between 2oC and 8oC. The cycle of managing vaccines in the appropriate temperature is called Cold Chain (CC).

According to WHO, the Cold Chain involves a complex logistics of ensuring vaccines reach children at the correct temperature, hence, with quality for providing the desired immunization. Several checkpoints need to be reinforced: 1) from the manufacturer to the airport, 2) to the central store, 3) to the district, 4) to the Health Unit and finally, 5) to the children. Figure 2 illustrates the procedures, transportation and stakeholders involved in the distribution process.

Figure 2. The Cold Chain

Source: PATH/WHO.

In Sao Tomé and Príncipe, the CC transportation to STP is assured by airplanes: vaccines are transported by land to the EPI (Expanded Program on Immunization) Distribution Center and then distributed to districts every month. From the districts, vaccines are sent to health units once or twice a month. The logistics involved need to ensure that the temperature is correct, the quality of the vaccines is maintained and delivered on time. During this process, a number of stakeholders involved, Gavi Alliance, UNICEF, government officials, and partner companies must be considered during the evaluation process.

Theory of Change

Based on desk review, interviews, and discussion within the evaluation team, a Theory of Change (ToC) of the intervention was reconstructed (See Figure 3). The ToC was initially drafted during the inception phase and helped with broadening the perspective of all the factors involved in the CC management process.


22

22

Figure 3. Theory of Change for the Cold Chain in Sao Tome and Principe

The major activities and the assumptions involved in the ToC for the Cold Chain are described in Appendix G. This ToC drafted in the inception phase highlights critical issues under consideration and will be discussed throughout the report. More specifically, the ToC enabled identification of the stakeholders’ preparedness to carry out the procurement of the Cold Chain. This preparedness included a number of assumptions (availability of data, well-functioning information systems, staff preparedness, sound information analysis, availability of resources and legislation in place). The communication among the various stakeholders was considered during data collection when the evaluation participants were interviewed. During the data collection, the ToC was used to reflect a number of factors involved in the management of the CC equipment, such as the use of a generator of an independent thermometer to check the temperatures in the districts and Health Units (which is not in place yet).

During data collection and data analysis, the ToC was used by the evaluators to identify key steps around the proper management of the CC equipment and actual implementation (use of management protocols, record on vaccine arrival and transportation, storage of vaccines in the right temperature, regular maintenance of equipment and proper daily record of temperatures). The greatest value added behind this ToC was to allow the evaluation team a broad overview of the whole process and the factors that were considered.


23

23

3. Evaluation Purpose This evaluation has two purposes: accountability and learning. This evaluation will provide the donor (vertical accountability) and the expected beneficiaries (horizontal accountability) solid evidence on the extent to which the Cold Chain put in place as part of the EPI attained the projected objectives. This evaluation is expected not only to inform evaluators and stakeholders of the Cold Chain implementation strategies and EPI more broadly in the years to come, but to shed light on the potential corrective actions that may be explored further in the future.

More specifically, this evaluation is expected to generate recommendations that will help UNICEF Country Immunization/Health program staff as well as other in-country partners to adapt the implementation of Cold Chain mechanisms and EPI strategies to the emerging and country-specific needs in this area. The target audience for this evaluation are represented in Table 2.

Table 2. Evaluation Users and Uses

Evaluation Users Evaluation Uses UNICEF health staff in STP and Deputy Representative

UNICEF will have the chance of linking the Cold Chain with Health System Strengthening and identifying lessons which could help shape future health agenda, possibly share with other countries and give insights to sustainability.

Other UNICEF Staff in STP

Other UNICEF Staff will be able to identify possible linkages with the Cold Chain and immunization program as a way of achieving Country Program Goal 1 under United Nations Development Assistance Framework (UNDAF).

UN and other development partners

The evaluation will help partners to reflect upon their practice, the agenda on immunization and linkages with the broader health system. The evaluation will also engage with common UNDAF goals and point at possible joined action for the future.

Government The evaluation will help to move forward in the goals set in the National Health Plan and help review the EPI for the period 2021-2024.

Civil Society Organizations (CSOs)

This is a potential user, as the country office does not directly engage with CSOs in the intervention. However, there is potential engagement for increasing immunization coverage and the evaluation might be a tool for broadening the scope of partnership with this sector as outlined by the Country Office report of 2018.

4. Evaluation Objectives The objectives of the evaluation, as stated in the Terms of Reference, are:

⇒ To measure the effects of the national Cold Chain operating system on the country´s immunization coverage;

⇒ To identify the shortcomings of the ́ vaccine chain´, the constraints and challenges which restrict current progress of the Vaccination Program towards universal coverage;


24

24

⇒ To establish recommendations for immediate, medium and long-term actions to achieve universal coverage and Health System Strengthening (HSS) based on the final conclusions and drawing on the experiences of other countries in the sub-region.

5. Evaluation Scope

5.1. Thematic scope

The evaluation focused on:

⇒ The Immunization Program with a focus on its CC and its key challenges for expanding coverage;

⇒ The effectiveness of the vaccine procurement services, including the CC equipment selection process;

⇒ The efficiency of the current transport system and exploration of the feasibility of the set-up of a possible storage mechanism in the Africa sub-region;

⇒ The frequency of the vaccination services; ⇒ The degree of availability and conservation of vaccines; ⇒ The extent to which vaccine distribution reflects equity concerns; ⇒ The changes that occurred in immunization coverage, and how that contributes

to HSS; ⇒ The key features of the overall immunization service system, including vaccine

procurement, transportation, storage and distribution mechanisms; and ⇒ The total costs of the supply chain (including the costs of immunizing a child).

5.2. Chronological scope

The evaluation covered the implementation period between September 2017 and April 2018. However, activities carried out after this period, through to September 2019, were also evaluated.

5.3. Geographical scope

The evaluation covered the installation and use of new solar panels refrigerators in each of the country’s 7 districts (Água Grande, Cantagalo, Caué, Lembá, Lobata, Me-Zochi e Príncipe).

6. Evaluation Criteria To fulfil the two envisioned purposes, the evaluation was guided by 5 different criteria: four of the five evaluation criteria suggested by the Development Assistance Committee of the Organization for Economic Cooperation and Development (OECD/DAC) (Relevance, Efficiency, Effectiveness and Sustainability) and one


25

25

additional criterion, pertaining to Gender, Equity and Human Rights. The Impact criterion was not considered. Besides not being included in the ToR, impact was left out as it would have been difficult to attribute any of the program long-terms results (e.g. the well-being of children) to the Cold Chain component of the Immunization program, which rests on a complex intervention model.

Evaluation Questions

The key evaluation questions included in the ToR are presented below (See Table 3). Adjustments were made to the evaluation questions during the inception phase. Some questions were either placed under a different criterion or merged with other questions. Once the list of evaluation questions was narrowed down, the evaluation team developed an evaluation matrix to address the 9 key questions more systematically (See Appendix C).

Table 3. Evaluation Criteria and Questions

Criteria Evaluation Questions

Rele

vanc

e

• To what extent have the vaccine Procurement Services rendered by UNICEF (the only operator in this area nationwide) met the country´s needs, especially during the transition period towards Vaccine Independence Initiative (VII), defined by UNICEF Supply Division jointly with Gavi Alliance?

Effe

ctiv

enes

s

• To what extent have CCE investment contributed to improving immunization service delivery (e.g. changes in frequency on immunization sessions, changes in frequency of vaccine collection/distribution, averted missed opportunities due to a better vaccine availability)?

• What are the factors (e.g. equipment types and/or model, health care workers conducting preventive maintenance tasks, installation method(s), verification systems, service delivery models etc.) that have facilitated and hindered healthcare delivery?

• What are the unexpected outcomes (positive and negative) produced with the scope of the procurement and distribution of vaccines?

Effic

ienc

y

• With respect to the transport mechanisms put in place, were there other alternative strategies that could have been put in place to achieve the same level of results but at a lesser cost, especially in light of the country´s current transition towards Vaccine Independence?

• To what extent were financial resources, human resources and supplies: a) Sufficient (quantity); b) Adequate (quality);


26

26

c) Distributed/deployed in a timely manner? • To what extent could an existing central warehouse at the regional

level (WCAR) produce transportation benefits for the whole country?

Sust

aina

bilit

y • What are the actual prospects for the government to be able to manage the CC after UNICEF´s support and how did UNICEF incorporate measures for the activities funded by UNICEF to be continued without UNICEF support in the future?

Gen

der,

Equi

ty

and

Hum

an

Righ

ts

• To what extent were Gender, Human Rights and Equity Principles dully integrated in the design and delivery of the Cold Chain Interventions? To what extent did the EPI program tackle the barriers that prevent girls´ and women´s access to the services in the targeted communities?

As previously stated, the evaluation served as an entry point for a broader reflection on the role of the Cold Chain towards Health System Strengthening (HSS) and, put forward strategic recommendations on how to bring about systemic changes in the country’s health system.

7. Methodology This evaluation team opted for a predominantly qualitative design but also relied on the analysis of secondary quantitative data to complement the findings stemming from the qualitative data collected. The evaluation was highly participatory in that it sought and combined the views of the different stakeholders, including government officials, UNICEF, implementation partners and rights holders. In line with the equity-based and humanitarian work carried out by UNICEF, the evaluation aimed to identify the reasons why rights stakeholders might not be accessing the second and third doses of vaccines recommended.

7.1. Methods for data collection Triangulation of data and resources was implemented to ensure that the findings and related conclusions yielded solid and credible results. Triangulation is key to allowing multiple sources to converge and yield reliable, accurate evaluation results. In addition, international quality instruments were also used by the evaluation team to ensure the quality and rigor of the evaluation process: the UNEG Checklist on Quality Evaluation Reports the GEROS Quality Assessment Criteria (followed closely by the evaluation team, item by item) and the UNEG Guide on the Integration of Gender


27

27

Equality and Human Rights into Evaluation. The following data collection methods were used in the process:

Desk Review The desk review concerned 70 programmatic documents, national strategies and other relevant contextual information. The primary objective was to provide the evaluation team with the necessary background of the evaluation and to prepare for the field work. This crucial step was taken during the inception phase, and throughout the evaluation process as new documents emerged during the field work and the drafting of the evaluation report. The documents were analyzed, and key information was analyzed to build a narrative of program context, challenges involved, activities carried out and results achieved, including the Theory of Change drafted in the inception phase. Check list for Health Units Drawing on the technical knowledge of the immunization specialist on the evaluation team, a checklist was developed to verify the availability and status of key points/issues at each Health Unit. The instrument consisted of 4 items: 1) Temperature chart recorded on the fridge or nearby; 2) Recorded Temperature: a log should be kept of the refrigerator’s temperature, meant to be measured twice daily; 3) Vaccine expiration dates: the dates in a sample of the boxes inside the arks were checked; and 4) Vaccine Vial Monitor: a technology designed to report whether vaccine are good. For the sake of precision, the lighter the inner square compared to the outer

circle, the better the performance against the specific item in question (See Figure 5 for details). The checklist was used during the visits to a sample of Health Units and the related findings will be presented in the Findings section.

Figure 4. Guidance on Vaccine Vial Monitor Source: Gavi Alliance.

Focus Groups The focus groups’ discussions (FGD) were important to identify key problems that prevented caretakers’ direct access to Health Units to vaccinate their children. They helped pinpoint key challenges, including on the equity front, which will be discussed under Evaluation Question 9. FGD involved groups of rights holders, mainly mothers and fathers and, in some cases, other caretakers, who also provided some unique perspectives on the program. FGD helped highlight the particular challenges involved in each one of the seven districts included in the sample. Four FGD were carried out with stakeholders in the districts of Caue, Lembá, Me-Zochi and R.A.P. In total, 20 stakeholders took part in the focus groups and were able to describe challenges involved in vaccinating their


28

28

children and suggest how to improve healthcare services and increase vaccination rates in their communities. Individual semi-structured interviews The stakeholders’ perspectives were collected through individual semi-structured interviews. Approaching diverse stakeholders on their own helped increase comfort to express their views. The semi-structured interviews were key in generating data and enriching the learning process. As people express their views, they reflect upon their own practice and this increases the learning component of the evaluation. A political component emerged, whereby the evaluation helped to foster dialogue among the various stakeholders and provoked critical thinking on the crucial challenges that emerged during the data collection. UNICEF´s personnel, government officials from the various ministries, other organizations involved, and influencers were interviewed individually (in pairs or small groups) to report on their perspectives concerning the initiative. The respondents’ feedback was crucial to build a narrative of the challenges and achievements behind the Cold Chain initiative and the country’s immunization process.

7.2. Data Analysis The evaluation team conducted analysis of both quantitative and qualitative analysis.

1. The evaluation has combined a number of methods of analysis: 1) Identification of key themes and contents in the desk review; 2) Descriptive statistics when analyzing the immunization data and lastly, 3) a standard method used by the senior evaluation consultant when dealing with qualitative data which is explained below and summarizes the evaluation process. First review of individual interviews and data from focus groups: the text from the interviews and focus groups, which was recorded, was initially reviewed (typos were corrected, and missing text duly added). At this stage some preliminary patterns were identified, and quotes were highlighted in the transcripts. Data was organized according to the evaluation questions and indicators of the evaluation matrix. This step helped the evaluator to review the key points that emerged. The insights and patterns identified were reported in an ´evaluation diary` with the primary objective to build an incipient body of key evaluation messages. These patterns identified during this first phase were gradually moved to the findings sections and started signaling possible conclusions and recommendations to be further elaborated through the analytical process.

2. Organization of report by evaluation question and indicators/criteria: the structure

of the report was set according to the evaluation questions and indicators/criteria presented in the inception report. During this stage, key patterns and insights from step 1 were already placed there as bullet points to be further developed. The


29

29

sections on “Lessons learned” and “Recommendations” were also set for allowing the evaluator not to lose any insights and data identified in this area.

3. Insertion of qualitative data by evaluation question: as the structure had been set

and key points identified, relevant parts of the interviews and focus groups were placed in the report to be used as quotes to support the arguments and key ideas already identified in Steps 1 and 2. Contrasting views were presented to give a more accurate picture of what was found during the fieldwork. Along with the qualitative analysis of indicators/criteria, there was also a quantitative analysis in order to complement the arguments around the key findings presented.

After the field visit, some questions remained unanswered due to the lack of documentation and unavailability of some of the key participants for the interviews. As the analysis progressed, other missing points were also identified. In order to fill in this gap, further documentation was requested from UNICEF and other stakeholders and 7 further stakeholders were interviewed. During the writing of the evaluation report, the evaluation team members were in close contact to discuss the data found and co-construct the narrative emerging from new information that emerged. Weekly meetings were held among the team members to keep track of progress and ensure that the report would come together in a collaborative effort.

Apart from the initial field visit by the evaluation team leader and the national consultant during the period of Oct 15th-22nd, a second field visit was carried out by the Thematic Expert from Nov 27th to 29th. Both visits were crucial for collecting data and identifying challenges and achievements. After the draft report, additional data were collected with the assistance of the national consultant.

Analysis of Secondary Data on Immunization in the country

Descriptive statistics was used to carry out the analysis of data from the population in question, considering differences between target groups by location, economic status, and education level. The development goal of the intervention was to help increase immunization rates in the country, especially for second and third doses. With that in mind, the data collection and data analysis of secondary data on immunization in the country was key to showcase trends and to explore, in conjunction with qualitative data yielded from interviews, a possible link between the installation of the new Cold Chain equipment and the changes in the country’s immunization rates. The data for Immunization, considered for this evaluation stemmed from the Government of STP, namely the EPI Distribution Center. Studies have evidenced little discrepancy between different sources of vaccination in the country.

7.3. Data Sources and Sampling strategy The major data sources were described in item 7.1. under evaluation design. However, it is important to emphasize how the evaluation team considered the perspective of all stakeholders involved in the evaluation. A total of 65 people was interviewed or


30

30

were involved in focus group discussions (FGD). Contrasts were identified during the triangulation process and a harmonized view was presented in the report, which reflects with a good degree of confidence (considering the number and variety of stakeholders) what is happening on the ground.

More specifically, Triangulation of data was used throughout data collection and analysis to reinforce the rigor and quality of the data in alignment to the evaluation purpose. Various sources of primary and secondary data (interviewees, program documents, statistical reports) were used to yield solid evidence of the findings. Two examples illustrate the approach taken: a) There were nurses who reported greater availability of vaccines and those who had reported no impact on availability of vaccines. Hard data on vaccine availability was used to check the information and it was found that in some cases, there was greater availability of vaccine and in others, there was not, showing then, a limited impact b) In one of the health units, one nurse reported that there were cases when the father did not like the mother to take the children to vaccinate. There were also cases where mothers did not report any obstacles posed by their husbands to vaccinate their children. In order to address this, an exploratory focus group was carried out only with fathers to ask for their opinion and the same issue was raised in the following focus groups which were conducted, and it was found that there were issues with the fathers against vaccination, but they were not spread as a general culture, but restricted to a smaller group.

Table 3 presents a list of interviewees by type. The key informants were identified through purposive sampling during the desk review and field visit. In purposive sampling, the informants were selected by the evaluation team after the desk review and during field visits as new information surfaced. Considering that the sampling strategy is relevant in general and for each sub-category of population in particular, the evaluation team was very careful about ensuring that all the key participants involved were heard and considered from different levels (from top government officials, to nurses at a local level, and caretakers who take their children to the Health Units).

Data gathering is a dynamic process in a mixed methods evaluation. One interviewee helped to lead to another interviewee. The inception report had mapped and planned to interview 45 stakeholders. During field visits and the completion on the data collection process, sixty-five stakeholders took part in the interviews and focus groups. A stakeholder analysis was carried out and, during this process, the interviewees were selected.

Table 4. Categories of Stakeholders Interviewed

Category Number

National Government Officials

15

District Level Government Officials

8

Local Government staff 10


31

31

UNICEF staff 8 Development Partners 2 Rights holders 20 Independent Experts 1 Implementing partner 1 Total 65

All of the 7 Districts of the country were visited. The main center was considered and then a further Health Unit in each one of the districts was selected for visitation. The selection took into account the Health Units, which had reported problems for installation of the solar panels and also logistics during the field visit. In addition, the Island of Principe, which was not in the original plan, was also visited to ensure that all of the 7 Districts were covered.

As for the rights holders, they were called to compose focus group discussion at the Health Units at the time of the evaluation visits. Rights holders who were in the HU taking their children were the ones invited for the exercise. The number of participants in each focus group varied, as reflected in the table below. The following procedures for a focus groups were used: 1) Briefing the participants about the purpose of the focus group, procedures and respect for each individual in case of stating or not stating their opinion; 2) Open questions for participants; 3) Moderated interaction among the various participants; 4) Calling on people who were less participatory and 5) Creating an atmosphere of respect and trust. All of the participants were genuine rights holders as they were in the HU at the time of the visit, except for the focus group in Cauê, where they were not in the HU, but were fathers who were close to the HU at the time of the visit.

The number of stakeholders involved in a focus group discussion varies from between 8 to 12 members. However, this varies significantly according to the purpose of the focus group, the topics to be debated, the deepness expected and the availability of the stakeholders. For non-commercial purposes, the number of people in a focus group discussion may be from 5 to 8 and mini focus groups are becoming increasingly common considering the context and sometimes difficulties in recruitment (Krueger and Casey, 2014). In this case, there were challenges in recruiting the rights holders. The focus groups varied between 3 and 8 participants. There was coherence and convergence among the different focus groups, which allowed the identification of key issues among them. However, on one particular issue of why family planning practices may hinder vaccination uptake, there was neither convergence nor a sufficient number of participants contributing their perspectives so as to warrant a convergent theme. This explains why a special disclaimer was made on the lack of conclusive evidence in relation to the EQ 9. Further investigation would be needed to support more substantive conclusions.

In the case of the focus group discussion in Me-Zochi, despite the participation of only 3 rights holders their statements were in alignment with the statements of rights holders from other Districts, which suggested that their perspectives should be considered. Beyond their coincidence of opinions, all the required care and procedures were


32

32

followed just as with the other focus group discussions. Furthermore, although the focus of this evaluation was the CC equipment and, therefore, the major informants in this respect were the government officials, the focus group discussions with the rights holders played an important role. They were exploratory in nature and allowed capturing their perceptions of the expected beneficiaries surrounding issues directly and indirectly related to the CC equipment (broader issues of the EPI program). Table 4 shows the number of participants in each mini focus group.

Table 5. Number of participants in the mini focus groups (by district)

District Number of participants Cauê 4 Lembá 5 Me-Zochi 3 Príncipe 8 TOTAL 20

In the focus group discussions, participants were asked to report their opinions on other rights holders who might not be attending the Health Units. They were also briefed about how their opinion would be considered anonymous and would contribute to improve vaccination rates in the country.

7.4. Limitations of the evaluation process The evaluation team experienced some challenges in the course of the evaluation process. Below is a description of the limitations encountered and of the mitigation strategies put in place by the evaluation team:

Table 6. Limitations of the evaluation and mitigation measures

Limitations Mitigation measures Difficulties in obtaining data from Health Units as data is not stored electronically, but only on paper.

The national consultant started collecting the missing data from as many Health Units as possible after the delivery of the first draft and following reviews.

Access to one specific and important stakeholder (manager from the MoH) was not possible as the person was out of the country and on medical leave.

The evaluation team was able to interview the Minister, the direct manager of the EPI program and several other staff members from the Ministry of Health.

The thematic expert joined the team after the mission to STP had been planned and could not join the first mission and interact with the team in person.

The thematic expert had the change of going to STP afterwards and his visit was a great complement to the first one. It helped to bring in more specific technical details about the Cold Chain equipment and overall EPI management.

Despite the challenges mentioned, no major obstacles were faced during the evaluation process. All of the team members had no previous involvement in the


33

33

initiative and no vested interests in the evaluation results. The evaluation was conducted through an unbiased lens.

8. Ethical Considerations

8.1. Key ethical evaluation principles The United Nations Evaluation Group had launched in 2005, the ́ Norms and Standards for Evaluation´ which has served as a landmark document for the United Nations and beyond. This document is in the process of being updated. The latest revised draft (2016) is the one retained by the evaluation team. The 10 general norms for evaluation are considered in Appendix A, which include ethical criteria for evaluators.

8.2. Ethical Safeguards Throughout the evaluation process, the evaluation team was very respectful to all the stakeholders involved and used a gender-sensitive approach that recognized the unique roles held by women and girls in creating social change, as per the United Nations guidelines. Beneficiaries and officers were orally informed that all the information would be used anonymously. Special consideration was given during focus groups to ensure stakeholders that no correct or incorrect answers existed and that participants were free to speak or not to speak when they wished. All of the photographs taken do not identify the people involved. There were no children involved in the interviews.

The interview team was careful and mindful about power relations during the focus groups and was mindful about listening to all participants and enabling stakeholders to feel comfortable about expressing their views.

8.3. Human rights and participation of stakeholders According to the UNEG guidelines for Integrating Human Rights and Gender Equality in Evaluations (2014), a number of considerations should be in place. Below are the guidelines that the evaluation team incorporated into the report.

⇒ Fostering inclusive participation: The evaluation process was inclusive and led to the direct participation of all the stakeholders (from donors to UN Agencies, governments agencies and rights holders). All the perspectives were heard and included in the report.

⇒ Ensuring respect for cultural sensitivities:


34

34

The evaluation team had three different nationalities, including from STP, and the field visit was carried out with a non -judgmental mindset to allow people to express themselves freely and report on their thoughts.

⇒ Human Rights (HR) and Gender Analysis (GA): This analysis was carried in relation to EQ9, where special consideration was given to the theme during field visits. However, all team members made sure to maintain the HR and GA lens throughout the evaluation.

⇒ Acknowledging the aim for the progressive realization of all HR & GE: The inclusion of all children of the country speaks for the target of full realization of HR and GE without any restrictions. The report focused on the missing children, identifying what worked well, connected to the CC, and on improvements to benefit the unvaccinated children.

⇒ Giving equal weight to the outcomes and the process: Beyond reaching higher immunization rates, the gender issues related to family planning were included in the report, although it came from a limited sample of stakeholders. It is not enough to immunize children. Rights holders must be treated well and respected as human beings and have their voices heard.

⇒ Identifying relevant rights-based and gender-sensitive indicators: Rights-based and gender sensitive indicators were used in EQ9. However, they were also discussed in the EQ related to effectiveness.


35

35

9. Findings and Preliminary Conclusions (by criterion)

The findings are presented according to each one of the five evaluation criteria: Relevance, Effectiveness, Efficiency, Equity, and Sustainability. The evaluation questions were grouped under each criterion and were addressed individually. For each evaluation question, all of the corresponding indicators were listed. Next, the key findings were presented. For each group of findings, some preliminary conclusions (placed in shaded textboxes) were scattered throughout the section. This section on “Findings and Preliminary Conclusions” is the heart of the evaluation report since the final conclusions and recommendations (both strategic and operational) are found toward the end of the report.


36

36

9.1. Relevance

Question under Relevance:

To what extent have the vaccine Procurement Services related to the Cold Chain (CC) rendered by UNICEF (the only operator in this area nationwide), met the country´s needs, especially during the transition period towards Vaccine Independence Initiative (VII), defined by UNICEF Supply Division jointly with Gavi Alliance? (EQ1)

Indicators 1.1. Participation of national government officials in the selection of SDD equipment.

1.2. Alignment between government´s expectation in the procurement process and actual practice.

Picture 2. Presidential Palace of Sao Tome and Principe


37

37

1. UNICEF has been a development partner of São Tomé and Principe since 1978. A

protocol was signed between UNICEF and the Government of Sao Tome and Principe (GoSTP) that appointed UNICEF as the only organization to purchase vaccines for the country. Initially, the vaccines were purchased with resources from UNICEF. Gavi later became a key funder and UNICEF began working with GAVI to manage all the procurement and delivery process. STP has entered a phase of accelerated transition to become independent of Gavi Funding in 2018 and is expected to graduate by 2023.

2. UNICEF supports the country with the Vaccine Independence Initiative (VII), a rotating fund that helps countries meet their funding needs to purchase vaccines and related equipment. STP has been receiving resources from the VII which allows timely vaccines purchased by the UNICEF Supply Division. The country has to pay back but is allowed more time to pay after vaccines are delivered. The Supply Division carries out procurement with advanced payment only. As the country prepares to leave the category of Least Developed Countries in 2024, the country is expected to gain more financial autonomy from the rest of the global community in the future. When it comes to the procurement of the Cold Chain equipment, the interviews with the central government as well as the exchange held with UNICEF staff from various offices and with external partners, attested that the selection of the CC equipment was highly participatory and a result of a close and consensual dialogue among the various partners (I1.1). In fact, the Planning of the Expanded Program on Immunization (EPI) for 2016-2020 contained the goal of ´equipping 100% of the districts with Cold Chain´. In STP, the supply interval is six months and the stock reserve is 3 months. See Appendix J for information on buffer stock before and after the new CCE.

3. The evaluation team found that there was coherence and alignment between expectations of the government officials and UNICEF during the selection process of the CC equipment (I1.2). The process of identifying the need of new equipment was jointly carried out by partners and was the result of the Study ´Evaluation of the Effective Management of Vaccines´ from WHO and UNICEF in 2011 that reflected the need to change the CC outdated equipment for new equipment powered by solar energy. The equipment in place required costly maintenance for replacement of damaged parts. In addition, the equipment was dependent on electric energy which was often down.

4. Furthermore, the study suggested the replacement of the CC for equipment with solar panels would offer more autonomy to operate. The report suggested replacement but not what to purchase, neither where to purchase it. Meanwhile, Gavi wanted to support the agenda of Health System Strengthening (HSS) though vaccination. The government informed Gavi about the recommendation of changing the CC equipment to solar panels. The Alliance said they could support the implementation of solar panels and included the funding in one of their activities. The Government contacted UNICEF to see if they could assist. UNICEF,


38

38

then, consulted the international catalogue of the Supply Division, and in dialogue with the Regional Office, discussed with the government. The government counterparts conducted research and the Supply Division sent the Cost Estimate for the purchase.

5. The World Health Organization gives guidance on Performance, Quality and Safety (PQS) of health equipment which meets international quality standards. They verify new equipment launched in the market and companies can apply to see if they meet the criteria. These standards help countries and the development community to make informed choices at the time of procurement. The old equipment was no longer qualified by WHO; the temperature was not adequate and there were severe problems with the energy supply.

6. From 2014 on, the new CC plan was drafted. During a field visit, the evaluation

team discovered that the equipment chosen was the result of a consensus among various partners. However, during one of the visits to the HU the equipment did not seem adequate because of the influence of seaside air in STP as it was not resistant stainless steel. Picture Box in Appendix H shows quick deterioration of CC equipment after less than two years of use. As a refrigerator had not yet been installed in this HU before, information on previous CC devices that were damaged by seaside air was not available.

7. ´The work of UNICEF in the support of the vaccination program and the CC was well assessed by its partners. The change of the CC equipment was timely. STP suffers from lack of fuel and frequents blackouts which affect the whole territory and can be fatal for the vaccines which need to be stored in a temperature between 2o and 8o C. Thus, the equipment fueled by solar energy was deemed more appropriate. However, the absence of stainless-steel grids in the equipment accelerates its deterioration specially with the influence of seaside air. There is recognition from various partners about the relevant work of UNICEF not only on vaccination, but also on HSS, infrastructure, and capacity building.

REL 1: The evaluation found that there was close collaboration between UNICEF´s various divisions, Gavi Alliance and the Government of Sao Tome and Principe, to purchase the Cold Chain equipment. The need was identified by a study carried out by WHO and UNICEF and in the Planning of the Expanded Program on Immunization for 2016-2020. There is evidence, however, that the equipment purchased was at least for one HU not completely adequate to the seaside air of STP as the grids were not made of resistant stainless steel.

Preliminary Conclusion


39

39

9.2. Effectiveness Questions under Effectiveness: To what extent have CCE investment contributed to improving immunization service delivery? (e.g. changes in frequency on immunization sessions, changes in frequency of vaccine collection/distribution, averted missed opportunities due to increased vaccine availability)?

What are the factors (e.g. equipment types and/or model, health care workers conducting preventive maintenance tasks, installation method(s), verification systems, service delivery models etc.) that have facilitated and hindered healthcare delivery?

What are the unexpected outcomes (positive and negative) produced within the scope of the procurement and distribution of vaccines?

Picture 3. Health Unit of Lembá


40

40

To what extent have CC investment contributed to improving immunization service delivery? (e.g. changes in frequency on immunization sessions, changes in frequency of vaccine collection/distribution, averted missed opportunities due to a better vaccine availability). (EQ2)

8. The evaluation team found a consensus among the stakeholders in terms of the significant improvement in the management of the vaccines caused by the new CC equipment. As the storage capacity increased in the Health Units, they conducted fewer visits to the District Centers. As the equipment was moved by solar panels, it was more reliable in face of frequent energy blackouts.

9. The stakeholders from the Heath Posts reported an improvement in vaccine availability, which can be seen in the Table below in a sample of Health units, mostly from Água Grande and one Health Unit in Cauê. There was a positive variation in all Health Units for which data were available except for Measles in S. Marçal. For further information, please see Appendix J.

Table 7. Availability of Penta and Measles by Health Units (2017 and 2019)*

District Centre/Health Unit

Availability of Vaccine Penta Measles

2017 2019 Variation (%)

2017 2019 Variation (%)

Água grande

CNSR 1615 2568 59 860 2170 152 Água Arroz 333 443 33 300 360 20 S.Marçal 257 383 49 300 280 -7 Pantufo 173 287 66 230 300 30 Vila Fernanda 342 516 51 280 320 14 P. Gamboa 279 367 32 190 410 116

Cauê P.Alegre 145 152 5 110 150 36

*January to June 2017 and 2018. Source: EPI Program.

10. With the new Cold Chain equipment, more vaccines were stored in the districts and were available daily despite power blackouts. This prevented caretakers from leaving for lack of vaccines at the time of their arrival. Before, nurses would have to withdraw the vaccines from the refrigerators in case of power failure and return vaccines to the District Center. With the new equipment, the previously mentioned

Indicators 2.1. Increase in vaccine availability

2.2. Increase of immunization sessions after use of refrigerators purchased by the program.

2.3. Increase in frequency of vaccine distribution.


41

41

time-consuming procedure was no longer in place and vaccines were readily available locally. Table 6 shows the increase in storage capacity in all districts of the country. There were three days of unavailable stock for VAA in 2017 and no events in 2018 (SMT, 2017 and 2018). At a local level, information on unavailable stock is not available. In terms of vaccine availability, when one compares data from 2017 and 2018 (considering the installation of the CCE), improvement is apparent in the surplus of all the vaccines during that period (see Appendix I).

Table 8. Storage Capacity by District - 2017 and 2018 in temperature of 5oC

District Storage Capacity (liters)

2017 2018 Água-Grande 55 99 Mé-Zóchi 55 99 Lobata 55 99 Lembá 17 99 Cantagalo 55 99 Caué 17 99 R.A.P 17 99 Total 271 594

Source: Stock Management Software – World Health Organization.

11. In terms of immunization sessions after use of refrigerators purchased by the program (I2.2), please, see the graphs supporting the following paragraphs in Appendix J. There has been an increase in the case of VAR1 between 2017 and 2018, followed by a slight decrease up to July 2019. The national vaccination rate for VAR1 has moved up from 89.9% to 95% between 2017 and 2018. Almost all the Districts have shown an increase or maintained stable levels, with the exception of Cauê and RAP which have slightly decreased (0.7 and 0.5%, respectively). With the decline of immunization rates between 2016 and 2017, UNICEF was cautious to mobilize with other partners towards reverting this tendency. See Appendix J for more details.

12. The latest data available for 2019 for the period Jan-July 2019, in comparison to

data from Jan-July 2017, when the new CC equipment was not installed show that the average between January and July 2017 and 2019 has been 95% and 92% respectively, reflecting a slight decrease. In 2018, not only the new CC equipment was installed, but Gavi helped to equip the Health Units with vehicles to conduct home visits. The gains between 2017 and 2018 may have been due to a number of factors combined, including the new CC equipment, the vehicles, and national mobilization, considering the poor performance of the year before. However, if mobilization is not maintained, a decrease may be felt, as it is the indication for 2019 (further information in Appendix J).

13. In the case of VAR 2, the average rate for Jan-July 2017 has been 76% while for

Jan-July 2019 there was an increase to 84% (see Appendix J). It is important to


42

42

highlight that 90 to 95% of the children need vaccination twice against measles to protect them and to avoid measles outbreaks. Even though this data have not been consolidated by the EPI program, data show a tendency of increase between both years. This is significant, as one of the challenges of the EPI program in STP is to increase vaccination rates for second doses.

14. When it comes to DTP3 vaccination rates, there was a very slight national

decrease between 2017 and 2018 from 95.4 to 95% (only 0.4). However, it shows an increase in vaccination for all the Districts, except Me-Zochi and Lembá. However, even for these districts the decrease has been minimal (0.5% for Me-Zochi and 0.7% for Lembá). It is important to mention that 79% of the CC equipment was installed by February 2018, which means that the data for 2018 are valid for capturing possible impacts of the CC equipment over vaccination rates. Moreover, there are contradictions. In RAP for instance, the equipment was installed only in June 2018 and it had the second increase in vaccination rates in comparison to other districts (from 95% in 2017 to 97 in 2018, only behind Cantagalo which increased vaccination rate by 2.8%). See Appendix J.

15. On the evolution of RRI vaccination rates for the period 2015-2018, there was a decrease from 2016 to 2017 and a significant increase from 2017 to 2018 (5.1 percentage points). Me Zochi comes in again with the lower take ups for RR1. These figures reflect that in 2018, a greater effort was in place to increase vaccination rates to the same level or higher in relation to 2016 (further in Appendix I).

16. Even with the achievements of 2018 and tendencies for 2019, there are still challenges in having children complete the vaccination cycle. Only 62% of the children in the country had completed their vaccination cycle in 2018. This shows much potential for improvement in second and third doses, despite the increase in VAR2. This is an issue to be further considered under the EQ9 on Equity.

17. When one considers the data at a local level, some impact can be seen in the Health Units where the suitcase was in use and was changed to the new CC equipment as it is the case of Monte Café and V. D´Ámerica. In Monte Café, there was an increase of number of doses of Penta 3 distributed between 2017 and 2019 (from 25 to 34, a variation of 36%). For Measles, there was a slight increase of 12.90% from 2017 and 2019. In V. D´América, there was an increase in Penta 3 from 45.45% between 2017 and 2019 and for Measles of 100% in the same period. At the same time, there was actually a decrease of vaccination sessions in the Health Units in St. Cecilia and Picão which used suitcases before. In St. Cecília, there as decrease of 60% for Penta 3 and of 38.46% for Measles between 2017 and 2019. In Picão, there was a decrease for Penta 3 of 20% and 34.78% for Measles between 2017 and 2019. So, no clear impact can be identified (See Appendix I). In the Health Units which had CC equipment before and changed to a new one, vaccination sessions have either kept stable or decreased. The fact is that


43

43

immunization rates were already high. To reach the remaining children, more focused and directed effort will have to be made.

18. When it comes to the vaccine distribution (2.3) from the central store to the districts, with the new CC available it has decreased from the year 2017 to 2018 from 118 to 109 distributions per year (Stock Management Software, WHO). Figures of the distribution from District to the HU are not available. The decrease of collection visits represents a gain to Health Units, more vaccine was available with fewer costs of transportation and staff time involved.

Preliminary Conclusion EFF 1: There is evidence that the CC investment contributed to increasing overall vaccine availability in Health Units, except for bVPO, as the new equipment allowed for increased and steady storage capacity. There was evidence that vaccination rates have increased or remained stable between 2017 and 2018 with a tendency to remain stable or increase for some vaccines in 2019 (according to the data available). However, this increase or stability in vaccination may not be attributed directly to the CC, as it is the result of a number of other factors, including the CC. On a local level, the impact of the CC is also mixed. In some Health Units, the new equipment has brought increase in vaccination sessions, in others it has not. There was a decrease in vaccine collection visits which represented a gain for the HU staff which had to do less frequent visits to the District Centers, allowing them to direct their time to other activities (paragraphs 11-18).

What are the factors (e.g. equipment types and/or model, health care workers conducting preventive maintenance tasks, installation method(s), verification systems, service delivery models etc.) that have facilitated healthcare delivery? What are the factors have hindered healthcare delivery? (EQ 3)

Indicators 3.1. Improvement of health care delivery due to type of equipment.

3.2. Qualification of health care workers in conducting preventive maintenance tasks.

3.3. Improvement of health care delivery due to installation method and verification system.

19. Health Care delivery involves a complex number of factors, such as infrastructure, knowledge and training of health workers, availability of instruments, well established protocols according to the best international practice, monitoring and evaluation, motivation and leadership, among other factors. CC is only one component of this complex system on the part of infrastructure. Of course, a system entails an integration of all of its parts and what happens in infrastructure or leadership will affect the whole system. However, one element will not transform the quality of health care.


44

44

20. There is concern from STP Ministry of Health key staff about the quality of health

care in the country. They report that patients may sometimes ask for assistance in other locations due to the poor health care in the place close to their homes. In fact, the field visit of the evaluation showed that there are discrepancies in training and infra-structure across the country. Considering the country is small and one can access different districts, moving to a different Health Unit is expected to happen. Nevertheless, even though the nurses and heads of districts did not report improvements in the quality of health care delivered due to the CC as the protocols of health care remained the same. They reported impact in other subtle questions related to work conditions (I3.1) – such as more time available to do other things and a more comfortable environment in the Health Unit as the new devices were prettier and did not have smoke coming out of them. They also reported increased trust they had on the maintenance of the vaccines, as the equipment worked better.

21. The evaluation team found mixed evidence of the healthcare workers’

qualifications and abilities to conduct preventive maintenance tasks (13.2). If on one hand, nurses reported abilities with maintenance procedures, the field visits showed that they were not always well-informed of the maintenance procedures and obvious gaps in maintenance practices were evident. As the new equipment was installed, the health professionals received basic training from the company Electrofrio for the maintenance, cleaning, etc., which was reinforced by employee training provided by the central government. However, questions on procedures remain the most effective way of handling the new equipment. The Information Box 1, in Appendix I, shows basic information about TCW 40 SDD that the evaluators found nurses at Health Units unaware of the basic instructions. For example, some professionals from the Health Units were not aware that the equipment had no battery storage. Others were unaware that the TCW 40 SDD can freeze water packs to be used for the small vaccine boxes for home visits. This was verified because some of the nurses were using other refrigerators for the freezing of extra water packs.

22. Maintaining the instructions on hand for easy access revealed challenging for workers. During a field visit, evaluators discovered a maintenance worker at Electrofrio had taken the manual to the head office and had not returned it. Often the manuals were not easily accessible; the manuals were protected by a plastic cover, and did not show signs of use, such as dirt on paper, wrinkles, normal wear and tear of paper. This was unexpected as the CC equipment had been in use for over a year and a half. On placing distance holders on the back of the equipment, they were delivered, however their installation was not checked by individual Health Units.

23. In terms of maintenance, the drainage plug should be opened daily while the

machine is operating, to evacuate the condensation. When the lid is opened, the cold surface attracts moisture inside. The condensation is collected at the bottom


45

45

of the inner cabinet and must be cleaned on a regular basis, using the drainage plug on the front side of the cabinet. Evaluators observed that in at least two Health Units the drainage plugs were open during operation. The open plug hole enables the outside air to enter where condensation forms, causing the unit to work extra to maintain appropriate temperatures. Also, some nurses were not aware of which container was used for collecting the condensed water under the drainage plug. Some showed a cup from a sink which might be also used for drinking. In the event that a stored vaccine container might be damaged and not identified on time, its contents may mix with the condensed water. In the worst-case scenario, the content may leak in the cup and, if it were a live vaccine, this may cause an equivalent disease after drinking from this cup.

24. Another observation was training for correct storage of vaccines when conducting house visits. Vaccines must not be exposed to direct heat as identified by the VVM. In this instance, the vaccines had been stored in a small vaccine carrier without enough cooled water packs as cited in Module 2 of the Vaccine Cold Chain Instruction Manual (Page 2/12). The vaccines stored at room temperature demonstrated a color change of the VVM. Too much heat exposure changed the color of the inner square of the VVM to the same color or darker than the outer circle. When this happens, the vaccine should be discarded.

The evaluation team took to the field a check list which contained the following items: a) Temperature record chart which should be on the fridge or nearby b) a log of the refrigerator temperature, measured twice daily; c) Expiration dates of vaccines in a sample inside the arks and d) Vaccine Vial Monitor. For item a, the evaluation team found that the temperature log was visible in only some of the Health Units visited. The pictures in Appendix I show the temperature chart covered by plastic with a tape coming out. The evaluation team verified that the light of the temperature chart, which was at a low distance from the ground chart tended to attract the attention of children who visited the Health Units. The pictures on Appendix I show an open drainage plug in CC equipment.

25. For letter b - recorded temperatures, a log was found to be kept either on top of the fridge (99% of the cases) or on a table nearby. For letter c, in the case of vaccines expiration dates, the samples were all within the designated date. In the case of the Vaccine Vial Monitor, the evaluation team found only one Health Unit in Cauê with the color of the VVM changed and still in the fridge. These vials had been in the field and returned to the mini-arks. Overall, some nurses experienced challenges adhering to the required maintenance procedures. Further, training may be required to ensure that maintenance procedures are adhered to properly.

26. Moreover, the evaluation team evidenced improper cleaning of one condenser, as shown in Appendix I (condensers showing signs it had been cleaned in a vertical line and not horizontal as recommended). The evaluation team saw some of the brushes for the condensers, but they seemed to be unused and connected


46

46

to a possibly too small rope behind the front ventilation grid to brush the condensers without damage). It was also identified that because of the weather the screw for opening the front ventilation grid in one HU was very rusty and might affect access to the front ventilation grid and may lead to problems with cleaning in the future. The proper, monthly cleaning protocol: a) Clean the unit from dust; 2) Switch off the TCW 40 SDD and disconnect from power socket; 3) Clean the ventilation grids and the condenser with the supplied brush or a vacuum cleaner; 4) Reconnect and switch on; 5) Clean the lid seal. No specific training was identified for safely cleaning the solar panels and roof structure.

27. In terms of the improvement of health care delivery due to installation method and verification system (I3.3), no impact was identified. All the Health Units visited had a form to keep temperature control of the vaccines, but this did not seem to be a new practice for them as the old equipment also had temperature control. What some of them reported was that it was easier in some cases to see the temperature inside as opposed to the old equipment.

Preliminary Conclusions EFF 2: Quality of health care is the result of a number of factors, from which CC is only one component. There was no evidence of improvement of health care delivery due to type of equipment or installation method and verification system. There is evidence that the new equipment helped by allowing health workers additional time to carry out their duties (paragraph 20). EFF 3: There are challenges in carrying out proper maintenance of the CC equipment due to health workers’ lack of knowledge and the lack of specialized care for the solar panels (paragraphs 21-27).

What are the unexpected outcomes (positive and negative) produced within the scope of the procurement and distribution of vaccines? (EQ 4)

Indicators 4.1. Catalyzation of innovations by CC investments and procedures in continuous improvement of supply chain.

4.2. Catalyzation of innovations by CC investments and procedures in leadership of supply chain.

4.3. Catalyzation of innovations by CC investments and procedures in data systems of supply chain.

28. The evaluation team looked at unexpected outcomes produced within the scope of the procurement and distribution of vaccines. The indicators were related to catalyzation of innovations in various areas. When it comes to unexpected outcomes, the evaluation team could identify the nurses’ pride in having new


47

47

equipment in their Health Units, which revealed easier maintenance and an attractive layout. The new equipment boosted the staff’s morale and facilitated their work. This was an unexpected effect that was identified during the interviews.

29. The evaluation team did not identify any negative outcomes. Several suggestions for improvement of health assistance as a whole, have been detailed in other parts of the report. The evaluation team identified a well-functioning structure in place that involved a complex network of people, presented in the Theory of Change. From the airport to the customs office, from transportation to the central warehouse, to the health districts, and to the heath posts, cooperation was apparent. A procedure was previously established for the vaccines with the old equipment. With the new devices, the greatest gain was at local level, where Health Units conducted fewer visits to the central warehouse.

30. STP has developed a structure to plan for the vaccines, control stocks, and monitor the temperatures and equipment of the Cold Chain. In this process, the contribution of major partners such as Gavi, UNICEF, and WHO has proved very beneficial. This contribution has been highlighted especially in terms of financial resources and technical assistance to help the country have access to the best international quality parameters (such as equipment and guidance for data control). One key aspect far beyond the CC, is the leadership identified in the supply chain (I4.2) in the Distribution Center and in various districts, even though this cannot be attributed to the CCE per se. An important level of cohesion was seen throughout the country in various site visits – the same type of guidance and policies are repeated throughout the country. This shows that despite the reported problems that may be identified with work morale in some cases, there is coordination in place and some degree of unity that may help explain the success of the country in the area of vaccination.

31. Some important innovations beyond the CC were also found on the ground (I4.1). Various Health Units in the attempt of reaching the caretakers tried out different methods: a) Registering the telephone numbers of mothers and calling them to remind them of the next vaccination session or asking them to come if they did not show up when expected; b) Registering the telephone number of caretakers’ relatives (as it can be seen in Appendix I). There were benefits identified in the area of planning as there was better forecasting in terms of the supply of vaccines in each health district because of increased storage capacity (See Statements in Appendix I).

32. When it comes to innovation in data systems (I4.3), no contribution of the CC was identified. In fact, the new equipment did not come with a central remote temperature control which was about to arrive in the country when the field mission was being carried out. However, an important process was in place with the support from UNICEF and WHO. That is, the development of the SIS, an information system to help manage health data in the country. All of the Health Units had received a computer and a modem to access the internet to use this


48

48

system. However, this was still in pilot phase. At the time, most of the information was still processed using paper.

Preliminary Conclusions EFF 4: There have not been innovations identified from the point of view of improvements in the supply chain or data systems. There was an unexpected outcome which was the impact of the new CCE in the work environment of health agents (paragraphs 29, 30 and 31). EFF5: Innovations have been identified across districts in reaching unvaccinated children. Different actions are being taken such as calling caretakers, calling their relatives and reaching out in different ways to have the children vaccinated (paragraph 32).

Table 9 summarizes the results expected and their level of Achievement.

Table 9. Results expected and level of achievement under effectiveness (by result)

Envisaged Result Observed Results on the ground

Level of Attainment

R1: CCE investments contribute to increase vaccine availability.

There was an increase in vaccine availability in the Health Units. Achieved

R2: CCE contributes to increasing immunization sessions.

There was an increase in immunization sessions in some of the Health Units with new CCE and in others there was not.

Partially Achieved

R3: CCE investments contribute to improve health care delivery.

There was evidence that the CCE helped nurses increase their time available in Health Units.

Partially achieved

R4: Health care workers are qualified to conduct preventive maintenance tasks in CCE.

There were several knowledge gaps identified in the maintenance of the new equipment.

Not achieved

R5: Health Care delivery is improved due to installation method and verification system

There was no evidence found to indicate impact in this area. Not

achieved

R6: CCE investments catalyze innovations in continuous improvement of procedures, leadership and data systems of supply chain.

There was no evidence in this area. Not

achieved


49

49

9.3. Efficiency Questions under Efficiency: With respect to the transport mechanisms put in place, were there other alternative strategies that could have been put in place to achieve the same level of results but at a lesser cost, especially in light of the country´s current transition towards Vaccine Independence?

To what extent could an existing central warehouse at the regional level (WCAR) produce transportation benefits for the whole country?

To what extent were financial resources, human resources, and supplies, sufficient (quantity), adequate (quality) and distributed/deployed in a timely manner?

Picture 4. Airport Storage Room


50

50

With respect to the transport mechanisms put in place, were there other alternative strategies that could have been put in place to achieve the same level of results but at a lesser cost, especially in light of the country´s current transition towards Vaccine Independence? (EQ 5)

33. STP is an insular country with great challenges in terms of transportation costs. This

evaluation question will be discussed in alignment with the question of a possible central warehouse with joined preliminary conclusions. STP imports most of its goods and pays high freight prices for goods arriving by ship or airplane. According to the UNICEF Country Office, the costs of transportation for needles and vaccines double the costs of these products. The vaccines and supplies come by airplane as the small amounts do not fill a container and take about 6 months from placement of the until reaching the country. However, in the case of CCE, it is not often purchased, has a volume high enough to fill a container and comes by ship. Thus, the transportation costs of CCE over its total cost is only about 5% (considering the cost estimate of the Supply Division sent to the GoSTP which was actually incurred in at the time of purchase). This 5% includes not only freight, but also insurance and inspection (I5.1). In the case of I5.2, percentage of transportation costs in similar Gavi Alliance programs, this information was not publicly available, and the cost of mining would be too high for the relevance of this indicator in the report. Thus, the question of alternative modes and routes of transportation is discussed in the next evaluation question.

To what extent could an existing central warehouse at the regional level (WCAR) produce transportation benefits for the whole country? (EQ7) Indicators 7.1. Unmet regional demand for vaccines

7.2. Capacity level of UNICEF Regional Office in procurement services

7.3. Cost of transportation between Sao Tomé and other countries of the region

34. There are debates among stakeholders about a possible Regional Warehouse

accessible to STP. In exploring possibilities, I7.1 looked at unmet regional demand for vaccines. In the case of Gabon and Guinea Equatorial, they both have vaccination rates for MCV1, DTP3 and POL3 much lower than STP as it can be seen in Table 10. This shows a potential increase for vaccines in the Region for the future (I7.1).

Indicators 5.1. Cost of transportation of CEE over total cost of program.

5.2. Percentage of transportation costs in similar Gavi Alliance programs.


51

51

Table 10. Immunization Rates 2018 in Gabon, Guinea Equatorial and Sao Tome and Principe for selected vaccines

Country MCV1 DTP3 POL3 Gabon 59 70 64 Guinea Equatorial 39 46 45 Sao Tome and Principe 95 95 95

Source: Immunization Estimates 2018, WHO.

However, the field work showed that there are different views of a Potential Regional Warehouse, where costs could decrease. Officials from the government are concerned about the location of a regional warehouse that could benefit the STP. Within UNICEF, perspectives also vary – some believe it would be beneficial, others do not see a clear rationale for a warehouse and point at the difficulties of coordination among the various countries. This is a topic that would have to be further discussed among the various stakeholders. A misguided perception held is that when goods go to nearby countries, they take longer to arrive in STP and are subjected to non-reliable flights. Thus, stakeholders should consider a Regional Warehouse would require: 1) Flight routes; 2) Airline costs and 3) Airlines transportation policy. Overall, there is a general distrust over Regional flights. The majority of stakeholders deem TAP a trusted airline.

35. When it comes to UNICEF capacity to carry out procurement procedures and distribute vaccines and alike items, the indicator proposed was to examine the Regional Office capacity to possibly manage more complex procurement which could be at a Regional Scope. However, evaluators discovered the entire procurement process was managed mostly by the UNICEF Supply Division. The UNICEF Offices in STP and Gabon only assist in the process. In the case of Gabon, the procurement team is composed of two people and in STP, there is one staff member with that mandate. In case of collective procurement, if UNICEF were to assist in this process, capacity would have to be developed within the UNICEF Office in Gabon. However, as it has been previously discussed, without reliable airline routes, the project does not seem a realistic option.

36. When it comes to I7.3 - Cost of transportation between Sao Tomé and other countries of the region, STP has only direct flights to Ghana, Gabon, Guinea Equatorial, Angola and Portugal. There are no frequent and reliable flights between STP and Gabon and Guinea Equatorial. Angola is distant from the Region and Ghana has flights to STP, but not to Gabon or Guinea Equatorial. In this case, the question of price becomes irrelevant if transportation is not in place for the special needs of vaccine cargo. Another potential country to consider would be Cameroon which neighbors the other countries and is deemed by one of the key stakeholders to have a stable transportation system. In fact, Cameroon has direct flights to Malabo and Livreville, but not to STP. The airlines for these routes are Cargolux Italia, Air France, Afrijet, Camair Co and RwandAir, which would bring viable options, but still out of the direct interest of STP.


52

52

37. In addition, an evaluation interviewee highlighted the issue with coordination among various countries in the Region. Different modalities of pharmaceutical setups reflect distinct cultures and history; STP with a Portuguese background, Gabon with a French influence and Guinea Equatorial with a Spanish influence. The differences in culture and background bring to the forefront a challenge of procedural harmonization and dialogue among these three countries.

38. One government official interviewed expressed concern that if transportation

costs were shared among other countries, STP would benefit and pay less – paying, for instance, the flight from Gabon or Guinea Equatorial, instead of the freight from Europe to STP. In addition, the theme of collective purchase (further discussed in EQ7) is under debate in the country between stakeholders at the Ministry of Health and UNICEF. As the evaluation report was being drafted, an agreement was under discussion to promote this modality of purchase. However, as discussed, beyond the cost issue is the availability of flights and the reliability of airlines in terms of flights and logistics (handling sensitive packages). Specific guidelines and careful handling of the vaccines must be followed by personnel.

Preliminary Conclusion

EFFI 1: Costs of transportation for STP are considerably high due to its location and the relatively small supply quantity ordered. However, there is no availability and reliability of airlines within the Region to allow for collective purchase of vaccines and a regional warehouse at the moment. In addition, the three countries considered (STP, Gabon and Guinea Equatorial have very different pharmaceutical set ups due to history and culture which would pose high costs of coordination and transaction (paragraphs 44 and 45).

To what extent were financial resources, human resources, and supplies, sufficient (quantity), adequate (quality) and distributed/deployed in a timely manner? (EQ 6) Indicators 6.1. Alignment of financial resources in relation to program design

6.2. Implementation of program activities in relation to the predicted budget

6.3. Adequacy of human resources in relation to program design and implementation

6.4. Timeline of program implementation against initial targets

39. Efficiency covers the analysis of how financial, human resources, and supplies were deployed in the implementation of project activities. In this case, funds from Gavi Alliance were transferred to the GoSTP for the purchase of the equipment. In addition, UNICEF Funds were used to hire Electrofrio to install the CC Equipment. Moreover, human resources were deployed from Government and UNICEF staff members, which managed the purchase and installation of CC equipment within the framework of the partnership in place.


53

53

40. The equipment purchased by the government through UNICEF amounted to US$ 296,143.00 (two hundred ninety--six thousand and one hundred and forty-three dollars) and included 7 TCW 2000 SDD, 1 spare set of parts for the first lot, 32 TCW 40 SDD, 3 spare sets of parts for the second lot and 32-day logs fridge tags. The contract with Electrofrio was of approximately US$ 20,000.00 (448.261.000 dobras as of Dec. 12, 2017) totaling about US$ 216,143.00. The UNICEF Supply Division in Denmark in 2018 alone procured $3.486 billion worth of goods and services globally for all its worldwide interventions from over 11,000 businesses (UNICEF, 2018). This capacity makes UNICEF an important player in the vaccination market with increased capacity to negotiate. The administrative costs were absorbed by the organizations involved. The purchase and installation of CC was part of the daily routine of the staff of Unicef and GoStp, increasing efficiency of the work (I6.1, I6.3). Moreover, the overhead charged by the UNICEF Supply Division in this case nearly equals what is charged when the purchase is made with resources from the organization.

41. Before the vaccines and CC arrive in the country, the UNICEF Office is notified and informs the government. There is an expedited customs process for vaccines and drugs in general. Government officials go to the airport together with UNICEF staff to receive the vaccines and then UNICEF transports them to the Distribution Center. Later, the fees are paid. The State forwarding agent takes care of the customs procedures in conjunction with the Ministry of Health.

42. Vaccine cargos usually arrive two to three times a year. One possible issue of concern in this process is the type of transportation from the airport to the Central Warehouse. The transportation is made via regular vehicles and not at the right temperature. The distance is short between the airport and the Distribution Center, but there are questions from drug experts in the country whether a proper vehicle should be used. The same applies to the airport, where there is no cold chamber on the premises, but considering the small number of arrivals, building one is not a high priority for the airport management.

43. The Evaluation team followed the whole arrival chain (airport, customs office, forwarding agents and transportation) and found that the Director of Health Care must issue an authorization so that the vaccines may be released. Even when this authorization is not issued, the Customs Director looks after the supplies and contacts the organization who has ordered them. In this case, they are stored in the office of the Customs area in the airport. The evaluation found that there is close collaboration among the chain of stakeholders involved.

44. With regards to the human resources, it is important to note that the purchase and installation of the CC is part of a process which UNICEF is used to. There are well established procedures in place. The process starts with the government requesting a purchase to the Country Office. The Country Office consults the Regional Office when it comes to technical CC issues, and the national office asks the Supply Division for a quote which is sent straight to the government. The


54

54

government pays for the products in advance and the Supply Division makes the purchase, sends it to the country where the products are received by the government with a follow-up by the UNICEF country office. All of the parties interviewed informed the procedures were well-followed and no problems were reported during the process (I6.3).

45. The challenges came in the installation of the equipment. The equipment was purchased in 2017 and the installation took place in 2018, with pending issues for 2019. The delays were due to infrastructure problems in the Health Units of Diogo Vaz, Ubua Budu, Sundy and Nova Estrela in Principe Island. In some cases, the roofs were not appropriate to receive the solar panels. Appendix I shows Health Units with poor infrastructure for installation of CCE. There was one further contract with Electrofrio due to the restoration of the HU which did not have conditions for installing solar panels (I6.4). Table 11 shows the number of equipment installed in relation to the target. There are still 3 refrigerators to be installed in STP almost two years after delivery.

Table 11. Targets and number of CC equipment installed in STP

Type of Equipment

Target Number installed

Timeline Problems faced

TCW 2000 SDD 7 7 Partially on schedule Infra-structure TCW 40 SDD 32 29 Partially on schedule Infra-structure

Source: EPI.

46. Other challenges of implementation include the training in reading the temperature. Each fridge has a thermometer inside. In at least one case, the nurse had difficulties managing it and credited it to a lack of proper explanation in the training she had had. There is also a datalogue which was in the cost estimate of UNICEF and had not been delivered. Temperature data are collected and stored from the different CC equipment and monitors temperature of the CC at a central level.

47. The office of UNICEF in STP is under the same operations and representative than the office in Gabon. In this case, the approval in the system for more complex operations needs to involve Libreville. However, the process between STP and Gabon is standard. In the case of purchasing of CC and vaccines, the office of STP contacts the Supply Department and maintains Gabon informed of the operation, which facilitates the transaction.

Preliminary Conclusions EFFI 2: The process of acquiring the CC is a standard procedure to which all the parties involved are used to and have reported a smooth journey. There were gains in efficiency as all the parties are part of a larger partnership where overhead costs are already paid for. The amount of staff involved was adequate in relation to the actions planned (paragraphs 46-52).


55

55

EFFI 3: Implementation challenges were due to lack of infrastructure in the HU that did not meet minimum standards for installation. Most solar panels and equipment were installed according to the timeline; however, three refrigerators remain uninstalled nearly two years after arrival (paragraph 54).

48. When it comes to the costs of vaccinating a child in STP, this calculation was

carried out and an analysis was conducted to simulate the possible challenges of funding vaccines in the country. According to the National Plan for Health Development 2017-2021, the GoSTP has agreed to finance vaccines and consumables according to the guidelines given by the Global Alliance for Vaccination and Immunization (Gavi). Table 12 shows Gavi´s support to co-finance different types of vaccine in STP since 2003.

Table 12. Gavi´s support in co-financing vaccines in STP

Vaccines Type Year(s) of Gavi support

Co-financing required

Yellow Fever Routine 2003-present* Yes Pentavalent Routine 2009-present* Yes PCV Routine 2012-present* Yes Rotavirus Routine 2016 -present* Yes MR Routine 2018 - present* Yes

Source: GAVI, Sao Tome and Principe, Key information and co -financing

*As of December 2019.

49. Table 13 and 14 show the costs for immunization of Gavi co-financed vaccines in the country from 2013 to 2017 and 2020 to 2024 and the percentage the government has been contributing to pay for them. The data is taken from the EPI Program and shows the yearly costs for the specific vaccines supported by GAVI. Each year, GAVI reduces the financial support for these vaccines. In the year 2013, for example, it paid for 94% of the vaccine costs, while in in 2017, it paid for only 82% of the total costs of vaccines it supports. In 2023, it is predicted that Gavi’s support will be 0%. Although, there has been an increase in the percentage of contribution from the STP Government to pay for the vaccines, from 6% in 2013 to 18% in 2017, full funding remains unforeseen. This may be a cause for concern with regards to the sustainability of vaccine supply in the country.

Table 13. Percentage of government contribution for paying for Gavi co-financed vaccination costs in STP (in US$)

Year/Source of funding

2013 2014 2015 2016 2017

Government expenditure

10,500 14,468 14,575 23,830 29,744

Total expenditure

187,481 186,823 141,228 235,746 166,645

Government as % of total

6% 8% 10% 10% 18%

Source: EPI Program.


56

56

50. In planning for the future, Table 14 below shows the vaccination costs foreseen for STP for the period 2020-2024 for vaccines currently funded by GAVI (Penta, PCV, Rota, YF routine, MR follow-up). The source is the EPI Program. Considering the deadline that was given to the GoSTP (assume the costs of vaccine from 2023 on), the GoSTP would have to pay from that year on, approximately US$155,600 to pay only for the 5 vaccines listed, not including the other costs involved (needles, indirect costs, transportation etc.)

Table 14. Vaccination costs foreseen for STP for the GAVI co-financed vaccines 2020-2024 (in US$)

VACCINE TYPE/YEAR

2020 2021 2022 2023 2024

Penta 8,815 12,233 15,715 21,257 21,517 PCV 27,582 38,367 49,348 63,155 63,925 Rota 26,745 37,203 47,852 62,575 63,338 YF routine 3,118 4,174 5,325 8,613 8,718 MR follow up

- 735 - - -

Total 66,260 92,713 118,240 155,600 157,498 Source: Co-financing information sheet Gavi, STP, last updated 15th Dec 2018

51. According to the STP vaccination calendar, for every child up to 2 years old, 20 vaccines should be given: 1x BCG, 5x oral Polio, 1x Hep B, 3x Penta (DPT-Hep B-HiB), 3x PCV, 3x Rota, 1 VPI (inactivated Poliovaccine), 2x Measles/Rubela-Combination and 1x Yellow Fever. Table 16 shows the costs of vaccines for one child in STP from birth until 2 years of age, considering available figures from 2018. In this case the costs are calculated considering that not even one dose of the vaccines gets lost ("best case"). The cost for all of the vaccines for one child has been estimated at US$ 30.97.

Table 15. Vaccine Costs per child from birth until 2 years old for 2018 when all doses even of the multi-dose vaccines are completely used

Type of Vaccine Cost in $/dose Recommended doses

Total Costs in $

Penta 0,92 $ 3 2,76 $ PCV 3,30 $ 3 9,9 § Rota 3,92 $ 3 11,76 $ YF routine 0,98 $ 1 0,98 $ VPI 2,80 $ 1 2,80 $ RR 0,64 $ 2 1,28 $ BCG 0,17 $ 1 0,17 $ OPV 0,18 $ 5 0,90 $ Hep B 0,42 $ 1 0,42 $

Total 20 30,97 $ Source: costs in $/dose based on Outil De Gestion Informatise des Stocks -SMT, Fiche Recapitulative De Gestion des Stocks 1. Jan 2018 - 31st Dec 2018, recommended doses based on the updated vaccination calendar in, STP, year 2018 and total costs in $ based on own elaboration.


57

57

52. Table 16 shows the vaccines which come in multi-dose vials. The waste of multi- dose vaccine will vary between the different multi-dose vaccines.

Table 16. Multi-dose vials used in STP for Children up to 2 years

Name of Vaccine Status Dose per vial BCG Lyophilized 20 RR Lyophilized 5 or10 Yellow fever Lyophilized 5 bVPO Liquid 10

Source: Own elaboration.

53. Table 17 shows the rate of vaccine loss between 2008 and 2013 (excluding 2011) which will be considered later for calculating the costs of vaccinating a child.

Table 17. Rate of vaccine loss between 2008 and 2013 (excluding 2011) * in percentage

Vaccine 2008 2009 2010 2012 2013 YF routine 19.9% 22.8% 22.8% 15.7% 21.4% VAR (now

RR) 43% 39% 39.1% 36.2% 31.8%

BCG 56% 57. 3% 57.3% 58.9% 44.7% OPV 9.9% 9.2% 9.2% 5.10% 10.1%

*There are no data available for 2011. Source: modification based on PLAN PLURIANNUEL COMPLET, 2016 -2020, STP, Janiver 2015

54. The next table shows the cost of vaccinating a child from birth until 2 years old when only one dose of the corresponding multidose vial is used. In this case, the whole vial has to be calculated (see explanation below).

Table 18. Vaccine Costs per child from birth until 2 years old from multi-vial vaccines when only one dose is used

Type of Vaccine

Cost in $/ dose

Recommended doses

Doses per vial

Total Costs in $

Penta 0,92 $ 3 2,76 $ PCV 3,30 $ 3 9,9 § Rota 3,92 $ 3 11,76 $ YF routine 0,98 $ 1 5 4,90 $ VPI 2,80 $ 1 2,80 $ RR 0,64 $ 2 10 12,80 $ BCG 0,17 $ 1 20 3,40 $ OPV 0,18 $ 5 10 9,00 $ Hep B 0,42 $ 1 0,42 $

Total 20 57,74 $ Source: doses by vial based on EPI Program.


58

58

55. A newborn will be vaccinated with BCG and for the next 6 hours this baby will be the only one. According to the information of the producer the reconstituted vaccine shall be used during a period of time no longer than 6 (six) hours. That means the other 19 doses have to be destroyed after 6 hours and we have to calculate in this case 20 doses for this one baby.

56. A child will be vaccinated with RR twice (month 9 and month 18) and for the next 6 hours, each time this child will be the only one. According to the information of the producer the reconstituted vaccine shall be used during a period of time no longer than 6 (six) hours. That means the other 18 doses have to be destroyed after 6 hours and we have to calculate in this case 20 doses for this one child.

57. A child will be vaccinated against yellow fewer and for the next 6 hours this child will be the only one. According to the information of the producer the reconstituted vaccine shall be used during a period of time no longer than 6 (six) hours. That means the other 4 doses have to be destroyed after 6 hours and we have to calculate in this case 5 doses for this one child.

58. A child will be vaccinated against Polio 5 times (day of birth, 6 weeks, 10 weeks,14 weeks and one year after the 14 weeks) with bOPV. According to the producer, once opened, multi-dose vials should be stored under appropriate Cold Chain conditions. Multi-dose vials of OPV from which one or more doses of vaccine have been removed during an immunization session may be used in subsequent immunization sessions for up to a maximum of 4 weeks, provided that all of the following conditions are met.

i. The expiraton date has not passed;

ii. The vaccines are stored under appropriate Cold Chain conditions;

iii. The vaccine vial septum has not been submerged in water;

iv. Aseptic technique has been used to withdraw all doses;

v. The vaccine vial monitor (VVM) has not reached the discard point.

59. That means if something went each time wrong during the handling, during storage or was open for longer than 28 days while only one child was vaccinated the open vials, altogether 45 doses have to be destroyed and 50 doses have to be calculated for this one child. Therefore, the cost of 20 vaccines for one child in this case will be US$ 57.74.

60. The best case and the worst case with losing the leftovers of the multi-dose vials are extremes, therefore the cost for the 20 vaccines in STP for one child up to 2 years must be calculated between US$ 30.97 and US$ 57.74.

61. In case we calculate the annual Target Population with the formula total Population x 3%, in STP, our target population would be 6,000.00 children


59

59

(200,000.00 inhabitants x 3%). See the Immunization in Practice, WHO, 2015 update. Thus, we can estimate the resources to vaccinate all children up to 2 years old. That means for 6,000 newborns, the country would spend between US$ 185,819.00 (6,000.00 newborns x 30.97) and US$ 346,440.00 (6.000 newborns x 57.74).

62. Neither information about direct costs nor indirect costs for immunization (such as consumables, salaries, petrol and others) is available. But this type of exercise is fundamental to see that beyond the cost of vaccines, there are many other costs involved which should be factored into the long-term, budget planning of the GoSTP.

Preliminary Conclusion EFFI 6: The cost of the vaccines for a child in STP until 2 years varies between US$ 30.97 and US$ 57.74. Beyond the costs of vaccines, there are many other costs involved which should be factored into the long-term, budget planning of the GoSTP.


60

60

9.4. Sustainability

Questions on Sustainability: What are the actual prospects for the government to be able to manage the CC after UNICEF´s support?

How did UNICEF incorporate measures for the activities funded by UNICEF to be continued without UNICEF support in the future? *These two questions were merged in the evaluation matrix. Indicators 8.1. Measures taken by UNICEF to ensure sustainability of program

8.2. Financial capacity of Sao Tome and Principe´s Government to manage procurement of vaccines and Cold Chain equipment

8.3. Staff knowledge of Sao Tome and Principe´s Government on how to manage procurement of vaccines and Cold Chain equipment

8.4. Political will of Sao Tome and Principe´s Government to assume procurement of vaccines and Cold Chain equipment

Picture 5. Unicef´s Deputy representative in STP and Minister of Health at the occasion of UNICEF´s commitment to fight Malaria in the country


61

61

63. Funding through Gavi has been decreasing over time and UNICEF has been supporting the government through the Vaccine Independence Initiative (VII)2. Gavi is expected to withdraw its funding by 2023 and STP is preparing to leave the category of Least Developed Countries. This increased pressure for more autonomy as the country relies heavily on the support of Gavi, UNICEF and WHO. The quotes in Appendix I illustrate the concern in relation to STP having to pay for the vaccines in the future.

64. Even the Unicef support through the VII, which has been crucial to the country, if watched closely, shows that there are many challenges. The first financial assistance given took much longer than expected to be paid back and after much pressure from the partners. In addition, the money which was paid back came from Chinese support instead of the GoSTP national budget. Moreover, the 620,000,000 Dobras paid by the Government to Vaccines in 2018 came from resources of the World Bank directed via General Budget Support.

65. In terms of the Cold Chain equipment, after its installation, Gavi funded a training on management of CC equipment in Benin where two government staff attended. Although there has been investment in providing training to government staff to manage the CC equipment, there are concerns about the capacity of the government to actually provide regular maintenance as opposed to helping with emergencies as they arise. A CCE can last for up to15 years, but proper maintenance needs to be undertaken. Negotiations remain open between UNICEF, Gavi and the government.

66. When it comes to financial capacity, there are questions about STP´s capacity to fully fund vaccines and WHO has been supporting the government to find alternative sources of funding, but they still are at an early stage (I8.2). In addition, the national budget of STP, according to one government staff member includes sources of funding from partners which are not secured. This leads to a misleading budget prediction and an expenditure rate of 70%, which means that only 70% of what had been budgeted for is actually spent at the end of the year. There are erroneous projections about the revenues that might be raised and that do not reach the Public Treasure.

67. The other component is the financial capacity of the government to fund the vaccines and CC and to manage the procurement process. International procurement is a complex process that demands intensive research of the market involved, transparent and well-established procedures, availability of resources and timely payment. There is a consensus from UNICEF and the government that UNICEF STP should continue to manage the vaccine and CC procurement procedures. There is a perception from government officials that if the

2 The Vaccine Independence Initiative (VII) is a rotating fund that helps countries meet their funding needs to purchase vaccines and related equipment.


62

62

government took over vaccine procurement, they would pay more for the same goods.

68. The capacity of the government to pay for drugs should be watched closely, so that hardly won gains are not withdrawn. From the part of the government, some measures are in motion, such as factoring the vaccines in the national budget. One concern is the dialogue between the Ministry of Health and Finance, as there are great pressures for fiscal responsibility from the part of the International Monetary Fund which tends to press for cuts in public expenditure. In this sense, efficiency of the health public expenditures becomes even more crucial, thus making the monitoring and evaluation components increasingly important.

69. The evaluation team found that the National Health Fund has some capacity in place for overall general procurement services aligned with procedures from international organizations (I8.3). However, vaccine procurement processes are complex, and there is a large operating capacity already installed within UNICEF. Hence, a strategic focus of sustainability could be financial capacity to fund the supplies needed in the medium term and procurement capacity in the long term.

70. There are debates within the country to create a National Health Committee with sub-commissions for vaccination which could address the purchase of various types of drugs. It would be composed by, amongst other members, the Pharmaceutical Department and a representative of the Information System. There are also debates about alternatives for funding vaccines in the country through: a) an unified tax over consumption; b) raising the taxes over alcoholic beverages; c) taxes on luxury products; d) taxes on petroleum exploration; e) taxes collected from private companies; f) new funding sources from the international community and g) installing electronic systems to better control payment of taxes by businesses. As of January 2020, some of these measures were already in the process of implementation.

71. One of the priorities of the United Nations Development Assistance Framework (UNDAF) for the country is to help with economic development, which seems to be key for the country – generate financial capacity to promote its own development. STP is a country that requires frequent support from the international community.

72. In terms of political will (I8.4) for funding, purchasing vaccines and assuming maintenance of equipment, the evaluation team found that there is more pressure from the international community for a vision of independence and sustainability than a national will in place to achieve it. There is high-level commitment from staff at middle management level and considerable commitment from staff at a local level. However, allocation of resources for vaccination in face of limited funds requires political will not only from the Ministry of Health, but the entire government.


63

63

73. The EPI Planning document for 2016-2020 predicted expenditures of US$ 10,590,667.00 and when it raises the question of funding sources, only the traditional sources (Gavi, UNICEF and the government) are considered. The document gives comments only about the need of better coordinating the support received, organizing management processes, and using resources for increasing efficiency. However, no clear sustainability strategies are devised which is a theme to carefully consider for the elaboration of the new EPI Plan for the period 2021-2025.

Preliminary Conclusion SUST 1: Sustainability is one of the key elements to be observed, not only of the CC, but of the whole vaccine management process in STP. The graduation of STP from the category of Least Developed Countries is making traditional partners withdraw their support. The GoSTP is actively looking for alternative sources of funding (e.g. new taxes, new partners), however, vaccination is only one more element of dispute among the various demands of the government (paragraphs 74 and 75).


64

64

9.5. Human Rights, Gender and Equity

Questions under Human Rights, Gender and Equity: To what extent were Gender, Human Rights, and Equity Principles dully integrated in the design and delivery of the Cold Chain Interventions? To what extent did the EPI program tackle the barriers that prevent girls´ and women´s access to the services in the targeted communities? (EQ 9)

Indicators 9.1. Integration of Gender, Human Rights and Equity principles in Cold Chain management through specific action

9.2. Integration of Gender, Human Rights, and Equity principles in EPI through specific action

9.3. Access to vaccines by gender and economic status

9.4. Measures taken by the Ministry of Health to increase accessibility of Health Units to the caregivers

Picture 6. Focus Group of Rights holders in R.A.P.*

*Health Unit of Santo Antonio in the Island of Principe. Picture edited for now allowing identification of the participants.


65

65

74. The process of Cold Chain management is very technical and specific. During the desk review and field work, no specific issues related to equity and technical aspects were identified. The aim of the CC equipment was to make vaccines readily available to girls and boys equally and to reach the most disadvantaged children in the remote areas of the country, which now have equipment to guarantee its own supply of vaccines for everyone (I9.1).

75. The rates for full immunization in the country are higher in the rural areas, slightly higher for boys (61.62% in comparison to 61.58% for girls) and higher for mothers with no schooling or only primary education (63.66% in comparison with 60.03% for educated mothers). This shows that common inequalities may not explain the reasons why children are not being vaccinated, as they are spread evenly among rural/urban population, boys and girls, and in families of educated/non-educated caretakers (See Figure 5).

Figure 5. Proportion of children completely vaccinated by placel of residence, gender and level of education of mothers in STP in 2017

Source: ECV 2017.

76. ´Equity and equality of access to vaccination are guaranteed to all children, teenagers and adults.’ This is the vision of the EPI Planning for 2016-2020. However, despite the close rates of vaccination between boys and girls, and according to the same planning strategy, young girls face the problems of early and unwanted pregnancies, low levels of education, poverty, and limited job opportunities. There are also limitations in the provision of and access to services and information to young mothers, as they are the ones to access the HU with their children (GoSTP, EPI Planning 2016-2020).

77. In the case of integrating gender, human rights, and equity principles in EPI

through specific action (I9.2), the EPI Plan for 2012-2020 sees the need of reaching non-covered populations through an increase in advanced and mobile strategies to reach the population. Further, ineffective sensitization campaigns and high costs of printing materials are also pointed as fragile aspects of the EPI work. Although this was projected in 2015, the field work shows that some of these


66

66

problems remain as it has been widely reported by nurses and rights holders when suggesting the need of expanding the reach of existing campaigns. The weak data analysis capacity with regards to non-reached target populations in the country given the lack of analytical capacity has been recognized as an area for improvement.

78. As non-vaccinated children are evenly distributed in the overall population

statistics, the evaluation team focused on investigating the reasons why caretakers might not take their children to vaccinate. In the process of asking caretakers their reasons and perceptions, various important issues surfaced. The common reasons mentioned were:

a. Access to Health Units is difficult due to distance, price paid for transportation

and precarious roads (more difficult to access with small children); mentioned by nurses and caretakers as one of the problems for not taking the children for vaccination (I9.4). Access may be hampered by: 1) cost of transportation or 2) location of a Health Unit in an area with mud creating difficult access with young children. The strategy of the Minister of Health has been to go to families as opposed to increasing accessibility to the Health Units. Vehicles are now in use to help reach the families. However, combined actions of improving urbanization close to Health Units could also help to improve access and thus facilitate caregivers to access the locations.

b. Caretakers do not like to go to the Health Units as they feel they lose too much

time increasing their workload: there may still be limited knowledge by a subgroup of the population about vaccine benefits, leading to little encouragement to go to Health Units for vaccination. Although there is a tradition of promoting campaigns in the country, most respondents during field work stated how more campaigns were needed to help educate the population about vaccination benefits and guide them through the process. There is also a concern that the campaigns should be intensified, using radios, encouraging health staff to go talk to caretakers and also involving other actors such as the Church and local Civil Society Organizations. Furthermore, there is an acknowledgment of the need of explaining to mothers how necessary the second and third doses are (information and education for health). A suggestion given by one caretaker was to make Health Units more attractive for mother and children (e.g. having a playground for the children etc.). The quality of health facilities varies significantly throughout the country.

c. Mothers do not like the vaccine reactions, because as children get

vaccinated, they may cry and disrupt the family routine. When it comes to vaccine management, there were various complaints about the side effects of vaccination. In this case, the evaluation team could identify the need to clearly explain the side effects to mothers so that they feel better prepared to deal with them. A possible strategy would be to give Penta preferably in the


67

67

morning and inform the mother of a possible painkiller if the child cries or exhibits additional side effects.

79. Other relevant issues that might be preventing health care in the country include

limited hours of service. Caretakers may be working and have difficulties in taking their children to Health Units during work hours. In addition, nurses have reported limited resources for fuel when it comes to field visits. Another matter that was observed and that would require further investigation is the lack of doctors in the Health Units. Not all Health Units had doctors available at the time of the field visits. They were absent for personal reasons or time shifts, for example. This would require further investigation. There was also one participant of a focus group who reported religious reasons for caretakers rejecting vaccines. She said that fathers may report that vaccines do not protect the children, rather God who protects the children.

80. A specific gender equity issue was identified in the focus groups. Even though the

sample was very small (as discussed in the methodology section in the Appendix I), it deserves attention of an issue for future investigation. The evaluation team found that fathers may sometimes be opposed to vaccination. When asked why, mothers reported that it is due to the disruptions it causes in the family and fear that mothers may receive services of family planning without their consent. Some of the fathers reported that women might end up cheating on their partner because of condom usage or that, after having a baby, mothers should stay at home and take care of the children where there is less risk of infidelity. This issue of equity needs to be considered in future research. However, it has to be reported that even though, the evaluation team found evidence of this resistance from fathers of family planning, this seems to be residual, as vaccination rates are high in the country. The Textboxes 5 and 6 in Appendix I illustrate this perception by fathers.

81. When caretakers were asked about effective ways of encouragement to take

their children to the Health Unit to be vaccinated, some type of recognition was suggested which was also reinforced by other consultants that came to the country (Firmin, 2017). Possibilities included: putting a star on the vaccination card, giving out a bag for those who had fully completed the vaccination cycle for their children.

82. One crucial policy for increasing vaccination in STP and that contributes to equity

is the active search for children. It works through the following actions: 1) Call caretakers for second doses when they have not come to the Health Unit at the predicted date for vaccination; 2) Visit children and caretakers for vaccination; 3) Approach mothers and ask them to come to the Health Unit for vaccination when they are met on the street or when a neighbor is met. These simple, but crucial actions demonstrate great effort to find unvaccinated children. There are also joined actions with other sectors. In order to enroll in school in STP, for instance, it is mandatory to show the child´s vaccination card.


68

68

83. Other reasons for high vaccination in the country reported by various stakeholders

are: stock planning capacity, frequent campaigns, motivation of key staff members and leadership. In the case of STP, there is an understanding by various actors that vaccination has been a priority for the government which has been consistent overtime and with support from important donors like Gavi and traditional supporters such as UNICEF and WHO. There is also commitment from key staff in the government in favor of home visits. In the process of reaching children for vaccination, there is the involvement of external partners, the Red Cross and the MARAPA. The first works in the interior of the country and the latter on the coast.

84. Seeking the unvaccinated children in a country with 95% coverage rate poses an

additional challenge of identifying the children and reaching them. In this case, quality of information and planning, such as micro-planning, becomes crucial. There is also a perception from key stakeholders from the international community and also influencers in the country that not enough capacity is in place in the Health Units to conduct analyses of coverage by District and sub-districts to evaluate the remaining population that needs vaccinating. There are also gaps of information in terms of the target population. The lack of integrated information systems has been identified as a critical issue. There is not a single database that allows children to be accounted for wherever they vaccinate. Each Health Unit has a target of children to vaccinate according to the number of births in each region. Each child has a health booklet and the Health Units register the vaccination and other services on it. However, if children get vaccinated outside their district, the information is not always passed from one Health Unit to the other.

Preliminary Conclusions

EQU1: Important obstacles were identified that might prevent caretakers from vaccinating their children. These include: a) difficult access to Health Units; b) time spent at Health Units; and c) adverse events following immunization (AEFI) as children may cry and disrupt the family routine. EQU2: Important issues related to gender equity were found out, not in relation to the CC alone, but in relation to a smaller group of males who may pose obstacles to vaccination due to the concern their spouses may be receiving family planning advise and drugs without their consent. Attention should be paid to this subgroup of caretakers where the relation between immunization and family planning may be negative. EQU3: Factors that may be hindering health care pass through lack of integrated information systems, proper feedback to mothers and quality of treatment that will encourage mothers to come back.


69

69

10. Final Conclusions

Conclusion 1: UNICEF, Gavi and WHO have been critical partners in support of the expansion of immunization in STP through funding, procurement and technical assistance in the case of CCE and beyond. Without such resources, procurement systems and guidance on international best practices and data systems, STP would have probably progressed much slower in the area. With respect to the CCE selection, there was alignment among the three partners. However, a gap was identified: the lack of stainless-steel materials, which would benefit stakeholders in a country like STP, which is heavily exposed to the corrosive effects of salty seaside air. REL1: The evaluation team found that there was close collaboration between UNICEF´s various divisions, Gavi Alliance and the Government of Sao Tome and Principe to purchase the Cold Chain (CC) equipment. Such need was identified by a World Health Organization (WHO) study and became apparent during the planning of the Expanded Program on Immunization (EPI) for 2016-2020. The CCE was outdated with difficulties in repairing pieces, especially at the local level. There was evidence that the equipment purchased was inadequate for resisting corrosion caused by the salty seaside air of STP, as its grids were not made of stainless steel.

Conclusion 2: The CC investment has contributed to increasing vaccine availability at the HU level and free up health staff time, due to the less frequent visits to the District vaccines collection points away from the HU. However, the CC program has neither directly impacted the quality of health care or increased vaccination rates in a context that feature an already high vaccination rate. EFF 1: There is evidence that the CC investment contributed to increasing overall vaccine availability in Health Units, except for bVPO, as the new reliable equipment allowed for increased and steady storage capacity. There is evidence that vaccination rates have increased or remained stable between 2017 and 2018 with a tendency to maintain or increase stability for some vaccines in 2019 (according to the data available). However, this increase or stability in vaccinations may not be attributed directly to the CC, as it is the result of a number of other factors, including the CC. On a local level, the impact of the CC is also mixed. In some Health Units, the new equipment has brought increase in vaccination sessions, in others it has not. There was a decrease in vaccine collection visits which represented a gain for the HU staff which had to do less frequent visits to the District Centers, allowing them to direct their time to other activities (paragraphs 11-18).

EFF 2: Quality of health care is the result of a number of factors, from which CC is only one component. There was no evidence of improvement of health care delivery due to the type of equipment or installation method and verification system. There is


70

70

evidence that the new equipment helped by enabling more time for health workers to carry out their duties (paragraph 20).

EFF 4: There have not been innovations identified from the point of view of improvements in the supply chain or data systems. There was an unexpected outcome which was the impact of the new CCE in the health agents’ work environment. (paragraphs 29, 30 and 31).

Conclusion 3: The programme was efficient in acquiring the CC equipment at a rather more affordable cost than what the Government by itself or UNICEF would have incurred by itself. However, the roll-out of the installation process was less efficient, mainly due to infrastructure issues and inadequate quantity and quality of the training provided to nurses that were responsible for the CCE maintenance. Maintenance was also a key area of concern on the efficiency front as the equipment was deteriorated in one of the country’s District because of the corrosive effects of sea air and the lack of adapted maintenance. REL 1: (…) There is evidence, however, that the equipment purchased was at least for one HU not completely adequate to the sea air of STP as its grids were not made of stainless steel. (Paragraphs 3, 4, 5, 6).

EFF 3: There are challenges in carrying out proper maintenance of the CC equipment due to lack of technical knowledge among health workers and the lack of specialized care for the solar panels (paragraphs 21-27).

EFFI 2: The process of acquiring the CC is a standard procedure in which all of the parties involved are used to. There were gains in efficiency as all the parties are part of a larger partnership where overhead costs are already paid for. The amount of staff involved was adequate in relation to the actions planned (paragraphs 46-52).

EFFI 3: Implementation challenges primarily included the lack of infrastructure in the HU that did not meet minimum standards for installation. Although most solar panels and equipment were installed according to the timeline, there were 3 refrigerators that remain uninstalled nearly two years after delivery(paragraph 54).

Conclusion 4: As the trasnsportations costs to STP are high, discussions were held to foster collective purchase and the set-up of a Regional Warehouse. However, these ideas pose great challenges in terms of airline connections between STP and its “neighboring” countries in the Region and did not resonate particularly well with the unique history, culture and pharmaceutical structures featured by the countries in question. EFFI 1: Transportation costs in STP are considerably high due to its location and the relatively small supply quantity ordered. However, there is no availability and reliability of airlines within the Region to allow for collective purchase of vaccines and a regional warehouse at the moment. In addition, the three countries considered (STP, Gabon and Guinea Equatorial) have very different pharmaceutical set ups due to history and


71

71

culture which pose high costs of coordination and transaction (paragraphs 44 and 45).

Conclusion 5: Although the GoSTP is making a great effort to reach all children via home visits and calling the mothers, the collection and harmonization of data across the country represent a great challenge to identify missing children. Such data is crucial to develop a micro-planning strategy which will be handed over to the nurses to support appropriate and focused actions.

EFF5: Innovations have been identified across districts in reaching unvaccinated children. Different actions are being taken such as calling caretakers, calling their relatives and reaching out in different ways to vaccinate the children (paragraph 32).

EQU3: Factors that may be hindering healthcare are a lack of integrated information systems, proper feedback to mothers and quality treatment that will encourage mothers to return.

The data problem makes the missing children invisible, as they may be in places where HU are not aware of them and at the same time may have been accounted for in other places. These missing children are the most critical group and their data are lost in the middle of the way. Getting to the remaining 5% of children that needs to be vaccinated will require effort in terms of data gathering and also data analysis to make focused interventions to reach them.

Conclusion 6: The funding available for the purchase of vaccines i and other related supplies in STP may be in danger in face of Gavi withdrawal of the country by 2023. SUST 1: Sustainability is one of the key elements to be observed, not only of the CC, but of the whole vaccine management process in STP. The graduation of STP from the category of Least Developed Countries has caused traditional partners to withdraw their support. Although the GoSTP is actively looking for alternative sources of funding (e.g. new taxes, new partners) vaccination is only one more element of dispute among the various demands of the government (paragraphs 74 and 75).

Conclusion 7: There are still important obstacles for caretakers to vaccinate their children and increase immunization in the country. Accessibility, quality of information, and health care and adverse associations between vaccination and family planning were identified. EQU1: Important obstacles were identified that might prevent caretakers from vaccinating their children. These include: a) difficult access to Health Units; b) time spent at Health Units; and c) Adverse events following immunization (AEFI) as children may cry and disrupt the family routine.

EQU2: Important issues related to gender equity were found out, not in relation to the CC alone, but in relation to a smaller group of males who may pose obstacles to vaccination due the concern their spouses may be receiving family planning advise and drugs without their consent. Attention should be paid to this subgroup of


72

72

caretakers where the relation between immunization and family planning may be negative.

Conclusion 8: The Theory of Change underlying CC activity and processes is rather complex. Overall, the Government manages well the CC activities and processes that are under its responsibility. However, there are several CC activities and processes that the Government still does not have the capacity to manage, namely the procurement and maintenance of CC equipment.

The Theory of Change behind CC involves a sequence of actions (each of whom involves a plurality of stakeholders) from the generation of data on CC needs to the launch of the procurement process, followed by the arrival of the CC equipment at the airport, the related customs clearance, transportation, installation, maintenance and final distribution of vaccines (See Fig. 1). The evaluation team found that the management of the CC, in terms of installation, guidance, training and daily maintenance are well taken care of by the government. However, it is the broader chain, which spans from procurement to regular maintenance that seems to be beyond the reach of the government.

International procurement takes much effort. Procedures and regular maintenance are key to keep equipment functioning well. The Theory of Change is helpful in terms of helping government officials and UNICEF identify the gaps of capacity building that need to be examined. This conclusion is connected to the findings EFF4 and SUST1.


73

73

11. Lessons Learned This section provides a quick overview of the three lessons learned that the evaluation team was able to identify during the fieldwork. Such lessons learned are all the more important as they may inform the planning and implementation of CC programs among Gavi, UNICEF and other partners not only in STP but also the rest of the region.

1. The installation of CC equipment may increase vaccine availability to the public and positively impact quality of the work environment.

The evaluation found an unexpected outcome of the CC in STP. The new equipment helped improve the work environment. The nurses enjoyed having equipment which had a prettier layout, did not emit smoke, was more reliable and easier to take care of. This is unexpected, as one might consider solely the functionalities of the equipment. However, impact on the quality of the workplace increased positive feelings and satisfaction for professionals about their work environment.

2. In a country with high vaccination rates such as STP, where availability of vaccines is already higher-than-average, the impact of new CC equipment may be limited.

Differently from other countries in the Region, vaccination rates for STP are much higher. There was already availability of CC equipment in STP, with gaps predicted for the introduction of new vaccines. The equipment was old, but still working. In this sense, one cannot expect great impact on immunization rates. Finding the remaining 5% of unvaccinated children in STP will require much more strategic and focused types of interventions.

3. The linkages between vaccination and family planning could increase undesired effects that require close monitoring.

The evaluation team found that there may be negative associations between vaccination and family planning that need be monitored. In STP, both vaccination and family planning services are managed by the same department of the Ministry of Health. These problems may be due to low quality of services or lack of dialogue among men and women when carrying out family planning. The evaluation team identified the importance of specific ´mental barriers` (cultural barriers) against vaccination that should be identified and worked through in the community with informative, culturally sensitive campaigns.


74

74

12. Recommendations Based on the different findings and conclusions and in line with the table of “Evaluation users and uses” presented in the introductory section of this report, the evaluation team put forward a number of recommendations (strategic and operational) whose objective is to contribute to future program improvements These recommendations were validated during several exchanges held among the evaluation team and UNICEF Country Office staff as well as other stakeholders. These exchanges focused on two particular aspects of the recommendations, namely feasibility and relevance to the implementation context. This process also contributed to the prioritization of the recommendations in question. All of the recommendations were rigorously based on findings from the evaluation process and the dialogue with the different evaluation participants which helped to identify what could be the right timing for action (a window of opportunity).

Recommendation 1, Priority 1 Type: Strategic Recipient: Government and UNICEF Recommendation: Explore alternative funding mechanisms for vaccination in the country in conjunction with advocacy for immunization. Action points: The government is making progress to increase its tax base to expand revenues in the country. However, immunization is one more demand among many the country faces. A conversation should be carried out with Gavi Alliance on this topic and assistance of the Regional Office is recommended to help with the identification of alternative funding sources. In addition, this recommendation should be read with Recommendation 2 which is about advocacy for this agenda in the country.

Rationale: This recommendation was based on dialogue with different partners from the government (Gavi, UNICEF, WHO) and staff from the government in various areas and at different levels (national, district and local) and on Conclusion 6: The funding of vaccines in STP and related supplies may be in danger with the potential withdrawal of Gavi from the country by 2023.

Recommendation 2, Priority 2 Type: Strategic Recipient: Government and UNICEF Recommendation: Promote high level advocacy for immunization in the country and promote STP as a stellar source for vaccination coverage to other countries of the region (which should also include learning with other countries in areas such as data analysis, district planning and dialogue with public stakeholders). Action points: Exploring funding alternatives is not sufficient, if there is not a vision that immunization is important. More advocacy is needed to help the immunization agenda be prioritized, not only by the Ministry of Health, but by the Prime Minister´s office, the President, and key political parties in the country. It should be part of the political debate in the country. ´A reason of pride for STP that should be cared for`.


75

75

The development of an advocacy strategy using the experience of UNICEF in communication and a South-South Cooperation exercise is recommended in this case. Rationale: This recommendation was based on Conclusion 5 and evidence from the report around the experience of STP in promoting high immunization rates. In addition, it was based on the dialogue with the Country Office and government staff members in other areas beyond Health. RSTP has the highest immunization rates in its Region and is a good example to the continent. However, since it is a small country, its merits of achieving this high coverage are not well acknowledged by other partner countries. There is great effort in increasing immunization in the country which has the potential of being shared with other countries. This protagonist role of the country could help with the internal advocacy for vaccines and boost morale, thus creating a virtuous cycle for the theme in the country. In addition, there are emerging experiences in the Region that could be beneficial to STP, such as planning by District in Senegal and dialogue with local leaders and organizations in Cameroon.

Recommendation 3, Priority 3 Type: Strategic Recipient: Government and UNICEF Recommendation: Improve quality of data and implement an effective information system with Monitoring and Evaluation mechanisms to help reach the remaining unvaccinated children. Action points: The central government could stimulate health workers to use and send reliable data, how to interpret them and how to develop a strategy especially in low performing areas. This will enable them to use limited resources (e.g. petrol or human power) efficiently to vaccinate in regions with high dropouts and missed opportunities. Further support to DHIS2 should be given and a monitoring and evaluation strategy for the EPI should be devised with training mechanisms and in dialogue with the Recommendation 4 of developing microplanning for the Districts. Rationale: This recommendation was based on Conclusion 5 and extensive evidence from the report and dialogue with rights holders, local health staff, national EPI staff and the Country Office. Data of missing children are not accurate as families migrate and there is not good transfer of information among the HU. There is already a mobilization from various partners around District Health Information Software 2 (DHIS2) with good prospects of expansion, however the system is still at an early stage and health agents are not fully equipped to use it. Recommendation 4, Priority 3 Type: Operational Recipient: Government and UNICEF Recommendation: Reinforce capacity building of data analysis and micro-planning (by District and Health Unit) to reach unimmunized children.


76

76

Action points: Training should be given for professionals at a local level to help them process immunization data and devise specific strategies to locate unvaccinated children. Exchange with other countries of the Region who have experience with microplanning is advised. Rationale: This recommendation was based on Conclusion 5, dialogues with the Regional Office and the revision of Regional experiences where microplanning was identified as a practice in other countries. Recommendation 5, Priority 4 Type: Operational Recipient: Government and UNICEF Recommendation: Increase communication campaigns and education for health in the area of immunization and develop direct methods to communicate with families (e.g. exploring new technologies). Action points: More communication should be in place in schools and with civil society organizations to enforce sensitization campaigns. UNICEF has a global expertise in the area which may be helpful for the government. In addition, new ways of communication with mothers could be further explored, such as the use of applications for cell phones to remind of vaccination dates. The draft of a new communication strategy for the EPI is advisable.

Rationale: This recommendation was based on Conclusion 7, dialogue with rights holders, influencers in the country, and local health staff. This recommendation came directly through the opinions of rights holders. There are already communication efforts in place, but more needs to be done and continuously on the importance of immunization and the vaccination calendar.

Recommendation 6, Priority 1 Type: Operational Recipient: Government and UNICEF Recommendation: Check all the new CC equipment on the impact of the seaside air and draft an appropriate maintenance plan which may include the hiring of a maintenance company for the solar panels and install the three remaining CC equipment. Future purchases should include the requirement of stainless steel materials.

Action points: The installation of stainless walkway grids is recommended for the solar panels to prevent falls and to ensure the roof is not damaged while the solar panel cleaner is trying to reach or clean the solar panels. A note for the Supply division should be sent about the importance of using stainless steel equipment for the Cold Chain in the Region, to avoid risks of corrosion caused by the seaside air.

Rationale: There is no planning in place to help maintain the CC equipment. This should be urgently considered for guaranteeing long term use of CC acquired.


77

77

Recommendation 7, Priority 5 Type: Operational Recipient: Government and UNICEF Recommendation: Develop a management protocol for the CC which includes forms for daily, weekly and monthly maintenance, location of equipment and frequency of vaccine collection (explore developing a quality management according to ISO 9001:2015). Action points: This management protocol and training should include: - An agreed procedure about how often the drainage plug should be opened (once

a day or once a week and/or depending on the condensation). - A working sheet (like the one for the temperature) for the daily maintenance, weekly

maintenance and monthly maintenance - so somebody can ensure the task was completed.

-The location of the CC equipment should be away from the children, to make sure they do not harm the equipment (e.g. by having access to the temperature adjustment panel or the drainage plug) or take appropriate measures such as putting a staple protective shield in front of the temperature panel to make sure no kid can accidently change the temperature. - The lot number of vaccines written in the baby book. If there were a recall of a lot,

the attendant should be able to understand where the vaccine is still available or to whom the vaccine has been given. As mothers go to different Health Units for vaccination it might be a time-consuming procedure to find the information. Also, in case of side effects of a vaccine, follow-up is crucial if there have been side effects observed by other children vaccinated with the same lot. The same procedure should be introduced to the TT- vaccine program for pregnant mothers.

- In case of Health Unit’s, there is enough storing capacity in the new CC for 2 (two) months. The frequency of vaccine collection to the Health Centers can be reduced.

Incentives should be considered for following the protocol that could be monitored by the Central Government. Possible incentives could include training opportunities, tokens, public recognition in monthly or bi-monthly national or district meetings, priority during maintenance visits. Rationale: This recommendation was based on Conclusion 3 and the dialogues with the local professionals (e.g. nurses in the Health Units visited).The key needs were identified from various stakeholders and summarized above. Recommendation 8, Priority 4 Type: Operational Recipient: Government and UNICEF Recommendation: Promote training on management of CC equipment, managerial skills and health care. Action points: Training should carefully include guidance on: - Storage of vaccines (e.g. to avoid oral vaccine in the Hep B packing equipment).


78

78

- Use of suitable container under the drain hose to catch the water and make sure that the appropriate container is not accidentally used by a human or animal to drink fresh water afterwards.

- Minimize the amount of expired vaccines by wrong handling of the CC. - Cleaning the ventilation grids and the condenser with the supplied cleaning brush

or a vacuum cleaner to avoid condensers being damaged. Brushes should be checked and the length of the rope to ensure adequate length for cleaning without damaging the equipment.

Rationale: This recommendation was based on Conclusion 3 and 5, dialogue with the local health professionals, national EPI staff and stakeholders from the private sector. The government is only assuming a small part of the CC cycle. Thus, building capacity in various areas are important to reinforce visualization of protocols and manuals in place, reinforce training of nurses to guide caretakers to recognize harmful symptoms after vaccination and to improve training on monitoring vaccine´s second and third doses.

13. Annexes

Appendix A. UNEG Evaluation norms and standards

Table 19. Compliance to UNEG evaluation norms and standards

UNEG Norm Compliance measures in the context of the evaluation Norm 1: Internationally agreed principles, goals and targets

The inception report discussed the context of the evaluation in relation to Sustainable Development Goal 4 and targets 3.2 and 3.9. b

Norm 2: Utility During inception phase and field work, the evaluation team talked to all the relevant stakeholders to ensure that the focus of the report meet their needs.

Norm 3: Credibility The evaluation methodology was developed according to international evaluation practice and with the collaboration of UNICEF to ensure that trust was developed between the commissioning organization and the evaluation team and the products met UNICEF’s quality standards.

Norm 4: Independence

None of the evaluation team members were involved with the design or implementation of the CC initiative or had worked with UNICEF STP previously or the EPI program in the country.

Norm 5: Impartiality All members of the evaluation team have a record of working with different stakeholders with diverse interests and are able to dialogue without compromising their integrity and impartiality.

Norm 6: Ethics The aim of the initiative and evaluation process was to increase immunization rates in STP. This was repeated to all the interviewees and the evaluation process was carried out with this aim in the mind. Confidentiality and anonymity were also ensured to respondents at all times during the data collection.

Norm 7: Transparency

All the evaluation phases were developed with the acknowledgement of the UNICEF team in STP and all the information collected was documented and will be delivered at the end of the evaluation process.

Norm 8: Human rights and gender equality

Explicit concern with human rights and gender equality was addressed in EQ 9.

Norm 9: National evaluation capacities

This evaluation did not have the aim of specifically targeting this topic, however, it is in the context of immunization in STP to address Monitoring and Evaluation (M&E) for health and this was addressed in the report.

Norm 10: Professionalism

The products were delivered on-time and in the case of the second draft of the report, a time extension was requested due to difficulties of data collection. The team made changes to the products as requested and the relationships built around the process were cordial and empathetic.

Norm 14: Evaluation use and follow-up

During the inception phase and according to the TOR, the uses of evaluation were discussed. The follow-up will be the mandate of UNICEF to carry out. Evaluation findings will also be validated by a Steering Group representing stakeholders, to ensure ownership of recommendations.


80

80

Appendix B. Stakeholder Analysis

The key stakeholders of the initiative were initially identified in 1.1., however, in this section, this topic is further detailed to ensure that all of the participants involved were properly considered and consulted. The stakeholders were classified according to their level of involvement in the initiative:

a. Implementing UN agency: this refers to the main UN stakeholder who is in charge of the initiative, which is UNICEF in this case.

b. Development partner: it refers to all the development partners involved in the initiative who could be in advisory roles, carrying out joined projects or as donors.

c. Implementing partner: this includes all the partners on the ground that have had an active role in any actions directly involved with the initiative.

d. Primary duty bearer: this refers to the public stakeholders who are responsible for the initiative and promoting the rights of the population, in this case, right to health and vaccination (e.g. Ministry of Health).

e. Secondary duty bearer: this refers to public stakeholders as stated in letter d, but who have a secondary responsibility in promoting the rights of children (e.g. Departments of the Government of STP not directly related to health like the Ministry of Finance).

f. Rights holders: this refers directly to the end-beneficiaries, in this case, caretakers and children.

g. Influencer: this includes external stakeholders to the initiative who may have some degree of influence over the CC, the EPI, or the health agenda in the country as a whole.

Table 19 details the stakeholders and their roles involved in the CC management. All of the stakeholders mentioned have been considered during the evaluation process in one degree or another.


81

81

Table 20. Stakeholders involved in the CC management and their roles

N Name Position/ Organization

Role Contribution

1 Mariavittoria Ballota

Deputy Representative

Implementing UN Agency

Overall supervision of the initiative

2 Luis Bonfim, Health specialist Implementing UN Agency

Interface with the GoSTP, direct contrition to the initiative

3 Jean-Cédric Meeûs

Regional Chief of Supply in West and Central Africa


Advise to the initiative

4 Serge GANIVET

Cold Chain Specialist


Adviser to the initiative

5 Claudina Augusto da Cruz

Focal Point WHO Development Partner

Partner of UNICEF and the GoSTP in the EPI

6 Thierry Vincent

Focal Point Development Partner

Donor of the Cold Chain and HSS

7 Edgar Neves

Minister Primary Duty Bearer

Overall supervision of the initiative within the Ministry workplan

8 Solange Barros

Coordinator of Reproductive Health Program and EPI

Primary Duty Bearer

Supervision of the CC

9 Vladimir Sousa

National Cold Chain manager

Primary Duty Bearer

Direct manager of the CC

10 Ineida Monte Verde

Coordinator of SIS Secondary Duty Bearer

Interface with the initiative, key system for data of children

11 Hugo Coordinator of HSS in the Ministry of Health

Secondary Duty Bearer

CC is under a broader support of HSS managed by this stakeholder

12 Carlos Benguela

Director Customs Office

Secondary Duty Bearer

CC acquisition and vaccines involve the relation with the Customs office to

enter the country 13 José

Eugénio Silva

CEO Development Partner

Installation of CC equipment.

14 Heads of Districts

Primary Duty Bearer

Overall supervision of HU in the district, collection of vaccines in the

Central Distribution Centre and distribution to HU

15 Nurses of Health Units

Primary Duty Bearer

Direct care of the CC equipment

16 Mothers Rights holders Direct beneficiaries of the CC with their children

17 Fathers or other relatives of the children

Rights holders Direct beneficiaries of the CC with their children


82

82

Appendix C. List of people interviewed Table 21. List of people interviewed

N Name Title Organization 1 Mariavittoria Ballota Deputy Representative UNICEF Sao Tome and Principe 2 Luis Bonfim, Health specialist UNICEF Sao Tome and Principe 3 Alan Batista Human Resources

Officer UNICEF Office Sao Tome and

Principe 4 Madgzer de Souza Administrative Assistant UNICEF São Tome and Principe 5 Alain da Cruz Deputy Representative

Operations UNICEF Area Office for Gabon and

Sao Tome & Principe 6 Rachel Lebomo Operations assistance UNICEF Area Office for Gabon and

Sao Tome & Principe 7 Jean-Cédric Meeûs

Regional Chief of

Supply in West and Central Africa

UNICEF

8 Serge GANIVET Cold Chain Specialist UNICEF WCARO 9 Claudina Augusto

da Cruz Focal Point World Health Organization

10 Thierry Vincent Focal Point Gavi Alliance 11 Edgar Neves Minister Ministry of Health 12 Neurice Ramos Director National Drugs Fund, Ministry of

Health 13 Solange Barros Coordinator of

Reproductive Health Program and EPI

Ministry of Health

14 Vladimir Sousa National Cold Chain manager

Ministry of Health

15 Carlos Benguela General Director Customs Office 16 Mandu Customs Officer Airport 17 Jersey Cady Public Forwarding

Agent Customs Office

18 Américo Mendonça Morais

Public Forwarding Agent

Customs Office

19 Carlos Alberto Joaquim da Costa

Head of the Department of Budget

Management

Ministry of Finance

20 Ineida Monte Verde; Coordinator of SIS Ministry of Health 21 Hugo Coordinator of HSS Ministry of Health 22 Marisa Conceição

Doctor, Health officer Health Unit Madre Deus

23 Idjamila Semedo Nurse, Focal Point Health Unit Madre Deus 24 Samoa Espírito

Santo Director Drugs Department, Ministry of Health

25 Vania Castro Pharmacist Drugs Department, Ministry of Health 26 Jerisson Ramos Health expert Drugs Department, Ministry of Health 27 Brigite Pires do

Santos Health expert Drugs Department, Ministry of Health

N Name Title Organization 28 Adilé Conceição Coordinator, Head

nurse, Água Grande Distribution Centre

29 Silvio Vera Cruz Doctor, Health officer Hospital R.A.P.


83

83

30 Lourdes Managem Head Nurse, Coordinator of Principe Distribution Centre

Príncipe Distribution Centre

31 Manuel Tomé Lima Health Officer District of Lembá 32 Jessica das Neves Nurse Lembá Distribution Centre 33 Maria de Jesus Alves Nurse Lembá Distribution Centre 34 Nicolau Martina Nurse, focal point Diogo Vaz Health Unit 35 Sorteana Lima Head nurse Coordinator of Lobata Distribution

Centre 36 Igualter Daio Nurse, Focal Point Desejada Health Unit 37 Preciosa Barreto Nurse Desejada Health Unit 38 Elanauria Choi Coordinator Me Zochi Distribution Centre 39 Maria Tomé Nurse Bom Bom Health Unit 40 Pedro Mateus Head nurse,

Coordinator Caué Distribution Centre

41 Ulisses Lima Nurse Ribeira Peixe Health Unit 42 Cleider Afonso Nurse Ribeira Peixe Health Unit 43 Maida Ceita Head Nurse,

Coordinator Cantagalo Distribution Centre

44 José Eugénio Silva CEO Electrofrio 45 António Lima Director Vale Flôr Institute 46 Jajalina da Graça Rightsholder, mother Focus Group Neves 47 Joaneza Santos Rightsholder, mother Focus Group Neves 48 Judite Lázaro

Afonso Rightsholder, mother Focus Group Neves

49 Julieta nascimento Rightsholder, mother Focus Group Neves 50 Maria Diogo Rightsholder, mother Focus Group Neves 51 Atanizia Lorenzo Vaz

Alcantara Rightsholder, mother Focus Group Trindade

52 Ilodizecia José da Costa

Rightsholder, mother Focus Group Trindade

53 Vaikeny Sacramento Afonso

Rightsholder, mother Focus Group Trindade

54 Adelina Soares Rightsholder, mother Focus Group Santo Antonio 55 Andreza Paraiso do

Espirito Santo Rightsholder, mother Focus Group Santo Antonio

56 Dagma Maya Rightsholder, mother Focus Group Santo Antonio 57 Lourenço Santa

Rosa Rightsholder, mother Focus Group Santo Antonio

58 Feliciana Vaz do Rosario

Rightsholder, mother Focus Group Santo Antonio

59 Pascoa Bom Jesus de Almeida


60 Ironia da Cruz Cravid


61 Josefina Tavares Leal


62 Carlos dos Anjos Rightsholder, father Focus Group, Angolares 63 Juvenal Paiva Rightsholder, father Focus Group, Angolares 64 Olimpo Glória Rightsholder, father Focus Group, Angolares 65 Wilker Monteiro Rightsholder, father Focus Group, Angolares

Appendix D. Evaluation Matrix

Evaluation question 1: To what extent have the vaccine Procurement Services related to the Cold Chain rendered by UNICEF (the only operator in this area nationwide), met the country´s needs, especially during the transition period towards Vaccine Independence Initiative (VII), defined by UNICEF Supply Division jointly with Gavi Alliance?

DAC Evaluation Criterion covered by this Evaluation Question: Relevance Evaluation Question Background (short narrative): This question will look at to which extent UNICEF´s provision of procurement services related to Cold Chain in São Tomé met the government´s needs and expectations.

Indicators/Criteria Source of Information

Data Collection

Tool

Preliminary Findings

1.1. Participation of national government officials in the selection of SDD equipment.

- Central Government officials

- UNICEF Staff - Procurement

documents

- Semi-structured interviews

- Desk Review

The National Health Strategy for Immunization considers the need of the increasing the number of refrigerators to store the vaccines at a central level but does not mention the type of equipment to be acquired. The participation of the government in the choice of purchases will be further investigated in the field.

1.2. Alignment between government´s expectation in the procurement process and actual practice.

- Central Government officials

- UNICEF Staff - Procurement

documents


- Desk Review

UNICEF is a long-time partner of the government in São Tomé. It is expected that there is already alignment in terms of working modality between stakeholders. To which extent this alignment is in place will be investigated during field work.


85

85

Evaluation question 2: To what extent have CCE investment contributed to improving immunization service delivery? (e.g. changes in frequency on immunization sessions, changes in frequency of vaccine collection/distribution, averted missed opportunities due to improved vaccine availability)?

DAC Evaluation Criterion covered by this Evaluation Question: Effectiveness Evaluation Question Background (short narrative): This question will explore data of vaccination collected before and during field visit to analyze changes in vaccine availability, increase in immunization sessions and frequency of vaccine distribution.


Data Collection Tool

Preliminary Findings

2.1. Increase in vaccine availability

- Administrative Data from Health Units

- Families accessing Health Units

- Desk review during field work

- Focus groups with mothers

There are no data available during this moment of inception. The data will be collected during field visits. The capacity of the storage of the country has supposedly doubled after the purchase of the new refrigerators, however, the field work will show to which extent the refrigerators were replaced, or the capacity actually increased and led to increase in availability of vaccines.

2.2. Increase of immunization sessions after use of refrigerators purchased by the program.

- Administrative Data from Health Units



There are no data available during this moment of inception. The data will be collected during field visits.

2.3. Increase in frequency of vaccine distribution.

- Administrative data from main distribution center and district centers


There are no data available during this moment of inception. The data will be collected during field visits.


86

86

Evaluation question 3: What are the factors (e.g. equipment type and/or model, health care workers conducting preventive maintenance tasks, installation method(s), verification systems, service delivery models, etc.) that have facilitated healthcare delivery? What factors have hindered the healthcare delivered?

DAC Evaluation Criterion covered by this Evaluation Question: Effectiveness Evaluation Question Background (short narrative): The factors that have both facilitated and hindered healthcare delivery will be explored, including but not limited to: type of equipment, qualification of health care workers, installation method and verification system an overall methods of service delivery.

Indicators/Criteria Source of Information Data Collection Tool Preliminary Findings 3.1. Improvement of health

care delivery due to type of equipment.

- Government officials at a central level - Government officials at district level

Government officials at local level

- Semi-structured interviews with government officials at the three different levels


There are no data available during this moment of inception. The data will be collected during field visit.

3.2. Qualification of health care workers in conducting preventive maintenance tasks.

- Government officials at a central level - Government officials at district level - Government officials at local level



3.3. Improvement of health care delivery due to installation method and verification system.

- Government officials at a central level - Government officials at district level

Government officials at local level



Evaluation question 4: What are the unexpected outcomes (positive and negative) produced within the scope of the procurement and distribution of vaccines?

DAC Evaluation Criterion covered by this Evaluation Question: Effectiveness


87

87

Evaluation Question Background (short narrative): This question will examine unexpected outcomes with a focus on innovation of the supply chain.

Indicators/Criteria Source of Information Data Collection Tool Preliminary Findings 13.1. Catalyzation of

innovations by CC investments and procedures in continuous improvement of supply chain.

- Government officials at a central, district and local level

- Companies involved in the supply chain

- Reports from UNICEF and Gavi Alliance


- Focus groups - Desk Review


13.2. Catalyzation of innovations by CC investments and procedures in leadership of supply chain.







13.3. Catalyzation of innovations by CC investments and procedures in data systems of supply chain.







Evaluation question 5: With respect to the transport mechanisms put in place, were there other alternative strategies that could have been put in place to achieve the same level of results but at a lesser cost, especially in light of the country´s current transition towards Vaccine Independence?

DAC Evaluation Criterion covered by this Evaluation Question: Efficiency Evaluation Question Background (short narrative): Transportation is a key factor for the CC system. Possible alternatives for transportation will be explored in this item and will help to allocate subsidies for a possible regional storage center.


Data Collection Tool Preliminary Findings


88

88

5.1. Cost of transportation in CC cycle over total cost of program

- Detailed budget of program

- UNICEF Program Officer

-

- Desk Review - Semi-structured

interview


5.2. Percentage of transportation costs in similar Gavi Alliance programs.

- Gavi Alliance Reports

- Gavi Alliance Program Officer

- Desk Review - Semi-structured

interview


Evaluation question 6: To what extent were financial resources, human resources, and supplies, sufficient (quantity), adequate (quality) and distributed/deployed in a timely manner?

DAC Evaluation Criterion covered by this Evaluation Question: Efficiency Evaluation Question Background (short narrative): This question covers the key items of efficiency: use of financial resources, human resources, delivery in a timely and cost-effective manner.

Indicators/Criteria Source of Information Data Collection Tool Preliminary Findings 1.1. Alignment of financial

resources in relation to program design

- Detailed budget of program

- UNICEF Reports - UNICEF Program

Officer

- Desk Review - Semi-structured interviews


1.2. Implementation of program activities in relation to budget predicted

- UNICEF Reports - UNICEF Program

Officer




89

89

1.3. Adequacy of human resources in relation to program design and implementation

- UNICEF Reports - Government officials - UNICEF Program

Officer



1.4. Timeline of program implementation against initial targets

- UNICEF Reports - Gavi Alliance Reports - UNICEF Program

Officer - Government Officials



Evaluation question 7: To what extent could an existing central warehouse at the regional level (WCAR) produce transportation benefits for the whole country?

DAC Evaluation Criterion covered by this Evaluation Question: Efficiency Evaluation Question Background (short narrative): This question will explore the pros and cons of a central warehouse at the regional level, using objective data and also the perspective of key stakeholders involved.

Indicators/Criteria Source of Information Data Collection Tool Preliminary Findings 7.1. Unmet regional demand

for vaccines - WHO Regional Health Reports - UNICEF Regional Health Reports - Reports of similar types of

arrangements in other regions (PAHO)

- Semi-structured interviews with UNICEF staff


7.2. Capacity level of UNICEF Regional Office in procurement services

- Regional Adviser - UNICEF Representative São Tomé



7.3. Cost of transportation between Sao Tomé and other countries of the region (possible hub locations like Ghana)

- Regional Adviser - Regional transportation Companies - Government officials at central level

- Semi-structured interviews - Desk review



90

90

Evaluation question 8: What are the actual prospects for the government to be able to manage the CC after UNICEF´s support? How did UNICEF incorporate measures for the activities funded by UNICEF to continue without UNICEF’s future support?

DAC Evaluation Criterion covered by this Evaluation Question: Sustainability Evaluation Question Background (short narrative): Sustainability is a crucial aspect of program, as the government relies intensely on UNICEF´s and Gavi Alliance´s contribution. The question will explore not only the prospects of sustainability, but measures that may have been taken to ensure sustainability..

Indicators/Criteria Source of Information Data Collection Tool Preliminary Findings 8.1. Measures taken by

UNICEF to ensure sustainability of program

- UNICEF Program manager - Government officials at a

national level Semi-structured interviews

- Semi-structured interviews with UNICEF staff


8.2. Financial capacity of Sao Tome and Principe´s government to manage procurement of vaccines and Cold Chain equipment


national level - National Budget - National Health Plan

- Semi-structured interviews with government officials and UNICEF staff

- Desk review


8.3. Staff knowledge of Sao Tome and Principe´s government on how to manage procurement of vaccines and Cold Chain equipment


national level

- Semi-structured interviews with government officials at a central level and UNICEF Supply Division in Copenhagen


8.4. Political will of Sao Tome and Principe´s government to assume procurement of vaccines and Cold Chain equipment


national level

- Semi-structured interviews with government officials



91

91

Evaluation question 9: To what extent were Gender, Human Rights, and Equity Principles dully integrated in the design and delivery of the Cold Chain Interventions? To what extent did the EPI program tackle the barriers that prevent girls´ and women´s access to the services in the targeted communities?

DAC Evaluation Criteria covered by this Evaluation Question: Gender, Equity and Human Rights Evaluation Question Background (short narrative): This question will explore to which extent there were gender, human rights and equity principles embedded in the program design and implementation. Data on background of children vaccinated will be taken into account whenever possible.

Indicators/Criteria Source of Information Data Collection Tool Preliminary Findings 9.1. Integration of Gender,

Human Rights and Equity principles in Cold Chain management through specific action

- UNICEF Program Officer - Government Officials at

central, district and local levels - Mothers and girls benefited by

program



9.2. Integration of Gender, Human Rights and Equity principles in EPI through specific action

- UNICEF Program Officer - Government Officials at

central, district and local levels - Mothers and girls benefited by

program




9.3. Access to vaccines by gender and economic status

- Administrative records of district centers

- Desk review There are no data available during this moment of inception. The data will be collected during field visit.

9.4. Measures taken by the Ministry of Health to increase accessibility of Health Units to the caregivers


- Semi-structured interviews with government officials at a central, district and local level


Appendix E. Interview guide

The interview guide below will be tailored for each specific group at the moment of field work.

Relevance

To what extent do you think the vaccine Procurement Services rendered by UNICEF met the country´s needs?

How do you assess the participation of national government officials in the selection of SDD equipment?

Do you think the participation of the government in defining the equipment to be purchased met your expectations?

Effectiveness

To what extent have CC investment contributed to improving immunization service delivery in your opinion? Do you think there was an impact in:

Increase in vaccine availability?

Increase on immunization sessions after use of refrigerators purchased by the program?

Increase in frequency of vaccine distribution?

What are the factors that have facilitated healthcare delivery in your opinion?

Improvement of health care delivery due to type of equipment?

Qualification of health care workers in conducting preventive maintenance tasks?

Improvement of health care delivery due to installation method and verification system?

What are the factors that have hindered healthcare delivery in your opinion?

What are the unexpected outcomes (positive and negative) you identify with the scope of the procurement and distribution of vaccines?

Catalyzation of innovations by CC investments and procedures in continuous improvement of supply chain.

Catalyzation of innovations by CC investments and procedures in leadership of supply chain.

Catalyzation of innovations by CC investments and procedures in data systems of supply chain.


93

93

Efficiency

With respect to the transport mechanisms put in place, what would be other alternative strategies that could have been put in place to achieve the same level of results but at a lesser cost, especially in light of the country´s current transition towards Vaccine Independence?

What is the Cost of transportation in CC cycle over total cost of program?

What is the percentage of transportation costs in similar Gavi Alliance programs?

In your opinion, to what extent:

Was there Alignment of financial resources in relation to program design?

The Implementation of program activities took place in relation to budget predicted?

Were the human resources adequate in relation to program design and implementation?

Was the program implementation timeline in accordance to initial targets?

Do you think a central warehouse at the regional level (WCAR) could produce transportation benefits for Sao Tome and Principe?

Do you know what is the unmet regional demand for vaccines?

What is the capacity level of UNICEF Regional Office in procurement services?

What is the cost of transportation between Sao Tomé and other countries of the region (possible hub locations like Ghana)?

Sustainability

What are the actual prospects for the government to be able to manage the CC after UNICEF´s support in your opinion? How do you assess:

Financial capacity of Sao Tome and Principe´s government to manage procurement of vaccines and Cold Chain equipment?

Staff knowledge of Sao Tome and Principe´s government on how to manage procurement of vaccines and Cold Chain equipment?

Political will of Sao Tome and Principe´s government to assume procurement of vaccines and Cold Chain equipment?


94

94

How did UNICEF incorporate measures for the activities funded by UNICEF to be continued without their support in the future?

Gender, Equity and Human Rights

To what extent were Gender, Human Rights, and Equity Principles integrated in Cold Chain management through specific action? Please, give examples.

To what extent were Gender, Human Rights, and Equity principles considered in EPI through specific action? Please, give examples.

How do you see the differences in vaccination rates by gender and economic status?

What measures have been taken by the Ministry of Health to increase mothers’ accessibility to Health Units?


95

95

Appendix F. Interview Guide for Focus Groups

Name:

Number for Children:

• When it comes to your youngest child, please, report the vaccines you remember he/she has taken.

• Where do you live and where have you vaccinated your children? • How do you assess the quality of the health care of the Health Unit you

come to? What works well and what could be improved? • From March 2018 on have you felt any difference in the care you have

received? • In your opinion, why do some mothers/caregivers not vaccinate their

children? • Why do mothers not take their children to take the second and third doses

of the vaccine? (What are the challenges they face to do so?) • What could be done to encourage mothers/caregivers to take their

children for vaccination?


96

96

Appendix G. Check List for Health Units

Item

Recommended

Not Recommended

Temperature record

chart

On fridge or near by

no

Recorded Temperature

Daily and safe range

no

Expired dates of

vaccines

Ok

expired

Vaccine Vial Monitor

VVM

Inner square lighter than

outer circle

Inner square same color

or darker


97

97

Appendix H. Description of the Theory of Change

1. Planning of Procurement: all the procurement for vaccines is currently done by UNICEF through the Supply Division in Denmark. Such procurement is complex and involves a number of factors which are indicated as assumptions (availability of data on demand for vaccines, information systems which can help with accurate and timely data, staff preparation to analyze data and conduct complex international procurement, availability of resources and legislation that allows for efficient and timely procurement).

2. Carrying out procurement procedures: in this case, after planning of procurement comes the implementation of the procurement process – publication of bidding documents, receipt of proposals, analysis of documents, contact and negotiation with companies, signing of contracts and following up on delivery of products, which involves all of the assumptions named in Step 1 above. This part was summarized in one item as the focus of the process more on the management of the Cold Chain within the country. Nevertheless, the country may soon take up procurement responsibilities, so they should be considered as part of the Cold Chain.

3. Production of vaccines: this key step is part of the process of the Cold Chain, but it is an item that is under the responsibility of the partner companies, which should ensure the quality of the antigens produced and stored. A basic quality assurance (which could be checking grades of companies) should be carried out to certify that appropriate procedures are conducted. An external audit is recommended to ensure the Good Manufacturing Practice - GMP of companies.

4. Overseas transportation of vaccines: this is a crucial step of the Cold Chain. Vaccines arrive in STP via the Portuguese airline, TAP. Vaccines should come in an appropriate refrigerated box to ensure temperatures stay within the range of 2 to 8 degrees Celsius.

5. Arrival at airport: the airport does not have the capacity of storing cold goods, so vaccines should stay at the airport as little as possible. This arrival needs to be well managed to ensure the vaccines do not reach a temperature beyond 8˚ C. The assumption is that communication takes place between supplier, air company, UNICEF, and government.

6. Customs Clearance: Customs clearance should be done in a timely manner to allow the vaccines to reach the ground at the airport and be directly transported to the EPI Distribution Center.

7. Transportation to EPI Distribution Center: the vaccines need to be transported at the shortest possible time span to reach the distribution center with no delay. One important assumption here is the availability of timely transport whenever needed. Communication plays an important role between the supplier, the air company, UNICEF and the government.


98

98

8. Inspection: once the vaccines reach the Central Distribution Warehouse, there should be a checking procedure to verify the quantity and quality requested of the vaccines. The new supplies should be registered and informed back to customs so that other customs procedures may be carried out.

9. Storage of Vaccines: the vaccines which have arrived should be stored in the appropriate refrigerators. A number of assumptions are in place in this case: vaccines are kept in the appropriate temperature, staff knows where to place each type of vaccine, a generator is in place and working well, log is kept of all the vaccines received, equipment is available and functioning properly for vaccine storage, staff is trained to keep the temperature monitored, regular maintenance of equipment is carried out, the EPI Distribution Center monitors the temperature at Health Units regularly.

10. Transportation to Districts: the transportation to districts needs to be done in appropriate arks and supply the districts according to their demands. Adequate means of transportation need to be in place.

11. Storage of Vaccines in Districts: the same cycle described in step number 9 applies.

12. Distribution of Vaccines to Health Units: the same cycle described in step number 10 applies.

13. Storage of Vaccines in Health Units: same cycle described in step number 9.

14. Distribution of Vaccines in Health Units: three critical assumptions are made: availability of Health Units close to children and their caretakers to benefit them, record of vaccines distributed is well managed and capacity of health workers to manage vaccines.


99

99

Appendix I. Additional Information by Evaluation Criterion

Relevance

Picture Box 1. Deterioration of CC Equipment


100

100

Effectiveness

The tables next show key data of stock management in STP for the years 2017 and 2018 and point at an improvement in vaccine stock levels in some cases and a decrease in others, evidencing a mixed impact, considering the vaccination rates of STP which are already high.

Table 22. Vaccine Stock Management, STP, year 2017 Routine

Immunization Presentation Estimation of needs for recipients CRITICAL STOCK LEVELS

Yearly Supply Period Stock_reserve Stock_re-order Stock_maxi

BCG 20 11.100 5.600 2.800 5.600 8.400

bVPO 10 33.300 16.700 8.400 16.800 25.100

DTC-HepB-Hib 1 18.500 9.300 4.700 9.400 14.000

PCV-13 1 18.500 9.300 4.700 9.400 14.000

VPI 1 6.400 3.200 1.600 3.200 4.800

Td 10 15.200 7.600 3.800 7.600 11.400

Rota_liq 1 18.500 9.300 4.700 9.400 14.000

VAA 5 7.400 3.700 1.900 3.800 5.600

RR 10 14.600 7.300 3.700 7.400 11.000

Source: SMT_STP 2017.

Table 23. Vaccine Stock Management, STP, year 2018 Routine

Immunization

Presentation

Estimation of needs for recipients CRITICAL STOCK LEVELS

Yearly Supply Period Stock_reserve Stock_re-order Stock_maxi

BCG 20 26.700 13.400 6.700 13.400 20.100

bVPO 10 28.800 14.400 7.200 14.400 21.600

DTC-HepB-Hib 1 16.100 8.100 4.100 8.200 12.200

PCV-13 1 16.100 8.100 4.100 8.200 12.200

Rota_liq 2 16.100 8.100 4.100 8.200 12.200

VPI 1 5.700 2.900 1.500 3.000 4.400

RR 10 11.700 5.900 3.000 6.000 8.900

Td 10 15.600 7.800 3.900 7.800 11.700

VAA 5 6.100 3.100 1.600 3.200 4.700

HPV 2 5.700 2.900 1.500 3.000 4.400

HepB 1 5.700 2.900 1.500 3.000 4.400

Source: SMT_STP 2018.

The textbox below illustrates the perception of increase of availability by one of the nurses of the District of Lobata.


101

101

Textbox 1. Quote from Nurse of LOBATA about the CC

Picture Box below shows maintenance instructions for CC equipment and solar panels.

Picture Box 2. Maintenance Instructions for CC Equipment and solar panels

The following tables show the figures for vaccine distribution by District for selected types in a comparison between 2017 and 2018 – before and after the introduction of the new Cold Chain Equipment.

There were power blackouts, there was another one which was based on petroleum. With the solar mini-ark, there was great improvement. And with the mini-

ark, we have the equipment available throughout the day and the good conservation of the vaccines with the constant availability in the sector. Mothers

come to vaccinate without previous notice. (…) We don´t have problems because the vaccines froze or the power went off.

Nurse, female, District of Lobata.


102

102

Table 24. Vaccine Distribution by District for selected type - Part I - 2017 and 2018*

Source: EPI. *The correct data for Cauê were not available.

Table 25. Vaccine Distribution by District for Selected type - Part II - 2017 and 2018*

Source: EPI. *The correct data for Cauê were not available.

The following Figures 6-12 demonstrate the Evolution of vaccination rates of VAR1, VAR2, DTP3, RR1, HUV1 and the proportion of children completely vaccinated with variations from 2015-2019 according to available data.

Figure 6. Evolution of VAR1 Vaccination Rate 2015-2018

Source: EPI.

District/ Vaccines

BCG bVPO DTC-HepB-Hib PCV-13

Year 2017 2018 2017 2018 2017 2018 2017 2018

Total 29.520 51.600 92.420 66.210 21.309 20.722 22.866 28.064

Água-Grande

6.880 8.200 27.650 14.540 7.370 7.658 7.316 7.733

Mé-Zóchi 500 600 11.350 4.110 2.922 2.750 2.872 2.700

Lobata 660 840 5.820 2.360 1.624 1.715 1.600 1.608

Lembá 720 960 4.520 3.880 1.559 1.349 1.450 1.419

Cantagalo

140 380 4.920 4.430 1.463 1.664 1.578 1.708

R.A.P 480 600 2.780 1.220 755 892 756 912

District/ Vaccine VPI Rota_liq VAA RR Year 2017 2018 2017 2018 2017 2018 2017 2018

Total 4.505 9.000 24.076 28.351 8.720 9.005 26.620 26.240

Água-Grande

799 3.018 7.830 8.100 2.960 3.045 4.890 5.640

Mé-Zóchi

275 958 2.968 2.800 1.190 1.140 3.070 3.010

Lobata 250 496 1.623 1.618 620 715 1.440 1.350

Lembá 160 438 1.266 1.334 550 500 1.140 920

Cantagalo

361 560 1.452 1.563 635 695 1.090 960

R.A.P 50 290 800 918 460 260 730 510

ÁguaGran

de

MéZochi

Lobata

Cantagalo

Lembá Caué RAP Nacin

al

2015 86.2 93.2 99.5 99.3 98.4 99.4 98.2 92.62016 91.7 90.7 95.3 95.8 93.2 95.9 97.8 92.72017 81.3 84.2 92 88.9 90 95.7 97.5 89.92018 96 90 92 99 96 95 97 95

020406080100120


103

103

Figure 7. Immunization Rates for VAR 1 in STP - Jan-July 2017 and 2019

Source: EPI.

Figure 8. Vaccination Rates for VAR2 in STP - Jan-July 2017 and 2019

Source: EPI.

Jan Feb Mar Apr May Jun Jul AverVAR1 - 2017 98% 99% 98% 117% 80% 89% 87% 95%VAR1 - 2019 96% 90% 93% 91% 90% 89% 93% 92%

0%

20%

40%

60%

80%

100%

120%

140%

VAR1 - 2017 VAR1 - 2019

Jan Feb Mar Apr May Jun Jul AverVAR2-2017 53% 67% 89% 158% 45% 49% 71% 76%VAR2-2019 92% 87% 76% 85% 85% 83% 82% 84%

0%

20%

40%

60%

80%

100%

120%

140%

160%

180%

VAR2-2017 VAR2-2019


104

104

Figure 9. Evolution of DTP3 Vaccination Rate 2015-2018

Source: EPI.

Figure 10. Evolution of RR1 Vaccination Rates – 2015-2018

Source: EPI.

Figure 11. Coverage of HUV 1st dose – 2018.

Source: EPI.

ÁguaGrand

e

MéZochi Lobata Canta

galo Lembá Caué RAP Nacinal

2015 94.1 98.6 96.9 97.1 97.7 97.6 97.3 96.32016 94.6 93.5 98.5 99.1 97 97 97.3 95.62017 95.9 92.5 93.7 94.2 96.7 96.1 95 95.42018 97 92 95 97 96 98 97 95

889092949698100

ÁguaGrande

MéZochi Lobata Cantag

alo Lembá Caué RAP Nacinal

2015 86.2 93.2 99.5 99.3 98.4 99.4 98.2 92.62016 91.7 90.7 95.3 95.8 93.2 95.9 97.8 92.72017 81.3 84.2 92 88.9 90 95.7 97.5 89.92018 111 59 86 104 114 95 120 95

020406080

100120140

Perc

enta

ge

Água-Grand

e

Mé-Zochi Lobata Lembá Cantag

alo Caué RAP Total:

Taxa Cobertura 98.1 92.7 100.3 106.4 92.4 117.5 104.4 98.1

0

20

40

60

80

100

120

140

Perc

enta

ge


105

105

Figure 12.Proportion of children 12-23 months completely vaccinated in 2018

Source: EPI.

Table 26 shows vaccination sessions by Health Unit and status of CC equipment comparing January-July 2017 to January-July 2019, before and after the installation of the new CC equipment.

Table 26. Vaccination sessions by Health Unit and status of CC equipment - January-July - 2017 and 2019

District Status Health Unit Penta 3 Measles 2017 2019 Variation 2017 2019 Variation

Agua Grande

New health Unit Madredeus N/A 50 100.00 N/A 56 100.00

Me-Zochi From Suitcase to Refrigerator

Monte Café 25 34 36.00 31 35 12.90

Same equipment

Bombom 105 91 -13.33 105 89 -15.24

Lobata From Suitcase to Refrigerator

Conde 37 40 8,11 43 35 -18.60

Same equipment

Desejada 53 39 -26.42 26 39 50.00

Lembá Not installed Diogo Vaz 1 2 100.00 2 1 -50.00 New equipment Neves 187 162 -13.37 182 176 -3.30

Cauê New equipment Angolares 53 43 -18.87 52 53 1.92 Same

equipment R. Peixe 37 21 -43.24 34 24 -29.41

Cantagalo From Suitcase to Refrigerator

V. D´América

33 48 45.45 20 40 100.00

From Suitcase to Refrigerator

St. Cecília 15 6 -60.00 13 8 -38.46

New equipment R. Afonso 23 23 0.00 33 12 -63.64 RAP From Suitcase to

Refrigerator Picão 15 12 -20.00 23 15 -34.78

New equipment Nova Estrela 13 11 -15.38 17 N/A 0.00 Source: EPI.

78 75 71 71

5138

62

0

20

40

60

80

100

RAP CAUE ME-ZOCHI LEMBA CANTAGALO LOBATA National

% E

nfan

ts v

acci

nné

Distrit


106

106

The following Textbox illustrates the perception of a manager about the improvement in the vaccine distribution caused by the new CC equipment.

Picture Box 3. Temperature chart and open lid in CC equipment

Picture Box 4. Pictures of damaged condenser and rusty screw

The following Info Boxes illustrate guidance for maintenance of CC equipment

Info Box 1. Cleaning monthly guidance for CC equipment

Source: CC Manual.

• Clean the unit from any dust: • Switch off the TCW 40 SDD and disconnect it from power socket. • Clean the ventilation grids and the condenser with the supplied brush or a vacuum cleaner. • Reconnect and switch on again. • Clean the lid seal.


107

107

Info Box 2. Basic Information about management of TCW 40 SDD

Source: https://www.unicef.org/supply/files/TCW_40_SDD.pdf

Sustainability

The following quotes illustrate the challenges of sustainability around EPI and CC in STP.

Textbox 2. Quote from Management Staff on Sustainability

Textbox 3. Quote from Nurse, Distribution Center on Sustainability

GENERAL INFORMATION

1. The TCW 40 SDD is a combined refrigerator and waterpack freezer for Solar Direct Drive without battery storage.

2. Keep these operating instructions available and leave them with the unit, so that all users can read about the functions and safety regulations.

3. Ensure that there is sufficient room around the unit for air circulation. Please refer to the installation information regarding this point.

´In the forums we go, STP is a middle-income country - ´we are going to focus in low

income countries`. I don´t see the country mobilize resources to purchase all the

vaccines the country needs. Even though the country has a good vaccination culture,

these achievements and other ones may be at risk´.

Management staff, female, Ministry of Health

´Even though STP can be considered a middle-income country, the economic

capacity of the county in terms of health is still very limited. I feel that we need support from UNICEF and other institutions, we might

get back in terms of health indicators.´

Nurse, male, Distribution Center

https://www.unicef.org/supply/files/TCW_40_SDD.pdf


108

108

Textbox 4. Quote from father, 29 years, District of Cauê on Family Planning

Equity

The following quote illustrate the perception of fathers towards family planning in one of the focus groups promoted during the evaluation mission.

Textbox 5. Quote from Father, 36 years, District of Cauê on Family Planning

´

´Women who attend family planning do not stay well. Pills bring harm.`

Father, District of Cauê, 29 years old

´The use of condoms bring less pleasure. Also, the period does not come´.

Father, District of Cauê, 36 years old


109

109

Appendix J. Additional information on Immunization, Health System Strengthening and the Regional Experience

In 2015-2016 Gavi decided to fund the HSS project using the vaccination as the entry point. The proposal was to improve the vaccination service and help improve the whole Health Unit. A total of US$ 3 million was approved for a period of 5 years. The key objectives were: 1) Planning and capacity building on financial management; 2) Supply of vaccines which included the Cold Chain; 3) Communication for immunization which included training activities; and 4) the Information system (DHS2). As the government changed leaders, a new perspective blossomed toward change and improvement of the HSS via other actions such as infra-structure building/ development. There were changes in the initial agreement between the Government and Gavi to also support the purchase of vehicles and vaccine incinerators for each district.

In dialogue with the Sustainable Development Goal, "Leave no one behind: all children immunized and healthy," the Ministry of Health (MoH) should present clear courses of action which are evidence-based, adapted to the country, economically viable and implemented after a sound analysis. In relation to UNICEF`s four-step approach to improve the management capacity of the country on health, four axis were identified:

⇒ Diagnose the most deprived populations and systems challenges ⇒ Intervene with solutions to overcome bottlenecks ⇒ Verify progress through timely monitoring, and ⇒ Adjust solutions as needed to optimize effectiveness and efficiency

When it comes to this four-step approach, STP has been working to improve its planning capacity upon a diagnosis of the country. One example has been the elaboration of the EPI plan for the Period 2016-2020. However, there are still gaps for identifying the most deprived populations per district and monitoring their inclusion into the system. Great efforts have been made by healthcare professionals and vested stakeholders in the country to reach people on the ground through home visits and follow-up in Health Units. Nevertheless, more effort is needed to collect sound and coherent data, analyze outcomes and review courses of action.

An analysis of the Health System as a whole, reflects an outdated legislation in STP, which creates health problems (e.g. outdated legislation to punish drugs sold without license). The country lacks an institutional framework for health. Also, the STP does not have a sanitation regulatory body in place, which poses health threats to the public. The Drug Department which authorizes drugs in the country is under the Ministry of Health and has no legal autonomy to carry out its work. These are broader issues of HSS which should be considered to advance the health system, to not only immunize each child but to promote more complex types of health care in the future, which will be officially covered by the institutional framework in the country.


110

110

Reflecting on the Regional Experience, STP could gain valued knowledge from the research to connect immunization with HSS. The UNICEF Annual Reports for Gabon, Guinea Equatorial, Ghana, and Cameroon for 2018 were reviewed and lessons identified. In the case of Gabon, the country was certified as free from Polio toward the end of 2017. However, a stagnation of DTP3 Coverage at 70.6% evidences that progress is slow regarding DTP3.

In Guinea Equatorial, a 2016 evaluation of the EPI data reflected 24 % of children under 12 months old received all of the recommended vaccines. Data from The Ministry of Health in 2018 indicated that the percentage of children receiving DTP3 had risen from 36% in 2016 to 48%, reflects progress, however, still far below STP. Guinea Equatorial has challenges with data management and there are challenges with securing the funding of all the components of the EPI Program, although the government finances all vaccine costs. For one, a challenge to overturn the belief that vaccinations cause sicknesses is prevalent. According to the UNICEF 2018 Report, the level of vaccination coverage is still too low and presents a serious risk of surging epidemic outbreaks of polio and measles. Part of the reason of poor immunization performance in Guinea Equatorial is attributed to Cold Chain and vaccine management, reduction of vaccination staff, and limited use of Cold Chain services. There is a systematic loss of vaccines and medication due to poor management and maintenance of Cold Chain facilities.

Conversely, Ghana represents higher immunization rates in comparison to Guinea Equatorial and Gabon. Full immunization coverage for children 12-23 months rose from 71 %, according to data in the Demographic and Health Survey 2014, to 77% in the MICS 2017/18. This is an achievement, despite disparities among different groups in various regions, the privileged and underprivileged, and the educated and uneducated. UNICEF, the government and Gavi, among other development partners have joined efforts to promote analyses of immunization coverage in priority districts, thus developing micro-plans to work in these districts. UNICEF has supported the country through an inventory of Cold Chain equipment to advance developments of a Cold Chain equipment optimization platform.

In Cameroon, immunization coverage has decreased over the period 2016-2018 nationwide, with Penta3 dropping from 85% in 2017 to 78% in 2018. Explanations for the decrease include poor management of Cold Chain, insufficient outreach activities, and socio-cultural adverse beliefs and rumors. UNICEF is advocating for solar powered refrigerators to be allocated to the most vulnerable regions.

A common problem reported in STP, Guinea Equatorial, and Cameroon was cultural resistance towards vaccines which arose from perceptions that vaccines cause more harm to children’s health. In Cameroon, UNICEF is exploring the potential alternative methods to overcome unwarranted beliefs, by promoting dialogue with CSOs and local leaders. The dialogue among CSOs and UNICEF in STP has been limited and was reported as a possible area of expansion in the Country Office Report of 2018. Another learning aspect for the region as a whole would be the choice of solar powered refrigerators. The corrosive impact on the equipment of the seaside air in one particular case in Cameroon should be considered. STP performs better than other countries in the Region with vaccination


111

111

rates, Cold Chain management, and public outreach services. However, data demonstrated that there is potential for learning in areas where they lag behind, such as analysis and coverage plans by the district similar to Ghana, and finally dialogue with civil society in Cameroon.

Appendix K. Agenda of Field Visit

Agenda

Atividades

DATA manhã tarde

16/10/2019 (quarta

feira)

UNICEF São Tomé

• Administrativos da UNICEF – Alan/Madger/Liudmir.. 8h30

• Mariavittoria Ballota – Representante Unicef São Tomé; 10h00

• Gestor da cadeia de frio nacional: Vladimir Sousa; 11H00

UNICEF Regional – Oficial de Saúde; UNICEF Global – Oficial de Saúde; Gavi Alliance

• Ex-Diretora dos Cuidados de Saúde: Dra. Marisa Conceição (foi quem começou o programmea); 14H00

• Posto de Saúde de Madre Deus: Enf. Resposável, Idjamila Semedo

• WHO – Dra. Claudina – 15h30 • Gavi Alliance – Dr. Thierry Vincent – focal point – 17h

(skype);

17/10/2019 (quinta feira)

Ministério da Saúde – Figura política, equipe técnica, OUTROS

MINISTÉRIOS

• Michele Tarsilla – M and E Regional Unicef – 8h00 • Diretor Geral das Alfândegas – Eng. Carlos

Benguela; 9h00 • Jersey Cady/Ivanildo – despachante do estado;

09h30 • Alfândega aeroporto – Técnico Responsável,

Mandu; 10h00

CENTRO DE CHEGADA DAS VACINAS – AEROPORTO - TÉCNICOS ETC

Centro regional Água grande:

• Responsável do programmea de saúde reprodutiva/programmea alargado de vacinação; Enf. Adilé Conceição; 14h00

• Diretora do Fundo Nacional Medicamentos: Dra. Neurice Neto; 15h00

• Diretor Geral da ELECTROFRIO: José Eugénio da Silva – 16H30


113

113

• Autoridade Reguladora de Medicamentos/Departamento Farmacêutico: Sra. Simoa/Vania;10h30

• Dr. Lima (Especialidade em ortopedia) visão de fora; 11H30

18/10/2019 (sexta feira)

VISITA AO CENTRO REGIONAL – Região Autónoma do Príncipe - Ilha do Príncipe • Responsável do programmea saúde reprodutiva/programmea alargado de vacinação/ Posto de Saúde de S.

António: Enf. Lurdes Managem; 9h00 • Grupo Focal: Mães e encarregadas de Educação das crianças; 9h30 • Delegado de Saúde: Dr. Sílvio Vera Cruz; 10h00

21/10/2019 (segunda

feira)

VISITA AOS CENTRO REGIONAIS - ZONA NORTE : LEMBA:

• Delegado de Saúde: Dr. Tomé; 7h30 • Centro de Distribuição Lembá – Programmea de

Saúde Reprodutiva: Enf. Jessica das Neves e Enf. Maria de Jesus Alves; 8h15

• Posto de saúde Diogo Vaz: Nicolau Martina; 9h15 LOBATA

• Responsável do Centro: Enf. Reaponsável, Sorteana Lima; 10h00

• Posto de saúde de Desejada: Enf. Resp. Igualter Daio; 11h00

VISITA AOS CENTRO REGIONAIS - ZONA CENTRO :

MÉ-ZOCHI: • Responsável do Centro: Enf. Responsável, Elanauria

Choi; 14H00 • Grupo Focal: Mães e encarregadas de Educação

das crianças; 14h30 • Posto de saúde Bom bom: Enf. Resp. Maria Tomé;

15H30 • Diretora do programmea de saúde

reprodutiva/programmea alargado de vacinação: Dra Solange Barros; 16h00

• Health specialist UNICEF: Luís Bonfim, 17h00


114

114

22/10/2019 (terça feira)

VISITA AOS CENTROS REGIONAIS - ZONA SUL:

CAUÊ

• Responsável do Centro: Enf. Pedro Mateus; 8h15 • Posto de saúde Ribeira Peixe: Enf. Cleider Afonso;

9h30

CANTAGALO

• Responsável do Centro de saúde reprodutiva/ posto de Água Izé: Enf. Maida Ceita; 11h00

TRABALHO INTERNO – REUNIÕES INTERNAS - REUNIÕES EVENTUAIS – DEBRIEFING COM UNICEF

• Debriefing – MariaVittoria Ballota; 14h30

12/11/2019

• Coordenadora do SIS – Ineida Monteverde; 14h30 • Coordenador do RSS – Reforço de Sistema de Saúde

– Hugo; 15h00

13/11/2019 • Chefe do Departamento de Orçamento do Ministério

das Finanças: Carlos Alberto; 14h30

Appendix L. Main Documents Reviewed

Gavi Alliance, 2019. Country Profile Sao Tome and Principe. Gavi Alliance: Geneva. Gavi Alliance, 2015. Country Programmes Strategic Issues. Gavi Alliance: Geneva. Gavi Alliance, 2017. Consent Agenda: Review of Cold Chain, Equipment Optmisation Platform. Gavi Alliance: Geneva. Gavi Alliance, 2018. Rapport d'évaluation conjointe de 2018. Gavi Alliance: Geneva. Krueger, R. & Casey, M. A., 2014. Focus Groups, A Practical Guide for Applied Research. Fifth Edition, University of Minnesota: Minneapolis.

Ministere de la Sante, 2015. Plan Pluriannuel Complet 2016 – 2020. Ministere de la Sante: Sao Tome. Ministério da Saúde, 2017. Plano Nacional Do Desenvolvimento Da Saúde 2017 – 2021. Ministério da Saúde: São Tomé e Príncipe. WHO & UNICEF, 2011. Evaluation de la Gestion Efficace des Vaccins. Republique Democratique de São Tome et Principe. Ministere de la Santé: São Tome. WHO & UNICEF, 2011. Plan d´Amélioration de Gov. Republique Democratique de Sao Tome et Principe. Ministere de la Santé: São Tome. WHO, 2015. Immunization in Practice A practical guide for health staff 2015 update. WHO : Geneva. The Department of Health, 2015. Cold Chain and Immunisation Operations Manual. The Department of Health: Republic of South Africa. République démocratique Gouvernment of STP, Gavi & WHO, 2018. Rapport de l´analyse des performances et de revue documentaire des données due Programme Élargi de Vaccination São Tomé et Príncipe 2018, STP. UNICEF, 2018. Country Office Annual Report 2018. UNICEF: Cameroon. UNICEF, 2018. Country Office Annual Report 2018. UNICEF: Gabon. UNICEF, 2018. Country Office Annual Report 2018. UNICEF: Ghana. UNICEF, 2018. Country Office Annual Report 2018. UNICEF: Guinea Equatorial.


116

116

UNICEF, 2018. Country Office Annual Report 2018. UNICEF: São Tome and Principe. Autres documents : Analyse des performances de l’immunisation et revue documentaire des données Plan de Transition, CDF – STP, 2016. Plano de instalação de painéis solares. Relatório de montagem dos painéis solar nos Centros e Postos de Saúde.


117

117

Appendix M. Terms of Reference Evaluation of the Cold Chain in Sao Tome and Principe (STP)

UNICEF works in some of the world’s toughest places, to reach the world’s most disadvantaged children.

For every child, a fair chance

1. Evaluation Object.

In STP, immunization services are made possible as part of the Expanded Programme of Immunization (EPI). Available in 39 health facilities, EPI aims to ensure that vaccination coverage to all population groups, regardless of sex, place of residence or family income.

The procurement and supply of vaccines is managed through UNICEF, based on a complete multi-year plan (cMYP).

Between 09/2017 - 04/2018, EPI allowed the installation of 39 new SDD refrigerators in a variety of health units nationwide. Prior to the installation of the new equipment, the country’s Cold Chain Park consisted of only 27 refrigerators RCW 42 EK and 5 VESTFROST.

2 years into the implementation of the EPI program and in light of the country’s higher vaccination coverage than in the rest of the region, UNICEF has commissioned this evaluation to assess the extent to which the existing cold chain may have contributed to higher immunization coverage.

2. Evaluation Purpose

This evaluation will have 2 aims: accountability and learning.

It will provide the donor (vertical accountability) and the expected beneficiaries (horizontal accountability) some solid evidence on the extent to which the Cold Chain (CC) put in place as part of the EPI attained its envisaged objectives. This evaluation is expected to inform the CC implementation strategies in the years to come but it will also shed some light on some potential corrective actions that may want to be explored further.

This evaluation is expected to generate recommendations that will help UNICEF CO Health programme staff and other partners to adapt the implementation of CC mechanisms to the emerging and country-specific needs in this area.

Table 1. Evaluation Users and Uses

Evaluation Users

Evaluation Uses

UNICEF Health/Immunization Section Staff


118

118

By better understanding the contributions of the CC to the health system.

Other UNICEF Sections Staff

Define a better coordination strategy with the Health team and identify concrete modalities of strategic collaboration towards the attainment of KRC1

UN and other development partners

The renewal of the CC as one of the recommendations of the EVM must reflect the securing vaccine supply costs and ensure strengthening vaccines'squality.

Other partners can take this opportunity to strengthen their collaboration with the MoH and their involvement in the immunization issues.

Government (Health and other line ministries)

Will better define the terms of collaboration with UNICEF with respect to the attainment of the relevant goals set in the National Development Plan

NGOs/CBOs

Mainstream the good practices identified during the evaluation and address the weaknesses emerged during the analysis**

3.Evaluation objectives**

1. Measure the effects of the national CC operating system on the country’s immunization coverage,

2. Identify the "vaccine chain"[1] shortcomings, constraints and challenges which are holding back the current progress of the Vaccination Programme towards universal coverage;

3. Establish recommendations for immediate, medium and long-term actions to achieve universal coverage and health systems strengthening (HSS).

4. Evaluation Scope

Thematic Scope

· Measuring the effectiveness of the vaccine procurement services and the CEE equipment selection process;

· Gauging the efficiency of the current transport system and exploring the feasibility of the set-up of a possible storage mechanism in the Africa sub-region;

· Determining the total costs of the supply chain (including the costs of immunizing a child);

· Measuring the frequency of vaccination services;

· Estimating the degree of availability and conservation of vaccines;

· Measuring the equity in the distribution of vaccines to target groups;


119

119

· Determining the improvement of immunization coverage, the vaccination coverage and its relationship with HSS

· Estimating the immunization service system, including vaccine procurement, transportation, storage and distribution mechanisms.

Geographical Scope

The desk review to be conducted as part of this evaluation is expected to cover all the activities implemented as part of the supply chain nationwide. However, the data collection will concentrate on a smaller sample of intervention sites to be defined by the consultancy.

Chronological Scope

The evaluation will make sure to capture the essence of the activities implemented between September 2017 and April 2018.

5. Evaluation Context

6.Evaluation Criteria

This evaluation will be guided by 5 evaluation criteria: 4 OECD/DAC criteria2 and an additional one on Gender, Equity and Human Rights.

7. Evaluation Questions

The Evaluation will seek to answer the following questions (the evaluation consultant may suggest some different ones in his/her technical proposal; questions will be finalized with the UNICEF Country Office staff upon signature of the contract).

7.1. Relevance

• To what extent have the vaccine Procurement Services rendered by UNICEF, met the country's needs, especially during the transition period towards Vaccine Independence Initiative (VII).

7.2.Effectiveness**

· How did the national authorities participate in the selection of SDD equipment? Did the type and level of participation modality correspond to the Government’s expectations?

· What is the level of functionalities of delivery services CCE across all levels of healthcare delivery?

· To what extent have CCE investments contributed to improving immunization service delivery? (e.g. changes in frequency of immunization sessions, changes in frequency of vaccine collection/distribution, Averted missed opportunities due to a better vaccine availability)?

https://uncareer.net/vacancy/Relevance,%20Effectiveness,%20Efficiency%20and%20Sustainability


120

120

· To what extent have CCE investments catalysed innovations in other supply chain fundamentals e.g. continuous improvement, leadership, data systems and system design?

· What are the factors (e.g. equipment type and/or model, health care workers conducting preventive maintenance tasks, installation method(s), verification systems, service delivery models etc.) that have facilitated the level of functionalities observed across all levels of healthcare delivery?

· What are the factors that have hindered the level of functionalities observed across all levels of healthcare delivery?

· What are the unexpected outcomes (positive and negative) produced with the scope of the procurement and distribution of vaccines?

7.3. Efficiency

· With respect to the transport mechanisms put in place, were there other alternative strategies that could have been put in place to achieve the same level of results but at a lesser cost, especially in light of the country’s current transition towards Vaccine Independence?

· To what extent were financial resources, human resources and supplies:

-sufficient (quantity)?

-adequate (quality)?

-distributed/deployed in a timely manner?

· To what extent could an existing central warehouse at the regional level (WCAR) produce transportation benefits for the whole country?

7.4. Sustainability

· How did UNICEF incorporate measures for the activities funded by the UNICEF to be continued without UNICEF support in the future?

7.5. Gender, Equity and Human Rights

• To what extent were Gender, Human rights and Equity principles duly integrated in the design and delivery of the Cold Chain interventions?

• To what extent did the EPI program tackle the barriers that prevent girls’ and women’s access to the services that it made available in the targeted communities?

8. Evaluation methodology **

The evaluation will be based on a participatory methodology using quantitative and qualitative data collection methods (mixed methods) on a multivariate analysis.

Based on the review of the existing program documentation, the evaluation consultant would identify information needs and gaps in appraisal data. In doing so, the


121

121

evaluation consultant would also analyse all the variables for which monitoring, and evaluation indicators have been collected at the district and national levels, including, equity, gender and immunization coverage. In doing so, the evaluation consultant would also conduct semi-structured interviews with the staff working for the EPI at Central level, and at the District level. Present the design of the study including evaluation method, tools of data collection, analysis and reporting. Tools to share the generated evidence and their wide use for planning and decision-making. Direct observations at health sites and focus groups discussion could also be used to collect and triangulate data.

The consultants are strongly encouraged to propose the use of innovative methodologies in their technical proposal.

The data collection and analysis as well as the different deliverables produced as part of this consultancy assignment, will draft report and final report will be consistent with the international evaluation quality standards namely: the UNEG Checklist on Quality Evaluation Reports[4], the GEROS Quality Assessment Criteria[5], and the UNEG Guide on the Integration of Gender Equality and Human Rights in Evaluation[6].

9. Schedule of Tasks, Deliverables, Duty-Station & Timeline

The consultancy will be three months in duration in August-October and will consist of three main phases:

Number of Days of Work

ACTIVITIES

International Team Leader

Consultant 1

Phase I

Development of inception report (this will include the development of the evaluation design and the data collection tools) + Inception Meetings

8

5

Phase II

Data collection and Field work + Debriefing on preliminary findings

12

12

Phase III

Data analysis, report writing (draft and final), validation and dissemination


122

122

15

8

Total

35

25

Deliverables:

1) Inception Report, including a detailed description of the methodology, data collection tools, and suggested work plan;

2) Power Point summarizing key preliminary findings and conclusions (to be held before the international consultant leaves the country);

3) First Draft of the evaluation report in English.

4) Recommendation validation workshop (to be facilitated remotely or by national consultant, as agreed with UNICEF);

5) Final Evaluation report in both English and Portuguese (max 40 pages with the rest to be placed in annexes) incorporating the commented made by UNICEF staff and the Reference Group members;

6) Power Point Presentation which summarizes the Evaluation Report with slide(s) of Key findings and recommendations;

7) Raw data in electronic medium, data collection instruments in electronic medium, transcripts in electronic medium, completed data sets, etc.

8) An Infographic summarizing the key findings and conclusions

The contractor will need to make sure that the draft report and final report will be consistent with the international evaluation quality standards namely: the UNEG Checklist on Quality Evaluation Reports, the GEROS Quality Assessment Criteria, and the UNEG Guide on the Integration of Gender Equality and Human Rights in Evaluation.

Duty-Station

The Consultant will be based in STP during the primary data collection phase and will work remotely (in his/her home country) during the rest of the assignment when physical presence in the country is not required. This will be proposed by the Consultant in the bid document and discussed and agreed between the UNICEF and the Consultant.

10. Governance of the evaluation

The contractor will be supervised by and report to the UNICEF STP Country Office M&E Specialist. A steering committee composed of Health Reproductive Programme/EPI, UNICEF and WHO will will be set up to validate the evaluation protocol, provide


123

123

oversight to the Evaluation and provide comments to the different deliverables (inception, draft and final report). The Regional Evaluation Adviser based at the UNICEF Regional Office for West and Central Africa (WCARO) will also provide technical oversight over the entire evaluation process, including on the different evaluation products (inception report, draft and evaluation.

11. Estimated duration of contract; Remuneration; Other Terms and Conditions

The contractor will be engaged under short-term individual contract on a full-time basis, immediately after the completion of the contracting procedure, for an estimated period of 30 days spread over a period of 12 weeks. The exact number of days to be proposed by the contractor and discussed with and confirmed, including the specific deadlines, by UNICEF when signing the contract. The Terms of Reference is an integral part of the individual contract signed with the contractor.

A national consultant will be hired to support the international consultant during the data collection and the scheduling of interviews with various health units and communities as well as to contribute to the work of data collection and analysis.

UNICEF reserves the right to withhold all or a portion of payment if performance is unsatisfactory, if work/outputs:

1. is incomplete, 2. does not meet the quality standards of both UNICEF and the Government of XXX, 3. is not delivered or has failed to meet deadlines 4. (fees reduced due to late submission: 5 days – 10%, 15 days – 20%; 1 month –

50%; more than 1 month – payment withheld).

Other terms:

1. UNICEF CO will assist the consultant in contacting with local health authorities. 2. UNICEF CO will provide an office for consultant installation and work during their

stay in Sao Tome 3. The consultant is committed to having his own computer and phone. 4. A national consultant will be hired by UNICEF CO to support on the displacement

to the Health Units with new Cold Chain equipment, the contacts with the health authorities, including the Districts, on the interviews and contribute to the work of data collection and analysis.

[1] The "vaccine chain” would include, inter alia: Purchasing services / Choice of equipment / External transport mechanism / Features of delivery services / Impact of the installation of a new CEC on the delivery of immunization services

[2] \OECD DAC Evaluation Criteria.pdf

[3] UNICEF’s Vaccine Independence Initiative (VII)] is a key element in UNICEF’s supply financing portfolio. It provides technical support to governments and short-term


124

124

bridge financing to help countries procure vaccines and other immunization-related supplies.

[4] http://www.unevaluation.org/document/detail/607

[5] https://www.unicef.org/evaluation/files/GEROS_Methodology_v7.pdf

[6] http://www.uneval.org/document/detail/980

To qualify as an advocate for every child you will have…

12. Qualifications and specialization/experience required

An International individual consultant to lead the evaluation with the following profile:

1. Advanced degree in Public Health, Immunization or another related field 2. At least 5 years’ experience in developing impact studies and/or evaluations a

copy of an evaluation report, which the applicant has been a primary author of, will need to be submitted a part of the application);

3. Experience working in a multisectoral environment 4. Speak and write fluently the English language; fluency in Portuguese will be an

advantage 5. Good IT Skills including a good knowledge of MS Word, Power Point and Excel;

13. Technical and Financial Proposal

UNICEF accepts applications from individual contractors.

All applications should contain the following documents:

I. Technical Project Proposal (max 2 pages), which would include at least the following (applicants are strongly encouraged not to repeat the text from Terms of Reference but rather to demonstrate a critical understanding of it):

1. Understanding of the evaluation purpose 2. An adequate conceptual framework and evaluation methodology 3. Consultant’s profile/portfolio 4. Proposed timeframes (hour/days) 5. Names and contact details of reference persons 6. List of past evaluation reports (if applicable) 7. Any other additional information to support the application (optional). 8. Financial Proposal:

The financial proposal should be a standalone document (using excel), which should include full details of financial offer: daily fees and the components of financial offer. Applicants are strongly encouraged to email their technical and financial evaluation offers (proposals should be submitted separately, to the following email address www.unicef.org/about/employ.

http://www.unevaluation.org/document/detail/607

https://www.unicef.org/evaluation/files/GEROS_Methodology_v7.pdf

http://www.uneval.org/document/detail/980

http://www.unicef.org/about/employ


125

125

1. Air ticket to and from home base using the most direct and economical routes 2. UNICEF does not provide or arrange health insurance coverage for the consultant. 3. DSA will be paid as per the International Civil Service Commission

For every Child, you demonstrate…

View our competency framework at

http://www.unicef.org/about/employ/files/UNICEF_Competencies.pdf

Remarks:

Mobility is a condition of international professional employment with UNICEF and an underlying premise of the international civil service.

Only shortlisted candidates will be contacted and advance to the next stage of the selection process. **

http://www.unicef.org/about/employ/files/UNICEF_Competencies.pdf