JPF South Sudan Program Evaluation

Final Report

IC Net Limited June 15, 2020

2

Table of Contents 1. Executive Summary .......................................................................................................................... 4

2. List of Abbreviations ........................................................................................................................ 6

3. Introduction ....................................................................................................................................... 8

3-1. Purpose, Scope, and Objectives ................................................................................................... 8

3-2. Overview of JPF .......................................................................................................................... 8

3-3. Context......................................................................................................................................... 9

3-4. Methodology and Limitations .................................................................................................... 10

3-4-1. Methodology ...................................................................................................................... 10

3-4-2. Limitations ......................................................................................................................... 11

4. Analysis and Findings ..................................................................................................................... 12

4-1. Relevance/ Appropriateness ...................................................................................................... 12

4-1-1. Relevance to the Ministry of Foreign Affairs’ and United Nations’ Strategy .................... 12

4-1-2. Contribution to the UN and Host Country Goals ............................................................... 14

4-1-3. Appropriateness of the Projects .......................................................................................... 14

4-2. Efficiency................................................................................................................................... 15

4-2-1. Allocation Per Sector, Per Country, and Per Member NGO .............................................. 16

4-2-2. Project Period ..................................................................................................................... 20

4-3. Coordination .............................................................................................................................. 20

4-4. Effectiveness .............................................................................................................................. 22

4-4-1. Number of Beneficiaries Planned and Reached ................................................................. 23

4-4-2. Number of Most Vulnerable Beneficiaries Reached (2016-2018) ..................................... 25

4-4-3. Overview of Outputs per Sector ......................................................................................... 25

4-4-4. Factors that Hinder/Promote Project Effectiveness ............................................................ 26

4-5. Impact ........................................................................................................................................ 26

4-5-1. Impact of Program Outcomes ............................................................................................. 26

4-5-2. Impact with Regards to CHS/Sphere Standards ................................................................. 28

4-6. Connectedness and Sustainability .............................................................................................. 29

4-7. Observance of CHS and Sphere Standards ................................................................................ 30

5. Lessons Learned .............................................................................................................................. 33

6. Recommendations ........................................................................................................................... 35

6-1. Specific Recommendations ....................................................................................................... 35

6-2. Summary of the Program Evaluation ........................................................................................ 36

Annex 1: Evaluation Grid .................................................................................................................. 38

Annex 2: Key Informant Questionnaires (JPF, UN, and NGOs) ................................................... 40

3

Annex 3: UN Responses to the Key Informant Questionnaire ....................................................... 50

4

1. Executive Summary Japan Platform (JPF)’s humanitarian assistance towards South Sudan has been ongoing for the past 14 years. Though there were some early signs of improvement, especially after the independence in 2011, there have been sporadic but extremely violent confrontations that affect the most vulnerable people, mainly the women, children and the disabled and the elderly. The projects under JPF target these populations in the countries where they reside in camps and temporary shelters.

The IC Net Team analyzed the overall South Sudanese refugee/Internally Displaced People (IDP) program based on the evaluation criteria set forth by JPF which was partly based on the Organisation for Economic Co-operation and Development (OECD) Development Assistance Committee (DAC) Evaluation Committee’s Guidance for Evaluating Humanitarian Assistance in Complex Emergencies (2001).

The program has been implemented with the JPF’s defined purpose to “respond to humanitarian needs in displaced locations and deteriorated humanitarian conditions, and strengthen resilience for the target population.”

Relevance/ Appropriateness: The JPF’s South Sudan Program is satisfactorily relevant to the international community programs but highly relevant to the Ministry of Foreign Affairs (MOFA)’s humanitarian aid policy. The program is designed to be in alignment with MOFA’s humanitarian aid policy as the budget is determined per country or region. On the other hand, the JPF’s response plan is developed based on the United Nations High Commissioner for Refugees’ (UNHCR) Regional Refugee Response Plans (RRRP), United Nations Office for the Coordination of Humanitarian Affairs’ (OCHA) Humanitarian Response Plans (HRP) and host country’s refugee response plans. The relevance and appropriateness of the program are also enhanced through needs assessments and implementing the Nongovernmental Organization (NGO)’s early coordination with local stakeholders. Therefore, the program had achieved high appropriateness.

Efficiency: The evaluation of efficiency was conducted by analyzing two factors, the fund allocation and project period. There is no delay in the release of funds, and the disbursement schedule is evaluated as appropriate to finance the projects by its member NGOs, as was the degree to which the funds were actually executed. The average of the actual project period was 118% of the planned period with minimum extension to deal with unexpected external factors. Therefore, the program has a high efficiency.

Coordination: Though not quantifiable, there were many forms of coordination and collaboration initiated by the member NGOs and the key stakeholders (UN, the national and local governments, etc.) that most likely enhanced or leveraged the project efficiency and effectiveness. Both the UNHCR officers in Uganda and Kenya evaluated JPF funded projects highly. Therefore, the program has high coordination.

Effectiveness: The number of planned and reached beneficiaries were used as an indirect measurement to achieve the results. Most of the projects accomplished to reach out to more people than planned, and consequently, the number of actual beneficiaries were 26% more than planned for the program as a whole. Moreover, including the vulnerable people, that are elderly, women, children, disabled, etc., 87% of the program’s total beneficiaries enhanced its effectiveness. Therefore, the program achieved high effectiveness.

Impact: Evaluating the impact of humanitarian work has been challenging due to the nature of its purpose and the duration of the projects. For this evaluation, impacts and secondary effects reported in the NGO final reports were used to measure the impact for this evaluation purpose with additional information gathered from the UNHCR. Those extracted were grouped into seven sectors (i.e., education, food, peacebuilding, WASH, shelter and NFI, and protection and psychosocial support). Each of them falls in one of the characteristics (i.e., horizontal ones, those beyond intended outcomes, those related to awareness and behavior change, and significance in quantity), which shows that the projects brought certain types of impacts pertinent to each sector though the impacts and secondary effects described in the NGO reports are not compatible with quantitative measurement. While some negative impacts are found, overall, many cases of impacts, most notably moderate awareness and

5

behavior changes, that can lead to strengthened resilience of beneficiaries were reported. Therefore, the program could potentially generate impacts that have contributed to the achievement of the program's purpose. As these were hypothetical, best-case scenarios, the program has an inconclusive impact.

Connectedness and Sustainability: Similar to impact, given that projects were funded on a single year basis and each project was implemented in a year, there is an inherent difficulty in evaluating sustainability and connectedness of the program and the study found only a few examples that showed significance or credible signs for sustainability. The final reports also indicated that there is an insufficient understanding of the concept of these two among member NGOs; most described how to ensure lasting outcomes through collaboration with and capacity development of the stakeholders. Other aspects of sustainability, such as that of financial, have very few NGOs reported on that. Thus further analysis could not be carried out. In sum, though all the projects were designed and implemented aiming at ensuring sustainability and connectedness, the programs’ sustainability and connectedness are inconclusive.

Observance of Core Humanitarian Standard (CHS) and Sphere Standards: There were inconsistencies in the reporting as different versions of the Sphere Standards were used by NGOs, and mentioning of the Core Humanitarian Standard were omitted. Considering the given condition of the South Sudanese IDPs and refugees’ support, meeting the minimum standards of Sphere handbook’s technical chapter is virtually impossible to achieve. Therefore, it is important to note that all the projects implemented under the program strived to adhere to the minimum condition and included it in their project design. Nonetheless, the program has a low observance of CHS and Sphere Standards because the program itself does not require that NGOs observe the standards fully, but rather recognize them and try to meet them. It is also important to note that using the metrics set forth by the CHS and Sphere Standards are appropriate as the standards to abide by for the program evaluation.

In this program evaluation, it was revealed that there was a need for further deliberation of concepts such as synergy, secondary effects, etc., and goal-setting for the program. While autonomy of the member NGOs is important for project efficiency and effectiveness, it would be useful to have a clearer understanding on what to be achieved by the program and how to achieve it among JPF and the member NGOs as JPF is in the humanitarian business for the world’s most needy due to natural and man-made disasters. . It is important to provide measurable and tangible impacts to the South Sudanese refugees, the IDPs, and host communities in the surrounding countries with a long term, overarching goal. That will help JPF to conduct a more accurate program evaluation.

6

2. List of Abbreviations AAR: Association for Aid and Relief, Japan

ADRA: Adventist Development and Relief Agency Japan

ALNAP: Active Learning Network for Accountability and Performance in Humanitarian Action

CCCM: Camp Coordination and Camp Management

CFS: Child-Friendly Spaces

CHS: Core Humanitarian Standard

CN: Concept Note

COVID-19: Severe Acute Respiratory Syndrome Coronavirus 2 (SARS-CoV-2)

CPMS: Child Protection Minimum Standard

CRRP: Country Refugee Response Plan

DAC: Development Assistance Committee

ECCD: Early Childhood Care and Development

HRP: Humanitarian Response Plan

IDP: Internally Displaced People

INEE: Inter-Agency Network for Education in Emergencies

JCCP: Japan Center for Conflict Prevention

JISP: Japan International Support Program

JPF: Japan Platform

JPY: Japanese Yen

KII: Key Informant Interviews

KIQ: Key Informant Questionnaire

GOE: Government of Ethiopia

GOJ: Government of Japan

GOK: Government of Kenya

GOU: Government of Uganda

GBV: Gender-Based Violence

MOFA: Ministry of Foreign Affairs

MOU: Memorandum of Understanding

MPJ: Millennium Promise Japan

MSF: Médecins Sans Frontières

NGO: Nongovernmental Organization

NFI: Non-Food Items

NRC: Norwegian Refugee Council

7

OECD: Organisation for Economic Co-operation and Development

PDM: Project Design Matrix

PLAN: Plan International Japan

PSN: Persons with Special Needs

PSS: Psychosocial Support

PTSA: Parent, Teacher and Student Associations

PWJ: Peace Winds Japan

RCM: Refugee Coordination Model

RRC: Regional Refugee Coordinator

RRRP: Regional Refugee Response Plan

SCJ: Save the Children Japan

SENS: Standardized Expanded Nutrition Survey

SGBV: Sexual and Gender-Based Violence

SPJ: Sustainable Development Goals Promise Japan

UNICEF: United Nations Children's Fund

UNHCR: United Nations High Commissioner for Refugees

UNOCHA: United Nations Office for the Coordination of Humanitarian Affairs

WASH: Water, Sanitation & Hygiene

WVJ: World Vision Japan

8

3. Introduction 3-1. Purpose, Scope, and Objectives IC Net carried out a program evaluation of the programs on response to the South Sudan Humanitarian Crisis as well as project monitoring and evaluation of the 16 projects being implemented or recently completed in South Sudan and its surrounding host countries.

The main purposes of the evaluation activity are to:

1. Carry out a desk review of all the projects funded by JPF under these programs to analyze the allocation of funds per sector, country and member NGOs and collaborations between NGOs, local governments and the UN;

2. Analyze the program strategy by comparing with the project achievements and outcomes;

3. Compare the outcome of JPF funded projects with the yearly Country Response Plans and needs assessment reports published by OCHA and other UN Agencies to analyze the relevance of the program with the international community programs; and

4. Analyze the effectiveness and performance of the program according to the program strategy such as allocation of funds and releasing funds on time, the relevance of projects, completion of projects, and target population and locations.1

This JPF’s program evaluation’s scope involved the three programs, namely the South Sudan Assistance Program (2016-2018), South Sudan Humanitarian Response (2018-2020), and South Sudan Refugees Emergency Assistance Program (2018-2020). As agreed during the inception phase, the three programs will be treated as a single comprehensive program for the purpose of this evaluation as it encompasses JPF’s comprehensive assistance to the South Sudanese IDPs and refugees. This assignment identified specific issues and successes of the JPF’s humanitarian assistance in South Sudan and provided a basis for future JPF’s program evaluations.

The scope of the program evaluation included using the definitions used by both JPF and DAC-OECD2. The team believes that the relevance/appropriateness touches upon the issue of policy “coherence,” as defined by the Active Learning Network for Accountability and Performance in Humanitarian Action (ALNAP) guideline3, but no analysis was conducted because the main use of that evaluation criterion was “Joint evaluations, large-scale evaluations, and those with a focus on policy.” 4 Additionally, coverage/targeting is used mainly for “all evaluation types except those with a mainly institutional focus.” 5Hence this criterion was also excluded from the program evaluation.

As South Sudan has been the longest recipient of the JPF funds, it is imperative to carry out an evaluation that encompasses the breadth and depth of the assistance and experience that JPF and its implementing partners have accumulated to date.

3-2. Overview of JPF The Japan Platform (JPF) is an international emergency humanitarian aid organization founded in 2000, and serves as an intermediary support organization providing various assistance to member NGOs in Japan.

1 Request for Proposal Terms of Reference for South Sudan Program Evaluation (JPF-GNR-20-001) 2 OECD (2001), Evaluation and Aid Effectiveness No. 1 - Guidance for Evaluating Humanitarian Assistance in Complex Emergencies, OECD Publishing, Paris, https://doi.org/10.1787/9789264033818-en [Accessed March 14th, 2020] 3 Overseas Development Institute (2006), Evaluating Humanitarian Action Using The OECD-DAC Criteria- An ALNAP Guide for Humanitarian Agencies. 4 Ibid, p. 21. 5 Ibid

9

JPF offers the most effective and prompt emergency aid in response to global developments, focusing on issues of refugees and natural disasters utilizing a tripartite cooperation system where NGOs, business communities, and the Government of Japan (GOJ) work in close cooperation, based on equal partnership, and making the most of the respective sectors’ characteristics and resources.

JPF has supported the aid activities of 43 member NGOs, each with its own set of diverse strengths6. It has delivered humanitarian assistance to 55 nations and regions over 1,500 projects, with a total financial contribution of 60 billion yen to date. JPF has built a strong reputation based on trust by promoting cooperation among private sectors and NGOs and by reporting all of its activities accurately.

JPF has started the “South Sudan Assistance Program” in 2006 and been implementing projects for government and communities to promote the Consolidation of Peace in South Sudan. Starting in 2014, JPF has been offering emergency response for those who have become refugees and IDPs due to the newly erupted civil war. JPF conducted a field assessment in 2016 to ascertain the validity of extending assistance for this protracted humanitarian crisis and to determine the program policy based on this assessment. As the heavy fighting erupted in July of 2016, the number of refugees increased dramatically, causing a large influx of refugees to neighboring countries of Uganda, Ethiopia, Kenya, and Sudan.

3-3. Context Since her independence, South Sudan has been in a protracted humanitarian crisis for nearly a decade. According to the latest Humanitarian Response Plan 20207, “the cumulative effects of years of prolonged conflict, chronic vulnerabilities, and weak essential services have left 7.5 million South Sudanese people—more than two-thirds of the population—in need of humanitarian assistance. Nearly 4 million people remain displaced: 1.5 million internally and 2.2 million as refugees in neighboring countries. The country remains in a critical period of unprecedented severe food insecurity, with 6.4 million people considered food insecure, and with malnutrition rates of 16 %. Protection concerns remain significant, with affected populations expressing fear over persistent insecurity, protection threats, human rights violations, and gender-based violence (GBV).” Figure 1 below shows the magnitude of South Sudanese Refugees’ presence in the neighboring countries.

According to the UNHCR’s 2019-2020 Regional Refugee Response Plan for the South Sudan Situation8, the UNHCR “sought to mount a regionally coherent inter-agency response supported by host governments in five countries of asylum, including Ethiopia, Uganda, Kenya, and Sudan, over the next two years. The shift of time span for the Regional RRP, changing from the past practice of a one-year

6 Th figure is based on the JPF website. 7 UNOCHA, Humanitarian Response Plan South Sudan 2019-2020 8 UNHCR, January 2019 - December 2020 South Sudan Regional Refugee Response Plan, pp.7-8.

Figure 1: Refugees and asylum-seekers from South Sudan (as of 30 April 2020), UNHCR Operational Portal (https://data2.unhcr.org/en/situations/southsudan)

10

planning cycle to two years, was introduced with a view to ensuring longer-term predictability in the planning and programming of life-saving and resilience needs of South Sudanese refugees in the region.”

The UNHCR reports that an estimated 790,000 South Sudanese refugees stayed in Uganda. During 2018, new arrivals dropped to 34,000, which was a huge reduction from 2,000 per day on average in 2016 and 2017. Food ration for the refugees who arrived before June 2015 stopped, but the land given by the Government of Uganda did not yield sufficient produce for refugees.

Over 840,000 South Sudanese refugees lived in Sudan in 2019. The estimate of the number of South Sudanese refugees by the Government of Sudan was over 1.3 million, although refugee verification was not able to confirm this figure. Standardized Expanded Nutrition Survey (SENS) for refugee camps in White Nile State of June 2018 warned critical global acute malnutrition with severe acute malnutrition above emergency levels in both refugees and the host community.

Over 420,000 South Sudanese refugees formed the largest refugee population in Ethiopia. Violence in Upper Nile, Jonglei and Unity States of South Sudan continued and pushed around 20,000 new refugees in 2018. The majority settled in the expanded Nguenyyiel camp in the Gambella Region, and a small part of them was relocated to Gure Shembola camp in May 2017. Intra-communal tensions were rampant in 2018 due to scarcity of resources, land disputes, and demographic changes in the Gambella region.

Most of the approximately 114,000 refugees from South Sudan living in Kenya were placed in Kakuma camp and Kalobeyei settlement in Turkana county. Kalobeyei settlement was opened to ease the pressure on the Kakuma camp and move from an aid-based model of refugee assistance to that with more focus on self-reliance. In 2018, the Kenyan government took more responsibility in protection services deliver and made significant achievements in strengthening the national refugee management system.

According to the OCHA 9 , the constraints faced in delivering humanitarian aid typically include bureaucratic impediments, operational interference, and violence against humanitarian personnel and assets. Poor road conditions, compounded by unprecedented heavy rains and floods during the rainy season, can significantly limit the ability of the humanitarian organizations to reach people in need. This is further exacerbated by checkpoint difficulties, including demands to search for personnel and vehicles. Active hostility, military operations, and intercommunal conflict led to the disruption of aid delivery.

3-4. Methodology and Limitations 3-4-1. Methodology

As a framework for program evaluation, the team:

1. Conducted a literature review of all of the relevant documents which includes program purpose, strategy, roles, and responsibilities to clarify the program evaluation objectives;

2. Developed a list of all of the projects undertaken during the 2016-2020 period, excluding the three needs assessments undertaken in 2016;

3. Revised the evaluation grid (refer to Annex 1) for the program;

4. Developed the key informant questionnaires for the NGOs, UN, and JPF based on the literature review;

5. Carried out the key informant questionnaires (KIQs) and key informant interviews (KIIs) with the available officers of the stakeholders: the NGOs, the UN, and JPF; and

6. Analyzed the qualitative and quantitative data independently and comprehensively and weighed them by evaluation criterion to come up with the final results.

9 UNOCHA (2018), Humanitarian Response Plan South Sudan 2019-2020

11

The analysis was qualitative in nature but did attempt to incorporate as many quantitative data. In particular, as mentioned in 6 above, the team utilized a weighing system to place a score on each of the seven evaluation criteria that were used. The specific weights placed differ from criterion to criterion, keeping in mind the balance of quantitative (objective), self-reported (subjective), and third-party (objective) data available.

As was agreed in the inception report, the evaluation criteria were the following: relevance/appropriateness, connectedness, efficiency, effectiveness, and impact. The DAC criteria are intended to be a comprehensive and complementary set of measures.

The team set out to tally the quantitative data derived from the financial reports from the applications to the final reports to collect fund allocation information for each component. The components were also categorized into the 15 sectors that the JPF had assigned in the 2017 and later versions of the application. For the 2016 application, the components were categorized using the 2017 application template as a guide to keep its coherence.

The KIQs (Annex 2) were drafted using the key questions that were presented in the evaluation grid (Annex 1). These questions were further distilled to be quantitative in some aspects and include multiple-choice answers to increase the response rate. In order to allow for further explanation, open-ended questions were included subsequent to the multiple-choice questions.

Once all of the responses to the KIQs and KIIs were gathered, the team considered the number of responses received from the stakeholders, in its totality; out of the five UNHCR and country offices that we contacted10, only two responded; out of the 16 JPF-funded projects, 16 responses were obtained; of the JPF officer contacted, we had one KIQ, one Skype follow up session and one email response. For the UN responses to the KIQ please refer to Annex 3. During the analysis, characteristics of the responses were examined and weighed to see whether there were important themes that were repeatedly mentioned to be included in the final report. After discussing the themes, the team drafted each of the chapters.

3-4-2. Limitations In carrying out the program evaluation, the team faced the following limitations.

1. Due to the COVID-19 pandemic, there were limitations to the data that could be collected. For example, the team was not able to obtain all of the responses from the UN in the time allotted.

2. Furthermore, the response obtained by the UN field officers has been, though intended to be applicable for all the JPF-funded projects in the country the entire country, focused on a few specific projects.

3. In order for the program years to be treated separately and not to draw conclusions from incomplete information, the final reports submitted in the year 2019-2020 was not included in this analysis.

10 The UNHCR and OCHA country officers in the program countries were contacted via email and were introduced through their colleagues to ask questions specific to the JPF. The questions include those on relevance, appropriateness, effectiveness, coherence, aid coordination, impact..

12

4. Analysis and Findings 4-1. Relevance/ Appropriateness Relevance, according to the OECD/DAC, ALNAP and JPF’s definition for evaluation, is “concerned with assessing whether the project is in line with local needs and priorities as well as donor policy,” while appropriateness “is the tailoring of humanitarian activities to local needs, increasing ownership, accountability accordingly.” This part of the evaluation includes desk research to seek the alignment of the donor policies, namely the Ministry of Foreign Affairs’ and United Nations’ strategies with the JPF South Sudan program strategy, as well as an in-depth examination of the United Nations Country Refugee Response Plans.

4-1-1. Relevance to the Ministry of Foreign Affairs’ and United Nations’ Strategy The JPF’s assistance policy to South Sudan is very much aligned with the GOJ’s humanitarian aid policy, as the earmarked funds are determined by the MOFA. The ordinary budget process is approved in May-June every year, while the supplementary budget planning process that is approved at the end of the year and JPF follows that budget cycle. The MOFA identifies the country or the region where the supplementary humanitarian assistance funds for that year should strategically go. The supplementary budget is not provided in its entirety as the ordinary budget could be but is determined by the country. After the amount of money for each program and country has been decided for both the original budget and supplementary budget, a request for Concept Notes (CN) is carried out, the CNs submitted and evaluated for ranking them in highest order by score. After that, the exact amount of funding is determined to start with the highest bidder.

JPF bases its yearly strategic response plan on the South Sudanese needs information and humanitarian response strategy presented by the UNHCR’s South Sudan Situation Regional Refugee Response Plan and OCHA’s Humanitarian Response Plan. Both of these response plans are published every year in accordance with its programming cycle and changing situations on the ground while highlighting the acute and chronic needs that have been reported by the UN and its partner NGOs in the previous reporting period. As the member NGOs are encouraged to formulate their CNs based on the most recent RRRPs and HRPs, the JPF ensures that the latest information in the respective country and that the CNs are aligned to and responsive to the needs identified by the international community.

The selection of refugee and IDP camps that the member NGOs work in are relevant because these are where there are the highest needs; Jonglei, Western Equatoria, and Upper Nile in South Sudan, as well as Adjumani, Kiryandongo, Arua, Bidibidi, Yumbe in Uganda, Kakuma camp and Kalobeyei in Kenya, and Gambella, Kure, Jewi, Terekidi camps in Ethiopia and the While Nile in Sudan. Specifically, the Upper Nile was identified as one of the three northern states in South Sudan that had the most pressing need for humanitarian interventions. The member NGOs have implemented projects within South Sudan in areas considered “critical levels of intersectoral needs” in states of Central Equatoria (including Juba), parts of Jonglei including Bor, parts of the northeast Upper Nile bordering Ethiopia and Sudan. In Kenya, the member NGOs are working in the two largest camps, Kakuma and Kalobeyei, while in Ethiopia, the NGOs work in the Gambella region camps. In Uganda, the northern districts of Yumbe, Adjumani, and Arua host most of the South Sudanese arriving from food insecure Central and Eastern Equatoria states. In Sudan, the White Nile state is where many of the residents of the Upper Nile state flee from due to food insecurity there.

Moreover, most JPF projects had two to three components that spanned across one to two complementary sectors. This multisectoral approach is in line with the UN’s strategy, as one of OCHA’s three strategic objectives aims to provide “timely and integrated multisector assistance to reduce acute needs.”11 This is important in the sense that the member NGOs are working to provide the same beneficiaries with different types of aid, to sustain the benefit. The school buildings, with training on improving pedagogy, protection issues, and awareness of adolescent girls’ issues, are all important components for children’s educational attainment.

11 UN OCHA (2019), Humanitarian Response Plan, Monitoring Report January-March 2019

13

4-1-1-1. Relevance to National Strategies, Guidelines and Policies For the projects where data was available, the team compared the outcomes of JPF funded projects with the yearly Country Response Plans published by the UN Agencies to analyze the relevance of the program with the international community programs.

Ethiopia: The Government of Ethiopia (GOE) made nine pledges12 to comprehensively respond to refugee needs and is formulating a National Comprehensive Refugee Response Strategy (NCRRS), which includes enrollment increase in primary, secondary and tertiary schools. The UNHCR’s Ethiopia Country Refugee Response Plan January – December 2018 (2017) stated that “the South Sudanese are the largest refugee population in Ethiopia, totaling approximately 420,000 persons at the close of 2017.” Of the new arrivals to the Gambella region where most of the South Sudanese refugees sought shelter, approximately 86% are women and children, with women accounting for 62% of the adult population, many of whom are heads of households. 23% of the new arrivals are youth that has specific needs. The risk of epidemics remains high due to ongoing cholera outbreaks and poor WASH conditions within the camps. The WASH project implemented in the Gambella region is consistent with the needs identified and prioritized by the UNHCR, while the secondary education project is supported by the GOE in its NCRRS.

The GOE’s livelihood approaches and responses are stated in its 2019–2020 Country Refugee Response Plan (RRP), where they identify the key components such as the expansion of access to education at primary, secondary, and tertiary levels by integrating refugees in the national educational system. Specific targets included: increasing the enrolment of: pre-school aged refugee children from approximately 46,000 (44 %) to roughly 63,000 (60 %); primary school-aged children from approximately 97,000 (54 %) to 137,000 (75 %); secondary school-aged refugees from approximately 3,800 (9 %) to roughly 10,000 (25 %); and for opportunities for higher education enrolment from 1,600 to 2,500 students.

Kenya: The Government of Kenya (GOK) made significant progress in the inclusion of refugees in the County Integrated Development Plans and the UN Development Assistance Framework 2018-2022 by including refugees as a target population. The Government of Kenya, with the support of partners, has taken significant steps to include refugees in national systems, such as in education and health. The JPF projects in Kenya (2016-2018) were mainly in shelter, WASH, protection, and involved refugees settled in Kakuma, Turkana, and Kalobeyei camps, and many were multi-year projects.

Uganda: Uganda has received the highest number of South Sudanese refugees since 2016. From January to March 2017, the country had over 180,000 new arrivals, 60% of the initially expected figures for the year 2017. As of January 2020, almost 870,000 of the approximately 1.4 million refugees and asylum seekers are South Sudanese (62%)13.

The Government of Uganda (GOU) developed its Comprehensive Refugee Response Plan in 2017, Refugee Response Plan 2018, and Refugee Response Plan 2019-2020, by analyzing the humanitarian needs and requirement of the refugees. Through these Response Plans, the GOU emphasized the importance of providing support to the host community and pursuing the 30- 70 principal, where 30 % of all assistance towards the refugee should benefit the host community to maintain a peaceful

12 The pledges are: 1. To expand the “out-of-camp” policy to benefit 10% of the current total refugee population; 2. To provide work permits to refugees and those with permanent residence ID; 3. To provide work permits to refugees in the areas permitted for foreign workers; 4. To increase enrolment of refugee children in preschool, primary, secondary and tertiary education, without discrimination and within available resources; 5. To make 10,000 hectares of irrigable land available, to enable 20,000 refugees and host community households (100,000 people) to grow crops; 6. To allow local integration for refugees who have lived in Ethiopia for over 20 years; 7. To work with industrial partners to build industrial parks to employ up to 100,000 individuals, with 30% of the jobs reserved for refugees; 8. To expand and enhance basic and essential social services for refugees. 9. To provide other benefits, such as issuance of birth certificates to refugee children born in Ethiopia, and the possibility of opening bank accounts and obtaining driving licenses. 13 UNHCR Uganda Factsheet January 2020, http://reporting.unhcr.org/sites/default/files/UNHCR%20Uganda%20Fact%20Sheet%20%20-%20January%202020_0.pdf [Accessed on May 17, 2020]

14

coexistence. Considering the fact that 86% of South Sudanese refugee are women and children, their strategic priorities goes to protection interventions, including the prevention and response to child protection risks and sexual and gender-based violence (SGBV).

From 2016-2018, JPF funded 15 projects in Uganda, of which nine had a protection component within their projects (60%). Through the KIQ for the NGOs currently implementing projects, it was confirmed that all were respecting the Government of Uganda’s 30-70 principals, and designed their projects to ensure that the 30% of the benefit goes to the host community.

Because the JPF projects are moderately relevant to each country’s response plans and the regional plans, the JPF program’s relevance to the international community programs is satisfactory.

4-1-2. Contribution to the UN and Host Country Goals

The JPF member NGOs have leveraged its strengths to contribute to the UN’s goals. As a matter of fact, 15% of the 200 indicators that the OCHA and the UNHCR objectives have in their 2018 HRP and RRRPs concur with the results that are tallied from the outcomes of the JPF funded projects. Though most of the projects include activities that could not be matched one to one to the indicators, they serve as activities to reach the outcome indicators stated in the HRPs and RRRPs.

The NGOs have contributed to sectors such as food, where the WFP is the dominant player and shelter and NFIs where the UNHCR is the dominant player, where there are not many other NGOs working. This dramatically increases the NGOs and, consequently, the JPF’s visibility and adds value to the contribution.

At the same time, in terms of the UN strategic objective indicators set for each country had numerical targets to reach, such as the number of students enrolled or the number of case management, the number of latrines rehabilitated or constructed, etc., the JPF-funded projects tended to have a lot of awareness campaigns in their activities. Some of the activities were to introduce and encourage usage of the facilities that were rehabilitated or built in a more effective manner, something that forms that basis for impact. In other words, these awareness campaigns aimed to change the behavior of the beneficiaries to understand the importance of child protection prevention and therapy, early childhood education, matriculation, etc. so that the beneficiaries used CRC/CFSs, made use of the educational services provided, etc.

However, as the UN tended to use specific indicators that could be standardized in other contexts. For health, the indicators were usually the child or pregnant women’s mortality rate, immunization rate, malnutrition rate, etc. The JPF funded projects would include campaigns that changed behavior of the beneficiaries to use the health facility through nutrition sessions, a positive impact would eventually lead to lower mortality rates in children and pregnant women. Therefore, the activities carried out by the member NGOs demonstrate a direct linkage to the output indicators set forth by the UN. In that regard, as long as the project impacts can be adequately measured, the JPF program’s contribution to the UN and host country goals can be achieved. So the leverage that the JPF program has is that for certain types of assistance, it directly contributes to the UN’s goals, while in others, it provides the essential “building blocks” for the results chain to function for the beneficiaries to strengthen resilience.

4-1-3. Appropriateness of the Projects

The response from the field officer at UNHCR Uganda to the question asking to what extent has the JPF funded projects supported beneficiary targeting and contributed to improved geographic coverage to ensure that the most vulnerable groups’ needs are addressed there, was “very appropriate.” Regarding whether UNHCR Uganda thinks that JPF funded project’s approach to addressing the needs on the ground was appropriate, the response was, “Yes, they provided safe spaces for children and youth through the Early Childhood Care and Development (ECCD) and Child-Friendly Spaces (CFS).” The response from the UNHCR field officer in Kenya was even stronger; “Yes. JPF funded project was appropriate as it addressed housing conditions for refugees, sanitation behaviors, and service for both in schools and settlement as well as access to water hence improving sanitation within the settlement

15

and surrounding host community. They are coordinating well on the COVID-19 preventive measures, including producing comic books on good hygiene practices for children.”

Additionally, in response to the question about how has the JPF funded projects contributed the most to the humanitarian assistance, the Uganda field officer’s response was “in Adjumani, they helped provide support to children at risk,” while the Kenya field officer’s response was “through Japan platform, PWJ has demonstrated ability in bridging the gaps in the areas of shelter, water, and sanitation aimed at improving the living environment of refugees and host communities holistically and sustainably in Kalobeyei Integrated settlement. PWJ mainstreamed disability in their programming as well as benefiting PSNs living in the Kalobeyei settlement. They championed the CLTS (community-led total sanitation) programme leading to Open Defecation Free (ODF) zones within the Kalobeyei settlement and will work together with NRC to roll out the CLTS to Kakuma camp where over 150,000 refugees reside.”

As the geographic coverage of assistance has been well-coordinated through each NGO’s sector working groups, which are organized and led by the UN sector leaders, there were very few duplications. The JPF response strategy is adequate in that it summarizes the needs that the UN has identified and puts together a list of the priority areas that its member NGOs can implement on. The NGOs have the autonomy of working in the geographic areas that they deem important, having already been vetted by the UN. Therefore, the program had achieved high appropriateness.

4-2. Efficiency According to the OECD/DAC ALNAP and JPF definition of efficiency, it “measures the outputs – qualitative and quantitative – achieved as a result of inputs. This generally requires comparing alternative approaches to achieving an output to see whether the most efficient approach has been used.”

The evaluation of efficiency was conducted by analyzing two factors, the fund allocation and project period. The quantitative data, mostly financial and project implementation period information in the CN and in the final reports of the NGOs were used to compare the planned and actual figures by sector (education, food, peacebuilding, water and sanitation, shelter and NFIs, protection/psychological support), by country, and by years (2016-2018). Aggregate data was compiled to monitor trends, and where there were interesting findings, the results are reported on.

During the 2016-2019 period for all of the projects that have been completed, the JPF had allocated approximately 3.5 billion JPY to its member NGOs. Three were dedicated to needs assessments during the fiscal year 2016-2017 and, as mentioned above, and have been excluded from the program evaluation since the nature of the assessment was different from the implementation of the humanitarian assistance projects.

The funding has increased every year and has increased by about 50% over the three years from JPY969,008,309 (2016) to JPY1,473,376,217 (2018), as shown in Figure 2 below.

16

Figure 2: The Number of Projects and Funding by Year

Of the JPY3,532,129,978 disbursed during the years 2016-2018, 39 projects were implemented, and the average project distribution was JPY90,567,435. During this period, nine-member NGOs received funds from the JPF; from the currently implementing projects, no NGO reported any issues with the timing of the fund disbursement or any mismanagement of the funds.

The disbursement rate for these three years is high: 97.40%, 98.65%, and 98.71%. Though there is no significant difference in the rates by year, it is gradually increasing. This may indicate that the member NGOs’ budget planning and fund execution are improving in precision and thus improving inefficiency, though a long-term data trend should be considered for a stronger conclusion to be drawn.

Table 2: Fund Disbursement Rate (2016-2018)

Year Rate

2016 97.40% 2017 98.65% 2018 98.71%

The JPF funded projects were implemented in the following countries: 14 in South Sudan by NGOs WVJ, PWJ, JCCP, 25 in Uganda by NGOs SCJ, AAR, PWJ, PLAN, WVJ, MPJ, and SPJ, eight in Kenya by NGOs PWJ, JISP, and eight in Ethiopia by NGOs WVJ, ADRA, and PWJ. The projects were in the following sectors: shelter and non-food items (NFI), health, WASH, social protection, peacebuilding, education, food, as well as education and health, and WASH, and NFI.

4-2-1. Allocation Per Sector, Per Country, and Per Member NGO

During the three years from March 1, 2016, to September 31, 2019, JPF funds enabled member NGOs to provide assistance to more than 1.4 million people. The funds were allocated the most to the WASH and education projects composed of 39% and 30%, respectively, followed by social protection, shelter and NFI composed 10% each (see Figure 3). The amount was calculated only by direct expense, excluding common costs and indirect expenses such as travel cost and operation cost.

JPY 0

JPY 500,000,000

JPY 1,000,000,000

JPY 1,500,000,000

JPY 2,000,000,000

201614 Projects

201712 Projects

201813 Projects

Total Funding by Year

Amount Planned Amount Executed

17

Figure 3: Amount Executed by Sector

In terms of the per sector spending by the component of each project, the highest was in education (JPY34,389,752), followed by WASH (JPY29,256,521), then education and health (JPY26,306,450) as described in Table 2 below.

Table 3: Per Component Funding by Sector (2016-2018)

Sectors Per Component Fund

(JPY) Shelter and NFI 20,569,606 Social Protection 10,010,929 Peacebuilding 3,485,960 Education 34,389,752 WASH 29,256,521 Food 12,437,179 Health 14,810,969 Education and Health 26,306,450 WASH and NFI 11,664,000

Most of the projects for education and WASH included the building and rehabilitation of the latrines and classrooms. Therefore, it is reasonable that the fund per component for these sectors is relatively larger than others.

In terms of funding per sector per year, the largest amount executed in 2018 was WASH (JPY 261,135,565), followed by education (JPY 258,014,117) and social protection (JPY 107,693,474), as shown in Figure 4 below. Though social protection project spending is much lower compared to the other WASH and education, it is almost twice as much compared to the executed amount in 2016. Considering it was only the 5th largest sector in terms of spending in 2017, the proportion of total spending towards this sector has also increased.

10%

10%1%

30%39%

3%2%

4% 1%Amount Executed by Sector

Shelter and NFI Social Protection Peacebuilding

Education WASH Food

Health Education and Health WASH and NFI

18

Figure 4: Amount Executed by Sector by Year (2016-2018)

JPF’s response plans—both South Sudan Humanitarian Response 2018 and South Sudan Refugee Emergency Assistance Program 201814—clearly stated the urgent need to increase assistance for the protection sector. The amount of funding increase described above demonstrates that the NGOs followed the JPF’s strategy to provide assistance in that sector.

Therefore, the increase of the fund towards this sector is in alignment with the JPF’s response plan.

During the 2016-2018 period, 38 % (JPY1,361,219,958) of the budget was allocated to South Sudan, followed by 37% (JPY1,324,312,525) in Uganda, 22% in Ethiopia (JPY 775,560,179) and 3% in Kenya (JPY95,996,671), as shown in Figure 5 below.

Figure 5: Amount Executed by Country (2016-2018)

Most of the funding is allocated for Uganda and South Sudan (38% and 37%, respectively). This is in alignment with the highest number of refugees and IDPs (in the case of South Sudan) compared to its neighboring countries. Sudan has had the third greatest number of refugees, closely followed by

14 According to the response plan, the demand for assistance for protection in South Sudan is the highest with 6.4 million people (92% of the total number of people who needs humanitarian support) requiring the protection support.

050,000,000

100,000,000150,000,000200,000,000250,000,000300,000,000

Amount Executed by Sector per Year (2016-2018)

2016 2017 2018

Uganda38%

South Sudan37%

Ethiopia22%

Kenya3%

Amount Executed by Country (2016-2018)

19

Ethiopia, and then by Kenya. There is a discrepancy between the number of projects carried out in Sudan15 compared to the number of actual refugees.

JPF’s largest funding was made towards PWJ in 2018 for a WASH and health project in South Sudan for JPY326,094,166. The smallest funding, on the other hand, was made towards JISP in the same year for child protection and psychosocial support project in Kenya for JPY5,128,818. Incidentally, those NGOs with larger project funds were the ones who had operated the largest number of projects. See Table 4 for funding executed by NGO for years 2016-2018.

Table 4: Fund Executed by NGO (2016-2018)

Name of NGO Number of Projects Average Funding per Project (JPY)

AAR 4 97,625,408 ADRA 4 109,106,307 JCCP 5 41,120,904 JISP 2 6,323,057 MPJ 1 8,223,911

PLAN 2 67,812,985 PWJ 10 108,967,409 SCJ 4 79,898,000 WVJ 7 133,405,216

Figure 6: Fund Executed by NGO (2016-2018)

As seen above, there is a marked difference in the funding sizes of the projects due to how projects are being funded; the past performance and the current experience in the specified location of the NGO as an organization is 55% and for the South Sudan Refugee Program 2020, 1. the amount of funds the organization has received by the UN through its Implementing Partner agreements among others, the GOJ’s NGO scheme and donations, etc. (20%); 2. how much the project contributes to the HRP (in the case of South Sudan) and RRRP (in the case of Uganda, etc.) strategic objectives and sector outcomes (25%) and 3. Whether the project emphasizes protection of the refugees, IDP, and the host community members.

15 No projects were implemented in Sudan from 2016 to 2018, and only one in 2019.

AAR11%

ADRA12% JCCP

6%

JISP0%

MPJ0%PLAN

4%PWJ31%

SCJ9%

WVJ27%

Fund Executed by NGO

20

4-2-2. Project Period

In terms of the planned project period and the actual project period for years 2016-2018, the actual project period (in days) was 118% of its planned period, with an average of 222 planned days compared to 261 actual days. Overall, the reasons for the delay were predominantly independent of the management project, such as weather, violence, interference from state and non-state actors, receiving project approval from governments and the UN, etc.16 These are unforeseeable project delays, and in terms of being able to recover from those setbacks and be able to complete the project within a reasonable amount of time were important factors to consider. According to the KII with JPF, it is common for humanitarian assistance projects to be implemented with delays.

4-3. Coordination According to the JPF Evaluation Guideline, coordination is defined as “the initial response and emergency assistance project adheres to the assistance policy. The evaluation serves to verify if it was implemented effectively.”

For this evaluation, the description of coordination in the application and final reports of the NGOs was extracted and grouped to tease out certain themes on which recommendations could be based.

Ideally, in order for the JPF program to be most effective, projects should be designed to include activities that can be carried out with the key partners such as NGOs, the UN, etc. This is already built into the program design: in the application and the final reports, NGOs clearly describe their efforts with aim to (1) Position the project well with the cluster aid policies and plans; (2) Coordinate with the host government; (3) Coordinate with other actors. However, from a strategic perspective, the idea of “coordination” per se was only expressed in the JPF Response plans of 2018 and 2019-2020.

Overall, coordination was well implemented and was achieved. The coordination among the United Nations, local governments, and international and local NGOs are critical to creating a bigger impact and providing effective and efficient support despite finite funds and resources. For this program, as is the case of the most refugee situations, the UNHCR coordinates the overall response with Regional Refugee Coordinator (RRC), who is responsible for the coordination of the emergency response to address specific refugee needs regionally17.

All of the NGOs that responded to the KIQ were participating in the monthly cluster meetings held by the UNHCR, sub-sector working groups, information sharing platforms, district inter-agency coordination meetings, and/ or settlement coordination meetings, on top of regular information sharing with stakeholders including the JPF. The NGOs responded saying that attending these meetings was demarcate the role, create the referral network, and seek possible collaboration that could lead to synergistic effects.

Specifically, the team found the following examples of effective coordination and collaboration with the key stakeholders by sector that materialized into effective projects:

16 OCHA (2019) reports that in South Sudan, there are “bureaucratic impediments, operational interference, and violence against humanitarian personnel and assets. The difficult physical environment, including poor road conditions, was the most prevalent access challenge reported. This was compounded by unprecedented heavy rains and floods since July 2019, significantly limiting humanitarians’ ability to reach people in need.” 17 UNHCR (2019), Refugee Coordination Guidance, https://www.unhcr.org/en-us/protection/basic/5d7b50e74/refugee-coordination-guidance.html [Accessed on 17th May, 2020]

One of the NGOs working for the protection of children stated, “Inter-sectoral coordination mechanism could have been strengthened in close collaboration with other sectors such as Education and Livelihood working groups, which are essential to promote children’s protection in the target areas.”

21

Education: The NGOs have coordinated and collaborated with the national ministry of education to state-level education departments in the construction and operation of the schools. After the completion of this project, it is planned to proceed with UNHCR and Administration for Refugee & Returnee Affairs, and with the possibility of handing over to the State Education Bureau in the future, with the other related organizations.

Another member NGO collaborated in the capacity building of South Sudan’s local government officials, community leaders, and parent organizations on fostering ownership and ensuring project sustainability. The NGO provided training on school management, monitoring, and supervision to county education department officers, parents, and community members. Also, one of the NGOs has organized and conducted Training for Teachers with Solidarity with South Sudan, a member of the education cluster, who has experience in providing Training that the Government of South Sudan certified. The implementation of joint- training with the organization with local experience surely enhanced the effectiveness of the training.

WASH: The NGOs have collaborated with NGO sector leaders such as OXFAM. For example, in a latrine construction project in Ethiopia, the UNHCR provided supplies such as toilet bowl cleaning tools, UNICEF provided pamphlets and posters on hygiene awareness, and OXFAM provided chlorine.

Other examples of collaboration include the immediate hand over the clinic to local authorities who have been involved in the project. The NGO has been implementing the project in close collaboration with local NGOs, while coordinating with UNHCR, Médecins Sans Frontières (MSF) and ADRA. This collaboration and coordination enabled the smooth operation and the White Nile State Ministry of Health and Sudan Red Crescent Society managed the constructed clinics together, immediately after the construction work finished.

In terms of coordination, one member NGO decided to withhold construction of new latrines in schools because OXFAM and the Norwegian Refugee Council had already made plans on doing the same. Additionally, there was a case reported that briefing was conducted from the JPF’s implementing NGO that has been active in the same region to the NGO that newly participated. The coordination between the implementing NGOs has occurred spontaneously, and greatly contributed to the smooth execution of the project.

Protection and Psychosocial Support: In Uganda, there were several projects in which the collaboration between the local government, such as the local department of the Office of the Prime Minister’s Directorate of Refugees, attend training sessions of how to continue the child protection work after the NGO completes its project. Another member NGO collaborated with Transcultural Psychosocial Organization Uganda, one of the biggest local NGOs for mental health care in the country, to take over beneficiaries after the project conclusion.

In Kenya, the member NGO coordinated their activities with an NGO called Humanity & Inclusion and conducted play therapy sessions targeting disabled children at the Child Resource Center that was constructed in the previous project.

One officer of the UN in the KIQ said that case management could be improved with better coordination with the UNHCR partners.

As mentioned above, the member NGOs have achieved effective operations through coordination which can contribute to generate a positive impact. .

Beyond the collaboration with the local governments and international and national NGOs, the team included the overall rating by the UN country officers. The two UNHCR field officers have rated the projects’ coordination with them as “very high.” UNHCR officer in Uganda stated that “the JPF funded projects are well coordinated with the UN to avoid gaps and overlap in coverage in the humanitarian assistance.” This overall sentiment was echoed by the UNHCR’s Kenya field officer who said, “…PWJ coordinated very well with other stakeholders through WASH and shelter working groups as well as

22

with the senior management of UNHCR to avoid overlapping of the activities in Kalobeyei settlement while ensuring efficient use of resources. PWJ is very transparent on its resource allocations and has ensured that its funds are spent on gaps identified and not a duplication of assistance… PWJ attends all the coordination forums, including the protection working groups. They are also very much involved in the Kalobeyei Integrated Socio-Economic Development Plan (KISEDP) and are members of the thematic working group where they also implement and complement the flagship projects in shelter and WASH.18”

However, according to our KIQ and KII, NGOs have been facing some limitations/ problems of the current coordination system in the area of information asymmetry. Information gaps in coordination occurred for the following reasons:

- Frequency of the meeting: as the coordination meetings are held as frequent as every two weeks to quarterly with the average of every month, the information and data shared at the meeting can be a month to three-months-old and not always up-to-date.

- Frequent changes of staff: due to the nature of the humanitarian work, changes of the staff and urgent matter happen more often than development work, which makes it unable for designated staff to attend meetings all the time. When new/ tentative member participates the meetings, they are not likely to be able to share as much information as other participants hope.

- Difference in engagement level: some NGOs mention that not all participants are well engaged in these meetings and not actively share their plans and information they have.

- Limitation of the information within the sector: as the inter-agency meeting is usually set by the sector, the information beyond the sector that could be beneficial for the effectiveness of the project cannot be shared.

As the timeliness, accuracy, and adequacy of the information are very important to adjust to constantly changing situations on the ground, the information gaps create extra work for the NGOs, such as reconfirming and verifying the information/ data. Other notable limitations/ problems reported included a lack of initiative towards problem-solving within the working group. However, overall, effective coordination is carried out by the NGOs.

Therefore, the program has high coordination.

4-4. Effectiveness JPF’s definition of effectiveness is the “degree of achievement of the assistance program.” In other words, it means to ask questions such as “have the (intended) results been generated from the outputs? Was the assistance program provided in a timely manner?” As the issue of the timeliness is mentioned here, it is implied as part of its effectiveness.19 Please refer to Section: 4-2-4 above for further analysis on timeliness.

For the effectiveness criterion, the team used the number of beneficiaries as an indirect measurement for the achievement of results for the program evaluation, as each project had several activities for each

18 The Kenya UNHCR officer specifically referred to the PWJ implemented projects. 19 Overseas Development Institute (2006), Evaluating Humanitarian Action Using The OECD-DAC Criteria- An ALNAP Guide for Humanitarian Agencies, p.21.

One of the member NGOs stated, “there are no guidelines regarding allowances paid to whom engaged in the projects. Consequently, different organizations have different guidelines, so we had to take time to negotiate with the community on the issue of allowances during project implementation. We repeatedly requested UNHCR to negotiate with local government and set the guideline, but they did not take any action.”

23

project component in which outcomes were measured, as well as the outputs per sector (education, food, peacebuilding, water and sanitation, shelter and NFIs, protection/psychological support). More discussion on the outputs is in 4-4-3.

Overall, the outputs for each project were mainly achieved though there were sometimes more or fewer beneficiaries reached due to unplanned events/incidents. Only a few of the projects did not reduce or increase the number of components. Most changes in the project plan—whether it be in the cost, project period, or in the activities or yet the personnel—were approved in a timely manner.

4-4-1. Number of Beneficiaries Planned and Reached The number of beneficiaries targeted during the four years was approximately 1.43 million. 20 Compared to sectors that involve mainly capacity building, the WASH sector reaches a wide range of people as the latrine and water facilities are used by all. Figure 7 shows the number of planned beneficiaries for year 2016-2018.

Figure 7: Number of Planned Beneficiaries (2016-2018)

The number of beneficiaries that benefitted from the assistance was nearly 1.3 million, with the following breakdown, as seen in Table 5 below.

Table 5. Number of Planned and Actual Beneficiaries

Year Number of

Planned Beneficiaries

Number of Actual

Beneficiaries Difference

2016 410,914 557,597 36% 2017 294,108 317,061 8% 2018 321,744 421,198 31%

20 A note on figures: Each sector of activity presented in this report includes net figures for the number of people reached with JPF funds. Within the same project, some double-counting persist across components; however, for the purpose of comparing the planned and actual figures, it did not pose a problem. All figures reported have been rounded down.

Shelter and NFI15%

Social protection11%

Peacebuilding3%

Education27%

WASH38%

Food3%

Health2%

Education and health WASH and shelter/NFI

0%

Planned Beneficiaries (2016-2018)

24

Compared to the total planned number of beneficiaries of 1.03 million, there were 26% more actual beneficiaries. These are thanks to the member NGOs’ efforts to identify and reach more beneficiaries once the project commenced, and the beneficiaries’ willingness to partake in NGOs’ activities even though they may have or may have not been part of the consultations during the planning phase. As seen earlier, the WASH sector was the one that had the most beneficiaries every year, demonstrating the NGOs’ preference to carry out WASH activities in all of the program countries (Figure 8).

Figure 8. Number of Beneficiaries per Year per Sector (2016-2018)

In the year 2018, there was an increase in the number of beneficiaries in the protection and psychosocial support projects as well in the health and WASH projects. In addition, there was a modest increase in the number of beneficiaries from shelter and NFI projects.

Thus, the program has a high effectiveness.

18120 18450 9961 9140 1675 1974114540 20630

9661 10482

88784

378511875 1330

3269 2734

2037

24711445640890

34843 27678

50023

54659

355603

468779

179185

244135

149830 255713

6320

7033

1099

1883 700

1042

0

485

51045

15092 15000

10987

0

0

50455917 11395

478560

0

00 2300

8645

0

100000

200000

300000

400000

500000

600000

Planned Actual Planned Actual Planned Actual

2016 2017 2018

Bene

ficia

ries

Year

Number of Beneficiaries per Year, per Sector

WASH and NFI

Education and health

Health

Food

WASH

Education

Peacebuilding

Protection and psychosocial support

Shelter and NFI

25

4-4-2. Number of Most Vulnerable Beneficiaries Reached (2016-2018) In terms of the most vulnerable beneficiaries that were reached, out of the 576,741 program beneficiaries that were identified in the 2016-2018 JPF projects, a staggering 87% of them, or 499,320, were considered being “most vulnerable,” defined as women, children, elderly, and the disabled. This is the result of well-executed targeting of the most vulnerable and in need of each project site, as well as the sectors covered (such as education). In terms of the JPF, ensuring that the NGOs carry out the identification and the targeting of the most vulnerable population, this has been done in a highly satisfactory manner.

4-4-3. Overview of Outputs per Sector For this evaluation criterion, the description of outputs in the final reports of the NGOs was extracted and grouped by sector. In general, the projects had the following outputs.

Education: Generally, the actual number of classrooms built and classroom furniture provided was in accords with the original plans. The number of participants who attended teacher training, life skills coaching, girls support groups, and other groups such as PTAs established, were implemented as planned.

Food: In general, the food distribution had reached more than the originally planned households due to the influx of the new refugees in the 2016 food distribution project administered by JCCP. In the four-year peacebuilding/vegetable production project, which was also administered by the JCCP, the training, rehabilitation of plows and other gardening equipment, were all carried out that exceeded its planned outputs.

Peacebuilding: In the four-year food project mentioned above, the youth leaders were mentored and trained to resolve conflicts early, patrol the neighborhood at night, etc. In terms of the participation rate, it was lower than planned, though the outcomes of the training were as originally planned.

WASH: There was a high degree of exceeding the planned number of water points repair, shower construction, community latrine rehabilitation, and construction, septic tank construction, waterborne disease training (such as cholera) though there was one project (WASH and Shelter Intervention for the Refugees from South Sudan in Northern Uganda) in which the water tank construction was halted due to other organizations taking over the task to construct the tanks, as well as the refugees’ delay in moving into their shelters. In another WASH project (The set-up of Household latrines for South Sudanese refugees in Tierkidi refugee camp of Western Ethiopia in Gambella region Phase 2 administered by ADRA), 50 latrines were built as part of a new project component after consultation with the UN, which resulted in serving over 50,000 newly entering refugees.

Shelter and NFI: Generally, the number of shelters that were completed was according to the original plan, though in some projects such as the 2018 Improving Sanitation, Hygiene Environment, and Provide Shelter Assistance for the PSN Refugees from South Sudan in Northern Uganda administered by PWJ, the number of shelters PSN shelters constructed exceeded the original plan by 71%.

Protection and Psychosocial Support: The actual number of case management provided to children is roughly the same as the original.

In terms of evaluating the program’s synergistic effects, the intention was to assess whether there were synergies through collaboration and cooperation with other institutions, not these effects beyond the framework of individual projects. However, there was only one example of synergy that was borne out of JPF funded projects that were collocated In Juba, South Sudan has been mentioned by the JPF during the KII, though there are not enough examples to be included in this program evaluation. The Team therefore focused their effort on finding either case for further analysis.

The other example of synergies produced through coordination was the WASH project in the White Nile state of Sudan that included coordinating with key actors to provide complimentary assistance to potentially produce synergistic effects on the project, though it has not been measured as it is still ongoing. The few times synergy and their effects were mentioned in the NGO final reports, it meant synergy between the infrastructure and the training components working together to promote better

26

utilization of the facility (school). Therefore, the impacts, or synergistic effects produced by either collaboration or through NGO projects were too few and not quantifiable to report.

Taking all of the above into consideration, the program has high effectiveness.

4-4-4. Factors that Hinder/Promote Project Effectiveness Overall, the projects currently implemented reported that factors such as having a long-standing relationship with the communities and maintaining close communication with its community leaders, local authorities, and beneficiaries have been factors promoting project effectiveness. In complex humanitarian crises, it is imperative to build trust between the NGO and the host and refugee communities. These, along with capable local NGO staff, ensure smoother operations in the field despite many setbacks that are unforeseen and uncontrollable.

On the other hand, the fear of the spread of COVID 19 was the most mentioned and serious factor that suspended activities in mid-March 2020 for projects that were slated to end later in 2020. Otherwise, heavy rain, occasional violence in the community, and approvals from the local government for the project to commencement or for international staff to travel to the project site were mentioned as main factors that hindered project implementation, both by the NGOs and the UNHCR. The MOFA travel advisories that the Japanese staff have to abide by are much more stringent than other countries, making it impossible to carry out monitoring missions that were mentioned by the JPF officer.

The above reasons are all justifiable and are external to the project team. It is important that each project team resolves the issues in a timely manner, which the records had indicated. Additionally, the two KIQ with the UNHCR officers revealed that the JPF funded NGOs’ accountability and risk management framework(s) and practices were very appropriate (100% of the respondents) and that the NGO’s project implementation process were very timely (100% of the respondents).

However, the multi-year project scheme has not been recognized by the NGOs as truly multi-year because the funding nor the application process is not guaranteed for three years. As a result, their programming may be myopic and could hinder program effectiveness.

4-5. Impact 4-5-1. Impact of Program Outcomes

The JPF defines impact as “A wide range of effects for individual beneficiaries (such as women, elderly, community and organizations), social, economic, technical and environmental, both intended and unintended at the macro (sector) level and micro (household level) included.”

This definition is in line with that of OECD. The OECD also adds that impact “addresses the ultimate significance and potentially transformative effects of the intervention. It seeks to identify social, environmental, and economic effects of the intervention that are longer-term or broader in scope than those already captured under the effectiveness criterion.”

It is widely recognized that the impact evaluation of humanitarian work is particularly challenging. The settings in which humanitarian aid operates are quite different from that of development work under peace and stability. Humanitarian assistance cases typically have no baseline data, being pressed to respond to life-saving needs against imminent threats and risks.

Thus, one definition suggested by Oxfam, ‘significant or lasting changes in people's lives, brought about by a given action or series of action (emphasis added),”21 maybe more pertinent for evaluation of

21 Roche, C. (1999) Impact Assessment for Development Agencies: Learning to Value Change. Oxford: Oxfam, Novib. p.21.

27

humanitarian work. This is based on the recognition that “in humanitarian response, saving someone’s life is significant, even if the effect is not lasting.22”

From this perspective, it seems that the definition of impact with significance—in terms of quantitative volume or qualitative value—can be linked to JPF’s first purpose of the program, the number 1 below. The impact of lasting change/effects can be liked with the second purpose, number 2 below.

1. Provide humanitarian assistance for refugees and those living in deteriorated living conditions; and

2. Strengthen the resilience of the refugees in preparation for repatriation and reintegration.

Given the sector and nature of the projects under the JPF program, the impact corresponding to the first purpose, humanitarian response with significance would be found in the responses with a high coverage or innovation such as latrine or shelter construction. The second type of impact would be found in lasting changes in skills, attitudes, and behavior of individuals or in its social infrastructure/systems through the capacity building component of the projects.

It is also challenging to evaluate mid- to long-term impacts of humanitarian assistance. Because projects tend to be short-term, and it is unclear whether the effects are long-lasting. Then, what can be done is to ‘expect’ lasting impacts without hard evidence. In order to claim the high possibility of such impacts to be realized, ideally, NGOs can provide some data or facts that can make their claim plausible.

For this evaluation, the description of impacts and secondary effects in the final reports of the NGOs were extracted and grouped by sector.23 Each of them can be grouped into four types of impacts: (1) horizontal impact (e.g., influence on non-beneficiaries or other donors), (2) identified effects beyond intended outcomes, (3) changes in awareness and/or behavior, (4) significance in quantity (e.g., beneficiary coverage). The distribution of these four types differs somewhat from sector to sector, probably because of the nature of the activities in each sector.

Education: Among the impacts and secondary effects reported in all project reports, those related to education were the highest in number. Both impacts and secondary effects are mostly about changes in attitude and behavior of beneficiaries and stakeholders (e.g. understanding about importance of education), and identified effects beyond the planned outcomes or spillover effects to non-beneficiaries (e.g. eased tension between the host and refugee communities after provision of secondary education to both). When compared with those recorded as secondary effects, those recorded as impact are mostly about awareness and behavior changes. The majority of the secondary effects are the effects beyond the planned outcomes with a relatively low significance level (e.g. temporal employment for school construction).

Food: The impacts and secondary effects related to food are unplanned effects on school meals, food consumption, and income from sales of vegetables resulting from vegetable production activities of the peacebuilding projects.

Peacebuilding: The impacts related to peacebuilding are about changes in awareness or attitudes, reduction of fighting in the camp (unplanned effects), and positive influence on neighboring communities (horizontal impact).

WASH: It is noted that those related to water, sanitation, and hygiene have many cases of high coverage of well or latrine construction as well as changes in awareness and attitudes. Given that there are high needs for family latrines in settlements, it is natural for this type of impact to be recorded. Other impacts include improved access to education, acquired income generation skills (skills for maintenance and repair of wells/latrines), and reduced number of GBV cases. Similar impacts are listed as secondary

22 C. Hofmann et al. (2004) Measuring the impact of humanitarian aid: A review of current practice. p.7. 23 There were gross inconsistencies in the reporting by the NGOs regarding impact and secondary effect. Some NGOs reported the impacts as secondary effects and other vice versa.

28

effects as well. In fact, “significance” can be context-dependent (e.g., the significance of the reduction of GBV cases may need to be set in the context of South Sudanese society where SGBV is rampant) and the line between impacts and secondary effects are not always clear-cut.

Shelter and NFI: Similar to latrine construction, the shelter, and NFIs projects tended to measure impact through coverage or quantitative outputs. Some unplanned outcomes are reported as well, rather as secondary effects but not as an impact (e.g. employment opportunities). It is notable that a shelter design introduced by PWJ in Kenya was adopted by other donors, generating a horizontal impact. The KIQ with the UNHCR officer in Kenya revealed that “strengthening PWJ’s reporting, as they do a lot of good work both in the refugee setting and for the host community which is not well documented” as an activity that can improve their impact to achieving UNHCR’s response plan.

Protection and Psychosocial Support: The impacts related to protection and psychosocial support are improvement in the family environment and better attitudes towards learning.

Five cases of negative impacts were reported. Among them, two were about the dissatisfaction of refugees who did not receive relief goods when the support was focused on more vulnerable refugees or new refugees in the camp. Two cases are about aid dependency; in one case, parents made incorrect claims or stopped caring for children so that they could receive support from the NGO. The 4th case mentioned the possible attraction for IDPs to stay in the camps for fear of insecurity at home. In the 5th case, clean water was used to clean up latrines at home in spite of the general lack of potable water in the area. These issues are inherent to humanitarian assistance, and NGOs must continue to work through these issues with special care.

Although some negative impacts are found, overall, there are quite many cases of awareness and behavior changes that could lead to strengthened resilience of beneficiaries, and it can be said that the program generated impacts that have contributed to the achievement of the program's purpose.

The KIQ with the UNHCR Uganda officer also revealed that the projects had contributed significantly to the collective results of the humanitarian assistance of the South Sudan Refugee/IDP. Additionally, the officer mentioned that the JPF program contributed significantly to the South Sudan refugee/IDP humanitarian assistance.

Nonetheless, this study was not able to quantify and gauge the extent of the contribution of such impacts. The inconsistency in reporting the impact and secondary effects, together with the issue of causal linkages, could be dealt with strong leadership and facilitation by JPF.

Therefore, the program has had an inconclusive impact.

4-5-2. Impact with Regards to CHS/Sphere Standards

In regards to the KIQs administered to the NGOs currently implementing projects, 100% of the NGOs answered that their project resulted in conflict prevention, in relation to the “do maximum good” principle.

However, there was more hesitance amongst the NGOs on how the project had resulted in resolving the root cause of violence in South Sudan, especially when the refugees were exiled in neighboring countries. Most NGOs referred to their projects that shielded children from entering gangs, promote peaceful coexistence with the host community through working together, equal employment opportunities, etc. as ways to resolve the root cause of intercommunal violence.

In regards to the question of whether there were any instances in which there was harm caused, in relation to the “do no harm” principle, there was one instance in which a teacher that was trained in an education project had used corporal punishment. The NGOs dismissed the teacher and ensured that there were to be only positive disciplining in the classrooms. Many of the NGOs used feedback mechanisms (CHS

One NGOS said, ‘We try to mitigate the risk of possible conflicts between refugees and host communities and/or even among refugees by providing equal opportunities in work (temporary labor in construction sites), accessibility of facilities provided and benefit borne from the project. We are always aware of Ugandan’s refugee policy of “30-70 Principle”, which requests that 30% of benefit should go to host communities

29

Commitment 524) to receive the grievances so that they would be addressed in a timely and appropriate manner.

4-6. Connectedness and Sustainability Connectedness is defined by the JPF as “how connected the short-term humanitarian assistance is to long-term activity.” It asks if the emergency assistance project was implemented with mid- and long-term issues in mind. Similarly, sustainability is defined as “whether the benefits and effects are sustainable after the end of assistance, how probable it is to keep the long-term benefit.” It also pertains to the resilience to risks of losing the project benefits as time passes.

Looking at the measures taken by the NGOs to ensure connectedness and sustainability, their major targets are the beneficiary and host communities, the governments and its agencies, and to a lesser extent, other aid organizations. Engagement of these actors in project activities is expected to increase ownership and capacity of beneficiaries and the governments, which can lead to sustaining the benefits brought by the projects. Relations with these actors and their roles are summarized below in Table 6.

Table 6. Relationship between the Actors and their Roles in Achieving Connectedness and Sustainability

Actor Assistance Roles

Refugees (including community leaders)

Capacity building, training Involvement in activities as members of PTSA, student clubs, advocacy committees, sanitation committees

Involvement as field workers of NGOs

Involvement in block making, construction, etc.

Operation and maintenance of infrastructure (e.g., well management committees)

Host community (including community leaders)

Capacity building, training (e.g., integration of refugees)

Government, government institutions, and administrative offices (including schools and health centers)

Strengthening organizational structure,

Leadership training

Coordination with NGOs for the adoption of government standards (e.g., school building design, school curriculum)

Taking over management of undertakings such as schools and training/workshops

Coordination, collaboration and information sharing with stakeholders

International organizations and aid organizations

Cooperation, collaboration, information sharing

Taking over of operation

Provision of funds (e.g., UNHCR)

Child protection information management system

It may be worth noting that community members are often not homogeneous, and some projects intentionally tried to involve different segments of the community into the project activities (e.g., a Plan International Japan’s education project in Arua District of Uganda).

24 Complaints are welcomed and addressed.

30

All projects attempted to involve different actors, as shown in the above table, in order to ensure sustainability and connectedness. Thus, it can be said that the projects under the JPF program appropriately plan and implement activities aiming at improving their sustainability and connectedness.

However, it is difficult to judge the level of sustainability and connectedness of each project. It may be partly because that activities were implemented on a one-year-cycle. Although many NGOs in their KIQs mentioned that multi-year programs make it easier to monitor and evaluate the sustainability of project effects, NGOs were not guaranteed that there would be funds in the following year. Moreover, three years may not be long enough to ensure the lasting effects of capacity building and consequent behavioral changes; it was not possible to be sure at the end of the project if the effects would be sustained for years.

Thus, it is not surprising that few NGOs reported that the effects brought by the project ‘are expected to continue.’ In such a situation, at a minimum, it would be advisable to take measures to increase the possibility of sustained effects and benefits and report some results which can make those statements credible. Several reports present such results, in the case of community members, who started taking the initiative for fundraising and cleaning the area surrounding the school that was built, for example.

About three-fourths of the NGOs with on-going projects also emphasized capacity building and engagement of beneficiaries and stakeholders, including community members, social workers, and government institutions, to ensure sustainability. Of that, roughly half of them see the continued engagement of beneficiaries and stakeholders and monitoring and follow-up as a strategy to ensure sustainability. Other strategies include the utilization and strengthening of an existing system, officialized cooperation such as MOUs, and handing over to other organizations which could fund the activities.

Connectedness is defined as a criterion that measures “[t]he extent to which activities of a short-term emergency nature are carried out in a context that takes longer-term and interconnected problems into account. [That] [r]eplaces the sustainability criterion used in development evaluations.25” However, the majority of NGOs seem to understand it as cooperation and collaboration with other projects, aid organizations, local government institutions and beneficiaries that may lead to sustainability of project outcomes. Some NGOs even used permanent buildings, the long service life of latrine facility, or a good relationship with communities where the NGO has been working for successive years as indication of connectedness.

Only a few NGOs described connectedness and sustainability. Its contents are, as mentioned above, mostly about continuation of services and outcomes through cooperation with other organizations, capacity building and organizational building (e.g., PTSA, hygiene committee) through community/beneficiary involvement. There was no report on environmental sustainability. For financial sustainability, there were three entries; one NGO received subsequent funds from JPF, one NGO used their own funds for follow-up activities, and one carried out fund-raising to continue their activities.

Therefore, there seems to be insufficient understanding about sustainability and connectedness among the member NGOs. It appears that the member NGOs with long experience have a better grasp of the concepts of sustainability and connectedness. This shows the need for JPF to facilitate a better understanding among the member NGOs through planning discussions and monitoring.

4-7. Observance of CHS and Sphere Standards The observance of the Core Humanitarian Standard26 and the Sphere standard was challenging to verify, due to the fact that the reporting format was not consistent throughout the years: for example, there was

25 ALNAP (2016) Evaluation of Humanitarian Action Guide. ALNAP Guide. London: ALNAP/ODI. p.114. 26 The CHS is comprised of nine commitments; 1. Communities and people affected by crisis receive assistance appropriate and relevant to their needs.2. Communities and people affected by crisis have access to the humanitarian assistance they need at the right time. 3. Communities and people affected by crisis are not

31

no place to discuss the NGO’s adherence to the CHS during the 2016 program year. Additionally, the Sphere Project integrated the CHS’s nine commitments into its Handbook, substituting it for the Sphere Handbook’s Core Standards.

In theory, both the CHS and Sphere Core Standards adhere to the principles of “humanity, impartiality, independence, and neutrality.” However, differences exist in how the rights and principles framework is described. The 2011 version of the Sphere’s six Core Standards describe “processes that are essential to achieving all the minimum Sphere standards, which are focused on meeting the urgent survival needs of people affected by disaster or conflict.” 27 Furthermore, “its guidance notes bring specific points to the attention of the practitioner applying the standards,” 28 in cases of the four technical chapters consisting of Water Supply, Sanitation and Hygiene Promotion (WASH), Food Security and Nutrition, Shelter and Settlement, and Health. On the other hand, the CHS describes “what organizations and individuals involved in humanitarian response commit to doing in order to improve the quality and effectiveness of the assistance they provide.” 29 Given that their focuses are slightly different, it was imperative that member NGOs understand the difference, and then provide sufficient explanation as to how their projects would observe both of the CHS and the Sphere minimum standards using their latest versions.30 Given the above, the description on CHS and Sphere minimum standards in the final reports of the NGOs were extracted and where possible, grouped by sector.

The member NGOs reported that they observed the CHS and Sphere Standards in their work, from the project design phase to its implementation phase. In that regard, each project has carried out the work to identify the vulnerable population, engage its stakeholders from the planning phase, recruit competent staff that can work in these environments where the humanitarian principles are usually not met. In doing so, the NGOs have intended to meet the nine CHS commitments, with the most emphasis on commitments 4 and 5, then 1, 2, and 3 followed by 6, 7, and 9. The least mentioned CHS commitment was 8. JPF could encourage its members to elaborate on commitment 8 for the future.

In general, the child protection and psychosocial support projects reported more in detail in terms of its adherence to the Child Protection Minimum Standard (CPMS), while the education projects tended to report that they met certain Inter-Agency Network for Education in Emergencies (INEE) standards. According to the KII with JPF, there was an awareness that meeting the minimum standards of the technical chapters of the Sphere Handbook was quite difficult to achieve. However, it is important to mention that all of the projects were striving to meet the Sphere minimum conditions in one way or another, though they may have been referring to completely different sets and versions of the Handbook. Using these technical chapters of the Sphere Standards as a guide is imperative to carry out any humanitarian assistance projects, and it is important that the NGOs are given clearer guidelines as to which versions of the handbooks and standards to utilize for reporting. A detailed discussion with the member NGOs will be useful to clarify how to utilize the latest handbook to better plan and report on their activities.

In sum, though it was evident that the NGOs strived to meet the minimum standards set forth technical chapters of the sphere standard and managed the project design, planning, and implementation keeping

negatively affected and are more prepared, resilient and less at-risk as a result of humanitarian action; 4. Communities and people affected by crisis know their rights and entitlements, have access to information and participate in decisions that affect them.5. Communities and people affected by crisis have access to safe and responsive mechanisms to handle complaints; 6. Communities and people affected by crisis receive coordinated, complementary assistance.7. Communities and people affected by crisis can expect delivery of improved assistance as organisations learn from experience and reflection. 8. Communities and people affected by crisis receive the assistance they require from competent and well-managed staff and volunteers; 9. Communities and people affected by crisis can expect that the organisations assisting them are managing resources effectively, efficiently and ethically. 27 The Sphere Project: The Core Humanitarian Standard and the Sphere Core Standards Analysis and Comparison Interim Guidance, March 2015 (version 2). 28 Ibid. 29 Ibid. 30 However, the format of the 2016 final reports did not include observance of the CHS commitments, so that was eliminated from the analysis for that year.

32

the CHS commitments in mind, it could be improved with a stronger leadership by the JPF to communicate how to report on it. Therefore, the program has a low observance of the CHS and Sphere standards because the program itself does not require that NGOs observe the standards, but rather recognize them and try to meet them if possible. In that regard, it is important to note that using the metrics set forth by the CHS and Sphere Standards are appropriate as the standards to abide by for the program evaluation.

33

5. Lessons Learned Overall, the South Sudan refugee projects have been implemented with high flexibility to respond to the changing needs of the refugees and host communities and shifting security concerns, while achieving the projects’ objectives. The JPF has been very responsive to give the green light in a timely fashion to change the implementation plan.

For this evaluation, the description of lessons learned in the final reports of the NGOs was extracted and grouped by sector (education, food, peacebuilding, water and sanitation, shelter and NFIs, protection/psychological support). The following is not an exhaustive list of the lessons learned, but ones that tended to be mentioned repeatedly, and sometimes across the sectors.

Education: It is essential to strengthening the capacity of the regional Department of Education as well as the school district and teachers while constructing the educational facilities and introducing the new curriculum. The involvement and understanding of parents and communities to encourage children to attend school will enhance the utilization of the facilities. Establishing a monitoring/ follow-up system will bring out the best benefit and ensure the sustainability of the project. The workshops should be limited to a smaller number of participants to encourage their full engagement. Lastly, it is imperative to plan the construction and training activities, including workshops around the school calendar, such as events and exams.

Food/ Agriculture: Agricultural/ food processing training should be tailored to fit the culinary culture of that region.

Peacebuilding: Creating guidelines and providing training will help youth leaders to conduct awareness activities. Careful attention must be paid to the problems that are hard to see from outside, such as domestic violence. The training schedule should not conflict with the youth leaders’ other engagements, such as school and work.

WASH: Excavation and construction of latrines should be planned in accordance with the soil properties and the climate. Construction also needs to be completed during the dry season; otherwise, it will be difficult to secure access to the site and to continue the construction in the rainy season. The participation of the beneficiaries from other target areas and people who are NOT direct beneficiaries of the activity is also a major factor contributing to successful project implementation. For example, the men’s participation in women’s hygiene activities will develop their understanding towards women’s problems, and that will contribute to reach the higher achievement of the activity as the men are usually the head of the house. Therefore, these efforts should be incorporated into the various phase of the project.

There is a lack of feeling of ownership of water supply and sanitation services among its beneficiaries. They are reluctant to contribute to the repair and maintenance aspects of the services, hence continued dialogue and workshops will be essential to enhance the community’s ownership and sustainability of the project.

Shelter and NFI: Various verification methods should be used to identify new refugees and IDPs for the distribution of goods. It was useful to prepare different sizes of NFI packages to accommodate the actual size of the households, instead of the standardized one size fits all method.

The most up to date needs of beneficiaries should always be assessed to deliver the most appropriate aid while receiving information and data from coordinating organizations such as the UNHCR. Creating a support system within the community is essential to ensure sustainability, as it is hard for some households to maintain the shelter.

Protection and Psychosocial Support: It is critical to secure skilled staff and a few back-up candidates that have the exact competency to carry out the specific activities. Also, the capacity building of the caseworker is essential to maintain the quality of case management. When establishing the committee,

34

group, task team, etc., for the project, gender balance needs to be taken into consideration to reflect the voice of vulnerable groups (e.g., women, girls, etc.).

Health: The continuous coordination within the health cluster and with the local government enabled them to receive the necessary medical supply that leads to boost the quality of the project. Capacity-building activities such as training enhance the effectiveness of the infrastructure. This also contributes to the longevity of the infrastructure, thus, increases the impact of the project.

The lessons learned mentioned above sometimes appear across the sector, for example, “plan the construction schedule in accordance with local climate, especially dry/ rainy season, for the efficiency” and “involvement of community for the construction for ownership and sustainability” are always mentioned in the project with construction component.

Moreover, the following three lessons learned were mentioned in almost all sectors.

1. Including the host community as beneficiaries are critical to mitigate unnecessary conflict between refugees and host community members.

2. Coordination at all of the levels that the stakeholders are involved in is key for the smooth and successful implementation of the project. This includes the coordination with high-level players such as the United Nations and host countries’ governments, down to the local actors such as local governments, international and national NGOs, local communities, associations, and other relevant authorities.

3. Design the project and plan the schedule with some flexibility to respond to unexpected events, such as conflict, weather, changes in social climate, etc.

35

6. Recommendations 6-1. Specific Recommendations In light of Chapter 5, the team makes the following recommendations for JPF and its member NGOs to improve their project and program implementations. Moreover, it is the hope of this evaluation to lay the foundation for modifying cooperation framework for not only the JPF but with the NGOs and host countries for the next program cycle.

Structure and Design of the Program

1. JPF should have a standardized system of tallying the number of beneficiaries in order to make it clearer for the NGOs to report on them. In the final reports, for example, if the beneficiary of different activities in the same component is double-counted or not, if the indirect beneficiary is mentioned, and the number of most vulnerable people among the beneficiary is mentioned, are varied by the NGO. Therefore, setting a common standard on how and what to count and report as a beneficiary will improve the accuracy of the data and enable better data monitoring for JPF.

2. The overall goal for the program is necessary for the detailed evaluation of the efficiency; thus, we suggest that JPF consider setting an overarching goal for the purpose of making causal linkage or results chain of all of the individual projects related to South Sudanese refugees and its host communities and have realistic expectations that future evaluations can better evaluate.

Capacity Building of and Accountability Towards Member NGOs

3. The merits of JPF’s multiple-year should be clearly stated and shared with member NGOs. Due to the funding volatility, the advantages may not be easily recognized by the NGOs. Some member NGOs do not realize that their funding could be based on a multi-year program, which can lead to rather myopic programming.

4. It is important that all of JPF and the implementing partners have a common understanding of the impact and intended/unintended outputs, secondary effects, and causal linkages to the impact. There seemed to be an inconsistency in the types of outcomes/impacts that were listed. Therefore, they should be standardized and made clear to all. The same could be said about the synergistic effects that were not clearly defined nor measured on which the NGOs could report.

5. It would be useful if the NGOs, along with JPF, can discuss developing measurable, quantitative impacts that lead to the achievement of the overall program purpose. One way to do that would be to clarify the causal linkages to the project purpose from outcomes at the onset of the project, and evaluate if the benchmark of each pathway is achieved, and to what extent. Those impacts not part of the pathways can be categorized as unintended impacts.

6. It would also be useful to discuss connectedness and sustainability to clarify what is expected to be reported. If details of sustainability (e.g. financial and so forth) are to be reported, the reporting format should have entries for that.

7. Capacity building training and some support could be provided to the relatively smaller/ newer NGOs. As JPF uses the CNs for the allocation of funds and selection of the projects, some NGOs expressed their frustrations in which their organizational capacity was weak; they were not obtaining enough funding. As an umbrella organization, the JPF should also support its members through continuous capacity building that they already have in the “NGO Capacity Building Program.”

8. In terms of the observance of the humanitarian standards, commitment 8 of the CHS was not reported on by any of the member NGOs. There could be improvements made to encourage NGOs to elaborate in relation to commitment 8 to ensure the quality of the project. Also, in many of the NGO’s reports, there seemed to be a lack of discussion on specific minimum standards or commitment to which they intended to observe. To mitigate this issue, there could

36

have been a checklist of the most important aspects of the nine commitments that the JPF could have selected for the member NGOs to which they could adhere.

Funding

9. It is important for JPF to secure more flexible funding sources. As JPF organizes the system of tripartite cooperation among NGOs, the business community, and GOJ to conduct humanitarian aid, more efforts to secure a variety of funds, especially from the business community is recommended. Diversification of funding sources will enhance the organizational strength, and it could potentially allow NGOs to make more flexible decisions regarding the start date of the project, etc., to bring the best outcome.

6-2. Summary of the Program Evaluation As this was a preliminary attempt to carry out JPF’s program-wide evaluation, the team encountered a few setbacks that were inherent to the humanitarian program evaluation.

First, it was important to use the evaluation criteria definition of the OECD/DAC, ALNAP, and compare them to the JPF definition at the beginning to identify which criterion was applicable, and in what way (its main use of the criterion—whether it be with mainly institutional focus or with single-sector/single-agency evaluations), and which was not, given the purpose of the evaluation. Once the criteria were made clear, the evaluation grid was drafted.

Secondly, in terms of the KIQs and KIIs, it was very important to obtain responses from all of the stakeholders. As this was not possible due to COVID-19 outbreak and time limitations, we advise that efforts be made to increase the response rates for the KIQs and KIIs to be able to increase the weighing of the score.

Only in its 2019-2020 South Sudan Refugee Emergency Support Response Plan does JPF include aims to generate synergistic effects; they are found in Kenya and in Sudan through collaborating with “JICA (in the case of Kenya), the UN, Japanese companies and other NGOs” as well as “implementing the JPF projects in the same area by placing NGOs, companies, and governments.” Additionally, the “Implementation Guideline for Concept Note Selection Methodology for JPF Projects Funded by the 2019 Supplemental Budget” has no selection criterion that encourages synergy by the NGO themselves. If not in the CN selection methodology, it would be necessary for the JPF to come up with a CN and final report format that encourages NGOs to describe possible synergies, reports on them, and rewards synergistic effects when materialized. It would also be important for the JPF to come up with specific metrics to measure these effects.

Here are criterion-specific areas where there could be improvements made in the program evaluation process.

Relevance/Appropriateness: According to ALNAP, in relation to the logframe results chain, relevance can be used to evaluate the wider elements of the intervention, such as the overall goal or outcome, while appropriateness can be used to evaluate inputs and activities. However, the JPF does not set an overall goal for the program, as their strategy is to give each NGO the autonomy to design their projects that makes the best use of their organizational capacity and core competency in a certain area where they have a long history of providing assistance and credibility with the local authorities. While understanding the importance to balance the member NGOs autonomous undertaking, setting common overall goals or outcomes will help evaluate the program.

Connectedness/Sustainability: determining the time horizon for sustainability is one of the most difficult but important aspects for NGO s to plan and implement activities, as well as evaluators who collect data on them. Though this program evaluation did not include how long these behavioral changes or infrastructure to last, it would be important to set standards for each sector.

Impact: Some key aspects of the program evaluation required on the ground information gathering, namely in impact, how the school infrastructure is currently being used, the attendance rates of both the students and teachers, etc. Therefore, it may be important to deploy field evaluators to visit several

37

selected sites for projects already concluded. The evaluation will have a set timeframe and a plausible assumption that the intervention contributed to the impact achieved.

In conclusion, it is advised that JPF and the member NGOs should further strengthen its cooperation to ensure a common understanding, define clear objectives and directions to address specific needs and accelerate progress towards the programme goal. The NGOs should take advantage of the JPF's position and competencies with key stakeholders, including national governments, communities, UN and other NGOs, to advocate and promote the humanitarian goal. It is also necessary to ensure that aspects such as synergy, coherence, and harmonization of programs and project implementations are in line with the JPF response plan, donors’ policies, national priorities and UNHCR and clusters’ strategies.

38

Annex 1: Evaluation Grid Evaluation Criteria Evaluation Question Sources/ Methods for Data

Collection and its scoring method Relevance/ Appropriateness Relevance to the Japanese Government’s Strategy

Is the program aligned with the priority areas of the MOFA's Humanitarian Aid Policy?

JPF Response Plans (30%), questionnaire and interviews with JPF staff (70%)

Appropriateness of the Program

Does the humanitarian assistance match the needs of the region, and does that assistance increases independence and the NGO’s credibility?

Host Countries’ Refugee Response Plans (20%), and Regional Refugee Response Plans (20%), JPF Response Plans (20%), Questionnaire and interviews with JPF staff (20%) questionnaire and interviews with UN staff (20%)

Relevance to the needs of the target group/area and the policies of the international community

To what extent were response and activities in line with the identified needs of the populations, priorities, and capacities, and its program designed based on a good quality context analysis?

Host Countries’ Refugee Response Plans (25%), and Regional Refugee Response Plans (25%), JPF Response Plans (25%), Questionnaire and interviews with JPF staff (25%)

Efficiency

Progress of the projects/ program

Have the projects been properly implemented to achieve the program goals?

Questionnaire and interviews with JPF staff (100%)

Has the program been agile and responsive to changes in the field during implementation?

Questionnaire and interviews with NGO staff (100%)

Relationship between inputs and activities

What internal and external factors contributed to issues in the implementation of the project?

Questionnaire and interviews with NGO staff (100%)

What are the best practices and the lessons learned from that?

Questionnaire and interviews with JPF staff (20%) and NGO staff (40%), Project final report (40%)

Fund allocation and disbursement

Is the fund from JPF released in a timely manner to achieve the expected goal?

Questionnaire and interviews with NGO staff (100%)

Are the funds from the Japanese government to JPF allocated and disbursed in a timely manner?

Questionnaire and interviews with JPF staff (100%)

Effectiveness

Program Achievement

What are the main outcomes (including positive/ negative, and intended/ unintended outcomes) for affected populations, by sub-groups (such as by country, refugee/host populations, gender, ethnicity)?

Project Report (80%)/ questionnaire and with NGO staff (20%)

What are the factors that contribute to or inhibit the effectiveness of the program?

Project Report (80%)/ questionnaire and interviews with JPF Staff (20%)

What was the level of synergy and multiplying effect between the activities in the program with activities of other projects? Project Report (50%), Questionnaire

and interviews with JPF staff (50%) Have the multi-year programs brought any positive outcomes to the overall program achievement?

Impact Examine the positive changes brought about by the programs.

Project final report (60%), questionnaire with NGO staff (40%)

39

Positive/ negative and impact brought by the program

Examine the negative changes brought about by the programs.

Impact against CHS/Sphere Standard

Has your program resulted in conflict prevention (against the “do maximum good” principle)?

Questionnaire and interviews with JPF staff (20%) and NGO staff (80%)

How has your program resulted in resolving the root cause of violence? Are there instances in which there was potential harm caused (against the “do no harm” principle)? How do you mitigate and prevent this from occurring?

Coordination

Coordinating with other actors/ implementation partners

Do partnership agreements include clear definitions of the roles, responsibilities, and commitments of each partner, including how each partner will contribute to jointly meeting humanitarian principles?

Questionnaire and interviews with JPF staff (25%), UN (25%) staff and NGOs (50%) Is there an information-sharing system

with NGOs that JPF can obtain the latest update on each project site?

Connectedness and Sustainability

Institutional Aspect

Has organizational capacity been built within the communities to sustain mechanism developed by projects under the program?

Questionnaire and interviews with JPF (20%) staff and NGO staff (80%)

External Factors that Affect Sustainability

What are the external factors that may influence (both positively and negatively) the sustainability of the projects?

Questionnaire and interviews with JPF (20%) staff and NGO staff (80%)

Observance of CHS/Sphere Standards

CHS/Sphere Standards

How do you ensure that each project complies with the CHS?

Questionnaire and interviews with JPF staff (100%)

What are some of the issues that make it difficult to meet the CHS/Sphere standards?

Questionnaire and interviews with JPF staff (50%) and NGO staff (50%)

40

Annex 2: Key Informant Questionnaires (JPF, UN, and NGOs)

Draft Key Informant Questionnaire (KIQ) for JPF

Instructions for the KIQ: The key informant should be someone who has been working on the JPF South Sudan humanitarian assistance program from 2016 onwards. It should be someone who has

extensive knowledge in the proposal, monitoring, and implementation of the projects.

IC Net has been commissioned to carry out a Program Evaluation for the South Sudan Refugee Humanitarian Assistance Program for the years 2016-2020. The questionnaire should not take more than 90 minutes of your time. Also, please provide the main point of contact’s phone number. Please return the completed form by April 10, 2020.

Please feel free to contact us at oleynikov.makiko@icnet.co.jp for any questions.

Name of the main point of contact

Title

How long have you been working for the South Sudan refugee program?

(Month/ Year) ～ (Month/ Year)

Phone number: __________________________________________

Best time to reach you: ____________________________________

Relevance/ Appropriateness

1. Did JPF design the Program to be aligned with the priority areas of the MOFA's Humanitarian Assistance Policy? (Yes/ No) Please explain __________________________________________________________________

2. It seems that coordination with the UN agencies as well as NGOs is not specifically described in the program strategy. Is coordination with other aid agencies still put importance within JPF program? (Yes/ No) Please explain __________________________________________________________________ 2a. If yes, what roles JPF play in coordination of the program project with UN agencies and other NGOs? ______________________________________________________________________________

3. Is the South Sudan program clearly articulated in JPF's global humanitarian assistance plan? (Yes/ No) Please explain __________________________________________________________________

4. There does not seem to be an overall goal for the Program? (Yes/ No) 4a. If yes, what is the goal? Please explain____________________________________________ 4b. If there is a goal, are there indicators that JPF measures to see if the overall goal has been achieved? (Yes/ No)

41

Please explain __________________________________________________________________ 5. Can the role and impact of each project in the Program be explained clearly, and is it well

organized within the overall Program strategy? (Yes/ No) Please explain___________________________________________________________________

6. Are Program strategies and actions to reduce risk and build resilience designed in consultation with, or guided by, affected people and communities? (Well consulted/ somewhat consulted/ Average/ slightly consulted/ not consulted at all)

7. Does the organization have the funding, staffing policies and programmatic flexibility to allow it to adapt to the changing needs? (Yes/ No) Please explain___________________________________________________________________

8. How has JPF specified its location of operations? Please explain __________________________________________________________________ 7a. Specifically, why has Sudan not been included in the programming for the 2018 Humanitarian Crisis Plan even though it was listed as one of the countries for assistance? Please explain __________________________________________________________________

9. The 5th program strategy states, “Utilize the advantages of being able to respond to multiple years and the relationships that have been built up with the experience so far, work on activities that contribute to resilience at the grassroots level.” What advantages/strengths are implied here? Please explain __________________________________________________________________

Efficiency

10. What is the rationale for conducting preliminary studies on certain projects? Please explain___________________________________________________________________

11. Is regular Program monitoring carried out and the monitoring results reviewed by the JPF’s Program Manager? (Yes/ No) Please explain___________________________________________________________________

12. Was the Program response plan (adoption and implementation of each project, securing budget, etc.) implemented as planned? (Yes/ No) Please explain __________________________________________________________________

13. Do the funds have adequate control and oversight mechanisms to address inefficient use of funds and mismanagement of funds (including corruption)? (Yes/ No) Please explain __________________________________________________________________

Effectiveness

14. Is the overall goal something that can be achieved by implementing multiple projects? If not, how can it be improved? (Yes/ No) Please explain___________________________________________________________________

15. How much of the program goal has been achieved? __________%

16. What are the main outcomes (including positive/ negative, and intended/ unintended outcomes) of the program for affected populations, by sub-groups (such as by country, refugee/host populations, gender, ethnicity)?

42

Sub-group_______________ Positive/ Negative outcome________________________________ Sub-group_______________ Positive/ Negative outcome________________________________ Sub-group_______________ Intended/ Unintended outcome_____________________________ Sub-group_______________ Intended/ Unintended outcome_____________________________

17. What are the factors that contribute to or inhibit the effectiveness of the program? ______________________________________________________________________________

18. What was the level of synergy and multiplying effect between the activities in the program with activities of other projects? (Very high/ High/ Average/ Low/ Very low) Please give an example___________________________________________________________

19. Have the multi-year programs brought any positive outcomes to the overall program achievement? (Yes/ No) Please explain __________________________________________________________________

Impact 20. Has the Program resulted in conflict prevention (according to the “do maximum good”

principle)? (Yes/ No) Please explain __________________________________________________________________

21. How has the Program addressed the root cause of conflict? Please explain __________________________________________________________________

22. Are there instances in which there were potential harm caused (against the “do no harm” principle)? How do you mitigate and prevent this from occurring? (Yes/ No) Please explain __________________________________________________________________

Aid Coordination and Coherence

23. Is there a clear commitment in organizational policies and/or strategies to work in collaboration with other actors? (Yes/ No) Please explain __________________________________________________________________

24. Have criteria or conditions for partner selection, collaboration and coordination been established? (Yes/ No) Please explain __________________________________________________________________

25. Do partnership agreements include clear definitions of the roles, responsibilities and commitments of each partner, including how each partner will contribute to jointly meeting humanitarian principles? (Yes/ No)

26. Is there an information sharing system with NGOs from which JPF can obtain the latest update on each project site? (Yes/ No) Please explain __________________________________________________________________

27. Have existing coordination structures been identified and supported? (Yes/ No) Please explain __________________________________________________________________

28. Are gaps and duplication in project coverage identified and addressed? (Yes/ No) Please explain __________________________________________________________________

43

Connectedness and Sustainability

29. How does the program relate with policies/ strategies of the global response as well as with the policies and strategies of the target country? _____________________________________________________________________________

30. Has organizational capacity been built within the communities to sustain mechanism developed by projects under the program? (Yes/ No) _____________________________________________________________________________

31. Has the program brought about behavioral changes or social changes in beneficiaries (e.g. hygienic behavior)? (Yes/ No) Please explain __________________________________________________________________

32. What are the external factors that may influence (both positively and negatively) the sustainability of the projects? Positive influence ______________________________________________________________ Negative influence _____________________________________________________________

33. How does the program ensure that the host community secures the budget to continue each project under the program? _____________________________________________________________________________

34. What is the role of JPF in ensuring that each projects’ plans and implementation of its exit strategy in an appropriate and sustainable manner? _____________________________________________________________________________

35. What is the exit strategy of the Program itself? _____________________________________________________________________________

Observance of CHS/Sphere Standards

36. What are some of the issues that make it difficult to meet the CHS/Sphere standards? _____________________________________________________________________________

37. In those cases, when the Standards are not met, how does the JPF address them? _____________________________________________________________________________

Please return the completed questionnaire by April 10, 2020, to oleynikov.makiko@icnet.co.jp

Thank you for your cooperation!

IC Net Limited Program Evaluation Team

44

Key Informant Questionnaire (KIQ)

for the Office of the United Nations High Commissioner for Refugees

Ethiopia Country Office

Instructions for the KIQ: The key informant should be someone who has knowledge of the Japanese NGO’s work under the Japan Platform (JPF) South Sudan humanitarian assistance program from

2016 onwards.

IC Net has been commissioned to carry out a Program Evaluation for the South Sudan Refugee Humanitarian Assistance Program for the years 2016-2020 of JPF. The questionnaire should not take more than 30 minutes of your time. Please provide the main point of contact’s Skype handle. Please return the completed form by April 27, 2020.

Please feel free to contact us at oleynikov.makiko@icnet.co.jp for any questions.

Name of the main point of contact

Title

How long have you been working in this position? (Month/ Year) ～ (Month/ Year)

Skype name: ________________________________________

Best time to reach you: ________________________

The JPF funded projects are listed here:

Project Name NGO Implementation Period

Project for improving educational and hygiene environment for South Sudanese refugees in Ethiopia

World Vision Japan

2016 July -2017 September

The set-up of Household latrines for South Sudanese refugees in Tierkidi refugee Camp of Western Ethiopia in Gambella region Phase Ⅱ ADRA 2016 October-2016

November The set-up of Household latrines for South Sudanese refugees in Tierkidi refugee Camp of Western Ethiopia in Gambella region Phase 2 ADRA 2017 August - 2017

September Project for improving educational environment for South Sudanese refugees in Ethiopia Phase 2

World Vision Japan

2016December -2017 May

The Hygiene and sanitation project for South Sudanese refugees at Kule refugee camp in Gambella region of Ethiopia. ADRA 2017January -2017May

The Hygiene and sanitation project for South Sudanese refugees at Kule refugee camp in Gambella region of Ethiopia phase 2 ADRA 2018 August －

2019 February Project for improving Secondary education environment in Jewi refugee camp, Gambella, Ethiopia

World Vision Japan

2018 August－ 2019 May

The Hygiene and sanitation project for South Sudanese refugees at Kule refugee camp in Gambella region of Ethiopia

Peace Winds Japan 2017June -2018 May

45

I. Relevance/ Appropriateness 1. How relevant are the objectives of the JPF funded projects implemented by the Japanese NGOs to

the humanitarian needs in the respective countries? Please select from the options. (very relevant, somewhat relevant, average, somewhat not relevant, not relevant at all)

2. To what extent are the JPF funded projects aligned with the RRRP? (very aligned, somewhat aligned, average, somewhat not aligned, not aligned at all).

3. In what regard has the JPF funded projects contributed the most to the humanitarian assistance? Please explain. ___________________________________________________________

4. Do you think JPF funded project’s approach to address the needs on the ground was appropriate? (Yes/No) 4a. Please explain._______________________________________________

II. Effectiveness 5. To what extent has the JPF funded projects implemented by NGOs supported beneficiary

targeting and contributed to improved geographic coverage to ensure that the most vulnerable groups’ needs are addressed? Please select from the options. (very targeted, somewhat targeted, average, somewhat not targeted, not targeted at all)

6. Are the JPF funded NGOs’ accountability and risk management framework(s) and practices appropriate? Please select from the options. (very appropriate, somewhat appropriate, average, somewhat not appropriate, not appropriate at all)

7. How timely is their project implementation process? (Very timely/Somewhat timely/Average/Somewhat not timely/Not timely at all)

III. Coherence/ Aid Coordination 8. Have the Japanese NGOs coordinated well with the UN to avoid gaps and overlap in coverage in

the humanitarian assistance? Please select from the options. (very coordinated, somewhat coordinated, average, somewhat not coordinated, not coordinated at all)

IV. Impact 9. To what extent has the JPF funded projects contributed to the collective results of the

humanitarian assistance of the South Sudan Refugee/IDP? Please select from the options. (contributed a lot, contributed somewhat, average, somewhat not contributed, did not contribute at all)

10. How does the JPF program/ scheme contribute to the South Sudan refugee/IDP humanitarian assistance? Please select from the options. (contributed a lot, contributed somewhat, average, somewhat not contributed, did not contribute at all)

11. What are the gaps that you see between their activities and the UNHCR’s response plan? ________________________________________________________________

12. What activities can improve their impact to achieving UNHCR’s response plan? ________________________________________________________________

If you have any specific suggestions on the evaluation criteria above related to improving the JPF program, please feel free to write below.

46

All of your answers will be kept anonymous. Please return the filled our questionnaire by April 27, 2020, and email to oleynikov.makiko@icnet.co.jp.

Thank you for your cooperation!

IC Net Limited Evaluation Team

47

Key Informant Questionnaire (KIQ)

for NGOs with Projects Currently Being Implemented

Instructions for the KIQ: The key informant should be someone who has been working on the JPF South Sudan humanitarian assistance program from 2016 onwards. It should be someone who has

extensive knowledge in the proposal, monitoring, and implementation of the projects.

IC Net was commissioned to carry out a Program Evaluation for the South Sudan Refugee Humanitarian Assistance Program for the years 2016-2020. The questionnaire should not take more than 45 minutes of your time. The information and opinions you provide us will be kept anonymous. Your name or organization will not be used in any reporting of survey results. In case we need to follow up, please provide the main point of contact’s phone number. Please feel free to add space/lines for your answers.

Please return the completed form by April 20, 2020.

Please feel free to contact us at mims.yukari@icnet.co.jp for any questions.

Name of the main point of contact

Title and Organization

Name of the Project: _____________________________________________

How long have you been working for the project? (Month/ Year) ～ (Month/ Year)

Phone number: ________________________________________

Best time to reach you: ___________________________________

Relevance/ Efficiency

38. Can the achievement of project goals be verified using data? (Yes/No)

39. Are the strategies and actions to reduce risk and build resilience designed in consultation with, or guided by, affected people and communities? (Yes/No) If yes, please explain the consultation process.________________________________________________________________________

Location and Targeting of Beneficiaries

40. How has the organization identified and specified its site of operation?

41. How much coordination is there with the other agencies? ______________________________________________________________________________

4a. What could have been improved?________________________________________________

42. Were you able to reach out to the most vulnerable groups of people in your project? (Yes/No)

5a. If yes, how do you identify and include them in the project? ______________________________________________________________________________

48

5b. Please list the most vulnerable groups that you monitor. ______________________________________________________________________________

5c. What is the percentage of the people who belong to the vulnerable groups in relation to the overall beneficiaries? ____________%

Fund Allocation

43. Have the funds from the JPF been released on time? (Yes/No)

44. Is the disbursement schedule adequate to finance the project? (Yes/No)

7a. If No, please explain the problem your organization faced. ______________________________________________________________________________

45. How does your organization coordinate funding from other donors with the funds from JPF?__________________________________________________________________________

Project Implementation

46. What percent of the project is completed in March 2020? ______ % with projected end date ______, 2020.

47. Is this project on schedule?

Select from the following options: (behind schedule/ on schedule/ ahead of schedule)

48. What are the factors that inhibited the project from achieving the results so far? ______________________________________________________________________________

49. What are the factors that helped the project to achieve the results so far?

______________________________________________________________________________

50. What are some of the unintended outputs (both positive and negative) so far?

13a. Positive: ___________________________________________________________________

13b. Negative: __________________________________________________________________

Aid Coordination and Complementarity

51. Are the services rendered by the project designed to have complementarity with other international aid providers? (Yes/No)

52. How are gaps and duplication in project coverage identified and addressed?

______________________________________________________________________________

49

Connectedness/ Sustainability

53. How do you ensure and measure sustainability in the projects?

______________________________________________________________________________

54. Compared to single-year projects, how is the JPF multiple-year program positively or negatively affecting your projects?

______________________________________________________________________________

55. How do you ensure that the exit strategy for the project is realized?

______________________________________________________________________________

Observance of CHS/Sphere Project Standards

56. How do you monitor whether the project resulted in conflict prevention about the “do maximum good” principle?

______________________________________________________________________________

57. How do you monitor whether your project resulted in addressing/alleviating the root cause of conflict? ______________________________________________________________________________

58. Are there instances in which there was harm caused (against the “do no harm” principle)? (Yes/No)

21a. If yes, please explain.

______________________________________________________________________________

21b. How do you mitigate and prevent harm from occurring?

______________________________________________________________________________

If you have any specific suggestions on the themes above related to improving the JPF program, please feel free to write below.

All of your answers will be kept anonymous. Please return the completed questionnaire by April 20, 2020, to mims.yukari@icnet.co.jp

Thank you for your cooperation!

IC Net Limited Program Evaluation Team

50

Annex 3: UN Responses to the Key Informant Questionnaire I. Relevance/

Appropriateness UNHCR Kenya UNHCR Uganda

1. How relevant are the objectives of the JPF funded projects implemented by the Japanese NGOs to the humanitarian needs in the respective countries? Please select from the options. (very relevant, somewhat relevant, average, somewhat not relevant, not relevant at all)

Very relevant very relevant

2. To what extent are the JPF funded projects aligned with the RRRP? (very aligned, somewhat aligned, average, somewhat not aligned, not aligned at all).

Very much aligned with the RRRP very aligned

3. In what regard has the JPF funded projects contributed the most to the humanitarian assistance? Please explain.

Through Japan platform, PWJ has demonstrated ability in bridging the gaps in the areas of shelter, water and sanitation aimed at improving living environment of refugees and host communities holistically and sustainably in Kalobeyei Integrated settlement. PWJ mainstreamed disability in their programming as well benefiting PSNs living in the Kalobeyei settlement. They championed the CLTS (community led total sanitation) progamme leading to Open Defacation Free (ODF) zones within the Kalobeyei settlement and will work together with NRC to roll out the CLTS to Kakuma camp where over 150,000 refugee reside.

In Adjumani the helped provide support to children at risk.

4. Do you think JPF funded project’s approach to address the needs on the ground was appropriate? (Yes/No) 4a. Please explain.

Yes. JPF funded project was appropriate as it addressed housing condition for refugees, sanitation behaviours and service for both in schools and settlement as well as access to water hence improving sanitation within the settlement and surrounding host community. They are coordinating well on the COVID-19 preventive measures including producing comic books on good hygiene practices for children.

Yes, they provided safe spaces for children and youth throught the ECCD and CFS.

II. Effectiveness UNHCR Kenya UNHCR Uganda 5. To what extent has the JPF funded projects implemented by NGOs supported beneficiary targeting and contributed to improved geographic coverage to ensure that the most vulnerable

very targeted very targeted

51

groups’ needs are addressed? Please select from the options. (very targeted, somewhat targeted, average, somewhat not targeted, not targeted at all)

6. Are the JPF funded NGOs’ accountability and risk management framework(s) and practices appropriate? Please select from the options. (very appropriate, somewhat appropriate, average, somewhat not appropriate, not appropriate at all)

very appropriate very appropriate

7. How timely is their project implementation process? (Very timely/Somewhat timely/Average/Somewhat not timely/Not timely at all)

Somewhat timely Delays in construction of mainly host community shelters due mostly to late approvals from the County Government Authorities, resulting in delays in completion of the construction projects.

Very timely

III. Coherence/ Aid Coordination UNHCR Kenya UNHCR Uganda

8. Have the Japanese NGOs coordinated well with the UN to avoid gaps and overlap in coverage in the humanitarian assistance? Please select from the options. (very coordinated, somewhat coordinated, average, somewhat not coordinated, not coordinated at all)

very coordinated To achieve the operation’s needs, PWJ coordinated very well with other stakeholders through WASH and shelter working groups as well as with the senior management of UNHCR to avoid overlapping of the activities in Kalobeyei settlement, while ensuring efficient use of resources. PWJ is very transparent on their resource allocations and has ensured that their funds are spent on gaps ideintified and not duplication of assistance. PWJ attends all the coordination forums including the protection working groups. They are also very much involved in the Kalobeyei Integrated Socio-Economic Development Plan (KISEDP) and are members of the thematic working group where they also implement and complement n the flagship projects in shelter and WASH.

very coordinated

IV. Impact UNHCR Kenya UNHCR Uganda

52

9. To what extent has the JPF funded projects contributed to the collective results of the humanitarian assistance of the South Sudan Refugee/IDP? Please select from the options. (contributed a lot, contributed somewhat, average, somewhat not contributed, did not contribute at all)

contributed a lot contributed a lot

10. How does the JPF program/ scheme contribute to the South Sudan refugee/IDP humanitarian assistance? Please select from the options. (contributed a lot, contributed somewhat, average, somewhat not contributed, did not contribute at all)

contributed a lot contributed a lot

11. What are the gaps that you see between their activities and the UNHCR’s response plan?

PWJ has demonstrated transparency in the activities implemented by JPF funds and ensured that they are aligned to UNHCR response plan.

The activities in Child Protection in terms of case management were not comprehensive at times they needed to refer cases to UNHCR partner. For instance when it case to alternative care arrangements, this was an issue at the beginning.

12. What activities can improve their impact to achieving UNHCR’s response plan?

Sterngthening their reporting as they do a lot of good work both in the refugee setting and for the host community which is not well documented.

Need to better coordinate with UNHCR partners to avoid overlapping especially in case management. This will help maximaze on the scarce funding and reach more refugees

13. If you have any specific suggestions on the evaluation criteria above related to improving the JPF program, please feel free to write below.