This report was commissioned by Action Against Hunger | ACF International. The comments contained herein reflect the opinions of the Evaluators only.

Mitigating the Impact of the Ebola Virus Disease on the Most Vulnerable Households through an Integrat-ed Food and Nutrition Security Intervention in the District of Moyamba, Sierra Leone

Funded by

IRISH AID By Tristan Dumas

INDEPENDENT FINAL PROJECT EVALUATION MAY 2016

‘Mitigating the Impact of the Ebola Virus Disease on the Most Vulnerable Households through an

Integrated Food and Nutrition Security Intervention in the District of Moyamba, Sierra Leone’

Independent final project evaluation report

Intervention Name Mitigating the Impact of the Ebola Virus Disease on the Most Vulnerable Households through an Integrated Food and Nutrition Security Intervention in the District of Moyamba, Sierra Leone

Contract Number ACF SL 15 01

Partners (if applicable) none Location (country/ies, region/s)

Moyamba district, Sierra Leone

Duration 12 months (+ 45 days of no-cost extension) Starting Date 15th May 2015

Initial Ending Date 14th May 2016

Intervention/ Country Office Language

English

Donor and Contribution/s Irish Aid: € 804,000

Country Office administering the Intervention

ACF Sierra Leone

Responsible Action Against Hunger HQ

ACF France

Evaluation Type Independent Final Evaluation Evaluation Dates 06th April – 25th May 2016

This report was commissioned by Action Against Hunger International. The comments contained herein reflect the opinions of the evaluator only.

Submitted by Tristan Dumas

2

Table of contents List of Acronyms .......................................................................................................................... 3

Executive Summary...................................................................................................................... 4

Background Information .............................................................................................................. 7

Evaluation Background................................................................................................................. 8

Methodology ............................................................................................................................... 9

Evaluation Findings ..................................................................................................................... 11

Overall assessment ................................................................................................................. 11

Analysis per evaluation criteria ................................................................................................ 12

Design - Rating (1: low, 5: high): 3 ........................................................................................ 12

Relevance/Appropriateness - Rating (1: low, 5: high): 4 ......................................................... 14

Coherence - Rating (1: low, 5: high): 4 .................................................................................. 15

Coverage - Rating (1: low, 5: high): 4..................................................................................... 16

Efficiency - Rating (1: low, 5: high): 2 .................................................................................... 17

Effectiveness - Rating (1: low, 5: high): 3 ............................................................................... 19

Sustainability and Likelihood of impact - Rating (1: low, 5: high): 3 ......................................... 21

Conclusions................................................................................................................................ 22

Lessons Learnt and Good Practices ............................................................................................. 23

Recommendations ..................................................................................................................... 24

Annexes..................................................................................................................................... 25

Annex I: Evaluation Criteria Rating Table ...................................................................................... 26

Annex II: Good practice .............................................................................................................. 28

Annex III: list of interviewed persons .......................................................................................... 29

Annex IV: list of document reviewed .......................................................................................... 29

Annex V: evaluation matrix ........................................................................................................ 29

Annex VI: interview guidelines ................................................................................................... 29

Annex VII: primary data analysis................................................................................................. 29

Annex VIII: Terms of reference ................................................................................................... 29

3

List of Acronyms ACF-SL: Action Contre la Faim, Sierra-Leone office ACF-UK: Action Contre la Faim, London office CSI: Coping strategy index DHMT: District Health Management Team ELA: Evaluation, Learning and Accountability (unit based in ACF-UK) EVD: Ebola Virus Disease FBO: Farmer- based organization (formal group) FCS: Food consumption score FGD: focus group discussions FSL: Food security and Livelihoods GAM: Global acute malnutrition IYCF: Infant and young child feeding MAFFS: Ministry of Agriculture, Forestry and Food security NACSA: National Commission for Social Action SSI: Semi structured interview VSLA: Village Savings and Loans Association VPG: Vegetable Production Groups SAM: Severe acute malnutrition WHO: World Health Organization

4

Executive Summary This project is being implemented between May 2015 and May 2016 with the financial support of Irish Aid (804 000 €), with the aim to contain the negative socio-economic impact of the Ebola outbreak on the food and nutrition security status of vulnerable households in Sierra Leo ne. The specific objective of the project is to ensure an economic income and improve the diversity of food intake of the targeted vulnerable households in Moyamba District. The proposed intervention aims first to give these households the means to access sufficient food during the lean season through unconditional cash transfer. Project’s initial target was to support 1,000 households in Moyamba over 3 months, with a monthly cash transfer of 50 USD (which is equivalent to 100% of the cost of a family-month Energy only diet and to 66% of the cost of a family-month Nutritional diet). Then, ACF planned to provide the same beneficiaries with the inputs and knowledge to produce vegetables in order to increase both their nutritious food consumption and their income. These groups were expected to be sensitized on the importance of a diversified and nutritious diet, and trained to apply it into their practices and habits. Finally these groups were expected to be trained on group savings and loans management (Village saving and loan association).

The overall purpose of the evaluation is to assess the overall performance of the project and to determine if the intervention has reached its intended outputs and objectives. In particular, it assesses to what extent (and the reasons why) the project’s outputs have ensured an economic income and improved the diversity of food intake of vulnerable households in Moyamba District. This evaluation focuses on the entire project funded by Irish Aid. It covers all geographical areas and all selected target groups of beneficiaries and examines the implementation of all activities and the degree of achievement of all outputs and objectives. Finally, the evaluation provide s key recommendations towards sustainability.ACF uses the OECD-DAC criteria approach to evaluate its projects. The criteria used are the following: Design, Relevance/Appropriateness, Coherence, Coverage, Efficiency, Effectiveness, Sustainability and Likelihood of Impact.

The methodology for this final evaluation is a systematic approach inspired by the qualitative research methodology. Data were directly collected from groups of beneficiaries, key informants (i.e. village leaders and technical partners), ACF staff members and triangulated with secondary data (i.e. reports provided by ACF).The sampling strategy was designed to cover a maximal diversity of situations regarding the project implementation. After a preliminary discussion with ACF-SL, the chosen criteria for village selection were access to markets and level of damages caused by grasshoppers on vegetable production. Because all beneficiaries had been involved in same project activities, the main driver of beneficiaries’ diversity was gender. Hence, focus groups were conducted separately between men and women. A total of 38 beneficiaries (66% women)were interviewed during 5 FGD.A total of 12 SSI were also conducted with village leaders (5), ACF FSL HoD (1), ACF PM (1), DHMT Moyamba (1), MAFFS Moyamba (1), Fakunya Paramount Chief (1), Lower Banta chiefdom section chief (1), Food Security Working Group coordinator (1).Primary data analysis followed a ‘content analysis’ approach. All ideas expressed by respondents were coded and entered in an analysis table, showing which categories of respondents is the source of information. Close ideas were then merged into categories (which are more conceptual) and then categories were grouped into themes to answer the evaluation questions. This process is transparent, replicable by another evaluator and makes it possible to link findings with categories of respondents. Main limits of this evaluation are (1) quantitative data were not all available during the evaluation. Baseline survey was still at draft stage for CSI indicator; endline survey together with the KAP survey and Impact survey will be conducted after this evaluation, which made it impossible for the evaluator to use these data to triangulate the qualitative information collected during the evaluation. (2) Collected data during the evaluation is not statistically valid, because of the qualitative sampling strategy design. (3) Due to a limitation of time and budget, there was no household interview during this evaluation. Information was collected through FGD only with beneficiaries. (4) FDG were designed to collect gender-sensitive information. The time limitation did not allow organizing FGD based on respondents’ age. There was no data collection with non-beneficiaries neither. (5) Comparison between approaches (Efficiency analysis) was based on discussions with project team and partners. No external documentation was reviewed in addition to this data collection due to time constraint.

This project was very relevant in the context of early recovery from the post-Ebola outbreak. Vulnerable households were in needs of cash to buy food and restart their livelihoods, especially

5

women. The project design included activities to address immediate needs (unconditional cash transfers) and activities to improve food security and nutrition in the longer term (support to vegetable production and local financial services, sensitization to improved nutrition practices), which were greatly appreciated by targeted communities. The exit strategy was to leave formal groups of vegetable producers, linked with markets and able to manage a local savings and loans fund.

Most of the activities were implemented and output targets were achieved. Nutrition sessions were still on-going and VSLA trainings were just starting during the evaluation period, which may be a problem for the sustainability of the results if VSLA’s final shares are not monitored. Outcome indicators are not yet available to see if expected results on food security and nutrition were achieved.

As a result, we can say that:

- Beneficiaries used the cash transfers to buy food, pay school fees and pay laborers in agricultural lands. This was a great help for them even if they would have preferred to receive the money during the lean season;

- Vegetable production was heavily impacted by grasshopper attacks. Two factors must be taken into account on this: (1) the late sowing because of the delay in seed distribution exposed young plants to grasshoppers and (2) it is the first time that villagers experience so many grasshoppers in their environment. As a consequence, vegetable productions vary a lot between groups depending on their exposure to grasshoppers. Most of the groups had a small harvest of okra, cucumber and pepper. Most groups did not get enough production to keep seeds for the next season. However, beneficiaries are very interested in vegetable productions for both consumption and income generation.

- Nutrition sensitization was really appreciated by both beneficiaries and local authorities. The outcomes of this activity should be measured by ACF during the endline and KAP surveys.

- Beneficiaries did not talk about VSLA as they were not yet trained on this during the data collection.

The general quality of the intervention suffers from various implementation delays. Most of the activities were implemented late because of different reasons:

- Cash transfers were discussed for two months1 with IrishAid and other stakeholders (NACSA, Cash transfer working group, etc.) before the project was officially validated. Finally transfers happened in September and October 2015, just after the lean season.

- Seed distribution, IYCF sessions and VSLA trainings were delayed because of internal procurement process. The national context (cf. Ebola-related restrictions) made it difficult for ACF to get supplies on time and hire qualified staff.

The integration of both short-term activities (Cash transfers) with development activities (vegetable production and nutrition, VSLA) has a strong potential for improving vulnerable households’ food and nutrition security. However, this potential couldn’t be fulfilled because of delays in project implementation and an above-average attack of grasshoppers. For future FSL projects, it would be important to review and monitor closely the procurement process and deve lop a low-cost strategy to mitigate grasshopper damages (early planting, local varieties less palatable for grasshoppers, etc.).

The complain mechanism implemented by ACF is a very interesting innovation in Moyamba district. This first experience has shown that beneficiaries are willing to provide feedbacks to implementing agencies; however this system was not always functioning as it should. ACF should now improve this system to make it effective: review the boxes collection system and follow-up, review the monitoring system to ensure that answers are given to communities.

1 ACF proposed to provide 50 USD/month for 3 months during the lean season which is equivalent to 100% of the cost of a family-month Energy only diet, while other operators preferred to provide 30USD/month during 9 months.

6

Generally, the information management could be improved. Data from the field were barely available and used by managers to take timely decisions, monitoring activities were designed for reporting rather than management. Developing adequate databases to store information at household level, presenting their selection criteria and activity results could help to track progress depending on categories of beneficiaries (ex. woman-headed households versus man-headed households, etc.).

Implementing a phase II support would be very relevant to strengthen these first results. The additional support should focus on timely vegetable seed provision (ACF should assess which local varieties less palatable to grasshoppers are available), quality seed production and storage, cooking demonstrations with the whole communities (including better-off households) with the aim to promote diet diversification and stimulate local vegetable market, FBO recognition and monitoring VSLA activity till the final shares (expected in February 2017).

The following lessons learnt have emerged from this evaluation:

- Procurement process monitoring is essential to ensure quality outputs in food security projects

- Grasshoppers must now be identified as a risk for dry season productions in future food security projects in Sierra-Leone

- Seasonal outcome data collection (FCS, CSI) should be planned and implemented in a way that allows data to be compared

The following good practises have emerged from this evaluation:

- Selection process of beneficiaries involving local institutions and authorities, cross -checked by two rounds of field visits was essential to get a quality targeting within the communities

- Implementing a complain mechanism is a relevant idea, which should be improved to ensure a quality information flow between ACF and targeted communities

Consequently, the recommendations are:

#1: Review procurement process and its monitoring for FSL projects [high priority]

#2: Prepare a phase II project to strengthen results’ sustainability [high priority]

#3: Review the complain mechanism to improve the information flow between ACF and communities [medium priority]

#4: Review the M&E plan for future FSL projects [low priority]

7

Background Information Hunger and malnutrition indicators in Sierra Leone showed challenging figures prior to the Ebola crisis (see table 1). In Moyamba district, 75% of the population was food insecure 2 (45% nationally) before the EVD outbreak.

Table 1: Malnutrition indicators at national and Moyamba district levels before EVD outbreak

Surveys GAM rate SAM rate Stunting rate National Nutrition Survey3 (2014)

4.7% (4.3%-5.2%, 95% C.I)

1% (0.9%-1.2%, 95% C.I.)

28.8% (27.5%-30.2%, 95% C.I.)

National Nutrition Survey4for Moyamba district(2014)

4.4% (WHZ) -4.9% (MUAC)

1.5% (WHZ) – 1.9% (MUAC)

34.5%

SMART Moyamba conducted by ACF (2013)

6.7% (5.1%-8.6%, 95% C.I.)

0.6% (0.2%-1.7%, 95% C.I)

38.0% (33.7% - 42.4%, 95% C.I.)

On the 26th of May 2014, Sierra Leone declared the Ebola outbreak in the country. Since then, a total of 14124 cases were confirmed in Sierra Leone (209 in Moyamba district) for 3956 deaths5.

Moyamba district was first hit by the epidemic in July 2014. In consideration of the high number of confirmed cases and their concentration in specific areas, some of its chiefdoms were quarantined following the Presidential decision adopted on September 25th 2014. Further, the closure of the markets, in particular the lumas, the restriction in movements within and outside the district (and the consequent price increase and extension of the time spent on the road by the traders), and the restriction to public gatherings have deeply impacted on the harvest season and on the trading activities of agricultural and non-

agricultural products of the district. Following the decrease of number of cases in 2015, most of these restrictions were ended in August 2015 and in March 2016, Sierra Leone was declared Ebola-free by WHO for the second time6.

This project is being implemented between May 2015 and May 2016 with the financial support of Irish Aid (804 000 €), with the aim to contain the negative socio-economic impact of the Ebola outbreak on the food and nutrition security status of vulnerable households in Sierra Leone. The specific objective of the project is to ensure an economic income and improve the diversity of food intake of the targeted vulnerable households in Moyamba District. The proposed intervention aims first to give these households the means to access sufficient food during the lean season (peak of hunger gaps and most acute prevalence of malnutrition) through unconditional cash transfer. ACF supported 1,000 households in Moyamba over 2 months, with three cash transfers of 50 USD (which 2 The State of food security and nutrition in Sierra Leone, CFSVA-WFP 2011 3 Sierra Leone, National Nutrition Survey 2014, Gvt of Sierra Leone, Irish Aid, UNICEF, Oct 2014 4 Sierra Leone, National Nutrition Survey 2014, Gvt of Sierra Leone, Irish Aid, UNICEF, Oct 2014 5 World Health Organization website, http://apps.who.int/ebola/ebola-situation-reports (10-05-2016) 6The fi rst time was in November 2015; but, a fter that, few EVD cases popped up again

Figure 1: location of Moyamba district and number of EVD cases (source:

http://maps.who.int/SimpleViewer_WHO/?webmap=5d6e620da51c41679dc3fe2b87fd8619 )

8

is equivalent to 100% of the cost of a family-month Energy only diet and to 66% of the cost of a family-month Nutritional diet7). Then, ACF provided the same beneficiaries with the inputs and knowledge to produce vegetables in order to increase both their nutritious food consumption and their income. These groups were sensitized on the importance of a diversified and nutritious diet, and trained to apply into their practices and habits. Finally these groups areexpected to be trained on group savings and loans management (Village saving and loan association), which was on-going by the time of this evaluation.

This project is managed from Moyamba ACF office, with a remote support from ACF coordination in Freetown and ACF Paris HQ.

At the time of this evaluation, a no-cost extension of 45 days was requested by ACF and accepted by Irish Aid, to enable the implementation of last activities (VSLA and nutrition trainings, monitoring activities). Other activities (cash transfers, vegetable production trainings and input distributions) have been implemented earlier.

Evaluation Background The overall purpose of the evaluation is to assess the overall performance of the project and to determine if the intervention has reached its intended outputs and objectives. In particular, it assesses to what extent (and the reasons why) the project’s outputs have ensured an economic income and improved the diversity of food intake of vulnerable households in Moyamba District.

The evaluation focuses on the entire project funded by Irish Aid. It covers all geographical areas and all selected target groups of beneficiaries and examines the implementation of all activities and the degree of achievement of all outputs and objectives. Fi nally, the evaluation provides key recommendations towards sustainability.

Direct users of this evaluation are ACF France HQ (ACF France Pool Desk Staff), ACF-SL team (including coordination and field levels) and Irish Aid. Main indirect users of this evaluation are: ACF International Network, Ministry of Agriculture, Forestry and Food Security (MAFFS), Ministry of Health and Sanitation (MoHS), others (donors, partner organizations, federal, regional and local governments, ministries, UN agencies and Global Clusters, NGOs and NGO Consortiums as well as humanitarian learning platforms (such as ALNAP)).

ACF uses the OECD-DAC criteria approach to evaluate its projects. The criteria used are the following: Design, Relevance/Appropriateness, Coherence, Coverage, Efficiency, Effectiveness, Sustainability and Likelihood of Impact. For each criteria, a series of 36 evaluation questions have been developed by ACF Sierra Leone and ACF UK (ELA unit) which are presented in the evaluation matrix (annex III).

The evaluation was conducted in Sierra Leone between 19th April and 04th May 2016.

7Cost of Diet Analysis, Irish Aid/ACF, December 2013

9

Methodology The methodology for this final evaluation is a systematic approach inspired by the qualitative research methodology. The main objective of the qualitative data collection is to get an in-depth understanding of the changes inducted by the project implementation. It also gives the opportunity to all concerned stakeholders to express their perceptions about the project’s results.

Data were directly collected from groups of beneficiaries, key informants (i.e. village leaders and technical partners), ACF staff members and triangulated with secondary data (i.e. reports provided by ACF). An evaluation matrix (see annex V) presents for each evaluation question the corresponding performance indicators, data sources (primary and secondary) and data collection methods (focus group discussion, semi-structured interview, document review). This matrix makes possible to ensure that data will be triangulated from different sources.

The sampling strategy was designed to cover a maximal diversity of situations (or “heterogeneous sampling”) regarding the project implementation . After a preliminary discussion with ACF-SL, the chosen criteria for village selection were access to markets and level of damages caused by grasshoppers. Because all beneficiaries had been involved in same project activities, the main driver of beneficiaries’ diversity was gender. Hence, focus groups were conducted separately between men and women.

To select respondents, the evaluation team asked the available beneficiaries to select 5 to 8 respondents to participate to the FGD. There always were more people available so a self -selection was necessary. A total of 38 beneficiaries were interviewed during these FGD (66% women).

Table 2: Sampled villages with their selection criteria and number of respondents interviewed in FGD

Chiefdom / Section / Community Access to markets

Vegetable production FGD only

Men Women Lower Banta / Mokotawa / Kortumahun

remote place Best yield detected 0 8

Kaiyamba / Koromboya / Service street

urban area important damages by

grasshoppers 5 0

Fakunya / Njawa / Salina Approx. 5 miles

to a market 90% damaged by

grasshoppers 0 9

Bumpeh / Greema / Magazine new site

Secondary town important damages by

grasshoppers 8 0

Kaiyamba / Waliwahun / Kpatema Approx. 5 miles

to a market moderate damages by

grasshoppers 0 8

Interview guidelines were developed for each type of respondents targeted for the evaluation (see annex VI). These guidelines show a correspondence between interview questions and evaluative question to make sure that all of them are covered by the data collection. Questions on same topics will be asked to different groups of people to triangulate information and understand potential differences of perceptions between respondents. Questions were opened, clear, neutral and non-oriented.

A total of 12 SSI were also conducted with village leaders (5), ACF FSL HoD (1), ACF PM (1), DHMT Moyamba (1), MAFFS Moyamba (1), Fakunya Paramount Chief (1), Lower Banta chiefdom section chief (1), Food Security Working Group coordinator (1) 8.

8 See annex III for the detailed list of interviewed persons.

10

Primary data analysis followed a ‘content analysis’ approach. All ideas expressed by respondents were coded and entered in an analysis table, showing which categories of respondents is the source of information. Close ideas were then merged into categories (which are more conceptual) and then categories were grouped into themes to answer the evaluation questions (see annex VII). This process is transparent, replicable by another evaluator and makes it possible to link findings with categories of respondents.

Secondary data analysis was conducted according to the same structure. Information was extracted from a series of documents provided by ACF and filled into an analysis table following the evaluation matrix.

The documents analyzed are: - ACF 9 internal monitoring reports (from July 2015 to march 2016) - ACF interim report to Irish Aid (December 2015) - ACF Post-distribution monitoring report (December 2015) - ACF studies on Ebola impact on food security (2014 updated in 2015) - Project proposal, logical framework, workplan and budget - Project contract - ACF Gender policy and toolkit

Quantitative information was provided by ACF SL Field Office. At the time of the evaluation,

a draft report of the baseline survey and the post-distribution monitoring report were available. The endline and KAP surveys were expected to be implemented after the evaluation.

Finally, findings were triangulated between primary data analysis and secondary data analysis. They were presented to ACF during a stakeholder workshop in Freetown (2nd May 2016).

Main limitations to the methodology

1. CSI indicator from baseline survey was still at draft stage at the moment of the evaluation. Moreover, the endline survey together with the KAP survey and Impact survey will be conducted after this evaluation, which makes impossible for the evaluator to use these data to triangulate the qualitative information collected during the evaluation. This is a major limitation to be noted.

2. The sampling strategy here was to collect qualitative information (see above); hence quantitative information collected during this evaluation is not statistically valid. However this information will provide a good understanding of trends which will be then triangulated with quantitative information collected during the endline survey by ACF.

3. Due to a limitation of time and budget, there was no household interview during this evaluation. Information was collected through FGD only with beneficiaries.

4. FDG were designed to collect gender-sensitive information. The time limitation did not allow organizing FGD based on respondents’ age. There was no data collection with non-beneficiaries neither.

5. Comparison between approaches (Efficiency analysis) was based on discussions with project team and partners. No external documentation was reviewed in addition to this data collection due to time constraint.

11

Evaluation Findings

Overall assessment9

This project was very relevant in the context of early recovery after the Ebola outbreak. Vulnerable households were in needs of cash to buy food and restart their livelihoods, especially women. The project design included activities to address immediate needs (unconditional cash transfers) and activities to improve food security and nutrition in the longer term (support to vegetable production and local financial services, sensitization to improved nutrition practices) , which were greatly appreciated by targeted communities.

Most of the activities were implemented and output targets were achieved. Nutrition sessions were still on-going and VSLA trainings were just starting during the evaluation period, which may be a problem for the sustainability of the results. Outcome indicators are not yet available to see if expected results on food security and nutrition were achieved.

As a result, we can say that:

- Beneficiaries used the cash transfers to buy food, pay school fees and pay laborers in agricultural lands. This was a great help for them even if they would have preferred to receive the money during the lean season;

- Vegetable production was heavily impacted by grasshopper attacks. Two factors must be taken into account on this: (1) the late sowing because of the delay in seed distribution exposed young plants to grasshoppers and (2) it is the first time that villagers experience so many grasshoppers in their environment. As a consequence, vegetable productions vary a lot between groups depending on their exposure to grasshoppers. Most of the groups had a small harvest of okra, cucumber and pepper. Most groups did not get enough production to keep seeds for the next season. However, beneficiaries are very interested in vegetable productions for both consumption and income generation.

- Nutrition sensitization was really appreciated by both beneficiaries and local authorities. The outcomes of this activity should be measured by ACF during the endline and KAP surveys.

- Beneficiaries did not talk about VSLA as they were not yet trained on this during the data collection.

The general quality of the intervention suffers from various implementation delays. Most of the activities were implemented late because of different reasons:

- Cash transfers were discussed for two months10 with Irish Aid and other stakeholders (NACSA, Cash transfer working group, etc.) before the project was officially validated. Then ACF discussed implementation modalities with the mobile operator SPLASH before transfers could be organized. Finally transfer happened in September and October 2015, just after the lean season.

- Seed distribution, IYCF sessions and VSLA trainings were delayed because of internal procurement process. The general context (mainly restrictions caused by the Ebola outbreak) made it difficult for ACF to get supplies on time and hire qualified staff. The procurement process now could be improved to avoid delays in future projects.

The exit strategy was to leave formal groups of vegetable producers, linked with markets and able to manage a local savings and loans fund. These expectations are not all realistic in a 12 months project. Formalizing groups as FBO would require additional support and should be conducted with MAFFS. A phase II project is strongly recommended to sustain the project’s results, which are potentially very positive for beneficiaries’ food security. A phase II should focus on timely seed distribution (a benchmark of available local varieties –potentially resistant to grasshoppers- is recommended), quality seed production and storage, cooking demonstrations with all villagers to encourage them to buy local productions, formalization of groups into FBO, and monitoring of VSLA implementation till the final share. 9 Please refer to annex 1 for the ranking table. 10 ACF proposed to provide 50 USD/month for 3 months during the lean season which is equivalent to 100% of the cost of a family-month Energy only diet, while other operators preferred to provide 30USD/month during 9 months.

12

Analysis per evaluation criteria

Design - Rating (1: low, 5: high): 3

The following questions were included in the analysis: Are beneficiaries needs (by sex and age) well identified and in which way? What was the level of beneficiary participation in project design? Is gender properly taken into account in project design? Is the design of the exit strategy realistic? Are project objectives and indicators SMART? Are sources of verification realistic? Is there a good design of the M&E system in place?

Needs were assessed at both local and national levels, but communities did not participate in the activity design.

ACF conducted FGD in Moyamba during January 2015 with villagers (more than 50% of respondents were women) to assess their situation and detect main important needs. Meanwhile, ACF participated in various coordination mechanisms (cash transfer working group, food security working group) and met MAFFS both at national and district levels to get a good understanding of needs.

As a consequence, ACF had a good understanding of vulnerable households’ early recovery needs in the post-Ebola context. During the evaluation, beneficiaries confirmed that their main need in May 2015 was to get cash to buy food. That is why cash transfers were particularly appreciated, especially by women who are more concerned by paying school fees and labourers in their garden.

However, ACF did not involve targeted communities in the activity design. Local authorities and beneficiaries said that ACF came to explain what will be implemented in their village. Moreover, these communities do not have an overview of all components of the project. Activities are presented to villagers one after the other without presenting links between them. This top -down approach may be a limit of this intervention and could be improved to promote project’s appropriation by the targeted communities now that the context enables more active participation from the communities.

On the other hand, ACF worked jointly with local State services (MAFFS and DHMT) to develop trainings (vegetable production; nutrition sensitization) to ensure that the project is in line with their strategy.

Gender was central in the project design.

ACF targeted a minimum of 50% women beneficiaries for this project. Actually, 55% of the 1 000 beneficiaries are women. It is not clear how many of them are women head of households and how many are married, the corresponding database does not present this information. This segregation was meaningful because women seem more involved in organizing dry season productions to get incomes.

During the cooking demonstration sessions, ACF asked the men beneficiaries to call their wife to attend the training. Husband were encouraged to participate in the training too.

Project’s objectives were not realistic for a 12 months period.

The exit strategy was to (1) have formal vegetable production groups (2) linked with markets and (3) able to manage locally savings and loans through VSLA.

This exit strategy, as it was designed, is not realistic in a 12 months period. First, create and formalize groups of farmers in such a short timing is not possible. ACF initiated a process in this direction with MAFFS, but finally it was not finalized. Requirements to recognize VPG as FBO is still not clear for ACF. It is recommended first to clarify what are these requirements.

Connecting VPG with markets is possible for groups located near cities, but is much more challenging for remote groups. ACF managed to connect 4 VPG with local private customers (Rutile Mining Company). Promoting vegetable marketing within local communities could be an alternative to this lack of access to main markets.

Finally VSLA trainings were initially planned to happen between September 2015 and May 2016, which gives a 7-8 months monitoring from ACF when a full VSLA cycle is 9 months. It is important for implementers to plan an on-going monitoring of VSLA until the final share to ensure that the

13

methodology is fully understood by members. There is a risk of unexpected negative effects on social cohesion if the final share is not properly conducted. Actually, VSLA trainings were started in May 2016 for an end of project expected end of June 2016. ACF is planning to monitor this activity on its own until end of August, which should be prolonged to end of February 2017 to cover the full cycle.

Indicators were mainly collected for reporting. The information reliability is weak.

Regarding the project logical framework and associated workplan, the following observations can be done:

- Selected result statement and corresponding indicators are relevant to track project progress. No result statement was chosen to reflect the expected increase of incomes (through sales of vegetables and income generation activities created thanks to VSLA). However, increase of incomes is a difficult indicator to track in a 12 months project.

- The expected increases of FCS and CSI (indicators for purpose and output 1) are not quantified;

- The data collection plan does not allow attributing any potential increase of FCS and CSI to the project implementation. Baseline data was collected in August which is the middle of the lean season. Post distribution monitoring was done in October, but FCS and CSI were collected only in January on recall. This information is biased and probably not reliable. Moreover, October and January are not in the lean season, it is expected that FCS and CSI would increase without any project implementation. Same problem will happen with endline data which is expected to be collected in May 2016.

- The sampling design is not presented in the methodology section of the baseline report. Baseline report says that the sample was designed to get a 95% confidence interval, without specifying which the variable of interest is. ACF used a sample size table provided by PENNSTATE cooperation extension11. This document should be hyperlinked in the baseline and endline reports.

Monitoring information from the field was summarized by team leaders. There is no database available to monitor project implementation at beneficiaries’ level, but only at village level. It is impossible to compare project’s results between categories of beneficiaries (ex. women -headed households and men-headed households) neither to run random check to verify information at beneficiary level.

In general, information was collected mainly for reporting purposes rather than management considerations. Output indicators were reported in monthly APRs which clearly show progress achieved, but with no specific link to management decision.

At outcome level, indicators are collected for donor reporting only (baseline, PDM and endline), which is fine for a 12 months project.

Finally, ACF invested a lot in the targeting and verification of selection processes, but no database was created to inform this process. Field team spent two months to conduct household visits and check their vulnerability status based on specific criteria. It would have been interesting to enter this data to understand who the project beneficiaries are.

11 http://www.extension.psu.edu/evaluation/pdf/TS60.pdf

14

Relevance/Appropriateness - Rating (1: low, 5: high): 4

The following questions were included in the analysis: Were the actions undertaken relevant and appropriate given the local context and needs of the target population? Was the assistance relevant and appropriate in relation to the practices / culture of the target population? To what extent were the needs of beneficiaries and stakeholders taken in to account in project implementation?

Integration of cash transfer, vegetable production groups, VSLA and nutrition sensitization is very relevant to support early recovery process.

The Ebola context was characterized by a decrease of incomes for targeted households. They had poor harvests so few products to sell and markets were largely disrupted. Food was available in some markets but vulnerable households faced difficulties to buy it.

Initially, cash transfers were expected to happen during the lean season (i.e. June-July-August) to improve food access for targeted vulnerable households. Due to the initial delay of 2 months (discussion with Irish Aid and other cash transfer stakeholders then discussion with the mobile company SPLASH) and to the length of the selection and verification process, these cash transfers happened in September and October. Most respondents agree that this money would have been more useful during the lean season; however this income was still very relevant and useful for the targeted households. They declared to have used it to buy food, pay school fees and pay labourers to prepare agricultural lands. PDM report confirms that the money was used to buy food (40% of the money) and pay school fees (11% of the money). However, PDM highlights that 11% of the money was used for debt repayment which was not declared by respondents during the evaluation. Labour payment does not appear in the PDM report, it may be included in the ‘small trading’ category (7% of the money used). Women interviewed said that they pay each labourer 7 000 SLL per day (1.6€) to clean the land and 10 000 SLL (2.2€) to prepare for planting, plus food provision12. For women, cash transfers were very useful to restart properly their agricultural production.

Hence, the support to VPG was very relevant too. In some villages, vegetable production was new, in others it was done before the project implementation but “with very low profit”. All interviewed beneficiaries appreciated to be trained on vegetable production and to receive inputs and tools.

However, potential improvements for vegetable production support were proposed by local communities and stakeholders:

- Seeds should be provided in September, not later. A late sowing exposes the plants to grasshoppers, important evapo-transpiration and risk of drought.

- Choice of varieties to be provided to farmers should take into account: (1) their resistance to grasshoppers, where local varieties seem to have a better potential; (2) the availability of local seed producers, who cannot compete with free distribution to farmers by iNGOs.

- It would be relevant to include pulses in the seed package provided to beneficiaries, so they could produce and consume proteins.

- There is a need to improve service provision to farmers, on a more demand-driven approach. This could be done by involving more MAFFS field staff in project activities and shifting from a top-down approach to a real farmer field school (FFS) approach.

Cooking demonstrations and nutrition sensitization were also perceived as very relevant by both local communities and local authorities. The only remark is that in some remote villages, some of the ingredients used for the cooking demonstrations are not available (onions, Maggi).

Finally, strengthening VPG by training them on VSLA is a very good idea. Unfortunately, due to implementation delay, expected effects of this activity on beneficiaries’ economic empowerment were not evaluable.

12 1 € = 4510 SLL (12/05/2016)

15

Project design and implementation enabled to avoid tensions in the targeted communities

No problem occurred in the targeted communities because of the project implementation. ACF team made sure that husbands agree on the fact that their wife were participating in project activities. Local elders were asked to help to find a land for VPGs. Initially, these lands were supposed to be provided for free but actually, most groups have to pay an annual rent (estimated 10 000 – 50 000 SLL per member per year [2.2- 11€]13) to use the garden land. This is not a big issue as it ensures that members have access to lands if they have access to incomes or loans in September when the rent is due. In future vegetable support projects, it will be important to include this consideration in the project design (cash transfer, VSLA active in this period, etc.).

Village leaders were asked to participate in the beneficiaries’ selection. Different leaders met during the evaluation said that non-beneficiaries come to them to complain. Sometimes they just answer that “they will be selected in the next project”, which could be a risk for social cohesion in future intervention.

Working at group level was very relevant. Village leaders note a better social cohesion in their community. Beneficiaries discuss between them and work together, and come more ofte n to discuss with the leaders. The evaluation did not collect much information on social cohesion as it was not an expected effect of the intervention. It would be interesting for ACF to investigate this result further.

Other detected needs and opportunities.

Many interviewed beneficiaries, and especially women, asked for a support to produce groundnuts during the rainy season. Groundnut seems to be a valuable cash crop, traded to Guinea.

A support for rice production was also requested by beneficiaries.

MAFFS Moyamba advised to include chicken production to the set of FSL activities, to promote proteins consumption, income generation and manure production for vegetable production.

Coherence - Rating (1: low, 5: high): 4

The following questions were included in the analysis: Are other stakeholders informed or aware about ACF activities/approach/strategy of the project? How activities of this project have been integrated with other ACF sectors/ programs in the operational area? Do project team members feel they are working towards a common goal with respect to other departments (Nutrition, WASH, Health)?

Please note that no other iNGO working in Moyamba district was met to discuss coherency of actions between iNGOs. The meeting with Plan was cancelled because Plan staffs were busy.

This project is in line with ACF strategy in Sierra-Leone

Since 1991, ACF is working in Sierra-Leone to participate in decreasing the incidence of malnutrition. Actions are preventive and /or curative. ACF implements different projects in Moyamba district: Health, Epidemiologic surveillance (EVD, malaria, etc.), Nutrition (NUT),WASH and lately Food security (FS). The Irish Aid project evaluated here is the only food security project conducted by ACF in Moyamba district since 2013.

This project is in line with ACF’s goal to decrease malnutrition by improving access to food and improving food utilization. There was no formal coordination with other ACF departments during the project implementation, but FS and NUT field teams worked together when activities related to nutrition sensitization were implemented. ACF Moyamba NUT team participated in the training design, together with Moyamba DHMT team.

ACF is seen differently by other stakeholders depending on actions implemented locally

Local stakeholders and authorities do not have a broad understanding of ACF’s strategy in Sierra -Leone. Their perception is directly related to local ACF projects. Hence ACF action is seen “to remove poverty”, “to fight children malnutrition”, “to ensure health surveillance” depending on local actors.

13 1 € = 4510 SLL (12/05/2016)

16

Coverage - Rating (1: low, 5: high): 4

The following questions were included in the analysis: Were the most affected groups covered with the limitation of the resources available? Was the geographical coverage of the project appropriate? Were beneficiaries correctly and fairly identified and targeted? How the targeting was understood or perceived by local communities? Were gender and vulnerable populations within the target community considered in ACF’s assessment/identification of the beneficiary and in the implementation of the project? Did the project include special components for women, if so; were these systematically designed and monitored during implementation?

Selection of chiefdoms and villages was based on specific criteria

Chiefdoms were selected based on specific criteria: poverty mapping, EVD cases monitoring, number of population. Some criteria were informed before the EVD outbreak (ex. the poverty mapping was realized in 2012-2013), while others were directly related to the outbreak (ex. number of cases). As a consequence, some chiefdoms which were food insecure but with less numbers of EVD cases were not selected for this intervention (ex. Timdale).

Same approach was applied to select vil lages in the four selected chiefdoms (Kaiyamba, Fakunya, Bumpeh and Lower Banta). In this process, ACF requested local authorities to participate in the village selection. Specific additional indicators were added to select appropriate villages: availabili ty of lands for vegetable production, willingness of population to be involved in agricultural activities. Both remote and suburban villages were selected, for a total of 45 villages.

Local authorities think that it is important to target remote villages where needs are more important in terms of food security and nutrition. They would like this kind of intervention to be scaled -up in non-targeted chiefdoms and villages. Moreover, they say that very few iNGOs are active in Moyamba district so the coverage is weak.

Selection of beneficiaries: most of the very poor households were targeted in the selected villages

To select beneficiaries within the selected 45 communities, ACF implemented an iterative process with local authorities. First ACF came to village leaders explaining that they will support only the most vulnerable households of the communities. Village leaders provided a list of potential beneficiaries (for an estimated total of 2 000 households). However, ACF team found some errors in this list, and decided to run a control for all households. They were surveyed for specific criteria: EVD cases in the households, women-headed households, food insecurity level, presence of elders and children, etc. Most of these criteria come from the cash transfer working group, which are validated by NACSA and in line with MAFFS’ strategy of intervention. Then ACF explained again to the community that they could only support the most vulnerable and that there is a limit of number of beneficiaries per village. This limit in numbers was given by the use of parameters prior of the EVD outbreak (ex. poverty mapping), which was not always coherent with the post-crisis context.

Finally, ACF team had to run a second check to ensure that no better-off households were selected. MAFFS team was involved in this verification process in the field. This process took two months to implement, which caused delays in the cash transfer implementation. In visited villages during this evaluation, village leaders agreed to say that actually most of the very poor households had benefited from the intervention in their community. And beneficiaries agree that only vulnerable households were selected.

These two rounds of verification and explanations were necessary, because beneficiaries did not understand the process after the first meeting. No non-beneficiaries were interviewed during this evaluation, but village leaders say that non-beneficiaries complained to them for not having been selected. They sometimes had to give false expectations to them, saying that they would be selected in a future project. Same situation happens with non-selected villages’ representatives complaining to section chiefs.

17

Women were specifically targeted but monitoring tools are not designed to track results between genders

ACF targeted specifically women and women-headed households. A total of 55% of the beneficiaries are women. It is not clear how many of them are heads of households.

On monitoring and evaluation side, only training attendance indicators are disaggregated by sex. Outcome indicators such as production, consumption and sales of vegetable , FCS and CSI are not disaggregated, which does not allow comparing project’s results between men and women.

Efficiency - Rating (1: low, 5: high): 2

The following questions were included in the analysis: Were the resources properly allocated to reach the objectives? How efficiently are the project implementers utilizing the pro ject’s inputs to conduct activities and achieve the project’s intended results? How efficient is the overall management set up of the project, or in other words, how is the suitability of management arrangements in place? Is the project being implemented in the most efficient way compared to other eventual alternatives (e.g. cash transfer, inputs purchased and distributed, training and staff)? Are the project activities being implemented as planned and scheduled?

A realignment of budget lines and a no-cost extension were necessary to finalize the project implementation

Activity costs may have been underestimated to enable an extensive support to beneficiaries. Specifically, budget for support of vegetable productions was short and didn’t allow implementing multiple sessions of trainings to ensure a good appropriation of knowledge. Trainings of trainers took sometimes several days when trainings to beneficiaries were designed to be conducted in 1 day only (ex. For business management: training of trainers took 3 days for ACF staff and VPG representatives, training for beneficiaries was conducted in 1 day). A no-costs extension was proposed by ACF and accepted by Irish Aid in May 2016. This was the opportunity to re-organize some budget lines from logistic budget to activity budget. Taking into account these changes, budget lines use in March 2016 is presented in table 3. Salaries line was only used at 56% when activities line was already used at 89%, with nutrition trainings going on and VSLA trainings pending. Table 3: Use of budget lines in March 2016 Budget lines(realigned after the no-cost extension) % of budget spent

March 2016

Salaries 56% Activities 89%

Cash transfers 98%

Vegetable production (includes VSLA activity) 92% Nutrition promotion 45%

Support costs 76%

Total budget 72%

18

Management organization is flexible but information flows could be improved (procurement & monitoring processes)

The new managerial organization promoted by ACF, which links the project manager (PM) directly to the base field coordinator (FieldCo) with a remote technical assistance from the head of department (HoD) is functioning well. This requires a close follow-up between HoD, PM and FieldCo, which was the case for this project. However, some aspects of this organization could be improved. This evaluation detected some inconsistencies in the monitoring and procurement processes.

At base level, the information management was challenging for the PM. Information from the field was collected weekly by field staff, summaries were entered by team leaders every Friday and submitted to PM on Monday morning, before field staff had to go back to remote villages. This gave a short window to PM to take management decision regarding activities implementation.

Then weekly summaries were aggregated into the monthly Activity Progress Report (APR), drafted by PM and validated by FieldCo with a technical support from HoD when needed. Once validated by FieldCo, APR is sent to the Grant & reporting officer who will forward it to Paris HQ.

For specific challenges regarding project implementation (ex. grasshopper damages), PM, FieldCo and HoD communicated directly and took decisions to adapt activities. This flexibility was very positive to implement relevant activities to beneficiaries. On the other hand, there is no evidence that APR were useful to take timely decision (ex. delays in seed distribution, VSLA trainings). APR clearly presents progress achieved for each indicator and challenge s encountered during the implementation, but they are more a reporting tool rather than a monitoring tool. Involving top managers (ex. Deputy country director) in the APR review could help to ensure that programme needs are addressed on time.

Procurement process was challenging during the project implementation. Several reasons were pointed out during this evaluation: (1) the complexity of the early recovery context (movement restrictions, supply availabilities and human resources management) and consequently (2) the inadequacy of the procurement process to address timely requests from the programme team.

Several tools are already developed by ACF to manage procurement processes: PPP at project design stage, PR during project implementation, PFU at both coordination and base levels to monitor requests and deliveries. PM, FieldCo, logistician and finance officers (at both base and coordination levels) are involved in procurement processes. Unfortunately, this was not enough to ensure a timely delivery of needed supplies (seeds, trainings kits). A first discussion was started on how to improve this process during the debriefing workshop with all concerned staff, which should now be deepened by ACF team. Primary recommendation to avoid similar delays in future FSL projects is to include a ‘procurement delay risk’ in the project risk assessment and select a top manager to closely monitor this risk (with a focus on needs’ anticipation).

Most activities were delayed, which had a negative effect on project’s results

As described previously, cash transfers were delayed because of the initial discussions between ACF, Irish Aid and other cash transfer stakeholders in Sierra Leone (NACSA, etc.) about the proposed methodology and objectives of these transfers which differed from safety nets projects implemented by other agencies. Because of that as well of the long selection and verification process, cash transfers missed the lean season. ACF decided to group the two last cash transfers in October, which was a wise decision regarding beneficiaries’ needs (payment of school fees and agricultural labour).

Seeds were provided late to VPGs. Beneficiaries report a delivery period around December 2015 – January 2016, while the planting season is around October. Moreover, preparing a garden is a real investment (renting costs and labour costs)14; so many groups waited to receive the seeds from ACF before cleaning the land, to make sure that this investment would be useful. This delay in seed delivery jeopardized vegetable production. Unfortunately, the above average attack of grasshoppers was an additional factor of failure, because it happened while vegetable plants were still young.

VSLA trainings were supposed to start in September 2015 but were finally implemente d in May 2016 because of a delay in VSLA kits delivery. This may jeopardize groups’ sustainability if there is no 14 See section on Relevance for renting and labour costs.

19

regular follow-up till the final shares (expected in February 2017). IYCF trainings were also delayed for 3 months because of training kits delivery.

Effectiveness - Rating (1: low, 5: high): 3

The following questions were included in the analysis: What is the quality of the project outputs and/or project activities? What are the major internal and external factors influencing the achievement or non- achievement of the intended outputs and objectives? How effectively have the project performance and its outputs and objectives’ indicators being monitored? Did the project management use the collected data to take timely decision regarding activities’ implementation? How is the adequacy of control mechanisms to limit fraud and corruption? How has the feedback mechanism in place worked? What could be improved? How was the project team able to adapt to the constraints of the project? What steps were taken by the implementing Agency (ACF) to ensure that its responses were coordinated with other organizations and local authorities? To what extent does ACF take part in technical coordination mechanism at all level of project implementation?

ACF has implemented most of the planned activities (at the time of the evaluation), the output quality is satisfactory but could be improved for some activities

ACF is an active member of the Cash Transfer Woking Group and so participated actively in the development of the set of criteria to select eligible vulnerable households. After the selection and verification process, ACF distributed an ID card to each beneficiaries presenting their name, picture and reference of a secondary recipient. Cash transfers were implemented in September and October 2015 with 1000 beneficiaries, through a private telecommunication company (Splash). Mobile transfers were not suitable due to poor network coverage and rare mobile ownership (8% of beneficiaries only own a mobile phone)15, so hand to hand transfers were done in cluster distribution points. Splash agents could check beneficiaries’ identity by inserting in the smartphone’s software the code number of the ID card, which made the beneficiaries’ picture to pop-up on their smartphone. Beneficiaries all received 220 000 SLL (50 USD) in September and 440 000 SLL (100 USD) in October 2015. Even if received after the lean season, these cash transfers were greatly appreciated by beneficiaries (see section on Relevance). Some interviewed women expressed that reaching the distribution point was not easy, but they managed by paying transportation. No fraud was detected, no complain was received during the evaluation process.

A total of 46 VPG were created in the 45 selected villages, which cover 100% of the cash transfer beneficiaries. These groups were not formalized as Farmer-based organizations. All VPG received tools (hoes, shovel, machete, watering can, sprayers) in October 2015, fertilizers and seeds (Okra, Cucumber, Lettuce, Pepper, eggplant, amaranthus, and tomato) between December 2015 and January 2016. All VPG were trained in vegetable production (1 day), post-harvest and marketing (1 day) and business management (1 day). These trainings were elaborated jointly with MAFFS in Moyamba. They were conducted in school first then in gardens to apply the new techniques. This is not a FFS approach because beneficiaries were not involved in the activity design, monitoring, evaluation and reporting to other farmers. However these trainings were appreciated by beneficiaries and new production techniques are applied. Vegetable productions were heavily impacted by the late delivery of seeds and grasshopper attacks.ACF and MAFFS tried to help farmers to face the grasshoppers by training them on organic-repellent production, but this solution was not sufficiently effective to decrease damages. Many of the VPGs were able to get a small harvest of okra, pepper, cucumber (sometimes of local varieties replanted for a second cycle, which seem more resistant to grasshoppers), but most groups don’t have seeds for next year. 4 of these VPG were put in relation with a private company (Rutile Mining Company) to sell their vegetables, but marketing is very challenging for most of remote VPGs. All these groups should be trained in VSLA approach after the evaluation period.

IYCF and care practices trainings were implemented late (started in January 2016). To date, only 250 sessions are done for a target of 600 (30% of the objective reached in April 2016). This activity was strongly supported by DHMT in Moyamba, which provided some IYCF kits to ACF to start the activity implementation, and trained ACF staff together with Cause Canada staff on nutrition issues. All beneficiaries attended a cooking demonstration where diversification of meals was put in practice.

15 Data source: Baseline study (2015)

20

These nutrition-related trainings were greatly appreciated by both beneficiaries and their local authorities. The only remark from interviewed women living in remote villages was that some of the ingredients used for the demonstration are not available locally (onions, Maggi).

ACF tested a complain mechanism to ensure quality of project outputs

A complain mechanism was implemented for the first time in Moyamba during this project. It consisted in two main tools:

- A phone number was available to call ACF HR in Moyamba in case of problem. A follow-up database was created to register main call information (name and contact of caller, subject, etc.). Unfortunately, columns concerning feedback provided to callers were not systematically completed.

- Wood boxes, closed with a padlock were distributed to all 45 villages of intervention. These boxes were meant to receive written complaints from beneficiaries. Only PM or DPM were allowed to collect these boxes (supposedly on a monthly basis) and only ACF HR in Moyamba was allowed to open them. Unfortunately, no follow-up mechanism of boxes collection and opening was settled. Many boxes were never collected and no answer was provided to beneficiaries where boxes were collected.

This complain mechanism is an interesting innovation and was appreciated by all stakeholders involved in this project. After this first experience, ACF should now improve it to ensure its effectiveness. Follow-up of calls and boxes collection must be done properly, with complete databases presenting all needed information and answers provided to the communities. Boxes collection system should be reviewed to make sure that all boxes are collected, at least on a quarterly basis.

A monitoring system was set-up but for reporting purpose mainly

ACF PM developed monitoring tools to collect information on beneficiaries’ attendance to distributions, surfaces cropped for vegetable productions, volumes of vegetable produced, consumed and sold. The monitoring data management was done by project te am, as follows:

- Field staff collected information on papers and gave them to their team leader every Friday;

- Team leader were supposed to enter this information and provide an electronic document to the PM and DPM on Friday, however they generally provided only summaries of information (aggregation at group level of main indicators) the following Monday. No database was informed presenting this information at beneficiary level;

- PM and DPM had hardly one hour to review these summaries and take decisions before the field team was leaving to villages for the week.

This information was then aggregated in the monthly Activity Progress Report (APR) sent to the management line for reporting purposes (see Design and Efficiency sections for more details).

Interestingly, MAFFS in Moyamba conducted monitoring visits on their own. They drafted a report including recommendations for ACF, but these visits and the report were not yet discussed between MAFFS and ACF. Involving MAFFS field staff in ACF monitoring system could be a way to get timely information for managers.

ACF coordinated its action with local stakeholders

ACF coordinated this project with NACSA, MAFFS and DHMT to make sure that it would be coherent with their own actions and strategies, and to avoid overlap with other stakeholders. MAFFS was also involved in the selection process of beneficiaries, design of agricultural trainings and germination test of provided seeds. Coordination meetings were conducted with MAFFS during the project implementation, unfortunately not on a regular basis. DHMT provided some IYCF kits to ACF and trained ACF staff together with Cause Canada staff on nutrition related trainings.

There are very few other NGOs in Moyamba district: Plan (working on food security), World Vision (implementing cash transfers), Action Aid (working on WASH and Education) and Cause Canada (on nutrition). No specific coordination mechanism was described with them during this evaluation.

21

Sustainability and Likelihood of impact - Rating (1: low, 5: high): 3

The following questions were included in the analysis: How and when does the project intend to withdraw its resources? What plans are in place to ensure that the achievements of the project are not jeopardized by the time of project phase out? Was the project assistance provided in a way that took account of the long term context? How suitable are these plans and are they being implemented? Did the partnership or local community based organizations established at local level contribute to the sustainability of the work? To what extent are the project results likely to be sustained in the long term? To what extent is the project contributing to improved socio-economic impact of the Ebola outbreak on the food and nutrition security status of vulnerable households? What does the comparison between baseline and endline suggest?

Sustainability strategy did not include a procurement delay risk

The risk assessment did not identify potential delays in procurement to be a risk for results’ sustainability. As we have seen, a late delivery of seeds exposes farmers to get a poor harvest and even less if grasshoppers are numerous. Seed production was strongly disturbed this last season, which makes most of VPG unable to restart a vegetable production next October by themselves. VSLA kits delivery was also delayed and so the activity implementation. This also could put th e sustainability at risk as VSLA is central in the exit strategy if final shares are not correctly monitored (expected in February 2017).

Sustainability strategy was not realistic, however most VPGs are likely to survive

Groups of beneficiaries were expected to be sustainable thanks to:

- A free access to gardening lands: We have seen that actually most beneficiaries have to pay an annual rent to get access to lands. ACF should now include this fact in its strategy of intervention and make sure that VPG members have access to cash when the rent is due. For example, VSLA activity should be implemented in order to have loans available in September (which was not the case in the initial plan but should be in September 2016).

- A formal FBO recognition: This was not achieved and the process of formalization is not even clear. FBO recognition could help VPGs to be part of MAFFS plan of development, being linked with agri -business centres (ABC supported by MAFFS), etc. ACF should keep working with MAFFS on that, looking at requirements to get formal FBO recognition.

- An improved access to financial services through VSLA: VSLA activity implementation is late and so was not included in this evaluation. As said before, it is recommended that ACF keeps monitoring these groups till the final shares expected in February 2017 to make sure that the methodology is totally understood and applied.

- An improved access to markets: Access to markets is challenging for remote groups. Targeting main local markets is maybe not the most relevant strategy considering the lack of transportation means in the area. An alternative way to increase marketing of produced vegetables could be to include all communities in the cooking demonstrations – including better-off households- promoting the use of local vegetables.

However, many interviewed VPG members plan to keep working within their group, on vegetable production or other crops. One VPG visited is planning to work on groundnut production during this rainy season; another will continue to rent a land during next dry season to grow local varieties of vegetables (mainly pepper and okra). This shows that this project was very relevant for targeted population, and it would be worthy to plan a phase II project to sustain these results.

22

Vegetables produced are sold and consumed by beneficiaries but potential impact on diet diversification and incomes is limited because of late planting and grasshopper attacks

Most interviewed beneficiaries declared that they managed to get a small harvest of okra, cucumber and pepper. A part of it was sold in local markets which generated some incomes for these households. They also consumed a part of their production and thanked ACF to have organized cooking demonstrations. Local leaders see an improvement with many people “getting healthier” and “babies getting fat”. These increases in incomes and diet diversification should be cross-checked with data from the endline survey when it is available.

Unfortunately, the expected effect on incomes and diet diversification was strongly limited by the grasshopper attacks and the late planting. It would be relevant to keep supporting these groups for another season, providing timely grasshopper-resistant varieties to these VPG and training them on seed production and storage.

Conclusions This project was very relevant in the early recovery context of the Ebola outbreak. Cash transfers enabled targeted households to restart their livelihoods; vegetable production groups helped them to improve their vegetable production, consumption and sales. Nutrition-related trainings were greatly appreciated and seem to be well assimilated by beneficiaries. VSLA trainings were not yet implemented at the time of this evaluation but are expected to sustain VPGs in the future. There is no quantitative data available to cross-check these findings to date.

The integration of both short-term activities (Cash transfers) with development activities (vegetable production and nutrition, VSLA) has a strong potential for improving vulnerable households’ food and nutrition security. However, this potential couldn’t be fulfilled because of delays in project implementation and an above-average attack of grasshoppers. For future FSL projects, it would be important to review and monitor closely the procurement process and develop a low-cost strategy to mitigate grasshopper damages (early planting, local varieties and of less palatability for grasshoppers, etc.).

The complain mechanism implemented by ACF is a very interesting innovation in Moyamba district. This first experience has shown that beneficiaries are willing to provide feedbacks to implementing agencies; however this system was not always functioning as it should. ACF should now improve this system to make it effective: review the bow collection system and follow-up, review to monitor system to ensure that answers are given to communities.

Generally, the information management could be improved. Data from the field were barely available and used by managers to take timely decisions, monitoring activities were designed for reporting rather than management. Developing adequate databases to store information at household level, presenting their selection criteria and activity results could help to track progress depending on categories of beneficiaries (ex. woman-headed households versus man-headed households, etc.).

Implementing a phase II support would be very relevant to strengthen these first results. The additional support should focus on timely vegetable seed provision (ACF should assess which local varieties resistant to grasshoppers are available), quality seed production and storage, cooking demonstrations with the whole communities (including better-off households) with the aim to promote diet diversification and stimulate local vegetable market, FBO recognition and monitoring VSLA activity till the final shares (expected in February 2017).

23

Lessons Learnt and Good Practices The following lessons learnt have emerged from this project evaluation:

- Procurement process monitoring is essential to ensure quality outputs in food security projects(for ACF SL and HQ)

Livelihoods activities are generally seasonal. For implementing agencies, this means that the window for activities’ implementation may be narrow and so requires a good preparation. Support to agricultural productions must be coherent with the local seasonal calendar. Seeds must be provided before the planting season to make sure that communities will prepare their lands on-time (especially if land renting and preparation are costly). Selection of varieties must take into account their adaptation to the local context (including grasshopper pressure, local food habits, agronomical potential, etc.) and availability in Sierra-Leone16. VSLA activity must be planed and implemented in a way that makes possible to monitor groups’ activities till the final shares.

- Grasshoppers must now be identified as a risk for dry season productions in future food security projects in Sierra-Leone (for ACF SL)

Grasshoppers were not identified as a major risk for agricultural productions in Moyamba. This last dry season has shown that they should be. Instead of relying on importations of pesticides as MAFFS tried to do in 2015-2016, it is recommended to develop a low-cost strategy to decrease grasshoppers’ damages. This would include selection of local and resistant varieties, early planting cycles, etc.

- Seasonal outcome data collection (FCS, CSI) should be planned and implemented in a way that allows data to be compared(for ACF SL and HQ, donors and partners)

To be able to compare FCS and CSI indicator between baseline and endline, the data collections should happen in similar periods (ex. lean season for both surveys). In a 12 months project, that requires to plan and budget this data collection during a no-cost extension in accordance with the donor. To be able to detect the effect of one activity on these indicators, the data collection should happen during the same period too (ex. During the first cash transfer as a baseline in lean season; during the second cash transfer for the follow-up in lean season too).

The following good practises have emerged from this project evaluation:

- Selection process of beneficiaries involving local institutions and authorities, cross-checked by two rounds of field visits was essential to get a quality targeting within the communities (for ACF SL)

This long process enabled ACF to avoid tensions within local communities during activities implementation. However, village and section chiefs noted that non-beneficiaries were complaining for not being selected. It would be relevant to provide a support to local authorities which have to answer to these complains (ex. providing them a document with the criteria list).

- Implementing a complain mechanism is a relevant idea, which should be improved to ensure a quality information flow between ACF and targeted communities(for ACF SL and HQ, local authorities)

The complain mechanism was a relevant idea implemented by ACF. This enabled beneficiaries to contact ACF about activities’ implementation and could be used to identify potential frauds (this was not the case during this project). This system can be improved by reviewing the collection process and monitoring tools. This good practise is developed in Annex II.

16ACF-SL has already started a process of developing a framework agreement with the certified seed supplier to avoid future delays in seed supply.

24

Recommendations #1: Review procurement process and its monitoring for FSL projects [high priority] To ensure better quality of FSL outputs in the future, it is strongly recommended that ACF SL reviews its procurement process. Delays in procurement process should also appear in future risk analysis for FSL project. This risk should be closely monitored by top managers and mitigated if necessary.

This review should happen before a new FSL project is implemented.

#2: Prepare a phase II project to strengthen results’ sustainability [high priority] Because of the poor harvest during last dry season, most beneficiaries do not have seeds available for next season. Moreover, VSLA activity would require an on-going monitoring till February 2017.

ACF SL should develop a phase II project in order to strengthen the first results. This phase II should focus on:

- Assess availability of local vegetable seeds, including their palatability for grasshoppers; - Provide seeds to existing VPGs in early September 2016; - Include pulses in the gardening production to improve proteins consumption; - Train beneficiaries in production and storage of quality seeds; - Develop a low-cost strategy to decrease grasshopper damages with MAFFS; - Involve better-off households in a second round of cooking demonstration and nutrition

trainings; - Work on formalization of VPGs as FBOs with MAFFS; - Monitor and support VSLA till the final shares

This project should be implemented in before September 2016.

#3: Review the complain mechanism to improve the information flow between ACF and communities [medium priority] To improve the current complain mechanism, ACF SL should look at:

- How the complain boxes are collected and how frequently. The box collection system should enable ACF SL to collect each box quarterly and open it in a safe way to protect potential sensitive information;

- Develop complete follow-up tools enabling to track box collections, received calls, and feedback provided to communities.

#4: Review the M&E plan for future FSL projects [low priority] The following improvements in M&E are recommended:

- CSI and FCS indicators should be collected during a similar period (ex. lean season) to be able to detect progress achieved. In a 12 months project, that requires to plan and budget this data collection during a no-cost extension in accordance with the donor;

- Monitoring databases should be developed at beneficiary level, presenting their selection criteria and activity results;

- Survey reports must present sampling strategy and design; - Involve MAFFS field team in activity monitoring to avoid duplication of monitoring

systems.

25

Annexes

26

Annex I: Evaluation Criteria Rating Table Criteria Rating

(1 low, 5 high) Rationale

1 2 3 4 5 Design X The design of the project has a great potential to

improve food and nutrition security status of vulnerable households. However, exit strategy is not realistic for a 12 months only project. M&E system shows important weaknesses: monitoring databases at beneficiary’ level are missing, outcome indicators are not timely collected and are not partially reliable (cf. recall survey in January for FCS& CSI).

Relevance/Appropriateness X Integration of cash transfer together with livelihood recovery (vegetable production and VSLA) and nutrition sensitization is very good. Working at group level is relevant and improved social cohesion. Support to vegetable could be improved: seed delivered earlier (September), choice of varieties, VSLA enabling loans in September to pay the land rent.

Coherence X The project objectives and activities are coherent with ACF goal in Sierra-Leone to decrease malnutrition incidence. National stakeholders (including local authorities) do not have a broad understanding of ACF goal and strategy.

Coverage X ACF implemented an important selection process based on specific and transparent criteria. ACF and its partners invested a lot in verifying this selection to ensure that only vulnerable households were selected. This process was long (2 months) which caused delays in the activity implementation. Most of very poor households of selected villages were included in the action.

Efficiency X Most activities were delayed which had a negative effect on project’s results. Especially, implementation delays in seed delivery and VSLA trainings may jeopardize results’ likelihood of impact and sustainability. Procurement process should be reviewed and closely monitored for future FSL projects (at both programme and logistic levels)

Effectiveness X Most of the output targets are achieved by the time of this evaluation. Quality of outputs could be improved by implementing FFS approach rather than top-down approach in agriculture. The complain mechanism is an interesting innovation in Moyamba, which now can be improved to ensure its effectiveness.

Sustainability and Likelihood of Impact

X Likelihood of impact and results’ sustainability are strongly impacted by late vegetable planting, grasshopper attacks and late implementation of VSLA activity. However it seems that interesting results are obtained where even small harvests were obtained, and that groups are likely to survive and implement other activities. It would be very relevant to plan a phase II project to sustain these results.

27

Guidance for rating the evaluation criteria:

Rating Definition

1. Unsatisfactory Performance was consistently below expectations in most areas of enquiry

related to the evaluation criteria. Overall performance in relation to the

evaluation criteria is not satisfactory due to serious gaps in some of the areas.

Significant improvement is needed. Recommendations to improve

performance are outlined in the evaluation report and Action Against Hunger

will monitor progress in these areas.

2. Improvement

needed

Performance did not consistently meet expectations in some areas of

enquiry– performance failed to meet expectations in one or more essential

areas of enquiry. Some improvements are needed in one or more of these.

Recommendations to improve performance are outlined in the evaluation

report and Action Against Hunger will monitor progress in these key areas.

3. On average

meets

expectations

On average, performance met expectations in all essential areas of enquiry

and the overall quality of work was acceptable. Eventual recommendations

over potential areas for improvement are outlined in the evaluation report.

4. Meets

expectations

Performance consistently met expectations in all essential areas of enquiry,

and the overall quality of work was fairly good. The most critical expectations

were met.

5. Exceptional Performance consistently met expectations due to high quality of work

performed in all essential areas of enquiry, resulting in an overall quality of

work that was remarkable.

28

Annex II: Good practice

Title of Good Practice

Implementing a feedback mechanism with local communities

Innovative features and key characteristics Enabling local communities to provide feedbacks on ACF activities creates a trust environment between beneficiaries and the implementing agency. This practice differs from what is sometimes implemented by agencies during cash transfers to detect frauds (random interviews or calls, etc.).

Background to the Good Practice The idea from ACF was to settle a system to avoid potential frauds during the cash transfers. But instead of implementing a specific and short-term complain mechanism, it was decided to implement a more durable system for the time of the project implementation. This system was also meant to be used by beneficiaries in vil lages where ACF implements other projects (nutrition, WASH).

Further explanation of the chosen good practice ACF implemented two channels for this complain mechanism:

- A hotline was made available (the ACFHR line in Moyamba) - Boxes closed with a padlock given to each community where the project was implemented. - These boxes were supposed to be collected by PM / DPM only, and given to ACF HR in

Moyamba for opening. - A follow-up of received calls was done, which could be improved.

It is the first time that such feedback system was implemented in Moyamba. Local communities appreciated this initiative. They see it as a proof of ACF seriousness. However it must be noted that not all boxes were collected by ACF, which could have disappointed some communities. Practical/Specific recommendations for roll out ACF should first review the box collection system – PM/DPM only are not enough – and information flow between communities and ACF. ACF must ensure that all feedbacks are collected and treated. Databases must present all information received and date of reception with contact of the person. It must be clear if an answer was provided to the community or not. To implement this improvement, ACF should invest in HR resources at base level. Hiring a M&E officer who would take the lead of this task together with other M&E activities could be a solution to avoid excessive workload for PM or HR officer.

How could the Good Practice be developed further?

This system may be a good opportunity for ACF to collect not only complains from communities but also their ideas and suggestions which could be integrated in future projects. Involving local communities in project design is a good way to ensure results’ appropriation and sustainability.

29

Annex III: list of interviewed persons

Name of interviewed person Organization

M. Sulaiman Sankoh MAFFS Moyamba

M. Anderson Sam Fannah MAFFS Moyamba

M. Foday M. Sheriff MAFFS Moyamba

Ms. Kadie Kandeh DHMT Moyamba M. Abu Himgbatei Lower Banta section chief

M. Joseph Atie Kavura Kongormor II Fakunya chiefdom paramount chief

M. Felix Gossrau FAO, Food security working group

M. Alessandro Dalle Carbonare ACF-SL, HoD FSL

M. Emmanuel D. Kemoh ACF-SL, PM

Ms. Isotta Pivato ACF-SL, DCD

Annex IV: list of document reviewed The following documents were reviewed during this evaluation:

- ACF 9 internal monitoring reports (from July 2015 to march 2016) - ACF interim report to Irish Aid (December 2015) - ACF Post-distribution monitoring report (December 2015) - ACF studies on Ebola impact on food security (2014 updated in 2015) - Project proposal, logical framework, workplan and budget - Project contract - ACF Gender policy and toolkit

Annex V: evaluation matrix Please refer to the attached excel file.

Annex VI: interview guidelines Please refer to the attached excel file.

Annex VII: primary data analysis Please refer to the attached excel file.

Annex VIII: Terms of reference Please refer to the attached pdf file.

1

TERMS OF REFERENCE

For the Independent Final Evaluation of ACF’s project

‘Mitigating the Impact of the Ebola Virus Disease on the Most Vulnerable Households through an Integrated Food and Nutrition Security Intervention in the District of Moyamba, Sierra Leone’

Programme Funded by

Irish Aid

Contract Reference ACF SL 15 01

08/02/2016

2

Summary Table

Project Name Mitigating the Impact of the Ebola Virus Disease on the Most Vulnerable Households through an Integrated Food and Nutrition Security Intervention in the District of Moyamba, Sierra Leone’

Contract Number ACF SL 15 01

Partners (if applicable) None

Sector Food and Nutrition Security

Location (country/ies, region/s)

Moyamba district, Sierra Leone

Duration 12 months

Starting Date 15th May 2015

Ending Date 14th May 2016

Project Language English

Donor & Contribution/s Irish Aid: € 804,000

Country Office administering the Project

ACF Sierra Leone

Responsible ACF HQ ACF France

Evaluation Type Independent Final Evaluation

Evaluation Dates 05th April – 13th May 2016

3

1. PROJECT BACKGROUND 1.1. Map of Project Area

1.2. Rational for the Project Since March 2014, the outbreak of Ebola and spreading in West Africa countries (Guinea, Liberia and Sierra Leone) is one of most sever and complex epidemic in history. According to the Sierra Leone Ministry of Health and Sanitation, the country recorded a cumulative (laboratory confirmed) infected cases of 8,704, confirmed deaths of 3,589 individuals (of which 221 were healthcare workers), and 4,051 survivors living in the country presently. The socio-economic impact of the EVD outbreak has been substantial: national economies have become affected in the key sectors; livelihoods of households and communities have deteriorated, as large segments of the active population have lost employment, and agricultural fields have been abandoned in the most affected rural areas. As a result, ACF has started in May 2015 a food security and nutrition project in Sierra Leone. The project intends to contribute to resolve the negative socio-economic impact of the Ebola outbreak on the food and nutrition security status of vulnerable households in Moyamba district. The project has three interrelated components or interventions; provision of unconditional cash transfer, promoting vegetable gardening and improving awareness on nutrition and care practice in target areas. Currently, 1000 households have been targeted and benefited in four selected chiefdoms (Bumpeh, Fakunya, Kaiyamba and Lower Banta; 45 communities) in Moyamba district. 1.3. Project Objectives The general objective of the project is to contribute to contain the negative socio and economic impact of the Ebola outbreak on the food and nutrition security status of targeted vulnerable households in Sierra Leone.

4

The specific objective is to ensure an economic income and to improve the diversity of food intake of vulnerable households in Moyamba District. Specifically the project aims to achieve the following outputs or results:

1. Targeted vulnerable households are given the means to access sufficient food consumption through Unconditional Cash Transfer during the lean season;

2. Targeted vulnerable households are given the inputs and knowledge to produce vegetables to increase both their nutritious food consumption and their income;

3. The awareness of targeted vulnerable households on the importance of a diversified and nutritious diet is improved and reflected in to their practices.

The direct targeted beneficiaries of this project are 1,000 vulnerable households living in Moyamba district and indirectly affected by the EVD outbreak. The intervention aims first to give these households the means to access sufficient food during the lean season (peak of hunger gaps and most acute prevalence of malnutrition) through unconditional cash transfer. The intervention supports these beneficiary households with a monthly cash transfer of 50 USD (which is equivalent to 100% of the cost of a family-month Energy Only Diet and to 66% of the cost of a family-month Nutritional diet1). Then, ACF provides the same beneficiaries with the inputs and knowledge to produce vegetables in order to increase both their nutritious food consumption and their income. Lastly, the third project component, the Health and Nutrition promotion, is implemented in the framework of the other two, to enhance the respective results. The awareness and education sessions have a strong focus on improving infant and young child feeding and care practices, promoting balanced diet based on local seasonal food calendar, processing and conservation of food. A detailed list of activities can be found in the project logframe in the Annex I. In September 2014, ACF conducted a Food Security Rapid Assessment on behalf of FAO in Moyamba2, whose findings are still relevant for the proposed intervention. Please refer to the Annex II: List of project documents for the desk review for more details. The management set up for the project involved ACF France, in charge of the technical support, the Country Office in Freetown, in charge of the donor reporting, the technical and overall coordination, the Field Office in Moyamba, in charge of the field coordination and the implementation, involving a Project Operational Team based in Moyamba and formed by a Project Manager (1), a Deputy Project Manager (1), Head of Projects (3), Team Leaders (2), Field Monitors (8), Agriculture Technicians (8) and Social Mobilizers (3). 1.4. Project Current Status The Unconditional Cash Transfer component (Output 1) was successfully completed in October 2015. At the moment the other two project components are still ongoing: vegetable production and nutrition sensitization. Currently all the 1,000 beneficiaries (organized in 46 Vegetable Producer Groups) are cultivating vegetables on a total area of about 100 hectares, and some of the crops (e.g. lettuce, cucumber) are being harvested and both consumed and sold in the local market and to local traders by the beneficiaries. Nutrtion awareness and sensitization are being carried out, including IYCF training and cooking demonstrations. Also VSLA training is ongoing.

1 ACF – Sierra Leone Study: A Cost of Diet Analysis - Slums of East Freetown, December 2013, funded by Irish Aid 2 ACF, “Rapid Assessment on the Effect of Ebola in Agriculture and Food Security organized by MAFFS, FAO, WFP, ACF, WHH, CWW,

BalMed” (Case Study of Three chiefdoms in Moyamba District), 11 – 15 September 2014.

5

2. PURPOSE AND OBJECTIVES OF THE EVALUATION 2.1. Rational for the Evaluation This evaluation is conducted as an exercise of accountability towards the donor and the beneficiaries. It is also expected to contribute to better understand the strengths and weaknesses of the intervention, drawing lessons learnt and making operational and strategic recommendations that can be used to improve the implementation of a potential next phase or similar interventions in the future. 2.2. Objectives of the Evaluation The overall purpose of the evaluation is to assess the overall performace of the project and to determine if the intervention has reached its intended outputs and objectives. In particular, it will assess to what extent (and the reasons why) the project’s outputs have ensured an economic income and improved the diversity of food intake of vulnerable households in Moyamba District. 2.3. Users of the Evaluation Direct users: ACF France HQ (ACF France Pool Desk Staff), ACF-SL Country Director, ACF-SL Food Security and Livelihoods Head of Department, ACF-SL Advocacy Expert, ACF-SL Health and Nutrition Head of Department, ACF-SL Moyamba Field Coordinator, ACF-SL Moyamba Food Security Project Manager and Irish Aid. Indirect users: ACF International Network, Ministry of Agriculture, Forestry and Food Security (MAFFS), Ministry of Health and Sanitation (MoHS), others (donors, partner organisations, federal, regional and local governments, ministries, UN agencies and Global Clusters, NGOs and NGO Consortiums as well as humanitarian learning platforms (such as ALNAP)). 2.4. Use of the Evaluation Learn from experience to develop new strategies at the global level, collect lessons learnt and good practices for future projects in country.

3. EVALUATION SCOPE

3.1. Evaluation Focus The evaluation will focus on the entire project funded by Irish Aid. It will cover all geograpghical areas, looking at different levels of the intervention (community level, district level, and national level) and at the links between those levels. It will also cover all selected target groups of beneficiaries and will examine the implementation of all activities and the degree of achievement of all outputs and objectives. Finally, the evaluation should provide key recommendations towards sustainability. Moreover, it will identify and recommend potential exist strategies. 3.2. Cross-cutting issues As per ACF Gender Policy and Toolkit, the evaluation is expected to carry out a gender analysis, also assessing the strategies implemented to address gender inequalities. 3.3. Elements not covered by the evaluation There are no specif elements not covered by the evaluation.

6

4. EVALUATION CRITERIA AND QUESTIONS As per ACF Evaluation Policy and Guidelines3, ACF adheres to the Development Assistance Committee (DAC) criteria for evaluating its programmes and projects. Specifically, ACF uses the following criteria: Relevance/Appropriateness, Coherence, Coverage, Efficiency, Effectiveness, Sustainability and Likelihood of Impact4. To the latter list ACF adds an additional criterion, Design. ACF also promotes a systematic analysis of the monitoring system in place within the aforementioned criteria. Evaluation questions have been developed to help the evaluator assess the project against these criteria (Refer to Annex V). The evaluator may adapt the evaluation criteria and questions, but any fundamental changes should be agreed between the ELA at ACF-UK and the evaluator and reflected in the inception report. All independent external evaluations are expected to use DAC criteria in data analysis and reporting. In particular, the evaluator must complete the DAC criteria rating table (Refer to Annex V) and include it as part of the final evaluation report.

5. EVALUATION METHODOLOGY This section outlines the suggested methodological approach for the evaluator to collect quantitative and qualitative data. The evaluator will to the extent possible develop data gathering instruments and methods which allow collecting sex-disaggregated data. The instruments need to make provision for the triangulation of data where possible. 5.1. Evaluation Briefing Prior to the evaluation taking place, the evaluator is expected to attend an evaluation technical briefing with the ELA ACF-UK. Briefings by telephone must be agreed in advance.

5.2. Desk review

The evaluator will undertake a desk review of project materials, including the project documents and

proposals, progress reports, outputs of the project (such as publications, communication materials,

videos, recording etc.), results of any internal planning process and relevant materials from secondary

sources.

5.3. ACF HQ Interviews As part of the evaluation, the evaluator will interview HQ stakeholders to get preliminary information about the project being evaluated. Briefings by telephone must be agreed in advance.

5.4. Inception Report At the end of the desk review period and before the field mission, the evaluator will prepare a brief

inception report. The report will be written in English and will include the following sections:

Key elements of the TORs to demonstrate that the evaluator will adhere to the TORs;

Present the methodological approach to the evaluation (including an evaluation matrix in

annex to specify how the evaluator will collect data to answer the evaluation questions) and

point out the limitations to the methodology if any and the choice of sites per field visit;

3 http://www.alnap.org/resource/6199 4 The criterion has been rephrased to “Likelihood of Impact” as a thorough impact assessment is linked to the estimation of attribution, which can only be measured through experimental or quasi experimental evaluation designs. The evaluation design for carrying out a performance evaluation would not be suitable to determine the effects attributed to the project.

7

Provide a detailed evaluation workplan and;

State adherence to ACF Evaluation Policy and outline the evaluation report format.

The inception report will be discussed and approved by the ELA in ACF-UK.

5.5. Field Mission

Primary data collection techniques As part of the evaluation, the evaluator will interview key project stakeholders (expatriate/national project staff, local/national representatives, local authorities, humanitarian agencies, or donor representatives) as per the list in Annex IV. The evaluator will use the most suitable format for these interviews as detailed in the inception report. The evaluator is also expected to collect information directly from beneficiaries. Towards enriching triangulation, if budget and timeframe allows, the evaluator could also conduct Focus Group Discussions (beneficiaries, non-beneficiaries, key informants – health workers, teachers and leaders) and household surveys. Field visits The evaluator will visit the project sites and the facilities provided to the beneficiaries (if any). Secondary data collection techniques: Desk review The evaluator will further review complementary documents and collect project monitoring data or of any other relevant statistical data. Debriefing and stakeholders workshop The evaluator shall facilitate a learning workshop in country to present preliminary findings of the evaluation to the project and key stakeholders (including National actors); to gather feedback on the findings and build consensus on recommendations; to develop action-oriented workshop statements on lessons learnt and proposed improvements for the future. 5.6. Evaluation Report The evaluation report shall follow the following format and be written in English:

Cover Page;

Summary Table to follow template provided

Table of Contents

Executive Summary must be a standalone summary, describing the project, main findings of the evaluation, and conclusions and recommendations. This will be no more than 2 pages in length.

Background Information

Methodology describe the methodology used, provide evidence of triangulation of data and presents limitations to the methodology

Findings includes overall assessment of the project against the evaluation criteria, responds to the evaluation questions, all findings are backed up by evidence, cross-cutting issues are mainstreamed and; unintended and unexpected outcomes are also discussed

Conclusions Conclusions are formulated by synthesizing the main findings into statements of merit and worth, judgements are fair, impartial, and consistent with the findings

Lessons Learnt and Good Practices presents lessons that can be applied elsewhere to improve project performance, outcome, or impact and; identify good practices: successful practices from those lessons which are worthy of replication; further develop on one specific good practice to be showcased in the template provided in Annex VI

Recommendations Recommendations should be as realistic, operational and pragmatic as possible; that is, they should take careful account of the circumstances currently prevailing in the context of the action, and of the resources available to implement it both locally. They

8

should follow logically from conclusions, lessons learnt and good practices. The report must specify who needs to take what action and when. Recommendations need to be presented by order of priority

Annexes These should be listed and numbered and must include the following: Good practice template (annex VI), Evaluation Criteria Rating Table (annex V), list of documents for the desk review (annex II), list of persons interviewed (annex IV), data collection instruments and evaluation TORs.

The whole report shall not be longer than 30 pages, 50 pages including annexes. The draft report should be submitted no later than 10 calendar days after departure from the field. The final report will be submitted no later than the end date of the consultancy contract. Annexes to the report will be accepted in the working language of the country and project subject to the evaluation. 5.7. Debriefing with ELA ACF-UK The evaluator should provide a debriefing to the ELA in ACF-UK to discuss any issues related to the evaluation report. 5.8. Debriefing with ACF HQ The evaluator should provide a debriefing with the relevant ACF HQ on her/his draft report, and on the main findings, conclusions and recommendations of the evaluation. Relevant comments should be incorporated in the final report.

6. KEY DELIVERABLES The following are the evaluation outputs the evaluator will submit to the ELA in ACF-UK:

Outputs Deadlines

Inception Report 07/04/2016

Stakeholders workshop 22/04/2016

Draft Evaluation Report 30/04/2016

Final Evaluation Report 13/05/2016

All outputs must be submitted in English and under Word Document format. The quality of the inception report and the evaluation report will be assessed by the ELA in ACF-UK. The evaluator is expected to follow the format, structure and length as defined under section 5.4 and 5.6 above.

7. MANAGEMENT ARRANGEMENTS AND WORKPLAN These evaluation TORs have been developed in a participatory manner, by the ELA in ACF-UK based on inputs from relevant stakeholders. The evaluator will directly report to the ELA in ACF-UK. The evaluator will submit all the evaluation outputs directly and only to the ELA in ACF-UK. The ELA in ACF-UK will do a quality check (ensure required elements are there) and decide whether the report is ready for sharing. The ELA will forward a copy to key stakeholders for comments on factual issues and for clarifications. The ELA will consolidate the comments and send these to the evaluator by date agreed between the ELA and the evaluator or as soon as the comments are received from stakeholders. The evaluator will consider all

9

comments to finalize report and will submit it to the ELA who will then officially forward to relevant stakeholders. Once the evaluation is completed the ELA ACF-UK will prepare the management response follow-up form to track implementation of the recommendations outlined in the evaluation report. A review of the follow-up process will be undertaken six months after the publication of the evaluation report. 7.1. Tentative Workplan NOTE: Consultants are expected to work 6 days a week (either Sundays/Fridays or whatever day the field office has off will not be paid) during their consultancy contract. Travel days are not paid as they are not working days as such.

Activities Evaluator Working Days

Dates Day

Evaluation briefing with ACF-UK ELA 0.5 05/04/2016 Tue

Interviews with HQ 0.5 05/04/2016 Tue

Desk review, preparation of field work and prepare Inception Report

2 06/04/2016 - 07/04/2016

We -Thu

Review of the Inception Report 08/04/2016 Fr

Travel to the Sierra Leone 1 09/04/2016 Sat

In country interviews with project staff 1 11/04/2016 Mo

Travel to the field (Moyamba) 0.5 12/04/2016 Tue

Field work, collection and analysis of secondary data & meeting with stakeholders

9 12/04/2016 - 21/04/2016

Tue - Thu

Travel back from the field (Moyamba) 0.5 21/04/2016 Thu

Stakeholders Workshop in country 1 22/04/2016 Fri

Travel back from Sierra Leone 1 23/04/2016 Sat

Evaluation debriefing with ACF-UK ELA 0.5 25/04/2016 Mo

Evaluation debriefing with HQ 0.5 25/04/2016 Mo

Draft Report 5 26/04/2016 -30/04/2016

Tue - Sat

ACF-UK: Quality check and initial review by ELA, circulate draft report to key stakeholders, consolidate comments of stakeholders and send to evaluator

02/05/2016 -10/05/2016

Mo – Tue

Final report on the basis of stakeholders, Mission, HQ, and ACF-UK comments

3 11/05/2016 13/05/2016

We - Fri

Total: 26

7.2. Profile of the evaluator The evaluation will be carried out by an international evaluation consultant with the following profile:

Knowledge in Food Security and Livelihood sector with particular experience on cash-based-interventions, agriculture, nutrition security and nutrition.

Significant field experience in the evaluation of humanitarian / development projects;

Relevant degree / equivalent experience related to the evaluation to be undertaken;

Significant experience in coordination, design, implementation, monitoring and evaluation of programmes;

Good communications skills and experience of workshop facilitation;

Ability to write clear and useful reports (may be required to produce examples of previous work);

Fluent in English

Understanding of donor requirements;

Ability to manage the available time and resources and to work to tight deadlines;

10

Independence from the parties involved.

8. LEGAL AND ETHICAL MATTERS The ownership of the draft and final documentation belong to the agency and the funding donor exclusively. The document, or publication related to it, will not be shared with anybody except ACF before the delivery by ACF of the final document to the donor. ACF is to be the main addressee of the evaluation and its results might impact on both operational and technical strategies. This being said, ACF is likely to share the results of the evaluation with the following groups:

Donor(s)

Governmental partners

Various co-ordination bodies For independent evaluations, it is important that the consultant does not have any links to project management, or any other conflict of interest that would interfere with the independence of the evaluation. 8.1. Intellectual Property Rights All documentation related to the Assignment (whether or not in the course of your duties) shall remain the sole and exclusive property of the Charity.

9. ANNEXES TO THE TORs

I. Project Logframe II. List of Project documents for the desk review

III. Evaluation Criteria and Detailed Evaluation Questions IV. List of people to be interviewed V. Evaluation Criteria Table

VI. Good practices Template

11

Annex I: Project Logframe Project’s Title: Mitigating the Impact of the Ebola Virus Disease on the Most Vulnerable Households through an Integrated Food and Nutrition Security

Intervention in the District of Moyamba, Sierra Leone

Narrative Summary Indicators Data Sources Assumptions

Goal

Contribute to contain the negative

socio and economic impact of the Ebola

outbreak on the food and nutrition

security status of targeted vulnerable

households in Sierra Leone.

- -

Purpose

Ensure an economic income and

improve the diversity of food intake of

vulnerable households in Moyamba

District.

- By the end of the project, at least 80% of targeted beneficiaries have increased their Food Consumption Score (FCS) from the baseline survey.

- At least 80% of beneficiaries maintain or improve their Coping Strategies Index (CSI)

- Baseline and final surveys - ACF activity & monitoring

reports - Impact Survey

Access to field and

beneficiaries is granted all

over the project period

and free from

governmental restrictions

The security conditions in

the country remain stable

and allow the

implementation of the

activities

12

The targeted population is

not affected by any other

outbreaks or disasters

during the project period

(Cholera, floods, etc.)

Output 1

Targeted vulnerable households are

given the means to access sufficient

food consumption through

Unconditional Cash Transfer during the

lean season

- During the lean season, 80% of the households have a higher and more diverse food intake than before the program (FCS)

- ACF registration and Distribution Reports

- ACF activity & monitoring reports

- Post Distribution Monitoring Report

- Impact survey

Inflation rate remains

under control and does

not adversely affect the

purchasing power of the

targeted population

Movement and trade

restrictions imposed to

the local population in the

areas of intervention are

not tightened up

The targeted population is

not affected by any hazard

or disaster during the

project period (Floods,

Pests, etc.)

Output 2

Targeted vulnerable households are

given the inputs and knowledge to

produce vegetables to increase both

their nutritious food consumption and

their income

- During the dry-season, at least 80% of targeted households have a vegetable production (e.g. okra, amaranth, sweet pepper, cucumber, hot pepper, tomato, onions, and/or others to be agreed with the beneficiaries) that insure micronutrient-rich food consumption and income (increasing their FCS)

- Distribution lists and reports - Training lists and evaluation

of the knowledge acquired - ACF activity & monitoring

reports - Impact survey

Output 3

The awareness of targeted vulnerable

households on the importance of a

diversified and nutritious diet is

- By the end of the project, at least 70% of targeted households demonstrate a better knowledge on nutrition and malnutrition.

- By the end of the project, at least 70% of the targeted households

- Post Distribution Monitoring Report

- Impact survey

13

improved and reflected in to their

practices

have applied at least two of the lessons learnt at the Nutrition Education trainings and sessions

The vegetable gardens and

the vegetable production

is not affected by acts of

vandalism or theft

Local politicians and other

stakeholders will not

unduly influence the

selection process of

beneficiaries and the

implementation of the

project

Restriction on Cash

transfer activities are not

imposed on NGOs by the

GOSL (government of

Sierra Leone)

Charges on money

transfer are not raised to

unforeseen levels due

14

Money transfer service

providers continue to be

operational during the

project period

Activities

A 1.1 Identification and selection of

beneficiaries

A 1.2 Registration of selected

households and identification cards

distribution

A 1.3 A coordination mechanism among

the different actors involved is put in

place

A 1.4 Selection of Cash Transfer service

provider based on technical capacity

A 1.5 Put in place a complaint

mechanism

A 1.5 Direct cash transfer activities

implemented by a cash transfer service

provider

- 1000 beneficiaries received the beneficiary cards

- 1000 households have benefited from 3 monthly unconditional cash transfer during the lean season

- 1000 households have been contacted by the project’s team after each cash transfer

- 3000 cash transfers have been performed by the service provider during 3 months

- Complaint boxes (one per each targeted community) and a hotline have been put in place, and beneficiaries’ complaints have been addressed

- The coordination among the actors in the same areas is assured and no overlapping for the same activities is registered

- Market prices have been monitored monthly along the project period in order to assess and ensure the food availability and the level of food access of the beneficiaries

- Baseline survey - Post distribution monitoring

reports and interviews with beneficiaries

- Cash transfer records - Monthly internal report - Regular coordination

meetings and exchange of information

- Case studies

Targeted beneficiaries do

not migrate to districts out

of the intervention area

during the implementation

of the action

The coordination

mechanism with the

relevant stakeholders is

efficient and no

overlapping or important

gaps are registered

Targeted beneficiaries are

interested and participate

actively to the trainings

15

A 1.6 Cash Transfer monitoring and

evaluation

A 1.7 Market monitoring

Activities

A 2.1 Purchase of agricultural inputs

A 2.2 Distribution of agricultural inputs

kits for vegetable production

A 2.3 Collection of soil samples and soil

analysis

A 2.4 Assess training needs, develop

training modules, and train

beneficiaries (vegetable production,

Good Agricultural Practices (GAP),

homemade composting and organic

pesticides, post-harvest and marketing,

etc.)

A 2.5 Technical assistance for

beneficiaries for vegetable production

- 1000 beneficiaries received agricultural inputs kits

- 1000 vegetables gardens have been prepared

- 15 soil samples in 15 different locations have been analyzed

- By the end of the project, at least 85% of beneficiaries have been trained on the different aspects of vegetable production and marketing

- By the end of the project, at least 75% of beneficiaries have produced vegetables for income and food consumption

- By the end of the project, 90% of beneficiaries received technical assistance and monthly field visits from the project technical staff

- At least two marketing opportunities have been identified and linked with beneficiaries in each community targeted

- At least 50% of the targeted households have been organized in

- Distribution lists - Soil analysis lab results - Monthly internal report - Attendance List of trainings - Monitoring reports and

impact survey - Case studies - Impact Survey

16

A 2.6 Monitoring and evaluation, data

collection and analysis of the vegetable

production

A 2.7 Market monitoring and

identification of marketing

opportunities and linkages

A 2.8 Support the formation of

Vegetable Producers Groups in the

targeted communities and initiate them

to Village Savings and Loan Association

approach (VSLA)

A 2.9 A coordination with line Ministry

(MAFFS) and other actors operating in

the district is put in place

Vegetable Producers Groups, and at least one Vegetable Producer Group per community has been created

- At least 50% of the Vegetable Producers Groups initiated Village Saving and Loans Activities putting in place a revolving fund for loans

- The coordination among the actors in the same areas is assured and no overlapping for the same activities is registered

A 3.1 Actively participate in the

nutrition security coordination

meetings at district and national level

advocating for nutrition security to be

mainstreamed in district and national

programmatic plans

A 3.2 Active participation in nutrition

and food security CSO platform at

- Capitalization and lessons learned are produced and shared with stakeholders

- By the end of the project, 100% of the beneficiaries have participated to at least one awareness raising activity on health and nutrition

- By the end of the project, 80% of the targeted households have benefited of at least one cooking demonstration session

- By the end of the project, at least

- Monthly internal report - Attendance List of awareness

raising activities, trainings, and cooking demonstration sessions

- Monitoring reports and impact survey

- End line KAP

17

district level for common advocacy and

awareness raising activities

A 3.3 Design and printing of illustrated

educational material and purchase of

utensils to be used for the cooking

demonstrations

A 3.4 Organization of awareness raising

sessions on health and nutrition-related

issues

A 3.5 Organization of nutrition, IYCF

and care training rounds with targeted

communities and households

A 3.6 Promotion of nutritional diet and

complementary feeding recipes

through cooking demonstration

sessions at community level

A 3.7 Carry out an end line KAP survey

50% of the targeted households have participated to training rounds on nutrition, IYCF and care

- An end line KAP survey has been performed in the targeted communities

18

Annex II: List of Project documents for the desk review The following documents will be reviewed by the evaluator during the desk review phase:

Document

ACF Evaluation Policy and Guideline

Project Proposal

Project interim report

Project Cash Transfer PDM report

Project Baseline survey report

Government of Sierra Leone, National Food and Nutrition Security Implementation Plan 2013-17

WFP Comprehensive Food Security and Vulnerability Assessment (CSFVA) 2010/11

WFP Emergency Food Security Assessment (EFSA) 2015

Agriculture Sector Post Ebola Virus Disease Response Programme (ASERP) 2015

19

Annex III: Evaluation Criteria and Detailed Questions To assess the project against each evaluation criteria, the evaluator will respond to the following evaluation questions: Design: A measure of whether the design is logical, allows for RBM and include a sustainability strategy involving local partners and beneficiaries

Are beneficiaries needs (by sex and age) well identified and in which way? What was the level of beneficiary participation in project design?

Is gender properly taken into account in project design?

Are project objectives and indicators SMART? Are sources of verification realistic?

Is the design of the exit strategy realistic?

Is there a good design of the M&E system in place?

Relevance/Appropriateness: A measure of whether interventions are in line with local needs and priorities (as well as donor policies, thus increasing ownership, accountability, and cost-effectiveness)

Were the actions undertaken relevant and appropriate given the local context and needs of the target population?

Was the assistance relevant and appropriate in relation to the practices / culture of the target population?

To what extent were the needs of beneficiaries and stakeholders take in to account in project implementation?

Coherence: A measure of whether interventions are consistent with existing interventions, global and national policies and strategies to ensure consistency, maximize synergies and minimize duplication

Are other stakeholders informed or aware about ACF activities/approach/strategy of the project?

How activities of this project have been integrated with other ACF sectors/ programs in the operational area?

Do project team members feel they are working towards a common goal with respect to other departments (Nutrition, WASH, Health)?

Coverage: A measure of whether interventions meet the need to reach major population groups facing life threatening suffering wherever they are

Were the most affected groups covered with the limitation of the resources available?

Was the geographical coverage of the project appropriate?

Were beneficiaries correctly and fairly identified and targeted?

How the targeting was understood or perceived by local communities?

Were gender and vulnerable populations with in the target community considered in ACF’s assessment/identification of the beneficiary and in the implementation of the project?

Did the project include special components for women, if so; were these systematically designed and monitored during implementation?

Efficiency: A measure of how economically resources/inputs (funds, expertise, time, etc.) are converted to results, includes value for money analysis

Were the resources properly allocated to reach the objectives?

How efficiently are the project implementers utilizing the project’s inputs to conduct activities and achieve the project’s intended results?

How efficient is the overall management set up of the project. or in other words, how is the suitability of management arrangements in place?

20

Is the project being implemented in the most efficient way compared to other eventual alternatives (e.g. cash transfer, inputs purchased and distributed, training and staff)?

Are the project activities being implemented as planned and scheduled?

Effectiveness: A measure of the extent to which the interventions’ objectives were achieved, or are expected to be achieved, taking into account their relative importance and illustrating the effectiveness of ACF approach

What is the quality of the project outputs and/or project activities?

What are the major internal and external factors influencing the achievement or non- achievement of the intended outputs and objectives?

How effectively have the project performance and its outputs and objectives’ indicators being monitored?

How is the adequacy of control mechanisms to limit fraud and corruption? How has the feedback mechanism in place worked? What could be improved?

How was the project team able to adapt to the constraints of the project?

What steps were taken by the implementing Agency (ACF) to ensure that its responses were coordinated with other organizations and local authorities?

To what extent does ACF take part in technical coordination mechanism at all level of project implementation?

Sustainability. A measure of whether the benefits of an activity are likely to continue after donor funding has been withdrawn and project activities officially cease.

How and when does the project intend to withdraw its resources?

What plans are in place to ensure that the achievements of the project are not jeopardized by the time of project phase out?

Was the project assistance provided in a way that took account of the long term context?

How suitable are these plans and are they being implemented?

Did the partnership or local community based organizations established at local level contribute to the sustainability of the work?

To what extent are the project results likely to be sustained in the long term? Likelihood of impact: Early signs of positive and negative, primary and secondary, short, mid and long-term effects produced by an intervention, directly or indirectly, intended or unintended

To what extent is the project contributing to improved socio-economic impact of the Ebola outbreak on the food and nutrition security status of vulnerable households? What does the comparison between baseline and endline suggest?

21

Annex IV: List of people to be interviewed The evaluator will interview the following stakeholders:

Internal

Name Position and Organisation Contact

Sasha Ekanayake Head of Mission - ACF 076601723 / 099901800 hom@sl.missions-acf.org

Isotta Pivato Deputy Country Director - ACF 078151217

dcd@sl.missions-acf.org

Alessandro Dalle Carbonare

FSL HoD - ACF 076563122

fsl-hod@sl.missions-acf.org

Geoffrey Olunga Field Coordinator Moyamba - ACF 079404516

fieldco-moyamba@sl.missions-acf.org

Emmanuel Kemoh FSL Project Manager - ACF 076574038

dpmfsl-mo@sl.missions-acf.org

Samuel Pyne FS HoP - ACF 076830934

fshop-mo@sl.missions-acf.org

Abie Nylander Nut HoP - ACF 078524735

nuthop-mo@sl.missions-acf.org

External

Name Position and Organisation Contact

Gerald A. Sama Moyamba District Agricultural

Officer - Ministry of Agriculture, Forestry and Food Security (MAFFS)

076651260 / 077877842

Kaddie Kandeh District Nutritionist - District Health

Medical Team (DHMT) 078818695

Prof. Herbert Bob Kandeh Chairman, Moyamba District

Council 076808305

PC Foday Momoh Gulama III

Paramount Chief, Kaiyamba Chiefdom

076791498

PC Joseph Alie Kavura Kongormor III

Paramount Chief, Fakunya Chiefdom

076668803

PC Charles Caulker Paramount Chief, Bumpeh

Chiefdom 076777776

PC Jibao Russell Paramount Chief, Lower Banta

Chiefdom 076613854

22

Annex V: Evaluation Criteria Table The evaluator will be expected to use the following table to rank the performance of the overall intervention using the DAC criteria. The table should be included either in the Executive Summary and the Main Body of the report.

Criteria Rating (1 low, 5 high)

Rationale

1 2 3 4 5

Design

Relevance/Appropriateness

Coherence

Coverage

Efficiency

Effectiveness

Sustainability

Likelihood of Impact

23

Annex VI: Good Practice Template The evaluation is expected to provide one (1) key example of Good Practice from the project. This example should relate to the technical area of intervention, either in terms of processes or systems, and should be potentially applicable to other contexts where ACF operates. This example of Good Practice should be presented in the Executive Summary and the Main Body of the report.

Title of Good Practice

(Max 30 words)

Innovative Features & Key Characteristics

(What makes the selected practice different?)

Background of Good Practice

(What was the rationale behind the good practice? What factors/ideas/developments/events lead to this particular practice being adopted? Why and how was it preferable to other alternatives?)

Further explanation of chosen Good Practice

(Elaborate on the features of the good practice chosen. How did the practice work in reality? What did it entail? How was it received by the local communities? What were some of its more important/relevant features? What made it unique?)

Practical/Specific Recommendations for Roll Out

(How can the selected practice be replicated more widely? Can this practice be replicated (in part or in full) by other ACF programmes? What would it take at practical level? What would it take at policy level?)

How could the Good Practice be developed further?

(Outline what steps should be taken for the practice to be improved and for the mission to further capitalise on this good practice)