evaluation of the winter cash programme for lebanese poor ... · 1.2 programme overview- the...
Post on 04-Jun-2018
214 Views
Preview:
TRANSCRIPT
Evaluation of the Winter Cash Programme for Lebanese Poor
Children and their Families
Evaluation Report
Submitted by
Awny Amer Morsy
Independent Monitoring, Research and Evaluation Consultant
Submitted to
UNICEF Lebanon- MENA Region
Jan, 2017
Disclaimer: The opinions expressed in this report are those of the evaluation consultant who
conducted and led the evaluation process that was captured from the views of the project
stakeholders interviewed. The evaluation consultant takes responsibility for any errors reported
herein that are based on its own independent data collection.
Awnyamer22@gmail.com
Egypt
Mobile: 002-01020525500
Skype: awny2210
2
The final evaluation report of the Winter Cash Programme-UNICEF Lebanon. Developed by AwnyAmer- Jan, 2017 Page 2
Acknowledgements
The consultant would like to thank all those who provided their input, feedback and genuine support
during the different phases of the evaluation process.
Most importantly, the consultant would like to thank all the actors and key partners working with
UNICEF Lebanon on the Winter Cash programme, especially the Ministry of Social Affairs (MoSA),
the National Poverty Targeting Program (NPTP) team, the Coordination Management Unit (CMU) at
the Presidency of Council of Ministries (PCM), and the Word Food Programme (WFP). In this
regard, the consultant is grateful for the assistance provided by the programme section at UNICEF as
well as the Planning, Reporting, Innovations and M&E section (PRIME) for the whole-hearted
support during the evaluation process. Appreciation also goes out to those who facilitated the
logistical requirements for the completion of this task and who provided professional guidance to the
evaluation process.
Special thanks are extended to the Directors of the Social Development Centres (SDCs) and their
teams, including the field coordinators and social workers who played a vital role in the evaluation
process, whether in organising and coordinating the facilitation of the Focus Group Discussions
(FGDs) with the direct beneficiaries or by providing their time, insights, views and feedback during
the interviews conducted with them. Their high level of transparency, commitment and
understanding for the evaluation purpose were indispensable to the success of the evaluation.
Finally, the consultant is very grateful to the direct beneficiaries who took part in the evaluation
process and provided their feedback, views, insights and recommendations during the FGDs, which
were facilitated in the four governorates and the sub-areas during the data collection phase of the
evaluation process.
3
The final evaluation report of the Winter Cash Programme-UNICEF Lebanon. Developed by AwnyAmer- Jan, 2017 Page 3
Table of Contents
LIST OF ACRONYMS …………………………………………………………………………………..4
0. Executive summary…………………………………………………………………………………….6
1. OBJECT OF EVALUATION………………………………………………………………………..13
1.1 Introduction/background…………………………………………….…….………………………13
1.2 Programme overview- The expected results chain………………………………………………..13
1.3 Key partners……………………………………………………………………………………….15
2. EVALUATION PURPOSE, OBJECTIVES AND SCOPE OF WORK…………………………16
2.1 Overall goal of evaluation………………………………………………………………………..16
2.2 Specific evaluation objectives……………………………………………………………………16
2.3 Scope of work…………………………………………………………………………………….16
2.4 Evaluation criteria/questions……………………………………………………………………..18
2.5 HRBA/Gender equality & equity principles in the current evaluation……………………….….19
3. EVALUATION METHODOLOGY Methodology………………………………………………..20
3.1 Introduction/overview…………………………………………………………………………….20
3.2 UNEG Norms & Ethical standards……………………………………………………………….20
3.3 Data collection methodologies and tools…………………………………………………………22
3.4 Limitations and challenges of evaluation…………………………………………………………25
4. KEY FINDINGS…………………………………………………………………………………….26
4.1 Relevance………………………………………………………………………………………..26
4.2 Effectiveness…………………………………………………………………………………….35
4.3 Efficiency………………………………………………………………………………………..46
4.4 Sustainability…………………………………………………………………………………….49
4.5 Coordination……………………………………………………………………………………..51
5. CONCLUSION………………………………………………………………………………………54
6. LESSONS LEARNED………………………………………………………………………………55
7. KEY RECOMMENDATIONS………………………………………………………………….….56
Annexes………………………………………………………………………………………………….60
Annex 1: The Terms of Reference (TOR) .................................................................................................. 60
Annex 2: List of stakeholders interviewed ................................................................................................. 68
Annex 3: List of documents consulted ........................................................................................................ 70
Annex 4: Data collection instruments – Methodologies & tools’ guides ................................................... 71
Annex 4.1 - Focus Groups Discussions (FGDs) Guide .............................................................................. 71
Annex 4.2 – Semi-Structured Interview Guide ........................................................................................... 72
Annex 4.3 – Semi-Structured Group Interview Guide................................................................................ 73
Annex 5: Evaluator Bio Data ...................................................................................................................... 75
Annex 6: Evaluation Matrix ........................................................................................................................ 76
Annex 7: Results framework ...................................................................................................................... 80
Annex 8: Summary of the stakeholders’ feedback and insights about the restricted cash modality versus
unrestricted cash modality .......................................................................................................................... 81
Annex 9: Summary overview about the Post Distribution Monitoring (PDM) process ............................ 84
4
The final evaluation report of the Winter Cash Programme-UNICEF Lebanon. Developed by AwnyAmer- Jan, 2017 Page 4
List of Acronyms
ALNAP Active Learning Network for Accountability and Performance
ATM Automated Teller Machine
CMU Central Management Unit
CPD Country Programme Document
CRC Convention on the Rights of the Child
CTP Cash Transfer Programming
DAC Development Assistance Criteria
FGD Focus Group Discussion
GDP Gross Domestic Product
HACT Harmonised Approach to Cash Transfer
HH Household
HRBA Human Rights Based Approach
HRRP Humanitarian Response and Resilience Plan
IT Information Technology
M&E Monitoring and Evaluation
MDG Millennium Development Goal
MEHE Ministry of Education and Higher Education
MoJ Ministry of Justice
MoSA Ministry of Social Affairs
MOU Memorandum of Understanding
NPTP National Poverty Targeting Programme
OECD Organisation for Economic Cooperation and Development
OMR Optical Marketing Recognition
PCM Presidency of the Council of Ministries
PDM Post Distribution Monitoring
PIN Personal Identification Number
PMT Proxy Means Test
SDC Social Development Centre
SDG Sustainable Development Goal
SMS Short Message Service
TOR Terms of Reference
UNEG United Nations Evaluation Group
5
The final evaluation report of the Winter Cash Programme-UNICEF Lebanon. Developed by AwnyAmer- Jan, 2017 Page 5
UNHCR United Nations High Commissioner for Refugees
UNICEF UN Children’s Fund
USD United States Dollar
WFP World Food Programme
6
The final evaluation report of the Winter Cash Programme-UNICEF Lebanon. Developed by AwnyAmer- Jan, 2017 Page 6
0. Executive summary Addressing the seasonal needs of families in Lebanon requires a multi-sectorial approach. During the
harsh winter months, vulnerable families rely on winter cash assistance to address their basic needs, often
at the expense of purchasing other key items. Poor Lebanese and refugee households are highly affected
by inconsistent flows of income, which shift season to season and year to year. Employment is largely
concentrated in the informal jobs market and remains unreliable. In particular, during the winter months,
the availability of work is less which directly impacts the earning ability of vulnerable families (reduced
by 40 per cent as reported by Lebanese poor households1).
Cash Transfer Programming (CTP) has rapidly become an important and often preferred mode of
assistance for humanitarian actors around the world. Organisations operating in areas with conflict-
affected or displaced persons and with functioning markets are increasingly using cash-based assistance
to meet the needs and improve livelihood outcomes of targeted populations.
The Winter Cash programme supports economically vulnerable Lebanese children and their families. It
was designed through a collaborative process between UNICEF, National Poverty Targeting Programme
(NPTP), Ministry of Social Affairs (MoSA), and World Food Programme (WFP), the Office of the Prime
Minister and the Lebanese Presidency of the Council of Ministers (PCM). The objective of the
programme was to introduce cash-based assistance as a pilot to further strengthen the existing social
safety nets and social protection mechanisms in Lebanon. Implementation is the responsibility of the
NPTP, Central Management Unit (CMU) at the presidency of the Council of Ministries and MoSA, with
technical support provided by UNICEF and WFP.
The Winter Cash programme relied on the Proxy Means Test (PMT) score to target the most socio-
economically vulnerable Lebanese households with children. MoSA social workers updated data in the
NPTP database which identifies the most vulnerable households. They visited these households to gather
updated information (specifically the number of children in the household and telephone numbers) which
was then entered into an Optical Marking Recognition (OMR) form, which ranked and selected the most
vulnerable families corresponding to the target of 75,000 children. At the end of this process, 26,052
families were selected as beneficiaries of the Winter Cash programme. A cash grant of USD 40 per child
was transferred to beneficiary households. This value is based on the estimated cost of the winter
clothing kit which was distributed during the 2014/2015 UNICEF winter campaign. The payment was
transferred through ATM cards issued by WFP and Banque Libano-Francaise. Distribution of cards took
place between February 15th and March 10th 2016 by MoSA social workers in 22 selected SDCs, out of a
total 220 existing in the country.
The specific objectives/expected outcomes of the Winter Cash programme were focused on two key objectives - (1) Poor Lebanese children and families provided with the means to cover a portion of their winter expenses to help their children cope with the winter cold in Lebanon exposed to seasonal hazards and unexpected winter emergencies are able to maintain safe access to goods and services; (2) Strengthened delivery mechanisms of the NPTP through the incorporation of new systems such as a Cash-based programme. The overall purpose of the programme
evaluation was to assess (or measure) the outcome and effectiveness of the winter cash programme in
order to strengthen the joint planning, design and implementation capacity of the NPTP, MoSA, PCM and
UNICEF as well as meet the needs of the children by providing appropriate quality assistance in a
dignified manner.
1Poverty, Inequality and Social Protection in Lebanon, January 2016, Oxfam, AUB
7
The final evaluation report of the Winter Cash Programme-UNICEF Lebanon. Developed by AwnyAmer- Jan, 2017 Page 7
The key objectives of the evaluation were guided by OECD/DAC criteria with special focus on assessing
the overall effectiveness of the winter cash modality in Lebanon; and the appropriateness and
acceptance of the programme from the perspective of the target population. Additionally, the evaluation
aimed to assess the efficiency of joint collaboration among the different actors in addition to providing
concrete recommendations for the improvement of planning and implementation of joint programme
targeting for vulnerable Lebanese children.
The key programme beneficiaries (vulnerable Lebanese poor children and their families) were selected
from the following targeted working areas: Bekaa, Mount Lebanon, Nabatieh, Akkar, Tripoli and South.
The key target audience of the evaluation was the NPTP and MoSA as well as other stakeholders
including the PCM, CMU, SDC directors, social workers & field coordinators. WFP in addition to the
households under “Hala” programme having benefited from the programme; children boys & girls and
their families.
Evaluation Methodology:
In order to address the key questions outlined in the TOR, the consultant used formative evaluation
practices combined with participatory approaches, depending on the type and category of the different
stakeholders (including programme beneficiaries). The evaluation process was initiated by carrying out a
desk review of ten of the key documents outlined in the TOR. Thirteen semi-structured individual
interviews were conducted with UNICEF staff and key partners (Five males and eight females). Seven
semi-structured group interviews were conducted with a total of 24 participants. The participatory
approach was applied by conducting Focus Group Discussions (FGDs) to capture the outcomes and key
changes impacting the quality of life of the targeted groups. A total of 90 stakeholders were interviewed
during the evaluation process in addition to 101 who were interviewed during the Post Distribution
Monitoring (PDM) process which was conducted in 2015. Therefore, a total of 191 individuals were
interviewed in order to assess the winter cash programme.
Additionally, the survey that was conducted by a third party during the PDM, using Computer Assisted
Telephone and face to face interviews, was also used in this evaluation as part of the triangulation process
to validate the data collected and the key findings captured. A total of 457 out of 500 beneficiaries were
interviewed in Bekaa, Mount Lebanon, North and South Lebanon.
Key Findings and Conclusions
Relevance
The Winter Cash programme focused mainly on addressing the rights and the needs of Lebanese poor
children and their families in targeted areas across Lebanon. Overall, the evaluation found that the
programme intervention reached the most socio-economically vulnerable Lebanese households and met
most of their basic winter needs.
The evaluation revealed the absence of the capacity building component from programme intervention
strategies. Despite the fact that the Winter Cash programme is a pilot project, this discovery led to the
recommendation for the inclusion of capacity development in future programmes. Capacity needs to be
strengthened to ensure fairness in the provision of cash assistance.
The cash grant of USD 40 per child was found to be insufficient to meet the basic needs of the most
socio-economically vulnerable Lebanese households. Moreover, the programme excluded a large number
of vulnerable families who were also in need of financial assistance due to their harsh economic
conditions owing to the pilot nature of the Winter Cash programme in addition to the available financial
resources.
8
The final evaluation report of the Winter Cash Programme-UNICEF Lebanon. Developed by AwnyAmer- Jan, 2017 Page 8
The selection process and criteria were developed jointly by the NPTP, UNICEF, and CMU/PCM.
However, four major pitfalls were reported – firstly, there lacked clear documentation and understanding
of how the selection process worked. Secondly social workers and field coordinators did not participate in
setting the selection criteria or other relevant questionnaires. Thirdly, the data used in the selection
process relied on outdated sources (as far back as 2011) in addition to omitting some of the important
variables and criteria in the selection process. Lastly, the age of selected children (aged up to 15 years)
was the key selection criterion. The programme did not consider gender as one of the selection variables
and therefore a gender perspective was not incorporated in the targeting process. Although the payment
list of beneficiaries includes the sex of the ATM cardholder, there exists no disaggregated data for the
children supported per family. As a result, no sex-disaggregated data for children who benefitted from the
programme is available.
The programme was successful in identifying the most relevant partners for the project, with each of the
five partners (UNICEF, MoSA, CMU, NPTP and WFP) playing a significant role in bringing an added
value to the implementation of the programme.
Effectiveness
The evaluation revealed that the majority of beneficiaries who took part in the FGDs expressed a high
degree of satisfaction with their participation in and benefits from the Winter Cash programme.
Dissatisfaction regarding the value of the cash transfer was reported amongst a number of beneficiaries
who believed it to be insufficient for meeting their needs. Others suggested that the assistance should be
recurrent on a monthly or quarterly basis throughout the year. The evaluation also found that the
programme did not take into account the special needs of families who have children with disabilities and
did not adequately consider other vulnerable groups such as working women, orphans and families who
do not have children.
Due to a delay in the implementation of the project, ATM cards were not distributed until March instead
of during the previous winter. This hindered the achievement of the main planned objective of the
programme which was “to provide the aid in the winter season” in order to reduce the vulnerability of
Lebanese poor families. UNICEF, with the support of other key partners, developed a well-structured
system to track the distribution cohorts, delivery system process, hotline utilisation, and question and
answer mechanism, and complaint mechanism, which are guided and led by WFP. However, there is a
lack of monitoring and post follow-up activities by social workers and field coordinators to assess the
changes in the lives of the beneficiaries as a result of their participation in the programme and collection
of information on the spending patterns of the cash amounts.
Some beneficiary families (particularly in the Akkar and Mount Lebanon governorates) received their
ATM cards without a pin code. To respond to this issue, a complaint mechanism was introduced.
However, it was weak and the programme ended before the issues were resolved.
The distribution mechanism was considered as one of the key challenges in the Winter Cash programme
especially for those living far from SDC selected distribution points and who incurred additional
transportation fees.
Communication issues relating to the SMS notification which alerted beneficiaries to the distribution of
ATM cards were reported. Many selected families could not understand the SMS for three primary
reasons: (1) They were illiterate, (2) They presumed the message to be an advertisement, and (3) The
model of their phone could not read Arabic messages.
Although the grant assisted families in securing some basic needs for their children, it cannot be
substantiated that the cash grant of USD40 per child resulted in significant changes, or directly impacted
on the lives of the beneficiaries.
9
The final evaluation report of the Winter Cash Programme-UNICEF Lebanon. Developed by AwnyAmer- Jan, 2017 Page 9
Based on these findings, no clear conclusion can be drawn on the success of unrestricted cash transfer as
opposed to restricted cash transfer. Some stated their stance based on their points of view, insights and
previous experiences from other similar programmes with poor vulnerable groups. All partners including
WFP, NPTP, MoSA staff, six SDC directors, five field coordinators as well as 14 out of the 20 social
workers interviewed during the evaluation process expressed their preference for restricted cash transfer
in e-voucher form in exchange for food, clothes or fuel. They indicated that if it was not possible for the
unrestricted cash transfer assistance to be recurrent, it would be better to have the restricted e-vouchers
for food, fuel, clothes or other essentials.
On the other hand, the majority of direct beneficiaries and few of the social workers and field
coordinators interviewed during the evaluation process expressed their preference for
unconditional/unrestricted cash transfers. This corresponds with the results of the PDM survey showing
that the majority of beneficiaries were satisfied with the cash assistance modality (63 per cent)2 compared
to those who were very satisfied (32 per cent) and those who were not satisfied (5 per cent). MoSA staff
and the direct beneficiaries interviewed during the PDM FGDs indicated that the cash modality was
perceived as a better tool, allowing beneficiaries to purchase the essentials most applicable for their
personal needs. Spending patterns of the priority five spending areas3 among the participants interviewed
are highlighted as follows:were: Winter Household expenses (56 per cent), food (55 per cent), education
fees-including the transporation fees to school (30 per cent), repaying of debts (20 per cent), health
expenses (20 per cent), electercity bills (12 per cent), heating- fuel or others (11 per cent) and rent (11 per
cent).
Capacity building activities are crucial for programme development and investment in enhancing the
skills and abilities of social workers in the field is crucial. Not only will this lead to improvement of
programme delivery, but it will also promote ownership of the NPTP.
Efficiency
The successful collaboration between implementing partners demonstrated an excellent example of the
partnerships model. Each of the key stakeholders were well-informed, roles and responsibilities were
agreed upon in advance, and specific measures were set in order to maintain cost-efficiency during each
phase of the programme. Most of the participants stated that they consider the unrestricted cash transfer as
the preferred modality of assistance due to its cost-efficiency, in comparison to the restricted cash transfer
(e-vouchers). The prevailing opinion of beneficiaries, captured during the FGDs, was that the cash value
meant that they felt more respected and were treated “like human beings”.
As the beneficiaries could withdraw the cash grant directly, using ATM cards, the need to engage a third
party was null, thus reducing costs and risk of wasted resources. The programme was less efficient
during the verification phase, which took much longer than planned leading to the delayed delivery of the
cash grant.
Sustainability The absence of a sustainable strategy to address the graduation approach in some of the targeted areas
was unveiled during the evaluation. Lack of skills training and activities aimed at improving
beneficiaries’ ability to secure employment opportunities were not evident. Although beneficiaries
learned how to use the ATMs, illiteracy impacted negatively on the results of this training.
2Statistics Lebanon Ltd_ Results of the survey findings _Winter Cash Programme-PDM Report-2016_Slide# 72 3Statistics Lebanon Ltd_ Results of the survey findings _Winter Cash Programme-PDM Report-2016_Slide# 69
10
The final evaluation report of the Winter Cash Programme-UNICEF Lebanon. Developed by AwnyAmer- Jan, 2017 Page 10
Coordination
Coordination during each implementation phase was aided by communication between all partners.
Weekly meetings were organised and attending by representatives from each of the five partners to ensure
follow up on the implementation of the programme, resolution of problems in a timely manner, and
maintain effective communication between staff. This mechanism will be referred hereafter as the
“steering committee”. However, they lacked a well-designed management and coordination system with
clearly defined roles and responsibilities of each partner.
Although the SMS as a communication mechanism was very well received by the respondents, MoSA
staff did not approve this mode of communication due to the belief that beneficiaries did not read or
understand the messages. The MoSA staff did not provide full approval of the brochures and flyers being
distributed to beneficiaries for reasons including the fact that social workers had arrived late in this
process and sometimes during the phase of distribution of the ATM cards which means that it was not
helpful and came at the wrong time.
The complaint mechanism was identified as one of the gaps in the programme. Issues such as the absence
of ATM pin codes, deposit of incorrect cash amounts and other technical issues, were not resolved
through this structure. Another key challenge in the coordination of the programme was the poor selection
of distribution centres which resulted in lengthy and difficult travel conditions for many beneficiaries.
Key Recommendations
Below are the key recommendations following the evaluation:
Programme approach, design and interventions Adopting a multi-dimensional approach when developing cash-based programme interventions
within NPTP is highly recommended. This approach comprises an integrated and comprehensive
package of activities and does not focus only on the cash transfer amount as a single activity. This
means that programme developers during the design phase should consider the “Cash for
what?” question, as the cash modality should not be seen as an end in itself or as a stand-alone
intervention.
Special attention should be given to developing a separate, well-structured programme proposal
or concept note for the Winter Cash programme, considering that it is part of the Country
Programme Document (CPD).
A gender sensitive strategy should be devised for the programme. The strategy should state how
the programme will be guided by equality and equity measures during each phase and should
outline the importance of disaggregated data for all the beneficiaries.
The target age group of children who are eligible to benefit from cash transfer assistance should
be revised and extended to include children aged up to 18 years. (In compliance with the
definition of the child adopted by UNICEF).
The cash transfer amount should be revised, ensuring that it is sufficient to secure winter basic
needs and also considers the transportation fees of those who must travel to selected distribution
centres.
Capacity building Investment in capacity building activities for field coordinators and social workers is highly
recommended. Enhancing their abilities and skills in the areas of communication, field work,
monitoring, mentoring, follow-up, data collection, data integrity, data validity, child rights, and
research ethics will lead to greater institutional and human resource sustainability. This could be
organised in the form of a four to five days training workshop to cover basic information and
skills which they can also apply to other projects managed by the NPTP.
Awareness raising sessions should be provided for parents to educate them on how to spend the
cash grant appropriately, focusing on the essential needs of their children.
11
The final evaluation report of the Winter Cash Programme-UNICEF Lebanon. Developed by AwnyAmer- Jan, 2017 Page 11
Selection process In order to gather accurate and updated baseline data, it is recommended to conduct a
vulnerability assessment to inform on specific vulnerability and welfare indictors of families and
communities registered in the NPTP. The information collected will provide important data
which is vital for implementation of similar programmes in addition to the winter cash
programme.
Promoting greater involvement of field coordinators, social workers and other stakeholders in the
selection process and development of selection criteria is recommended in order to strengthen
their sense of cooperation - not only being seen as implementers but as implementing partners.
The questionnaire currently does not include the minimum assessment requirements and
variables. Revision of the questionnaire tool will enable the appropriate identification of
vulnerable poor families referring to the vulnerability variables.
Coordination A well designed communication and coordination plan is recommended. The plan should include
specific milestones and measures outlining the specific roles and responsibilities to guide each
phase of the programme: planning, distribution, monitoring and post-delivery activities.
Establishing unified guidelines and principles before the implementation phase will help to avoid
addressing issues which arise in an ad hoc manner.
Cash modality Joint partnership with other organisations, such as WFP, where each partner complements the
other to address the needs of the NPTP beneficiaries is recommended. This means that UNICEF,
for example, can provide cash assistance while WFP can provide conditioned e-vouchers under
the same programme in a systematic way throughout the year, combining both restricted and
unrestricted cash transfer modalities.
It is highly recommended that NPTP adopts the unrestricted cash transfer, which is considered to
be the most favourable cash modality by a majority of beneficiaries. Certain conditions are
encouraged to be incorporated including a revision of the cash grant amount; adoption of a
recurrent policy throughout the year and replacement of cash grants with an e-food voucher on
quarterly basis.
Monitoring & Evaluation A well-structured M&E system should be developed comprising various methodologies and tools,
as well as community reflection tools which are linked to a combination of qualitative and
quantitative indicators.
Best practices should be supported, documented and shared regularly for enhanced learning
purposes.
Special attention should be given to increasing the involvement of field coordinators and social
workers in the different phases of programme implementation.
Joint partnerships
Developing joint partnerships with other organisations working with the same vulnerable families
registered in the targeted areas is strongly recommended. Through collaboration and utilisation of
data collected during the vulnerability assessment, integrated, joint interventions can be
developed reducing duplication of efforts.
The evaluation suggests investigating the possibility of targeting the winter needs of groups
whose profiles do not meet the current selection criteria or the mandate of partner organisations.
12
The final evaluation report of the Winter Cash Programme-UNICEF Lebanon. Developed by AwnyAmer- Jan, 2017 Page 12
Scaling up opportunities
In order to promote the institutional sustainability of the NPTP and strengthen its ownership of other
similar programmes, special attention should be given to adopt a well-structured strategy that includes
the following sub-strategies:
Develop a comprehensive, well-designed capacity building training in advance of project
implementation.
Provide awareness sessions for targeted parents and/or caregivers to ensure the correct use of cash
transfer assistance to secure the basic needs of their families during the winter season.
Conduct a vulnerability assessment to collect accurate and updated data on vulnerability and
welfare variables.
Promote collaboration with the other organisations working with the same targeted groups to
maximise the use of resources.
Set clear and structured communication and coordination mechanisms for all stakeholders
including key partners and the participating bank. This will increase the efficiency of the
programme; improve timely delivery of activities and help to identify and address challenges
promptly.
Develop a well-structured monitoring and quality assurance framework with clear mechanisms,
methodologies and tools to track the progress during the project lifecycle.
13
The final evaluation report of the Winter Cash Programme-UNICEF Lebanon. Developed by AwnyAmer- Jan, 2017 Page 13
1. Objective of the Evaluation
1.1 Introduction/Background
Today, Lebanon’s four million citizens struggle to withstand the social, economic and political
impact of the Syrian conflict. A stagnant economy, coupled with the threat of insecurity, increases
fears that children and families living in poverty will sink further into deprivation. As a consequence
of the on-going Syrian crisis, population augmentation, of both host and displaced communities has
resulted in gradually shrinking spaces for livelihood and income-generation which significantly
impacts the ability for vulnerable children and their families to secure their basic needs.
In 2016, President Michel Aoun was elected as Head of State after a 29-month political deadlock and
presidential vacuum. Despite a magnitude of challenges, the Government with UNICEF Lebanon’s
financial and technical investments in strengthening public institutions and their delivery systems
contributed to over one million vulnerable children being reached.
Nearly half of those affected by the Syrian crisis are children and adolescents and are currently
growing up at risk, deprived, and with an acute need for basic services and protection4. Vulnerable
families are subject to seasonal hazards and a substantial proportion of these have had to increase
their spending patterns to cover winter needs such as heating fuel, shelter repairs and winter clothing.
Addressing the seasonal needs of families in Lebanon requires a multi-sectorial approach. During the
harsh winter months, vulnerable families rely on the winter cash assistance to address their basic
needs, often at the expense of purchasing other key items. Poor Lebanese and refugee households are
highly affected by inconsistent flows of income, which shift season to season and year to year.
Employment is largely concentrated in the informal jobs market and remains unreliable. In particular,
during the winter months, the availability of work is less which directly impacts the earning ability of
vulnerable families (reduced by 40 per cent as reported by Lebanese poor households5).
Cash Transfer Programming (CTP) has rapidly become an important and often preferred mode of
assistance for humanitarian actors around the world. Organisations operating in areas with conflict-
affected or displaced persons and with functioning markets are increasingly using cash-based
assistance to meet the needs and improve livelihood outcomes of targeted populations.
1.2 Programme overview
The Winter Cash programme supports economically vulnerable Lebanese children and their families.
It was designed through a collaborative process between UNICEF, National Poverty Targeting
Programme (NPTP), Ministry of Social Affairs (MoSA), and World Food Programme (WFP), the
Office of the Prime Minister and the Lebanese Presidency of the Council of Ministers (PCM). The
objective of the programme was to introduce cash-based assistance as a pilot to further strengthen the
existing social safety nets and social protection mechanisms in Lebanon. A cash-based intervention
was considered to be the most relevant and effective mechanism for reaching 75,000 vulnerable
children across Lebanon. Implementation is the responsibility of the NPTP, Central Management
Unit (CMU) at the presidency of the Council of Ministries and MoSA, with technical support
provided by UNICEF and WFP. The programme was designed to be implemented by the NPTP, the
4LCRP 2015-2016 p.9 5Poverty, Inequality and Social Protection in Lebanon, January 2016, Oxfam, AUB
14
The final evaluation report of the Winter Cash Programme-UNICEF Lebanon. Developed by AwnyAmer- Jan, 2017 Page 14
Central Management Unit (CMU) at the presidency of the Council of Ministries and Ministry of
Social Affairs (MoSA) with technical support from UNICEF and WFP. The NPTP established in
2011, aims to support vulnerable Lebanese families in meeting their most urgent needs. To date,
86,000 Lebanese households have been identified as living below the poverty line (USD3.84/day), of
which nearly one-third (25,000 households) live in absolute poverty (below USD2/day)6. The NPTP
provides health and education subsidies to eligible beneficiaries, and food assistance through e-
vouchers is provided to 10,000 (27,000 extremely poor individuals).
The NPTP, PCM and the World Bank developed a Proxy Means Test (PMT) formula to target NPTP
beneficiaries. Inclusion of applicant households was based on a proxy score that calculates the
welfare of individuals by estimating their consumption per day as a proxy measure of income per
day. The test included questions to assess the applicant’s standard of living, such as the employment
status, level of education, marital status, physical ability, housing conditions, assets owned and their
geographical location. It is agreed that if the score is under a certain determined threshold that
corresponds to the Lower Poverty Line, then the household was included as a beneficiary of the
NPTP. This was followed by providing the head of the household or the applicant with a “Halla”
card, which can be used to access the education and health services at any of the 220 SDCs, and local
clinics.
The Winter Cash programme relied on the Proxy Means Test (PMT) score to target the most socio-
economically vulnerable Lebanese households with children. MoSA social workers updated data in
the NPTP database which identifies the most vulnerable households. They visited these households
to gather updated information (specifically the number of children in the household and telephone
numbers) which was then entered into an Optical Marking Recognition (OMR) form, which ranked
and selected the most vulnerable families corresponding to the target of 75,000 children. At the end
of this process, 26,052 families were selected as beneficiaries of the Winter Cash Programme. A cash
grant of USD 407 per child was transferred to beneficiary households. This value is based on the
estimated cost of the winter clothing kit which was distributed during the 2014/2015 UNICEF winter
campaign. The payment was transferred through ATM cards issued by WFP and Banque Libano-
Francaise. Distribution of cards took place between February 15th and March 10th 2016 by MoSA
social workers in 22 selected SDCs, out of a total 220 existing in the country.
Expected outcomes/results
Subsequent to the Winter Cash programme project proposal, the Humanitarian Response and
Resilience Plan 2016 (HRRP) and other project documents, the following key expected outcomes
were developed: which targets
Key Objectives/ Expected Results (medium and short-term results)
Outcome 1:
Lebanese children and the families vulnerable to seasonal hazards and unexpected winter
emergencies are able to maintain safe access to goods and services.
Expected Outputs:
1.1: Households in the targeted areas at risk of seasonal and unexpected hazards survive without
6 Poverty, Inequality and Social Protection in Lebanon, January 2016, Oxfam, AUB 7 The US$40 was the monetized amount of the children’s winter clothing kits and e-vouchers which UNICEF had provided to children of NPTP
support families in previous years.
15
The final evaluation report of the Winter Cash Programme-UNICEF Lebanon. Developed by AwnyAmer- Jan, 2017 Page 15
adverse effects.
Planned activities
1.1.1: Provide cash grants for seasonal hazards affected households (USD40/child in the targeted
families)
Outcome 2:
Strengthened delivery mechanisms for the NPTP via the incorporation of new systems: Cash-
based programme
Expected Outputs:
2.1: Strengthened child-focused delivery mechanisms for NPTP via the incorporation of new child-
sensitive systems, including cash-based programmes
Planned activities
2.1.2: Conduct a joint study with MoSA/NPTP on outcomes and impacts of winter cash-based
programme for socio-economically vulnerable children and their households.
1.3 Key programme stakeholders – including key partners
Table 1 below shows a brief mapping of the stakeholders involved in the programme. It lists the
different key actors/stakeholders – including the key partners who were targeted during the data
collection phase.
# Key stakeholder Actors involved
1 UNICEF Lebanon
staff
UNICEF Programme Specialist Cash and Vouchers
UNICEF Winter Focal Points
UNICEF Chief of Field Operations
2 NPTP Director of the NPTP
NPTP Business IT
SDC directors
3 MoSA MoSA Winter Cash Transfer Programme Focal Point
MoSA Field work coordinators
MoSA social workers
Adviser of the Minister of Social Affairs
4 PCM/CMU CMU Director
PCM/CMU Statistician
5 WFP WFP Cash and Vouchers Unit Programme Associate(s)
6 Direct beneficiaries Sampled groups from the poor vulnerable families (parents and
caregivers)
Table1. Key Stakeholders
16
The final evaluation report of the Winter Cash Programme-UNICEF Lebanon. Developed by AwnyAmer- Jan, 2017 Page 16
2. Evaluation Purpose, Objectives and Scope of Work
Referring to the agreed scope of work with UNICEF & other stakeholders, UNICEF commissioned
the consultant to conduct “The final evaluation of the winter cash transfer programme for
Lebanese poor children & their families”.
2.1 Overall Goal of Evaluation
The overall goal of the programme evaluation is to assess (or measure) the outcome and
effectiveness of the Winter Cash programme in order to strengthen the joint planning, design and
implementation capacity of the NPTP, MoSA, PCM and UNICEF as well as meet the needs of the
children by providing appropriate quality assistance in a dignified manner.
2.2 Specific Evaluation Objectives
In order to provide a robust evaluation of the joint NPTP / UNICEF winter cash response, the
specific objectives of the programme evaluation are as follows:
Assess the overall effectiveness of the Winter Cash programme modality in Lebanon.
Assess the appropriateness and acceptance of the Winter Cash programme modality from
the perspective of the target population.
Assess the joint collaboration of the overall implementation of the Winter Cash programme.
Provide concrete recommendations to improve planning and implementation of joint programmes
and deliver optimal results for the most economically vulnerable Lebanese children.
2.3 Scope of Work
In order to achieve the specific objectives, the evaluation covered the entirety of the programme
including the following components (in reference to the project cycle process):
The design & planning phase: This includes the selection process, payment modality and
repartition of the duties and responsibilities among the different actors as well as the planning
phase.
The Implementation phase: This includes all training conducted, distribution of ATM
cards, feedback mechanisms and monitoring.
It is important to note that the scope of work in the current evaluation focused mainly on poor
vulnerable Lebanese families only and did not consider Syrian and Palestine refugees addressed
under the winter HRRP-2016.
17
The final evaluation report of the Winter Cash Programme-UNICEF Lebanon. Developed by AwnyAmer- Jan, 2017 Page 17
The geographic scope of the evaluation:
The evaluation focused primarily on the targeted areas of programme implementation shown in Table
2 below.
The key programme beneficiaries (vulnerable Lebanese poor children & their families) were selected
from the following targeted working areas8:
S Governorate Cazas Targeted # Of SDCs involved
1 Beirut Beirut 4
Sub-Total 4
2 Bekaa Baalbek 9
Hermel 1
Rashaya 4
West Bekaa 4
Zahle 3
Sub-Total 21
3 Mount-Lebanon Aley 6
Baabda 6
Chouf 10
El Metn 5
Jbeil 4
Kesrouan 5
Sub-Total 36
4 Nabatieh Hasbaya 1
Marjayoun 3
Nabatiyeh 3
Sub-Total 7
5 Akkar Akkar 10
Batroun 4
Bchare 1
Koura 2
Sub-Total 17
6 Tripoli Minieh-Dannieh 3
Tripoli 3
Zgharta 2
Sub-Total 8
6 South Bint Hbeil 4
Jezzine 1
Saida 5
Sour 5
Sub-Total 15
Total 26 108
Table2. Geographic scope of the evaluation
8List of Distribution of UNICEF_NPTP beneficiaries by Governorate_Gaza_SDC.
18
The final evaluation report of the Winter Cash Programme-UNICEF Lebanon. Developed by AwnyAmer- Jan, 2017 Page 18
Time period covered by the evaluation: The evaluation covered the period between August 2015
and May 2016, starting from the first meetings held between PCM, MoSA, WFP and UNICEF
pertaining to the design of the programme and ending with the last Post Distribution Monitoring
(PDM) process.
2.4 Evaluation Criteria & Questions
With reference to the TOR and the approved inception report, the evaluation is primarily based on
the DAC criteria with a special focus on Relevance (including appropriateness), Effectiveness
(including acceptance), Efficiency, Sustainability and Coordination reflected in the key questions
outlined in Table3 below.
Evaluation
criteria Key questions / areas of discussion
Relevance
(including
Appropriateness)
1. To what extent were the project intervention strategies, project design and
approaches appropriate and relevant to promoting the intended objectives of
the programme? And to what extent were the needs of the most socio-
economically vulnerable Lebanese households met?
2. To what extent were the criteria of the selection process developed to ensure
that the most vulnerable and poor children were reached and benefited
properly from the programme?
3. To what extent has the programme been successful in identifying the most
relevant partner/actors for this project and what was the added value of these
partnerships?
4. How gender sensitive was the programme when reaching the most
vulnerable children?
5. Do the current Log frame and/or results framework and relevant indicators
in the Winter Cash programme document need improvement to better serve
the programme objectives?
Effectiveness
6. To what extent were the vulnerable children and their families targeted by
the Winter Cash programme reached (as planned)? How satisfied were they
about the programme?
7. To what extent did the programme have an impact on beneficiaries in
improving access to essential goods and services of their choice in a safe,
dignified, and empowered manner? (Key changes affecting the lives of
beneficiaries as result of the programme)?
8. Are there other unintended or unforeseen changes evident in the lives of the
targeted groups of poor children & their families? Are there any unintended
beneficiaries of the programme?
19
The final evaluation report of the Winter Cash Programme-UNICEF Lebanon. Developed by AwnyAmer- Jan, 2017 Page 19
9. What was the level of acceptance of the cash modality as a tool to support
poor Lebanese households to cover their winter needs (winterisation) from
the point of view of all the different actors: MoSA, NPTP, SDCs, field staff
and the beneficiaries?
10. To what extent were the capacities of the different actors involved -
particularly the NPTP and MoSA - adequate and efficient to ensure the
quality of the programme implementation? What can be improved in this
regard?
11. To what extent does the programme have an effective monitoring and
evaluation mechanism (including reporting, quality assurance and
reflection)? Are there proper processes on different levels (UNICEF, NPTP,
and CMU/PMC) in place? How has the programme used information
generated to inform and make programmatic adjustments & corrective
actions during the life of the project?
12. To what extent did the project consider gender equality in the programme
implementation – including the selection process (e.g. is USD40 an
equitable amount for both boys and girls? On what basis was the amount
calculated and does it take into account gender-based requirements?)
Efficiency 13. To what extent was the Winter Cash programme cost-efficient – including
optimal usage of programme resources & time efficiency as opposed to
other modalities/programmes that could be designed to cover the same
needs of the Lebanese vulnerable poor children and their families?
Sustainability 14. To what extent did the programme equip service providers (NPTP/MoSA)
and beneficiaries with the required capacities (skills, knowledge and
sustainability), resources (connections) and confidence (institutional and
financial sustainability) to sustain the programme implementation?
Coordination
15. How effective was the management and coordination system - as part of
joint programme partnership between UNICEF, NPTP, MoSA, CMU/PCM
and WFP - in achieving results for the target population?
Table3. Evaluation Criteria and Key questions / areas of discussion
2.5. HRBA, Gender Equality and Equity Principles in the Current Evaluation The current evaluation took into consideration the following key Human Rights Based Approach
(HRBA), gender equality and equity principles to ensure that they were mainstreamed in the
evaluation process as follows9:
9 United Nations Evaluation Group (2014). Integrating Human Rights and Gender Equality in Evaluations. New York: UNEG- page 32.
20
The final evaluation report of the Winter Cash Programme-UNICEF Lebanon. Developed by AwnyAmer- Jan, 2017 Page 20
Inclusion: The evaluation investigated which groups benefitted from the programme and
targeted them during the assessment process. The evaluation developed disaggregated data
of all the groups interviewed as shown in Annex 2 of this report. Participation: The evaluation adopted and followed a participatory evaluation approach,
applying methodologies and data collection tools engaging the programme stakeholders in a
consultative manner. The evaluation measured the level of their participation and
involvement in the project design, planning, and monitoring activities. Fair power relations: The evaluation focused specifically on assessing the degree to which
power relations among the different programme stakeholders and implementers supported or
undermined programme results. Additionally, the evaluation considered the status of groups
in the targeted population within the programme.
3. Evaluation Methodology
3.1 Overview
With reference to the TOR, the evaluation process was guided by the following four internationally
recognised standards:
Utility – Evaluation findings and recommendations will serve the different stakeholders.
Feasibility – Realistic, prudent, diplomatic and frugal.
Propriety – Conducted legally, ethically and with due regard for the welfare of those involved in the
evaluation process, as well as those affected by its results.
Accuracy – Reveal and convey technically adequate information about the features that determine
the worth or merit of the programme being analysed in the evaluation process.
3.2 UNEG Norms and Ethical Standards
On the other hand, the evaluation was guided in all its processes with the key UNEG norms and
ethical standards10 as follows:
Intentionality of evaluation - by ensuring the utility and necessity of the information and
results captured from the current evaluation for the different stakeholders and target
audiences.
Key obligations11 to the evaluator:
Independence - both behavioural and organisational independence.
Impartiality- objectivity, professional integrity and absence of bias.
Credibility – transparent evaluation processes and inclusive approaches involving relevant
stakeholders.
Conflict of interest- Ensure neutrality in all activities related to the programme design,
planning and implementation.
Honesty and integrity
Accountability
10Linda G. MorraImas&Ry C. Rist: The Road to Results- The world bank – Washington- 2009- Page 509-510 11United Nations Evaluation Group (2016). Norms and Standards for Evaluation. New York: UNEG- page 11-12.
21
The final evaluation report of the Winter Cash Programme-UNICEF Lebanon. Developed by AwnyAmer- Jan, 2017 Page 21
Key obligations to participants:
Respect for dignity & diversity
Rights
Confidentiality
Avoidance of harm
In order to address the key questions outlined in the TOR, the consultant used formative evaluation
practices combined with participatory approaches when engaging the different stakeholders and
beneficiaries.
As the programme was a pilot project, the scope of the evaluation was mainly focused on the
learning purpose rather than applying it for accountability processes.
The evaluation process was initiated by conducting a desk review of ten key documents
including: the TOR; basic assistance/winter HRRP 2016; implementation plan; PDM final
products; distribution documents; forms and sheets; lessons learnt document as part of the
PDM process; complaint mechanism document; and NPTP card management in addition to
the respective questionnaires. (See Annex 3).
Thirteen semi-structured individual interviews were conducted with relevant UNICEF staff
and key partners. (Five males and eight females).
Seven semi-structured group interviews were conducted with a total of 24 participants (Five
males and nineteen females) with relevant MoSA staff in the five targeted working areas
located in four governorates.
One group interview took place with relevant WFP staff.
The participatory approach was applied by conducting FGDs to capture the outcomes of
programme activities and any significant impact on the quality of life of the beneficiaries.
Five FGDs were facilitated with five sample groups of beneficiaries from five working areas
affiliated to five SDCs located in four governorates with a total of 53 parents & caregivers.
The current evaluation used eight FGDs (men and women) and four key informant interviews
(with SDC directors, social workers and field coordinators) which were conducted last year
during the Post Distribution Monitoring process.
Ninety stakeholders were interviewed in the current evaluation process in addition to the 101
who were interviewed during the Post Distribution Monitoring process last year. A total of
191 were therefore interviewed to assess the Winter Cash assistance programme. Those
interviewed were involved in the programme either as partners or as direct programme
beneficiaries (Refer to Annex 2 for more details).
Additionally, the survey that was conducted by a third party during the Post Distribution
Monitoring - using Computer Assisted Telephone and face to face interviews - was also used
in this evaluation. A total of 457 out of 500 beneficiaries were interviewed in Bekaa, Mount
Lebanon, North and South. Triangulating the data to draw comparisons from the information
was essential in order to validate the findings.
Table4 below shows the number of stakeholders interviewed and consulted during the current
evaluation process disaggregated by sex and type of evaluation methodology/tool. The details of this
list are presented in Annex 2.
22
The final evaluation report of the Winter Cash Programme-UNICEF Lebanon. Developed by AwnyAmer- Jan, 2017 Page 22
S Type of evaluation
methodology/tool
# of
interviews/FGDs
Classification of participants by
sex
Male Female Total
1 Individual semi-structured interviews 13 5 8 13
2 Group semi-structured interviews 7 5 19 24
3 Focus Group Discussions 5 16 37 53
Total 25 26 64 90
Table4. Type of evaluation methodology/tool
3.3 Data Collection Methodologies and Tools
3.3.1. SOURCES OF DATA COLLECTION
The following are the key sources of information referred to in the current evaluation process:
- Information collected and captured from FGDs with direct beneficiaries.
- Information collected from the individual and group interviews conducted with the key
stakeholders including the main partners of the Winter Cash Programme.
- Post Distribution Monitoring (PDM) findings.
The following key criteria were applied in order to select the sample of SDCs that were to
facilitate the interviews and FGDs:
The SDCs which have the highest number of the beneficiaries referring to the above actual
distribution schedule.
Ensure that the sample of SDCs, representing each of the rural/semi-rural/urban/semi-urban
and/or slums, is useful for analysing the spending patterns and sufficiency of the cash transfer
amount to secure their winter basic needs.
This sample was developed after the initial recommendation from UNICEF o and the adviser of the
NPTP and MoSA staff.
The following criteria were applied for the selection of participants in FGDs: The
participants were selected from the beneficiaries of the Winter Cash programme in each
SDC, ensuring that all targeted working areas were represented under the SDC.
The number of children per household who benefitted from the Winter Cash programme was
considered during the selection process to ensure a diverse group of beneficiaries from those
with only one child to those with several children aged up to 15 years. This proved to be
helpful in capturing different views during the evaluation process.
Access to targeted working areas was a factor due to security issues in some areas.
Other basic criteria are outlined in the FGD guide included in Annex 4.
23
The final evaluation report of the Winter Cash Programme-UNICEF Lebanon. Developed by AwnyAmer- Jan, 2017 Page 23
3.3.2. DATA COLLECTION TOOLS
A. SEMI-STRUCTURED INTERVIEWS (GROUP AND INDIVIDUAL INTERVIEWS)
The evaluation included a series of semi-structured interviews with key partners and stakeholders
involved in the programme implementation. These interviews were conducted with NPTP, MoSA,
WFP, SDCs, field coordinators, Social Workers, and CMU/PMC in addition to UNICEF. Participants
were selected based on TOR requirements and consultations with UNICEF and the desk review
findings. The selected individuals were interviewed for their first-hand knowledge about the
programme and their involvement in the programme implementation itself.
The interviews were roughly structured with a list of topics for discussion, and an interview guide to
allow a free flow of ideas and information. The consultant framed questions spontaneously, probed
for information and took notes accordingly. (Details of the interviews are available in the tools’
guides & protocol section in Annex 4).
Twenty interviews (13 individual interviews and 7 group interviews) were conducted with a total of
37 stakeholders (Refer to Annex 2 for more details).
B. FOCUS-GROUP DISCUSSIONS (FGDS)
The consultant facilitated FGDs in four selected governorates across Lebanon. The purpose of the
FGDs was to capture the most significant impacts, if any, of participation in the programme and to
gain a better understanding of their opinions, insights, ideas, and challenges encountered, and
recommendations based on their level of satisfaction.
The advantage of FGDs is that they allow for open dialogue and discussions that helps to gather
insightful information for the evaluation. .
Table 5 below shows the number of beneficiaries/households (HHs) per governorate, Caza and SDC
and was used as a guiding reference to identify the targeted working areas and the beneficiaries who
took part in the FGDs.
S Governorate Caza Targeted # Of SDCs
involved
# of HHs
benefitted
1 Beirut Beirut 4 46
Sub-Total 4 46
2 Bekaa Baalbek 9 2982
Hermel 1 534
Rashaya 4 122
West Beqaa 4 552
Zahle 3 572
Sub-Total 21 4762
3 Mount-Lebanon Aley 6 737
Baabda 6 921
Chouf 10 306
El Meten 5 33
Jebeil 4 69
24
The final evaluation report of the Winter Cash Programme-UNICEF Lebanon. Developed by AwnyAmer- Jan, 2017 Page 24
Kesrouan 5 105
Sub-Total 36 2171
4 Nabatieh Hasbaya 1 8
Marjayoun 3 3
Nabatiyeh 3 149
Sub-Total 7 160
5 North Akkar 10 7568
Batroun 4 124
Bchare 1 59
Koura 2 182
Minieh-Dannieh 3 3085
Tripoli 3 4959
Zgharta 2 679
Sub-Total 25 16656
6 South Bint Hbeil 4 322
Jezzine 1 4
Saida 5 1069
Sour 5 862
Sub-Total 15 2257
Total 26 108 26052
Table 5. Number of beneficiaries/households per governorate, Caza and SDC
The following table (Table 6) shows the results of the selection process and number of beneficiaries
per target area. The consultant selected four governorates with the highest number of HHs that
benefitted from the programme. Referring to Table 5 above, the key specific areas in the targeted
governorates with the highest administrative level and highest number of beneficiaries were: Baalbek
SDC – Bekaa governorate (2,982 families), Rahba SDC- Akkar governorate (7,568 families), Tripoli
SDC – North governorate (4,959 families) in addition to the others mentioned above.
After consultations with UNICEF and considering any possible security & safety limitation during
the data collection phase, Table 6 below shows the breakdown of the sample:
S Governorate CAZA(s)
Targeted
SDC(s)
targeted
# of FGDs/
interviews Target groups # of
participants
1 Mount
Lebanon
Baabda Ghobeiri 4
SDC director
Field coordinators
Social workers
Direct beneficiaries
1
2
5
8
Sub-total 16
2 Bekaa Baalbek Baalbek 5 SDC director
Director of SDCs -
Beqaa
Field coordinators
Social workers
Direct beneficiaries
1
1
1
7
13
Sub-total 23
3 Akkar Akkar Rahba 4 SDC director 1
25
The final evaluation report of the Winter Cash Programme-UNICEF Lebanon. Developed by AwnyAmer- Jan, 2017 Page 25
Field coordinators
Social workers
Direct beneficiaries
1
2
10
Sub-total 14
4 Tripoli Tripoli El-Mina 3 SDC director
Social workers
Direct beneficiaries
1
2
22
Bab El
Tabbaneh 3 SDC director
Field coordinator
Social workers
1
1
4
Sub-total 31
Total 19 84
Table 6. Breakdown of the sample
Specific FGD guide was developed and included under annex 4 attached to this report.
3.4 Limitations and Challenges of the Evaluation Process
The following are some of the limitations faced during the evaluation process:
Availability of programme documents: During the initial phase of the evaluation, it was not possible for the consultant to access or
reference the selection criteria. This was rectified when they were provided by the CMU and
the SDCs.
There was difficulty accessing relevant versions of the questionnaire pertaining to the
analysis of the selection process, which were also later provided by the CMU and SDCs.
The HRRP 2016 includes the results and the respective indicators for poor Lebanese families,
Syrian and other refugees. This was in fact one of the challenges encountered especially for
the logical framework that should be referred to in the current evaluation. Therefore, if we
have one programme document addressing the winterization basic needs targeting both
Syrian or other refugees and the poor Lebanese, we have to have separate results framework
and respective activities to be referred during the evaluation process.
It was difficult to conduct planned FGDs due to security reasons in some of the targeted areas
(e.g. Sir El-Doniyeh-Tripoli). However, this issue was addressed by conducting additional
FGDs in other similar areas (El-Mina-Tripoli).
The FGDs relied on participants’ memory only to recall the impact of the programme which
was implemented approximately one year ago. However, the information previously
collected during the PDM process played a vital role in triangulating all of the data. There
was a strong correlation and similarity between the key findings captured from the two
processes.
High turnover of staff involved in programme implementation, especially at MoSA and
NPTP, led to certain gaps in information. However, this issue was overcome by organising
meetings with them and other key staff members such as field coordinators who had
transferred to another SDC.
26
The final evaluation report of the Winter Cash Programme-UNICEF Lebanon. Developed by AwnyAmer- Jan, 2017 Page 26
4. Key findings
The following sections review the data collected and provide an analysis of the key findings based on
relevance, effectiveness, efficiency, coordination and sustainability criteria. Refer to Annex 6 for a
detailed copy of the evaluation framework & matrix with the respective questions for each indicator.
4.1. Relevance
Question1: To what extent were the project intervention strategies, project design,
interventions, and approaches suited to the priorities, needs and rights of the targeted children
and their families; as well as appropriate (and relevant) to promoting the intended objectives of
the programme? And to what extent were the needs of the most socio-economically vulnerable
Lebanese Households met?
The overall goal of the Winter Cash programme was to introduce cash assistance to serve as a pilot to
further strengthen existing social safety nets and social protection mechanisms in Lebanon. In order
to do that, the Winter Cash programme provided one-off cash assistance to 75,000 children through a
cash grant of USD 40 per child to secure basic winter needs. The expected objectives/outcomes of the
winter cash programme mainly focused on the following two objectives:
Enable Lebanese children and their families to cover a portion of their winter expenses to help
their children cope with the winter cold in Lebanon.
The Winter Cash programme aimed to strengthen delivery mechanisms of the NPTP via the
incorporation of new child-sensitive systems, including cash-based programmes like the
unconditioned cash transfer assistance modality.
The strategy adopted in the Winter Cash programme mainly focused on the collaboration between
UNICEF, NPTP, MoSA, WFP, the office of the Prime Minister and the Lebanese Presidency of the
Council of Ministries (PCM). The programme relied on the PMT score to target the most socio-
economically vulnerable Lebanese households with children, referring to some key variables in the
final selection process such as the employment status, level of education, marital status, physical
ability, housing conditions, assets owned and their geographical location.
Analysis of cash transfer modalities adopted in other similar programmes addressing the winter
basic needs for vulnerable groups revealed that alternative conditioned cash transfers are used,
mostly by other organisations. These modalities include the e-voucher for food, paper vouchers for
food, and paper vouchers for fuel or for clothing kits. The current Winter Cash programme adopted
the unconditioned cash transfer modality as a pilot intervention managed by NPTP/MoSA with the
technical support of UNICEF, WFP and other actors. Families have the opportunity to secure and
purchase their basic winter essentials in a more flexible, unrestricted manner, respecting both their
rights and needs.
Overall, the Winter Cash programme has focused on addressing the rights and the needs of the
Lebanese Poor vulnerable children and their families in the targeted areas across Lebanon. In
general, the evaluation found that the programme intervention could reach the most socio-
economically vulnerable Lebanese Households significantly and was successful in meeting most of
their basic winter needs.
27
The final evaluation report of the Winter Cash Programme-UNICEF Lebanon. Developed by AwnyAmer- Jan, 2017 Page 27
The evaluation found that the intervention strategy implemented by UNICEF and its partners helped
vulnerable children and their families to meet some of their basic winter and emergency needs such
as food, medication and winter clothes. Without this assistance, their level of poverty would be
further exacerbated. The evaluation found that the NPTP succeeded through the implementation of
the Winter Cash programme in securing most of the basic winter needs of the targeted families.
The two areas of intervention highlighted as needing additional support were: (1) Capacity building
and (2) Raising awareness.
Capacity building
The evaluation revealed that the capacity building component was missing in the following phases of
programme design, intervention, and activities, which directly affected achievement of programme
objectives. Strengthening this component is considered a pressing issue, to promote fairness in
delivery mechanisms and must be addressed in future programme design.
Capacity building of Field Coordinators and Social Workers
Although the Winter Cash programme aimed at strengthening delivery mechanisms and enhancing
the skills of NPTP staff, the evaluation found that the capacity development component was largely
absent. Through interviews conducted with staff from NPTP and MoSA, it was reported that not all
of the social workers had a scientific professional qualification, thus limiting their ability to play a
full role in programme implementation, particularly when working directly with beneficiary families.
An Advisor at MoSA indicated that the supply of social workers is also much greater than the
demand for their skills. This was validated through interviews conducted with three SDC directors
during the evaluation process.
“There are a large number of human resources represented in the category of the social workers
and field workers but they are not utilised completely, fairly and effectively” - CMU/PCM staff
interviewed during the evaluation process commented.
Limitations found in the knowledge, capabilities and skills of social workers indicate an urgent need
to include a training component in the programme design in order to ensure the quality of the
delivery process and to arrive at improved socio-economic results for the beneficiaries.
SDC directors, NPTP staff, MoSA staff and WFP partners recommended the following areas in
which to develop capacity, enabling improved performance of social workers to effectively deliver
the Winter Cash programme: communication skills, social work & research ethics, data collection,
data reliability and validity, data integrity, and child rights.
Such capacity building activities are essential to reduce the gaps and individual differences among
social workers who represent the key personnel in this programme in terms of knowledge, skills and
their performance in the field where they work with targeted communities and vulnerable families.
One SDC reported that questionnaires distributed as part of the selection process were filled by
telephone by two social workers, indicating a possible lack of credibility of certain information
collected during the implementation phase. Although this was found in only one out of four SDC
visited it highlights the need to include sufficient training for social workers and field coordinators to
ensure that they suitably reach the most vulnerable families. The capacity building component must
also include a monitoring mechanism enabling greater awareness of social workers to the needs of
28
The final evaluation report of the Winter Cash Programme-UNICEF Lebanon. Developed by AwnyAmer- Jan, 2017 Page 28
beneficiary families and establish strong relations with them. This will help to ensure accurate
reporting on the status and changing conditions of the families.
Training and workshops
When social workers obtained the ATM cards at the beginning of the distribution process, they also
received an introductory administrative awareness session which included an explanation on the
complaint mechanism. NPTP and MoSA staff interviewed during the evaluation process indicated
that they would - rather have received a capacity building training on communication and facilitation
skills which they deemed to be more appropriate for the distribution process.
Beneficiaries’ awareness session
Feedback collected from MoSA staff, SDC directors and field coordinators, during the PDM process
indicated that although the workshops were well received, the beneficiaries were not interested in
taking part. The majority of beneficiaries did not attend these sessions. It is therefore highly
recommended that a demonstration on the use of ATM cards be provided during the ATM card
distribution itself, rather than in a separate workshop12.
Capacity building of MoSA/NPTP staff
To ensure the sustainability of the programme; promote its ownership at the level of MoSA and the
NPTP; and to enable them to replicate the programme it in the future, MoSA staff should receive
capacity building to enable them to monitor the implementation of the programme and raise their
awareness on the importance of follow-up and presence in the field. This will maintain the accuracy,
effectiveness, efficiency of the programme and increase its credibility among beneficiaries.
Parents’ awareness
During FGDs in Mount Lebanon, Akkar and Tripoli it was reported that on a few occasions, parents
did not use the cash grant appropriately, leading their children to miss out on the benefits of the
assistance. Examples of how the grant was spent are cigarettes, and going to the hairdresser.
Although these cases cannot be generalized, the inclusion of awareness raising sessions for parents is
recommended for future programmes, to educate them on how to appropriately use the cash grant to
meet the needs of the child - the main objective of the programme. Outlining the correct use of the
cash grant was largely conducted in an informal manner, when parents received the ATM cards. This
proved to be inadequate for communicating the intended purpose of the grant. Revisiting the “Cash
for What?” question for multi-purpose and integrated programmes, to better identify the expectations
of parents are advised.
Definition of the child in the Selection criteria of the Winter Cash Programme
At the global level there is considerable debate around the definition of a child, compared to that of
an adolescent. UNICEF considers a child to be under the age of 18 - as defined in the Convention on
the Rights of the Children (CRC). However, the Winter Cash programme targeted children aged up
to 15 years due to limited funding and the priorities of the most vulnerable age groups.
12InfoPro_ Winter Cash Assistance- Final report -Post Distribution Monitoring Report-2016_page 13
29
The final evaluation report of the Winter Cash Programme-UNICEF Lebanon. Developed by AwnyAmer- Jan, 2017 Page 29
The justification and the rationale behind this age limit was not present in any of the programme
documents reviewed, nor was it explained to the social workers, field coordinators or the SDCs
directors.
“I have to treat my children equally and that’s why I added and/or borrowed additional money to
buy the winter clothes for all my children instead of buying it for all my younger children up to 15
years” - participants in Akkar, Mount Lebanon and Tripoli.
It is recommended to provide a clear explanation on how the target age was reached, to both
implementers and beneficiaries, to reduce confusion and encourage a better understanding of the
programme.
The needs of most socio-economically vulnerable Lebanese families
It was concluded that the objective of the programme in meeting the seasonal needs of the most
socio-economically vulnerable Lebanon households through the provision of USD 40 per child was
not achieved. The cash grant was insufficient in securing the basic needs related to seasonal hazards,
such as food, medication and clothes. This was particularly highlighted by participants during FGDs
in Mount Lebanon and Bekaa. All SDCs’ directors, field coordinators and social workers interviewed
during the current evaluation process indicated that despite the fact that the programme managed to
benefit many families, it excluded a large number of vulnerable families who were also in need of
financial assistance. This was validated again by the beneficiaries during the FGDs.
The majority of stakeholders and direct beneficiaries interviewed in the evaluation process reported
that in order to solve this, there are two possible suggested scenarios:
1- To increase the amount of money and decrease the number of families so as to ensure that
more of their needs are met;
2- To include all families who carry the “Hala” card and are classified as vulnerable families,
while decreasing the amount of the grant with the aim of serving more.
Two solutions are recommended in order to meet the needs of vulnerable families: (1) Increase the
total value of the grant provided, to cover the needs of entire families, and (2) Engage more partners
in the programme with special support from MoSA and other ministries to provide cash or in-kind
assistance to the programme, such as vouchers for food, clothes, fuel, or medications, ensuring that
the all needs of the most socio-economically vulnerable families are met sufficiently and in a
dignified manner.
Meeting the needs of more vulnerable families in the future would only be possible by ensuring
accurate and updated data to better inform programme decisions. The data used in the selection
process of the current programme was outdated and was not validated by social workers.
Question2: To what extent were the criteria of the selection process developed to ensure
that the most vulnerable and poor children were reached and benefited properly from the
program?
The selection process was developed jointly between the NPTP, UNICEF, and CMU/PCM
including the selection criteria. However, there was no clear documentation or understanding of
how the selection process worked. Social workers & field coordinators did not participate in
setting the selection criteria and other relevant questionnaires. Additionally, the data referred in
30
The final evaluation report of the Winter Cash Programme-UNICEF Lebanon. Developed by AwnyAmer- Jan, 2017 Page 30
the selection process relied on outdated sources (from2011) in addition to omitting some of the
important variables and criteria in the selection process.
Selection criteria
The evaluation found that there was no clear documentation or understanding of how the selection
process worked. It is worth noting that the NPTP, PCM and the World Bank developed a Proxy
Means Test (PMT) formula to target beneficiaries whether through the current Winter Cash
programme or other programmes. However, the Winter Cash programme relied on the PMT score to
target the most socio-economically vulnerable Lebanese households with children to include them in
the programme. In order to proceed to the targeted HHs, MoSA social workers updated the NPTP
database of the 35,000 identified most vulnerable households and entered it into an Optical Marking
Recognition (OMR) form, which was then inserted into the live database.
Thereafter, the selection process was purely scientific and based on a statistical, computer-based
system developed by experts. This system analysed 60 indicators and criteria which defined the
situation of each household through information collected from field and household visits, and which
was then entered into the computer system to generate a final score. This score dictated eligibility for
the programme. It was found that the data used in the selection process was outdated. In addition, the
new questionnaire, which was used as a validation tool to select the poor vulnerable families, did not
take into consideration new-borns.
The use of an outdated list during the selection process meant that the poorest families were not
always reached. MoSA staff who were interviewed in the PDM process indicated the strong belief
that the selection process was “unfair”, and that many families who should have been eligible were
excluded, while others who were less in need received the cash assistance.
In addition, some SDC directors, social workers, field coordinators and FGD participants - especially
in Akkar, Tripoli and Beqaa/Baalbek - indicated that the equitability of the selection process was an
issue of concern. They implied the existence of political bias and that many families who were
eligible to receive the grant did not benefit from it, while others who were less needy did. Moreover,
NPTP field workers assured the 35,000 households that were surveyed and verified using the NPTP
PMT formula that they would receive the ATM cards, however only 27,000 cards were printed.
Furthermore, not all benefitted because of additional challenges encountered during the distribution
phase, such as distribution without pin codes or other technical problems. In the final analysis the
actual number of families who benefitted from the Winter Cash programme was 24,601 (94 per
cent3) out of the 26,052 selected households. Therefore, 5.6 per cent of the targeted children and
their families were not reached.
This figure is considered as a no-show rate because families called for the distribution phase did not
show up after two rounds of notification SMSs, telephone calls and home visits. It is also probable
that some families did not go to the distribution centres to receive their ATM cards for reasons other
than what was reported by SDC staff (e.g. families involved in disputes, and individuals who
separated or divorced during the distribution process. The latter would lead to a case in which the
mother would have the “Hala” card while the father (the beneficiary name) could not receive the
ATM card without the “Hala” card). The evaluation found that 5.6 per cent, which represents
approximately 5,000 children and their families, could be accepted as a normal rate. However,
31
The final evaluation report of the Winter Cash Programme-UNICEF Lebanon. Developed by AwnyAmer- Jan, 2017 Page 31
additional steps should be taken by WFP and NPTP to use more effective means of communication
with beneficiaries in future similar cash programmes.
Findings from the PDM process showed that the majority of social workers and field coordinators
were not aware of who would receive cash assistance and created unrealistic expectations amongst
possible recipient households. 13. This shows the weakness of the training provided by NPTP to the
social workers and field coordinators and highlights the need for quality capacity building activities
for implementing staff. This would help to ensure more effective communication and interaction with
the targeted families.
Some social workers in the North (including Akkar and Tripoli governorates) who interviewed
during the PDM14 also reported that members of the police force were interfering in the selection
process to help people that they knew, indicating favouritism and exclusion towards others15. Other
political and community considerations were also revealed.
The questionnaire used in the selection process
The evaluation found that the questionnaire used in the selection process was inadequate to identify
vulnerable families and their children.
As explained above, the winter cash programme relied on the Proxy Means Test (PMT) formula was
developed by the NPTP, PCM and the World Bank to target the most socio-economically vulnerable
Lebanese households with children to include them as a programme beneficiary. Then, MoSA social
workers updated the data of 35,000 most vulnerable identified households included in the NPTP
database and entered them into an Optical Marking Recognition (OMR) form. After updating the
assessment performed, the CMU ranked and selected the most vulnerable families corresponding to
the target of 75,000 children (with total 26,052 families) referring to the PMT formula.
Another concern raised by MoSA staff was the irrelevance of some of the selection criteria while
other crucial elements were missing completely from the questionnaire.
The questionnaire is linked to a PMT formula, including 60 variables identified by the data collection
team. Most of the MoSA staff, field coordinators and social workers who were interviewed reported
the following missing variables in the selection process:
Deceased husband
Children or persons with disability in the family
Unemployment rate of the family
The employment type and whether it is permanent or seasonal type
The income level(s) of the family
The illness history of the family- how many family members are suffering from chronic
diseases
The educational status of the family – how many children are studying
Whether the house is rented or owned by the family
The status of orphans in the family (due to a deceased husband)
13InfoPro_ Winter Cash Assistance- Final report -Post Distribution Monitoring Report-2016_page 9 14Ibid_ page 9 15Ibid _ page 10
32
The final evaluation report of the Winter Cash Programme-UNICEF Lebanon. Developed by AwnyAmer- Jan, 2017 Page 32
It should be noted that the size of a beneficiary’s apartment or house is not considered as a robust
criterion since the house could be inherited.
The new questionnaire, administrated during the selection process, was used as a verification tool to
update some of the basic existing data included in the NPTP database. The existing data had been
used to issue the “Hala” card and included similar vulnerability and welfare variables. The NPTP
recognised some issues which had been reported in the interviews conducted during the evaluation
process and accordingly, they started to redevelop the PMT, questionnaires and other forms to ensure
an improved selection process. Most of the MoSA staff and the field staff indicated the importance of
involving and consulting them during the programme design and implementation phases and to take
into account their feedback on the selection criteria and the decisions taken. Their insights would be
useful 16as they work directly with the targeted families17.
Other selection criteria/variables
The selection process was mainly based on the Proxy Mean Test (PMT) formula to assess and
estimate the families’ consumption per day as a proxy means referring to some variables such as the
employment status, level of education, marital status, physical ability, housing conditions and
geographical locations. This led to a selection process based on data that was not updated with
regards to any significant change in some of the key variables such as the employment status, level of
income, housing conditions and other relative sub-variables as indicated above.
Social workers and field coordinators’ involvement in setting the selection criteria and
questionnaire
The evaluation found that the lack of involvement of the social workers, field coordinators and other
SDC staff in reviewing and checking the questionnaire greatly impeded the selection process and was
considered to be a major gap. All of the 20 social workers and field coordinators reported that they
were not informed about the selection criteria variables, except that the beneficiary family should
have children aged up to 15 years.
Although they received and reviewed the revised questionnaires, providing feedback and
highlighting certain issues, this was not taken into consideration. This inevitably has a negative
impact and created unfavourable relations between the targeted families and the SDCs – particularly
the social workers and field coordinators as they were unable to respond adequately to the families
who dissatisfied with their exclusion from the programme.
Although, the questions and answers mechanism was developed to respond to these issues, there was
a noticeable gap in the capacity of social workers and field staff to adequately respond. This, once
again, highlights the need to train personnel before programme implementation.
16Beneficiaries’ feedback document__Winter cash programme_ MoSA staff- Summary report page 4. 17Ibid_page 10
33
The final evaluation report of the Winter Cash Programme-UNICEF Lebanon. Developed by AwnyAmer- Jan, 2017 Page 33
Question 3: To what extent has the programme been successful in identifying the most relevant
partners/actors for this project and what was the added value of these partnerships?
Partners’ identification
The evaluation found that UNICEF consistently selected appropriate and relevant partners for this
programme and it was evident that UNICEF had a clear vision regarding its partner selection.
UNICEF has been working with Government partners represented in NPTP/MoSA, and has also
been supplementing this work by collaborating with a series of key partners with considerable
expertise and strong commitment to the implementation of the Winter Cash programme such as
WFP, and the CMU/PCM.
Complementarity component among the different key partners
The National Poverty Targeting Programme was established in 2011, with the aim of supporting
vulnerable Lebanese families in meeting their needs. Although the NPTP works under MoSA, it has
its own separate, permanent structure as a semi-entity. For this reason, the Winter Cash programme
was designed in a collaborative manner between UNICEF, NPTP, MoSA), WFP, and CMU/PCM
(with separate roles for each). For example, MoSA and the CMU were the official entities
responsible for the provision of information and data on the most socio-economically vulnerable
families in Lebanon needed during the selection process.
The programme was successful in identifying the most relevant partners for implementation and each
of the five partners (UNICEF, MoSA, CMU, NPTP and WFP) played a significant role bringing their
own added value.
The NPTP is the official Government arm supporting underprivileged Lebanese families living in
harsh economic conditions (below the poverty line). Its importance has increased, especially due to
the ongoing Syrian conflict that has resulted in further deterioration of conditions for many Lebanese
families. Syrians willing to work for fewer wages than their Lebanese counterparts have gained
access to the labour market. This has undermined the situation of many Lebanese wage-workers,
threatening already vulnerable families with falling further into poverty.
In addition, WFP has a good reputation and strong outreach and technical expertise, particularly
among the group targeted by the Winter Cash programme where they implement e-food programmes.
Overall, the evaluation found that the selection of the partners was well-informed and successfully
boosted the trust of the beneficiaries in the programme.
Question 4: How gender sensitive was the programme when reaching the most vulnerable
children?
Based on the findings of the evaluation, the programme did not incorporate a gender perspective
when targeting the most vulnerable children. This is due to the fact that the selection criteria of the
families in the poor vulnerable families did not consider gender as one of the selection variables. The
age of the selected children (up to 15 years) was the key selection criterion. No sex-disaggregated
data for children who benefitted from the programme was available. The payment list of beneficiaries
of the Winter Cash programme included the sex of the ATM card holder without having
disaggregated data of the children supported per family.
34
The final evaluation report of the Winter Cash Programme-UNICEF Lebanon. Developed by AwnyAmer- Jan, 2017 Page 34
Moreover, the gender criterion was not included in the questionnaire that was administrated to the
target groups during the selection process. As a result, little or no information was collected on the
number of female children who benefited from the programme. There was a noticeable gap in the
knowledge of when one should apply gender sensitive and equity measures. It is therefore
recommended that a gender component be included in any future programmes.
Question 5: Do the current log frame and/or results framework and relevant indicators in the
winter cash programme document need improvement to better serve the programme objectives
Programme proposal/concept note
A basic Concept Note on the 2015/2016 Winter Programme was developed and endorsed by the UNICEF
Country Management Team and shared with the NPTP management. The Lebanese component of the
UNICEF assisted winter programme was also included in the HRRP and Lebanon Crisis Response Plan.
At the start of the planning, a detailed action plan was developed by UNICEF and NPTP with the
partners.
The evaluation, however, found that a programme proposal, considered to be a fundamental
document, was missing from the programme. The programme proposal, plays a major role in
outlining the programme objectives, activities, outcomes, monitoring and evaluation (M&E)
mechanism, duration, target group and sustainability mechanism for the partners and other
stakeholders.
Providing this document at the start of the programme ensures that the role of each partner is well
explained and that the partners are aware of their responsibilities in the implementation. Despite the
absence of a concept note, the consultant managed to collect most of this information during the
inception phase of the evaluation and during the desk review of secondary data.
The Humanitarian Response and Resilience Plan (HRRP) 2016
The evaluation found that the HRRP 2016 lacks a separate log frame with specific objectives for the
Winter Cash programme targeting the poor Lebanese families, without reference to Syrian refugees.
This is considered to be a shortcoming in the design of the programme. To ensure efficient strategic
planning for the programme in the future, a well-structured log frame should be incorporated from
the very beginning, with a results framework which follows the results chain approach. There is no
clear documentation that shows the results framework as part of a complete Results-Based
Monitoring and Evaluation system or that includes an M&E matrix explaining the roles and
responsibilities for each stakeholder (who will do what, when, how and where).
However, the consultant extracted the key expected outcomes (results chain) from other documents
that combine two programmes, the first addressing and targeting Lebanese poor vulnerable families,
and the other addressing Syrian refugees. It is valuable to focus on the scope of work of the
programme component that addresses vulnerable Lebanese children and their families. Therefore, it
is critical to differentiate between dealing with and addressing the priorities and needs of refugees
versus tackling these same needs of the vulnerable poor Lebanese families as explained above.
This is important because for a programme that addresses refugees’ basic needs, evaluation criteria
must be followed (mainly the ALNAP criteria as part of the OECD-DAC criteria). This is different
from those used for programmes addressing the poorest Lebanese vulnerable groups (mainly DAC
35
The final evaluation report of the Winter Cash Programme-UNICEF Lebanon. Developed by AwnyAmer- Jan, 2017 Page 35
criteria). The normal DAC criteria includes: Relevance, effectiveness, efficiency, sustainability and
impact, while the OECD-DAC criteria for humanitarian aid programmes are18:
Relevance/appropriateness, connectedness, coherence, coverage, efficiency, effectiveness and
impact. The TOR for the current evaluation includes a combination between the OECD/DAC criteria
with the absence of the connectedness, coherence, and impact while the current evaluation addressed
the coordination as a separate indicator.
The evaluation found that an operational plan was developed and updated on a bi-weekly basis.
However, it was revealed that there was a delay in distribution of ATM cards. This is not due to lack
of unclear timeliness but was because of the time allocated to the data collection by the field workers
and social workers in addition to the time spent in the long negotiation process with the SDCs to
define where the distribution will take place. However, one of the lessons learnt- considering that this
is pilot project- is that the estimated time needed for each of the different programme phases should
be re-visited for better time management in the future implementation of similar cash programmes.
4.2. Effectiveness
Question 6: To what extent were the vulnerable children and their families targeted by the
Winter Cash programme reached- (as planned)? How satisfied were they about the
programme?
Level of satisfaction among the direct beneficiaries
FGDs revealed that the majority of beneficiaries expressed a high degree of satisfaction with their
participation in the Winter Cash programme. These results were consistent with satisfaction shown
by beneficiaries who either experienced or observed changes in their sons, daughters or other family
members.
‘We really appreciate the efforts and aid provided by UNICEF through the cooperation of NPTP
via the “Hala” card which was our channel to benefit from the Winter Cash programme,” –
participants in the FGDs.
The evaluation indicates that the programme was very useful and that beneficiaries were satisfied
that it helped them secure some of their children’s and families’ basic needs such as food, clothes,
and medication.
Other FGD participants from the programme beneficiaries (particularly in Mount Lebanon and Bekaa
governorates) expressed how the cash transfer was exceptionally helpful and necessary given the fact
that it covered a large segment of the most vulnerable and disadvantaged families who were
registered in the NPTP. This was particularly true in the two above mentioned governorates where
scarcity of work opportunities, particularly in the winter season is a regular characteristic.
One of the rewarding observations that emerged during the FGDs was that the beneficiaries
associated the “Hala” card as the key reference for the Winter Cash programme rather than UNICEF.
This attitude corresponds with the key findings captured during the PDM19. The majority of
18Evaluating Humanitarian action using the OECD-DAC criteria- Overseas Development Institute- London, March 2006- page
20-21
36
The final evaluation report of the Winter Cash Programme-UNICEF Lebanon. Developed by AwnyAmer- Jan, 2017 Page 36
beneficiaries discussed their appreciation of the cash assistance during the FGDs and how much the
support was necessary.
The adequacy of the cash transfer amount
The evaluation found that some of the beneficiaries were not satisfied with the value of the cash
grant, deeming it insufficient in meeting their children’s needs, while others suggested that it should
be distributed on a monthly or quarterly basis throughout the year. These responses correspond to
those collected during the PDM which showed that 42 per cent of beneficiaries believed the
assistance to be sufficient in covering their children’s winter needs, while the majority, 58 per cent20,
thought it was insufficient.
One of the interesting comments reported by some of the participants during the FGDs - especially in
Baalbeck was “I’m earning at least 20,000 LL on daily basis as I’m a taxi driver and that’s why
the amount allocated in the Winter Cash programme per child up to 15 years means nothing”.
Participants in all five FGDs indicated the need to either increase the amount of the cash grant, or
receive the assistance on a recurrent basis (i.e. quarterly).
Timing of the winter cash assistance
A delay in the implementation of the programme led to the distribution of ATM cards in March
instead of winter. This hindered achieving the main planned objective of the programme which is “to
provide the aid in the winter season to reduce the vulnerability of Lebanese poor families that is
exacerbated by the increase of expenditures and prices and is paralleled with a reduced flow of
income due to low availability of work opportunities during the winter”. This corresponds to the
results of the PDM survey that revealed 59 per cent21 of beneficiaries believed that the assistance
was performed in a timely manner, 24 per cent stated that it was a bit too late, 14 per cent stated that
it was quite late, and 3 per cent stated that it was really too late.
Families who have children with disabilities – Other vulnerable groups
Other beneficiaries, especially in Bekaa, Mount Lebanon and Akkar governorates expressed their
dissatisfaction as the programme did not take into account the special needs of families who have
children with disabilities. They emphasised that this should be included as one of the key selection
criteria, especially because these families incur higher expenses for additional care and medication.
Family members spend more caring for children with special needs, impeding their employment
opportunities and ability to earn a steady income.
Despite the fact that the number of children with disabilities in Lebanon is not documented, it is
highly recommended to consider this category as part of the key selection criteria for the Winter
Cash programme.
Working women and orphan children
The evaluation concluded that the Winter Cash programme did not give special consideration to
categories such as working women or orphans. Participants on the FGDs suggested that this should
20Statistics Lebanon Ltd_ Results of the survey findings _Winter Cash Programme-PDM Report-2016_Slide# 61 21Statistics Lebanon Ltd_ Results of the survey findings _Winter Cash Programme-PDM Report-2016_Slide# 65
37
The final evaluation report of the Winter Cash Programme-UNICEF Lebanon. Developed by AwnyAmer- Jan, 2017 Page 37
be included as one of the key variables in the main vulnerability assessment selection criteria. This
variable was omitted from the questionnaire when selecting the most vulnerable families. In addition,
the number of employed family members who earn an income inside the family was also not
considered.
“We don’t know on what basis those who benefited from the Winter Cash programme were
selected,” – One FGD participant. .
Other vulnerable families- families with no Children
Other concerns were recorded which raised dissatisfaction over omitting families who do not have
children and who were excluded completely from the programme. Despite the fact that the Winter
Cash programme with the support of UNICEF Lebanon targets poor children and their families (in
reference to the mandate and scope of work of UNICEF), the FGD participants in addition to the
SDC staff pointed to the importance of considering them in the other similar future projects.
Other fairness aspects in the selection process
Some families who benefitted from the Winter Cash programme (especially in Akkar and Mount
Lebanon governorate) - received their ATM cards without a pin code. To address this issue, the
programme adopted a complaint mechanism to report these cases (with the support of the field
coordinators and social workers in the respective SDCs). However, these problems were not
resolved. Furthermore, these families did not benefit from the programme due to the weaknesses of
communication within the complaint mechanism - including delays in response. The programme
ended before these issues were resolved.
“What is the mistake I made that deprived my children from the Winter Cash programme like
the rest of the children in my community/neighbourhood? “ - Two participants of the FGDs in
Akkar governorate.
Distribution mechanism – Organisational issues
The distribution mechanism was considered as one of the main challenges of the Winter Cash
programme as reported by some of the participants, especially those living far from the SDC selected
as a distribution point. This corresponds with the results of the PDM process that showed: 69 per
cent stated that it was too far from their place of residence, 14 per cent stated that it was unsafe, 13
per cent waited too long, 3 per cent stated that it was not accessible to elderly, pregnant women, or
disabled individuals, and 1 per cent stated that they had to go to the site several times22.
During the current evaluation, some of the FGD participants reported that additional transportation
fees were incurred by families living far from the distribution centre. The following examples were
captured:
In Akkar Governorate, only three SDCs were selected to distribute the ATM cards
The number of ATM cards distributed in Rahba SDC covered beneficiaries from five SDCs -
Bebnin (1,534 ATM cards), El Bireh (551 ATM cards), Halba (1,025 ATM cards), Qoubaiyet
22Statistics Lebanon Ltd_ Results of the survey findings _Winter Cash Programme-PDM Report-2016_Slide# 39
38
The final evaluation report of the Winter Cash Programme-UNICEF Lebanon. Developed by AwnyAmer- Jan, 2017 Page 38
(269 ATM cards) and Rahba (465 ATM cards) totalling 3,844 ATM cards. The distance
between Rahba SDC and Qoubaiyat SDC is at least one-hour drive and a family travelling
between the two would incur additional transportation fees of approximately 20,000 LL
(USD 13.50). The same was found in Bekaa governorate (Baalbeck SDC).
“If I have only one child up to 15 years old, I would receive USD40 This means that I have to
spend 35 per cent of this amount before coming to receive the ATM card,” - Participants of the
FGDs commented in both Akkar and Baalbek.
These comments corresponded with the results of the PDM process. The results of the survey
conducted during the PDM23 show that 89 per cent of participants confirmed that the distribution site
was satisfactory in comparison to 11 per cent who found it to be unsatisfactory. With respect to why
they felt dissatisfied, the PDM results showed that 78 per cent24 indicated that the organisation of the
distribution process was weak. The related issues were reported by the beneficiaries interviewed
during the PDM process as follows:
The distance between beneficiaries’ homes and their assigned centres was an issue (cost of
the trip).
The distribution was cumbersome, often disorganised and extremely exhausting for social
workers.
Some centres were overloaded and could not cope with the large number of beneficiaries
coming every day. Staff had to deal with problems on the spot and be reactive rather than
proactive25.
The evaluation considers this issue as one of the gaps in the coordination among different actors and
a drawback negatively impacting the cost efficiency of the programme.
SMS
Another coordination issue that was raised related to UNICEF’s role in informing the eligible
families about the ATM cards. The evaluation found that many of the selected families did not open
the SMSs that were sent to them, which notified them about the distribution of the cards. The reasons
behind this were that they were either illiterate and could not read the SMS, they thought that the
message was an advertisement, or because the models of their phone could not read Arabic messages.
This corresponds with the findings of the PDM process as reported by MoSA staff26.
Question 7: To what extent did the programme have an impact on beneficiaries in improving
access to essential goods and services of their choice in a safe, dignified, and empowered
manner while decreasing socio-economic vulnerability?: Key changes affecting the lives of
beneficiaries as result of the programme.
23Statistics Lebanon Ltd_ Results of the survey findings _Winter Cash Programme-PDM Report-2016_Slide# 40 24Statistics Lebanon Ltd_ Results of the survey findings _Winter Cash Programme-PDM Report-2016_Slide# 41 25 Beneficiaries’ feedback document__Winter Cash programme_ MoSA staff- Summary report page 4 26InfoPro_ Winter Cash Assistance- Final report -Post Distribution Monitoring Report-2016_page 14
39
The final evaluation report of the Winter Cash Programme-UNICEF Lebanon. Developed by AwnyAmer- Jan, 2017 Page 39
Key changes
The programme succeeded in serving 75,000 children from the most socio-economically vulnerable
families in Lebanon and provided each child, aged 0-15 years with a USD40 cash grant to help
secure food, clothing and other winter basic needs. Those beneficiaries who were interviewed
expressed their satisfaction with the programme, stating that the money came exactly on time as they
were in a serious need of extra support due to their economic conditions and lack of employment
opportunities during the winter season. The evaluation revealed that the majority of the beneficiaries
spent most of the cash transfer on medical care, education and food followed by heating oil, winter
clothes27, rental fees and repayment of some loans. The cash grant gave beneficiaries the freedom to
spend their money as they needed and according to their own priorities in a safe, dignified and
empowered manner. Nevertheless, although the grant helped families to secure basic needs for their
children, it cannot be confirmed that the USD 40 managed to invoke key changes, or significantly
impact the lives of the beneficiaries, other than fulfilling a serious need for some basic services
which they could not afford due to their poverty.
On the other hand, some beneficiaries felt that the amount was insufficient for securing their basic
needs, especially as the price of food and clothing in the winter tends to increase. In addition, others
stressed that the assistance should be provided on a continual basis and if it was merely a one-time
grant then no significant change would be incurred. It was also reported that because of the delay in
delivering the ATM cards, after the winter season, some families used the cash grant to cover other
basic needs.
Spending patterns
The current evaluation revealed that the majority of beneficiaries spent the cash amount on food and
rent, namely in Mount Lebanon and Tripoli, and on winter clothing especially in areas like Baalbek
and parts of Tripoli. A minority of beneficiaries reported that they spent the cash assistance on
medical expenses - in Mount Lebanon, Akkar and Baalbek - to address chronic cases of illness within
the family. Finally, few beneficiaries reported that they spent the cash support on tution fees and
purchasing fuel for heating purposes in the winter season in addition to other purposes.
“The cash transfer I received through the Winter Cash programme came on time. It protected
the health of two of my sons and saved the life of my husband as I used all the amount to buy
the medicines they need,” Participant in a FGD.
This confirms the results captured in the PDM survey. The results showed that the five most common
expenditures 28 among the beneficiaries interviewed were as follows: Winter Household expenses
(56 per cent), food (55 per cent), education fees-including the transporation fees to schools (30 per
cent), repaying debts (20 per cent ), health expenses (20 per cent), electercity bills (12 per cent),
heating- fuel or others (11 per cent) and rent (11 per cent).
Question 8: Are there other unintended or unforeseen changes shown in the lives of the
targeted groups of poor children & their families? Also, are there any unintended beneficiaries
in the programme?
27InfoPro_ Winter Cash Assistance- Final report -Post Distribution Monitoring Report-2016_page 28-29. 28Statistics Lebanon Ltd_ Results of the survey findings _Winter Cash Programme-PDM Report-2016_Slide# 69
40
The final evaluation report of the Winter Cash Programme-UNICEF Lebanon. Developed by AwnyAmer- Jan, 2017 Page 40
Although, the Winter Cash programme was designed to secure the basic winter needs of poor
vulnerable children and their families, the unrestricted cash transfer was used as an added value to
cover other expenditures. The below chart # 7 shows how the unrestricted cash transfer was used to
cover other needs including education, health, rent and others. The spending patterns revealed that
the cash grant was used only for the basic needs of winter. Participants indicated that the cash
assistance had a positive effect on the child’s schooling, through positive impacts29 such as
purchasing clothes for school (29 per cent), paying transportation fees (27 per cent), paying school
fees (22 per cent), purchasing stationery (20 per cent), and purchasing books (16 per cent). However,
the chart below shows how the programme, by adopting the unconditioned cash transfer modality,
helped in securing other spending aspects as a kind of unintended change.
Moreover, the unrestricted cash transfer assistance in the current Winter Cash programme that was
designed to secure the winter basic needs for the vulnerable children up to 15 years and their families
was helpful in securing the winter and other basic needs for the other family members.
Table 7. Summary of the spending and expenditures patterns by order of priority30
Question 9: What was the level of acceptance of the cash modality as a tool to support poor
Lebanese households to cover their winter needs (winterisation) from the points of view of all
the different actors: MoSA, NPTP, SDCs, field staff and the beneficiaries?
29Statistics Lebanon Ltd_ Results of the survey findings _Winter Cash Programme-PDM Report-2016_Slide# 67 30Statistics Lebanon Ltd_ Results of the survey findings _Winter Cash Programme-PDM Report-2016_Slide# 68
41
The final evaluation report of the Winter Cash Programme-UNICEF Lebanon. Developed by AwnyAmer- Jan, 2017 Page 41
The level of acceptance of the unrestricted cash transfer
Cash support is considered unrestricted and this form of assistance was praised by the majority of
respondents as their preferred means of support. The same attitude was also expressed by MoSA
staff, who felt that this was a more appropriate wayto help those in need and allow them to address
their own priorities. This opinionwas applicable in all of the groups, with no exception.
“It allowed me to prioritise and buy what the family needed”, said one man in Baalbek.
Men in Halba also agreed that the needs vary from one household to another, and that while one
might need clothes, others might need food products.
“With the money you can always purchase anything that you need”, said one man in Mount
Lebanon.
“We can always secure food, but how can we go to the doctor’s if we do not have money?” commented a woman from Mount Lebanon.
However, others pointed to the need for other types of assistance. Men in Mount Lebanon agreed
that the e-food voucher was also a great tool that could secure food and beverages on a monthly basis
and that other types of support were also helpful provided that they were appropriate and needed. A
man from Tripoli said that if the cash assistance was only going to reach them once, then it would
make no difference.
The findings of the evaluation show that opinions vary with regards to unrestricted cash transfer and
restricted cash transfer.
Partners from WFP, NPTP, and MoSA, as well as six SDC directors, five field coordinators and 14
social workers who were interviewed during the evaluation reported their preference for restricted
cash transfers (in the form of e-vouchers). They indicated that if it was not possible to implement the
unrestricted cash assistance on a recurrent basis, then it would be better to provide e-vouchers for
food, fuel and clothing. During the FGDs participants commented that the one-off winter cash grant
of USD40 per child once a year was not enough to secure their basic needs. They also commented
that since the cash grant was at the end of the winter season, it was instead used to purchase other
essentials and was still not adequate to meet all basic needs. These participants recommended that the
cash assistance be provided on a recurrent basis.
On the other hand, the majority of the direct beneficiaries and a few social workers and field
coordinators who were interviewed expressed their preference for the unrestricted cash transfer. This
corresponds with the results of the PDM survey showing that the majority of beneficiaries are
satisfied with the cash modality (63 per cent)31 compared to those who are very satisfied (32 per cent)
and those who are not satisfied (5pe cent). The MoSA staff and direct beneficiaries interviewed
during the PDM indicated that the unrestricted cash modality was a better tool, to allow beneficiaries
spend according to their personal needs.
During the PDM, other participants indicated the need for other types of support and expressed
criticism of the “Hala card” which they believe was not useful in accessing medical care32
particularly to cover medication fees.
31Statistics Lebanon Ltd_ Results of the survey findings _Winter Cash Programme-PDM Report-2016_Slide# 72 32InfoPro_ Winter Cash Assistance- Final report -Post Distribution Monitoring Report-2016_page 31
42
The final evaluation report of the Winter Cash Programme-UNICEF Lebanon. Developed by AwnyAmer- Jan, 2017 Page 42
However, some of the programme’s actors have gone for the restricted cash modality (vouchers for
food, clothes, medication or fuel) in order to ensure the delivery of the assistance to its target group,
without leaving a space for the irrationality of some parents who might misuse the money and spend
it on inappropriate items, and consequently, hinder the delivery of the money to the children. The
results show that the ones who preferred other kinds of modalities mainly chose vouchers (59%), in
kind (30%), and fee waiver (11%) as reported by the FGDs and survey respondents targeted in the
PDM process33.This indicates their preference for restricted cash modality (e- voucher for food, fuel,
clothes, etc.) in the case of not having the possibility to receive the cash transfer assistance in a
recurrent way because of the highest guarantee in benefiting vulnerable children and avoiding some
of the bad practices of misuse of the cash transfer amount by few parents as reported earlier in this
report and as summarized below.
For more details on the advantages of using the unrestricted/or the restricted cash transfer as
captured and collected from the interviews with the different stakeholders and FGDs with the
programme beneficiaries, please refer to annex 8 attached to this report.
Six social workers in addition to the majority of direct beneficiaries interviewed during the
evaluation showed great acceptance of the cash modality as a support tool, giving families the
freedom of choice and power to purchase the goods and services they need according to their own
priorities.
Combined modalities recommended by the beneficiaries
Some suggested combining the in-kind and the cash modality so as to ensure that the children and
families receive the in-kind assistance -while not depriving them from receiving cash to choose and
purchase what they need according to their personal priorities. Some commented that the cash
transfer assistance is the best cash modality, but on one condition -that it is received regularly on a
quarterly or monthly basis.
One interesting observation emerging from the FGDs in Mount Lebanon was that the husband
preferred the restricted cash transfer, especially the e-food voucher on a regular basis, while the wife
favoured the unrestricted cash transfer as the preferred cash modality, allowing them to set and
prioritise their own needs.
Some participants in FGDs, in Baalbek and Mount Lebanon, highly recommended introducing the e-
health voucher where families can receive medical services especially medication in case of its
unavailability within the health section in the SDC. In this manner, the parent/caregiver can continue
to work and earn money while he/she remains in a good health.
Finally, the evaluation found that introducing a multi-dimensional cash modality based on accurate
and updated data using a well-structured vulnerability assessment is highly recommended.
Question 10: To what extent were the capacities of the different actors involved, particularly
the NPTP and MoSA, adequate and efficient in ensuring the quality of the programme
implementation? What can be improved in this regard?
33Statistics Lebanon Ltd_ Results of the survey findings _Winter Cash Programme-PDM Report-2016_Slide# 75
43
The final evaluation report of the Winter Cash Programme-UNICEF Lebanon. Developed by AwnyAmer- Jan, 2017 Page 43
Although NPTP and MoSA have a large number of social workers in the field, their skills need to be
improved and strengthened in order to deliver quality programme activities. Many do not possess a
scientific degree in social services and sciences creating a gap in skills and knowledge. As previously
discussed, there is a need to strengthen and enhance their capacities so as to promote programme
ownership of the NPTP and establish it as a model for replication and scale-up in the future. In
addition, the number of senior NPTP and MoSA staff involved in the programme was limited
compared to the number of social workers and the number of beneficiaries. The evaluation concludes
that investments should be made to improve the skills of field coordinators and social workers in
order to deliver a quality programme.
Capacity building – What is needed?
The following areas in need of capacity building are those specified by key partners and
stakeholders:
Communication skills - to improve interaction with the other stakeholders in the programme
ensuring enhanced cooperation.
Social work skills - including case management skills to enable social workers to
professionally interact with the beneficiaries during the programme implementation phase.
Field work skills - including field visits and working with vulnerable groups. Research ethics
- to ensure credibility, privacy and accuracy when distributing questionnaires to targeted
families.
Data collection methodologies and tools - including surveys, questionnaires, FGDs and
interviews.
Data reliability, validity, data integrity.
Knowledge of child rights - to enable a better understanding of the linkages between the
impact of the programme and the rights of the child. This will be helpful in the design of
other questionnaires/assessment forms that include different indicators.
Basic monitoring skills
The recommendations listed above could be organised in 4-5 working days to ensure basic
knowledge of each area and that all social workers and field coordinators have the same level of
understanding of the scope of work.
Other specific and structured short training packages should be provided for other MoSA staff and
partners involved in the programme.
Question 11: To what extent does the programme have an effective monitoring & evaluation
mechanism (including reporting, quality assurance and reflection)? Are there proper processes
on different levels (UNICEF, NPTP, CMU/PMC) in place and how has the programme used
information generated to inform (make) programmatic adjustments and corrective actions
during the life of the project?
Monitoring and Evaluation (M&E)Mechanisms
The Winter Cash programme has specific reporting and monitoring activities:
UNICEF Monitoring & Reporting component including FGDs, third monitoring
process and UNICEF programme monitoring activities.
44
The final evaluation report of the Winter Cash Programme-UNICEF Lebanon. Developed by AwnyAmer- Jan, 2017 Page 44
Partner reporting component including Government reports, quarterly and final
narrative reports and Activity info reporting.
The evaluation found that UNICEF, with the support of key partners, developed a well-structured
system to track the distribution cohorts, delivery system process, and the questions and answers
mechanism, in addition to the complaint mechanism, which was guided and led by WFP. There are
86,000 families registered in the NPTP and other vulnerable lists of the poorest vulnerable families in
the current Winter Cash programme who were selected using the PMT formula with technical
support from the CMU/PCM (guided by the questionnaire and associated forms).
One of the notable M&E achievements was the PDM process which included FGDs with direct
beneficiaries, and key informant interviews conducted with MoSA staff- including the social
workers, field coordinators and the SDC directors. Additionally, a survey was conducted with
beneficiaries using the computer assisted telephone and face to face interviews.
The results framework of the Winter Cash programme referred mainly to the HRRP 2016 where
expected outcomes were identified with planned activities under each objective. The evaluation
recommended that a separate results framework be used, instead of combining it with other activities
addressing the winter basis needs of both vulnerable Lebanese and refugees in Lebanon.
Risk and assumption analysis
Considering the nature of the Winter Cash programme, some of the challenges encountered during
the implementation phase were as follows:
As noted before, 5.6 per cent (approximately 5,000 HHs) of the selected families was not
reached during the distribution phase after two rounds of SMS, telephone calls and field
visits. This highlights the importance of conducting a simple risk analysis and assumption
component as part of the M&E system during the programme design phase.
The evaluation revealed that the reason beneficiaries who received ATM cards, but had
issues with their pin code or other technical issues was due to the lack of an effective
monitoring and follow-up mechanism to track the communication between WFP and the
bank when responding to the complaint mechanism cases reported.
45
The final evaluation report of the Winter Cash Programme-UNICEF Lebanon. Developed by AwnyAmer- Jan, 2017 Page 45
Table8. Monitoring and reporting mechanism
Post distribution monitoring activities
There is a lack of monitoring and post follow-up activities by social workers and field coordinators
that assess changes in the lives of the beneficiaries as a result of their participation in the programme
as well as information on the spending patterns of the cash grants. Based on the absence of a proper
follow-up mechanism, families were not provided with sufficient guidance on the proper use of these
grants to secure the basic needs of their children during the winter season.
Involvement of field coordinators and social workers in M&E activities
The current evaluation highlights a need for the NPTP to enhance the role of social workers and field
coordinators in setting and developing the selection criteria and respective questionnaires and forms
in addition to considering their feedback in a meaningful manner. This corresponded with the results
found in the PDM.
There is an urgent need to train field staff, especially field workers and social workers, on the key
monitoring skills for tracking changes during the life of the programme as a community M&E team.
In this way, the database of targeted families would be updated on a bi-annual or annual basis.
Moreover, they should be involved in the development of the vulnerability assessment process that
includes all the vulnerability indicators.
46
The final evaluation report of the Winter Cash Programme-UNICEF Lebanon. Developed by AwnyAmer- Jan, 2017 Page 46
Question 12: To what extent did the programme consider gender equality in the programme
implementation – including the selection process (e.g. is USD40 an equitable amount for both
boys and girls? On what basis was it calculated and does it take into account gender-based
requirements?)
Gender and selection process
As indicated previously the Winter Cash programme did not specify gender as a variable of the
selection criteria of poor vulnerable families. The Proxy Mean Test formula was used instead to
estimate a family’s consumption per day by referring to other variables such as the employment
status, level of education, marital status, physical ability, housing conditions and geographical
location. Both boys and girls aged 0-15 years were eligible for the cash grant.
Additionally, the evaluation demonstrated the absence of sex disaggregated data among the children
who benefited from the programme. The final payment transfer list includes sex disaggregated data
for the ATM card holder, but not for the beneficiary children.
Following the evaluation, it is highly recommended that the NPTP includes a gender sensitive
component in all cash transfer programmes and that gender be considered not only in the programme
design and planning, but also during the programme implementation phase with special emphasis on
sex disaggregated data.
Gender sensitivity and the adequacy of the cash transfer amount
Regarding the adequacy of the cash transfer amount of USD40, for covering and securing basic
winter needs, the majority of the participants interviewed agreed that there was no significant
difference between the cost of winter clothes items for boys and for girls. They reported that the main
winter item bought in response to the seasonal hazards and unexpected winter emergencies was an
outer winter jacket which can be worn by both male and female children and which can be
exchanged between the two genders.
Gender sensitivity and spending patterns
Interestingly, there were divergent views between men and women in terms of the preferred cash
modality. Direct male beneficiaries, particularly in Mount Lebanon, preferred the restricted cash
transfer especially the e-food voucher on regular basis, while female beneficiaries favoured the
unrestricted cash transfer to set and prioritise their needs. This corresponded to the results of the
PDM process34.
The female participants (parents/caregivers) who took part in the FGDs expressed that they shared
each other’s’ experiences and views that they were the decision makers in terms of the family’s
needs and priorities. It could be concluded that women preferred the unrestricted cash modality as it
was a source of empowerment and engagement in decision-making activities at home. Restricted
cash transfers would isolate a woman from any decision-making process.
34InfoPro_ Winter Cash Assistance- Final report -Post Distribution Monitoring Report-2016_page 31
47
The final evaluation report of the Winter Cash Programme-UNICEF Lebanon. Developed by AwnyAmer- Jan, 2017 Page 47
4.3. Efficiency
Question 13: To what extent was the Winter Cash programme cost-efficient – including
optimal use of the programme resources and time efficiency as opposed to other
modalities/programmes that could be designed to cover the same needs of the Lebanese
vulnerable poor children and their families?
Based on the evaluation, there was a good selection of partners and it was considered as a good
example of a partnerships model. Each of the key partners and actors were well-informed, their main
roles and responsibilities were agreed upon in advance, and specific measures were set to maintain
cost-efficiency during the different phases of the winter cash programme. This model ensures that the
minimum requirements can be realized.
As explained previously in the report, the winter cash programme was designed to provide monetized
assistance to 75,000 children all across Lebanon through a one-off winter grant of USD 40 per child.
Since the programme includes only one activity targeting poor vulnerable Lebanese children and
their families, Table 9 shows the cost transfer per child/family:
Total amount of cash transferred to the bank
Planned
Transferred
Actual delivered
USD
3,000,360
USD
2,800,000
Planned Actual
1 Total number of programme beneficiaries - HHs 26,052 24,601
2 Total number of vulnerable beneficiaries of children 75,009 70,000
3 The average number of children/family 2,87 2.84
4 The average cost per family(HH) USD 115.10 USD 113.80
5 The average cost per child USD 40 USD 40
Table9. Cost transfer per child/family
Cost efficiency and the cash modality
As indicated previously, 88 per cent of the surveyed respondents (programme beneficiaries)35 as well
as the majority of the FGD participants stated their preference for the unrestricted cash transfer
indicating that in their opinion it was more cost-efficient than the restricted cash transfer (e-vouchers)
especially in the absence of WFP as an effective implementing partner.
Cost efficiency and programme outreach
Despite the fact that the majority of the participants during the FGDs and other key partners indicated
that it would be better to target all the families (86,000 HHs) registered in the NPTP, even with an
amount less than USD 40 per child, this would not acceptable in most cases because most, if not all
of the families already have e-cards from WFP and other organisations in addition to the winter cash
35Statistics Lebanon Ltd_ Results of the survey findings _Winter Cash Programme-PDM Report-2016_Slide# 74
48
The final evaluation report of the Winter Cash Programme-UNICEF Lebanon. Developed by AwnyAmer- Jan, 2017 Page 48
grant. This would therefore not be the most cost effective way of reaching vulnerable families.
Offering an amount less than USD 40 might also result in additional problems, since Palestine and
Syrian refugees receive USD 40 per child up to 15 years and could lead to unnecessary competition
and increased tensions between vulnerable groups.
The most prevalent response from beneficiaries regarding the cash grant was the comment that they
felt more respected and treated “like human beings”. “The card is a more respectful tool rather than
having cash handed out to us,” said the SDC director of Baalbek. However, beneficiaries
recommended better management of the distribution process, advising to minimise the flow of people
so that huge numbers do not show up on the same day at the same distribution centre. Respect for
human dignity implies commitment to creating conditions under which individuals can develop a sense of self-worth and security in such situations36.
Cost efficiency and the selection process
It is highly recommended to allocate additional resources for the design of a well-structured,
comprehensive vulnerability assessment which includes all those registered in the NPTP and which
covers all vulnerabilities and welfare variables. This would reduce the risk of selecting those families
who are not in need of the grant and would ensure optimal use of resources.
Cost efficiency and capacity building
It is strongly recommended to provide capacity building for implementing stakeholders - especially
field coordinators, social workers and other MoSA staff. Developing skills in the areas of
communication, interaction, reporting, monitoring, and field work in addition to child rights and
research ethics is most necessary. By building capacity of these stakeholders, implementation of the
programme would be more efficient, reducing errors and leading to greater cost efficiency.
Cost efficiency and the cash transfer process
Collaborating directly with Banque Libano-Francaise enabled beneficiaries to directly withdraw the
cash grant using an ATM card thus reducing any third party costs. This not only ensured the security
of resources, but also reduced the risk of wasted resources during the implementation process,
excluding the need for procurement operations.
Cost and time efficiency
The evaluation found that the verification phase took much longer than planned, leading to a
significant delay in the distribution of ATM cards. Issues included cash withdrawal errors, lack of a
functioning complaint mechanism and lack of coordination and efficient communication between the
bank and implementing staff. The objective of the programme required timely delivery of cash
support to secure winter basic needs. This delay undermined the quality of programme
implementation and beneficiary satisfaction.
Cost effectiveness and value for money
There was a surplus amount of approximately USD 200, 360, reported by the MoSA focal point in
the NPTP for the Winter Cash programme. However, the surplus was a combination of an
36http://hdr.undp.org/en/content/human-development-and-human-dignity- Human development and human dignity- By Aung San
SuuKyi, Winner of the Nobel Peace Prize, 1991
49
The final evaluation report of the Winter Cash Programme-UNICEF Lebanon. Developed by AwnyAmer- Jan, 2017 Page 49
overestimation of the WFP budget and the funds corresponding to the no-show cases of
approximately 5.6 per cent (5,000 children). Greater attention should be given to ensure the full
amount is allocated to the selected vulnerable families as planned or a selection of other eligible
families is conducted in order to reduce the surplus amounts, especially considering the large number
of poor vulnerable Lebanese families.
Due to the large number of beneficiaries visiting the SDC distribution centres, it was considered
impossible to organise the distribution process in a sufficient manner to reach all families with ATM
cards in one day. Additional transportation fees were incurred by families living far from the
distribution centres, which in some cases cost 33 per cent (20,000 LL) of the cash grant. In this
regard, some believed that it would be better to have 50,000 LL (less than the grant amount of USD
40) instead of withdrawing part of this amount and leaving the remainder as some of the other
families had commented. 37.
Some of the key partners (particularly WFP staff) interviewed during the evaluation indicated that
setting up an ATM-card or e-card/voucher system is too costly to for it be cost efficient for a one–off
transfer. Card issuance fees are between USD 4-5 per card in Lebanon in addition to other bank
charges and operational costs. This further confirms the need to maximise the coordination of NPTP,
Government and other actors serving the same vulnerable groups, to consolidate the vulnerability
needs of the beneficiaries based on the results of the vulnerability assessment and (as recommended
above) to ensure that the minimum requirements are taken into consideration and measures are set for
the provision of cash for greater cost-efficiency.
Cost efficiency and the distribution mechanism
As indicated earlier in this report, the distribution mechanism was considered one of the key
challenges in the Winter Cash programme, negatively impacting the cost effectiveness and value for
money of the cash transfer amounts allocated and invested in this programme.
Cost efficiency and risk analysis
The programme design lacked a risk analysis component, which is considered critical for any type of
resources transfer process. Risk analysis is a method that is used to guide the development of
standard operating procedures for cash transfer including threat analysis and acceptable risk
threshold. One relevant example which emerged from the evaluation was the illiteracy of some
beneficiaries who could not use the ATM cards, especially in Akkar and other areas. This
observation confirms that such challenges were not addressed during the trainings provided to social
workers and coordinators before the implementation of the programme.
Finally, there are both advantages and disadvantages to restricted and unrestricted cash assistance.
Unrestricted support shows that distribution is more timely and cost effective, whereas restricted
assistance incurs other logistic activities with additional costs.
37 Beneficiaries’ feedback document__Winter cash programme_ MoSA staff- Summary report page 4
50
The final evaluation report of the Winter Cash Programme-UNICEF Lebanon. Developed by AwnyAmer- Jan, 2017 Page 50
4.4. Sustainability
Question 14: To what extent did the programme equip service providers (NPTP/MoSA) and
beneficiaries with the needed capacities (skills and knowledge) (Human resources'
sustainability), resources (connections) and confidence (Institutional and financial
sustainability) to sustain the programme implementation?
Within the framework of the Winter Cash programme, the NPTP social workers were trained on the
distribution of ATM cards. However, the programme failed to provide well-designed capacity
building training on essential topics, such as: communication skills, teamwork, social work, research
ethics, data collection, data reliability and validity, data analysis, child rights, and monitoring skills.
Without these skills, the programme is not sustainable. It is highly advised to raise the capacities of
the NPTP and MoSA workers to ensure full programme ownership by the NPTP. The aim is to
strengthen capacity and empower personnel in such a way that they can carry out similar future
programmes on their own without receiving technical assistance from partners such as WFP or
UNICEF. The Winter Cash Programme, as a pilot, shed light on the essential capacities if a more
holistic programme is rolled out in the future.
Human resources and institutional sustainability
The evaluation affirms that the capacity building component promoting human resource
sustainability is the most appropriate mechanism in the medium and long-term and would also
contribute in the bolstering institutional sustainability. However, a capacity building approach
requires significant investment in order to provide on-going training for partners in order to develop a
team of field coordinators and social workers who can deliver effective policy making, programming
and application of cash transfer support to poor vulnerable families. Well-trained social workers and
field coordinators optimise key human resources not only within the programme, but also would
enable them to play a vital role as a community monitoring & evaluation team.
Considering the winter cash grant is a one-off cash transfer there is a difficulty in developing
sustainability measures. However, other scaling up mechanisms should be developed on the
institutional and human resources levels (as indicated above).
Although the scope of work of the Winter Cash programme focused on providing monetised
assistance through a one-off winter cash grant of USD 40 per child to secure their basic winter needs
and to help mitigate the effects of seasonal hazards and unexpected emergencies, the evaluation
found that there was an absence of a sustainability strategy to address the graduation approach in
some of the targeted areas – to raise the skills of the targeted groups and helping them to secure
employment opportunities. The evaluation regards this aspect as one of the fundraising mechanisms
to promote financial sustainability where the budget can be invested and allocated to serve other
vulnerable groups.
The programme provided beneficiaries with training on how to use the ATM cards to facilitate
withdrawal of the cash grant. Nevertheless, illiteracy among some beneficiaries- especially in the
Akkar governorate - impacted negatively on the result of the training, and they continued to have
difficulties using the cards. This corresponded with the results of the PDM survey which showed one
51
The final evaluation report of the Winter Cash Programme-UNICEF Lebanon. Developed by AwnyAmer- Jan, 2017 Page 51
of the main reasons for not finding information on the card was illiteracy (16 per cent of interviewed
beneficiaries)38.
4.5. Coordination
Question 15: How effective was the management and coordination system - as part of the joint
programme partnership among UNICEF, NPTP, MoSA, CMU/PCM and WFP- in delivering
the set results to the target population?
Coordination and complementarity among partners
Given the fact that the Winter Cash programme is a pilot project, the evaluation found that the
programme succeeded in identifying and selecting a genuine and strong model of partnership
combining Government entities represented by the NPTP, and the CMU under the Presidency of
Council of Ministries (PCM) in addition to other UN organisations such as UNICEF and WFP. In
this regard, the roles and responsibilities for each partner were more or less defined clearly and
contributed to the realisation of complementarity principles in this joint partnership. Overall, partners
and other different actors had smooth communication between one another throughout the lifecycle
of the programme. However, there was a lack of a well-designed management and coordination
system, particularly with respect to clear identification of the roles and responsibilities of each
partner.
Steering committee
Regular meetings were organised on a weekly basis by representatives from the five partners to
follow-up on the implementation of the programme, ensure that problems were solved in a timely
manner, and maintain effective communication between the implementing staff. The steering
committee also served as a reference for the implementing staff whenever they needed advice,
assistance, or certain exceptional interventions to solve problems or deal with challenges
encountered.
Communication mechanisms
1. SMS
One area of concern was the role of UNICEF in informing the selected eligible families about
the distribution of the ATM cards. The evaluation found that many of the selected families
did not open the SMSs that were sent to them notifying them of the distribution of the cards.
The reasons for this were: they were either illiterate or unable to read the SMS; they thought
that the message was an advertisement; or because their phones could not read Arabic
messages. This corresponded with the findings of the PDM process as reported by the MoSA
staff39.
This is considered one of the mitigation measures taken by UNICEF to ensure the highest attendance
rate of beneficiaries during the distribution process. Consequently, social workers telephoned the
families who were not present at the distribution centres, and visited those who did not have
38Statistics Lebanon Ltd_ Results of the survey findings _Winter Cash Programme-PDM Report-2016_Slide# 29 39InfoPro_ Winter Cash Assistance- Final report -Post Distribution Monitoring Report-2016_page 14
52
The final evaluation report of the Winter Cash Programme-UNICEF Lebanon. Developed by AwnyAmer- Jan, 2017 Page 52
telephones to inform them about the distribution of the cards. This corresponded with the results of
the PDM survey, which showed that 92 per cent40 of individuals received an SMS to collect their
cards compared to 8 per cent who did not.
Although the SMS was very well received by the respondents, many of the MoSA staff disapproved,
as reported in the FGDs facilitated in the PDM41, due to a variety of factors among which was the
belief that many beneficiaries did not read the SMSs or understand them. Additionally, in some
cases, information brochures arrived late and SMSs were delayed. On the other hand, the entire
process had a very short timeframe and that also affected the delivery of information and hindered
the opportunity of creating awareness about the programme in advance as reported by the majority of
beneficiaries in the FGDs facilitated in the PDM process.42
2. Brochures and flyers:
The MoSA staff interviewed in the PDM process indicated that they did not fully approve the
brochures and flyers distributed to beneficiaries for reasons including their late inclusion in this
process; as well as some not being involved in the distribution process until much later thus limiting
their inputs. Other reasons included the illiteracy of many beneficiaries which led to a lack of
interest in the brochure and flyer. This created a feeling of disappointment and lack of trust among
many of the community members towards the coordinators since they expected to benefit from the
programme through the flyers and visits43.
3. Complaint mechanism/Hotline:
The MoSA staff interviewed in the PDM process indicated that many of the beneficiaries had called
the complaint mechanism/hotline phone numbers several times and had to pay money for the
communication, without solving their problems44. Also, many participants reported that they would
have preferred one to one interactions or reaching out to the centres and people they knew better for
help45.
Moreover, the current evaluation observed the absence of following some of the coordination steps
which should have been taken, as well as lack of activation of the complaint mechanism in a timely
manner; this resulted in some families receiving cash transfers that were less than the planned. In this
regard, there were problems in terms of withdrawing the entire amount of money, as beneficiaries
were notified to withdraw the cash in USD 20 notes, and yet these were not available at all ATMs
due to the variation of the denominations from one machine to another. One of the reasons behind
this was the absence of daily or even weekly updates of the bank account balances, demonstrating a
vital communication obstacle with the bank.
Additionally, one of the participants in the FGDs indicated that not all villages and areas of residence
had ATMs close by, which meant that beneficiaries had to travel to the ATMs, sometimes for long
distances, spending a significant amount on transportation fees..
40Statistics Lebanon Ltd_ Results of the survey findings _Winter Cash Programme-PDM Report-2016_Slide# 16 41InfoPro_ Winter Cash Assistance- Final report -Post Distribution Monitoring Report-2016_page 14 42 Beneficiaries’ feedback document__Winter cash programme_ MoSA staff- Summary report page 4 43InfoPro_ Winter Cash Assistance- Final report -Post Distribution Monitoring Report-2016_page 14 44Ibid_page 15 45Ibid_page 24
53
The final evaluation report of the Winter Cash Programme-UNICEF Lebanon. Developed by AwnyAmer- Jan, 2017 Page 53
The evaluation found that the complaint mechanism was one of the gaps identified under the
coordination component. In practice, very few issues (such as the absence of pin codes , or the
incorrect cash amount or other technical ATM related issues) were addressed or resolved when
following the existing complaint mechanism with delays experienced by many.
This observation corresponded with the results of the PDM findings. According to the results, out of
those who confirmed that their case was solved (56 per cent), 60 per cent 46 of them verified that the
processing of their complaint was done in a timely manner, while 40 per cent stated that the process
was not time efficient. For those who were informed about the problem, the majority confirmed (82
per cent) that the interlocutor took their claim into consideration, while the remaining (18 per cent)
did not.
Coordination and distribution process
The current evaluation revealed that the geographical distribution of centres selected as distribution
points was considered a key challenge in terms of good communication. This was associated with
lengthy and difficult travel conditions for many beneficiaries. The distance between many of the
beneficiaries’ homes and their assigned centres was too long, which made the trip tedious and costly.
This resulted in many not attending the distribution of ATM cards, particularly those who had only
one child and others who went to the centres and spent time, effort and money on the trip and who
were dissatisfied and confused by the programme.
Moreover, the operation of distribution at the centres was a cumbersome process, often disorganised
and extremely exhausting for social workers. Some centres were often overloaded and could not cope
with the large number of beneficiaries coming every day. Therefore, it is recommended- as reported
in the current evaluation process and in the PDM process - that other mechanisms be devised either
for wider distribution of all the SDCs or covering the transportation fees for those for whom it has
been proved that they are too far from the distribution centre.
Coordination and follow-up meetings
Weekly meetings were held among the key partners through the steering committee to manage and
steer the programme activities in a timely manner. A similar meeting should be organised on a
regular basis between the WFP and the bank to get ensure cash transfer balances of the bank accounts
of the ATM holders. This would enable those responsible for the complaints mechanism to track
complaints concerning any of the ATM related issues. It is worth noting that the WFP staff
interviewed in the current evaluation indicated that specific agreements were consequently made
with the bank to ensure updates are received on these balances for any future cash transfer
programmes.
46Statistics Lebanon Ltd_ Results of the survey findings _Winter Cash Programme-PDM Report-2016_Slide# 56
54
The final evaluation report of the Winter Cash Programme-UNICEF Lebanon. Developed by AwnyAmer- Jan, 2017 Page 54
5. Conclusion
Overall, UNICEF and its key partners have successfully achieved the outcomes anticipated for the
Winter Cash programme. There was a high level of satisfaction among the direct beneficiaries
commenting that the assistance came at a right time considering the challenges that are being
encountered to find job opportunities during winter in some regions. However, complaints
highlighted the insufficiency of the cash amount. On the other hand, the ATM card was perceived as
tool that made them feel proud, free, respected, safe and valued not to have to wait in line and beg
for support as happens with other modalities. However, some beneficiaries commented that they
were unable to withdraw the full amount due to the different denominations. Others who could not
use the ATM cards sought help from their neighbours, children and family members.
Both MoSA staff and the direct beneficiaries perceived the selection process as unfair and having left
many of those in need with no support, while some who were very less vulnerable received the
assistance. Additionally, the selection process was not well understood by social workers and field
coordinators who were not aware of the selection criteria. They expressed their willingness to take
part in the selection process and the added value their knowledge of communities and vulnerable
families would bring to the process. Many believed that the selection process should be more based
on more accurate information and closer observation of each household’s needs and priorities.
The flier and SMS, used as communication tools, were criticized as being insufficient in meeting the
needs of the beneficiaries. Illiteracy among vulnerable groups and lack of feedback from the
complaint mechanism led to dissatisfaction among beneficiaries and MoSA staff. In some cases, the
brochures arrived late and SMSs were delayed. The programme was conducted within a short time
frame and this hindered the ability to create much need awareness about the process and objectives.
In terms of the distribution process, the majority of beneficiaries were satisfied with the steps taken
including the SMS, direct contact by telephone and direct communication in some cases. However,
the selection process was widely criticised by other beneficiaries and MoSA staff because of the
distribution locations and regrouping of several smaller community-based centres into one large
centre, making travel burdensome, and internal organisation issues difficult to manage. It was
described as chaotic and disorganised in some SDCs. Other tensions occurred between members of
communities and the SDCs workers when individuals did not understand why they had been
excluded from the programme and SDC staff was unable to explain the selection criteria and process.
Finally, the unrestricted cash transfer modality was perceived as a positive method of support by
most of the beneficiaries and SDC staff enabling families to spend the cash on what they think is
important to them at any time, based on their priorities. However, they indicated the need to revisit
the cash amount and the possibility of receiving it on a recurrent basis. Contrary to this, most of the
MoSA, NPTP and WFP staff in addition to some SDC workers who were interviewed preferred the
restricted cash transfer, especially the e-food voucher and e-vouchers for clothing or fuel in order to
ensure the best usage of the cash amount to secure the basic needs of the vulnerable groups
particularly children.
55
The final evaluation report of the Winter Cash Programme-UNICEF Lebanon. Developed by AwnyAmer- Jan, 2017 Page 55
6. Lessons Learned
During the implementation of the current Winter Cash programme evaluation the following lessons
learned were identified based on several observations gleaned from the key informant interviews and
FGDs conducted in addition to the data captured from the PDM process-including the survey
findings and FGDs facilitated with a sample of the programme beneficiaries:
Selection of the most appropriate cash modality and delivery mechanism should be made in
consultation with the field coordinators, social workers and other relevant actors, including a
sample of the direct beneficiaries, given the mixed opinions that exist between beneficiaries
and the NPTP and considering the risks associated with both the restricted and unrestricted
cash modalities.
The lack of involvement of field coordinators, social workers and other SDC staff in the
selection criteria and process is a lesson to be learned. It is strongly recommended that they
be involved and consulted through different feedback and reflection activities in order to
ensure a higher level of fairness in the data collection and selection processes.
The absence of accurate, up-to-date and complete data on the poorest vulnerable families
registered in the NPTP led to concerns about the relevancy of the selection process, which
might not have taken into account the poorest vulnerable families- including children with
disabilities, orphans, the unemployed parents and/or caregivers, and others.
Lack of coordination and the importance of the complaint mechanism as a means for
effective management requires improved follow-up with the bank, by WFP, in order to
receive updated balances of the bank accounts.
The lack of relevant skills of the field coordinators and social workers played a vital role in
understanding the training provided by NPTP. This requires special attention to strengthen
the human and institutional capacities of the NPTP towards improved management of future
cash programmes, particularly with regards to skills related to teamwork, communication,
monitoring and field work.
NPTP should develop their own effective monitoring system that includes quality assurance
and a performance management component. In addition, there is a need for better knowledge
management within the NPTP to ensure the quality of the data and information.
56
The final evaluation report of the Winter Cash Programme-UNICEF Lebanon. Developed by AwnyAmer- Jan, 2017 Page 56
7. Key recommendations
Based on this review and assessment of the Winter Cash programme, the consultant proposes the
following recommendations for improvement of the programme, should it be expanded or
replicated in other areas in the future. The majority of these recommendations were derived from
the analysis of the data collected during the interviews and FGDs with the different actors. These
recommendations will be presented in sub-categories in bullets as follows:
Programme approach, design and interventions
It is highly recommended that a multi-dimensional approach be adopted when developing
cash-based programme interventions within NPTP. This approach comprises an integrated
and comprehensive package of activities. Programme developers should answer the “Cash
for what?” question during the design phase, as the cash modality should not be seen as an
end in itself or as a standalone programme. (Relevance & & appropriateness findings #1-page 26)
Special attention should be given to developing a separate, well-structured programme
proposal or concept note, considering that it is part of the Country Programme Document
(CPD). This plays a major role in explaining the programme to partners and other
stakeholders. This document should include the programme’s objectives, activities, outputs,
outcomes, M&E mechanism, logical framework, duration, target groups and sustainability
mechanism. (Relevance & appropriateness findings#5-page 34)
Having a complete form of the theory of change that combines between vertical & horizontal
logic template and the traditional logical framework matrix is helpful in humanitarian aid
programmes in order to track whether the outputs are achieved and to identify the means of
verification through which data and evidence are communicated to confirm the indicators. In
addition, risks and assumptions that might be faced or encountered as a result of external
factors during the implementation of the program should be stated.
A gender component should be devised for the programme, and managed and led by NPTP.
This should state how the programme will be guided by equality and equity measures relating
to gender during the different phases and should include the need for disaggregated data. (Relevance & appropriateness findings#4-page 33 + Effectiveness findings # 12-page46)
The age of eligibility of children should be revisited, with the possibility to extend assistance
to children aged up to 18 years (in compliance with the definition of a child adopted by
UNICEF) and/or agree jointly on the maximum number of children who will be supported
per family considering the available financial resources. (Relevance & appropriateness findings#1-
pages 28-29)
The cash transfer amount should be revised to ensure that it is adequate to secure basic winter
needs of vulnerable children. Consideration should also be given to the supplementation of
transportation fees of those who travel to collect the ATM card from their identified SDC.
)Effectiveness findings # 6-page 36)
Capacity building
It is highly recommended that the NPTP provide capacity building activities for field
coordinators and social workers and invest in raising their capacities and skills in the areas of
communication, field work, monitoring, mentoring, follow-up, data collection, data integrity,
data validity, child rights, and research ethics. With these skills, they can play a vital role in
updating the existing data regarding vulnerable families in Lebanon and work as a permanent
community M&E team not only for the Winter Cash programme but also for other future
programmes. This could be organised in the form of a 4-5 day training workshop to cover the
57
The final evaluation report of the Winter Cash Programme-UNICEF Lebanon. Developed by AwnyAmer- Jan, 2017 Page 57
basic information and skills under each of these areas. Investing in capacity development
will also promote enhance institutional and human resources sustainability. (Relevance &
appropriateness findings#1-page 27 + Effectiveness findings # 10-page 43)
Awareness raising sessions for parents should be included in the design of the programme to
educate them about spending the money appropriately on the needs of the children, and
explain to them the main objective of the programme, highlighting children as the main target
group. (Relevance & appropriateness findings#1-page 28)
Selection process
Immediate resources should be allocated by the NPTP to develop a well-structured
vulnerability assessment for the communities and families registered in the NPTP programme
to gather accurate baseline data that can be referred to when needed in other similar
programmes.. This requires the compilation of accurate, complete and updated data that
addresses all the vulnerability and welfare indicators (employment and income status,
number of family members, illness status, disability, gender variables, education status,
orphans’ status, house conditions and others).The NPTP has already started this process. (Relevance & appropriateness findings#2-pages31- 33 + Effectiveness findings pages 35-37)
It is highly advised to promote and increase the involvement of the field coordinators, social
workers and other actors in the SDCs in the selection process starting from jointly developing
the selection criteria and including the SDCs in the implementation phase as partners and not
only as implementing mechanisms. (Relevance & appropriateness findings#2-page 32)
The questionnaire must be revised as it currently does not include the minimum assessment
requirements and variables; this will enable the accurate identification of vulnerable poor
families referring to specific vulnerability variables. (Relevance & appropriateness findings#2-page
31 + Effectiveness findings #1- page 37)
Coordination
It is recommended to continue working with the key partners following the format of the
steering committee - which represents all the partners and different actors. - A
communication and coordination plan outlining specific roles and responsibilities to guide
them during the different phases of the programme should be developed. The steering
committee should meet regularly and on a timely basis to immediately deal with the
challenges and problems encountered. UNICEF has announced that these meetings will
continue in future cash programmes. (Relevance & appropriateness findings#2-page 32 + Effectiveness
findings page 39)
It is highly advised during the implementation to set unified guidelines and principles for all
the SDCs to avoid inconsistency when finding solutions to issues. For example, in some of
SDCs field coordinators monitored the social workers in the field while others did not follow
this approach. (Coordination findings # 15-pages 51-52 + Efficiency findings- pages 36-37)
Cash modality
If in the future the restricted cash transfer (e-vouchers) - based on the recommendations of
some groups interviewed in the current evaluation process – is adopted, it is highly
recommended to engage in a joint partnership with other organisations such as WFP and
UNHCR where each partner can complement the other to address the needs of beneficiaries. (Effectiveness findings # 9- pages 41-42 + Annex 8 attached to this report)
It is highly recommended that NPTP continues to implement the unrestricted cash transfer,
which is the preferred cash modality of the majority of beneficiaries, but with some
conditions/or alternatives included:
58
The final evaluation report of the Winter Cash Programme-UNICEF Lebanon. Developed by AwnyAmer- Jan, 2017 Page 58
Ensure the selection process reaches the poorest vulnerable families fairly.
Provide the cash transfer on a monthly or quarterly basis to secure basic needs.
Increase the cash transfer amount significantly in order to secure basic needs especially in the
winter season when employment opportunities are scarce. (Effectiveness findings # 9- page 42 +
Annex 8 attached to this report)
The majority of the participants indicated that they prefer the e-food voucher if they can
receive it regularly and not once per year, as it represents the best e-voucher modality
compared to the e-voucher for fuel or clothes as mentioned in the report. This preference is
explained by the fact that they need to have control in the malls to ensure that they deliver
food items not other non-food items in a corrupt manner. (Effectiveness findings # 9- pages 41-42 +
Annex 8 attached to this report)
Monitoring & Evaluation
Special attention should be given at the NPTP level to developing a well-structured M&E
system that comprises methodologies and tools, including some of the community reflection
tools linked to a combination of qualitative and quantitative indicators.. Accordingly, the
selected field coordinators and social workers in each community/area per SDC will work as
a community M&E team by incorporating a reflection and reporting component when
meeting with the direct beneficiaries and make corrective action when needed. (Effectiveness
findings # 11- pages 43-44)
Best practices should be supported and documented. These examples or success stories can
be shared with others for learning purposes. (Effectiveness findings # 11- pages 43-44)
Special attention should be given to increasing the involvement of the field coordinators and
social workers in the different phases of programme implementation. (Effectiveness findings #
11- page 45)
Joint partnerships It is recommended to develop joint partnerships with organisations who work with the same
vulnerable families registered in the NPTP in the targeted areas. Through this partnership, the
results of the vulnerability assessment can be used and the areas of programme interventions
can be jointly developed referring to the scope of work of each organisation. Based on such
practice, duplication of services can be avoided and organisations can complement each other
through an integrated approach. (Relevance & appropriateness findings#2-pages 31- 33)
The evaluation suggests investigating the possibility of targeting the winter needs of some
groups whose profile might not be consistent with the scope of work or the mandate of some
of these organisations. (Effectiveness findings # 6- page 36)
Scaling up opportunities In order to promote the institutional sustainability of the NPTP and strengthen its ownership in other
similar programmes, special attention should be given to adopt a well-structured strategy that
includes the following sub-strategies (Sustainability findings # 14-page 50):
Implement a comprehensive well-designed capacity building training in advance, before the
beginning of the project implementation.
Provide awareness sessions for targeted parents and/or caregivers to ensure that they will use
the cash transfer assistance appropriately and secure the basic needs of their children during
the winter season.
Conduct a vulnerability assessment to collect accurate and updated data on vulnerability and
welfare variables.
59
The final evaluation report of the Winter Cash Programme-UNICEF Lebanon. Developed by AwnyAmer- Jan, 2017 Page 59
Promote integration with the other organisations working with the same targeted groups to
maximise the optimal use of financial resources.
Set clear, structured communication and coordination mechanisms among the different actors
including the key partners and the bank. This will be helpful in effective implementation of
programme activities and in a timely manner and identify and manage gaps.
Develop a well-structured monitoring and quality assurance framework with clear
mechanisms, methodologies and tools to track the progress of the programme.
60
The final evaluation report of the Winter Cash Programme-UNICEF Lebanon. Developed by AwnyAmer- Jan, 2017 Page 60
Annexes
Annex 1: The Terms of Reference (TOR)
61
The final evaluation report of the Winter Cash Programme-UNICEF Lebanon. Developed by AwnyAmer- Jan, 2017 Page 61
62
The final evaluation report of the Winter Cash Programme-UNICEF Lebanon. Developed by AwnyAmer- Jan, 2017 Page 62
63
The final evaluation report of the Winter Cash Programme-UNICEF Lebanon. Developed by AwnyAmer- Jan, 2017 Page 63
64
The final evaluation report of the Winter Cash Programme-UNICEF Lebanon. Developed by AwnyAmer- Jan, 2017 Page 64
65
The final evaluation report of the Winter Cash Programme-UNICEF Lebanon. Developed by AwnyAmer- Jan, 2017 Page 65
66
The final evaluation report of the Winter Cash Programme-UNICEF Lebanon. Developed by AwnyAmer- Jan, 2017 Page 66
67
The final evaluation report of the Winter Cash Programme-UNICEF Lebanon. Developed by AwnyAmer- Jan, 2017 Page 67
68
The final evaluation report of the Winter Cash Programme-UNICEF Lebanon. Developed by AwnyAmer- Jan, 2017 Page 68
Annex 2: List of stakeholders interviewed
S Name Sex Position Affiliation /
Governorate
Date of
interview/F
GD
M F Total
(1) Key partners ( Individual & group interviews) 1 Mrs. Marie Camaghia 0 1 1 Winter Cash programme Focal
Point NPTP/MoSA 16/11/2016
2 Mr. Ramzy Fanous 1 0 1 CMU/PCM statistician CMU/PCM 24/11/2016
3 Mr. Michele Sarkis 1 0 1 IT Business officer NPTP/MoSA 28/11/2016
4
Mrs. Fouz Quebesi 0 1 1 Programme Policy officer-
Formerly cash & transfer associate WFP 29/11/2016
Mr. Charbel Habib 1 0 1 Head of Cash Transfer
Programming Unit - Formerly cash
& transfer associate
5 Dr. Bashir Osmat 1 0 1 The adviser of the MoSA MoSA 29/11/2016
Sub-total 4 2 6
(2) Other stakeholders ( Individual, group interviews & FGDs)
A. Individual interviews:
6 Mrs. Naziha Dakroub 0 1 1 Chiyah SDC Director Mount Lebanon 24/11/2016
7 Mrs. Rajia Bitar 0 1 1 Rahbe SDC Director Akkar 2/12/2016
8 Mrs. Sahar Alawani 0 1 1 Al Mina SDC Director Tripoli 6/12/2016
9 Ms. Hoda Said 0 1 1 Baalbek SDC Director Bekaa/MoSA
20/12/2016
10 Mr. Wael Assaf 1 0 1 Director of SDC directors of
Bekaa 11 Mrs. Fatima Sawrli 0 1
1 Ex- Field coordinator - Chiyah
SDC
Mount Lebanon
24/11/2016
12 Mrs. Fatma Zeater 0 1 1 Field Coordinator - Chiyah SDC 1/12/2016
13 Mr. Hayssam Habib 1 0 1 Field Coordinator – Rahbe SDC Akkar 2/12/2016
14 Mona Raad 0 1 1 Field Coordinator- Baalbek SDC Bekaa 20/12/2016
Sub-total 2 7 9
B. Group interviews: 15 1. Hussein
Tabaja
2. Amira Khalil
3. Fatma Gaafar
4. Maha
Dakrouni
5. Amel Merwa
1 4 5 Social workers-Goberiri SDC
Mount Lebanon
1/12/2016
16 1. Khetam
Mohamed
2. Wafaa Elias
0 2 2 Social workers-Rahbe SDC
Akkar 2/12/2016
17 1. Selvia
Antonios
2. Dina Gamous
0 2 2 Social workers-Al-Mina SDC
Tripoli 6/12/2016
18 1. Amira
Sharamnt
2. Wael Negro
3. Belal Kadri
0
1
1
1
0
0
1
1
1
- SDC Director- Bab E-
Tabaenh
- Field coordinator – “ “
- 4 Social workers - “
Tripoli 13/12/2016
69
The final evaluation report of the Winter Cash Programme-UNICEF Lebanon. Developed by AwnyAmer- Jan, 2017 Page 69
4. Fawaz
Moreaby
5. Nivine El-
Gendy
6. Farah Saleh
1
0
0
0
1
1
1
1
1
“
19 1. Mary
Ghanem
2. Walaa El-
Nemr
3. Nour El-Hoda
4. Laila Asaf
5. Rawia Saleh
6. Waad El-Haj
7. Zeinab Ismail
0
0
0
0
0
0
0
1
1
1
1
1
1
1
1
1
1
1
1
1
1
Social workers- Baalbek SDC
Bekaa 20/12/2016
Sub-total 4 1
8
2
2
C. Focus Group Discussions with sample of programme beneficiaries: 20 FGD with sample of
Programme
Beneficiaries
(All were parents
and/or caregivers)
2 6 8 Ghobeiri SDC Mount Lebanon 1/12/2016
21 2 8 10 Rahbe SDC Akkar 2/12/2016
22 1 12 13 Al-Mina SDC Tripoli 6/12/2016
23 0 9 9 24 11 2 13 Baalbek SDC Bekaa 20/12/2016
Sub-total 1
6
3
7
5
3
Total 2
6
6
4
9
0
Classification of stakeholders interviewed per category
S Stakeholders category/type # of
interviewed
1 Key partners 6
2 SDC/MoSA directors 6
3 Field Coordinators 5
4 Social Workers 20
5 Beneficiaries 53
Total 90
70
The final evaluation report of the Winter Cash Programme-UNICEF Lebanon. Developed by AwnyAmer- Jan, 2017 Page 70
Annex 3: List of documents consulted
The following are the key documents consulted and referred to during the evaluation process:
TOR of the evaluation of the Winter Cash programme.
Overview about the Winter Cash programme “Leaflet”.
Basic Assistance/winter HRRP 2016.
NPTP – UNICEF Winterisation programme Implementation Plan.
PPT about the Post Distribution Monitoring- Lessons learnt document- April, 2016.
PDM Final products including FGD Summary Reports and Survey findings.
Distribution documents including: banners, card issuance (household registry statement),
card loading (payment list), card transportation (delivery notes, Clustering (clustering of
NPTP UNICEF beneficiaries, distribution cohorts, questions and answers and programmatic
information for social workers, templates and standards operation procedures distribution
and timeline & distribution rate tracking.
Complaint mechanism hotline including logbook & procedures.
Card Management NPTP.
Questionnaire administrated to select the poor vulnerable families.
71
The final evaluation report of the Winter Cash Programme-UNICEF Lebanon. Developed by AwnyAmer- Jan, 2017 Page 71
Annex 4: Data collection instruments – Methodologies & tools’ guides
Annex 4.1 - Focus Groups Discussions (FGDs) Guide
(With sample of programme beneficiaries: Children & their families)
Key Guidelines
Focus group discussions each had a maximum of 8-12 participants with an average of 10
persons including parents and/or caregivers from the Lebanese poor vulnerable families. The
actual number of participants ranged between 8-13 participants.
At the beginning of the FGD, the consultant/facilitator provided a short introduction to
explain the purpose of the evaluation process, and the objective of conducting the FGD and
why they were selected to take part in the activity.
The FGD was conducted by one facilitator and one junior researcher, who documented the
responses, allowing the facilitator to focus on managing and facilitating the FGD.
The facilitator asked participants for their consent to use the information collected during the
FGD for inclusion in the evaluation.
The facilitator conducted the FGD in an interactive manner, giving participants the
opportunity, to express their views, insights and ideas freely.
The seating arrangement was shaped in a semi-circle to allow all participants to take part in
the FGD and to allow the facilitators to observe any non-verbal communications.
A documentation form was developed to document the name, age, profession and sex of
participants and other basic data about the participants in the FGDs (the names will be
referred for internal use only- when needed).
The duration of the FGD was between 60-90 minutes.
At the end of the FGD, the facilitator summarised the information recorded and thanked the
participants for their feedback and time.
Selection criteria
The total number of participants involved in each of the FGDs ranged from 8-13 participants.
All the participants were selected from the parents and caregivers of poor vulnerable families.
Gender of participants was considered equitably to include both male and female participants.
Despite the fact the all participants are supposed to be among the most/vulnerable poor
children and their families, they were in fact selected from different socio-economic status.
Despite the fact that the initial criterion was intended to include at least one person with
disabilities, this was not implemented in practice as it was not possible.
Key issues/questions that were discussed in the FGDs facilitation process
Discuss the orientation received about the selection & distribution processes.
Discuss the support received from the SDCs and social workers involved in the distribution
process
How did you spend the cash transfer amounts and in which aspects of your lives (e.g.
education, services, health, transportation, food, clothes, etc.)
What are the key changes captured in your lives as a result of your participation in this
programme? Elaborate.
72
The final evaluation report of the Winter Cash Programme-UNICEF Lebanon. Developed by AwnyAmer- Jan, 2017 Page 72
As one of the recipients of cash transfer assistance, are you satisfied about the programme
implementation?
To what extent do you think that the selection process of the targeted poor families followed
what was explained at the beginning of the process through the orientation sessions?
What do you think about the timing of the cash transfer process referring to the winter
season?
What do you think about the level of support received from the MoSA (SDCs social workers
during the different phases of the programme implementation?
What were the key challenges and weaknesses encountered and how were they addressed?
Who did you prefer to contact to deal with this issue?
How well did the complaint mechanism work from your point of views? Explain with some
examples?
What is the level of acceptance of the cash transfer modality from your point of view?
If yes, why do you see this modality as the best one in the winter cash assistance programme?
If not, from your point of view, what are the other modalities that can work much better? And
why? Explain with some examples.
From your point of view, in case of replicating the same programme, what we can be done in
a better way to improve the quality of the programme implementation?
--------------------------------------------------------------------------------------------------------------------------
Annex 4.2 – Semi-Structured Interview Guide
(Key partners/stakeholders: NPTP, MoSA, CMU/PMC, WFP and others)
Interviewee Name: Age:
Position: Sex:
Affiliation:
Date:
The venue of the interview:
Interviewer:
Note Taker (Documenter
At the beginning of the interview, the consultant/interviewer provided a short introduction to
explain the purpose of the evaluation process, and the objective of conducting the interview
and why they were selected to take part in this activity and how the information collected will
be used.
The interview lasted between 60 to 75 minutes, in a location preferable to the interviewee.
Key issues/questions that were discussed with the interviewee
Discuss the role of the interviewee and his/her affiliation in the programme implementation?
From your point of view, to what extent were the programme design and planned activities
adequate promoting the achievement of the programme objectives?
To what degree did the programme meet the needs of the most socio-economically
vulnerable Lebanese Households?
To what extent was the criteria of the selection process developed while ensuring that the
most vulnerable and poor children were reached and benefited properly from the programme?
73
The final evaluation report of the Winter Cash Programme-UNICEF Lebanon. Developed by AwnyAmer- Jan, 2017 Page 73
What is the added value of having such an integrated approach with the different selected
partners/actors in this joint programme?
If there are some specific successes, what are the key factors that led to that success?
What are the key challenges/constraints (hindering factors) that influenced the achievements
of the programme objectives? How were they addressed?
How effective was the management and coordination system - as part of the joint programme
partnership among UNICEF, NPTP, MoSA, CMU/PCM and WFP in delivering the set
results to the target population?
From your point of view, to what extent has the planned and implemented response reduced
vulnerability among the targeted groups?
What is your level of acceptance of the cash transfer modality from your point of view?
If yes, why do you see this modality as the best one in the Winter Cash programme?
If not, from your point of view, what are the other modalities that can work much better? And
why? Explain with some examples.
To what extent were the capacities of the different actors- particularly the NPTP and MoSA -
adequate and efficient to ensure the quality of the programme implementation? What can be
done in a better way in this regard?
To what extent was the Winter Cash programme cost-efficient – including the optimal usage
of the programme resources & time efficiency as opposed to other modalities that could be
designed to cover the same needs of the Lebanese vulnerable poor children and their
families?
To what extent did the programme equip service providers (NPTP/MoSA) and beneficiaries
with the capacities (skills and knowledge) (Human resources' sustainability), resources
(connections) and confidence (Institutional and financial sustainability) to sustain the
programme implementation?
What are the key lessons learnt from the programme implementation?
What can be done in a better way to improve the programme effectiveness?
Others (E.g. consider the gender equality during the evaluation process?
Annex 4.3 – Semi-Structured Group Interview Guide
(With sample of MoSA social workers and SDCs coordinators)
Date:
The venue of the interview:
Interviewer:
Note Taker (Documenter
S Interviewee name Age Sex Position Affiliation
74
The final evaluation report of the Winter Cash Programme-UNICEF Lebanon. Developed by AwnyAmer- Jan, 2017 Page 74
At the beginning of the interview, the consultant/interviewer provided a short introduction to
explain the purpose of the evaluation process, the objective of conducting the interview, why
they were selected to take part in this activity and how the information collected will be used.
The interview lasted between 60 to 75 minutes, in a location preferable to the interviewee.
Key questions/issues will be discussed with the interviewee
The role of the interviewee and his/her affiliation in the programme implementation?
From your point of view, to what extent were the programme design and planned activities
adequate in promoting the achievement of the programme objectives?
To what degree did the programme meet the needs of the most socio-economically
vulnerable Lebanese households?
To what extent were the capacities of the different actors- particularly the NPTP & MoSA -
adequate & efficient to ensure the quality of the programme implementation? What can be
done in a better way in this regard?
To what extent were the criteria of the selection process developed while ensuring that the
most vulnerable and poor children were reached and benefited properly from the programme?
What are the key challenges/constraints (hindering factors) that influenced the achievements
of the programme objectives? How were they addressed?
From your point of view, to what extent has the planned and implemented response reduced
vulnerability among the targeted groups?
What is your level of acceptance of the cash transfer modality from your point of view?
If yes, why do you see this modality as the best one in the Winter Cash programme?
If not, from your point of view, what are the other modalities that can work much better? And
why? Explain with some examples.
What are the key lessons learnt from the programme implementation?
What can be done in a better way to improve the programme effectiveness?
75
The final evaluation report of the Winter Cash Programme-UNICEF Lebanon. Developed by AwnyAmer- Jan, 2017 Page 75
Annex 5: Evaluator Bio Data
Awny Amer is working as an independent Research, Monitoring & Evaluation consultant, based in
Egypt. He has 30 years of experience in development & evaluation. Before, he worked with Plan
International Egypt for about 22 years before starting his role as independent R&E consultant starting in 2009.
Through his career in the R&E field, he had the opportunity to lead and conduct different R&E
processes, surveys, and situations analysis for diverse programmes addressing children, youth,
women and other vulnerable groups with most of the UN and INGOs at global, regional and national
levels- including UNICEF, UNDP, UN Women, WFP, ESCWA, Oxfam, IDRC, CARE, Save the
Children, Inter News, Search for Common Ground (SFGG), UNRWA, and Plan International among
others. He has good regional experience (MENA) region where he worked in different MENA
regional countries.
In this regard, he always adopts the application and usage of a combination of the qualitative &
quantitative with special focus on the participatory M&E approach, methodologies and tools
including Most Significant Change (MSC) technique, storytelling techniques, longitudinal study
processes, FGDs, case studies, community reflection tools and others in a consultative manner during
the evaluation process to promote meaningful participation and strengthen the accountability practices on different levels.
Currently, he is acting as board member of the International Development Evaluation Association
(IDEAS), African Policy Centre (APC), the Egyptian R&E network in Egypt (EREN) and the
Community of Evaluators (CoE) in addition to acting as one of the founders of the EvalMENA and
active member at AfrEA. He had the opportunity to develop different papers and abstracts where
they were submitted to different international evaluation conferences. Also, he had the opportunity to
translate some of the key R&E publications such as the “The Road to Results” book, the Most Significant Change technique (MSC) guide among others.
76
The final evaluation report of the Winter Cash Programme-UNICEF Lebanon. Developed by AwnyAmer- Jan, 2017 Page 76
Annex 6: Evaluation Matrix
Criteria Evaluation Question(s) Method(s) Target
Groups(s)
Triangulation
Relevance 1. To what extent were the
scope of project
intervention strategies,
project design and
approaches appropriate and
relative to promoting the
intended objectives of the
programme? And to what
extent the needs of the most
socio-economically
vulnerable Lebanese
households were met?
2. To what extent were the
criteria of the selection
process developed while
ensuring that the most
vulnerable and poor
children were reached and
benefited properly from the
program?
3. How gender sensitive was
the programme in reaching
the most vulnerable
children?
4. To what extent has the
programme been successful
in identifying the most
relevant partner/actors for
this project and what was
the added value of these
partnerships?
5. Do the current Log frame
and/or results framework
and relevant indicators in
the Winter Cash
programme document
need improvement to
better serve the
programme objectives?
Semi-
structured
interview
Relative
programme staff
of UNICEF
Lebanon
Semi-structured
interviews with
key partners (in
some/all parts of
these issues)
FGDs with the
targeted groups of
the vulnerable
families
The individual and
group interviews
with the field
coordinators and
social workers.
77
The final evaluation report of the Winter Cash Programme-UNICEF Lebanon. Developed by AwnyAmer- Jan, 2017 Page 77
Effectiveness
6. To what extent did the targeted
vulnerable children & their
families of the Winter Cash
programme were reached by
the programme as planned?
How satisfied were they about
the programme?
7. To what extent did the
programme have an effect on
beneficiaries in improving
access to essential goods and
services of their choice in a
safe, dignified, and empowered
manner while decreasing socio-
economic vulnerability? (Key
changes impacted on the lives
of beneficiaries as result of the
programme implementation)?
8. Are there other unintended or
unforeseen changes shown in
the lives of the targeted groups
of poor children & their
families? Also, are there
unintended beneficiaries in the
programme?
9. What was the level of
acceptance of the cash modality
as a tool to support poor
Lebanese households to cover
their winter needs
(winterisation) from the point
of views of all the different
actors: MoSA, NPTP, SDCs,
field staff and the beneficiaries?
10. To what extent the capacities of
the different actors- particularly
the NPTP & MoSA actors
involved were adequate and
efficient to ensure the quality of
the programme
implementation? What can be
done in a better way in this
regards?
11. To what extent does the
programme have an effective
Focus-
Group
Discussion
s
Key direct
beneficiaries
(Lebanese poor
vulnerable
children and their
families)
(Some of the
questions
discussed with
the beneficiaries
will be discussed
with partners:
UNICEF, WFP
staff)
1-Semi-Structured
interviews with the
concerned
UNICEF Lebanese
Staff and WFP
2-Secondary
analysis review
3- Semi-structured
interview with
decision makers in
MoSA and NPTP
78
The final evaluation report of the Winter Cash Programme-UNICEF Lebanon. Developed by AwnyAmer- Jan, 2017 Page 78
monitoring & evaluation
mechanism (including
reporting, quality assurance and
reflection) processes on
different levels (UNICEF,
NPTP, CMU/PMC) in place
and how has the programme
used information generated to
inform programmatic
adjustments and corrective
actions during the life of the
project?
12. To what extent did the project
consider gender equality in the
programme implementation –
including the selection process
(e.g. is USD40 an equitable
amount for both boys and girls?
On what basis was it calculated
and does it take into account
gender-based requirements?)
Efficiency 13. To what extent was the
Winter Cash programme
cost-efficient – including
the optimal usage of the
programme resources &
time efficiency as opposed
to other
modalities/programmes that
could be designed to cover
the same needs of the
Lebanese vulnerable poor
children and their families?
Semi-
structured
interviews
Key partner(s)
(NPTP and
MoSA)
Secondary data
analysis & review
Semi-structured
interviews with
key UNICEF and
WFP programme
staff involved in
the program
Sustainability 14. To what extent did the
programme equip service
providers (NPTP/MoSA) and
beneficiaries with the capacities
(skills and knowledge) (Human
resources' sustainability),
resources (connections) and
confidence (Institutional and
financial sustainability) to
sustain the programme
implementation?
Semi-
structured
interviews
Key partner(s)
(NPTP & MoSA)
Secondary data
analysis & review
Semi-structured
interviews with
key UNICEF &
WFP programme
staff involved in
the programme
79
The final evaluation report of the Winter Cash Programme-UNICEF Lebanon. Developed by AwnyAmer- Jan, 2017 Page 79
Coordination
15. How effective was the
management and
coordination system - as
part of the joint programme
partnership among
UNICEF, NPTP, MoSA,
CMU/PCM and WFP in
delivering the set results to
the target population?
Semi-
structured
interviews
Key partner(s)
(NPTP and
MoSA- including
the SDCs
directors & field
coordinators)
Secondary data
analysis & review
Semi-structured
interviews with
key UNICEF &
WFP programme
staff involved in
the programme
80
The final evaluation report of the Winter Cash Programme-UNICEF Lebanon. Developed by AwnyAmer- Jan, 2017 Page 80
Annex 7: Results framework
Results Chain (Operational & Developmental results)
Overall goal: “Improve the current conditions of the vulnerable & most vulnerable Lebanese poor children
and their families”
Outcome 1: Lebanese children and the families vulnerable to seasonal hazards and unexpected winter
emergencies are able to maintain safe access to goods and services.
Outcome 2 – Strengthened delivery mechanisms for the NPTP via the incorporation of new systems : Cash
based programme
A.1 Inputs A.2 activities A.3 Outputs A.4 Outcomes A.5 Impacts
Funding and
human resources
on different levels
(UNICEF staff,
NPTP, MOSA
and other actors &
partners) to
support the
implementation of
the programme
activities aimed to
improve the
current situation
of the Lebanese
poor children &
their families in
the targeted areas
of the programme.
Cash transfer
processes and other
relative supportive
guidance steps
provided by/and to
the key partners to
promote use of these
resources to help
achieve programme
outcomes & outputs
and establish efficient
distribution and
delivery
unconditioned cash
systems & respective
mechanisms.
Overall Rating:
MOUs
developed,
budget
allocated, ATM
cards
distribution
completed and
complaint
process and
mechanism set
in the targeted
working areas
in Lebanon
Overall Rating:
Increased the
efficiency of the
Delivery and
Distribution
systems on the
NPTP/MoSA
level.
Overall Rating:
Increased
sustainability
& ownership
(on the
national &
governmental
level) of
programmes
that targeting
the
vulnerable
poor
Lebanese
children and
their families
Overall
Rating:
81
The final evaluation report of the Winter Cash Programme-UNICEF Lebanon. Developed by AwnyAmer- Jan, 2017 Page 81
Annex 8: Summary of the stakeholders’ feedback and insights about the restricted cash modality versus unrestricted cash
modality
S Unrestricted cash modality
Restricted cash Modality
The following is a summary of the justified reasons for the preference
for the unrestricted cash transfer from the point of view of the
majority of the direct beneficiaries (parents/caregivers) compared to
the other stakeholders interviewed in the current evaluation process.
This corresponded with the PDM findings as the majority does not prefer
other kinds of modalities in comparison to those who do (88per cent
versis 12 per cent)47.
The following is a summary of the justified reasons of
stakeholders preferences for the restricted cash transfer
assistance to cover the winter needs provided to the poor
Lebanese households (advantages of adopting the e-vouchers or
restricted cash transfer as opposed to the restricted cash transfer
modality as reported by all the partners (WFP, NPTP and MoSA
staff) interviewed and 12 per cent48 of the FGD participants in the
PDM process as indicated inside the report :
1 The unrestricted cash transfer assistance modality gives
families the freedom and flexibility of choice, and the
power to purchase the goods and services they need as per
their own priorities instead of receiving clothes or others
items they do not need. This is particularly of great value
in the case of spending this cash on covering medication
expenses for chronic diseases or disabilities within the
family especially if this is not considered in the selection
process.
The possibility to misuse the cash amounts in purchasing
inappropriate items by the cardholder.
2 It is worth noting that most of the targeted poor vulnerable
households in the programme have access to other e-
vouchers, provided by WFP, UNHCR or others, which
create duplication of the services, provided resulting in a
gap in the cost effectiveness.
When cash transfer is adopted as the preferred modality,
there is no assurance that the children will benefit from this
programme because of the unconditional nature of this
method.
3
Discussions with two of the SDC directors interviewed
reported that the unrestricted cash transfer is cost-effective
The burden of additional expenses such as bank charges in
addition to other relative fees to issue the ATM cards on
47S4tatistics Lebanon Ltd_ Results of the survey findings _Winter Cash Programme-PDM Report-2016_Slide# 74 48 Ibid- slide # 74
82
The final evaluation report of the Winter Cash Programme-UNICEF Lebanon. Developed by AwnyAmer- Jan, 2017 Page 82
in comparison to the other alternatives of the restricted
cash transfer (restocking, seed distribution, etc.) especially
in view of the presence of the WFP as an effective partner
and which already has such assets through other similar
programmes. This was validated by some of the SDC
directors (e.g. Baalabek SDC) when interviewed in the
current evaluation report.
the accounts created for the holders of the ATM cards.
4 Additionally, two of SDC directors interviewed during the
evaluation process indicated that the unrestricted cash
transfer assistance is a human-based approach compared
to the e-voucher to buy clothes- for example. “ It is not
acceptable to let children wear the same type of winter
jacket, for example, which makes them feel like they
are stigmatised and conveys the image that they are
either poor/vulnerable or refugee. This has a negative
impact on social cohesion within the community
especially if they feel that you are violating their
dignity,” one of the interviewers commented.
It might lead to inflation and increased local prices in the
markets.
5 In this regard, other technical distribution related issues
such as the low quality level of materials and what could
be provided is too expensive. “With the e-voucher
provided to buy the clothes from specific shops, we can
buy the same and maybe double the number of items
with the same amount”. “Actually, this is against
respecting our privacy” other participants commented.
This means that the participants feel a great deal of
independence and privacy when they can buy the clothes
wherever they like and select the models they prefer
instead of going to specific places that offer models they
might not like.
Women may not retain control over the income or on the
amounts transferred. During the FGDs, some cases
revealed disputes between the two parents which resulted
in losing the benefit of the cash transfer assistance
especially if the ATM card was with the husband while the
“Hala” card was with the mother. Children may suffer the
consequences and be deprived from benefiting from the
programme. In fact, this was reported by some of the FGD
participants as explained earlier in this report.
6 Potential benefits of cash injection on local markets and
trade.
Potential security issues especially for those who cannot
read and write well.
7 There could be some kind of corruption in the restricted
cash transfer (ex. e-food voucher). Some of the
participants in the FGDs reported that “some markets
Two of the SDC directors interviewed indicated that in
some cases, they noticed that the unrestricted cash transfer
may result in dependency behaviours from some of the
83
The final evaluation report of the Winter Cash Programme-UNICEF Lebanon. Developed by AwnyAmer- Jan, 2017 Page 83
provide different non-food items to the holder of the e-
voucher - including cigarettes as per his request while
it is registered as foods”.
parents who are not working seriously or might result in
preference to bearing more children, in order to benefit
from this programme. 8
It is easily invested in livelihood security as long as the
transfers are sufficient.
9 Can improve the status of women and marginalised groups
if they use it to raise their own small projects or are even
being involved in the decision-making process inside the
family. Two of the female FGD participants in Mount
Lebanon and Tripoli- Al-Mina SDC indicated: “I feel
empowered when I have the cash transfer amount in my
hands. I can select what my children need myself and
this makes me feel that I’m the owner of the decision
taken in this regard.”
Sometimes, the provision of cash may cause other social
problems such as family disputes and domestic violence.
(E.g. some of the families that took part in the FGDs
indicated that some of their neighbours divorced and
separated with the father possessing the ATM card while
the mother holds the Hala card. As a consequence, they
were not able to use the cash transfer amount). Other
family members pointed to the occurrence of family issues
because of the bad use of the cash transfer amount by the
father. However, this happened rarely and in few cases as
indicated before in this report.
10 By adopting the unrestricted cash transfer assistance,
projects benefit the community as a whole and promote
the internal trade and market movement within a small
community.
11 Finally, it could be quicker to mobilise than alternatives
such as food purchase and transport. This means that the
other unrestricted cash transfer might include logistical
steps with additional cost compared with the unrestricted
one.
84
The final evaluation report of the Winter Cash Programme-UNICEF Lebanon. Developed by AwnyAmer- Jan, 2017 Page 84
Annex 9: Summary overview about the Post Distribution Monitoring (PDM) process
Feedback from the Ministry of Social Affairs (MoSA) staff and beneficiaries of the Winter Cash
programme is a crucial matter for UNICEF in order to evaluate its performance, redirect, improve
and upgrade the programme for any future assistance it might provide.
UNICEF engaged InfoPro Research (a third party entity) to conduct a series of focus group
discussions (FGDs) to assess the perception of the target population and field actors; programme
implementation; processes; use of assistance; and the general satisfaction and perception of
effectiveness of the cash modality.
The qualitative information collected through the FGDs was triangulated with the quantitative data
collected through the questionnaires on a statistically representative sample of the beneficiary
population. The objective was also to gather information on the possible areas of conflict and/or
sensitivities regarding the programme and more specifically on possible tensions that could have
arisen among the communities.
InfoPro held twelve focus group discussions; four with MoSA staff and eight with beneficiaries. The
focus groups with the MoSA staff were split as follows: one with directors, one with coordinators,
and two with social workers, one of which was in North Lebanon and the other in Mount Lebanon.
As for the beneficiaries, the focus groups were split as follows: four with benefiting females and four
with benefiting males residing in Mount Lebanon, Tripoli, Halba and Baalbek. Each focus group was
attended by six to nine participants whose ages ranged between 24 and 53 years49.
InfoPro was provided with contact lists of MoSA staff and Lebanese beneficiaries by UNICEF. Prior
to each discussion group session, InfoPro screened and evaluated all the lists provided in order to
confirm that all participants had the appropriate profile. A total of ten MoSA staff and an average of
twelve Lebanese people were invited to each session in order to ensure a minimum of eight
attendees. The focus group discussion guide was drafted and developed by the UNICEF
Communication for Development (C4D) section in Arabic and translated into English by InfoPro. It
is included in the Appendix. Each focus session lasted a maximum of one and a half hours. As for the
FGDs, before the start of the discussions, InfoPro team and UNICEF’s Monitoring and Evaluation
Consultant, C4D section’ representative and Programme Specialist assisted in a training session to
elucidate the content of the discussion guide.
49 InfoPro_ Winter Cash Assistance- Final report -Post Distribution Monitoring Report-2016_page 6 & 35
top related