project/assignment title - daleel madani · 2020. 1. 22. · project/assignment title: evaluation...
TRANSCRIPT
UNICEF Lebanon Terms of reference: Accelerated Immunization Activities evaluation
PROJECT/ASSIGNMENT TITLE: Evaluation of the Accelerated Immunization Activities within the Health and
Nutrition programme at UNICEF country programme in Lebanon (2017-2019)
BRIEF SUMMARY
The evaluation of the Accelerated Immunization Activities (AIA) covering the period from November 2017 till June 2019 is aimed to
understand primarily the impact of such intervention and secondarily how the approach and strategy shifted from EPI System
Strengthening (in phase I) to mitigating the spread of measles cases (in Phases II and Phase III).
The AIA started in November 2017, aiming to strengthen the existing Expanded Programme of Immunization (EPI) of the Ministry
of Public Health (MoPH), and to prevent the transmission of Polio from Syria in 2017 and other vaccine preventable outbreaks in
Lebanon (such as measles) through working on both the supply and quality of service at the health center level and to increase
demand through finding children defaulting from immunization and enhancing the immunization seeking behavior at community
level. AIA was first piloted in Bekaa and Mount Lebanon in September 2017 for almost one-month period before expanding to more
areas. However, with scarcity, unpredictability and the increase in earmarked funding, UNICEF now would like to evaluate the
Accelerated Immunizations Activities to measure their effectiveness, efficiency, relevance, sustainability and impact of the approach
both on the measles outbreak response, as well as their contribution to Immunization System Strengthening and to provide specific
recommendations on the way forward to strengthen the MoPH immunization programme.
The evaluation is planned to start in December 2019 for 3 months period. The evaluation team will be supported by and reporting to
the M&E Specialist in coordination with the Health and Nutrition section.
BACKGROUND
During the development of the UNICEF Lebanon Country Programme Document (2017-2020), all stakeholders (MoPH, UN
agencies and implementing partners) agreed that immunization should be a main pillar under the Health and Nutrition programme for
the coming 4 years. As such, UNICEF in its CPD, added Outcome (RAM) indicators to measure the progress towards children’s
immunization, against Penta 1 and Penta 3 and measles.
Throughout various country programmes, UNICEF has been supporting the MoPH for both Routine Immunizations and Campaigns
through procurement of routine vaccines and vaccination commodities, in the quality cold chain at the EPI points for quality storage of
vaccines, and campaigns implementation. In addition, UNICEF in past years, supported MoPH and UN agencies (namely WHO) in
conducting several immunization campaigns. Lebanon constantly reported high national immunization coverage (JRF reports).
However, the EPI cluster survey (WHO, 2016), identified low immunization cadasters in Lebanon. In addition, a circulating vaccine
derived Polio outbreak in Syria was declared in 2017, with high risk of contamination to Lebanon. All of this indicated that Lebanon’s
existing EPI system needs further strengthening.
The Accelerated Immunization Activities project was initiated to originally target the low immunization coverage cadasters (identified
in the EPI cluster survey and in consultation with central and regional MoPH focal points). The AIA was modified a bit later in 2018
to respond to the measles outbreak and mitigate the spread of measles cases and try to control the outbreak. In 2019 (Phase III) and due
to continued increase in measles cases, especially in Great Bekaa and Great North, AIA was prioritized in these governorates to control
the spread of cases.
Overall, the AIA implementation was prioritized in cadasters, with the following consideration:
- Equity: based on areas with low immunization coverage cadasters, to reach vulnerable children defaulting from immunization
- Quality: under the leadership of the MoPH, to provide quality vaccines, with priority to Measles and Polio containing
vaccines, but with a referral to the EPI system, to ensure access to quality vaccines and quality vaccine service provision and
full immunization of the child.
- Vulnerability: most of the areas identified as low immunization coverage, were also part of the vulnerable localities identified
in the UN vulnerability mapping
- Engagement with different stakeholders: to implement AIA, the Ministry of Public Health at central and regional levels, with
UNICEF support, involved implementing partners (national and international NGOs) and local authority (such as
municipalities, gatekeepers) and with the personnel at the identified primary health care facilities (PHCC or a dispensary).
The AIA intervention, is aligned with meeting the indicators set in the CPD and are aligned to the SDG 3.
AIA Conceptual Framework – Theory of Change:
The AIA is divided into 3 phases:
-Phase I: November 2017 till December 20171
-Phase II: June 2018 till December 2018
-Phase III: January 2019 till current
When AIA Phase I started, in 2017, its main aim was to: strengthen the existing MoPH EPI system and prevent Polio transmission
from Syria and vaccine preventable outbreak in Lebanon. As such, the AIA Phase I approach focused on:
- Targeting low immunization cadasters identified in the EPI cluster survey conducted in 2016 (WHO) and cadasters there were
identified as vulnerable based on joint discussions between UNICEF and MoPH central and regional level.
- Trained community outreach workers, data entry clerks, vaccinators from identified PHC facilities, qadaa level MoPH staff,
on Interpersonal communication skills, AIA approach, data organization and flow of work
- Implementing partner, jointly with MoPH PHC coordinators and public health officers, scheduled meetings with local
authorities (mainly municipalities) and briefed them about the AIA project and asked them to support in accessing households
in their localities.
- Implementing partner mapped each cadaster
- Outreach teams visited all houses in the cadasters and screened the vaccination cards of all children under age of 15 years old;
identify defaulting children and provide them with a referral voucher with unique barcode.
- Referred children go to the identified PHCs near them and they are provided with needed vaccine and their barcode and their
information is registered
- Children who were identified as defaulters but did not go to the identified PHCs for their vaccination as advised had their data
shared with the implementing partner to follow-up on their immunization
- Implementing partner followed-up on children through phone calls and visits
1 Some implementing partners continued till February 2018 (based on extensions of their PDs)
Also, during Phase I, WHO with MoPH mapped all Informal Tented Settlements (ITSs) into 2 categories A&B, where by all ITSs
under category A (essential): children under 15 years of age, were provided on-spot immunization for defaulting children (measles,
MMR, PCV 13, OPV and IPV) and referred defaulting children to identified PHCs for follow-up on remaining vaccines. And children
in category B were screened and referred to nearest PHC (treated as a Household visit).
AIA phase II – specifically after measles outbreak; MoPH with support of UNICEF adapted the original AIA approach, to include
‘campaign mode’ option in areas with high measles attack rate and that do not have a nearby PHC facility. As such the AIA phase II
approach focused on:
- First, cadasters with measles cases; second, cadasters with low immunization coverage that were targeted in AIA phase I, but
were not completed; finally, cadasters with low immunization coverage that were not approached in Phase I.
- During planning phase jointly with MoPH and implementing partner(s), each cadaster was assigned as potentially needing a
Mobile Vaccination Unit or only outreach workers screening and referring children to nearest PHC and then following up on
them.
- Data is collected electronically using KOBO with direct link to MOPH at central level.
AIA Phase III: UNICEF continued to focus on areas with measles cases; as such the intervention in first 6 months of 2019 was like
that adopted in phase II. In addition:
- UNICEF with MoPH mobilized the health sector, especially in Great North and Great Bekaa, where some sector partners
adopted AIA approach and supported MoPH in targeting children in areas with measles cases.
- UNICEF started discussions with UNICEF partners from different sections, who already conduct outreach, to also support
without additional cost in reaching out to children under 15 years of age and referring defaulters to nearest identified PHC.
- All data is collected through MERA at outreach and center level and data is triangulated at MOPH central level to provide
feedback to health centers and partners on children defaulting from immunization who did not attend the service following
the referral.
Objectives, Purpose & Expected results:
Purpose:
UNICEF is commissioning a summative evaluation on the accelerated immunization activity program to generate substantial
knowledge and learning on the results of AIA project in order to understand how the project (i) reached its objective of strengthening
MoPH national EPI system; (ii) increased immunization coverage in Lebanon, and (iii) had an effect on the measles outbreak
(controlling the spread of cases, or decreasing the number of measles cases, or had no effect on the outbreak). The evaluation is also
intended to propose recommendations for such a project, to make it sustainable, cost effective and included within MoPH EPI
system.
UNICEF will be the primary user of the evaluation report. Ministries, donors and implementing partners will be the secondary users.
Objectives:
The evaluation will explore how the AIA project has contributed towards improving access and coverage of routine immunization and
its effect on the measles outbreak.
More specifically, the objectives of the evaluation are to:
1. Assess the effectiveness of different AIA approaches (door to door mobilization, temporary vaccination sites, mobile
vaccination units…) on enhancing routine immunization, including uptake by most vulnerable girls and boys and attitudes,
knowledge and behaviors.
2. Assess how AIA II and III had any effect on controlling the spread of measles cases in Lebanon; and if AIA I had any
‘protective factor’ on some cadasters and prevented or at least delayed the onset of measles outbreak
3. Assess the effect of AIA on strengthening the MoPH expanded programme of immunization (EPI) system; including the
effect of AIA on the enhancement of EPI data/registry system through KOBO/MERA application.
4. Provide recommendations on how to move forward with an immunization strengthening approach, which is relevant, efficient,
cost-effective and sustainable, especially with less and less funding available in the health sector. These recommendations
will be utilized bythe MoPH, UNICEF and the main health stake-holders to feed into future strategies for the Health and
Nutrition programme and upcoming UNICEF Lebanon Country Programme Document and could influence the health chapter
of the LCRP.
a. Scope of work:
The scope of the evaluation will focus on the UNICEF Lebanon AIA project, from 2017 till June 2019 in all governorates of Lebanon
including Akkar, North, Bekaa, Baalbeck-Hermel, Beirut, Mount Lebanon, South and Nabatieh.
The evaluation will specifically look at the following pillars:
1. Sustainable strengthening MoPH national EPI system;
2. Increasing immunization coverage in Lebanon;
3. Controlling the measles outbreak
The evaluation will take into consideration, the CPD as the highest level of outcome, whereby the AIA project falls as one of the tools
to reach the CPD Child Survival outcome. In addition, the rolling work plans of Health & Nutrition section of 2017, 2018 and 2019
will be considered when measuring the outcome of AIA project.
b. Time covered by the evaluation
The period 2017-2019 will be considered as the time frame for the evaluation which includes the 3 phases (AIA I+II+III)
c. Timing of the evaluation
The evaluation will be used to inform next years rolling work plan and shape the needed Immunization strategy. It will also guide in
the planning of the next country programme document (2022-2026). The start of the evaluation will be December 2019 – for a period
of 3 months.
Evaluation questions:
The following questions for this evaluation are initial suggestions formulated based on the OECD DAC criteria, and will be further
developed by the research institute/consultancy firm, that will contribute in developing the evaluation methodology and instruments
(including questionnaires). Overall, the evaluation aims to answer the following questions focusing to understand the relevance,
effectiveness, efficiency, coverage, and sustainability of the project:
1. How relevant was the AIA project to MoPH EPI national priorities and the needs of the most vulnerable boys and girls in Lebanon?
a. To what extent did the AIA project suit the priorities of the national MoPH strategy, MoPH EPI programme, UN LCRP,
Health sector plan and UNICEF CPD, and annual and rolling workplans?
b. To what extent are the objectives of the AIA programme still valid and will be valid - in regards to epidemiology (outbreaks)
and immunization coverage of children in Lebanon - for the coming country programme document?
c. To what extent are the activities and outputs of the programme consistent with the overall goal and achievement of its initial
objectives considering the shift from phase I to II and III? (Was the shift in the approach successful and useful or UNICEF
just had to remain in the initial AIA approach?)
2. To what extent was the project effective in meeting the intended goal?
a. To what extent was the project effective in strengthening the functionality of the MOPH EPI system?
b. How did provision of immunization data to health centers enable them to know the immunization coverage within their
catchment area and informed them on the children defaulting from immunization within their area?
c. What real difference has the project made to targeted PHC staff and centers in terms of immunization knowledge, motivation
and ownership?
d. To what extent did UNICEF collaborate with the right (key) partners to ensure achievement of the results? Were there any
challenges in achieving the intended goal?
e. To what extent did the AIA project result in changes in the knowledge and behavior of caregivers of children (boys and girls)
under 15 years of age towards immunizing their children and following up on their routine vaccination? And increasing their
trust in PHCs?
f. To what extent did the AIA project protect from vaccine preventable outbreaks? And to what extent did it control the spread
of measles outbreak?
3. To what extent did the AIA project increase routine immunization coverage at national level, or at least at targeted cadasters?
a. To what extent was the project effective in reaching most vulnerable children (boys and girls) in Lebanon including children
defaulting from routine vaccines? To what extent was the project generating evidence to inform equity EPI priorities? To what
extent was it gender equitable?
b. What reasons accounted for not reaching an identified defaulter child (boys and girls) with measles and polio vaccination?
What lessons learnt can be drawn and applied for strategy revision/strengthening in the near future?
c. To what extent was the project tailored to meet the needs of Lebanese and non-Lebanese children(boys and girls) in rural and
urban setting?
d. To what extent was the project tailored to target children (boys and girls) with disabilities?
4. To what extent was the project cost efficient (reflecting any differences between urban and rural areas; between one implementing
partner and another)
5. To what extent is the AIA project sustainable?
a. To what extent did the benefits of AIA project continue when interventions (due to limiting funding) stopped? Or intervention
was moved from one area to another? (what does sustainability for a vaccination programme look like? Will MoH continue
the programme? Are vaccines available without UNICEF project? Will the staff continue to use the knowledge?)
-If yes: What were the major factors that influenced the achievement of sustainability of the project?
-If no: what were the major factors that influenced the non-achievement of the sustainability of the project?
b. To what extent have the AIA project contributed to or embed sustained local ownership and involvement of local
communities in increasing demand and utilization of routine immunization services considering gender disaggregation of
caregivers?
c. Did the AIA project contributed to sustainable strengthening of the MoPH EPI system (in capacity building, in community
outreach, in EPI registry system)?
d. To what extent did MoPH, PHCs and local authority have sufficient capacity to take on tasks without UNICEF support
(especially that with decrease in funds, UNICEF will not be able to provide HR support to MoPH and PHCs)?
Evaluation stakeholders:
A preliminary mapping of relevant stakeholders of the AIA programme identified the below list of stakeholders. In addition, these
stakeholders are divided into four sub-categories depending on their relative level of “Interest” and relative “Influence”. These
stakeholders are at the core of the design of the programme with whom the team will engage.
Lo
w
I
NF
LU
EN
CE
H
igh
Municipalities Ministry of Public Health;
World Health Organization
Municipalities
Primary Health Care Center Directors
Primary Health Care Center Directors
Caregivers of children under 15
Primary Health Care Center Directors
Implementing partners
Caregivers of children under 15
Low INTEREST High
Evaluation Methods:
The methodology for this evaluation method will be a mix of quantitative (analysis of existing secondary data) and qualitative
tools (KII, FGDs). . The overall methodology should be participatory and should ensure that the various stakeholder including
the most vulnerable communities, municipalities, mayors, partners and others will be reached and their voices elicited.
It is expected that the evaluation will use the following methods:
- Desk Review of key documents: analysis of data from partners and MoPH reports, national surveys and internal compilation
of data and documentation of AIA.
- Key informant interviews with:
• Staff from MoPH PHC department (head of PHC department, senior PHC coordinator, EPI focal point), MoPH qada
offices (qada physicians, PHC coordinators, supporting staff), MoPH IT department (IT project manager and IT AIA
focal points), PHCs (nurses, data entry clerks, directors).
• Project manager from main UNICEF implementing partners and non-implementing partners: WHO, IOCC,
Makhzoumi, Amel, Islamic health association, Al Resaleh Scouts, Al Midan, LRC, Medecin sans frontiers, Nudge. Staff
from UNICEF Health & Nutrition, UNICEF Planning, Monitoring and Evaluation and Communication for
Development and others if needed.
• Municipalities (Member, Mayor, staff involved in AIA, Internal Security Forces)
• Geographical area: Akkar, North, Bekaa, Baalbek Hermel, Beirut, Mount Lebanon, South and Nabatieh considering
partners that targeted Palestinian camps (Burj El Barajneh, Chatila)
- Focus group discussions with:
• Parents/caregivers (men and women)
• Children defaulters and vaccinated (10 to 15 years old)
• Geographical area: Akkar, North, Bekaa, Baalbek Hermel, Beirut, Mount Lebanon, South and Nabatieh considering
targeted Palestinian camps (Burj El Barajneh, Chatila)
The team can further elaborate these requirements in the inception report as appropriate and needed.
List of documents to inform evaluation planning and desk review (This list is non-exhaustive; the evaluation team will be expected to
expand upon this list during their planning phase and desk review.)
The following are some of the key information sources for the evaluation:
- Programme document from different implementing partners and budgets
- LCRP strategy and progress reports (2017 – 2020)
- Presentations on AIA and other internal AIA documents.
- Monthly reports received by MoPH on AIA data throughout the 3 phases
- Partner reports
- Nudge final report
- Lists of municipalities involved in AIA and list of PHCCs and dispensaries that participated in AIA
- Etc…
Limitations:
Some of the foreseeable limitations to this evaluation include in some cases the absence of good quality data (different formats of
same data, incomplete data).
Gender considerations:
- The research team, involved in data collection should be gender balanced, with the division of responsibilities as equally divided
as possible.
- The research team will ensure that locations for FGDs are easily accessible for female participants and individuals with disabilities
and that the date and time for FGDs will be set so as not to interfere with women/girls, men/boys’ routines. Same-sex focus group
discussions are generally more valuable as women/girls may feel more comfortable speaking about certain topics without the
presence of men. Female moderators will be assigned to female participants in the FGDs.
Ethical considerations:
In line with the Standards for UN Evaluation in the UN System (developed by the UN Evaluation Group), all those engaged in
designing, conducting and managing evaluation activities will aspire to conduct high quality and ethical work guided by professional
standards and ethical and moral principles. The proposal must identify actual or potential ethical issues, as well as measures and
methods adopted to mitigate against these issues. All interviewees will be informed with the purpose of the study and their role and
what information is required specifically from them. No deceptive practices are adopted in the research methodology. Confidentiality
of participants and right of withdrawal are ensured. If interviewees will include minors, a written consent should be taken from the
persons in charge of their care. Most importantly, the research methodology and every personnel involved in the study must ensure and
abide by the “Do No Harm” principle. This means that during all the stages of the research, the team needs to avoid putting the
participants at any risk, providing false information, or giving false promises. The harm can be physical, psychological, social, or
financial. The three pillars of the Do No Harm are Respect, Beneficence and Non-maleficence, and Justice. The least to ensure Do No
Harm, the research team must abide by the aforementioned considerations (see UNICEF Procedure for Ethical Standards in Research,
Evaluation, Data Collection and Analysis for details). All the documents, including data and fieldwork instruments, developed in the
course of this consultancy are the intellectual property of UNICEF.
All tools developed must be in line with the Institutional Review Board (IRB) or the Ethical Review Board (ERB). The IRB is a
constituted review body established or designated by an institution to protect the rights and welfare of human subjects recruited to
participate in biomedical or behavioral or social science research. IRBs attempt to ensure, both in advance and by periodic review,
protection of subjects by reviewing research proposals and related materials. IRB protocols assess the ethics of research, evaluations
or data collection and analysis and their methods, promote fully informed and voluntary participation by prospective subjects capable
of making such choices (or, if that is not possible, informed permission given by a suitable proxy), and seeks to maximize the safety of
subjects. The below provides more details on the main ethical aspects, that are further explored in details in the UNICEF Procedure for
Ethical Standards in Research, Evaluation, Data Collection and Analysis.
A. Informed Consent
The purpose of the informed consent is to assure that the fundamental rights and welfare of subjects are protected. Signing the informed
consent ensures that the document has been provided to a prospective subject, risks and benefits have been explained, participants agree
to participate, and a contact has been given incase of any complaints or queries. As per the IRB guidance, the informed consent should
include the following:
An age-appropriate statement that the study involves research, an explanation of the purposes of the research and the expected
duration of the subject's participation, a description of the procedures to be followed, and identification of any procedures which
are experimental;
- A description of any reasonably foreseeable risks or discomforts to the subject;
- A description of any benefits to the subject or to others which may reasonably be expected from the research;
- A disclosure of appropriate alternative procedures or courses of assistance, if needed, that might be advantageous to
the subject;
- A statement describing the extent, if any, to which privacy and confidentiality of records identifying the subject will
be maintained, and any limitations to confidentiality (e.g., mandatory reporting of abuse, etc.);
- For research involving more than minimal risk, an explanation of the risk involved and any support services that will
be made available;
- An explanation of whom to contact for answers to pertinent questions about the research and research subjects' rights, and
whom to contact in the event of a research-related injury to the subject; and
- A statement that participation is voluntary and negotiable, refusal to participate or choose not to respond to a particular
question will involve no penalty or loss of benefits to which the subject is otherwise entitled, and the subject may discontinue
participation at any time without penalty or loss of benefits to which the subject is otherwise entitled.
B. Protection of Data
To maintain the integrity of stored data, project data should be protected from physical damage as well as from tampering, loss, or theft
by limiting access to data. Principal Investigators should decide which project members are authorized to access and manage stored
data. Data stored on paper, such as surveys or notebooks should be kept together in a safe, secure location away from public access,
e.g., a locked file cabinet. Confidentiality and anonymity can be assured by replacing names and other information with encoded
identifiers, with the encoding key kept in a different secure location. Ultimately, the best way to protect data may be to fully educate
all members of the research team about data protection procedures. Data protection should be a part of every project's plan for data
storage. The best way to protect data, whether in written or electronic form, is by limiting access to the data. Electronic data storage
offers many benefits but requires additional consideration and safeguards. Theft and hacking are concerns with electronic data. Many
research projects involve electronic collection and maintenance of human subjects’ data and other confidential records that could
become the target of hackers.
C. Protection of Human Subjects’ Identities
If it is essential to collect and link identifying information (e.g., subjects' names) to subjects’ responses (e.g., questionnaire answers),
researchers must do their best and may need to be creative to provide the utmost confidentiality of subject data. Providing subject
anonymity of information collected means that either the project does not collect identifying information of individual subjects (e.g.,
name, address, email address, etc.), or the project cannot link individual responses with participants’ identities Maintaining
confidentiality of information collected from research participants means that only the investigator(s) or individuals
collecting/analyzing data can identify the responses of individual subjects. However, the researchers must make every effort to prevent
anyone outside of the project from connecting individual subjects with their responses. Examples of increases the level of
confidentiality is use a unique subject code instead of recording identifying information and/or encrypt identifiable data.
D. Protection of Human Subjects’ Safety
The primary concern of researchers should be subject safety. Protecting subject safety requires researchers to use all available
information to identify potential risks to subjects, to establish means of minimizing those risks, and to continually monitor the ongoing
research for adverse events experienced by subjects. Researchers must be prepared to stop the study if risks arise. To assess risks and
benefits, researchers need to explain and potential risks, as well as benefits, that are encountered by the research project. The term risk
refers to the possibility that harm might occur. There are many kinds of risks, such as psychological, physical, legal, social and
economic hardship. The term benefit in research refers to something positive as related to health or welfare. Risks and benefits affect
not only subjects, but also their families and society at large. The IRB will decide whether the risks to the subjects are justified.
Three Basic Ethical Principles:
- Respect for subjects: protecting the autonomy of all people and treating them with courtesy and respect and allowing for
informed consent. Researchers must be truthful and conduct no deception.
- Beneficence: The philosophy of "Do no harm" while maximizing benefits for the research project and minimizing risks to the
research subjects. Subjects are treated in an ethical manner not only by respecting their decisions and protecting them from
harm, but also by making efforts to secure their well-being.
- Justice: ensuring reasonable, non-exploitative, and well-considered procedures are administered fairly — the fair distribution
of costs and benefits to potential research participants — and equally to each person an equal share.
- The basics of all human research subject protections is the fulfillment of nine functions:
- The risks of the research are minimized;
- The risks to subjects are reasonable in relation to anticipated benefits;
- The selection of subjects is fair;
- Each participant gives a voluntary and informed consent;
- When appropriate, the research plan makes adequate provisions for monitoring the data collected to ensure the safety of
subjects;
- There are adequate provisions to protect the privacy of subjects and to maintain the confidentiality of data; Conflicts of interest
are transparent and appropriately managed;
- Consideration is given to what additional protections, if any, are needed for vulnerable populations; and
- Proper training in human subjects’ protections is provided for research personnel.
Management of the evaluation:
The planning, monitoring and evaluation specialist will directly manage the evaluation team in coordination with the Health and
Nutrition Officer to ensure neutrality of the evaluation. MoPH and the Communication for Development Section as well as the Health
& Nutrition Section will also be involved in the evaluation and the H&N section.
The Planning, Monitoring and Evaluation specialist with the Health and Nutrition Officer will give final approval for all the
deliverables, including inception report and the final evaluation report (considering MoPH and communication for development focal
points inputs) prior to final payment.
In addition, UNICEF team (Planning, monitoring and evaluation specialist and health and nutrition officer) will support the
coordination of the evaluation, by facilitating the evaluation team and providing necessary assistance and information to effectively
support the AIA programme evaluation.
UNICEF team will be responsible in the evaluation of bids and act as the selection panel following the rules and regulations of UNICEF,
which will be the contacting party.
Deliverables:
The tasks to be completed by the contractor (or team of consultants) includes, but are not necessarily limited to the following:
Phases Tasks Deliverables
Timeline
(Working
Days)
Phase 1 Review background documentation, including
analysis of data from partners and MoPH
reports, national surveys and internal
compilation of data and documentation of AIA.
Bibliography 5 days
Phase 2 Develop an inception report including
evaluation design and detailed
methodology/tools, work plan for data
collection, and an evaluation matrix.
Inception Report with
annexes submitted to UNCEF
Lebanon country office
10 days
Presentation of Inception report to Health &
Nutrition and Planning, Monitoring and
Evaluation sections.
Feedback on Inception report
provided by UNICEF, MoPH
7 days
Research team to address inputs provided by
UNICEF
Feedback on Inception report
addressed
5 days
Phase 3 Desk Review;
Data collection: Meet/interview/group
discussion with relevant key stakeholders
and beneficiaries
Desk review, Interviews and group
discussion with key stakeholders
conducted in due time.
20 days
Phase 4 Perform analysis and produce draft
preliminary findings and Produce draft
evaluation report
Preliminary findings and draft
report available and shared with
UNICEF Lebanon country office
and MoPH.
20 days
Presentation of draft report to UNICEF
Leabanon Country Office and MoPH, as
well stakeholders including WHO and
Implementing Partner Organizations.
Feedback on final report provided
by UNICEF, MoPH
7 days
Research team to addresse inputs provided by
UNICEF LCO comments
Feedback on final report addressed 5 days
Evaluation team presents the findings of the AIA
program evaluation final report.
Final report is shared with UNICEF
Lebanon Country Office and MoPH
(The final report should be 40 to 60
pages.). A workshop will be
organized to present key findings of
the evaluation.
10 days
Responsibilities:
The research institute/consultancy firm is responsible to produce the following by the stated deadline:
1. Desk review and preparation of the inception report: 27 days
2. Data collection: 20 days
3. Final report draft submission: 32 days
4. Final report: 10 days
Payment schedule:
Payment proportion Phases
30% of total cost Upon delivery of deliverables in phase 1 & 2
30% of total cost Upon delivery of deliverables in phase 3
40% of total cost Upon delivery of deliverable in phase 4
Reporting requirements:
- The research institute/company needs to meet UNICEF’s Quality Assurance requirements to an evaluation. Guidance and
tools relevant to these requirements at any stage of work and respective deliverable: Inception report and full evaluation
report, as well as ethical review requirements, will be shared with the selected service provider (Evaluation quality
assurance/ Ethical review (UNICEF), integrating human rights and gender equality in evaluations (UNEG).
- The report structure will be as per UNEG required standard. UNICEF-Adapted UNEG Evaluation Reports Standards will
be shared with the research company.
- The Consultant will report to the Planning, Monitoring and Evaluation Specialist.
- The reports will be electronically submitted to the Planning, Monitoring and Evaluation Specialist.
- The final report will be submitted in the evaluation database Evidence Information System Integration (EISI). The Global
Evaluation Reports Oversight System (GEROS) will report on the quality of the evaluations, by reviewing and assessing
the quality of final evaluation reports commissioned by UNICEF Offices. The quality of the evaluation report is then
reported to senior management mainly through three channels: a) annual report of the Director to the Executive Board; b)
the Global Evaluation Dashboard, and c) inclusion of this information in the Global Evaluation database;
Qualification Requirements:
This evaluation is looking for research institutes/companies with proven expertise in programme evaluation in an emergency and
development context, and especially within the Health and Nutrition field. UNICEF will ensure neutrality and no conflict of interest
in the recruitment process.
The staff from the research institute/company should have the following qualifications:
Team of
consultants
Work Experience and academic qualification
Team Leader
• Advanced University degree, ideally MPH/PhD, in a relevant Health discipline (for instance,
Public Health, Epidemiology, Bio-Statistics, Health Management) with at least 10 years of
relevant work experience (both international and national) in health, specifically in
immunization programming;
• A minimum of 8 years of practical experience in implementing evaluations (having conducted
evaluation on Immunization programme (routine and campaigns) and Immunizations System
Strengthening is an asset);
• Qualitative and quantitative methods expertise;
• Excellent report writing and analytical skills in English;
• Knowledge of Arabic language is mandatory;
• Familiarity with UNICEF's programming strategies and organizational culture;
• Previous experience in undertaking evaluations of health programmes or similar work
experience in the region specifically in Lebanon will be an asset;
• Understanding of human rights-based approaches to programming, gender considerations and
participatory approaches considered in programme evaluations;
• Strong inter-personal, teamwork and organizational skills;
• Familiarity with information technology, including proficiency in word processing,
spreadsheets, and presentation software.
Overall team
requirements
• Master’s degree in one of the disciplines relevant to public health (for instance epidemiology,
bio-statistics, Health Management, health promotion).
• Good understanding on the use of evaluation methodologies including both quantitative and
qualitative methods for data collection and analysis;
• Previous experience in undertaking evaluations especially for health programming including
Immunizations is considered an advantage;
• Understanding of human rights-based approaches to programming’, gender considerations and
participatory approaches considered in programme evaluations;
• Familiarity with UNICEF's programming strategies and organizational culture;
• Strong inter-personal, teamwork and organizational skills;
• Familiarity with information technology, including proficiency in word processing, spreadsheets,
and presentation software;
• Fluent in both English and Arabic languages.
Evaluation process of the bids:
1-Technical Evaluation Criteria:
Technical Evaluation Criteria Max. Points
Obtainable
1 Overall Response
1.1 Completeness of response and Understanding of UNICEF requirements. 5
5
2 Overall Experience of Research Institute/Company and Key Personnel Proposed for the assignment
2.1
Proven experience with UN agencies (The team leader having conducted at least 2 projects) (2.5 points
per previous contracts).
Familiarity with UNICEF’s health programming and UNICEF’s organizational culture.
5
2.2
Evidence of experience in conducting evaluation of humanitarian and/ or development projects in
Lebanon and internationally (including MENA).
(The team leader having conducted at least 4 projects) (2.5 points per project)
10
2.3 Evidence of previous experience in undertaking evaluations of health programmes specifically
Immunizations. 10
2.4 Proven experience in similar methods (qualitative and quantitative) as proposed with analytical skills 10
35
3 Proposed Approach
3.1 Logical proposal of deliverables and timelines for the consultancy respecting the proposed milestones. 10
3.2 Adequacy of the evaluation proposal and work plan, as per requested methodology including
evaluation questions. (Focus on language and local expertise) 20
30
TOTAL TECHNICAL SCORE 70
Technical evaluation is composed of 70 points
Minimum successful score for the technical evaluation is 49 points
2- Financial Evaluation Criteria:
- Only bidders obtaining the minimum pass mark in the technical evaluation (49 points) will be considered for the financial
evaluation.
- Financial evaluation is composed of 30 points. The lowest financial offer will obtain 30 points.
- Service Providers should fill the attached table for calculation of the financial evaluation.
Timing/Duration of Contract:
UNICEF will receive the final evaluation report 3 months from the date the contract is signed.
Duty Station: Beirut
Administrative issues:
It is not mandatory for the entire team (team leader) to be based in Lebanon on the condition that regular visits to Lebanon are
conducted.
Project management:
The project will be managed by PRIME and Health and Nutrition units of UNICEF.
ANNEX: Financial Proposal