mid-term evaluation of strengthening the global wash...
TRANSCRIPT
Mid-Term Evaluation of
Strengthening the Global
WASH Cluster Rapid
Assessment Team Project
2
June 2014
This publication was produced at the request of CARE US. It was prepared independently by Kay D. Mattson,
Independent International Public Health/WASH consultant.
3
MID-TERM EVALUATION OF
THE STRENGTHENING THE
GLOBAL WASH CLUSTER RAPID
ASSESSMENT TEAM PROJECT
June 2014
A qualitative evaluation of the Strengthening the Global WASH Cluster Rapid Assessment
Team project funded by the United States Agency for International Development Office of
Foreign Disaster Affairs (Agreement Number: AID-OFDA-G-11-00133) with CARE , Oxfam
and the International Federation of Red Cross.
DISCLAIMER
The author’s views expressed in this publication do not necessarily reflect the views of the United States
Agency for International Development or the United States Government.
4
Table of Contents
EXECUTIVE SUMMARY ............................................................................................................................... 7
ACRONYMS AND TERMS ........................................................................................................................ 11
1. PROJECT BACKGROUND .......................................................................................................... 12
2. EVAULATION METHODOLOGY ............................................................................................... 14
2.1 Evaluation Purpose .............................................................................................................. 14
2.2 Evaluation Questions .......................................................................................................... 14
2.3 Evaluation Methods & Limitations ...................................................................................... 15
3. FINDINGS and DISCUSSIONS ..................................................................................................... 16
3.1 Demographic Data of Evaluation Participants .................................................................... 17
3.2 Project Achievements Linked with Project Plan ................................................................. 18
3.3 SWOT Analysis ................................................................................................................... 28
3.4 Project Adaptation & Relevancy to GWC Needs/Plans/Mechanisms ................................ 29
3.5 Project Implementation, Coordination and Management .................................................. 32
4. CONCLUSIONS AND RECOMMENDATIONS ......................................................................... 37
4.1 Conclusions and Findings .................................................................................................... 37
4.2 Recommendations ............................................................................................................... 41
ATTACHMENTS ........................................................................................................................................ 46
Attachment 1 - Secondary References..…………………………………………………………………………………………………46
Attachment 2 - Terms of Reference ……………………………..……………………………………………………………………...47
Attachment 3 - Evaluation Plan Time Line……………………………………………………………………………………………..54
Attachment 4 - Matrix of Evaluation Reviewed RAT Documents……………………………………………………………55
Attachment 5 - RAT Mid-Term Evaluation On-Line Survey Questionnaire………………………………………………58
Attachment 6 - Key Informant Questions Guide…………………………………………………………………………………….68
Attachment 7 - Evaluation KI and On-Line Survey Participant Contacts………………………………………..……….70
Attachment 8 – Complete Survey Monkey Aggregate Survey Results ………………………………….……………….72
Attachment 9 - Disclosure of Conflicts of Interest form…………………………………………………………………………89
5
TABLE of TABLES
Table 1 - Evaluation Participant Contacts by Method and Totals…………………………………………………………..16
Table 2 - Survey Respondents by Deployment Country Emergency……………………………………………………….17
TABLE of FIGURES
Figure 1 - Evaluation Contributors by Agency…………………………………………………………………………………………17
Figure 2 - Rating of overall effectiveness of the project at improving the quality of WASH sector assessments in emergency operations…………………………………………………………………………………..19
Figure 3 - Rating of RAT Projects Overall Effectiveness at Adding Value through Produced Documents…23
Figure 4 - Rating of perspective of added value and usefulness of deployment assessment reports by
deployment respondents……………………………………………………………………………………………………….24
Figure 5 - Review of Randomly Selected RAT Reports against Five Criteria…………………………………………….26
Figure 6 - Rating of Relevancy of the RAT Project to Current GWC Work……………………………………………….30
Figure 7 - Rating of Effectiveness of RATs Coordination with Stakeholders during Deployments…………..33
Figure 8 - Rating of Importance of Phase I Deployment Activities for Phase II……………………………………….35
Figure 9 - Rating of Importance of Non-Deployment Activities for Phase II……………………………………………36
6
7
EXECUTIVE SUMMARY
PROJECT BACKGROUND
The Inter-agency Rapid Assessment Team (RAT) project is an initiative of the Global WASH Cluster
supported by a consortium of three agencies, CARE, Oxfam and the International Federation of Red
Cross (IFRC). The USAID/OFDA funded project was initially for two years (Phase I) October 1, 2011
through September 31, 2013, and has been extended through September 2015. The overall purpose of
the project is to improve the quality of WASH sector assessments in emergencies by producing
consolidated and comprehensive assessments that include data on the numbers, location and condition
of affected populations. The RAT project was originally designed to respond to large scale rapid on-set
emergencies, however during its first phase no such emergencies took place, thus the project somewhat
modified its focus to fill assessment gap in other types of emergencies and undertook efforts to increase
assessment capacity among National WASH Clusters. The primary desired result of the RAT project is
a fundamental change in the way emergency response are planned and implemented. The project
consists of a team of three RAT Coordinators from the consortium agencies, which are available for
rapid mobilization to support the design and implementation of WASH assessments. In addition to
responding to emergencies the RAT project identified a number of non-deployment initiatives focused
on coordination with multi-sector assessment initiatives, capacity building among WASH cluster agencies
and overall development of a knowledge base of tools, techniques for assessments in general as well as
for specific contexts (e.g. urban ad flooding). The RAT project is overseen by a steering committee of
the three agencies global WASH advisors. During the first phase and up through the time this evaluation
was conducted a total of 12 deployments had taken place.
There is significant focus at this time within the GWC to enhance and strengthen the effectiveness of
the FST, which is comprised of the RRT, RECA and RAT projects (3Rs). A Concept Paper (Global
WASH Cluster, May 2014) has been developed to guide the integration of the 3Rs in this effort.
There were multiple goals and objectives included in the projects proposal, with overlap between them:
Goal(s):
To support WASH Cluster operations in major emergencies by providing timely, systematic, and
comprehensive data on numbers, location and condition of affected populations.
To strengthen emergency WASH response planning and coordination by improved collection and
dissemination of information.
Objective(s):
To support emergency WASH response operations in major emergencies by providing timely, systematic, and
comprehensive data on numbers, location and condition of affected populations
To improve quality of WASH sector assessments in emergency operations.
EVALUATION PURPOSE/METHODS
The evaluation was implemented to contribute to the analysis of the project experiences to date and to
provide evidence-based findings and lessons learned to further inform the projects second phase as well
as overall Global WASH Cluster strategic decisions. The evaluation called for an assessment of the
project’s progress towards meeting its objectives and intended results, identification of the projects,
strengths, weaknesses, opportunities and threats and a review of the relevancy of its current activities in
8
relationship to the Global WASH cluster and National Coordination platform needs, with a particular
emphasis on the latest developments in the formation of the FST under the Global WASH Cluster.
The methods used for the evaluation included review of over 100 documents produced by the project
or other documents associated with the project’s work, 34 key informant interviews, development of
an on-line survey tool, which 58 stakeholders responded to and one field visit to the Philippines, the
project’s latest deployment in response to Typhoon Yolanda. The evaluation was conducted over 25
days between April 8 and June 30, 2014 by an individual independent external evaluator. Results
captured in this report were written up over seven days and presented to the evaluation steering
committee of CARE staff and subsequently presented in Geneva to the GWC and members of the
Strategic Advisory Group, OFDA and the RAT Project team.
CONCLUSIONS AND KEY FINDINGS
The RAT project’s purpose to improve the overall effectiveness of WASH assessments in emergencies
was very ambitious. The collaborative consortium management model was also ambitious and not
effective as implemented to support the obtainment of the broader project’s purpose. As a pilot the
project experienced multiple challenges and successes. These experiences provide the project with
multiple lessons learned and a series of identified tasks which, if applied, can result in improved
outcomes during the project’s second phase. The timing for focusing on improving assessment efforts is
ripe given the movement towards a fully integrated Field Support Team under the GWC, for which the
activities of the RAT project will play an integral role.
Overall, across the identified evaluation questions results were overall positive with responses to the
on-line survey rated questions rated as extremely or moderately effective or extremely or moderately
relevant to the GWC’s work. The evaluation confirmed that stakeholders see the project as very
relevant to current GWC needs (83.3% of survey respondents) and that there is overall agreement that
making improvements to WASH assessments in emergencies remains a critical GWC need. It was also
noted that this was the first time such a focus has been placed on improving assessments at the global
level using a more systems focus approach. Unsatisfactory ratings and/or feedback about the project
were most often associated with respondents experience with, or knowledge of, individual deployments
that were not seen as effective. Key findings include:
Finding 1 – The project completed 12 deployments in a wide range of emergencies
Finding 2 – The project and the need for improved assessments in emergencies is seen as very relevant
to the GWC
Finding 3 – The project was worked to improve the quality of WASH sector assessments in
emergencies
Finding 4 – The project utilized a variety of assessment methods that provide the sector with promising
approaches to assessing emergency needs
Finding 5 – As a pilot the project provides a wealth of information to inform future assessment efforts
Finding 6 – The project’s purpose needs to be re-clarified
Finding 7 – The project goals, objectives and indicators are not well developed or ‘SMART’
Finding 8 – The project’s usefulness to national cluster platforms is unknown
Finding 9 – The projects management model is not adequate for the ambitious project
Finding 10 – Processes and mechanisms for requesting RAT services are not well understood or
well developed
9
Finding 11 – There is a need for increased assessment capacity among RAT coordinators
Finding 12 – The assessment capacity within the GWC and National WC platforms is limited
Finding 13 – The overall quality of RAT assessments needs improvement and the project lack sufficient
internal quality control measures
Finding 14 – Publicly available materials about the RAT are considerably out of date
Finding 15 – More effective multi-sector collaborations to enable assessments of WASH needs in
context are needed-
Finding 16 – There is a need to prioritize and clarify the activities conducted by RAT coordinators
during deployments
RECOMMENDATIONS
Strategic changes recommended for the projects second phase include:
R1 – Explore and prioritize types of emergencies that the project will respond to
R2 – Increase the assessment skills of the RAT coordinators
R3 – Improve the quality of assessments and associated reports with increased use of evidence-based
methods
R4 – Update project information and prepare/implement a promotion plan that will more effectively
inform stakeholders of the available RAT services.
R5 – Prioritize the project’s multi-sector assessment collaborations
R6 – Clarify and prioritize RAT coordinator assessment activities
Recommended project plan, management and coordination changes for phase two:
R7 – Refine the projects work plan and associated M&E plan
R8 – Establish an effective management structure that will support the realization of the project’s
intended results.
With respect to how the project is embedded in the existing GWC structure the following
recommendations are made:
Current plans outlined by the GWC and the FST look to be moving in the right direction. This concept
paper was in part informed by an evaluation of the FST results and contributions to supporting national
coordination platforms since the GWC initiated its global surge capacity system in 2007 (Avenir, 2014).
This evaluation identified similar gaps and findings identified in the Concept Paper and Avenir evaluation,
which serves in part to validate the evaluation’s findings. It will be important that the RAT projects
assessment focus and over all original intended purpose of the RAT Project to “not get watered down”
as several stakeholders stated, in this new frame work. High quality assessments are critical to
determining needs and designing response efforts that meet those needs effectively. There is the
potential that the new structure could become overly bureaucratic and focused too much on process,
particularly during this period of transition. While process is important and there looked to be not
enough process for the early phases of the RAT project, this should not trump execution of
assessments. Further the GWC structure needs to ensure that it has sufficiently qualified managers that
are highly skilled in assessments to ensure that assessment designated team members have the proper
support and that quality control processes are put in place.
10
11
ACRONYMS AND TERMS
ACAPS Assessment Capacities Project
AWG Assessment Work Groups
Barangay Smallest administrative division in the Philippines
CAST Cluster Advocacy and Support Team
CARE International NGO
CC Cluster Coordinator
DRR Disaster Risk Reduction
EOM End of Mission Report
FST Field Support Team
GWC Global Wash Cluster
IFRC International Federation of Red Cross
IASC Inter-Agency Standing Committee
IDP Internally Displaced People
IM Information Manager
KI Key Informant
L3 Level 3 Emergency
M&E Monitoring and Evaluation
MIRA Multi-Sector Initial Rapid Assessment
NGO Non-Governmental Organization
NWC National WASH Cluster
OFDA US Office of Foreign Disaster Assistance
OCHA Office for the Coordination of Humanitarian Assistance
OXFAM GB Great Britain NGO
PMT Evaluation Project Management Team
RAT Rapid Assessment Team
RCRC Red Cross/Red Crescent Movement
RECA Regional Emergency Coordination Advisors
RRT Rapid Response Teams
SAG Strategic Advisory Group
SMART Specific, Measureable, Attainable, Realistic, Timely – goals/objectives
TOR Terms of Reference
TWiG Technical Work Groups
UN United Nations
UNICEF United Nations Children Fund
UNHCR United Nations High Commission for Refugees
USAID U.S. Agency for International Development
WASH Water, Sanitation and Hygiene
3Rs Combined RAT, RECA and RRT teams
12
1. PROJECT BACKGROUND
The Inter-agency Rapid Assessment Team (RAT) project is an initiative of the Inter-Agency Standing
Committee (IASC) Global Water, Sanitation and Hygiene (WASH) Cluster (GWC)1, supported by a
consortium of three agencies, CARE2, Oxfam Great Britain (GB) and the International Federation of Red
Cross (IFRC). CARE United States (US) serves as the contract administrator. The project is funded by
USAID/OFDA. The project was initially funded for two years (Phase I) October 1, 2011 through September
31, 2013. The first phase of the project was seen as a learning project3 and was proposed as a pilot project
in the 2011-2015 Global WASH Cluster Strategic plan (GWC, 2011, page 13). The project received a two
year cost modification/project extension by OFDA up through September 2015. The overall objective of
the project as identified in the evaluation Terms of Reference (TOR) was to improve the overall quality of
emergency operations in the WASH sector by providing timely, systematic and comprehensive data
regarding the number, location and condition of affected populations to humanitarian agencies (United
Nations-UN and Non-Governmental Organizations-NGOs), national authorities and donors at the
beginning of large, rapid onset emergencies. The project aimed to provide a mechanism that could facilitate
the rapid mobilization and deployment of a team of three skilled personnel, hosted by the three agencies to
support the design and implementation of systematic and comprehensive assessments to identify WASH
needs that could be readily disseminated to WASH coordination platforms. During Phase I of the project
the RAT project evolved to fill wider assessment gaps in emergencies, such as working to increase
assessment capacity of National WASH Cluster (NWC) partners and participation in multi-cluster needs
assessments and responding to slow on-set emergencies.
As a consortium project each agency has been responsible for supervision of its individual RAT
coordinator, with direction provided by their respective agencies global WASH advisors. Initially the
decision to deploy the RAT to an emergency was determined by a consensus decision of the three-person
steering group made up of the global WASH advisors among the three participating agencies. As a
consortium the project did not have one identified team project manager for most of Phase I of the
project. In year two of the project one of the three RAT coordinators was designated as a lead RAT
coordinator and over the last year the Rapid Response Team (RRT)4 manager also took on management
duties of the RAT. During Phase I the project experienced a fairly high level of turn over for a small team,
with six different RAT coordinators and change in one of the agencies global WASH advisors. At the time
of the evaluation the project was in the process of filling the vacancy of its CARE RAT coordinator, left
vacant in January by its longest serving, and identified lead RAT coordinator.
At the time of the evaluation significant discussions were underway to continue to modify the structure of
the RAT project, moving it more formally under the Global WASH Cluster as part of the Field Support
Team (FST). The FST comprised of the RAT coordinators, RRTs and the Regional Emergency Cluster
1 Interviews with key informants identified some differences of opinion in the formation and purpose of the project, with a few people stating that the project evolved outside of the formal GWC structure and that the purpose of the RAT was to provide skilled staff to areas not being assessed
by others. 2 Consisting of CARE Australia, CARE Germany and CARE USA 3 While the original proposal did not specifically identify the project as a pilot, the funding agency (OFDA) and other’s involved in the consortium or
GWC structure discussed the project as a pilot during interviews. 4 The RRT is another initiative of the IASC Global WASH Cluster. The RRT is comprised of WASH professionals that are available as surge support to national WASH clusters during emergencies. The RRT’s were established in 2007. See http://www.washcluster.info/?q=content/rapid-response-
team-rrt
13
Advisors (RECA)5. The current FST concept paper (May 2014 for 2015-2016) has the FST completely
integrated under the direct management of the GWC Cluster Advocacy and Support Team (CAST). Under
the current FST structure the GWC Deputy Manager has started to take on a more active role
coordinating deployment of the RAT team members to emergencies.
The goal, objective, indicators and expected project results for Phase I of the RAT project:
The project proposal was not consistent in the identification of the project’s goal and objective. Two
project goals were identified in the proposal:
1- Goal: To support WASH Cluster operations in major emergencies by providing timely, systematic, and
comprehensive data on numbers, location and condition of affected populations. (proposal page 3); and
2- Program Goal: To strengthen emergency WASH response planning and coordination by improved
collection and dissemination of information. (proposal page iii)
Similarly two different objectives were included in the proposal, these were not identified as two separate
objectives, but rather one objective described in two different ways:
1- Objective: To support emergency WASH response operations in major emergencies by providing timely,
systematic, and comprehensive data on numbers, location and condition of affected populations (page iii –
note this is the same as the first goal above); and
2- Objective: To improve quality of WASH sector assessments in emergency operations. (page 5)
Phase I Project Indicators and associated targets:
1. Number of assessments carried out from which preliminary data is shared with cluster members
and other stakeholders within 15 days of emergency onset and comprehensive data within 30 days
(Target – 9 deployments)
2. Number of organizations utilizing assessment results to design response strategies (Target – 20
organizations)
3. Number of assessment protocols developed (Target – 4 protocols)
Phase I Proposed Expected Results:
The primary result anticipated from implementing a consolidated and comprehensive assessment will be a
fundamental change in the way emergency responses are planned and implemented. The WASH Cluster
Coordinator (CC) and the Cluster Strategic Advisory Group (SAG)6 will, for the first time, have access to a
timely and reliable picture of the scale of the emergency, the specific needs of the affected population and
their numbers and locations.
Sharing these data with responding agencies will allow response priorities to be established and
agencies to coordinate their response efforts in advance.
Greater equity will be achieved by the humanitarian community being able to respond in the
knowledge that the cases of greatest need have been identified.
Resource mobilization can proceed with improved understanding of the full dimensions of the
response requirements
5 The RECAs are tasked to deliver emergency capacity building and preparedness support to national WASH clusters and national coordination
platforms. - See more at: http://www.washcluster.info/?q=content/regional-emergency-cluster-advisors-recas#sthash.yXu64fcH.dpuf 6 The SAG is comprised of a small number of cluster partners nominated by the wider global level membership from 3 NGOs, one international
organisation, two UN organisations and one agency or individual representing the roll out of cluster activities at a regional or country level. The
SAG is tasked with representing the GWC agencies in various events and negotiations (2011-2015 GWC Strategic Plan, page 12)
14
Assessment effort previously expended in piecemeal assessments can be diverted to delivering
assistance to the affected population
Responding agencies not normally participating in the WASH Cluster have an incentive to
coordinate through the cluster
Follow up multi-sectoral assessments will have the benefit of good preliminary data and access to
valuable site-specific logistic and operational guidance
Donations-in-kind can be more reliably assessed for suitability
Beyond these operational enhancements, the RAT was expected to become a repository of expertise
relating to policy and practice for the gathering of high-value information in emergencies. The project
activities were expected to enhance the capability of humanitarian agencies in WASH preparedness and
response to disasters.
2. EVAULATION METHODOLOGY
The following section outlines the purpose for the mid-term evaluation and the methods employed:
2.1 Evaluation Purpose
The evaluation was implemented to contribute to the analysis of the project experiences to date and to
provide evidence-based findings and lessons learned to further inform project activities in the proposed
second phase of the project and overall Global WASH Cluster strategic decisions. The specific objectives
for the evaluation outlined in the Terms of Reference (TOR):
1. To determine the extent to which the original project fulfilled its objectives and achieved its
results;
2. To identify the strengths, weaknesses, opportunities and threats of the first phase of the project
(October 1st, 2011 – September 30th, 2013);
3. To review the evolution of the RAT project through phase 1 and determine if current activities are
relevant to: a) Global WASH Cluster needs and b) National Coordination platforms needs
4. To provide evidence based findings and lessons learned to further inform project activities in the
second phase of the project (October 1st, 2013 – September 30th, 2015);
5. To propose, what if any strategic, structural and operational changes should be introduced in the
second phase of the project (October 1st, 2013 – September 30th, 2015) with particular reference
to the latest developments towards establishing the FST within the GWC.
2.2 Evaluation Questions
The TOR identified five focus areas for the evaluation:
1. Project Achievements Linked with Project Plan: Review of the project’s achievements based
on reports and stakeholder feedback in relations to its results and activities as laid down it the
project plan.
2. SWOT Analysis: Identification of the projects strengths, weaknesses, opportunities and threats.
3. Project Adaptation and Relevancy to Current GWC Needs/Plans/Mechanisms: Review
of the evolution of the RAT project and current relevancy.
4. Project Implementation, Coordination and Management: Evaluation of the
implementation, coordination and management of the project.
15
Each of the above areas had specific questions, which are listed and addressed in Section 3 – Findings and
Discussion. A complete list of the evaluation questions are in the TOR in Attachment 2. In addition to the
above focus areas the TOR requested that realistic, operational and as pragmatic as possible
recommendations be made to address identified conclusions to inform the projects second Phase. These
are outlined in Section 4 –Conclusions and Recommendations,
2.3 Evaluation Methods & Limitations
The mid-term evaluation of the RAT project primarily utilized a qualitative evaluation approach. This report
documents findings based on 25 days of work between April 8, 2014 and June 30, 2014, including travel to
the Philippines (See Evaluation Plan Time Line Attachment 3). The evaluation included the following
activities and methods:
Briefing by RAT consortium Project Management Team (PMT) David Gazashvili (CARE US), Georg
Nothelle (CARE Germany) and Nicholas Brooks (CARE-Australia).
Desktop review of over 100 documents. Including documents created by the RAT project or
associated FST documents (See Attachment 4 Matrix of Evaluation Reviewed RAT Project
Documents) and secondary documents (See Attachment 1 Secondary References).
Development and implementation of an on-line survey (See Attachment 5 for the RAT Mid-Term
Evaluation On-Line Survey Questionnaire).
Semi-structured Key Informant (KI) interviews over Skype with stakeholders and others engaged in
GWC assessment activities. (See Attachment 6 for the KI Question Guide).
Field visit to Manila and Tacloban, Philippines to interview stakeholders involved in three RAT
project deployment activities between May 20 and May 30, 2014, including travel.
The on-line survey used both quantitative (ranking) and open ended qualitative questions. While names
were collected, respondents were informed that individual responses tied to names would not be seen by
anyone other than the evaluator. Survey questions focused the overall project’s strengths, weaknesses,
opportunities and threats (SWOT), assessment of the project’s relevancy to the GWC’s current needs and
an assessment of the project outputs related to deployment and non-deployment activities. The survey also
asked for respondents recommendations for the projects second phase. Potential respondents were
identified by the PMT, through the evaluator’s review of project documents, and by the project’s current
and former staff. The survey was open for 22 days from May 1, 2014 to May 23, 2014. 235 people were
invited to complete the survey, including contacts taken from available National CC contact lists where the
RAT had deployed. It was acknowledged that many of these contacts had no direct contact with the RAT
project, but were included to be inclusive. A total of 58 persons completed the on-line survey. Of those 19
also completed a KI interview.
Interviews with KIs were held between May 13 and June 4. In total 34 interviews were held, including 17 in
the Philippines. Field visits were initially scheduled to take place in two deployment sites; however this was
reduced to one as the other locations identified had too few people who remained in the field to justify in-
country interviews. The Philippines was targeted as it was the most recent RAT deployment, with three
different deployments by two RAT coordinators in response to Typhoon Yolanda between November and
May 2014. Interviews were held with stakeholders in Tacloban as well as WASH CC and Information
Manager (IM) staff in Tacloban, Cebu and Manila.
16
Table 1 provides a summary of all evaluation contacts.. Attachment 7 (Evaluation KI and On-Line Survey
Participant Contacts) provides a list of the names of persons, and their corresponding organizations that
participated in the evaluation. Interview responses, as well as open ended survey responses were analyzed
for common themes using key word searches. The evaluator worked to triangulate conclusions whenever
possible looking to multiple inputs (interviews, survey results and documents) to support findings.
Table 1 – Evaluation Participant Contacts by Method and Totals
Method # of Persons
Survey Respondents 58
Key Informant Interviews 17
Philippine Field Key Informant Interviews 17
Total 92
Total unduplicated persons 72
Total number of organizations7 26
Limitations
There are several limitations to the evaluation. One primarily limitation was the lack of information
available to objectively assess how assessments and other work completed by the RATs during
deployments resulted in “…. a fundamental way emergency responses are planned and implemented”, the
identified primary result of the RAT project. This limitation is related to: a) limited information on how
completed assessments and other work was used by stakeholders following deployments, b) information
that is available is primarily antidotal or subjective, and b) recall bias (asking about work completed up to
two years ago, c) and/or the inability to interview people involved in deployments due turnover and/or
inability to locate involved contacts. Another limitation is only one field visit was conducted. Thus the
report has significantly more information for that particular deployment, which may not be representative
of other deployments. Another limitation is that the survey collected a limited number of responses from
persons directly involved in deployments outside of the Philippines; with53.6 percent of survey respondents
having no direct experience with a deployment. In addition, many of the KI’s interviewed also had no direct
experience with a deployment. While this is a limiting factor survey respondents, as well as KI’s, with no
direct field experience provided their feedback on their overall knowledge, and experience of the RAT
project based on their review of available reports and information obtained about deployments and other
work conducted by the RAT team via other avenues, e.g. meetings, briefings, reports from national and
GWC groups, and feedback from staff from their respective organizations, etc. Another limiting factor is
the limited amount of time (5 days allocated) to conduct analysis and write up of the extensive material
reviewed (over 100 documents), interviews conducted (34) and survey responses (58).
3. FINDINGS and DISCUSSIONS
This section outlines the findings derived from the methods discussed in the Section 3. Comment boxes
are inserted throughout this section to illustrate key points or themes captured by stakeholder and to
7 These represent global organizations, e.g. CARE Australia, US and Germany are counted as one organization.
17
provide a sense of the variability in comments received during the evaluation. Summary findings, conclusion
and recommendations are in Section 4.
3.1 Demographic Data of Evaluation Participants
A total of 72 persons provided input into the evaluation; representing 26 organizations (see Figure 1).
Figure 1 - Evaluation Contributors by Agency
Of the 58 respondents that completed the survey, 26 (46.4%), had direct involvement in at least one of 12
deployments8, with the largest number involved in a Philippine deployment (Table 2).
Table 2 - Survey Respondents by Deployment Country Emergency
Respondents by Deployment % N
Philippines Hurricane Yolanda (November-December 2013) 34.6% 9
Philippines Hurricane Yolanda (March to present) 26.9% 9
Syria Conflict (Remote from Turkey) (November-December 2013) 19.2% 5
South Sudan Conflict (February-March 2014) 15.4% 4
Lebanon Refugee Crisis (February-March 2013) 15.4% 4
Burkina Faso Refugee/Food/Nutrition Crisis (July 2012) 15.4% 4
Mali Food/Nutrition Crisis (May 2012) 15.4% 4
Myanmar Ethnic Conflict Rakhine State (January-March 2013) 11.6% 3
Chad Food/Nutrition Crisis (March 2012) 11.5% 3
Namibia Drought (July 2013) 7.7% 2
Syria Conflict (Remote from Jordan) (November-December 2012) 7.7% 2
Madagascar Cyclone Giovanna (February-March 2012) 3.9% 1
Total Respondents: 26 out of 58 Total Survey Respondents 44.8% 26
Positions held by evaluation participants ranged from front line technical field based WASH responders to
head quarter (HQ)/desk officer based WASH experts, National WASH CC’s, WASH donors and FST
8 Multiple responses allowed.
18
The purpose on paper is quite useful as
there is a valid need in the WASH sector to improve data collection and analysis for needs assessments. (Survey
respondent)
managers and staff, including current and former RAT team members. The majority of respondents to the
on-line survey (42.2%) worked for an international NGO or UN agency, with 24.4 percent serving as a
member of the GWC SAG.
3.2 Project Achievements Linked with Project Plan
The RAT project plan and overall strategy was seen as very ambitious, not only because it was developed
to improve on-the-ground in-emergency WASH assessments but also because the project set out to bring
about a “fundamental way emergency responses are planned and
implemented” (2010 proposal to OFDA). For the most part
evaluation respondents’ own description of the project’s purpose
aligned fairly closely with the proposed project purpose “to
improve WASH assessments in emergencies”. In addition there was overall agreement that improvement
of WASH assessments in emergencies is very much needed, even if there was sometimes disagreement on
how to go about addressing this need. There were a couple of differences that were identified between
what was in the proposal and some participant’s perspective of the purpose of the project, which were
often associated with operational aspects of the project, e.g. where assessments were to be conducted (“in
underserved or hard to reach areas not currently being reached by current assessment efforts”, KI).
Evaluation participant’s expectations for the project were often more narrowly defined in comparison to
the proposed project results (see Section 1 Project Background) and were seen as realistic and feasible.
Below is a summary of the most frequently identified evaluation participant’s expectations of the project:
Summary of evaluation stakeholder expectations for the RAT project
To provide rapid deployable personnel as surge support to fill gaps in needs
assessments, especially in the early days of an emergency,
To provide assessment experts with highly skilled analytical skills who are seen as
credible (in particular credible to donors),
To provide high quality primary and secondary data and analysis for planning purposes,
To build or strengthen rapid assessment capacity in the field,
To provide assessment support to national CC’s by providing information and analysis
of information to inform WASH goals or strategic plans,
To provide objective, reliable, valid and evidence-based information
To assist in having WASH needs effectively included into multi-sector assessments
While there is some overlap between responding stakeholder’s expectations and the proposed results,
stakeholders tended to focus more narrowly on inputs (deployments and/or reports/assessments produced
by the RAT), rather than on the impact that occurred as a result of the RAT project’s work. As such,
whether or not the project was seen to achieve its intended purpose and desired results is very dependent
upon individual perspectives and varies greatly from deployment to deployment. As a whole the project
looks to have fallen short of achieving many of its primary targeted results. However, as a pilot, the project
made progress conducting activities necessary to see these results realized.
As this section asks for an assessment of achievements laid down in the project plan it is important to point
out that the project proposal had a fairly loosely described strategy, and the corresponding activities
identified in the proposal are not incorporated into a well formulated plan. There is evidence of some
19
“The strategy and concept of the project is quite
straightforward when written on paper. However, the
unforeseen challenge has been whether or not to link it directly to GWC and UNICEF. This has caused issues with the project
strategy being realistic and feasible, as in the early stages, it was thought to be independent of the coordination structure, however, over time the project leads have understood the
importance and need for integration to increase project
feasibility and acceptance.” (Survey respondent)
Information is “…not in a format that easily allowed decision-makers to
make informed judgment about appropriate responses” … information
needs to be available and catered to program managers in the field. (KI)
project work plans9 being developed by the RAT team. However, what was available for review did not
cover all quarters. Also it is not clear how these were used in planning project activities or for monitoring
progress of the RAT team’s work. Therefore it’s somewhat difficult to measure achievement linked to the
project plan as the plan is not well defined. This in part may have played a role in the project falling short
of achieving it defined project results in some areas.
Survey respondents were asked whether
or not they felt the project, with the
principle goal and objective10 in mind had
been effective at “improving the quality of WASH sector assessments in emergency situations” (Figure 2).
Among survey respondents 64.3 percent rated the project as either Extremely Effective (2.4%) or
Moderately Effective (61.9%). Eight people (19.0%) rated the project as either Moderately Ineffective or
Very Ineffective. Among the eight, three experience was with the Burkina Faso deployment, cited by many
as perhaps the RATs least successful deployment on many fronts (response time, coordination and
produced product utility), with one of the eight having no direct deployment experience..
Figure 2 - Rating of overall effectiveness of the project at improving the quality of WASH sector assessments (N=42)
The questions identified in the TOR for this section and corresponding findings are discussed below.
Was the overall strategy of the project realistic and feasible?
Among survey respondents (N=42) 72.4
percent either Agreed Completely (21.4%) or
Moderately Agreed (50.0%) that the overall
project strategy was realistic and feasible.
However, many found that implementation of
the strategy and obtainment of overall project
objectives were not fully realized by the RAT
project during its first phase. Many reasons were cited for this shortfall. The most frequent reasons cited
9 Five work plans were made available covering five quarters, with limited differences among them. 10 This was defined in the survey as to support WASH Cluster operations in major emergencies by providing timely, systematic, and comprehensive data on numbers, location and condition of affected populations, with the principle objective to improve the quality of WASH Sector assessments in
emergency operations.”
20
“More work up front needed to be done on the development of the management structure,
particularly related to management of work during non-deployment times.” (KI)
“Committee management doesn’t work so well – worked well on deployments 60-70% of the time. Doesn’t work so well for non- deployment time” (KI)
were: a) the project was not well aligned initially within the GWC emergency WASH Cluster response
structure, and b) the project’s consortium management structure inhibited the project’s ability to
effectively implement the strategy. With respect to a) respondents felt more alignment within the GWC
structure would have led to more rapid deployments and better defined TORs, particularly in the early
days of the project when such issues were seen as more problematic. That said, issues were identified with
the deployments even when the GWC was more engaged. For example there were deployments that were
recommended and arranged by the GWC without fully orienting the National CC as to the RAT’s role and
purpose, which led to confusion and implementation delays and/or to less than satisfactory deployments,
e.g. in Mali, South Sudan, the first Philippines deployment in November.
With respect to b) the project was identified as
lacking consistent strong leadership responsible for
insuring that the overall project was implemented in a
cohesive, consistent manner during deployment and in
particular during non-deployment periods. The
consortium reliance on the agency based WASH focal
points to oversee the work of a new three person team was not a realistic or feasible management
strategy. Particularly given the focal points had other responsibilities, limiting the amount of time they could
commit to the project. As a new and ambitious project, the team required more support, particularly in the
early stages when both protocols and processes need to be developed and marketed to the GWC
stakeholders. This was recognized by the project during its first year and steps were made to adjust the
leadership structure; however the project is still be experiencing challenges in this area.
Other KI feedback captured related to whether the project strategy was realistic and feasible include:
The original project assumptions were too narrow. It worked out differently than designed, and the project had to
evolve. While it has not come around to what was envisioned, there is still potential and positive lessons learned to
inform the work as it moves forward.
Yes, the project filled a gap. It’s clear no one had the job of finding and identifying the needs of the whole affected
population in a systematic way. The project however did not fully meet the intended results.
When designed there were huge gaps in assessments. This was the first time there has been a focus on assessments.
There has been improvement, but work remains, especially in consolidating partner assessment methodologies.
Yes – it has had its challenges, especially as a multi-agency consortium. It does serve a purpose. It’s a good move that
they are being more integrated into the FST- it has been an evolution.
As a new initiative confusion existed related to the RAT’s purpose specific to deployments, clarification was/is needed.
Yes and No - previous efforts by the GWC to address assessment needs were “upstream” and were not going
anywhere, tools developed, but no use of the tools or outcomes as a result. This RAT focused more on a bottom up
approach, building capacity at the local level. However, more work is needed both structurally to insure that the FST is
effective, and not bogged down in bureaucracy, with assessments led by and meeting country level response needs.
More work needs to be done to capitalize on the RATs expertise in strengthening the capacities in the sector,
particularly at the national level; also the work of the RAT needs to be readily available so that global understanding
can occur – too few results from deployments and lessons learned from the project are available publically.
Were the original objectives11 SMART12? To which degree were they achieved?
The project proposal was not consistent in identifying its stated objective as noted in Section 1 – Project
Background. These objectives are very different from one another, with the first one more adept to being
11 This was originally stated as “were the original results SMART” and has been revised accordingly with approval by the evaluation PMT. 12 SMART objectives are identified as Specific, Measureable, Attainable/Achievable, Relevant and Time bound.
21
“At first I did not know there was a RAT did not understand the alphabet soup – RAT, RECA, RRT, SAG. I was happy and to learn that there were these resources to assist us from the GWC,
available at no cost to us, but it wasn’t clear at first what the RAT’s
role was and how it could help us.” (National CC)
“We requested an RRT, but were told one was not available, but that a RAT was. I wasn’t sure what a RAT was or could do. As I needed the additional surge assistance, I said send them in, without fully understanding their purpose, which contributed to a less than
successful deployment. In hindsight we should have spent more time
defining the TOR before deployment occurred” (National CC)
more “SMART” in its specificity. However, neither objective meets the definition of being SMART. In
particular neither objective is easily measureable, or time bound. The second objective that identifies
improving assessments is difficult to measure, given the lack of any baseline data/information to compare
and/or to measure improvement over time and is perhaps overly ambitious for an objective. Further
assessments as described in the proposal were to be “… a systematic and comprehensive assessment of
the WASH needs of the affected population, including identifying those most in need and vulnerable.” Few
assessments met this definition. Given the projects very ambitious goal and objective, the project
indicators were not well aligned with expected results and not adequate to enable measurement of
progress throughout the project’s first phase. The status of the projects identified three indicators with
two of the indicator targets closely or partially met is as follows at the time of the evaluation:
Indicator 1 # of assessments carried out and shared: The project reported that 8
deployments were conducted as of the end of phase 1 (September 31, 2013). The target was 9 for
the first phase, with 12 deployments made up through the time of the evaluation. However, all
deployments did not result in an assessment, nor is it clear how the completed work was shared
with the cluster members and other stakeholders. In addition there is limited information to
indicate whether or not the deployments were made within 15 days of emergency onset, or
whether comprehensive data has been gathered and shared within 30 days of deployment.
Indicator II # of organizations using results to design response strategies: There is
limited measureable data to support whether or not this indicator was met. The project measures
this indicator by counting the number of organizations the RAT worked with during deployments.
This is measurement of an input and does not adequately provide information on whether or not
organizations are “using the results to design response strategies”. The field visit to the Philippines
found that the initial work conducted by the RATs first deployments were not very helpful or used
in the design of response strategies. The latest assessment (WASH Baseline Barangay Assessment
Phase I: Interim Technical Report, April 2014) was identified as useful (discussed further below),
but should have been conducted much earlier.
Indicator III # of assessment protocols developed: The project has developed 4 protocols
as of September 31, 2013. However, the Cluster Advocacy and Support Team (CAST) has not
responded to, or provided feedback on the proposed protocols and thus they have not been
adopted or implemented and their utility to the broader GWC is unknown at this time.
Were the targeted beneficiaries reached by the project? If not, why not?
The project proposal identified its direct
beneficiaries as “…responding
organizations participating in national
WASH Cluster (NWC) and other national
emergency WASH coordinator platforms.”
Indirect beneficiaries were identified as
“beneficiaries of WASH projects designed
and implemented by these member
organizations.” Overall key informants,
including staff, held a common understanding of who the project’s beneficiaries were as identified in the
proposal. The degree to which the project reached and worked to meet the needs of the direct
beneficiaries, particularly during RAT deployments again looked to vary by deployment. In several
deployments, particularly early on in the RAT project the National CC were not well served and/or the
22
deployment activities were not timely to meet the NWC assessment needs. In addition, many people
interviewed were not aware of the RAT project even up to the point the RAT was deployed. Even at the
time of the evaluation, year three of the project, some evaluation participants indicated that they were not
fully aware of the RAT project’s purpose, or how to request a RAT coordinator or the deployment
process, with some (in the Philippines) not aware of the RAT project at all13. In addition, materials that are
publically available at the time of the evaluation were very out of date14. During the field visit to the
Philippines National WC and partner agencies had varying feedback as to whether or not the work of the
RAT met their needs. All those interviewed that were knowledgeable about the initial deployment
(November 2013) immediately following Typhoon Yolanda indicated that the deployments15 in general
were not very useful to the responding WASH partners. Respondents indicated that while the RAT
worked to include WASH indicators (4 were eventually included) within the MIRA assessment, they found
that assessment was not useful to first responders, as it was too narrow and did not use good WASH
indicators, i.e. respondents were asked “Does your water appear to be clean?”. Many see the MIRA’s as
being useful to informing potential funders as to general WASH needs (one of the desired results for the
project), but not useful to first responder designing response interventions. The RAT’s participation in the
WASH-Shelter assessment was also seen as not very useful, primarily because the assessment was more
focused on Shelter, than WASH and that all areas impacted by Yolanda were not assessed. Given the high
level of turnover of WASH respondents16 many of the KI’s interviewed had not heard about or seen these
initially RAT contributed to, documents. Most, but not all, of those interviews had heard of the more
recently produced report (WASH Baseline Barangay Assessment Report, April 2014). This report was
produced with a high level of involvement by both the National CC as well as Sub-national CCs in
collaboration with REACH17 and overall met the needs of cluster coordinators; however they indicated
that this report should have been conducted much earlier. WASH Cluster partners indicated that they had
used this report primarily in three ways: to compare with and validate their own assessment findings, to
identify new intervention areas for their WASH programming, and as a source of data in proposals to
funders. Several of the partner agencies interviewed stated that they were waiting on Phase II of this report
which would include all barangay’s, as the first report’s data did not cover their targeted communities. KI’s
identified having the spreadsheet that contained all the data as particularly helpful and many saw this as
more useful to them than the written report. The assessment relied on Barangay18 captains as the KI
sources for the assessment. Several indicated that due to extensive geographic spread of the emergency
this was an appropriate approach; however there were some concerns as to whether or not the captains
were the most reliable source for WASH information19.
13 In addition to the Philippines a half dozen people responded to the survey monkey stating that they did not know what the RAT was or had
worked with them during (these were contacts taken off RAT deployment NWC Contact Lists) 14 The Global WASH Cluster maintains a website which has a page for the RAT. The RAT briefing document is very out of date, with all of the
identified RAT team and their contact information not having been with the project for months to over a year.
http://www.washcluster.info/content/rat-briefing-document 15 Two rats were deployed to the Philippines. 16 This is not unique to the Philippines Yolanda response. Frequently first responders to emergencies are agencies lead WASH emergency focal
points, who are there for a limited time until an agency determines next steps, which are implemented by second and later responders. 17 Reach is a joint initiative of two international NGOS that is focused on the development of information tools and products that enhance the
capacity of aid actors to make evidence-based decisions in emergency, recover and development contexts. For more information about REACG -
http://www.reach-initiative.org/reach-overview 18 Smallest administrative division in the Philippines 19 It was noted that the captains had recently been elected to these positions just prior to the typhoon and that there was varying degrees of
WASH knowledge among the captains. This limitation is noted in the report.
23
“…it has been quite hit and miss depending on the deployment and team member(s) carrying it out. The lack of access
to the outputs from the RAT deployments has also contributed to this as even if useful information has been produced, if partners cannot access it because they are unaware of it, then, it becomes quite ineffective.” (Survey Respondent)
Survey respondents rated the RAT project as fairly effective at adding value to producing useful deployment
assessment products/data/reports that met respondent and organization’s needs (Figure 3). However,
again respondents indicated that reports could have added more value if they had been conducted earlier.
In addition, how the work of the RAT is disseminated or presented, or the lack thereof, also had an impact
on
whether the report was used by in-country partners. One respondent captured this finding, expressed by
many in the box above. In a review of the RAT produced products, there is little indication of how or
who the information was presented during the deployment, or any post-deployment follow-up activities.
There is some discussion related to how some reports were distributed in the project’s Quarterly
Progress Reports, but it is not captured systematically, or in a way that enables evaluation of report use.
Figure 3 - Rating of RAT Projects Overall Effectiveness at Adding Value through Produced Documents
The evaluation did not look at the effectiveness of each individual deployment or ask survey respondents to
rate the effectiveness of individual deployments. In an attempt to get some perspective on the effectiveness
of individual deployments analysis included in Figure 4 is presented. This figure captures the effectiveness of
deployments rated by all the survey respondents who indicated involvement in that particular deployment.
Their ratings were provided for the “overall RAT project” therefore this should not be taken as a definitive
rating for any specific deployment, as many were involved in multiple deployments. In addition, the range in
the number of people rating a particular deployment is small (ranges from 1 in Madagascar to 7 in the 2014
Philippines) and the ratings include RAT project staff as well as local partners and CC’s. The figure is useful
in showing the wide range of ratings for all deployments.
The degree to which indirect beneficiaries were served by the project is mostly unknown. There is limited
information that indicates how the assessments and other work conducted by the RAT project was used in
the design and implementation of WASH projects, outside of the Philippines, which is limited.
24
While movement towards the FST structure is in general a good idea, it will be important to remain
flexibility for the RATs and that the “rapid” aspect of their role is retained. UNICEF had not always been able to facilitate rapid assessments, this will
need to continued to be addressed. (KI)
Figure 4
Were risks sufficiently considered in the project plan and the right assumptions made?
Were any crucial elements overlooked in the project design?
The project proposal identified three critical assumptions:
The Cluster Lead Agency(CLA) (UNICEF) will support the function of the team through its
established relationships with national authorities
National authorities will permit access to affected populations and the collection of relevant data
Physical access to the affected population is possible
There looked to be variation in the level of support from
the CLA and national authorities among deployments.
However, for the most part the above identified
assumptions were accurately identified. Of the above
assumptions, the last was not experienced for all
deployments and was a barrier to conducting timely assessments in some deployments. In multiple
deployments (Chad, Mali, Syria, Myanmar and South Sudan) physical access was limited, or not possible due
to a variety of reasons, e.g. road conditions, travel restrictions, safety/security and/or conflict concerns,
vehicle availability and lack of area personnel support structure, etc. In addition in the early days of the
Philippines typhoon response access was not possible, due to conditions as well as insufficient
transportation options. In some of these deployments the RAT project undertook remote assessments,
e.g. in Syria, to address the access issue, which proved to be effective. In the Philippines this lack of, or
limited, access was in part what led to the delayed assessments and also contributed to the eventual use of
the methods (KI interviews using mobile phone technology) throughout the broad impacted geographical
area.
KI’s identified a variety of other assumptions and elements not sufficiently considered for the project:
Deployments did not always have sufficient time to conduct some assessments, particularly large-scale deployments,
although the project proposal did not have a defined number of days for assessments. The project assumed that the RAT staff would have a higher level of expertise in assessment skills than some have.
The project did not adequately identify what the assumptions were for non-deployment and how much non-
deployment time the RAT would have. It’s not clear what RAT does when not deployed and what benefit they provide
when they are not deployed.
25
“In general yes. Could have been done differently. Would have been easier to have a
“lead RAT” from the beginning and to have all employed by agency rather than three. Advantages +/- to different structures. It was
set up as a pilot project – and there has been a fair amount of learning along the way.” KI Interview
At the time the RAT was being formulated the tools that were available
were the RAT/CAT. They had limited success, were very complicated, insane, not friendly, the excel platform didn’t work half the time. To have a person to work with in-country was helpful. Added value. For someone
who knows, has expertise in how tools work and can assist in conducting an assessment that is relevant to that situation is a big change.
Were the original project timeframes and resources appropriate?
The project was built around the goal of addressing large rapid
onset scale emergencies, however no such emergencies
occurred until Typhoon Yolanda in November of 2013, after
the end of Phase II of the project. As a result, the project
modified its scope to allow for responses to smaller scale or
slow on-set emergencies, e.g. droughts and more chronic
emergencies which were not “rapid onset large scale emergencies”. The strategies and associated
timeframes for these types of emergencies, as well as perhaps the availability of more non-deployment
time, were not anticipated and therefore not considered in the original project proposal. It was noted that
OFDA has been flexible in allowing modifications and enabled the project to make adjustments along the
way. Some KI’s felt that while the more “chronic short term emergencies” are appropriate for the project,
they shouldn’t be the focus of the RAT project. To address the assessment needs of these types of
emergencies some KI’s suggested that the RAT project focus more on developing the in-country capacity
to assess such emergencies, as many of these are re-occurring within these countries, e.g. drought in Chad,
rather than continue to assesses such emergencies themselves.
Evaluation respondents, including project staff, felt that the funding was appropriate. The inclusion of funds
to support the implementation of assessments, e.g. hiring of local enumerators, is seen as a critical RAT
project resource. With few exceptions KI’s felt that the number of RAT deployable team members (3) was
appropriate. A few KI’s felt that the project might benefit from one to two more RAT team members, to
be more in-line with regional FST focused efforts. Evaluation participants both within and outside of the
RAT project felt that the RAT team needed a stronger manager or management structure, than what was
available through the consortiums steering committee. Not having this was seen as impacting the projects
ability to effectively undertake non-deployment activities specifically focused on identifying and addressing
capacity needs associated with WASH in emergency assessments. In addition, several key informants
indicated that phase two of the project needed to be more strategic about measuring results, particularly in
relationship to how reports are used (Indicator 2); and to be more proactive in seeking feedback from end
users about the produced products.
What evidence is there of changes in improved WASH assessments in terms of quality
and availability (in time) of data resulting from project activities?
This question cannot be directly
answered with any level of
confidence, as there is little evidence,
e.g. baseline data, from which to
compare whether the project
resulted in improvements in the quality or availability (in time) of WASH assessments. KI’s perceptions of
whether or not the RAT assessments were an improvement over assessments conducted prior to the
project were mixed, and depended on individual deployment experiences.
26
The review found a high level of variability in the quality of the products produced by the RAT project.
Initially the reports also lacked a consistent format and/or methods. While this appears to have improved
overtime, there remains an overall need for increased standardization and improvement in RAT products.
For example EOM reports and lesson learned documents are not available for all deployments. In addition
some deployments resulted in a separate assessment report, where for others the assessment findings
were only included in an EOM report. Based on a review of documents and reports produced by the
project the RAT staff (former and current) look to have significant technical WASH skills; however there is
great variability in skills as assessment professionals in both conducting and effectively documenting their
work in reports. This variability was also frequently cited by KIs as a project shortcoming.
As the evaluation lacked a baseline to compare RAT assessments pre/post implementation a simple matrix
was developed from which a random sample of seven of 12 deployment reports were considered against.
The matrix (Figure 5) includes some key criteria that the RAT proposal stated assessments would address,
or include (criteria 1, 4 and 5), and a couple of generally accepted assessment/research best practices20
(criteria 2 and 3). Areas were then coded as follows: green identified as having been generally met, yellow
identified as having gaps or limitations, and red insufficiently addressed or not included. There evaluator
acknowledges that there are limitations to this matrix as the criteria do not cover all areas important to a
high quality assessment, or identified in the RAT proposal and therefore it is not comprehensive. In
addition this review was conducted very rapidly. In addition there were considerable differences between
the individual deployment’s purposes and thus corresponding aims of the results/reports, making some
criteria not as applicable to some deployments. Of the reports reviewed below none addressed all criteria.
None of the assessments disaggregated data by gender21 and almost all did not include available tools or
data bases developed as attachments.
Figure 5 - Review of Randomly Selected RAT Reports against Five Criteria
Deployment Type of
Report
Comprehensive
Scope (1)
Methods
Described (2)
Findings
Referenced (3)
Disaggregated
by Gender (4)
Tools/Data base
Attached (5)
Philippines - Typhoon (2014) Assessment
South Sudan - Conflict (2014) EOM
Namibia - Drought (2013) EOM
Lebanon - Syrian Conflict (2013) EOM
Myanmar (2013) EOM
Burkina Faso (2012) EOM
Chad (2012) EOM
(1)Comprehensive – assessment covered the entire identified emergency target area, (2) Methods – specific methods used for the assessment are clearly identified and described, e.g. if survey’s, FGDs or KI interviews the questions covered and/or specific tools are included, (3) Findings –
references are cited in the document to enable verification and sources for conclusions, (4) Gender disaggregated data- data that presents the different WASH needs for male and females, (5) Attached tools – are included to enable capacity building as well as enable replication.
KEY Met: Gaps: Insufficient:
A summary of observations of a random selection of deployment reports are included in boxes below to
provide a few case studies of some of the shortcomings, issues and strengths of produced reports. These
deployments were selected for their diversity in geography and in the type of emergencies responded to.
The RAT team employed a variety of assessment methods including KI interviews, secondary data review,
individual/household surveys and Focus Group Discussions (FGDs). As indicated above, most of these
20 See ACAPS (ACAPS, 2013, 2013 and 2014) and Darcy 2014 documents in the references. 21 In the evaluation review meeting in Geneva July 1 and 2 participants discussed that this may not be a realistic for rapid assessments.
27
Namibia (EOM Report) Drought Assessment. This report looked at secondary data, interviewed stakeholders, and
conducted direct observation of water facilities. There was some data collection from affected population using participatory
techniques. The report indicates that there was “assessment fatigue among the affected communities” which begs the question as to
why another assessment was being conducted. It also discusses a future assessment focused on nutrition survey, by which WASH
questions will be included. This was perhaps a missed opportunity for the RAT project, as it is does not appear that the RAT was
involved in assisting with development of WASH questions for inclusion in this assessment. Of the seven objectives identified in the
TOR, only one was addressed and was limited as an assessment deployment. There are no assessment instruments (e.g. survey
questions for KI interviews) or other methods outlined in the report or other documents available on this deployment for review.
Myanmar (EOM Report) Conflict Assessment – It is not clear why was this deployment was requested/conducted. The TOR
identified need for assessments, but when RAT arrived it was identified that a significant number of assessments had already been
conducted. The report identifies a high number of WASH projects (986) were completed between 2003-2012. The RAT did try to
collect assessments – but there was reluctance of WASH actors to share assessment (only 2 were ultimately obtained). The TOR is
ultimately revised in agreement with the CC to focus on providing technical assistance (e.g. identifying methods for desluding of
latrines, design of a flood resistant latrine pit, and a market survey for WASH items). However, the report also includes WASH
assessment data, so it is not clear if this was an assessment or a technical assistance deployment. There is a considerable amount of
data in the report. However, it is not well organized and the findings are not well documented. Sources and dates for data are not
cited throughout the report, e.g. “Overall, 84.6% of the population of Myanmar live in households with improved sanitation facilities,
this figure decreases to 48% in Rakhine State.” Another example of a conclusion, with no reference or data to support how this
conclusion was derived - “The communities have very limited knowledge on water borne disease, as they ignore that diarrhea is
related to consumption of water unclean and malaria and stagnant water are related”. Only one Annex is referenced in the report –
Annex 9, however neither this, nor other annexes (presumably 1-8) were available in the project’s Drop Box for review.
reports did not provide a copy of the tools in the associated report, limiting the ability of the project to
serve as a capacity building mechanism for the broader GWC, and the ability to replicate methods.
There is no baseline data to measure improvement in availability (in time). The RAT protocols established a
target for deployment within 72 hours of request (baring visa constraints). In the GWC 2011-2015
Strategic Plan targeted response time is identified to be within 48 hours of agreement for deployment.
Limited data is available to measure whether either of these targets was met. However, there were a
number of examples of it not being met per the RAT guideline, e.g. Burkina Faso, Chad and Mali. There
were many reasons for this in addition to visa issues ranging from security issues, issues associated with
hosting the RAT, language barriers, e.g. not having a French speaking RAT coordinator available and
wrangling over development of an agreeable TOR by all parties, etc. Many of these were beyond the
control of the RAT.
In addition to the above it appears that further discussion is needed to talk about when is the most
appropriate time to deploy the RAT in a given emergency. There was considerable feedback from KIs
about the response time not only being too late in some cases, but also in the case of the first Philippine
deployment that it was perhaps too early to enable the implementation of a broad scale systematic
assessments given first responders were focused on initial life saving measures. KIs in the Philippines
indicated that perhaps there is a need for staging of RAT responders in large scale rapid on-set
emergencies. For example, an initial response within the first few days that results in a very rapid
Syria Remote from Jordon (EOM and supporting Annexes). The TOR was very clear – a remote WASH assessment to be
conducted from Jordan. The EOM report is well laid out, includes presentation of information in different formats (tables, graphs
and maps) and is well cited, e.g. “90% of the population uses improved water sources, with 93% in urban areas and 90% in rural
areas (UNICE/WHO, JMP 2012).” Methodology used is well presented and detailed, limitations are discussed. Sphere standards are
cited alongside data of current status to provide some sense of the extent of gaps. A summary of risk are included, which identifies
where the greatest needs are, followed by recommendations making the report potentially useful to decision makers. Annexes are
referenced and available, including assessment tools developed/used. A Lessons Learned document is available.
28
“Rat members are WASH specialist with a lot of WASH implementation experience. However, they are not necessarily viewed as survey methodology specialists or
data specialists. The RAT members have struggled to establish themselves as data specialists…” (KI)
assessment to identify immediate needs and a later response (e.g. from three to five weeks, depending
upon the emergency) that assesses longer term recovery needs.
3.3 SWOT Analysis
The on-line survey was the primary method used to assess the SWOT of the RAT project. Overall
assessment of the project’s success is mixed. The mixed reviews by stakeholders were again associated
with different deployment experiences and/or associated outcomes as well as individual perspectives of the
project, which changed for some over time. The following summarizes evaluation respondent’s feedback as
well as information gathered from project documents.
What areas of the project were particularly successful and why?
Among the strengths identified were the project’s capacity to be flexible and its ability to provide surge
support for the specific purpose of conducting assessments in emergencies. Respondents report RAT’s
involvement in other field-based multi-sector assessments (e.g. multi-cluster assessments, MIRA and MINA)
as strength, even if all such assessments were not always useful to responders in the field. Respondents felt
that having the RAT involved in these efforts increased the focus on WASH, which was identified as not
always been a high priority in these assessments. The RAT’s use of mobile phone technology in
collaboration with REACH in the most recent Philippines deployment was seen as a positive use of
technology that holds great potential for future assessments. Overall despite its challenges and
shortcoming the project was seen as generally successful, particularly given it was a pilot. Ratings for all
questions posed in the online survey were on aggregate also generally positive (complete aggregate results
provided in Attachment 8). Many KI’s indicated that as a result of the pilot there is an increased awareness
and acknowledgement that improvements need to be made in WASH assessments and in addition there is
greater understanding of the specific barriers and challenges to implementing high quality assessments. This
was seen as a positive outcome as prior to this project there was not a collective response to conducting
WASH assessments, which made analysis of the issues more difficult given most assessments were being
conducted by individual agencies, if at all.
What areas of the project were less successful and why?
There were a variety of areas where the project was
less successful than desired, some of which have already
been highlighted. The following were identified as areas
where the project was less successful and why:
RAT deployments and produced assessments were not always successful. Quality was lacking in many reports in terms
of either scope of the assessment, methods used, and/or quality of the written report. Reports did not always meet
the requesting organization(s) and/or national WASH cluster needs in terms of scope, coverage or timeliness. Some
deployments did not result in a written report, beyond an EOM report, or as one survey respondent described “a
boiled down concept paper” rather than an assessment of emergency needs and conditions.
The RAT project has not served all regions consistently well. RAT Coordinators have inadequate French language skills
to meet Francophone Africa country’s needs.
The project was not well integrated into the FST structure, particularly in year one, and other institutional barriers
among coordinating bodies prevented the RAT project from being successful in some deployments.
29
Project has not been transparent as a pilot project to enable the GWC to learn from and apply lessons learned.
Non-deployment time is not well utilized and/or WASH actors are unaware of what is done during non-deployment.
What were and are the opportunities and threats for the project in relation to
implementation, coordination and management and how were they taken into account
during phase 1?
The main opportunity identified is the depth and amount of lessons learned from its first phase, which
places the project in a good position to implement actionable changes during phase two. The realignment
under the FST was cited as an opportunity to potentially remove management and institutional barriers
experienced by the project and to begin to have the various components of the FST operate as one team
focused on improving WASH in emergencies (FST Concept Paper, May 2014). Many cited phase two as an
opportunity to work on increasing the capacity of WASH actors, and in particular National WASH cluster
assessment skills. Many felt that the project would be most effective in doing this if it aligned itself and
coordinated more with other organizations seen as experts in assessment, e.g. ACAPS, REACH, CDC, etc.
In addition, given the RAT’s development of, and exposure to, a broad array of assessments22 and
assessment methodologies, they are in a potentially unique position to refine and provide WASH actors
with effective, proven tools and approaches that could be easily adapted to different settings23.
Among the identified threats for the project is the possibility that inter-agency politics and process issues
associated with integration of the FST will overshadow the RAT work.24 In addition, some cited that some
in the GWC structure see no need to improve assessments. Both of these are seen as potential threats
which may inhibit the RAT project from focusing on and achieving its intended purpose. Another often
cited threat was the RAT was frequently not seen as project comprised of assessment experts. If this real,
or perceived, threat is not addressed it may impact future requests for assistance by National CC’s. In its
February 2014 lessons learned document the RAT project acknowledged the need to increase RAT
coordinator skills in assessment methodologies, sampling techniques and software, and other statistical and
epidemiology methods. To date some limited steps look to have been taken to address this25.
3.4 Project Adaptation & Relevancy to GWC Needs/Plans/Mechanisms
Evaluation participants were asked if they thought the project was relevant to the current GWC’s current
work. Among survey respondents 83.3 percent thought the project was either Extremely Relevant (52.4%)
or Moderately Relevant (31.0%) (Figure 6). Of the eight respondents identified earlier (see Figure 2) who
rated the project as either Moderately or Very Ineffective, six of the eight rated the project as Extremely
22 The RAT project reported to have reviewed and rated the “most useful” tools from 33 existing rapid WASH assessment tools collected by RRT
IMs at the GWC level. (Oct-Dec 2012 Quarterly Report). Its not known how this work was shared with Global WASH Cluster actors. 23 In its February 2014 Lessons Learned document the RAT project indicated that it “…has embarked on the development of a generic standardised
data collection method for the WASH sector almost like similar initiatives undertaken by other sectors, such as the development of the SMART
methodology by the nutrition sector, and the Household Economic Approach (HEA) for the food security sector.” The status of this effort is
unknown. 24 The 2011-2015 Global WASH Cluster Strategic Plan cited a similar critical issue identified in the 2010 evaluation that needed to be addressed by
clusters (Streets e. al. Cluster Approach Evaluation 2 –Synthesis Report, URD and GPPI, 2010) that of “Focus on cluster processes rather than
humanitarian action” (page 9), so this is not a new issue. 25 The Oct-Dec 2012 Quarterly report states that one member of the RAT participated in in the Training on Coordinated Assessments in
Humanitarian Crises followed by a Training of Trainers (TOT in Dakar hosted by OCHA in Oct. 2012
30
or Moderately Relevant. The remaining two, rated it as Extremely Not Relevant; with both having had no
direct RAT deployment experience.
Figure 6 - Rating of Relevancy of the RAT Project to Current GWC Work (N= 42)
Did the RAT project adapt effectively to meet the needs of the GWC and national
coordination platforms?
Overall, the project has worked to better align itself with changes occurring within the GWC. In October
2013 during the first joint 3R’s meeting held in Jakarta to discuss the synergies among the 3R’s there was
general agreement and consensus to “harmonize” the RAT project with the RRT and RECA into one newly
conceived “FST” to support WCs. In 2014 at the RAT’s Toulouse Workshop the team, minus the IFRC and
Oxfam Steering Committee Focal Points, further discussed their alignment within the FST and need for
structural changes to support the team’s development. As of this date this is a work in progress. In
addition, the project has made efforts throughout its operation to work with the CAST, e.g. through
participation in GWC meetings and preparation of project protocols referenced earlier; however it was
reported by the RAT project that the CAST has not always responded to or fully engage the project.
Whether or not the project adapted to effectively meet the needs of GWC or national coordination
platforms during deployments looks to again vary depending upon the deployment. In some deployments
inadequately formed TORs were modified after the RAT arrived leading to deliverables that were helpful to
the WC and partners, even though the tasks varied significantly from the original TOR. However, there are
other examples either due to delayed deployment, unclear TORs, and/or in some cases fit between RAT
coordinators and hosting agency staff, where the RAT was not able to adapt and meet the needs of the
requesting organizations. Further, in some cases National CC have felt that the RAT project was “pressed
upon” them from the global level without their fully understanding the role and purposes and potential
benefits of hosting a RAT coordinator. The RAT did take steps throughout its first year to address
identified issues and improvements were noted by KIs, however more remains to be done on both the
RAT project’s, as well as GWC/CAST and national cluster coordinator’s end to ensure that the RAT
project meets the needs of coordination platforms.
Are current activities appropriate considering current GWC architecture and GWC
strategic plans
31
“To address the critical shortfall in data from an emerging crisis, a small dedicated Rapid Assessment Team (RAT) is proposed through an 18 month pilot. This would comprise of three individuals hosted by
cluster agencies, to be initially deployed to gather field-level information against which the WASH cluster agencies could make informed priority
decisions whilst awaiting a full picture of the particular emergency. The RAT concept is intended to help overcome lack of an overall picture created by
the tendency of agencies to cease collecting data beyond their own capacity to respond.” (2011-2015
GWC Strategic Plan, page 13)
The RAT project is among one of six strategic priorities
identified in the Global WASH Cluster’s 2011-2015
strategic plan (see box). The formation and
implementation of the RAT project by the consortium of
three agencies worked to support the realization of
Outcome 2: Timely operational support to National
WASH clusters. At the end of Phase 1 the project was
extended for another 24 months by OFDA enabling
continued deployments by the RAT project to assist
National WASH clusters as well additional time to continue to revise and improve overall RAT services.
Key Informants, including survey respondents reported a high level of consensus that the RAT project is
very relevant (see Figure 6) to the GWC and its relevancy is further reinforced in how it is being
incorporated into FST restructuring efforts under the GWC. That said there was considerable variability
among evaluation participants perspectives as to whether the various activities currently conducted by the
RAT are either relevant to the GWC and its strategic plans, or are activities that the RAT should be
engaged in as “assessment experts”. For more details, including ratings on the specific activities currently
being conducted by the RAT and their importance to Phase II of the project see below under the question
subheading: What changes should be introduced in the implementation, coordination and management of the
project under a second phase?
Could the services that the RATs provide be delivered through a better mechanism?
This is addressed more thoroughly under recommendations as well as in Section 3.5. There were different
opinions among KIs as to what the best fit was for the RAT project, with the majority stating that the RAT
will be more effective under the FST structure. However, it remains to be seen if the RAT concept
provides the best mechanism for assessment of WASH needs in emergencies under the FST. For example
is it best to coordinate assessments under the FST structure, or could assessments be done more
effectively and efficiently by a separate independent organization that is focused solely on assessments in
WASH, as a technical focus area, similar to for example the REACH project?26
In addition, while moving the RAT project under the FST makes sense from a 3R’s structural standpoint
adequate, involved and skilled-in-assessment management to support the still developing assessment
services provided by the RAT need to be in place. Several KI’s indicated that more emphasis needs to be
placed on developing assessment capacity at the local level and that the RAT project should focus on this,
particularly during non-deployment periods. While the RAT project undertook some initiatives during
Phase I that worked on this area, e.g. Lebanon 2013 deployment, this is a greater identified need in the
WASH sector. Further discussions are needed to determine if this is the RAT’s role and secondarily, if yes,
whether the right skill set exists among the current RAT team to enable the development and delivery of
assessment capacity building efforts.
KI’s frequently cited the need for the RAT to either coordinator with the Health Sector and/or incorporate
more health focused assessment tools, questions and methods, e.g. epidemiology, into their overall
26 More about the REACH project can be found here http://www.reach-initiative.org/
32
assessment work. There were several reasons this was identified as a needed service. One is that the
underlying purpose of WASH interventions is to improve the health of affected populations by reducing
morbidity and mortality among WASH associated diseases, e.g. cholera, diarrhea, Hepatitis E, etc. The
second reason was that the Health Sector, including the field of nutrition are assessment experts and skilled
at conducting surveillance work that could benefit and improve WASH assessments. However, whether or
not these activities should be delivered by the RAT directly or better incorporated through coordination
with health experts should be explored. Currently the RAT coordinators are WASH experts and in the
process of building assessment expertise, asking them to add this skill to their “tool box” may over extend
their capacities and may be better served by coordinating with others, e.g. CDC.
3.5 Project Implementation, Coordination and Management
Was the involvement of project partners appropriate to the needs of the project?
This question is addressed in other areas with respect to broad project partners within GWC platforms.
In terms of involvement of the three consortium project partners more work could have been done to
facilitate the development of a cohesive team and development of a common set of project
standards/approaches. Most of the work was done remotely over teleconference and e-mails. This is
perhaps not the best mechanism for a project that is technically focused. As early as the second quarterly
report (April-June 2012 Quarterly Progress Report) the project had identified the need for a workshop to
focus on the development of assessment protocols and project documents. At the time they indicated that
the use of conference calls and e-mails were not sufficient to do this work27. However, it does not appear
that this session took place. There were a variety of planning and/or learning sessions over the course of
Phase I up through February 2014 documented in quarterly reports (see below). These meetings focused
on various project aspects, mostly on external relations and/or coordination.
Induction Meeting. (Jan 23-24, 2012 in Geneva)
One day visit by OFDA WASH advisor to review progress of project, including lessons learned
from deployments, coordination and integration of the project within the GWC. (July 2012)
A RAT Coordinator participated in a meeting with REACH and ACAPS to discuss project
synergies. (Dec. 2012)
Half-day Midterm Review with project team followed by half-day meeting with members of the
3R’s to discuss potential opportunities for joint collaboration. (October 15-16 2012)
Presentation by the IFRC focal point at a sector wide assessment initiative meeting hosted by
OCHA (March 15, 2013). Project update and review sessions with OFDA and the GWC in
Geneva (March 18, 2013) by the RAT CARE and IFRC focal points.
RAT Coordinators participated in the RRT Retreat in Geneva. (June 19-21, 2013)
RAT project meeting with RAT Consortium Focal Points, SAG and CAST members and the
OFDA WASH Advisor focused on Phase II of the project and integration with RECA and RRT.
RAT Coordinators participated in series of project meetings focused on the FST and integration of
the RAT project with the RRT and RECA. (Oct. 28-30, 2013, Jakarta)
One RAT Coordinator attended ACAPS Assessment Lessons Learned workshop.
RAT Toulouse Workshop. Oxfam and IFRC focal points absent. (Jan 15-17, 2014 Jakarta).
27 The quarterly report indicates that the project had not included a budget for such planning meetings.
33
What evidence is there of increased collective planning, coordinated implementation and
overall coherency in emergency WASH responses as a result of project activities
The evaluation considered how effective the RAT was at coordinating with stakeholders during
deployments. 51.2 percent or survey respondents rated it as Extremely or Moderately Effective with
17.1% as Moderately or Extremely Ineffective (Figure 7). The degree of effectiveness again looks to be
dependent on deployment experience. Of those rating it as being Ineffective, two had no direct
deployment experience, with three having deployment experience in either Mali or Burkina Faso and two
in the Philippines, along with one RAT staff member. Overall most respondents, regardless of rating
indicated that the RAT project has made improvements in this area over time. KI’s and survey
respondents indicated that the RAT’s ability to coordinate looks to be somewhat dependent upon the
individual RAT team member, with some being more effective, or skilled at, collaborating with others.
Several KI WCCs indicated that RAT coordinators needed to be more self-directed and problem solvers,
particularly when originally agreed to TOR’s did not go as planned after arriving in the field.
Figure 7 - Rating of Effectiveness of RATs Coordination with Stakeholders during Deployments (N=41)
There is evidence that several deployments were very effective in bringing together stakeholders to
conduct assessments, particularly the Syria remote from Jordon and Turkey deployments. Deployments
identified as having good coordination experiences were identified to have started off with a well-defined
TOR, an initial meeting with key players at the onset of the deployment, and a well formed team most
often consisting of a WCC, a RRT and in some cases a Regional WASH Cluster Advisor. Some of the
more successful identified deployments also formed Assessment Work Groups (AWG) or Technical Work
Groups (TWiGs) to guide the assessments.
As with other indicators there is no baseline data to be able to measure an increase in this area over time.
Also, with only 12 deployments, the number of deployments is limited to measuring this indicator with any
level of confidence. Key informants did indicate that the RAT project has reinforced the need for
increased coordination among stakeholders in emergency WASH assessments. Several also pointed to the
RAT’s projects coordination with REACH and MIRA as examples of potential opportunities for increased
coordination, particularly across clusters. Others also indicated that more work is needed in this area,
particularly with respect to working with the Health Clusters in emergencies. Additionally, there was
considerable agreement that there remains a lot of work in the WASH sector for increased synergy and
sharing of assessment data that enables comparisons of data collected from multiple responders. Finally,
34
as noted above there has been increased collective planning associated with the merger of the RAT under
the FST as identified above in meetings as early as October 2012.
Were the governance and executive structures conducive to an optimal implementation
of project activities?
This is also addressed below. The evaluation did not review the MOU’s or other agreements between the
consortium partners as part of the evaluation. However, both internal and external KI’s interviewed
indicated that management by steering committee was not the most conducive structure for this project.
In addition, a review of quarterly reports indicates that designated WASH Focal Points were not as engaged
in the project as they were described they would be in the proposal.
Were all parts of the project implemented in a coordinated and coherent manner?
The project’s underlying structure as a consortium provided both opportunities and challenges to the
project’s implementation. As a collaboration among three agencies it allowed for a potentially more
flexible and expansive approach, both philosophically and geographically than could perhaps be provided by
one organization. As one survey respondent indicated “I like the collaborative approach as it forces
organizations out of their comfort zones and into solving problems associated with working together which
they would otherwise avoid.” On the downside the collaboration was less efficient. For example issues
associated with deployments, i.e. visas and security28 had to be addressed among three agencies as opposed
to one agency. Management by steering committee is also more time consuming and less efficient. This
approach also looked to be particularly problematic in providing sufficient direction and accountability to
the RAT coordinators during non-deployment periods. The following is a review of the project proposal’s
implementation activities (page 8 and 9 of the OFDA proposal):
A. Establishment of the Team – Staff were successfully hired, however the team has experienced
turnover throughout its first phase and there is a wide range of assessment skills among past and
current team members. Language skills, particularly French, have not been sufficient to meet needs
of French speaking countries. The original management structure (focal point steering committee)
was identified to not work as designed. This resulted in the designation of one of the RAT
coordinators as a lead coordinator to guide the team. However, this position did not have
authority over other staff, which limited their effectiveness. With this in mind the steering
committee identified the need for a RAT project manager as early as October 2012; however this
was not made official until October 2013 and the manager was not able to meet with the full team
until January 2014. While implemented, this position again has had limited authority over the team
as coordinators continue to report to their agency WASH focal points. In addition the RAT
management responsibilities were assigned to the RECA manager on top of their existing duties.
B. Deployment – The project, after a slow start conducted twelve deployments up to the point of the
evaluation. There has not been consistency in how deployments have been implemented or in how
the deployments have been documented. As a pilot the project has worked to address identified
issues, however there looks to be continued inconsistency in produced RAT projects. For example
Situation Reports (SitRep) reports and EOM reports are not always completed, in addition it does
not appear that there is any internal review of produced documents to determine if protocols and
established guidelines are consistently applied.
28 The project identified the need to develop an agreement regarding legal and security protocols, including liability and responsibility between all
three agencies for deployment situations. A draft protocol was developed and commented on in 2012; however this looks to remain unresolved.
35
C. Non-Deployment – While some of the work identified under this activity in the proposal has been
worked on, it looks to have been somewhat piece meal rather than strategic. Work plans covering
some of year two were available and progress reports for all quarters were available for review.
Based on these there is not a good link between work plans and reported progress, or internal
documents indicating progress on assigned non-deployment work activities by individual team
members or overall. In addition, there is inconsistency in the activities and how the activities are
reported on, between what was proposed and progress reports. The project has developed four
protocols29, and submitted these to the CAST for review and consideration.
What changes should be introduced in the implementation, coordination and
management of the project under a second phase?
Survey respondents were asked to rate on a scale of one to five (1 lowest to 5 highest) the importance of
activities30 undertaken by the RAT project during deployments to the projects second phase of work.
Figure 8 presents the results of these rankings, starting with the activities rated least important to highest.
Figure 8 - Rating of Importance of Phase I Deployment Activities for Phase II
Conducting or providing technical WASH expertise inputs was rated the lowest. In exploring this rating
futher with key informants, all KI’s overwhelmingly thought this was both not that important to the RAT’s
work, but distracted from the RAT’s purpose of conducing assesments. While the RAT’s WASH technical
expertise was seen as necessary to conducting a meaningul WASH assessment, technical input during the
deployment was seen as something that should only be provided after an assessment had been completed,
and only if time was available to do so. Several respondents indicated that RATs have in some deployments
fallen back on this area of expertise, somewhat to the demise of needed assessment work. A close second
for least importance was the conducting of orangizational mapping exercises, which some respondents felt
was more the responsibility of other FST members. Ranked as most important is the core function of the
RAT project – development and implementation of WASH assessments, closely followed by
development/implementation of multi-sector assessments and training of others on assessment approaches,
designs, and methodologies. During KI interviews the later was seen as critical to moving the overall
WASH sector forward and to building local capacity. Some went so far to say if the RAT is sucessful they
29 Deployment Protocols, Lessons Learnt, Countries Prone to Disaster database and Good Enough Guide or WASH Assessments 30 For a table of the individual ratings for all of the activities please see Attachment 8 Complete Survey Monkey Aggregate Results
36
would work themselves out of a job. Five respondents provided other activities and associated comments.
These comments included having the RAT focus on “real time evaluations”, “training of actors as part of
the prepardness phase”, and “recommendation of outcomes and output targets based on needs
assessments”. In addition, one respondent recommend that assessments be more strategic and asociated
with GWC and NWC appeals/decisions/orientations. Finally, some identified an overlap between the IMO
and RATs roles, and requested that the duties of each be clarified.
Survey respondents were also asked to rate on a scale of one to five (1 lowest to 5 highest) the importance
of activities31 undertaken by the RAT project during non-deployment time for the projects second phase
of work. Figure 9 presents the results of these rankings, starting with the activities rated least important
through activities rated the highest. Development and dissemination of survey methodologies and
technicques for both WASH and multi-sector assessment was rated the highest. Seven respondents
provided other activities and associated comments. Among them were statements that some of these
activities were more of the IMO’s role and duties (specificially countries prone to diasters data base).
Figure 9 - Rating of Importance of Non-Deployment Activities for Phase II
Evaluation respodents were asked what changes, they would recommend to the RAT to make the project’s
work more effective specific to how it is implemented, coordinated and managed during Phase II. The
following provides a summary of the recommendations made for each area.
Implementation:
Increase awareness of and clarify the RAT’s project purpose, specific to its goals, objectives and role among
national CC/platforms and global WASH partners. Clarify, better describe and inform all actors of the RAT
deployment process. Expand promotion of the availability of the RAT beyond GWC meetings.
Increase the level of coordination and input provided by national coordination platforms, prior to and during
deployments, including improvement in hosting arrangements. Decrease the amount of time it takes to
make a decision and deploy a RAT to an emergency. TORs need to be well developed and agreed upon
prior to deployment and reviewed with all participating partners upon the RATs arrival in country, while
flexibility is important modifications to TOR’s should be agreed to by all.
31 For a complete view of the ranked activities please see Attachment 9.
37
Streamline and improve the management of the RAT team members and their work during deployment
and non-deployment. Develop and implement quality control measures for products produced by the RAT
and insure that accountability mechanisms are in place to insure that standards are met by all team
members. Insure that RAT team members are experts in assessments, survey methodologies, data collection
processes, etc. and that there is a more systematic approach to their work.
Place more emphasis on building the assessment capacity of WASH partners.
Insure that produced products are publicly32 available and disseminated to increase visibility, knowledge,
application of lessons learned and capacity both of the project and overall WASH assessments in
emergencies. Undertake measures to validate findings and ensure methods/indicators can be replicated
later for monitoring and evaluation efforts by the CCs, IMs etc.
Coordination:
Also see above related to coordination with national coordination platforms and TORs.
Increase inter-sectoral assessment coordination, particularly with Health, Nutrition, Education and Shelter.
Continue efforts to move and integrate the RAT’s work under the FST. However, continue to work to insure
flexibility, certain level of independence from UNICEF and rapid deployment under this structure. Work to
reduce or remove other potential agency deployment barriers33.
Align the RAT team regionally within the FST structure to develop and improve relationships and knowledge
of regional issues that can inform deployment and improve assessment outcomes. Insure that that the 3R’s
work as one “harmonized unit”.
Establish working relationships, MOUs and other partnerships with other assessment initiatives and
organizations that have expertise in conducting assessments, e.g. ACAPS, REACH, CDC to improve the
quality of the RATs work and overall assessment capacity of WASH partners.
Management:
Also see above related to management.
Have all RATs managed by a single agency to improve communication, faster decision making, and stronger
accountability.
Clarification and notification to partners of who the current manager is as well as the management
structure that is currently in effective as well as future plans being discussed under the FST.
4. CONCLUSIONS AND RECOMMENDATIONS
4.1 Conclusions and Findings
The RAT project’s purpose to improve the overall effectiveness of WASH assessments in emergencies was
very ambitious. Not only were the stated purpose and desired results ambitious, but undertaking such a
project using a collaborative consortium management model among three participating organizations was
also seen as very ambitious. Within this context the two year pilot project has had many successes and
positive outcomes as well as shortcomings. Overall feedback received from stakeholders engaged in the
evaluation was on aggregate positive, with clear areas identified for improvement. As a pilot project the
project was adept at conducting assessments in multiple emergency settings, far beyond its original focus of
32 Some indicated that in some rare cases this might not always be possible, given security concerns (e.g. Syria). 33 Several respondents indicated that IFRC’s governance inhibits the ability of the IFRC RAT to be deployed in some situations (e.g. cross boarder
emergencies as was the case in Turkey), which potentially reduces where the RAT team can deploy to.
38
“rapid on-set” emergencies, which did not occur during its first two year of operations. This demonstrated
a great deal of flexibility on the part of the project, as well as enabled the project to test out assessment
strategies for different types of emergencies that the GWC responds to. This provides the GWC a
broader array of lessons learned that will be useful to informing assessment efforts in different types of
emergencies. The RAT’s involvement in multi-cluster assessments was reported by many to have played a
role at increasing greater emphasis on WASH needs in these assessments, cited often as previously lacking.
The project assisted in further identifying assessment capacity gaps in the sector and in several deployments
worked to improve assessment capacity of NWCs. Finally, the project provided surge capacity specific to
conducting assessments, a resource previously not available to the sector, which worked to provide data to
national WASH cluster platforms
Among the more positive findings are the following:
F1 – The project completed 12 deployments in a wide range of emergencies - The project
conducted 12 deployments in a wide range of emergency situations in regions throughout the globe, thus
increasing the surge capacity specific for the purpose of conducting assessments in emergencies. These
deployments demonstrated that the project was flexible and able to adapt to changing circumstances by
responding to a broad range of emergencies beyond what was identified in the proposal.
F2 – The project and the need for improved assessments in emergencies is seen as very
relevant to the GWC - The evaluation confirmed that stakeholders see the project as very relevant to
current GWC needs (83.3% of survey respondents) and that there is overall agreement that making
improvements to WASH assessments in emergencies remains a critical GWC need. It was also noted that
this was the first time such a focus has been placed on improving assessments at the global level using a
more systems focus approach.
F3 – The project was worked to improve the quality of WASH sector assessments in
emergencies - 64.3 percent rated the project as either extremely or moderately effective at improving
the quality of WASH sector assessments in emergencies. In addition, there was also overall agreement that
the specific activities that the RAT project has been involved in both deployment and non-deployment
times are important to the GWC’s work.
F4- The project utilized a variety of assessment methods that provide the sector with
promising approaches to assessing emergency needs - The project utilized a variety of techniques
and approaches that were seen as particularly effective and promising for future assessments, including
collaboration with REACH in using mobile phone technology to collect quantitative KI interview data,
implementation of remote assessments in areas inaccessible due to security reasons, and the provision of
raw data to all partners in a format that can be used by partners to identify service gaps, validate other
assessments, and used in project proposals.
F5 – As a pilot the project provides a wealth of information to inform future assessment
efforts - Finally, the project made improvements to its work and evolved over time and identified many
lessons learned and other critical issues that will be informative to the GWC and the FST in the years
ahead. This information is particularly timely as the GWC is the process of formalizing the FST structure,
with assessments identified as one of the seven key surge service to be provided under the FST.
While there were many positive findings the project had multiple challenges and areas in which its intended
purposes and targets were not met. Specifically the project looks to have fallen very short of the outlined
39
desired expected results (outlined on page 12 and 13). The following are the findings identified from the
evaluation where further work is needed to strengthen the project as it moves forward: F6 – The
project’s purpose needs to be re-clarified – As indicated the proposed project was very ambitious
and comprehensive in its stated purposes. When the projected large rapid on-set emergencies did not
occur the project altered its course to focus on conducting assessments in L2 or more chronic, slow on-set
emergencies or crisis. These emergencies presented considerable different deployment and assessment
needs. While the project, either out of default for the lack of large rapid onset emergencies or
purposefully, worked to meet these assessment needs. As a pilot this provided for perhaps a broader array
of learning opportunities, however also compounded perhaps the projects ability to be more effective. The
project needs to consider whether it can continue to be so ambitious in its stated purpose as well as
effectively equipped to address the assessment needs for the significantly different types of emergencies.
F7 – The project goals, objectives and indicators are not well developed or ‘SMART’ - The
overall purpose of the RAT project is well understood. However, the specific identified project goal,
objective and associated indicators upon which the project should be implemented, and evaluated on, were
not clear or well defined as “SMART”. This looked to have had an impact on the project’s ability to
develop and implement some of its project deliverables. This also left the project without a cohesive M&E
strategy that was well understood by the consortium partners or external stakeholders.
F8 – The project’s usefulness to national cluster platforms is unknown - Whether or not the
activities conducted by the RAT project met the needs of National WC and local partners is not known or
well understood. The project has obtained little feedback from the field on whether assessments met their
needs and if and how the assessments were used. While the evaluation worked to gather such feedback
given the nature of the RAT project such feedback, so long after most deployments were completed, was
difficult to obtain. Given the nature of the RAT’s work, and the high incidence of turnover in WASH field
staff in emergencies, feedback needs to be collected much more rapidly after a response.
F9 – The projects management model is not adequate for the ambitious project - The
Consortium Management WASH Focal Point Steering Committee Management model was not and
continues to not be an effective model for the project as a whole. The model did not take into
consideration the time needed to implement the project by the focal points outside of activities specifically
focused on deployments, nor the array of challenges that deployments would pose. The focal points added
this ambitious project to already heavy workloads. The project lacked an overall leader through many
phases of the project who was responsible for guiding the development of the project, as well as insuring
that all involved were tasked with and delivering their assigned tasks during both deployment and non-
deployment periods.
F10 – Processes and mechanisms for requesting RAT services are not well understood or well
developed - Despite the project having responded to twelve emergencies and an increased understanding
of the RAT project’s purpose over time, there remains considerable lack of clarity on stakeholders’ behalf
on the processes and mechanisms for requesting the services of the RAT. This was found for both
deployment and non-deployment activities. Further there is a significant need for the creation of well
formulated deployment TOR’s that are well understood by all parties and are feasible given the time and
resources available within the given context of the targeted emergency. With respect to non-deployment
activities there is a considerable gap in awareness of what the RAT does during non-deployment time and
40
many who felt this time could be better utilized to address the multitude of identified assessment needs not
tied to a specific deployment.
F11 – There is a need for increased assessment capacity among RAT coordinators - The
evaluation identified a need for increased skill development among the RAT Coordinator’s specific to
design, methods and data analysis, application of evidence-based assessment methods and reporting of and
presenting results. The RAT coordinators are seen as highly skilled technical WASH experts, but not
necessarily assessment experts, to increase their effectiveness they will need to be highly skilled in both
areas to be fully utilized as WASH assessment experts.
F12 – The assessment capacity within the GWC and National WC platforms is limited - The
project reinforced the need for overall increased WASH assessment skills and assessment capacity among
National WASH Clusters and GWC partners. As many assessments continue to be done by independent
organizations and/or are initiated by national WASH clusters the need for assessments extends far beyond
the RAT project’s capacity.
F13 – The overall quality of RAT assessments needs improvement and the project lacks
sufficient internal quality control measures – There is considerable variability in the quality of
produced RAT assessment reports, which does not lend the project to being seen as assessment experts
or enable the project to meet the project’s proposed expected results. In addition the project is lacking
sufficient internal quality control measures to ensure that produced products are implemented using valid
and reliable methods and evaluated against a common, agreed upon of set of high quality standards.
F14- Publicly available materials about the RAT are considerably out of date – Available
materials about the RAT project are considerably out of date and are not known by many to be available at
the national and local level.
F15– More effective multi-sector collaborations to enable assessments of WASH needs in
context are needed- Evaluation participants, as well as RAT staff, identified multi-sector assessments,
particularly among the Health/Nutrition and Shelter sectors, given the inter-relatedness of these sectors
and WASH interventions as needing improvement. Health/Nutrition and Shelter specific multi-sector
assessments were either not conducted, when needs were identified, e.g. Chad, or were heavily weighted
to one sector over another (e.g. first Philippines deployment multi-sector Shelter/WASH assessment). In
addition broad multi-sector assessments, e.g. MIRA, need to capture meaningful indicators for the WASH
that extend beyond serving the interests of potential funders. There were few well-conceived multi-sector
collaborative assessments which identified findings that could be used to improve emergency WASH
responses, as well as to serve as a baseline to enable the sector to measure impact of interventions.
F16 – There is a need to prioritize and clarify the activities conducted by RAT coordinators
during deployments - A significant amount of work by RAT Coordinators during deployments was
focused on other activities that are not assessment related, such as conducting technical assistance to
WASH interventions. These activities, while important to emergency WASH responses, look to pull the
RAT Coordinators focus away from assessments, the primary purpose of the RAT project. The RAT
Coordinators have come to this project with their primary skills as technical WASH experts; skills that are
very critical to their role as RAT coordinators, however many did not see the RAT coordinators as
41
“assessment experts”. This potentially impedes the projects ability to meet its overall purpose to increase
the effectiveness of assessments in emergencies.
4.2 Recommendations
There is significant focus at this time within the GWC to enhance and strengthen the effectiveness of the
FST, which includes the RAT project. A Concept Paper (Global WASH Cluster, May 2014) has been
developed to guide the further integration and work of the 3Rs in this effort. The concept paper was
informed by an evaluation conducted by Avenir, 2014. This evaluation has identified some similar gaps and
findings in these two documents, which serves in part to validate the evaluation’s findings. While
integration of the RAT project into the FST is already underway and the findings and recommendations will
inform those processes, it is important that the findings from this evaluation be applied to RAT project
efforts currently underway.
The recommendations below are aligned with the above findings (see brackets).
What strategic changes should be introduced under the second phase of the project?
R1 – Explore and prioritize types of emergencies that the project will respond to - Recommend
that the project further explore and prioritize the types of emergencies the project will respond to and the
associated assessment strategies best suited for these types of emergencies (e.g. length of deployment,
most effective time for deployment, type of assessment methodologies needed for the prioritized
emergency types, etc.). For types of emergencies that are not prioritized, recommend that the project
focus on the development of tools and other resources during non-deployment periods to increase
National WC capacity for conducting their own assessments in these emergencies. For example if its
identified that re-occurring chronic crisis such as droughts (e.g. Chad), or longer-term internal conflict
crisis (e.g. Myanmar), are not RAT project emergency response priorities what can the project do to equip
in-country capacity within National WC platforms to be better prepared to launch their own effective and
evidence-based assessments (F6, F8, F12, F16)
R2 – Increase the assessment skills of the RAT coordinators– Recommend that in conjunction with
the prioritized emergency types that the project undertake an internal assessment of staff technical
assessment skills and gaps for meeting the identified priorities. Where gaps are identified for prioritised
emergency types (e.g. nutrition/famine, infectious disease epidemiology) capacity building should be
undertaken to strengthen those skills within the RAT and/or partnerships formed to address unmet needs.
(F11, F13)
R3 – Improve the quality of assessments and associated reports with increased use of
evidence-based methods In addition to assessment skills specific to particular types of deployments,
assessments need to be conducted using evidence-based designs, methods and approaches as well as
documenting methods used in written reports. This will serve to increase the overall evidence base to
enable the evaluation of WASH interventions in emergencies and throughout the recovery phase and assist
in providing information that can be useful to measuring the longer term impact of response efforts by the
WASH cluster and its partners, e.g. NGO’s. This is essential as currently the WASH Cluster/Sector is most
42
focused on measuring inputs (e.g. 3Ws) and identifying numbers affected, with limited focus on outcomes.
Specific suggestions on how to improve assessments are identified further below. (F7, F13)
R4 - Update project information and prepare/implement a promotion plan that will more
effectively inform stakeholders of the available RAT services The publicly available materials
on the GWC website need to be updated to better inform stakeholders of the availability of
RAT emergency assessment services. Recommend that when the project has re-clarified its purpose
for phase two the project that it informs its current and potential stakeholders of any changes made. While
there are significant changes underway associated with the restructuring and further integration of the FST
the work of the RAT continues. It is recommended that a basic, but strategic promotion strategy be
designed that acknowledges these changes but also works to inform potential stakeholders of the available
services at this time and how they are accessed. It is also recommended that the project better capture,
report and make available resources and reports produced by the project during both deployment and
non-deployment periods to provide stakeholders with a more complete picture of what the RAT project’s
work. The promotion plan should extend outside the typical GWC mechanisms (e.g. meetings and
website) as the evaluation found that many either do not attend these meetings or the information is not
filtered down to them. The project should include efforts to measure these promotion efforts in their M&E
plan. (F6, F11, F14)
R5 – Prioritize the project’s multi-sector assessment collaborations - Identify which WASH multi-
sector assessments are the projects priority for phase two of the project, e.g. is it health/nutrition, shelter
or broader MIRA type assessments? Once prioritized, in collaboration with key assessment stakeholders
and associated experts (e.g. health/nutrition or Shelter global cluster leads, OCHA, and organizations such
as CDC, ACAPS, REACH, etc.) identify what the specific interests, needs and gaps are for both sectors that
would increase the effectiveness of the targeted the multi-sector assessment and develop an associated
strategic plan to address identified interests, needs and gaps. Include this in the projects work plan and
establish a time line for working to see the end results realized. This should be done in collaboration with
FST integration efforts to inform long term GWC assessment needs. (F7, F9, F15, F16)
R6 – Clarify and prioritize RAT coordinator assessment activities - Identify what activities are not
critical to the RAT Project’s assessment activities and discontinue doing them. Insure that all requesting
organizations are aware of what a RAT Coordinator can and does not work on in deployments and work
to develop TOR’s that are well defined in advance with these considerations in mind. (F6 and F16)
What changes should be made to the project plan in terms of results and activities and to
the project setup in terms of management and coordination for implementation under a
second phase of the project?
R7 – Refine the projects work plan and associated M&E plan – It is recommended that after the
project has clarified its purpose that the project work with OFDA to develop a revised work plan that
includes revised indicators and an M&E plan for measuring the project in meaningful ways, beyond just the
three identified indicators. Also recommend that a mechanism for gathering feedback from deployments
within an established time period shortly after a deployment has been completed. Recommend that the
M&E plan be developed with the future of the FST restructuring in mind. (F7, F8)
43
R8 – Establish an effective management structure that will support the realization of the
project’s intended results. While the Focal Point Steering Committee has a continuing role in guiding
the project particularly related to determining and arranging for deployment field support, e.g. housing and
security, etc. it is recommended that the project address the need for a stronger management structure.
The current structure lacks someone who has operational authority over the entire team to ensure that
project activities both related to deployment and non-deployment are implemented to a higher degree of
quality and continuity among all project staff across the entire project. (F7 and F9)
In addition to the above recommendations the following are suggested to support the
implementation of the above recommendations.
Data needs:
Recommend that every deployment develop a list of contacts worked with and/or who are potential
end users of the RAT’s assessment to enable follow-up and evaluation of project efforts. It is
recognized that there is significant turn over in the field following an emergency, but such contact lists
will aid in the ability to contact stakeholders that were directly involved in the field during deployment.
Develop a system (see Figure 5 as an example of a starting point) to capture and enable the M&E of
whether or not established criteria, seen as important to improved assessment reports, are
implemented in RAT produced products over time.
Improving assessment reports and assessment approaches:
The RAT deployment team would benefit by increasing its overall assessment and evaluation skills in
terms of use of methods and writing up of results. Participation in seminars and/or completion of
courses offered by leading assessment or research organizations would not only benefit the project, but
ultimately work to improve WASH assessments in general and potentially overall capacity in the field.
Consider partnering with other organizations whose expertise lies in assessments, such as
ACAPS and the CDC. Work to development skills in specific methods, e.g. CDC’s Community
Assessment for Public Health Emergency Response (CASPER) (CDC, 2012) applied in the
America Samoa 2009 tsunami (Choudary, 2012) that could be adopted and/or modified to
WASH emergencies.
Consider that for existing RAT Coordinators each have their own “capacity development plan”
to support their obtainment of new assessment specific technical skills and abilities that will
increase their and the overall RAT and FST teams effectiveness over time, based on
project/FST needs. This will also enable the RAT team to better market specific services, e.g.
use of mobile phones, GIS mapping, survey development, statistical analysis, etc… to National
WC. For future RAT team members it is recommended that job descriptions incorporate
technical assessment skills and competencies for these positions that extend beyond WASH
specific assessment skills.
Assessment findings in reports need to be better documented to indicate what data led to a specific
finding (citations). This is not only good assessment practice when multiple data sources are used, but
also provides the end user with the data source for the purposes of validating or updating/comparison
purposes later on. Other recommendations to improve deployments and the quality of reports include:
44
TOR’s need to be included for all deployments. Overall TOR’s need to be better defined and
attainable within the allowed timeframes. Modifications made to the TOR, including the reasons
for the modification need to be agreed on, and clearly identified. (see F2)
Limitations need to be included in all reports.
Methods used need to be documented and more descriptive, e.g. if KIs or FGDs were
conducted include what specific questions were asked, how many KI’s and FGD’s were
conducted, with whom and where. Include tools and questions used as attachments to assist in
building the availability of tools for future assessments.
Findings and conclusions need to be well documented (source and date).
Analysis of data should be included, which highlights the gaps and priority needs supported by
the data and go beyond just presenting lists of data.
Assessments should go beyond the who, what, where to look at other issues of why and how.
Recommend that report writers work to challenge their assumptions and biases as well as
those providing qualitative input. Use triangulation methods for example while reliance on KI’s,
is very acceptable it is important to know and identify limitations of KI’s. Wherever possible
use other methods (e.g. convenience sample of some areas to validate KI responses)
Include how the report was presented in the field and to whom, and how received.
Lessons learned should be captured for all deployments - If feasible, given timing, include how
the report’s findings were used or were planning to be used.
Recommend that the project consider different assessment model frameworks for different phases and
types of emergencies. Initial assessments shortly after the onset of an emergency may not be
conducive to comprehensive assessments, however may be more needed within a few weeks of the
emergency.
A review of reports identified several deployments that were determined to already have been
assessed and/or had numerous assessments but not in a format that could be considered as a whole. In
these cases there remained a need for a comprehensive understanding of the needs as well as
identification of assessment gaps. To address these needs and to better enable identification of gaps
using evidence based methods it is recommended that the project look to different methods for
synthesizing data/reports from multiple data sources and multiple partners, perhaps using systematic
review techniques. There may be opportunities from the Cochrane Collaboration34 or other evidence-
based institutes to explore and to identify and adapt techniques that would be useful to the RAT
project.
How should the project be best embedded in the existing GWC structure?
Current plans outlined by the GWC and the FST look to be moving in the right direction. It will be
important for the RAT projects assessment focus and over all original intended purpose of the RAT Project
to “not get watered down” as several stakeholders stated, in this new frame work. High quality
assessments are critical to determine needs and designing response efforts that meet those needs
effectively. For example a recent report coming out of Haiti (Crawford, Kennedy and Killing, 2014)
indicates that poorly executed assessments, in part, had a long term ripple effect that was not only costly,
34 More about the Cochrane Collaboration can be found here http://www.evidenceaid.org/ and the associated Evidence Aid work can be found here
- http://www.evidenceaid.org/
45
but impeded recovery efforts over the long term. While this was not a WASH focused report, the risk
remains the same. In addition there is the potential for the new structure to become overly bureaucratic
and focused too much on process, particularly during this period of transition. While process is important
and there looked to be not enough process for the early phases of the RAT project, this should not trump
execution of assessments. Further the GWC structure needs to ensure that it has sufficiently qualified leads
and managers that are highly skilled in assessments to ensure that assessment designated team members
have the proper support and that quality control processes are put in place.
46
ATTACHMENTS
ATTACHMENT 1
Secondary References and Other Sources of Information Reviewed
Avenir Analytics (2014). Global WASH Cluster – Evaluation of the Support Provided to National
Coordination Platforms Draft Final Report. ACF International
ACAPS. (2013) Compared to What? Analytical thinking and needs assessment – Technical Brief.
Retrieved June 11, 2014 from
http://www.acaps.org/resourcescats/downloader/compared_to_what_analytical_thinking_and_needs_assess
ment/191
ACAPS. (2013), How sure are you? Judging quality and usability of data collected during rapid needs
assessments, Technical Brief. Retrieved June 11, 2014 from
http://www.acaps.org/resourcescats/downloader/how_sure_are_you_judging_quality_and_usability_of_dat
a_collected_during_rapid_needs_assessments/194/1402601312
ACAPS (2014), Secondary Data Review – Sudden Onset Natural Disasters Technical Breif,
Retrieved June 11, 2014 from
http://www.acaps.org/resourcescats/downloader/secondary_data_review_sudden_onset_natural_disasters/
233
Choudhary, Ekta, et. al (2012) Public Health Needs Assessment of Tutulia Island, America Samoa,
After the 2009 Tsunami. Disaster Medicine and Public Health Preparedness, 6: 03, 209-216
Darcy, James, Stobaugh, Heather, Walker, Peter and Maxewell, Dan. (2013). The Use of Evidence
in Humanitarian Decision Making, ACAPS Operational Learning Paper. Retrieved June 11, 2014 from
http://www.acaps.org/resourcescats/downloader/the_use_of_evidence_in_humanitarian_decision_making/1
51
Center for Disease Control and Prevention (CDC). (2012) Community Assessment for Public
Health Emergency Response (CASPER) Toolkit: Second edition. Atlanta (GA). Retrieved June 15, 2014
from
http://emergency.cdc.gov/disasters/surveillance/pdf/CASPER_Toolkit_Version_2_0_508_Compliant.pdf
GWC (2010). Global WASH Cluster Meeting 20-21 September 2010 at WEDC, Loughborough
Mock, N. and R. Garfield. (2007). Health Tracking for Improved Humanitarian Performance.
Prehosptial and Disaster Medicine 22(5): 377-383.
GWC (2011). Global WASH Cluster Strategic Plan 2011-2015. Retrieved May 15, 2014 from
http://www.washcluster.info/?q=content/strategic-framework-2011-2015
O’Donnel, Vicki. (2014) WASH Cluster Partner Information Management Consultation. Project:
Consolidation of the WASH Sector/Cluster assessment and information management procedures during
preparedness, response and transition phases and development of a capacity building strategy for the
Global WASH Cluster partners
O’Donnell. (2014) WASH Information Management Inception Report.
Crawford, Kate, Kennedy, Jim and Killing, Alison (2014) Path-dependency culture in
humanitarian decision-making: why it was hard to change direction in Haiti, Humanitarian Exchange Magazine,
Issue 61, Retrieved June 15, 2014 from http://www.odihpn.org/humanitarian-exchange-magazine/issue-
61/path-dependency-culture-in-humanitarian-decision-making-why-it-was-hard-to-change-direction-in-haiti
ATTACHMENT 2
ToR RAT Project Evaluation – page 1
Terms of Reference (ToR)
Project Evaluation
Title of Project: Strengthening the Global WASH Cluster Rapid Assessment Team
Starting Date: 3rd March 2014
End Date: 30th April 2014
Number of Days: 30
I. Background
The Inter-agency Rapid Assessment Team (RAT) is an initiative of the IASC Global WASH Cluster (GWC),
supported by a consortium of three agencies, CARE, Oxfam GB and IFRC. The core objective of the initial
project was to improve the overall quality of emergency operations in the WASH sector by providing
timely, systematic and comprehensive data regarding the number, location and condition of affected
populations to humanitarian agencies (UN and NGOs), national authorities and donors at the beginning
of large, rapid onset emergencies. The project aimed to provide a mechanism that could facilitate the
rapid mobilisation and deployment of a team of three skilled personnel, hosted by three agencies, CARE,
Oxfam GB and IFRC, to support the design and implementation of systematic and comprehensive
assessments to identify WASH needs that could be readily disseminated to WASH coordination
platforms. During the implementation of this project the RAT have evolved to fill wider gaps in the
assessment process and created a space to support GWC related assessment functions.
The principal goal of the project was to support WASH Cluster operations in major Emergencies by
providing timely, systematic, and comprehensive data on numbers, location and condition of affected
populations, with the principle objective to improve the quality of WASH sector assessments in
emergency operations.
The primary result anticipated from implementing a consolidated and comprehensive assessment will
be a fundamental change in the way emergency responses are planned and implemented. The WASH
Cluster Coordinator and the Cluster Strategic Advisory Group will, for the first time, have access to a
timely and reliable picture of the scale of the emergency, the specific needs of the affected population
and their numbers and locations:
47
Sharing these data with responding agencies will allow response priorities to be established and
agencies to coordinate their response efforts in advance;
ToR RAT Project Evaluation – page 2
Greater equity will be achieved by the humanitarian community being able to respond in the
knowledge that the cases of greatest need have been identified;
Resource mobilization can proceed with improved understanding of the full dimensions of the
response requirements;
Assessment effort previously expended in piecemeal assessments can be diverted to delivering
assistance to the affected population;
Responding agencies not normally participating in the WASH Cluster have an incentive to
coordinate through the cluster;
Follow up multi-sectoral assessments will have the benefit of good preliminary data and access
to valuable site-specific logistic and operational guidance;
Donations-in-kind can be more reliably assessed for suitability.
Beyond these operational enhancements, the RAT was expected to become a repository of expertise
relating to policy and practice for the gathering of high-value information in emergencies. Their
activities were to enhance the capability of humanitarian agencies in WASH preparedness and response
to disasters.
The project was designed to end in September 2013, following a 24 month intervention period.
However, the RAT consortium has received from the donor a cost modification / project extension for
the second phase of the project for another period of in total 24 months, which started October 1st,
2013.
This project evaluation is designed
to contribute to the analysis of the experiences to date,
providing evidence based findings and lessons learned,
to further inform project activities in the proposed second phase of the project and Global WASH Cluster
strategic decisions.
II. Overall Objective of the Project Evaluation
The evaluation will review project experiences made during the pilot phase 1 and up to today through a
comprehensive analysis of relevancy and efficiency of services. The results of this analysis will be
evidence based findings and lessons learned to inform project activities in the second phase of the
project on the most effective mechanism through which RAT services are to be provided.
III. Specific Objectives of the Project Evaluation
48
Analyse the project Phase 1:
3. To determine the extent to which the original project fulfilled its
objectives and achieved its results; ToR RAT Project Evaluation – page 3
4. To identify the strengths, weaknesses, opportunities and threats of the first phase of the project
(October 1st, 2011 – September 30th, 2013);
5. To review the evolution of the RAT project through phase 1 and determine if current activities are
relevant to:
a. Global WASH Cluster needs
b. National Coordination platforms needs
With the aim:
6. To provide evidence based findings and lessons learned to further inform project activities in the
second phase of the project (October 1st, 2013 – September 30th, 2015);
7. To propose, what if any strategic, structural and operational changes should be introduced in the
second phase of the project (October 1st, 2013 – September 30th, 2015) with a particular reference
to the latest developments in establishing Field Support Teams of and for the Global WASH Cluster
IV. Scope of the Evaluation
a. Review the achievement of the project based on reports and stakeholder feedback in relation to
its results and activities as laid down in the project plan
Was the overall strategy of the project realistic and feasible?
Were the original results SMART? To which degree were they achieved?
Were the targeted beneficiaries reached by the project? If not, why not?
Were risks sufficiently considered in the project plan and the right assumptions made? Were
any crucial elements overlooked in the project design?
Were the original project timeframes and resources appropriate?
What evidence is there of changes in improved WASH assessments in terms of quality and
availability (in time) of data resulting from project activities?
b. Identification of the strengths, weaknesses, opportunities and threats of the project
What areas of the project were particularly successful and why?
What areas of the project were less successful and why?
What were and are the opportunities and threats faced by the project in relation to
implementation, coordination and management and how were they taken into account during
phase 1?
Based on this SWOT what can be taken as determinant(s) of project success?
c. Review the evolution of the RAT project and current relevancy
49
Did the RAT project adapt effectively to meet the needs of the GWC and national coordination
platforms?
ToR RAT Project Evaluation – page 4
Are current activities appropriate considering current GWC architecture and GWC strategic
plans?
Could the services that the RATs provide be delivered through a better mechanism?
d. Evaluation of the implementation, coordination and management of the project
Was the involvement of project partners appropriate to the needs of the project?
What evidence is there of increased collective planning, coordinated implementation and overall
coherency in emergency WASH responses as a result of project activities
Were the governance and executive structures conducive to an optimal implementation of project
activities?
Were all parts of the project implemented in a coordinated and coherent manner?
What changes should be introduced in the implementation, coordination and management of the
project under a second phase?
e. In light of the above, derivation of recommendations for a better informed and an improved
implementation of Phase 2
What strategic changes should be introduced under the second phase of the project?
What changes should be made to the project plan in terms of results and activities and to the
project setup in terms of management and coordination for implementation under a second phase
of the project?
How should the project be best embedded in the existing GWC structure?
V. Methodology
Final methodology to be chosen will be proposed by and agreed upon with the evaluator based on a
respective technical and financial proposal. However, the following corner stones will have to be
included:
1. Briefing (1 day)
At the start of the evaluation, the evaluator(s) will be expected to attend a briefing with CARE USA as
the consortium lead, the CARE Senior WASH Advisor as the technical supervisor, and the current RAT PM
(all in person) in a joint meeting.
50
ToR RAT Project Evaluation – page 5
2. Desk Review (4 days)
Prior to the implementation of field visits and interviews, the evaluator(s) will be expected to review all
project documentation, including (but not limited to)
Project Proposal
Quarterly and Annual Narrative Reports
End of Mission / Deployment Reports
Monthly Reports (RAT Coordinators)
Stakeholders´ feedback
Lessons Learned documents (internal / external)
Quarterly contributions to the quarterly GWC SOF monitoring reports
3. Key Informant Interviews (4 days)
As a mean of more direct input the evaluator(s) will have to undertake interviews via telephone or skype
with all RATs and relevant decision-making partners of the project, particularly:
Focal points of Consortium partners: Oxfam GB, IFRC
Members of the Strategic Advisory Group (SAG)
Representative(s) GWC Cluster Advocacy and Support Team (CAST)
Representative WASH in Emergencies from UNICEF Programme Division (New York)
The evaluator(s) will also be expected to communicate with OFDA representatives in Washington D.C.
4. Stakeholder Survey (2 days)
A stakeholder survey will be undertaken through a (email/online) questionnaire to all stakeholders in all
deployment countries who have or should have benefitted from the project, and by direct interviews via
telephone or skype of selected appropriate stakeholders with a particular relevance to key issues of RAT
deployments. Stakeholders comprised in the survey should be, among others, the following:
Regional Emergency Coordination Advisors (RECA)
Rapid Response Teams (RRT)
UNICEF representatives (regional and country level)
WASH Cluster/platform Coordinators
Members of WASH coordination clusters/platforms in the countries of executed RAT
deployment
project partner agency representatives in the countries of executed RAT deployment
5. Field visits (12 days)
The evaluator(s) will be expected to travel to 2 regions/countries where RAT deployments have been
carried out. These field visits should be used to discuss vis-à-vis with regional and/or national WASH
51
cluster / coordination mechanisms and WASH partners how they perceive the quality and relevance of
the services rendered as well as their satisfaction and/or recommendations.
ToR RAT Project Evaluation – page 6
6. Report Compilation (5 + 1 days) including Debriefing (1 day)
After finalisation of the evaluation analysis a draft report will have to be compiled and submitted to the
consortium lead at a date agreed upon between the evaluator and the consortium within the timeframe
of the planned evaluation. After submission the consortium has one week to review the draft report and
come back with comments. This will be followed by a one day debriefing meeting between the
evaluator(s) and CARE USA to finalise the discussion around the findings and recommendations. Based
on the results of the debriefing meeting the evaluator will amend the report and submit the final
version.
VI. Profile of the Evaluator
Proven experience and knowledge in carrying out needs assessments in the WASH sector in
different contexts;
Proven experience and knowledge in project management, particularly for that of consortium
related projects;
Prior project evaluation experience a must;
Significant field experience in the evaluation of humanitarian / development projects
Relevant degree / equivalent experience related to the evaluation to be undertaken
Significant experience in coordination, design, implementation, monitoring and evaluation of
programmes;
Good communications skills and experience of workshop facilitation;
Ability to write clear and useful reports (may be required to produce examples of previous
work);
Fluent in English required; and French an asset;
Understanding of donor requirements, particularly that of OFDA;
Ability to manage the available time and resources and to work to tight deadlines;
Independence from the parties involved.
VII. Outputs
Prior to the final submission, a debriefing meeting will be held between the RAT Consortium, RAT/RECA
Project Manager, CAST and the evaluator(s) to discuss the preliminary findings of the evaluation and
clarify open issues.
Afterwards the evaluator(s) will produce one final evaluation report, following the format:
Cover Page
Table of Contents
52
Executive Summary: must be a standalone summary, describing the programme, main findings
of the evaluation, and conclusions and recommendations. This will be no more than 3 pages in
length.
ToR RAT Project Evaluation – page 7
Main Body: The main body of the report shall elaborate the points listed in the Executive
Summary. It will include references to the methodology used for the evaluation and the context
of the action. In particular, for each key conclusion there should be a corresponding
recommendation. Recommendations should be as realistic, operational and pragmatic as
possible; that is, they should take careful account of the circumstances currently prevailing in
the context of the action, and of the resources available to implement it.
Annexes: Listed and correctly numbered.
Format for the main body of the report is:
o Background Information
o Methodology
o Findings and Discussions
o Conclusions and Recommendations
o Annexes
The report should not be longer than 30 pages excluding annexes. The final report will be submitted no
later than the end date of the consultancy contract. Annexes to the report will be accepted in the
working language of the country and programme subject to the project evaluation.
VIII. Level of Effort
(1) Briefing: 1 day
(2) Desk Review 4 days
(3) Interviews: 4 days
(4) Survey: 2 days
(5) Field visits: 12 days
(6) Data analysis and draft report writing: 5 days
(7) Debriefing: 1 day
(8) Final report preparation: 1 day
Total: 30 days
CARE USA, 9.2.2014
53
4/9/144/21/14
Phase IEvaluation Prep/Lit Review
Dev Eval Plan14‐Apr
Initial Skype Call16‐Apr
Literature Review2‐M
ay4
Draft Online Survey Interview
Questions
28‐Apr0.25
Second Skype Call(0800 Pacific Tim
e)29‐Apr
0.25Total Days
Phase II Stakeholder on‐Line Survey
29‐AprFinalize Survey/questions
2‐May
Set up Survey on Survey Monkey/Release
2‐May
Survey data collection3‐M
ay13‐M
ayTotal Days
Phase IIIKey Inform
ant Interviews
Draft interview questions
5‐May
Send e‐mails to RAT team
for review/feedback over e/m
ail/Skype as needed6‐M
ay7‐M
ayKey Inform
ant Interviews betw
een dates*8‐M
ay16‐M
ayTotal Days
Phase IVField Visits
KI Interviews Phllippines
5/20/145/30/14
Total DaysPhase V
Report Writing
Analysis of results5/28/14
6/19/14Preparation of Draft Report
9‐Jun19‐ Jun
Debriefing with CARE
(& Others in G
eneval TBD)**1‐Jul1 day
Comment period
TBDReport Finalization
TBD1 day
Total DaysTotal Evaluation D
ays30
*Some of these w
ill also be conducted during field visits where relevant
DRAFT RAT Mid‐Term
Evaluation Timeline/W
ork Plan
54
5
4
Home
Home
0.5
7
Field/Home
7
Field Visits
up to 12
122
Home
2+
Introduction e‐mail to KI's/Survey contacts by RAT
54
Proposal/Planning
Doc
TOR
Progress/ EO
M/Field
Report
LL Doc
Trip Report
Assessment
Report
Assessment
Tool(s)/ supports
Indicator tools
Data base/
spreadsheetTraining
Map
Other
Deploym
ent Docum
ents
Philippines #2 (March to M
ay 2014)
WASH Baseline Barangay Assessm
ent Phase I: Interim Technical Report
X
WASH Assessm
ent Evaluation ToolX
WASH Assessm
ent Raw and Aggregate data spreadsheet
XWASH Baseline Survey Basic W
ASH training for enumerators
XSouth Sudan (February‐M
arch 2014)SS RAT EO
M Report
XSyria Rem
ote from Turkey (N
ovember‐D
ecember 2013)
TOR (4‐5 w
eeks) X
EOM Report (20 days)
XX
01/12/2013 RAT Mission SitRep 1
X
09/12/13 RAT Mission SitRep 2
X
FSTReport_RAT Deployment_Turkey
X
Draft Continuous Needs M
onitoring Framew
ork (Annex C)X
X
WASH Indicators and Data Sets (Annex B)
XLessons Learned Doc
XPhilippines #1 (N
ovember‐D
ecember 2013)
FST Report by RATX
FST Report by RATX
RAT EOM Report Yolanda Em
ergency‐PhilippinesX
EOM Report Philippines IFRC
X
MIRA Initial Rapid Assessm
entX
RAT YOLAN
DA Basic WASH Training
XDebriefing RECA SEAT RAT‐O
xfamX
Final REACH Shelter/WASH Assessm
ent Methodology/Tool
X
Shelter and WASH Rapid Assessm
ent Typhoon Hiayan, Philippines Final ReportX
Shelter and WASH Response M
onitoring Final ReportX
REACH BOHO
L Assessment Tools (HH, KI and FG
D)X
Lessons LearnedX
Nam
ibia (July 2013)Draft TO
R Nam
ibia Drought Response X
EOM Report
XX Lebanon (Refugees) (Feb‐M
arch 2013)
Attachment 4 M
atrix of Evaluation Reviewed RAT Produced or FST RAT Associated D
ocuments
Docum
ent/Report Type
Type of Docum
ent
55
TOR W
ASH Needs Assessm
entX
TWG Assessm
ent Workplan G
nant ChartX
EoM Report Final
X
Household Level Assessment Tool
XX
Municipal Level Structured Q
uestionnaire (also includes Community and HH)
X
WASH W
orking Group Intervention/Planning Spread Sheet
WASH indicator Data Sets (Annex C)
X
Sampling M
ethods (Annex C)X
Data Collection Methods (Annex C)
X
WASH Assessm
ent Report Template (Annex C)
X
MAP W
ASH Sector Response Assessment (Annex C)
X
Workshop: Standardized Assessm
ent Framew
ork Agenda X
Minutes Standardized Assessm
ent Framew
ork Workshop (Annex H)
XCapacity Assessm
ent Tool (Survey Monkey) (Annex E)
X
WASH Activities M
atrixX
Myanm
ar (January‐March 2013)
EOM (includes TO
R)X
XX Syria/Rem
ote from Jordan (Syria Conflict) N
ov‐Dec 2012)
TOR
X
EOM Report
XX
021212 RAT Mission SitRep 1
X091212 RAT M
ission SitRep 2X
Annex C MLA Structured Q
uestionnaireX
Annex B HLA Structured Questionnair e
XX
WASH W
orking Group 3W
reportX
Lessons LearnedX
X Burkina Faso (influx of Malian refugees and food/nutrition crisis) (July 2012)
RAT EOM Report
X
16‐07‐12 RAT Mission SitRep 1 BF
X18‐07‐12 RAT M
ission SitRep 2 BFX
Lessons Learnt
Mali (Com
plex Emergency food and nutrition) (M
ay 2012)EoM
Report (includes TOR)
XX
XX
Lessons Learned X
X Chad (food and nutrition crisis) (March 2012)
RATCHAD EOM Report Final
X
18‐03‐12 RAT Update N
o 1X
23‐03‐12 RAT Update N
o 2X
Annex 1 Assessment Tool/FG
D (English/French)X
56
Field Visit Maps (N
= 3)X
Annex 8 _ Cholera Incidence ChadX
Annex 7 WASH_N
utrition ChadX
Lessons LearntX
X Madagascar (Cyclone G
iovanna) (Feb‐March 2012)
RAT Madagascar EO
M Report
XX
XUpdate N
o 1 Site Rep ReportX
Update N
o. 2 Site Rep ReportX
Lessons Learned Doc General Project D
ocuments
OFDA Proposal
XResults of RAT w
orkshop Toulouse X
RAT Lessons Learned October 2011‐Dec 2013
XX
Results RAT Meeting Jakarta Final
X
EOM Report IASC/M
IRA Workshop Indonesia
XQuarterly and Annual O
FDA Reports
Oct. 1‐Dec. 31, 2011
XAnnual Report Septem
ber 20, 2011 ‐ September 30, 2012
XOct. 1‐Dec. 31, 2012
XJuly‐Sept 2012
XApril 1‐June 30, 2012
XJan‐M
arch 2012 X
Quarterly Report O
ct. 1‐Dec. 31, 2013X
July‐Sept 2013 X
April‐June 2013 X
Quarterly Report Jan 1‐M
arch 31 2013 X
Quarterly Report Jan 1 ‐M
arch 31, 2014X
RAT/FST ProductsRAT Brochure
X
Protocol 1 Deployment Procedures
XProtocol 2 Lessons Learnt Docum
entsX
Protocol 3 Countries Prone to Disaster X
Protocol 4 ‐ Good Enough Assessm
ents‐ Draft Indicator Assessment
spreadsheet and Summary: M
ost useful Tools for Rapid WASH N
eeds Assessm
entsX
Template for EO
M/Handover reports for the FST
XRegional Support Package Plan in regional FST sub‐team
sX
Assessment Fram
ework for W
ASH Assessments (U
rban, Flooding, Drought)X
FST Concept Paper, May 2014
X
Key:Project
EAPROWCARO
EASAROMEN
A
57
Greetings WASH Interagency Rapid Assessment Team (RAT) Project Stakeholders, This survey is being implemented as part of the Midterm Evaluation of the Interagency Rapid Assessment Team (RAT) Project currently being conducted to assess the first phase of the RAT Project (October 1, 2011September 30, 2013) and up through today. Some basic information about the RAT project can be found here http://www.washcluster.info/?q=content/rapidassessmentteamrat. Your input is being sought as you were identified to have had contact with, or worked with the RAT team or overall RAT process. The purpose of the MidTerm Evaluation is to provide an analysis of the project todate using evidencebased findings and lessons learned through desk review, key informant interviews and this online survey. Evaluation findings will be used to inform the second phase of the RAT Project, as well as overall Global WASH Cluster Strategic decisions related to WASH assessments in emergencies. On average the survey can be completed in 15 minutes. Your thoughtful and honest feedback is greatly appreciated. Survey data results will be presented in the final report in aggregate form. No names or organizations will be tied to specific survey responses and comments by individual respondents will only be known to the evaluator, so anonymity will be retained. You are requested to please complete the survey no later than the end of the day on May 13, 2014. Thank you for taking the time to complete the survey! Kay Mattson Independent External WASH Project Evaluation Consultant
58
Please provide the following demographic information. Respondent names will be included in the final report to capture overall survey respondents, but names/organizations will not be tied to any comments. Contact information is requested in case there is a need to followup, or clarify, any of your responses.
1. What is your name? (Your responses will be kept anonymous in the final analysis)
2. What is your email address?
3 What organization did you work for when you worked with the RAT Project (Please write the complete name unless the acronym is VERY well known. Thanks!)
3a Do you still work for this same organization?
4. What was/is your role in relationship to the RAT project (Multiple responses accepted):
DEMOGRAPHICS/RESPONDENT INFORMATION
Yes
nmlkj
No
nmlkj
RAT Consortium agency partner
gfedc
Member of Global WASH Cluster (GWC) SAG
gfedc
GWC CAST member
gfedc
RECA Advisor
gfedc
RRT Cluster Coordinator
gfedc
RRT Information Manager
gfedc
National WASH Cluster Coordinator
gfedc
National WASH Cluster Information Manager
gfedc
National WASH Cluster/coordination platform Member
gfedc
National NGO or National Government implementing partner
gfedc
International NGO or UN Agency representative/implementing partner
gfedc
RAT Project Donor
gfedc
Other (please specify)
59
5. What is your main emergency response role/position?
6. Have you worked directly with a RAT deployment?
Technical WASH expert field responder
nmlkj
Technical WASH expert desk officer/HQ focal point
nmlkj
Other Sector expert (Please indicate which Sector in Other)
nmlkj
Management or administrator
nmlkj
Donor
nmlkj
Other (please specify)
Yes
nmlkj
No (Skip to Question 8)
nmlkj
60
7. If YES to Q 6 The following is a list of RAT deployments since the project began on October 1, 2011 (starting with the most recent), please check all that you have had direct working experience.
Philippines Hurricane Yolanda (March to present)
gfedc
South Sudan Conflict (FebruaryMarch 2014)
gfedc
Syria Conflict (Remote from Turkey) (NovemberDecember 2013)
gfedc
Philippines Hurricane Yolanda (NovemberDecember 2013)
gfedc
Namibia Drought (July 2013)
gfedc
Lebanon Refugee Crisis (FebruaryMarch 2013)
gfedc
Myanmar Ethnic Conflict Rakhine State (JanuaryMarch 2013)
gfedc
Syria Conflict (Remote from Jordan) (NovemberDecember 2012)
gfedc
Burkina Faso Refugee/Food/Nutrition Crisis (July 2012)
gfedc
Mali Food/Nutrition Crisis (May 2012)
gfedc
Chad Food/Nutrition Crisis (March 2012)
gfedc
Madagascar Cyclone Giovanna (FebruaryMarch 2012)
gfedc
61
The following are open ended questions to capture your general perceptions of the RAT project and your thoughts on the Strengths, Weaknesses, Opportunities and Treats (SWOT) related to the RAT project. You are encouraged to use specific examples, whenever possible with your response to the questions below. 8. How would you describe the purpose of the RAT project in your own words?
9. What do you think the RAT Project has been particularly successful at (its strengths) todate (specific examples appreciated)?
10. What do you think the RAT Project has been less successful at (its weaknesses) to date (specific examples appreciated)?
11. What do you see as the POTENTIAL OPPORTUNITIES for the RAT project during the project’s second phase?
12. What do you see are the greatest CHALLENGES or THREATS for the RAT project during the projects’ second phase and on into the future?
ASSESSMENT OF OVERALL SWOT OF THE RAT PROJECT'S WORK
55
66
55
66
55
66
55
66
55
66
62
The following questions ask you to rate various aspects of the RAT Project including how effective the project has been at working to achieve its goal/principle objective, the project’s strategy, relevancy to the Global WASH Cluster current needs, coordination with stakeholders, impact on WASH in emergency assessments for deployments conducted by the RAT project, and importance of activities conducted by the RAT Project for Phase II of the project. 13. “The PRINCIPLE GOAL of the RAT project was to support WASH Cluster operations in major Emergencies by providing timely, systematic, and comprehensive data on numbers, location and condition of affected populations, with the PRINCIPLE OBJECTIVE to improve the quality of WASH Sector assessments in emergency operations.” With the this statement in mind please rate how effective the project has been overall at “improving the quality of WASH sector assessments in emergency operations”:
14. To what extent do you agree with the statement “The overall RAT project strategy is realistic and feasible”?
ASSESSMENTOF THE RAT PROJECTS RELEVANCY, GOALS AND OUTPUTS
1 Extremely Effective
nmlkj
2 Moderately Effective
nmlkj
3 Neither Effective or Not Effective
nmlkj
4 Moderately Ineffective
nmlkj
5 Very Ineffective
nmlkj
6 Unknown/not able to say at this time
nmlkj
1 Agree Completely
nmlkj
2 Moderately Agree
nmlkj
3 Neither Agree or Disagree
nmlkj
4 Moderately disagree
nmlkj
5 Disagree completely
nmlkj
6 Unknown/not able to say at this time
nmlkj
Any Comments about why you ranked this question this way?
55
66
63
15. How relevant do you think the project is to current Global WASH Cluster Work?
16. Overall how effective was the RAT at Coordinating with Stakeholders during deployments?
17. How effective was the RAT project at adding value to producing useful deployment assessment products/data/reports that met you/your organization’s needs?
1 Extremely Relevant
nmlkj
2 Moderately Relevant
nmlkj
3 Neither Relevant or Not Relevant
nmlkj
4 Moderately Not Relevant
nmlkj
5 Extremely Not Relevant
nmlkj
6 Unknown/Not able to say at this time
nmlkj
Any Comments about why you ranked this question this way?
55
66
1 Extremely Effective
nmlkj
2 Moderately Effective
nmlkj
3 Neither Effective or Not Effective
nmlkj
4 Moderately Ineffective
nmlkj
5 Extremely Ineffective
nmlkj
6 Unknown/not able to say at this time
nmlkj
Any Comments about why you ranked this question this way?
55
66
1 Extremely Effective
nmlkj
2 Moderately Effective
nmlkj
3 Neither Effective or Not Effective
nmlkj
4 Moderately Ineffective
nmlkj
5 Extremely Ineffective
nmlkj
6 Unknown/not able to say at this time
nmlkj
Any Comments about why you ranked this question this way?
55
66
64
18. The following is a list of some activities undertaken in some, but not all RAT deployments, please rate on a scale of 1 to 5 (one lowest, five highest) how important you think these activities are to the overall RAT Project Strategy going into Phase 2 of the project SPECIFIC TO EMERGENCY RAT DEPLOYMENTS
Lowest Low Maybe Important High Very High No opinion
a. Development/Implementation of WASH assessments
nmlkj nmlkj nmlkj nmlkj nmlkj nmlkj
b. Development/implementation of and contributing to Multi Sector assessments (e.g. WASH and food/nutrition focused assessments, WASH and infectious diseases, WASH in shelter, WASH in schools, WASH in health care, etc.) together with others (REACH, ACAPS, etc.)
nmlkj nmlkj nmlkj nmlkj nmlkj nmlkj
c. Participation in larger MultiCluster assessments (e.g. MIRA,)
nmlkj nmlkj nmlkj nmlkj nmlkj nmlkj
d. Review of secondary data nmlkj nmlkj nmlkj nmlkj nmlkj nmlkj
e. Training others on WASH assessment approaches, designs, methodologies
nmlkj nmlkj nmlkj nmlkj nmlkj nmlkj
f. Conducting or providing technical WASH expertise inputs (e.g. assessment of specific water system or latrine rehabilitation needs/materials, water quality testing, etc.)
nmlkj nmlkj nmlkj nmlkj nmlkj nmlkj
g. Conducting WASH organizational capacity mapping
nmlkj nmlkj nmlkj nmlkj nmlkj nmlkj
h. Development of continuous needs monitoring frameworks
nmlkj nmlkj nmlkj nmlkj nmlkj nmlkj
i. Other (describe below and rate here)
nmlkj nmlkj nmlkj nmlkj nmlkj nmlkj
Other (please specify)
65
18a. Same as above, Please rate on a scale of 1 to 5 (one lowest, five highest) how important you think these activities are to the overall RAT Project Strategy going into Phase 2 of the project BROADER WASH IN EMERGENCY ACTIVITIES NOT RELATED TO A SPECIFIC DEPLOYMENT
Lowest Low Maybe Important High Very High No opinion
j. Further development of Countries Prone to Disasters database
nmlkj nmlkj nmlkj nmlkj nmlkj nmlkj
k. Development of rapid WASH assessment tools/questionnaires
nmlkj nmlkj nmlkj nmlkj nmlkj nmlkj
l. Development and dissemination of specific survey methodologies and techniques for WASH and multisector needs assessments
nmlkj nmlkj nmlkj nmlkj nmlkj nmlkj
m. Training/capacity building related to conducting effective WASH assessment (e.g. approaches, designs, methodologies, etc.)
nmlkj nmlkj nmlkj nmlkj nmlkj nmlkj
n. Development of Emergency WASH Indicator data sets for assessments and M&E
nmlkj nmlkj nmlkj nmlkj nmlkj nmlkj
o. Development of Assessment Frameworks for specific types of disasters (e.g. urban, flooding, droughts, etc.)
nmlkj nmlkj nmlkj nmlkj nmlkj nmlkj
P. Other (Describe below and rate here)
nmlkj nmlkj nmlkj nmlkj nmlkj nmlkj
Other (please specify)
66
The survey is almost complete! The following question ask you to provide input on what changes you might recommend for Phase II of the project as well as an opportunity to provide any additional comments you may have. 19) What, if any, changes do you recommend to the RAT Project to make the project’s work more effective specific to the following areas
19a. How the project is IMPLEMENTED:
19b. How the project COORDINATES WITH OTHERS:
19c. How the project is STRUCTURED or MANAGED:
20. Any final comments or feedback you would like to share?
The survey is now complete. Thank you for your time!
RECOMMENDATIONS/FINAL COMMENTS
55
66
55
66
55
66
55
66
67
Attachment 6
RAT Project Mid-Term Evaluation Key Informant Interview Question Guide
(Estimated time 45 min)
The RAT Projects Principle Goals and Objective:
“The PRINCIPLE GOAL of the RAT project was to support WASH Cluster operations in major Emergencies
by providing timely, systematic, and comprehensive data on numbers, location and condition of affected
populations, with the PRINCIPLE OBJECTIVE to improve the quality of WASH Sector assessments in
emergency operations.”
Demographics (some of these maybe already answered in on-line Survey)
1. How long have you been involved in the RAT project?
2. Who do you work for now? Is this the same as when working w/ RAT?
3. What is your role(s) in relationship to the RAT project?
Project Scope/Achievements/Results
4. Do you think that the RAT project strategy, as designed was realistic and feasible?
a. Why/why not?
5. Were the right assumptions made about the project to adequately and effectively implement it?
a. Why/why not?
b. What if anything, was not considered or unforeseen? (Are these issues still relevant to the
project at this time?)
6. Were the original project timeframes and resources adequate to meet the projects goals?
a. If not, what needed to be done differently, or what resources were needed? Are these still
relevant today for the project?
7. What considerations, if any, could have been/needed to be addressed differently in the project’s
design specific to the following areas: (Note these are also covered in the On-line Survey. IF the interviewee has answered these in the survey will discuss more here their
responses, as necessary.)
a. How the project is being implemented?
b. How the project is managed?
8. Who do you see as the project beneficiaries?
a. Do you think their needs, in relationship to WASH assessments, were/ are being met by this
project?
b. Why/Why not? Examples?
9. What do you think are the main results coming out of the first two years of the project and up
through the first of May?
a. Do you think the project has increased “collective planning, coordinated implementation
and overall coherency in emergency WASH responses” as a result of the RAT project’s
efforts?
b. Has the RAT project achieved its purpose and main objective “to improve the quality of
WASH Sector assessments in emergency operations”? If yes, how. If not, why not?
68
RAT and the GWC Structure
10. How well does the RAT project currently fit into the Global WASH structure (e.g. coordination
platforms, strategic plans)?
a. Is there a role for the RAT project in this structure?
b. For the RAT to be more effective are there any modifications that need to be made to better
align the project with the GWC Structure?
Deployments
11. What RAT deployments have you been directly involved in, if any?
12. What has worked well in these deployments?
13. What did not worked so well? Specific examples are appreciated.
a. What suggestions do you have to improve deployments?
Non –Deployment Activities
14. What other RAT project activities (non-deployment) have you been involved in or aware of, if
any?
15. Please discuss these activities and your feedback on their effectiveness and relevance to the
overall GWC WASH in emergency assessment/other efforts?
Measurement of Success
The project is now into the beginning of year three of four years (Phase II). How will we know if the
project has been successful at the end of the Phase II of the project? What will be different about the
project and the overall GWC assessment in emergencies system/structure, than how it looks now, if
anything?
69
ATTACHMENT 7
Mid-Term Evaluation Contributors by Method
Name Survey Skype KI
Interview
Field KI
Interview
Organization worked for associated
with the RAT Project
Abdallah Abdelrassoul X Oxfam GB
Abel Augustinio X 5/22/2014 IFRC
Abraham Varampath X Save the Children
Andi Dyah 5/28/2014 Save the Children
Andreas Nilsson X Swedish Civil Contingencies Agency
Andrew Parker X 5/15/2014 UNICEF
Andy Bastable X Oxfam GB
Aurelien Read 5/29/2014 Solidarities International-Philippines
Azadeh Hasani 5/27/2014 Relief International
Bagus Setyawan 5/27/2014 Oxfam-Tacloban, Philippines
Camelia Marinescu 5/26/2014 IFRC - Philippines
Chery McDonald X Tearfund
Chris Seremet X Catholic Relief Services
Claire Gaillardou X Action Contre la Faim
Colin McCubbin X Medair
Damien Brosnon X 5/14/2014 UNICEF- CARE
David Alford X 5/20/2014 GWC RRT
David Gazashvili X 6/4/2014 CARE USA
Denis Heidebroek X ECHO
Doddy Suparta 5/25/2014 Mercy Corps-Cebu
Emily Beadle 5/28/2014 Acted
Emma Tuck X Solidarites / UNICEF - RRT
Esther Kabahuma 5/27/2014 Oxfam-Tacloban, Philippines
Fidel Pena 5/29/2014* American Red Cross
Francesco Dotto X 5/16/2014 Oxfam GB
Franck Bouvet X 6/3/2014 UNICEF
Francois Bellet X UNICEF WCARO & WASH Regional Group
Georg Nothelle X 5/27/2014 CARE
Gudrun Bertinussen X Norwegian Church Aid
Hasan Hamou X 5/14/2014 RECA/IFRC
Jainil Didaraly X RECA WCARO ACF
Jean Lapegue X Action Contre la Faim
Jesse Pleger X 5/28/2014 UNICEF
Johann Hervoche 5/27/2014 Relief International
John Akudago X Samaritan's Purse
70
Name Survey Skype KI
Interview
Field KI
Interview
Organization worked for associated
with the RAT Project
Jola Miziniak X UNICEF
Jon Barden X Department for International Development
Jules X ABMS
Kit Dyer X 5/16/2014 UNICEF
Lino Deng Akot X ACF-USA (action controlia firm)
Loles Solis 5/25/2014 ACF
Luca Pupulin X ACTED
Mark Buttle X Save the Children
Monica Ramos X 5/16/2014 CARE
Mugur Dumitrache X Mercy Corps
Niall Boot X GOAL
Nick Brooks X 5/19/2014 CARE
Nicole Hahn X 5/26/2014 UNICEF
Nina Odling X 5/27/2014 Oxfam
Oliver Wright X South Sudan WASH Cluster / Medair
Pablo Alcalde X ACF
Pankaj Kumar Singh X UNHCR
Patrick Laurent X UNICEF
Paul Gerard Escoton 5/25/2014 Mercy Corps - Tacloban
Paul Shanahan 6/2/2014 UNICEF
Prasad Bhagwan Sevekari X Norwegian Church Aid
Renaud Zambeaux 5/24/2014 REACH
Richard Tracey X UNHCR
Rick Bauer X Oxfam
Robert Fraser X 5/19/2014 IFRC
Robert Ikoha X UNHCR
Roberto Arranz X Acción Contra el Hambre (ACF-Spain)
Rodolphe Houlsonron X UNICEF
Rory Villaluna X 5/29/2014 UNICEF Phillipines
Shenna Calub 5/29/2014 UNICEF Philippines
Shuuya Mathew X Namibia Red Cross Society
Silvia Ramos Martinez X UNICEF/NCA
Souleymane Sow X 5/23/2014 CARE Germany-Luxemburg
Subodh Vijapure X Concern
Trevor White X 6/2/2014 USAID/OFDA
Vincent Annoni X REACH initiative
Yvan Grayel X ACF Contract - RRT Deployment in UNICEF
Total 58 17 17 Unduplicated: 74 People/27 Organizations
Yellow highlight - incomplete survey respose *Correspondence
71
Inter-AgencyRapidAssessmentTeam(RAT)ProjectMid-TermEvaluationStakeholderSurvey
4/29
89.09% 49
10.91% 6
Q43aDoyoustillworkforthissameorganization?
Answered:55 Skipped:3
Total 55
Yes
No
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
AnswerChoices Responses
Yes
No
72
Inter-AgencyRapidAssessmentTeam(RAT)ProjectMid-TermEvaluationStakeholderSurvey
5/29
13.33% 6
24.44% 11
4.44% 2
11.11% 5
8.89% 4
4.44% 2
Q54.Whatwas/isyourroleinrelationshiptotheRATproject(Multipleresponses
accepted):Answered:45 Skipped:13
RATConsortiumagencypartner
MemberofGlobalWASH...
GWCCASTmember
RECAAdv isor
RRTClusterCoordinator
RRTInformation...
NationalWASHCluster...
NationalWASHCluster...
NationalWASHCluster/coor...
NationalNGOorNational...
InternationalNGOorUN...
RATProjectDonor
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
AnswerChoices Responses
RATConsortiumagencypartner
MemberofGlobalWASHCluster(GWC)SAG
GWCCASTmember
RECAAdvisor
RRTClusterCoordinator
RRTInformationManager
73
Inter-AgencyRapidAssessmentTeam(RAT)ProjectMid-TermEvaluationStakeholderSurvey
6/29
11.11% 5
0.00% 0
2.22% 1
2.22% 1
42.22% 19
6.67% 3
TotalRespondents:45
NationalWASHClusterCoordinator
NationalWASHClusterInformationManager
NationalWASHCluster/coordinationplatformMember
NationalNGOorNationalGovernmentimplementingpartner
InternationalNGOorUNAgencyrepresentative/implementingpartner
RATProjectDonor
74
Inter-AgencyRapidAssessmentTeam(RAT)ProjectMid-TermEvaluationStakeholderSurvey
7/29
38.00% 19
36.00% 18
4.00% 2
16.00% 8
6.00% 3
Q65.Whatisyourmainemergencyresponserole/position?
Answered:50 Skipped:8
Total 50
TechnicalWASHexpertfield...
TechnicalWASHexpertdesk...
OtherSectorexpert(Plea...
Managementoradministrator
Donor
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
AnswerChoices Responses
TechnicalWASHexpertfieldresponder
TechnicalWASHexpertdeskofficer/HQfocalpoint
OtherSectorexpert(PleaseindicatewhichSectorinOther)
Managementoradministrator
Donor
75
Inter-AgencyRapidAssessmentTeam(RAT)ProjectMid-TermEvaluationStakeholderSurvey
8/29
46.43% 26
53.57% 30
Q76.HaveyouworkeddirectlywithaRATdeployment?
Answered:56 Skipped:2
Total 56
Yes
No(SkiptoQuestion8)
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
AnswerChoices Responses
Yes
No(SkiptoQuestion8)
76
Inter-AgencyRapidAssessmentTeam(RAT)ProjectMid-TermEvaluationStakeholderSurvey
9/29
34.62% 9
26.92% 7
19.23% 5
15.38% 4
Q87.IfYEStoQ6-ThefollowingisalistofRATdeploymentssincetheprojectbeganonOctober1,2011(startingwiththemostrecent),pleasecheckallthatyouhavehad
directworkingexperience.Answered:26 Skipped:32
PhilippinesHurricane...
PhilippinesHurricane...
SyriaConflict(Remotefrom...
SouthSudanConflict...
LebanonRefugeeCris...
BurkinaFasoRefugee/Food...
MaliFood/Nutriti...
MyanmarEthnicConflict...
ChadFood/Nutriti...
NamibiaDrought(Jul...
SyriaConflict(Remotefrom...
MadagascarCyclone...
0% 10% 20% 30% 40% 50%
34.62%
26.92%
19.23%
15.38%
15.38%
15.38%
15.38%
11.54%
11.54%
7.69%
7.69%
3.85%
AnswerChoices Responses
PhilippinesHurricaneYolanda(November-December2013)
Phil ippinesHurricaneYolanda(Marchtopresent)
SyriaConflic t(RemotefromTurkey)(November-December2013)
SouthSudanConflic t(February-March2014)
77
Inter-AgencyRapidAssessmentTeam(RAT)ProjectMid-TermEvaluationStakeholderSurvey
10/29
15.38% 4
15.38% 4
15.38% 4
11.54% 3
11.54% 3
7.69% 2
7.69% 2
3.85% 1
TotalRespondents:26
LebanonRefugeeCrisis(February-March2013)
BurkinaFasoRefugee/Food/NutritionCrisis(July2012)
MaliFood/NutritionCrisis(May2012)
MyanmarEthnicConflic tRakhineState(January-March2013)
ChadFood/NutritionCrisis(March2012)
NamibiaDrought(July2013)
SyriaConflic t(RemotefromJordan)(November-December2012)
MadagascarCycloneGiovanna(February-March2012)
78
Inter-AgencyRapidAssessmentTeam(RAT)ProjectMid-TermEvaluationStakeholderSurvey
16/29
2.38% 1
61.90% 26
0.00% 0
11.90% 5
7.14% 3
16.67% 7
Q1413.“ThePRINCIPLEGOALoftheRATprojectwastosupportWASHClusteroperationsinmajorEmergenciesbyprovidingtimely,systematic,and
comprehensivedataonnumbers,locationandconditionofaffectedpopulations,withthePRINCIPLEOBJECTIVEtoimprovethequalityofWASHSectorassessmentsinemergencyoperations.”Withthethisstatementinmindpleaseratehow
effectivetheprojecthasbeenoverallat“improvingthequalityofWASHsectorassessmentsinemergencyoperations”:
Answered:42 Skipped:16
1-ExtremelyEffective
2-ModeratelyEffective
3-NeitherEffectiveor...
4-ModeratelyIneffective
5-VeryIneffective
6-Unknown/not...
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
AnswerChoices Responses
1-ExtremelyEffective
2-ModeratelyEffective
3-NeitherEffectiveorNotEffective
4-ModeratelyIneffective
5-VeryIneffective
6-Unknown/notabletosayatthistime
79
Inter-AgencyRapidAssessmentTeam(RAT)ProjectMid-TermEvaluationStakeholderSurvey
17/29
Total 42
80
Inter-AgencyRapidAssessmentTeam(RAT)ProjectMid-TermEvaluationStakeholderSurvey
18/29
21.43% 9
50.00% 21
7.14% 3
7.14% 3
4.76% 2
9.52% 4
Q1514.Towhatextentdoyouagreewiththestatement“TheoverallRATproject
strategyisrealisticandfeasible”?Answered:42 Skipped:16
Total 42
1-AgreeCompletely
2-ModeratelyAgree
3-NeitherAgreeor...
4-Moderatelydisagree
5-Disagreecompletely
6-Unknown/notabletosay...
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
AnswerChoices Responses
1-AgreeCompletely
2-ModeratelyAgree
3-NeitherAgreeorDisagree
4-Moderatelydisagree
5-Disagreecompletely
6-Unknown/notabletosayatthistime
81
Inter-AgencyRapidAssessmentTeam(RAT)ProjectMid-TermEvaluationStakeholderSurvey
19/29
52.38% 22
30.95% 13
2.38% 1
2.38% 1
4.76% 2
7.14% 3
Q1615.HowrelevantdoyouthinktheprojectistocurrentGlobalWASHCluster
Work?Answered:42 Skipped:16
Total 42
1-ExtremelyRelevant
2-ModeratelyRelevant
3-NeitherRelevantor...
4-ModeratelyNotRelevant
5-ExtremelyNotRelevant
6-Unknown/Notabletosay...
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
AnswerChoices Responses
1-ExtremelyRelevant
2-ModeratelyRelevant
3-NeitherRelevantorNotRelevant
4-ModeratelyNotRelevant
5-ExtremelyNotRelevant
6-Unknown/Notabletosayatthistime
82
Inter-AgencyRapidAssessmentTeam(RAT)ProjectMid-TermEvaluationStakeholderSurvey
20/29
12.20% 5
39.02% 16
12.20% 5
14.63% 6
4.88% 2
17.07% 7
Q1716.OverallhoweffectivewastheRATatCoordinatingwithStakeholdersduring
deployments?Answered:41 Skipped:17
Total 41
1-ExtremelyEffective
2-ModeratelyEffective
3-NeitherEffectiveor...
4-ModeratelyIneffective
5-ExtremelyIneffective
6-Unknown/notabletosay...
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
AnswerChoices Responses
1-ExtremelyEffective
2-ModeratelyEffective
3-NeitherEffectiveorNotEffective
4-ModeratelyIneffective
5-ExtremelyIneffective
6-Unknown/notabletosayatthistime
83
Inter-AgencyRapidAssessmentTeam(RAT)ProjectMid-TermEvaluationStakeholderSurvey
21/29
11.90% 5
45.24% 19
7.14% 3
7.14% 3
4.76% 2
23.81% 10
Q1817.HoweffectivewastheRATprojectataddingvaluetoproducinguseful
deploymentassessmentproducts/data/reportsthatmetyou/your
organization’sneeds?Answered:42 Skipped:16
Total 42
1-ExtremelyEffective
2-ModeratelyEffective
3-NeitherEffectiveor...
4-ModeratelyIneffective
5-ExtremelyIneffective
6-Unknown/notabletosay...
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
AnswerChoices Responses
1-ExtremelyEffective
2-ModeratelyEffective
3-NeitherEffectiveorNotEffective
4-ModeratelyIneffective
5-ExtremelyIneffective
6-Unknown/notabletosayatthistime
84
Inter-AgencyRapidAssessmentTeam(RAT)ProjectMid-TermEvaluationStakeholderSurvey
22/29
Q1918.Thefollowingisalistofsomeactivitiesundertakeninsome,butnotallRATdeployments,pleaserateonascaleof1to5(onelowest,fivehighest)how
importantyouthinktheseactivitiesaretotheoverallRATProjectStrategygoinginto
Phase2oftheprojectSPECIFICTOEMERGENCYRATDEPLOYMENTS
Answered:40 Skipped:18
0.00%0
0.00%0
12.50%5
27.50%11
60.00%24
0.00%0
40
4.47
0.00%0
2.50%1
10.00%4
35.00%14
52.50%21
0.00%0
40
4.38
0.00%0
10.00%4
17.50%7
50.00%20
22.50%9
0.00%0
40
3.85
a.Development/...
b.Development/...
c.Participatio...
d.Rev iewofsecondarydata
e.TrainingothersonWA...
f.Conductingo...
g.ConductingWASH...
h.Development...
i.Other(describebe...
0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5
Lowest Low MaybeImportant
High VeryHigh
Noopinion
Total AverageRating
a.Development/ImplementationofWASHassessments
b.Development/implementationofandcontributingtoMulti-Sectorassessments(e.g.WASHandfood/nutritionfocusedassessments,WASHandinfectiousdiseases,WASHinshelter,WASHinschools,WASHinhealthcare,etc.)togetherwithothers(REACH,ACAPS,etc.)
c.Partic ipationinlargerMulti-Clusterassessments(e.g.MIRA,)
85
Inter-AgencyRapidAssessmentTeam(RAT)ProjectMid-TermEvaluationStakeholderSurvey
23/29
2.50%1
0.00%0
25.00%10
45.00%18
27.50%11
0.00%0
40
3.95
2.50%1
5.00%2
10.00%4
52.50%21
30.00%12
0.00%0
40
4.03
15.00%6
10.00%4
25.00%10
25.00%10
25.00%10
0.00%0
40
3.35
2.50%1
17.50%7
40.00%16
20.00%8
20.00%8
0.00%0
40
3.38
0.00%0
0.00%0
32.50%13
37.50%15
27.50%11
2.50%1
40
4.00
0.00%0
0.00%0
28.57%2
0.00%0
42.86%3
28.57%2
7
4.71
d.Reviewofsecondarydata
e.TrainingothersonWASHassessmentapproaches,designs,methodologies
f.ConductingorprovidingtechnicalWASHexpertiseinputs(e.g.assessmentofspecificwatersystemorlatrinerehabil itationneeds/materials,waterqualitytesting,etc.)
g.ConductingWASHorganizationalcapacitymapping
h.Developmentofcontinuousneedsmonitoringframeworks
i.Other(describebelowandratehere)
86
Inter-AgencyRapidAssessmentTeam(RAT)ProjectMid-TermEvaluationStakeholderSurvey
24/29
Q2018a.Sameasabove,Pleaserateonascaleof1to5(onelowest,fivehighest)howimportantyouthinktheseactivitiesaretotheoverallRATProjectStrategy
goingintoPhase2oftheprojectBROADERWASHINEMERGENCYACTIVITIESNOTRELATEDTOASPECIFICDEPLOYMENT
Answered:40 Skipped:18
5.00%2
7.50%3
35.00%14
30.00%12
22.50%9
0.00%0
40
3.58
0.00%0
10.00%4
17.50%7
40.00%16
30.00%12
2.50%1
40
3.98
0.00%0
0.00%0
17.95%7
43.59%17
35.90%14
2.56%1
39
4.23
2.50%1
5.00%2
12.50%5
40.00%16
40.00%16
0.00%0
40
4.10
2.50%1
7.50%3
22.50%9
35.00%14
32.50%13
0.00%0
40
3.88
0.00%0
5.00%2
20.00%8
32.50%13
40.00%16
2.50%1
40
4.15
j .Furtherdevelopment...
k.DevelopmentofrapidWAS...
l.Development...
m.Training/cap...
n.Development...
o.Development...
P.Other(Describebe...
0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5
Lowest Low MaybeImportant
High VeryHigh
Noopinion
Total AverageRating
j .FurtherdevelopmentofCountriesPronetoDisastersdatabase
k.DevelopmentofrapidWASHassessmenttools/questionnaires
l.Developmentanddisseminationofspecific surveymethodologiesandtechniquesforWASHandmulti-sectorneedsassessments
m.Training/capacitybuildingrelatedtoconductingeffectiveWASHassessment(e.g.approaches,designs,methodologies,etc.)
n.DevelopmentofEmergencyWASHIndicatordatasetsforassessmentsandM&E
o.DevelopmentofAssessmentFrameworksforspecific typesofdisasters(e.g.urban,flooding,droughts,etc.)
87
Inter-AgencyRapidAssessmentTeam(RAT)ProjectMid-TermEvaluationStakeholderSurvey
25/29
0.00%0
0.00%0
28.57%2
0.00%0
42.86%3
28.57%2
7
4.71
P.Other(Describebelowandratehere)
88
ATTACHMENT 9
DISCLOSURE OF ANY CONFLICTS OF INTEREST
Name Kay D. Mattson
Title Consultant
Organization Independent
Evaluation Position? X Team Leader Team member
Evaluation Award Number (contract or other instrument) N/A
USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable)
Strengthening the Global WASH Cluster Rapid Assessment Team project funded by the United States Agency for International Development Office of Foreign Disaster Affairs (Agreement Number: AID-OFDA-G-11-00133) with CARE, Oxfam and the International Federation of Red Cross.
I have real or potential conflicts of interest to disclose. Yes X No
If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID
operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated.
2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation.
3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project.
4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated.
5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated.
6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation.
I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished.
Signature
Date June 15, 2014
89