Evaluation Report

Evaluation of the UNICEF Sanitation Programme at Scale in

Pakistan (SPSP) – Phase 1 (2013-14)

Final Version: Dec, 2014

[ii]

Disclaimer: The views and opinion expressed in this report are of the consultants and do not necessarily reflect the views or policies of UNICEF and/or other organizations involved in the programme.

[iii]

List of Acronyms and Abbreviations BCC Behaviour Change Communication

BHU Basic Health Unit

BRSP Baluchistan Rural Support Programme

CAP Community Action Plan

CATS Community Approach to Total Sanitation

CCD Climate Change Division

CEDAW Convention on the Elimination of All Forms of Discrimination against

Women

CLTS Community-Led Total Sanitation

CNA Capacity Needs Assessment

CRC Convention on the Rights of the Child

CRP Community Resource Person

DAC Development Assistance Committee

DFID Department for International Development

DRR Disaster Risk Reduction

DSI Development Solutions International

FATA Federal Administered Tribal Areas

FGD Focus Group Discussion

FRDP Fast Rural Development Program

GDP Gross Domestic Product

GoP Government of Pakistan

GSF Global Sanitation Fund

HACT Harmonised Approach to Cash Transfers

HCD Human Centred Design

HH Household

HRBA Human Rights Based Approach

IEC Information Education Communication

IP Implementing Partner

IRSP Integrated Regional Support Program

JMP Joint Monitoring Programme

KAP Knowledge Attitude and Practice

KP Khyber Pakhtunkhwa

LGRDD Local Government & Rural Development Department

LHW Lady Health Worker

MDG Millennium Development Goals

MEAL Monitoring, Evaluation, Accountability and Learning

MICS Multiple Indicator Cluster Surveys

MIRA Multi-cluster Initial Rapid Assessment

MoCC Ministry of Climate Change

MoRES Monitoring Results for Equity Systems

[iv]

MOU Memorandum of Understanding

NCBP Nations Capacity Building Programme

NDMA National Disaster Management Agency

NDWP National Drinking Water Policy

NFC National Finance Commission

NOC No Objection Certificate

NRSP National Rural Support Programme

OD Open Defecation

ODF Open Defecation Free

OECD Organisation for Economic Co-operation and Development

PATS Pakistan Approach to Total Sanitation

PCA Programme Cooperation Agreements

PDHS Pakistan Demographic and Health Survey

PHED Public Health Engineering Department

PMER Planning, Monitoring, Evaluation & Reporting

PRA Participatory Rural Appraisal

PSLM Pakistan Social and Living Standards Measurement Survey

RSPN Rural Support Programme Network

RUSFAD Rural Sanitation in Flood Affected Areas

SACOSAN South Asia Conferences on Sanitation

SAFWCO Sindh Agricultural and Forestry Workers Coordinating Organization

SLTS School-Led Total Sanitation

SMC School Management Committee

SO Social Organizer

SPSP Sanitation Programme at Scale in Pakistan

SPSS Statistical Package for the Social Sciences

SRSP Sarhad Rural Support Programme

TOC Theory of Change

TOR Terms of Reference

TPM Third Party Monitors

UC Union Council

UN United Nations

UNDAF United Nations Development Action Framework

UNEG United Nations Evaluation Group

UNICEF United Nations Children's Fund

VSC Village Sanitation Committee

WASH Water Sanitation and Hygiene

WDCO Women Development & Community Organization

WSP Water and Sanitation Program

WSSCC Water Supply and Sanitation Collaborative Council

[v]

[vi]

Acknowledgement The Consultants (AAN Associates) are extremely thankful to all those who contributed to the evaluation. Special thanks to Department for International Development (DFID), Government of United Kingdom, for funding the programme. We are thankful to the Implementing Partners (IPs) for coordinating field activities with communities and local stakeholders; and for sharing their own experiences with the evaluators. The consultants are thankful to the members of the ‘Evaluation Reference Group’, which included representatives from DFID, provincial and federal governments and UNICEF for their valued contributions through workshop on initial findings of evaluation. This is to acknowledge the continued support and guidance extended by UNICEF Pakistan and its various programme staff in Islamabad and the provincial offices. We would also like to thank UNICEF evaluation offices at the country and regional level for their consistent engagement, valuable feedback and advice. This may not have been possible without the hard work, resourcefulness and commitment of AAN’s team and the consultants engaged. Thank you, Wajahat Anwar, Salman Mahmood, Kashif Pervaiz, Zaki Ullah, Asmat Gill, Arsalan Kashfi and Arbab Qadir. We wish UNICEF and all other WASH stakeholders’ success with their work and hope to see a country where everyone has access to desirable and affordable WASH services. Nadeem Haider Team Lead/AAN Associates Islamabad Pakistan

[vii]

Table of Contents List of Tables ............................................................................................................ viii

List of Figures ............................................................................................................. ix

List of Annexes ........................................................................................................... x

1. EXECUTIVE SUMMARY .................................................................................... xi

2. OBJECT OF THE EVALUATION ........................................................................ 1

2.1. Programme Overview ....................................................................................... 1

2.2. Country Context ............................................................................................... 5

2.3. Theory of Change ............................................................................................. 8

3. EVALUATION PURPOSE, OBJECTIVES AND SCOPE .................................. 12

4. METHODOLOGY ............................................................................................. 15

4.1. Evaluation Conceptual Framework ................................................................. 15

4.2. Data Collection Methods ................................................................................ 15

4.2.1. Quantitative Evaluation Methods .......................................................... 16

4.2.2. Qualitative Evaluation Methods ................................................................ 17

4.3. Evaluation Constraints, Limitations and Mitigation ......................................... 19

4.4. Ethical Considerations .................................................................................... 22

4.5. Quality Assurance Mechanisms ..................................................................... 22

4.6. Evaluation Management and Stakeholder Participation ................................. 23

5. FINDINGS ........................................................................................................ 25

5.1. Relevance ...................................................................................................... 25

5.2. Effectiveness .................................................................................................. 32

5.2.1. Programme Design ............................................................................... 32

5.2.2. Programme Management & Coordination ............................................ 34

5.2.3. Programme Components & Results ..................................................... 35

Output 1: Demand for Sanitation ....................................................................... 35

Output 2: Supply Side Interventions .................................................................. 42

Output 3: Health and Hygiene Promotion (H&HP) ............................................ 50

Output 4: Linkages with Duty Bearers ............................................................... 53

Output 5: Attaining Total Sanitation ................................................................... 55

Output 6: Knowledge Management and Accountability ..................................... 57

5.2.4. Stakeholders Contribution .................................................................... 59

5.2.5. Gender Equality, Equity & Disability ..................................................... 61

5.3. Efficiency ........................................................................................................ 64

[viii]

5.4. Sustainability .................................................................................................. 68

5.5. Unforeseen Events ......................................................................................... 71

5.6. Human Rights Based Approach ..................................................................... 72

5.7. Partnership and Financial Management ......................................................... 77

5.8. Unexpected Findings ...................................................................................... 80

6. CONCLUSIONS & LESSONS LEARNED ........................................................ 81

7. RECOMMENDATIONS .................................................................................... 83

List of Tables Table 1: Programme Coverage in Terms of Beneficiaries per Area/Province. ........... 4

Table 2: SPSP Theory of Change ............................................................................ 11

Table 3: Evaluation Criteria and Core Questions ..................................................... 14

Table 4: Sampling Calculations for Post KAP Household Survey ............................ 17

Table 5: Summary Table for Instrument, Source and Limitations ............................. 19

Table 6: Evaluation Constraints, Limitations and Mitigation ..................................... 21

Table 7: Community Mobilisation Activities & Training ............................................. 36

Table 8: SLTS Training and Events ......................................................................... 36

Table 9: BCC Campaigns Outreach (Target versus Actual Number of People) ....... 37

Table 10: Supply Side Training Events for Masons and Entrepreneurs ................... 43

Table 11: Types of Low Cost Latrines ...................................................................... 44

Table 12: Number of Latrines Constructed (Target versus Actual) .......................... 45

Table 13: Outreach of Health & Hygiene Messages through Various Mediums ....... 50

Table 14: Capacity Building & Advocacy for Government ........................................ 53

Table 15: Villages Supported in attaining Total Sanitation Status ............................ 55

Table 16: Knowledge Management and Accountability Events and Activities .......... 57

Table 17: Cost Per ODF Declared Beneficiary ......................................................... 66

Table 18: Cost per ODF Declared and Verified Village ............................................ 67

Table 19: Evaluation Recommendations .................................................................. 88

[ix]

List of Figures Figure 1: SPSP Rural Districts and Agencies ............................................................. 4 Figure 2: Budgetary Trends for Water & Sanitation in Pakistan Source: Water Sector Report 2011 ............................................................................................................... 7 Figure 3: Evaluation Data Collection Methods ......................................................... 16 Figure 4: SPSP Contribution to Sanitation Target .................................................... 25 Figure 5: Province Based Access to Sanitation Facility ........................................... 29 Figure 6: Number of Sources of Health & Hygiene Messages ................................. 37

Figure 7: Health & Hygiene Awareness Messages .................................................. 38 Figure 8: Health and Hygiene Messages that the Respondents Remembered ........ 38 Figure 9: Respondents Able to Recall Three Key Hygiene Messages ..................... 38 Figure 10: Source of Health & Hygiene Messages in Schools ................................. 39 Figure 11: Topics Covered during Health & Hygiene Sessions in Schools .............. 39

Figure 12: Awareness of Hazards of Open Defecation ............................................ 40

Figure 13: Awareness of the Need for Achieving ODF Status .................................. 40 Figure 14: Proportion of Population Practicing Open Defecation ............................. 41

Figure 15: Willingness to Construct a Latrine (including methods of cost bearing) .. 41 Figure 16: WASH Facilities Constructed with External Support ............................... 46 Figure 17: Latrines Constructed By Respondent over Last Year and Type of Support Received .................................................................................................................. 46

Figure 18: Availability of Sanitation Supplies and Trained Masons .......................... 47 Figure 19: Population Using an Improved Drinking Water Source ........................... 47

Figure 20: School WASH Facilities .......................................................................... 48 Figure 21: Availability of WASH Facilities in Boys, Girls and Co-Education Schools 49 Figure 22: Household Water Treatment and Safe Water Storage ............................ 50

Figure 23: Proportion of Respondents who Treat Water .......................................... 50 Figure 24: Hand Washing Place & Soap Availability ................................................ 51

Figure 25: Critical Moments for Hand Washing ........................................................ 51

Figure 26: Knowledge of Water Related Diseases and Incidence of Diarrhea ......... 52

Figure 27: Government Expenditure on WASH ........................................................ 54 Figure 28: Households Using Underground Drainage .............................................. 55 Figure 29: Gender, Disability and Equity Based Access to Latrines ......................... 62 Figure 30: School Facilities Attributes ...................................................................... 75

[x]

List of Annexes Annex 1: Terms of Reference – Evaluation of SPSP Phase 1 Annex 2: Illustration of SPSP Programme Components Annex 3: SPSP Rural Results Framework Annex 4: Stakeholders & Roles Annex 5: Evaluation Matrix Annex 6: Key Indicators and Source of Information Annex 7: List of People Met for SPSP Evaluation 2014 Annex 8: List of Participants in Provincial Bottleneck Workshops Annex 9: List of Villages for Post KAP - SPSP Final Evaluation Annex 10: List of Documents Reviewed Annex 11: Evaluation Toolkit Qualitative Instruments Annex 12: Evaluation Toolkit Quantitative Instruments Annex 13: SPSP Outputs - Target versus Actual Annex 14: TORs for the Evaluation Reference Group Note: Consolidated version of all annexes is attached as Volume II.

[xi]

1. EXECUTIVE SUMMARY

Evaluation Object ‘Sanitation Programme at Scale in Pakistan - Rural’ (SPSP Rural)’ is a multiyear (2013-17) programme with first phase implemented through 2013-14. The programme targets rural parts and particularly the flood and security affected (insecure) areas. It builds on the experiences and learning drawn from UNICEF’s Post Floods Recovery (2010) Programme i.e. ‘Rural Sanitation in Flood Affected Districts’ (RuSFAD) implemented through 2010-12. The programme aims that; ‘By 2017, most vulnerable and marginalized children and women will be using safe drinking water, improved sanitation and hygiene practices’. The SPSP Phase I (the Year I) intended to reach out to 1.3 million people by implementing the Pakistan Approach to Total Sanitation (PATS)1. The PATS combines the School and Community Led Total Sanitation (SLTS & CLTS) approaches (referred to as Community Approaches to Total Sanitation (CATS) by UNICEF). The programme consists of six interlinked components, which include: i) Demand for sanitation; ii) Supply facilitation; iii) Health and hygiene promotion; iv) Linkages with duty bearers; v) Attaining total sanitation and; vi) Knowledge management. The programme design and delivery involved multiple stakeholders. UNICEF is the lead agency with contributions to programme design, resource mobilization, contracting the implementing partners (national and international non-profit organizations) and technical oversight. The field implementation has been carried out by non-profit partner agencies, and been referred to as ‘Implementing Partners (IPs)’. The programme has leveraged partnership with government agencies (mainly Public Health Engineering and Local Government Depts.) for some component. The field monitoring functions were performed by a ‘Third Party Monitors (TPM)’ in assistance to UNICEF. Moreover, the research and capacity building functions have been undertaken by the Development Solutions International (DSI - a consulting company) and Rural Support Programme Network. The programme activities have been carried out between February 2013 and March 2014. The Phase I had been rolled out with major funding of US $ 13.69 million, contributed by DFID UK. This phase was implemented in thirteen (13) districts and agencies i.e. 10 districts in Punjab, Sindh, Khyber Pakhtunkhwa, Balochistan and 3 Agencies from the Federally Administered Tribal Areas (FATA).

Evaluation Purpose, Objectives and Scope

The evaluation was commissioned by UNICEF Pakistan with the purpose to assess the programme achievements and to use the findings to inform the next phase. Objectives as stated in the TORs include, determining the relevance, effectiveness, efficiency and sustainability (as per UNEG standards), whilst also assessing the extent to which UNICEF core programme strategies and normative principles

1 Pakistan Approach to Total Sanitation (PATS) is a country specific model that evolved from Community Approaches to Total Sanitation (CATS), through a consultative process during 2008-09.

[xii]

(Human Rights Based Approach, Impact of Unforeseen Events and Financial Risk Management) were met. The evaluation has been carried for the activities undertaken in 13 districts and (FATA) agencies during Phase I (2013-14). The primary audiences or users of the evaluation include DFID, UNICEF, Implementing Partners (IPs) and government stakeholders. The secondary audiences include WASH sector partners and beneficiary communities.

Evaluation Methodology The evaluation methodology was evolved with a considered view of this being a formative evaluation and a clear focus on documenting key learning coupled with identifying areas for improvement. The intent was to have the next phase informed of the opportunities to improve both the design and delivery. The evaluators used mixed method approach for the evaluation. To map the extent of change for key quantitative output/result indicators, quasi-experimental research design was used to collect the information and hence draw comparison with baseline results. For this, a representative Post-KAP survey was administered in programme areas. The evaluation design however, excludes the creation of counterfactual; primarily for being formative evaluation rather an impact. The other methods used include secondary sources review, in-depth interviews, focus group discussions, bottleneck analysis (workshops) and observation checklists. For the evaluation design and implementation, the consultants followed UNICEF adapted United Nations Evaluation Guidelines (UNEG), accredited researching norms and ethics. An Evaluation Reference Group (ERG) with representatives from UNICEF, DFID, and government departments, which provided evaluation oversight and shared their own experiences with the evaluators. The report comprises of two volumes, one being the report itself and second includes all annexes.

Key Findings and Conclusions Relevance: The section carries findings and commentary on the level of coherence (consistency) of programme objectives and approaches to national and provincial policies and priorities, needs of men and women beneficiaries (including vulnerable) and total sanitation approaches. Also level of integration for UNICEF core strategies or crosscutting priorities i.e. gender equality, equity, and disability in programme design and implementation. By and large, the SPSP Phase 1 design was found coherent and consistent with the national policies and development commitments. The geographic prioritization seems appropriate, for the fact that most of regions picked up for programme implementation showed having lower sanitation coverage than the national average. The design carries evident departures from the conventional approaches like Community Approach to Total Sanitation (CATS), such as provision of subsidies (for latrine construction) following the ‘Disaster Response Approach’ within PATS which allows provision of Humanitarian assistance in post disaster/insecurity environment.

[xiii]

The design by and large, offers relevance to the UNICEF core strategies where it entails expressed focus on equity (subsidy to reach the extremely vulnerable and vulnerable families). Moreover, balancing gender consideration at all levels such as staffing, beneficiaries, forming men and women VSCs, hiring CRPs and in for some interventions positively discriminates to favour the marginalized like prioritising girls schools for rehabilitation of WASH services. In comparison to the equity and gender equability, the focus on disability was relatively less as evident from non-inclusive latrine design for rehabilitated WASH facilities (at schools and health outlets).

Effectiveness: The effectiveness of the programme has been assessed with respect to achieving outputs/results, actual application of the proposed theory of change. Each component has been assessed individually and all been assessed collectively also. Moreover, the section carries commentary on programme effectiveness with respect to implementing the integration of core strategies or cross cutting priorities i.e. gender equality, equity, & disability. The programme has performed well as the results indicate implementers being successful in achieving or in most exceeding the set (output) targets. The indirect implementation approach, where IPs were contracted, appears to have worked well to achieve targets in short time span. The limited engagement of public sector in programme delivery appears as a weak link. This has not helped much with preparing public agencies for PATS replication and scale-up. The partners and the partnerships for action research, third party monitoring and centralized training components, appear effective. This is evident form the fact that these partners been able to design and implement a parallel programme with different delivery approach, gather and document useful field learning (evaluators were told that the document is work in progress), facilitate timely information collection to inform management decision making and preparing and mentoring IPs staff respectively, to help achieve the set targets. The policy advocacy, for which UNICEF has been lead, has shown some positive results. These include selective roll-out of PATS in Punjab, WASH Master Plans, progress with respect to provincial sanitation policies, substantial sanitation budgetary increases for FATA and others. There was no evidence to suggest if it happened only for UNICEF advocacy efforts. There are multiple other WASH stakeholders involved in public advocacy, though with limited coordination and this could be attributed as a result of collective efforts. The strategies evolved for community mobilization Behaviour Change Communication (BCC) were found to be most effective in organizing, educating and sensitizing beneficiaries. For instance, 41% more people shared to have received health and hygiene messages than the baseline numbers. Similarly, proportions of those claiming to have adopted improved hygiene practices increased by 39% (compared to baseline). Receptiveness to the messages as well as ability to recall (including practice) has shown significant increase. In programme areas, 65% population now has access to sanitation facilities, significantly higher than the national average and two per cent short of the MDG sanitation targets.

[xiv]

Largely, the demand side activities found to be more effective than the supply side interventions. The programme may need to revisit the subsidy element (in supply side) to address equity and vulnerability. The parallel use of SLTS and CLTS helped in leveraging the complementarities and proved effective.

Efficiency: This section explores and shares evaluators’ views on how well the various activities have transformed the available resources into the intended results or outputs, including quantity, quality and timeliness. The programme implementation was found efficient especially where all (significant) performance targets have been met, despite delayed start. However, the fact that implementers were able to exceed most of the output indicators raises questions on whether realistic targets were set and corresponding resource calculations were appropriate. The section carries efficiency calculations mainly drawn through statistical analysis for cost per beneficiary and village for raising awareness and help achieving the ODF status. The analysis offers comparison with regional countries. Some inefficiencies were noted with contracting arrangements for hardware support in the programme i.e. rehabilitation of WASH facilities. The readers are advised to take caution while reading the efficiency analysis, as this has been carried out using budgeted figures, as consolidated expenditure statements were not available by the time of analysis.

Sustainability: The section carries findings and commentary around extent to which various stakeholders including the public sector and civil society are likely to sustain the positive outcomes of the project after the end of phase 1. The programme had envisaged using SLTS (school teachers and students) and CRPs as potential agents for sustaining the momentum. This however, found not to have been translated into an exit strategy or plan across sites and partners. The IPs found no or little time for post ODF monitoring and evaluators could take note of slippage signs. The linkages with duty bearers’ component could achieve little in terms of fostering ownership amongst public agencies to either takeover responsibility and/or scale-up the efforts. The community structures and mechanisms, such as CRPs and VSCs are likely to sustain but with limited success. The sustainability of sanitation mart owners as well as entrepreneurs created/promoted during the programme is also uncertain due to lack of access to loans/finance as well as demand issues. The evaluators are of the view that the SLTS component without continued support and engagement of district education offices is less likely to sustain either. The rehabilitated WASH facilities at schools and health outlets may not sustain without proper O&M mechanisms. The changes brought in the knowledge, attitudes and behaviours amongst men, women and children, are the most likely and significant results that may sustain.

[xv]

Gender Equality, Equity & Disability: This section carries commentary on how well the programme has implemented the integration of cross cutting priorities for gender equality, equity and disability. The design and PCA largely refer to balancing gender considerations in terms of boys/girls schools, men and women CRPs and VSCs, as well as IP field staff. The BCC campaigns found responsive to the information needs and preferred mediums of men, women and children. The programme also considered women and girls privacy and safety aspects (though the elevated WASH facility design has drawn criticism with respect to privacy). However, there were evident gaps around formal gender analysis before undertaking field interventions as to assess if interventions may not add to workload (burden) on the rural women and possibly children for additional requirement of water. This is significant in the context where gender roles place water fetching, cleaning of latrines as the domain of women and children. The programme demonstrated the equity focus by using PRA techniques to identify the vulnerable and extremely vulnerable and then extending household subsidy for 10% beneficiaries. However, the household subsidy component found to be challenging both in terms of design and implementation (creation of dependence in few cases and community friction at identification stage etc). It merits a serious review and revision. Addressing of disability, however, remained weaker both at community as well as school level in terms of sensitisation, infrastructure designs and sanitation supplies. Overall in terms of access to sanitation facilities, results of the post-KAP survey confirm an increase for women, as well as people in the lowest income quintile.

Unforeseen Events: PCAs in general seemed to lack any planning or comprehensive risk assessment of the programme areas and overall security environment. Fortunately, nothing significant happened (natural and human induced disaster of noticeable scale) during the course of implementation carrying potential to have had derailed the programme implementation. IPs knowledge of local communities and in most cases having local staff helped in ensuring smooth running of activities. However, given the environment within which the programme is implemented, there is a need to have a risk/disaster strategy in place with clear action plan, as part of the PCAs.

Partnership & Financial Risk: Largely, the partners’ selection was found to be appropriate with some exceptions (elaborated in detail in main report), However, TPM reports were able to identify operational and programme delivery challenges with the umbrella arrangement in Sindh. Financial risks were mitigated through some basic level capacity building. UNICEF did well to have conducted a baseline of partners’ capacities and developing plans, however it appears that this was not followed up to facilitate partners capacity development. Similarly, the programme seemed to have achieved limited results with building capacities of the relevant government departments. At provincial level, there was disconnect with duty bearers in terms of taking them on board, engaging them in planning to execution level activities and introducing rewards or incentives through the departments.

[xvi]

Lessons The key lessons learnt are outlined below; a) Access to Sanitation Facilities and Reduction in Violence Against Women. The evaluation has contributed to unravelling a critical link between open defecation and extreme cases of gender violence in rural Pakistan. A significant number of respondents shared that the availability of latrines inside the house, has significantly reduced the incidences of extreme gender violence such as Karo Kari (honour killing). To them, the construction of latrines have reduced women’s need to step out (as for defecation they were coming out at odd hours), thus the protection risk (or probability) of being accused of illicit relations (with men), and consequently being subjected to extreme violence. The lesson that is derived from this instance is that though an indirect result of WASH intervention, the benefit of contributing to a reduction in violence is a substantially important aspect and should inform the future programme design and delivery. b) Shame, Shock and Disgust versus Pride, Self-Respect and Dignity The two approaches generally are not used together for triggering sanitation, however, SPSP Rural I demonstrated that by using them intelligently (where shame, shock and disgust follows the other), better results could be achieved. c) Business Incentive and Opportunity Is a Larger Motive for Response. DSI’s action research suggests that manufacturers, entrepreneurs and mart owners (with business risk bearing attributes) see supply to the underserved areas as an opportunity to expand their business further. Given purely business incentives, they are more likely to sustain sanitation supply in order to meet the heightened demand and maintenance requirements in communities. Hence, it could be argued that the programme may benefit more, if it may focus on identifying and working with manufactures/entrepreneurs, with a clear intent to help them realise the real market potential and opportunity offered by these underserved areas for business continuation and expansion. This in the long run shall contribute to ensure technological innovation, continued and sustained supplies facilitation. d) Using Subsidy to Address Equity Can Lead to Dependency To a degree the subsidies, allowed in PATS in form of humanitarian assistance in post disaster/insecurity affected community, have served the equity purpose. However, the subsidy planning and delivery mechanisms have had some inherent challenges, which need deeper thinking and innovative solutions. The key problems that need to be addressed are;

a) Subsidies reducing the willingness of other poor families to put their own very limited resources into latrine facility construction, given the possibility of subsidy may become available to them, thus acting as a detrimental incentive.

b) By the end of the project, in few cases, those who have not received this subsidy, remain dependent and wishful for similar other interventions.

[xvii]

Main Recommendations Following are key recommendations across design, implementation and policy areas; 1) Future programming should provide adequate time for all stages of the

programme i.e. inception phase, PATS implementation phase and exit phase. This should include defining the duration it takes for a standard PATS implementation cycle.

2) The programme Theory of Change needs to be articulated with clear linkages, pre-requisites, risks and assumptions. Similarly a sustainability strategy, focusing on sustaining ODF status should be developed. PCAs should include specific IP exit plans, and stakeholders who will be ultimately responsible for sustainability should be clearly informed of their role along with their consent to delivery it (community as well as role expected from Government Departments).

3) Following aspects are recommended for the supply facilitation component, a) Sanitation Mart Owner/Entrepreneur Distinction between Sanitation Mart Owners and Entrepreneurs should be reassessed and if required dropped. The DSI action research points to the fact that market forces or business opportunity works well with sustained supply of materials, given, those in the business are made to realize that how CLTS and SLTS approaches can contribute to increasing demand for sanitation. This may need to be focused in the next cycles. b) Household Subsidy Modality and intended benefits of providing a household level subsidy should be reassessed, including possible detrimental effects on the community. Other mechanisms should be tested (such as providing collective incentives at the community level). For equity purposes and the needs of the vulnerable revolving credit/contributory fund can be considered as an option to be managed by the VSC.

4) The future programme design must clearly articulate and provide adequate implementation guidance for integration of gender equality, equity and vulnerability including addressing the needs of the disabled. UNICEF may need to take lead to provide guidance to the IPs as to how should these be incorporated into future PCAs. Also, provide adequate guidance for implementing those with the aim to achieve provision of inclusive WASH services.

5) UNICEF may need to provide greater guidance on engaging with public stakeholders (preferably at provincial levels) to create linkages, find sustainable exit (where responsibility to sustain the change shifted to relevant public authorities).

6) UNICEF may need to work more on harmonization of sector partner efforts and advocacy initiatives. This could be best delivered by providing strategic support to relevant public authorities (provincial level in particular) to help them realize the potential and leverage comparative strengths and resources of different WASH sector partners for shared goals.

7) In SPSP Phase II, UNICEF should consider piloting PATS implementation with public stakeholders in at least one province, receptive and willing to contribute to the idea.

[1]

2. OBJECT OF THE EVALUATION

2.1. Programme Overview ‘Sanitation Programme at Scale in Pakistan - Rural’ (SPSP)’ was launched by UNICEF in 2012. It is a multi-year national sanitation programme focusing on rural parts of flood and insecurity affected areas (2013-17). It builds on lessons and experiences of UNICEF’s ‘Rural Sanitation in Flood Affected Districts’ (RuSFAD) programme (2010-12). Purpose The purpose of the programme as stated in the project documents is; ‘By 2017, most vulnerable and marginalized children and women will be using safe drinking water, improved sanitation and hygiene practices’. The aim of the programme is to reach 1.3 million people by the end of year one or first phase2. The programme designed is built on the principles of Community Approaches to Total Sanitation (CATS), which with slight adaptations for Pakistan is referred to as Pakistan Approach to Total Sanitation (PATS). The PATS model advocates use of both School Led and Community Led Total Sanitation (SLTS & CLTS) approaches, and has been applied in parallel, for the programme under evaluation. Components and Indicators There are six components or pillars in the programme design. In the results matrix, these have been equated with outputs (with 18 output indicators) contributing to the programme result or outcome statement. The outcome has 3 indicators to be realized by 2017. The six components/outputs are as follows:

1. Demand for sanitation to achieve open defecation free status created in targeted communities

2. Sanitation demand sustained with supply side interventions 3. Improved Health and Hygiene behaviours promoted with the target

community 4. Linkages developed with duty bearers/public institutions to support PATS 5. Villages supported in attaining Total Sanitation status 6. PATS programme knowledge management and accountability Framework

implemented The outcome indicators as appear in the results framework are as below (result framework attached as Annex 3):

1. Proportion of the population using an improved drinking water source increases to 94% in rural areas.

2. Proportion of population practicing open defecation reduced to 35% in rural areas.

3. Government expenditure on WASH as a proportion of total government expenditure increases by 1%.

2. The terms SPSP Year 1 and SPSP Phase 1 have been used interchangeably, however both imply the implementation period February 2013 to March 2014. The evaluation covers the programme activities implemented through this period.

[2]

The overall programme outcome is to be achieved through phased implementation of yearlong or more implementation rounds. Hence, this is the evaluation of activities undertaken during year 1 (implemented during 2013-14) and excludes progress assessment on some of the impact/outcome level indicators. This remains a formative evaluation for UNICEF with the clear focus on learning documentation to inform the programme design and implementation for follow-up phases for the period of 2014-17. Stakeholders The programme design and delivery involved multiple stakeholders ranging from relevant public agencies to civil society organizations, research & monitoring organizations to leading capacity development networks (refer to Annex 4 for detailed breakdown of stakeholders and respective roles). Significant public agencies involved in the programme (first phase) include: Climate Change Division (CCD, formerly the Ministry of Climate Change) at the federal level, and public health engineering (PHED), health, education, and local government (LG) departments at provincial and district levels. The programme design offers a clear delineation of roles between stakeholders including coordination arrangements. For instance, the programme envisaged CCD to facilitate public sector coordination, oversight and provide policy guidance with respect to programme coherence with PATS principles and guidelines. The public stakeholders at provincial and district levels such as health, PHED, LG, education were expected to provide field implementation oversight including implementation of selective hardware/engineering components (identification of schools for schools sanitation interventions, planning and execution of water schemes rehabilitation), distribute and contribute to rewards and provide ODF certification. UNICEF contracted Development Solutions International (DSI) Pakistan, a marketing research and training enterprise to develop and implement (on pilot scale) a Sector Mechanism for Sanitation Marketing (SMSM). Moreover, DSI provided technical support for latrine designs. The Rural Support Programme Network (RSPN), a leading national network for technical advice and capacity development took lead in training and mentoring related capacity development of the Implementing Partners (IPs), Community Resource Persons (CRPs), government departments and officials on PRA, community mobilization methods, PATS/CATS approaches including IEC material. APEX, a consulting company was contracted for ‘Third Party Monitoring (also referred to as TPM) services. The role included developing a complete process and output driven M&E system, assisting UNICEF in undertaking field monitoring at all project locations and providing performance assessment reports for management decision making. Knowledge management was one of the key functions to be performed by TPM.

[3]

The non-profit organizations, contracted as implementing partners (IPs) took lead in field implementation3. These included national and international NGOs, with portfolio of WASH projects in designated geographic regions. UNICEF’s Role & Contribution The programme envisaged multiple roles for UNICEF mainly technical & operational oversight and backstopping, resource mobilization, stakeholders’ coordination, sub-contracting and contract management, advocacy and knowledge management. The following description elaborates on specific UNICEF contributions. Programmatic Redesign & Management of TPM: This included redesigning of the integrated sanitation model and programme implementation framework incorporating learning and recommendations from the RuSFAD programme. In addition to this UNICEF also contracted and managed an Independent third party monitoring system within the MoRES framework. Capacity Building: This involved building a cadre of PATS master trainers in all the provinces, while also liaising with and building capacity of government departments to implement and prioritize PATS. Policy Advocacy & Integration into Provincial WASH Master Plans: UNICEF provided policy oversight and contributed to decisions ensuring that the Provincial Government’s Master plans incorporated PATS programme implementation. Knowledge Management: Activities included preparing PATS training documentary, commissioning thematic issue papers, action research by DSI, Menstrual Hygiene Management (MHM) research and WASH Equity study, Development of PATS training manual for master trainers and guidance booklets for CRPs, SOs, School teachers, WASH clubs, Masons, entrepreneurs and Village Sanitation Committees (VSCs) Financial: This included capacity building of IPs and Government counterparts, as well as focusing on lowering third party independent monitoring costs to 3% of overall programme cost. Duration & Resources The evaluated programme is part of the UNICEF’s medium term rural sanitation programme planned for 2013-17. The phase evaluated commenced in February 2013 and activities were completed by 31 March 2014. The allocated budget for the first phase (2013-14) was US $ 13.69 million. Funds were provided by Department for International Development (DFID – UK) from two humanitarian funding streams; one for assisting flood affected areas while other targeted people in security (militancy) affected areas.

3 Muslim Aid, Qatar Charity, TVO (Trust for Voluntary Organization), BRSP (Baluchistan Rural Support Programme), WDCO (Women Development & Community Organization), NCBP (Nations Capacity Building Programme), NRSP (National Rural Support Programme), SAFWCO (Sindh Agricultural and Forestry Workers Coordinating Organization), Sabawoon (Sindh Agricultural and Forestry Workers Coordinating Organization), SEED (Social Efforts for Education & Development), IRSP (Integrated Regional Support Program), SRSP (Sarhad Rural Support Programme)

[4]

Implementation Status The activities for the first phase (2013-14) were completed on 31 March 2014. However, the government partners in KP province and FATA regions, mandated to restore/rehabilitate water schemes, were given a no-cost extension until end of July 2014 to facilitate complete utilization of funds. Coverage During Phase 1 of the programme, SPSP was implemented in 13 districts and agencies (10 districts in Punjab, Sindh, Khyber Pakhtunkhwa and 3 Federally Administered Tribal Area Agencies - FATA). Find below the details of programme coverage in terms of geographic spread with corresponding beneficiaries (intended) in different regions. Province/Area District/Agency Number of Beneficiaries

Flood Affected

Punjab Sindh Baluchistan

Dera Ghazi Khan, Rajanpur Jacobabad, Kashmore, Shikarpur Jaffarabad, Naseerabad

680,000

Security Affected

Khyber Pakhtunkhwa FATA

Kohat, Tank, DI Khan Kurram , Mohmand and Bajur

620,000

Table 1: Programme Coverage in Terms of Beneficiaries per Area/Province.

The following map (Figure 01) illustrates the geographic coverage of the programme.

Figure 1: SPSP Rural Districts and Agencies

[5]

2.2. Country Context Pakistan has an estimated population of close to 180 million, of which almost 1/3rd population lives below the poverty line. With a population growth rate of more than 2.9% a year, every year an additional 4.0 million people4 are added; needing additional clean water and sanitation facilities. This has long been established that lack of access to clean water and sanitation facilities carries wide variety of impacts. The data and evidence to establish the extent of the burden imposed on the people of Pakistan has been limited, which to a degree contributed to lower prioritization and consequently resource allocations to water and sanitation in public services delivery. Moreover factors such as insecurity leading to internal displacement, recurrent/annual floods and cyclic pattern of disasters have further aggravated both the access and quality issues. It is therefore not surprising that Pakistan lags behind its committed MDG (relating to sustainable access to safe drinking water and basic sanitation, MDG 7-C) target for sanitation outreach and access to 67% of population by 2015. The Joint Monitoring Programme (JMP) 2012 report indicates that only 48% people are using improved sanitation (72% urban and 34% rural), while 23% are still practicing open defecation. Moreover, poor sanitary conditions remain one of the major contributing factors to modest progress on key health indicators. Diarrhoea caused by these conditions is one of the leading causes of mortality for children under 5. Children also suffer disproportionately from sickness due to diarrheal diseases with almost 25 million cases (PSLM 2006-7) reported annually. UNICEF’s ‘Investing Wisely, Sanitation & Water, Saving Lives’ report5 indicates that between 60 to 75 million people are affected by diarrheal diseases annually, which costs the country 3.94% of gross domestic product (GDP) in terms of health expenditure. Another report (DFID 2013) states that, it is diarrhoea that is causing severe malnutrition (amongst 5% at least) and moderate malnutrition (amongst 20%) of the children under five, which places Pakistan behind Sub Saharan countries e.g. Ghana (DFID 2013). In addition to health concerns, lack of water and sanitation services is also associated with socio-economic disadvantages. A recent DFID study6 points out that evidently lack of availability of water, sanitation and latrine facilities can create gender gap at homes as well as significantly decrease girls’ attendance at schools. The social analysis of WASH illuminates gender and age disparities, limited community engagement in services delivery. For instance, within gendered domains, mostly women and sometimes children are tasked to fetch water from nearest water point. The RuSFAD programme evaluation raised concern around added workload for women for houses that constructed pour-flush latrines. Similarly, where open defecation is a norm, the standards for observing privacy are relatively strict for women. Hence, left with the option to go out in the dark, which again poses both physical and protection risks. Children and elderly people are often excused for age and mobility considerations. During field evaluation, some communities even

4 WSP: Economics Impacts of Inadequate Sanitation in Pakistan 5 Investing Wisely Sanitation and Water Saving Lives, Pakistan Sector Status Report 2012, UNICEF

6 Water, Sanitation and Hygiene, Evidence Paper, DFID 2013

[6]

Price for Poor Sanitation: 116,013 children (under five) die due to diarrhoea each year in Pakistan – approximately 13 deaths per hour - many of these deaths can be prevented by adequate sanitation, safe drinking water and improved hygiene (Pakistan Demographic Health Survey (PDHS) 2007-8). In Pakistan poor sanitation and hygiene accounts for an annual loss of PKR 343.7 billion. (Pakistan Sector Status Report, 2012). “Pakistan is one of the few fateful countries in the world that have witnessed large-scale outbreak of cholera and typhoid fever”. (Mukhtar et al as quoted in DFID 2013).

referred to men preferring open defecation despite having latrines at home, primarily due to the embarrassment they felt of sharing the same space with family women. In terms of involving users or communities in services such as design and management, very few examples are available where public agencies may have successfully engaged communities for design and management of water points/services. The policy initiatives undertaken in the past, have largely failed to translate into successful examples or models for limited technical capacities within public agencies for community mobilization, sustained engagement and unwilling users to contribute (financially) to the operations and maintenance of water services. Through the last decade, the prioritization of water and sanitation sector has led to steady increases in the budgetary allocations (refer Figure 2), a trend noted across all provinces. However, unfortunately increased financing has not translated into improved service delivery for multiple reasons. Some of those include under-utilization for delayed releases, budgetary cuts and diversions. The joint budgeting/allocation, as is practiced in most provinces, is skewed in favour of water in relation to sanitation, this is due to absence of need based resource allocation by the relevant government agencies, political influence and lack of credible data on district wise sanitation coverage. Moreover, the anecdotal accounts from the field indicate steady increase in latrine construction costs (on average between US$ 50-100), which in a way discourages people, more so the poor, to invest in constructing latrines. Those at risk to economic shocks (for natural disasters and insecurity) are also reluctant to invest in latrines. As for the groups who could afford, latrine construction rests at a much lower priority compared to food, clothing or entertainment. On a positive note, the last decade has seen some encouraging public policy shifts. For instance, formulation and approval of the National Sanitation Policy (2006) set the goal to reduce the number of people without sustainable access to improved sanitation by 2015, and provide improved sanitation to 100% population by 2025. The policy, while referring to strategic areas of intervention, mentions application of Community Based/Led Total Sanitation (CLTS) approaches to achieve desired results. Similarly Balochistan following on the line of the National Policy approved their Provincial Sanitation Policy in 2008. In order to contextualize communities' involvement in sanitation, a Pakistan-specific model of Community Approaches to Total Sanitation (CATS) was evolved, i.e., ‘Pakistan Approach to Total Sanitation (PATS), through a consultative process during 2008-09. The PATS, though an evolution of CLTS entails additional features than traditional CLTS approaches; for instance, introduction of sanitation marketing,

[7]

component sharing (public-private partnership) and others. PATS approach has a geographic focus on peri-urban and rural areas. It aims to end the practice of open defecation while transforming hygiene behaviour. PATS has also be endorsed in provincial sanitation policies by both Punjab and KP respectively. Another policy milestone achieved was the National Drinking Water Policy (NDWP 2009), which acknowledges water as a fundamental right of every citizen and aims ‘to improve the quality of peoples’ lives by reducing water-borne diseases with the provision of safe and sustainable drinking water by 2025 at affordable cost’. In last few years, the public sector spending on water and sanitation has seen a significant increase, which also reflects prioritization of water and sanitation on the public policy agenda (Pakistan Sector Status Report, 2012).

Figure 2: Budgetary Trends for Water & Sanitation in Pakistan Source: Water Sector Report 2011

The 18th Constitutional Amendment & 7th National Finance Commission Award (NFC) stand out as major developments, offering huge potential for devolved planning and management of civic services including water and sanitation. This shall give provinces greater access to resources and mandate to make their own specific policies and action plans. It offers the opportunity in terms of greater ownership at local level and drawing need based and context specific plans and interventions. However, the dividends of positive policy shifts are yet not visible on the ground in terms of improved service delivery. The dismal situation of sanitation and clean access to water (as highlighted above) was further exacerbated in Pakistan, due to the repeated floods during 2010, 2011 and 2012, which affected at least 20 million people in the country. Some have been affected repeatedly. According to the National Disaster Management Authority (NDMA) Pakistan, the heavy monsoon flooding in 2012, affected an estimated 4.8 million people in the provinces of Punjab, Baluchistan and Sindh. According to a Multi Cluster Rapid Needs Assessment (MIRA) an estimated 1.4 million children (of which 392,000 under five) were directly affected by the floods. These floods have resulted in contamination of wells and other sources of drinking water, further compromising the sanitation and hygiene environment. This has contributed to visible surge in water, sanitation and hygiene related diseases such as diarrhoea. Moreover, the insecure environment along the western borders and security operations in the Federally Administered Tribal Areas (FATA) has resulted in mass displacement across the region. Millions have been displaced since 2008. The authorities struggled to tend to the humanitarian needs of the displaced. The

[8]

displacement further strained the natural resource base and services delivery in host areas and communities. UNICEF conceptualised and rolled out SPSP to address the unmet sanitation needs in the rural parts of the insecurity and flood-affected areas.

2.3. Theory of Change The project documents shared for review did not include an explicitly defined ‘Theory of Change (TOC)’ for the programme7. However, as required the consultants in consultation with UNICEF and IPs have developed the TOC for the programme evaluated, derived from the SPSP flow chart of programme activities, results framework, details of project components and adaptation of the Global CATS TOC. The SPSP design is drawn on PATS framework with some adaptations. The SPSP design is drawn on a set of core assumptions that explain what changes should happen and why in order to lead to an ODF environment with improved health and hygiene practices and access to water. Moreover, there are assumptions, which explain how the envisaged changes happen and how they can be catalysed at each level (community, at local, provincial and national level). These constitute the building blocks of the overall envisaged/assumed change process. Based on the understanding of why, what and how changes may happen with respect to sanitation behaviours culminating in creation of Open Defecation Free (ODF) environment, the consultants have developed intervention logic. The logic includes strategies and activities that trigger the change process. The description below entails core assumptions and the two parts comprising a matrix and an illustration, which cumulatively explain the TOC. Core Assumptions The programme is based on the following core assumptions, with a logical flow between them from 1 to 5 (that is 2 logically follow 1, it does not represent hierarchy i.e. 1 is not more important than 2) describing the process of change: 1. Open defecation causes serious health and environmental concerns. It is an

individual and a collective community issue. Its practice should be abolished and a new social norm of ODF environment should be instituted.

2. The decision to end Open Defecation (OD) has to be collective, explicit and widespread, thus giving individual families the confidence that others are also abandoning the practice. When social pressure is transformed into collective commitment by the community to end the practice of open defecation, it can become self-sustaining. Once it reaches the tipping point, the change is more rapid and far-reaching.

3. Open defecation happens because of lack of awareness, resources and socio-cultural reasons. Even with awareness and resources, it may continue because of the socio-cultural acceptance within the community. This needs to be replaced and addressed through recognition of new behavioural expectations.

4. Due to socio-cultural and collective dimensions, the decision to end open defecation lies with the community. Ending open defecation is therefore a

7 UNICEF Pakistan WASH Security 2012 Proposal

UNICEF Pakistan WASH Flood 2012 Proposal

[9]

collective social change process led by informed and empowered communities that challenge and change pre-existing normative beliefs and attitudes.

5. A population wide shift to a new social norm of not continuing with open defecation requires an enabling environment. Firstly, a clear recognition of the hazards and consequences of open defecation to communities, who then consider it a practice not to be engaged in. Secondly, the enabling environment needs to provide for sustainability in terms of options (low cost options, drainage, supplies accessibility etc.) as well as connect with duty-bearers so that community is protected from future hazards of open defecation.

[10]

Activties

Outputs Outcomes Result (Impact)

Short Medium Long

- Formative research feeds into BCC .

- BCC campaigns launched. - SOs facilitate PATS in

communities and schools. - Wash Clubs formed and

strengthened. - Participatory Health and

Hygiene Promotion activities conducted.

- School WASH Facilities provided.

- CRPs trained and triggering CLTS.

- VSCs formed and their capacity built.

- Community Action Plans facilitated.

- School teachers and education officials trained on facilitating SLTS.

- Masons, Entrepreneurs, Sanitation Mart Owners Trained.

- Community informed on technical options.

- Demo Latrines installed - Water facilities rehabilitated and

built. - Linkages for accessing loans to

create household sanitation facilities.

- Capacity development and advocacy events with Duty Bearers.

- Advocacy for rewards by government for ODF communities.

- Advocacy with duty bearers to support Total Sanitation programmes.

- Mass Media BCC campaigns launched.

Demand for sanitation to achieve open defecation free status created in targeted communities.

Sanitation demand sustained with supply side interventions.

Improved Health and Hygiene behaviours promoted with the target community.

Linkages developed with duty bearers/public institutions to support PATS.

Villages supported in attaining Total Sanitation status.

PATS programme knowledge management and accountability Framework implemented.

At & Across Community Level

Community aware, mobilized and empowered to take action (through VSC), CRP & VSC formed, CAPs facilitated.

Community and School and engaged in BCC campaigns & attaining ODF status.

Low cost options, supplies and labour introduced.

ODF Certifications and Incentives awarded.

Demonstration of model ‘Total Sanitation’ villages.

Linkages with Duty Bearers.

Improvement in health condition and reduction in diarrheal diseases.

Safe water usage given households are treating water and storing safely.

At School Level

Health and Hygiene Promotion, formation of Wash Club, school taking responsibility to ensure ODF status in community.

Functional WASH facilities help maintain ODF status in school.

Schools Duty Bearers trained to take action.

Children, especially girls are regularly attending school.

At Provincial/District Level

Improved Provincial/District budget allocation to WASH specific basic social services.

PATS supported/implemented by the government.

Change in social norms. The decision to abandon open defecation has to be collective, explicit and widespread thus giving individual families the confidence that others are also abandoning the practice. When social pressure is transformed into collective commitment by the community to end the practice of open defecation it can become self-sustaining, once it reaches the tipping point, the change is more rapid and far-reaching and those who chose to practice OD are denounced. Social norm related to OD is addressed through recognition of new behavioural expectations as OD happens because of lack of awareness, resources and socio-cultural reasons. Even with awareness and resources it may continue because of the socio-cultural acceptance within the community. Due to socio-cultural and collective dimensions, the decision to end open defecation lies with the community. Ending open defecation is therefore a society level change in social norm led by informed and empowered communities.

A population wide shift to a new social norm of not practicing open defecation requires an enabling environment, firstly a clear recognition of the hazards and consequences of open defecation to communities, who then consider it a practice not to be engaged in. Secondly the enabling environment needs to be provided for sustainability in terms of options (low cost options, drainage, supplies accessibility etc.) as well as a connection with duty-bearers so that community is protected from future hazards of open defecation.

By 2017, most vulnerable and marginalized children and women are using safe drinking water, improved sanitation and hygiene practices

Open defecation causes serious health and environmental concerns. It is an individual as well as a collective community issue. Its practice is abolished and a new social norm of Open Defecation Free Environment is instituted. Access of targeted and neighbouring communities to safe drinking water, improved sanitation and hygienic practices, leading to conditions that do not cause diarrhoea and other water borne diseases. Sustainable ODF communities climbing up the sanitation ladder with duty bearers responding to infrastructure development needs through allocation of resources in an equitable manner.

SPSP Rural – Theory of Change – Part 1

[11]

Table 2: SPSP Theory of Change

Table 2: SPSP Theory of Change

SPSP Rural – Theory of Change – Part 2

Enabling Environment Assumptions and Risks

Changing Policy and Institutional Environment (18th Amendment, Local Government Elections, Provincial Policies and Programmes)

Proper Identification of Partner with capacity, outreach and ability to deliver with accountability.

Support at implementation level from the various departments of District and Sub-District Government.

Quality advice and facilitation, by UNICEF, IPs and involved experts.

Identification and effective use of natural leaders (whether in Government, Community (CRPs, VSC Member, Religious Clergy or Teachers etc.)

Proper sanctions (authority to work, NOCs, registration of organizations (IPs)) in place.

Key local community persons engaged through the process.

Availability of sound technical advice (UNICEF Senior Management, Subject Experts, 3rd Party Entities, Field level advice from IPs and input from Government (at provincial, district and sub-district level)).

Programme implementers with appropriate capacities to engage people in programme activities i.e. CRPs, Mason, Mart Owner, VSC Members, and Teachers etc.

Programmatically sound and locally viable interventions.

Effective financial controls and M&E arrangements in place (Financial management and reporting capacities of IPs, M&E arrangements for soft activities as well as technical activities).

Qualified staff and implementing partner’s identified/recruited/trained for the programme, having the requisite skills and knowledge.

Adequate availability and utilization of resources (both in terms of human resources as well as financial resources).

The programme implementation (its various phases community mobilization, triggering, BCC etc.) is not impeded by influential or other locals.

Community engages fully with the various programme interventions and stages.

Programme interventions are supported at collective as well as individual level, to ensure desired change occurs and necessary actions follow.

Programme based participatory techniques (whether through community, CRPs, VSC, government etc.) encourage and ensure individual and collective action.

Programme activities are not hindered by environmental and ecological constraints or issues, in which case alternate approach would have to be identified.

Resources for the programme remain accessible through the life time of the first phase, including financial as well as human resources.

Technical options and interventions are appropriate for the use and local context.

Security situation in few selected areas may delay achievement of results.

[12]

3. EVALUATION PURPOSE, OBJECTIVES AND SCOPE

Purpose This is formative evaluation with the purpose to assess programme design and delivery to inform the follow-up phases of SPSP. The evaluation is commissioned with the primary intent to accumulate learning, whereas accountability appears to be the secondary purpose. The evaluation TORs underlines assessment of programme achievements especially in terms of reaching out to the most vulnerable children and women. The assessment around choice to implement PATS with government and non-profit partners and integration of UNICEF programming priorities (normative principles) such as human rights and gender. The primary users or audiences of the evaluation include DFID, UNICEF, Implementing Partners (IPs) and government stakeholders. Each of these stakeholders has something to gain from the evaluation. For instance, the findings may facilitate UNICEF readjust the design and guide IPs with effective and efficient programme delivery. The government stakeholders may pick ideas of how PATS implementation could be adopted and up-scaled within public sector. WASH sector partners as secondary users could benefit in terms of picking up best practices for replication elsewhere. Objectives The evaluation TORs outline the specific objectives (refer to Annex 1 for complete TORs), which are reproduced below. These are:

1. Determining the relevance, effectiveness, efficiency and sustainability of UNICEF SPSP-Rural Phase I programme (as per UNEG standards).

2. Assessing the extent to which UNICEF core programme strategies were met. 3. Assessing how UNICEFs normative principles were reflected in the approach. 4. Identifying bottlenecks and success factors for programme implementation. 5. Commenting on the constraints and opportunities of programme

implementation. 6. Assessing effectiveness of the strategic partnership for process monitoring

and of the collaborative relationship for programme implementation (RSPN for Training).

7. Providing recommendations to strengthen PATS programme. 8. Assess the integration of UNICEF normative principles i.e. human rights

based approach, gender equity and environmental sustainability.

Scope The evaluation was commissioned by UNICEF Pakistan. The evaluation covered the SPSP implementation (activities undertaken) for the period of February 2013 – March 2014, hence could be referred to as SPSP Phase or Year 1 evaluation. As desired, the evaluation focused on determining the relevance, effectiveness, efficiency and sustainability of UNICEF SPSP-Rural Phase/Year I. The evaluation scope included assessment of all six key programme areas/outputs (explained in the introduction section) with more focus on areas where design had changed based on

[13]

learning from RuSFAD III evaluation (RuSFAD III preceded SPSP I roll-out and implemented in flood and polio affected regions). The evaluation scope included assessment of UNICEF’s choice for strategic partnership for process monitoring (TPM) including the collaborative relationship for programme implementation. Moreover, assessing the SPSP-Rural’ coherence (in design and implementation) to UNICEF core programme strategies i.e.

i) Capacity assessment and capacity development ii) Service delivery iii) Advocacy iv) Strategic partnership v) Knowledge management vi) Communication for development

The evaluation TORs prescribed the evaluation scope in terms of methods, which included:

i) Representative assessment of post KAPs (post KAP for communities and non-representative post KAP for schools) that helped assess the progress or extent of change on quantifiable indicators (thirteen outcome and output level indicators),

ii) Qualitative assessment of programme achievements and learning as per selected OECD-DAC criteria,

iii) Evaluation of capacity development of SPSP-rural non-governmental and governmental stakeholders,

iv) PATS related bottleneck analysis for each province. The geographic scope of the evaluation includes thirteen districts and agencies (where SPSP-Rural Phase I implemented) of Baluchistan, Punjab, Sindh, KP and FATA. The evaluators interacted with/consulted stakeholders at national, provincial district/Sub-district and community levels. Evaluation Framework The evaluation followed the UNICEF adapted United Nations Evaluation Guidelines (UNEG) norms, standards and ethical guidelines. The evaluation TORs outlined evaluation purpose, objectives, scope and criteria, broadly found consistent with OECD-DAC evaluation framework i.e. relevance, effectiveness, efficiency and sustainability. Assessment of impact was not a requirement given the short duration of project implementation during Phase 1 and the fact that the phase had recently ended. However, the evaluation included additional criteria including assessment of financial risks, integration of human rights and the impact of unforeseen events on programme delivery. Furthermore, evaluators assessed the gender equality and equity aspects, as crosscutting theme for the programme design and delivery across the other criteria. The element of financial risk assessment was added to evaluate UNICEF’s ability to enable IPs to demonstrate transparent and accountable management of programme funding. The aspect of human rights was assessed to give a measure of how design and delivery incorporated WASH as rights issue, given that Pakistan has ratified several international treaties, under which the provision of/access to safe drinking water and improved sanitation is guaranteed. It

[14]

was included to help assess the translation and application of HRBA approach in WASH services both by UNICEF and implementing partners8. Lastly, the impact of unforeseen events was assessed in view of repeated floods (in some areas) as well as the insecure environment. The evaluators are of the view that TORs could have been made more explicit in terms of defining evaluation questions and sub-questions. During inception phase of the evaluation, the consultants together with relevant UNICEF team members reviewed and revised the evaluation questions to enable more precision in terms of information needs for the evaluation. The evaluation matrix drawn describes the information needs, sources and methods for data collection (refer to Annex 5 – Evaluation Matrix for detail breakdown). The Annex 6 provides a complete list of key result indicators used for assessment of programme achievements (refer to Annex 6 – Key Indicators and Sources of Information). The following table outlines the core evaluation questions.

OECD-DAC Evaluation Criteria

Core Questions

Relevance To what extent the programme design and implementation was relevant to national/provincial priorities and needs of men and women beneficiaries in terms of achieving expected results across all socio-cultural groups including the vulnerable in the targeted communities?

Effectiveness To what extent were project implementation strategies effective and successful in achieving the planned outcomes/results for rich and poor men and women across all socio-cultural groups including the vulnerable in the targeted communities?

Efficiency To what extent have the outputs of the project been achieved (in terms of quality and quantity) with the allocated resources/inputs (such as funds, time, and procedures)?

Sustainability To what extent various stakeholders including the public sector and civil society are likely to sustain the behaviour change related to the goals of the programme after it has ended?

Note: Assessment of Impact was not required given the recent completion of project activities as well as the short duration of Phase 1.

Additional Evaluation Criteria

Core Questions

Unforeseen Events To what extent was the project implementation affected/influenced by unstable, insecurity or emergency situation?

Equity and Gender Equality

Though initially listed as separate criteria in the original TORs, these were included as cross cutting elements across the other evaluation criteria following UNICEF Team review and advice.

Human Rights Based Approach

To what extent did the project observe HRBA and/or contributed to observance of Human Rights in Pakistan as an implicit objective of the project?

Financial Risk Management

To what extent the project has observed and applied UNICEF’s standard financial management practices with Implementing Partners as well as with Government?

Table 3: Evaluation Criteria and Core Questions

8 These include the International Covenant on Economic, Social and Cultural Right (ICESCR), the Convention on the Elimination of All Forms of Discrimination against Women (CEDAW), and the Convention on the Rights of the Child (CRC).

[15]

The evaluation design took note of established research and evaluation principles and standards such as UNEG standards, norms and ethical guidelines. These standards have been complied with at varied levels such as design of evaluation framework, tools, training of staff, field data collection. For instance, requesting prior consent from respondents for research tools application, assuring data anonymity and others (more details in the methodology section).

4. METHODOLOGY

4.1. Evaluation Conceptual Framework The evaluation conceptual framework has been developed while keeping in view the formative nature of the evaluation, as outlined in the TORs. The evaluators used the quasi-experimental research design, where Pre and Post-KAP results have been compared to assess programme/approach effectiveness for quantitative result indicators. This was done with the view to assess effectiveness of PATS approach in terms of improved health and hygiene awareness, access to water and sanitation and use of improved/unimproved sanitation. However, keeping in view the formative nature of the evaluation, the quasi-experimental design was chosen with one exception i.e. exclusion of control group. Hence, the readers may not find any reference to creation of counterfactual for quantitative data analysis. The evaluation design is drawn on mixed method approach to leverage the complementarities of secondary and primary data. The primary information collection entailed use of both quantitative and qualitative methods. The quantitative information is gathered through representative household survey i.e. Post-KAP surveys and school sanitation surveys. The qualitative methods include in-depth interviews, focus group discussions, field observations and consultative workshops with key stakeholders. For data analysis, the evaluators used data triangulation techniques to further substantiate commentary and analysis. This facilitated in measured assessment of the extent and mapping the reasons for change. To understand the programme effectiveness in terms of inclusion/equity (as per UNICEF’s MoRES framework), the quantitative analysis offers useful segregation at income levels, gender and disability. In addition to gender and equity, it also examines vulnerability (including for the women, children, disabled and the elderly) and focuses on elements such as provision of subsidy and construction of demo latrines for the vulnerable, as well as on household and school wash facilities (taking into account the needs of the disabled). To map programmes contributions to larger context related changes (in comparison to the baseline), the evaluators have drawn province based comparative bottleneck analyses. For this consultative workshops have been organized at provincial level.

4.2. Data Collection Methods The data or information was gathered through both secondary and primary information sources. The secondary sources included project documents, national and regional research reports, published and unpublished data, web search and review of data-bases. For primary data collection, both quantitative and qualitative methods were used. The quantitative data was gathered through a representative

[16]

household level Post KAP survey. For assessment of school WASH facilities, a non-representative school sanitation survey was undertaken. The quantitative tools used for the SPSP Baseline were used for the evaluation, with minimal changes. The data entry was done on Census and Survey Processing System (CSPro), specialized software for data entry. The data was then exported to Statistical Package for Social Sciences (SPSS) and MS Excel, the software used for the data analysis. The qualitative tools included IDIs, FGDs, interactive workshops, field observations checklists, and photographs. The qualitative tools were applied to a wide variety of stakeholders (directly and indirectly involved in the programme) to get their reflections around programme design and delivery. The qualitative tools were used across all regions/provinces, where at least one site/community was picked up from each province/region. Although, there are no prescribed standards for sampling for application of qualitative tools, however, the evaluators are confident that the scale and breadth, at which quality tools have been applied, enabled to capture the depth and diversity of views in varied contexts. The evaluators have reached out to all key stakeholders (some even multiple times), which is quite reassuring. The figure below illustrates the range of research methods used for the evaluation.

Figure 3: Evaluation Data Collection Methods

4.2.1. Quantitative Evaluation Methods Post-KAP Household Survey Post-KAP survey was administered at the household level to collect representative quantitative data, which enabled to draw trends and patterns analysis with respect to changes in knowledge, attitudes and practices around water, sanitation, health and hygiene aspects. The analysis has been undertaken to capture trends for equity and gender aspects. The Post-KAP administered on 5185 households, where respondents were selected through multi-stage random sampling (refer Annex 9 for more details on sampling approach). Equal numbers of men and women respondents were interviewed in all districts, however in FATA only men were interviewed.

Evaluation

(UNEG, OEAD/DAC Criteria, UNICEF Core Strategies, Normative Principles)

(disaggregated by gender, income quintile and vulnerability status)

Quantitative Data

Post KAP Household

Survey

Post KAP School Survey

Qualitative Data

Secondary Data

Review

In-Depth Interviews

Focus Group

Discussions

Bottleneck Analysis

Workshops

School WASH FAcility

Observation

Checklists

[17]

Find below the sampling calculations, where detailed list of villages is attached as Annex 9.

Categories of

Respondents

No. of districts/

locations

Total Remarks

Male

13 + 3 (additional

FATA regions)

3189 In 3 FATA regions data

was collected only from

male respondents that’s

why male respondents

are calculated against

16 districts/locations and

female respondents

against 10 districts and

locations.

Female

13 – 3* = 10 (*FATA

regions)

1996

Total Sample 13 5,185

Table 4: Sampling Calculations for Post KAP Household Survey

Note: The above calculations were based on 13 districts/agencies considering, the overall margin of error for target sample as 1.35% at 95% confidence level.

Post-KAP School Survey Non-representative School Sanitation Survey was also administered in 78 schools, where at least 6 schools were selected (randomly) in each district/agency. Only those schools were selected, which had received assistance to rehabilitate the WASH services. The respondents comprised school head teachers or senior teachers. The inclusion criteria for schools included equal number of boys and girls (except FATA, where only boys schools were selected) schools at primary and secondary level. 4.2.2. Qualitative Evaluation Methods Secondary Data Review The evaluation involved extensive desk review of relevant documentation comprising Government Policies (PATS, NSP), UNICEF policies and programme documents (such as gender equality, HRBA, CATs, Child Friendly Schools etc.), sector reports, databases, TPM reports, national and regional survey reports etc. The extensive review helped understand the programme, larger environment in which it was implemented and the challenges. Also, it fed into the evaluation design especially tools development. The numeric data (secondary) was used to draw comparison with field survey findings and referenced as such (refer Annex 10 carrying details of the documents reviewed). In-Depth Interviews In-depth interviews (IDIs) were carried out with range of stakeholders primarily at national, provincial and district level public officials (MoCC, PHED, Health, Local Government, Education and others), managers of project partners (including UNICEF), non-profit organizations active in sanitation, political and community leaders, masons, third party monitors, mart owners and others. In total 36 IDIs were conducted where provincial and districts level public officials were covered multiple

[18]

times (refer Annex 7 – List of People Met for the SPSP Evaluation). The senior members of the evaluation team administered interviews to ensure consistency and quality. The instrument design took note of generating complementary information to help validate Post KAP results and triangulate information to deepen the analysis. The focus was to establish how far and in which ways the programme interventions brought changes in engaging public stakeholders and delivery of WASH services. The sample was evenly spread across the regions (across districts/agencies in each province/area) to capture regional disparities and reflect as such in the analysis and recommendations. Focus Group Discussions Like IDIs, the evaluators administered FGDs to draw complementary primary information to deepen commentary and analysis. Varied community or beneficiary groups comprising members of village sanitation committees (VSCs), members of school health club (children), community resource persons (CRPs), and School Management Committees (SMCs) participated in the FGDs. The discussions were carried out to understand the involvement, contributions and results of these efforts in changing knowledge, attitudes and practices around WASH. In total 40 FGDs were undertaken in five selected communities in as many districts/agencies i.e. one each in the provinces and an additional in the FATA agency. Please refer Annex 7, which carries the list of people who participated in the FGDs. Bottleneck Analysis Workshops The evaluators facilitated day-long bottleneck workshops, one each at provincial headquarters (four in total), in line with UNICEF MoRES framework to assess the extent change that SPSP I has been able to bring to the larger sanitation environment at provincial level. The participants included both public officials from relevant depts. and IPs from respective provinces. Find attached the reports that offer analysis as to the extent of change and causes in the larger sanitation sector (in relation to what transpired during baseline stage) including list of participants i.e. Annex 8.

School WASH Facility Observation Checklists To assess the quality and consistency with child friendly WASH facility standards, evaluators developed and administered WASH facility observation based checklists. The checklists were used to assess whether the facilities met the design specifications as provided in the manuals, and establish facility usability, standard of maintenance and likelihood to sustain/remain functional. The checklists were administered in selected schools.

[19]

Find below the table that offers summary of the methods/tools used, sources/respondents and limitations.

Instrument Source Limitation

Post-KAP Household Survey Men and Women at household level from target geographical locations

Absence of Control Group Some instances of Security Incidents and inaccessibility to areas, requiring a change in sample villages.

Post-KAP School Survey Teachers and School Children No Specific Limitation

Secondary Data Review Various Reports, Assessments and Policy Documents

Dependency on the existing body of literature, data and statistics. In some cases data desegregation is not available for comparison (geographically in the context of districts or in terms of gender).

In-Depth Interviews Government Departments (Health, Education, PHED, LGRDD at Provincial and District Level)

Availability of Government Officials in all project locations for interviews during the evaluation period. Frequent change of officials during the project life.

Focus Group Discussions Children (boys and girls), VSCs (Men and Women), Entrepreneurs and Mart Owners, CRPs (Men and Women)

Availability and sufficiency of numbers to allow for FGDs. In some instances due to limited attendance FGDs had to be managed as Interviews.

Bottleneck Analysis Workshops Public Sector Partners (Government Officials) drawn together at the provincial level.

Official deputed from relevant department may not have been involved during the project cycle.

School WASH Facility Observation Checklist

Actual visits to School WASH Facilities (Evaluation Team Observations)

Checklist is focused on assessing whether provided criteria and design specifications were observed or not. In some cases requires qualitative comments.

Table 5: Summary Table for Instrument, Source and Limitations

4.3. Evaluation Constraints, Limitations and Mitigation The evaluation went largely as per plan. As happens with most of the evaluations, the evaluators faced some constraints and challenges, for which appropriate remedial actions were adopted to not compromise quality. The key constraints and challenges are listed below together with mitigation measures undertaken by the evaluators.

[20]

Constraints/Limitations Mitigation Measures

Terms of Reference: Ambiguities around evaluation key questions and sub-questions.

Consultants sought clarifications and convened meetings with UNICEF team (including PMER section) to get clarity on key evaluation questions and sub-questions, which later were made part of the inception report.

Theory of Change: Shared documentation and programme proposals lacked an articulated Theory of Change.

The consultants in consultation with relevant UNICEF staff developed Theory of Change, and used as such for evaluation.

RBM Logic: The results framework found Inconsistent with RBM principles e.g. missing outcome statement.

This could partly be attributed to gaps in the donors’ results framework template; however evaluators designed comprehensive tools (post-KAP, IDIs and FGDs) to capture all results indicators desegregated across income quintiles, gender, and disability.

Creation of Counterfactual: The evaluation design excluded establishing counterfactual to assess the element of attribution.

The evaluation being formative in nature did not require establishing counterfactuals (as it mostly required for impact evaluations), hence excluded from the evaluation design. Moreover, it might have a challenge in itself to find a reliable control group, given the fact that communities (in project areas) are receiving humanitarian assistance from multiple stakeholders.

Security Situation Unstable security environment constrained accessing some project sites and caused delays.

A robust security management system established with senior resources on the ground taking lead in security assessment, liaison with local authorities and taking on the spot decision around withdrawal, rescheduling and changing field sites/villages.

Gender Context: Beliefs and cultural sensitivities especially in parts of FATA constrained access to women beneficiaries.

To overcome local sensitivities, locally recruited female enumerators and FGD facilitators were involved to administer household interviews and FGDs from female beneficiaries.. However, in view of the particular context in FATA agencies, the consultants in consultation with UNICEF agreed to interview male respondents only.

Methodological & Individual Biases: Methodological and individual (core team members and field staff) biases potentially affecting evaluation findings and analysis.

The evaluation took note to address this at all stage such as using both quantitative and qualitative methods, use of data triangulation techniques for analysis, instruments design with the intent to limit human biases, deploying experienced and trained resources for qualitative data collection. Moreover, extensive consultations within the core evaluation team, with UNICEF staff (including evaluation unit) and later the presentation to the Evaluation Reference Group (including the UNICEF Evaluation

[21]

Unit) helped keep continuous check to overcome methodological constraints and ward off the personal biases to keep into the analysis and recommendations.

Non-availability of Some Key Project Documents: Consultants did not receive the key project documents such as expenditure statements constraining desegregated cost benefit analysis.

The consolidated expenditures statements were not available hence evaluators used the budget figures gathered from PCAs.

Table 6: Evaluation Constraints, Limitations and Mitigation

[22]

4.4. Ethical Considerations The evaluators take note of and followed the UNICEF adapted United Nations Evaluation Guidelines (UNEG), standard research norms and ethics, for evaluation design, field data collection and reporting. Some of these include demonstration of impartiality, seeking prior consent from the respondents, ensuring data and informant anonymity. The evaluators sought prior permission to record conversations (digital recording) and for field photography. UNICEF focal points were kept informed of the progress and where required consulted during field data collection, analysis and reporting. The members of reference group were also kept informed mostly through UNICEF focal points.

4.5. Quality Assurance Mechanisms To ensure consistency to the accredited and agreed research and evaluation standards, the evaluators developed a set of quality assurance mechanisms for data collection, field work, data analysis and reporting. Some of those mechanisms are listed below; a. Where respondents agreed and allowed, the IDIs and FGDs were digitally

(audio) recorded and transcribed to avoid loss of field information;

b. To the extent possible, the evaluators hired the local youth/resources and

rigorously trained them before having them involved in field work. Female

local resources hired to overcome cultural sensitivities around interacting with

outsiders;

c. The field training of enumerators was conducted by senior evaluator, involved

in evaluation and instrument design. The training and contents and delivery

approach was standardized. Moreover, to ease work of field teams, field

guidelines were developed for field staff comprising District Coordinators,

Supervisors, Enumerators and Data Entry Operators;

d. The evaluators used the frequently used and reliable industry

software/packages for data entry (CSPro) and data analysis such as SPSS

and Excel;

e. 5% of the Post KAP forms were validated through field quality assurance

staff;

f. Distinguished sector professionals with complementary training and field

experience were engaged in the evaluation design, field data collection,

analysis and reporting;

g. Field application of IDIs and FGDs (the key quality tools) by senior evaluators

(some involved in design also) to ensure consistency and quality

administration;

h. Security protocols were developed and implemented throughout field data

collection. Regular contact was maintained within the team members, field

teams and local authorities (through respective IPs and UNICEF staff).

[23]

4.6. Evaluation Management and Stakeholder Participation

The evaluation was commissioned and managed by UNICEF Islamabad Office. The WASH Team was in the lead with technical oversight from the Evaluation Unit. The UNICEF Provincial Offices extended field level coordination and facilitated in organizing province based bottleneck analysis workshops. UNICEF teams at varied levels assisted in linking up the evaluation team with respective stakeholders (Government officials, IPs and community/beneficiary groups). A regular contact was maintained between UNICEF and evaluation team focal points. Significant time was taken during inception phase to clarify expectation, responsibilities, methodology, process and the output. The field activities were primarily facilitated by the IPs. This included scheduling interviews with local officials, FGDs with school children, VSCs, CRPs, SOs and School Teachers and application of school based observation checklist. By and large, the national and district level government officials made themselves available to discuss in detail and respond to evaluation questions comprehensively and candidly to help better inform the evaluators on their perspective of what worked well and what could be improved in the programme. At national level, UNICEF facilitated initial contacts whereas in the field IPs helped connecting to the local officials. Community was involved in various forms in the overall evaluation process, including making themselves available and giving time to the various evaluation processes and use of instruments (such as household members giving time and sharing their views and feedback in the household survey and similarly for children and school teachers during the School Survey). FGDs were organized with groups formed or supported during the project such as CRPS, VSCs, sanitation mart owners, entrepreneurs and masons. The IPs facilitated gathering community members at common spot (mostly guest room of local influential) to help evaluators interact with respondents. The female FGDs were largely performed inside the houses. It was encouraging to note that women and girls shared their views openly. An Evaluation Reference Group was formed (refer to Annex 14 for Evaluation Reference Group TORs) comprising representatives from UNICEF (WASH Team, Evaluation Unit), DFID, CCD and relevant provincial government departments. The evaluation team maintained regular contact and sought inputs both from the Reference Group and UNICEF’s Evaluation Management Committee (mandated for quality assurance) through PMER Section. A two days Reference Group consultation session was organized towards the end, where the evaluation team presented the key findings, analysis, conclusion and recommendations. Over the course of the consultation session, the recommendations were refined and translated into action plans for UNICEF to incorporate into the next cycle of SPSP programme. Third Party Monitors, Apex Consultants, on the advice of UNICEF monitored and reported on compliance to agreed standards. For instance, the TPM monitors

[24]

participated in the enumerators’ trainings, observed the implementation of the Post-KAP household and school surveys (by observing the conduct of field enumerators) and other field activities. The AAN consultants were responsible for evaluation design, field planning, organising field resources, training, logistics and security management. The firm was also responsible for conducting a professional evaluation, and producing high quality evaluation report.

[25]

5. FINDINGS

This section of the report offers key findings and analysis. The description is structured around key evaluation criteria and questions.

5.1. Relevance Evaluation Q1: To what extent the programme design and implementation was relevant to national/provincial priorities and needs of men and women beneficiaries in terms of achieving expected results across all socio-cultural groups including the vulnerable in the targeted communities? The evaluators have evaluated the relevance of the programme against a range of aspects including, relevance to international commitments, national and provincial policies and priorities, to the needs of men and women beneficiaries including the vulnerable, to total sanitation approaches and also to the integration of gender equality, equity and disability. Government’s International Commitments The programme objectives, selection of approach and geographic prioritization all point to relevance to Pakistan’s international commitments such as MDGs and SACOSAN. The relevant MDG sets the target to achieve 67% sanitation coverage by 2015, which at present stands at 50%. The programme was planned to increase sanitation coverage in the selected target villages and communities in rural parts of the country and successfully increased access to sanitation facility to 65% within the project areas thus also contributing to the overall MDG target (SPSP Evaluation Post KAP results - refer to Figure 4).

67%

50% 44%

65%

0%

10%

20%

30%

40%

50%

60%

70%

80%

MDG Target Current National Coverage

SPSP Baseline SPSP Evaluation

Proportion of Population having Access to Sanitation Facility

SPSP Contribution to Sanitation Target

Figure 4: SPSP Contribution to Sanitation Target

[26]

SPSP contributes to the following WASH related CCCs: Commitment 3: Children and women access toilets and washing facilities that are culturally appropriate, secure, sanitary, user-friendly and gender-appropriate; Commitment 4: Children and women receive critical WASH-related information to prevent child illness, especially diarrhoea; and Commitment 5: Children access safe water, sanitation and hygiene facilities in their learning environment and in child-friendly spaces.

Constitution of Pakistan does not recognise sanitation as a right yet, however, the national sanitation policy recognises that right to health cannot be achieved without sanitation. (http://www.washwatch.org/southern-asia/pakistan ) Following SACOSAN 4, Pakistan (including other SAARC Countries) adopted Colombo Declaration which commits governments to legislate and make the right to sanitation a part of the constitution.

The programme also conforms to the Kathmandu Declaration issued at the end of SACOSAN V, where Pakistan pledged to contribute to the ODF status for South Asia by 2023. The SPSP made a noticeable contribution by having successfully reached to 1.2 million people (men, women, boys and girls) with demand creation for sanitation messages and increasing the number of people living in ODF villages by 711,467. UNICEF Global and National Commitments Programme objectives also have close relevance to UNICEF’s Pakistan country programme commitments and to its global Core Commitments to Children (CCCs). The objective is to ensure children and women access to safe, sufficient and reliable WASH facilities. UNICEF’s work in Pakistan is guided by cooperation programme for 2013-2017, aimed at improving the lives of children and women in Pakistan. WASH is reflected as one of the priority areas for UNICEF and interventions are guided by UNICEF CCC and United Nation Development Assistance Framework i.e. UNDAF. Hence, it could be argued that SPSP I demonstrates coherence to organizational commitments and standards and larger UN mandate and priorities. National and Provincial Policies The programme has relevance to the National Sanitation Policy 2006 and draws on the Pakistan Approach to Total Sanitation (PATS) 2009. The programme design is found consistent with PATS approach, which is integral part of both national and provincial sanitation policies, even if available in draft form. The PATS approach places communities (beneficiaries) at the centre of driving and sustaining changed sanitation behaviours. The programme is strategic in terms of rolling out PATS and engaging the relevant public sector stakeholders selectively to work together with civil society to better understand PATS principles and components, and in the due course get prepared for complete ownership, replication and scale-up. Particular components and principles were found aligned to other relevant national level polices e.g. Environment Policy 2005, Drinking Water Policy 2009 and Climate Change Policy 2012.

[27]

In the wake of devolution of public functions and resources to provinces after 18th Constitutional Amendment and 7th National Finance Commission (NFC) respectively, provincial WASH policies and plans have either been approved or being finalized. Across all provinces, PATS remains the key policy priority for sanitation services, which again points to consistency with the evolving policies within provinces. Sector Coordination and Harmonization Activities The programme apparently has made limited success with sector coordination. UNICEF is recognized as key sector player and is involved in coordination, however that role transcends beyond programme. The focus remained largely on national level coordination with limited attention and resources for provincial and district levels sector coordination. For instance, UNICEF organized Sector Partners Workshop in June 2013, which brought the stakeholders together and event was used to showcase UNICEF work and findings of the RuSFAD III Evaluation and SPSP I Baseline. There are multiple thematic coordination forums, mostly at the national level where UNICEF is represented and participates regularly. UNICEF leverages its presence to share learning from SPSP and other programmes to inform sector discussions and policy decisions. The forums available are as under (some have become dormant also);

National Drinking Water & Sanitation Committee

National Sanitation Policy Implementation Committee

PACOSAN

CLTS Core Group under PATS (UNICEF is one of the members) As far as harmonization of efforts is concerned, neither federal government nor UNICEF is actively involved to facilitate harmonization of WASH stakeholders and activities. The situation is more or less the same in all provinces with exception of Punjab, where Planning & Development department has recently formed a Provincial Sanitation Committee to guide and coordinate the sanitation sector interventions. For programme interventions, the evaluators did not find evidence as to allocation of resources and active contributions by relevant district authorities such as contributions to rewards, though envisaged in the programme design. This may be the result of limited guidance by UNICEF provincial offices and lack of focus by IPs on mobilizing resources from concerned local officials. This may need a different delivery approach by IPs and considerate guidance and support from UNICEF provincial offices. The evaluators suggest that the programme roll-out in following years must prioritise and set guidance for formal coordination at national, and more importantly provincial and district levels. The relevant departments such as PHED or LGRDD (as varies in different provinces) must be given due support to take initiative for harmonization of efforts and resources of WASH stakeholders. The provincial level coordination and harmonization by lead public agencies may create enabling environment for IPs to work with district officials and cultivating greater sense of ownership amongst public agencies on ground.

[28]

CATS Essential Element 4: Subsidies – whether funds, hardware or other forms – should not be given directly to households. Community rewards, subsides and incentives are acceptable only where they encourage collective action in support of total sanitation and where they facilitate the sustainable use of sanitation facilities. CATS Essential Element 5: CATS support communities to determine for themselves what design and materials work best for sanitation infrastructure rather than imposing standards. External agencies provide guidance rather than regulation. Thus, households build toilets based on locally available materials using the skills of local technicians and artisans. UNICEF Field Notes – CATS

Total Sanitation Approaches The integration of SLTS and CLTS approaches where SLTS is seen to contribute to the continuity of WASH work in communities, found consistent with both PATS and CATS. The simultaneous application of SLTS and CLTS was found complementary and contributed to reinforcing the message. The provision of subsidy to poorest, though appear relevant on equity grounds, together with construction of demo-latrines, remain the major departures from the CATS/PATS essential elements. Moreover, the use of pride, respect and dignity as integral to triggering appear another departure from the conventional approaches. Overall, the use of SLTS and CLTS contributed to complementing project interventions and also reaching out to all family members, hence resulted in reinforcing the message at household level. FGDs and feedback from IPs highlights several challenges around provision of subsidies 3% and 7% subsidies to the vulnerable and extremely vulnerable respectively. Some relate to distinguishing between vulnerable and extremely vulnerable, which at few places caused frictions in the community. Some IPs referred to vulnerable being so poor (or in some case lack of interest or commitment) left the latrines incomplete (as IPs supported latrine construction except super-structure). Some even not used at all and most being unusable anymore for lack of cleaning. The future programme design may need to take more considered approach with this. Justification for use of subsidy was under ‘Disaster Response Approach’ within PATS which allows subsidy in form of humanitarian assistance in case of post disaster/insecurity affected communities. PATS includes human rights perspective in sanitation programming. Use of ‘Pride, Self-Respect and Dignity’ instead of conventional ‘Shame, Shock and Disgust’ approach remains another key departure. The evaluator found it relatively more consistent with rights based programming, which remains cardinal to UNICEF programme design. The post-KAP results point to greater dependency amongst respondents, no matter what economic strata they belong. This is evident from survey results where overall numbers for people willing to construct latrines have increased with a corresponding increase in people asking for external assistance/cost sharing. It is difficult to establish in concrete terms what may have contributed to this, however apparently it may have been for project subsidy/demo latrines and larger environment where non-profit organizations have been distributing free of cost latrines.

[29]

The consultants propose review of subsidy delivery mechanics with greater involvement of VSCs and communities to decide about subsidies and rewards. One option could be to transfer funds to these community forums to prioritize localized interventions, which may very well include latrine construction for poor and extremely poor households. Masons were trained, as part of supplies facilitation to help improve supply of trained and locally available masons. The evaluators noted departure from CATS application which emphasizes engagement of local skills (to support local livelihoods) whereas the programme required IPs to engage contractors (not necessarily local) for hardware components such as rehabilitation and construction of school and health facility latrines. Future programme design may need to take note to address this.

Community Priorities and Programme Area Selection The programme areas demonstrated greater deprivation in terms of poverty, insecurity and gaps with respect to sanitation coverage in comparison to urban areas. The programme areas found falling far below the national targets set for MDG Goal 7. Moreover, districts illustrated low scores on UNICEF’s Child- Wellbeing Index, which as per UNICEF contributed to their prioritization. In essence, programme coverage demonstrates coherence to larger programme goal/aim. The multiple information sources such as Joint Monitoring Programme (JMP 2012), Pakistan Social and Living Standards Measurement Survey (PSLM) and UNICEF Multiple Indicator Cluster Survey (MICS), all show the selected districts being low on WASH indicators. The Fig # 5, offers comparative analysis for MICS and SPSP pre-KAP (baseline) results, and illustrates the significant gaps that exist in terms of access to sanitation in urban and rural areas. The baseline results indicate these all largely being the poorest districts across Pakistan. The numbers indicate that by and large the programme prioritised districts, which had low sanitation figures including poor and security affected. Moreover, the interaction with both public officials and communities (varied groups) in programme districts point to sanitation being a priority need, particularly for women, children and poor.

The UNICEF team shared that as children focused organization, they selected districts based on child wellbeing index, which points to the larger deprivation of these districts.

96%

72% 77%

91% 93%

38%

64%

26% 29%

68%

33% 22%

50% 45%

61%

0%

20%

40%

60%

80%

100%

120%

Sindh Punjab FATA Balochistan KPK

MICS Urban MICS Rural SPSP Baseline

Figure 5: Province Based Access to Sanitation Facility

[30]

Gender Equality, Equity & Disability Integration The programme design and implementation demonstrate a clear focus on gender integration and targeting the most vulnerable i.e. equity. In comparison, the focus and the demonstrated integration for disability have been far less e.g. in BCC, latrine designs and others. The programme did demonstrate incorporation of gender equality at varied levels. These include gender being a key element in defining vulnerability, thus ensuring access to subsidized sanitation inputs. The women and girl centred BCC components supported by women staff and volunteers. Moreover, creation of men and women community groups, which demonstrates recognition that women are equally important to take this forward, as men. The collection of gender desegregated data during baseline, evaluation and during monitoring to track progress on gender equality. Women were part of the delivery apparatus, where field staff had healthy women representation. As described above, UNICEF justifies inclusion of subsidy (a departure from PATS and CATS) for equity consideration, to support and enable poorer segments to have access to sustainable sanitation. The poverty context in rural Pakistan and programme focus on rural poor, to a degree justifies the provision of subsidy, despite the fact that this at times creates frictions (in community) and element of dependency amongst communities (as results of post KAP illustrate) to seek outside assistance for construction of latrines. The programme design refers to activities being designed and implemented with due regard to addressing the needs of the disabled. The elements assessed such as BCC; triggering; and education and awareness were all found less disability informed. Similarly, neither prescribed latrine designs nor products available, appeared disability friendly. The design and delivery of future programme may need to demonstrate greater focus on disability integration particularly with respect to BCC campaign, latrine designs and products.

Theory of Change The programme documents did not have an articulated ‘Theory of Change (TOC)’. However, as part of the evaluation, a TOC was documented by holding repeated consultations with relevant UNICEF staff. The TOC found largely consistent with PATS and SPSP results framework.

The IPs and Third Party Monitors (TPM) shared that the programme documents may not have an expressed TOC; however, the essence of it was verbalized during the Inception Workshops while explaining the programme implementation plan. This helped to create a common understanding between UNICEF and the IPs on inputs, outputs, outcomes and possible challenges and risks. This to a degree helped delivery of programme in line with verbalized TOC, where some incidences of variations noted amongst IPs. Key variations noted around departure from the set sequence to roll-out BCC. UNICEF is advised to either incorporate the TOC in the programme concept note or emphasize incorporation of

[31]

TOC in future PCAs. This shall facilitate common understanding of results, approaches, interventions, inputs and risks and minimize implementation level variations, unless essentially required for context reasons. Overall, the programme design found largely relevant and consistent with national and international commitments and policy frameworks including those of UNICEF. The geographic prioritization found extremely relevant, as evident from the low progress vis a vis MDG 7 targets. By and large the design found consistent with PATS and CATS approaches, with evident departures from key conventions such as provision of subsidies, non-engagement of trained masons/skilled workers and others. Both the design and delivery merit re-consideration to achieve consistency with principles of CATS and PATS. The evaluators found an expressed focus on equity and gender equality, however with relatively less focus on disability integration. The programme documents lack an articulated TOC, which to a degree compensated by its verbalization during inception phase.

[32]

5.2. Effectiveness Evaluation Q2: To what extent were project implementation strategies effective and successful in achieving the planned outcomes/results for rich and poor men and women across all socio-cultural groups including the vulnerable in the targeted communities? The evaluators have assessed effectiveness of the programme across varied programme aspects, i.e. design, programme pillars/interventions and strategies (demand creation, supply facilitation, health and hygiene promotion, linkages with government, attainment of total sanitation, knowledge management and accountability) as well as effectiveness of integration of gender equality, equity & disability.

5.2.1. Programme Design The programme achievements as evident from comparative analysis of pre & post KAP results, point to effectiveness of PATS/CATS especially the parallel implementation of SLTS and CLTS. This helped leveraging complementarities and to a degree has planned to set foundations for continued community mobilization through schools. UNICEF used indirect implementation approach, whereby it contracted International and national non-profit organizations to lead implementation. Similarly, arrangements were made with relevant public agencies to work closely with these IPs. The implementation modalities did help with efficient delivery, however limited engagement of public agencies remains a concern (at both provincial and district levels). This may affect having willing and capacitated public sector for PATS replication and scale-up. UNICEF and other development partners such as WSP, WSSCC, WaterAid and others, are advocating PATS adoption at varied levels across the country. There are developments that point to the success of advocacy such as increased budgetary allocations for FATA, selective rollout of PATS in Punjab and others. UNICEF surfaces as a major player; however given the involvement of other stakeholders in advocacy attribution to UNICEF efforts alone is difficult. . The evaluators did not find reference to stakeholders leveraging their engagement and resources to piece together a collective advocacy plan to maximize impact. UNICEF may need to prioritize this for subsequent cycles of SPSP rural. UNICEF contracted specialized partners for action research, third party monitoring and centralized training, which largely proved effective. DSI representative, the partner contracted for action research shared that there are several key points that UNICEF can integrate in the next phase. For instance, broadening the scope of BCC materials by adding multi-dimensional focus e.g. emphasizing sanitation vis a vis health, privacy, religion and others. Moreover, DSI provided entrepreneurs with business and motivational training. APEX Consultants, the TPM worked as per the framework available providing real-time monitoring information to help UNICEF keep track of progress and make informed decisions. While centralised training by RSPN was effective in terms of standardization, deployment of dedicated teams for capacity building across different functions and layersthe RSPN staff and IPs both pointed to gaps in implementation mainly for overstretched RSPN staff. This may

[33]

have been because of under-estimation of human resources required on the part of RSPN. The training materials/manual produced will be available for future SPSP cycles. The programme design offers expressed focus on integration as to gender, equity and disability. For some elements, the inadequacies as to operational guidelines and standards (PCAs offer limited information) resulted in in-consistent implementation across sites and partners. For instance, a girl school visited did have elevated latrine including water points, where the students and teachers complained that as the boundary wall is low, they feel more exposed to people outside whenever they move up either to collect water or use latrines. This may have helped securing the WASH facilities from floods; however in absence of elevated walls, they feel that their privacy has been compromised. The future programme design and delivery may need to consider this and couple elevated WASH structures with raising boundary walls, particularly at girl schools. Similarly, for equity integration household subsidies were provided, however identification of vulnerable and extremely vulnerable individuals was done during the PRA exercises carried out before VSC formation, which caused friction at later stages. There are indications that the subsidy provisions are working to discourage communities (even the well-off) to construct latrines on their own, as evident from the survey results (shared above). For disability integration, the WASH manual (latrine designs) provides guidelines; however evaluators did not find much evidence suggesting the disability integration for facilities restored at schools, communities and health centres. The evaluators feel that ambitious targets were set especially with respect to time allocated. The partners were stretched further for delayed start, owing to PCA approvals and seeking field level authorization from district authorities. This left not more than 9 months (on average) for actual field implementation, making it even more ambitious. The field interaction with communities points to design and planning inadequacies around taking note of and adjusting activities to the community work cycle (e.g. harvesting activities, closure of schools during summer holidays), despite the fact that seasonal calendars were prepared through PRA. This contributed to IPs circumventing (to the extent of compromise) the laid processes to save time, especially capacity development, post ODF monitoring and others. This needs serious consideration from UNICEF to ensure that each IP gets at least 12 months of implementation time. For lack of clarity as to what it entails (from UNICEF) for the first programme component or pillar i.e. coordination with public stakeholders, the IPs used varied interpretations and implemented whole range of different activities. The evaluators did not get more clarity from UNICEF either as to what it entailed. If it meant more structured and systematic engagement (with respect to contributions to rewards, increased budgetary allocations etc.) with local and provincial public agencies, the results fall short of expectations. The evaluators noted gaps in communication chain (both at horizontal and vertical levels), again probably result of un-explained pillar. The provincial PHED which is the lead agency in Punjab also communicated that reporting needs to improve ensuring regular reporting and monitoring of field progress.

[34]

UNICEF appear to have consulted stakeholders particularly public agencies, however the extent to which these consultations contributed or informed programme design was difficult to ascertain. The other stakeholders like WSP and WaterAid did refer to being aware of the programme, however none referred to having been consulted for the design. UNICEF apparently guided and supported IPs for the design of PCAs, however how much have IPs contributed to the design is unknown. The evaluators view is that a programme like SPSP with a range of partners and of significant magnitude may need to evolve mechanisms to foster consultation and wider ownership by the stakeholders. This would surely have bearing on both the design and implementation.

5.2.2. Programme Management & Coordination Overall arrangements for programme management and coordination were found appropriate and relevant to the nature and scale the programme. The timely completion and achievement of results point to these being effective also. UNICEF WASH Unit based at Islamabad worked as the lead management and coordination point. Support was provided by the UNICEF Provincial Offices for coordination with public stakeholders, oversight of province level IPs, field monitoring and troubleshooting. Most of the decisions, in many instances were delayed for instance, delay in capacity building PCA resulting in a 3 month waiting period before RSPN could provide training to partners, delayed guidance on voucher system as well as delays caused by withdrawal/disengagement from loans and grants to entrepreneurs. The IPs shared that they were given sufficient flexibility, which they appreciated given the diversity of contexts. IPs referred to UNICEF being supportive through the implementation. Also, that they were able to communicate with confidence and freely, and received timely guidance and support with any issues or bottlenecks highlighted. For instance, an IP in Sindh shared that demo latrine cost revised upwards as were found to be under-budgeted in the original budget. The umbrella arrangements as used in Sindh added an extra layer with raised operating costs; however evaluators were unable to find a clear reference or evidence to the value addition of the umbrella partner. In absence of any significant value addition, UNICEF is advised to re-consider umbrella arrangements. More analysis around this is available in the partnerships section. The practice of monthly substantive monitoring reports helped keep track of progress, take note of deviations and constraints and take necessary management decisions for course correction. The evaluators were told of some of key challenges that programme faced, which are as below;

i) The TPM were contracted centrally by the Islamabad Office and relationship featuring direct reporting to WASH Unit Islamabad completely excluding the provincial UNICEF offices. The review of the monitoring framework (templates and tools available to TPM) indicates lesser flexibility and scope to highlight and incorporate the operational constraints. This IPs and UNICEF provincial offices referred to the process

[35]

being rigid and less sensitive to the field context they operated in. The UNICEF provincial offices raised concerns around being out of reporting loop for TPM, which UNICEF Islamabad office may need to look into. The consolidated versions could be shared with provincial offices for information purposes.

ii) UNICEF (both at Islamabad and provincial offices) could have played more effective role with public sector advocacy, had more structured process followed for public sector coordination at varied levels. The limited success with budgetary (provincial) increases, contributions by public sector into rewards and others, demonstrate UNICEF and in particular provincial offices may need to take more hands on approach for public sector coordination. This impacted IPs engagement with district level public stakeholders and level of ownership of the programme by PHED and LGRDD in districts.

iii) For lack of systematic and structured coordination, the approvals for implementation by the district authorities got delayed invariably for all partners. This delayed the project kick-off thus stretched the IPs resources further to deliver on committed results. One such example is the NOC for Muslim Aid in DG Khan, which was never resolved and left the project exposed to a range of risks. Moreover, the relationship and terms of engagement (including the focal agencies) between IPs and district governments kept on evolving. Some IPs were lucky to find receptive focal points, thus were able to mobilize public sector support, whereas for others it was struggle all along.

5.2.3. Programme Components & Results Note: The results framework lacked outcomes statements and indicators, hence evaluators used outputs indicators (which in strict results term could be equated to outcomes) to map the extent of change the programme implementation brought (by comparing pre and post KAP results).

Output 1: Demand for Sanitation SPSP used both CLTS and SLTS approaches, which helped to sensitize and educate communities and thus resulted in increased demand. The results are indicative of the approaches complementing each other, which was supported further by the cyclic BCC campaigns. The objective to reduce practice of open defecation, raise community’s awareness of the need to achieve ODF status in their villages and to create demand for sanitation facilities by the target population, all showed encouraging upward change towards the end of the year (end of Phase 1). Community Mobilisation The decision to engage RSPN (to help train SOs, CRPs etc.) and draw on RSPs experiences and approaches for community mobilization (in use for over three decades) was well-considered. However several aspects including sufficiency of training team in terms of numbers, as well as better scheduling could have led to greater effectiveness in mobilisation.

[36]

The results in Table 7 points to IPs exceeding targets mostly for community mobilization and sensitization related indicators.

Community Mobilization Activities & Training Target Actual

SOs Training for PATs (participatory tools, IEC material) 8 85

CRP Training on Conducting CLTS 46 67

PATS Facilitation Session by SOs in Villages 230 911

CRPs facilitation session on CLTS 1,385 3,471

VSCs Formed and Trained 1,385 2,994

Community Action Plan (CAP) Developed 1,385 1,878

Table 7: Community Mobilisation Activities & Training

Generally the feedback has been positive on the quality, duration and content of the RSPN Social Organisers (SOs) training. The SOs shared that the training helped them get prepared for the expected work, however highlighted that one extra day for field work (practical demonstration) may have added more value. Moreover, they shared that the scale and frequency (considered excessive) of mentoring activities and refreshers (to be undertaken by RSPN) affected SOs mobilising schedule. Most of the SOs were found satisfied with the workload, however for delayed start, they were left with no option but to execute their field plan, even if at times it clashed with community activity cycle (rural). The availability of both men and women SOs were appreciated to help overcome the gender divide. Again, the programme performance found encouraging where the IPs exceeded almost on all targets (refer Table 8).

School Led Total Sanitation Training and Events Target Actual School Teacher Training 23 84

WASH Clubs Formed in Schools 692 881

Events Conducted by School Wash Clubs 886 1,433

Boys and Girls reached with sanitation messages in Schools. 103,000 176,022

Table 8: SLTS Training and Events

Behaviour Change Communication (BCC) The post KAP results and interaction with stakeholders involved including communities point to the effectiveness of BCC strategies and interventions. The key factors that appear contributing to the success are i) use of a variety of mediums targeting varied groups (IEC material, theatres & mass media) and; ii) repetitive rounds of multiple activities. They helped targeting different intended population groups with varied messages and mediums (matching the information preferences and interests) and in a way re-assured that message gets across to everyone at least once. The involvement of opinion leaders such as Imam Masjid (prayer leaders through Friday sermons), health workers, and teachers further strengthened the communication. Refer table 9 for details. The reports indicate IPs being able to meet the targets for most of the activities.

[37]

BCC Campaigns Outreach Target Actual

Children (boys and girls) reached with demand creation for sanitation messages in Schools. 103,000 176,022

People (men, women, boys, girls) reached with demand creation for sanitation messages. 776,000 1,200,529

People reached with demand creation for sanitation messages through electronic/print media. 970,000 540,786

People reached with demand creation for sanitation messages through community events (theatres, puppet shows). 291,000 195,450

Table 9: BCC Campaigns Outreach (Target versus Actual Number of People)

The effectiveness of BCC strategy is evident from the number of people claiming to have received the health and hygiene messages in last one year. The KAP results indicate that this has increased from 18% (during baseline) to 59% (pre-post KAP results). This shows that within the span of 8-10 months, the number have risen from 1 to 3 (for every five) people having received at least one health and hygiene message. The effectiveness of the mixed medium and repetitive BCC campaigns contributed to a significant proportion of respondents claiming to have received information from one or more sources (36%, 37% and 27% claimed to have received information from one, two and three sources respectively). It means that those who received messages, 2 out of three had received from two or more sources (refer Figure 7).

The awareness and knowledge factor (in terms of being able to remember and recall messages) has also been effectively improved from 71% to 98%. The most significant is the change in behaviours, where 92% respondents claimed that they have adopted the practice (following receipt of message) compared to 53%, at baseline (refer to Figure 7).

One Source, 36 Two Sources, 37

Three Sources, 27

Number of Sources of Health & Hygiene Messages

One Source

Two Sources

Three Sources

Figure 6: Number of Sources of Health & Hygiene Messages

[38]

Some elements of the BCC campaigns have been appreciated and being practiced more than others e.g. 10 Steps of Hand Washing. The Figure 8 illustrates that 93% respondents were able to remember/recall health and hygiene specific messages compared to 67%. Similarly 66% and 40% (from 56% and 24% respectively) of the respondents were able to remember/recall messages about significance of clean and safe water and water borne diseases. .

The number of people who could recall three key hygiene messages doubled over the course of year (refer Figure 9). The Punjab province is in the lead followed by Sindh and Baluchistan.

18%

71% 53% 59%

98% 92%

0%

20%

40%

60%

80%

100%

Health & Hygiene Message Received

Remember H&H Messages Practice H&H Messages

Health & Hygiene Awareness Messages

Figure 7: Health & Hygiene Awareness Messages

67% 56%

24%

93%

66%

40%

0%

50%

100%

Personal Hygiene Clean And Safe Water Water Born Diseases

Health & Hygiene Messages that the Respondents Remembered

Health and Hygiene Messages that the Respondents Remembered

Baseline Evaluation

Figure 8: Health and Hygiene Messages that the Respondents Remembered

24% 43%

23% 26% 23% 27%

80% 64%

48% 46% 33%

51%

0%

50%

100%

Punjab Balochistan Sindh KPK FATA Total

Proportion of Population Recalling Three Key Hygiene Messages

Baseline Evaluation

Figure 9: Respondents Able to Recall Three Key Hygiene Messages

[39]

The BCC strategy has also been effective in delivering health and hygiene messages at schools, as noted in Figure 10. 77% schools confirmed having received health and hygiene sessions over the last six months, whereas 41% schools reported to have had sessions conducted in-house by the school teachers. Gender focus in terms of coverage of boys and girls schools also comes across as sufficiently balanced.

Health and hygiene topics covered through school sessions, where message around personal hygiene were appealed the most. This is evident from the fact that 73% school teachers were able to recalling delivery of this message, followed by environmental and domestic hygiene i.e. 69%.

In terms of effectiveness of BCC mediums, face to face community sessions by the SOs and the CRPs were referred to as being most effective in terms transfer of knowledge. The respondents shared that these were more useful as it provided an opportunity for them to interact with the facilitators, ask questions and get response for their queries. It was also noted by a range of stakeholders (SOs, children, VSC members and CRPs) that theatres, cricket matches, key events celebrations (including world water day, environment day and hand washing day) and use of mass media, was also effective though with varying degree of results and responsiveness. These elements need to be kept and further strengthen in the subsequent years.

77%

41% 47% 27% 27%

59%

9% 31%

0% 20% 40% 60% 80%

100%

Imp

lem

enti

ng

Par

tner

Sch

oo

ls

Bo

ys

Co

Ed

uca

tio

n

Gir

ls

Bo

ys

Co

Ed

uca

tio

n

Gir

ls

Implementing Partner School

Source of Health and Hygiene Messages in Schools

Figure 10: Source of Health & Hygiene Messages in Schools

73%

58%

36%

69%

42%

42%

0% 10% 20% 30% 40% 50% 60% 70% 80%

Personal Hygiene

Clean & Safe Water

Water Born Diseases

Environmental & Domestic Hygiene

Safe Disposal of Human Excreta

Solid Waste Disposal

Figure 11: Topics Covered during Health & Hygiene Sessions in Schools

[40]

The BCC campaigns proved effective in terms of making people understand and recall hazards of open defecation. Similarly, a significant increase is noted in terms of better awareness as to the types of hazards associated with open defecation. Figure 12 highlights the proportion of post-KAP respondents, who remembered and were able to recall various hazards related to open defecation.

The extent of change in terms of respondents being able to realize and communicate their desire to make their settlement/community ODF, points the success of BCC. Results indicate significant increase across all provinces (refer Figure 13).

Another measure of success is the overall decrease in number of people defecating in open, refer Figure 14. The numbers have come down from 68% to 47% overall, with Punjab and Sindh showing the most significant changes. The results are negative for FATA, which are due to limited data collection at baseline stage and need to be probed further. Limited data collection in Mohmand Agency of FATA was due to security reasons.

14%

34%

54%

24% 10%

54% 62%

77%

47%

18%

0%

20%

40%

60%

80%

100%

Awareness Message about

Hazards of Open Defecation

Dirty Surroundings Diseases Environmental Problems

Health Issues Related To Women

Hazards of Open Defecation

Awareness of Hazards of Open Defecation

Baseline Evaluation

Figure 12: Awareness of Hazards of Open Defecation

14% 14% 24% 25% 1% 10%

61% 73% 93% 84%

32% 39%

0%

50%

100%

Total Punjab Balochistan Sindh FATA KPK

% of targeted population aware of the need for achieving ODF status in their settlement

Proportion of population aware of the need for achieving ODF status in their settlement

Baseline Evaluation

Figure 13: Awareness of the Need for Achieving ODF Status

[41]

Another measure of success comes in the form of increase in numbers for respondents willing to construct a latrine due to the hazards associated with open defecation (refer Figure 15). The respondents willing to construct a latrine have increased from 61% (baseline) to 86%. Majority of those referred to cost-sharing arrangements as preferred option to construct latrines, whereas the numbers for self-construction have reduced from 18% to 10%.

As highlighted above, this trend could be for subsidy factor or for the provision of free latrines by other stakeholders (particularly after floods). However, this merits further exploration as to whether household subsidy is playing a positive role in increasing the sanitation coverage or creating a dependency factor. UNICEF is advised to explore alternative methods for incorporation of equity principles especially in view of the excessive poverty in rural Pakistan. The BCC campaign faced some constraints particularly around short time-frame (for implementation), timely production and delivery of IEC materials and sequenced implementation of BCC campaigns. The IPs underlined that delayed delivery of IEC

83%

56%

85%

47%

70% 68%

44% 39% 35% 34%

76%

47%

0%

20%

40%

60%

80%

100%

Punjab Balochistan Sindh KPK FATA Total

Proportion of Population that Practices Open Defecation

Baseline Evaluation

Figure 14: Proportion of Population Practicing Open Defecation

61% 28% 25% 23%

86%

10% 20%

62%

0% 20% 40% 60% 80%

100%

Yes Self-finance Loan Cost-Share Basis

Willing to Construct a Latrine Due to Hazards

of Open Defecation

Method of Bearing the Cost of Construction

Willingness to Construct a Latrine (Including Methods of Cost Bearing)

Baseline Evaluation

Figure 15: Willingness to Construct a Latrine (including methods of cost bearing)

[42]

materials shortened the time for complete BCC campaigning. This pushed to have additional per week time engagement with the communities (also affected by community’s unavailability during the harvesting season), and accelerated delivery to a degree lead to information overload and reduced absorption time for beneficiaries. Within the mediums used, theatres performances though liked by children and adult men, excluded women as in most cases these were organized at cluster (village) level, where participation required considerable travel – a constraining factor for women for limited mobility. The gender exclusion also constrained mixed gathering for theatres. The next phase may need to consider easier access and if possible more privacy for women (may also consider women exclusive theatre performances). The engagement of religious leaders and institutions proved useful in adding the religious element that appealed to the religious groups, a significant majority in the rural Pakistan. School health and hygiene sessions (plays, puppet shows and others) were found effective in terms of packaging messages in child friendly manner. The use of mass media was relatively less, which may need to be looked into for future.

Output 2: Supply Side Interventions Overall supply side interventions remained effective in terms of meeting the increased demand for supplies. Some of the interventions could have proven more effective given greater clarity and guidance such as involvement of trained masons for hardware (project supported) components, loans and grants for entrepreneurs and marts. The range of supply facilitation interventions included training of masons (for better design and low-cost sanitation solutions) training of entrepreneurs and sanitation mart owners (to encourage business activity and ensure sustained and subsidized supply of materials near community), and construction of demonstration latrines for both demonstrative and equity purposes (by providing subsidy or full costs for vulnerable and extremely vulnerable families). Moreover, it included rehabilitation of child friendly WASH facilities at selected schools and at health service centres. Training of Entrepreneurs, Sanitation Mart Owners and Masons The RSPN developed training contents and trained entrepreneurs and sanitation mart owners, through Master Trainers. The IPs and trainees both point to ineffectiveness of the training, mainly for weak training contents and limited facilitation capacities of Master Trainers - lack of understanding of marketing and supply chain issues. Realizing the gap, UNICEF contracted DSI to develop contents and train the trainees through refreshers (refer results in Table 10). These were apparently more effective in articulating expectations. The key learning from the experience include a full day training and training/sensitization of relevant IP staff to enable productive engagement (post training) with the entrepreneurs and marts.

[43]

In Kohat a local mart owner said that he was dealing in cement, sand and bricks. After the entrepreneurs training, he stocked supplies and was able to sell items for approximately 150 latrines through the voucher system. From this he was able to purchase more latrine supplies and sustain the sanitation business. However, he was of the view that availability of loans/grants would have been helpful, given village level vendors are generally cash-strapped and capital to invest in new products is not available. He also felt that IPs should have focused more on creating demand and publicising their sanitation business to the communities.

Training Events for Mason and Entrepreneurs Actual Target No. of masons trained on sanitation construction and using their learnt skills

692 1,250

No. of entrepreneurs/manufacturer trained and producing low-cost sanitation solutions 692 469

No. of facilitated sanitation marts doing profitable business 346 82

Table 10: Supply Side Training Events for Masons and Entrepreneurs

DSI also established a business helpline for entrepreneurs and sanitation mart owners for all 13 SPSP programme districts as part of active research and support. The helpline was launched to provide the entrepreneurs with the opportunity to seek business related advice from DSI SanMark expert and engineers. The records indicate that maximum numbers of calls were received in October and November 2013, mostly regarding the withdrawal of loan / grant by UNICEF and IPs, financing options, and supply and pricing issues. This service was considered by entrepreneurs and sanitation mart owners to have been effective in helping them with teething problems of setting up the marts. Masons trainings proved relatively less effective for largely being informed of the training contents, before taking up the sessions (as DSI research). The programme may have benefitted more in case it had employed the trained masons to rehabilitate the school and health WASH infrastructure. Financial Support for Entrepreneurs and Sanitation Marts The programme envisaged supporting entrepreneurs and marts with loans, which was not appreciated much. More interest was shown in grants, which ran against UNICEF rules prohibiting use of funds for setting up for profit businesses. In view of the resistance, the loan element was withdrawn, which impacted negatively given expectations had been raised that loan/grant facilities may be available. As the component was withdrawn during implementation, where some Provinces were found to have had advanced into implementation. For those, UNICEF created modality of vouchers and distributed to the most vulnerable/vulnerable groups to get supplies from the supported marts and entrepreneurs. This to a degree proved useful in terms of localized availability of supplies and at subsidized rates. With programme closure the voucher system has stopped, and this shall have affect on their businesses. The component needs deeper analysis and more innovative approach to facilitate both the users and the business involved. UNICEF could leverage the findings of DSI action research.

[44]

Design Options and Manuals for Low-Cost Sanitation Facilities UNICEF engaged DSI to help develop low cost latrine designs. DSI is a consulting company (for profit), contracted by UNICEF for the work. DSI used Human Centred Design (HCD) process to developed latrine design, which are as below (refer to Table 11) Type Description Cost Factor (maximum)

Latrine Type A Ventilated Brick-Lined Direct Pit Latrine

PKR 4,000

Latrine Type B Pour Flush Off-Set Lined Pit Latrine

PKR 7,000

Latrine Type C Pour Flush Latrine With Septic Tank

PKR 10,000

Table 11: Types of Low Cost Latrines

For field interventions, another manual i.e. Latrine Construction Manual, was developed for sub-engineers, masons and local facilitators in local (Urdu) language. This manual was used for field training and for project supported hardware interventions. The document was found comprehensive and incorporated disability and age considerations. The evaluators did technical assessment of the manual and deeply looked at constructed latrines, which are outlined below as key observations/comments.

The latrine construction manual is comprehensive, however it be more useful if additional information regarding latrine super-structure options as well as low cost options for pit lining (using local materials) could be added. Translation in Urdu contributed to improved delivery and understanding of trainees.

DSI’s report lacks design recommendations related to disabled/elderly persons’ needs. The manual however covers these elements comprehensively. It appears that the rehabilitation work overlooked the manual prescriptions around designs.

Communities (in hot zones) referred to latrines being small in size, low roof and low walls, hence lead to discouraging some for fear or perception of lack of privacy. Although designs prescribed local adaptation, which to a greater degree not been complied with. UNICEF and IPs are advised to adapt design with respect to roof height, larger ventilation points and higher side walls.

The prescribed pit size for the demo latrines highlighted being insufficient by the communities. There are reported cases where community tried to dig bigger pits, however the cover provided (by the project) found to be of smaller. This alludes to the fact that in some cases the process of agreeing a participatory design with community was not followed.

Life cycle cost analysis of the latrine options also seems not to have been done. Depending on the material used, design life of the latrine superstructure is expected to vary considerably.

SPSP aims to improve peoples’ access to safe drinking water and sanitation. It merits further researching as to what impact increased sanitation coverage has had on water quality and accessibility in target.

Design of hand wash facility needs further clarity and needs to be communicated to the IPs particularly in areas where water is being accessed

[45]

through hand pumps. It merits consideration whether to have separate hand wash facility or not, and if is not required then suggest what actions to take.

School WASH facilities were found not being disability friendly both due to stairs leading up to the hand wash facility, as well as the design of the hand wash facility itself, which in most cases requires children to squat and reach out for the tap.

The evaluators did use an observation checklist (in line with standard design guidelines and standards followed worldwide). Significant deviations were noted and in some cases communities and teachers shared that for limited school space, the compliance to the set standards (for instance the minimum or safer distance prescriptions) was not possible anyways. Local Context and Issues The evaluators found quite a few non-functional demo-latrines, which amounts to lack of investment (cost sharing required) from communities and in some case discouraging attitude from the landlords (in Sindh particular). Some of the community feedback articulating the reasons for non-functioning of demo-latrines is outlined below (all by and large point to non-compliance to prescribed participatory design process):

Some shared that even if the sides of the latrines are covered with cloth/provide some form of superstructure, the females were reluctant to use for privacy reasons for low height of boundary walls.

Some referred to land ownership issues where landlords (particularly in Sindh) did not allow the tenants to construct latrines on their lands by tenants/ workers. There was visible reluctance from the landlords to allow seasonal tenants to construct permanent latrines on their land.

The implications of the points highlighted above include women being dissuaded to use the latrine due to lack of privacy and a possibility of conflict with landlords. The larger implication is that, a facility that has been built is not used, but still counts to be contributing in efforts to achieve ODF targets. The IPs may need to be advised to integrate local context and follow the prescribed process to have people’s views before such interventions. This is likely to contribute to continued and effective use of the facilities or investments made. For latrine construction, outputs exceeded than the targets (see table 12).

Latrines Constructed under SPSP Target Actual

No. of latrines (other than demo latrines) constructed by masons trained under PATS programme

13,857 20,040

No. of complete demonstration latrines constructed for extremely vulnerable families as per agreed criteria with community (maximum 3%)

4,157 5,212

No. of demonstration latrines constructed through materials provision/vouchers for vulnerable families as per agreed criteria with community (max 7%)

9,700 8,581

Table 12: Number of Latrines Constructed (Target versus Actual)

For supply facilitation (as noted in Figure 16), the results indicate an overall 39% of the respondents found using latrine facilities constructed with external support in the form of subsidy, loan, materials or trained skilled labour (please note that for this indicator the baseline value was assumed to be 0%).

[46]

The results indicate increase in people claiming to have constructed latrine in last year. The numbers have gone up from 26% to 47%, which is significant. The respondents shared that supplies facilitation as major support is available for latrine construction (only for those who had constructed latrines last year). .

The key challenges include withdrawal of grant/loan for sanitation mart and entrepreneurs. This affected the supplies facilitation element negatively; however the positive results (around latrine construction and availability of materials) indicate that existing mart owners were able to meet up the increased demand. The increase for availability of materials and skilled labour both point to larger effectiveness of the interventions (see Figure 18).

The programme may need to evolve more effective and both user and entrepreneur friendly solutions for this. UNICEF may need to look into DSI’ work for this including work in the region.

39% 45%

17% 18%

53% 55%

0%

50%

100%

Total Balochistan FATA KPK Punjab Sindh

Proportion of Population Using WASH Facilities Constructed With External Support

Total Balochistan FATA KPK Punjab Sindh

Figure 16: WASH Facilities Constructed with External Support

26%

4% 2% 8%

47%

12% 6%

31%

0%

10%

20%

30%

40%

50%

Latrine Constructed in the Last One Year in Respondents House

Demo Latrine Cost Share Supply Facilitation

Type of Support Received by the Respondent for latrine construction

Figure 17: Latrines Constructed By Respondent over Last Year and Type of Support Received

[47]

Overall the proportion of people having access to and using improved drinking water source has improved from 76% to 82% (refer to Figure 19). This points to the success of programme with respect to increasing access to drinking water (excludes water quality).

School WASH Facilities Post-KAP results show a positive trend in all key aspects, including 97% access to latrine facilities and 79% access to water facilities (in some cases new water schemes were not installed for poor water). The results indicate a significant increase in availability of soap close to larine, which increased from 11% to 30%. This points to huge space/need and potential for improvement in this regard.

16%

53%

27%

60%

0%

10%

20%

30%

40%

50%

60%

70%

Availability of Sanitation Supplies in the Village Availability of Trained Skilled Mason/Labour in Village to Construct the Latrine

Figure 18: Availability of Sanitation Supplies and Trained Masons

76% 60%

82% 82%

53%

96% 82%

68% 86% 85%

66%

99%

0% 20% 40% 60% 80%

100% 120%

Total FATA KPK Punjab Balochistan Sindh

Proportion of the population using an improved drinking water source

Population Using an Improved Drinking Water Source

Baseline Evaluation

Figure 19: Population Using an Improved Drinking Water Source

[48]

The delivery mechanisms must have given consideration to the customary summer holidays, which apparently was not given much thought into while laying the plans. In line with the ‘Islamabad Commitment (2010)’, signed by all provinces for Child Friendly Inclusive Schools, the programme did effort and largely remained successful to improve school WASH facilities to make them child friendly9. The programme prioritised girls’ school for WASH facilities rehabilitation. It was found more relevant for the fact that culturally availability of safe and secure facilities at schools is relatively more significant for girls. The results indicate better performance in girls’ schools compared to boys such as availability of soap, though still insufficient. This merits greater attention from teachers and availability/use of SMC funds to ensure continued availability of soap, as well as proper functioning and cleaning of the facilities. Positive developments in this regards include Punjab government’s approval to use SMC funds for repair and maintenance, and the change in resourcing criteria for SMCs in Sindh from a fixed yearly amount to a per child allocation basis. Despite policy changes, the actual impact in terms of funds getting spent is quite limited. Often these remain unused or under spent for procedural reasons (possible audit objections, low engagement and capacities of SMCs, limited interest of school head teacher and others), hence the future programme design may need to prioritize addressing those procedural bottlenecks.

9 According to ‘Development of a framework and an action plan for school sanitation and hygiene education’ (SSHE) in Punjab, Analytical Report on Current

Status of School WES in Punjab School; Final Report December 2007. UNICEF Punjab, child friendly facilities have the following attributes,

easy and comfortable seating

proper ventilation and light arrangement

appropriate space inside the toilet

special non-slip ceramic tiles in pour flush latrines

provision for looking glass (mirror)

suitably designed hand washing points

appropriate height of door knobs/locks

appropriate height of hand washing basin

provision of shelf and hook to safely keep belongings while using toilet

provision for children with special needs

74% 53%

31% 11%

97% 79% 77%

30%

0% 20% 40% 60% 80%

100%

Schools with Latrine Facility

Schools with Drinking Water Points

Schools with Hand Washing Facilities

Schools with Soap Close to the Latrines

School WASH Facilities

Baseline Evaluation

Figure 20: School WASH Facilities

[49]

85% 97%

77%

23%

74%

95%

74%

26%

65%

100% 80%

45%

0%

20%

40%

60%

80%

100%

Availability of Drinking Water

Availability of Latrine Facility

Handwashing Facilities Availability of Soap

Boys Co-Education Girls

Figure 21: Availability of WASH Facilities in Boys, Girls and Co-Education Schools

[50]

Output 3: Health and Hygiene Promotion (H&HP) The results indicate largely positive performance for health and hygiene promotion component. However, for some indicators the results are not very encouraging.

The following table illustrates the mixed performance for some of the health and hygiene activities.

Description Target Actual

No. of people reached with WASH BCC messages through electronic/print media. 970,000 609,530

No. of people reached with WASH BCC messages through community events (theatres, puppet shows). 183,300 165,092

No. of school children (boys and girls) reached with messages through SLTS activities. 63,941 154,526

No. of people (men, women, boys, girls) reached with WASH BCC messages through CRPs. 291,000 846,444

Table 13: Outreach of Health & Hygiene Messages through Various Mediums

Overall the results are positive with respect to increasing knowledge and bringing positive attitudinal and behavioural changes. For instance, water treatment and storage practices show improved results where those treating water (through multiple tools) increased from 4% to 13% (a three-fold increase). Similarly, those storing water has increased from 74% to 91%. Balochistan province leads in terms of extent of change followed by Punjab (see Figure 22 & 23).

4% 13%

2%

41%

18%

0%

26%

74%

13% 23%

10%

48%

1% 5% 13%

91%

0%

20%

40%

60%

80%

100%

Water Treatment

Boil Add Bleach & Chlorine

Strain it through a

Cloth

Use a Water Filter

Solar Disinfection

Let it Stand & Settle

Water Storage Practice

Water Treatment Methods '

Figure 22: Household Water Treatment and Safe Water Storage

5% 5% 9%

3% 0% 8%

13% 18%

34%

10%

2%

11%

0%

10%

20%

30%

40%

Total Punjab Balochistan Sindh FATA KPK

% of respondents who treat drinking water

Baseline Evaluation

Figure 23: Proportion of Respondents who Treat Water

[51]

Another measure of success and effectiveness of H&HP is increase in the number of households reporting availability of soap in the house. Similarly, there is an increase in designated hand washing places (available). The increases indirectly points to heightened awareness around hand washing and linkages to diseases (refer Figure 24 for details). The availability inside or near latrines have not increased much though. This can be attributed to the fact that people in rural areas tend not to leave the soap near or inside the latrine for risk of dissolution (waste).

The encouraging increase in hand washing practice at critical moments, points to effectiveness of H&HP. The results are particularly more encouraging for post defecation (refer to Figure 25) compared to other elements such after urination, cleaning child etc.

19%

47%

27% 32%

60%

25%

0%

20%

40%

60%

80%

Separate Handwashing Place Anywhere in the dwelling Near/Inside Latrine

Soap

Hand Washing Place & Soap Availability

Baseline Evaluation

Figure 24: Hand Washing Place & Soap Availability

50%

29% 20%

15% 15% 10%

62% 68%

22% 14% 12%

7%

0%

10%

20%

30%

40%

50%

60%

70%

80%

Before Eating After Defecation

After Urination Before Cooking After Cleaning a Child that has

Defecated

Before Breastfeeding a

Child

Respondents Practice of Hand Washing at Critical Moments

Hand Washing at Critical Moments

Baseline Evaluation

Figure 25: Critical Moments for Hand Washing

[52]

The effectiveness is evident from encouraging increase on awareness around water related diseases. The heightened awareness however has not contributed to any significant reduction of diarrhoea incidence though. This could have the seasonal element also; as numbers increase in summers anyways (refer Figure 26 for details).

52% 28% 27% 22% 27%

79%

44% 44% 33% 24%

0% 20% 40% 60% 80%

100%

Diarrhea Cholera Hepatitis Skin Disease Incidence of Diarrhoea in Last

Two Weeks

Knowledge of Diseases Caused by Using Unsafe Water '

Knowledge of Water Related Disease and Incidence of Diarrhoea

Figure 26: Knowledge of Water Related Diseases and Incidence of Diarrhea

[53]

Output 4: Linkages with Duty Bearers The level of interest and engagement of public sector entities varied significantly across provinces as well as IPs. However generally there was not much success with respect to fostering ownership of PATS and mobilizing resources for planned rewards/incentives. The district governments, however, helped with selection of villages, identification of schools and health facilities which needed rehabilitation, subsequently they were engaged at the ODF certification process, involving a range of government officials from the departments (LGRDD, PHED, Education and Health). Overall the interventions still contributed to raising knowledge and awareness of public officials both at the provincial and district levels though the extent of it merits further investigation. Bottlenecks identified through the provincial workshops as part of the baseline did not translate into activities to address them; this may need to be prioritized in future programming. This component aimed at developing linkages with duty bearers and public institutions to support PATS to positively influence budgetary increases (at provincial level) for WASH services, encourage PATS adoption in the public sector and finally leverage public funds as rewards for villages turned ODF. Multiple activities carried out for this and largely the results are encouraging (refer Table 14). Description Target Actual Percentage

Capacity Needs Assessments (CNA) 14 6 43%

Capacity Development Plans 14 6 43%

Capacity Development Events 14 15 107%

Number of Government officials and political leaders that participated in capacity development and key events of the programme.

185 230 124%

Advocacy initiatives undertaken with duty bearers/political leadership to support PATS

14 17 121%

Villages jointly selected for PATS programme by relevant Government duty bearers with IP

1,385 1,791 129%

ODF communities received rewards/incentives by government 39 62 159% Table 14: Capacity Building & Advocacy for Government

The level of interest and engagement of public sector entities i.e. Public Health Engineering Department (PHED), Local Government (LG), Health and Education departments, varied significantly across provinces. For limited guidance and standardization, the focal points in the public sector were found to be different for different IPs (for some it was DC office, others worked with either PHED or LGRRD). The IPs were found to be connected to bottom tiers of government officials. In some cases, where the (public sector) focal points were more sensitive and aware of potential NGO contributions, showed greater interest, ownership and leveraged the partnership for common objectives. The district governments helped with selection of villages and education and health authorities facilitated identification of schools and health facilities needing rehabilitation. The interventions did contribute to raising knowledge and awareness of public officials (particularly around PATS), both at the provincial and district levels. To what extent to which capacities been raised and contributing to improved public services is unknown and merits investigation.

[54]

The achievement of leveraging public resources both in terms of increased budgetary allocations (at provincial level) and mobilizing funds for rewards at districts are insignificant. This again reflects lack of planning and guidance for the intervention area (more details in Figure 27).

The frequent transfers of the public officials also did affect the efforts, as those who were sensitized and trained got posted elsewhere, leaving IPs to work with new officials with no/limited exposure to programme and areas where it needed their support. This element together with short time frame (requiring focus on delivering outputs) apparently work to discourage IPs to work more closely with local officials. Bottleneck analysis workshops conducted at provinces as part of the evaluation, which were attended by LGRDD, PHED and Education Departments. It was noted that not much progress been made with respect to actions identified in the bottlenecks carried out during baseline. This may need to prioritize for the coming years.

0.28% 0.39% 0.37%

0.12%

0.00%

0.50%

KPK Punjab Balochistan Sindh

Government expenditure on WASH as a proportion of total government expenditure

Figure 27: Government Expenditure on WASH

[55]

Output 5: Attaining Total Sanitation The programme appears effective with respect to achieving the targets for transforming the targeted communities or villages into ODF. This marks progression on the sanitation ladder. There are, however, concerns as to communities being able to sustain the ODF status. The table below presents the reported results with respect to the total sanitation component. The programme has achieved (overachieved) on all sub-indicators. Villages supported in attaining Total Sanitation status Target Actual

No. of households with new latrines constructed after start of the PATS programme 17,182 33,673

No. villages declared as ODF by community 831 1,436

No. of villages Verified as ODF by Implementing partner and independent monitors 831 1,250

No. of people (men, women, boys, girls) in communities declared as ODF 581,700 711,467

No. of people (men, women, boys, girls) in communities verified as ODF 581,700 679,967

No. of model villages with laid sewers, WW collection and treatment system through PATS programme 4 18

No. of advocacy initiatives undertaken to promote communal underground drainage system linked WW treatment system and communal SWM system 2 22

No. of ODF villages provided with underground drainage and WW treatment facility by relevant public sector stake holder/duty bearer 2 0

Table 15: Villages Supported in attaining Total Sanitation Status

This performance however comes with caveats around sustainability. In general, there is an understanding that the focus was on delivery rather sustainability and for shortage of time the IPs were left with no time to monitor and to sustain ODF status in the certified villages. The communities have started showing signs of slippage. The programme has also been able to demonstrate healthy increase overall in communities/households connected to under-ground drainage (details in Figure 28). The evaluators did however notice significant variations in terms of what constitutes drainage system.

27%

52%

0%

15%

35% 31% 44%

65%

33% 43%

58%

24%

0%

20%

40%

60%

80%

Total FATA KPK Punjab Balochistan Sindh

Households Using Underground Drainage

Households Using Underground Drainage

Figure 28: Households Using Underground Drainage

[56]

A CRP told the evaluation team that in Uthman Khel village a household latrine was being drained directly on to the street. Due to this the area was very dirty and no one was able to walk past that part of the street. After the PRA exercise the people contributed through VSC to construct drainage / sewer along the streets. Therefore, the streets are now clean.

The evaluators noted heightened

awareness evident from the identification

from the communities themselves that the

latrines with single pits designs; do not

seem to have taken note of the local

demographic context (where people living

in joint and extended family arrangements)

and the single pits may fill in very quickly.

Without adequate cleaning/disposal

arrangements, overflowing pits may cause

of pollution and spread diseases.

[57]

Output 6: Knowledge Management and Accountability The knowledge management component being an integral part to the programme design and delivery has contributed to critical learning around use of subsidy, sanitation marketing and low-cost sanitation options. The third party monitoring mechanism instituted for the programme proved to be particularly useful in real time monitoring and complemented decision making. The new learning resulting from this phase of the programme have already been fed into the design and modalities of the next phase of SPSP, and offers the opportunity and potential of informing a PATS review. Overall, the component contributed to improving the programme implementation and interventions by informing management and programmatic decisions through real time monitoring of programme processes, challenges and issues whilst helping in course correction. The component also focused on programme learning and documentation for future design, possible replication and scaling up through several lesson learning workshops, baseline and evaluation studies and action research. The achievements for the components are presented in the following table. Knowledge Management and Accountability Actual Target

No of Inception workshops conducted in the beginning of programme 12 12

No. of villages with baseline established in the beginning of programme by IP 1,385 7,448

No. of joint progress monitoring reports (IP and public institution) 80 39

No. of monthly substantive monitoring reviews’ visits conducted for confirming/measuring performance indicators in results framework 80 216

No. ‘monthly substantive monitoring review’ reports shared among partners 63 55

% of responsive progress reports submitted as per MEAL framework by IP - 33

No. of quarterly progress reviews conducted with IP and minutes documented 2 15

No. of mid-term review workshop report endorsed by UNICEF 1 4

No. of Human interest stories documented by IP and accepted as per UNICEF guidelines 80 78

No. of end line survey reports accepted by UNICEF 11 2

No. of Lessons sharing workshop conducted and feedback documented 1 1

Table 16: Knowledge Management and Accountability Events and Activities

Third Party Monitors (contracted for process improvement, verification and feeding into management decision making) work was a major factor in redesigning programme activities based on ground realities that emerged over the course of programme implementation. This monitoring process allowed UNICEF to monitor and oversee ground implementation in real time and push for more effective and efficient implementation through IPs (particularly those that were lagging). It also helped inform the management of the need for corrective measures where weak performance of IPs was spotted. Third Party Provincial Monitoring meetings (with IPs and UNICEF) also allowed for active and effective debate that helped keep the project on track. Monitoring was of four types, planned, verification purposes, spot checking and follow-up. Regular monthly reviews helped in making the programme more effective whilst ensuring efficiencies were achieved. Whilst most of the Third Party Monitors work proved effective, there was some feedback that at times monitoring took the form of policing. Moreover, there was confusion in respective roles of several firms contracted for the range of monitoring.

[58]

These aspects need to be clarified and addressed in the next phase and in future programmes. Given the delayed start of the project, tight timeline for activities and an ambitious monitoring schedule, more focus and emphasis remained on monitoring and less on knowledge management and documenting of key learning during the programme implementation period. However, the component did contribute to knowledge management by commissioning useful baseline and end of phase evaluation. The baseline findings and learning have been fed into the implementation of Phase 1 of SPSP, whilst the evaluation findings and lessons learnt have been used to inform the design and interventions of Phase 2 (during the presentation and workshop with the Evaluation Resource Group, UNICEF Staff and other Stakeholders). The component also made available data and evidence on the use of PATS in Pakistan, with a potential of using this information in a structured way to inform PATS and sector stakeholders of the experiences, challenges, and successes of SPSP implementation. Information is now available, however it needs to be consolidated, disseminated as well as used for a possible review of PATS based on the experiences, and lessons learnt through the implementation of SPSP Phase 1, particularly around low-cost sanitation options, use of subsidy for household latrines, and sanitation marketing. Moreover, the action research commissioned to DSI should also be proactively shared with stakeholders to help inform their interventions.

[59]

5.2.4. Stakeholders Contribution10 Multiple stakeholders contributed to the successful completion of year 1 activities. UNICEF has largely been successful in awarding roles (while taking note of partners strengths), creating synergies, complementarities and coordination mechanisms, and documenting the expectations and responsibilities between stakeholders. The review of the programme achievements indicate that most have been able to deliver on expectations. The description below captures the key contributions, successes and challenges (or opportunities) with respect to the key stakeholders involved. UNICEF (Islamabad and Provincial Offices) provided effective programmatic and management oversight to ensure that the programme was delivered according to the specific PCAs with IPs within the available resources. The number of actual beneficiaries as well as villages exceeds the planned targets. UNICEF also put in place a robust monitoring function for the overall programme which helped inform better decisions and course correction during implementation. Throughout the programme, UNICEF offered a range of capacity building interventions which helped develop the community’s capacity, build a cadre of PATS master trainers, increase capacity of non-profit entities as well as government departments (though the focus on government capacity building could have been greater) to implement sanitation responses. Though it is difficult to attribute all improvements and positive policy measures taken at the federal and provincial level to one stakeholder i.e. UNICEF, however there is evidence in the form of Provincial Government’s Master Plans that incorporate PATS model, as well as in some cases where government allocations to WASH sector have increased. However engagement with government needs more concerted effort from UNICEF Islamabad as well as Provincial Offices particularly in resource mobilisation for PATS. Other UNICEF contributions included a focused emphasis on Knowledge Creation including project and sector specific training manuals, IEC material, training documentary, producing thematic issue papers, focusing on issues such as Menstrual Hygiene Matters (MHM) research and WASH Equity study. Lastly, UNICEF’s contribution has been of raising sufficient funds for the two programme contexts i.e. follow up in flood affected areas as well as funding for WASH sector in the insecurity affected areas. UNICEF’s contribution had an impact across all the outputs of the programme. DSI’s specific contributions relate to sanitation marketing, design manual for latrine facilities and focusing on action research in SPSP pilot area. Sanitation marketing is an evolving process in WASH sector in Pakistan, and DSI demonstrated their flexibility in capacity building interventions during SPSP where circumstances necessitated that they provide recurring capacity building support. Moreover, offering a business advice phone service to the Mart Owners and Entrepreneurs specifically contributed to building the confidence as well as helping through the initial teething issues. DSI’s contribution resulted in building and sustaining demand through supply side interventions.

10 TORs required the evaluation to focus on a sample of partners; there was no requirement to conduct a comparative analysis of IPs.

[60]

RSPN’s capacity building activities (in particular social mobilisation training, use of PRA and mentoring activities) played a critical part in the delivery of results as it helped a range of non-profit organisations to deliver the SPSP programme across geographic regions using a uniform implementation methodology to carry out SLTS and CLTS. Some points have been highlighted above regarding changes that can be incorporated in the training, mentoring approach as well as social mobilisation to ensure it is more effective for future phases, however in the context of Phase 1, the activities were carried out effectively both in terms of number of activities and beneficiaries as well as in terms of delivery of results. RSPN’s contribution resulted in generating demand for sanitation. APEX contribution to the programme remained its regular monitoring reports and feedback which informed real time decision making to keep the programme on course both in terms of its delivery of outputs and achievement of results. Recommendations have been made to address issues that were highlighted in reference to monitoring i.e. rigidity of approach, which if addressed will further improve the role of third party monitoring in the next phases of the programme. APEX’s direct contribution is to the knowledge and accountability output, however due to their regular monitoring the results of their contribution have an across the programme impact for all output areas. Government Departments also played a contributory part although with substantial variation across geographic regions, i.e. in some cases close coordination and engagement with the project, but through lower tiers of the departments (e.g. Dera Ghazi Khan District), whilst in other areas limiting departmental engagement to only certification of ODF villages through senior district officials e.g. ADC in Jacobabad. However this allowed government departments to become familiarised with the PATS approach as well as to the issues on the ground in the villages. Across SPSP, in certain parts where government was more responsive, water supply schemes have been rehabilitated, whilst in other areas government allocation in its budget to WASH sector has increased, however direct attribution to engagement through SPSP is difficult to do.. Government’s direct contribution has been to the linkages output, with community-government, non-profit –government department links being formed to deliver PATS. IPs (range of non-profit organisations) contribution has been across the results framework in all output areas. The extent, degree as well as success varies across the IPs, however all have been able to create and sustain demand for sanitation, while also ensure supply facilitation, health and hygiene promotion and total sanitation results were also attained. Specific areas have been highlighted in the findings section as well as more specific detail is provided in the lessons and recommendations section to help improve IPs delivery and impact for the next programme phase. In summary the required number of activities and outputs were delivered, benefitting a larger number of villages and beneficiaries than targeted, with many result indicators showing positive trends i.e. increased access to sanitation facilities (for men, women, children as well as an improvement in the equity context). Similarly ODF declaration, verification and certification have also been achieved though slippage will need to be monitored and controlled.

[61]

Community stakeholders (CRPs, VSCs, Children’s School WASH Clubs, Teachers as well as masons, entrepreneurs and sanitation mart owners) have played the most vital part in helping achieve the results. The various community stakeholders stated have actively participated in the programme activities, given their valuable time to the interventions, facilitated IPs, APEX and UNICEF. Moreover, their willingness and acceptance of wanting to improve their and their community’s lives helped make this programme an effective one that also achieved its targets. Their contribution can be mapped across the range of the outputs.

5.2.5. Gender Equality, Equity & Disability In line with UNICEF policies, the programme design and interventions have incorporated principles relating to gender equality, equity and disability. In terms of gender, the programme planning and implementation demonstrates balanced focus on men, women, girls and boys in terms of having balanced field teams, creating of women/men VSCs, children clubs, and CRPs, gender and age focused BCC strategies and interventions, prioritization of girls’ schools for rehabilitation of WASH facilities. The focus on equity is evident from use of PRA for poverty ranking, and provision of subsidies to the most poor. However the integration of disability remained weak as evidenced in the construction and access aspects of WASH facilities. The selection of schools both for boys and girls, Social Organizers, Community Resource Persons (CRPs) and Village Sanitation Committees (VSCs) ensured participation of men and women in the programme. To address differential information needs, access to information sources and physical accessibility -- mainly for cultural seclusion between sexes -- the programme evolved responsive BCC strategies in terms of content, design and medium of transmission, e.g. female CRP sessions, and theatre performances for children. The programme was also in line with the needs identified by women, including children’s health due to the unhygienic conditions as well as hazards of open defecation, difficulties faced by women particularly due to privacy needs which force them to defecate either very early in the morning or to hold until it is evening causing health complications. It was also mentioned that going at night exposes them to a range of risks including being bitten by insects or attacked by animals, to even being harassed and physically attacked by men. The programme design therefore clearly identified gender considerations such as appropriate design of latrines for women and girls to ensure privacy and safety, appropriately designed water collection points as well as role in collecting water. Gender disaggregated data was effectively collected and used in the program. The programme design also effectively addressed equity considerations by including provision of low-cost environment friendly and culturally appropriate demonstration latrines for the poorest of the poor communities. The programme had a 3% (for the extremely vulnerable) and 7% (for the vulnerable) approach to address the sanitation needs, whereby the extremely vulnerable received subsidy for the whole latrine, whilst the vulnerable had to construct the super structure using their own resources. SPSP design also considered addressing the needs of the disabled; however, this approach lacked clarity in terms of expectation and implementation.

[62]

From a gender perspective the results of the post-KAP survey demonstrate that women’s access to latrines has increased by 30%. Moreover in numerous interviews and meetings with a range of stakeholders (both men and women), the evaluators were told about instances of violence as well as harassment towards women due to their vulnerable situation while going out at night for defecation. In the gravest of situations, false allegations of sexual indiscretion have even led to instances of Karo Kari.

In terms of equity, the lowest and the 2nd Quintile seems to have benefitted most in terms of access to latrines as intended by the programme. However, as highlighted above household subsidy as a method to respond to equity needs will need to be revised or replaced with innovative methods. Lastly in terms of the disabled, access to household latrine has increased to 66% during the evaluation as compared to 44% during the baseline; however whether these latrines cater to the special needs of the disabled is not confirmed, given this component remained weak during the design as well as implementation stages. Moreover, there is substantial evidence that suggests that the programme’s successes, as secondary impact, have also contributed to reinforcing a gender divide. For instance, the programme interventions have added to the workload of women and children being responsible for cleaning latrines and fetching additional water required due to the introduction of flush latrines. There has been some feedback on the DRR designs of the latrines as well as water facilities which have been designed to be elevated, and which have reduced the privacy factor for some. This is of particular issue where such facilities have been constructed without taking community into confidence about the design aspects of the facilities; otherwise an alternate site for the facility could have been facilitated by the community.

44% 34%

27% 32%

13% 7% 9%

66% 64%

40% 37%

12% 5% 6%

0% 10% 20% 30% 40% 50% 60% 70%

Disabled Having

Access to Latrine

Women Having

Access to Latrine

Lowest 2nd 3rd 4th Highest 5th

Income Quintile

Gender, Disability & Equity Based Access to Latrines

Baseline Evaluation

Figure 29: Gender, Disability and Equity Based Access to Latrines

[63]

Also there is however a strong perception by a range of stakeholders (CRPs, VSCs, Government Officials) that having access to latrines has reduced instances and opportunities for harassment and violence against women that was always a possibility when women were defecating in the open particularly given the privacy needs at night time.

[64]

5.3. Efficiency11

Evaluation Q3: To what extent have the outputs of the project been achieved (in terms of quality and quantity) with the allocated resources/inputs (such as funds, time, and procedures)? The evaluators have looked at the element of efficiency in terms of progress or performance against key results, adequacy of resources (time, human and financial) and coherence to the set/available quality standards. The evaluators did do value for money analysis with respect to cost per beneficiary (for varied results) and basic analysis around costs for partnership management. The programme has achieved efficiency in terms of performance against target indicators, Cost per beneficiary and villages achieving ODF declaration stage demonstrates efficiencies against the planned costs. However, some indicators underperformed, while concern around planning, target setting and best use of resources is raised on indicators that substantially overachieved their targets (this leads to the understanding that either there was a cost and resource implication or a quality impact). Project results indicate adequacy of human resources, however the results exceeded by a number of times pointing to gross miscalculation as to planned targets and resources required (financial and human). Instances such as contractors charging the same amount of funds for building new and rehabilitation old school wash facilities suggests areas of financial inefficiencies. Highly variable cost of latrines raises the question of affordability for the vulnerable across SPSP programme areas. Performance against Target Indicators The efficiency analysis with respect to performance against key indicators suggests that the IPs have been able to achieve several of major targets. The Programme Output Matrix (updated version available as of March 2014) shows that out of 60 output indicators, the programme exceeded targets for 41 indicators. Furthermore, within these there are 10 output indicators, where results exceeded by over 300%. These however, raise concerns around set targets being reasonable, over-budgeting and given these were achieved in shorter duration, the quality and sustainability of the results. However, for some indicators the performance has been below par. For example, reaching out to beneficiaries through media & theatres, provision of rewards and incentives (with the target to provide 158 critical community infrastructure schemes as incentives/rewards, only 2 were delivered). Furthermore, two programme components indicate little progress i.e. provision of loan and provision of underground drainage. This point to limited success for fostering partnerships (with relevant public authorities) and mobilizing public resources.

11 Note: The evaluators’ ability to conduct a detailed analysis was constrained by the fact that project expenditure sheets were unavailable (not compiled by the time evaluation was in progress). The calculations and analysis done below relies on budget estimates rather actual expenses, drawn either from PCAs or UNICEF submissions to DFID (UK).

[65]

Financial, Human & Time Resources The review of programme performance for key output indicators suggest satisfactory performance, and this at another level points to adequacy of resources to carry out tasks. However, the fact that the programme has exceeded most of the targets by 20% to 100% raises concerns around realistic costing by the IPs (refer Annex 13: Programme Target versus Achievements). The evaluators noted some costing oversights where they found that contractors (for rehabilitation of WASH services) charging the same amounts for building new facilities as for the rehabilitation of old ones at school and health outlets. Some IPs communicated that the umbrella arrangements contributed to raising operational expenses (where both umbrella and implementing partners charged operational expenses), which points to the inefficient use of resources. The staffing arrangements were found to be adequate by and large. The field staff especially SOs did raise concerns of the availability of time for quality work as mobilization and exit planning. To them, this contributed to pushing them work extra hours to meet the targets. Some IPs referred to this as key factor for high staff turnover. SPSP Phase 1 was designed for a period of one year. The field level implementation in some cases started with a delay of up to 4 months (PCAs with IPs were signed all through February to June 2013) which allowed some IPs implementation period of only 6-8 months against the planned one year. Despite the fact that IPs were able to achieve most of the set targets, there are concerns with respect to work planning & time availability (to IPs for implementation). Reportedly, there were oversights with respect to timing activities with community life-cycle (despite the fact that the information was available from village PRA exercises) and with those of public services outlets such as schools and health centres. This to a degree pushed IPs staff to use short-cuts to complete activities e.g. BCC, and left them with little time to plan exit and leave behind arrangements to sustain the achievements and take it further. WASH Facilities Cost Comparison Across IPs and Geographic Spread: The documents reviewed indicate significant cost variations for different interventions such as demo-latrines (varies between Rs. 9,500 - 20,000), hand pumps (from Rs. 18,000 - 69,590). Whilst a price differential is expected across various regions, it raises question on the definition of low-cost in terms of latrines. For instance, the costs range between Rs. 9,500 and Rs. 20,000 across Kashmore and Jaffarabad respectively, though the monthly household income of the lowest quintile across the two districts varies only by Rs. 292, thus questioning the appropriateness as well as cost efficiency for demonstration latrines for the vulnerable in Jaffarabad and likewise for other districts.

[66]

Cost Analysis This section offers cost/beneficiary analysis with respect to different output/outcome level results. The financial information used has been drawn from all the PCAs (Project Cooperation Agreements) signed between UNICEF and the Implementing Partners (IPs) for this phase of the SPSP Programme. Also, it offers comparative analysis with regional countries. Cost Per Beneficiary The budgeted figures indicate that for SPSP Phase 1, a budget of US$ 7.5 million was set aside for PCAs with IPs to reach 1.3 million people (men, women, boys, and girls) in 13 target districts/agencies. Reportedly (refer Indicator No. 2 in Annex 13), the project reached out to 1.2 million people with sanitation messages. Hence, the programme cost per beneficiary comes out to be US$ 6.2 (against a planned cost of US$ 5.8). Total Budget in PCAs US$ 7,500,724

Planned Beneficiaries in ODF villages 581,700

Actual Beneficiaries Reached 711,467

Per Capita Costs

Planned Beneficiaries Actual Beneficiaries

$ 13 $ 11 Table 17: Cost Per ODF Declared Beneficiary

The cost per beneficiary (Table 17) for all components of programmes in a village achieving ODF declaration for actual number of project beneficiaries is US$ 11, whereas the planned cost for the targeted beneficiaries was US$ 13, thus demonstrating efficiency gains of US$ 2 per beneficiary. However, the actual cost per capita for Social Mobilization12 is US$ 1.52 as compared to planned cost per capita of US$ 1.40 for the same, demonstrating reasonable control over planned costs during programme implementation. Cost Per Community/Village Another key result was for communities/villages to achieve ODF status. These include villages which declared (self-declared) ODF status and were verified by IPs and TPFMs.

12 Breakup costs for Social Mobilization Package comprising of salaries and training of SOs & PM, Incentive and training of CRPs, Training of VSC and Training of Govt Officials. This also includes proportionate direct and indirect programme support costs.

[67]

Total PCAs Budget 6. US$ 7,500,724

No. of Planned Target Villages* 1,385

No. of Actual Villages Reached* 1,878

No. of Planned ODF Villages 831 No. of Declared ODF Villages 1436 No. of Verified ODF Villages 1250

(Planned) Cost per village for total target no. of ODF villages

Cost per village to declare it as ODF village

Cost per village to Verify it as ODF Village

US$ 9,026 US$ 5,223 US$ 6,001 Table 18: Cost per ODF Declared and Verified Village

In this context the programme has overachieved both targets i.e. number of people living in ODF settlements, as well as the number of ODF declared villages. This has resulted in a substantial efficiency of 42% in the cost of having a village declared as ODF. In comparison to SPSP, the Total Sanitation and Sanitation Marketing (TSSM) programme in India which was implemented in two states Himachel Pradesh and Madya Pradesh13 with an actual budget of $1.9 million had a cost per person living in an ODF declared village of US$9 only. The programme documents describe the costs promoted for building a new latrine into three categories equalling $38 for category ‘A’ toilet, $70 for category ‘B’ toilet and $99 for category ‘C’ toilet. Using the average cost of these 3 categories, the average cost of constructing a new latrine equals $69. A study conducted by WaterAID for CLTS programmes across Bangladesh, Nepal and Nigeria suggests that the latrine (lined pit, concrete slab and concrete/ceramic pan, brick or concrete superstructure) costs $62-100 in the plain areas and US$150-187 in the hills. As such the three latrine designs are well within comparable costs. Overall, the evaluators feel that the time-lag in approving PCAs did affect the quality delivery of project activities especially for IPs whose contracts were approved late. The over-achievements by 100% or more for few indicators reflect under-targeting, as in most cases those results were achieved within the given time and budgets. Hence, UNICEF is advised to work more closely to set realistic activity targets and corresponding budgets.

13 Water, Sanitation and Hygiene Portfolio Review; Department for International Development (DFID); March 2012, available at https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/67433/DFID_20WASH_20Portfolio_20Review.pdf

[68]

6.1. Sustainability Evaluation Q 4: To what extent various stakeholders including the public sector and civil society are likely to sustain the behaviour change related to the goals of the programme after it has ended? Programme design excludes any articulated and expressed exit plan or sustainability strategy. Knowledge, attitudinal and behavioural changes that this programme has been able to bring amongst men, women and children, are the most significant and likely results that may sustain. Whilst the community structures and mechanisms such as CRPs and VSCs are likely to sustain but with limited success. However, IPs found no/little time to prevent the process of slippage following ODF declaration and certification. Linkages with duty bearers component could achieve little in terms of fostering ownership amongst public agencies. The sustainability of sanitation mart owners as well as entrepreneurs created/promoted during the programme is also uncertain due to lack of access to loans/finance as well as low demand issues. Lack of having instituted an O&M mechanism for rehabilitated WASH facilities at schools and health outlets suggests a high probability for them to become defunct in time. According to the PATS approach achievement of an ODF environment (declared or certified) is not sufficient, it needs to be sustained. Moreover, there needs to be an enabling environment at local, provincial as well as national level to ensure long term sustainability. The evaluators have therefore assessed sustainability across a range of aspects i.e. sustainability as part of the design, level of commitment amongst stakeholders to sustain the ODF status in project areas, contribution of SLTS in sustaining ODF status in communities, partnerships established during programme implementation (with government, donors, civil society organizations, and other stakeholders) and lastly government commitment for PATS scale-up. The design excludes any articulated and expressed exit plan or sustainability strategy. Some references are made to sustainability in PCAs, though they appear more of a requirement, as none of the IPs, the evaluators interacted with, shared evolution of exit plan and its implementation. The exit strategy is neither defined nor possible stakeholders identified for sustainability, possible scale-up or even moving up the sanitation ladder. The implication of lack of an expressed sustainability plan or strategy is that there is no standard approach that can be followed and put in place for and by the various IPs. Moreover interventions and project aspects that could otherwise be brought to sustain may be affected by the omission of a focused and planned effort which would have been followed given a strategy/plan had been articulated, put in contract and monitored over the course of the programme. A further implication is leaving the community with unanswered questions e.g. who will continue the work? How other poor people will get access to a facility who did not receive it in the current phase? The programme has however contributed to development of training and IEC materials, latrine guides etc., which are available with partners. The applied

[69]

In response to a question on sustainability and continuity of

WASH Club work, one of the students from Dera Ghazi Khan said

‘ I am graduating to the 6th Standard and will be going to a new school but will continue to share my learning with the children over there

and also volunteer to be part of their Wash Club’.

In another instance, children imposed on their mother to wash her hands before she kneaded the

wheat dough for making roti.

knowledge acquired through practical application of PATS approaches; principles and standards will likely sustain and be used to further build on. In terms of stakeholders, several IPs have expressed interest to continue working in sanitation sector, though this is at their individual level and not attributed to efforts for sustainability through the programme. Moreover continuity and scale of interventions will be dependent on resources available to IPs. Most of the Programme Staff and SOs have also moved on to other programmes; however they will remain available in the market for other WASH programmes, drawing on their SPSP experience. CRPs are referred to as a cadre of human resource for sensitizing communities on the adoption of improved sanitation and safe hygiene practices by PATS. This cadre has been successfully created by the programme, and while all may not continue, feedback from most is that they will continue to raise awareness as it is a social good and in the benefit of their communities. With time this cadre of local resource persons may either themselves or through functional VSCs work on larger community development agenda, including water and sanitation. VSCs have developed CAPs and there is some sense of communal benefit for VSCs to continue beyond end of programme, their ability to sustain as VSCs/CBOs has not been strengthened and most of these VSCs are likely to either disband or become dormant. However the knowledge and experience gained by individual members will still be there and future interventions may be able to revive or strengthen them for developmental purposes. Sustainability of entrepreneurs and sanitary marts is not very likely, this is due to several reasons, first the way they were identified and mobilised (initial concept was to provide loans/grants which was dropped by UNICEF), secondly their innate nature to conduct business activities which don’t carry a possibility of risk, and lastly access to required finance. SLTS has played a complementary role to CLTS; however it is too early to say whether it will help in sustaining ODF status. However the change that has been evidenced in children’s behaviour, increased knowledge and practices will continue as well as be passed on from student to student, to household members and to community in general. Also as observed in the field they will convince their household members as well as community of the significance of health and hygiene as well as maintaining an ODF status. Longer lasting contribution will come through sustainability of Children’s WASH Clubs which depends on continued and proactive involvement of teachers (as WASH Club Supervisors). Therefore, it is important that teachers remain effective and

[70]

receive periodic refreshers. In the longer term the importance of health and hygiene needs to be incorporated into the curriculum, this can be raised by UNICEF and IPs as a policy need in the next phase. Sustainability of physical infrastructure (school latrines and wash facilities, household latrines and community water schemes) is going to have mixed results. School WASH facilities as well as community water schemes depend on continuous operation and maintenance (O&M) provisions without which they are likely to become defunct in time. At school level, O&M provision should have been institutionalised or linked with use of SMC funds through a mechanism; whilst a community based O&M system was necessary for sustainability of communal water supply schemes. Household latrine facilities are likely to sustain until their expected life span (depending on how soon the pit fills up and whether the household is able to afford another pit or cleaning of the first one). The behavioural changes amongst men, women and children, as also noted in health and hygiene section from the post KAP results, are likely not only to sustain but have a multiplier effect on the community that they interact with (including hand washing practices, construction of latrines for family (particularly women)). ODF status is likely to sustain but not at the current level, evaluators field observation as well as feedback from several FGDs suggests slippage (already visible in some villages), this is due to the fact that programme delays in the beginning left little margin for supporting the sustainability of ODF status. Most of the partnerships developed during the implementation phase are not going to sustain beyond project end, without a continued funding relationship. The implication of this will be that several of the IPs may end up not continuing their work on PATS and WASH sector unless they are able to secure funds from elsewhere, reducing the efforts to a projectized approach which lasted only one year, whereas the situation that SPSP is trying to address i.e. access to sanitation and achievement of ODF status is a much longer term effort as also reflected in the Results Framework, therefore it should be UNICEF’s effort to secure a minimum funding commitment for the entire period of the programme, moreover partnerships should continue with the IPs subject to continued funding. A further implication of operating this way is duplication of effort and costs of training IP staff each time a phase ends since the trained staff from previous phase is laid off by the IPs at the end of each phase of the programme given the uncertainty of partnership and funding. However key partnerships that were developed during SPSP Phase 1 include, DSI’s action research work that contributed to PATS learning for UNICEF, and RSPN’s role as master trainer which has been effective and may likely continue in the next phase, though by bringing on board some improvements. In terms of Government commitment to scale-up only Punjab as a province has shown commitment and has taken on the responsibility of sanitation sector with the intent of achieving ODF status. Therefore even in the absence of a stated sustainability strategy several critical areas are likely to sustain. The programme sustainability could have improved had there been more paced time to see through the post ODF status, also involvement of

[71]

school teachers should’ve been more and instead of SO’s providing the training in schools, teachers should have been used. As also highlighted in the PATS, CRPs could have a sustained role to play, particularly where they can pick up entrepreneurship or sanitation mart, given their own incentives, this area should be explored in the next phase.

6.2. Unforeseen Events Evaluation Q 5: To what extent was the project implementation affected and/or influenced by unstable, conflict or emergency situation? Programme design, TOC and PCAs lacked planning or comprehensive risk assessment of the security environment in the programme areas. Fortunately nothing significant happened that would have derailed the programme. IPs knowledge of local communities and recruitment of local staff helped in smooth running of activities. However there is a need to have a Risk/Disaster Strategy in place in future programme design/PCA. Although the programme was focusing on working in flood and security affected areas, the programme design, TOC as well as PCAs lacked any planning or comprehensive risk assessment of the programme areas and overall security environment. There was no detailed natural disaster risk assessment done, however, the latrine construction guidelines contain safety measures to counter impacts of natural disasters such as floods and there is a section detailing DRR in WASH. The PCAs also did not have a risk assessment or mitigation measurement in the event of security problem. Fortunately nothing significant happened during the course of implementation and the programme interventions were not derailed. There were a few instances reported where IPs staff was threatened or attacked e.g. one of the ‘Prepared’ staff member was shot at, in another incident a contractor in District Kohat was unable to complete the latrine construction job due to security situation, similarly programme activities were halted due to security threats in Tank and there was also an issue with Muslim Aid not being able to obtain NOC from the government. Other unforeseen issues noticed were the delays in supplies and release of funds. Generally, even without having any strategy in place the IPs managed to respond to the situation in a manner to reduce any significant impact on the programme implementation. Feedback from the IPs suggests that they used the knowledge of local communities to make decisions on day to day basis, and in most cases having local staff also helped in ensuring smooth running of activities. IPs who had prior presence in district were better able to deal with any such incidences. Considering the above mentioned issues there is a need to have a Risk/Disaster Strategy in place in the programme design/PCAs so that it could be used at the time of implementation. The strategy should have clear protocols to respond to situations. The strategy should be made clear to all and applied from field to office. Similarly, there should be a clear policy on IP staff security, possible insurance and clarity of responsibility and duty of care.

[72]

6.3. Human Rights Based Approach Evaluation Q 6: To what extent did the project observe HRBA and/or contributed to observance of Human Rights in Pakistan as an implicit objective of the project? Ethical Programme Design The design and implementation did not formally articulated ethical considerations though they were adhered to in practice, including protection of the dignity, rights, and welfare of human subjects, particularly children, and respect for the values of the beneficiary community. It is however recommended that ethical considerations be made a formal and written part of design documents as well as implementation arrangements including in PCA documents with partners for future programmes. The evaluation TORs did not specify conducting an analysis of how ethical the programme design was, rather the focus for evaluators was limited to ensuring that the evaluation was carried out under the UNEG ethical guidelines and Guidance Note. However for completeness of the evaluation process consultants have reviewed the programme in line with ethical considerations both in terms of design and implementation. The evaluation report mainstreams human rights and ethical considerations across the report, however following sections present a detailed and consolidated view. At the design stage there is clear focus on addressing the needs of the vulnerable and extremely vulnerable. The programme also ensures that the range of primary stakeholders i.e. the beneficiaries of the programme are reached and their needs addressed through specific mechanisms and approaches for instance a mix use of CLTS and SLTS focusing specifically on the needs of the community and directly on the needs of children through school led approach. Similarly interventions and operational methods were designed so that women, men and children were approached directly so that their views and input as well as their specific needs and concerns could be addressed directly. The programme also demonstrates focus on specific issues such as disability and MHM in design though less so during the implementation stage. Process of using PRA to ensure vulnerability assessment is carried out as well as approaching community through Men and Women Social Organisers is a good practice and considerate of local context and cultural sensitivities. Moreover the use of local resource in terms of CRPs, Mart Owners and Masons ensured the involvement and ownership of the community in the programme process. Children’s involvement is clear in the SLTS process and their views and participation in the process has been ensured. There was no particular mention of violations or breach in terms of ethical considerations during interaction with community, schools and children. However, some operational oversights have occurred particularly in the areas of constructing DRR specific elevated water supply facilities and latrines, in which case certain privacy aspects (particularly for girls in schools) have been infringed, this could be addressed in the future through better planning and involvement of respective stakeholders.

[73]

Having reviewed both the design and implementation, the evaluators argue that although not formally articulated ethical considerations were adhered to in practice, including protection of the dignity, rights, and welfare of human subjects, particularly children, and respect for the values of the beneficiary community. However it is recommended that ethical considerations be made a formal and written part of design documents as well as implementation arrangements including in PCA documents with partners for future programmes. Programme Design and Implementation UNICEF uses a human rights based approach (HRBA) and works in partnership with communities, especially women and children (in planning, implementing and maintaining water and sanitation systems). The design of SPSP does not explicitly state this, however the design elements comply mostly with the HRBA and this was made part of the programme following recommendations from the RuSFAD evaluation. The design and PCA refer to gender considerations as well as demonstrated equity focus At the contractual stage, all IPs were required to follow Millennium Declaration, Core Commitments for Children (CCCs), the Convention on the Rights of the Child (CRC) and the Convention on the Elimination of All Forms of Discrimination against Women (CEDAW). At the implementation stage, expected HRBA approaches and techniques are observed as strengths of the programme, including,

- Participatory rural appraisal. - Participatory situational assessment. - Community dialogues (within villages, and with district officials). - Village Sanitation Committees (allowing to build village consensus, promotion

of hygiene and responsibility of liaising with officials). - CAP (articulating community’s development needs and creating an action

plan to move towards these). - Joint activity with programme IPs for ODF declaration and certification. - Children’s WASH Clubs, helping them take charge of their health and hygiene

conditions in the school. As already noted above this was not done through an explicit statement of HRBA, therefore there are certain weak areas where the programme could have done more and better. The programme does not use a rights based language, therefore the articulation of water and sanitation as a right is not made, nor is the emphasis clear on the roles and responsibilities of the right holder as well as the duty bearer who is responsible for provision and upholding of the right. Also the programme could have done more in linking the VSCs with Duty Bearers and could have tried to access resources (water and sanitation) in an effort to demonstrate the HRBA. Instead the programme provides most of the water and sanitation facilities through IPs and school facilities through independent contractors, in many ways not in line with HRBA principles. Moreover the staff and IPs do not seem to be fully aware of HRBA as an approach and therefore areas which could have been stronger, come across as not being addressed. One such example is the lack of policy influencing and push on providing WASH facilities in government schools through the education department. Another, mobilizing communities to advocate and ask relevant agencies to rehabilitate

[74]

dysfunctional water schemes. The limited engagement of communities in programme monitoring, involvement in identification, bidding and rehabilitation of community schemes/infrastructure also points to limited understanding (in terms of translating HRBA into activities) of HRBA amongst partners. Monitoring Arrangements UNICEF’s approach in this programme was to mainstream HRBA across the programme design and implementation framework, including even in the context of monitoring. Third Party Monitors were therefore required not only to monitor the quality of processes but also on progress in activities and outputs under the framework including aspects such as proper identification of extremely vulnerable in the community. Similarly monitoring that the PRA process was followed as per the UNICEF checklist to ensure inclusion of marginalized in the planning process. The monitoring checklist included other HRBA related indicators, which was further strengthened through the Programme Results Framework which mainstreamed income quintile based desegregation as well as gender based desegregation. Other aspects including a monitoring focus on access for the disabled. In addition to this, UNICEF also conducted the WASH Equity study which helped informed UNICEF of the public spending in all districts of Pakistan on WASH as well as access to WASH facilities in terms of income quintiles, which could be compared with baseline and end of phase evaluation figures to demonstrate increased access to water and sanitation (declared as basic rights in the various treaties that Pakistan has ratified and is a signatory of). Contribution towards MDGs PATS is the formal strategy by Government of Pakistan towards attaining MDGs related to sanitation. The programme has been successful in increasing the proportion of population having access to sanitation facility to 65% from 44% at the time of baseline in the programme areas. Moreover this achievement is higher than the current national coverage of 50% (including both rural and urban areas), thus one can argue that there has been some appreciable increase in access to sanitation facility resulting from this programme even at the national level. Child Friendly Facilities In line with the Islamabad Commitment, 2010 which was signed by all provinces relating to Child Friendly Inclusive Schools, the programme has been successful in providing child friendly WASH facilities.

[75]

The programme has been able to provide 97% of the schools with latrine facilities, with 77% having access to water for latrines (including piped as well as through bucket). In addition to this SPSP was able to form WASH Clubs, train teachers, and reactivate SMCs which should help in addressing issues to ensure the school facilities remain child friendly. Disability Addressing of disability remained a weak aspect of the programme at both communities as well as school level in terms of sensitisation as well as actual infrastructure designs and sanitation supplies. At the proposal stage the programme clearly stipulates that all activities will be designed and carried out with due regard to disability. The baseline study also highlighted the need to collect desegregated data for the disabled, additionally Third Party Monitoring mechanisms also included monitoring of facilities including assessing their suitability for the disabled. Lastly, under the SLTS component the facilities were to be child friendly including accessibility to children who are disabled. However this emphasis has not been translated into action during programme implementation. None of the school or household latrines meets the requirements for being accessible to the disabled. The hand washing points are based on traditional design which means it can be limiting for a child having physical disability. Due to DRR concerns both water as well as Sanitation facilities have been designed to be elevated from the ground, however no arrangements were made to provide suitable ramps, instead both facilities have stairs. While visiting some hand pump facilities which did have ramps (although very steep), the evaluators were told that these ramps are provided so that community can bring wheel barrows/trolleys up to the hand pump so that it is easier to carry the water, failing again to highlight any possible link to disability.

95%

64%

42%

30%

73%

77%

97%

77%

77%

0% 20% 40% 60% 80% 100% 120%

Well Lit Classrooms

Ventilation in Classrooms

Dustbins in Classrooms

Availability of Soap

Is Latrine In Usable Condition

Water Availability in Latrines

Availability of Latrine Facility

Handwashing Facilities

Availability of Drinking Water Points

School Facilities Attributes

Figure 30: School Facilities Attributes

[76]

Duty Bearers Capacity Building The programme has carried out orientation and capacity building activities with duty bearers (including district budget seminar). However they have not been focused on HRBA principles and they do not formally and explicitly hold duty bearers responsible for respecting people’s rights, or identify delivery of basic services such as sanitation and access to water as a basic right for the people. Moreover, whilst the programme created community structures such as the VSC as well as the CRPs, there has been little effort and progress in connecting them to duty bearers or political individuals to ensure a statement of rights is placed in front of them, or resources and services are demanded of them. Formal and Informal Mechanisms In some cases IPs with the cooperation of district government were able to set up District Committees to oversee the progress, implementation of the SPSP programme, however in most cases the relationship remained informally structured with IPs picking and choosing who to interact and deal with given the responsiveness of the duty bearer (interaction varies significantly from District Commissioner down to Union Council Secretaries).

The programme however encourage Duty Bearers to interact and visit the communities through the orientation sessions, opportunity to visit and certify ODF Villages, engagement with IP field staff vis-a-vis Union Council Secretaries, BHU Staff, and Govt Field Extension Workers. At the community level, CRPs and VSCs (men and women) have been developed as the informal mechanisms for articulating community needs and rights. Similarly at the school level, Children’s WASH clubs (for boys and girls) help raise health and hygiene as well as WASH issues. However the process of articulating and voicing rights needs more capacity building to ensure these mechanisms remain effective as well as sustain. Similarly more work is required in building duty bearers understanding and ability to respect, protect as well as provide for people’s rights.

[77]

6.4. Partnership and Financial Management Evaluation Q 7: To what extent the project has observed and applied UNICEF’s standard financial management practices with Implementing Partners as well as with Government? UNICEF guidelines for partner selection were mostly complied with, though the process could have been made clearer, in terms of decisions and the process used to achieve them. The partner selection nevertheless was appropriate with partners having relevant WASH experience and geographical presence. The umbrella arrangement posed operational and implementation challenges and needs to be reassessed. Consultation with Government Departments and their involvement in the programme remained weak, while issues identified during the bottleneck analysis seem not to have been adequately addressed. One of the provincial team (Baluchistan) refers to self-defined scoring system under which points are awarded during CSP mapping exercise for the review and recommendations by the committee comprising of members from different thematic areas. Evaluators were not provided any approved policy document or decision from Islamabad to legitimate the scoring system and the factors defined under mapping exercise. Evaluators were interested to see how partners were selected for SPSP project and for this, a complete list of CSPs under consideration during mapping exercise for SPSP project was acquired with final scores from UNICEF team. It was found that out of three selected partners for SPSP in Baluchistan, two organizations (BRSP and NCBP) were not part of the list provided by provincial office. On further review of the mapping list provided, there were many organizations who had obtained more scores and had geographical presence with experience with UN and Government agencies and were not selected for partnership as compared to the organization (WDCO) which was finally selected. This indicates a requirement of making more informed and well documented decision making with a clear rational for selecting and not selecting the potential partners. The guidelines provide less guidance on how the committee should have made a decision, on what basis to shortlist and select partners particularly in the event of qualification of greater number of organizations than required. The proceedings of committee deciding on final partners were not made available for evaluation. Evaluators also noted some inconsistencies in scoring vis-a-vis final selection of partners. However information was not made available at the time of the evaluation to determine how these inconsistencies had affected the programme during the course of programme implementation. To a large extent appropriate partners were selected (though no specific capacities beyond an element of WASH experience and geographical presence/outreach in the area were assessed) while filtering for partners. Moreover, evaluators were able to identify an exception to geographical presence criteria in one province.

[78]

Before signing off the PCA with NRSP, micro assessment was conducted and NRSP was rated as low risk CSP. However as per joint proposal three consortium partners (Sub Implementing Partners) were selected which are FRDP, PREPARED and Sukkar Foundation. This umbrella PCA arrangement did not consider conducting micro assessment for these Sub Implementing Partners posing a considerable risk for the project. Even the funding amount is less than USD 100,000 for each sub IP however, as a matter of best practice; thorough assessment could have addressed any legal, financial and programme risks. Records were not available to confirm what type of comprehensive legal arrangements was made by NRSP to avoid any risks as well as how the capacities of these organizations were assessed before awarding them the sub IP status. Evaluators were able to identify through TPM reporting that there were operational and programme delivery challenges with the umbrella arrangement in Sindh (NRSP as lead agency with three sub-IPs) as well as in the partnership in Baluchistan (NCBP with WDCO). Working through sub IP is more costly as at least two organizations and their

resources are involved to deliver on the project. The process on umbrella PCA

arrangements needs to be defined clearly as under what circumstances, this

arrangement can be considered, how sub CSPs can be assessed and selected,

how legal risks can be addressed in case of poor or non-performances etc.

Micro assessment methodology comes across having its own limitation, however through it; UNICEF did identify the areas of improvement. Though it seems UNICEF ended up picking and choosing among the various risk areas for interventions and no record was available to explain this selective approach and the circumstances that contributed to it. PCAs contain sufficient clauses on anti-corruption and bind the CSPs to follow the UNICEF policies. CSPs are also contractually required to follow Millennium Declaration; Core Commitments for Children (CCCs), the Convention on the Rights of the Child (CRC) and the Convention on the Elimination of All Forms of Discrimination against Women (CEDAW). However evaluators do not have any mechanism to verify what orientation and alignment initiatives have been taken to ensure that CSPs and GPs are on board and fully understand these policies and comply with in their routine operations. This fact is not checked during spot checks. Some basic level capacity building was done which helped to mitigate the project

related financial risks (including training of the finance teams and introducing

independent internal audit functions). The evaluators also found that the

recommendations made by the EY during financial micro assessment were not

fully picked up and addressed under the Risk Mitigation plans and hence there is a

strong likelihood that the same issues will be raised during audit process. There

was no specific information available to confirm how provincial level assurance

committees have made a decision to pick and choose between different

recommendations as identified by CA firm.

[79]

Government Partners Coordination and Management Arrangements UNICEF provincial offices were not able to produce enough documents where coordination meetings at provincial and district levels and issues related thereof could have been reviewed. Evaluators had to rely on the interactions with UNICEF staff and Provincial Government department’s representatives. Provincial Government departments were not adequately consulted and were not taken on-board in the programme design even to the extent that details of roles and responsibilities expected from them had not been properly shared. For example, under the SLTS component, the education department at provincial level was not on board in terms of selection of schools, teachers training and formation of Wash clubs as well as details of operation and maintenance of latrines that were constructed. While orientation on PATS was provided in the beginning of the programme, neither the government departments were fully aware of key programme activities nor was an approach agreed with them to ensure the sustainability of the programme after exit phase. Evaluators found that situation at district level, was relatively better where IPs were more active in engaging with district management. The government departments however found to have little knowledge and involvement around incentives and rewards (including expectations from relevant departments to contribute). The engagement level at district level varied from district to district and was more dependent on personal and organizational reputation of the implementing CSP. Capacity development plans for provincial level departments in 2013 were not rolled out with allocation of budgets to carry out the activities nor the bottleneck analysis that were conducted with government departments in four provinces were fed into the programme or reflected fully during programme implementation. Financial Risk Management PCAs contain sufficient clauses on anti-corruption and bind the NGO partners to follow the UNICEF policies. However evaluators do not have any mechanism to verify what alignment initiatives have been taken to ensure that NGO partners and Government partners are on board and fully understand these policies and comply with them in their routine operations. This fact is also not checked during spot checks. Limited sharing of further information makes a conclusive assessment difficult. Procurement Procedures for Goods and Services The evaluators neither have the mandate nor access to key information on procurements by UNICEF and NGO partners as this comes under the financial audit domain. However as a process, supply management assessments were carried out for NGOs as mentioned on PCA-Review Committee checklist before signing off the PCA.

[80]

6.5. Unexpected Findings The unexpected findings resulting from the implementation of SPSP Phase include unravelling of the critical link between open defecation exposes women to accusations of illicit relations and can lead to being subjected to extreme violence. Access to Sanitation Facilities and Reduction in Violence against Women The evaluation findings have contributed to unravelling a critical link between open defecation and extreme cases of gender violence. A significant number of respondents (including Government Officials, CRPs and Women) across different geographical locations (Sindh and Punjab) shared that the construction of latrines has reduced the incidences of violence against women such as Karo Kari (honour killing). To them, the construction of latrines has reduced the need for women to step out, and hence reduced the probability of being accused of illicit relations and being subjected to violence. In a narrated instance a mother married off her daughter to a mentally handicapped person in order to save her daughter from Karo Kari. There is other emerging evidence for instance in May 2014, two girls were gang-raped and hanged in Uttar Pradesh, India where they had gone out to look for a place to defecate. A WaterAid study in the slums of Lagos in Nigeria also showed that a quarter of women, who lacked access to sanitation, had first or second-hand experience of harassment, threats of violence or actual assault linked to their lack of a safe, private toilet. Amnesty International has released similar studies from Kenya and the Solomon Islands. Results of the SPSP Phase 1 post-KAP survey suggest that 83% of the respondents feel it is unsafe for women to go out to defecate in the open. Though an indirect result, it is a significant aspect and therefore should be incorporated in a considered way into the evolution of BCC strategies and demand creation interventions. Shame, Shock and Disgust versus Pride, Self-Respect and Dignity The two approaches used for triggering sanitation demand, however normally have not been used together. SPSP Rural I, by using them together in certain order (where shame, shock and disgust follows the other), has demonstrated that if used together these two approaches could help deliver better results. Business Incentive and Opportunity Is a Larger Motive for Response. DSI’s action research suggests that manufacturers, entrepreneurs and mart owners (with business risk bearing attributes) see supply to the underserved areas as an opportunity to expand their business further. Given purely business incentives, they are more likely to sustain sanitation supply in order to meet the heightened demand and maintenance requirements in communities. Hence, it could be argued that the programme may benefit more, if it may focus on identifying and working with manufactures/entrepreneurs, with a clear intent to help them realise the real market potential and opportunity offered by these underserved areas for business continuation and expansion. This in the long run shall contribute to ensure technological innovation, continued and sustained supplies facilitation.

[81]

7. CONCLUSIONS & LESSONS LEARNED

The evaluation concludes that the SPSP Phase 1 was implemented successfully though with varied levels achievements in terms of programme components, interventions and results. This conclusion is made based on observations from field visits, discussions held with beneficiaries in visited villages, government staff and IP’s in the four provinces and FATA as well as from consultations at national level, desk reviews and comparison of the Pre-Post KAP. Under the programme, the poor and extremely poor communities in the rural areas benefitted from access to sanitation services, improved access to water, health and hygiene promotion and through supply side sanitation interventions. The programme maintained a gender and vulnerability focus which is demonstrated in operations and results. CLTS and SLTS were found to be complimentary to each other while community mobilisation training and capacity building proved effective in achieving ODF declaration status. The field interaction and involvement of government representatives during the ODF verification/certification process comes across as a strong point. However the programme could have developed stronger and better engagement with government departments with particular focus on drawing down public sector funds from provincial to district level and ensuring that communities received awards/ incentives following ODF verification/certification. The presence of a well-defined monitoring and evaluation system since the beginning of the Programme resulted in improved quality of progress reporting in terms of adequacy and consistency of information as well as informing management decisions and course correction during implementation. The programme also contributed in developing WASH training manuals, health and hygiene promotion material and conducting action research. Moreover, the baseline and evaluation studies commissioned during the programme have created evidence and provided data that can be used to inform future programme direction as well as contribute to review of PATS. Future programme would however need to focus more on modality of subsidy provision, methods of demonstration latrines, sanitation marketing (particularly developing sanitation marts and entrepreneurs), and availability of low-cost technologies for sanitation. The evaluation identifies lessons and proposes recommendations that the programme can take into consideration while designing the next phase or future programme. Overall the experience of SPSP Phase I confirms that PATS can be implemented on a large scale with key requisites such as increased capacity of and advocacy with government, sector stakeholders and communities to apply PATS approach effectively.

[82]

Lessons Learned Following four key lessons emanate from having implemented the programme, which may have broader implications and also help add to the general understanding and body of knowledge around these issues (both in the context of future WASH programming by UNICEF or others). These could also inform planned PATS review and UNICEF CATS approach. Access to Sanitation Facilities Has a Role in Reducing Violence against Women The evaluation has contributed to unravelling a critical link between open defecation and extreme cases of gender violence in rural Pakistan. A significant number of respondents shared that the availability of latrines inside the house has significantly reduced the incidences of extreme gender violence such as Karo Kari (honour killing). To them, the construction of latrines have reduced women need to step out (as for defecation they were coming out at odd hours), thus the protection risk (or probability) of being accused of illicit relations (with men), and consequently being subjected to extreme violence. The lesson that is derived from this instance is that though an indirect result of WASH intervention, the benefit of contributing to a reduction in violence is a substantially important aspect and should inform the future programme design and delivery. Shame, Shock and Disgust versus Pride, Self-Respect and Dignity The two approaches used for triggering sanitation demand, however normally not been used together. SPSP Rural I, by using them together in certain order (where shame, shock and disgust follows the other), has demonstrated that if used together these two approaches could help deliver better results. Business Incentive and Opportunity Is a Larger Motive for Response. DSI’s action research suggests that manufacturers, entrepreneurs and mart owners (with business risk bearing attributes) see supply to the underserved areas as an opportunity to expand their business further. Given purely business incentives, they are more likely to sustain sanitation supply in order to meet the heightened demand and maintenance requirements in communities. Hence, it could be argued that the programme may benefit more, if it may focus on identifying and working with manufactures/entrepreneurs, with a clear intent to help them realise the real market potential and opportunity offered by these underserved areas for business continuation and expansion. This in the long run shall contribute to ensure technological innovation, continued and sustained supplies facilitation. Using Subsidy to Address Equity Can Lead to Dependency To a degree the subsidies have served the equity purpose. However, the subsidy planning and delivery mechanisms faced challenges, which need deeper thinking and innovative solutions. The solutions may need to take note of the evaluation finding that such incentives contribute to reducing willingness of families to make financial contributions to construct latrines.

[83]

8. RECOMMENDATIONS

Following are key recommendations along with priority and identified stakeholder(s). The recommendations were drawn through a comprehensive and systematic process, including drawing on lessons learnt by the stakeholders (UNICEF, DSI, RSPN, Range of IPs, other development partners, government representatives as well as community representative) and their propositions that could improve the programme. Following this process the evaluators were able to consolidate key recommendations and present to the Evaluation Reference Group in a plenary session (including UNICEF representative from Regional Office, Islamabad and Provincial Offices, Government Officials as well as Donor Representative) during the two day workshop. The recommendations were then discussed within smaller groups, arguing their respective importance, clarifying their objective and then agreed through a mutual process with agreement of the Evaluation Reference Group.

No. Recommendations Priority Stakeholder

1. Informing PATS Policy

UNICEF has garnered PATS related experience through implementation of RuSFAD and SPSP (for instance whether to use Respect and Dignity or Shock, Shame and Disgust method), moreover lessons and findings are also coming out of the Action Research carried out under SPSP by DSI (suggestive that involving manufacturers with clearly identified incentives and opportunities goes a longer way in facilitating sanitation supply). Similarly other development partners such as WaterAID have tried other variations (for instance not offering household level subsidy) while interpreting and implementing the PATS approach. It is therefore recommended that UNICEF commission a joint research (involving relevant stakeholders including other development partners) to study the various PATS models being implemented in Pakistan and the adaptations that have been brought about to ensure ODF status. The research should also highlight the differences in approaches due to contextualising of cultural and geographical considerations and sensitivities such as found across the Provinces. This research should thus be used to inform and revise the PATS document and depending on scheduling be showcased in the next PACOSAN. Dissemination of the study would also help bring attention to PATS and the effort to make progress towards MDG targets.

Medium Term UNICEF, WASH Sector Development Partners, CCD and relevant Provincial Departments

[84]

No. Recommendations Priority Stakeholder

2. Knowledge Management

It is clear from the SPSP evaluation as well as from interacting with the various stakeholders that there is substantial and good work being done on PATS (e.g. WaterAID, plans of WSSCC as well as work being carried out by Punjab). This work is also being documented and adding to the existing knowledge base. It is therefore recommended that UNICEF takes a lead role (along with other development partners) to ensure that an online repository is created, which makes programme documents, evaluations, action research etc. relevant to PATS programmes (including those generated during SPSP Phase 1, as well as from RuSFAD) widely available. The repository should provide open access, be user-friendly and allows authorised people to upload documents.

Immediate UNICEF (along with WASH Sector Development Partners)

3. Harmonisation & Coordination

From the interaction with various stakeholders as well as based on the evaluation findings it is clear that the sanitation sector critically lags behind other sectors in terms of harmonisation as well as coordination mechanisms. It is recommended that UNICEF should use its stature and leverage its relations with other development partners, Federal and Provincial governments to ensure sector wide coordination forums at national and provincial levels are formed/made active for advocacy, policy making and resource mobilisation/allocation. Furthermore for coordination at project level, it is recommended that Provincial and District Coordination Committees be formed/notified through government with PHED/LGRDD as lead to leverage provincial and district level support and ownership, in cases where similar committees exist they may require reactivation and reconstitution to make them functional.

Medium Term UNICEF, WASH Sector Development Partners, CCD and Relevant Provincial Departments

4. Policy Influencing & Advocacy for Government Allocations

Based on the evaluation findings following things are evident, a) drawdown of government resources remained weak even though it is considered intrinsic to

PATS that communities are linked to Government Departments at district and sub-district

Immediate UNICEF

[85]

No. Recommendations Priority Stakeholder

level for continued work in achieving and maintaining ODF status, for which government resources should also be mobilised,

b) also the scale and magnitude of the sanitation challenge in Pakistan necessitates the increased involvement of Government along with the will to ensure scale-up and sustainability.

It is therefore recommended14 that UNICEF should use its position and influence and work top to bottom from province to ensure budgetary provision for awards, incentives and rewards to departments at the district level. A standardised approach and methodology should be adopted by UNICEF in Islamabad as well as the Provincial Office for policy influencing at provincial level and implementation at district levels (with variations to suit provincial or geographic context). This should be based on written MOU or agreement between UNICEF and Government Departments. In SPSP Phase II, UNICEF should pilot PATS implementation with Public Stakeholders in at least one Province (which is receptive and committed to addressing sanitation issues).

5. Integration with Other Sectors It is evident from the evaluation as well as through interaction at the ground level with Government Departments that to achieve sanitation targets, engagement and committed action on a multi-stakeholder basis will be required. There is a clear need to advocate mainstreaming of sanitation across other sectors (including Health, Nutrition, Education and Government Departments). This should include cross sector convergence on WASH policy and programmes, which can include advocating for hygiene to be led by Health Department, while being mainstreamed into the education curriculum. Moreover, it can look into options for improved usage of SMC funds while ensuring regular monitoring of Schools including monitoring of WASH facilities to identify and address defunct facilities. It is therefore recommended as a minimum that UNICEF engages with these departments both at the Provincial and District level, while ensuring that they come together (it is notified as part of SPSP

Medium Term UNICEF, WASH Sector Development Partners and Relevant Government Departments

14 The evaluators note that this is an ambitious recommendation, however it is considered necessary for scale-up and sustainability so that in the longer term progress can be made at the national level in meeting MDG Target.

[86]

No. Recommendations Priority Stakeholder

Phase 2 agreement with government) at the provincial level as a Coordination and Steering Committee headed preferably by PHED, and at district level as an Implementation Committee headed and reporting to the District Administrative head. This would also allow other stakeholders to use similar government structures for better coordination and implementation of their respective WASH programmes.

6. Project/Programme Duration The evaluation findings and the two-day workshop with the Evaluation Reference Group highlighted that there was no specific time framework (minimum incubation period) for PATS. Examples from different provinces as well as IPs ranged from as short as 3-4 months to 6-9 months. However based on substantial and in-depth discussion a clear recommendation for future programming was agreed which should take into account and provide adequate duration for the following phases of the programme,

- Start-up/inception phase (PCA signing, hiring of staff, IP inception workshops etc.) - Implementation phase of 12 months or more for CLTS and SLST (based on time required for

effective community mobilisation, as well as factoring in community life/work cycle aspects, school holidays etc.)

- Exit phase (should include exit planning, sustainability of interventions as per programme strategy, and post ODF monitoring).

Immediate UNICEF

7. Gender Equality, Equity, HRBA (Disability) While SPSP Phase 1 was clear in mandating and focusing on gender equality, equity and disability, evaluation findings suggest more clarity; working mechanisms and definitions need to be articulated, along with describing the process that will ensure these aspects are factored into the programming and implementation. It is therefore recommended that clear processes need to be articulated in the programme design, the TOC should explain the change process and results, while PCA should include sections detailing specific interventions, criteria and processes that IPs have to implement ensuring inclusiveness (such as for PRA, School WASH Facilities, more targeted BCC, disability integration in Latrines) in line with UNICEF guidelines to address gender equality, equity and vulnerability along with special needs of the disabled.

Immediate UNICEF, Relevant Government Departments at Provincial and District Levels.

[87]

No. Recommendations Priority Stakeholder

8. Exit Strategy & Sustainability

The evaluation highlights the lack of clarity about the mechanisms and time duration that is required to ensure a smooth exit and put in place sustainability mechanisms.

It is recommended that for SPSP Phase 1 implementing partners a minimal operational support for six months is provided in the districts of operation to ensure gains of the implementation phase (ODF Status - declaration and certification) are not lost to slippage. For the next phase and future programmes, it is then recommended that a Theory of Change needs to be articulated, and a Sustainability Strategy drawn in line with it, focusing on sustaining ODF status (for which a study can also be commissioned examining what needs to be done to sustain ODF status). PCAs should include specific IP exit plans, and those stakeholders who will be ultimately responsible for sustainability should be clearly informed of their role along with their consent to deliver it (CRPs, VSCs, Wash Clubs as well as role expected from Government Departments and their extension workers such as Community Development Officers, Union Council Secretaries, LHWs and School Teachers etc.).

Immediate UNICEF

9. Training & Mentoring

Findings suggest that using a central training organisation proved effective; though with some constraints which if addressed should help make it even more effective. It is therefore recommended that the programme should continue the use of a central training organisation, however the level of resource provision, staffing should be according to the magnitude of activities over the duration of the programme. Mentoring should be need based and resourced from the training organisation based on IPs need or as identified through progress monitoring. Training events should be announced in a timely manner to allow maximum attendance and should benefit all who are relevant, while taking into account aspects such as community work cycle and availability.

Immediate UNICEF and Training Partner Organisation

[88]

No. Recommendations Priority Stakeholder

10. Supply Facilitation Sanitation Mart Owner/Entrepreneur Distinction between Sanitation Mart Owners and Entrepreneurs should be reassessed and if required dropped. Innovative results emanating from Action Research carried out by DSI need to be examined to evolve an effective role for entrepreneur (including the element of risk taking, market forces of demand and supply, as well as issue of financing). Community Resource Persons (CRPs) The concept of service fee for Community Resource Persons (CRPs) worked and should continue for the period of mobilisation. Options should be considered to utilise these resources for sustained focus on sanitation by transforming them into small entrepreneurs and help expand the capacity of the market to supply services and products (this could be made possible by giving them a reward once they have achieved their agreed targets as an incentive). Household Subsidy Household subsidy could either be withdrawn and other mechanism such as support with material through voucher system in post humanitarian crisis situation and collective incentives can be tested. For equity purposes and the needs of the vulnerable revolving credit/contributory fund can be considered as an option to be considered managed by the VSC to ensure the sanitation requirements of vulnerable are also met, this can be done through options such as setting up revolving credit with the VSC.

Immediate to Medium Term

UNICEF (with advice of DSI, RSPN)

11. Low-Cost Sanitation and Drainage

More options for low-cost sanitation need to be explored. This should include options for low cost pits (possibly using low cost pit lining, increased use of local materials), as well as for superstructures (which should make use of local materials while ensuring usability of the facility by the target community). It is recommended that options for low cost end of pipe solutions should be identified (if required a study be conducted to inform options) and new ideas piloted in some locations given the demand for drainage articulated by community during SPSP Phase 1.

Continuous UNICEF

Table 19: Evaluation Recommendations

[89]

Annex 01 Terms of Reference

Technical Assistance for SPSP-Rural End of Programme Evaluation

1. Programme information:

Commissioning office: UNICEF Paksitan - WASH Programme, Islamabad Evaluation Title: UNICEF’s Sanitation Programme at Scale in Pakistan (SPSP) – Rural

Through Pakistan Approach to Total Sanitation (PATS) 2. Background of the Programme:

In Pakistan, diarrhoea is the leading cause of mortality for children under 5 (PDHS 2007-8) where 116,013 children under the age of 5 die due to diarrhoea each year, translating into the loss of life of 13 Pakistani children per hour (PDHS 2007-8). Children also suffer disproportionately from sickness due to diarrheal disease with almost 25 million case (PSLM 2006-7) reported annually. Major reasons for diarrhoea are lack of access to clean drinking water, poor hygiene practices and, poor sanitation. The already dismal situation of sanitation in Pakistan was further deteriorated by the devastating floods of 2010/2011/2012 which affected at least 20 million people across the country, some of them twice or thrice consecutively. The 2012 heavy monsoon flooding affected an estimated 4.8 million people in the provinces of Punjab, Balochistan and Sindh according to the National Disaster Management Authority (NDMA). Sectarian and tribal violence, violence by other non-state entities and military operations in the Federally Administered Tribal Areas (FATA) have caused displacement. There are about one million people registered as displaced who have still not returned to their area of origin. Most of the displaced people in flood affected districts have returned to their damaged or destroyed homes, infrastructure and livelihoods while most of the insecurity affected displaced families are still sheltering in relief camps or with relatives. UNICEF with the Government of Pakistan, through its partner organizations, launched a large scale Rural Sanitation Programme in 2010 - flood affected and high polio risk districts (RusFAD) to mitigate the grave needs of people related to sanitation. The program was implemented in three phases i.e. Phase I, II and III. In 2012, based on learning from the successful implementation of the RusFAD programme, UNICEF with the Government of Pakistan and its Implementing Partner (IP) organizations launched the ‘Sanitation Programme at Scale in Pakistan (SPSP)’ for 2012 - flood affected and high Polio risk and insecurity affected areas. The first phase of SPSP was implemented in fourteen districts to reach 1.3 million people of Punjab, Sindh, Khyber Pakhtoonkhwa, FATA and Balochistan aiming at by 2017, most vulnerable and marginalized children and women are using safe drinking water, improved sanitation and hygiene practices. The National Sanitation Policy of Pakistan, approved by the Federal Government in 2006, envisions the creation of an open defecation free environment with safe disposal

[90]

of liquid and solid waste and the promotion of health and hygiene practices in the country. Further to this policy, a holistic vision of sanitation was developed through an integrated Total Sanitation model resulting in the formulation of a Pakistan Approach to Total Sanitation (PATS) as a comprehensive strategy for promoting improved sanitation and hygiene outcomes in peri-urban and rural areas that focuses not just on ending the practice of open defecation and transforming hygiene behaviour, but also sanitation marketing, waste water collection, solid waste disposal and drainage (Government of Pakistan, Ministry of Environment, 2008). Operationalizing PATS, sanitation demand is created by applying Participatory Rural Appraisal (PRA), Community Led Total Sanitation (CLTS), and School Led Total Sanitation (SLTS) tools. It focuses on respect, dignity and pride as motivations for change rather than the conventional shock, shame and disgust methodology usually employed by CLTS projects. The programme also promotes the use of safe hygienic latrines and encouraged improved hygiene behaviour through Information, Education & Communications (IEC) campaigns while supporting the establishment of markets for low-cost sanitation goods and services. Trained masons demonstrate the construction of latrines to showcase low cost options. The strategy also includes pilot projects to demonstrate decentralized constructed wetlands for attaining 100 per cent drainage and wastewater treatment in select villages in all provinces. The conceptual framework of project and activities within the six pillars of PATS are illustrated below, which can be helpful to the evaluators in developing a Theory of Change for the evaluation mission.

Sanitation

Demand

Creation for

ODF

Communities

Technical

trainings of

masons

Mobilizing

communities

through school

teachers

/children and

social organizers

adapting a

cascading model

Launching

localized demand

creation C4D

campaign ( based

on KAP &

Formative

Research)

Construction of

low cost Demo

Latrines

(environment

friendly, gender

appropriate,

safe)

Facilitating

sanitation marts

and trainings of

sanitation

entrepreneurs

Develop

microfinance

options

Provide

incentives and

subsidies (7% +

3%)

Locally

customized C4D

material Packs

developed on key

health and

hygiene

messages with a

communication

strategy

Mass Media,

Community,

School children,

Religious and

Political leaders

led thematic

campaigns

Promotion of

low-cost

appropriate and

informed

sanitation

solutions

Ensuring adequate

drainage design with

the support of PHED

Advocacy for

wastewater

treatment through

Oxidation

ponds/Constructed

Wetlands

Advocacy for

underground

/covered drainage

system

Advocacy for waste

water disposal in a

hygienic way

Demonstrating

drainage and

wastewater

treatment in

selected villages

Sustaining

Demand

through

Supply Side

Interventions

Participatory

Health &

Hygiene

Promotion

Attaining

100% Adequate

Drainage and

Wastewater

Treatment

Linkages

development

with duty

bearers

Knowledge

Management and

Accountability

Capacity

development of

duty bearers to

enhance their

understanding

and ownership

of PATS

Advocacy with

duty bearers to

take policy

decisions in

support of

PATS

interventions

(e.g. selection

of intervention

areas, M&E,

provision of

rewards and

incentives)

Learning PATS:

Research, Inception &

Lessons sharing

Monitoring:

Baseline, Endline

Pre/Post KAP

UNEG Evaluation

Monthly Process

Monitoring,

Mid term review

workshop

C4D/Institutional

Learning:

Replication and

institutional memory

(Case studies, Human

interest stories ,

Technical papers,

Documentaries, online

PATS DMS)

Formation of

Village Sanitation

Committees

Development of

Community

Action Plans

(PRA, Transsec

Walk)

[91]

3. Purpose of Evaluation

The major purpose of the assignment is to conduct a final evaluation of the process outputs and outcomes of SPSP-Rural taking into account UNICEF guidelines for programme evaluation following OECD Development Aid Criteria. The final (end term) evaluation will examine the results of this programme with the purpose not just to to assess and review the outputs, outcomes and impact, but also to capture the lessons learnt in order to make use of them for recommending improvement in the evidence-based planning and decision-making in similar future initiatives. A large scale Rural Sanitation Programme in 2010 was launched in flood affected and high polio risk districts (RusFAD) to mitigate the grave needs of people related to sanitation. The programme was implemented in three phases i.e. Phase I, II and III. In 2012, based on learning from the successful implementation of the RusFAD programme, UNICEF with the Government of Pakistan and its Implementing Partner (IP) organizations launched the ‘Sanitation Programme at Scale in Pakistan (SPSP)’ for 2012 - flood affected and high Polio risk and insecurity affected areas. The first phase of SPSP was implemented in fourteen districts to reach 1.3 million people including most vulnerable and marginalized children and women. As the project is ending, now, is the most pertinent time to undertake an end of project evaluation in order to assess the outcomes for safe drinking water, improved sanitation and hygiene practices. The evaluation is expected to evaluate achievement of the planned results, whether processes were efficient and participatory and also assess UNICEF’s specific role and contribution and propose measures to increase relevance, effectiveness and efficiency of the revised PATS approach in order to strengthen the PATS strategy in the context of working in partnership with other stakeholders and draw lessons to direct future programme efforts. The primary audience of the evaluation are the DFID, UNICEF staff; IP staffs and government officials who are involved in programme re-design at next stage. The secondary audiences are the WASH sector partners and communities who can learn how areas of improvement in this programme can be adopted and best practices replicated elsewhere while taking sanitation programmes at scale. It is therefore very important to capture both analytical critique as well as best practices of the project. 4. Evaluation Objectives Specific evaluation objectives include:

I. Determining the relevance, effectiveness, efficiency, impact and sustainability of UNICEF SPSP-Rural Phase I programme (as per UNEG standards given under section 10);

II. Assessing the extent to which UNICEF core programme strategies; i) capacity, development, ii) service delivery, iv) advocacy, v) strategic partnership, vi) knowledge management and vii) communication for development were met;

III. Assessing how UNICEFs normative principles, specifically i) equity and human rights based approach, ii) gender equality mainstreaming and iii) environmental sustainability were reflected in the programme approach.

[92]

IV. Identifying bottlenecks and success factors for programme implementation including a detailed analysis of key determinants or factors leading to achievements, taking into account the framework conditions of the programme as part of Early Recovery following the 2012 floods (Humanitarian – Development Nexus);

V. Commenting on the constraints and opportunities of programme implementation under unstable and emergency situation;

VI. Assessing effectiveness of i) strategic partnership for process monitoring (Third Party Monitoring by Apex and other technical monitoring Contractors) and ii) collaborative relationship for programme implementation (RSPN for Training);

VII. Providing recommendations to strengthen PATS programme operational implementation to further scale up rural sanitation in Pakistan.

5. Scope of Evaluation

While reviewing the overall PATS programme approach, the following components relating to UNICEF’s efforts in improving its effectiveness and efficiency in implementing SPSP-Rural based on the learning of Phase III RuSFAD and its change in methodological approach shall be given priority:

‒ Improved institutional linkages & capacity development for governmental counterparts and implementing partner

‒ Campaigning for improved hygiene behaviour and polio eradication

‒ Attaining total sanitation

‒ Knowledge management

‒ Effectiveneness of SLTS

‒ Subsidy for the extremely vulnerable and vulnerable households

In the light of the above objectives, the evaluation shall comprise of a i) representative assessment of post KAPs prevailing in the targeted communities of SPSP-rural. In addition, the evaluation will also include establishing a ii) end line against outcome and output level indicators given in the ‘results framework of SPSP-Rural’ and a iii) capacity assessment of SPSP-rural non-governmental and governmental stakeholders, iv) PATS related bottleneck analysis for each province, using the OECD/DAC criteria. UNICEF intends to enter into an agreement for twelve months for 30 work days. Contractor would be required to start the assignment in mid-January 2014, The end of project evaluation will be limited to project areas covered by SPSP-Rural in its first phase including fourteen (14) districts of Balochistan, Punjab, Sindh, KP and FATA, targeting about 1.3 million beneficiaries in all selected districts of Sindh (Jacobabad, Kashmore and Shikarpur districts), Punjab (DG Khan, Rajanpur districts), Balochistan (Jaffarabad and Naseerabad districts) and Kyber Pakhtunkhwa (DI khan, Tank, Kohat, and Kohat) and FATA (Mohmand Agency, Kurram Agency and Bajaur Agency).

[93]

6. Evaluation Methodology

The evaluation will use both qualitative as well as quantitative approaches and inform itself by triangulation of data. Quantitative design will be used for statistically assessing the situation in SPSP-Rural targeted communities against outcome and output indicators in the results framework. After programme implementation, the qualitative design will give an in-depth understanding of the context and common perception, concerns and aspirations. The qualitative approach shall include focus group discussions and interviews with key implementing partners’ staff, community leaders, local government officials, group of men and women in villages to analyse existing information and comment on their capacity to implement SPSP-Rural in relation to base line findings. In order to uphold the principles of equity, purposive sampling shall be used to ensure participation from all sociocultural groups. Relevant data collected under this evaluation shall be disaggregated by gender and socio-economic (income quintile and vulnerability) status. The data shall be collected from a number of beneficiaries (to be determined later) which will enable a statistically representative sample, at household level, stratified by two (2) levels: male/female and districts (14 districts) to help make results statistically valid and reliable with confidence intervals around the means. For districts with small population size the sample should not be below 3%. The evaluation team will review already existing end of programme questionnaire. Additional questions shall be included to address the objectives of this evaluation, as outlined above, to gather relevant field data within the sampled households. The following strategies for data collection and analysis shall be applied:

i. Desk review of relevant existing information sources, project documents,

partner’s assessments and databases and monitoring reports.

ii. Discussion with WASH and Evaluation staff of UNICEF and implementing

partners

iii. Interview with key informants

iv. Field visits to targeted villages observations.

v. Workshop for capacity assessment guided and structured interviews and focus

group discussions with implementing partner’s staff, beneficiary communities

including women, men, children, district and provincial govt.

vi. Household survey questionnaire to gather quantitative data.

vii. School Survey questionnaire for students, school teacher and head

master/mistress to gather quantitative data.

viii. Workshop for validation of draft findings.

Following documents will be provided to the Evaluation Team:

[94]

i. Programme Results Framework, incl. baseline indicator for outcome and output of SPSP rural programme

ii. Work Plans of project/activities flow chart iii. Questionnaires for Pre-KAP study for SPSP rural; and post-KAP study for

RuSFAD iv. Final draft evaluation report RuSFAD Phase III v. Pre-KAP and Baseline study reports incl. results of bottleneck analysis and

capacity assessment vi. Methodology on how to conduct a bottleneck analysis vii. Copies of relevant sections from PCAs with IPs viii. UNICEF Evaluation Guidelines and supporting documents for evaluation ix. UNICEF branding template x. MoRES Approach xi. Relevant UNICEF policies

All interviews, field visits, focus group discussions, questionnaires and meetings are to be used to update information in relation to baseline conducted at the beginning of SPSP-rural including local sources of knowledge about contextual factors which might influence outcomes later. 7. Evaluation Criteria and Questions

The evaluation will be guided by OECD/DAC criteria and will strictly follow the UNICEF adapted UNEG standards covering the following questions related to the criteria of relevance, efficiency, effectiveness, impact and sustainability:

Efficiency The evaluation will:

Assess the efficiency of the project by comparing to the allocated resources with results obtained.

The evaluation should present a measured opinion on whether the results obtained justified the costs incurred.

Examine delivery of outputs in terms of quality and quantity. The evaluation will give a considered opinion of the total scope of work was as could be expected in view of the inputs and whether the initiatives have produced high quality results and outputs.

Effectiveness The evaluation will:

Assess the programme design as outlined in original documents and in view of the experience with implementation to consider whether the design from the outset could have offered more in terms of efficiency and effectiveness for the programme. The inception report will include a Theory of Change and will give a measured option as to how clear was the programme design in articulating Theory of Change (how change comes about) and how effectively was it used to achieve desired outcomes.

[95]

Assess the implementation as experienced in practical terms to see if the methods applied worked as intended. Special emphasis will be given to considering whether the programme outputs presented in the original plan/proposal were successfully and effectively produced.

Examine whether all parties involved in implementation had the necessary professional prerequisites, experience, expertise and attitudes to produce planned results in an effective and timely manner.

Relevance The evaluation will:

Consider the intended outputs and form an opinion on whether these have practical relevance to the overall goals as set out in the programme document. If so, specify what this relevance is and, if so, how this can be presented to a larger audience.

Consider what lessons could be drawn from the experience with this programme?

Evaluate the strategies used in this programme in light of national (GOP) and other policies and initiatives related to PATS.

Assess whether the GoP and its partners have a clear understanding as to how to harmonize support with the national development priorities as well as other organizations working on similar issues?

Sustainability The evaluation will:

Assess political will on part of GoP and other partners to sustain such an initiative without external assistance (ref. national policies and political discourse at the policy level).

Other related concerns:

Cross-cutting themes (Gender and Equity):

View the programme through a gender lens and consider whether all aspects of UNICEF policy relating to gender issues in development projects have been adequately handled in this initiative or sub-project related to this programme.

Determine whether programme used equity approach and how far it was successful in achieving results related to UNICEF’s equity goals for children and women across all regions and sociocultural groups.

Consider whether the prgramme observed HRBA or contributed to observance of Human Rights in Paksitan as an implicit objective of programme

Impact of unforeseen events/circumstances on programme:

View the influence of unstable and emergency situation on programme implementation.

Financial and Risk Management:

[96]

Measure and analyze influence of unstable and emergency situation on programme implementation

Give a measured opinion on whether the project(s) has dealt efficiently and effectively with all matters related to financial risk management.

Consider whether all matters related to financial management, including procurement procedures for goods and services, have been appropriately handled in light of the UNICEF’s zero tolerance to corruption policy.

Consider all aspects related to the programme’s cost effectiveness and evaluate whether alternative implementing methods could have been applied to increase the cost effectiveness.

Give an opinion on possible strengths and weaknesses in UNICEF/GOP management routines and administrative procedures for implementing the project effectively.

Consider whether UNICEF/GOP have networked and consulted sufficiently with other relevant agencies in the process of programme formulation and implementation.

The consultant is advised to take into account the following aspects in refining and devising evaluation questions in consultation with UNICEF and other key stakeholders during the inception workshop with a view:

To provide all parties/stake-holders with an understanding of the current achievements, successes and areas for future improvements.

To gain an understanding as to whether the programme was optimally implemented with regards to efficient and effective use of available resources.

To assess the programme’s contribution towards creating and maintaining gender equity among the implementing partners and the target beneficiaries;

To assess the implementing partners’ capacity, monitoring mechanism and orientation on applying a result-based approach as well as ability to documenting results;

UNICEF role in supporting the initiative and suggest recommendations for future better support mechanisms.

8. Major Tasks and Deliverables

No. Major Task Deliverables

1 Inception workshop Inception report (D1)

2 Statistically representative assessment of post KAPs, by sex and districts, prevailing in the target communities of SPSP-rural.

Revised post KAP Tools (D2) Populated SPSS database (D3) Post KAP report for statistically relevant number of beneficiaries (D4)

3 Data on SPSP-rural outcome and output level indicators

Output and outcome report for statistically representative sample at household level including interviews and verification of data submissions by IP (D5)

4 Capacity assessment of SPSP-rural non- government and government

Workshop with provincial governmental stakeholders to identify improved capacity

[97]

stakeholders based on a capacity assessment tool.

to deliver basic WASH services in rural areas (D6)

5 PATS related bottleneck analysis for each province.

Bottleneck analysis conducted and documented during same workshop in each provinces (D7)

6 Evaluation as per UNICEF/UNEG criteria

Included in final Evaluation Report (D8)

7 Assessment of effectiveness of i) strategic partnership for process monitoring and ii) collaborative relationship for programme implementation

Included in final Evaluation Report (D9)

8 Reporting Draft report submitted for comments

Final End of Programme Evaluation

Report using UNICEFs branding

template for reports (D10) while giving

consultancy which conducted the

evaluation sufficient visibility (logo)

ppt in UNICEF with government logo

branding template summarizing main

findings (max. 15 slides) (D11)

9. Reporting Standards

The Final Evaluation report outline MUST be based on UNICEF evaluation report standards. Inception Report The evaluation team will prepare an inception report before going into full-fledged evaluation exercise in the field. The inception report will follow the guidelines and proposed table of contents provided by UNICEF (PMER) mainly entailing the following:

‒ Contractor’s understanding on SPSP-Rural operationalizing PATS,

‒ Outline of evaluation methodology, proposed sources of data and data collection

methods/tools etc.

‒ A Gantt chart to define timeline on tasks planned for the assignment

‒ A proposed list of the team members and team leader, their qualification, roles

and responsibilities.

‒ Tools for end of programme evaluation

Draft and Final evaluation report and presentation

‒ The consultant will prepare a draft evaluation report presenting all Key

Deliverables as outlined above which shall then be presented and discussed in

consultative session organized by UNICEF/Implementing partners/Evaluation

Reference Group

[98]

‒ Consultant will incorporate changes agreed in consultative session or through

email for UNICEF comments.

‒ Consultant will incorporate comments by UNICEF and Reference Group on draft

report and submit a final report to be agreed by UNICEF and handover organized

evaluation data (soft and hard copies to UNICEF).

10. Evaluation Ethics:

Evaluation team will adhere to UNICEF ethical principles as outlined in UNEG ‘Ethical Guidelines for Evaluations (to be shared by PMER, UNIECF). The procedures to safeguard the rights of and confidentiality of information providers, e.g., legal issues pertaining to issues such as provisions to collect and report data will be observed, particularly, permission needed to interview or obtain information about children and young persons, storing and maintaining security of collected information and protocols to ensure anonymity and confidentiality. 11. Evaluation Work plan

Evaluation team activities Time

Desk review of existing project documents (as detailed above) 2 day

Inception meeting 1 day

Finalize evaluation design, tools, work plan, database, submission of inception report

4 days

Bottleneck analysis and interviews with IPs 2 days

Capacity needs assessment 3 days

Key informant field interviews, FGDs with partners/ community, HH and School Survey

10 days

Data entry and analysis 4 day

Draft Report/debriefing/Finalization of report 4 day

Total 30 days

i. Management Arrangements:

To facilitate the process, UNICEF will assist in linking up the evaluation team with respective stakeholders. UNICEF will provide any coordination support needed and ensure participatory study process. UNICEF will suggest key partners for interview and consultations with the team. However, UNICEFs involvement will not influence the process in anyway as the team is expected to retain their full integrity and flexibility to determine the best approach to collection and analyzing data for the evaluation.

Evaluation Reference Group (RG): UNICEF, MoCC and provincial governmental departments will form evaluation Reference Group. The programme Focal Person would ensure seeking nominations of members from the key stakeholders and subsequent involvement of the members of the RG in delivering a high quality

[99]

evaluation. UNICEF’s ToRs for RG for evaluations would guide and steer the work of RG. The RG will provide inputs on relevant and strategic issues. The evaluation team will liaise with the RG and provide regular evaluation progress updates to them against the agreed work plan during different phases of evaluation process. The Evaluation Team will be liaising with RG and UNICEF’s Evaluation Management Committee (mandated for quality assurance) through PMER Section and responds to queries and incorporate feedback as necessary through the focal person designated by Chief, WASH.

The final draft report will be submitted to the RG for review and comments for onward transmission to PMER (UNICEF) and other relevant stakeholders.

Specific Responsibilities:

UNICEF, including field teams in provinces:

- Facilitate meetings and interactions with stakeholders

- Comment on the draft report.

- Approve final report

- Follow up on recommendations/ management response

Implementing partners/ stakeholders:

- Participate in meetings/ consultations.

- Participate in a workshop to share study findings.

- Provided needed information and documents related to project outcome

- Comment on draft report.

Government counterparts:

- Serve on Evaluation Reference Group

- Participate in meetings/consultations

- Participate in a workshop to share study findings

- Provided needed information and documents related to project outcome

- Comment on draft report.

[100]

ii. Evaluation team criteria:

All UNICEF HR business rules and workflow processes would apply in selection of the consultants’ team. The evaluation team (with a desired gender balance) shall be composed of individuals with most appropriate experience and knowledge on conducting evaluations and technical skills in the areas being evaluated including adequate knowledge and experience on gender, human rights and equity issues, given as annex 1.

Prepared by:

Kamran Naeem Signature:_______________________ WASH Specialist, Islamabad Date:

Reviewed by

Signature ________________________ Chief WASH, Islamabad Date:

Cleared by:

Signature:_____________________ Chief of Supply & Logistics Date:

Approved by:

Signature:_____________________ Deputy Representative Date:

[101]

Annex. 1. Qualification Requirement for Evaluation Team Team Leader assigned to evaluation should meet the following requirements and qualification:

‒ At least 8-10 years experience in development sector of which at least 5 years experience of evaluations and research.

‒ Experience of providing leadership in similar assignments. ‒ Other attributes e.g. analytical skills, communicating and negotiation skills ‒ Experience with KAP studies and end line evaluations is must ‒ Experience with managing/leading implementation/evaluation of PATS

programmes is an added advantage Key individuals assigned to evaluation should meet the following requirements and qualification:

‒ At least 5-10 years’ experience in community health research including gender and equity issues.

‒ Experience in report writing and analytical skills ‒ Experience in the use of statistical software e.g. SPSS ‒ Other attributes e.g. analytical skills, communicating and negotiation skills ‒ Experience with KAP studies and end line evaluations is must ‒ Experience with implementation/evaluation of PATS programmes is an added

advantage

[102]

9. Annex 02 Illustration of SPSP Programme Components/Design

[103]

10. Annex 03

Programme Results Framework SPSP-Rural

Programme Result

Project Outputs

Project Activities Indicators Baseline Target MoVs Frequency of reporting and data

collection

By 2017, most vulnerable and marginalized children and women are using safe drinking water, improved sanitation and hygiene practices

1. Government expenditure on WASH as a proportion of total government expenditure.

4%, 28 Billion PKR, 2010/2011 ADP Punjab = Balochistan Ratio Water/Sanitation: 3/1

5% (1% increase)

- WASH Sector study report -PSLM/JMP -PSDP/ADP -MICS

Once a year/project duration

2. Proportion of the population using an improved drinking water source (disaggregated by residence and wealth

National :91% RURAL:90% Wealth Quintile: Poorest, 2nd, 3rd, 4th, richest RURAL: HH: 6% 14% 24% 31% 38% 6.50% Other improved: 81%, 74%, 68%, 65% , 60%, 0.70% Total: 0.87%, 0.88%, 0.92%, 0.96%, 0.98%, 0.072%

National :95% National RURAL: 94%

WASH Sector study report -PSLM/JMP -PSDP/ADP -MICS

Once a year/project duration

[104]

3. Proportion of population that practices open defecation (disaggregated by residence and wealth quintile).

23% RURAL: 38.7% Wealth Quintile: Poorest, 2nd, 3rd, 4th, richest RURAL: OD: 91%, 64%, 40%, 14%, 2%, 0.2%

22% reduction in OD at national level RURAL: 35.0%

WASH Sector study report -PSLM/JMP -PSDP/ADP -MICS

Once a year/project duration

1. Demand for sanitation to achieve open defecation free status created in targeted communities

1. Proportion of targeted population that practices open defecation (disaggregated by sex, residence and wealth quintile).

2. % of targeted population willing to self-finance construction of a latrine (disaggregated by sex, residence and wealth quintile).

3. # of CBO’s and/or WUA (Water User Associations) in the targeted areas that are functioning and utilized to claim equitable access to WASH services

4. % of targeted population aware of the need for achieving ODF status in their settlement (disaggregated by sex residence and wealth quintile).

HH survey Once a year/project duration

2. Sanitation demand sustained with supply side interventions

1. Proportion of the targeted population using an improved drinking water source (disaggregated by sex, residence and wealth quintile)

2. # of latrines in the targeted constructed

[105]

with external support in form of subsidy, loan, materials or trained skilled labour.

3. % of population using WASH facilities constructed with external support in form of subsidy, loan, materials or trained skilled labour source (disaggregated by sex, residence and wealth quintile)

4. % of school children in the targeted areas with access to child friendly WASH facilities in their schools and child-friendly spaces(disaggregated by sex and age)

5. # of basic social service delivery offices that use innovative technologies to improve service delivery.

3. Improved Health and Hygiene behaviors promoted with the target community

1. % of households in the targeted areas practicing household water treatment and safe water storage/using improved drinking water who use an appropriate treatment method

2. % of households in the targeted areas with soap anywhere in the dwelling

3. % of households in

[106]

the targeted areas using and maintaining hand washing facilities with soap close to their latrines

4. % of schools in the targeted areas using and maintaining hand washing with soap facilities close to their latrines

5. % of the targeted population aware of at least one design option for constructing low cost latrine (disaggregated by sex, residence and wealth quintile)

6. % of the targeted population recalling three key hygiene messages (disaggregated by sex and wealth quintile)

4. Linkages developed with duty bearers/public institutions to support PATS

1. % of provincial budgets allocated to WASH specific social services, with separated budget lines

2. # of ODF villages through PATS programmes implemented by the government institutions

5. Villages supported in attaining Total Sanitation status

1. % of population in the targeted areas using underground drainage and waste water treatment system (disaggregated by

[107]

residence and wealth quintile)

6. PATS programme knowledge management and accountability Framework implemented

1. Demand for sanitation to achieve open defecation free status created in targeted communities

1.1 Conduct formative research to develop/customize BCC (if required)

1.1.1 Formative research to guide BCC conducted with clear guidelines to develop communication materials in the start.

- Formative research report - Payment to consultant/Partner

- One time activity

1.2 Develop/customize standard BCC package based on formative research findings and results of KAP baseline survey (if required)

1.2.1 No. of items in standard BCC package materials (IEC, electronic, print, community shows) developed/customized in local context

- Standardization of BCC package report - Payment for printing of IEC materials etc. - Standardization workshop/meetings minutes

- One time activity - Payments and meetings as per required need

1.3 Launch BCC campaigns in targeted communities and schools to create demand for sanitation.

1.3.1 No. of children (boys and girls) reached with demand creation for sanitation messages in Schools. 1.3.2 No. of people (men, women, boys, girls) reached with demand creation for sanitation messages 1.3.3 No. of people reached with demand creation for sanitation messages through electronic/print media. 1.3.4 No. of people reached with demand creation for sanitation messages through community events (theatres, puppet shows)

-Monthly progress reports - Monthly substantive review reports - CRP follow up reports - Schools events reports - VSC meeting minutes - VSC register - Media reports - Community event reports, photographs and videos - CRP reports - SLTS monitoring reports - Third party monitoring reports - Human interest stories

- One event for demand creation conducted in each school of targeted community (boys and girls) in the beginning of project - One event at each selected religious worshiping places of community in the beginning of project - One community demand creation Event in each selected community in the beginning of project - One door to door demand creation campaign for housewives in the beginning of project - One 15 days cycle of publishing advertisement in local newspapers observed for demand creation

[108]

- At least one TV/Radio promo/spot aired in the beginning of project for demand creation

1.4 Train school teachers and education officials on facilitating SLTS

1.4.1. No. of training events conducted for school teachers (men, women) on facilitating SLTS

- Training attendance sheets -Photos of training events -Monthly progress reports -SLTS training reports -Monthly substantive review reports

- One training batch for 20 selected teachers of schools in communities conducted as per size of targeted area in the beginning of project

1.5 Form and strengthen WASH clubs in schools

1.5.1 No. of WASH clubs formed in schools and strengthened with planned follow up capacity development sessions 1.5.2 No. of events conducted by school wash clubs (boys and girls) as per plan

- SLTS events reports - Monthly Progress reports - SLTS follow-up plans - Monthly substantive review reports -WASH club activity register, photographs

-One WASH club will be formed in all selected schools for SLTS in the beginning of project - Each school will have a plan for events agreed with SLTS facilitator SO throughout the project cycle

1.6 Train social organizers on facilitation of PATS in communities and schools

1.6.1 No. of training events conducted for social organizers on participatory tools and IEC material for facilitating PATS 1.6.2 No. of social organizers successful using participatory tools and IEC material for facilitating PATS in villages according to developed criteria

- Master trainers and social organizers training reports - Training attendance sheets -Training evaluation reports -Reports on capacity development events -Monthly substantive monitoring reviews - Monthly progress reports -Third party monitoring reports -MIS/GIS Database - Partner vouchers on training event costs

- One training batch for trainers/Social organizers in the beginning of projectOne training batch for CRPs/Activists in the beginning of project -One monthly substantive review will be conducted in one selected district in the programme area every month

1.7 Train CRPs on facilitation of CLTS in communities

1.7.1 No. of training events conducted for CRPs on conducting CLTS in villages 1.7.2 No. of CRPs successfully facilitating CLTS in villages according to developed criteria

- CRPs and Activists training reports -Training evaluation reports -CRPs follow up reports -criteria for evaluating CRP facilitation

[109]

1.8 Identify existing/form new and develop capacity of VSCs

1.8.1 No. of VSCs existing/formed and trained

- PRA reports - ToPs for formation of VSCs - VSC register - Knowledge management - Database for PATS - VSC's CNA reports - VSC training reports, attendance sheets, evaluations - VSC meeting minutes - Reports on other capacity development events - Documented CAPs - Photographs - Monthly progress reports - Monthly substantive monitoring review reports - Monthly Progress reports - MIS/GIS Database

- One VSC formed for - One village will receive two trainings in the beginning of project and other as refresher - PRAs will be conducted in the beginning of project - VSC meetings are recorded in VSC register as an ongoing activity -One time activity with one community/village/cluster

1.9 Develop Community Action Plans (CAPs) in targeted villages

1.9.1 No. of villages/clusters with developed community action plan 1.9.2 No. of CAPs with prioritized need for action by community on Hygiene Behaviors

- Documented CAPs One time activity with one community/village/cluster

2. Sanitation demand sustained with supply side interventions

2.1 Conduct technical trainings for masons in low-cost sanitation construction

2.1.1 No. of masons trained on sanitation construction and using their learnt skills 2.1.2 No. of latrines (other than demo latrines) constructed by masons trained under PATS programme at the end of project

- Masons training reports - Training attendance sheets -Training evaluation reports -Monthly substantive monitoring reviews - Monthly progress reports - Implementing partners vouchers on training event costs - Photographs -Masons follow-up FGD

-Training of masons is one time activity in the early stages of project, however several training events can be planned depending upon No. of masons being trained and typology of latrines which might vary from area to area.

2.2 Construct low-cost, environment friendly, gender appropriate and secure demonstration latrines

2.2.1 No. of complete demonstration latrines constructed for extremely vulnerable families as per agreed criteria with community (maximum 3%) 2.2.2 No. of demonstration latrines constructed through materials provision for vulnerable families as

- Photographs - Monthly progress reports - Community Action Plans - MIS-GIS database - BOQs and drawings of demonstration latrines - Demo latrines beneficiary data

Monthly reporting and data

[110]

per agreed criteria with community (maximum 7%)

2.3 Train entrepreneurs for establishing sanitation marts and sanitation enterprises

2.3.1 No. of entrepreneurs/manufacturer trained and producing low-cost sanitation solutions 2.3.2 % of trained entrepreneurs doing sanitation related business

- Entrepreneur’s training reports - Training attendance sheets -Training evaluation reports -Monthly substantive monitoring reviews - Monthly Progress reports - Human interest stories - Trained entrepreneurs/manufacturer follow up FGDs

One time activity

2.4 Provision of support for establishing sanitation marts and sanitation enterprises as profitable businesses

2.4.1 No. of sanitation marts provided with financial support by VSC revolving sanitation fund 2.4.1 No. of facilitated sanitation marts doing profitable business

-ToPs signed with entrepreneurs -Business plan for sanitation marts in place -Sales record of sanitation marts - Monthly progress reports -Third party monitoring reports - Case studies / human interest stories - MIS – GIS database - photographs and postal addresses of Sanitation Marts

Establishment of sanitation mart in a target area is one time activity but monitoring for profitable business is an ongoing activity and needs continuous reporting.

2.5 Develop linkages to facilitate community in accessing loan for construction of sanitation facilities

2.5.1 No. of lending institutions successfully approached to give loans for sanitation facilities 2.5.2 No. of households received sanitation loan from some lending institution facilitated by PATS programme

- Lending agreement - Monthly Progress reports - Monthly Substantive monitoring reviews

Quarterly

2.6 Provide communal/collective incentives to communities achieving ODF status

2.6.1 No. of communal water facilities constructed as incentives/rewards 2.6.2 No. of communal sanitation facilities constructed as incentives/rewards 2.6.3 No. of house sanitation facilities constructed as incentives/rewards 2.6.4 No. of house water facilities constructed as incentives/rewards 2.6.5 No. of critical community infrastructure constructed as

- ToPs for construction of communal facilities - BoQs and Drawings of communal facilities -Taking over and O&M agreement for communal facilities with relevant government department or community - Photographs of communal facilities -GIS coordinates - Monthly progress reports - Case studies/human interest stories - ODF certification of community receiving incentives

Monthly

[111]

incentives/rewards 2.6.6 No. of health and hygiene packages given as incentives/rewards

3. Improved Health and Hygiene behaviors promoted with the target community

3.1 Customize communication strategy for BCC standard package

3.1.1 WASH BCC strategy in place for PATS programme.

- Communication strategy document - Comments by C4D section on communication strategy

One time activity in the beginning of project

3.2 Launch mass media BCC campaigns

3.2.1 No. of people reached with WASH BCC messages through electronic/print media. 3.2.2 No. of people reached with WASH BCC messages through community events (theatres, puppet shows)

-Recorded TV Promos and radio spots/talk shows - Media rating -Pre and Post KAP in community and schools - Monthly progress reports - Community event reports - CRP/social organizer report

Ongoing activity as per communication Plan

3.3 Launch BCC campaigns on promotion of low-cost, appropriate and informed sanitation solutions

3.3.1 No. of school children (boys and girls) reached with messages through SLTS activities. 3.3.2 No. of people (men, women, boys, girls) reached with WASH BCC messages through CRPs

- Monthly progress reports - Community event reports -SLTS events reports - CRP/social organizer report - Human interest stories -Monthly Substantive monitoring visit reports - Water quality testing reports

Ongoing as per communication strategy/plan

4. Linkages developed with duty bearers/public institutions to support PATS

4.1 Conduct capacity development events to enhance ownership of PATS by government officials and political leaders

4.1.1 No. of capacity needs assessments (CNA) conducted for relevant public sector stake holders 4.1.2 No. of capacity development plans developed for relevant public sector stake holders by Implementing Partner 4.1.3 No. of capacity development events undertaken for relevant public sector stake holders by Implementing Partner 4.1.4 No. of government officials and political leaders participated in capacity development and key events of the project. 4.1.5 No. of advocacy initiatives undertaken with duty bearers/political

-Stake holder analysis report -CNA report -Stake holder capacity development plans -Minutes of meetings with the government and political leaders - Duty bearers training attendance sheets -Training evaluation reports -Notifications issue by government -Minutes of meeting confirming agreement on advocacy agendas -Reports on capacity development events -Copies of MoUs signed by government departments - Strategy/Plan developed by IP for each stake holder as part of stake holder analysis

- Monthly - Event specific

[112]

leadership to support PATS 4.1.6 No. of MoUs signed by implementing partner with relevant public institutions responsible for sanitation 4.1.7 No. of villages jointly selected for PATS programme by relevant Government duty bearers with IP

4.2 undertake advocacy to facilitate rewards by government for ODF communities

4.2. 1 No. of ODF communities received rewards/incentives by government

-Village assessment and selection reports notified/endorsed by government -Annual Development Plans of government reflecting funding for PATS project and Rewards for ODF -Political leadership/Government official’s public/media announcements -end of project evaluation -monthly progress reports -Quarterly review reports -end of project report -Joint progress monitoring reports generated

5. Villages supported in attaining Total Sanitation status

5.1 Mobilize communities for ODF status through SLTS, CLTS or other Community Approaches to Total Sanitation (CATS)

5.1.1 No. of households with new latrines constructed after start of the PATS programme 5.1.2 No. villages declared as ODF by community 5.1.3 No. of villages Verified as ODF by Implementing partner and independent monitors 5.1.4 No. of people (men, women, boys, girls) in communities declared as ODF 5.1.5 No. of people (men, women, boys, girls) in communities verified as ODF

- CRP reports -monthly progress reports -midterm review report -end of project report -PCRWR reports -Photos, BOQs and drawings -IP sitreps on ODF status -ODF verification reports by IP -ODF certificates -ODF notifications by government

Monthly Quarterly Annual/project duration bound

5.2 Demonstrate total sanitation by

5.2.1 No. of model villages with laid sewers, WW

-Photos, BOQs and drawings of WW collection and treatment

[113]

constructing WASH infrastructure in selected ODF certified villages, as model villages

collection and treatment system through PATS programme

facilities -Meeting Minutes with government -IP sitreps on ODF status -monthly progress reports end of project report

5.3 Undertake advocacy with duty bearers to support total sanitation projects

5.3.1 No. of advocacy initiatives undertaken to promote communal underground drainage system linked WW treatment system and communal SWM system 5.3.2. No. of ODF villages provided with underground drainage and WW treatment facility by relevant public sector stake holder/duty bearer.

- Monthly progress reports -Quarterly review reports -End of project report -Annual Development Plans of government reflecting funding for PATS project and Rewards for ODF villages -Political leadership/Government official’s public/media announcements -Joint progress monitoring reports

Monthly

6. PATS programme knowledge management and Framework implemented

6.1 Launch PATS project inception workshop

6.1.1 Inception workshop conducted in the beginning of project

- inception/launching workshop report/proceedings

One time activity

6.2 Establish baseline against indicators in results framework

6.2.1 Baseline established in the beginning of project

-Baseline documented -Baseline dissemination/inception/launching workshop

One time activity

6.3 Conduct regular joint ‘Monthly Substantive Monitoring Reviews’ with implementing partners in the field

6.3.1 No. of joint progress monitoring reports (IP and public institution) 6.3.2 No. of monthly substantive monitoring reviews’ visits conducted for confirming/measuring performance indicators in results framework 6.3.3 No. ‘monthly substantive monitoring review’ reports shared among partners

- Substantive monitoring visit reports -Knowledge management data base on PATS -Progress reports -Minutes of Quarterly Review meetings -Mid-term review reports

Monthly

6.4 ensure all reports in M,E&L framework are timely received and are responsive to agreed formats and quality

6.4.1 % of responsive reports submitted as per M,E&L framework

- Progress reports -Minutes of Quarterly Review meetings

As per reporting schedule

6.5 Conduct structured quarterly review meetings with field staff

6.5.1 No. of quarterly progress reviews conducted with IP and minutes documented

- Minutes of Quarterly Review meetings - third party monitoring reports

Quarterly

6.6 Conduct internal 6.6.1 mid-term review -Mid-term review report Six monthly

[114]

midterm review workshop with IPs, field staff, CRPs and community representatives/activists

workshop report endorsed by M&E section of UNICEF

- Comments by M&E section

6.7 Document best practices for communication and replication

6.7.1 No. of Human interest stories documented and accepted as per UNICEF guidelines and disseminated

-Knowledge management data base on PATS -Progress reports -Minutes of Quarterly Review meetings -Mid-term review reports -Human interest stories published on UNICEF website, Annual WASH reports -End of project lessons sharing workshop report -End of project report by IP -practices/documentaries

Monthly Quarterly Six monthly End of project

6.8 Conduct end line survey

6.8.1 accepted end line survey report

- End line survey report - Lessons sharing workshop report

One time activity at the end of project

6.9 Conduct end of project lessons sharing workshop with all stakeholders

6.9.1 Lessons sharing workshop conducted and feedback documented

- Lessons sharing workshop report -List of participants in lessons sharing workshop - Photos - Payments made for lessons sharing workshop

One time activity at the end of project

6.10 Provide data to knowledge management databases on developed formats

6.10.1 Up to date knowledge database available in UNICEF for access to all stake holders at the end of project 6.10.2 Up to date GIS database available with government for access to all stake holders at the end of project

- Database up to date in government custody - on-line link to database

One time activity at the end of project

[115]

11. Annex 04 Stakeholders & Roles Following key stakeholders were involved during the implementation of SPSP-Rural. The role and geographic coverage of each stakeholder is highlighted in the table below.

Stakeholder Role

National Level

- Climate Change Division, The Cabinet Secretariat

- Policy guidance & formulation, monitoring of programmatic approaches

- Water and Sanitation Programme (World Bank), Water Supply and Sanitation Collaborative Council (WSSCC), Global Sanitation Fund (GSF), Plan International, Water Aid (UK) Plan Pakistan

- To coordinate WASH/PATS efforts/plans, share UNICEF learning, Advocate PATS implementation, organize learning events

- DSI Pakistan - Technical advice on sanitation designs and sanitation marketing

- RSPN - Capacity development of IPs and government on PATS/CATS approaches, PRA, community mobilization

- APEX Consulting - Third party monitors for field monitoring

Provincial Level

- Provincial Government Departments (PHED, Health, Education and LGRDD)

- Provincial policy advocacy/development, capacity development,

District Level

- Government Departments at local level (PHED, Health, Education and LGRDD)

- Planning and Implementation of selective hardware components (identification of schools for schools sanitation interventions, planning and execution of rural water schemes rehabilitation, rewards, field monitoring and ODF certification,.

Implementing Partners

- Muslim Aid - Qatar Charity - TVO (Trust for Voluntary

Organization) - BRSP (Baluchistan Rural Support

Programme) - WDCO (Women Development &

Community Organization) - NCBP (Nations Capacity Building

Programme) - NRSP (National Rural Support

Programme) - SAFWCO (Sindh Agricultural and

Forestry Workers Coordinating Organization)

- Sabawoon (Sindh Agricultural

- D. G. Khan Punjab - RajanPur Punjab - RajanPur Punjab

- Jaffarabad Balochistan

- Jaffarabad Balochistan

Naseerabad Balochistan

- Jacobabad & ShikarPur Sindh

- Kashmore Sindh

- Kohat KPK

[116]

Stakeholder Role

and Forestry Workers Coordinating Organization)

- SEED (Social Efforts for Education & Development)

- IRSP (Integrated Regional Support Program)

- SRSP (Sarhad Rural Support Programme)

- D. I. Khan KPK

- Mohmand, Bajour (FATA)

- Tank (KPK) – Kurram (FATA)

[117]

13. Annex 05 13.1. Evaluation Matrix for Inception Report - SPSP Evaluation

1. Relevance

Evaluation Q1: To what extent the programme design and implementation was relevant to national/provincial priorities and needs of men and women beneficiaries in terms of achieving expected results across all sociocultural groups including the vulnerable in the targeted communities?

Sub-questions What to Check (Indicators)

Data sources

Methods of Data Collection

To what extent were the project objectives and strategies aligned with government priorities, commitments and harmonisation efforts and also to PATS? To what extent the analysis of then-exiting WASH needs of community (if conducted) informed programme design and TOC? How far programme objectives were consistent with priorities and needs identified by men and women beneficiaries across different sociocultural groups? How aligned were programme objectives toward UNICEF’s equity focus for its integration into systems/processes and project deliverables? To what extent was the project aligned with UNICEF’s CATS’ principles?

- What lessons can be drawn from the introduction of CLTS triggering methods later in programme cycle, particularly of potential stigma for those who could not construct toilets vis-à-vis the ethical and human rights principles? To what extent use of PATS master trainers was relevant and beneficial to training cadre of human resources in sanitation sector?

Evidence of relevance at the various tiers of operation (provincial, district and community levels) as well as relevance to geographic areas, including views of key stakeholders on targeting Evidence of contextualising and aligning with national/provincial strategies as well as with the PATS approach. Evidence of alignment of the goals and results with identified needs of the targeted community. Evidence of using UNICEF’s equity focused approach in the project design. Evidence of contextualising and aligning with CATS approach. Evidence of stigma with late/no toilet construction for non-adherent families. Evidence of CLTS in harmony with ‘do no harm’ principle of ethics. Evidence of PATS master trainers delivering standard trainings and maintaining knowledge base for sector

- Pre-KAP & Post KAP Comparison

- SPSP Results Framework Review

- UNICEF - IPs - MoCC - PHED - Local

Government

- Evaluation Consultants

- Pre-Post KAP Results

- WSP/WB (CR)WAP

- PIEDAR

- Baseline

study –

context

analysis

report

- PCAs with

IPs and

RSPN

- Training

reports

- Literature Review - Survey (HHS,

SSS)

- Interview - UNICEF

- Interview – IPs - Interview –MoCC - Interview –

PHED

- Interview – Local Government

- Interviews with

Donors/INGOs

(WSP/WB,

RSPN)

- Interview

Delegation Lead

SACOSAN

Member of

National

parliament

- Interview RSPN management and PATS master trainers

[118]

2. Effectiveness

Evaluation Q2: To what extent were project implementation strategies effective and successful in achieving the planned outcomes/results for rich and poor men and women across all sociocultural groups including the vulnerable in the targeted communities?

Sub-questions What to Check (Indicators) Data sources Methods of Data Collection

What were the most effective activities/outputs within overall programme which contributed to behaviour change against OD, and why? How effectively activities and outputs were designed and conducted from the outset to involve men, women from various sociocultural and age groups? How the implementation strategies could have been improved/altered for them to have been more effective in achieving the results? To what extent the equity principles were integrated in the project implementation (social mobilization, BCC, supplies facilitation, and others) and substantiated by project results (e.g. targeting of poorest of poor for demo latrines, change in access to WASH services and facilities for the lowest quintile)? How was the gender analysis carried out within the communities and used to inform programme implementation? Can the results be gender disaggregated?

To what extent has the project delivered on its outputs and achieved any early outcomes? How do the results compare across the project? What factors helping/impeded the delivery of the outputs and achievement of early outcomes and whether were used for improvement?

Evidence of what worked effectively / identified best practice in terms of approaches/outputs linked to behaviour change against OD. Evidence of a logical and robust results framework with a clear attribution and vertical logic. Evidence of the validity of the theory of change including any changes over the implementation period (Including views of stakeholders). Evidence of what worked effectively in terms of approaches and as per guidelines - including departure/variation from project design during implementation. Evidence of use of gender analysis and use of Gender Equality Policy for WASH. Evidence of operationalizing the Equity Approach in the various project work streams and identifying benefits. Evidence of outputs delivered (NOTE: Analysis will be done using Output Indicators presented in Annex 6 for all outputs. Results will be further disaggregated on the basis of gender, geographical location, wealth quintiles and disability. Including pre-post KAP scenarios. Evidence of learning lessons and their use.

- UNICEF Project Documents

- UNICEF - Implementing

Partners - Post KAP results

- Pre-KAP & Post KAP Comparison

- VSC (men,

women), SOs,

CRPs, Masons,

Marts/Entreprene

urs, SMCs,

Teachers,

Students, Wash

Club

- Review of

partnership

guidelines, micro

assessments and

baseline partners’

capacity

assessment &

development

report.

- Men and women beneficiaries’ feed back

- UNICEF Staff

- Literature Review - Survey

(Household, School)

- Interview with UNICEF

- Interview with Implementing Partner

- FGDs with VSCs, SOs, CRPs, Masons, Marts/Entrepreneurs, SMCs, Teachers, Students, Wash Club

[119]

How has the project changed the perception of improved services and who has benefitted the most (men, women, children, or others including the disabled) and in what ways?

To what extent did the combination of SLTS along with CLTS resulted in enhancing effectiveness of the programme design in achieving the desired results? To what extent SPSP-Rural Phase I flow of activities was more effective in mobilizing people to construct toilets as compared to CLTS programmes in similar settings?

UNICEF and IPs views on project achievements, factors helping or hindering project outcomes. Views of CRPs, VSC, School Teachers, Children etc. to check change in perception in relation to availability of services. Evidence of the complementarity of SLTS and CLTS approach and its overall relevance within the PATS model. Comparison scenarios between SPSP and CLTS activities

[120]

3. Efficiency

EQ3: To what extent have the outputs of the project been achieved (in terms of quality and quantity) with the allocated resources/inputs (such as funds, time, and procedures)?

Evaluation Questions and Sub-questions

What to Check (Indicators) Data sources Methods of Data Collection

Were the resources allocated adequate in terms of quantity to help achieve project outputs?

- To what extent were the outputs/approaches delivered in line with planned quality standards?

- Was the project duration sufficient/adequate for implementation of PATS in delivering the expected outputs?

- How does the project implementation of PATS compare in terms of efficiency with other institutions within country?

- What was the project

investment for each toilet constructed?

- How much clarity was offered by UNICEF in partner selection guidelines about the capacities required for project implementation? To what extent were these adhered to? Whether these helped in achieving results effectively and in time?

- What areas can be improved for partners’ selection in future programmes?

Evidence of resource

efficiency – comparison of planned vs. actual expenditures for varied outputs.

Evidence of minimum

quality standards (such as for inputs, outputs and strategies, and their

application. Evidence of efficient use of time (in terms of project duration) and timely delivery of outputs (in terms of activities). Evidence of efficiency with

other national Sanitation

programmes.

Cost incurred on project against each toilet constructed in project by IPs or community. IPs and UNICEF Staff views on effectiveness of partnership guidelines and identification of weaknesses or need for change. IPs and UNICEF Staff views on application of partnership guidelines in view of PATS project requirements

- Project Documents – project proposal & reports, budget, expenditure tracking sheets,

- Post KAP Results

- 3rd Party Reports

- Evaluation Consultant

- BoQs of latrines

-

- Review of Project Documents

- Review of 3rd Party Reports

- Survey (Household, School)

- Literature Review

- Interview with UNICEF and IPs

- FGDs - IDIs - Checklists

[121]

4. Sustainability

Evaluation Q 4: To what extent various stakeholders including the public sector and civil society are likely to sustain the behaviour change related to the goals of the programme after it has ended?

3.

Evaluation Questions and Sub-questions

What to Check (Indicators) Data sources Methods of Data Collection

Was sustainability aspect of the programme a part of the design from the outset?

What is the level of commitment amongst stakeholders to sustain the ODF status in project areas? Are there any challenges? How far SLTS has contributed to sustaining ODF status in communities? To what extent have partnerships (with government, donors, civil society organizations, and other stakeholders) been established to foster sustainability of effects? To what extent political will and government commitment exist for PATS scale-up (in terms of increased budgetary allocation, organisational restructuring, improved harmonisation and coordination)?

Evidence of sustainability initiatives, strategy and approach in the project design. Stakeholders views on the sustainability of strategies used under the joint programme. Evidence of challenges to sustainability. Evidence SLTS ensuring ODF in communities. Evidence of strengthened involvement/buy in/ownership of the project in the relevant stakeholders. Evidence of Government interest and priority of PATS (possibility of Government ownership, leadership and scale-up).

- IPs - MoCC Officials - PHED - Local

Government

- MNAs/MPAs - Communities - IP Reports

- Third party

reports

- Interview with MoCC Officials

- Interview with PHED (Province & District)

- Interview with Local Government (Province & District)

- Interview with Political Representatives (District Level)

- Interview with IPs

- FGDs with VSCs - FGDs with

WASH clubs

5. Impact of Unforeseen Events on Project

Evaluation Q 5: To what extent was the project implementation affected/influenced by unstable, conflict or emergency situation?

Sub-questions What to Check (Indicators) Data sources Methods of Data Collection

- Did the project (including IP PCAs) have a Risk Assessment and undertook mitigation measures (both for natural disaster and conflict)?

- Was the project implementation affected at any time by security, conflict or any disaster/emergency situation? If so, what aspects were affected and what measures were taken to

Evidence of Risk Assessment and Mitigation Strategy and measures in design and at operational level. Views of UNICEF Staff and IPs on any incidents (security, conflict, disaster etc) that affected project delivery and how it was dealt with including evidence of lesson learning.

- UNICEF Staff - Implementing

Partner

- IP and Third party reports

-

- Interview with UNICEF

- Interview with Implementing Partner

- Literature review

-

[122]

mitigate/respond to the situation?

- What are the lessons learnt to plan for unforeseen in future?

6. Cross Cutting Theme: HRBA

Evaluation Q 6: To what extent did the project observe HRBA and/or contributed to observance of Human Rights in Pakistan as an implicit objective of the project?

Evaluation Questions and Sub-questions

What to Check (Indicators) Data sources Methods of Data Collection

- Did programme made efforts to developing capacity of duty bearers to take responsibility and empowering communities and contributing to addressing immediate needs.

- To what extent did the project support Government through the use of PATS approach in making progress on the MDG 7 (Target 10): (Halving, by 2015, the proportion of people without sustainable access to safe drinking water and basic sanitation)

- To what extent has the project ensured that all schools in the target area have adequate child friendly water and sanitation facilities and hygiene education project?

- To what extent was there a common understanding of HRBA in UNICEF, in context of Right to Water and Right to sanitation? With Staff? With Government? With Implementing Partners? And among sanitation sector stakeholders? If not, why?

- What formal and informal

mechanisms were put in place/existed for duty bearers and rights holders to participate/have a voice in the various activities/interventions of the project in terms of

Evidence of integrating and using HRBA in the project. Evidence of focused attention to child related issues. Stakeholders views on rights based approach used in the project. Stakeholder views on right holder/duty bearers linkage.

- MoCC Officials - UNICEF Staff - PHED

(Provincial level)

- Local Government (Provincial level)

- Education Department

- Health Department

- Implementing

Partners

- Children

- School

Teaches/SMC

- Interview with MoCC Official

- Interview with UNICEF

- Interview with PHED

- Interview with Local Government

- Interview with Implementing Partners

- Interview with Children

- Interview with Teachers/SMC FGD

- Interview with UNICEF Staff

- Interview with IPs

[123]

programming planning, implementation, monitoring and evaluation?

7. Financial Risk Management

Evaluation Q 7: To what extent the project has observed and applied UNICEF’s standard financial management practices with Implementing Partners as well as with Government?

Evaluation Questions and Sub-questions

What to Check (Indicators) Data sources Methods of Data Collection

What was the financial impact of security, conflict or any disaster/emergency situation during the implementation of the project? (including cost extension, slow implementation rate, associated security costs etc)

Was UNICEF’s zero tolerance to corruption policy applied in the project?

To what extent alternative implementing methods could have been applied to increase project’s cost effectiveness?

To what extent micro assessment process remained significant during the implementation of the project in terms of managing financial risks and did these assessments help in identifying key issues at the beginning of relationships with IPs and Sub IPs? What have been the areas of improvement from UNICEF and IP perspective?

To what extent the project (IPs and Sub-IPs) remained compliant with UNICEF Policies? What organizational arrangements were introduced for transparent procurements?

To what extent value for money was a consideration during implementation of project outputs and how do the results relate to this?

Evidence of security/conflict/disaster/ Emergency impact on the project. Evidence of application of UNICEF’s zero tolerance corruption policy. Evidence of cost effective methods. Evidence of significance and usefulness of conducting micro assessments process. Views of UNICEF and IP on improvements in financial risk management. Evidence of compliance with procurement policies and procedures. Note: Adherence to procurement policies & procedures is an auditing domain. Evidence of cost effectiveness and application of value for money principles. Evidence of cost effectiveness in terms of comparative approach and projects across partners and government.

- Project documents and others resources

- MOCC, PHED and LG Depts.

- Development partners (WSP, RSPN)

- UNICEF Staff - Implementing

Partner

- Review of previous Audit Reports, Spot checks, micro assessments and baseline partners capacity assessment & development report

- UNICEF Bottleneck Analysis and Minutes of coordination and other meetings between UNICEF and GOP

- Literature Review (project reports and expenditure statements)

- Interview with UNICEF

- Interview with Implementing Partner

- Documents Review as mentioned in Data sources

- Interview with MOCC, PHED, LG

- Interview with development partners (WSP and RSPN)

[124]

To what extent the implementation modality of working through CSOs and public sector partners prove to be cost efficient and effective? If not, why not?

What were the UNICEF/GOP coordination, management and administrative arrangements for this project? What were the key strengths and areas of improvement in the operational and administrative procedures/arrangements?

Views of GoP and UNICEF Staff on coordination, management and administrative arrangement.

[125]

15. Annex 06 Key indicators for the evaluation include the output, outcome and goal indicators which are drawn from the SPSP-Rural results framework. A baseline exists against them and a post-KAP will be carried out during the evaluation which will form the basis

Results Hierarchy

Operational Indicators Survey Instrument

Data Source/ Respondent

By 2017, most vulnerable and marginalized children and women are using safe drinking water, improved sanitation and hygiene practices

4. Government expenditure on WASH as a proportion of total government expenditure.

Secondary Sources Review

Secondary Sources

5. Proportion of the population using an improved drinking water source (disaggregated by residence and wealth Quintile)

HHS HH Respondent

6. Proportion of population that practices open defecation (disaggregated by residence and wealth quintile).

HHS HH Respondent

1. Demand for sanitation to achieve open defecation free status created in targeted communities

5. Proportion of targeted population that practices open defecation (disaggregated by sex, residence and wealth quintile).

6. % of targeted population willing to self-finance construction of a latrine (disaggregated by sex, residence and wealth quintile).

7. # of CBO’s and/or WUA (Water User Associations) in the targeted areas that are functioning and utilized to claim equitable access to WASH services

8. % of targeted population aware of the need for achieving ODF status in their settlement (disaggregated by sex residence and wealth quintile).

HHS HHS HHS HHS

HH Respondent HH Respondent HH Respondent

HH Respondent

2. Sanitation demand sustained with supply side interventions

6. Proportion of the targeted population using an improved drinking water source (disaggregated by sex, residence and wealth quintile)

7. # of latrines in the targeted constructed with external support in form of subsidy, loan, materials or trained skilled labour.

8. % of population using WASH facilities constructed with external support in form of subsidy, loan, materials or trained skilled labour source (disaggregated by sex, residence and wealth quintile)

9. % of school children in the targeted areas with access to child friendly WASH facilities in their schools and child-friendly spaces(disaggregated by sex and age)

10. # of basic social service delivery offices that use innovative technologies to improve service delivery.

HHS HHS HHS SSS KII District Officials

HH Respondent HH Respondent HH Respondent Head/Senior Teacher PHED/LG&RD District Officials

[126]

3. Improved Health and Hygiene behaviours promoted with the target community

7. % of households in the targeted areas practicing household water treatment and safe water storage/using improved drinking water who use an appropriate treatment method

8. % of households in the targeted areas with soap anywhere in the dwelling

9. % of households in the targeted areas using and maintaining hand washing facilities with soap close to their latrines

10. % of schools in the targeted areas using and maintaining hand washing with soap facilities close to their latrines

11. % of the targeted population aware of at least one design option for constructing low cost latrine (disaggregated by sex, residence and wealth quintile)

12. % of the targeted population recalling three key hygiene messages (disaggregated by sex and wealth quintile)

HHS HHS HHS SSS HHS HHS

HH Respondent HH Respondent HH Respondent Teacher HH Respondents HH Respondents

4. Linkages developed with duty bearers/public institutions to support PATS

3. % of provincial budgets allocated to WASH specific social services, with separated budget lines

4. # of ODF villages through PATS programmes implemented by the government institutions

Secondary Sources Review HHS

Context Analysis HH Respondents

5. Villages supported in attaining Total Sanitation status

2. % of population in the targeted areas using underground drainage and waste water treatment system (disaggregated by residence and wealth quintile)

HHS HH Respondent

[127]

16. Annex 07 16.1. List of People Met For SPSP (Rural) Final Evaluation

Summary Location Interviews/KIIs FGDs

Islamabad 8

Sindh 8 9

Balochistan 5 10

Khyber Pakhtunkhwa 8 3

Punjab 14 7

Total 43 29

Islamabad Interviews/KIIs S #

Name Designation/Organization Interview District/Village

1 Imtinan Ahmed Development Solutions International Interview Islamabad

2 Tania Khan WSSCC (Water Supply and Sanitation Collaborative Council)

Interview Islamabad

3 Shandana Khan CEO RSPN Interview Islamabad

4 Bashir ud Din RSPN Interview Islamabad

5 Irfan Tariq Ministry of Climate Change Interview Islamabad

6 Farhan Sami WSP (Water and Sanitation Program) Interview Islamabad

7 Muhammad Hafeez

WaterAid Islamabad

8 Tahir Jelani Apex Interview Islamabad

Kamran Naeem WASH Specialist, UNICEF, Islamabad Islamabad

Total # of Interviews = 8

Sindh – FGDs FGD – Community Resource Persons -

Male Ahmedabad, Misripur, Jacobabad FGD – Children (Girls) - Dao Syed

Bukhari Ghari Khero, Jacobabad

1 Ahmed Ali 1 Aisha

2 Zulfiqar Ali Jakhrani 2 Fareeda

3 Muhammad Raheem 3 Haleeman

4 Karimdad 4 Hira

5 Ali Murad 5 Asma

6 Shabir Ahmed 6 Tabasum

7 Hafiz Abdul Hameed 7 Hameeda

8 Ghaus Bux 8 Kainat

9 Nazar Muhammad 9 Shumaila

10 Naghma

FGD – Teachers Female – Dao Syed Bukhari Ghari Khero, Jacobabad

FGD – Community – New Goth Jakhro Ghari Khero, Jacobabad

1 Zahida Perveen 1 Sadiq Ali

2 Jameela 2 Muhammad Bux Mirali

3 Rani 3 Hussain Bux Malgani

4 Ghaus Bux Maachi

5 Abdul Wajid Jakhro

FGD – Village Sanitation Committee - Men New Goth Village – Alan Pur - Jacobabad

FGD – Social Organizers – (Male/Female) Jacobabad

1 Tahir Ali 1 Ghazala SO SUKAAR

2 Abdul Waheed 2 Safdar Abbas SO SUKAAR

3 Muhammad Ishaque Brohi 3 Naseebullah SO PREPARED

[128]

4 Muhammad Aslam 4 Taj Muhammand SO PREPARED

5 Shaban Ali Katohan

6 Abdul Wahab Jakhero

7 Muhammad Anwar

8 Abdul Wahab Rind

FGD – Teachers (Male) – GPS - QadirPur City - Jacobabad

FGD – Children (Male) – GBPS Jacobabad

1 M Nawaz Khoso 1 Mohkamdin Mugheri

2 Syed Bashir Ahmed 2 Sajjad Ali Khoso

3 Mehboob Ali Khoso 3 Ayaz Hussain Bhand

4 Muhammad Jan Bhand 4 Shaib Ahmed L ashari

5 Qurban Ali Maghsi 5 Liaqat Ali Khoso

6 Ahmed Ali Bhand 6 Irfan Ali Khoso

7 Nisar Ahmed Chohan 7 Sajid Ali Chamdani

8 Barkat Ali Brohi 8 Bilal Ahmed Brohi

9 Ali Jan Khoso

FGD – Village Sanitation Committee - Female - Jacobabad

1 Kaz Bano

2 Noor Bano

3 Anwari

4 Wazeeran

5 Shehzadi

6 Noor Jehan

7 Rani

8 Mai Birai

9 Azeeman

10 Shahnaz

[129]

Sindh – Interviews/KIIs

S #

Name Designation/Organization Interview District/Village

1 Dr. Abdul Aziz Hambi

Health Officer – District Health Office Interview Jacobabad

2 Sabir Ali Sheikh XEN PHED (JCB Division) Interview Jacobabad

3 Azizullah Odho DEO Education – Education Department Interview Jacobabad

4 Jamil Ahmed Rakhsani

Mason – New Goth Jakhro Ghulab Mari

Interview Jacobabad

5 Iftikhar Ahmed Mart Owner – Gari Khero – Meran Pur Interview Jacobabad

6 Syed Shahnawaz Mart Owner – Gari Khero – Alan Pur Interview Jacobabad

7 Syed Bashir Ahmed Shah

Political Activist Interview Jacobabad

8 Gul Wali IP Staff Interview Jacobabad

Total # of Interviews

Total Interviews = 8

Total # of FGDs & Groups

Total FGDs = 9

Balochistan – FGDs FGD – Teachers - Female –

Murad Ali - Jaffarabad FGD - School Children (Girls),

SohbatPur – GGS, Jaffarabad

1 Rukiya 1 Saleema

2 Azra 2 Resham

3 Nasreen 3 Yasmeen

4 Rizwana 4 Saima

5 Raheela 5 Afshan

6 Abida Perveen 6 Hira

7 Benazir Parveen CRP 7 Naima

8 Zubeda 8 Faryal

9 Rukiya Bibi 9 Raesa

10 10 Nageena

FGD - School Teachers (Male), Govt Boys Primary School, Goth Mehar Ali Khosa – SohbatPur Jaffarabad

FGD - School Children (Boys), Meher Ali Khoso - Jaffarabad

1 Ghulam Haider 1 Muhammad Asif

2 Nabi Bux Khoso HM 2 Sabir Ali

3 Abdul Sattar Brohi 3 Din Muhammad

4 Attaullah Brohi 4 Kareem Bux

5 M Anwar Brohi 5 Sajid Ali

6 Ghulam Fareed

7 Bakhtiar

8 Zahoor Ahmed

9 Hafeezullah

10 Izhar Hussain

FGD – Village Sanitation Committee (Male/Female) – Azam Khan - Jaffarabad

FGD – Village Sanitation Committee (Male) – Meher Ali Khoso - Jaffarabad

1 Muhammad Panah 1 M Naseeb Meher Ali

2 Rahim Bux 2 Sardar Khan Khoso

3 Shah Nawaz 3 M Anwar Pechoho

4 Zubeda Khatoon 4 Raza Muhammad

5 Muhammad Siddique 5 Shahad Khan Khoso

6 Naseeban (Female) 6 Allah Bux Sarki

7 Muhammad Hassan 7 Ali Nawaz Sarki

8 Fida Hussain Sarki

FGD – Masons – Jhat Pat Town – WDCO Office – Jaffarabad

FGD – Community Resource Person – (Male) – Meher Ali Khoso – Jaffarabad

1 Muhammad Bux 1 Ali Hassan

[130]

2 Ghulam Muhammad 2 Zafrullah

3 Saddam Hussain 3 Mehar Khan

4 Abdul Rashid 4 Barkat Ali

5 Allah Bux 5 Ali Murad

6 Inayatullah 6 Muhammad Umer

7 Muhammad Kabeer 7 Rahib Khan

FGD – Social Organizers – (Male/Female) BRSP Office - Jaffarabad

FGD – Community Resource Person – (Female) – Azam Khan – Jaffarabad

1 Haider Ali 1 Asia Bano

2 Jam Muhammad 2 Bhagul

3 Tahira (F) 3 Benezir

4 Amina (F) 4 Shareefan

5 Arbela (F) 5 Gull Bano

6 Fareeda 6 Arbab Khatoon

7 Abdul Hameed 7 Zainab

8 Muhammad Nawaz 8 Benezir Bibi

Balochistan – Interviews/KIIs S. #

Name Designation/Organization Interview District/Village

1 Dr. Jamal Din Jamal District Health Officer (DHO) Interview Jhatpat, Jaffarabad

2 Rafique Ahmed Khoso

JWP District Leader – Political Activitist

Interview Jhatpat, Jaffarabad

3 Rafique Kurd DPM BRSP Interview Jaffarabad 4 Alam Gurgag PM WDCO Interview Jaffarabad 5 Tariq Magsi Training Coordinator WDCO Interview Jaffarabad

Total # of Interviews Total Interviews = 5

Interview Jaffarabad

Total # of FGDs & Groups

Total FGDs = 10

Interview Jaffarabad

[131]

Punjab – FGDs FGD – Community Resource Person -

Male Kot Chuta – DG Khan FGD – School Children – (Boys)

Mukhtar Arain – Guristaniwala – Mutafarik Chohan DG Khan

1 Abdul Shakoor CRP Birmani Gharbi 1 M. Ahmed

2 Riaz Hussain CRP Birmani Sharki 2 M. Idrees

3 Habib Ullah Basti Hasanabad Colony 3 M. Owais

4 4 M. Shahzad

5 5 M. Rab Nawaz

6 6 Sheryar Ahmed

7 7 Ali Haider

8 8 Husnain Abbass

9 9 M. Shahzad

FGD – Teachers (Male) – Basti Birmani Kot Chuta – DG Khan

FGD – Village Sanitation Committee - Male – Kot Chuta - DG Khan

1 Ashiq Husain 1 Sadiq Hussain (CRP Phatuwala)

2 M. Afzal 2 Zulfiqar Phutuwala

3 Imtiaz Ahmed 3 Nadeem

4 Munir Ahmed 4 Mir Hazaar (VSC Member)

5 Irshad Husaain 5 Haqim Husaain

6 Rashid Ahmed 6 Munir Ahmed (VCS Member)

7 7 Zahid Hussaing (VCS/MO)

FGD – Teachers (Female) – Basti Birmani Kot Chuta – DG Khan

FGD – Village Sanitation Committee - Female – Kot Chuta - DG Khan

1 Rukhe Zara 1 Samrina Bibi

2 Mrs. Ghulam Hafsa 2 Khurshid Bibi

3 Nargis Zahra 3 Naheed Riaz

4 Zahida Perveen 4

5 Tasleem Zahra 5

FGD – Children (Girls) – Mukhtar Arain – Guristaniwala – Mutafarik ChohanDG Khan

1 Aneela Aslam 8 Tahira Batool

2 Asifa Batool 9 Shaista Atta

3 Arifa Batool 10 Saira Manzoor

4 Uzma Karim 11 Azra Bibi

5 Rukia Bibi 12 Kanwal Tehseen

6 Nasreen 13 Nazia Mukhtar

7 Mehwish 14

Punjab – interviews / KIIs – updated asmat

S # Name Designation/Organization Interview District/Village

1 Younas Leghari Secretary Union Council Mutafariq Chohan UC

Interview DG Khan

2 Muhammad Iqbal

Secretary Union Council Nawan UC

Interview DG Khan

3 Nazim Abbas CRP Interview Mazari wala DG Khan

Nasir Abbas Supervisor Health & Nutrition Interview Choti Zeren DG Khan

4 Asif Qureshi DCO – DG Khan Interview DG Khan

5 Amjad Tareen ADC Interview DG Khan

6 Saif ur Rehman AC Revenue Addition Director of ADLG&CD

Interview DG Khan

7 Rafia Bibi Primary Class Teacher Interview DG Khan

8 Tariq Leghari Community Resource Person Interview DG Khan

9 Sehar Abrar Community Resource Person Interview DG Khan

10 M Akhtaer Supervisor Head Teacher Interview DG Khan

11 Mushtaq Ahmed AEO Dept. of Education Interview Kot Chuta/Choti

[132]

Talpur DG Khan

12 Zahid Hussain Mart Owner Mutafariq Chohan Interview Mutafariq Chohan DG Khan

13 M Yousaf Entrepreneur Interview Kot Chuta/ Mutafariq Chohan DG Khan

14 Zubeda Bibi CRP Interview Mutafariq Chohan DG Khan

Total # of Interviews

Total Interviews = 14

Total # of FGDs & Groups

Total # of FGDs = 7

Khyber Pakhtunkhwa – FGDs FGD – Village Sanitation Committee -

Kohat FGD – Children (Boys)- village Ali

Zai,Kohat

1 Israr Din 1 Muqadas Ali

2 Mohammad Ishtiaq 2 Mohammad Ali

3 Sabz Ali Khan 3 Zain Hassan

4 Gul Mohammad 4 Ali Haider

5 Mohammad Naeem 5 Shaiq Hussain

6 Qambar Gul 6 Noor Hussain

7 Adil 7 Sadaqat Ali

8 Sajjad 8 Anwar Iqbal

9 Ghazanfar Abbas

10 Asher Hussain

FGD – Children (Girls) - Kohat FGD – Community Resource Person (Girls) - Kohat

Fareeda Musarrat

Fatima Basharat

Lal meena Uzma

Leeda Rabia

Subeena

Zarmeena

Rubeena

Sakeena

Nazia

Laila

Khyber Pakhtunkhwa KIIs / Interviews S # Name Designation/Organization Interview District/Village

1 Hazrat Gul ADEO/Education department IDI Kohat

2 Mohamad Hayat Secretary UCs/Local Government IDI Kohat

3 Abdul Wahab Social organizer IDI Kohat

4 Qamar Gul Community resource person IDI Kohat

5 Abdur Rehman Market owner/entrepreneur IDI Kohat

6 Mohammad Yousaf Teacher IDI Kohat

7 Noor Naz Social Organizer IDI

8 Shamim Akhter Teacher

Total # of Interviews Total Interviews = 8

Total # of FGDs & Groups Total # of FGDs = 3

[133]

[134]

17. Annex 08 17.1. List of Participants Provincial Bottleneck Analysis &

Government Partners Capacity Development Workshops

KPK-FATA-Bottleneck Analysis and Capacity Development Workshop - 8th April, 2014

Sr. No NAME DESIGNATION ORGANIZATION

1 Haroon Rashid S.O K.P.H

2 Majeed Gul AEO Kurrum Agency EDUCATION DEPARTMENT

3 Azhar Iqbal Team Leader SRSP

4 Qabil Shah MT (health) MT (health)

5 Mohammad Arif Team Leader SRSP

6 Alam Zeb Assistant Director LG FATA

7 Shah Behram TANK PHED

8 Khalid Naeem M.O.I MC Tank

9 Abdul Hafeez ADO EDUCATION DEPARTMENT

10 Sanaullah S.M.O Health

11 Nahid Khan S.D.O LG & RDD

12 Najeeb ullah Sub Engineer PHE BAJAUR

13 Muhammad Idrees Director Programme IRSP

14 Zaki ullah Consultant AAN ASSOCIATES

15 Hazrat Gul ADEO Kohat EDUCATION DEPARTMENT KOHAT

16 Muhammad Hayat LG & RDD UNION COUNCIL

17 Liaqat Zaman Sub Engineer PHE KOHAT

18 Anwar Zaib Programme Manager SABAWON

19 Sahibzada M.Shoaib CEO SEED

20 Shahzad Hadi Assistant Director (tech) LG & RDD

21 Sajjad Akbar UNICEF

22 Zain Raza Project Coordinator AAN ASSOCIATES

23 Nadeem Haider MD AAN ASSOCIATES

24 Zarqa Khan PO Operations SRSP

25 M.Shakaib Jan WASH Officer UNICEF

[135]

Balochistan Bottleneck Analysis and Capacity Development Workshop – March 13, 2014 Sr. No NAME DESIGNATION ORGANIZATION

1 Abdul Qayyum Director Finance PHED 2 Fatehullah Manager Field

Operation BRSP

3 Rafique Khan DPM BRSP 4 Aftab Aziz ADLG LG 5 Abdul Khaliq AIE LG LG 6 Ahmed Nawaz CEO -NCBP NCBP 7 Mohammad Yousaf Chief Health Educator Health DEPT

8 Zaqir Hussain Training Coordinator WDCO 9 Alam Khan Programme Manaer WDCO 10 Khalil Khetran Feid Monitor APEX 11 Mohammad Ahsan R.O APEX 12 Amanullah S/M BRSP 13 Syed Shaed Nawaz M&E Officer BRSP 14 Amoon Pervez Finance Officer NCBP 15 Abid Hussain H-R Officer NCBP 16 Naqeeb Agha PA,WASH UNICEF 17 Shazia Aman WASH Officer UNICEF 18 Nadeem Haider MD Consultant AAN ASSOCIATES 19 Mukhtar WASH UNICEF

Sindh - Bottleneck Analysis and Capacity Development Workshop – March 25, 2014

Sr. No NAME DESIGNATION ORGANIZATION

1 Kamran Naeem WASH Specialist UNICEF 2 Nadeem Haider Consultant AAN ASSOCIATES

3 Zahida Jamali Manager WASH SAFWCO 4 Qurban Ali ED SAFWCO 5 Rafique Ahmed G.M Operation SAFWCO 6 Khurram Aslam Chief Munciple Office LG DEPARTMENT 7 Mubashara-Iram WASH Officer UNICEF 8 Zahid Durrani WASH Specialist UNICEF 9 Amjad Khan Regional General

Manager NRSP

10 Qaiser Shah Wash Officer UNICEF 11 Muhammad Usman Executive Engineer PH.E.D 12 Engineer Ali Director (tech) PH.E.D 13 Faheem Junejo LG M&E LG Department

[136]

Punjab - Bottleneck Analysis and Capacity Development Workshop Sr. No NAME

1 Zulqarnain (SO(PH))

2 Abdullah Fahad

3 Tariq Mahmood

4 Sabahat Ambreen

5 Najeeb Aslam

6 Nadeem Haider

7 Salman Yousuf

8 Mustajab Ahmad

9 Syed Waseem Hassan

10 Abdullah Asad Khan

11 Umer Ali Aziz

12 Faisal

13 Aqeel

14 Atif

15 Usman

[137]

18. Annex 09 Sample Size and List of Villages for Post-KAP

Actual Sample Size

Province District Female Male Grand Total

Punjab

DG Khan 200 200 400

Rajan Pur 200 200 400

Sub Total - Punjab 400 400 800

KPK

D.I.Khan 200 200 400

Kohat 199 201 400

Tank 200 200 400

Sub Total - KPK 599 601 1200

Balochistan

JAFFARABAD 200 200 400

NASIRABAD 193 206 399

Sub Total - Balochistan 393 406 799

Sindh

JACOBABAD 206 197 403

KASHMORE 199 199 398

SHIKAR PUR 199 201 400

Sub Total - FATA 604 597 1201

FATA

Bajur Agency 0 400 400

Kurram Agency 0 385 385

Mohmand Agency 0 400 400

Sub Total - Punjab 0 1185 1185

GRAND TOTAL 1996 3189 5185

[138]

Sindh Jacobabad

IP Name Union Council Village Name

SUKKAR Misripurr Ahmed Abad

Habibullah Bugti

Muhammad Abid Khoso

Piyaro Khoso

Ramzan Jakhrani

Hussain Bux Ghunio

Wakil Ahmed

PREPARED Allan Pur New Goth Jakhro

Noor Muhammad Brohi

Hazar Khan Brohi

Taro Khan Dasti

Dawoo Syed bukhari

Meeran Khan Luhar

FRDP Qadirpur Nazeer Husain Mugheri

Qadir Bux Bhayo

Mundar khoso

Mazar khan Mirali

Qadir pur city-2

Hazoor Bux Mugheri

Haji Fakhruddin khoso

Kashmore

Union Council Name of Village Village Changed

Lalao

Rabban Khan Dahani

Mohd Mureed Bajkani Sahab Khan Bhalkani

Hadi BuxBahalkani

Abdul Haq Bahlkqni

Jeeand Khan Dahani

Saifal

Naik muhammad bajkani Piyaro Khan Nindwani

Allah wasayo nindwani

Tagio bijarani

Yar muhammad nindwani

Baland khan nindwani

Suhliani Gulan Khan Dahani Motio Lashari

Molvi Suraj Din Khoso 1

Ghulam Nabi Jakhrani

Nazar Mohammad Gujrani

Mir Sobho Sadique Khoso

Tangwani Imam Bux Bajkani

Mohd Hassan Malak

Yar Mohd Golo

Sounharo Malak

Kareem Bux Nindwani

[139]

Shikarpur

Tehsil Union Council Village

Khanpur

Khanpur faqir Wahid bux

Moor Unchar

Taj Muhammad brohi

Eid Muhammad brohi

Muhammad Hassan

Mian Sahib Abdullah khoharo

Faiz Muhammad Meer badar

Naik Muhammad brohi

Mageera Wahi

Din muhammad panwar

Shikarpur Jahan Wah Agha Jan Akhtar

Aghan Jan Pathan

Rasool Bux chandio

Ali Bahar (Gull Muhammad Marfani)

Sardar Naseer, Muhammad Wesar

Sultankot Qabil khan chandio

Aitbar Ali

Abdil

Fatah muhammad brohi

Suhbat Machi

[140]

Punjab Dera Ghazi Khan

UC Name Village/Basti Name

Mutfariq Chahan Norang Shah (Gharbi)

Basti Haji hussain Khan

Basti Mazari

Gadi wala

Basti Pushiya Sharqi

Basti Pushiya Gharbi

Mana Ahmedani Ramay Wala

Chah Charhoye wala

Basti Qureshi

Chakir Wala

Dubba Duri

Basti Changwani (Budhan)

Nawan Basti Hajan Muraad Khan(Masu wani+lashari+bhatti+Faqeeri)

Dodhay Wala

Khan Wala(Wahnes Wala+G Serwer+G Yasin Aslam+Shumla Khosa)

Gadray Wala(tariq Bajwa+Ahmad Bux+Abdul aziz+Rahim bux)

Basti Muhammad Afzal Khan

Basti Haji Faiz Muhammad+ Durrhatta

Rajanpur

UC Name Village/Basti Name

Noshehra Gharbi / Burrey wala

Basti Channar # 0

Basti Channar # 1

Basti Ada Khalil Abad

Mud Bhutta

Basti Rasool Bakhsh

Burrey wala Basti Haji Riaz Sirla

Basti Peer Bakhsh jogyani

Dur Muhammad Bazgir

Basti Nazoo Khan Bazgeer

Basti Ameer Shameer

Jahanpur Basti Nazeer Ahmad

Basti Muhammad Ramzan

Basti Ghulam Husain Sandhi/Mumtaz Lohar

Basti Abdul Sattar Nariwal

Basti Pati Juma Arain

Rakh Fazilpur Basti Iqbal jut

Basti Haji Jind wada/ hasil/Saich

Dunia Pur

Basti Shah pur

Basti Ghulam Mustafa Horra

[141]

Balochistan Naseerabad

UC Name Village/Basti Name Villages Changed

Bedar Goth Shakoor Khoso

Goth Haji Sobdar Khoso

Goth Ghoram Chinjni

Goth Nassur Din Khoso

Changaiz Khan Sasoli Lanja Khan

Quba Sher Khan Goth Bangul Khan Sasoli

Goth Badal Khan Nichari

Goth Bilawal Khan Pyanhar

Goth Haji M sadiq Talani

Goth Haji Shair Dil Nicharri

Manjhoti Sharqi Goth Chaker Khan Jhakrani

Goth Ali Madad Umrani/Peral

Goth Dildar Khan Umrani

Raja KHAN Umrani

Haji Hazaar Khan Umrani

Shori Drabhi Goth Abdul Rehman Shar

Goth Jamal Din Baghdar

Goth Musa Khan Bugti Goth: Peer Bux Shar

Sardar Allah Waraya Shar Goth:Ghulam Muhammad Shar

Abdul Kareem Domki

Jaffarabad

UC Name Village Name Replacement

Khuda e Dad Darya Khan Javeed Abad Muhammad Ismail Langhu Sohrab Khan Nizam ud Din / Abdul Aziz / Nisar Ahmad Name Change

Noz Band Nozband

Masood Abad Azam Khan Izhar Khan Khoso Mujeeb Khan Khoso Hamid Pur Jan Mohammad Mohammad Salah Abdul Aziz Meeral Khan Goth Kareem Baig Sohabat Pur City Goth Nawab Khan Bugti Mohalla Goth Ghulam Qadir Brohi

[142]

Goth Izhar Khan Khosa

Goth Maher Ali Khsa

Khuda Bux Mengel

[143]

Khyber Pakhtunkhwa Kohat

UC Name Village/Basti Name

Urban VI Afridi Banda

Khwaja Abad

Gul Abad

Eidgah colony merozai

Afzal Abad

Tappi

Muhammad Zai Muhammad Zai

Naway Kally

Kooz Kaley

Rajgaan Banda

Haider Rehman Banda

Speen Jumaat Mohalla

Ali Zai Ali Zai

Narey Kale

Yourd Kaley

Zear Kamar

Gul Kada (Kana)

Chashma Metha Khan

Dera Ismail Khan

UC Name Village/Basti Name

Parova Parova 1

Babber Kacha

Chah Pattar

Jhok Jhandir

Jhok Sikhani

Naiwela Jatta

Naiwela City

Gara Aashiq

Gara Tahir

Kotha Kumar

Lunda Sharif Umar Boba

Lunda Sharif

Shameer

Babbi

Rang Pur Janubi

Mahra Sikandar Janubi 2

Jhok Daruli

Lunda Para

Pachani

Dhawa

[144]

Tank

UC Name Village/Basti Name

Uttar Uttar

Shieikh Sultaan

Garra Matta

MamraizPathan

Jammal Awan

Gul Imam Daraki

Ghazi khel

Gul Imam

andary

Naway gara

Jattatar Tatoor

Adam Abad

Garra Hayyat

Garra Budda

Garra Pathaar

Gara Baloch Proper Gara Baloch

Razzaq colony

Dhari aba khel

Kot piran

Kirri saidal

FATA

Kurram Agency

Tehsil / Area Village/Basti Name

Lower Kurram

Baghzai Kalay

Dagar Kalay

Jabba Kalay

Baleesh Khel

Ibrahim Zai

Lower Basatu

Central Kurram Khapa Kalay

Johdara

Said Ali Mela

Tanorak

Kotkai

Koda Kalay

[145]

Mohmand Agency

Village/Basti Name

Monda/sapare

Dew Dhere

Kirra

Khadi Kas(Khas)

Palosay kally

Speena Khawra/Prang ghar

Boocha kally

Toor Zerik Boocha

Nao Kally

Sroo Shaheed Kally

Pata Saly/Patta Salle

Karkarna kally

Bajaur Agency

Tehsil UC Name Village/Basti Name

Khar Loe Sum Rashakai

Miangano Kalay

Loe Sum

Zega Dherai

Zoor Bandar Gundo

Shawo

Zoor Bandar

Sasmasai

Nawagai Kotkai Sharif Khana

Kotkai Charmang

Ghari Kalay

Shingar

Nawagai Redawan

Lawargai

Anar Bagh

Charmango Kalay

List of Villages Replaced Province / District

UC / Area Sampled Village/Basti Name

Replaced Villages

Sindh

Kashmore Lalao Mohd Mureed Bajkani Sahab Khan Bhalkani

Saifal Naik muhammad bajkani Piyaro Khan Nindwani

Suhliani Gulan Khan Dahani Motio Lashari

[146]

20. Annex 10 List of Documents Reviewed

(2010). Design of School WES.

González, A. H. (2013). 5th South Asian Conference on Sanitation.

Group, U. M. (2012). Global Evaluation of the Application of a Human Rights Based Approach in

UNICEF Programming - Final Report Vol II.

Pakistan Bureau of Statistics, G. (2011-12). Pakistan Social Living Measurement Survey.

SAARC. (2013). The Kathmandu Declaration. Kathmandu.

Syed Shah Nasir Khisro, I. (n.d.). CLTS Intervention in Post Emergency Context in Pakistan.

(n.d.). The 'shame question' in CLTS.

(n.d.). The 'shame question' in CLTS.

UNDP. (n.d.). General Guidelines for Inception Reports.

UNEG. (2005). Norms for Evaluation in the UN System.

UNEG. (2005). Standards for Evaluation in the UN System.

UNEG. (2008). Ethical Guidelines for Evaluation.

UNICEF & Ministry of CA&D, G. (n.d.). Child Friendly School Standards.

UNICEF. (2007-08). Multiple Indicators Cluster Survey .

UNICEF. (2009). Community Approaches to Total Sanitation.

UNICEF. (2010). Policy on Gender Equality and the Empowerment of Girls and Women.

UNICEF. (2011). Guidance on Conducting a Situation Analysis of Children’s and Women’s Rights.

UNICEF. (2011). WASH in Schools, Monitoring Package.

UNICEF. (2011-12). End of Programme Evaluation, Rural Sanitation in Flood-affected Districts

(RuSFAD III).

UNICEF. (2012). Funding Proposal to DFID UK for Life-saving WASH and Child Protection Assistance to

Flood Affected Populations in Balochistan, Punjab and Sindh Provinces.

UNICEF. (2012). Funding Proposal to DFID UK for WASH Humanitarian Assistance to Insecurity

Affected People in the Federally Administered Tribal Areas (FATA) and Khyber Pakhtunkhwa

(KP).

UNICEF. (2012). UNICEF Funding Proposal to DFID UK.

[147]

UNICEF. (2012). UNICEF Funding Proposal to DFID UK for Life-saving WASH and Child Protection

Assistance to Flood Affected Populations in Balochistan, Punjab and Sindh Provinces.

UNICEF. (2013). Baseline Report for Sanitation Programme at Scale in Pakistan – SPSP Rural.

UNICEF. (2013). Bottleneck Analysis for Sanitation Programme at Scale in Pakistan – SPSP Rural.

UNICEF. (2013). Contextual Analysis for Sanitation Programme at Scale in Pakistan – SPSP Rural.

UNICEF. (2013). Strategic Guidance Note on Institutionalizing Ethical Practice for UNICEF Research.

UNICEF. (n.d.). Global CATS Evaluation: Draft Theory of Change.

UNICEF. (n.d.). Programme Results Framework SPSP-Rural.

UNICEF. (n.d.). Terms of Reference (ToR) Technical Assistance for SPSP-Rural End of Programme

Evaluation.

UNICEF. (n.d.). Water, Sanitation & Hygiene (WASH) in Schools.

[148]

22. Annex 11 Research Instruments

End of Programme Evaluation UNICEF’s Sanitation Programme at Scale in

Pakistan (SPSP) – Rural Through Pakistan Approach to Total Sanitation (PATS)

INDEPTH INTERVIEW UNICEF & PARTNERs

Introduction to the Interviewee: I am ,a member of the evaluation team deployed by an independent consulting entity i.e. the AAN Associates, who has been contracted for the End of Programme Evaluation of UNICEF’s Sanitation Programme at Scale in Pakistan. This programme aims to improve sanitation practices, using Community-led Total Sanitation (CLTS) and School-led Total Sanitation (SLTS) approaches under the overarching PATS strategy to ensure that the most vulnerable, marginalized children and women are using safe drinking water, improved sanitation and hygiene practices. The target areas include flood affected, high Polio risk and insecurity affected areas. I am here to get your reflections, experiences and recommendations about the programme, which will help us to assess, analyse and document programme progress, results and areas for further improvement. This will help at large in designing or developing future sanitation programmes in the country and/or extension or scale-up of this initiative. I thank you for taking time for this interview today. I shall be asking you questions about how the programme activities were implemented in your area and what changes this programme brought in the lives of the community in regard to improved access and availability of sanitation services. I anticipate the interview will last about 60 minutes, and appreciate any information you can provide. Please note that this interview is important for the success of this assessment. Your answers are completely confidential and shall remain with the evaluation team. At the start we seek your consent to make use of this information for analysis and reporting and may also cite it in the report while making reference to your name, if needed. Also, we are recording the proceeding to avoid any loss of information and seek your consent to record our discussion. Finally, we would request you to please allow us to make photographs of project interventions and yourself for evidence and communication purposes.

[149]

Note: This is to underline that the IDI instrument has been designed as such to offer flexibility to the interviewers to adapt questions as to the specific role of the organization, respondents profile, understanding of programme operations and overall knowledge of sector. Hence one consolidated tool has been developed for multiple stakeholders (as listed above).

Name of Respondent: _____________________ Designation _____________________ Organisation _____________________ Location: _____________________ Date of Interview: _____________________

Name of the Interviewer: _____________________

Interview Questions: Please listen carefully to the questions and provide your candid feedback.

RELEVANCE Q 1. Was WASH the priority/one of the priority needs in rural context? If

yes, how was this established by UNICEF? How does the

programme design addresses the needs of the rural poor (equity)?

Q 2. To what extent the SPSP results framework is consistent with

results based management principles & approach (impact,

outcome, outputs, inputs)? If so, how does this approach manifest

and to what extent the vertical logic was valid (at design stage)?

Q 3. What was the process and basis/criteria used by UNICEF and IPs

to select the geographic regions where the programme has been

implemented (for instance selection of provinces, districts,

agencies, tehsil, UCs and villages)? In your view if the most

deserving regions were selected for programme interventions?

[150]

Q 4. To what extent you find the two models i.e. Community and School

Led Total Sanitation (CLTS & SLTS) complementary? Did the

programme achieve desired results by using both models, please

elaborate?

Q 5. What experiences/lessons were learned by UNICEF and IPs around

programme design and implementation (list what worked, what did

not and why? Substantiate with measure of success, scale, and

lessons learnt for future)?

Note: The analysis to be desegregated for following (components/subcomponents)

1. DEMAND CREATION

What Worked What Didn’t Work Lessons Learnt Success Scale

Rank

Highly

Successful

Moderate

Poor

Fail

[151]

2. SUPPLY FACILITATION

What Worked What Didn’t Work

Lessons Learnt Success Scale

Rank

Highly

Successful

Moderate

Poor

Fail

3. HEALTH AND HYGIENE PROMOTION

What Worked What Didn’t Work

Lessons Learnt Success Scale

Rank

Highly

Successful

Moderate

Poor

Fail

4. LINKAGES WITH DUTY BEARERS FOR PATS

What Worked What Didn’t Work

Lessons Learnt Success Scale

Rank

Highly

Successful

Moderate

Poor

Fail

5. VILLAGES SUPPORTED IN TOTAL SANITATION STATUS

What Worked What Didn’t Work

Lessons Learnt Success Scale

Rank

Highly

Successful

Moderate

Poor

Fail

[152]

6. PATS KNOWLEDGE MANAGEMENT FRAMEWORK

What Worked What Didn’t Work

Lessons Learnt Success Scale

Rank

Highly

Successful

Moderate

Poor

Fail

7. PARTNERSHIP MANAGEMENT

What Worked What Didn’t Work

Lessons Learnt Success Scale

Rank

Highly

Successful

Moderate

Poor

Fail

8. INTEGRATION OF CROSS CUTTING PRIORITIES

(Gender & Equity, Unforeseen Events, Rights Based Approach) What Worked What Didn’t

Work Lessons Learnt Success

Scale Rank

Highly

Successful

Moderate

Poor

Fail

9. SCHOOL LED TOTAL SANITATION

What Worked What Didn’t Work

Lessons Learnt Success Scale

Rank

Highly

Successful

[153]

Moderate

Poor

Fail

Q 6. In the provinces where project was implemented, what is the

status of water and sanitation policies? Are these

compliant/consistent with Pakistan Approach to Total Sanitation?

Q 7. In which way and to what extent the current programme design

(SPSP Rural) relates to PATS, please elaborate? In what respects

the programme differs/introduces elements different to PATS?

Q 8. Are you aware of any mechanisms (coordination meetings/forums

etc) that relevant government departments (at national and

provincial levels) are using to align the efforts of government and

development partners in WASH (as per PATS)? To what extent are

[154]

they successful in their efforts? In which ways UNICEF contributed

to harmonization and what could be improved further (including

the role of development partners)?

EFFECTIVENESS

Q 9. To what extent the programme design (UNICEF project document

to DFID and IP PCAs) offered clarity into project implementation –

defining expectations, standards of achievement, contributions,

coordination, etc required from Implementation Partners?

Q 10. To what extent the actual programme implementation followed the

design and how much has been achieved as to the stated outputs?

Were there any departures from the planned

methods/implementation plan? What were the points of departure,

and what were the reasons for this? To what extent did this

departure help in achieving the programme results?

[155]

Q 11. Did the programme design have a clear and articulated Theory of

Change (with clarity around various stages)? Can you elaborate on

the theory of change? What inputs contributed to which outputs

and on to Outcomes/Goals? Did the change happen as per the

assumed/perceived theory? If not what changes need to be

factored into future programme design/theory of change?

Q 12. To what extent has the project delivered on its outputs and

achieved the outcomes/results through the various interventions

undertaken? How do the results compare geographically (in

provincial context) and across partners?

[156]

Q 13. What factors helped/impeded the delivery of the outputs and

achievement of the outcomes across regions and partners?

Q 14. How has it changed the perception of improved services and who

has benefitted the most (men, women, children, or others

including the disabled) and in what ways?

Q 15. How far did the UNICEF partners’ selection guidelines offer clarity

(processes, controls and accountabilities) about the requisite

capacities (experience, expertise, geographic footprint, attitudes)

amongst partners for PATS programme implementation?

Q 16. To what extent were partner selection guidelines/standards (of

UNICEF) adhered to? And to what extent did this help in achieving

results effectively and in time? In which areas UNICEF may need to

improve the process/guidelines and controls for partners selection

particularly for WASH/PATS programmes?

[157]

EFFICIENCY Q 17. Has the project achieved the desired outputs in time, technical

resources & allocated financial resources? If not what elements

constrained the project delivery & why?

Q 18. Were any quality standards prescribed and set for outputs/pillars

before implementation? To what extent the outputs were delivered

in line with the prescribed quality standards e.g. community

mobilization (in terms of strategy, PRAs, time available, skills of

SOs, status of VSCs, role of NRSP trainers), BCC (IEC materials,

communication guidelines, etc.), supplies facilitation (training of

masons, latrine designs, demo latrines, marts/entrepreneurs),

health and hygiene awareness and rewards?

Q 19. To what extent the 3rd Party Process Monitoring (guidelines/tools,

reports and actual application) contributed to management

decision making in terms of efficient delivery (timeliness and

resources utilization), consistency with standards/quality? In

[158]

which ways the third party monitoring process/implementation

could be improved to help decision makers?

Q 20. To what extent the knowledge management interventions were

carried out as planned (please elaborate what were the

interventions - technical reviews, baseline/evaluation results, third

party monitoring, best practices, periodic reviews, research and

policy advocacy)? Which ways it helped accumulating

knowledge/resources, dissemination (internally and externally)

and informed the PATS standardization and delivery across

Pakistan (for all stakeholders involved in PATS)?

Q 21. To what extent the centralized capacity building arrangements

(through a partners NRSP) facilitated building capacities of

implementing partners to what extend it helped achieving time and

resource efficiencies (including contract management)? What were

the advantages/disadvantages of the centralization capacity

building approach and what needs to change to effectively build

capacities (of IPs and government) on sustainable basis?

[159]

SUSTAINABILITY Q 22. Did the project design have a sustainability approach/strategy for

the various initiatives undertaken? To what extent those been

implemented successfully, if not what needs to be

incorporated/changed in sustainability design and

implementation?

Q 23. [NOTE: From Implementing Partners Only]

What is the level of commitment amongst stakeholders (i.e. GoP, IPs, communities) to sustain the initiative without external support? In your view which of the project interventions and results may sustain whilst others may not? And why and what could be done to improve the sustainability of those elements in future?

UNFORESEEN EVENTS Q 24. Did the programme design (UNICEF and IP PCAs) have a Risk

Assessment and Mitigation Strategy (both for natural disaster and

conflict)? Was it clearly articulated and to what extent was it

implemented effectively?

[160]

Q 25. Was the project implementation affected at any time by security,

conflict or any disaster/emergency situation? If so, what aspects

were affected and what measures were taken to mitigate/respond

to the situation? Were the measures taken successful, if not then

why not? What are the lessons learned for future?

Note: Please ask if previous natural disaster response/recovery context was taken into account for design of interventions/approach in SPSP Rural.

GENDER & EQUITY Q 26. To what extent outputs and outcomes were designed from the

onset to contribute to attainment of gender (men, women, children,

older person, disabled, adolescent and others) equality/equity (in

relation to UNICEF policy)? Did the planned programme results

(including M&E systems) clearly state in measureable ways

contribution to gender equality, and reduction of possible gender

discrimination in any particular aspect? Can results of the

programme be disaggregated by sex, age, and disability?

[161]

Q 27. Did UNICEF or IPs undertake structured/guided (UNICEF and/or IP

guidelines) gender analysis for programme design and delivery (at

critical programme stages)? To what extent it helped mapping the

possible effect on burden on women/vulnerable groups, and/or

possible benefits (e.g. possible increase in girls’ retention in

schools)? In which ways the programme strategies and

interventions were realigned to the findings of gender analysis?

Q 28. In which ways the SPSP Rural facilitated larger policy and

implementation related changes as to promote gender equity (with

Government, Implementing Partners, Beneficiaries)? For instance,

in terms of gender policy formulation, implementation, positive

discrimination in staffing recruitments (equal number of field staff

SOs, beneficiary groups VSCs), project/intervention specific

gender analysis?

Q 29. To what extent the equity (vulnerability) principles were integrated

in the social mobilization, BCC, supplies facilitation, and others

and substantiated by programme results e.g. targeting of poorest

of poor for demo latrines, change in access to WASH services and

[162]

facilities for the lowest quintile? What processes evolved for

equity integration, how successful were those and what may need

to change in future?

Q 30. To what extent the programme supported the Government’s use of

PATS approach in making progress on the Environmental

Sustainability MDG i.e. Target 10: Halving, by 2015, the proportion

of people without sustainable access to safe drinking water and

basic sanitation?

Note: Did the programming try to address, directly or indirectly, the issues identified by the CRC, CEDAW and/or UPR in SPSP-Rural?

Q 31. To what extent and in which ways the HRBAP (Human Rights

Based Approaches to Programming) is manifested in SPSP Rural

programme design? Give examples of proposed approaches,

interventions, and results? What challenges if any were faced in

application of HRBAP?

Q 32. To what extent there was consistency in understanding of HRBAP

by UNICEF, IPs and government partners? If not, what was done to

bring all at same level and how successful has that been?

[163]

Q 33. What formal and informal mechanisms were put in place/existed

for duty bearers and rights holders to participate/have a voice in

the various activities/interventions of SPSP-Rural in terms of

programming planning, implementation, monitoring and

evaluation? In what ways participation was ensured?

Q 34. Has the programme ensured that all schools in the target area

have adequate child friendly water and sanitation facilities and

hygiene education programme? How was children participation

ensured in the process?

Q 35. How do you compare the application of the ‘respect, integrity’

approach vs ‘shame shock disgust’ approach in WASH in

Pakistan? Which one has been more effective and why?

[164]

Thank You for your time and candid response.

[165]

End of Programme Evaluation UNICEF’s Sanitation Programme at Scale in

Pakistan (SPSP) – Rural Through Pakistan Approach to Total Sanitation (PATS)

INDEPTH INTERVIEW

GOVERNMENT & POLITICIANS

Introduction to the Interviewee: I am , a member of the evaluation team deployed by an independent consulting entity i.e. the AAN Associates, who has been contracted for the End of Programme Evaluation of UNICEF’s Sanitation Programme at Scale in Pakistan. This programme aims to improve sanitation practices, using Community-led Total Sanitation (CLTS) and School-led Total Sanitation (SLTS) approaches under the overarching PATS strategy to ensure that the most vulnerable, marginalized children and women are using safe drinking water, improved sanitation and hygiene practices. The target areas include flood affected, high Polio risk and insecurity affected areas. I am here to get your reflections, experiences and recommendations about the programme, which will help us to assess, analyse and document programme progress, results and areas for further improvement. This will help at large in designing or developing future sanitation programmes in the country and/or extension or scale-up of this initiative. I thank you for taking time for this interview today. I shall be asking you questions about how the programme activities were implemented in your area and what changes this programme brought in the lives of the community in regard to improved access and availability of sanitation services. I anticipate the interview will last about 60 minutes, and appreciate any information you can provide. Please note that this interview is important for the success of this assessment. Your answers are completely confidential and shall remain with the evaluation team. At the start we seek your consent to make use of this information for analysis and reporting and may also cite it in the report while making reference to your name, if needed. Also, we are recording the proceeding to avoid any loss of information and seek your consent to record our discussion. Finally, we would request you to please allow us to make photographs of project interventions and yourself for evidence and communication purposes. Note: This is to underline that the IDI instrument has been designed as such to offer flexibility to the interviewers to adapt questions as to the specific role of the organization, respondents’ profile, and understanding of programme operations and overall knowledge of sector. Hence one consolidated tool has been developed for multiple stakeholders (as listed above).

Name of Respondent: _____________________ Designation _____________________ Organisation _____________________ Location: _____________________ Date of Interview: _____________________

Name of the Interviewer: _____________________

[166]

Interview Questions: Please listen carefully to the questions and provide your candid feedback.

Section 1: RELEVANCE Q 36. Is WASH one of the priority needs in rural (your district/province)

context? If yes, which regions/groups need the services most and

why (probe men, women, equity)? Which regions and groups are

at greatest risk for poor WASH conditions and what are the

implications (including potential)?

Q 37. Are you aware of UNICEF’s SPSP-Rural programme? Was there

any coordination between your department, UNICEF and the

Implementing Partners on geographic area selection (for instance

selection of districts, agencies, Tehsil, UCs and villages),? If yes,

was this based on any criteria? In your view do you feel that the

most deserving regions were selected for programme

interventions?

Q 38. What is the role of your dept. /organization in delivery of WASH

services especially PATS operationalization? Do you feel the

UNICEF SPSP is aligned to PATS principles and pillars (explore if

he/she looks at logic elements as - partnerships, capacity building,

demand creation, facilitation of supplies) and how?

Q 39. Have you/department been involved in SPSP design and

implementation, if yes how? Also, share your thoughts on

your/depts. role at any/various stages of the programme

(consultation, design, assessment, planning, co-ordination &

information sharing, sharing of technical resources, services

delivery, monitoring, evaluation and learning, policy advice)?

Q 40. Can you share your thoughts if this project i.e. SPSP (design and

implementation) could have benefitted more from engaging with

your organization and other public agencies for PATS

implementation & potential replication/scale-up, and how??

Q 41. In which ways and extent your department benefitted from the

engagement with the SPSP project e.g. District level capacity

enhancement, improved planning and identification and others?

What areas should the future programme focus (in terms of

capacity development) to help government departments adopt,

[167]

replicate and scale-up PATS? Probe more for ideas for future

programme design?

Q 42. To what extent is the project design consistent with National

Sanitation Policy (2006)/PATS and/or to Provincial policies on

water and sanitation? Are PATS or Provincial Strategy/Action

Plans available (for national and provincial stakeholders)?

Q 43. Is your department/organization involved in harmonizing efforts

for PATS implementation and if yes what practical steps been

taken (for instance working groups, coordination meetings etc) to

align the efforts of government and development partners in

WASH/PATS? If these been successful, how and if not what needs

to be done and how UNICEF/other development stakeholders

could facilitate that?

Q 44. PATS outlines engaging multiple stakeholders and also

introducing CATS and SLTS? Do you find two approaches

complementary? Did SPSP achieve desired results by using both

models, please elaborate and how your dept contributed or

potentially contribute to application of either one or both?

[168]

Section 2: Effectiveness

Q 45. To what extent you see the SPSP programme (in particular your

organization given engagement) has been able to achieve the

desired outputs/outcomes? In which particular areas the SPSP

performed better and less in others and why?

Q 46. To what extent the SPSP has been in application of various PATS

pillars i.e. mobilization, demand creation, supplies facilitation,

achieving and sustaining ODF, rewards? Probe into the reasons

for their assessment/perception and share your dept./organization

contributions? Which areas (design and implementation) the next

project needs to improve to achieve better results?

Q 47. To what extent this project has contributed to building capacities

of public sector partners in particular your organization in order to

achieved desired results? How capacity needs were appraised,

what strategies were evolved to address needs (training, systems

development, equipment, finances and others)? Which ones

worked well & why – why others were not as successful?

Q 48. To what extent the engagement in SPSP and capacity development

have lead to institutionalization of PATS in terms of new policies,

structures, systems, practices within your department? What are

the factors that are still hindering PATS institutionalization?

Section 3: Efficiency

Q 49. To what extent do you find that there was a balance between the

project targets and resources allocated/utilized (technical,

financial and time)? Probe more, if respondent is aware only of

public organizations targets, times and resources?

Q 50. In your view were any measures put in place to avoid/control

wastage and misuse of resources at all levels (performance

management, audits, others) while working with partners as well

[169]

as government? What were those measures and to what extent

those measures contributed to achieving transparency,

accountability and cost efficiencies, please explain?

Q 51. Are you familiar with any cost benefit analysis undertaken of

CATS/CLTS implementation (within public sector or other non-

profit sector inside Pakistan or abroad) approaches in terms

reaching out with messages, convincing to construct latrine and

others? if yes please share references/documents?

Q 52. What changes could be introduced in the project design and

implementation to achieve maximum resource efficiencies? If yes,

please share what are those and how would these contribute to

achieving resource efficiency?

Section 4: SUSTAINABILITY

Q 53. In your view, if SPSP had an articulated sustainability

approach/strategy for project interventions particularly those in

which your organization partnered? How successful has that been

implemented and if that links it with relevant government depts.?

Q 54. What is the level of commitment amongst stakeholders (i.e. GoP,

IPs, communities) to sustain the initiative without external

support? In your view which of the project interventions and

results may sustain whilst others may not? And why?

Q 55. What is the level of GOP commitment to PATS/Rural sanitation

replication & scale-up in terms of increased budgetary allocation,

organizational restructuring, improved multi-stakeholder

coordination for planning, implementation and monitoring, intra-

government understanding of PATS to deliver better results)?

Section 5: GENDER, EQUITY & RIGHTS BASED APPROACH Q 56. To what extent the SPSP design and implementation (mobilization,

BCC, supplies facilitation, rewards) is informed of gender analysis

in terms of engaging and benefitting varied groups particularly

vulnerable (children, women, older person, men, disabled)? In your

[170]

view if any of the project interventions caused any negative impact

on gender relations & exclusion, please elaborate?

Q 57. To what extent the SPSP design and implementation (mobilization,

BCC, supplies facilitation, rewards) is informed of equity

considerations in terms of engaging and benefitting poor and

marginalized (poor, disabled)? In your view if any of the project

interventions caused any negative impact on equity divide &

exclusion, please elaborate?

Q 58. Did the programme support Government’s use of PATS approach

in making progress on the Environmental Sustainability MDG i.e.

Target 10: Halving, by 2015, the proportion of people without

sustainable access to safe drinking water and basic sanitation?

Did the programming try to address, directly or indirectly, the

issues identified by the CRC, CEDAW and/or UPR in SPSP-Rural?

Thank You for your time and candid response.

[171]

End of Programme Evaluation

UNICEF’s Sanitation Programme at Scale in Pakistan (SPSP) – Rural Through

Pakistan Approach to Total Sanitation (PATS)

FOCUS GROUP DISCUSSION (FGDS) CHILDREN

(Boys & Girls)

Introduction to the Interviewee: I am , a member of the evaluation team deployed by an independent consulting entity i.e. the AAN Associates, who has been contracted for the End of Programme Evaluation for UNICEF’s Sanitation Programme at Scale in Pakistan. This programme aims to improve sanitation practices, using Community-led Total Sanitation (CLTS) and School-led Total Sanitation (SLTS) approaches under the overarching PATS strategy to ensure that the most vulnerable, marginalized children and women are using safe drinking water, improved sanitation and hygiene practices. The target areas include flood affected, high Polio risk and insecurity affected areas. We thank you for taking time for this focus group today. We shall be asking you questions about how the programme activities were implemented and what changes this programme brought in your life. During the FDG, please help yourself to tea and water. If you need to get up to use the rest room, feel free. We expect the focus group to take 1 to 1-½. We shall record/document the proceedings and seek your consent to allow us to use this discussion and/or interpretation for our report. Purpose: Before we get started, we would like to give you a little background about focus groups and why we are doing this focus group discussion with you. Focus groups are intended to give us a sense of how a group of people feels about a certain subject. We often want confirmation that others feel this way, so we might follow up a question with something like “Do others feel this way” or “Is this important for other people to know about?” We want to hear from everyone, and so at times we may interact to help the conversation keep moving. We will be taking your comments and summarizing them to share with the evaluation team and the programme implementers. You and your individual opinions won’t be identified in any way.

[172]

Location: Village ____________________

Tehsil ____________________ District ____________________

Date: / / Participants List:

Name Age Name Age

1. 2.

3. 4.

5. 6.

7. 8.

9. 10.

Guidelines For Participants: Before start of discussion, we would like to let you about some recommended guidelines or “ground rules” that help establish the group norms so that the group proceeds smoothly and respectfully for all participants:

Only one person talks at a time.

Be sure about the Confidentiality. “What is shared in the room stays in the room.”

It is important for us to hear everyone’s ideas and opinions. There are no right or wrong answers to questions – just ideas, experiences and opinions, which are all valuable.

It is important for us to hear all sides of an issue – both the positive and the negative.

Focus Group Questions: Please listen carefully to the questions and provide your feedback as per the guidelines provided.

Q-1: Over the last year has there been any changes/improvements in

the school water and sanitation facilities? What has

changed/improved and share if you know which organization/s

have helped improve these facilities?

[173]

Q-2: Can you please share your impressions (to assess child friendly

facilities)?

a) Are the facilities sufficient in number (or are there long queues)?

b) Do the facilities have soap, running water, and water/ablution

jug/mug?

c) Is the height of the water stand post accessible for children?

d) Are the latrines properly ventilated and clean at all time?

[174]

e) Can all children (girls, boys as well as disabled children) access

the Water and Sanitation facilities equitably?

f) Is the area around the latrine and water stand post kept clean?

g) Do the latrine facilities provide privacy?

h) Are all facilities still functional and in use of school children? Are

there any which are either non-functional or inaccessible to you

(used by teachers or others)?

Q-3: Can you share if you/children were involved in the process of

identification, design, implementation and sustainable use of

facilities (design, location, number etc.)?

Q-4: In which you/children benefitted from improved facilities – probe in

terms of benefits for boys and girls, children of different age

groups, reduced student sickness and absenteeism, lower

dropouts, reduction in open defecation and others?

[175]

Q-5: Is there a Wash Club in your school? If yes, how & when was it

formed and who are its members? What has it done for highlight

water, sanitation and health/hygiene issues during the

implementation of SPSP, Please elaborate?

Q-6: Have you received health and hygiene messages in last one year,

what were those messages about and which organizations

provided that information? Probe what messages and delivery

methods were found useful and why? Ask children to recall and

share at least 1 message/ learning with regards to water, sanitation

and health/hygiene?

Q-7: How many of you have adopted health and hygiene learning

(messages/practices) in school? If yes how? If not, why? Have you

passed on this information to your families/village, and did you

see any changes in family/neighbours health and hygiene

practices? Probe: how many of the students had washed hands

that day? When and how? Why is essential to wash hands? List

the most essential times when we must wash our hands? Are their

children at school and adults in your community who still continue

to defecate in open? Why?

Thank You for your time and candid response.

[176]

End of Programme Evaluation

UNICEF’s Sanitation Programme at Scale in Pakistan (SPSP) – Rural Through

Pakistan Approach to Total Sanitation (PATS)

FOCUS GROUP DISCUSSION (FGDS) SOCIAL ORGNIZERS (SOs)

(MEN AND WOMEN)

Introduction to the Interviewee: I am ,a member of the evaluation team deployed by an independent consulting entity i.e. the AAN Associates, who has been contracted for the End of Programme Evaluation for UNICEF’s Sanitation Programme at Scale in Pakistan. This programme aims to improve sanitation practices, using Community-led Total Sanitation (CLTS) and School-led Total Sanitation (SLTS) approaches under the overarching PATS strategy to ensure that the most vulnerable, marginalized children and women are using safe drinking water, improved sanitation and hygiene practices. The target areas include flood affected, high Polio risk and insecurity affected areas. We thank you for taking time for this focus group today. We shall be asking you questions about how the programme activities were implemented and what changes this programme brought in your life. During the FDG, please help yourself to tea and water. If you need to get up to use the rest room, feel free. We expect the focus group to take 1 to 1-½. We shall record/document the proceedings and seek your consent to allow us to use this discussion and/or interpretation for our report. Purpose: Before we get started, we would like to give you a little background about focus groups and why we are doing this focus group discussion with you. Focus groups are intended to give us a sense of how a group of people feels about a certain subject. We often want confirmation that others feel this way, so we might follow up a question with something like “Do others feel this way” or “Is this important for other people to know about?” We want to hear from everyone, and so at times we may interact to help the conversation keep moving. We will be taking your comments and summarizing them to share with the evaluation team and the programme implementers. You and your individual opinions won’t be identified in any way.

[177]

Location: Village ____________________

Tehsil ____________________ District ____________________

Date: / / Participants List:

Name Age Name Age

11. 12.

13. 14.

15. 16.

17. 18.

19. 20.

Guidelines For Participants: Before start of discussion, we would like to let you about some recommended guidelines or “ground rules” that help establish the group norms so that the group proceeds smoothly and respectfully for all participants:

Only one person talks at a time.

Be sure about the Confidentiality. “What is shared in the room stays in the room.”

It is important for us to hear everyone’s ideas and opinions. There are no right or wrong answers to questions – just ideas, experiences and opinions, which are all valuable.

It is important for us to hear all sides of an issue – both the positive and the negative.

[178]

Focus Group Questions: Please listen carefully to the questions and provide your feedback as per the guidelines provided.

Q-8: What were priority development needs in the community? Was

Water, Sanitation and Health & Hygiene (sanitation in particular)

amongst the priority needs of the people? If yes, what are the

causes of poor WASH conditions and how/who gets affected the

most by that?

Q-9: What is your understanding of the intended role of SOs (trigger

agents, motivation, social mobilization, demand creation, bridge

between the sanitation mart, VSC and beneficiaries, and others)?

To what extent you feel that SOs have been able to deliver on their

envisaged/expected role?

Q-10: Did the processes undertaken during the PRA exercises (Poverty

and Social Mapping) and Social Mobilization help identify and

target the poorest (lowest quintiles), the vulnerable including the

disabled as a priority? If so, what was done to address their WASH

related needs? How was the equity focus maintained during the

identification, selection and implementation?

[179]

Q-11: To what extent were the mobilization and facilitation strategies

(including topics/issues chosen for community awareness)

relevant to community needs (of different ages/sex)? Did your

training contents, training delivery, duration, IEC materials

(messages, medium, materials), community session plan

(duration, schedule and coverage area), help in raising your

understanding & ability (facilitated) to counsel/educate community

beneficiaries (men, women and others) & How?

Q-12: To what extent was the social mobilization, VSC, CRP formation,

construction of demo latrines, and linkages with supply side

interventions effective? What can be done differently in the future

to make it more effect?

Q-13: To what extent was your message responded to by different

groups? Which groups (of men, women, children and others

including the disabled) were more receptive and why? What

messages & issues people (men, women, children and others

including disabled) were more attentive to and adapted early and

why? Which mediums of communication (talks, posters, electronic

messages – radio, tv, theatre and others) were people (of age and

sex) more receptive of and why?

[180]

Q-14: To what extent you feel you were provided necessary/adequate

support (e.g. training, time, IEC materials, coverage area and

supervision) by the project to deliver on the expected roles? If not

what were the constraints?

Q-15: How has the project changed the perception of improved WASH

services and who has benefitted the most (men, women, children,

or others including the disabled) and in what ways? What changes

have been/can be observed as a result of the various programme

outputs? How were these changes brought about in your villages?

Q-16: To what extent you feel that the project approaches/interventions

were most suited/appropriate and of the quality to address the

varied needs (of different groups),

a) Community mobilization (including the strategy, use of PRAs,

usage and availability of time, skills of SOs/CRPs, status of

VSCs, role of NRSP Trainers)

b) Behavior Change Communication (demand generation)

activities (IEC, Print, Electronic, Community Shows),

[181]

Q-17: What were the key features of Community Action Plans (CAPs)

developed by the VSC? To what extent interventions of different

community groups were included in the plan e.g. Joint activities of

male & female VSC, SLTS and others? To what extent did project

provide adequate and timely resources (community

mobilization/education, technical training, and availability of low

cost toilet supplies) for implementation of this plan?

Q-18: To what extent project interventions were designed and

implemented from the onset to contribute to attainment of gender

(men, women, children, older person, disabled, adolescent and

others) equality/equity (in relation to UNICEF policy)? Was a

gender analysis done? What positive changes have been

supported through this programme (with Government,

Implementing Partners, and Beneficiaries) that have contributed to

gender equality? Were there unintended/negative effects of the

programme implementation (such as increase in work burden)?

Q-19: Did the programme have an equity focus (as per UNICEF

guidelines for instance vulnerability criteria)? To what extent the

equity principles were integrated in the social mobilization, BCC,

supplies facilitation, etc? How did the interventions cater to the

needs of the disabled or women/children?

[182]

Q-20: Did your village achieve the ODF status, is it certified /notified by

Government? What rewards and incentives has your village been

able to draw down from the Government? Are there any issues in

accessing the rewards/incentives? In your opinion has the

rewards/incentives approach been an effective one?

Q-21: In your view which of the project interventions and results may

sustain (ODF status, CRPs, VSC, hand wash practice, construction

of low cost latrines, sanitary marts, entrepreneurs)? Which

interventions may not sustain and why? What should be done

differently in future to achieve better results?

Q-22: What experiences/lessons were learned around programme design

and implementation (list what worked, what did not and why?

Substantiate with measure of success, scale, and lessons learnt

for future)?

Note: The analysis to be desegregated for following (components/subcomponents)

10. DEMAND CREATION

What Worked What Didn’t Work

Lessons Learnt Success Scale

Rank

Highly

Successful

Moderate

Poor

Fail

[183]

11. SUPPLY FACILITATION

What Worked What Didn’t Work

Lessons Learnt Success Scale

Rank

Highly

Successful

Moderate

Poor

Fail

12. HEALTH AND HYGIENE PROMOTION

What Worked What Didn’t Work

Lessons Learnt Success Scale

Rank

Highly

Successful

Moderate

Poor

Fail

13. LINKAGES WITH DUTY BEARERS FOR PATS

What Worked What Didn’t Work

Lessons Learnt Success Scale

Rank

Highly

Successful

Moderate

Poor

Fail

14. VILLAGES SUPPORTED IN TOTAL SANITATION STATUS

What Worked What Didn’t Work

Lessons Learnt Success Scale

Rank

Highly

[184]

Successful

Moderate

Poor

Fail

15. PATS KNOWLEDGE MANAGEMENT FRAMEWORK

What Worked What Didn’t Work

Lessons Learnt Success Scale

Rank

Highly

Successful

Moderate

Poor

Fail

16. PARTNERSHIP MANAGEMENT

What Worked What Didn’t Work

Lessons Learnt Success Scale

Rank

Highly

Successful

Moderate

Poor

Fail

17. INTEGRATION OF CROSS CUTTING PRIORITIES

(Gender & Equity, Unforeseen Events, Rights Based Approach) What Worked What Didn’t

Work Lessons Learnt Success

Scale Rank

Highly

Successful

Moderate

Poor

Fail

[185]

18. SCHOOL LED TOTAL SANITATION

What Worked What Didn’t Work

Lessons Learnt Success Scale

Rank

Highly

Successful

Moderate

Poor

Fail

Thank You for your time and candid response.

[186]

End of Programme Evaluation

UNICEF’s Sanitation Programme at Scale in Pakistan (SPSP) – Rural Through

Pakistan Approach to Total Sanitation (PATS)

FOCUS GROUP DISCUSSION (FGDS) COMMUNITY RESOURCE PERSONS (CRPs)

(MEN AND WOMEN)

Introduction to the Interviewee: I am , a member of the evaluation team deployed by an independent consulting entity i.e. the AAN Associates, who has been contracted for the End of Programme Evaluation for UNICEF’s Sanitation Programme at Scale in Pakistan. This programme aims to improve sanitation practices, using Community-led Total Sanitation (CLTS) and School-led Total Sanitation (SLTS) approaches under the overarching PATS strategy to ensure that the most vulnerable, marginalized children and women are using safe drinking water, improved sanitation and hygiene practices. The target areas include flood affected, high Polio risk and insecurity affected areas. We thank you for taking time for this focus group today. We shall be asking you questions about how the programme activities were implemented and what changes this programme brought in your life. During the FDG, please help yourself to tea and water. If you need to get up to use the rest room, feel free. We expect the focus group to take 1 to 1-½. We shall record/document the proceedings and seek your consent to allow us to use this discussion and/or interpretation for our report. Purpose: Before we get started, we would like to give you a little background about focus groups and why we are doing this focus group discussion with you. Focus groups are intended to give us a sense of how a group of people feels about a certain subject. We often want confirmation that others feel this way, so we might follow up a question with something like “Do others feel this way” or “Is this important for other people to know about?” We want to hear from everyone, and so at times we may interact to help the conversation keep moving. We will be taking your comments and summarizing them to share with the evaluation team and the programme implementers. You and your individual opinions won’t be identified in any way.

[187]

Location: Village ____________________

Tehsil ____________________ District ____________________

Date: / / Participants List:

Name Age Name Age

21. 22.

23. 24.

25. 26.

27. 28.

29. 30.

Guidelines for Participants: Before start of discussion, we would like to let you about some recommended guidelines or “ground rules” that help establish the group norms so that the group proceeds smoothly and respectfully for all participants:

Only one person talks at a time.

Be sure about the Confidentiality. “What is shared in the room stays in the room.”

It is important for us to hear everyone’s ideas and opinions. There are no right or wrong answers to questions – just ideas, experiences and opinions, which are all valuable.

It is important for us to hear all sides of an issue – both the positive and the negative.

Focus Group Questions: Please listen carefully to the questions and provide your feedback as per the guidelines provided.

Q-23: Are you familiar with UNICEF SPSP? What has been its objectives

and approach, please share your understanding? To what extent

the programme has been able to achieve desired results?

[188]

Q-24: Is WASH (particularly Sanitation) one of the priority needs in the

rural context (your villages/regions)? If yes, which groups need the

services most and why (probe men, women, children, poor)?

Which groups are at greatest risk for poor WASH (sanitation in

particular) conditions and what are its implications (including

potential) for rural communities?

Q-25: What is your understanding of the intended role of CRPs (trigger

agents, motivation, demand creation, bridge between the

sanitation mart, VSC and beneficiaries, and others)? To what

extent you feel that CRPs have been able to deliver on their

envisaged/expected role? If yes how and if not why?

Q-26: To what extent the topics/issues chosen for community awareness

were relevant to community needs (of different ages/sexes)? Did

your training contents, training delivery, duration, IEC materials

(messages, medium, materials), community session plan

(duration, schedule and coverage area), help in raising your

understanding & ability (facilitated) to counsel/educate community

(men, women and others) & How?

[189]

Q-27: Which mediums of communication (talks, posters, electronic

messages – radio, tv, theatre and others) were people (of age and

sex) more receptive of and why? How could this be made more

effective?

Q-28: Please share your impressions of the most visible changes that

this programme overall and CRP lead interventions/BCC have

been able to achieve? Which groups (e.g. men, women, children,

older people) have been relatively more receptive to BCC/H&H

awareness (attentive, demonstrated attitudinal and behavioural

change) and why others been not so receptive?

Q-29: To what extent you feel you were provided necessary/adequate

and timely support e.g. training, financial compensation, time, IEC

materials, coverage area, supervision) to enable you to deliver on

the expected roles? If yes, please elaborate and if not why/how?

Q-30: Which of the project achievements e.g. community groups, ODF,

changed H&H behaviours, CRPs and others, may sustain after

completion of the programme and why? Why others may not

sustain? What needs to be done to create self-reinforcement and

sustaining the programme achievements within communities

please elaborate?

[190]

Thank You for your time and candid response.

End of Programme Evaluation UNICEF’s Sanitation Programme at Scale in

Pakistan (SPSP) – Rural Through Pakistan Approach to Total Sanitation (PATS)

FOCUS GROUP DISCUSSION (FGDS)

[191]

Marts Owners/Entrepreneurs

Introduction to the Interviewee: I am , a member of the evaluation team deployed by an independent consulting entity i.e. the AAN Associates, who has been contracted for the End of Programme Evaluation for UNICEF’s Sanitation Programme at Scale in Pakistan. This programme aims to improve sanitation practices, using Community-led Total Sanitation (CLTS) and School-led Total Sanitation (SLTS) approaches under the overarching PATS strategy to ensure that the most vulnerable, marginalized children and women are using safe drinking water, improved sanitation and hygiene practices. The target areas include flood affected, high Polio risk and insecurity affected areas. We thank you for taking time for this focus group today. We shall be asking you questions about how the programme activities were implemented and what changes this programme brought in your life. During the FDG, please help yourself to tea and water. If you need to get up to use the rest room, feel free. We expect the focus group to take 1 to 1-½. We shall record/document the proceedings and seek your consent to allow us to use this discussion and/or interpretation for our report. Purpose: Before we get started, we would like to give you a little background about focus groups and why we are doing this focus group discussion with you. Focus groups are intended to give us a sense of how a group of people feels about a certain subject. We often want confirmation that others feel this way, so we might follow up a question with something like “Do others feel this way” or “Is this important for other people to know about?” We want to hear from everyone, and so at times we may interact to help the conversation keep moving. We will be taking your comments and summarizing them to share with the evaluation team and the programme implementers. You and your individual opinions won’t be identified in any way. Location: Village ____________________

Tehsil ____________________ District ____________________

Date: / / Participants List:

[192]

Name Age Name Age

31. 32.

33. 34.

35. 36.

37. 38.

39. 40.

Guidelines for Participants: Before start of discussion, we would like to let you about some recommended guidelines or “ground rules” that help establish the group norms so that the group proceeds smoothly and respectfully for all participants:

Only one person talks at a time.

Be sure about the Confidentiality. “What is shared in the room stays in the room.”

It is important for us to hear everyone’s ideas and opinions. There are no right or wrong answers to questions – just ideas, experiences and opinions, which are all valuable.

It is important for us to hear all sides of an issue – both the positive and the negative.

Focus Group Questions: Please listen carefully to the questions and provide your feedback as per the guidelines provided.

Q-31: Are you familiar with UNICEF SPSP, if yes, what do you know

about the programme? To what extent the programme has been

able to achieve desired results?

[193]

Q-32: How have you been approached/engaged (in supplies facilitation)

in the SPSP? To what extent has this project increased

community demand for sanitation supplies? Probe if number of

latrine materials sold has increased, ask for monthly sale of latrine

materials?

Q-33: In your view how has the introduction of entrepreneurs/marts

benefitted the communities (for mart owners probe further on

people, price, promotion, product and placement)? To what extent

has it contributed to facilitating community access to latrine

material supplies (lower costs, less transportation, year round

availability, quality controls and others) Probe further and ask for

examples?

Q-34: To what extent the project allocated resources (technical, financial

and time) for supplies facilitation were adequate e.g. number of

marts, entrepreneurs, products diversity, timing of establishing

facilitation chain, promotion, and others? In case of any disruption

in supply chain, was project able to identify and resolve the supply

chain disruption in time?

[194]

Q-35: What changes have happened in the latrine/ sanitation supplies

market because of this project (prices, products diversity,

competition, quality)? What could have been done to achieve

better results?

Q-36: Do you plan to continue after project completion? Do you foresee

the demand for latrines would continue to increase or otherwise

after completion of the project? What could this project do (or

future phase) for continued demand creation and provision of

materials?

Q-37: What do you feel were some of the factors that either facilitated or

hindered your work? What can be done in to enable easy access to

products/services for construction of household latrines?

[195]

Thank You for your time and candid response.

[196]

End of Programme Evaluation UNICEF’s Sanitation Programme at Scale in

Pakistan (SPSP) – Rural Through Pakistan Approach to Total Sanitation (PATS)

FOCUS GROUP DISCUSSION (FGDS)

MASONs

Introduction to the Interviewee: I am , a member of the evaluation team deployed by an independent consulting entity i.e. the AAN Associates, who has been contracted for the End of Programme Evaluation for UNICEF’s Sanitation Programme at Scale in Pakistan. This programme aims to improve sanitation practices, using Community-led Total Sanitation (CLTS) and School-led Total Sanitation (SLTS) approaches under the overarching PATS strategy to ensure that the most vulnerable, marginalized children and women are using safe drinking water, improved sanitation and hygiene practices. The target areas include flood affected, high Polio risk and insecurity affected areas. We thank you for taking time for this focus group today. We shall be asking you questions about how the programme activities were implemented and what changes this programme brought in your life. During the FDG, please help yourself to tea and water. If you need to get up to use the rest room, feel free. We expect the focus group to take 1 to 1-½. We shall record/document the proceedings and seek your consent to allow us to use this discussion and/or interpretation for our report. Purpose: Before we get started, we would like to give you a little background about focus groups and why we are doing this focus group discussion with you. Focus groups are intended to give us a sense of how a group of people feels about a certain subject. We often want confirmation that others feel this way, so we might follow up a question with something like “Do others feel this way” or “Is this important for other people to know about?” We want to hear from everyone, and so at times we may interact to help the conversation keep moving. We will be taking your comments and summarizing them to share with the evaluation team and the programme implementers. You and your individual opinions won’t be identified in any way.

[197]

Location: Village ____________________

Tehsil ____________________ District ____________________

Date: / / Participants List:

Name Age Name Age

41. 42.

43. 44.

45. 46.

47. 48.

49. 50.

Guidelines for Participants: Before start of discussion, we would like to let you about some recommended guidelines or “ground rules” that help establish the group norms so that the group proceeds smoothly and respectfully for all participants:

Only one person talks at a time.

Be sure about the Confidentiality. “What is shared in the room stays in the room.”

It is important for us to hear everyone’s ideas and opinions. There are no right or wrong answers to questions – just ideas, experiences and opinions, which are all valuable.

It is important for us to hear all sides of an issue – both the positive and the negative.

[198]

Focus Group Questions: Please listen carefully to the questions and provide your feedback as per the guidelines provided.

Q-38: Are you familiar with UNICEF SPSP, if yes, what do you know

about the programme? To what extent the programme has been

able to achieve desired results?

Q-39: How have you been approached/engaged (in supplies facilitation)

in the SPSP? To what extent has this project increased

community demand, supplies facilitation including creation of

skilled labour force? Probe if number of latrine constructed every

month increased in the last one year?

Q-40: To what extent you find masons’ training useful/effective/required (

in terms of introducing new knowledge & skills, timing and

duration, contents and methodology, skills of trainer) in

addressing design/implementation issues to construct safer

latrines? In your view what could be improved in future e.g.

additional contents, different methods, and time/duration?

[199]

Q-41: To what extent the programme interventions (especially design of

latrines, masons training, and marts/entrepreneurs) have helped

making the availability of hardware and skills for safer, quicker,

contextually/environmentally relevant, low cost latrines? Probe

how?

Q-42: To what extent has this project been able to introduce new/low

cost/context specific and long lasting (including environment

friendly) technologies and designs? Which designs and products

are more in demand and why? Are these latrines durable (life

span), what are the tentative maintenance/utility costs and can

community afford that?

Q-43: To what extent the project allocated resources (technical, financial

and time) for supplies facilitation were adequate e.g. number of

marts, entrepreneurs, products diversity, timing of establishing

facilitation chain, promotion, and others? In case of any disruption

in supply chain, was project able to identify and resolve the supply

chain disruption in time?

[200]

Q-44: Who (groups and individuals) has benefitted most because of

latrines construction and how? Are you familiar with project

financed demo latrines and subsidies, if yes, to what extent those

been given to the most deserving? Did it contribute to replication

(at least design and technologies), if not why?

Q-45: To what extent do you feel that this increased demand, improved

supplies facilitation (hard ware and skills) would continue after

closure of the programme? If yes how/why and if not why?

Thank You for your time and candid response.

[201]

End of Programme Evaluation UNICEF’s Sanitation Programme at Scale in

Pakistan (SPSP) – Rural Through Pakistan Approach to Total Sanitation (PATS)

FOCUS GROUP DISCUSSION (FGDS)

Teachers (Male and Female) School Management Committees

Introduction to the Interviewee: I am , a member of the evaluation team deployed by an independent consulting entity i.e. the AAN Associates, who has been contracted for the End of Programme Evaluation for UNICEF’s Sanitation Programme at Scale in Pakistan. This programme aims to improve sanitation practices, using Community-led Total Sanitation (CLTS) and School-led Total Sanitation (SLTS) approaches under the overarching PATS strategy to ensure that the most vulnerable, marginalized children and women are using safe drinking water, improved sanitation and hygiene practices. The target areas include flood affected, high Polio risk and insecurity affected areas. We thank you for taking time for this focus group today. We shall be asking you questions about how the programme activities were implemented and what changes this programme brought in your life. During the FDG, please help yourself to tea and water. If you need to get up to use the rest room, feel free. We expect the focus group to take 1 to 1-½. We shall record/document the proceedings and seek your consent to allow us to use this discussion and/or interpretation for our report. Purpose: Before we get started, we would like to give you a little background about focus groups and why we are doing this focus group discussion with you. Focus groups are intended to give us a sense of how a group of people feels about a certain subject. We often want confirmation that others feel this way, so we might follow up a question with something like “Do others feel this way” or “Is this important for other people to know about?” We want to hear from everyone, and so at times we may interact to help the conversation keep moving. We will be taking your comments and summarizing them to share with the evaluation team and the programme implementers. You and your individual opinions won’t be identified in any way.

[202]

Location: Village ____________________

Tehsil ____________________ District ____________________

Date: / / Participants List:

Name Age Name Age

51. 52.

53. 54.

55. 56.

57. 58.

59. 60.

Guidelines for Participants: Before start of discussion, we would like to let you about some recommended guidelines or “ground rules” that help establish the group norms so that the group proceeds smoothly and respectfully for all participants:

Only one person talks at a time.

Be sure about the Confidentiality. “What is shared in the room stays in the room.”

It is important for us to hear everyone’s ideas and opinions. There are no right or wrong answers to questions – just ideas, experiences and opinions, which are all valuable.

It is important for us to hear all sides of an issue – both the positive and the negative.

[203]

Focus Group Questions: Please listen carefully to the questions and provide your feedback as per the guidelines provided.

Q-46: Over the last year has there been any changes/improvements in

the school water and sanitation facilities? What has

changed/improved and share if you know which organization/s

have helped improve these facilities?

Q-47: Can you please share your impressions (to assess child friendly

facilities)?

i) Are the facilities sufficient in number (or are there long queues)?

j) Do the facilities have soap, running water, and water/ablution

jug/mug?

k) Is the height of the water stand post accessible for children?

l) Are the latrines properly ventilated and clean at all time?

[204]

m) Can all children (girls, boys as well as disabled children) access

the Water and Sanitation facilities equitably?

n) Is the area around the latrine and water stand post kept clean?

o) Do the latrine facilities provide privacy?

p) Are all facilities still functional and in use of school children? Are

there any which are either non-functional or inaccessible to

you/children (used by teachers or others)?

Q-48: In which ways and extent you/children benefitted from improved

facilities – probe in terms of benefits for boys and girls, children of

different age groups, reduced student sickness and absentism,

lower dropouts, reduction in open defecation and others?

Q-49: Can you share if you/children were involved in the process of

identification, design, implementation and sustainable use of

facilities (design, location, number etc.)?

[205]

Q-50: Is there a SMC/Wash Club in your school? If yes, how & when was

it formed and who are its members (both SMC/WC)? What has it

done for highlight/address the water, sanitation and health/hygiene

issues in last one year, Please elaborate?

Q-51: Have you/children received health and hygiene messages in last

one year, what were those messages about, which organization/s

provided that information and by whom (teachers, CRP, SOs

others)? Probe what messages and delivery methods were found

useful and why?

Q-52: In your view has the approach of engaging schools (teachers,

children) played a role in improving knowledge and helping

change practices around of water, sanitation and health/hygiene

both at school and community levels?

Q-53: How schools (children/teachers/parents) could be involved more

effectively to improve school WASH and catalysing change in the

larger community (around WASH)?

[206]

Thank You for your time and candid response.

End of Programme Evaluation UNICEF’s Sanitation Programme at Scale in

Pakistan (SPSP) – Rural Through Pakistan Approach to Total Sanitation (PATS)

FOCUS GROUP DISCUSSION (FGDS)

VILLAGE SANITATION COMMITTEE (VSC) (MEN AND WOMEN GROUPS)

Introduction to the Interviewee: I am , a member of the evaluation team deployed by an independent consulting entity i.e. the AAN Associates, who has been contracted for the End of Programme Evaluation for UNICEF’s Sanitation Programme at Scale in Pakistan. This programme aims to improve sanitation practices, using Community-led Total Sanitation (CLTS) and School-led Total Sanitation (SLTS) approaches under the overarching PATS strategy to ensure that the most vulnerable, marginalized children and women are using safe drinking water, improved sanitation and hygiene practices. The target areas include flood affected, high Polio risk and insecurity affected areas. We thank you for taking time for this focus group today. We shall be asking you questions about how the programme activities were implemented and what changes this programme brought in your life. During the FDG, please help yourself to tea and water. If you need to get up to use the rest room, feel free. We expect the focus group to take 1 to 1-½. We shall record/document the proceedings and seek your consent to allow us to use this discussion and/or interpretation for our report. Purpose: Before we get started, we would like to give you a little background about focus groups and why we are doing this focus group discussion with you. Focus groups are intended to give us a sense of how a group of people feels about a certain subject. We often want confirmation that others feel this way, so we might follow up a question with something like “Do others feel this way” or “Is this important for other people to know about?” We want to hear from everyone, and so at times we may interact to help the conversation keep moving. We will be taking your comments and summarizing them to share with the evaluation team and the programme implementers. You and your individual opinions won’t be identified in any way.

[207]

Location: Village ____________________

Tehsil ____________________ District ____________________

Date: / / Participants List:

Name Age Name Age

61. 62.

63. 64.

65. 66.

67. 68.

69. 70.

Guidelines For Participants: Before start of discussion, we would like to let you about some recommended guidelines or “ground rules” that help establish the group norms so that the group proceeds smoothly and respectfully for all participants:

Only one person talks at a time.

Be sure about the Confidentiality. “What is shared in the room stays in the room.”

It is important for us to hear everyone’s ideas and opinions. There are no right or wrong answers to questions – just ideas, experiences and opinions, which are all valuable.

It is important for us to hear all sides of an issue – both the positive and the negative.

[208]

Focus Group Questions: Please listen carefully to the questions and provide your feedback as per the guidelines provided.

Q-54: What are priority development needs in your community? Probe if

Water, Sanitation and Health & Hygiene (sanitation in particular) is

one amongst the priority needs? If yes, what are the causes of

poor WASH conditions and how/who gets affected the most by

that?

Q-55: When was the VSC formed and how was it formed? Probe if VSC

process of participatory and all inclusive (all key groups are

represented)? What were the key functions of VSC and what have

you achieved with respect to the anticipated roles (assessment,

identification of vulnerable households, village development

planning, management of interventions, rewards management?

Please elaborate? To what extent VSC was involved in design of

interventions and implementation?

Q-56: Which organizations/individuals have worked on WASH with you

over the last year? To what extend did you find the interventions

and implementation approach relevant to your needs/context

(mobilization, demo latrines construction, supply side facilitation,

BCC etc.) and been able to successfully address the issues?

Probe if they feel there were any gaps, constraints and

weaknesses in the design and implementation?

[209]

Q-57: To what extent did organization/project provide adequate and

timely resources (community mobilization/education, technical

training, availability of low cost toilet supplies) for implementation

of this plan? Were the most deserving received the demo latrines

and subsidies (assess if VSC are satisfied with targeting)?

Q-58: What has changed in the village because of the project/VSC

efforts? Probe more with respect to village achieving ODF,

changes in the village environment, changes in people’s

perception and practices, impact on health and reduced diseases,

work load for men/women)? Which groups i.e. men, women,

children, or others including the disabled benefitted and in what

ways? Probe if there were no project what may have been the loss

or may have not changed?

Q-59: Did your village achieve the ODF status, is it certified /notified by

Government? If yes, did you receive rewards, what are those &

how were those identified? Probe if women and children were part

of rewards consultations and if any group is excluded from the

benefits of rewards?

Q-60: In your view which of the project interventions and results may

sustain (ODF status, CRPs, VSC, hand wash practice, construction

[210]

of low cost latrines, sanitary marts, entrepreneurs)? Which

interventions may not sustain and why? What should be done

differently in future to achieve better results?

Thank You for your time and candid response.

End of Programme Evaluation UNICEF’s Sanitation Programme at Scale in

Pakistan (SPSP) – Rural Through Pakistan Approach to Total Sanitation (PATS)

INDEPTH INTERVIEW

SANITATION MARKET Consultant Introduction to the Interviewee: I am , a member of the evaluation team deployed by an independent consulting entity i.e. the AAN Associates, who has been contracted for the End of Programme Evaluation of UNICEF’s Sanitation Programme at Scale in Pakistan. This programme aims to improve sanitation practices, using Community-led Total Sanitation (CLTS) and School-led Total Sanitation (SLTS) approaches under the overarching PATS strategy to ensure that the most vulnerable, marginalized children and women are using safe drinking water, improved sanitation and hygiene practices. The target areas include flood affected, high Polio risk and insecurity affected areas. I am here to get your reflections, experiences and recommendations about the programme, which will help us to assess, analyse and document programme progress, results and areas for further improvement. This will help at large in designing or developing future sanitation programmes in the country and/or extension or scale-up of this initiative. I thank you for taking time for this interview today. I shall be asking you questions about how the programme activities were implemented in your area and what changes this programme brought in the lives of the community in regard to improved access and availability of sanitation services. I anticipate the interview will last about 60 minutes, and appreciate any information you can provide. Please note that this interview is important for the success of this assessment. Your answers are completely confidential and shall remain with the evaluation team. At the start we seek your consent to make use of this information for analysis and reporting and may also cite it in the report while making reference to your name, if needed. Also, we are recording the proceeding to avoid any loss of information and seek your consent to record our discussion. Finally, we would request you to please allow us to make photographs of project interventions and yourself for evidence and communication purposes. Note: This is to underline that the IDI instrument has been designed as such to offer flexibility to the interviewers to adapt questions as to the specific role of the organization, respondents profile, and understanding of programme operations and overall knowledge of sector. Hence one consolidated tool has been developed for multiple stakeholders (as listed above).

[211]

Name of Respondent: _____________________ Designation _____________________ Organisation _____________________ Location: _____________________ Date of Interview: _____________________

Name of the Interviewer: _____________________

Interview Questions: Please listen carefully to the questions and provide your candid feedback.

Q 59. Were there any quality standards for outputs & to what extent the

outputs have been delivered in line with planned quality

standards?

Q 60. What were the key quality aspects/elements of supply facilitation

interventions? Were there any issues encountered related to

specific quality standards?

1. Latrine designs

2. Quality of demo latrines

3. Mason training

[212]

4. Entrepreneurs/Marts

5. Rewards design/Implementation (for model villages)

6. Any Other Aspect

Q 61. Did the programme document clearly define the equity focus (as

per UNICEF guidelines for instance vulnerability criteria)? How

was it implemented and how were the systems/processes evolved

for equity integration into the project deliverables (how did the

interventions cater to the needs of the disabled or

women/children?

[213]

Q 62. To what extent the equity principles were integrated in the

approaches and deliverables i.e. BCC, supplies facilitation, e.g.

targeting of poorest of poor for demo latrines? Or bringing about a

change in access to WASH services and facilities for the lowest

quintile?

Q 63. What was done to ensure that the school based interventions in

the target area provide child friendly water and sanitation

facilities?

Thank You for your time and candid response.

[214]

End of Programme Evaluation UNICEF’s Sanitation Programme at Scale in Pakistan (SPSP) – Rural through Pakistan

Approach to Total Sanitation (PATS) INDEPTH INTERVIEW

APEX THIRD PARTY MONITORS

Introduction to the Interviewee: I am , a member of the evaluation team deployed by an independent consulting entity i.e. the AAN Associates, who has been contracted for the End of Programme Evaluation of UNICEF’s Sanitation Programme at Scale in Pakistan. This programme aims to improve sanitation practices, using Community-led Total Sanitation (CLTS) and School-led Total Sanitation (SLTS) approaches under the overarching PATS strategy to ensure that the most vulnerable, marginalized children and women are using safe drinking water, improved sanitation and hygiene practices. The target areas include flood affected, high Polio risk and insecurity affected areas. I am here to get your reflections, experiences and recommendations about the programme, which will help us to assess, analyse and document programme progress, results and areas for further improvement. This will help at large in designing or developing future sanitation programmes in the country and/or extension or scale-up of this initiative. I thank you for taking time for this interview today. I shall be asking you questions about how the programme activities were implemented in your area and what changes this programme brought in the lives of the community in regard to improved access and availability of sanitation services. I anticipate the interview will last about 60 minutes, and appreciate any information you can provide. Please note that this interview is important for the success of this assessment. Your answers are completely confidential and shall remain with the evaluation team. At the start we seek your consent to make use of this information for analysis and reporting and may also cite it in the report while making reference to your name, if needed. Also, we are recording the proceeding to avoid any loss of information and seek your consent to record our discussion. Finally, we would request you to please allow us to make photographs of project interventions and yourself for evidence and communication purposes. Note: This is to underline that the IDI instrument has been designed as such to offer flexibility to the interviewers to adapt questions as to the specific role of the organization, respondents profile, and understanding of programme operations and overall knowledge of sector. Hence one consolidated tool has been developed for multiple stakeholders (as listed above).

Name of Respondent: _____________________ Designation _____________________ Organisation _____________________ Location: _____________________ Date of Interview: _____________________

Name of the Interviewer: _____________________

[215]

Interview Questions:

Please listen carefully to the questions and provide your candid feedback.

1. Were you involved in initial orientation/training session introducing the

design and results framework of SPSP Rural? How and who was

involved from APEX? Was the introduction to the programme sufficient?

2. How were the TPM criteria developed and agreed? Was APEX involved

in developing the monitoring tools, procedures and checklists? Were

there any standard monitoring targets set for various programme

components including soft and hard components?

3. What were the most common and frequent observations and how/to

whom were these communicated (UNICEF HQ, Provincial Offices)?

How frequent was the reporting? How were these issues taken forward

and resolved? How long did it take from identifying the issue to it being

resolved?

4. What challenges were encountered during the TPM activities? Were

they resolved and how? What lessons/recommendations need to be

factored into future interventions related to TPM?

5. How was the overall ongoing coordination with UNICEF Isb, Provincial

Offices as well as the range of Implementing Partners?

6. What internal checks and processes were put in place in APEX to

ensure the quality of TPM activities related to SPSP? Were Monitors

given standardized trainings and understanding of how to apply TPM

criteria?

7. How was RusFAD learning incorporated in SPSP as part of the

Knowledge management process? What learnings have been

generated during the SPSP implementation?

8. How do you view the actual delivery by IPs (in terms of no’s and targets

i.e. Villages ODF etc)? Was quality a consideration and to what extent

these elements are likely to sustain?

[216]

9. Did APEX have a monitoring role in the Action Research undertaken by

DSI in SPSP?

10. What is your perception of training provided by RSPN and their

mentoring role? Were there any issues or challenges observed?

11. Was it in the ambit of APEX to look at partnership management

and risk management (particularly financial)? Was cost efficiency and

value for money aspects part of TPM?

12. How was Gender, Disability and Equity factored into the project

and picked up in monitoring activities? What are the observations in

terms of

13. How and to what extent has TPM affected/influenced decision

making in SPSP? What key factors improved as a result?

[217]

End of Programme Evaluation UNICEF’s Sanitation Programme at Scale in

Pakistan (SPSP) – Rural Through Pakistan Approach to Total Sanitation (PATS)

INDEPTH INTERVIEW

THIRD PARTY MONITOR

Introduction to the Interviewee: I am , a member of the evaluation team deployed by an independent consulting entity i.e. the AAN Associates, who has been contracted for the End of Programme Evaluation of UNICEF’s Sanitation Programme at Scale in Pakistan. This programme aims to improve sanitation practices, using Community-led Total Sanitation (CLTS) and School-led Total Sanitation (SLTS) approaches under the overarching PATS strategy to ensure that the most vulnerable, marginalized children and women are using safe drinking water, improved sanitation and hygiene practices. The target areas include flood affected, high Polio risk and insecurity affected areas. I am here to get your reflections, experiences and recommendations about the programme, which will help us to assess, analyse and document programme progress, results and areas for further improvement. This will help at large in designing or developing future sanitation programmes in the country and/or extension or scale-up of this initiative. I thank you for taking time for this interview today. I shall be asking you questions about how the programme activities were implemented in your area and what changes this programme brought in the lives of the community in regard to improved access and availability of sanitation services. I anticipate the interview will last about 60 minutes, and appreciate any information you can provide. Please note that this interview is important for the success of this assessment. Your answers are completely confidential and shall remain with the evaluation team. At the start we seek your consent to make use of this information for analysis and reporting and may also cite it in the report while making reference to your name, if needed. Also, we are recording the proceeding to avoid any loss of information and seek your consent to record our discussion. Finally, we would request you to please allow us to make photographs of project interventions and yourself for evidence and communication purposes. Note: This is to underline that the IDI instrument has been designed as such to offer flexibility to the interviewers to adapt questions as to the specific role of the organization, respondents profile, and understanding of programme operations and overall knowledge of sector. Hence one consolidated tool has been developed for multiple stakeholders (as listed above).

Name of Respondent: _____________________ Designation _____________________ Organisation _____________________ Location: _____________________ Date of Interview: _____________________

Name of the Interviewer: _____________________

[218]

Interview Questions: Please listen carefully to the questions and provide your candid feedback.

Q 64. What experiences/lessons were identified as part of the monitoring

process around programme implementation (list what worked,

what did not and why? Substantiate with measure of success,

scale, and lessons identified for future)? The analysis to be

desegregated for following (components/subcomponents)

19. DEMAND CREATION

What Worked What Didn’t Work

Lessons Identified Success Scale

Rank

Highly

Successful

Moderate

Poor

Fail

20. SUPPLY FACILITATION

What Worked What Didn’t Work

Lessons Identified Success Scale

Rank

Highly

Successful

Moderate

Poor

Fail

21. HEALTH AND HYGIENE PROMOTION

What Worked What Didn’t Work

Lessons Identified Success Scale

Rank

Highly

Successful

Moderate

Poor

Fail

22. LINKAGES WITH DUTY BEARERS FOR PATS

[219]

What Worked What Didn’t Work

Lessons Identified Success Scale

Rank

Highly

Successful

Moderate

Poor

Fail

23. VILLAGES SUPPORTED IN TOTAL SANITATION STATUS

What Worked What Didn’t Work

Lessons Identified Success Scale

Rank

Highly

Successful

Moderate

Poor

Fail

24. PATS KNOWLEDGE MANAGEMENT FRAMEWORK

What Worked What Didn’t Work

Lessons Identified Success Scale

Rank

Highly

Successful

Moderate

Poor

Fail

25. PARTNERSHIP MANAGEMENT

What Worked What Didn’t Work

Lessons Identified Success Scale

Rank

Highly

Successful

Moderate

[220]

Poor

Fail

26. INTEGRATION OF CROSS CUTTING PRIORITIES

(Gender & Equity, Unforeseen Events, Rights Based Approach) What Worked What Didn’t

Work Lessons Identified Success

Scale Rank

Highly

Successful

Moderate

Poor

Fail

27. SCHOOL LED TOTAL SANITATION

What Worked What Didn’t Work

Lessons Identified Success Scale

Rank

Highly

Successful

Moderate

Poor

Fail

Q 65. What tools/guidelines were designed/implemented as part of the

3rd Party Monitoring process? How successful was the actual

application of these tools and methods in contributing to ensuring

quality of outputs and processes? What needs to be done

differently in terms of monitoring to improve the implementation of

the programme?

[221]

Q 66. In terms of knowledge management, did the 3rd party monitoring

process inform and contribute to improved project implementation

and decisions? Please elaborate with examples.

Q 67. How did UNICEF and the Implementing Partners respond to the 3rd

party monitoring process? Were any issues/challenges faced in

the monitoring process?

[222]

Thank You for your time and candid response.

End of Programme Evaluation UNICEF’s Sanitation Programme at Scale in

Pakistan (SPSP) – Rural Through Pakistan Approach to Total Sanitation (PATS)

[223]

INDEPTH INTERVIEW BCC CONSULTANT

Introduction to the Interviewee: I am , a member of the evaluation team deployed by an independent consulting entity i.e. the AAN Associates, who has been contracted for the End of Programme Evaluation of UNICEF’s Sanitation Programme at Scale in Pakistan. This programme aims to improve sanitation practices, using Community-led Total Sanitation (CLTS) and School-led Total Sanitation (SLTS) approaches under the overarching PATS strategy to ensure that the most vulnerable, marginalized children and women are using safe drinking water, improved sanitation and hygiene practices. The target areas include flood affected, high Polio risk and insecurity affected areas. I am here to get your reflections, experiences and recommendations about the programme, which will help us to assess, analyse and document programme progress, results and areas for further improvement. This will help at large in designing or developing future sanitation programmes in the country and/or extension or scale-up of this initiative. I thank you for taking time for this interview today. I shall be asking you questions about how the programme activities were implemented in your area and what changes this programme brought in the lives of the community in regard to improved access and availability of sanitation services. I anticipate the interview will last about 60 minutes, and appreciate any information you can provide. Please note that this interview is important for the success of this assessment. Your answers are completely confidential and shall remain with the evaluation team. At the start we seek your consent to make use of this information for analysis and reporting and may also cite it in the report while making reference to your name, if needed. Also, we are recording the proceeding to avoid any loss of information and seek your consent to record our discussion. Finally, we would request you to please allow us to make photographs of project interventions and yourself for evidence and communication purposes. Note: This is to underline that the IDI instrument has been designed as such to offer flexibility to the interviewers to adapt questions as to the specific role of the organization, respondents profile, and understanding of programme operations and overall knowledge of sector. Hence one consolidated tool has been developed for multiple stakeholders (as listed above).

Name of Respondent: _____________________ Designation _____________________ Organisation _____________________ Location: _____________________ Date of Interview: _____________________

Name of the Interviewer: _____________________

Interview Questions: Please listen carefully to the questions and provide your candid feedback.

Q 68. Were any quality standards prescribed and set for Behavior

Change Communication (in line with UNICEF core strategy –

advocacy and communication for change) of communities for

improved health & safer hygiene practices/demand generation

[224]

interventions before implementation? To what extent the BCC

outputs were delivered in line with the prescribed quality

standards?

Q 69. What BCC strategies/instruments/messages/mediums were

evolved & to what extent were those effective? Which methods

received maximum attention, which groups of people

changed/responded, and which practices changed more? Which

BCC interventions worked well & why and why others failed?

[225]

Q 70. Did the programme design clearly define the equity focus (as per

UNICEF guidelines for instance vulnerability criteria)? How was it

incorporated into the BCC interventions and implemented (how did

the interventions cater to the needs of the disabled or

women/children?

Thank You for your time and candid response.

[226]

End of Programme Evaluation UNICEF’s Sanitation Programme at Scale in

Pakistan (SPSP) – Rural Through Pakistan Approach to Total Sanitation (PATS)

INDEPTH INTERVIEW

DONORS/INGOs

Introduction to the Interviewee: I am , a member of the evaluation team deployed by an independent consulting entity i.e. the AAN Associates, who has been contracted for the End of Programme Evaluation of UNICEF’s Sanitation Programme at Scale in Pakistan. This programme aims to improve sanitation practices, using Community-led Total Sanitation (CLTS) and School-led Total Sanitation (SLTS) approaches under the overarching PATS strategy to ensure that the most vulnerable, marginalized children and women are using safe drinking water, improved sanitation and hygiene practices. The target areas include flood affected, high Polio risk and insecurity affected areas. I am here to get your reflections, experiences and recommendations about the programme, which will help us to assess, analyse and document programme progress, results and areas for further improvement. This will help at large in designing or developing future sanitation programmes in the country and/or extension or scale-up of this initiative. I thank you for taking time for this interview today. I shall be asking you questions about how the programme activities were implemented in your area and what changes this programme brought in the lives of the community in regard to improved access and availability of sanitation services. I anticipate the interview will last about 60 minutes, and appreciate any information you can provide. Please note that this interview is important for the success of this assessment. Your answers are completely confidential and shall remain with the evaluation team. At the start we seek your consent to make use of this information for analysis and reporting and may also cite it in the report while making reference to your name, if needed. Also, we are recording the proceeding to avoid any loss of information and seek your consent to record our discussion. Finally, we would request you to please allow us to make photographs of project interventions and yourself for evidence and communication purposes. Note: This is to underline that the IDI instrument has been designed as such to offer flexibility to the interviewers to adapt questions as to the specific role of the organization, respondents profile, and understanding of programme operations and overall knowledge of sector. Hence one consolidated tool has been developed for multiple stakeholders (as listed above).

Name of Respondent: _____________________ Designation _____________________ Organisation _____________________ Location: _____________________ Date of Interview: _____________________

Name of the Interviewer: _____________________

[227]

Interview Questions: Please listen carefully to the questions and provide your candid feedback.

Q 71. What is your organization doing in the context of WASH Sector? Is

the focus rural or urban? Is PATS approach being utilized? Are

there any differing factors/context in your approach as compared

to the PATS approach?

Q 72. Are there any consultative processes/mechanisms (at varied levels

across different stages) that are/were used to take stakeholders on

board for programme design and implementation? Were you

consulted/involved in the UNICEF SPSP-Rural Programme in any

way? If so, how?

Q 73. Are you aware of any other mechanisms (coordination

meetings/forums etc) that relevant government departments (at

national and provincial levels) are using to align the efforts of

government and development partners in WASH and PATS?

[228]

Q 74. To what extent are they successful in their efforts? In which ways

it could be improved further and what could the development

partners contribute towards this? What in your view are key

learning around consultative planning and what needs to be done

in future to make it more consultative/inclusive, both at the design

as well as during the implementation stages (at varied levels)?

Q 75. Does your approach involve implementation through Government,

Implementing partners/Private Sector? What has been more

successful and which approach do you see being mose

sustainable in addressing WASH in particular Sanitation needs?

[229]

Q 76. What has been the impact of the 18th Amendment in terms of

WASH priorities at the provincial and district levels?

Q 77. How do your programme costs incurred for different WASH

components to achieve results (in terms of mobilization,

beneficiaries awareness, construction of latrines etc on per

head/per facility cost or per beneficiary cost basis etc.) compare

with a similar nature/scale of programme as UNICEF’s SPSP-

Rural? And/or govt and/or in regional countries (CATS, CLTS,

PATS)?

[230]

Q 78. What approaches can be developed going into the future to

address WASH and Sanitation in particular that are aligned to

PATS? What needs to be addressed at Policy level to improve the

approach?

Thank You for your time and candid response.

[231]

End of Programme Evaluation UNICEF’s Sanitation Programme at Scale in

Pakistan (SPSP) – Rural Through Pakistan Approach to Total Sanitation (PATS)

INDEPTH INTERVIEW

SANITATION MARKET CONSULTANT SANITATION ENGINEER

Introduction to the Interviewee: I am , a member of the evaluation team deployed by an independent consulting entity i.e. the AAN Associates, who has been contracted for the End of Programme Evaluation of UNICEF’s Sanitation Programme at Scale in Pakistan. This programme aims to improve sanitation practices, using Community-led Total Sanitation (CLTS) and School-led Total Sanitation (SLTS) approaches under the overarching PATS strategy to ensure that the most vulnerable, marginalized children and women are using safe drinking water, improved sanitation and hygiene practices. The target areas include flood affected, high Polio risk and insecurity affected areas. I am here to get your reflections, experiences and recommendations about the programme, which will help us to assess, analyse and document programme progress, results and areas for further improvement. This will help at large in designing or developing future sanitation programmes in the country and/or extension or scale-up of this initiative. I thank you for taking time for this interview today. I shall be asking you questions about how the programme activities were implemented in your area and what changes this programme brought in the lives of the community in regard to improved access and availability of sanitation services. I anticipate the interview will last about 60 minutes, and appreciate any information you can provide. Please note that this interview is important for the success of this assessment. Your answers are completely confidential and shall remain with the evaluation team. At the start we seek your consent to make use of this information for analysis and reporting and may also cite it in the report while making reference to your name, if needed. Also, we are recording the proceeding to avoid any loss of information and seek your consent to record our discussion. Finally, we would request you to please allow us to make photographs of project interventions and yourself for evidence and communication purposes. Note: This is to underline that the IDI instrument has been designed as such to offer flexibility to the interviewers to adapt questions as to the specific role of the organization, respondents profile, and understanding of programme operations and overall knowledge of sector. Hence one consolidated tool has been developed for multiple stakeholders (as listed above).

Name of Respondent: _____________________ Designation _____________________ Organisation _____________________ Location: _____________________ Date of Interview: _____________________

[232]

Name of the Interviewer: _____________________

Interview Questions: Please listen carefully to the questions and provide your candid feedback.

Q 79. Were there any quality standards for outputs & to what extent the

outputs have been delivered in line with planned quality

standards?

Q 80. What were the key quality aspects/elements of supply facilitation

interventions? Were there any issues encountered related to

specific quality standards?

7. Latrine designs

8. Quality of demo latrines

9. Mason training

[233]

10. Entrepreneurs/Marts

11. Rewards design/Implementation (for model villages)

12. Any Other Aspect

Q 81. Did the programme document clearly define the equity focus (as

per UNICEF guidelines for instance vulnerability criteria)? How

was it implemented and how were the systems/processes evolved

for equity integration into the project deliverables (how did the

interventions cater to the needs of the disabled or

women/children?

[234]

Q 82. To what extent the equity principles were integrated in the

approaches and deliverables i.e. BCC, supplies facilitation, e.g.

targeting of poorest of poor for demo latrines? Or bringing about a

change in access to WASH services and facilities for the lowest

quintile?

Q 83. What was done to ensure that the school based interventions in

the target area provide child friendly water and sanitation

facilities?

Thank You for your time and candid response.

[235]

Questions for RSPN 1. To what extent was the PCA clear about the role and expectations

of RSPN as an umbrella training and mentoring organization?

Were there any changes in the expected role during the

implementation of the programme?

2. How was the range of training designed? Who was involved in the

process (UNICEF, RSPN etc?) Was the training derived from the

Theory of Change? If so, which model (PATS, CATS etc) was

used as the basis?

3. What was the training model (TOT, or Number of Trainers)? Were

training activities carried out in standard format across Provinces

and IPs?

4. What was the requirement for trainings in terms of trainees

capacities, as well as numbers of participants per training?

5. Was DSI’s report on Low-Cost Sanitation option used in field?

6. Was the training conducted by diverse teams or same set of

trainers provided the whole range of training (mason, CLTS, SLTS,

SO, CRPS, mart/entrepreneurs)? If no, please elaborate?

7. Were the agreed number of training and targets delivered

according to the PCA and Results Framework? What constrains

were faced during the training activities? Did these constrains

affect the quality of the training? Were they discussed with

UNICEF and how were they addressed?

8. How was it ensured that the trainings were effective in their

purpose? Were there any follow up/mentoring activities planned?

What was the result of these in terms of improvement?

9. What is the sustainability of the capacity building that has resulted

from the range of training, in terms of IP Staff and community?

10. What orientation/trainings/capacity building activities were

undertaken with government? At what levels? What was the key

component of the training? What was their effectiveness and how

was it measured?

[236]

23. Annex 12 KNOWLEDGE, ATTITUDES AND PRACTICE (KAP) HOUSEHOLD SURVEY

QUESTIONNAIRE UNICEF’S SANITATION PROGRAMME AT SCALE IN PAKISTAN

EVALUATION FEB 2014 INFORMED CONSENT

I am ____________________, a representative of an independent consulting entity i.e. the AAN Associates, who has been contracted for the Evaluation of UNICEF’s Sanitation Programme at Scale (SPSP) in Pakistan using the Pakistan Approach to Total Sanitation (PATS). This programme aimed to improve sanitation practices, using Community-led Total Sanitation (CLTS) and School-led Total Sanitation (SLTS) approaches under the overarching PATS strategy.

We are conducting a household survey and would appreciate your participation. I would like

to ask about health, hygiene and sanitation related aspects of the community and your

household. This information will help Unicef and partners to assess achievements under this

project in health, hygiene and sanitation. The survey takes around 30-45 minutes to

complete.

Whatever information you provide, will be kept strictly confidential and will be dealt with

anonymity.

Participation in this survey is voluntary and you can choose not to answer any individual

questions. However, we hope that you will participate in this survey since your views are

important. At this time, do you want to ask me anything about the survey?

RESPONDENT AGREES TO BE INTERVIEWED

RESPONDENT DOES NOT AGREE TO BE INTERVIEWED END

Edited/Verified by: Approved By:

District Supervisor Provincial Coordinator

Signature: Signature:

Date: Date:

[237]

INSTRUCTIONS

RESPONDENT: IS TO BE SLECTED WITH THE FOLLOWING CRITERIA: o SHOULD BE WITH A FAMILY; AND o SHOULD BE OF 18-45 YEARS IN AGE FOR BOTH

MALE AND FEMALE. o IF YOU DON’T FIND A RESPONDENT MEETING

THIS CRITERIA MOVE TO THE NEXT HOUSE

AND SO ON UNTIL YOU COME ACROSS A

HOUSE WITH THE REQUIRED RESPONDENT

CRITERIA

HOUSEHOLD: IS DEFINED AS A GROUP OF FAMILY MEMBERS

CONTRIBUTING TO A SINGLE KITCHEN.

MULTIPLE CHOICE QUESTIONS: ARE NEEDED TO BE PROBED FURTHER WITH ALL

CHOICES MENTIONED. THE QUESTIONS ARE ALSO

MARKED WITH A CAPTION “RECORD ALL

MENTIONED”.

RESPONSE: FOR EACH QUESTION CIRCLE THE

APPROPRIATE RESPONSE/S.

THIS SECTION NEEDS TO BE COMPLETED PRIOR TO STARTUP OF THE FIELD WORK

RECORD NUMBER: (001-TEAM XX) INTERVIEW DATE: ________/________/________ (DATE / MONTH/ YEAR) INTERVIEWER’S NAME: _________________________ SUPERVISOR’S NAME: _________________________ PROVINCE: _________________________ DISTRICT NAME: _________________________ TEHSIL NAME: _________________________ UNION COUNCIL NAME: _________________________ VILLAGE NAME: _________________________

[238]

ALL QUESTIONS ARE TO BE ADDRESSED TO THE RESPONDENT

A) RESPONDENT DETAILS

RESPONDENT NAME:

SEX OF RESPONDENT: MALE 1 FEMALE

2 AGE OF RESPONDENT IN YEARS: MARITAL STATUS MARRIED 1 RESPONDENT’S LEVEL OF EDUCATION: DIVORCED 2 NONE 1 WIDOW 3 PRIMARY 2 SECONDARY 3 HIGHER OTHERS:

B) HOUSEHOLD HEAD’S DETAILS

HOUSEHOLD HEAD’S NAME:

SEX OF HOUSEHOLD HEAD: MALE 1 FEMALE 2 RELATIONSHIP OF RESPONDENT TO THE HEAD OF HOUSEHOLD:

C) FAMILY CHARACTERISTICS

FAMILY TYPE: NUCLEAR FAMILY 1 JOINT FAMILY 2 FAMILY SIZE: ______MEMBERS (I.E. FAMILY MEMBERS CONTRIBUTING/LINKED TO A SINGLE

KITCHEN NUMBER OF CHILDREN: _____ (BOYS _____ & GIRLS _____ )

ANY FAMILY MEMBERS WITH PHYSICAL DISABILITIES: YES 1 NO 2 Disable: Men _____ Women ____ Girl/s (under 18) ____ Boy/s (under 18) ____ Note: Nuclear Family = Family unit consisting of a mother and father and their children Joint Family = Family that consists of parents, children, and close relatives/multiple generations

D) House Structure

HOUSE STRUCTURE: (TICK ONE FOR EACH CATEGORY A-C) A) FLOOR TYPE KATCHA (RUDIMENTARY) 1

PAKKA (FINISHED) 2 B) WALLS KATCHA (RUDIMENTARY) 3 PAKKA (FINISHED) 4 C)ROOF KATCHA

(RUDIMENTARY) 5 PAKKA (FINISHED) 6

[239]

E) ASSETS & DURABLE GOODS

DO YOU OWN ANY OF THE FOLLOWING ASSETS? (TICK ALL MENTIONED) HOUSE 1

LAND 2 LIVESTOCK 3

DO YOU OWN ANY OF THE FOLLOWING DURABLE GOODS? (TICK ALL MENTIONED)

BICYCLE 1 MOTORCYCLE/SCOOTER 2

RADIO 3 TELEVISION 4 FRIDGE 5 OTHER ____________________

NOTE: FOLLOW SKIP PATTERN WHERE APPLICABLE

NO QUESTIONS AND FILTERS

CODING CATEGORIES SKIP

1. 2 What work do you do? Record Only One Option

Does Not Work…………..………..………………..1 Household Chores………....………….………......2 Self Employed ….……….................................3 Salaried Worker……………………..…...………...4 Retired ………………………………………………5 Other: _______________________________ 96 (Specify)

….Go to 4 ….Go to 4 ….Go to 4

2. 3

What is the nature of work you do? Record up to a Maximum of 3 Responses (Rank in priority 1,2,3) With 1 being the most important Migrant worker is defined as those who work overseas Skilled worker

Daily Labourer………………………….………..…..1 Harvesting/Farming………...…..……..….............2 Shopkeeper…………………………………..…….3 Street Vendor……………………. …….................4 Domestic Help/Household Worker………....…….5 Raising Live Stock…………………………….……6 Carpet Weavers……………………………….……7 Skilled Worker…………………………..……….….8 Migrant Worker……………………..………….……9

[240]

NO QUESTIONS AND FILTERS

CODING CATEGORIES SKIP

is defined as those including drivers, carpenters, cobblers, etc

Government Job…………….…………………..…10 Private Job……………………….……………....…11 Other: ________________________________96 (Specify)

3. 4 What is your average monthly income?

(Average will be used for those who work either on daily/weekly paid basis – an estimates average should be stated. In case of those earning on monthly basis then that should be stated).

Amount In PKR:

None: ...............................................................1 Did Not Respond.............................................98 DK...................................................................99

RESUME IN CASE OF SKIP FROM Q 1

4. Does your spouse work?

Yes……………………………………………………1 No……………………………………………………..2

….Go to 7

5. 3

What is the nature of work that your spouse does? Record up to a Maximum of 3 Responses (Rank in priority 1,2,3) With 1 being the most important Migrant worker is defined as those who work

Daily Labourer………………….…………………..1 Harvesting/Farming………...……....………........2 Shopkeeper……………………………..….……..3 Street Vendor……………………. …………...,....4 Domestic Help/Household Worker………..…….5 Raising Livestock…………………………………6 Carpet Weavers……………………..……………7 Skilled Worker………………………………….….8 Migrant

[241]

NO QUESTIONS AND FILTERS

CODING CATEGORIES SKIP

overseas Skilled worker is defined as those including drivers, carpenters, cobblers, etc

Worker………………..……….…….……9 Government Job………………….…………..…10 Private Job……………………….…………....…11 Other: _______________________________96 (Specify)

6. What is your Spouse’s average monthly income?

None..................................................................1 Amount In PKR: Did Not Respond.............................................98 DK...................................................................99

RESUME IN CASE OF SKIP FROM Q 4

7. What is your household’s average monthly income?

Note: Confirm using the ‘Contributors to one Kitchen expense’ definition whether anyone other than the respondent and his/her spouse are contributing to the household income

Amount In PKR: Did Not Respond.............................................98 DK........................................................99

Access To Water, Use Of Household Water Treatment And Safe Storage

8. Do you have access to drinking water? Note: PROBE IF NO. People would normally have some access even if poor or from outside.

Yes ..................................................................1 No ..................................................................2

….Go to 16

[242]

NO QUESTIONS AND FILTERS

CODING CATEGORIES SKIP

9. Where is the drinking water point located?

Inside the House...................................................1 Outside the house.................................................2 DK .................................................................99

10. 5 What is the main source of drinking water for your household? Record up to a Maximum of 3 Responses (Rank in priority 1,2,3) With 1 being the most important Surface water include: river, dam, lake, pond, stream, canal, irrigation channels HandPump: Note in Others 96

Piped Water Into Dwelling (House)..................….1 Piped Water To Yard/Plot………............………...2 Public Tap/Standpipe……………………..............3 Tube Well/Borehole……….............………….…...4 Protected Dug Well….……….........................…..5 Unprotected Dug Well………..........................….6 Protected Spring…………….............………….…7 Unprotected Spring ………............……………....8 Rainwater Collection………..............………..…..9 Bottled Water…………………..............………...10 Cart With Small Tank/Drum….............………....11 Tanker-Truck ……………….............…………....12 Surface Water………………………............……13 DK………………….....………...................99 Other______________________96 (Specify)

11. Do you treat your drinking water?

Yes……………………………………..1 No……………………………………2 DK…………………………………..99

….Go to 13 ….Go to 13

12. How do you treat your drinking water? Record up to a Maximum of 3 Responses

Boil……………………………………1 Add Bleach/Chlorine…………………….2 Strain It Through A Cloth……………….3 Use A Water Filter………………….…...4 Solar Disinfection……………………...5 Let It Stand And Settle………………...6

[243]

NO QUESTIONS AND FILTERS

CODING CATEGORIES SKIP

(Rank in priority 1,2,3) With 1 being the most important Water filter: Do they pass water through sand/gravel or charcoal/carbon or ceramic to clean it.

Don’t Know………………………………99 Other: __________________________96 (Specify)

RESUME IN CASE OF SKIP FROM Q 11

13. How do you store drinking water? Record up to a Maximum of 3 Responses (Rank in priority 1,2,3) With 1 being the most important Containers include: bucket, jerry can, jerkin, bottle, drum

In Containers……………………………..1 Roof Tank ……………….………………2 No Water Stored………………………….3 Don’t Know……………………………99 Other: __________________________96 (Specify)

….Go to 16 ….Go to 16 ….Go to 16 ….Go to 16

14. If in container, ask permission to see the containers and observe are they:

Narrow Mouthed……………………………...1 Wide Mouthed…………………………...2 Of Both Types…………………………...3 Not Observed…………………….……...4 Not Allowed………………………….……...5

15. If in containers, observe are they covered?

All Are……………………………….………1 Some Are…………………………………...2 None Are…………………….……………...3

RESUME IN CASE OF SKIP FROM Q 8 / Q13

16. Do you have access to water for other purposes? (‘Other

Yes ...........................................................1 No ............................................................2

….Go to 22

[244]

NO QUESTIONS AND FILTERS

CODING CATEGORIES SKIP

purposes’ such as cooking, hand washing and bathing)

17. Where is the location of water point for purposes other than drinking?

Inside the House........................................1 Outside the house.....................................2 DK ..........................................................99

18. 6 What is the main source of water for other purposes of your household? Record up to a Maximum of 3 Responses (Rank in priority 1,2,3) With 1 being the most important (Such as cooking, hand washing and bathing) HandPump: Note in Others 96

Piped Water Into Dwelling (House)...............….1 Piped Water To Yard/Plot…….....……...2 Public Tap/Standpipe………..…….......3 Tube Well/Borehole………........…...4 Protected Dug Well….……................…....5 Unprotected Dug Well………................….6 Protected Spring……………….......……7 Unprotected Spring ………….........….8 Rainwater Collection………….............9 Bottled Water……...……………….... ….10 Cart With Small Tank/Drum…….....…….…...11 Tanker-Truck ………………….…….....12 Surface Water………………….....…..13 DK……………...................……………..99 Other___________________________96 (Specify)

19. Who usually goes to fetch water for your household? RECORD ALL MENTIONED (RANK in priority)

Adult Woman………………………...…………….1 Adult Man………………..………………2 Female Child (Under 18 Years)……..……3 Male Child (Under 18 Years)……........………….4 DK…………………......................……99

20. How long does it take to go there, get water, and come back?

Less Than 15 Minutes…………………1 15 Minutes To 30 Minutes………………2 30 Minutes To 1 Hour……………………3 More Than 1

[245]

NO QUESTIONS AND FILTERS

CODING CATEGORIES SKIP

Probe: Try to probe the amount of time spent on socialization that should be excluded in the time value

Hour……………….………………..4 DK…………………..…………………..99

21. How many trips a day are made on average to collect water?

No. of Trips __________________________

RESUME IN CASE OF SKIP FROM Q 16

22. Are there any Water User Committees/CBOs in your villages that you use to claim/access WASH services?

Yes……………………………………...1 No………………………………………….2 DK………………………………….…….99 If ‘Yes’ What Services do you access, Water .......................................................1 Health & Hygiene Promotion Services.................2 Sanitation Services..............................................3 If ‘Yes’ Name of Organisation________________________

23. Are you paying for water for uses other than drinking?

Yes ..................................................................1 No.............................................................2 If ‘Yes’ To Whom Do You Pay _____________________ If ‘Yes’ Fixed Operation & Maintenance Cost.................1 Water Usage Based Cost...................................2 If ‘Yes’ How Much: PKR per month

Health & Hygiene Awareness Raising And Practice Of Hand Washing At Critical Moments

24. Are you aware of Yes …………………………………………..1

[246]

NO QUESTIONS AND FILTERS

CODING CATEGORIES SKIP

the need to achieve Open Defecation Free status in your village?

No ……………………………………..……….2 DK …………………………………………...99

25. 24a

Have you received any health and hygiene related message in the last 6 months?

Yes …………………………………………..1 No …………………………………..……….2 DK ……………………………………...99

….Go to 30 ….Go to 30

26. 19 Where did you get the message related to health, hygiene and sanitation? Record up to a Maximum of 3 Responses (Rank in priority 1,2,3) With 1 being the most important

LHWs………………………………………..1 Hospital/BHU……………….……………2 Health Staff……………………………....……3 Village Pharmacy/Shopkeeper……………………4 CBOs…………………………………..…….5 Religious Leaders……………………………..…...6 Friend/Neighbour………………..……..…7 Awareness Raising Session………………..….....8 Radio……………………………………..…9 TV……………………………………..…...10 Print Media/Material.………………..…...11 Folk Media/Theater…………………….…12 Mothers/Youth Groups…………..…..…..13 School Children………………………..……..….14 Don’t Know.…………….………………99 Others ________________________ 96 Specify

27. Which source of information do you prefer more? RECORD THREE MOST PREFERRED SOURCES (Rank in priority 1,2,3) With 1 being the most important

LHWs………………………………………...1 Hospital/BHU…………………….……….…2 Health Staff……………………………………….…3 Village Pharmacy/Shopkeeper………………....…4 CBOs……………………………….……..5 Religious Leaders……………….………….….…...6 Friend/Neighbor……………….…….…7 Awareness Raising Session……………...8 Radio………………………………………9 TV……………………………………....10

[247]

NO QUESTIONS AND FILTERS

CODING CATEGORIES SKIP

Print Media/Material.………….………...11 Folk Media/Theater……………………..…12 Mothers/Youth Groups………….…….....13 School Children……………..……….14 Don’t Know.…………………..….………99 Others ________________________ 96 Specify

28. 21 What health & hygiene related topics can you remember? RECORD ALL MENTIONED (Rank in priority 1,2,3) With 1 being the most important

Personal Hygiene……..…………..…………….…1 Clean And Safe Water....…………..……………..2 Water Born Diseases………….…………..3 Environmental & Domestic Hygiene…......4 Safe Disposal Of Human Excreta.... …...5 Solid Waste (Garbage) Disposal………..……….6 Do Not Know………………….….…....99 Other: _________________________ 96 (Specify)

29. 22 Are you practicing any of the messages you received? If “Yes”, please specify any three messages.

Yes ………………………………….………..1 No ……………………………………….2 DK …………………………………………...99 1.

2.

3.

RESUME IN CASE OF SKIP FROM Q 25

30. 26 Does your household have a separate/designated place for hand washing?

Yes………………………………….……….1 No……………………………………………2 DK………………………………………...99

31. Is soap available in your house? RECORD ALL MENTIONED Note: In Case of NOT AVAILABLE then this should be the only

Anywhere In The Dwelling....……………………….1 Near Hand Washing Place.…………………………2 Near Latrine……………………..………3 Inside Latrine..………… …………..……...4 Not Available…………..………………..5 Other: __________________________96 (Specify)

[248]

NO QUESTIONS AND FILTERS

CODING CATEGORIES SKIP

entry.

32. 27 (With Women Respondents Only) Ask to see the place used most often for hand washing and latrine to observe which of the above responses with regards to availability of soap are true: (Observe above responses only with Women Respondents)

Anywhere In The Dwelling....……………………….1 Near Hand Washing Place.……… ………2 Near Latrine………………………………3 Inside Latrine..……… …………………...4 Not Available………..…………………..5

33. 28 When do you usually wash your hand with soap? RECORD ALL MENTIONED Note: Number of. Critical Moments

Before Eating………………………….........1 After Eating………….....…………………2 Before Praying……...…..……………….3 Before Breastfeeding A Child…………… .4 Before Feeding A Child ……………………5 Before Cooking Or Preparing Food………6 After Defecation………………..…………...7 After Urination……………………………….8 After Cleaning A Child That Has Defecated/Changing A Child’s Nappy……9 When Your Hands Are Dirty………………………10 After Cleaning The Toilet Or Potty……………….11 DK……………………………..……….99 Other _______________________________ 96 Specify

34. Can household members with physical disabilities use the hand washing station?

Yes …………………………………………….1 No……………………………………....2

35. 29 Can you name the diseases caused by using

Diarrhoea………………………………….1 Cholera…………………………………...2 Hepatitis………………………………..…3

[249]

NO QUESTIONS AND FILTERS

CODING CATEGORIES SKIP

unsafe water? RECORD ALL MENTIONED (Rank in priority 1,2,3) With 1 being the most important

Skin Disease…………………….…….4 Don’t Know…………………………99 Other: _________________________96 (Specify)

36. 30 In the past two weeks has any child (less than 5 years) in your household suffered from diarrhea?

Yes………………………………………………1 No………………………………………………..2 DK………………………………………………..99 Was the child, Male ………………………………….…………1 Female …………………………………………2

37. 61 What did you do the last time your youngest child passed stools? MARK SINGLE RESPONSE ONLY

Put/Rinsed Into Latrine………………..…………..…1 Child Used Latrine……………………………………2 Thrown Into Garbage…………………….…………..3 Buried In Yard ………………………………………..4 Not Disposed Of/Left On The Ground……………..5 DK…………………….………………………………99 Other___________________________ ______ 96 Specify

Access To & Use Of Sanitary Facilities And Disposal Of Human Excreta

38. Do you have a latrine?

Yes …………………………………………………1 No…………………………………………………...2 How Many _____________________

….Go to 40

39. Where is the latrine located? In relation to their

Inside the house.....………………………………1 Outside the house….…………………………….2

[250]

NO QUESTIONS AND FILTERS

CODING CATEGORIES SKIP

specific house/property (not to be mixed with communal latrine)

RESUME IN CASE OF SKIP FROM Q 38

40. 31 Where do you defecate?

Latrine………………………………………...…...1 Open……………………………………………….2 Both……………………………………….……….3

…Go to 42

41. 33 Why do you practice Open Defecation? Record up to a Maximum of 3 Responses (Rank in priority 1,2,3) With 1 being the most important

No Latrine………………………..……….…..…….1 Latrines Are Dirty……………………….………….2 No Privacy……………………………….….………3 Latrine Far From House……………………..……4 Insufficient Numbers of Latrines………..………..5 Lack of Water for Defecation……………………..6 DK………………………………………………….99 Others________________________________96 Specify

42. Where do your household members defecate?

Latrine………………………………………...…...1 Open……………………………………………….2 Both……………………………………….……….3

….Go to 45

43. 35 Which household members practice open defecation (OD) more? RECORD ALL MENTIONED (Rank in priority 1,2,3) With 1 being the

Men………………………………………………...1 Women…………………………………………….2 Boys (5 Years And Above)…………….………..3 Girls (5 Years And Above)……………..……..…4 Boys (Under 5 Years)…………………..……..…5 Girls (Under 5

[251]

NO QUESTIONS AND FILTERS

CODING CATEGORIES SKIP

most important

Years)………………….…………6 DK……………………………………………….….99 Other_________________________________96 Specify

44. 33 Why are your household members practicing Open Defecation? Record up to a Maximum of 3 Responses (Rank in priority 1,2,3) With 1 being the most important

No Latrine…………………………………....…….1 Latrines Are Dirty………………………………….2 No Privacy…………..……………………..………3 Latrine Far From House………..…………...……4 Insufficient Numbers of Latrines……………..…..5 Lack of Water for Defecation…………………....6 DK………………………………………………….99 Others_______________________________96 Specify

RESUME IN CASE OF SKIP FROM Q 42

45. 34 Is there any designated place for open defecation?

Yes………………………………………………...1 No………………………………………………….2 DK…………………………………………………99 If Yes, Inside House ……………………………………1 Outside House ………………………………….2 Other ______________________________96

46. 36 Is it safe to go for open defecation? RECORD ALL MENTIONED (Mark for all

Men YES………………………………………………...1 NO………………………………………………….2 Women

[252]

NO QUESTIONS AND FILTERS

CODING CATEGORIES SKIP

types of family members)

YES………………………………………………...3 NO………………………………………………….4 Children YES………………………………………………...5 NO………………………………………………….6 Household Members with Physical Disability YES…………………………………………………7 NO…………………………………………………..8 DK…………………………………………………99

47. 41 What kind of latrine facility do members of your household usually use? Probe if “flush” or “pour flush” probe: Where does it flush to? Also observe the latrine Hanging Latrine: Where the latrine is built directly over body of water (river, stream) VIP Latrine (A pit latrine with ventilation in the form of a pipe from the pit to the outside of the roof taking the smell/flies outside) Composting

Flush/Pour Flush To: Piped Sewer System ……..……………………1 Septic Tank ………………,.…………………....2 Pit Latrine………………..…………………..…..3 Elsewhere……………..……………………..….4 Unknown Place/Not Sure/DK Where…………5 Ventilated Improved Pit Latrine………………....6 Pit Latrine With Slab……………………………...7 Pit Latrine Without Slab/Open Pit……….……...8 Composting Toilet……………………………..…9 Bucket …………………………………………....10 Hanging Toilet/Hanging Latrine…………….….11 Public/Communal Latrine…………………….…12 No Facilities Or Bush Or Field…………………13 Other ________________________________96

[253]

NO QUESTIONS AND FILTERS

CODING CATEGORIES SKIP

Toilet: A latrine where little or no water is used, and sawdust, peat moss (khus) may be mixed with excreta to help it compost)

(Specify)

48. 42 Which members of your immediate household use the latrine in the house? RECORD ALL MENTIONED

Male Adults……………………………….………....1 Female Adults ......................................................2 Male Children 5 Years Or Above .........................3 Female Children 5 Years Or Above......................4 Household Members with Physical Disability……5

49. 43 When did you construct the latrine in your house?

In the last year ……………………………………..1 In the last three years ……………………………..2 In the last 5 years…………………………………..3 DK………………………………………………….99

50. 44 Did you receive help from any organization for latrine construction in any of the following? RECORD ALL MENTIONED (Rank in priority 1,2,3) With 1 being the most important

Need Identification….……………………………..1 Supply Facilitation (Design, Material)……………2 Loan ………..…..…………………………………..3 Cost Share Basis…………………………………..4 Demo Latrine……………………………………….5 Subsidy……………………………………………..6 Trained Skilled Labour……………………………7 No External Support………………………………8

[254]

NO QUESTIONS AND FILTERS

CODING CATEGORIES SKIP

Other ________________________________96 DK ………………………………………………...99 Specify In case of facilitation, Name Of Organisation:_____________________________ How many latrines did you receive support for: ____

51. 51 Did you receive help from any organization for latrine construction in any of the following?

IGNORE REDUNDANT QUESTION

Need Identification….……………………………..1 Supply Facilitation (Design, Material)……………2 Loan ………..…..…………………………………..3 Cost Share Basis…………………………………..4 Communal Water Facility ……………………….5 Subsidy……………………………………………..6 Trained Skilled Labour……………………………7 No External Support………………………………8 Other ________________________________96 DK ………………………………………………...99 Specify In case of facilitation, Name Of Organisation:_____________________________ Type of Water Facility: ______________________

[255]

NO QUESTIONS AND FILTERS

CODING CATEGORIES SKIP

52. 46 Did you easily get the required supplies necessary for construction of latrines?

Yes…………………………………………………1 No…………………………………………………..2 DK………………………………………………….99

53. 47 From where did you get the supplies?

General hardware shop in Yes 1 No 2

the same village Shops in the nearby village Yes 1 No 2 Shops in the nearby city Yes 1 No 2

54. 48 Did you get any of the latrine supplies/material on loan?

Yes…………………………………………………1 No…………………………………………………..2 Dk………………………………………………….99

55. 49 Are trained skilled masons/labour available in your village that can construct a latrine?

Yes………………………………………………….1 No………………………………..…………………2 DK…………………………………………………99

56. 50 Is your household latrine cleaned regularly?

Yes ……..………………………………….……….1 No……..…………………………………………….2 DK…………………………………………………..99 Other_________________________________96 Specify

….Skip to 58 ….Skip to 58

57. 51 How frequently is your latrine cleared in a week?

____________________________

RESUME IN CASE OF SKIP FROM Q 56

58. 52 Who usually cleans the latrine in your house?

Boys………………………………………………..1 Girls………………………………..…………

[256]

NO QUESTIONS AND FILTERS

CODING CATEGORIES SKIP

RECORD ALL MENTIONED

…….2 Women…………………………………………….3 Men…………………………………………………4 Dk………………………………………………….99 Other______________________________ 96 Specify

59. 53 Is your latrine linked to drainage? CHECK: In Case of YES confirm with response for Q38. Observe Where Possible

Yes……………………………………………………1 No……………………………………………………..2 If YES, Out in the open………………………………………1 In to an Open Drain..………………………………..2 Underground Drain …………………………………3 Observation ____________________________

….Go to 61

60. 54 Are you aware if your latrine’s underground drain is linked to any sort of waste water treatment system?

Yes………………………………………1 No…………………………………………..2 DK……………………………………….99 If YES What Type? ______________________________

RESUME IN CASE OF SKIP FROM Q 59

61. Have you received any awareness message from any organization about the hazards of open defecation in the last One Year?

Yes……………………………………1 No…………………………………..2 If Yes Name Of Organization-----------------------

62. 37 What are the consequences/ hazards of open defecation? RECORD ALL

Dirty Surroundings……………………………..…1 Diseases…….…………………………..…2 Environmental Problems……………………..…..3 Physical Problems From Not Being Able To

[257]

NO QUESTIONS AND FILTERS

CODING CATEGORIES SKIP

MENTIONED

Go Out During The Day (Women)…………………….….4 DK………………………………………99 Other: ________________________96 (Specify)

63. 60a

Are you aware of any design option for constructing a low cost latrine?

Yes……………………………………….1 No……………………………………...2 If Yes What type of Latrine ________________ If Yes How did you get to know this option____________________________

64. 38 Given the consequences/ hazards of open defecation, would you be willing to construct a latrine for your household?

Yes……………………………………….1 No……………….……………………………2 DK…………………………………………99

….Go to 66 ….Go to 66

65. How would you bear the construction cost?

Self-finance……………………..…………1 Loan…….…………………………….……2 Cost-Share Basis………..…………………..……3 Other: ___________________________96 (Specify)

RESUME IN CASE OF SKIP FROM Q 64

66. Are there any communal latrines available? Communal Latrines : These are not household based latrines and are located at a common place for the use of several households, they are also usually

Yes……………………………………….1 No…………………………………………2 DK…………………………..…………...99

….Go to 70 ….Go to 70

[258]

NO QUESTIONS AND FILTERS

CODING CATEGORIES SKIP

more than one in number, e.g are latrines in schools, mosques..

67. 60 Do you or your household members use communal latrines?

Men Yes……………………………………...1 No………………………………………….2 Women Yes……………………………………...3 No………………………………………….4 Girls Yes…………………………….……….5 No…………………………………………6 Boys Yes………………………………………..7 No…………………………………………8 DK…………………………………………99

….Go to 70 ….Go to 70 ….Go to 70 ….Go to 70 ….Go to 70

68. 60 Are there separate communal latrines for men and women?

Men Yes…………………………………………...1 No………………………………………….2 Women Yes………………………………………...3 No……………………………………….4 DK…………………………………………99

69. Where are these communal latrines located?

Mosque…………………………………..…1 School………………………..……..……2 DK………………………………………99 Other__________________ ______ 96 Specify

RESUME IN CASE OF SKIP FROM Q 66-67)

70. 62 What do you do with your household garbage?

Open Pit………………………….………..1 Closed Pit………………………………….2 Anywhere………………………………….3 Burning ………………………………...….4 Garbage Collector……………………...…5

[259]

NO QUESTIONS AND FILTERS

CODING CATEGORIES SKIP

Communal Waste Disposal Ground…….6 DK……………………………….…..99 Other _______________________ 96 Specify

Thank you for taking the time to meet with me and answer these questions. The information you provided will be kept strictly confidential and will be used with anonymity

Do you have any questions or comments for us?

OBSERVATION BASED QUESTIONS (with women respondents only check household for the following questions) Q. 14 , Q 15 and Q 32

[260]

Public School Sanitation Survey Questionnaire Sanitation Programme at Scale in Pakistan (SPSP) – Rural through Pakistan

Approach to Total Sanitation (PATS) Evaluation Feb 2014

Introduction to the Interviewee (Informed Consent):

I am _________________________, a representative of an independent consulting entity i.e. the AAN Associates, who has been contracted for the Evaluation of UNICEF’s Sanitation Programme at Scale (SPSP) in Pakistan using the Pakistan Approach to Total Sanitation (PATS). This programme aimed to improve sanitation practices, using Community-led Total Sanitation (CLTS) and School-led Total Sanitation (SLTS) approaches under the overarching PATS strategy.

I would like to ask about health, hygiene and sanitation related aspects/services at schools and about children’s knowledge and practices. This information will help Unicef and partners to assess projects achievements and performance in health, hygiene and sanitation related services for the children. The survey takes around 30 minutes to complete.

Whatever information you provide, will be kept strictly confidential and dealt with anonymity.

Participation in this survey is voluntary and you can choose not to answer any individual questions. However, I hope you will participate in this survey since your views are important.

At this time, do you want to ask me anything about the survey?

RESPONDENT AGREES TO BE INTERVIEWED

RESPONDENT DOES NOT AGREE TO BE INTERVIEWED END

INSTRUCTIONS:

RESPONDENT: QUESTIONNAIRE IS TO BE ADMINISTERED WITH SCHOOL

PRINCIPAL/SENIOR TEACHER

MULTIPLE CHOICE QUESTIONS REQUIRE FURTHER PROBING AND ARE ALSO

HIGHLIGHTED WITH CAPTION OF “RECORD ALL MENTIONED”.

RESPONSE: FOR EACH QUESTION CIRCLE THE APPROPRIATE RESPONSE/S.

OBSERVATION WALK: AFTER THE INTERVIEW TAKE A WALK AROUND THE SCHOOL AND

RECORD OBSERVATIONS.

Edited/Verified by: Approved By:

District Supervisor Provincial Coordinator

Signature: Signature:

Date: Date:

GENERAL INFORMATION

[261]

RECORD NUMBER: (001-TEAM XX)

1 Date of visit ____/____/_____

2 Level of school Primary Secondary/High Secondary

3 Type of School Boys Girls Coed

4 Number of students Boys:__________ Girls:____________

5 Number of students with physical disabilities

Boys: __________ Girls:_________

6 Number of Teachers Male: __________ Female:_________

7 Name of the school _______________________________

8 Province _______________________________

9 District _______________________________

10 Tehsil _______________________________

11 Union Council _______________________________

12 Village _______________________________

13 Distance from Community

(in minutes by walk)

____________min ___________ max

Health and Hygiene

No Question Response Skip Pattern

14 Do the classrooms have the following? RECORD ALL MENTIONED For observation visit one random class room. (Ventilation – Class should not smell musty, or be suffocative)

Windows 1 Ventilation 2 Dust Bin 3

15

What is the condition of the classroom floor? For observation visit one random class room.

Earth 1 Concrete 2 Other ____________96 and Clean 1 Unclean 2

[262]

16 Does the school dispose (solid waste) garbage?

Yes ……………………1 No……………………..2 DK………………..…..99

Go to 18 Go to 18

17 How frequently does the school dispose (solid waste) garbage? And how?

Daily 1 Weekly 2 Irregular 3 No disposal 99 Other ____________ 96 Incinerate 1 Collection Service 2 Thrown/Piled outside of school premises 3 Other____________ 99

18 Does the school have any cleaning staff?

Full time 1 Part time 2 None 3

19

During the last six month did the school/any other organization organize health and hygiene awareness activity?

School Yes 1 No 2

Organization Yes 1 No 2 Name of Organization ___________________

If ‘No’ in both cases Goto 21

20 What topics were covered by the awareness raising exercise? Circle all mentioned

Personal Hygiene…...….1 Clean And Safe Water…2 Water Born Diseases…..3 Environmental& Domestic Hygiene….……………….4 Safe Disposal Of Human Excreta......................….5 Solid Waste (Garbage) Disposal………………….6 Do Not Know…………..99 Other: _____________96

OBSERVATION ONLY

21 Is the school compound free from solid wastes?

Yes 1 No 2

22

Does the school have flowing liquid waste/puddles of stagnant water?

Yes 1 No 2

WATER SUPPLY

[263]

23

Is water available in the school for the following purposes? Mention all and circle each response

Drinking 1 Hand washing 2 Latrine 3 Not Available 4

Go to 25

24 What is the water source? RECORD ALL MENTIONED

Piped Water Into School 1 Piped Water To Yard/Plot 2 Public Tap/Standpipe 3 Tube Well/Borehole 4 Protected Dug Well 5 Unprotected Dug Well 6 Protected Spring 7 Unprotected Spring 8 Rainwater Collection 9 Bottled Water 10 Cart With Small Tank/Drum 11 Tanker-Truck 12 Surface Water 13 Don’t Know 99 Other___________________ 96 (Specify)

25

What type of water storage does the school have? RECORD ALL MENTIONED

Overhead Tank 1 Underground Tank 2 Open Drum 3 Covered Drum 4 Other ____________ 96

26 Are drinking points available for children?

Yes 1 No 2

27

Are hand washing points available for children’s use in the school?

Yes 1 No 2

28

What do children do when school water point is not working?

For drinking:

For hand washing __________________________ __________________________

OBSERVATION ONLY

29 Is soap available at the hand washing point?

Yes 1 No 2

[264]

30 Is the area around the hand washing facility clean?

Yes 1 No 2

31

Is the area around the drinking source clean?

Yes 1 No 2

SANITATION

32

Is a latrine available in the school compound?

Yes 1 No 2

If No Goto 45

33 What is the total number of latrines for the following?

Teachers _______ Boys _______ Girls _______

34 Is the latrine in usable condition?

Yes 1 No 2 DK 99

35 What is the type of school latrine? Hanging Latrine: Where the latrine is built directly over body of water (river, stream)

Flush/Pour Flush To: Piped Sewer System 1 Septic Tank 2 Pit Latrine 3 Elsewhere 4 Ventilated Improved Pit Latrine 5 Pit Latrine With Slab 6 Pit Latrine W/O Slab/Open Pit 7 Composting Toilet 8 Bucket 9 Hanging Toilet/ Latrine 10 No Facilities/Bush/Field 11 Other___________________ 96 Hanging Latrine: Where the latrine is built directly over body of water (river, stream)

36 What type of floor does the latrine have?

Concrete 1 Slab 2 Earth 3

37 What is the structure of the latrine?

Walled 1 Has Door 2 Ceiling 3 Other _____________96

38 Does the latrine have a drain pipe?

Yes 1 No 2 DK 99

39 Is water available in the latrines for washing

Piped/Tap 1 Bucket 2

[265]

purposes?

Other ______________96

40 Is container (mug, pot) for ablution (Wazu) available?

Yes 1 No 2 DK 99

41 What do children do when latrines are not functioning/breakdown?

Open Defecate In School 1

Outside school 2 Go to home 3 DK 99 Other__________ 96

42 OBSERVATION ONLY

43 Is excreta visible around the latrine? Enumerator to check latrine facility

Yes 1 No 2

44 Was there excreta inside the latrine? Enumerator to check latrine facility

Yes 1 No 2

45 Are there children that defecate or urinate in the open?

Defecate Yes 1 No 2 Urinate Yes 1 No 2

Go to 47 Go to 47

46 Which students defecate in the open?

Boys (5 Years And Above)……..1 Girls (5 Years And Above)…..…2 Boys (Under 5 Years)………..…3 Girls (Under 5 Years)…...………4 DK……………………………….99 Other____________________96 Specify

47 Is soap available? Inside the latrine 1 Near Latrine 2 Not Available 3

48

Are children with physical disability able to use the hand washing facility on their own?

Yes 1 No 2 DK 99

49 Are children with physical disabilities able to use the

Yes 1 No 2

[266]

latrine on their own?

DK 99

50. OBSERVATIONS _________________________________________________________________________ _________________________________________________________________________ _________________________________________________________________________ _________________________________________________________________________

24. Annex 13 SPSP Outputs - Target Vs. Actual

No. Output Indicators Target

Actual

Difference

As of

30-Apr

A. Demand for sanitation to achieve open defecation free status created in targeted communities Target Actual

1 Boys and Girls reached with sanitation messages in Schools. 103,000 176,022 171%

2 Men, Women, Boys & Girls reached with sanitation messages 776,000 1,200,529 155%

3 People reached with sanitation messages through electronic/print media. 970,000 540,786 56%

4 People reached with sanitation messages through community events (theatres, puppet shows) 291,000 195,450 67%

5 No. of training events conducted for school teachers (men, women) on facilitating SLTS 23 84 365%

6 No. of WASH clubs formed in schools and strengthened with planned follow up capacity development sessions 692 881 127%

7 No. of events conducted by school wash clubs (boys and girls) as per plan 886 1,433 162%

8 No. of training events conducted for social organizers on participatory tools and IEC material for facilitating PATS 8 85 1063%

9 No. of social organizers successful using participatory tools and IEC material for facilitating PATS in villages according to developed criteria 230 911 396%

10 No. of training events conducted for CRPs on conducting CLTS in villages 46 67 146%

11 No. of CRPs successfully facilitating CLTS in villages according to developed criteria 1,385 3,471 251%

12 No. of VSCs existing/formed and trained 1,385 2,994 216%

13 No. of villages/clusters with developed Community Action Plan (CAP) 1,385 1,878 136%

14 No. of CAPs developed with prioritized need for action by community on Hygiene Behaviours 1,385 2,233 161%

B. Sanitation demand sustained with supply side interventions

15 No. of masons trained on sanitation construction and using their learnt skills 692 1,250 181%

16 No. of latrines (other than demo latrines) constructed by masons trained under PATS programme 13,857 20,040 145%

17 No. of complete demonstration latrines constructed for extremely vulnerable families as per agreed criteria with community (maximum 3%) 4,157 5,212 125%

[267]

18 No. of demonstration latrines constructed through materials provision/vouchers for vulnerable families as per agreed criteria with community (max 7%) 9,700 8,581 88%

19 No. of entrepreneurs/manufacturer trained and producing low-cost sanitation solutions 692 469 68%

20 % of trained entrepreneurs doing sanitation related business 80% 199 24930%

21 No. of facilitated sanitation marts doing profitable business 346 82 24%

22 No. of lending institutions successfully approached to give loans for sanitation facilities 4 8 200%

23 No. of households received sanitation loan from some lending institution facilitated by PATS programme 140 0 0%

24 No. of communal water facilities constructed as incentives/rewards 620 552 89%

25 No. of communal sanitation facilities constructed as incentives/rewards 159 88 55%

26 No. of house sanitation facilities constructed as incentives/rewards 11,179 1,202 11%

27 No. of house water facilities constructed as incentives/rewards 11,262 1,383 12%

28 No. of critical community infrastructure constructed as incentives/rewards 158 2 1%

29 No. of health and hygiene packages given as incentives/rewards 242,000 253,330 105%

C. Improved Health and Hygiene behaviors promoted with the target community

30 No. of people reached with WASH BCC messages through electronic/print media. 970,000 609,530 63%

31 No. of people reached with WASH BCC messages through community events (theaters, puppet shows) 183,300 165,092 90%

32 No. of school children (boys and girls) reached with messages through SLTS activities. 63,941 154,526 242%

33 No. of people (men, women, boys, girls) reached with WASH BCC messages through CRPs 291,000 846,444 291%

D. Linkages developed with duty bearers/public institutions to support PATS

34 No. of capacity needs assessments (CNA) conducted for relevant public sector stake holders 14 6 43%

35 No. of capacity development plans developed for relevant public sector stake holders by Implementing Partner 14 6 43%

36 No. of capacity development events undertaken for relevant public sector stake holders by Implementing Partner 14 15 107%

37 No. of government officials and political leaders participated in capacity development and key events of the project. 185 230 124%

38 No. of advocacy initiatives undertaken with duty bearers/political leadership to support PATS 14 17 121%

39 No. of MoUs signed by implementing partner with relevant public institutions responsible for sanitation 14 17 121%

40 No. of villages jointly selected for PATS programme by relevant Government duty bearers with IP 1,385 1,791 129%

41 No. of ODF communities received rewards/incentives by government 39 62 159%

E. Villages supported in attaining Total Sanitation status

42 No. of households with new latrines constructed after start of the PATS programme 17,182 33,673 196%

43 No. villages declared as ODF by community 831 1,436 173%

44 No. of villages Verified as ODF by Implementing partner and independent monitors 831 1,250 150%

45 No. of people (men, women, boys, girls) in communities declared as ODF 581,700 711,467 122%

46 No. of people (men, women, boys, girls) in communities verified as ODF 581,700 679,967 117%

[268]

47 No. of model villages with laid sewers, WW collection and treatment system through PATS programme 4 18 450%

48 No. of advocacy initiatives undertaken to promote communal underground drainage system linked WW treatment system and communal SWM system 2 22 1100%

49 No. of ODF villages provided with underground drainage and WW treatment facility by relevant public sector stake holder/duty bearer 2 0 0%

F. PATS programme Monitoring Evaluation and Learning Framework implemented

50 No of Inception workshops conducted in the beginning of project 12 12 100%

51 No. of villages with baseline established in the beginning of project by IP 1,385 7,448 538%

52 No. of joint progress monitoring reports (IP and public institution) 80 39 49%

53 No. of monthly substantive monitoring reviews’ visits conducted for confirming/measuring performance indicators in results framework 80 216 270%

54 No. ‘monthly substantive monitoring review’ reports shared among partners 63 55 87%

55 % of responsive progress reports submitted as per MEAL framework by IP 100% 33 3336%

56 No. of quarterly progress reviews conducted with IP and minutes documented 2 15 750%

57 No. of mid-term review workshop report endorsed by UNICEF 1 4 400%

58 No. of Human interest stories documented by IP and accepted as per UNICEF guidelines 80 78 98%

59 No. of end line survey reports accepted by UNICEF 11 2 18%

60 No. of Lessons sharing workshop conducted and feedback documented 1 1 100%

[269]

25. Annex 14

Reference Group for Evaluation

1. Why a Reference Group?

The draft Regional Evaluation Strategy requires all COs to form a Reference Group (RG) for each evaluation. According to the UNICEF Evaluation guidelines, RG is a small group of specialists and stakeholders related to an intervention to be evaluated, which is formed by the concerned programme in collaboration with the Evaluation Management Committee (EMC).

The RG should ideally consist of members from all key stakeholders, government, implementing partners and one or two technical experts relating to the sector under evaluation. For each evaluation, members of the RG should be selected carefully on the basis of relevance to the sector and intervention. The RG will support the internal EMC in overseeing the entire process of a specific evaluation and assist in ensuring quality, adherence to UNEG standards and making the evaluation useful for all stakeholders.

The RG is an interface between the Programme (focal person/s for evaluation) and the external evaluation team. Its members help the external evaluation team to identify data sources and to access them. They validate the evaluation questions and discuss the conclusions and recommendations. The reference group allows the variety of points of view on the evaluated intervention to be expressed.

2. Roles and Responsibilities Discuss and comment on the terms of reference drawn up by the evaluation

manager (focal person assigned by the programme/EMC). Aggregate and summarise the views of the commission services and act as an

interface between the evaluation team and the services, thereby supplementing bilateral contacts.

Help ensure that the evaluation team has access to and consults all information sources and documentation on activities undertaken.

Help validate the evaluation questions and methodology. Discuss and comment on notes and reports produced by the evaluation team.

Comments by individual members of the reference group are compiled by the focal person managing the evaluation process and subsequently transmitted to the evaluation team.

Assist in feedback of the findings, conclusions and recommendations from the evaluation.

3. Possible Membership Different types of actors can be invited to participate in the Reference Group. For evaluations managed in the country, membership may extend to: UN counterparts (where applicable) Government partners/services Selected development partners Selected civil society organizations

[270]

Experts15 (One or more members of the reference group may be invited as a resource person, for their expertise)

As per UNICEF guidelines, RG should be limited to 08 to 10 members for effective functioning.

4. Process

The Chief/Focal Person/Manager/ managing the evaluation:

Officially announces the launching of the evaluation (as per the IMEP);

Identifies the services, institutions or organisations (key stakeholders) and sends request to appoint a person as a member of the RG;

Request expert/s to join for technical support;

Shares a list of all members of reference group with EMC/PMER; and

Coordinates all activities of RG

5. When and how to involve the Reference Group? The RG is generally involved in the following way16:

Stage Meeting Role

Terms of reference Comments

Engagement of the external evaluation team Comments

Evaluation questions/methodology 1 Validation of ToR

Inception report Comments

Draft/First phase report (desk) 2 Discussion, comments

Field phase debriefing 1-2 Discussion

Draft final report, conclusions and recommendations

2-3 Discussion, comments

6. Opening the evaluation process for wider comments and advice

In certain cases, under the guidance of the RG and its inputs, the evaluation manager/programme may benefit from a wider variety of views through other channels like identification of and consultation with broader stakeholders by the evaluation team in the first stage of the process; participatory workshops, focus groups or expert panel(s) during the evaluation process; or discussion of the draft evaluation report in a widely attended seminar.

15 On engaging experts, further guidelines are available on

http://ec.europa.eu/europeaid/evaluation/methodology/methods/mth_exp_en.htm 16 For details on how to engage the Reference Group, detailed guidelines are available on

http://ec.europa.eu/europeaid/evaluation/methodology/methods/mth_stg_mee_en.htm