meta-evaluation and synthesis of the 2010 pakistan floods

Meta-evaluation and Synthesis

of the

2010 Pakistan Floods Response

by SHO participants

A Synthesis of Conclusions

Report phase 2

Verona Groverman

Joke Hartmans

5 December 2012

2

Content

Acronyms ................................................................................................................................................. 3

1. Introduction ..................................................................................................................................... 4

2. Background on the emergency situation ........................................................................................ 4

3. Synthesis: background and process ................................................................................................. 7

4. Synthesis of the main conclusions of the evaluations of responses to the flood disaster in

Pakistan............................................................................................................................................ 8

4.1 Relevance/ appropriateness .................................................................................................... 8

4.2 Effectiveness and impact ......................................................................................................... 10

4.3 Efficiency .................................................................................................................................. 11

4.4 Coverage ................................................................................................................................. 12

4.5 Connectedness: ....................................................................................................................... 12

4.6 Coherence ................................................................................................................................ 15

4.7 Standards ................................................................................................................................. 17

4.8 Cross-cutting themes: Gender and Vulnerable groups .......................................................... 20

4.9 Strengthening of responses ..................................................................................................... 23

4.10 Cooperation ........................................................................................................................... 25

4.11 Lessons Learnt ....................................................................................................................... 28

5. Concluding remarks ....................................................................................................................... 30

Annex 1: SHO Terms of Reference Meta-evaluation and synthesis of the 2010 Floods in Pakistan .... 33

Annex 2: List of evaluation reports and other literature used .............................................................. 39

Annex 3: Overview of provinces with districts in which SHO participants supported interventions and

the focus or sector of support ....................................................................................................... 40

3

Acronyms

AJK Azad Jammu and Kashmir ALNAP Active Learning Network for Accountability and Performance in Humanitarian Action CBO Community-Based Organisation CoC Code of Conduct (for the International Red Cross and Red Crescent Movement and Non-

Governmental Organisations in Disaster Relief) DRM Disaster Risk Management DRR Disaster Risk Reduction HAP Humanitarian Accountability Partnership (Standard in Accountability and Quality

Management) IASC Inter-Agency Standing Committee (international inter-agency coordination forum for

humanitarian assistance involving key UN and non-UN humanitarian partners) NDMA National Disaster Management Agency NFI Non-Food Items NGO Non-Governmental Organisation KPK Khyber Pakhtunkhwa PHF Pakistan Humanitarian Forum RTE Real Time Evaluation SHO Samenwerkende Hulporganisaties (Cooperating Aid Organisations) UN United Nations WASH Water, Sanitation and Hygiene promotion

Synthesis of conclusions on the 2010 Pakistan floods response by SHO members 4

1. Introduction

In July and August 2010 heavy monsoon rains swept over Pakistan, eventually causing catastrophic

floods in about one third of its geographical area. More than 18 million men, women and children,

i.e. 10% of the population, suffered from the effects of these floods which caused the death of

almost two thousand people and damaged buildings, infrastructure and livelihood assets. SHO raised

just over 27,5 million euro for this emergency: 25,5 million in donations from the Dutch public,

complemented with a contribution of 2 million from the Dutch government.i The participating SHO

members, Cordaid (Mensen in Nood), ICCO & Kerk in Actie, The Netherlands Red Cross, Oxfam Novib,

Save the Children, Stichting Vluchteling, Tear, UNICEF Nederland en World Vision, spent these funds

on housing, medical care, food, water, sanitation, education and livelihood opportunities.ii

In 2011 and 2012 each of the participating SHO members commissioned independent evaluators to

evaluate their responses. Building on the conclusions of these evaluations, the present report

summarizes and synthesizes the answers on key questions related to twelve issues, in accordance

with the Terms of Reference attached in Annex 1. A first set of six issues concerns evaluation criteria,

namely, appropriateness/relevance, efficiency, effectiveness and impact, coverage, connectedness

and coherence. The second set of six issues regards the compliance with internationally agreed

humanitarian standards (particularly CoC and Sphere), cross-cutting themes (gender, vulnerable

groups, strengthening of local responses), and organisational issues (cooperation and coordination,

lessons learnt from previous disasters). Chapter 4 of this report presents the summary of combined

conclusions that answer the key questions related to the twelve issues.

The synthesis concerns the second phase of a three-phased meta-evaluation. It builds on phase 1,

which is a quality assessment of the evaluation reports mentioned above. Phase 3 of the meta-

evaluation is meant for learning purposes of the SHO members and will build on the outcomes of the

previous phases.

For proper understanding of the information on the twelve issues presented in chapter 4, the report

first briefly describes the emergency context in Pakistan during 2010 and 2011 (chapter 2). This is

followed by a description of the synthesis’ background and process, which includes relevant

information on the quality assessment process (chapter 3). In the final chapter 5 we, two

independent consultants commissioned by SHO to conduct the meta- evaluation and synthesis,

Verona Groverman and Joke Hartmans, give some concluding remarks.

2. Background on the emergency situation

Mid July 2010 Pakistan experienced heavy monsoon rains that lasted for more than eight weeks,

causing floods that evolved from normal flash floods into a massive disaster affecting large parts of

the country. The floodwater waves washed down from north to south as the Indus River extended to

about forty times its usual size and - at one point - submerged a fifth of the country’s land mass. The

floods directly and/or indirectly affected 78 of Pakistan’s 121 districts, devastating and submerging

entire villages, washing away houses, health and education facilities, roads, bridges, water supply

and sanitation infrastructure as well as livestock and agricultural lands.


The Pakistan 2010 floods are considered amongst one of the major disasters of the 21st century due

to the disaster’s widespread geographical scale and distribution (from the Himalayan Plateau to the

Arabian Sea), the unprecedented caseload of affected population and its economic impact.iii

According to the Pakistan’s National Disaster Management Authority (NDMA), the 2010 floods

constitute the country’s largest disasteriv as over 18 million persons, almost 10% of the country’s

population, were affected. Despite the scale of the disaster, casualties remained relatively low: 1,985

people were killed and around 3,000 injured.

Initially, the provinces of Balochistan and Khyber Pakhtunkhwa (KPK) were flooded. In mid-August, as

flood waters flowed south, Punjab and Sindh provinces experienced resultant widespread flooding.

The impact of the flood was very diverse in each province due to the changing nature of the disaster,

the different levels of preparedness and the access to individual and common resources. Waters in

Balochistan and KPK receded within days; this took several weeks in Punjab and months in some

areas of Sindh. Sindh was the worst affected province, as the Indus River did not find an outlet due to

the flat topography of this area.

Where water receded rapidly most of the displaced population was able to return during the months

of August and September. However, in January 2011 four districts of Sindh and one district in Punjab

were still submerged. Over 75 percent of people affected were in Sindh and Punjab: Sindh had the

highest number of persons affected (7.2 million), followed by Punjab with 6 million people and KPK

with 3.8 million people. Areas in Sindh province were confronted with longer‐term displacements

and situations where most vulnerable parts of the population remained under severe difficulties

from recovering. Their livelihoods could not be re‐established as quickly as in other affected areas

due to lack of access to land.

The emergency was further compounded by pre‐existing chronic poverty, inequality, limited access

to basic services, inadequate governance, fragmentation of the state, military dominance and the

superposition of conflict and displacement in KPK and Balochistan provinces. Nationwide, the floods

washed out years of achievements through developmental efforts; the country suffered extensive

damage to crops and physical infrastructure. The total economic impact of the floods is estimated to

be around $10 billion.

The impact of the floods has worsened chronic poverty and inequality, especially among the most

vulnerable parts of the Pakistani population.v Some of the main reasons behind the already existing

poverty levels and inequalities are of a structural nature and related to restricted access to social

services and land due to a feudal system of landownership.vi Recovery of livelihood in Sindh, for

instance, was impeded by economic debts to landlords. Patriarchal socio-cultural structures and

religious fundamentalism hindered the effectiveness of the relief effort and the recovery capacity,

particularly, that of women and minorities. Relief work was also hampered by the difficult logistical

terrain, the destruction of infrastructure, government bureaucratic hurdles, lack of effective

coordination and the threat of terrorist attacks. These factors made this emergency response one of

the most difficult ones in recent times.vii


3. Synthesis: background and process

Prior to making this synthesis, we assessed the quality of sixteen evaluation reports provided by the

nine SHO participants mentioned aboveviii. An assessment grid was used with eight evaluation

criteria, namely, meeting needs (i.e. demands in the Terms of Reference), appropriate design,

reliable data, sound analysis, impartial conclusions, useful recommendations and clear report.ix The

synthesis draws on the conclusions provided in nine evaluation reports which proved to be of

satisfactory quality. Annex 2 gives the list of these reports. All of the nine selected reports scored

positive on impartiality of conclusions. These reports evaluated the interventions of six of the nine

SHO members that responded to the 2010 Pakistan floods emergency.x

In the search for information on the twelve issues to be addressed in the synthesis, we faced some

challenges. First, in six of nine reports, the conclusions were intertwined with the findings, mostly in

a clear sub-section but at times as statements in the main text of findings. This complicated the

extraction of the conclusions.xi Second, due to the different foci of evaluations, in many cases the

evaluations’ conclusions did not separately address the issues regarding the second set of issues

(standards, gender, vulnerable groups, strengthening of local responses, and so on). It required a

search in the full evaluation reports to get sufficient information about the issues concerned. These

two factors made the process of bringing the conclusions together rather time consuming.

Another, methodological issue concerns the extent to which information in different evaluations can

be summarized. First, although the Terms of References of the evaluations showed overlap, each

evaluation had its specific emphasis. Second, the evaluation reports showed variety in nature and

focus which required caution in comparing the information provided on the twelve issues. More in

particular, the role of the commissioning organisation in the delivery of the response was diverse:

some agencies were directly implementing activities, whereas others worked with local or

international implementing partners or the national branch of the international umbrella

organisation. As regards the focus, some evaluations merely reflected on programme interventions,

others also took organisational and institutional issues into account, and one evaluation focussed

mainly on institutional aspects of the emergency.

Lastly, a hindering factor for the whole exercise of synthesising conclusions of the evaluation reports

was the limited time available (six days including reporting). The synthesis has, therefore, the nature

of a summary of main conclusions on the twelve issues. It does not include an analysis of the

information brought forward in the reports. In only a few explicitly indicated cases we have added

our thoughts about an issue. Still, we expect that the outcome will give a good input for phase 3 of

the meta-evaluation which is about learning.


4. Synthesis of the main conclusions of the evaluations of responses

to the flood disaster in Pakistan

In this chapter twelve issues are addressed which, together, will provide an overview of the main

outcomes of the responses of the different SHO participants included in this synthesis. Each of the

twelve issues answers key questions as per the Terms of Reference of the Meta-evaluation and

Synthesis. The sections start with the main answer to the key questions based on our findings, which

is elaborated in the text that follows. We have put the references to the reports in endnotes (to be

found at the very end of this synthesis report). We made an exception for one report, i.e. the Inter-

Agency RTE. This Real Time Evaluation does not concern an evaluation of one specific SHO

participant but gives an overall assessment of the combined emergency responses (UN and NGO)

nationwide. It, therefore, provides interesting lessons about the relief efforts in general. The full

titles of the reports can be found in Annex 2.

We like to stress that the main conclusions of the nine evaluations formed the basis of the

description. For some issues though we had to rely on the findings presented because most reports

did not draw conclusions on the issue. This holds for connectedness, coherence, gender and

vulnerable groups, strengthening of responses and cooperation. We also like to underline that the

evaluations concerned six of the nine SHO members participating in the Pakistan floods response

(see Annex 2). Their evaluations mostly included various projects in a number of districts,

implemented by different local or international partners of SHO participants or national branches of

international umbrella organisations or international member organisations of international

networks to which two SHO participants belong.

4.1 Relevance/ appropriateness: Was the response tailored to local needs, enhancing ownership and accountability?xii

All evaluations addressed the relevance and appropriateness of the interventions. These issues are

presented and discussed at length in the findings sections of the reports, which we will not repeat in

this report. The evaluators concluded for both the relief and recovery phase that the overall flood

response was relevant and appropriate. Additional to our elaboration below we like to add that

various reports pointed at factors which contributed to the relevance, such as the selection of

interventions, the selection of targeted districts (see section 4.4), flexibility to address the needs in

different phases of the emergency (more in section 4.5), the involvement of local communities in the

programme, and, last but not least, emphasized by one evaluationxiii, the focus on gender and

vulnerability (see further in section 4.8). Higher levels of ownership and accountability were found in

those interventions which purposely included the involvement of local communities.

As regards the relief phase, people were said to be satisfied with the food packages and NFI packages

provided. However, two evaluations added critical commentsxiv. The first comment concerned the

relevance and appropriateness of the standard roll out of (four) clusters which the Government of

Pakistan, donors and implementing partners unanimously questioned as follows: the clusters

operated independently from contextual realities and to a large extent also to the phases of the

operation (more about the issue of cooperation in section 4.10). The second comment regarded a

few items in the NFI packages that were less appropriate and proper, such as blankets, knifes and


some expired goods. Another comment was about the need for cash. It appeared that there was a

high demand for cash in the communities, which was not provided by the agency due to past

negative experiences. According to the evaluators cash would have allowed the agency to respond

faster and cheaper and more in line with people’s specific needs in many though obviously not all

places.

The evaluations focusing on relief and recovery concerned interventions implemented by local or

international partners of SHO participants and by a SHO participant itself.xv These interventions were

based on the immediate needs of affected people and resulted in high community satisfaction.

According to these reports the most crucial needs addressed were related to food security and

livelihood, WASH and health. Distribution of NFIs and rehabilitation of shelter and water systems also

responded to the urgent needs of the victims. One evaluation critically commented on the provision

of shelters.xvi Firstly, the organisation concerned had overestimated the need for houses to assist the

urgent cases. Secondly, although the quality of the shelter was generally good, the culturally

appropriateness of the design left to be desired: the construction of the latrines not directly adjacent

to the house did not ensure the safety and privacy for women, and the provision of a kitchen was

insufficient. According to another evaluationxvii lands rights represent a key constraint both for

permanent shelter as many people returning home find themselves without having a place to plant

or to build a house. The same applied to livelihood restoration.

Looking at the level of beneficiary appreciation for different types of interventions, livelihood

services were rated among the highest because people felt that they increased their self-reliance,

helped them stand on their own feet and allowed them to obtain the cash needed for many of their

other needs. This conclusion is in line with the evaluation of the livelihood programme (recovery

phase)xviii in which seed and fertiliser distribution, tube-well and generator repair, and land clearance

were well in line with the expressed recovery needs and priorities of the affected people.

Interestingly, not all responses included livelihood support.

The extent to which accountability was enhanced in the local communities appeared to depend,

among others, on the degree of their involvement in the intervention. This is evidenced by different

findings in three evaluation reports.xix One evaluation of the relief phase that concluded a low level of

accountability, found that the agency did not consult people at all about their needs at the village

level. The people received the same general package given across all provinces. They were also not

informed about the exact items they would receive when tokens were given as other organisations

did. Other evaluators concluded that the implementing agencies addressed accountability concerns,

though there were still large percentages of people who reported not being consulted, informed

about the exact programme support planned or knowing the complaint mechanisms. Positive

evidence comes from a third evaluation report. It concluded that the involvement and active

participation of the affected communities from the very early stage of the response highly

contributed to better planning and response and the project’s success as a whole. It also pointed to

the importance of the use of data from other secondary sources and information generated by other

humanitarian actors, to ensure relevance and accountability during project implementation,

especially concerning NFI distributions, health and hygiene awareness-raising and sanitation

interventions.


4.2 Effectiveness and impact: Was the response effectively implemented?

Generally speaking, the evaluation reports concluded that the response in the relief and recovery

phases was effectively implemented and had a positive impact, but with clear variations between

locations and sectors. Overall, objectives and coverage targets had been achieved, in many cases

exceeding the planned targets. The reports presented and discussed many findings that underlie the

conclusions on effectiveness and impact, which we will not repeat in this report. Factors that

contributed to the interventions’ effectiveness and impact were, among others, prior presence of

implementing agencies in Pakistan, the decision to work through local partners, organisations’

experience in responding to previous emergencies, the ability to mobilize staff, teams, funds and

equipment internationally, strong appeal success, and, for some agencies, the large size of the

volunteer base. Various factors hindered effectiveness and impact, such as the security situation,

bureaucratic hurdles, delays in appeal funds release, high volunteer turnover and coordination

issues.xx The latter will be dealt with in section 4.10 on Cooperation.

According to various evaluationsxxi the interventions’ contribution was significant in operational

terms, i.e. meeting immediate needs related to food, non-food items, drinking water, health,

sanitation, shelter and livelihoods. More specifically on the latter, seed and fertiliser distribution and

land clearance scored (very) well, while tube-well and generator repair scored average on

effectiveness.xxii As mentioned earlier, effectiveness and impact varied between locations and

sectors. According to one evaluation focusing on the relief phase, not in all communities the

livelihoods and water needs were met, while another evaluation of the same phase gave examples of

less met needs in the WASH and health sector.xxiii Also, on timeliness evaluators found different

situations. For one programme considerable delays in delivery were reported for some districts,

while in other programmes the beneficiaries received help in the relief phase very quickly, but the

transition to early recovery was slow, and started almost a year after the floods.xxiv Findings in one

evaluation led to the conclusion that the interventions’ contribution was less successful in making a

strategic contribution in some areas, such as addressing pre-disaster vulnerabilities and enhancing

local capacities.xxv This brings us to the impact of the interventions.

In terms of impact on the people affected, various evaluators of the relief phasexxvi noticed an

increased food intake of women, men and children which positively affected their ability to engage in

repair and income activities. The tools supplied as NFIs greatly helped in repairs and livelihoods

activities. Food aid and NFIs distribution also reduced the need to take loans and to sell animals for

food purchases and replacing household items that were lost. Additionally, providing NFIs reduced

the burden on non-beneficiary households that were hosting them and using their scarce resources

to meet their immediate needs. People mentioned a sharp decline in child diseases due to the

preventive and curative health care provided and as a result of improved health and hygiene

practices. The overall environmental situation of the target villages also improved as a result of waste

management and sanitation facilities, which contributed to a reduction in cases of malaria and other

flood related diseases. One report concluded that women felt empowered due to their participation

in different project activities and their involvement in awareness-raising sessions and decision

making. According to us, such a statement is quite remarkable after just a few months of support. An

evaluation of full period of support found that women got confidence and started getting organized

themselves in groups with the help of partner organisations.xxvii (more about women in section 4.8).


Evaluations of the combined relief and recovery phasexxviii concluded the following about the

interventions’ impact: demand and need for WASH related programmes had increased; good hygiene

practices were noticed; the capacity of local and affected people were enhanced in terms of

humanitarian work; food security had improved; the scope and outreach of partner organisations

were enhanced, to mention some of the conclusions. These reports also added that by the time of

the evaluations two main sectors still needed more support, namely, livelihood (livestock) and

shelter, and that communities would have preferred more cash support, especially in the recovery

phase.

These examples of impacts were mostly noticed during field visits of the evaluators. We felt it

interesting that one evaluationxxix stated that it was difficult to verify the actual impact on meeting

basic needs and the restoration of livelihood options, as there were no quantifiable monitoring

systems measuring impact. In general, we did not find much information on monitoring and

evaluation systems in the evaluation reports (see also section 4.8).

4.3 Efficiency: Was the response efficiently implemented?

Five of the nine evaluation reports included, very briefly and without much reflection, the issue of

efficiency and cost-effectiveness. Moreover, they addressed it in entirely different ways. Therefore, a

concluding answer to the question ‘Was the response efficiently implemented?’ cannot be given.

Instead, we summarize the conclusions of the various reports below, which at the same time show

the variety in focus of the different reports:

- About one programme the conclusion was that it was implemented efficiently and also cost-

effective, and that it reflected a rational distribution of programmatic and operational costs. The

evaluators had calculated the cost per person which they assessed as cost effective. For another

agency the ratio of relief services and administrative costs had been calculated which compared

favourably with other agencies, concluded the evaluator.xxx

- Another evaluation focused on different types of interventions and concluded that the seed and

fertiliser distribution and land clearance scored (very) well and tube-well and generator repair

average on efficiency using a number of criteria.xxxi

- On the availability of (financial) data we found two observations.xxxii In one case, the agency had

not provided detailed budgets to the evaluator which hindered an analysis of efficiency in

various sectors or provinces. In another case, the evaluators found all the necessary documents

and data on program activities in digital form. They felt the program’s implementation

schedules, the project logical framework, and monitoring and evaluation mechanisms and tools

efficient and concluded that they presented an evident case of good practice.

- In two evaluation reports cost efficiency was related to the costs of materials.xxxiii For the WASH

sector it was found that indigenous resources like materials and local labour had not been fully

utilized. Thus, it was concluded that such utilization would have favoured cost efficiency.

Regarding infrastructure an evaluation team made efforts to compare costs but did not draw

conclusions on cost-effectiveness.


4.4 Coverage: Did the response reach major population groups facing life-threatening risk wherever they were? Was there appropriate geographical coverage within the affected region? xxxiv

When looking at coverage, it is relevant to distinguish between people who face life-threatening risk

but who are likely to recover relatively quickly due to resources and position and people who hardly

have adequate coping strategies. Most of the SHO participants focused their interventions on the

latter, the poor and most vulnerable groups, which generally refers to vulnerable in terms of

resources and position. Section 4.8 further elaborates on vulnerable groups and, also, on needs

assessment. In terms of geographical coverage, the conclusions of the nine evaluations were clear:

the interventions of the SHO participants took place in those areas that were most affected and,

generally, they reached the major population groups facing life-threatening risk. Where targeting of

beneficiaries took place in consultation with the community the most deserving were well reached.

Those agencies that worked more through government officials and local authorities, however,

covered most vulnerable as well as low vulnerability regions and families.

When looking at coverage in terms of people at risk, the evaluations of responses of SHO participants

- implemented directly or through local or international partners - showed that the flood responses

reached the most affected people both in the relief and in the recovery phase.xxxv Evidently, due to

the sheer scale of the flooding the organisations adopted a targeting mechanism to reach those who

were at greatest need.xxxvi The role of local village committees, at times especially set up under the

intervention, appeared to be of crucial importance in identifying the people mostly at risk.

The evaluations of responses of international umbrella organisations and their national branches,

which looked at the relief phase, were very critical of the coverage issue.xxxvii The evaluators spoke of

considerable variations in the vulnerability of the locations targeted and cases where aid mainly

reached people that were locally well positioned and/or aligned to political parties. Interference,

primarily at local levels from politicians, landlords or tribal leaders played a negative role in reaching

the most vulnerable people. Access issues in conflict affected areas, such as Khyber Pakhtunkhwa

(KPK) and parts of Balochistan proved another factor that influenced the coverage of assistance.

In terms of geographical coverage, most responses evaluated took place in provinces and districts

where the impact of the flooding was severe or extreme: Sindh with the highest number of people

affected (7,2 million) and largest area covered by flood, followed by Punjab (6 million affected

people) and Khyber Pakhtunkhwa (KPK, 3,8 million affected people). In Annex 3 a table can be found

which shows the districts in which the SHO participants supported interventions, either directly or

through local or international implementing partners or the national branch of the international

umbrella organisation.

4.5 Connectedness: Did the response ensure that activities of a short-term emergency nature are carried out in a context that takes long-term and interconnected problems into account?

Immediately after the floods, many people fled and lived dispersed in camps where emergency relief

was provided. After the first weeks (in KPK) or months (in Sindh) the camps had largely been

dismantled and the majority of the displaced people had returned home to their villages where relief


and recovery needs persisted. During the first phase of the emergency all the organisations focussed

on immediate lifesaving activities. Some of these relief activities, particularly in the WASH, health,

and in the NFI/ shelter sectors had a longer term effect (see below). There is not always a clear

demarcation between the relief and the recovery phase, however, over time and in accordance with

the changing needs of the affected population the activities of the SHO participants increasingly

focused on early recovery. With regard to the connectedness between relief and early recovery

activities, the evaluation reports demonstrated that most organisations made a smooth transition

from one phase to the other, although some delays in the implementation of essential recovery

activities were also reported. Most organisations purposely continued working in the same

communities, some even in the same sectors -particularly WASH and NFI/ Shelter - while others

started activities in new sectors -particularly livelihoods. Setting up community committees and other

CBOs, such as farmer groups, was generally seen as an important instrument to ensure sustainability

of the interventions and a smooth transition towards recovery and development, as will be

elaborated below.

Various activities carried out during the relief phase did create some long-term impact as we have

presented under section 4.2 Effectiveness and impact. To add some more examples, three

evaluations reported that activities in the WASH sectorxxxviii, such as hygiene promotion, had

improved personal hygiene practices in the long run. Beneficiaries - women in particular - also

stressed that the sanitation facilities installed had brought a positive change in the overall living

conditions of the communities, notably in the health and hygiene areas. One report clearly stated

that in the affected districts the situation in terms of safe drinking water and sanitation was poor

even before the floods occurred. Activities to improve these facilities, contributed to strengthening

of the resilience of the targeted communitiesxxxix.

The SHO participants used different approaches with regard to shelter. One organisation that

focussed on large scale relief provided affected communities with tarpaulin or tents for temporary

shelter as part of the NFI kit.xl Others were involved in the construction of more durable shelters out

of corrugated iron sheeting or local materials. Unfortunately, the programme using local building

materials encountered delays due to complex procurement procedures.xli It should be added that, as

one evaluator stated: “even the tents and tarpaulins - and other NFI items, such as tools, mosquito

nets, kitchen items - may last much longer than the 6 months life expectancy on paper because

people in poor communities are generally able to stretch the durability of these goods considerably

through careful use and repair”.xlii

The longer term effect of preventive and curative health care activities is evidenced by the wish of

communities that these services could continue. Where one SHO participant successfully managed to

increase governmental engagement for the sustainability of programme interventions another SHO

participant involved in the health sector experienced significant challenges in trying to ensure

sustainability of its interventions. The transfer of the health facilities developed by the SHO

participant to the Ministry of Health was not feasible due to structural organisational weaknesses of

those national structures. The organisation eventually secured continuity of its work by passing the

project on to another international medical organisationxliii

With regard to the connectedness between relief and early recovery the Inter-Agency RTE established

that most of the international responses focused on relief rather than on recovery activities and that


strategies related to recovery and rehabilitation were not timely and carefully planned for by most

clusters, because the clusters were to a large extent operating independently from the phases of the

operation. In addition, recovery had been dealt with from a cluster perspective rather than in a more

integrated fashion.

Most of the evaluation reports gave a more positive impression of the connectedness between the

two phases. Most organisations built their early recovery interventions on their relief activitiesxliv.

One report stated that the organisation wanted to move into the recovery phase with the same

communities, because the relief interventions had provided a solid foundation for the recovery

phase by cultivating a strong relationship between the organisation, its local partners and the

communities and by creating a high degree of trust.xlv The early recovery phase of another

organisation targeted some of the same areas where the relief activities had been conducted. The

evaluator continued stating that “the organisation will benefit from its prior knowledge about those

communities even though the high turnover of volunteer staff of the national organisation will

reduce some of this connectedness”.xlvi Another report stated that there was a time gap in moving

from relief towards the early recovery phase and that the delay proved especially critical for time-

bound activities like livelihoods, because delays in seed distribution caused problems in timely

plantation by communities and slowed down self-sufficiency in food productionxlvii.

Some SHO participants that initially focused on life-saving sectors moved almost (exclusively) to

livelihoods and reconstruction activities. One organisation, for instance, started to focus on providing

cash for agricultural inputs and shelter construction during the early recovery phase. The evaluator

assumed that the reconstruction activities would benefit from the tools given during the relief phase.

The same report also stated that the food aid given during the relief phase had given people more

physical strength and economic resources to engage in shelter construction and agricultural work.

Other organisations continued working in the same sectors but adjusted their activities. xlviii An

organisation that worked in the WASH sector, for instance, moved from the distribution of hygiene

kits towards activities which facilitated recovery, such as repair of irrigation and water infrastructure

at the household and community levels.xlix Another evaluation advised about the recovery phase

that, with the coming of winter and the start of a new cropping season, the emerging priority should

focus on winter NFIs, shelter and livelihood assistance, e.g. repair of irrigation channels, seeds,

fertilizer, water for crops and livestock. Other needs identified to support recovery were construction

and repair of water supply infrastructure and latrines. Furthermore, NFIs were still deemed relevant

because displaced people were returning and had the same basic needs for NFIs and shelter.l

Several evaluation reports praised the strategy of continuing to work in the same communities. One

evaluator stated that “the strong relationship developed with targeted communities during the relief

phase will provide a strong platform for humanitarian actors to build upon during the early recovery

phase and later for development interventions”.li Another report established that some

opportunities for early recovery and long term interventions were created by involving women from

the affected communities in relief activities, which gave them confidence and helped them organise

themselves.lii A third report stated that, for future sustainability and development, the organisation

had trained community groups at the village level and encouraged them to relate their development

needs with the relevant local government structures.liii Yet another evaluator expected that the large

participation of the beneficiaries in the rehabilitation works in the shelter and WASH sector would

result in a feeling of ownership, because the communities themselves will have to maintain the


systems.liv Finally, the report concerning a livelihood projectlv established that the organisation

developed a strong and innovative approach to recovery by utilising a one-time relief distribution to

create long-term development structures, such as farmer groups and a revolving CBO loan fund.

Accounts from beneficiaries and surveys indicated that both food security and the restoration of

livelihood options of the farmer households had significantly improved and that activities had been in

line with beneficiaries’ expressed recovery needs and priorities. Beneficiaries, male and female,

voiced an interest to pay back their loans. They, furthermore, saw the farmer groups as a means to

increase control over their lives and to lessen dependency on money lenders.

4.6 Coherence: Did the response take into account humanitarian and human rights considerations (assess security, developmental, trade and military policies, as well as humanitarian policies)?

The majority of the evaluation reports did not specifically look at coherence or observance of

fundamental humanitarian and human rights principles separate from checking adherence to the Red

Cross and NGO Code of Conduct. Four of the evaluations explicitly mentioned coherence, three of

which linked coherence to coordination and solely discussed whether relief efforts had been

appropriately and sufficiently coordinated. This may not be surprising for the following two reasons:

according to ALNAP ‘coherence’ is often confused with ‘coordination’lvi and ALNAP also states that

coherence may be less relevant for evaluating single-agency or single-project interventionslvii. The

issue of coordination and co-operation is discussed under section 4.10.

In our elaboration below we looked at the humanitarian and human-rights considerations, to which

the question on coherence refers. Basically , these are the first four fundamental humanitarian

principles of the Red Cross/NGO Code of Conduct (CoC), i.e. 1) humanity, meaning the centrality of

saving human lives and alleviating suffering wherever it is found; 2) impartiality, meaning the

implementation of actions solely on the basis of need, without discrimination between or within

affected populations; 3) neutrality, meaning that humanitarian action must not favour any side in an

armed conflict or other dispute where such action is carried out; and 4) independence, meaning the

autonomy of humanitarian objectives from the political, economic, military or other objectives that

any actor may hold with regard to areas where humanitarian action is being implemented.lviii As will

be set out in section 4.7, all evaluation reports confirmed a certain general level of adherence to the

CoC. However, the reports also mentioned some challenges that were encountered, particularly with

regard to impartiality and neutrality, but also to some extent with regard to humanity and

independence, as we elaborate below.

Regarding the humanitarian principle humanity, all the SHO participants provided life-saving

assistance, trying to make sure that the first basic needs of the affected households within their

reach were met with regard to physical safety, food, health, WASH, NFI and emergency shelter (the

relevance and effectiveness of this assistance is addressed in section 4.1 and 4.2 respectively). In one

case, the local implementing partners of an SHO participant were directly involved in the evacuation

of affected people to safer areas. They supported the local authorities and enabled the Pakistani

army to better fulfil their rescuing task by lending them their boats.lix

One evaluation report stated that due to the large scale of the disaster, and because of limited

presence and capacity, organisations had been confronted with challenges trying to deliver an

independent needs‐based response. Initially, the response was poorly prioritised, because “there


was an excessive rush to start without deciding on how and where to respond”.lx All evaluations

reported that the relief effort had been hampered by the destruction of infrastructure. Roads and

bridges had been washed away by the floods, which created problems to access the affected

communities. In some instances, people had to travel far to receive food and NFI assistance, because

implementing partners could not reach their villages. Therefore, several SHO participants supported

activities to rehabilitate physical infrastructure.lxi Some evaluators remarked that the relief response

was late in reaching people due to a combination of slow donor allocations, complex or overly-

stringent procurement policies and staffing issues.lxii

The evaluation reports give us the impression that most SHO participants and their implementing

partners made serious efforts to adhere to the principle of impartiality. However, this was a

challenge during the general confusion in the early stages of the large scale flood emergency and

within the cultural and political context of Pakistan. The latter relates to the issue of interference

from politicians, landlords or tribal leaders, which has been discussed under section 4.4.lxiii According

to the evaluations, most organisations carried out independent needs-assessments, involving the

affected communities.lxiv In how far this community involvement was representative, i.e. included

representatives from all different segments of the population (young, old, men, women, minorities,

people with disabilities, etc.) varied. The SHO participants which had strong longstanding partnership

relations with local organisations and provided their local partners with continuous guidance and

capacity support did particularly well in securing impartiality.lxv Yet, several evaluation reports

mention that vulnerable groups had not been reached.lxvi For more information on these groups we

refer to section 4.8.

Relief efforts in the provinces bordering Afghanistan were complicated by the prevalent political and

security situation, the presence of Taliban - both from Afghanistan and from Pakistan – and terrorism

threats. In KPK and parts of Balochistan, the impact of the flood was compounded by conflict and

insurgency; the situation in this part of the country is considered a complex emergencylxvii. The

security situation for international agencies in Pakistan and their ability to do their work unhindered

worsened after 1st of May 2011 when the US attacked the residence of Osama Bin-Laden in

Abbottabad and subsequently killed him. This complex emergency situation in KPK and parts of

Baluchistan created extra challenges to international aid organisations, including SHO participants, to

adhere to the fundamental humanitarian principles of neutrality. One SHO participant experienced

operational difficulties as a result of the ensuing security situation: in KPK international staff needed

security clearance from the government, a Non Objection Certificatelxviii, to permit them to travel to

Kohat and Shangla districts; they were not allowed to stay overnight in these districts.lxix

Referring to civil-military co-operation, the Inter-Agency RTE explicitly mentioned that the

humanitarian space had been compromised, especially in KPK and Balochistan. Where civilian assets

were insufficient and physical access constrained, military assets were sometimes used during the

rescue operation and the early stages of the emergency. The RTE stated that “civil-military

cooperation in emergencies is a sensitive matter” and recalled that the UN had previously

determined that military assets should not be used in areas such as Balochistan and KPK, where the

government or regional actors are party to the conflict. Although a set of country specific guidelines

for civil-military interventions had been adapted in early 2010, donors pressurized aid agencies to

use a NATO air bridge during the relief phase. The evaluators praised the UN Humanitarian

Coordinator, who stood up to this pressure.lxx


One evaluation report mentioned that the agency undertook advocacy through the national NGO

coordinating body Pakistan Humanitarian Forum (PHF) on several issues, including concerns related

to use of humanitarian space for counter-terrorism by the CIA.lxxi This indicates that the organisations

that implemented the programme had encountered challenges with regard to maintaining their

independence. In other evaluation reports no reference was found to attempts by foreign donors to

interfere with the humanitarian aid efforts.

4.7 Standards: Did the response adhere to the Code of Conduct and the Sphere Standards?

All the evaluation reports referred to the Code of Conduct for the International Red Cross and Red

Crescent Movement and Non-Governmental Organisations in Disaster Relief (CoC)lxxii and/or Sphere

Standardslxxiii, however, not all discussed or substantiated their conclusions with regard to adherence

to these standards. Perhaps this is due to the fact that it is well known that all SHO participants have

signed the CoC and committed themselves to adhere to the Sphere Standards. In some cases issues

relevant to adherence to the standards were included in reports’ sections on, for instance,

accountability, coordination and partnership.

The reports give us the impression that overall adherence to CoC and the Sphere Core Standards was

adequate. Due to the enormous scale of the disaster, adherence to the Sphere Minimum Standards

for the different sectors was posing some challenges, particularly those with regard to shelter and

NFI. All SHO participants appeared to have made serious efforts to fulfil the standards. Several of

them provided specific training on the standards as part of their quality and accountability training

for staff and partners and some put posters with core standards and principles up in their offices.lxxiv

Yet, despite having been briefed about them, some of the local partner organisations were not

familiar with widely used international standards and principles such as Sphere and CoC.lxxv

Code of Conduct (CoC)

Interestingly, one of the evaluation reports stated that the strong adherence to most of the

principles in the COC was one of the strengths of the response, thereby implicitly confirming the

importance of these principleslxxvi. The first four principles of the CoC, the humanitarian imperative,

impartiality, neutrality and independence have been discussed in section 4.6 on Coherence. Most

reports specifically paid attention to the remaining principles, particularly, building disaster response

on local capacities, participation of affected communities, reducing future vulnerabilities and

accountability, predominantly, towards beneficiaries. To a lesser extent consideration is given to

respecting local culture and custom. Obviously, some of these issues are strongly interrelated;

however we try to present them separately.

Where it comes to building on local capacities (i.e. human and material resources), the evaluation

reports show that most SHO participants worked with local partner organisations or their national

branch in the case of an umbrella organisation to implement their response programmes.lxxvii Most

local partners recruited their staff locally.lxxviii Two of the SHO participants included in this evaluation

implemented (parts of) their programmes directly. In one of these cases, the evaluation report

mentioned that local staff was employed and materials were purchased locally, but no efforts were

made to identify local NGOs having the capacity to assist in the implementation of the projects.lxxix

Most evaluations reported that NFI and building materials were purchased from local sources. In


some instances this caused delays, particularly, because the local market could not instantly fulfil the

enormous demand for building materials. Several reports also mentioned the benefits of local

procurement, such as people already being familiar with certain types of water filters. lxxx

With regard to participation of affected communities, the evaluations established different levels of

involvement of local communities in planning, implementation and monitoring of response. Some of

the implementing agencies/ partners ensured representation of people from all segments of local

communities in all stages of the project management cycle whereas others only involved the

leadership in some stages (see also section 4.8). Concerning the first category, one report found that

the participation of affected people, including women, in the identification and selection of

beneficiaries and the involvement of beneficiaries in distribution and purchasing processes

contributed to creating trust among the communities.lxxxi Another report, however, concluded that

people’s participation was low, information provision insufficient and complaint mechanisms absent

with the result that the agency’s programmes caused conflicts and grievances in 8 of the 18 places

visited.lxxxii

The evaluation reports found that implementing agencies/ partners’ ability to contribute to reducing

future vulnerabilities differed. To some extent this depended on whether they were primarily

involved in emergency relief or also focussed on recovery, but it also depended on the size of their

operation. One report established that insufficient attention could be paid to developing the

capacities of communities due to time constraints and the scale of operation during the relief

phase.lxxxiii Particularly, organisations implementing (early) recovery programmes appeared to

contribute to some extent to reinforcing disaster resilience of affected communities. This will be

discussed in further detail in section 4.9 on Strengthening of responses.

The evaluation reports showed remarkable differences between implementing agencies/ partners in

their ability to translate the principle of accountability towards beneficiaries to local realities. Most

organisations tried to address accountability concerns, however with varying success as has already

been outlined in the paragraph on accountability in section 4.1 Relevance/Appropriateness. One

report established that although accountability concerns had been addressed to some extent, there

were still large percentages of people not consulted or informed about the planned programme

support or knowing the complaint mechanisms, because local partners often did not understand the

overall objectives and values of accountability principles and took it as a routine activity. This

suggests, according to the evaluators, the need for far greater investment in developing a local

approach to accountability.lxxxiv Prove that this can be done is given by another report stating that the

SHO organisation had a clear position on ensuring the implementation of standards and

accountability mechanisms in humanitarian assistance, which led local partners to be proactive in

designing and implementing accountability towards beneficiaries and adapting strategies to create

awareness among the affected population about the relief packages, and how and where to lodge

complaints.lxxxv

The concept of accountability towards beneficiaries is further elaborated on and specified in the

Humanitarian Accountability Partnership Standard in Accountability and Quality Management (HAP).

Four evaluations specifically refer to the HAP Standard and mention is made of implementing

partners that acquired HAP certification.


Reference to respect for local culture and customs is particularly made in connection to the shelter

and NFI and the health sector, and is mainly related to the sheltered position of women. In the

shelter sector, the positioning of latrines was an important issue as was the placing of a kitchen and

the inclusion of certain items in the NFI kits; and in the health sector special ‘lady visitors’ were

trained and employed. This is described in detail in section 4.8 Women and Vulnerable groups.

Sphere Standards

Four evaluation reports discussed in more detail how adherence to Sphere was established, whereas

the other reports only mention that the response provided was in accordance with Sphere

Standards. One evaluation checked all the sectors against the Core Standards but did not reflect on

the Minimum Standards per sector.lxxxvi The other three evaluations also focused on adherence to the

Minimum Standards. Two of these stated that information was not always readily available but they

had made serious efforts to check on observance of the Standards for the different sectors.lxxxvii All

three reports mentioned that it appeared difficult to put the standards into practice. This was partly,

but not exclusively, due to the enormous scale of the disaster as a result of which resources were

spread thin. In addition, there were other factors that constituted difficulties in meeting the

Minimum Standards. One SHO participant, for example, made a serious effort to implement the

Standard requiring that houses should be disaster resilient by taking measures to reduce erosion on

the shelter sites and making efforts to build new shelters on alternative sites to reduce the risks of

future flooding. However, to build real safely one would have to look for completely new locations,

which was not easy because beneficiaries were attached to their properties and obtaining new land

for construction was a real challenge.lxxxviii

The Inter-Agency RTE concluded that the experiences from the Pakistani floods clearly showed the

need to adapt the response to the context and conditions on the ground, and have international

standards serve as guidelines for what should be achieved. Some clusters managed to agree on

adapted standards using Sphere as a guiding objective. One example is shelter. Many NGOs reported

difficulties in implementing the standards due to their relatively high costs and consequently decided

to go for cheaper and less permanent solutions, often inadequately prepared for future floods.

Eventually, the shelter cluster applied a one‐room standard policy for permanent shelter. lxxxix The

Inter-Agency RTE also revealed that the situation was further complicated because the Government

of Pakistan decided on standards other than those agreed to internationally and concluded that in a

country like Pakistan standards, principles and guidelines should be negotiated in due time with

relevant government entities and institutions, and not only be dealt with once the emergency

strikes.xc


4.8 Cross-cutting themes: Gender and Vulnerable groups xci

Gender: Was gender considered in the agencies’ emergency assessments? Did relief provision include special components for women, men, girls and boys and, if so, were these systematically monitored?

Vulnerable groups: Were the special needs of acute vulnerable groups (e.g. children/ elderly/people with disabilities, etc.) considered in the agencies’ emergency assessments and were they consulted in the same way as other groups? Did relief provision include special components for them and if so were these appropriate and systematically monitored?

Overall, the evaluated programmes paid attention to most vulnerable groups, but in very different

degrees. It concerned women, more particularly widows and other female-headed households, and

children, and to a much smaller extent, elderly and people with disabilities, while even less

consideration was given to religious, ethnic and tribal minorities. None of the interventions evaluated

included special components for women or specific vulnerable groups, although the health

programmes mostly targeted women. Needs of women and other vulnerable groups were usually

identified during the needs assessments. The assessments and, consequently, interventions carried

out by the national branches of international umbrella organisations showed considerable gaps in

terms of participation of vulnerable groups and addressing their needs, apart from some good

exceptions. Many programmes implemented by local or international partners of SHO participants

and by SHO participants themselves, however, integrated women´s concerns in the different stages of

the interventions and made efforts to increase their participation in consultations, activities and

decision making. In a few programmes a systematic approach was used to that effect. In the

evaluation reports of responses implemented by (partners of) SHO participants much less attention is

paid to other vulnerable groups than women. It is not entirely clear to us whether this limited

information is due to the lack of attention of the evaluators or to the programme itself. Most

evaluators reported without providing much detail that vulnerable groups were considered in the

response, more specifically in needs assessments. Children, people with disabilities and elderly people

were mostly mentioned. As regards monitoring, the evaluators found that for all implementing

agencies and partners monitoring from a gender or vulnerability perspective left to be desired.

Although in different degrees, all evaluations paid attention to vulnerable groups being the most

affected amongst those who suffered from the floods. In the reports, the term ‘vulnerable’ usually

refers to women, more particularly widows and other female-headed households, and children,

elderly people and people with disabilities, as well as religious, ethnic and tribal minorities. These

groups were considered more susceptible due to a lack of social services, and, especially in the case

of (young) women, deprivation of basic human rights due to cultural and social norms, and the threat

of exploitation by traffickers. As one evaluation pointed out, “the majority of the women who

reached to the camps, or lived in open areas particularly in Sindh and Punjab, had never experienced

displacement in their lives. Similarly, in districts of KPK, for many women it was the first time they left

their homes and had never been exposed to public life. They had to quickly cope with the situation

either living in the camps with other communities, castes and social groups or live in isolation to bear

the pain of disaster.”xcii


Because we noticed a difference in approach between interventions implemented by local or

international partners of SHO participants and by SHO participants themselves and those

implemented by the national branches of international umbrella organisations we address the

questions about needs assessment, relief provision and monitoring for each of the categories.

The two evaluations of the responses of international umbrella organisationsxciii concluded that the

interventions were generally ‘gender blind’ on most fronts. The evaluators observed insufficient

understanding of or attention to cultural and contextual factors which prevented women and other

vulnerable groups from accessing aid on equitable terms. Many people from ethnic and tribal

minorities and most vulnerable individuals and groups, such as widows or other female‐headed

households, were deprived from any assistance at all.xciv The evaluators traced this insufficiency from

the earliest stages of the responses, the needs assessments. Although efforts were made to identify

the needs of the most vulnerable people, the needs assessments showed general weaknesses which

affected the inclusion of vulnerable groups. In general, people were not consulted about their needs

at village level, let alone the most vulnerable, among which women. Due to political pressures the

identification of vulnerable groups was jeopardized (as mentioned in section 4.4 and 4.6) and due to

the absence of systematic registration or a verification process the targeting was weak, particularly,

of the most vulnerable groups. Yet another example refers to the area of nutrition, most important

for vulnerable people, especially children and women, which were not included in the needs

assessment. Some organisations though used assessment and funding proposal screening tools that

ensured attention to gender issues, which the evaluators assessed as ”a step in the right direction”.

Concerning interventions of the national branches of international umbrella organisations, the two

evaluations pointed at underlying issues related to women’s rights that were brought to the surface

during the relief provision. To give a few examples, some implementing agencies took corrective

measures to address women’s access to assistance, in the area of food distribution (e.g. issuing of

women ID cards), sanitation (separate latrines), and health (recruitment of female health workers to

reach out to pregnant and lactating mothers. These measures improved access to a part but not all

women who most needed the assistance.

Taking corrective measures relates to monitoring which is basically meant to identify what should be

done to keep the response on track. In general, the evaluators noticed a lack of consolidated

monitoring data. Agencies did not make a comprehensive assessment of how vulnerability and

gender issues had been addressed, which is required to ensure more gender‐sensitive responses and

adequately meeting needs of specific vulnerable groups. The evaluators that dealt with programmes

of local or international partners of SHO participants were more positive about monitoring in general.

Systems had been set up, which according to some evaluators were efficient to identify progress.

Field visits by staff were regarded the most useful tools to identify what was happening on the

ground. Most sections on monitoring in the evaluation reports, however, did not include information

about vulnerable groups, including women. The sections pointed at constraints in monitoring, such

as lack of staff’s time for such tasks during the busy project activities and gaps in capacities of local

implementing partners, more especially in the area of data analysis, documentation and reporting.

One of the few evaluations that paid attention to vulnerable groups stated that due to these gaps it

was difficult to analyse overall how many men and women, children, elderly people, people with

disabilities, widows, and minority groups benefited from the project activities.xcv This is in line with

the comment of the Inter-Agency RTE: “The response plan as such was not monitored which left


crucial questions widely unanswered, such as ‘Where did the aid go? Who benefitted and who did

not? Was it useful and did it cover needs (sometimes identified)?”xcvi

The responses implemented by local or international partners of SHO participants and by SHO

participants themselves showed a slightly different picture. Four evaluation reportsxcvii speak of

concerted efforts to include gender issues in the programme. These programmes were generally

characterized by a highly participatory approach throughout all stages of the response. One of these

programmes specifically focused on women, in the others gender was integrated in the different

stages of the programme. Three implementing partners (evaluated together) applied a gender

mainstreaming approach but, in practice, their programmes emphasized activities specifically for

women to a much lesser extent. Consequently, women’s participation lagged behind that of men.

One evaluation found that the project proposal lacked a clear gender dimension but that in the

course of the implementation measures were taken to get more women on board.xcviii

Six evaluation reports pointed out that the partners of SHO participants had taken specific actions

during the needs assessments to make the voices of women heard while identifying urgent and

practical needs for the immediate relief.xcix Particularly due to cultural restrictions, the participation

of women in such assessments varied by location. More successful efforts were characterized by

mobilisation of the existing community structures in the assessments, combined with awareness-

raising about the need of special care for vulnerable groups; by conducting structured camp

assessment; by engagement of separate male and female staff to reach both men and women; and

by including gender criteria for potential beneficiaries. As a consequence, interventions took care of

women’s needs such as inclusion of sanitary items in the hygiene kits, health issues (for example,

hygiene promotion, delivery kits, labour room equipment), shelter, the placement of hand pumps at

a distance convenient for women, tailored NFI distribution, separate latrine and bathing spaces for

women to maintain privacy of women, laundry spaces, cash for choice, and light tasks for widows

against the same payment as other workers. One evaluation found that “women were encouraged to

collect the relief items personally, also to ensure that widows and female headed households

received the support directly. As a result, many women came to the public places and distributions

points, got the opportunity to interact with diverse people and had access to information”’c The

same evaluation included some examples of overlooked women’s needs, such as in the shelter and

livelihoods programmes. Although all the women interviewed by the evaluators highly appreciated

the shelters as a primary need, they complained that the shelters did not include a kitchen and that

the latrine was not attached to the shelter, which raised the question of their protection and privacy.

In one of the livelihoods components women received vegetable seeds but it was found that

especially widows and the needy women were in high need of additional support in terms of

livelihood.

Due to the mobilization of women in committees (camp management committees, relief

committees, and so on) or the formation of special women’s groups the women remained active

participants of the programme interventions, also in the cases when they returned to their homes

from the camps. For many women such separate women’s committees were the first time to sit

together and discuss their issues.

Two evaluations explicitly concluded that women felt empowered due to their participation in

different project activities and their involvement in awareness-raising sessions and decision making.ci


Women clearly appeared to benefit from their participation in programme activities. As an example

we quote one evaluation, “Women in particular underlined that this awareness-raising and the

building of sanitation facilities had resulted in better health and hygiene conditions and lower rates

of diseases among children in the aftermath of the floods. …Women stated that they had benefited

the most from the health points set up by the organisation and requested it to establish prenatal and

postnatal health care facilities. Women stressed that, before the floods, they had to travel 15 to 40

km (5-10 hours) to access health care and that cost them at least 1500 PKR (18 USD) – in several

cases, beneficiaries did not have a health facility in their villages at all in the pre-flood period.”cii

Another example shows the opposite. The evaluation report stated that “The larger components of

the overall livelihoods programmes focused on agriculture and cash for work which by default

excluded women as these are generally men’s vocations in Pakistan. Special programmes targeting

women’s activities such as quilt-making, handicraft, goat and chicken rearing and home gardening

would have been more useful in empowering large numbers of women, especially those which are

sensitive to the norms of their community.”ciii

So far about women as vulnerable groups. Much less attention is paid to other vulnerable groups in

the evaluation reports of responses implemented by (partners of) SHO participants. Most evaluators

reported without providing much detail that other vulnerable groups were considered in the

response, more especially in needs assessments. Children, people with disabilities and elderly people

were mostly mentioned. It is not entirely clear to us whether this limited information is due to the

lack of attention of the evaluators or to the programmes themselves. A few evaluation reports only

elaborated a bit on the issue. Two evaluators felt that the specific needs of vulnerable groups had

been insufficiently addressed, while one of them noticed that staff had different understandings

about the concept vulnerability and its operationalization.civ Field visits by an evaluation team

showed that religious minorities in Sindh also benefited from the assistance, while another

evaluation team concluded that lower quality services had been provided to minorities.cv In WASH

related activities the first mentioned team found that concerns of elderly people and people with

disabilities had been overlooked. Moreover, there were issues of protection of children/ missing

children in camps, issues of girls’ education as girls’ schools remained closed for a long time which

had not been considered.

4.9 Strengthening of responses: Did the response of the SHO agencies strengthen and complement the response of local organisations and coping mechanisms, or hinder them?

Pakistan is a disaster hotspot exposed to recurrent hydrological and meteorological events; every

year when the monsoon season starts, the next disaster is ‘just around the corner’. The risk for a new

emergency situation is high due to the concentration of poor people living in hazardous areas along

river shores. Therefore, the strengthening of the capacities of local actors, Government of Pakistan,

local NGOs as well as local communities in the affected areas, to respond more adequately to disaster

is vital.

The evaluation reports pointed out that the civilian disaster management capacity of the Pakistani

government, particularly at the district and provincial level was insufficient. In general, the response

of SHO participants has strengthened local NGO capacities, which would enable them to respond

more adequately to future emergencies. Most SHO participants have partnerships with local


organisations. They continuously monitored and strengthened the capacity of their partners, using,

for example, on the job training and regular field visits. The response of most SHO participants also

strengthened the resilience of local communities and local coping mechanisms, particularly, because

the communities assisted became better organised. In some cases the assistance also had a positive

impact on the level of participation of women in decision making at the community level. Health,

hygiene, shelter, water and sanitation facilities and/or livelihood were rebuilt and, in several cases,

improved in comparison with the situation before the floods. People were able to rebuild their lives

better. But there is always room for improvement as some evaluators concluded. They indicated that

more specific activities directed at disaster preparedness and disaster response planning were needed

as well as more robust in-depth needs assessments with a stronger focus on the coping strategies

employed by the flood affected population.

The reports painted a consistent picture of insufficient capacity of the Pakistani government to deal

with the 2010 floods. Despite the existence of a national disaster management agency (NDMA) with

subdivisions at the provincial (PDMA) and district (DDMA) level and national and sometimes also

provincial plans, the capacity of the government to deal effectively with the emergency was

insufficient. The Inter-Agency RTE stated that the civilian disaster management capacity at provincial

and district level was low and the links between national and provincial disaster management were

generally weak. Disaster management had been reactive, preparedness deficient and investments in

disaster risk reduction (DRR) inadequate. Disaster risk management (DRM) is one of the five strategic

priorities of the UN in Pakistan; however, contingency planning with a clear division of labour among

the government and the international community was generally missing.cvi Other evaluation reports

did not mention any conscious involvement of SHO participants – probably with the exception of

UNICEF – in strengthening of DRM capacities of the Government of Pakistan. However one

evaluation recommended that local partners should help build DRM capacities of local authorities.cvii

Yet, most SHO participants were –with varying success- involved in strengthening the DRM capacity

of local NGOs. The majority maintained strong partnerships with local organisations; some had

already been working together in development programmes or in other emergencies, in particular in

KPK. One evaluator highly valued the SHO participant’s strategic engagement with local partner

organisations in preparing contingency plans and other disaster risk reduction (DRR) activities

including disaster preparedness. Due to the involvement of the implementing partner in DRR and

preparedness activities and stockpiling, the staff was able to quickly respond to the flood

emergency.cviii Another evaluator found it commendable that the SHO participant was able to scale-

up rapidly because the organisation could rely on the capacity of local partners. But it was added that

there was a need for the organisation to continue undertaking systematic capacity assessments of

partners and developing formal capacity-building strategies for them to enhance their long-term and

strategic capacity.cix Another SHO participant implemented all the projects in cooperation with local

partners which initially lacked experience in emergency response programmes, especially on a large

scale as required after the floods of 2010. It developed an ‘on the job training’ system providing close

supervision and gradually assigning more responsibility to the partners.cx These local partners were

involved in all stages of the project cycle, including needs assessment, implementation and

monitoring, and were directly engaged in disaster risk reduction.

One SHO participant as a consequence of its international organisational structure worked with a

large nationwide local organisation, however poorly organised. The capacity of the local organisation,


especially at the district levels, was low due to a shortage of paid staff with a strong humanitarian

background and high turnover among volunteers.cxi The poor performance of this local partner had a

significant effect on the international organisation’s ability to contribute effectively to disaster risk

management.

Moreover, evaluations pointed out that preparedness requires specific activities directed at disaster

preparedness and disaster response planning to strengthen community resilience for future

disasters.cxii Several evaluators assumed that the sustainability of the activities - in the recovery

phase and where possible in the relief phase - will strengthen local coping mechanisms and most SHO

participants made serious attempts and managed to provide assistance with some level of

sustainability, as was also outlined in section 4.5 Connectedness.cxiii For instance, evaluators

expressed that they expected an enhancing effect on coping mechanisms and increased abilities to

deal with disasters in the future from reinstating basic sanitation, water and hygiene facilities and

constructing flood resistant houses and water systems.cxiv Though, some evaluators felt that not

enough was done to support local coping strategies. cxv One report stated that - in future - the

organisation should implement more robust in-depth needs assessments with a stronger focus on

the coping strategies employed by the flood affected population.cxvi The organisation was also

advised to focus more on livelihoods activities during the emergency phase - with a special focus on

women - as this was expected to help in making communities self-reliant.

One evaluator asserted that good preparedness measures, including involving communities in

disaster preparedness planning, and focus on sustainability of programs is a must in countries like

Pakistan that experience an emergency almost every year.cxvii As evidenced by several evaluations,

this well sums up what is needed to enhance the coping mechanisms of affected populations and

strengthened community resilience for future disasters. Several evaluations concluded that the

positive effect of community mobilization on abilities to better cope with future disasters depended

on the level of participation of different segments, particularly women and vulnerable groups.

Evaluators found that community mobilization also had a positive effect on the intensity and

effectiveness of capacity building efforts.cxviii In this respect some SHO participants were more

successful than others. A positive example of creating enhanced capacity of local and affected people

is the following: affected people who worked as volunteers and gained experience of humanitarian

work could as a result be considered human assets to actively participate in early recovery and long

term humanitarian and development work in their respective areas.cxix

4.10 Cooperation: What was the level of co-operation in the field between the SHO agencies? Could more have been done to help improve the effectiveness of SHO agencies’ responses in terms of co-ordination, joint-logistics, communications packages, and information flows between the key relief players?

None of the evaluators specifically looked into cooperation between the SHO organisations in the

field, because it was not requested by their respective Terms of Reference. However, all evaluation

reports mentioned efforts by the (implementing partners of) SHO participants to coordinate their

relief activities with other stakeholders. During the 2010 Pakistan floods emergency multiple

coordination mechanisms existed at the national, provincial and district level and most (implementing


partners of) SHO participants took part in these institutionalised coordination meetings. The main

coordination forum was the UN cluster system, where coordination was organised by technical sector

(WASH, food, NFI/shelter, health, logistics), and which was established in collaboration with the

National Disaster Management Authority (NDMA) at the national, provincial and the district level.

Most of the larger organisations present in the country (SHO or implementing partners) participated

in the cluster meetings at the national level of the sectors relevant to them. However, others - both

local and international organisations - prioritised participation in coordination meetings at the district

or provincial level. Several evaluation reports also mentioned participation by international NGOs

(SHO or implementing partners) in the Pakistan Humanitarian Forum (PHF), a coordinating body that

coordinated and advocated with the national government to create a conducive environment for

humanitarian actors. Several (implementing partners of) SHO participants - at some level – also

collaborated with others and, particularly, within international umbrella networks of SHO participants

belonging to the same network specific efforts were made to promote voluntary cooperation.

The Terms of Reference that guided the evaluations did not request to attend to the issue of

cooperation between the SHO participants in the field and none of the evaluations paid attention to

the issue. Most SHO participants worked through their own international umbrella and/or network

structures and had their own international and/or local implementing partners. Priorities for

intervention areas - both geographical and sectoral - were set, based on criteria such as most urgent

needs, prior experience in the area, presence of (local) implementing partners, etc. (see section 4.4).

However, all evaluation reports mentioned efforts by (the implementing partners of) SHO

organisations to coordinate their interventions with other relevant stakeholders to strengthen the

effectiveness, efficiency, etc. of their programmes. One evaluator specified that the organisation

followed a three-pronged strategy and used external coordination to exchange information for

programme purposes, to promote relevant principles and standards and to access operational

support for example for logistics and government Non Objection Certificate issues.cxx

Several evaluators mentioned the importance of coordination with government departments and

authorities in order to ensure transparency and credibility and to foster sustainability.cxxi Local

partner organisations engaged with the government at the district level for effective service delivery.

Some organisations shared their expertise and experience with the government and in the early

stages of the emergency some also shared their transportation resources.cxxii

All reports mention that (the implementing partners of) SHO participants used the cluster meetings

for coordination with other NGOs, UN and government institutions. The various organisations,

however, employed different strategies depending on their capacity to participate and on their own

priorities. Local partners, generally, only participated at the district level, whereas at the national

level mainly international organisations took part.

Yet, several evaluators established that the government was unable to live up to its coordinating role

and that this created hindrances that affected relief efforts of the SHO participants.cxxiii Some

evaluation reports stated that coordination capacities of the government authorities, particularly at

the district and lower levels, were generally weak. They also found that the internal coordination of

the Disaster Management Authorities with line ministries was sometimes problematic. The Inter-

Agency RTE specifically focused on the UN’s relationship with Government of Pakistan counterparts -

who together were responsible for the cluster coordination - and found that this partnership was


characterised by diverging views on how to approach the emergency situation as well as on

leadership and accountability. Nevertheless, co‐leadership both at national and district level was

considered positive.

With regard to the functioning of the cluster system, the evaluation reports generally mentioned that

the cluster meetings were useful for the exchange of information between the different stakeholders

and to avoid duplications. One evaluation stated that the cluster system had been praised for its

important role in gathering data and providing evidence on the severity of the humanitarian needs.

However, there was also some criticism, expressed as follows: cluster meetings at the district level

were unable to keep a focused agenda; the clusters lacked strong leadership and strategic

prioritisation; cluster managers often lacked capacity or ignored their responsibilities; cluster leads

had a tendency to favour their own agencies’ interests rather than the sector priorities; the clusters

were operating independently from contextual realities; there were too many meetings and the

meetings were time consuming; the information flow was massive but not consolidated and

therefore of limited strategic use.cxxiv

Various evaluation reports mentioned that SHO participants and their implementing partners

engaged in advocacy efforts to create a conducive environment for humanitarian actors and improve

government and UN performance.cxxv Several evaluators referred to the Pakistan Humanitarian

Forum (PHF) in this respect. The PHF organised coordinated advocacy at the national level, taking

NGO concerns, for example regarding non-adherence to principles and standards, the violation of

humanitarian space, access and security issues, to the responsible government or UN entities.

Alongside the cluster system, other coordination forums also came into being like, for instance, the

Food Security Alliance Pakistan. Some evaluations also reported coordination and collaboration

between different NGOs on an ad hoc basis. cxxvi One of these reports mentioned that many NGOs

recognised the trend that more coordination among NGOs was required by donors and expected by

host governments. During the Pakistan floods emergency NGOs created voluntary coordination

mechanisms with other NGOs which were initially completely unrelated to them. Another report

stated that, to avoid duplication of efforts and resources, the SHO participant and its implementing

partners specially reached out to all organisations working in its target areas to build a relationship,

even if they were not participating in the clusters or other coordination mechanisms.

Also, SHO participants that belonged to the same international network cooperated more closely

with each other. Several evaluators commented on the coordination and collaboration within

international networks.cxxvii One evaluation report said that the memorandum of understanding

signed by the organisations within the international network encouraged voluntary coordination and

cooperation among the network members. During the Pakistan floods emergency, the network

worked well as an information exchange, security coordination and fundraising mechanism. One

member organisation, with well-trained and dedicated security staff, had the lead and coordinated

security issues of the other members of the international network present in Pakistan. Another

evaluator reported that the SHO participant took part in the meetings of the international network in

Pakistan. The evaluator established that in the early stages of the programme coordination at the

field level was lacking. This because competition and claiming of rights on intervention areas seemed

overriding concerns as one organisation opposed the shelter programme of the other fearing

duplication. Later on, an agreement was reached and the organisations agreed on a division of


labour. This example seems to illustrate clearly the statement of another evaluator that coordination

had obvious benefits, for instance, of reducing the chaos created by dozens of organisations working

haphazardly during emergencies. But, at the same time, there was a high degree of reluctance and

inertia created against coordination by the weight of separate line managements.cxxviii

4.11 Lessons Learnt: To what extent did responses reflect lessons learnt from previous similar disasters and more specifically from the SHO humanitarian aid action in East Asia in 2005?

With the last issue ‘Lessons Learnt’, we arrive at what could be the most interesting part of the

summary of conclusions. The question ‘Did, according to the evaluators, the SHO participants learn

from previous disasters’ refers to openness to learning. We used a proxy approach to answer the

question since it would have been an impossible task to assess the variety of responses of different

SHO participants against the numerous lessons learnt from previous similar disasters. First, we

screened the nine evaluation reports, including their Terms of Reference, on references made to

lessons learnt from previous disasters. Second, we compared the conclusions from the evaluation of

the SHO South Asia Earthquake action in 2005cxxix with the conclusions and, where necessary, the

findings in the different evaluation reports of the 2010 floods in Pakistan. We looked at the

conclusions of the meta-evaluation 2005 because the report did not include any recommendations or

lessons learnt. This brings us to the following observations. Except for two evaluations, the reports did

not include attention to lessons learnt from previous similar disasters. In a third case, the ToR

requested the evaluators to look into the issue, but they did not attend to it. This observation does not

mean, however, that the responses to the Pakistan floods did not take account such lessons learnt.

Comparing the (four) conclusions of the meta-evaluation of the 2005 SHO South Asia Earthquake

action with those of the 2010 floods evaluations we noticed some improvements in the areas of the

cluster approach, women’s participation and standards. Based on the reports considered, our overall

conclusion is that lessons learnt are poorly drawn which complicates a proper learning process on

emergency responses.

Only two evaluation reports referred explicitly to lessons learnt from previous disasters.cxxx Referring

to the question in one of the evaluations ‘To what extent were relevant recommendations from the

Review of the Pakistan Earthquake Operation: 2005 -2008 implemented’ the evaluator found that

“Not even one staff interviewed within Pakistan expressed familiarity with the recommendations of

the Earthquake 2005 evaluation report. Thus, overall, there has been no explicit attempt to keep

those recommendations in mind during the relief phase. Many of the lessons included in that report

also pertained to the early recovery phase: According to […] staff, they have been incorporated in the

early recovery framework, and […] the teams did keep these lessons in mind while undertaking their

assessments even though there is no official record of that. The report was also provided to deployed

staff and referenced by staff in the development of overall strategy”. Additionally, the evaluator

found that “there is an absence of formal and effective tools for incorporating past lessons and

documenting current lessons, except in the case of the WASH sector”. This made the evaluator

recommend that better systems for lessons incorporation and documentation should be

developed.cxxxi The Inter-Agency RTE found that “In Pakistan the humanitarian community did not

take stock of lessons learnt from prior evaluations (i.e. Government of Pakistan and a pervious Inter-

Agency RTE), including systematic follow up on previous recommendations”. The evaluators


concluded that “A more systemic follow up on previous recommendations is needed as it would help

improve the effectiveness and efficiency of the humanitarian system’s response to future disasters,

both within Pakistan and globally” and, consequently, drew some specific recommendations on how

to address the matter.

The meta-evaluation of the 2005 SHO South Asia Earthquake action does not provide lessons learnt

but includes four conclusions on general issues.cxxxii None of the nine evaluation reports including

their Terms of Reference refer to this document. The evaluations reports of the responses of the

2010 floods indicate that three general issues are better addressed, although improvements may still

be welcome as set out in the respective sections. The conclusions of the meta-evaluation read as

follows:

The participation of SHO-funded agencies in the IASC Cluster meeting at field level still needs to be

improved.

We addressed this issue in section 4.9 on Cooperation. Compared to 2005, the cluster approach

was much more common in 2010.

In many projects, it was difficult to overcome cultural constraints like very limited room for

women’s participation and domination of the committees by local elite groups. Overall, most of

the projects have paid surprisingly little attention to gender issues.

Section 4.8 above showed that more attention is paid to women participation in the responses.

As Sphere and Code of Conduct issues were mentioned in a minority of the evaluation reports only

(and if mentioned in many cases with very little detail), the impression exists that these important

topics did not receive the attention that is required for SHO-funded projects.

Section 4.7 proved that all SHO organisations paid attention to CoC and the Sphere Core

Standards with less attention to the Sphere Minimum Standards for the different sectors.

The establishment of different Village Committees for each project was an appropriate approach

for facilitating regular contact with the local population. However, it often resulted in a sectoral

approach which does not match with the overall strategy promoted by IASC and other agencies

focusing on integrated planning and implementation. The Village Committee approach also does

not contribute to sustainability as in most cases the committees are dissolved at the end of the

project period.

Since the evaluation reports of the 2010 floods did not address the issue in the way stated we

refrain from commenting.


5. Concluding remarks

In the chapter above we have presented the answers to a number of key questions concerning

twelve issues which SHO considered relevant for synthesizing and internal learning. The information

underlying the answers was taken from the conclusions of nine evaluations commissioned by six SHO

members who responded to the emergency situation in Pakistan caused by the 2010 floods. In the

next phase of the meta-evaluation it is foreseen that a thorough discussion among SHO members will

take place with the view of drawing lessons to improve future responses. We hope that the

information presented in this report will be a proper and inspiring input to the learning process of

the SHO and that the SHO members will draw recommendations to improve efforts in the emergency

sector.

To facilitate the learning process we summarize some main points that emerge from the different

key issues addressed.

1. The evaluators who conducted the nine evaluations concluded for both the relief and recovery

phase that the overall flood response was relevant and appropriate and generally, although with

local and sectoral variations, effectively implemented. In terms of impact, some of the relief

activities, particularly in the WASH, health, and in the NFI/ shelter sectors showed a longer term

effect. Some evaluators remarked that attention to livelihood/ food security and cash for work

during the relief phase, as far as the floods did allow these activities, would have facilitated the

recovery. In general, the response of SHO participants has strengthened local NGO capacities,

which enabled them to respond more adequately to future emergencies. According to the

evaluations, capacity strengthening of civilian government agencies responsible for disaster risk

management, however, received insufficient attention. Some of the factors that the evaluators

found positively contributing to both relevance and effectiveness were: prior presence of

implementing agencies in Pakistan; organisations’ experience in responding to previous

emergencies; work through local partners (most of the SHO participants evaluated have

partnerships with local organisations); the flexibility to address the needs in different phases of

the emergency and the ability to timely mobilize staff, funds and equipment internationally. A few

factors hindered an effective response, such as delays in appeal funds release, coordination issues

(particularly the weak management of the clusters), bureaucratic hurdles and - in PKP - security

concerns.

2. A picture that emerged from the evaluation reports is that the involvement of local communities

in all of the phases of the programme, starting with the needs assessment, greatly contributed to

effectiveness and sustainability. The same applied to setting up community committees and other

CBOs, such as farmer groups, which was generally seen as an important instrument to ensure

sustainability of the interventions and a smooth transition towards recovery and development.

The response of most SHO participants strengthened the resilience of local communities and local

coping mechanisms, particularly, because the communities assisted became better organised. In

some cases the assistance also had a positive impact on the level of participation of women in

decision making at the community level. People were able to rebuild their lives better. Some

evaluators indicated that more specific activities of both government agencies and NGOs directed

at disaster preparedness and disaster response planning were needed as well as more robust in-


depth needs assessments with a stronger focus on the coping strategies employed by the flood

affected population.

3. In terms of coverage, the SHO participants undertook interventions in those areas that were most

affected and, generally, they reached the major population groups facing life-threatening risk.

During the first phase of the emergency all the responses of SHO participants focussed on

immediate lifesaving activities. Most organisations made a smooth transition from relief to early

recovery, although some delays in the implementation of essential recovery activities with regard

to shelter and livelihoods/ food security were also reported. Most organisations purposely

continued working in the same communities, some even in the same sectors - particularly WASH

and NFI/ shelter -, while others started activities in new sectors - particularly livelihoods.

4. The evaluation reports showed that the most deserving were well reached where targeting of

beneficiaries took place in consultation with the community. Overall, the programmes supported

by SHO participants paid attention to most vulnerable groups, but in very different degrees.

Vulnerable groups concerned women, more particularly widows and other female-headed

households, and children, and to a much smaller extent, elderly people and people with

disabilitiescxxxiii, while even less consideration was given to religious, ethnic and tribal minorities.

Although none of the interventions evaluated included special components for women or specific

vulnerable groups, many programmes implemented by local or international partners of SHO

participants and by SHO participants themselves integrated women´s concerns in the different

stages of the interventions and made efforts to increase their participation in consultations,

activities and decision making. These (partner) organisations did much better in terms of

sensitivity to diversity than the international umbrella organisations. An issue of concern of

several evaluators was the lack of disaggregation in data collection and analysis to adequately

identify who was reached and who benefitted of the support provided.

5. The majority of the evaluation reports did not specifically look at coherence or observance of

fundamental humanitarian and human rights principles separate from checking adherence to the

Red Cross and NGO Code of Conduct. Those who specifically paid attention to coherence linked it

to coordination and solely discussed whether relief efforts had been appropriately and sufficiently

coordinated. Concerning humanitarian and human-rights considerations, some challenges were

observed, particularly with regard to impartiality and neutrality, but also to some extent with

regard to humanity and independence. The SHO participants having long-standing relations with

local partners performed better than those who did not have such partnerships. All the evaluation

reports referred to the Code of Conduct for the International Red Cross and Red Crescent

Movement and Non-Governmental Organisations in Disaster Relief (CoC) and/ or Sphere

Standards; however, not all discussed or substantiated their conclusions with regard to adherence

to these standards. SHO participants showed adequate overall adherence to CoC and Sphere Core

Standards and made efforts to fulfil these standards. Adherence to the Sphere Minimum

Standards for the different sectors was discussed by a few evaluators only and the

implementation thereof, particularly of some with regard to shelter and NFI, posed certain

challenges.


6. Not all the Terms of Reference (ToR) of the evaluations requested attention to the issue of

efficiency and those that did showed quite a variety between the questions the evaluators were

asked to address, ranging from cost-effectiveness to planning and even coordination. This

variation was reflected in the five evaluation reports which included the issue of efficiency,

however very briefly and without much reflection. Regarding cooperation, none of the evaluators

specifically looked into cooperation between the SHO organisations in the field, because this was

not requested by their ToR. Yet, all evaluation reports mentioned efforts by the (implementing

partners of) SHO participants to coordinate their relief activities with other stakeholders. During

the 2010 Pakistan floods emergency multiple coordination mechanisms existed at the national,

provincial and district level and most (implementing partners of) SHO participants took part in

these institutionalised coordination meetings.

7. Only two reports referred to lessons learnt from similar previous disasters. Yet, this does not

mean that the responses did not take lessons learnt into account, although we do not know what

has triggered the change. Compared to the conclusions of the 2005 SHO Earthquake Response,

improvements are notable in the areas of: participation in cluster meetings; attention to gender

and the inclusion of women in the implementation of relief efforts; adherence to the Code of

Conduct and the Sphere Standards. All these issues have improved in the past years throughout

the relief sector. Still, the two evaluation reports mentioned earlier recommended developing a

better system for systematic learning and documentation to help improve the effectiveness and

efficiency of the humanitarian system’s response to future disasters globally. We feel that the

planned phase 3 of the meta-evaluation, which focuses on internal learning of the SHO members,

is a good step in that direction.


Annex 1: SHO Terms of Reference Meta-evaluation and synthesis of the 2010 Floods in Pakistan

June 18, 2012

1) Introduction

On August 12, 2010 the SHO started a co-ordinated effort for the victims of the floods in Pakistan.

The participating organisations in this initiative were: Cordaid (Mensen in Nood), ICCO & Kerk in

Actie, The Netherlands Red Cross, Oxfam Novib, Save the Children, Stichting Vluchteling, Tear,

UNICEF Nederland en World Vision. The funds raised – 27,5 million euro – were spent on housing,

medical care, food, water, sanitation, education and livelihood opportunities. The initiative officially

ended in August 2011.

SHO participants agreed to carry out meta-evaluations for each action. SHO follows the ALNAP1

definition for meta-evaluations, adapted from Lipsey (2000): systematic synthesis of evaluations

providing the information resources for a continuous improvement of evaluation practice.’ This ToR

provides the guidelines and expected outcomes for the Pakistan meta-evaluation.

The meta-evaluation will therefore assess the quality of the evaluations carried out by the individual

SHO members with regards to their implemented activities in Pakistan. The report will be based on

existing information – i.e. the available project evaluations of the SHO members.

In addition to systematically analysing available evaluations to improve our evaluation practice the

SHO members want to broaden the scope of this exercise by summarizing and synthesizing the main

results and recommendations of the individual project evaluations. This summary/synthesis will be

used to inform the Dutch public and donors about the outputs and outcomes of our interventions at

an aggregated level (accountability). The possibility to summarize depends on the actual outcome of

the quality of the project evaluations (the meta-evaluation component), as well as the level of

information provided.

The expected added value of this meta-evaluation is thus twofold:

- Facilitate joint learning by examining the quality of evaluations - Accountability towards back donors and the general Dutch audience

Each SHO member applies its own PME system; all members account, through external project – or

program evaluations for at least 50% of SHO budget received (with a minimum of 250.000 euro).

These evaluations may cover direct humanitarian response and/or rehabilitation activities. Since the

SHO did not agree on the contents of the ToR for individual evaluations in advance, the evaluation

reports of the individual participating organisations are likely to have different objectives,

methodologies, focus, lay-outs and depth. Hence, it might be challenging to draw comparable data

from the individual evaluation reports.

1 ALNAP is the Active Learning Network for Accountability and Performance in Humanitarian Action and was established in

1997. ALNAP is a collective response by the humanitarian sector, dedicated to improving humanitarian performance through increased learning and accountability.


This ToR covers the following sections: main purposes and specific objectives (paragraph 2),

methodology (par. 3), organisational issues (par. 4), the budget & timeline (par. 5) and the selection

process of the consultant. Annex 1 provides the evaluation grid.

2) Main purposes of the evaluation As indicated in the introduction the main purposes of the evaluation are to provide an independent

assessment of the project evaluations by the individual SHO members on effectiveness and impact of

the SHO-funded responses. It serves not only joint learning by SHO members, but also the

accountability of the SHO.

The meta-evaluation takes into account the agreements made with regards to evaluations as stated

in the Organisation Protocol (“Organisatiereglement SHO, January 2010”, chapters 1.11 – 1.19). This

refers to:

- The OECD DAC criteria for disaster response evaluations, in accordance with the ALNAP Quality Proforma;

- The Sphere minimum standards in Humanitarian Response during the intervention; - The Principles of Conduct for The International Red Cross and Red Crescent Movement and

NGOs in Disaster Response Programmes during the intervention.

The main purposes and abovementioned standards and criteria are covered by the following three

specific objectives:

Objective 1: Assess the quality of the evaluations of the SHO-members.

Objective 2: Provide an overview of the findings of the different project or programme

evaluations that were undertaken/funded by the SHO-members.

Objective 3: Formulate lessons learnt and recommendations for SHO regarding future

humanitarian interventions and its methodology for evaluations of SHO-

funded projects/programs.

These objectives are further elaborated below.

Objective 1: Assess the quality of the evaluations of the SHO-members.

SHO members will provide their relevant evaluations to the consultant. Annex 1 provides a quality

assessment grid that will be used to assess the quality of these evaluations. This quality assessment

grid includes elements such as meeting of objectives indicated in the ToR, impact, quality of the

analysis and reliability of the data. The consultant will provide the rationale for scores given and

indicate strengths and weaknesses. The quality assessment grid will be used for all project evaluation

reports shared. Based on the outcomes a synthesis is written with specific attention for each

component in the assessment grid. Evaluations of sufficient quality will provide the basis for

objective 2. The exact minimum score will be defined in a session between the SHO members and

the consultant.

Objective 2: Provide an overview of the findings of the different project or programme

evaluations that were undertaken / funded by the SHO-members.

The overview of the findings will consist of two parts.


Part I refers to conclusions drawn in the project evaluations in relation to the following specific topics

– for each topic a specific question is defined:

Appropriateness/Relevance - Was the response tailored to local needs, enhancing ownership, accountability, and cost-effectiveness?

Efficiency - Was the response efficiently implemented?

Effectiveness and impact - Was the response effectively implemented?

Coverage - Did the response reach major population groups facing life-threatening risk wherever they were?

Connectedness - Did the response ensure that activities of a short-term emergency nature are carried out in a context that takes long-term and interconnected problems?

Coherence - Did the response take into account humanitarian and human-rights considerations (assess security, developmental, trade and military policies, as well as humanitarian policies)?

Fact-finding and the analysis to answer these questions will be based on the evaluations that have

proved to be of sufficient quality. The SHO Quality group will decide on actions to take if overall

information and quality is inadequate.

Part II will look at conclusions in the evaluations with regards to compliance with SHO agreed upon

standards, cross cutting themes and organisational issues. More specifically the following topics and

related questions are expected to be answered:

a) Standards

Did the response adhere to the Code of Conduct and the Sphere Standards? b) Cross-cutting themes

Gender - Was gender considered in the agencies’ emergency assessments? Did relief provision include special components for women, men, girls and boys and, if so, were these systematically monitored?

Vulnerable groups - Were the special needs of acute vulnerable groups (e.g. children/elderly/disabled etc) considered in the agencies’ emergency assessments and were they consulted in the same way as other groups? Did relief provision include special components for them and if so were these appropriate and systematically monitored?

Strengthening of responses - Did the response of the SHO agencies strengthen and complement the response of local organisations and coping mechanisms, or hinder them?

c) Organisational issues

Co-operation - What was the level of co-operation in the field between the SHO agencies? Could more have been done to help improve the effectiveness of SHO agencies’ responses in terms of co-ordination, joint-logistics, communications packages, and information flows between the key relief players?

Lessons learned - To what extent did responses reflect lessons-learned from previous similar disasters and more specifically from the SHO humanitarian aid action in East Asia in 20052?

Geographical coverage - Was there appropriate geographical coverage within the affected region?

Objective 3: Formulate lessons learnt and recommendations for SHO regarding future

humanitarian interventions and its methodology for evaluations of SHO-funded

projects/programs.

2 See meta-evaluation 2009


Based on the findings, results and conclusions of objectives 1 and 2, the meta-evaluation shall

present the lessons learnt and recommend future actions for SHO or its members to improve the

evaluation of future humanitarian interventions. The recommendations should include ideas or

proposals for learning activities for SHO.

3) Methodology and deliverables This meta-evaluation consists – in line with its main purposes - of three phases:

Phase 1: Quality check

Phase 2: Accountability phase

Phase 3: Learning among SHO members

Phase 1: Quality check

The findings and recommendations of the meta-evaluation will be based on a desk study of the

reports of external evaluations and reviews. Based on this desk study a first draft report will be

developed that will be discussed during a meeting with SHO members. This draft report should

consist of:

1. An analysis on the quality of the evaluation reports based on the assessment grid mentioned under objective 1;

2. A section with the conclusions, and lessons learned based on the above analysis; 3. Recommendations for continuation of this meta-evaluation (phase 2 and 3) and if necessary

an adapted work plan. The main outcomes and recommendations will also be used in the final meta-evaluation report.

During a meeting with the SHO directors, the consultant will present the deliverables of the first

phase research. Based on the collected information on the evaluations, the quality of the

content and the presentation of the consultant, the SHO directors will decide whether to proceed to

the second phase.

Phase 2: Accountability

During the accountability phase a synthesis report will be developed containing the following main

components:

1. Background information on the emergency context in Pakistan; 2. A short reflection on the evaluation methodologies used; 3. A synthesis presenting the main outcomes of the analyses based on the elements mentioned

under objective 2 This synthesis report will be used to inform the Netherlands Ministry of Foreign Affairs, interested

NGOs and the Dutch public on the outcomes of the questions mentioned under objective 2.

Phase 3: Learning The outcomes of the meta-evaluation as well as the synthesis report will be shared among the SHO

members. The need to further revise processes and procedures, concepts and axioms of disaster

response (1st, 2nd, 3rd loop learning) as well as our evaluation system will be discussed. The

consultant, in cooperation with ICCO and Kerk in Actie, will organize a learning event that shall

integrate the findings and recommendations of this meta-evaluation.

Overall report


Finally, an overall report will be made which contains the reports of phase 1 and 2 including a short

one-pager on the learning session. Besides the reports mentioned above, the final report will

contain:

1. An executive summary, including key recommendations (max. 3 pages); 2. A section with the lessons learned, conclusions & recommendations; 3. Appendices, including the Terms of Reference, maps, sample framework, summary of agency

activities, sub-team report(s), end notes (where appropriate) and bibliography.

After being selected, the consultant will develop a detailed work plan at the beginning of the process.

All materials collected in the undertaking of the evaluation process should be lodged with the SHO

Secretariat prior to termination of the contract.

4) Responsibilities participating SHO organisations

The process of developing and finalizing the meta-evaluation and synthesis is steered by representatives of the SHO quality group. ICCO & Kerk in Actie is responsible for the overall co-ordination of the meta-evaluation, supported by Stichting Vluchteling and Unicef through a Reference group. This Reference Group3 will:

Select and facilitate the communication with the evaluator(s);

Facilitate the administrative steps around this evaluation in coordination with SHO back office (contract, payment etc);

Support the consultant in assembling the relevant documentation (i.e. evaluation reports and reviews)

Organise preparation and progress meetings and a final learning event;

The members of the quality group will

Provide relevant information within their own respective organisations and send it to the evaluator(s);

Provide input into draft report;

Agree on the final report;

Participate in knowledge exchange and the final learning event.

Participating SHO agencies are required to submit the following material as input to the meta

evaluation (preferably both in hard copy an electronic format to the SHO secretariat)

- Key documents on the agency’s response to the emergency and their use of SHO funds (general project reports, overviews, SHO reports, etc.)

- Reports of external and internal evaluations and reviews

5) Expected workload Activity Number of days

Preparation and reading (basic documents) 2

Desk study (analysis different evaluations, +/- 20) 10

Writing meta-evaluation report 3

Presentation Report for SHO and processing final adjustments 2

Writing synthesis report 3

Contribute to facilitation of learning process 2

3 Members are: Wim Stellingwerf (St. Vluchteling), Martijn Engels (Unicef), Martijn Marijnis (ICCO & Kerk in Actie)


Total working days consultant 23 days

NB: in this table the expected workload for the consultation is mentioned. The final workload might decrease depending

on the decision on the go/no go of phase 2 of this meta-evaluation.

6) Timeline

Activity Week(s) Date

Approval of ToR 25 June 18

Deadline for application 26 June 27

First meeting Reference Group / Consultant 27 July 5 or 6

First draft meta-evaluation report

Go/no go phase 2

35

36

August 27 September

3

Writing of draft and final synthesis report

Learning event

To be discussed

Nov/ Dec 2012

Finalization of the process December 2012

6) Selection process and minimum requirements

SHO members will propose possible candidates. These candidates should be independent and have

not been involved in one of the project evaluations. The SHO quality group will make a shortlist of 3 -

5 candidates. The Reference group will cross-check the independency of the candidate and select

after consulting the quality group. These candidates will be invited to prepare a work plan and

budget.

The evaluator should have relevant skills and a proven background in humanitarian emergency

project implementation and/or evaluations. No specific sector expertise is required. The evaluator(s)

should be gender aware. The selection procedure will be elaborated in a specific document. In

summary, selection criteria are:

- Experience in humanitarian aid - Knowledge of the country/region - Gender awareness - Availability


Annex 2: List of evaluation reports and other literature used

List of evaluation reports:

Title, date and authors evaluation report Type of evaluation 1 Evaluation Report Oxfam Novib Flood Response August 2010 –

March 2011. June 2011. Institute of Social Policy, Islamabad Evaluation relief and early recovery phases

2 Pakistan Floods 2010. Evaluation of the Relief Phase of the International Federation of Red Cross Red Crescent Societies/ Pakistan Red Crescent Society Monsoon Flash Floods Operation. Undated (July 2011?). Dr Niaz Murtaza

Evaluation relief phase

3

Inter-Agency Real Time Evaluation of the Humanitarian Response to Pakistan’s 2010 Flood crisis. March 2011. Riccardo Polastro (team leader), Aatika Nagrah, Nicolai Steen, Farwa Zafar.

RTE

4 End of Programme Evaluation Report for DEC-funded WV Relief Program in Sindh. World Vision Pakistan, Flood Response Program – DEC phase 1, August 2010 – January 2011. July 2011. Sustainable Solutions Ltd

EvaluationPhase 1 of response

5 Final Evaluation: Pakistan Flood Emergency-Draft Report Act Alliance. “Together we are stronger” First Draft Report of External Final Evaluation of Pakistan Flood Emergency Appeal (Pak102) Implemented by CWS, DKH and NCA, 29th March 2012. Dr Niaz Murtaza, Khurshid Alam and Sahar Bhatti

Evaluation full period of response

6

Final report Evaluation of Cordaid’s Flood Response 2010 Programme Pakistan. December 2011. Ton de Klerk and Shahida Sultan

Evaluation full period of response

7 Evaluation of the ‘Support to Flood Affected Communities in Southern Punjab, Pakistan’. 1st of November 2010 – 11th of June 2011. August 2011. Gerrit-Jan van Uffelen

Evaluation of Phase 1 of a livelihood programme

8

Real Time Evaluation of CRS’ flood response in Pakistan, KPK and Baluchistan. October 2010. Clara Hagans and Loretta Ishida

RTE

9

Real Time Evaluation of CRS’ flood response Pakistan Jacobabad and Kashmore, Sindh. November 2010. Evaluators unknown

RTE

Other literature used:

Hoogendoorn, Annemieke. August 2009. “Meta-evaluation South Asia Earthquake SHO action. Final Report.”

SHO. December 2011. “SHO-eindrapportage Pakistan 12 augustus 2010 – 31 augustus 2011. SHO-actie ‘Help slachtoffers Overstroming Pakistan’.”

Report of the Secretary-General to the General Assembly (sixty-fifth session, Item 69 (d)). 7 March 2011. “Strengthening emergency relief, rehabilitation, reconstruction and prevention in the wake of devastating floods in Pakistan - Strengthening of the coordination of humanitarian and disaster relief assistance of the United Nations, including special economic assistance: special economic assistance to individual countries or regions “


Annex 3: Overview of provinces with districts in which SHO participants supported interventions and the focus or sector of support

SHO participant

Provinces with districts covered Focus/ sector of interventions

Oxfam Sindh: Ghotki, Jamshore, Kashmore, Thatta, Jacobabad, Shahdad Khot, Larkana, Dadu. Punjab: Multan, Muzaffargarh, Layyah, Bakhar, DG Khan, Rajanpur. KPK: Charsadda, Mardan, Kohistan, Shangla, Battagram, Mansehra. AJK: Muzaffarabad.

Early warning and evacuation, WASH, emergency food security, livelihood and advocacy.

Red Cross/ Red Crescent

KPK, Punjab, Sindh, Balochistan. Food, NFI, health, WASH, shelter (relief, recovery) and livelihood.

UNICEF Sindh, Punjab, KPK, Balochistan. (evaluation primarily focused on institutional aspects).

World vision

Sindh: Sukkur, Shikarpur, Qamber Shehdadkot. NFI, WASH, primary and preventive health.

Act alliance Balochistan: Sibi. KPK: Swat, Kohistan, DI Khan, Mansehra, Shangla, Nowshera, Charsaddha, Tank, Upper Dir, Lower Dir, Peshawar. Punjab: DG Khan, Muzzafargarh, Rajanpur Sindh: Khairpur, Sukkur, Thatta.

Food, NFI, temporary shelter, WASH health and livelihood.

Cordaid KPK: Shangla, Kohat, Kohistan, Swat. Sindh, Punjab.

Shelter (relief, rehabilitation), NFI’s, WASH and health.

ZOA Punjab: Rahim Yar Khan. Livelihood (seed and fertiliser distribution, land clearance, tube-well and generator repair).

CRS KPK, Balochistan. Sindh: Jacobabad, Kashmore.

NFI, transitional shelter, livelihood, hygiene promotion, infrastructure rehabilitation (cash‐for‐work).

Source: evaluation reports


End notes

i Publieksrapport over SHO Actie ‘Help slachtoffers Overstroming Pakistan’, SHO-eindrapportage Pakistan, 12 augustus 2010

- 31 augustus 2011, p.19. A detailed overview of the support provided can be found in this report. ii http://samenwerkendehulporganisaties.nl/actions/pakistan-2010/

iii The floods covered over 100,000 square km, and affected a wider area and more people than the Indian Ocean Tsunami

2004, the Pakistan 2005 and the Haiti 2010 earthquakes combined. iv

http://www.irinnews.org/Report/90115/PAKISTAN-Top-10-natural-disasters-since-1935 v Inter‐Agency Real Time Evaluation of the Humanitarian Response to Pakistan’s 2010 Flood crisis. Riccardo Polastro, Aatika

Nagrah, Nicolai Steen and Farwa Zafar. March 2011 p 20-21. vi

According to the World Bank, about 2 percent of Pakistani households control more than 45 percent of the land area. http://siteresources.worldbank.org/PAKISTANEXTN/Resources/293051-1264873659180/6750579-1291656195263/Ch4_PakistanFloodsDNA_December2010.pdf vii

Final Evaluation: Pakistan Flood Emergency-Draft Report Act Alliance. “Together we are stronger” First Draft Report of External Final Evaluation of Pakistan Flood Emergency Appeal (Pak102) Implemented by CWS, DKH and NCA, 29th March 2012. Dr Niaz Murtaza, Khurshid Alam and Sahar Bhatti p.7 viii

The reports reflected the evaluations as carried out by different evaluators according to specific Terms of Reference developed in line with the prevailing PME system of an individual SHO member. Some SHO members can fully influence an evaluation, others hardly or not because they are part of the evaluation system of the international umbrella organisation to which they belong, such as UNICEF and The Netherlands Red Cross. ix More information about the assessment grid and the assessment as a whole can be found in the report Meta-evaluation

and Synthesis of the 2010 Pakistan Floods Response by SHO members. Report Phase 1: Quality Assessment. Verona Groverman and Joke Hartmans. 8 October 2012. x The reports evaluating the emergency response programmes of the other three SHO participants scored unsatisfactory in

phase 1 of this meta-evaluation, the quality assessment. xi This observation holds for most of the 16 reports assessed.

xii The ToR of the Meta-evaluation and Synthesis also requested attention to the issue of cost effectiveness. Since the

evaluations reports considered this issue under the evaluation criterion efficiency, we refer to the section on efficiency in this report. xiii

Report # 1, see Annex 2. xiv

The comments are taken from the reports # 3 and # 2, respectively. xv

Reports # 1, #5, #6, #7, #8 and #9. xvi

Report # 6. xvii

Report # 3. xviii

Report # 7. xix

Report # 2, #5 and #4, respectively. xx

Taken from report # 2 and #5. xxi

Report # 1, #2, #4, and #5. xxii

Report # 7 xxiii

Report # 4 and #2, respectively. xxiv

Report # 6 and #5 respectively. See also section 4.5 on Connectedness. xxv

Report # 5 xxvi

Report # 2 and #4 xxvii

The text refers to report # 4 and #1. xxviii

Report # 1, #5, and #6 xxix

Report # 7 xxx

Report # 3 and #1, respectively xxxi

The arguments are given in the findings section of the report. Farmers preferred in kind distributions over cash payments to buy inputs themselves; generator and tube-well repair provided through the project saved farmers’ resources at a time that both money and time were important constraints to flood affected households; land preparation was done by sub-contracting relative better-off farmers in possession of a tractor, which is seen as an efficient means in the absence of viable alternatives. xxxii

Report # 4 and #2, respectively xxxiii

Report # 4 and #6 xxxiv

Under ‘coverage’ we address two issues that were mentioned separately in the ToR: coverage and geographical coverage. xxxv

Report # 1, #4, #5, #6, #7, #8 and #9 xxxvi

Illustrative is the classification of potential beneficiaries in two groups, applied by a SHO participant (found in Report # 6) They distinguished Category A with the following criteria: - a flood/rains affected HH with a fully damaged house owned by them; - beneficiary used the damaged house as his first home / not as commercial property; - should be a married person with a normal family structure; widows and widowers with unmarried sons or daughters qualified also; - he/she did

http://samenwerkendehulporganisaties.nl/actions/pakistan-2010/

http://www.irinnews.org/Report/90115/PAKISTAN-Top-10-natural-disasters-since-1935

http://siteresources.worldbank.org/PAKISTANEXTN/Resources/293051-1264873659180/6750579-1291656195263/Ch4_PakistanFloodsDNA_December2010.pdf

http://siteresources.worldbank.org/PAKISTANEXTN/Resources/293051-1264873659180/6750579-1291656195263/Ch4_PakistanFloodsDNA_December2010.pdf


not rebuild their house and was not covered by another NGO for provision of shelter, - was living with a host family or in a rented house; - had either own or donated safe and stable piece of land to build shelter. Category B criteria covered groups which were considered less urgent: - a flood/rains affected HH with partially damaged house. HH’s who hadn’t lost their house but were living in unsafe shelter due to land settlement/weak structures were also eligible; - a flood affected HH who was already covered by another NGO but the received shelter was not enough for this family considering its size and social norms (for example if a couple having young or mature kids are sharing single room or shelter or young brothers and sisters are sharing single room etc.); - a family who had migrated after floods due to house damages and now wanted to come back; - a person (married) who had a partially damaged house but was not able to repair it due to certain reasons, for example widows and people with disabilities. xxxvii

Report # 2 and 3 xxxviii

Report # 1, 4, and 5 xxxix

Report #1 xl Report # 2

xli Report # 6, 8 and 9

xlii Report # 2

xliii Report # 4 and 6

xliv Reports # 1,2,4,6,8 and9

xlv Report # 8 and 9

xlvi Report # 2

xlvii Report # 5

xlviii Report # 2 and 5

xlix Report # 5

l Report # 8 and 9 li Report # 6. p.10

lii Report # 1

liii Report # 5

liv Report # 6

lv Report # 7

lvi “Evaluating humanitarian action using the OECD-DAC criteria”, ALNAP, March 2006, p33,

lvii “Evaluating humanitarian action using the OECD-DAC criteria”, ALNAP, March 2006, p19,

lviii http://ochanet.unocha.org/p/Documents/OOM_HumPrinciple_English.pdf

lix Report #1,

lx Report #3, p36.

lxi Report #5 and 9

lxii Report #2 and 6

lxiii Report #3

lxiv An exception is presented in one of the evaluations (Report # 2) which established that the focus on the most vulnerable

regions and families was mixed with some low vulnerability regions and families selected along with those with high vulnerability. One of the reasons being that beneficiary lists for a large scale relief programme had been provided by local authorities and were not verified or prioritized by the organisations themselves. lxv

Report #1, report #5, report #6, report #8 and report #9 lxvi

Among which the reports #3 and #7 lxvii

A complex emergency is defined by the Inter Agency Standing Committee (December 1994) as “A humanitarian crisis in a country, region or society where there is total or considerable breakdown of authority resulting from internal or external conflict and which requires an international response that goes beyond the mandate or capacity of any single agency and/ or the ongoing United Nations country program”. lxviii

The authorities control movement of international organisations through the No-Objections Certificate system, and constrained access to the affected populations, partly for security and partly for other reasons. See: http://daraint.org/wp-content/uploads/2010/10/IARTE_PK_displacement_2010_report.pdf. lxix

Report #6. Another SHO participant apparently experienced similar problems, because the evaluators reported that the implementing partners undertook advocacy on issues related to access (Report # 5). lxx

Report #3 lxxi

Report #5 lxxii

http://www.ifrc.org/Docs/idrl/I259EN.pdf. lxxiii

The sphere handbook can be downloaded or ordered at http://www.sphereproject.org/handbook/. lxxiv

Report # 1, 5 and 6. lxxv

Report # 5. lxxvi

Report # 2. lxxvii

Reports # 1, 2, 5, 6, 7, 8 and 9 lxxviii

According to report # 6 the implementing partner of one SHO agency however brought its staff over from another region within Pakistan. lxxix

Report # 6.

http://ochanet.unocha.org/p/Documents/OOM_HumPrinciple_English.pdf

http://daraint.org/wp-content/uploads/2010/10/IARTE_PK_displacement_2010_report.pdf


http://www.ifrc.org/Docs/idrl/I259EN.pdf

http://www.sphereproject.org/handbook/


lxxx

Report # 1, #2, #4, #5, #8 and #9 lxxxi

Report # 1. lxxxii

Report # 2. lxxxiii

Report # 2. lxxxiv Report # 5. lxxxv

Report # 1. lxxxvi

Report # 4. lxxxvii

Report # 6 and #5, respectively. lxxxviii

Report # 6. lxxxix

The WASH cluster also managed to adapt standards to the context; the WASH organisations did seek to provide adequate sanitation facilities where more permanent shelters are made. (source: Inter Agency RTE) xc

The Inter Agency RTE refers to the 2010 IA‐RTE (Displacement Crisis) which already highlighted this issue. http://daraint.org/wp-content/uploads/2010/10/IARTE_PK_displacement_2010_report.pdf. xci

Since most evaluators combined the information about gender and vulnerable groups in the reports the two issues are covered in one section. xcii

Report # 1, p.12. xciii

Report # 2 and #3. These evaluations concerned the relief phase. xciv

Report # 3. xcv

Report # 1. xcvi

Inter Agency RTE, p. 52. xcvii

Report # 1, #4, #5, and #6. xcviii

Report # 7. xcix

Report # 1, #4, #5, #6, #8 and #9. c Report # 6, p.28.

ci Report # 1 and #4.

cii Report # 4, p.20 and 23.

ciii Report # 5, p.34.

civ Report # 6 and #7.

cv Report # 1 and #5.

cvi. Report # 3, p34.

cvii Report # 1

cviii Report # 1.

cix Report # 5.

cx Report # 8 and #9, also report # 6.

cxi Report # 2

cxii Report # 5 and #6

cxiii Reports #1, #4, #5, #6, #7, #8 and #9

cxiv Report # 4, #5 and #6

cxv Report # 2

cxvi Report #2

cxvii Report # 4.

cxviii Report # 1, #2, #4, #5, #6, #8 and #9

cxix Report # 1

cxx Report #5. The government body responsible for the coordination of the flood response was the National Disaster

Management Authority (NDMA). At the national level coordination was jointly managed by NDMA and the UN Humanitarian Coordinator; at the provincial and the district level the Provincial Disaster Management Authorities (PDMA), respectively the District Disaster Management Authorities (DDMA), were managing the cluster coordination meetings jointly with the UN Office for the Coordination of Humanitarian Assistance (OCHA). cxxi

Reports #1, #4, #5, #6, #8 and #9. cxxii

Report #1 and report #7 cxxiii

Reports #1, #3, #5, #8 and #9. cxxiv

Reports #1, #3, #5 and #6. cxxv

Reports #1, #5, #6, #8 and #9. cxxvi

Reports #1, #5 and #9. cxxvii

Reports #2, #5 and #6. cxxviii

Report #5. cxxix

The evaluation report Meta-evaluation South Asia Earthquake SHO action. August 2009 gives only conclusions and does not include lessons learnt. cxxx

Report # 2 and #3. The ToR from one other evaluation requested insight into the question ‘To what extent has the SHO organisation been building on lessons learnt from other emergency responses’ but the report ( i.e. report # 4) did not include an answer. cxxxi

Report # 2, p.22, 28 and 31.



cxxxii

The evaluation report also gives conclusions on sectoral performance issues. These are less relevant for the issue we have to address since they mainly focus on the implementation level. cxxxiii Concerning people with disabilities we like to refer to Nazmul Bari’s Participatory Evaluation Report “Mainstreaming

Disability in Emergency Response and Local Development” CHEF International, Pakistan (Project Number Light for the World: p00555) First draft . April 2012, which was part of phase 1 of the meta-evaluation but, unfortunately, did not qualify mainly due to its draft status.