urd-tdm-acf mongolia wash evaluation-final report …€¦ · external evaluation of acf’s...

64
External evaluation of ACF’s project, “Improve Access to Water, Sanitation and Hygiene in the Ger areas of Ulaanbaatar, by providing proven and innovative solutions” May 2009-April 2012 (first phase) May 2012-June 2015 (second phase) June 2015 Julie Patinet Anne Delmaire

Upload: vungoc

Post on 30-Aug-2018

224 views

Category:

Documents


3 download

TRANSCRIPT

Page 1: URD-TDM-ACF Mongolia WaSH evaluation-final report …€¦ · External evaluation of ACF’s project, “Improve Access to Water, ... methodological tool design, ... Ger Mongolian

External evaluation of ACF’s project,

“Improve Access to Water, Sanitation

and Hygiene in the Ger areas of

Ulaanbaatar, by providing proven and

innovative solutions”

May 2009-April 2012 (first phase)

May 2012-June 2015 (second phase)

June 2015

Julie Patinet Anne Delmaire

Page 2: URD-TDM-ACF Mongolia WaSH evaluation-final report …€¦ · External evaluation of ACF’s project, “Improve Access to Water, ... methodological tool design, ... Ger Mongolian

2

Final external evaluation of the ACF project Improve access to WASH in the Ger areas of Ulaanbaatar, Mongolia

Groupe URD - TDM | June 2015

Groupe URD is a support structure for the humanitarian sector. For over 20 years it has endeavoured to improve the aid practices

for the populations affected by crises, with several types of activity: operational research projects, programme

evaluations, methodological tool design, institutional support and training processes, both in France and abroad.

Toilettes du Monde has worked since 15 years in France and abroad with public and private sanitation players to improve the hygiene

and living conditions of underprivileged populations by promoting the development of ecological sanitation (dry

toilets, grey water) through professional training actions, action research, bibliographic studies, techniques,

regulations, socio-cultural actions, lobbying the authorities.

Note on the authors Julie Patinet is in charge of "Environment" research in Groupe URD. She studied Geography and has a Master’s degree

in Geomatics and Development. She is a qualified Risk Prevention Territorial Engineer and worked in French local

development associations before turning to the international aid sector, specializing in the field of ecological

sanitation.

Anne Delmaire is a benchmark in social project management concerning ecological sanitation and solidarity. She

joined the TDM team in 2010 (and is now the general manager) after working for over 7 years in Mexico as a

coordinator of water and ecological sanitation projects. She trains trainers in participatory methodologies such as

MARP, PHAST and SARAR for the social and technical implementation of WASH projects and has created tools for

workers on the ground. She has likewise carried out methodological support and distance techniques.

Acknowledgements I would like to express my sincere thanks to all those who took part in this evaluation and took the time to answer my

questions, as well as for the logistical support that I received, and the confidence that I was shown. I would

particularly like to thank Pier-Francesco Donati and Federica Voltolini for their extremely warm welcome, their

availability and the quality and nature of our discussions.

Groupe URD

La Fontaine des Marins

26170 PLAISIANS

Tel : + 33 (0)4.75.28.29.35

Fax : + 33 (0)4.75.28.65.44

E-mail : [email protected] | www.urd.org

Toilettes du Monde

28, place des Arcades

26110 NYONS

Tél : +33 (0)4 75 26 29 98

Fax : +33 (0)4 75 26 19 02

E-mail : [email protected]

http://www.toilettesdumonde.org

This evaluation was financed by ACF.

The opinions expressed in this report are those of the authors alone.

© Groupe URD | July 2015

Page 3: URD-TDM-ACF Mongolia WaSH evaluation-final report …€¦ · External evaluation of ACF’s project, “Improve Access to Water, ... methodological tool design, ... Ger Mongolian

3

Final external evaluation of the ACF project Improve access to WASH in the Ger areas of Ulaanbaatar, Mongolia

Groupe URD - TDM | June 2015

Table of contents

Abbreviations/Acronyms _____________________________________________________ 5

Executive summary __________________________________________________________ 6

1. The project and its context ________________________________________________ 8

Characteristics of the area of intervention _______________________________________ 8

Brief outline of the project ___________________________________________________ 10

2. Evaluation objectives and methodology ____________________________________ 12

3. Between the interim and final evaluations… _________________________________ 13

4. Results based on the OECD DAC criteria _____________________________________ 16

Coherence /Coordination: the project is coordinated well with its institutional environment

_________________________________________________________________________ 16

A project which is coherent with the new urban development plans ___________________________ 16

ACF an important player in WASH coordination ____________________________________________ 17

Internal coherence of the project ______________________________________________ 18

Relevance: decentralised solutions adapted to the capital’s Ger areas________________ 19

Effectiveness: an ambitious development project ________________________________ 20

Impact ___________________________________________________________________ 21

Impact on behaviour change at the household/community level ______________________________ 21

Impact on a larger scale and at the institutional level _______________________________________ 23

Impact of the advocacy strategy ________________________________________________________ 23

Development of the advocacy strategy ____________________________________________________ 24

Analysis of the content of the advocacy strategy____________________________________________ 24

Analysis of the advocacy strategy process _________________________________________________ 26

Degree of autonomy of local structures to take over kiosks, emptying services, and

composting sites ___________________________________________________________ 27

Lack of anticipation in ACF’s exit strategy _________________________________________________ 27

Multi-service water kiosk managed by Tolgoit for almost three years __________________________ 28

Major concern about the sustainability of MonESIC taking over the ecological sanitation service ____ 29

WaSH Action, a young NGO that is still fragile, but has a lot of potential ________________________ 37

Page 4: URD-TDM-ACF Mongolia WaSH evaluation-final report …€¦ · External evaluation of ACF’s project, “Improve Access to Water, ... methodological tool design, ... Ger Mongolian

4

Final external evaluation of the ACF project Improve access to WASH in the Ger areas of Ulaanbaatar, Mongolia

Groupe URD - TDM | June 2015

Conclusion and recommendations _____________________________________________ 41

Conclusion _________________________________________________________________________ 41

Recommendations for ACF: ____________________________________________________________ 41

Recommendations for WaSH Action: ____________________________________________________ 42

Recommendations for MonESIC: ________________________________________________________ 43

Recommendations for the Water Agencies: _______________________________________________ 43

Recommendations for all actors involved in advocacy: ______________________________________ 44

Annexes __________________________________________________________________ 45

Annex 1: Best practice: Hygiene promotion ______________________________________ 46

Annex 2: Classification according to the DAC criteria _______________________________ 48

Annex 3: People met ________________________________________________________ 50

Annex 4: timetable _________________________________________________________ 52

Annex 5: Bibliography _______________________________________________________ 53

Annex 6: Workshop with the ACF / WaSH Action national staff ______________________ 55

Annex 7: online survey _______________________________________________________ 59

Page 5: URD-TDM-ACF Mongolia WaSH evaluation-final report …€¦ · External evaluation of ACF’s project, “Improve Access to Water, ... methodological tool design, ... Ger Mongolian

5

Final external evaluation of the ACF project Improve access to WASH in the Ger areas of Ulaanbaatar, Mongolia

Groupe URD - TDM | June 2015

Abbreviations/Acronyms DAC Development Aid Committee

EPA U.S. Environmental Protection Agency (who regulates the maximum Ecoli quantity accepted)

GADA Ger Area development Agency

Ger Mongolian term for "yurt"

IEC Information, Education, Communication

Khooro neighbourhood

MIRIM Mongolian branch of IRIM: Independant Research Institute (marketing consultants)

M&E Monitoring & Evaluation

MNT Tugrik (Mongolian currency) 1€ = approx 2 000 MNT

NGO Non-governmental organisation

OECD Organisation for Economic Cooperation and Development

SDC Swiss Agency for Development and Cooperation

SWOT strengths, weaknesses, opportunities and threats

TDM Toilettes du Monde

Tolgoit Grassroots community organisation with the name of the neighbourhood

UB Ulaanbaatar

USTB University of Science and Technology Beijing

USUG Public company responsible for supplying water and sanitation services in the city of Ulaanbaatar

WASH Water, Sanitation and Hygiene

WC Water Closet (flush toilet)

WECF Women in Europe for a Common Future

Page 6: URD-TDM-ACF Mongolia WaSH evaluation-final report …€¦ · External evaluation of ACF’s project, “Improve Access to Water, ... methodological tool design, ... Ger Mongolian

6

Final external evaluation of the ACF project Improve access to WASH in the Ger areas of Ulaanbaatar, Mongolia

Groupe URD - TDM | June 2015

Executive summary

Main Observations / Conclusions Main Recommendations

Su

sta

ina

bil

ity

ACF’s exit strategy was not sufficiently anticipated and

will have a different impact on each local partner. ACF’s

withdrawal will have an obvious negative effect on

MonESIC, who will not meet all the challenges facing it

without external support. The pilot solutions have been

tested technically, but have not been tested socially for

long enough.

Tolgoit is likely to continue to manage the multi-service

kiosk in the long term.

For ACF: the next time ACF implements a project on

the margins of its mandate, it would be a good idea

to anticipate the exit by working with local partners

sufficiently in advance.

For ACF in connection with this project: maintain

strong links with WaSH Action, in the form of a

partnership.

ACF should invest one last time in the EcoSan sector

and support social project management activities,

without which the service is in danger of collapsing.

WaSH Action should also support the EcoSan sector

in the short term, as it will help to make it better

known.

For WaSH Action: in the short term : survive ! and

for the longer term: pick up the torch of WaSH

advocacy

WaSH Action has a lot of potential, but is still very

fragile and still has too little financial visibility. One of

the challenges for this young national NGO will be to

take over the advocacy strategy and to adapt it to their

means.

- the pursuit of the implementation of the advocacy

action plan should not be WaSH Action’s main

priority in the months ahead; but it should

nevertheless remain in the background (with

activities which do not require any budget) because

it is in keeping with Wash Action’s mission (vision)

and allows it to remain visible and involved vis-à-

vis key decision-makers.

For the Water Agencies and the other potential

financial partners: in order for advocacy to be able

to continue, it is NECESSARY to have financial

support for WaSH Action.

Imp

act

Several improvements to national regulations directly

attributable to ACF and its advocacy activities.

Sanitation: the project led to changes in behaviour

(acceptance) vis-à-vis the ecological sanitation

approach, but this remains at the pilot level, which

could well be abandoned if there is no investment in

social project management (households’ enthusiasm

for the EcoSan toilets proved to be reversible as soon

as they had to pay for the service).

Hygiene: excellent long-term impact, which has led to a

drop in the number of illnesses in the targeted schools.

Page 7: URD-TDM-ACF Mongolia WaSH evaluation-final report …€¦ · External evaluation of ACF’s project, “Improve Access to Water, ... methodological tool design, ... Ger Mongolian

7

Final external evaluation of the ACF project Improve access to WASH in the Ger areas of Ulaanbaatar, Mongolia

Groupe URD - TDM | June 2015

Co

he

ren

ce

ACF’s decentralised solutions remain relevant with

regard to the new urban development plans: the two

types of solution (large-scale decentralised public

infrastructure vs decentralised solutions at the

household level) are, in fact, complementary.

It is not likely that all the Ger areas will be transformed

into apartments and connected to the network.

The models that have been tested can meet the WaSH

needs of people in the Ger areas in the short/medium

term.

Recommendation for all actors involved in advocacy

activities: include the need for public showers in

advocacy vis-à-vis the institutions (to be included in

the urban development and facilities plans for the

Ger areas).

Long term recommendation for WaSH Action:

resume action research into improving the

management of grey water (look for partnerships

with researchers + volunteers +funding).

It would be useful, in terms of advocacy, to compare

the different WaSH solutions (economically,

environmentally, and in terms of job creation) and

how they meet the short/ medium/ long-term

needs of the inhabitants of the Ger areas.

Re

lev

an

ce/

ap

pro

pri

ate

ne

ss

The project meets people’s essential WaSH needs, even

though it does not cover needs in terms of public

showers or grey water management. ACF’s innovations

are promising and adapted to an extremely cold

climate, they have inspired large organisations like

UNICEF, and have been integrated into the urban

development preliminary studies for the2nd biggest

city in the country.

The relevance of the solutions is apparent also in the

amount of public demand which continues to grow.

This success needs to be qualified: the technical option

of urine infiltration into the ground is a source of

pollution, and is not coherent with the EcoSan

philosophy which campaigns for complete reuse of all

nutrients.

Eff

ect

ive

ne

ss The project was unable to meet its objectives in

terms of the dissemination/accompaniment of the

tested solutions for sanitation and water, which was

predictable with regard to the duration of the project.

This objective was too ambitious.

Page 8: URD-TDM-ACF Mongolia WaSH evaluation-final report …€¦ · External evaluation of ACF’s project, “Improve Access to Water, ... methodological tool design, ... Ger Mongolian

8

Final external evaluation of the ACF project Improve access to WASH in the Ger areas of Ulaanbaatar, Mongolia

Groupe URD - TDM | June 2015

1. The project and its context

Characteristics of the area of intervention

A capital city that is facing mass countryside-city migration

The country is mountainous and has an arid, almost desert climate. There is thus little farmed

agricultural land and, at present, the population of nomadic or semi-nomadic herders is

estimated as 30% of the Mongolian population. The recent evolution of the climate, with a

reduction in the rainfall, combined with the development of industrial activity (significant

exploitation of mineral resources and development of tanneries) has entailed deterioration in

the quantity and quality of the country’s water resources.

In 2013, Mongolia was divided into 29 river basins, each with a basin Agency, a model that is

very similar to the water agency management in France.

In the 90s, Mongolia swung from an economy based on the Soviet model to a free-market

economy. In addition to this major economic transformation and the weak environmental and

industrial policies, there was a quick succession of particularly harsh winters (dzuud), which

decimated the livestock of numerous nomadic families. Mongolia now has to confront a crisis

caused by a combination of many climatic, environmental and structural factors, which have led

to the rapid mass exodus of the populations to the capital. It seems that this trend is beginning to

slow down now.

Administrative complexity and political crisis

It is not easy to understand the coordination between responsibilities and regulations (even

for the Mongolian institutions themselves). The evaluation team recommends the institutional

analysis of the WaSH sector carried out in connection with the project (January 2014) and the

study, “Sanitation: experiences, challenges and recommendations” (January 2015), which are

both of very high quality.

At the political level, the government experienced a major crisis at the end of 2014 with the

forced resignation of the Prime Minister followed by a ministerial reshuffle. These changes and

the high turnover in key positions which exists in Mongolia, have resulted in certain decisions

and laws being held up.

The specific characteristics of the climate

Four seasons, but which can be summarized as two: a very cold winter period (temperatures

going down to minus 50°C at night and often minus 35°C during the day), and hot in the summer.

This is a central consideration in the Water, sanitation and hygiene sector, and is a major

technical constraint (the need to bury the pipes deep or to insulate them considerably above a

depth of 2.5 metres, all liquids freezing on the surface and at a shallow depth, making the ground

slippery and dangerous, etc.).

As a result, the majority of sanitation options which are known and well-understood in

temperate climates are not applicable in Mongolia. This is all the more true for low-cost local

Page 9: URD-TDM-ACF Mongolia WaSH evaluation-final report …€¦ · External evaluation of ACF’s project, “Improve Access to Water, ... methodological tool design, ... Ger Mongolian

9

Final external evaluation of the ACF project Improve access to WASH in the Ger areas of Ulaanbaatar, Mongolia

Groupe URD - TDM | June 2015

sanitation: there are too few tried and tested solutions for autonomous sanitation in areas where

there is an extremely cold winter. Difficulties to use onsite technologies suitable for the Mongolian

harsh climate conditions and the lack of consistency from WaSH actors in piloting options from

internationally acknowledged models have slowed down the introduction of sound onsite

sanitation technologies for Mongolia (SDC report, June 2015)

The study by the World Bank which is due to be published in September 20151 on local

sanitation in cold climates is a very interesting experience sharing initiative. We should also

point out here that WECF produced a very good technical guide in 2007 on EcoSan toilets, re-

using nutrients, sanitisation and regulatory aspects.

http://www.wecf.eu/cms/download/2007/WP-26_web-07.pdf

This specific climatic characteristic should be seen as an opportunity (for example, negative

temperatures make bacteria inactive).

In the Ger areas

There is a clear disparity between the city centre and the Ger areas in terms of access to

basic services (water, sanitation, communication channels, electricity, health, education).

Access to water is mainly through around 500 water kiosks (the overwhelming majority

public, managed by USUG), of which about half are connected to the mains, the other half being

supplied by water tankers. The management of the water kiosks located in the yurt area is loss-

making for the USUG, because the operating costs are significantly higher than the income from

the sale of water. In the winter, the absence of paved roads makes the journeys complicated and

dangerous, especially those to transport water.

In addition to the problem of quantity (some families live with less than 10L/person/day), there

is the bad quality of the water consumed, notably due to contamination from the transport or in

the home. The water containers used for transport/storage are often unsuitable, as they

previously contained harmful substances (reuse of containers from the petrochemical

industries).

Sanitation essentially consists of basic latrines which, once full, are “closed”. They then dig

another pit. On the one hand, the current system contaminates the land, and on the other hand it

is not sustainable, due to the limited space in individual homes, preventing them from changing

the pits regularly.

The inadequacy or bad quality of the WASH infrastructures, and the lack of knowledge about

how diseases are transmitted, combined with poor hygiene practices, contribute greatly to the

development of diseases, in particular hepatitis A virus.

1 The recommendations of the study were presented for the first time at a conference in June 2015.

Page 10: URD-TDM-ACF Mongolia WaSH evaluation-final report …€¦ · External evaluation of ACF’s project, “Improve Access to Water, ... methodological tool design, ... Ger Mongolian

10

Final external evaluation of the ACF project Improve access to WASH in the Ger areas of Ulaanbaatar, Mongolia

Groupe URD - TDM | June 2015

Brief outline of the project

The project evaluated is formally the one financed by the Seine Normandy and Artois-Picardie

Water Agencies2, which began in May 2012 for a period of 3 years (which ended on 30 June 2015

following a two-month no cost extension). It would, however, make little sense to ignore the

project financed by Swiss Cooperation– SDC (with co financing by ACF) with a 2-year duration

(May 2012-April 2014), the “research” aspect, as well as the “hygiene promotion” project funded

by UNICEF (dates?). Indeed, these three projects were conceived as complementary.

In simple terms (in order to give the reader some points of reference), at the time of ACF’s

withdrawal, the main (and frequently mentioned) achievements of the project are as follows:

Main outputs Comments

Organisation of high quality WASH events,

seminars, exhibitions, media campaigns, etc.

This mobilisation is based on WASH technical

expertise and increases ACF’s recognition in

Mongolia. In effect, these are the main

advocacy activities.

sanitation

Around 280 eco-toilets3 and testing of a peri-

urban emptying service

Robust, tried and tested solution4. The only

solution to date for local sanitation in the Ger

areas (solution recommended by the

engineers who carried out the pre-feasibility

study of WASH facilities for the 2nd largest city

in Darkhan region).

Cost recovery difficult for the emptying

service (only around 10% of users pay)

Need for significant support and

accompaniment (notably advocacy for grants)

before it can genuinely be scaled up.

Ability to produce compost in winter An impressive scientific exploit, but without

any operational use in the present context.

Production of compost in the summer Accumulated experience -> today there is

genuine expertise. Excellent visual and

olfactory quality of the finished product.

Laboratory analyses from June 2015 show

that there is a good level of sanitization, even

though there is still room for improvement

with regard to the summer composting

process (which is still very new) and the

analysis protocol.

2 Including co-funding from Cap Calaisis community agglomeration, ACF.

3 Of the 370 EcoSan toilets built by the project, this figure does not include all the initial prototypes and initial tests or

failures which have been abandoned. 4 Even though it is still possible to improve the designs and optimise the service (with a view to continuous improvement) in

order to optimise costs.

Page 11: URD-TDM-ACF Mongolia WaSH evaluation-final report …€¦ · External evaluation of ACF’s project, “Improve Access to Water, ... methodological tool design, ... Ger Mongolian

11

Final external evaluation of the ACF project Improve access to WASH in the Ger areas of Ulaanbaatar, Mongolia

Groupe URD - TDM | June 2015

Standard approved at the end of May 2015 for

the compost

A significant success which was needed in

order to be able to sell the compost

water

Model of autonomous multi-service water

kiosks for three years

Very promising model, which is economically

viable if it is run by a non-profit association

and we exclude the initial investment cost

(kiosk construction) which remains high.

The model could be refined further to achieve

a perfectly replicable solution.

No dissemination for the time being.

20 000 25L water containers made in UB, then

sold at subsidised prices

Major success in the sense that this activity

was mentioned by all those who were

interviewed for the evaluation. A Mongolian

company is now capable of producing this

type of container. The goal now should be to

analyse demand (a consumer or social

marketing study) in order to determine

whether the « business » should be developed

at cost price or whether to campaign for the

price of the containers to be subsidised.

Hygiene promotion

A lot of WASH awareness-raising materials in

a variety of formats, disseminated in an

innovative way5 (television, radio, social

media…)

One of the most successful outputs both in

terms of quantity and quality (see annex 1

best practice). Materials can easily be re-used

by organizations other than ACF/Wash Action

High quality toolbox for WASH training

courses validated by the official institutions

and integrated into health courses throughout

the country, widely disseminated and used

Long-term impact

Health clubs in 22 schools + networking and

creating Facebook pages for the clubs

Clubs being replicated in certain schools.

Other clubs will perhaps not continue due to

insufficient dynamism and support.

Contribution to major changes in regulations

at the national level

Long-term impact

As was predicted in the intermediate evaluation, it was not possible to achieve the specific

objective of “Accompanying the dissemination of tried and tested solutions which allow

sustainable access to water and sanitation for the inhabitants of Ulaanbaatar’s Ger areas”

(except for the hygiene promotion component): dissemination had hardly begun at the time of

ACF’s withdrawal, which is completely “normal” considering the overall duration of the project

and the real challenge of developing a peri-urban EcoSan sector from A to Z, in a country where

there is strong cultural resistance.

5 This was the first time that WASH related messages were disseminated in this way.

Page 12: URD-TDM-ACF Mongolia WaSH evaluation-final report …€¦ · External evaluation of ACF’s project, “Improve Access to Water, ... methodological tool design, ... Ger Mongolian

12

Final external evaluation of the ACF project Improve access to WASH in the Ger areas of Ulaanbaatar, Mongolia

Groupe URD - TDM | June 2015

Location

The two intervention districts already targeted by the previous research programme carried out

between March 2009 and August 2012 are Bayanzurkh and Songino Khairkhan. Relatively

quickly, activities were focused on Songino Khairkhan district alone, as this was the only district

involved in the second phase (2012-2015).

Map 1: location of ACF’s operational areas in Ulaanbaatar

2. Evaluation objectives and methodology

For us, the evaluation, as well as allowing accountability, must be useful for the project

stakeholders and their partners, enabling them to learn from their experiences and to improve

their actions. It is thus essential to start from their concerns that the evaluation should be

constructive, the players should take ownership of it and it must allow real progress to be made.

It was thus important for future users of the evaluation to be able to discuss the terms of

reference, perceive them as their own, explain the issues involved in the evaluation questions

and express the priority objectives of the evaluation for them. The main expectations which

were expressed by all ACF staff, both at headquarters and in the field, concerned the criterion of

sustainability (particularly for WaSH Action of Mongolia: expectation of recommendations in

order to be able to evolve in the best possible way after ACF’s withdrawal), and the criterion of

impact (for the Water Agencies: the issue of achieving results after a long-term engagement, and

for ACF: reviewing their involvement at the time of withdrawal). As the criteria of coherence and

relevance were covered a year and a half ago, and as there has not been any change since, they

are not developed at length in this report (see interim evaluation).

Page 13: URD-TDM-ACF Mongolia WaSH evaluation-final report …€¦ · External evaluation of ACF’s project, “Improve Access to Water, ... methodological tool design, ... Ger Mongolian

13

Final external evaluation of the ACF project Improve access to WASH in the Ger areas of Ulaanbaatar, Mongolia

Groupe URD - TDM | June 2015

The evaluation was carried out in the unusual context of the last month, when staff were clearing

up and leaving. Despite this, they did everything they could to be relatively available.

Classic evaluation methodology was applied (described in detail in the inception report): the

evaluation is based on a qualitative methodology and is based on the triangulation of

information (written and oral sources, and field observations) in order to draw conclusions.

- Interviews with a wide range of stakeholders: (see annex 3 for more details about the

people met)

- Gathering and analysis of documents (see annex 5) :

- Direct observation of project outputs (see annex 3)

As is often the case for a final evaluation, an online survey was sent to around 200 Mongol and

international representatives of institutions and partners working in Mongolia (see annex 7).

3. Between the interim and final evaluations…

The following table reviews the recommendations made in the interim evaluation: Have they

been implemented? Why? Are they still relevant in relation to the findings of the final

evaluation? Etc.

Crit

eri

on

Main observations Main Observations Reviewed in

June 2015

Ex

tern

al

co

he

ren

ce

The first official development

plan for the Ger areas was

established after ACF’s project

began

ACF’s local solutions remain

relevant considering the new

urban development plans

• All the Ger areas will

probably not be transformed

into apartments and

connected to the network

• A model which will allow the

WASH needs of Ger area

inhabitants to be met in the

short/medium term

• A relatively light and flexible

(adaptable) model in the case

of toilets and the emptying

service

ACF: Coordinate regularly with the municipal

authority, and the competent institutions,

particularly the Ger Areas Development

Agency, and establish contact between ACF’s

partners who are taking over the activities and

these institutions

Collaboration

with the GADA

proved to be

difficult

(turnover,

distrust from the

Agency)

Nevertheless, ACF

continued to

engage in

coordination

activities

ACF is a major WASH player in

Ulaanbaatar

Continue to integrate the tools (in particular

hygiene promotion) and skills into the

different Mongol institutions, and other WASH

This was done

remarkably well

Page 14: URD-TDM-ACF Mongolia WaSH evaluation-final report …€¦ · External evaluation of ACF’s project, “Improve Access to Water, ... methodological tool design, ... Ger Mongolian

14

Final external evaluation of the ACF project Improve access to WASH in the Ger areas of Ulaanbaatar, Mongolia

Groupe URD - TDM | June 2015

organisations In

tern

al

co

he

re

nc

e

Four complementary

components (Water Agency +

research + SDC + UNICEF)

Some progress made possible

by the research (development

of composting technique in a

cold climate, toilet survey,

assurance of scientific

reliability…)

But the combination of research

and operational activities was

very ambitious and

insufficiently prepared with all

the actors, which led to

misunderstanding and

frustration: it could have been

conducted more effectively

For the next time: prepare, clarify and

communicate the objectives with all the

stakeholders from the beginning: anticipate

and clarify how the research and field

activities are going to complement each other.

Ensure that information is regularly shared

between the field and research activities.

Continue to disseminate in Mongolia and

research results at a global level, and use the

results during the advocacy phase.

Recommendation

remains valid6

Pe

rtin

en

ce

The project meets the essential

WASH needs of the population,

even though it does not address

needs in terms of public

showers and managing grey

water

Pursue research into simple, low-cost solutions

for the management of grey water

Recommendation

remains valid, but

for the long term

(is not a priority

for WASH Action)

The innovations developed by

ACF are promising and adapted

to an extremely cold climate

Use multi-service kiosks to:

- provide an example to the population,

USUG and the other WASH stakeholders

(social marketing, promotion of the

multiservice aspect, EcoSan cycle, making

EcoSan toilets available, etc.)

- experiment (re-use grey water in summer,

recycle compost made from faeces…)

As part of the advocacy strategy: collect past

and future lessons from running multi-service

water kiosks, organize visits for WASH

organizations and conduct advocacy in favour

of a new legal standard for multi-service water

kiosks.

Recommendation

remains valid, but

depends on

obtaining funding

There is a good level of

acceptance amongst users and

this is gradually becoming more

widespread

Continue improving the comfort of EcoSan

toilet users to increase the level of acceptance

Recommendation

remains valid

Test the level of re-use of urine and grey water

per plot

Still valid, but not

a priority

Develop a strategy for promoting/selling

compost

Recommendation

remains valid

E F More fine-tuned approaches For a future project: Recommendation

6 The main initial misalignment was due to the different needs in terms of schedules and timeframe between the research

and the program. The research became operative on the ground only during the second year when most of the project

activities were already defined and no more open to technical inputs from the research.

Page 15: URD-TDM-ACF Mongolia WaSH evaluation-final report …€¦ · External evaluation of ACF’s project, “Improve Access to Water, ... methodological tool design, ... Ger Mongolian

15

Final external evaluation of the ACF project Improve access to WASH in the Ger areas of Ulaanbaatar, Mongolia

Groupe URD - TDM | June 2015

(more development oriented)

would probably have increased

the level of acceptance7 and

effectiveness:

- Creation of the supply

without working in parallel

on the demand

- The expats feel that they did

not properly engage with

the Mongolian culture, and

that this was a mistake

Include sociological expertise from the

beginning of the project

Co-create supply in a participatory manner so

that it corresponds to demand

remains valid

The project will probably have

difficulty reaching its objectives

in terms of disseminating tested

solutions by June 2015

Make the accompaniment and consolidation of

capacity building of local organisations a

priority for the last year of the project, and

extend the duration of the project

This was one of

the main

recommendations

. The

accompaniment

of local partners

was made a

priority during

the last year of

the project, but

ACF’s

(premature)

withdrawal may

weaken their

sustainability.

DU

RA

BIL

ITE

The handover to private sector

EcoSan partners is coherent,

but very new and fragile and

there is a danger that it will be

rushed if the accompaniment is

not extended beyond the

remaining year

IMP

AC

T

The project made a decisive

contribution towards changing

behaviour vis-à-vis the EcoSan

approach

Extend the project beyond the 6th year (3 + 3

years is not enough for a development project

of this kind)

As was acknowledged from the

beginning, the pilot project

could not be aimed at the most

vulnerable sections of the

population

Future strategies should look at solutions for

vulnerable households8

This is relevant

for all

humanitarian

organizations

(not a priority for

WaSH Action)

7 This concerns mainly acceptance at the level of decision-makers and institutions concerning the use of compost. 8 This concerns economic vulnerability (the poorest households): WaSH vulnerability is very high even amongst households

with an average income.

Page 16: URD-TDM-ACF Mongolia WaSH evaluation-final report …€¦ · External evaluation of ACF’s project, “Improve Access to Water, ... methodological tool design, ... Ger Mongolian

16

Final external evaluation of the ACF project Improve access to WASH in the Ger areas of Ulaanbaatar, Mongolia

Groupe URD - TDM | June 2015

4. Results based on the OECD DAC criteria

Coherence /Coordination: the project is coordinated well with its institutional environment

A project which is coherent with the new urban development plans

In general, ACF’s intervention in Ger area based on a decentralized logic (construction of a multi-

service water kiosk, renovation of two kiosks, establishment of an EcoSan branch…) is still

relevant when considering the urban development plans implemented by the Mongolian

institutions (6th Master Plan) and consisting broadly of progressively transforming the Ger areas

into blocks of apartments (whose appearance recalls the post-WWII low-income social housing

that was built in Europe) and connecting them to the centralized water and sewer mains.

It is indeed unlikely that it will be possible to transform all the Ger areas into apartments

connected to the mains, especially as new families arrive regularly, and it will take several

decades even for those that are transformed.

New apartment blocs & Gers

This point has been confirmed again and again over the past year and a half.

(SDC report) Ecosan Scale up in Darkhan

As mentioned in the context analysis CDIA proposed in the prefeasibility study for “Water Supply and

Sanitation Infrastructure Improvement Project (WSSIIP) for the City of Darkhan” the combination of onsite

sanitation solutions and infrastructure expansion. After a consistent assessment of sanitation options and the

comparison of MoMo project ecosan cycle with ACF one, the CDIA team selected ACF ecosan toilet and waste

management ACF ecosan cycle for the onsite sanitation project component in Darkhan. The prefeasibility

study foresee the construction of around 9300 ecosan toilet that initially will constitute the first access to

improved sanitation and that later will be slowly replaced, where orographic conditions allows, by sewerage

lines. The adoption of the ecosan technology in a big scale prefeasibility study leads to two considerations

about consistent piloting of actions and coordination of actors

Page 17: URD-TDM-ACF Mongolia WaSH evaluation-final report …€¦ · External evaluation of ACF’s project, “Improve Access to Water, ... methodological tool design, ... Ger Mongolian

17

Final external evaluation of the ACF project Improve access to WASH in the Ger areas of Ulaanbaatar, Mongolia

Groupe URD - TDM | June 2015

The two types of solutions and levels of intervention (large-scale decentralised public

infrastructure on the one hand, and decentralised household-level solutions on the other)

should no longer be viewed in opposition to each other, because they are complementary9.

However, issues of coordination and the development of a strategic vision remain a neglected

priority. The study, “Sanitation in Mongolia: Experiences, issues and recommendations”, which

was completed in January 2015, came to the same conclusions:

The work carried out by actors working on improving the housing and sanitation conditions in the ger area of

Ulaanbaatar is sometimes not welcomed by the authorities because these interventions might potentially increase

the investment done on the Khasha, its market value might also slow down the process for land acquisition from

construction companies redeveloping the Ger areas. At the same time actors working on the improvement of onsite

sanitation facilities in the Ger area are facing difficulties in engaging with households in improving their latrine

because households expect in a very short time to have their living environment connected to the water and

sewerage network. This is due to the lack of a vision strategically integrating the sector of service provision with

onsite sanitation solutions.

� Recommendation for all actors involved in advocacy, including WaSH Action: it

would be useful to compare different WaSH solutions (economically, environmentally

and in terms of job creation) with regard to meeting the short/ medium/ and long-term

needs of those who live in the Ger areas.

In addition, the fact that certain regulations have evolved in the last year (validation of a

standard for compost10, minimum WaSH dictates in schools, the establishment of a budgetary

line by the Ministry of Finance for all schools to have soap11) is directly linked to the work of

ACF. This also shows the relevance of the strategies that ACF has defended with regard to the

institutional context, despite the difficulties mentioned above.

ACF an important player in WASH coordination

Over and above participation in existing coordination bodies and working groups (e.g. WASH,

Education and Health clusters), ACF was clearly a driving force in the creation of ad-hoc working

groups and the organisation of experience sharing events.

Several indicators show that ACF’s WASH expertise is highly regarded:

-ACF was invited by the WHO to run a training course on water quality in 10 Khoroos involved in

the development of health security plans.

-ACF/WaSH Action were invited to take part in the municipal authority’s Waste Management

working group.

-ACF’s participation in a restricted meeting with the World Bank’s WaSH experts on 18 June to

discuss the initial recommendations of the study on sanitation (before its official presentation in

September), two weeks before the end of the mission, shows not only that ACF is recognized, but

also the quality of its networking which allowed it to be invited into this kind of network.

9 This is what the WASH pre-feasibility study for the city of Darkhan shows: decentralised solutions that are planned to be

temporary. 10 Standard which determines the characteristics of the product 11 These two last changes had not been fully officialised at the time of the evaluation.

Page 18: URD-TDM-ACF Mongolia WaSH evaluation-final report …€¦ · External evaluation of ACF’s project, “Improve Access to Water, ... methodological tool design, ... Ger Mongolian

18

Final external evaluation of the ACF project Improve access to WASH in the Ger areas of Ulaanbaatar, Mongolia

Groupe URD - TDM | June 2015

Internal coherence of the project

The first three years (2009-2012) were conceived without necessarily imagining a “continuation”, and

then at the end of 2011 discussions began with the Water Agencies on continuing the activities. A call

by the SDC (beginning of 2012) and a visit by the Water Agencies (March 2012) endorsed the

continuation and development of the project for a further 3 years (only 2 years with the SDC).

Between the end of the first Water Agency project (end 2011-beginning 2012), and the decision to

continue with the operations in Ulaanbaatar, there was a moment of hesitation between the exit

strategy and the new operational strategy, which could be called a “crisis”. This was at the beginning

of 2012 and was marked, among others, by the fact that the expatriate team was completely

renewed, and by several months without a coordinator (accompanied by a loss of institutional

memory).

The three components of the project evaluated are currently as follows:

1. Project financed by the Water Agencies, the regional authorities and ACF, April 2012-

June 2015 “Improve access to water, hygiene and sanitation in the Ger areas in

Ulaanbaatar, Mongolia, through the dissemination of innovative and proven solutions.”

2. Project financed by Swiss Cooperation and ACF, July 2012-February 2015: “Ger EAH

project: Sustainable Water, Sanitation and Hygiene services - “Improving Access to Safe

Water and Improved Sanitation, and Knowledge on Proper Hygiene Practices” for local

population in Songino Khairkhan district – Ulaanbaatar / Mongolia”

3. UNICEF: three consecutive projects between February 2013 and June 2015.

+Research aspect: “Sustainable Sanitation for vulnerable peri-urban population - Operational

Research Study in Mongolia”

The SDC project clearly allowed an extension of the activities which had been tested and

validated in the previous research phase (without being able to talk of ‘scaling up’), with the

construction of 250 eco-toilets from 2013 in the SDC project, which led to the recruitment of

around 2/3 of the employees.

Page 19: URD-TDM-ACF Mongolia WaSH evaluation-final report …€¦ · External evaluation of ACF’s project, “Improve Access to Water, ... methodological tool design, ... Ger Mongolian

19

Final external evaluation of the ACF project Improve access to WASH in the Ger areas of Ulaanbaatar, Mongolia

Groupe URD - TDM | June 2015

Since May 2012, the project has focused primarily on consolidation and handing over the

decentralized WASH solutions to local partners (full EcoSan cycle with supply of eco-toilets +

emptying service + composting site, construction and running of a multi-service water kiosk, a

vast hygiene-promotion campaign mainly aimed at schools, and the design, production and

distribution of containers for transporting water). The handing over of the EcoSan cycle to the

local partner also involved looking for ways to establish financial viability and transform the

beneficiaries into clients.

This second phase also involved developing an advocacy strategy which was implemented from

December 2014.

Only the experiment which consisted of building two treatment units for grey water in two

different households during the first phase was more or less abandoned because the technique

was not appropriate (too complex, not appealing) for the household level.

Relevance: decentralised solutions adapted to the capital’s Ger areas

Access to potable water and improved sanitation, and hygiene promotion are priority needs for

the population. The ACF programme is therefore relevant: there are no other solutions for a

large proportion of the Ger neighbourhoods other than to develop and reinforce appropriate

decentralized solutions. Even though one of the main needs remains unresolved: the absence of

solutions for grey water management (insalubrity and mosquito breeding pools in the summer,

and dangerous patches of ice in the streets in winter), and the lack of public shower facilities12;

the ACF programme provides alternative solutions which are adapted to the context and meet

the needs of the inhabitants of the Ger areas, whether temporary (in relation to the major urban

development plans for the capital) or long-term13. What is more, the solutions that have been

developed are capable of meeting the needs created by the mass arrival of new migrants

(estimated at 30 000 families each year) in the peripheral areas.

Ecological sanitation is also relevant due to the fact that there is a water shortage looming in

Ulaanbaatar, as well as low soil fertility and the need for organic enrichment14.

The relevance of the solutions can also be measured in terms of public demand which is growing

both for subsidized water containers and EcoSan toilets. In addition, those who use the emptying

service have expressed concern about it stopping due to ACF’s departure.

12 Activities for provision of bath houses had to be investigated at the beginning of the definition of pilot options and maybe

integrated to the concept of MSWK, nevertheless it has to be said that even nowadays the legal framework is not allowing

this kind of combination due to hygiene rules. 13 Other priority needs are solid waste management, the lack of charcoal for heating, the lack of schools and kindergartens,

unemployment, the poor state of the roads, and street lighting. 14 A large number of studies have shown that there is demand and a market for compost. For example, ACF took part in an

exhibition on ecology organized in April 2014 by the Ministry for the Environment and the municipal authority. The focus of

the event was primarily the production and use of compost. Apart from the public interest in this type of compost,

numerous gardening companies (20 in number) were also interested.

Page 20: URD-TDM-ACF Mongolia WaSH evaluation-final report …€¦ · External evaluation of ACF’s project, “Improve Access to Water, ... methodological tool design, ... Ger Mongolian

20

Final external evaluation of the ACF project Improve access to WASH in the Ger areas of Ulaanbaatar, Mongolia

Groupe URD - TDM | June 2015

Some solutions have inspired large organizations and have already been replicated: for example,

UNICEF outdoor emptyable pit toilets for rural areas schools.

This success nevertheless needs to be qualified in terms of households’ acceptance of EcoSan

and the emptying service: of the 370 toilets that have been built in connection with the project,

the real number of toilets used today in the spirit of the EcoSan cycle is quite disappointing. As

soon as the service becomes fee-paying, public support tends to be withdrawn.

Another qualification is technical: the project’s eco-toilets are based on the principle of urine

infiltration into the soil, which seems to be the most realistic option (based on previous

experience), and which was retained in the WASH pre-feasibility study for Darkhan.

Nevertheless, it is important to remember that this infiltration represents a source of pollution

depending on the type of soil and the level of the water table, and does not correspond to the

EcoSan approach which campaigns for the complete recycling of nutrients.

� Recommendation for all actors involved in advocacy15 : include public showers in the

advocacy work vis-à-vis institutions (to be included in urban development and facilities

plans for Ger areas)

� Long-term recommendation for WaSH Action: resume action research to improve the

management of grey water (look for partnerships with researchers + volunteers +

funding)16

Effectiveness: an ambitious development project

The specific objective of the “Agencies” project is as follows:

Accompany the provision of proven solutions permitting sustainable access to water and

sanitation for the inhabitants of the Ger areas of Ulaanbaatar

The expected results are detailed below with comments about how well they have been

achieved:

Result 1: Accompany the public

and private players to develop

decentralized water and sanitation

branches with the aim of meeting

the needs of the population of the

Ger areas of Ulaanbaatar.

As was predicted in the interim evaluation, concerning the

dissemination of multi-service water kiosks and EcoSan

(“the development of sectors of activity”), considering the

ambitious nature of the project and the “common”

difficulties encountered (cultural resistance, political

crises with numerous changes in terms of organisation,

responsibilities and personnel), several additional years

would be necessary to achieve sustainable results.

15 not applicable to WaSH Action in the short term, as it has other priorities 16 same remark: not a priority for WaSH Action (except, of course, if funding opportunities open up on this subject)

Page 21: URD-TDM-ACF Mongolia WaSH evaluation-final report …€¦ · External evaluation of ACF’s project, “Improve Access to Water, ... methodological tool design, ... Ger Mongolian

21

Final external evaluation of the ACF project Improve access to WASH in the Ger areas of Ulaanbaatar, Mongolia

Groupe URD - TDM | June 2015

Result 3: The transfer of technical

and economic skills in water,

sanitation and hygiene, to public,

private and association players

contributes to attaining the

Millennium Development Goals

Result broadly achieved: very good level of skills transfer

to a wide range of institutional actors, to USUGs, to

Tolgoit, and to the national ACF staff who are taking over

as a national NGO specialised in WaSH.

Result 2: The institutional

framework of the water, sanitation

and hygiene sector is analyzed and

reinforced on the national level.

It seems reasonable to say that the changes in national

policies and regulations directly related to the ACF project

go beyond the initial objectives which aimed to “analyse”

and “reinforce” the institutional framework.

Impact

Impact on behaviour change at the household/community level

Sanitation:

The interim evaluation stated that beneficiaries of pilot infrastructures were satisfied, proud and

talked about them to their immediate entourage (neighbours, family, and friends). Word of

mouth appeared to be working well. ACF had received requests from several districts of the

capital and from other Provinces.

At the time of the final evaluation, this trend remains true for the (rare) beneficiary households

who effectively use the emptying service. These same users say that they are worried about ACF

leaving as they are afraid that the service will stop one day. What is more, the municipal services

at the khooros level receive requests from households who want to install eco-toilets17.

However, considering the alarming drop in the number of toilets that are emptied (see

paragraph on MonESIC in the chapter “degree of autonomy of local structures”), it should be

noted that though households are happy with / interested in EcoSan toilets, they obviously have

not understood the importance of the emptying service. They are not prepared to pay for this

service which is nevertheless essential for the proper functioning of the system as a whole.

Though pilot solutions were tested technically and were accepted well, there are levers of a

more sociological nature that are missing to establish acceptance of the following idea: someone

has to pay so that the service can function. And, to some extent, this someone has to be the

beneficiary household.

Hygiene:

Cases of water-borne diseases have fallen considerably in the schools targeted by the project,

which is a very positive indicator of impact.

Water:

According to USUG, average water consumption in apartments has fallen. USUG attributes this

phenomenon to the numerous campaigns for saving water (of course, in this area ACF is just one

of several contributors).

17 The social worker from the 7th khoro says that he has recieved around 250 requests since January 2015 and thinks that

requests are going to rise considerably in summer due to the problems with pit-latrines (unpleasant smell, flies).

Page 22: URD-TDM-ACF Mongolia WaSH evaluation-final report …€¦ · External evaluation of ACF’s project, “Improve Access to Water, ... methodological tool design, ... Ger Mongolian

22

Final external evaluation of the ACF project Improve access to WASH in the Ger areas of Ulaanbaatar, Mongolia

Groupe URD - TDM | June 2015

Regarding the two resource centres set up in connection with the project, it is very difficult to

evaluate their real impact (no data about the number of visits to the centres). It is clear that the

library, the museum and the technical training room in the USUG training centre were clearly

improved as part of the ACF project. USUG is very proud and satisfied with this, and this can only

have a positive impact. Those who use it are USUG staff from the whole country, students, school

children and the general public. The small room in school n°106 is more like a meeting room

than a resource centre, but again, even though the real impact of the facilities provided during

the project is impossible to evaluate, despite its very small size, it is already a good thing that it

exists. The publications in the cupboard are not only about hygiene promotion, and are very few

in number. Even though communication through social media about materials, toolkits and key

publications, and the networking of groups via internet will probably have a much greater

impact than this room in an isolated school on dissemination and the reinforcement of health

clubs in schools: the two strategies are complementary.

Resource center at school number 106

Museum, library and technical training room at USUG training center

Page 23: URD-TDM-ACF Mongolia WaSH evaluation-final report …€¦ · External evaluation of ACF’s project, “Improve Access to Water, ... methodological tool design, ... Ger Mongolian

23

Final external evaluation of the ACF project Improve access to WASH in the Ger areas of Ulaanbaatar, Mongolia

Groupe URD - TDM | June 2015

Impact on a larger scale and at the institutional level

It should be underlined that the impact of ACF’s activities at the national level, which were a real

challenge, was remarkable. Here are some of the main changes attributed to ACF and mentioned

by partners directly (non-exhaustive list).

Contribution to changes in national regulations:

- Minimum requirements related to WASH in schools: ACF took part in the working group

organized by the Ministry of Education to define WASH requirements within schools, and

influenced the results significantly (in the process of being officially validated in June

2015). ACF provided training about these new requirements to heads of department

working for the Ministry of Education and Culture from the country’s 21 Provinces

(aimags) at the beginning of 2015 (150 people trained). These people will have begun

implementing concrete changes on returning to their Provinces (see list of changes with

Ariuna)18

- Integration of WaSH tools and training courses developed by ACF and validated by the

official institutions into health courses. These tools are widely disseminated and used.

- Setting up of health clubs in 22 schools (potential for autonomous replication)

- It is probable that (on the basis of ACF’s advocacy work) the Ministry of Finance (or the

schools themselves) will allocate an annual budget to buy soap for every school.

- The standard for compost as a finished product was successfully established, which will

make its commercialization much easier.

- In connection with the working groups created by the Ministry of Health in order to

prepare specific standards for non-collective sanitation19, ACF introduced the EcoSan

toilet model, which was accepted as a technical solution by the members of the working

group.

ACF’s influence will therefore continue after it has left.

Impact of the advocacy strategy

Definition of advocacy, from the ACF International Advocacy Toolkit, 201320

There are many definitions of advocacy but fundamentally, advocacy is a set of organised

activities designed to influence policies and actions of those in power, with the aim of achieving

lasting and positive change. Hence, advocacy requires a good understanding of power

relationships (who makes decisions and at what level, and who has the power to influence

them?) and sets out to effectively engage with and influence the multiple expressions of power

to achieve a desired change.

18 This happened thanks to the UNICEF project – There was nevertheless synergy with the projects implemented by ACF. 19 These standards should allow the standards established in 2008 for latrines and septic tanks to be updated. 20 Excellent guide (clear, simple, comprehensive and operational); it was a very good initiative to have translated it

into Mongolian.

Page 24: URD-TDM-ACF Mongolia WaSH evaluation-final report …€¦ · External evaluation of ACF’s project, “Improve Access to Water, ... methodological tool design, ... Ger Mongolian

24

Final external evaluation of the ACF project Improve access to WASH in the Ger areas of Ulaanbaatar, Mongolia

Groupe URD - TDM | June 2015

Development of the advocacy strategy

1. Draft Advocacy Paper drawn up by ACF headquarters and the WaSH coordinator

2. Support from Bolor LEGIEM, a very good local consultant, from July 2014 on the

basis of the Advocacy Paper and institutional analysis.

3. Exchanges between Bolor and ACF expat staff

4. 2014-2017 advocacy strategy finalised in September 2014

5. Elaboration of an action plan by the heads of project with support/coaching from

the consultant. This step was preceded by substantial and progressive capacity

building (training) in advocacy by the national team.

6. Finalisation of the action plan in the autumn of 2014 (even though it is

continually changing, in order to be able to seize any opportunities that arise).

7. Implementation of the action plan since the end of 2014, but which has been

perturbed by the decision to close the ACF mission and the creation of WaSH

Action in December 2014;

8. In recent months, a connection has begun to be established between the ACF

action plan and WaSH Action. This was not easy, and remains a challenge.

Analysis of the content of the advocacy strategy

Even though the strategy was formalised at the end of 2014, ACF had conducted advocacy

activities21 aimed at the general public and institutions since the beginning of the project which

contributed to ACF’s reputation in Mongolia and led to the remarkable progress made on

regulations already mentioned above.

21 See activity reports for descriptions of all these activities which allowed advocacy messages to be transmitted: world

water days, handwashing, toiltes, conferences, exhibitions, presence in the national media (television, radio), on the

internet and on social media.

Page 25: URD-TDM-ACF Mongolia WaSH evaluation-final report …€¦ · External evaluation of ACF’s project, “Improve Access to Water, ... methodological tool design, ... Ger Mongolian

25

Final external evaluation of the ACF project Improve access to WASH in the Ger areas of Ulaanbaatar, Mongolia

Groupe URD - TDM | June 2015

The objectives of the formalized advocacy strategy were as follows:

-progress on the ‘polluter pays’ law (strategic focus on water)

-standard for compost as a finished product (strategic focus on sanitation)

-HP curriculum is included in the National curriculum of primary schools and kindergartens

(strategic focus on hygiene promotion)

-Budgets for Schools and Kindergartens cover the hygiene management expenses (strategic

focus on hygiene promotion)

Globally, the 2014-2017 advocacy strategy document is relevant and of good quality. There are,

nevertheless, some comments to be made:

1. The objectives of the water section seem very ambitious (not coherent with the

capacities of the team) and are not always very relevant.

2. There is no advocacy in favour of subsidies to support both the emptying service and

the purchase of eco-toilets by residents of the yurt areas. And yet nothing else is likely

to be done on this important issue without ACF.

ACF’s mandate (as a humanitarian organisation whose mission it is to help the most

vulnerable people) raises questions about the fact that the advocacy strategy does not

include a “social” objective concerning access to services (particularly sanitation) for the

poorest sections of society, with subsidy mechanisms for the construction of eco-toilets

and the emptying service. Something similar was done for the cost of water in the SDC

project (see English/ Mongolian brochure)

3. The challenge is for WaSH Action, which has limited funds, to take over the action

plan which was developed from the perspective of an international NGO (ACF).

4. Several crucial issues affecting all three components of WaSH emerged from the

study, “Sanitation in Mongolia: experiences, challenges and recommendations,

January 2015”.

As this study was conducted after the advocacy strategy was finalized, the

recommendations of the study are not included in ACF Mongolia’s advocacy strategy.

The only activity that was carried out was the awareness-raising campaign aimed at the

general public and local authorities on decentralised sanitation (cf. Facebook).

The study (which is in the process of being published) was presented during the last

WaSH Forum organized by ACF (90 participants) and is going to be sent in the coming

months to the 60 members of parliament.

� Recommendation for all actors involved in advocacy (and, if possible, for WaSH

Action eventually, if they obtain funding): optimize the advocacy potential of this study

(dissemination, organisation of workshops, etc.).

� Recommendation for WaSH Action and all actors involved in advocacy: it would be

a good idea to include a specific section on developing the EcoSan sector in the advocacy

strategy: even though, ideally, the emptying service could eventually be financially

viable, the scaling up process and the social marketing campaigns that are required need

to be subsidised.

Page 26: URD-TDM-ACF Mongolia WaSH evaluation-final report …€¦ · External evaluation of ACF’s project, “Improve Access to Water, ... methodological tool design, ... Ger Mongolian

26

Final external evaluation of the ACF project Improve access to WASH in the Ger areas of Ulaanbaatar, Mongolia

Groupe URD - TDM | June 2015

Analysis of the advocacy strategy process

- Action plan developed exclusively by the Mongolian heads of project (which, in

retrospect, appears even more important due to ACF’s decision to withdraw and the

creation of WaSH Action). This process was chosen (rather than the development of an

action plan steered by the expat staff) with the aim of building the capacity of national

staff and no doubt also because ACF’s withdrawal was a possibility (even though the

decision had not been made officially).

- Long development process, but the time taken was essential to develop a strategy that

was adapted to the context.

- The implementation of the advocacy action plan only began at the beginning of 2015: it

would be a great shame (and this would raise questions about effectiveness) if all this

solid work, built on ACF’s years of experience in Mongolia, was not pursued.

- But, as stated above, the advocacy action plan was developed by staff working for ACF

(not with a view to becoming WaSH Action) : in its current state, it is too ambitious, and

as such it is not adapted to the capacities of a new local NGO with limited resources

(indeed, no resources for the time being). In any case, it is unlikely that it could be

implemented at the planned rhythm, but it may be possible gradually, little by little,

using WaSH Action’s solid network and by beginning with the “easiest” advocacy issues.

� Recommendation for WaSH Action: pursuing the implementation of the advocacy

action plan should not be an absolute priority for WaSH Action in the months ahead; this

should remain in the “background” (with activities which do not require any budget)

because it constitutes the mission (vision) of WaSH Action and allows it to remain visible

and involved vis-à-vis key decision-makers. For this, WaSH Action will need to adapt the

strategy and the action plan based on their capacity (having several scenarios at the

ready depending on their resources).

� For the Water Agencies and other potential financial partners: for advocacy to

continue, it is NECESSARY for WaSH Action to receive financial support.

Looking ahead, we unfortunately feel that the collaboration between Artois-Picardie Water

Agency and the Mongolian river basins authorities (notably on integrated watershed

management and environmental licence fees) appears to be compromised, considering the

reduced resources of the Agencies for international cooperation programmes.

� Adopt a clearer position about the possibility of pursuing the collaboration with the

Mongolian watershed authorities.

Page 27: URD-TDM-ACF Mongolia WaSH evaluation-final report …€¦ · External evaluation of ACF’s project, “Improve Access to Water, ... methodological tool design, ... Ger Mongolian

27

Final external evaluation of the ACF project Improve access to WASH in the Ger areas of Ulaanbaatar, Mongolia

Groupe URD - TDM | June 2015

Degree of autonomy of local structures to take over kiosks, emptying services, and composting sites

Lack of anticipation in ACF’s exit strategy

Even though ACF’s strategic decision to withdraw from Mongolia is perfectly justified22 :

1. The withdrawal is too early considering the nature and state of progress of the project:

the majority of WaSH solutions that have been developed are still at the pilot stage, and

have not yet reached their full potential in terms of dissemination and replication. It

would be a shame to abandon them in their current state.

2. The decision to leave was made late (in November 2014), which is a pity, as it was not a

surprise (questions were clearly already being asked a year and a half ago) and an earlier

decision would have made it possible to envisage local capacity building with the main

partner that was taking over (WaSH Action) more serenely.

ACF knew that the chances of receiving funding for new projects were slim (withdrawal of

international donors)… and even though they have intensified their efforts to set up projects

with WaSH Action in the last six months, in the end, what the WaSH Action team lacks is “the

ability to put together projects that can be accepted”: ACF should pursue its support of WaSH

Action in looking for and putting together projects so that they are equipped in relation to

international donors23.

As for the EcoSan sector, in order for scaling up to be able to happen, there needs to be progress

in the following areas:

-advocacy for subsidies to help individuals buy toilets

-advocacy for subsidies for the emptying service (which, like the sale of water at multi-service

kiosks, is similar to a public service)

-promotion / sale of the compost produced

-continuous improvement of toilet design

-using “local champions” to re-launch the process (construction, restoring the functioning of the

service)

Today, ACF still has a central role in activities which aim for replication: the khoroos send the

hundreds of people interested in EcoSan toilets to ACF rather than to construction companies.

The advertising for eco-toilets is done by ACF rather than by MonESIC (who, in any case, do not

have the time or the resources to take care of this kind of promotion and to deal with the people

who are interested), and ACF is supporting the emptying service financially (until the end of

June 2015).

22 Extract from ACF activity report for year 5: “This decision is based on in-depth analysis of the humanitarian situation in

Mongolia, ACF’s international operational context and the weak possibility of funding for new projects. ACF France must

also face multiple humanitarian crises at the global level which will require significant amounts of human and financial

resources. Mongolia has managed to recovery from the dzuds which affected the country in the 2000s, and although WASH

conditions remain worrying, the general situation has improved immensely”. 23 When the evaluation report was being finalised, WaSH Action were on the point of signing a MoU with ACF and the USTB

so that they would receive technical support (and perhaps financial support).

Page 28: URD-TDM-ACF Mongolia WaSH evaluation-final report …€¦ · External evaluation of ACF’s project, “Improve Access to Water, ... methodological tool design, ... Ger Mongolian

28

Final external evaluation of the ACF project Improve access to WASH in the Ger areas of Ulaanbaatar, Mongolia

Groupe URD - TDM | June 2015

What is more, the unsubsidised cost of the toilets seems prohibitive for households in the yurt

areas. This point needs to be qualified though, as households are capable of paying almost 4

million Tugrik (around 1 700 Euros) per family for new telephones to the detriment of their

toilets which are left in the same state, that is to say, without any improvements for a decade24. It

is clear that spending on “sanitation and health” is not among households’ priorities. This non-

demand for sanitation is a barrier for the development of ecological and sustainable solutions.

Demand needs to be generated by working more on Information, Education and Communication

(IEC) and increasing communication campaigns. If households do not see what they can get out

of it, demand will be limited to those who are “naturally” conscious of ecological issues.

Despite this, significant investment has been made in training and building the capacity of

partners since the decision was made, which has compensated somewhat for the lack of

anticipation.

� Recommendation pour ACF: the next time ACF implements a project which is on the

margins of its mandate, it would be a good idea to anticipate its exit by working with

local partners sufficiently early, in order to be able to accompany them (+ capacity

building) at a gradually decreasing level until the exit.

Multi-service water kiosk managed by Tolgoit for almost three years

Regarding the multi-service kiosk, the quality of Tolgoit’s commitment in running the service

should be underlined, and the fact that it continued despite problems (the economic crisis which

affected household budgets and made it difficult to rent spaces to artisans, and compromised its

financial stability).

It is therefore very probable that Tolgoit will continue to manage the kiosk in the long term. In

addition to the engagement of MonESIC, several factors will help to establish this long-term

stability:

- The training-coaching (development of the business-plan) carried out by MIRIM was

very useful to the extent that Tolgoit already was involved in the financial management

of the kiosk, and that the business plan was developed on the basis of everyday

experience. Another key factor to take into consideration is that the President25 of the

organisation (head of Tolgoit) already has solid grounding in financial management

(previous experience and personal experience).

- ACF was successful in campaigning the Water Service Regulatory Commission to

exonerate Tolgoit from paying tax on water sales. This non-profit NGO provides a public

rather than commercial service.

24 It is estimated that 4 million have been invested on mobile phones in the last 10 years (source: ACF analysis).

25 She has been the President for one and a half years.

Page 29: URD-TDM-ACF Mongolia WaSH evaluation-final report …€¦ · External evaluation of ACF’s project, “Improve Access to Water, ... methodological tool design, ... Ger Mongolian

29

Final external evaluation of the ACF project Improve access to WASH in the Ger areas of Ulaanbaatar, Mongolia

Groupe URD - TDM | June 2015

TOLGOIT headquarter multiservice waterkiosk

The evaluation team notes that success and long-term stability depends on choosing a key

partner, which was the case with Tolgoit. It should also be underlined that the time and

commitment of the members of the association, and particularly the President (since 1,5 year), is

voluntary, and is essential to make the service work. This needs to be taken into account in

terms of the replicability of the model.

It should also be underlined that the EcoSan toile at the kiosk site (which is used by the

caretaker’s family) was changed to a pit latrine about a year ago (a pit was dug in the space for

the bucket) and is neither cleaned nor maintained, which is a very negative advert for EcoSan.

Tolgoit has no doubt not embraced the idea of EcoSan enough, or it was not a priority for them.

Considering Tolgoit’s recent enthusiasm for planting plants and bushes (+ project to

commercialise plants, which is a very profitable activity in Mongolia), as well as the association’s

dynamism (and its audience in the community), it would have been/would be interesting for the

association to adopt the EcoSan approach. We feel that the recommendations from the interim

evaluation are still worth exploring: selling small quantities of compost and promoting the

EcoSan approach at the kiosk.

In the medium to long term it would also be interesting to test how the grey water produced by

the artisans is re-used (this currently only concerns the hairdresser because the washing

machine service has been replaced by a shoemaker).

Major concern about the sustainability of MonESIC taking over the ecological

sanitation service

The real investment and motivation of the people from MonESIC who were met during the

evaluation should be underlined. The composting technique and the final result of the mature

compost are excellent. One week after the compost pile was established in June 2015, the

temperature that was recorded in the pile on 8 June 2015 was 55°C. Congratulations for these

results.

Compost processing, June 2015

Page 30: URD-TDM-ACF Mongolia WaSH evaluation-final report …€¦ · External evaluation of ACF’s project, “Improve Access to Water, ... methodological tool design, ... Ger Mongolian

30

Final external evaluation of the ACF project Improve access to WASH in the Ger areas of Ulaanbaatar, Mongolia

Groupe URD - TDM | June 2015

Comments about the analysis of the compost

The analyses in September-October 2014 (4 weeks after the compost pile was put in place)

showed that there was a very high quality of product during the composting : very little EColi

(below the EPA threshold), Giardia lamblia in one sample and grysporidium in another. There

was no Strongyloides stercoralis (from the helminth egg family).

The analyses in June 2015 using samples taken from the same mature compost showed that all

the pathogens had disappeared (Giardia, Grysporidium), but a helminth egg was detected,

despite the fact that it had not appeared in the analysis at the beginning of the composting

process.

Helminth eggs can

sometimes end up in

the ground. They are

the most resistant

parasite -> an excellent

composting process is

needed: 50-60°C° for 7

consecutive days

evenly spread

throughout the pile

(without the

temperature falling

back down).

Several hypotheses are possible: either 1) the composting process did not maintain a sufficiently

high temperature for 7 consecutive days in the whole pile, which would explain why a helminth

egg was found, or 2) the presence of this helminth egg comes from the way of taking samples

(the contamination took place when the sample was being taken), or 3) the helminth egg was

present in the soil (which again raises questions about the sample taking protocol).

The finished product includes 87% of organic matter, which is excellent. The pH was already

basic (8.2): mineralisation was beginning to take place. The compost is still very good as a soil

enricher, but it will be important to check that the pH stabilizes and does not increase too much.

During the composting process, the compost goes through an acidic phase, then neutral, then

basic. When the pH result is acidic, the composting process is not finished. When the pH result is

too basic, this is like going past the optimal “use-by date”26.

26 Comment: in the sanitation sector there is often a focus on composting as a sanitising process, but not on optimising the

compost for its agronomic value with a view to its reuse for agriculture.

Page 31: URD-TDM-ACF Mongolia WaSH evaluation-final report …€¦ · External evaluation of ACF’s project, “Improve Access to Water, ... methodological tool design, ... Ger Mongolian

31

Final external evaluation of the ACF project Improve access to WASH in the Ger areas of Ulaanbaatar, Mongolia

Groupe URD - TDM | June 2015

� Recommendation for WaSH Action / MonESIC: the protocol for taking samples to

analyse the summer compost could probably be improved (how and when to take

samples).27

MonESIC still does not have a licence (in terms of being an emptying company), but this does not

appear to pose any problems.

Since the handover to MonESIC, ACF has mainly provided the NGO with technical support on the

compost production and management. However, there does not appear to have been enough

support in terms of social marketing or optimising the emptying service. The question of long-

term stability after ACF left was raised at each meeting between MonESIC and ACF (ACF

continues to fund the NGO’s running costs).

ACF leaving will have an obvious negative effect on MonESIC, who clearly will not be able to

meet all the challenges involved without external support.

Today, around 280 EcoSan toilets are effectively usable.

Ecotoilet from summer 2014, SKHD

Ecotoilet from 2011, SKHD

27 What is the protocol for taking samples? For example, was the spade clean (was the sample of mature compost taken

using a tool that was used in making the compost pile, and therefore had been in contact with fresh faeces)?

Do you have an initial analysis of each substance that goes into the compost: sawdust + a little earth (soil) due to the fact

that the sawdust is stored on the ground + faeces + other?? If these analyses are not carried out for each sample, then it is

impossible to know where these helminth eggs came from. It would be good to do this for the next analyses.

Page 32: URD-TDM-ACF Mongolia WaSH evaluation-final report …€¦ · External evaluation of ACF’s project, “Improve Access to Water, ... methodological tool design, ... Ger Mongolian

32

Final external evaluation of the ACF project Improve access to WASH in the Ger areas of Ulaanbaatar, Mongolia

Groupe URD - TDM | June 2015

When ACF was present, two people were responsible for the emptying round, but they were not

employed full time on this activity to the extent that the round was only carried out once a

quarter and lasted about twenty days.

The handover to MonESIC, when the service was ready to operate at full capacity, actually led to

a reduction of the service, and some hesitation. The emptying service and the composting are

still done manually, still on the same scale as for the pilot project.

Several worrying issues:

-Reduction of the number of toilets emptied: since people have had to pay for the emptying

service, most have returned to their old latrines (or have transformed their EcoSan toilet into a

traditional latrine);

-The service is managed in a somewhat passive way, whereas it should absolutely be managed

proactively;

-The eco toilet on the composting site is not looked after (broken ventilation, dirty) and is not

used much;

-A theft in the staff premises at the composting site.

Raw data:

Quantity of compost produced in August 2014: around 1 tonne (very little!), which corresponds

to about 150 toilets effectively used and emptied.

Since September 2014, around 40 buckets have been collected (as could be seen during the visit

to the composting site in June 2015, on the day when the compost pile was put in place), which

corresponds to roughly 2 to 2.5 tonnes of matter at the most28.

According to MonESIC, since January 2015 around 20 toilets have been emptied (3 rounds); ¾ of

the households paid for the service.

Since September 2014, around 40 buckets have been collected

28 By considering an average family = 4 people

Page 33: URD-TDM-ACF Mongolia WaSH evaluation-final report …€¦ · External evaluation of ACF’s project, “Improve Access to Water, ... methodological tool design, ... Ger Mongolian

33

Final external evaluation of the ACF project Improve access to WASH in the Ger areas of Ulaanbaatar, Mongolia

Groupe URD - TDM | June 2015

Fragilities:

-MonESIC does not have any human resources of its own (to use an image, the NGO is like an

empty shell);

-The financial support for running the service from ACF ends in June 2015;

-Only one person is fully competent in managing the composting process (a former ACF member

of staff), who is employed as a day labourer;

-The urine separation toilets are not used properly: the faeces that are collected are extremely

liquid.

Faeces that are collected are extremely liquid

Analysis:

The transition from being “beneficiaries” as part of a humanitarian project to “clients” of an

emptying service led to a very large number of households abandoning the service: indeed, it is

common that the shift from a free sanitation service to a fee-paying one does not work (unless

there are extremely solid support measures in terms of social marketing and communication).

Ideally, the households should never have been considered as “beneficiaries” of a free service.

They should have paid a fee for the service from the start.

There was no joint construction of demand or solutions with the households and the other

partners (which would have increased ownership and helped to establish long-term stability).

Even though the idea of developing the business is important in itself, the training course

provided by MIRIM to MonESIC was not appropriate.

The business plan which resulted from this coaching proposed to change the strategy again: the

new idea was to consider eco-toilet owners as suppliers of a resource (faeces) and therefore the

service should remain free. This change was very sudden and created confusion within the

MonESIC team (though this strategy is not absurd in itself29, it should have been worked on

much more in advance). Today this business plan is very attractive on paper, but remains

29 EcoSan systems like this exist in the world (in Kabul for example).

Page 34: URD-TDM-ACF Mongolia WaSH evaluation-final report …€¦ · External evaluation of ACF’s project, “Improve Access to Water, ... methodological tool design, ... Ger Mongolian

34

Final external evaluation of the ACF project Improve access to WASH in the Ger areas of Ulaanbaatar, Mongolia

Groupe URD - TDM | June 2015

virtual: there is a huge gap between the current situation with the emptying service run by

MonESIC and the business plan. The accompaniment from MIRIM did not make it possible to

plan in concrete terms the steps that are needed to establish a model of this kind.

MonESIC had more urgent needs:

- support in operational management to optimize the emptying service (this is what ACF

tried to provide in the last year and a half);

- help in consolidating (stimulating and maintaining) social demand, that is the will to pay

for the emptying service: also done by ACF, but insufficiently.

- Scaling up of the EcoSan cycle in the form of an increased number of users, which

requires the engagement of decision-makers at the highest level: ACF worked towards

this with the study “Sanitation in Mongolia: experiences, challenges and

recommendations”

There are different levels of targets to create or maintain demand (households / community

relays / institutions, decision-makers / the general public). All these different levels are

complementary. Obviously the government has a crucial role to play in order to achieve

sustainable change on a large scale.

The objective of social engineering (which usually takes the form of social marketing) is not that

households want subsidised (free) toilets, but rather that they understand why it is important to

have a toilet and the need to pay for the service.

Page 35: URD-TDM-ACF Mongolia WaSH evaluation-final report …€¦ · External evaluation of ACF’s project, “Improve Access to Water, ... methodological tool design, ... Ger Mongolian

35

Final external evaluation of the ACF project Improve access to WASH in the Ger areas of Ulaanbaatar, Mongolia

Groupe URD - TDM | June 2015

Focus on the EcoSan projects:

Sanitation is never a priority for households (a fundamental difference compared to access to

water). Transport is a recurring problem in this kind of project: experiences in other projects

show how important it is in terms of efficiency, and consequently in terms of viability/ long-

term stability to make sure that collection points are close to each other, to the treatment sites30,

and to the compost sales sites. The EcoSan service in Ulaanbaatar will need to eventually look

into this issue.

Even though the EcoSan approach is perfectly relevant, it is important to say that making a

success of an EcoSan project is always a major challenge; as such, it was completely relevant to

invest in the research aspect even though the linking up of the research and the programmes

was disappointing and did not lead to the expected results. If this EcoSan cycle experiment in

Mongolia becomes firmly established (dissemination, scaling-up and consolidation), it will gain

international recognition! Taking the time to fill in a SuSana project file, and disseminating it

globally, could boost alliances and partnerships with the EcoSan service project in Ulaanbaatar.

EXPERIENCES OF URBAN/PERI-URBAN ECOSAN SERVICE PROJECTS

WHAT LESSONS FOR THE MONGOLIAN CAPITAL?

SOURCES:

HTTP://WWW.SUSANA.ORG/EN/RESOURCES/CASE-STUDIES/DETAILS/1049 (FOR ERDOS PROJECT)

HTTP://WWW.SUSANA.ORG/EN/RESOURCES/CASE-STUDIES/DETAILS/791 (FOR SOUTH AFRICA PROJECT)

HTTP://WWW.SUSANA.ORG/EN/RESOURCES/CASE-STUDIES/DETAILS/84 (FOR OUAGADOUGOU PROJECT)

Globally, there have been very few experiments in urban and peri-urban “ecosan” services. We

can refer to three of these, and the lessons that have been drawn from them: (i) the ERDOS

project in Inner Mongolia in China; (ii) the e-Thekwini project, Durban municipality, South

Africa, and (iii) the EU ecosan project in Ouagadougou in Burkina Faso. All three have provided

lessons that are relevant to the Mongolian experience.

In each of these cases, urine separation toilets were installed and an external service was set up

to manage the faeces and/or the urine. In two cases (China and Burkina Faso), the service was

paying from the start; in the South African case, the initial premise was that the families would

empty the toilets themselves, but in practice they were not very willing to do so. As a result, local

micro-entrepreneurs were trained by the municipal authority to do the emptying, for which the

families paid a fee. The experience in Burkina Faso showed that the households were not very

willing to pay for the service (almost 80% of costs not recovered) and that there was not

sufficient demand from farmers for ecosan fertilizer, which meant that the municipal authority

had to subsidise the service at the end of the project, taking over from the international donor. In

China, technical problems in terms of toilet use (large-scale implementation from the beginning),

a lack of openings for ecosan fertilizers in agriculture and a low level of involvement of users at

the beginning led to households becoming demotivated and making the radical change from dry

toilets to WCs.

These different experiences, which are far from being negative, show that the question of an

ecosan service and even the use of dry toilets on a large scale is a process which needs to be

thought through in advance with interlocutors who are all equally concerned that the project

will be sustainable: users, local authorities, farmers, educators, technicians and companies. If

30 This can lead to the introduction of a collection point or a relay treatment point (buffer site).

Page 36: URD-TDM-ACF Mongolia WaSH evaluation-final report …€¦ · External evaluation of ACF’s project, “Improve Access to Water, ... methodological tool design, ... Ger Mongolian

36

Final external evaluation of the ACF project Improve access to WASH in the Ger areas of Ulaanbaatar, Mongolia

Groupe URD - TDM | June 2015

these connections have not been made from the beginning, the experiences in South Africa and

Burkina Faso show that it is possible to rectify the situation, but this implies spending more time

than initially planned for the ecosan service solution to be stable in the long term!

The positive points which have already been established by the Mongolian experience are as

follows: (i) the interface with users is accepted and (ii) the pilot phase and the test phase before

wider dissemination were done very well. However, to prevent the EcoSan service in

Ulaanbaatar from coming to an end, based on analysis of these 3 EcoSan services, we

recommend the following actions be taken to influence different factors affecting the long-

term stability of the service:

� Subsidise the functioning of the service immediately (fully or in part), to ensure that he

dry toilets are emptied and avoid them being abandoned. It is important to understand that if

the service is paying for households, it is wise to estimate the rate of non-recovery in the

service’s business plan at least 40%.

� Involving the local authorities is essential for the long-term stability of the service. The

government needs to support these projects and help to develop regulations to achieve a wider

coverage, which will then allow the significant scaling-up which goes hand in hand with a more

stable service in the long-term.

� Reinforce links with agriculture so that this outlet is accepted and easy to set up (within a

radius of 20km at most for reuse), notably through the sale of fertilisers from the dry toilets.

� To revitalise the process and improve the image of using ecosan toilets: (i) use local

champions, in the form of model users and influential people in the neighbourhood, and even

people who are well-known nationally to boost the campaign; (ii) in connection with the well-

known people nationally, carry out IEC activities (media, street theatre, and events in schools,

for example).

Recommendations:

� For ACF: Having invested so much (including 400 000€ on the PhD), it would really be

a shame to let the EcoSan experience die as is happening now: ACF should invest one last

time in the EcoSan area of activity and support social project management activities.

� For ACF, WaSH Action/MonESIC and all other actors involved in advocacy: it is

urgent to find subsidies for the construction of EcoSan toilets and social marketing

campaigns, without which scaling up is very unlikely to happen31.

� For MonESIC: Have an immediate survival strategy: be more proactive and develop

niche markets by optimising use of the equipment.

� For WaSH Action and MonESIC: Work in an integrated manner, as this will be to

everyone’s benefit: EcoSan is an innovative approach which could help to make WASH

Action known, but without the expertise and support of WASH Action, there is a danger

that the EcoSan adventure will come to an end.

31 If all goes well, the emptying service could eventually be financially viable, but it will probably need financial support

immediately to replace ACF’s subsidies which will stop and the end of June 2015, and until MonESIC has developed niche

markets to balance the service financially.

Page 37: URD-TDM-ACF Mongolia WaSH evaluation-final report …€¦ · External evaluation of ACF’s project, “Improve Access to Water, ... methodological tool design, ... Ger Mongolian

37

Final external evaluation of the ACF project Improve access to WASH in the Ger areas of Ulaanbaatar, Mongolia

Groupe URD - TDM | June 2015

WaSH Action, a young NGO that is still fragile, but has a lot of potential

The next six months will be decisive. This assertion is based on:

-informal discussions with staff (“6 months maximum”)32

-rental of a small office from CARITAS for WaSH Action (cost covered by ACF until the end of

2015)

-the Mongolian social protection system: 50% of the salary is covered for 3 months + an end of

contract bonus of one month’s salary which is given by the employer.

ACF Mongolia’s staff benefited from numerous training courses and capacity building activities

(in Ulaanbaatar, Paris, Madrid…) over several years.

What is more, when ACF made the decision to leave, capacity building activities were intensified,

notably with two national consultants:

- Bolor LEGIEM (who has already been mentioned in the paragraph in which the advocacy

strategy was analysed): her 2nd contract focused on capacity building (writing propels,

developing a Wash Action strategy, fundraising, etc.). Wash Action will receive support

from Bolor after ACF has left based on a tacit contract between ACF, Bolor and WaSH

Action (10 days for advocacy and 3 training courses on fundraising, external

communication and M&E).

- Development Solutions: ten workshops (around 120 hours in total) since February 2015.

A very positive point is that the company offered to pursue its coaching on a voluntary

basis until November 2015, which indeed seems essential, because it is when WaSH

Action will have to face reality on its own that it will have the greatest need for coaching.

SWOT analysis of WaSH Action in June 2015

Strengths

Motivation

Competencies, qualifications

Conscious of their real capacity and weaknesses

Team spirit and commitment

Excellent network (links and partnerships with

numerous organisations)

Expertise and knowledge of sellable

decentralised WaSH solutions

The only national WaSH NGO

Weaknesses

Lack of financial visibility

Key competencies still need to be acquired and

put into practice (see map of competencies

below), notably in organizational, administrative

and financial management

Opportunities

The existence of obvious and enormous WaSH

needs

The existence of a market (around 150

consultancy companies in Mongolia in the

environment sector, but integrated

competencies in WaSH are extremely rare)

Threats

Withdrawal of international donors and closure

of certain large WaSH projects

The sector is still not sufficiently prioritized at

the national level, and at the community level

The attraction of other organisations: risk of

losing the most competent members of WaSH

Action

32 confirmed by the final report of Development Solutions

Page 38: URD-TDM-ACF Mongolia WaSH evaluation-final report …€¦ · External evaluation of ACF’s project, “Improve Access to Water, ... methodological tool design, ... Ger Mongolian

38

Final external evaluation of the ACF project Improve access to WASH in the Ger areas of Ulaanbaatar, Mongolia

Groupe URD - TDM | June 2015

ACF national team, WaSH Action of Mongolia members

WaSH Action map of competencies

The years of experience with ACF combined with the intensive training courses received in

recent months have allowed staff to be trained with a view to acquiring essential competencies.

The table below summarises the competencies needed. Grey boxes signify gaps (the greyer the

box, the less they have of this competence).

Essential

competencies for

WaSH Action as an

autonomous local

NGO

ACF experience and

internal training

Support from Bolor

Legem

Support from

Development

Solutions

WaSH technical

competencies

Excellent level33

English 2 people speak fluent

English34

Networking as WASH

Action

Very good network at the national level, but

new to the international level, improving little

by little

Advocacy Beginner level

Project (cycle) management

Strategic planning The aim is to organise their ideas within the

academic frameworks of project management

and the reality of financial opportunities :

more or less beginner level (lack of

experience)

Communication

Proposal writing

Fundraising Beginner level

33 Though some expressed the need to increase their technical competencies to achieve the level of experts. 34 But this figure depends on the commitment of the people who are effectively going to work for WaSH Action in the

months ahead.

Page 39: URD-TDM-ACF Mongolia WaSH evaluation-final report …€¦ · External evaluation of ACF’s project, “Improve Access to Water, ... methodological tool design, ... Ger Mongolian

39

Final external evaluation of the ACF project Improve access to WASH in the Ger areas of Ulaanbaatar, Mongolia

Groupe URD - TDM | June 2015

Budget management

Establishment of a

budget plan

The Programme

Managers currently

have theoretical

competencies (no

practical experience)

but the former ACF

Financial Manager

has very good

capacity

Financial procedures

Logistical procedures

Management /

organisation / Senior

management35

Funding and development prospects at the time of ACF’s departure

-Several studies have already been carried out by WaSH Action (water monitoring tool,

menstrual and personal hygiene tools36)

-Ideas for project type funding:

- with Asia Foundation who propose to include WaSH Action in their advocacy activities

(UNICEF funding), which would be useful in terms of visibility even though it is not a

large contract (around 2 weeks, 800 USD in total).

- New Zealand small grant in July 2015

- The balance from the ACF PhD?

- UNICEF in September/October: pushed by the Ministry of Education who wanted WaSH

Action to pursue their work on the minimum WaSH requirements in schools, UNICEF

finally decided (at the end of June 2015) to evaluate WaSH Action’s capacity in August, to

continue working together.

- UNICEF Mongolia are planning to make WaSH a major part of their strategy for the next

6 years, and at the same time they seem to have realized the importance of reinforcing

the capacity of local actors in the current context of diminishing international funds.

-A few ideas of projects for consultants, such as facilitating the training module for USUG (WHO

funding) in the months ahead, or for a number of Ministries.

From ACF to WaSH Action

One of the main challenges and projects to be carried out by WaSH Action will be to adapt

(reduce) ACF’s strategies (for action and advocacy) so that they correspond to their real capacity

and priorities.

� Recommendations for WaSH Action :

In the SHORT TERM:

1. Survive!

This means investing in what are perhaps not the most relevant activities for the WaSH sector,

but in activities which will be the most lucrative, and will give WaSH Action some financial

visibility. From a tactical point of view, this will mean using the competencies and materials

which can already be mobilised (produce that is ready to sell), and focusing on areas in which

WaSH Action already has recognition and visibility.

35 Who is responsible for what? Human resource management, etc. 36 These are the first tools of this kind in Mongolia.

Page 40: URD-TDM-ACF Mongolia WaSH evaluation-final report …€¦ · External evaluation of ACF’s project, “Improve Access to Water, ... methodological tool design, ... Ger Mongolian

40

Final external evaluation of the ACF project Improve access to WASH in the Ger areas of Ulaanbaatar, Mongolia

Groupe URD - TDM | June 2015

- Have an action plan: who does what in the coming weeks, depending on the strategic

priorities and tactics which have been established.

- Adapt the advocacy strategy to continue at a minimum level (see detailed

recommendation in the advocacy paragraph)

- Be proactive to increase visibility within the international community (INGO, UN, World

Bank, Asian Development Bank, etc.)

- Give ACF and other partners news regularly. In the medium term (after the survival

phase), it would be useful to regularly analyse the WaSH sector and to send this analysis

to partners (and make it available on the website). The aim here would be to remain

visible and become an important local actor in terms of analysis.

2. Support the EcoSan sector

- Contribute to advocacy in favour of subsidies for EcoSan toilets and the emptying

service;

- Draw up a SuSanA file to share lessons learned, which could help to improve visibility at

the international level, and fund-raising.

In the LONG TERM:

3. Promote the WaSH sector in Mongolia, in line with what was done with ACF

- Optimise the 2014-2017 WaSH advocacy strategy, as well as the advocacy potential of

the study “sanitation in Mongolia: experiences, challenges and recommendations”

(dissemination, organisation of workshops, etc.)

- Improve the versatility of the WaSH Action staff (water, sanitation and hygiene);

- Develop a virtual Resource Centre (website) for the Mongolian WaSH sector;

- Resume action research to improve grey water management (look for partnerships with

researchers + volunteers + funding)37.

� Recommendation for ACF: Maintain strong links with WaSH Action, in the form of a

partnership, rather than a simple handover of the facilities38, as this would be to

everyone’s advantage.

Advantages for WaSH Action: 1/ getting advice, and even re-reading of the next proposals (in

English) which are going to be crucially important for the long-term stability of the structure; 2/

being able to show that they are in a partnership with a globally recognised international NGO

would be helpful;

Advantages for ACF: 1/ having invested a great deal in this project, it would be useful to monitor

its impact and future in terms of lessons learned and knowledge management; 2/ continuing to

support the project, even at a distance, is a way of optimising investments; 3/ keeping a local

partner in a country where there is a risk of earthquakes and Dzuud is good strategy for a

humanitarian organisation.

� Recommendation for ACF: Help WaSH Action to write a SuSanA file, underlining all the

research work that has been carried out (PhD and Masters).

37 same remark : this is not a priority for WaSH Action (except, of course, if funding opportunities open up for this subject) 38 This seems to be in the process of being signed (including USTB) at the time of writing this report.

Page 41: URD-TDM-ACF Mongolia WaSH evaluation-final report …€¦ · External evaluation of ACF’s project, “Improve Access to Water, ... methodological tool design, ... Ger Mongolian

41

Final external evaluation of the ACF project Improve access to WASH in the Ger areas of Ulaanbaatar, Mongolia

Groupe URD - TDM | June 2015

Conclusion and recommendations

Conclusion

To conclude, the evaluators would like to highlight both the relevance of the overall project and

its completely innovative nature in several respects, which logically leads us to encourage the

consolidation and dissemination of the WASH models that have been developed.

This has been a very ambitious project, which has achieved excellent results, notably due to the

complementary nature of the projects by the 3 main financial partners (Water Agencies, Swiss

Cooperation and UNICEF).

At the time of ACF’s withdrawal, despite the fact that solutions that have been developed are

very relevant, are tried and tested and can be replicated, there has not yet been any scaling-up of

the decentralized sanitation solutions (EcoSan cycle) or for the multi-service water kiosk.

The difficulties encountered on the EcoSan section are not surprising considering the challenge

that this kind of innovative project represents and the time that they require. What is more, it is

not realistic to think that this type of sector can become financially balanced within the space of

a few years (even though it is possible in theory).

The results in terms of the dissemination and sustainability of hygiene promotion messages

have been remarkable.

The work of the ACF team in building its reputation and visibility should also be underlined. This

constitutes a very favourable environment for the creation of WaSH Action. Even though there is

a risk that the initial months will be difficult and decisive, the new (and only) Mongolian NGO

specialised in WaSH has all the potential and competencies needed to succeed in this bold

gamble.

From a strategic point of view as regards Mongolia, the development of projects aiming to

improve health conditions in other provinces and secondary urban centres should be

encouraged, as should the advocacy for an improvement in access to WASH in rural areas, in so

far as this type of action could help to curb the mass rural exodus, which could lead to an urban

crisis, at the same time as the disappearance of the traditional nomadic culture.

Recap of detailed recommendations in the previous chapters:

Recommendations for ACF:

FOR A FUTURE PROJECT…

The next time ACF implements a project which is on the margins of its mandate, it would be a

good idea to anticipate its exit by working with local partners sufficiently early, in order to be

able to accompany them (+ capacity building) at a gradually decreasing level up to the exit.

IN CONNECTION WITH THIS PROJECT:

- Having invested so much (including 400 000€ on the PhD), it would really be a shame to

let the EcoSan experience die as is happening now : ACF should invest one last time in

Page 42: URD-TDM-ACF Mongolia WaSH evaluation-final report …€¦ · External evaluation of ACF’s project, “Improve Access to Water, ... methodological tool design, ... Ger Mongolian

42

Final external evaluation of the ACF project Improve access to WASH in the Ger areas of Ulaanbaatar, Mongolia

Groupe URD - TDM | June 2015

the EcoSan area of activity and support social project management activities (IEC) +

capacity building for WaSH Action in setting up projects for international donors.

- Maintain strong links with WaSH Action, in the form of a partnership, rather than a

simple handover of the facilities39, as this would be to everyone’s advantage.

Advantages for WaSH Action: 1/ getting advice, and even re-reading of the next proposals (in

English) which are going to be crucially important for the long-term stability of the structure; 2/

being able to show that they are in a partnership with a globally recognised international NGO

would be helpful;

Advantages for ACF: 1/ having invested a great deal in this project, it would be useful to monitor

its impact and future in terms of lessons learned and knowledge management; 2/ continuing to

support the project, even at a distance, is a way of optimising investments; 3/ keeping a local

partner in a country where there is a risk of earthquakes and Dzuud is good strategy for a

humanitarian organisation.

- Help WaSH Action to write a SuSanA file, underlining all the research work that has been

carried out (PhD and Masters).

- For ACF, WaSH Action and all other actors involved in advocacy: it is urgent to find

subsidies for the construction of EcoSan toilets and the emptying service, without which

scaling up is very unlikely to happen.

Recommendations for WaSH Action:

� Recommendations for WaSH Action:

FOR THE SHORT TERM:

1. Survive!

This means investing in what are perhaps not the most relevant activities for the WaSH sector,

but in activities which will be the most lucrative, and will give WaSH Action some financial

visibility. From a tactical point of view, this will mean using the competencies and materials

which can already be mobilised (produce that is ready to sell), and focusing on areas in which

WaSH Action already has recognition and visibility.

- Have an action plan: who does what in the coming weeks, depending on the strategic

priorities and tactics which have been established.

- pursuing the implementation of the advocacy action plan should not be an absolute

priority for WaSH Action in the months ahead; this should remain in the “background”

(with activities which do not require any budget) because it constitutes the mission

(vision) of WaSH Action and allows it to remain visible and involved vis-à-vis key

decision-makers. For this, WaSH Action will need to adapt the strategy and the action

plan based on their capacity (having several scenarios at the ready depending on their

resources).

39 This seems to be in the process of being signed (including USTB) at the time of writing this report.

Page 43: URD-TDM-ACF Mongolia WaSH evaluation-final report …€¦ · External evaluation of ACF’s project, “Improve Access to Water, ... methodological tool design, ... Ger Mongolian

43

Final external evaluation of the ACF project Improve access to WASH in the Ger areas of Ulaanbaatar, Mongolia

Groupe URD - TDM | June 2015

- Be proactive to increase visibility within the international community (INGO, UN, World

Bank, Asian Development Bank, etc.).

- Keep regular contact with ACF and other partners. In the medium term (after the

survival phase), it would be useful to regularly analyse the WaSH sector and to send this

analysis to partners (and make it available on the website). The aim here would be to

remain visible and become an important local actor in terms of analysis.

2. Support the EcoSan activity

- Contribute to advocacy in favour of subsidies for EcoSan toilets and the emptying

service;

- Work closely with MonESIC, as this will be to everyone’s benefit: EcoSan is an innovative

approach which could help to make WASH Action known, but without the expertise and

support of WASH Action, there is a danger that the EcoSan adventure will come to an

end.

- - Draw up a SuSanA file to share lessons learned, which could help to improve visibility at

the international level, and fund-raising.

LONG TERM:

3. Promote the WaSH sector in Mongolia, in line with what was done with ACF

- Optimise the 2014-2017 WaSH advocacy strategy, as well as the advocacy potential of

the study “sanitation in Mongolia…” (dissemination, organisation of workshops, etc.)

- Improve the versatility of the WaSH Action staff (water, sanitation and hygiene);

- Develop a virtual Resource Centre (website) for the Mongolian WaSH sector;

- Resume action research to improve grey water management (look for partnerships with

researchers + volunteers + funding)40.

Recommendations for MonESIC:

- Have an immediate survival strategy: be more proactive and develop niche markets by

optimising use of the equipment.

- Work closely with WaSH Action, as this will be to everyone’s benefit: EcoSan is an

innovative approach which could help to make WASH Action known, but without the

expertise and support of WASH Action, there is a danger that the EcoSan adventure will

come to an end.

- Co-develop an advocacy and fundraising action plan to help develop the EcoSan sector:

indeed even though the emptying service could ideally become financially viable in the

long term, the scaling up process and social marketing campaigns require subsidies.

Recommendations for the Water Agencies:

- Finance a small-scale project by WaSH Action to test an initial partnership (for around

50 000€)

- Support the implementation of the advocacy plan

- Adopt a clearer position about the possibility of pursuing the collaboration with the

Mongolian watershed authorities.

40 same remark : this is not a priority for WaSH Action (except, of course, if funding opportunities open up for this subject)

Page 44: URD-TDM-ACF Mongolia WaSH evaluation-final report …€¦ · External evaluation of ACF’s project, “Improve Access to Water, ... methodological tool design, ... Ger Mongolian

44

Final external evaluation of the ACF project Improve access to WASH in the Ger areas of Ulaanbaatar, Mongolia

Groupe URD - TDM | June 2015

Recommendations for all actors involved in advocacy41:

- Include public showers in the advocacy work vis-à-vis institutions (to be included in

urban development and facilities plans for Ger areas)

- It would be useful to compare different WaSH solutions (economically, environmentally

and in terms of job creation) with regard to meeting the short/ medium/ and long-term

needs of those who live in the Ger areas. This would not require a great deal of time to be

invested by WaSH Action in the short term to increase its reputation amongst WaSH

actors in Mongolia.

- Optimise the advocacy potential of the study, “Sanitation in Mongolia: experiences,

challenges and recommendations”: dissemination, organisation of workshops, etc.

41 Not applicable to WaSH Action in the short term, as they have other priorities

Page 45: URD-TDM-ACF Mongolia WaSH evaluation-final report …€¦ · External evaluation of ACF’s project, “Improve Access to Water, ... methodological tool design, ... Ger Mongolian

45

Final external evaluation of the ACF project Improve access to WASH in the Ger areas of Ulaanbaatar, Mongolia

Groupe URD - TDM | June 2015

Annexes

1/ Best practice: Hygiene promotion

2/ Classification based on the DAC criteria

3/ People met

4/ Field timetable

5/ Bibliography

6/ Contents of the workshop with the ACF / WaSH Action national staff

7/ Online survey

Page 46: URD-TDM-ACF Mongolia WaSH evaluation-final report …€¦ · External evaluation of ACF’s project, “Improve Access to Water, ... methodological tool design, ... Ger Mongolian

46

Final external evaluation of the ACF project Improve access to WASH in the Ger areas of Ulaanbaatar, Mongolia

Groupe URD - TDM | June 2015

Annex 1: Best practice: Hygiene promotion

TITLE Hygiene promotion in 22 schools in Ulaanbaatar, Mongolia: introduction of

sustainable changes at both the individual and institutional levels 2009-2015

HISTORY

- 2009: targeting of poor families

Result: Unsuccessful because the people felt stigmatised, and did not come to the

training courses.

- 2010: change of strategy: work at the school level

Introduction of child to child methods and health clubs

4 schools in SKHD, 1 school in BZD

Interesting impacts observed in that the children transmit good practice to their

families

Distribution of soap in schools to create a context that is favourable to good

hygiene practices

Result: Limited success as there was no high level institutional support (the

teachers were reluctant to have external interventions if they were not officially

recognised).

- Official authorisation signed with Municipal Education Department

- Since 2011 : scaling-up process (+5 new schools)

Development and continuous improvement of hygiene training toolkit (which

gradually becomes a WaSH training toolkit, rather than just about hygiene). The

improvement of the hygiene promotion kit was carried out in collaboration with

UNICEF.

Validation by the Education Research Institute of the changes to health curricula

proposed by ACF

� Integration into health curricula at the national level

- 2012: scaling up (made possible by funding from the Swiss aid agency): +

14 new schools in SKHD (22 in total42)

Key factor of the sustainability of the health clubs: support from the school boards

and the teachers

Main outcomes and impacts:

-a lot of awareness-raising materials (including high quality video spots)

-high quality WaSH toolkit and training courses validated by the official institutions

and widely disseminated and used

-the replication of numerous health clubs

-a resource centre in a model school

-health club Facebook page

-contribution to major changes to national regulations:

-WaSH Minimum requirements at school

-It is probable (based on ACF’s advocacy work) that the Ministry of Finance

(or the schools themselves) will allocate an annual budget to buy soap for

every school.

42 and not 24, because projects have continued in 2 of the schools targetted by previous projects

Page 47: URD-TDM-ACF Mongolia WaSH evaluation-final report …€¦ · External evaluation of ACF’s project, “Improve Access to Water, ... methodological tool design, ... Ger Mongolian

47

Final external evaluation of the ACF project Improve access to WASH in the Ger areas of Ulaanbaatar, Mongolia

Groupe URD - TDM | June 2015

INNOVATIVE

FEATURES

-Strictly speaking, the peer training method (child to child) and the health clubs are

not innovative, because they have been tried and tested in other countries, but they

were new to Mongolia.

-Similarly, the use of popular communication techniques for hygiene promotion

messages was new in Mongolia: the production of videos shown on the internet,

social marketing, campaigns on social media like Facebook and Twitter (hash tag

campaign), and flash mobs.

SPECIFIC

RECOMMEN

DATIONS

FOR

DISSEMINAT

ION

in Mongolia:

-Continue to invest in social media which appear to be very popular in Mongolia,

and not only amongst the young.

-Look into ways of raising awareness amongst young children, as early as

kindergarten, because habits are learned early, and it is easier to change behaviour

early on, rather than when bad habits have been adopted.

Page 48: URD-TDM-ACF Mongolia WaSH evaluation-final report …€¦ · External evaluation of ACF’s project, “Improve Access to Water, ... methodological tool design, ... Ger Mongolian

48

Final external evaluation of the ACF project Improve access to WASH in the Ger areas of Ulaanbaatar, Mongolia

Groupe URD - TDM | June 2015

Annex 2: Classification according to the DAC criteria

c note

(1 mini, 5 maxi) Explanation

1 2 3 4 5

Sustainability x

ACF’s exit strategy was not sufficiently anticipated. ACF’s

withdrawal will have an obvious negative effect on MonESIC, who

will not meet all the challenges facing it without external support.

The pilot solutions have been tested technically, but have not been

tested socially for long enough.

Tolgoit is likely to continue to manage the multi-service kiosk in

the long term

WaSH Action has a lot of potential, but is still very fragile and still

has too little financial visibility. One of the challenges for this

young national NGO will be to take over the advocacy strategy and

to adapt it to their means

Impact x Several improvements to national regulations directly attributable

to ACF and its advocacy activities.

Sanitation: the project led to changes in behaviour (acceptance)

vis-à-vis the ecological sanitation approach, but this remains at the

pilot level, which could well be abandoned if there is no

investment in social project management + subsidising of the

service

Hygiene: excellent long-term impact, which has led to a drop in the

number of illnesses in the targeted schools.

Plusieurs améliorations de la réglementation au niveau national

directement attribuables à ACF.

Coherence x ACF’s decentralised solutions remain relevant with regard to the

new urban development plans: the two types of solution (large-

scale decentralised public infrastructure vs decentralised

solutions at the household level) are, in fact, complementary.

It is not likely that all the Ger areas will be transformed into

apartments and connected to the network.

The models that have been tested can meet the WaSH needs of

people in the Ger areas in the short/medium term.

Coverage x Coverage is good in terms of the dissemination of innovations

(even though it is difficult for the economic model of the emptying

service to have toilets to empty which are scattered throughout

the district).

Relevance/app

ropriateness

x The project meets people’s essential WaSH needs, even though it

does not cover needs in terms of public showers or grey water

management. ACF’s innovations are promising and adapted to an

extremely cold climate, they have inspired large organisations like

UNICEF, and have been integrated into the urban development

preliminary studies for the2nd biggest city in the country.

Effectiveness x The project was unable to meet its objectives in terms of the

Page 49: URD-TDM-ACF Mongolia WaSH evaluation-final report …€¦ · External evaluation of ACF’s project, “Improve Access to Water, ... methodological tool design, ... Ger Mongolian

49

Final external evaluation of the ACF project Improve access to WASH in the Ger areas of Ulaanbaatar, Mongolia

Groupe URD - TDM | June 2015

dissemination/accompaniment of the tested solutions for

sanitation and water, which was predictable with regard to the

duration of the project. This objective was too ambitious.

Efficiency x The training courses for local partners were useful for USUG and

Tolgoit, but not appropriate for MonESIC.

The donors’ high level of flexibility and confidence allowed the

field staff to adapt the project strategy and activities in response to

opportunities and changes in the context, and thereby increase

efficiency as the main decisions made by the project team were

completely relevant. On the other hand, it is regrettable that

Artois-Picardie Water Agency had to repeatedly cancel its

participation in forums organised with the other institutional

actors.

Page 50: URD-TDM-ACF Mongolia WaSH evaluation-final report …€¦ · External evaluation of ACF’s project, “Improve Access to Water, ... methodological tool design, ... Ger Mongolian

50

Final external evaluation of the ACF project Improve access to WASH in the Ger areas of Ulaanbaatar, Mongolia

Groupe URD - TDM | June 2015

Annex 3: People met

Partenaires de hand-over (MonESIC,

Tolgoit)

-MonESIC: Myagmarsuren Shagdarjav, executive Director

(4/06)

-MonESIC: Zaya Munkhchimeg, financial management

(4/06)

-Tolgoit : Ms Suuri, head of CBO (11/06)

-Gardienne vendeuse d’eau du kiosque multiservice

(11/06)

Consultants mongols clefs ayant

accompagné ACF : MIRIM (Marketing

Innovation Research Institute of

Mongolia), Bolor Legjem, Development

Solution

-BAT-ORSHIKH Erdenebat, executive Director of

Development Solutions (2/06)

-Bolor LEGIEM (4/06)

-David CHACE, MIRIM manager (8/06)

-Yanjinpagma Nyamsuren, Project Manager MIRIM

Consultant (8/06)

Equipes ACF / WaSH Action of Mongolia -Pier Francesco Donati: EAH Head of Department

-Clément Philit: Directeur pays

-Bulganchimeg.B, Ecosan Officer

-Ariunaa, Hygiene Promotion Manager / WaSH Action

-Shijirtuya, Environmental Sanitation Manager / WaSH

Action

-Zolzaya, Finance manager/ WaSH Action

-Baatar, Driver logistics

-Bodigerel, Human Resource Officer/ WaSH Action

-Amarkhuu, Field officer

-Battseren, head of WaSH Action

-Khaliunaa, Communication officer/ WaSH Action

-Javzansuren, Water supply manager / WaSH Action

Bayrmaa.E, Field officer

Différentes catégorie de

« bénéficiaires »/partenaires :

propriétaires d’Eco-toilettes usagers du

service de vidange, écoles, usagers des

kiosques à eau, différentes personnes

formées par ACF (USUG, River basin,

Schools, etc…), institutions

-ecotoilets users 3° & 26° khoroo (5/06)

-schools 106: Tuvshinjargal, teacher; Erkhtuya, doctor;

Bayarmaa & Ariunzul, Health Club members (5/06)

-Director of Batsurkh school (5/06)

-USUG: Dagvasuren, manager + 2 teachers

Acteurs institutionnels43. Ces entretiens

viseront à comprendre l'implication de

ces partenaires et leur point de vue sur le

projet et ses réalisations (Ministères,

-Ms MYAGMAR Jadamba & DULAMKHAND, Ministry of

education and science - department of coordination of

policy implementation – division preschool and primary

education (4/06)

43 The categorisation of the actors met is more or less arbitrary to the extent that the majority of institutional actors at

different levels are also partners or beneficiaris of ACF either as targets of advocacy or capacity building. The same is true

for UNICEF for example, which has several roles.

Page 51: URD-TDM-ACF Mongolia WaSH evaluation-final report …€¦ · External evaluation of ACF’s project, “Improve Access to Water, ... methodological tool design, ... Ger Mongolian

51

Final external evaluation of the ACF project Improve access to WASH in the Ger areas of Ulaanbaatar, Mongolia

Groupe URD - TDM | June 2015

Municipalité d’UB, autorités du district et

sub-districts…)

-Ms MUNKHJARGAL, consultant (former health

department of UB Municipality) (4/06)

- 7th khoroo governor: Davaanyam, social worker(5/06)

+ online survey

Acteurs humanitaires et de

développement travaillant sur des

projets similaires dans les zones de

yourtes (JICA, Banque Mondiale, GIZ,

Croix Rouge, World Vision…)

� online survey

Expert -D. Myagmar Dovchin, (10/06)

Bailleurs44 -UNICEF: BATNASAN Nyamsuren

& Robin WARD, WaSH officer (9/06)

Universités et étudiants partenaires -Nazim Uddin, pHD reseracher, Université de Sciences

et Technologies de Pékin (by skype 2/06)

Observations

- site et mise en place du compost d’été (3/06)

- Zip-Zap toilet atEmelt compost site (3/06)

- resource center + museum + technical training room at USUG training center (10/06)

- resource center at school (5/06)

- school toilets rehabilitation (5/06)

- Eco-toilettes installées de 2011 à 2014 (5/06)

- MSWK (11/06)

- coaching meeting on business plan provided by MIRIM for MonESIC & TOLGOIT (8/06)

44 L’équipe d’évaluation recommande à ACF d’inviter les Agences de l’eau lors de la restitution finale à Paris (au moins via

skype ou téléconférence)

Page 52: URD-TDM-ACF Mongolia WaSH evaluation-final report …€¦ · External evaluation of ACF’s project, “Improve Access to Water, ... methodological tool design, ... Ger Mongolian

Final external evaluation of the ACF project Improve access to WASH in the Ger areas of Ulaanbaatar, Mongolia Groupe URD - TDM | June 2015

Annex 4: timetable

1st

we

ek

Monday 1er /06

Day OFF in Mongolia

Tuesday

02/06

Wednesday

3/06

Thursday

4/06

Friday

5/06

Sat

6/06

Sun

7/06

mo

rnin

g

Biblio + Inception

report

-Briefing HoM

-Meeting with

Development Solution

Compost site visit

Compost processing

-Bolor (consultant)

-education Minister

members

-shools (including health club,

resource center, toilets rehab)

Data analysis

+ biblio

aft

ern

oo

n

Briefing with ACF

mission

Planning of evaluation

ACF MonESIC -EcoSan toilets users in SKHD

-7th khoroo social worker

e v Skype with Nazim Data analysis + biblio Skype meeting with TDM

2d

we

ek

Monday 8/06 Tuesday 9/06 Wednesday

10/06

Thursday

11/06

Friday

12/06

Sat

13/06

Sun

14/06

mo

rnin

g

MIRIM workshop with

USUG, Tolgoit &

MonESIC

MIRIM meeting

Data analysis

UNICEF

-Myagmar Dovchin (expert

& consultant)

-Thibault (CARITAS)

TOLGOIT & Multiservice

Water Kiosk

Ministry of Environment, Green

Development & Turism (3 times

postponed …)

Preparation of debriefing seminar

off

off

aft

ern

oo

n

WaSH Action of

Mongolia workshop

(1/2)

WaSH Action of

Mongolia workshop

(2/2)

USUG training center Tuul River basin

authority

Debriefing (=evaluation

workshop) with ACF staff +

partners

ev

e

nin Skype meeting with TDM Data analysis + biblio

AP Water Agency A. Courtecuisse

(Skype) Skype meeting with TDM Skype meeting with TDM

Page 53: URD-TDM-ACF Mongolia WaSH evaluation-final report …€¦ · External evaluation of ACF’s project, “Improve Access to Water, ... methodological tool design, ... Ger Mongolian

53

Final external evaluation of the ACF project Improve access to WASH in the Ger areas of Ulaanbaatar, Mongolia

Groupe URD - TDM | June 2015

Annex 5: Bibliography

General reports from ACF

- ACF Mongolia, 2012/2014 Knowledge, Practice and Attitude KAP survey on Water,

Sanitation and Hygiene promotion in ACF targeted Ger area in Songino Khairkahn

District, Ulaanbaatar, Final findings report, 2014

Ecosan toilet monitoring report 2014 -> inclus dans cette KAP survey

- ACF Mongolia & WaSH Action of Mongolia, Sanitation in Mongolia: experiences,

challenges and recommendations “We can’t wait”, January 2015

- ACF Mongolia, APPENDIX L. GER AREA INVESTMENT PLAN AND PROGRAM Water

Supply and Sanitation Infrastructure Improvement Project for Darkhan Final Report,

September 2014 (=CDIA prefeasibility study 2014 (mentionné dans les TDR)

- ACF Mongolia, “Improve access to water, hygiene and sanitation in the Ger areas in

Ulaanbaatar, Mongolia, through the dissemination of innovative and proven solutions”

MAY 2013 – DECEMBER 2014, Water Agencies activities report January 2015

- ACF Mongolia, Franziska Bock, Institutional analysis of WATER SUPPLY, SANITATION

AND HYGIENE IN MONGOLIA, 2014

- ACF, Sustainable Water, Sanitation and Hygiene services, GER WASH PROJECT, SDC

FINAL REPORT, april 2015

- coordonnées GPS des éco-toilettes

- Mongolian University of Life Sciences, compost analysis, June 2015

- Enkhtungalag B (Central Laboratory of Agency for Specialized Inspection), Battseren

(ACF, Purevdorj (MNUMS, School of Public Health), Microbiological study of compost in

the open area, 2014

Hygiene Promotion

- ACF Mongolia “Ger WASH project: Sustainable Water, Sanitation, and Hygiene services”

2012-2014/ Hygienic training and public advocacy work and monitoring at the schools

- ACF Mongolia, Survey report on hand washing conducted among secondary school

students in Songinokhairkhan district, Ulaanbaatar, dec 2014

- matériel de formation et plaidoyer développé par ACF de 2009 à 2015 (supports de

cours, boîte à outils, posters, clips vidéo)

Advocacy

- ACF, Eric Reinshtein, Coordinator of WaSH project, advocacy document FOR AN

EQUITABLE AND SUSTAINABLE WATER TARIFF IN ULAANBAATAR, 2014

- ACF MONGOLIA MISSION, ADVOCACY STRATEGY, (2014-2016), sept 2014?

- ACF, ADVOCACY ACTION PLAN, Water, Change goal: National policies and plans are to be

implemented for water conservation, protection, water supply to face water pressure

due to human activities, population movements and poor resource management, déc

2014

- ADVOCACY ACTION PLAN, Sanitation (Change goal: Legal frame and standards for

sustainable sanitation are improved, déc 2014

- ACF, ADVOCACY ACTION PLAN, Hygiene, Change goal: National curriculum on HP

continuously disseminated in primary schools and kindergartens, déc 2014

- ACF International advocacy toolkit, 2013

WASH Action of Mongolia

Page 54: URD-TDM-ACF Mongolia WaSH evaluation-final report …€¦ · External evaluation of ACF’s project, “Improve Access to Water, ... methodological tool design, ... Ger Mongolian

54

Final external evaluation of the ACF project Improve access to WASH in the Ger areas of Ulaanbaatar, Mongolia

Groupe URD - TDM | June 2015

- WaSH Action of Mongolia, strategy Plan 2015-2017, January 2015

- SWOT analysis of WaSH action of Mongolia, early January

- site internet de WaSH Action

- facebook page of WaSH Action

MonESIC

- statuts

- organigramme

- business plan

- comptabilité (cash book) janvier, février, mars, avril 2015

Consultants

- BAT-ORSHIKH Erdenebat Development Solutions, final report of establishing and

strengthening the NGO through improving operational and financial management, 5 juin

2015

- http://www.irim.mn/index.php/en/mirim

- MIRIM, Business Plan Training Program for Sanitation Enterprises Final progress report

June 18th, 2015

Other:

- http://www.publicfinanceforwash.com/

- World Bank : report on onsite sanitation in cold countries (à paraître en septembre

2015)

Page 55: URD-TDM-ACF Mongolia WaSH evaluation-final report …€¦ · External evaluation of ACF’s project, “Improve Access to Water, ... methodological tool design, ... Ger Mongolian

55

Final external evaluation of the ACF project Improve access to WASH in the Ger areas of Ulaanbaatar, Mongolia

Groupe URD - TDM | June 2015

Annex 6: Workshop with the ACF / WaSH Action national staff

Monday the 8th June afternoon

& Tuesday the 9th June

I/ Review of past activities

Session 1 :

- Propose following categories (steps): testing solution-> solution selection -> scaling up -

> Advocacy -> adoption at large scale, explaining that these steps can sometimes carried

out in parallel. Give examples.

- Divide the group into three teams: water / sanitation / hygiene

- Ask each group to write on pieces of paper the main activities of the past 6 years,

indicating if possible the year or period of achievements of the activity and the contract

(WA, SDC, UNICEF, ACF )

- The groups will stick on the wall the different activities proposed in the 3 following

categories (1-solution testing / pilot project; 2 / scaling up; 3 / advocacy)

- in Plenary: all groups validate (or move) paper bonded by other groups. Then:

Discussion of the 3 different WaSH sectors in the progressivity of the test to the

widespread adoption through the scaling up. What are the most significant

achievements?

Page 56: URD-TDM-ACF Mongolia WaSH evaluation-final report …€¦ · External evaluation of ACF’s project, “Improve Access to Water, ... methodological tool design, ... Ger Mongolian

56

Final external evaluation of the ACF project Improve access to WASH in the Ger areas of Ulaanbaatar, Mongolia

Groupe URD - TDM | June 2015

Session 2 :

Plenary Intro: What was the initial strategy? What were the main adaptations (or shift) of the

strategy?

- Division in subgroup water / sanitation / hygiene

- Leave the product materials (papers with activities), and for each activity, consider the

following questions in groups:

1. What has still to be done in relation to what was planned (following the rationale of

the project)? (Optional question depending on available time)

2. In your opinion, what activities should be conducted (new business) to optimize the

work already done, in order to achieve a good impact -> write the new activities on

white paper

3. Do WaSH Action wants to carry out these activities that would maximize the impact?

4. What are the most difficult activities to finance?

5. Imagining that funding is available, does WaSH Action has the capabilities to carry

out these activities?

Each group is asked to prepare a synthetic restitution on paperboard

While the groups are working, and depending on the degree of work-progress, I bring new

questions in each subgroup: among all these activities, did you face failures? Disappointments?

Have you learned lessons? Would have it been possible to implement activities in a different

manner?

Page 57: URD-TDM-ACF Mongolia WaSH evaluation-final report …€¦ · External evaluation of ACF’s project, “Improve Access to Water, ... methodological tool design, ... Ger Mongolian

57

Final external evaluation of the ACF project Improve access to WASH in the Ger areas of Ulaanbaatar, Mongolia

Groupe URD - TDM | June 2015

II/ Advocacy45

� Working in small groups:

What does advocacy mean for you?

What are the advocacy actions that have succeeded and why? What advocacy actions have failed

and why?

How all ACF advocacy activities planned in the strategy fit in in with WaSH Action strategy?

Who are the others stakeholders doing advocacy in UB?

� Questions for plenary debrief:

-Do you monitor advocacy activities (with performance indicators)? (See Manual ACF)

-How will you promote / disseminate the study "we can not wait"? what are the links with your

advocacy strategy?

III/ Next steps: looking in the future

1. From ACF-> to WaSH Action

� 3 subgroups work on the following questions: What are the main changes

between WaSH Action nd ACF (+ and -)? What will be the similar activities? What

activities will you stop? What new activities will you implement?

cross fertilization for restitution

+ contracts, consultancies and proposals review

2. Competencies map

� Introduction in plenary intro: What are the skills needed to WaSH Action to be

viable?

� Mutual interview with neighbor on 3 questions (the neighbor will present the

answers):

1. In the last 6 years, what are the formations that were the most useful to you? (List)

why?

2. What are the skills that I will bring to WaSH Action

3. What could WaSH Action bring to me? (=Why I want to involve myself in WaSH Action?)

45 this workshop on advocacy was not done in group (lack of time)

Page 58: URD-TDM-ACF Mongolia WaSH evaluation-final report …€¦ · External evaluation of ACF’s project, “Improve Access to Water, ... methodological tool design, ... Ger Mongolian

58

Final external evaluation of the ACF project Improve access to WASH in the Ger areas of Ulaanbaatar, Mongolia

Groupe URD - TDM | June 2015

* What do you do in 1/3/6 months if Wash Action doesn’t succeed in getting substantial

funding?

How long can you stay without any salary?

From July 2015, what will be your level of investment? full-time? part-time?

Page 59: URD-TDM-ACF Mongolia WaSH evaluation-final report …€¦ · External evaluation of ACF’s project, “Improve Access to Water, ... methodological tool design, ... Ger Mongolian

59

Final external evaluation of the ACF project Improve access to WASH in the Ger areas of Ulaanbaatar, Mongolia

Groupe URD - TDM | June 2015

Annex 7: online survey

The low number of respondents meant that this could not be used to provide analysis for the

evaluation. The raw results are mentioned here for information purposes only.

Q2: What is your overall feeling about ACF WaSH programs ? / ACF-ийн WaSH

төслүүдийн талаарх Таны бодол ?

3 comments:

-Thank you for helping schoolchildren to create a clean environment for school children

-To keep activities continuesly. The project is only one project, which did improvement based

on the survey and pilot result. Also involved researchers.

-салбарын байгууллагуудтай хамтарч ажиллахад түлхүү анхаарах (Should focus more to

cooperate with relevant stakeholders)

Q3: What are the main changes in the WaSH sector in Mongolia in the last 10 years /

Сүүлийн 10 жилд Монголын WaSH салбарт ямар чиглэлд гол өөрчлөлтүүд гарсан бэ?

(1/2)

Page 60: URD-TDM-ACF Mongolia WaSH evaluation-final report …€¦ · External evaluation of ACF’s project, “Improve Access to Water, ... methodological tool design, ... Ger Mongolian

60

Final external evaluation of the ACF project Improve access to WASH in the Ger areas of Ulaanbaatar, Mongolia

Groupe URD - TDM | June 2015

3 comments :

Сургуулийн бие засах газрыг үлгэр жишээ болгон өөрчилж, удирдлагуудад хийх

боломжийг мэдрүүлсэн. (School toilet room was rehabilitated as a good practice for others

and school principal had good understanding of importance of the improvement.)

Established Water supply and sewerage regulation committee, improved legal environment.

used different types of technologies on waste water treatment and pit latrine. Missed

coordination and evaluation.

Монголын нөхцөлд тохирох ариун цэврийн байгууламж бий болгох нь бидний хамтын

зорилго, зорилт байх болно (Our main objective, challenge is develop toilet design that can

be adjusted Mongolian weather condition.)

Q4: What are the main changes in the WaSH sector in Mongolia in the last 10 years /

Сүүлийн 10 жилд Монголын WaSH салбарт ямар чиглэлд гол өөрчлөлтүүд гарсан бэ?

(2/2)

3 comments:

Pit latrine new standard approved.

Гар угаах сургалтын материалыг мэргэжлийн түвшинд хийж, сургалтыг явуулсан.

Сурагчдын клубыг бие даан ажиллахад нь дэмжлэг үзүүлсэн. (We have developed hand

washing training material and conducted the training)

In efficient use of budget not based on long term strategy in the WaSH

Q5: What are the changes due to ACF ? (among the previous ones) / Дээрх чиглэлүүдэд

гарсан өөрчлөлтүүдэд ACF –ийн оролцоотой өөрлөлт гарсан нь аль вэ? (1/2)

Page 61: URD-TDM-ACF Mongolia WaSH evaluation-final report …€¦ · External evaluation of ACF’s project, “Improve Access to Water, ... methodological tool design, ... Ger Mongolian

61

Final external evaluation of the ACF project Improve access to WASH in the Ger areas of Ulaanbaatar, Mongolia

Groupe URD - TDM | June 2015

2 comments:

Эрүүл мэндийн болон Боловсролын сайдын хамтарсан шаардлага гарсан. (Joint order

was approved by Ministery of Health and Ministry of Education.)

улам сайн технологи нэвтрүүлэх (More efficient technologies have to be implemented)

Q6: What are the changes due to ACF ? (among the previous ones) / Дээрх чиглэлүүдэд

гарсан өөрчлөлтүүдэд ACF –ийн оролцоотой өөрлөлт гарсан нь аль вэ? (2/2)

1 comment:

Боловсролын салбарт ялангуяа багш нарын "Гар угаах" сэдвийг заахад өөрчлөлт гарсан.

(Education of teachers, especially in "hand-washing" to teach the subject changes)

Q7: According to you, what will remain from ACF work here? / Таны бодлоор ACF –ийн

ажлуудаас үлдэцтэй ямар ажлууд байна вэ ?

Page 62: URD-TDM-ACF Mongolia WaSH evaluation-final report …€¦ · External evaluation of ACF’s project, “Improve Access to Water, ... methodological tool design, ... Ger Mongolian

62

Final external evaluation of the ACF project Improve access to WASH in the Ger areas of Ulaanbaatar, Mongolia

Groupe URD - TDM | June 2015

1 coment:

Бодлогын болон нэгжийн түвшинд ажиллаж чадсан. (Policy and worked with the unit level)

Q8: According to you, what will remain from ACF work here? / Таны бодлоор ACF –ийн

ажлуудаас үлдэцтэй ямар ажлууд байна вэ ? (2/2)

Q9: What are the main WaSH coordination events and bodies in Mongolia? / Монголд

WaSH чиглэлээр үйл ажиллагаа, ажил зохион байгуулдаг гол байгууллагууд нь юу вэ?

?

ACF

The Ministry of Health, MECS

MCUD, MOE, MOHealth,

олон байгууллага байгаа (many!)

Q10: Do you know WaSH Action of Mongolia ? / Та WaSH Action of Mongolia-г мэдэх үү?

100% : YES

2 comments:

But I need to learn more about WaSH Action of Mongolia.

Маш санаачлагатай, хүртээмжтэй ажилладаг байгууллага. (Very proactive and inclusive

organizations.)

Page 63: URD-TDM-ACF Mongolia WaSH evaluation-final report …€¦ · External evaluation of ACF’s project, “Improve Access to Water, ... methodological tool design, ... Ger Mongolian

63

Final external evaluation of the ACF project Improve access to WASH in the Ger areas of Ulaanbaatar, Mongolia

Groupe URD - TDM | June 2015

Page 64: URD-TDM-ACF Mongolia WaSH evaluation-final report …€¦ · External evaluation of ACF’s project, “Improve Access to Water, ... methodological tool design, ... Ger Mongolian

Groupe URD

La Fontaine des Marins

26170 PLAISIANS

Tel : + 33 (0)4 75 28 29 35

www.urd.org