· web viewwas implemented by ncdds in three local administrations in takeo province: doun keo...

Local Governments and Climate ChangeReport of Participatory EvaluationsApril 2013

LGCC Participatory Evaluations

Contents1 Introduction...................................................................................................................................1

2 Background....................................................................................................................................1

2.1 The LoCAL Global Project.......................................................................................................1

2.2 Local Governments and Climate Change Phase 1..................................................................1

2.3 Scope of Sub-Projects Supported...........................................................................................2

2.4 LGCC Phase 2.........................................................................................................................3

2.5 Timing of the Evaluations......................................................................................................3

3 Purpose of the Evaluations............................................................................................................3

4 Methodology of the Participatory Evaluations..............................................................................4

4.1 Steps......................................................................................................................................4

4.2 Selection of Sub-Projects.......................................................................................................4

4.3 Preparation............................................................................................................................5

5 Fieldwork.......................................................................................................................................6

5.1 Physical Inspection of Outputs...............................................................................................6

5.2 Focus Group Discussion.........................................................................................................6

5.3 Summing Up..........................................................................................................................6

5.4 Scoring By The Evaluation Team............................................................................................7

5.5 Provincial Reflection Workshop...........................................................................................10

6 Findings of Participatory Evaluations...........................................................................................10

6.1 Technical Quality.................................................................................................................10

6.2 Beneficiaries’ Perceptions of Benefits..................................................................................11

6.3 Negative Impacts.................................................................................................................12

6.4 Sustainability........................................................................................................................12

6.5 Relevance to Climate Change Adaptation............................................................................12

6.6 Impact on Beneficiaries’ Understanding of Climate Change Adaptation.............................13

6.7 Cost-Benefit Scores..............................................................................................................13

7 Summary Of Findings And Lessons Learned................................................................................13

7.1 Impact..................................................................................................................................13

7.2 Relevance to Climate Change Adaptation............................................................................14

7.3 Weaknesses Of the Sub-Projects.........................................................................................14

7.4 Key points to strengthen......................................................................................................15

8 Assessment of the Evaluation Methodology...............................................................................16


1 IntroductionThis report describes the process and results of participatory evaluations of sub-projects funded by the Local Governments and Climate Change (LGCC) project in Takeo province during March 2013. LGCC is a pilot in Cambodia of United Nations Capital Development Fund’s global “Local Climate Adaptive Living” (LoCAL) project. As such, the findings of the evaluation and the methodology used are of potential value for LoCAL country programmes elsewhere, in addition to the lessons learned which are directly applicable to the future implementation of LGCC.

The evaluations were undertaken by the Secretariat of the National Committee for Sub-National Democratic Development (NCDD-S) and the Takeo Provincial Administration with technical assistance from United Nations Capital Development Funds (UNCDF).

2 Background

2.1 The LoCAL Global ProjectThe LoCAL Global Project was designed to provide access to climate finance for local governments in least developed countries (LDC). The project is based on successful previous UNCDF experiences in public financial management; local investments and performance-based fiscal grants.

Climate change adaptation finance flows to central government agencies, and yet local governments have the mandate to deal with many of the issues directly affected by climate change such as drainage and water management, land use planning, road maintenance and other issues which effect food security, especially in the LDCs.

LoCAL addresses this issue by providing Performance Based Climate Resilience (PBCR) grants to selected local governments that show themselves able to meet a defined set of minimum conditions. PBCR Grants may be used to finance climate change adaptation measures that fall within a general Investment Menu based on a climate change vulnerability assessment, and that are identified through local participatory planning processes.

LoCAL transfers PBCR Grants to local governments through existing financial and public expenditure management systems. The performance-based nature of the grants provides local governments with an incentive to build their own capacity. In most country programmes UNCDF with its partners provide direct capacity development support for climate change adaptation while also learning lessons from LoCAL implementation and supporting policy development for climate change adaptation at local level.

LoCAL is directed by a Board consisting of participating countries and donors and is administered through a Secretariat. The Secretariat deploys specialist expertise to support LoCAL country programmes, facilitates sharing of experiences and manages the monitoring and evaluation system to ensure effective use of funds.

1


2.2 Local Governments and Climate Change Phase 1Phase 1 of Local Governments and Climate Change was supported by a grant of $US 250,000 from the Cambodia Climate Change Alliance Trust Fund and $US 50,000 funding from UNCDF. The project was implemented by NCDDS in three local administrations in Takeo Province: Doun Keo Municipality and Bati and Borei Chulsar Districts. Project implementation began in the final quarter of 2011 and ended in April 2013; i.e. the project financed a single cycle of PBCR grants in the three local administrations.

The project objective of LGCC was “to demonstrate the role of Local Governments in fostering climate change resilience and to identify practical ways to mainstream CC resilience into Sub-National planning and finance systems. An important focus is thus on how to strengthen institutions and systems at both National and Sub-National levels. The entry points are provided by the recently introduced Sub-National planning and Sub-National finance systems.”

LGCC had three expected outputs which were defined as:

1. Analysis of climate change awareness and integration into Sub-National plans and investment programs;

2. Piloting of systems to mainstream climate change resilience in Sub-National Finance and Planning Systems; and

3. Policy feedback to National and Sub-National authorities.

For the purpose of LGCC the three participating local administrations were considered as including the constituent Communes of the Districts and Sangkats of the Municipality. The PBCR Grant was considered as a single grant to the District or Municipality, but Communes or Sangkats, as well as the District and Municipal Administrations, could propose activities to be financed and if successful, could implement sub-project activities. In practice, infrastructure sub-projects were implemented by Communes and Sangkats following the well-established procedures of the Commune/Sangkat Fund, while non-infrastructure activities were implemented by the District and Municipal administrations

Representatives of the District and Commune levels participated in a planning exercise to identify the key climate change vulnerabilities and priority investments for climate change adaptation. The resulting workplan and budget was approved by the District/Municipal Council and submitted to NCDD-S, which approved the workplan and budget and transferred funds for those activities to be implemented by the Communes and Sangkats into the respective Treasury Accounts. For activities to be implemented at the District level, funds were transferred into a project bank account operated by the Provincial Administration.

2.3 Scope of Sub-Projects SupportedEligible expenditures of the PBCR grant funds under the Investment Menu comprised:

1. Strategic Analysis and Planning for Climate Change Adaptation;1. Climate Change Adaptation in Sector Service Delivery;2. Climate Proofing of Infrastructure and Additional Infrastructure for Climate Change

Adaptation.

2


In addition, Districts / Municipalities were permitted to spend up to 5% of the PBCR grant on administrative overheads and up to 15% of the costs of infrastructure investments on hiring technical assistance (a “Technical Support Consultant”) for design and supervision of the works.

A list of LGCC sub-projects is attached as Annex 1. A summary of grant allocations is shown in Table 1 below.

Investment Type Doun Keo Bati Borei Chulsar TotalCCA Planning $1,384 $4,102 $642 0 $6,128Sector Service Delivery 2 $4,477 2 $8,624 1 $4,149 5 $17,250

Climate Proofing Infrastructure

Roads 1 $2,629 2 $14,000 3 $16,629

Water Supplies 2 $16,100 2 $16,100

Irrigation 1 $10,000 2 $20,553 3 $30,553 Drainage 1 $6,400 1 $3,631 2 $10,031

Technical Support $3,158 $6,468 $3,111 0 $12,737Administration $1,492 $2,875 $1,383 0 $5,750TOTAL 5 $29,540 6 $58,722 4 $26,916 15 $115,178

The sub-projects were implemented between May and December 2012, with some construction activities not completed until the first quarter of 2013 due to the need to wait for wet season floods to recede.

2.4 LGCC Phase 2During 2012 funding was agreed between Swedish International Development Agency (SIDA), NCDDS and UNCDF for a second phase of LGCC, to be implemented during 2013-2014. LGCC Phase II supports the same local governments in Takeo Province as in Phase 1, plus five Districts (with constituent communes) in Battambang Province.

2.5 Timing of the EvaluationsThe participatory evaluations which are the subject of this report formed part of the work-plan of LGCC Phase 1. The participatory evaluations were carried out after all LGCC1 sub-projects had been completed and during the start-up phase of LGCC2. Therefore, the experience of implementing the sub-projects was fresh in the memory of the participants and lessons learned could be applied toLGCC2 implementation. However, the timing of the evaluations was such that the full impacts of some sub-projects might not be apparent (e.g. because an irrigation sub-project was evaluated before the end of the first full growing season using the sub-project outputs).

3 Purpose of the EvaluationsThe purpose of the participatory evaluations was to improve knowledge on local administration capacity and performance in responding to climate change adaptation

3


The purpose of the participatory evaluations was to improve knowledge, learn lessons and identify success in the implementation of LGCC, and thus to further the project objective of demonstrating the role of local governments in fostering climate change resilience. Specifically, the methodology of the participatory evaluations was intended to”

allow the sub-project beneficiaries to express their views on all aspects of sub-project implementation;

to identify strong and weak aspects of the implementation of the sub-projects; to identify the expected impacts (including any negative impacts) and sustainability of the

sub-projects; to assess the relevance of the sub-projects to local needs for climate change adaptation; to assess the extent to which implementation of the sub-projects assisted in building

awareness of climate change vulnerability and adaptation needs at local level; to assess whether the investments provided value for money as “no regrets” CCA

investments, meaning that the immediate and ongoing benefits of the projects are broadly commensurate with the project costs; and

to develop and field-test a simple, robust evaluation methodology that can be replicated for different types of sub-project and in different geographical areas.

Evaluation of small, diverse and geographically scattered sub-project outputs is a challenge and in particular it is difficult to use quantitative methods that practical to apply, yield results that can be meaningfully compared, and that yield useful information about the impacts of the project or activity being evaluated. Some methods that are routinely applied to large projects are too complex or expensive to be useful in this situation. When the desired impact is “climate change resilience” there is additional difficulty as resilience is the capacity to adapt to unknown future events.

The methodology of the participatory evaluations, described in detail in the following section of this report, was designed with these difficulties in mind. High priority was given to simplicity and replicability, while yielding useful information of relevance to climate change adaptation impacts.

4 Methodology of the Participatory Evaluations

4.1 StepsThe participatory evaluations were carried out according to the following steps:

1. Selection of sub-projects to evaluate;2. Preparation including summary of available information on each sub-project;3. Fieldwork;

a. Physical Inspectionb. Focus Group Discussionc. Summing Up

4. Scoring by the Evaluation Team;5. Provincial Reflection Workshop including:

a. Presentations by the beneficiaries based on focus group discussionsb. Presentation of scoring by the evaluation teamc. Reflection and lessons learned.

4


4.2 Selection of Sub-ProjectsSub-projects were selected to represent the range of sub-project types implemented across the three local administrations (District or Municipality). Because of the small total number of sub-projects this could not be achieved by random sampling. Therefore it was decided that in each local administration one road sub-project, one water sub-project (irrigation or water supplies) and one non-infrastructure sub-project would be selected. Selection of the sample sub-projects was done by the national level members of the evaluation team to reduce the risk of bias (as they were less familiar with the individual sub-projects than the Provincial team members).

The following sub-projects were selected for evaluation:

District Commune Type DescriptionBati Lumpong Pond Rehabilitation of a community pond 60m x 30m x

2m approx for water supplies and gardeningBati Thnowt Canal Rehabilitation of a canal 1450m long with one

culvert and repair of road alongside canalBati Pea Ream Non-

InfrastructureDemonstration of climate resilient rice seed variety at 4 locations

Borei Chulsar

Borei Chulsar Road Drainage structures and raised embankment on road in flood-prone area

Borei Chulsar

Kouk Phos Drainage Construction of culverts on line of an irrigation canal and providing access across the canal

Borei Chulsar

Borei Chulsar Non-Infrastructure

Training on water use and hygiene and provision of water filters to XXX families in a flood-prone village

Doun Keo Roka Knong Road Construction of two short lines of road crossing a flood prone area, with raised embankments and drainage structures

Doun Keo Baray Irrigation Construction of water gate on an existing canal line

Doun Keo Roka Krao Non-Infrastructure

Raising awareness on climate change and planting tree seedlings

4.3 PreparationPreparation for fieldwork included preparing a summary of available data on each sub-project. This information was obtained from the Project Information Database of NCDDS and from other sources by the Provincial Administration. The basic information on each sub-project consisted of:

Sub-project location; Implementing agency (Commune or District level) Sub-project name For infrastructure sub-projects:

o Contract Numbero Contractor Nameo Name of Technical Supervisor

Description of Outputs Contract Cost

5


Table of Outputs: Number, Description, Location (Village), Quantity Completion Date (Planned and Actual)

In addition to tabulating this information, copies of relevant documents including contract and design drawings (for infrastructure sub-projects) and work plan (for non-infrastructure sub-projects) were collected for reference in the field.

5 Fieldwork

5.1 Physical Inspection of OutputsFor infrastructure sub-projects the evaluation team inspected the outputs in the company of the local authorities and beneficiaries. The evaluation team attempted to determine whether the design was appropriate, the quality of the construction and whether the output was functioning correctly. The views of the beneficiaries were taken into account and were cross-checked with technical staff including the Provincial Infrastructure Adviser and the Technical Support Consultant who was responsible for design and construction supervision.

The evaluation did not include a complete “engineering” assessment of the structures or any physical tests.

5.2 Focus Group DiscussionThe focus group discussions were structured around a number of themes. The beneficiaries were assisted to summarise their ideas on each theme into a few sentences which one of them wrote down on flip-chart paper.

In following this procedure, it was important not to try to write down everything anybody said. The intention was to encourage an open discussion on each topic and then to summarise the views expressed into a few sentences.

The themes around which the discussion was structured were:

What were the sub-project activities (as perceived by the beneficiaries)? What impacts do the beneficiaries expect (including negative impacts) as a result of the sub-

project activities? How does the sub-project help to build climate resilience? What actions are needed to ensure the sustainability of the sub-project? What lessons can we learn that can be applied in future sub-projects?

It is emphasised that this list of “themes” is not a questionnaire requiring a set of direct, one-to-one answers. The purpose of the themes is to stimulate discussion.

The time allocated for the physical inspection and the focus group discussion was a maximum of two hours – this permitted three sub-projects to be evaluated in one day but more importantly, served to avoid excessive demands on the time of the beneficiaries.

6


5.3 Summing UpAt the end of the focus group discussion the beneficiaries read through and reviewed the summary sentences written on the flipchart paper. Finally, the beneficiaries were asked to select one man and one woman, who should not be people with an official position of any kind (i.e. not the village chief or assistant, or similar positions) who would attend the reflection workshop and present the findings of the discussion.

5.4 Scoring By The Evaluation TeamThe evaluation team (meaning the representatives of NCDDS and the Provincial Administration, plus the UNCDF advisers present) carried out scoring of each sub-project as soon as possible after completing work at the sub-project site.

Scores were awarded against the six criteria in the following table.

Technical Quality (max 10 points) Sustainability (max 10 points) Negative Impacts (max 10 points) Relevance to Climate Change Adaptation (max 20 points) Impact on Beneficiaries’ Understanding of Climate Change (max 20 points) Estimated Cost: Benefit Ratio (max 40 points)

Therefore, each sub-project was awarded a score out of a maximum 100 points.

For the first five criteria, the team arrived at the score by consensus. For each criteria, a table was provided defining the meaning of each score. The table for “technical quality” is reproduced below as an example. All the scoring tables are included in Annex 2 to this report.

Technical quality: Meaning of Evaluation Scores

Score Meaning

10 Very high technical quality: better than technical design and standards and no defects

8 High technical quality: conforms with the technical design and standards with no defects

6 Good technical quality overall. Some defects but these will not significantly reduce the benefits of the project

4 Poor technical quality. The benefits of the project will be reduced because of the low quality

2 Vey poor technical quality. The benefits of the project are much less than planned because of the poor technical quality.

0 Extremely poor technical quality. Because of the poor technical quality the project provides very small benefits or no benefits at all.

By ensuring that the scoring conformed fairly strictly to these tables, the scores awarded could be considered as comparable between projects of different types and potentially between different evaluation teams.

7


Technical Quality was considered mainly as meaning that the design and construction of the (infrastructure) outputs was suitable to ensure the intended sub-project benefits and the long-term sustainability of the sub-project. For non-infrastructure projects the team attempted to award equivalent scores based on the quality and effectiveness of services provided, for example the quality of a training as described by the beneficiaries and as evidenced by the beneficiaries’ understanding of the training topic.

Sustainability was defined in terms of the level of maintenance effort the community would have to provide to ensure that the sub-project would continue to provide benefits for 10 years or longer. Therefore a sub-project scored maximum points if benefits were expected to continue for 10 years without any specific maintenance effort being required.

Negative impacts were evaluated in terms of the seriousness of any negative impacts, compared to the benefits of the sub-project; whether there was any plan to mitigate or manage the negative impacts; and the feelings of the local community in respect of the negative impacts. A sub-project could score zero if “The negative impacts from the project are bigger than the good impacts and most members of the community are angry about the results of the project.”

Relevance to Climate Change Adaptation was evaluated on a scale in which the maximum score was awarded to a sub-project that directly responded to a direct impact of global climate change. Lower scores were awarded for sub-projects that responded to natural disasters (not necessarily directly linked to climate change); and to sub-projects that did not respond to climate threats directly but were located in areas that were vulnerable to climate change impacts.

The impact on beneficiaries’ understanding of climate change criterion awarded high scores to sub-projects that had a demonstrable impact on beneficiaries’ understanding. It was important to ensure that the evaluation team understood that points should be awarded for change attributable to the sub-project, not for the beneficiaries’ good understanding of climate change that they might have gained from participating in quite separate activities.

Estimation of the relationship of sub-project costs to benefits was the most difficult part of the evaluation as it required an estimation – either explicit or implied – of the money value of the benefits. Further, this required a methodology that could be applied with consistent results across a wide range of sub-project types.

Initially an attempt was made to obtain a valuation of the sub-project benefits from the beneficiaries by asking them to rank the benefits from the actual sub-project with easily understood benefits from a range of (hypothetical) example sub-projects. This method had been found to work well in Cambodia in the past but for whatever reason, it produced no useful results in this case as the beneficiaries routinely ranked the sub-project benefits higher than all the example sub-projects, even where this was clearly implausible1. Therefore this technique was abandoned and an alternative, based on relative ranking of the sub-project benefits according to the perceptions of the evaluation team, was substituted.

1 For instance, ranking the benefit obtained from a half-day CC awareness training above the grant of a hand-tractor. No farmer would make this choice in real life.

8


In this technique, each member of the evaluation team was asked to rank the value of the benefits obtained by one typical beneficiary household, from each sub-project, i.e. the sub-project with the highest benefits per household at the top and the sub-project with the lowest benefits per household at the bottom. Using a ranking method prevents the evaluation team from routinely inflating the values of the sub-projects – if one sub-project is marked up, another must be marked down, and because each team member ranked the sub-projects separately the impact of any individual favouritism would be reduced. It was important for the team members to understand that a high ranking does not necessarily mean that a sub-project is “better” than one with a low ranking – the costs as well as the benefits must be taken into account.

The rankings provided by each team member were combined into an average ranking for each sub-project2 and the sub-projects were then divided into three groups: the top third of the ranking table being considered as providing “high benefits per household”; the middle third as providing “moderate benefits per household” and the bottom third as “low benefits per household.” A further possibility, of a sub-project providing zero benefits (i.e. the sub-project completely failed) was allowed.

The costs per beneficiary household (i.e. the total cost of the sub-project, including the PBCR contribution and any co-financing, divided by the total number of households) were calculated, ranked, and divided into equal thirds with “high”, “medium”, or “low” cost per household.

The rankings of cost per household and benefits per household were then compared and a points score awarded to each project based on the table below.

Cost Per Household Benefits Per Household Points ScoreLow High 30Low Medium 25Medium High 25Low Low 20Low Low 20High High 20Medium Low 15High Medium 15High Low 10

No benefits 0

The score obtained from this system s relative: it only has meaning in comparison to the other sub-projects in the sample. Unless any sub-project fails completely, the average score for the whole sample will always be 20 points. The score is useful for determining which sub-projects and types of sub-project provided best value for money, but it cannot tell us anything about the overall performance of the programme.

With further development, a scale of cost per household could be developed. The scale would have to be appropriate to local prices and conditions: the scale for Cambodia would be very different from the scale for the Solomon Islands, for example.

2 This required some complex manipulation as the composition of the evaluation team varied between sub-projects and different evaluators had evaluated different numbers of sub-projects.

9


With greater difficulty, it might be possible to develop a scale of rating of sub-project benefits. One possible approach would be for the evaluation team to compare the benefits from the sub-project against other good events that could have an impact on a beneficiary household: for example, winning a significant amount in a lottery (high benefit); obtaining a microfinance loan (medium benefit); reduction in the price of petrol (low benefit).

The scores for all six criteria were totaled to give a score out of a maximum of 100 for each sub-project in the sample.

5.5 Provincial Reflection WorkshopA reflection workshop was held on 2nd April 2013 to present and discuss the results of the participatory evaluations. Participants included representatives of the beneficiaries who took part in the evaluations, Commune/Sangkat Councils, District/Municipal administrations, the Provincial Administration and also a delegation from the Provincial Administration and District Administrations in Battambang province that are beginning implementation of LGCC. Representatives of the donor (CCA Trust Fund) were also invited.

The agenda of the workshop included:

A summary of the results of the LGCC project, by the Provincial Administration; A presentation on the methodology of the participatory evaluations, by NCDDS; Presentations of the findings of participatory evaluations, by beneficiary representatives,

using the flip-chart summaries prepared in the field; Presentation of the scoring by the evaluation team, by UNCDF; Discussion and identification of lessons learned.

The presentations by the beneficiaries are attached as Annex 3.

6 Findings of Participatory Evaluations

6.1 Technical QualityTechnical quality of the infrastructure outputs was acceptable overall, but with a number of points noted on which improvements could have been made. Only one of the six infrastructure sub-projects (the road in Borei Chulsar Commune) were evaluated by the national team as scoring 8 / 10, i.e. “High technical quality, conforms to the design and with no defects.” In general the beneficiaries’ were aware of technical problems with the outputs and expressed views consistent with those of the evaluation team.

The water gate in Baray Commune was found to leak water, when closed, at a higher rate than should occur if the gate seals were properly constructed. It was also noted that there was a lack of any kind of energy dispersion features in the downstream side of the gate – these are normally included as standard in any irrigation structure and serve mainly to reduce erosion of the channel bed and side banks downstream of the structure. For these reasons the Baray water gate was scored only as 5/10 for technical quality by the evaluation team.

The beneficiaries were additionally disappointed in the Baray structure because they felt that the canal should have been excavated deeper – this would permit water to be stored for longer and

10


improve the potential for dry season cropping. This view is doubtful from a technical point of view – the volume of water that can be “stored” in a canal is small compared to the water demands of a rice crop, and the proper function of a canal is to transmit water from a source, not to store it – i.e. if water ceases to flow from the source, the canal is likely to become dry no matter how deep it is excavated. Nevertheless this complaint points to a need for better communication and discussion with the beneficiaries before implementing the sub-project.

The culverts at Kouk Phos were also scored only 5/10 for technical quality, because the slopes of the earth covering the culverts were much too steep and are likely to suffer erosion. There were varying technical views about the cause of this problem – probably the most sound engineering solution would have been to construct longer culverts, allowing the slope angles to be less steep.

No other serious defects were found in the infrastructure sub-projects – the remaining four projects were scored at 6/10 or 7/10, meaning that although there were minor defects these would not affect the benefits from the sub-project nor its useful life. The beneficiaries were critical of a number of points of detail, often concerning the quality of compaction and finishing of earthworks. One point that emerges from these discussions is that though the beneficiaries lack technical knowledge, they are interested in and perceptive of the quality of works so far as they are able to judge by appearances.

The non-infrastructure sub-projects were likewise judged to be of only acceptable technical quality. The tree-planting that was included with a climate change awareness raising in Doun Keo Municipality seems to have failed because the tree seedlings were eaten by grazing animals. A proportion (estimated at 30%) of the water filters supplied in Borei Chulsar, did not function correctly – the rate of filtration was very slow and in some cases the filtered water from these filters had a bad taste.

6.2 Beneficiaries’ Perceptions of BenefitsThe beneficiaries were in general very positive about the benefits obtained – or expected- from the sub-projects. It is important to note that the evaluation was conducted shortly after the sub-projects were completed, so before actual benefits could be expected in some cases, such as the canal and pond projects.

The beneficiaries expected the irrigation projects to bring benefits to substantial numbers of households: 173 in the case of the Baray water gate and 300 in the case of the Thnout canal, even though the number of families owning land directly accessible from the canals is probably rather lower. Beneficiaries noted that benefits could be improved by construction of tertiary or field canals to distribute water away from the main canal. Both the irrigation projects were expected to bring important secondary transportation benefits – due to the provision of a bridge on the water gate and a road running parallel to the canal.

The beneficiaries of the pond project in Lumpong commune strongly asserted the importance of the project, given the difficulties of obtaining water in the dry season in this area. Villagers from 5 villages are expected to use the pond, most likely by transporting water from the pond on motorcycles (the pond is some distance from any village centre, which caused the evaluation team to question the choice of location).

11


All the transportation (roads and culverts) projects were seen as providing high levels of benefits in terms of reduced time and difficulty of travel. In Borei Chulsar Commune, the road is expected to make a major difference in the wet season as the villages accessed by the road were previously cut off by floods for parts of the year.

The beneficiaries of the rice seed demonstration projects affirmed that the project had assisted them to improve their knowledge of rice production including selecting appropriate seeds and fertiliser use. It was not clear whether there would be a large number of farmers continuing to use the seed demonstrated, in part because of problems with the supply of seed.

Although the beneficiaries of the training on environmental awareness and climate change conducted in Rokar Krao Commune were highly appreciative of the sub-project, it was not so clear what specific benefits they had obtained, especially as the associated tree-planting activity, which was expected to provide shade, protect the canal bank and “attract rain” had failed as the seedlings had died or been eaten by animals.

The main benefits perceived by the beneficiaries of the health and hygiene sub-project in Borei Chulsar Commune were obtained from the water filters – using these allowed the families to reduce expenditure on bottled water and was said to have resulted in health improvements. The filters can potentially be transported to flood refuge areas in the event of a flood – this would be important as clean water supplies are a problem at flood refuge sites.

6.3 Negative ImpactsVery few specific negative impacts were identified from the sub-projects. The road in Borei Chulsar is likely to cause limited flooding during the coming wet season, however this is not directly attributable to the sub-project alone but to the nearby construction of a canal which is temporarily obstructing the drainage path. The beneficiaries expect the problem to be resolved once the construction of the canal is complete.

Seven of the sub-projects were assessed by the evaluation team as having no negative impacts (score 10/10) and the remaining two were assessed as having only minor negative impacts (8/10).

6.4 SustainabilityThere are concerns over the sustainability of all the infrastructure sub-projects, as there is only limited capacity for maintenance either at community or local administration level. The beneficiaries were able to propose measures that could assist in ensuring sustainability of the sub-projects, these were a mixture of physical measures (e.g. planting trees to protect canal banks), management measures (e.g. excluding animals from ponds and canals) and periodic maintenance works. However in most cases the resources to implement these measures are lacking. Only in the case of the pond at Lumpong the community state that they have a plan to construct a fence around the pond to manage access and exclude animals.

Sustainability scores awarded by the evaluation team ranged from 4/10 (poor sustainability as the community does not have the capacity to undertake actions needed to make the sub-project benefits sustainable) to 8/10 (high sustainability: the community has adequate capacity to undertake actions needed) with an average score of 6.4/10.

12


6.5 Relevance to Climate Change AdaptationAll the sub-project activities were of types that might well be prioritised by local communities irrespective of the threat of global climate change. The relevance of the sub-projects to climate change adaptation arises from the location of the sub-projects in areas that are vulnerable to climate change as well as the specific features of the sub-projects designed to reduce climate change impacts. The beneficiaries perceived climate change relevance in terms of securing water supplies in the dry season (4 sub-projects); reducing flood damage (3 sub-projects); improving rice production (5 sub-projects); improving access and providing refuge during floods (1 sub-project) and increasing understanding of climate change (1 sub-project).

The evaluation team scored the relevance of the sub-projects to climate change adaptation between 12/20 (principal relevance is that the sub-project is in an area that is vulnerable to climate change impacts) and 20/20 (responds directly to a climate change challenge) with an average score of 14.9/20.

6.6 Impact on Beneficiaries’ Understanding of Climate Change AdaptationWith the exception of the sub-project that directly provided awareness raising on climate change, the effectiveness of the sub-projects in increasing beneficiaries’ understanding of climate change appeared to be limited. The evaluation team scored the sub-projects between 4/20 (beneficiaries do not really understand how the sub-project relates to climate change) and 16/20 (beneficiaries have a good understanding but do not know what they can do to adapt to climate change) on this criterion with an average score of 10.7.

6.7 Cost-Benefit ScoresAs explained above the cost-benefit scoring methodology adopted is a relative one – it will always result in the same average score. Therefore the utility of the method is in examining which sub-projects were judged by the evaluation team to have produced a high level of benefits relative to the cost. There was not a clear pattern in the results, with the two lowest scoring sub-projects (15/30) being the Baray water gate and the rice seed demonstrations in Pea Ream. The two highest scoring sub-projects (25/30) were the Roka Knong road and the pond in Lumpong Commune.

7 Summary Of Findings And Lessons Learned

7.1 ImpactThe beneficiaries were in the main extremely positive about the benefits they had experienced or were expecting to experience from the sub-projects. The beneficial impacts of the sub-projects were also clear to the evaluation team in most cases. Where the beneficiaries expressed reservations, these generally concerned either technical quality or what they saw as missed opportunities to enhance the benefits through modified sub-project designs. There were few negative impacts from the sub-projects and in no case did the beneficiaries perceive that the negative impacts were such as to cancel out the benefits.

Sustainability of the sub-projects is clearly an issue. Sub-national administrations and beneficiaries are aware of the need for good management and maintenance of sub-project outputs but capacity and resources are generally lacking. Lack of regular maintenance is likely to lead to the deterioration

13


of sub-project outputs over time, leading to a reduction in the benefits and eventually to further rehabilitation works being required. It should be clearly understood that this lack of maintenance capacity does not imply that the sub-projects are not cost-effective or that they should not have been implemented, but only that the level of benefits obtained is less than would be the case if effective operation and maintenance arrangements were in place.

It was not possible to conduct a quantitative valuation of the benefits obtained from the sub-projects, so assessment of the relationship between costs and benefits is subjective. However, in the view of the evaluation team the actual benefits experienced and the expected future benefits from the sub-projects seemed more than adequate to justify the costs of the sub-projects in almost all cases.

7.2 Relevance to Climate Change AdaptationThe types of sub-projects funded by the Performance Based Climate Resilience Grants were not markedly dissimilar from the types of sub-projects prioritised by local communities and sub-natioal administrations for funding from the Commune/Sangkat Fund and other general funds for local development. The justification for regarding the PBCR grant investments as climate change adaptation investments is threefold. First, the sub-project types are all permitted according to the PBCR Grant Investment Menu which is an annex to the Memorandum of Understanding between NCDDS and UNCDF for implementation of LGCC. Second, the actual sub-projects were selected and prioritised in accordance with the District / Municipal Climate Change Adaptation Strategy, itself based on vulnerability reduction analyses and facilitated stakeholder dialogues on local climate change adaptation needs. Third, the treatment of the PBCR Grant contribution to infrastructure projects as “climate proofing costs” is underpinned by an effort to ensure that sub-projects are designed and constructed to climate resilient standards.

The infrastructure sub-projects in all cases were located in areas that are vulnerable to climatic disasters (floods and / or droughts) that are likely to be made more frequent or severe by climate change. The sub-projects aimed to reduce climate-related vulnerabilities directly, by ensuring improved water supplies for agriculture or domestic use, by improving drainage and by improving road access during flood seasons, as well as to ensure improved climate resilience of roads and canals.

The non-infrastructure sub-projects addressed direct climate change challenges including the need for more climate-resilient rice seed varieties and improved access to potable water during floods. One sub-project included climate change awareness raising as well as a tree-planting activity designed to improve the local environment.

The experience of planning and implementing the sub-projects has contributed to improved understanding of the challenge of climate change and of possible adaptation measures, amongst the sub-national administrations and the beneficiary communities. However more could be done to take advantage of the opportunity for education and awareness raising offered by the sub-projects.

7.3 Weaknesses Of the Sub-ProjectsSome of the concerns expressed by the beneficiaries indicated that there had been insufficient active participation by the beneficiaries in the design and implementation of the sub-projects. In some cases actual improvements to the designs could have resulted from closer consultations with

14


the beneficiaries; in other cases the beneficiaries would have gained a better understanding of the design and would not have been disappointed when the sub-project failed to fully meet their expectations (for example, the water gate in Baray Sangkat). The beneficiaries have a strong interest in the good quality of the outputs and are capable of identifying weaknesses in the quality of construction work even without expert technical guidance.

The technical quality of some of the works was slightly disappointing overall. None of the sub-projects were so badly constructed as to appear in any danger of imminent failure or so that the extent of benefits experienced would be severely reduced, but a number of cases were identified where simple improvements could have been made to the designs. Weaknesses in construction quality were evident to the (mainly non-technical) evaluation team and to the sub-project beneficiaries in a number of cases.

All the non-infrastructure sub-projects had weaknesses that reduced the sustainable benefits from the sub-projects. These weaknesses are linked to the technical capacity and resources available to the implementing agencies (the technical line offices at District level). These sub-projects serve a valuable purpose as they pilot the modalities of cooperation between the District Administration and the technical offices and the possibilities to extend the development activities of the sub-national administrations beyond infrastructure investments. The total cost of the non-infrastructure sub-projects and their cost per beneficiary were quite small. Nevertheless if this type of activity is to be supported at a larger scale, more technical support will be needed to ensure effectiveness.

7.4 Key points to strengthenThe findings of the participatory evaluations indicate that while the first round of pilot climate change adaptation sub-projects funded by the LGCC pilot was generally successful, there are a number of respects in which the quality of implementation could be strengthened.

The technical quality of both infrastructure and non-infrastructure sub-projects could be improved. There is a lack of capacity for technical design, particularly of more complex sub-project types. One option would be for the project to directly support technical assistance for sub-project design. The number of different sub-project types (infrastructure and non-infrastructure) is not very large and so it would be feasible for the project to support specialised technical assistance for the most common types, without restricting the freedom of the participating sub-national administrations to choose other types where these are a priority.

Stronger participation of local communities in the design stage of the sub-projects could lead to improved effectiveness and would also avoid disappointment when the sub-project is not able to provide all the benefits expected by the beneficiaries.

Technical quality of construction works should be improved by stronger monitoring arrangements. The sub-project beneficiaries are able to contribute usefully to monitoring of works under construction, when they are supported by a technical specialist.

More use could be made of the opportunities afforded by planning and implementation of the sub-projects, to raise climate change awareness amongst the beneficiary communities. This could be

15


done through distributing educational materials and by conducting participatory sub-project planning meetings at which the climate change adaptation purpose of the sub-projects would be clearly explained.

8 Assessment of the Evaluation MethodologyPart of the intention of the participatory evaluations was to develop and test a methodology for evaluations that would be appropriate to a wide range of different sub-project types, was not too costly or complex to carry out (taking into account the small size and cost of the sub-projects themselves) and would yield both useful insights into the strengths and weaknesses of the sub-projects and robust, cross-comparable assessments of sub-project impacts.

Using the evaluation methodology it was possible for a small team of 2 – 3 evaluators to conduct evaluations of 3 sub-projects per day. This is sufficient to allow a small but representative sample of sub-projects to be evaluated annually without excessive cost.

The beneficiaries were able to participate in and contribute perceptively to the evaluation. The constraint is not the understanding of the beneficiaries but their ability to express their views in a structured way. Structuring the discussion around a set of key themes and encouraging the beneficiaries to record in writing a short summary of views on each theme proved an effective approach. It is important for the facilitator to understand how to help the beneficiaries to express and record their views, including understanding the importance of hearing dissenting views and allowing negative aspects to be clearly represented in the results.

The evaluation team had the dual role of facilitating beneficiary discussions and also independently scoring the sub-projects against a set of pre-defined criteria. Using clear definitions and examples of the meaning of each score helps ensure consistency and cross-comparability between sub-projects and between evaluations by different teams.

The most difficult challenge in the evaluation was to develop a consistent approach to assessing the relationship between sub-project costs and benefits. Given the small size of the sub-projects, the limited resources for the evaluation and the wide range of different sub-project types it was not possible to carry out a quantitative valuation of sub-project benefits. Instead, the method used relied on the subjective assessments of the evaluation team members and on ranking the sub-projects in the sample to assess the relative level of benefits per beneficiary household. Using the method as applied here, the results of this “cost-benefit score” cannot be cross-compared with other samples (for example, to identify improvements from year to year). More work could be done on developing a robust and cross-comparable approach to estimating the value of sub-project benefits.

16


Annex 3: List of LGCC Sub-Projects 2012LGCC Infrastructure Projects In Takeo

No

Commune/Sangkat

Project Name CCA Relevance Expected Impact CCA Grant

Total

1- Borei Chulsar District ( Infrastructure)1.1

Borei Chulsar Comune

Restoring rural laterite road ( Increasing height of road)

This is an area affected by annual flooding. Road access is difficult during the flood season. This problem will become more severe with climate change.

At the design stage, the height of the road needed to avoid flood damage will be determined and the adequacy of cross-drainage will be checked. The road will be built up more resilent to the water level during flooded season by increasing height of road and it will enable local communities to easily access during the annual flood season. This also could open access for local communities to transport their agricultural product to market, childrend go to school, and access to health care center

$ 7,500.00

$ 22,500.00

1.2

Kok Por Commune

Restoring rural laterite road ( Increasing height of road)


At the design stage, the height of the road needed to avoid flood damage will be determined and the adequacy of cross-drainage will be checked. The road will be built up more resilent to the water level during flooded season by increasing height of road and it will enable local communities to easily access during the annual flood season. This also could open access for local communities to transport their agricultural product to market, childrend go to school, and access to health care center

$ 6,500.00

$ 19,500.00

1


1.3

Dong Kpos Commune

Installing water drainage/culvert

This is an area affected by annual flooding. Flood flows damage road infrastructure and standing floodwater damages crops. This problem will become more severe with climate change.

The project, implemented in conjunction with road repairs funded through the C/S Fund, will result in improved cross-drainage of roads and quicker relief of floodwaters, thus reducing damage to roads and to crops.

$ 3,631.00

$ 10,893.00

Sub-total $ 17,631.00

$ 17,631.00

2. Daunkeo Municipality ( Infrastructure)2.1

Baray Sangkat Building water gate

Increasingly irregular weather patterns are a problem for agriculture. Improved capacity for water management is a key climate resilience response.

91 ha of paddy field in two villages where are annualy flooded, will be rescued during the flooded season by regulating/draining water out of paddy field via the gate or retaining water when needed.

$ 10,000.00

$ 10,000.00

2.2

Roka Krao Sangkat

Building sewage water pipe

Increasingly severe storms result in flooding around the health centre. The floodwaters cannot drain away and become contaminated with sewage.

The culvert will relieve the polluted flood waters around the health centre. The local community will have better health, the water polution will be reduced and sewage water will be well manged during flood season

$ 6,400.00

$ 6,400.00

2.3

Roka Knong Sangkat

Restoring rural road ( Increasing height of road)


At the design stage, the height of the road needed to avoid flood damage will be determined and the adequacy of cross-drainage will be checked. The road will be built up more resilent to the water level during flooded season by increasing height of road and it will enable local communities to easily access during the annual flood season. This also

$ 2,629.00

$ 2,629.00

2


could open access for local communities to transport their agricultural product to market, childrend go to school, and access to health care center

Sub-total $ 19,029.00

$ 19,029.00

3. Bati District ( Infrastructure)3.3

Lumpong Commune

Restoring natural community pond

Longer and dryer dry seasons will reduce water availability, particularly for vegetable gardening and livestock.

Restoring the pond will increase its storage capacity and ensure that there is sufficent water to support vegetable gardening, livestock and fishing activities in the dry season

$ 6,100.00

$ 6,100.00

3.4

Krang Tnong Commune

Restoring canal

Longer, dryer dry seasons and shorter, more intense wet seasons will be a challenge for traditional agriculture. Improving community water management is a key CCA response.

The principal benefit of this canal is expected to be improved drainage for relief of floodwaters that cause damage to the wet season crop. The canal will have a secondary benefit as a supplementary water source for the wet season crop during drought periods. The community also hope to expand dry season cropping although whether this is realistic will be determined during the design phase.

$ 8,100.00

$ 8,100.00

3.5

Thnout Commune

Restoring natural community pond

Longer and dryer dry seasons will reduce water availability, particularly for vegetable gardening and livestock.

Restoring the pond will increase its storage capacity and ensure that there is sufficent water to support vegetable gardening, livestock and fishing activities in the dry season

$ 10,000.00

$ 10,000.00

3.6

Thnout Commune

Restoring canal

Longer, dryer dry seasons and shorter, more intense wet seasons will be a challenge for traditional agriculture. Improving community water management is a key CCA

The principal benefit of this canal is expected to be improved drainage for relief of floodwaters that cause damage to the wet season crop. The canal will have a secondary benefit as a supplementary water source for the wet season crop during drought periods. The community

$ 12,452.85

$ 12,452.85

3


response. also hope to expand dry season cropping although whether this is realistic will be determined during the design phase.

Sub-total $ 36,652.85

$ 36,652.85

Grand Total $ 73,312.85

$ 73,312.85

LGCC Service Projects in Takeo

No

Target Proposed Project

Implementer

CCA Relevance

Expected Impact Budget Total

1 Borei Chulsar District

Disiminating Sanitation/Drinking Water/Heath Care and distributing water filter

Health Office

Longer, dryer dry seasons will reduce potable water availability. Improved hygiene knowledge and household water treatment are key CCA responses.

Local community will be aware of how to use and drink cleaning water during flooded season and their health will get better. The incidence of water-borne diseases will be reduced.

$ 4,149.00

$ 4,149.00

Sub-total $ 4,149.00

$ 4,149.00

2 Daun Keo Municipality

Disiminating resilient rice seed

Municipality

Changing seasonal patterns are a challenge for traditional rice agriculture. Climate resilient seeds will reduce losses from flood or drought.

Local communities will got proper technique in growing drought/flooded resilience rice varieties and they could apply this technuques to grow these resilient rice verieties during drought and flood season. From this, they could secure their food

$ 2,277.00

$ 2,277.00

4


during flood/drought seasons and they could adapt to the Climate Change

Disiminating CCA and Environment Concept

Municipality

In order to build climate resilience, local communities need a better understanding of the climate and environmental challenges they are faced with.

Local community and relevant stakeholders will aware of the concept of CCA. From this, they could use these khnowledge in regard with problem/cause/solution analysis in their local planning process.

$ 2,200.00

$ 2,200.00


$ 4,477.00

3 Bati District

Disiminating Gender in CCA and distributing water filter

Women Affair Offce

Climate change will have specific impacts on women, particularly in the area of water availability and household hygiene and water treatment.

Local community will be aware of ToR of Gender in CCA and they understand how to use and drink cleaning water during flooded season and their health will get better. The incidence of water-borne diseases will be reduced.

$ 5,500.00

$ 5,500.00

Training on resilient rice technique

Agriculture Office

Changing seasonal patterns are a challenge for traditional rice agriculture. Climate resilient

Local communities will got proper technique in growing drought/flooded resilience rice varieties and they could apply this technuques to grow these

$ 3,124.00

$ 3,124.00

5


seeds will reduce losses from flood or drought.

resilient rice verieties during drought and flood season. From this, they could secure their food during flood/drought seasons and they could adapt to the Climate Change


$ 8,624.00

Grand Total $ 17,250.00

$ 17,250.00

6


Annex 3: Scoring Tables

Independent Evaluation by National Team

Province DistrictCommune Sub-Project TypeYearDescription of sub-project outputs

Description of benefits for average household (from participatory evaluation):

Criterion Value / ScoreNumber of beneficiary householdsCost of sub-project (Total)Cost of sub-project from PBCR grantEstimated value of benefits/HHEstimated Benefit: Cost RatioTechnical Quality (Max 10 points)Sustainability (Max 10 points)Negative Impacts (Environment and Natural Resources) (No negative impact: 10 points)Relevance to Climate Change Adaptation (Max 20 points)Impact on Beneficiaries’ Understanding of Climate Change (Max 20 points)Cost-effectiveness (based on benefit: cost ratio, max 30 points)Total score out of 100

Technical quality: Meaning of Evaluation Scores

Score Meaning

10 Very high technical quality: better than technical design and standards and no defects

8 High technical quality: conforms with the technical design and standards with no defects

6 Good technical quality overall. Some defects but these will not significantly reduce the benefits of the project

4 Poor technical quality. The benefits of the project will be reduced because of the low quality

2 Vey poor technical quality. The benefits of the project are much less than planned because of the poor technical quality.

0 Extremely poor technical quality. Because of the poor technical quality the project

1


provides very small benefits or no benefits at all.

Sustainability: Meaning of Evaluation Scores

Score Meaning

10 Very high sustainability. The project benefits will continue for at least 10 years with no further actions needed to ensure sustainability

8 High sustainability. Some activities are needed to ensure sustainability (project benefits continue for at least 10 years) but the community has enough capacity to implement these activities.

6 Good sustainability. The project benefits can continue for at least 10 years but only if appropriate activities are implemented. It will be difficult for the community to implement these activities.

4 Poor sustainability. The activities that are needed to ensure sustainability (continue project benefits for 10 years) are beyond the capacity of the community.

2 No sustainability. The project will provide benefits for maximum 5 years or less.

0 The project benefits will continue for maximum 1 year or less.

Negative Impacts: Meaning of Evaluation Scores

Score Meaning

10 No negative impacts from the project

8 Only very minor negative impacts from the project

6 There are some negative impacts but there is a plan to reduce the negative impacts

4 The project has some serious negative impacts and there is no plan to reduce the negative impacts

2 There are very serious negative impacts. Some people in the community are angry about the project because of these negative impacts.

0 The negative impacts from the project are bigger than the good impacts and most members of the community are angry about the results of the project.

Relevance to Climate Change Adaptation: Meaning of Evaluation Scores

Score Meaning

20 The project responds directly to a challenge caused by global climate change trends

16 The project responds directly to a challenge caused by the climate or by natural disasters (not directly caused by global climate change).

12 The project output is in an area that is vulnerable to a specific climate challenge and is

2


either (1) infrastructure designed and constructed to climate resilient standards or (2) a service activity that will help reduce the vulnerability of the community.

8 The project output is either (1) infrastructure designed and constructed to climate resilient standards or (2) a service activity that will help reduce community vulnerability but the project is not in an area that is vulnerable to any specific climate challenge.

4 The project makes only a limited and indirect contribution to building climate resilience.

0 The project does not help build climate resilience in any way.

Impact on Beneficiaries' Understanding of Climate Change: Meaning of Evaluation Scores

Score Meaning

20 Through planning and implementing the project, the beneficiaries have learned a lot about the challenges of climate change and about strategies to build climate resilience

16 Through planning and implementing the project the beneficiaries have a good understanding of the challenges of climate change but they do not know what they can do to build climate resilience.

12 Through planning and implementing the project the beneficiaries have learned more about climate change than they knew before but their understanding is still quite weak

8 The beneficiaries understand how the project helps build climate resilience but they have not really learned anything new from the project

4 The project beneficiaries have heard that the project is something to do with climate change but they do not really understand how the project builds climate resilience.

0 The project beneficiaries do not know that the project has anything to do with climate change

Benefit: Cost Ratio: Meaning of Evaluation Scores

Score Meaning

30 Low cost high benefit (per household)

25 Medium cost – high benefit or low cost – medium benefit

20 High cost – high benefit, medium cost - medium benefit or low cost – low benefit

15 High cost – medium benefit or medium cost – low benefit

10 High cost – low benefit

0 There are no valuable benefits from the project

3


Annex 3: Beneficiary Evaluations of Sub-Projects(As presented at the reflection workshop in Takeo on 2nd April 2013)

1

· web viewwas implemented by ncdds in three local administrations in takeo province: doun keo...

Documents