evaluating the impacts of participatory planning for urban water

Evaluating the impacts of participatory planning for urban water infrastructure and rural livelihoods adaptation in Indonesia Small Grant Report to the Australia Indonesia Centre

Seona Meharg, James Butler, Dewi Kirono, Neil Lazarow (CSIRO)

Hannah Barrowman (Australian National University)

Kate Duggan (Griffin NRM Ltd.)

October 2015

LAND AND WATER

Citation

Meharg, S., Butler, J., Kirono, D., Lazarow, N., Barrowman, H. and Duggan, K. 2015. Evaluating the

impacts of participatory planning for urban water infrastructure and rural livelihoods adaptation in

Indonesia. Small Grant Report to the Australia Indonesia Centre. CSIRO Land and Water, Canberra.

Copyright

© Commonwealth Scientific and Industrial Research Organisation 2015. To the extent permitted

by law, all rights are reserved and no part of this publication covered by copyright may be

reproduced or copied in any form or by any means except with the written permission of CSIRO.

Important disclaimer

CSIRO advises that the information contained in this publication comprises general statements

based on scientific research. The reader is advised and needs to be aware that such information

may be incomplete or unable to be used in any specific situation. No reliance or actions must

therefore be made on that information without seeking prior expert professional, scientific and

technical advice. To the extent permitted by law, CSIRO (including its employees and consultants)

excludes all liability to any person for any consequences, including but not limited to all losses,

damages, costs, expenses and any other compensation, arising directly or indirectly from using this

publication (in part or in whole) and any information or material contained in it.

CSIRO is committed to providing web accessible content wherever possible. If you are having

difficulties with accessing this document please contact [email protected].

mailto:[email protected]

4 | Evaluating the impacts of participatory planning for urban water infrastructure and rural livelihoods adaptation in Indonesia

Executive summary

In 2009-2014 the Department of Foreign Affairs and Trade (DFAT)-CSIRO Research for Development (R4D)

Alliance (‘the Alliance’) brought together the research skills of CSIRO and its international research partners

with the development knowledge and networks of the Australian aid program to enhance poverty reduction

in the Southeast Asian region. The partnership provided an opportunity to develop innovative approaches to

international aid delivery by better understanding the relationship between poverty and the environment.

Six major collaborative projects were implemented, covering Vietnam (two projects), Bangladesh (one),

Indonesia (two) and the Greater Mekong Region (one). The research investigated complex development

challenges in the domains of climate, water resources, sustainable cities, and food security.

This Australia Indonesia Centre (AIC) Small Grants project enabled us to refine an evaluation methodology

developed as part of the Alliance. In this case, the two Indonesian projects and partners were the focus:

Makassar Sustainable Urban Development (with Hasanuddin University), and Climate Adaptation Strategies

for Rural Livelihoods in Nusa Tenggara Barat Province (with the University of Mataram). The Small Grants

project involved repeating evaluation workshops and stakeholder interviews in April-June 2015 that had first

been undertaken at the end of the Alliance projects in April-June 2014, thus enabling the tracking of progress

along each project’s Impact Pathway 1 year after project completion.

A significant limitation amongst many current impact evaluation methodologies is that a narrow set of

approaches is often used to evaluate complex development issues. This means that impacts may be only

partially described, and this is compounded by a lack of clarity around the validation and attribution of

impacts to particular interventions. The Alliance approach applied mixed methods to overcome many of

these challenges, allowing us to evaluate and quantify the impact of the individual projects, and to provide

an opportunity for reflection by the project teams and hence further build capacity through learning.

Our approach was based on three facets:

1. A Theory of Change and Impact Pathway exercise and diagram which created a ‘roadmap’ for each

project’s assumed progress and related outputs, outcomes, impacts and goals.

2. A self-reflection workshop amongst the Indonesian research partners which mapped key

achievements against the project’s Impact Pathway, and discussed reasons for lack of progress and

necessary remedial actions.

3. An impact evaluation survey of the research team and boundary partners, which asked interviewees

to score 18 indicators linked to phases of the Theory of Change and Impact Pathway, providing

additional perspectives and triangulation of results from the self-reflection workshops.

The results of the exercises demonstrated that with a small additional investment, project impacts can be

effectively estimated and attributed. In addition, the process of self-reflection was shown to rekindle project

teams’ efforts to maintain momentum, and to tackle barriers to impact that they had identified. We suggest

that the Alliance methodology demonstrated by this Small Grants project could be usefully applied to other

projects in the AIC portfolio.

Evaluating the impacts of participatory planning for urban water infrastructure and rural livelihoods adaptation in Indonesia | 5

Recommendations

Reflections from the Indonesian and CSIRO research teams on the evaluation method enabled us to make

several recommendations to the AIC:

1. The Theory of Change and Impact Pathways concepts were easily understood and accepted as

‘roadmaps’ for the projects, and could be useful planning and evaluation tools for AIC projects.

2. The self-reflection workshops were very useful exercises which enabled the research teams to

analyse their achievements, and identify actions needed to maintain progress along their Impact

Pathways after project completion. However, clear actions and responsibilities must be decided

during these exercises to encourage the teams to follow-up.

3. The impact evaluation surveys revealed additional information from both researchers and boundary

partners which complemented the self-reflection workshops, and the indicators provided useful

graphic representations of progress along the Impact Pathway over time. However, Indonesian team

members undertaking the interviews must be adequately resourced to complete the interviews.

4. To have maximum utility for the Indonesian research teams, the results of the evaluations must be

returned to them following analysis within 2-3 months. If not, momentum and interest wanes, and

any actions to be followed-up are forgotten. Ideally, Indonesian researchers should be trained to

undertake the entire process independently.

5. The self-reflection workshops and impact evaluation surveys should be incorporated into projects

from the outset, rather than being conducted ex-post when the greatest opportunities for research

teams to respond have largely been lost.


Acknowledgments

This report has been funded by the AIC with in-kind support from CSIRO, and in partnership with

Hasanuddin University (UNHAS) and the University of Mataram (UNRAM). Our in-country project

partners included:

Sustainable Urban Development, Makassar: UNHAS, Pusat Pengelolaan Ekoregion Sulawesi Maluku,

Public Work Agency Makassar City, Makassar Environmental Agency, Makassar Water Utility Company,

Indonesian Bureau of Meteorology and Geosciences, DFAT.

Climate Adaptation Strategies for Rural Livelihoods in Nusa Tenggara Barat (NTB) Province: UNRAM,

Indonesian Agricultural Technology Assessment Agency, Indonesian Bureau of Meteorology and

Geosciences, NTB Climate Change Task Force, NTB Planning Agency, NTB Environmental Research Board,

NTB Food Security Agency, United Nations World Food Program, Indonesian Ministry of the Environment,

EcoRegions Indonesia, DFAT.


Contents

Executive summary ......................................................................................................................... 4

Recommendations .............................................................................................................. 5

Acknowledgments ........................................................................................................................... 6

List of acronyms and key terms ...................................................................................................... 8

1 Introduction ........................................................................................................................ 9

2 Approach ........................................................................................................................... 11

2.1 Overview .............................................................................................................. 11

2.2 Step 1 – Define Theory of Change for each project ............................................ 12

2.3 Step 2 – Impact Pathway Evaluation ................................................................... 13

2.4 Step 3 – Analysis .................................................................................................. 17

3 Projects under evaluation ................................................................................................. 18

3.1 Sustainable Urban Development, Makassar ....................................................... 18

3.2 Climate Adaptation Strategies for Rural Livelihoods in Nusa Tenggara

Province ............................................................................................................................ 19

4 Evaluation results.............................................................................................................. 21

4.1 Sustainable Urban Development, Makassar ....................................................... 21

4.2 Climate Adaptation Strategies for Rural Livelihoods in Nusa Tenggara Barat

Province ............................................................................................................................ 26

5 Discussion ......................................................................................................................... 32

6 References ........................................................................................................................ 35


List of acronyms and key terms

AIC Australia Indonesia Centre

BMKG Bureau of Meteorology, Climatology and Geophysics

CSIRO Commonwealth Scientific and Industrial Research Organisation

DFAT Department of Foreign Affairs and Trade (Australia)

ERI EcoRegions Indonesia

KPDT National Ministry for Regional Development (Indonesia)

NGO Non-government organisation

NTB Nusa Tenggara Barat (West Nusa Tenggara Province, Indonesia)

ODI Overseas Development Institute

R4D Research for development

ToC Theory of Change

UN United Nations

UNRAM University of Mataram

USAID United State Agency for International Development

WFP World Food Program

3iE International Initiative for Impact Evaluation

Key terms

Adaptive capacity – the potential for actors within a system to respond to changes, and to create changes

in that system (Chapin et al., 2006, p. 16641).

Adaptation strategies – describes the (coordinated) response to existing or anticipated effects of global

change in order to facilitate the adjustment of communities. In the context of climate change, this

generally refers to the need to adjust human society and ecological systems to altered climate regimes

(Mawdsley et al., 2009). For the purposes of this report, we infer that adaptation strategies also include

responses to significant or systemic societal shocks such as large economic or demographic changes.

Theory of Change – articulates a vision of social change, and systematically describes the assumptions

about the sequential relationships between interventions and change (Vogel, 2012; Bours et al., 2014).

Impact Pathway – mapping out causality between interventions and expected outcomes and impacts,

normally using boxes and arrows (Douthwaite et al., 2009).

Change agents – people who act as catalysts for change (Rogers, 2003).

Boundary partner – individuals with whom research teams interact directly and intend to influence (Earl

et al., 2002).


1 Introduction

International development is complex. Many assumptions about the direct relationship between an

intervention and its impact can be weak or even untested, presenting challenges for the thorough evaluation

of investment efforts. For example, investment in capacity building projects is usually intended to generate

impact over time, but will often demonstrate little positive return at the immediate completion of projects.

These issues are well illustrated through investment efforts that aim to assist vulnerable communities in

developing countries to adapt to global change (e.g. globalised markets, population growth and climate),

where responses need to be integrated across multiple actors (e.g. planning, emergency services, primary

industries), at multiple scales, and over time (Wise et al., 2014; Butler et al., 2014).

Development banks, the UN and other NGOs continue to develop guidance for impact evaluation; and

Australia’s 2014 Aid Strategy has also emphasised the need for improved evaluation of aid effectiveness.

These agencies conduct impact evaluation for multiple purposes, broadly classed as follows:

To inform policy, program and project direction;

To contribute to improved development outcomes;

To facilitate internal learning - improve effectiveness and efficiency of delivery; and

To meet regulatory or auditing obligations - such as governance and reporting requirements that

document project activity, outputs, and resourcing.

Most assessments seek to evaluate actual impacts measured with metrics described in terms such as health,

nutrition, education and improved economic outcomes for affected people; and relative to costs (or avoided

costs). The most challenging or contested component of these assessments has been the identification of

where pathways to impact have been identified or created, and future benefits are calculated.

Assessments often take place in complex policy settings and systems with multiple actors across levels and

sectors, as was the case for the DFAT-CSIRO R4D Alliance (‘the Alliance’) projects. Accurately defining,

measuring and attributing impacts are therefore central to describing and communicating the success of

these investments. Better quantification of impacts as a result of activities (e.g. research, vaccinations, policy

input) and identification of the contributory impact (i.e. attribution) when multiple actors influence the

outcomes (e.g. White, 2012), provides evidence for agencies to inform direction and priorities for aid

investment.

Single methods will rarely be able to adequately capture the societal benefits of R4D investment. Practice

leaders worldwide, including but not limited to DFAT, 3iE, ODI, World Bank, Oxfam and USAID, continue to

use ‘mixed’ or ‘multiple methods’ to evaluate the impact of research for development. These approaches

have been refined over the past 30 years in recognition of the need to better capture the impact of often

complex, multi-goal and multi-partner projects. ‘Mixed methods’ can refer to the data collection techniques

or the design of the evaluation - we use it to mean both.

Mixed method approaches integrate quantitative and qualitative approaches to data collection, analysis and

synthesis. By integrating several approaches, evaluators are attempting to capture the broader range of

impacts and challenges, while minimising the limitations of any one method or approach. Employing a mixed


method approach builds a more robust evidence base that could not be established otherwise (Wiek et al.,

2014). This also enables a triangulation of results, and can ‘ensure that understanding is improved by

integrating different ways of knowing’ (Better Evaluation, 2015). These approaches also generally promote

greater integration of stakeholder perspectives and a more participatory approach, which in itself can

generate further capacity building through learning (e.g. Bours et al., 2013; Butler et al., 2015a, b; Butler et

al., in review).

In 2009-2014 the Alliance brought together the research skills of CSIRO and its international research

partners with the development knowledge and networks of the Australian aid program to enhance poverty

reduction in the Southeast Asian region. The partnership provided an opportunity to develop innovative

approaches to international aid delivery by better understanding the relationship between poverty and the

environment. Six major collaborative projects were implemented, covering Vietnam (two projects),

Bangladesh (one), Indonesia (two) and the Greater Mekong Region (one). Research investigated complex

development challenges in the domains of climate, water resources, sustainable cities, and food security.

In 2010-2014, two of the Alliance projects saw the Hasanuddin University (UNHAS) and the University of

Mataram (UNRAM) collaborate with CSIRO to design adaptation strategies for urban water infrastructure

and agriculture to climate change and natural disasters in the provinces of South Sulawesi and Nusa Tenggara

Barat (NTB), Indonesia. Using a participatory planning process the projects engaged provincial and district

governments, NGOs, the private sector and communities to analyse potential future impacts from climate

change and extreme events, population growth and other drivers. Multi-disciplinary research teams were

established in UNHAS and UNRAM to tackle the complex and multi-scale issues involved. A systems-based

evaluation method was scoped and designed by the Alliance and applied to all projects, including the

Makassar and NTB projects, to assess the outcomes and impacts of the research and planning processes.

This AIC-funded Small Grants project built on the previous evaluation activities, and enabled further capacity-

building amongst the CSIRO, UNHAS and UNRAM research teams to develop and apply novel systems-based

approaches to project evaluation using Theory of Change (ToC) and Impact Pathways concepts. By re-

assessing the projects approximately 1 year after the prior Alliance evaluation, we hoped to learn more about

how participatory planning processes impacted stakeholders’ adaptive capacity and policy change, and

barriers to this process following project closure. Activities included workshops with cross-institutional

project teams and stakeholders to develop ToC and Impact Pathways to identify and tackle barriers to

implementation, and design frameworks for ongoing project monitoring and evaluation.


2 Approach

2.1 Overview

Impact evaluations are by their nature sequential. That is, change (an intervention) needs to be defined and

measured before impact can be described (attributed). With the benefit of a clearly articulated ToC narrative,

approaches and tools can be deployed to quantify research impact and improved pathways to impact (e.g.

Bours et al., 2014; Butler et al., in review). Through this process appropriate methods can be used to capture

different parts of the impact story. Different approaches to address the same question also enhance

confidence in evaluation, validity and reliability, and results from one approach may be used to inform the

design of future stages of the evaluation. In 2013-2014, we designed and implemented a mixed methods

approach to evaluate and quantify the impact of the Alliance at both project and program level.

Our approach was based on a clear set of criteria such as the ability to interrogate impact against the project

ToC, scalability of the approach, ability to assist with validation, ability to provide a mix of quantitative and

qualitative findings, and the inclusion of methods for reflection on pathways to impact by delivery teams and

boundary partners. Our approach was structured by a three step process, as depicted in Figure 1 below:

1. Define a ToC for each project to help explain conditions prior to and at the conclusion of the projects,

including project contribution;

2. Evaluate Impact Pathways through reviews, structured interviews, surveys and workshops against

measures of impact consistent with theory at project level, and

3. Reflection and learning with research teams and boundary partners to provide cross-Alliance project

design evaluation and describe key opportunities to realise improved development impact from R4D.

Figure 1. The Alliance impact evaluation steps


By re-surveying the same key partners and stakeholders approximately 1 year after the previous evaluation,

the AIC project enabled improved understanding of the effectiveness of the Alliance’s investment and impact

over time. Evaluating two case studies in Indonesia also provided an opportunity for comparison.

2.2 Step 1 – Define Theory of Change for each project

At completion of the Alliance projects in June 2014, the CSIRO project teams undertook a review of project

material to develop a framework of analysis consistent with overall project objectives, stakeholder

engagement and intended outcomes. Following from this, a structured interview process was designed to

compile ‘stories’ from inception and identification of a research idea or issue through research process into

dissemination of research findings and on towards its impact on the welfare of people and society. The

objective of the structured interviews was to facilitate articulation of hypothesised (at the project design

stage) and observed (after project implementation) impact and impact pathways considering the value chain

from research to impact. Project teams were asked to describe the links between:

1. Inputs, activities and outputs;

2. Outcomes (defined as the changes in behaviour that result from the building of knowledge, skills and

tools, evidenced in policy and practice that transform the way things are done); and

3. Impacts (defined as social and economic impacts beyond the immediate project participants).

A common interview structure enabled collection of information that was comparable across projects, and

enabled us to develop a ‘narrative’ for each project about impact. In addition to enumerating the wider

impacts of projects, the interviewers asked the CSIRO project teams to indicate how users of their research

(Australian and partner country governments, professional organisations and civil society groups) were likely

to rate the relative importance of different research themes such as climate, water and infrastructure. The

process required project teams to identify:

How their research has or might directly influence changes in policy, practices and behaviour and

how to best track these changes;

Where to look for research impacts (i.e. identify the research users where possible);

How best to assess the specific contributions made by their research (i.e. was the research really the

key factor in any changes observed?);

What has been the impact of publishing the results?

From these interviews a generic ToC was developed which reflected the intent and process for all Alliance

projects, consisting of three over-lapping stages:

1. Stage 1 Building Capacity: This involved project-funded activities and outputs, which build the

capacity of the research team and stakeholders by collating data and developing system analysis

tools and skills. These skills and tools are developed through a participatory process with

stakeholders which generates learning, new partnerships, innovation and collaboration to enable

policy, plans and programs to be formulated. A key outcome is the emergence of leaders or ‘change

agents’ amongst the research teams and other stakeholders. Stage 1 and its outputs either

immediately or subsequently trigger Stage 2.


1. Stage 2 Policy and Program Development: This is facilitated by change agents, and results in

outcomes such as altered decision-making, investments, policies and plans. In the medium to long

term this enables Stage 3.

2. Stage 3 Implementation, Adoption and Scaling-out: New programs and practices are implemented

which positively impact vulnerable beneficiaries (i.e. communities). Through this process the

adaptive capacity of all stakeholders is enhanced, while also achieving the project objective or goal.

An Impact Pathway was then described for each project which linked the stages of the ToC. The Impact

Pathway was context-specific and identified the causal relationships between the project’s activities,

outputs, outcomes and impacts related to the ToC and relevant boundary partners and beneficiaries. For

each project a diagram or ‘roadmap’ was created which illustrated the generic ToC stages and specific

Impact Pathway (see Figure 2).

2.3 Step 2 – Impact Pathway Evaluation

Evaluation of each project’s Impact Pathway was initially undertaken in April-June 2014, and was designed

to assess the extent of progress that had been made. The evaluation was structured along two activities: a

self-reflection workshop, and an impact evaluation survey. By including the views of key boundary partners

and change agents in the impact evaluation survey, the process enabled us to complement each research

team’s1 self-reflection process with a more objective assessment, which also provided triangulation.

2.3.1 Step 2A – self-reflection workshops

A 1 day workshop was held with in-country research teams and stakeholders directly involved in the project

in April 2014. The workshop was designed to encourage reflection on:

The research questions and objectives addressed over the life of the project;

The roles and expectations of project participants and boundary partners, and

The outputs, outcomes and impacts achieved along the expected Impact Pathway.

This discussion provided an opportunity for teams to share their knowledge of the project and lessons

learned. Table 1 outlines the self-reflection activity. Project teams were then asked a series of performance

questions to identify the outputs, outcomes and possible future impacts achieved by the project during the

three ToC stages. The teams were asked to map these outputs, outcomes and impacts along the project’s

Impact Pathway (see

1 Note that all research teams comprised CSIRO staff and in-country research partners.


Figure 2), relative to their original expectations. The teams mapped activities such as training and skills-

building events, outputs (such as reports and tools), and outcomes such as new management approaches,

through to new policy, plans and investment programs. They were also prompted to describe any claims of

wider impact through scaled-out implementation of new programs, and how this could be maintained in the

future.

Table 1 Steps in the self-reflection workshops

Research teams were asked to:

1. Re-articulate the research problem and goal for their case study.

2. Identify the participants and partners over the life of the project, and their expected roles. This activity also

identified new/unexpected boundary partners or changes in project design.

3. List the new skills, tools and capacity development, and map each activity and corresponding output or outcomes

on the project Impact Pathway (see Figure 2).

4. Discuss and reflect on why certain outputs and outcomes fell below or exceeded expectations, what could be

learned from this understanding, and how it could be used to inform the design of future research.


5. Identify what other changes were needed to maintain/strengthen the project’s Impact Pathway and achieve its

goal.

6. Reflect more broadly on the project, including the Impact Pathway at project completion. Specifically, was it

what they expected, and if not why, and what they would do differently if the project was repeated?

Figure 2. The expected Impact Pathway (line) and actual project achievements (numbers) relative to the ToC (boxes)

for the NTB Climate Futures project. Mapping on or near the impact Pathway line indicated that projects performed

as expected, above the line better than expected, and below the line more poorly than expected.

2.3.2 Step 2B – Impact evaluation surveys

A survey to measure progress in each stage along the Impact Pathway, and emergent capacity to respond

effectively to change (i.e. adaptive capacity) was developed and administered. The indicators and questions

drew upon the scientific literature on the adaptive co-management of social-ecological systems, social

change and collective action processes. The literature proposes that adaptive co-management evolves

through three stages similar to our ToC, and hence we matched indicators from this literature to our stages

(Table 2).

At each project’s completion in June 2014, structured interviews were carried out individually with research

team members and boundary partners. Interviewees were prioritised according to whether they had been

most directly involved in the project, and were therefore able to give the most comprehensive reflections


about Stage 1, and possible early progress in Stages 2 and 3. Interviewees were asked to discuss whether

they had observed evidence for each of the indicators (see Table 2) emerging amongst the stakeholders

concerned. The indicators were then presented as propositions, and interviewees were asked to respond

on a 5-point Likert scale from ‘strongly disagree’ to ‘strongly agree’ that the project had had any influence

on that indicator. For each statement the interviewee was asked to explain their score and provide

supporting evidence. If the interviewee ‘disagreed’ or only gave a weak ‘agree’, they were asked to identify

barriers or impediments to a higher score having been achieved. Surveys took 1 to 1 ½ hours and were

conducted in Bahasa Indonesia.

Interview data was sent to the CSIRO team members by the Indonesian members and analysed.

Interviewees’ scores for each indicator were later averaged, and then pooled as an average for each stage.

Results were subsequently returned to the Indonesian teams for verification.

Following the first interviews in 2014, the question sets for Stages 2 and 3 were reviewed and refined to

better investigate and diagnose impact in order to be more specific about how and under what conditions

impact had been achieved subsequently. This was also an opportunity to reinforce evaluation skills of our

in-country research partners who were re-trained in the interview process in order to assist with the data

collection.

Survey questions were designed to track impact over time, so wording differed slightly between 2014 and

2015. For example, in 2014, participants were asked the following question: ‘Thinking of the researchers

and boundary partners directly involved in the project do you think that their understanding and knowledge

about the problem has increased as a result of the project?’ Whereas in 2015 survey participants were

asked ‘Thinking of the researchers and boundary partners directly involved in the project, has knowledge

continued to increase as a result of the project?’

Table 2 The 18 indicators applied to evaluate progress along the Impact Pathway for each ToC stage and adaptive

capacity

Theory of Change (and adaptive co-management) stages

Indicator

Stage 1: Building Capacity

1. Leadership emerging

2. Trust created

3. Vision and goal for an alternative development pathway

4. New social networks established

5. Knowledge of the problem enhanced

6. Different knowledge types successfully integrated


Theory of Change (and adaptive co-management) stages

Indicator

7. Questioning of values, norms and governance underlying problem, and awareness of its complexity

8. Creative solutions and innovations developed

Stage 2: Policy and Program Development

1. Management plans and agreements

2. Resources made available for implementation

3. New projects triggered by the project in other problem areas

Stage 3: Implementation, Adoption and Scaling out

1. Implementation of innovations in arenas that can trial, monitor and learn from the experience

2. Enabling changes to existing or new institutions (formal and informal)

3. Cross-scale social networks established

4. New partnerships and cooperative initiatives

5. Empowerment of the most vulnerable beneficiaries (communities), including women and children

6. Enhanced self-organisation by beneficiaries (communities)

7. Enhanced beneficiary (community) capacity to live with change and uncertainty

Stage 1: Capacity Building (researchers and boundary partners)

Stage 2: Policy and Program Development

(researchers and boundary partners)

Stage 3: Implementation, Adoption and Scaling-out (researchers, boundary partners and beneficiaries)


Figure 3. Example of project influence in terms of indicators of adaptive capacity within the three stages of the ToC (see Figure 2). Scores are the averages given for each indicator by interviewed researchers and boundary partners. This example is the NTB Climate Futures project in 2014 (n = 17 interviewees).

2.4 Step 3 – Analysis

CSIRO team members later analysed the self-reflection and impact evaluation surveys. The surveys were

analysed using descriptive statistics, and key themes or issues from interviewees’ responses were collated

using NVivo software. The self-reflection results were then compared with the survey results to enable

triangulation, and therefore refine attribution for impact to the Alliance projects.

V. strong

Strong

Good

Some Infl

uen

ce


3 Projects under evaluation

3.1 Sustainable Urban Development, Makassar

Issues and approach

Makassar, the capital city of the South Sulawesi Province, is the most urbanised city in the eastern part of

Indonesia with a population of more than 1.4 million people in 2013. Makassar is already struggling to meet

demand for clean water; in 2012 only 62% of the population had access to mains water supply. This problem

will become increasingly challenging under future climate change. As a result, Makassar City faces critical

knowledge gaps relating to how future delivery of water and wastewater services can be achieved. This issue

is limiting the ability of city authorities to plan for growing demand and adapt to a potential decline in rainfall

and hence water resources.

This project (‘Makassar Sustainable Urban Development’) investigated the current and future water supply

challenges in Makassar, and provided new data including projections of local climate and its medium-term

impacts on streamflow of three rivers which are the region’s major water resources. The project also

examined the potential for bulk and clean water supply to meet the urban demand under a range of scenarios

based on population projections, water consumption patterns, leakage in distribution, infrastructure upgrade

plans and climate change. A participatory planning approach was used to engage decision-makers to

understand future implications, introduce total water cycle and integrated water management concepts and

to explore potential adaptation options to improve the sustainability of water supplies.

Outputs

Fine-resolution climate change simulation data and projections to inform climate change impact and

adaptation assessments at a local scale. This included the development of briefing papers, fact sheets,

reports, data and journal papers. Most publications were produced in English and Bahasa Indonesia.

Modelling tools to assist scenario planning activities. These tools allowed the researchers and

stakeholders to examine when and in what conditions the water supply may or may not meet demand.

The tools also allowed them to develop multiple adaptation strategies, targeting the local context, and

identifying when the risk management measures would be needed.

Adaptation options and implementation strategies to improve water provision sustainability for the city.

This included developing an understanding of Makassar’s current and future challenges associated with

water security, as well as assessing the suitability of the Makassar Infrastructure Master Plan to meet

demand, and demonstrating that infrastructure and population are the major drivers of water service

problems.


Key findings

Climate change projections suggest a decrease in rainfall over South Sulawesi, a shortening of the monsoon and a potential increase in evaporation. Changes in temperature and rainfall due to climate change are likely to change the streamflow in the catchments around Makassar. Combined with a projected 20% increase in population and 120% increase in mains water demand and no new infrastructure, water shortages will be common from 2020.

Addressing water security involves more than just building bigger dams. Better solutions can be attained by combining infrastructure and preventative measures, such as demand management and consumer behaviour changes.

Water stakeholders in Makassar now share similar views regarding water resources, which means that they are well-placed to start developing adaptation thinking and practices.

3.2 Climate Adaptation Strategies for Rural Livelihoods in Nusa Tenggara Province

Issues and approach

The islands of eastern Indonesia have some of the highest levels of poverty and food insecurity in the country.

In 2009, NTB had the second-lowest Human Development Index amongst Indonesia’s 33 provinces, reflecting

low levels of life expectancy, literacy rates, education and per capita income. Sixty-one percent of rural sub-

districts are reported to suffer chronic food insecurity, and the majority are on the island of Lombok. Most

people live in rural communities and derive their livelihoods from farming, fishing and small-scale local

industries. They are highly vulnerable to changes in rainfall and weather patterns that affect crop yields,

livestock and fisheries. They are also susceptible to natural disasters such as floods, drought and storms,

which are becoming more frequent and intense as the global climate changes, exacerbated by population

growth, fluctuating commodity prices and rising costs of living. Currently, there is little information available

to project the potential impacts of climate and other changes on rural communities, and no planning

processes which can pro-actively anticipate them.

This project (‘NTB Climate Futures’) engaged provincial and local government, NGOs, businesses and

communities to plan and test adaptation strategies that could enhance vulnerable communities’ incomes,

while building the adaptive capacity of all stakeholders to long-term change and uncertainty. The project was

intended to demonstrate an adaptation pathways planning approach that could be scaled out in other rural

regions of Indonesia.

Outputs

A participatory planning method that co-produced knowledge and learning, mitigated power imbalances

and created ownership of problems and solutions: This approach was developed by researchers from

UNRAM and CSIRO, and the Indonesian agencies for Agricultural Technology Assessment and

Meteorology and Geosciences (BMKG). This process integrated the data, tools and facilitation skills

necessary for adaptation pathways planning. The method was successively applied and refined by the

team through five sub-district case studies.

‘No regrets’ adaptation strategies: Innovative strategies were developed with vulnerable communities,

based upon their specific local needs that were ‘no regrets’ (i.e. that deliver benefits even in the absence


of climate change). The research team, local farmers and fishermen tested 12 strategies in Lombok.

Several have since been adopted, funded and scaled out by government agencies, the private sector and

communities. These included:

o Alternative bondre seaweed production that is more resilient to storms than traditional methods:

on average, this strategy could increase household welfare by $500-600 AUD per annum, which

is greater than the provincial GDP per capita.

o Inter-cropping of maize, castor and mung beans to increase productivity and reduce the risk of

crop loss from variable weather. On average, this strategy could increase household welfare by

$300-400 AUD per year, roughly equivalent to the provincial GDP per capita.

A Vulnerability Atlas of NTB was developed by combining a livelihoods’ typology with projected impacts

of climate change and population growth, and current adaptive capacity. The Vulnerability Atlas

highlighted sub-districts where adaptation planning should be prioritised. The United Nations World

Food Program (WFP) has applied the Atlas to guide their food security and resilience program, and has

secured funding for the implementation of adaptation strategies from the Adaptation Fund. The Atlas

was also incorporated into the NTB Government’s 2012 Food and Nutrition Action Plan, and the 2012

Strategy and Action Plan for Food Security and Climate Change.

Key findings

Climate change impacts vary widely across islands, requiring locally-specific adaptation planning. In many areas, population growth and the loss of agricultural land to urbanisation will have a far greater impact than climate change.

Government, NGOs and science and community stakeholders have very different perspectives of livelihood problems and solutions. This justifies multi-stakeholder planning processes which can understand and integrate the different views.

Multi-stakeholder planning, while time-consuming, enhances participants’ adaptive capacity by catalysing innovation, new partnerships and empowering vulnerable communities.

The causes of community vulnerability are highly complex and dynamic. Many values and rules governing decision-making are changing, such as women’s empowerment in Sumbawa. These trends also present some paradoxes. For example, the decline in traditional institutions precipitated by modernisation erodes customary ecosystem stewardship and mutual assistance practices that are important for the poor, but also enables women’s education and leadership.


4 Evaluation results

4.1 Sustainable Urban Development, Makassar

Impact pathway self-reflection workshop

The project’s objective was ‘to help the city understand the problem and to develop solutions to improve the

management of water resources’. The research team’s Impact Pathway linked the ToC stages and

cumulatively enhanced stakeholders’ sustainable water resource management skills (Figure 4). Stage 1 of the

ToC encompassed the project’s activities (solid line), but Stage 2 and 3 were out of the project team’s direct

control (dashed line). Consequently, the Impact Pathway was clearly mapped for Stage 1, but less so for

Stages 2 and 3.

Stage 1 Stage 2 Stage 3

Figure 4 Self-reflection comparisons for the Makassar Sustainable Urban Development project between 2014 and

2015. Red numbers are the project completion scores from 2014 and yellow numbers show any changes in the 2015

evaluation

Eleven team members attended the self-reflection workshop. The results of the 2014 evaluation were shared

with participants, followed by a refresher presentation on the project’s objective and activities. Participants

then re-evaluated project outputs, outcomes and impacts, mapping them against the 2014 results (Figure 4).

Overall, the results reflected positive progress since 2014, with a number of advances along the Impact

Pathway documented.


The majority of project achievements where ‘significant change’ (13 of 40 achievements) had been recorded

could be described as capacity-building activities (seven of the 13 achievements). Other significant change

achievements included those for data improvement (two), networking (two) and the establishment of new

projects (two). In 2015, five of the seven capacity-building achievements shifted from Stage 1 to Stage 2

Policy and Program Development, including the two achievements that participants ranked as having had the

greatest change. These latter two achievements included the identification of new skills for UNHAS

postgraduate students in project management and networking, and the observation that key stakeholders

viewed problems differently and more holistically as a result of their engagement with the project. For

example, participants stated that all UNHAS students who had worked on the project had now graduated to

related jobs in academia and the public sector.

The CSIRO-UNHAS research team

Impact evaluation surveys

In 2014, 16 researchers and boundary partners involved in Stage 1 were interviewed and asked to give their

assessment of change by scoring each indicator, and provided evidence for their scores. In 2015, 17

researchers and boundary partners were interviewed; one from the previous survey was unavailable and two

additional stakeholders were interviewed. Results showed that in 2015 there had been a positive influence

for all indicators in all stages, and the average score for each phase was higher, with small increases for Stages

1 and 3, and a significant improvement for Stage 2 (Figure 5).


Figure 5 Average scores and standard deviations for all indicators in the three stages of the Makassar Sustainable

Urban Development project’s ToC

Stage 1 Building Capacity

2014 survey

By project completion this stage had generated strong participatory engagement processes, capability

development and the provision of new information for decision-making. This was demonstrated by high

scores relative to the following two stages. A key aspect of this project was the exchange of ideas between

local government agencies and institutions through the project workshops and communication activities.

These relationships were also strengthened through training, such as a visit to Melbourne to undertake

climate projection downscaling, and involving boundary partners in the research activities. Consequently, an

overall improvement in cooperation between institutions and sharing of information was achieved. Overall,

the project had improved the trust and cooperation between individuals participating in the project, with

some enhanced trust at the institutional level and increased cooperation between UNHAS, Public Works and

BMKG. Such cooperation has led to an opportunity to recruit new employees to work in this field, which, in

turn, has led to an increase in the number of students at UNHAS. The majority of survey respondents

mentioned that theirs and others’ awareness of climate change and the subsequent impact on the

management of water in Makassar had increased as a result of this project.

What changed in the 2015 survey?

In 2015 the averaged indicator scores for Stage 1 revealed a small increase from ‘strong’ in 2014 towards

‘very strong’, and the standard deviation contracted. Overall, this reflected a more positive and consistent

view of the project’s achievements for Stage 1. There were slight variations in scores for most of the

indicators, apart from indicators for leadership and creative solutions, which had substantial increases

relative to the other indicators (‘strong’ to ‘very strong’, and ‘good’ to ‘strong’, respectively). Responses

contained some commonalities in terms of emerging leaders. There was also clear evidence of a maturing of

capacity at both junior levels (e.g. ex-students now in management/teaching roles) and also in more

substantive roles, which interviewees were able to trace back to the project.


2014 2015

V. STRONG

STRONG

GOOD

SOME

Infl

ue

nce


Stage 2 Policy and Program Development

In 2014, interviewees were asked three questions for Stage 2. As part of a review of the method in 2015 a

further three questions were added to the survey, giving a total of six questions for Stage 2.

2014 survey

Policy and program development had lower indicator scores in 2014 relative to Stages 1 and 3, highlighting

the lag time and potential barriers to policy outcomes. The Public Works Agency was reviewing the Makassar

Infrastructure Master Plan, and CSIRO had been asked to review the Clean Water Supply Master Plan. The

State Ministry of Environment and the Ecoregion Management Centre asked UNHAS to design a pilot

demonstration for rainwater harvest and water use. Funding to continue this work and to implement the

proposed actions was yet to be secured, however. The Makassar Public Works Agency was trying to

implement a new organisational framework, but has yet to source funding for extra staff.


Across the original three questions from 2014, the 2015 survey showed an increase in the average indicator

score from ‘some’ to ‘strong’. Substantial improvements were recorded for all three questions: with

‘management plans’ and ‘new projects’ moving from good to strong, and ‘resources’ moving from ‘some’ to

‘good’. These are now discussed in more detail:

Management plans: In 2014, interviewees were asked: ‘Have any management plans or agreements

(e.g. verbal discussions, minutes of meetings) to address the problem been developed as a result of

the project?’, and in 2015 this was followed up with: ‘What happened to the existing plans, and if

nothing, why not?’ Participants identified a range of diverse activities, including common initiatives

or extensions to existing activities, some of which were unknown to the CSIRO project team.

Examples extended from strategic planning and education through to on-ground activities such as

water quality monitoring and planting trees.

Resources: The same question was asked in 2014 and 2015: ‘Have they made resources (e.g. funding,

staff) available to implement the new management plans and tackle the problem?’ A review of the

range of answers indicated that in 2014, eight (of 13) respondents indicated that they ‘did not know’,

compared to five (of 14) in 2015. In 2015, eight interviewees ‘strongly agreed’ (i.e. gave a maximum

score) that resources had been made available. The high standard deviation suggested wide variation

in the scores. However, there was little overlap amongst the answers for this question for those who

‘strongly agreed’, signalling that the diversity of interviewees had captured a broad range of projects,

emphasising positive progress along the Impact Pathway.

New projects: The same question was asked in 2014 and 2015: ‘Have any new projects been designed

or implemented to deal with different issues as a result of this project?’ The responses showed a

substantial shift from ‘agree’ to ‘strongly agree’ (nine of 14 responses in 2015 compared to only two

responses in this category in 2014). Three broad types of examples were provided by respondents:

education (focusing on students); infrastructure; and management or professional development.

Education examples included the use of global climate data in research activities and the

incorporation of climate information into a range of other studies such as disaster reduction.

Infrastructure examples cited included the use of project data or skills in the development of

absorption wells, rainwater harvesting and waste water treatment. Finally, management and

professional development examples included further training, and the application of project

information in design and reporting.


Stage 3 Implementation, Adoption and Scaling-out

2014 survey

There was evidence of the project having facilitated cross-scale social networks. For example, there had been

cooperation between UNHAS and the Centre for Eco-region Management and the Agency of Environment of

Makassar City, as well as with the national government and their related agencies. There had been several

new partnerships created as a result of the project, including between researchers and the Public Works

Agency and the City Company for Drinking Water. At the provincial and district level, the researchers were

involved in the preparation of an action plan for the mitigation of greenhouse gas emissions. Another new

partnership was formed between UNHAS and the Centre for Eco-region Management of Sulawesi, Maluku

and Papua Provinces to establish a reservoir for rainwater capture. There had been little change in water

management for Makassar City, however. This was in part due to the short time frame between project

completion and the evaluation, and a lack of resources. The stakeholders noted that regulations needed to

be changed before changes in organisational function could occur, but this was complicated by bureaucratic

processes.


Overall the averaged indicator score for Stage 3 was somewhat higher in 2015 compared to 2014.

Improvements were recorded across all questions apart from that related to ‘enhanced beneficiary capacity’.

Most significant changes were:

Implementation: The same question was asked in 2014 and 2015: ‘Have any of the ideas and solutions

developed by the project been implemented? Have they been set up as trials with a monitoring and

learning design?’ Responses indicated a shift from ‘some’ to ‘good’. Responses had to be balanced

against the view espoused by one participant that there are always research, planning and

implementation activities in the region, which makes it difficult to attribute impact to the CSIRO

project. Respondents cited examples such as improved relationships between government and

academia, creation of new knowledge, community empowerment, development of an eco-

educational park, ongoing rainwater harvesting activities, improved data collection and analysis, and

no regrets activities such as tree planting to reduce solar exposure in residential areas. No specific

cases of monitoring and learning design were provided, however, it was possible that a number of

the examples cited included this aspect.

Enabling changes: The same question was asked in 2014 and 2015: ‘Have any changes been made to

organisations, rules or the usual practices?’ Responses indicated an improvement from ‘some’ to

‘good’. Interestingly, half of the respondents (seven of 14) answered ‘strongly agree’ and five

suggested ‘some’, indicating a polarisation amongst the answers. This was also reflected in the large

standard deviation. The most common response amongst those that ‘strongly agreed’ related to

improved sharing of information between agencies and organisations.

Cross-scale social networks: The same question was asked in 2014 and 2015: ‘Have you seen new

linkages and relationships growing between stakeholders from different levels (e.g. national

government, communities) as a result of the project, where they exchange information and ideas?’

Thirteen of the 15 respondents ‘strongly agreed’, the highest score for this stage by some margin.

Examples of improved networks included the national level (e.g. Agricultural Agency and National

Agency for Disaster Prevention), between Makassar Government and UNHAS, across faculties within

UNHAS, between regional governments, within civil society and also through a number of informal


channels. A good portion of this activity reflected UNHAS’s emergence as a centre of excellence for

climate adaptation and mitigation.

Enhanced beneficiary capacity: Most respondents answered ‘don’t know’ for this question (eight in

2014 compared to four in 2014). One interpretation of this, which is supported by the self-reflection

workshop, is that there has been a maturing of respondents’ understanding of impact and progress

along the Impact Pathway over time, and survey participants felt better equipped to answer this

question 12-18 months after major project deliverables were completed. Overall it was agreed that

it was too early to see tangible benefits for vulnerable local communities.

Summary

The results showed that the participatory planning approach and training significantly built the capacity and

social networks of the UNHAS research team to understand the problem and to develop solutions which

would improve the management of water resources for Makassar. These skills and the process influenced

management plans and resources in Stage 2, resulting in improved cross scale social networks and new

partnerships in Stage 3. However, in 2014 and 2015 tangible impacts had not yet reached vulnerable local

communities in Makassar.

Participants in the 2015 self-reflection evaluation process found it useful, as it increased their understanding

about sustainable urban development issues and potential solutions, raised their awareness about what each

other had been doing since the last evaluation at project completion in 2014, and opened up more

possibilities of maintaining and/or extending collaboration. The process facilitated the sharing of new ideas,

and encouraged participants to think innovatively. Importantly, the process motivated them to follow up

project recommendations and keep the process alive. The impact evaluation survey triangulated many of the

results provided by the self-reflection workshop, but because it included boundary partners, additional

outcomes emerged which the research team were not aware of.

4.2 Climate Adaptation Strategies for Rural Livelihoods in Nusa Tenggara Barat Province

Impact Pathway self-reflection workshop

The project’s goal was to increase the adaptive capacity of rural communities, boundary partners and

researchers to reduce vulnerability to adverse change. To achieve this, the research team developed a ToC

and Impact Pathway consisting of three linked stages (Figure 6). As for the Makassar Sustainable Urban

Development project, Phase 1 Capacity Building enabled the allocation of resources and development of

plans, agreements and new projects through Stage 2 Policy and Program Development. Following on from

Stage 2, Stage 3 Implementation, Adoption and Scaling-out would occur. These phases would cumulatively

build the adaptive capacity of the project stakeholders. Unlike the Makassar Sustainable Urban Development

project, because the project engaged beneficiaries directly in trialling adaptation strategies, the project was

expected to have a direct influence throughout each phase.


Figure 6 Self-reflection comparisons for the NTB Climate Futures project between 2014 and 2015. Blue numbers

show no change from the project completion scores in 2014, whereas yellow numbers reflect improvement and

green a decline in impact since the 2014 evaluation.

Fifteen stakeholders attended the Impact Pathway self-reflection workshop. The results of the 2014

evaluation were shared with participants, followed by a refresher presentation on the original project.

Participants then re-evaluated project outputs, outcomes and impacts, mapping them against the 2014

results. Overall, the results reflected positive progress, with some advances along the Impact Pathway.

Of the 39 project achievements listed and discussed, participants agreed that since 2014 four had decreased

slightly (green), there had been no change for 18 (blue), a positive change for 13 (yellow), and a significant

increase in impact for a further six achievements (also yellow). Of the four achievements that were scored

negatively, three were due to the cut in DFAT funding support in 2014 following the change in the Australian

Government in 2013. The fourth was due to the fact that the NTB Vulnerability Atlas had only been partially

used by the WFP. The WFP’s initial uptake of the Atlas was an unexpected and positive outcome in 2013, and

the slight decrease in this achievement reflected participants’ disappointment at not seeing this continued.

Positively, the four achievements where participants scored a significant increase indicated impact across a

diverse range of capacity-building initiatives. These included the ongoing use of climate projection data by

the BMKG to provide advice; an increase in adaptation strategy evaluation using a number of decision

support approaches; BMKG securing continued funding to support the development of climate change field

schools across a number of districts and regions in NTB; and an extension of trials for ‘climate ready’ farming

practices. As with the companion Makassar project, it was anticipated that capacity-building activities would

extend into Stage 2 and the self-reflection exercise would provide an evidence trail to support this, including

sufficient granularity to identify the changing nature of capacity-building activities.


The CSIRO-UNRAM research team

Impact evaluation surveys

In 2014, 17 researchers and boundary partners involved in Stage 1 were interviewed at project completion.

In 2015, 16 researchers and boundary partners were interviewed, with one research team member from the

previous survey unavailable. Results showed that there had been a positive influence for many indicators in

all stages, with the evaluation results indicating increases since the 2014 survey for Stage 1, and a maintaining

of influence in Stages 2 and 3 (Figure 7).

Figure 7 Average scores for all indicators in the three stages of the NTB Climate Futures project’s ToC


2014 2015

V. STRONG

STRONG

GOOD

SOME

Infl

ue

nce


Stage 1 Building Capacity

2014 survey

Stage 1 had the highest-scoring indicators. The project fostered the research team’s leadership skills, and

UNRAM team members were recognised amongst their peers, boundary partners and communities as

champions of inter-disciplinary adaptation and gender research. As a result, team members were employed

by the WFP’s Climate Change Adaptation Program in Jakarta and EcoRegions Indonesia (ERI), which facilitated

better connections between the project and these change agents. The project had improved trust and

cooperation between individual participants and institutions. Participants appreciated the opportunity for

training and evaluation to build their skills, and professional networks were developed through joint project

activities, with these interactions continuing beyond the project. At an institutional level, there was an

increase in understanding, trust and cooperation between agencies. The project also strongly influenced the

improvement and integration of knowledge. The majority of interviewees mentioned that their own and

others’ awareness of climate change and its impacts on livelihoods had increased. Several revealed that they

had gained a better understanding of the problem’s complexity, and appreciated learning a systems approach

with mixed methods including scenario planning.


Average indicator scores for Stage 1 showed an increase from ‘strong’ in 2014 towards ‘very strong’ in 2015,

as well as a slight contraction of the standard deviation. All individual indicator scores also increased in 2015.

The most significant changes were for the questions on ‘new social networks’ and ‘questioning values and

governance’, which moved from ‘strong’ to ‘very strong’, and ‘good’ to ‘strong’, respectively:

New social networks established or maintained: In 2015, 12 of 17 respondents scored this indicator

as ‘very strong’, compared to six of 16 in 2014. The responses indicated an increase in new projects

and collaborations between researchers and boundary partners; utilisation of CSIRO methods to

analyse climate change vulnerability and policy; and out-growth of networks to include the forestry

sector.

Questioning of values, norms and governance underlying the problem, and awareness of its

complexity: The overall averaged indicator score in 2015 was ‘strong’, compared to ‘good’ in 2014,

with a relatively high standard deviation for both years, indicating large variability across the

responses. Key issues raised by respondents varied. Where participants responded positively (i.e.

scored ‘very strong’), answers included improved awareness and approaches, and greater levels of

collaboration. Barriers to funding, lack of uptake by local government and the challenge of changing

behaviour (e.g. farmers’ cropping practices) were all listed as ongoing challenges that prevented

greater outcomes.

Creative solutions and innovations: The overall averaged indicator score moved up from ‘strong’ in

2014 to ‘very strong’ in 2015, and the standard deviation declined, highlighting a greater consistency

amongst interviewees’ answers. Positive influence was reflected through ongoing awareness and

understanding of causes, using the project’s science outputs to develop new projects, and the

ongoing out-scaling of adaptive farming practices (e.g. replacing rice with cassava in dry land areas,

adjusting planting regimes, introduction of higher yield varieties of maize and mung bean).


Stage 2 Policy and Program Development

2014 survey

In 2014 Phase 2 had slightly weaker indicator scores than Phase 1. Management plans which included

adaptation strategies had been produced, including the NTB Government’s 2012 Food and Nutrition Action

Plan, and the 2012 Strategy and Action Plan for Food Security and Climate Change. New resources were

created, with staff amongst boundary partner organisations allocated additional responsibilities to become

involved in climate adaptation planning with other institutions. For example, the WFP had employed a

research team member to assist with their climate adaptation projects, and ERI was funding a demonstration

site and community adaptation planning in one sub-district case study. The national Ministry for Regional

Development (KPDT) and the East Lombok District Government opened a ‘show room’ for seaweed

production in Lombok, highlighting the bondre technique developed by the project. Many new projects had

also commenced, including:

The WFP had introduced Payments for Ecosystem Services as a new food security strategy.

The Indonesian Ministries of Education and Forestry were undertaking research into climate change

and adaptation nationally, with the invited participation of a team member.

BMKG had joined the Asia Pacific Network Project, allowing them to further build their capacity for

climate projection modelling.

Funding had been allocated by the NTB Government for adaptation strategy development in the case

studies, including bee-keeping in North Lombok and maize inter-cropping in East Lombok.


The average indicator score for this stage revealed almost the same overall result in 2015 as in 2014, and a

slight widening of the standard deviation scores. Overall, there was minimal movement across the set of

individual indicators. The most significant changes were for ‘resources made available for implementation’,

which decreased from ‘strong’ in 2014 to ‘good’ in 2015; ‘changes to institutions’, which also decreased

slightly; and ‘cross scale social networks’, which increased slightly. Notably, the standard deviation for

‘resources made available for implementation’ increased considerably. In 2015, three participants indicated

‘don’t know’ for this question, compared to one in 2014. Of the three participants who scored down from

2014, only one provided an explanation, and this related to the loss of DFAT funding for the project. However,

two participants indicated that other sources of funding support had been maintained or increased over the

period, with one specifying the WFP Adaptation Fund project.

Stage 3 Implementation, Adoption and Scaling-out

2014 survey

The Stage 3 indicator score in 2014 was slightly lower than for Stage 2, but there was strong evidence of the

project having implemented strategies amongst vulnerable case study communities, including:

The bondre seaweed system was scaled-out from the initial focus village across East and West

Lombok Districts in 2013 and 2014, with support from KPDT and the district governments.

Project demonstration sites were set up by ERI and the East Lombok Climate Forum.

Maize inter-cropping was also scaled out in partnership with ERI in East Lombok.


The Payments for Ecosystem Services approach for conserving water resources under climate stress

was implemented in three villages in Central Lombok as part of WFP’s revised approach to food

security projects.

Vulnerable case study communities were empowered through their involvement in the project’s

participatory planning workshops and the trialling of adaptation strategies. Throughout the project, women

and other vulnerable groups were identified and invited to participate in the workshops, enabling them to

contribute equally to decision-making and planning. For example, women’s groups were processing cassava

chips as a substitute for rice in North Lombok, diversifying their families’ diets and providing alternative

employment at home. The bondre seaweed system and bee-keeping trials were also creating jobs for women

and children, especially in the wet season when stormy weather prevents fishing. Overall, the adaptive

capacity of communities scored relatively strongly, indicating that the outcomes and impacts in Phase 3 had

collectively made progress towards the project’s over-arching goal.


The average indicator score for Stage 3 showed almost the same overall result in 2015 as for 2014, and a

slight widening of the standard deviation scores. Overall, there was minimal movement across the full

range of indicators. The most significant change was recorded for the question on ‘implementation’, which

recorded a drop from ‘strong’ in 2014 to ‘good’ in 2015, coupled with an increase in standard deviation in

2015. In 2014 all participants answered the question, whereas in 2015, three respondents answered ‘don’t

know’; and one participant’s score moved from ‘very strong’ to ‘disagree’. Where positive scores were

provided, a number of answers focused on the implementation and extension of cropping trials.

Summary

The results showed that, as for the Makassar Sustainable Urban Development project, the participatory

approach and training had significantly built the capacity and social networks of the research partners, as

shown by the increased in scores for Stage 1 between 2014 and 2015. These skills and the project’s

participatory planning process have had varying influence over time on management plans and resources in

Stage 2, with external funding pressures and competing priorities leading to no change over time for project

results. This was also evident in the resulting mixed scores for Stage 3, with implementation slightly lower,

but improved cross-scale social networks, resulting in no further progression along the Impact Pathway.

Unlike the Makassar Sustainable Urban Development project, because the NTB Climate Futures activities

engaged community members directly in trialling adaptation strategies, impacts were clearly evident in both

2014 and 2015, albeit largely within the project’s case study sub-districts.


5 Discussion

5.1.1 Indonesian research partners’ reflections

Following the exercises in 2014 and 2015, the CSIRO teams sought reflections and feedback from their

Indonesian research partners about the strengths and weaknesses of the evaluation method and tools. The

strengths mentioned were:

The Alliance and AIC surveys were the first opportunities that they had had to reflect on their own

projects. Previously, their projects were evaluated by independent consultants linked to the donor

or funding body, and the researchers were largely dis-engaged from the process.

The researchers found the evaluation was useful, particularly the Impact Pathway self-reflection

workshops which allowed them the chance to reflect, discuss and learn with one another about

what worked and what did not. This process enhanced their reconsideration of actions that needed

to be taken to maintain their original Impact Pathway, and to keep the projects’ ‘spirit’ alive.

The chance to reflect collectively was also felt to enhance possibilities for maintaining and/or

extending collaborations.

The process also facilitated learning about other disciplines, the sharing of new ideas, and

encouraged researchers to think innovatively.

Weaknesses and limitations identified included:

The time lag between Phase 1 Building Capacity and Phase 2 Policy and Program Development and

Phase 3 Implementation, Adoption and Scaling-out was substantial, limiting clear evidence of

outcomes and impacts immediately after the projects’ closure (i.e. the 2014 survey) or 1 year later

(i.e. the 2015 survey). However, by engaging communities directly in planning and the trialling of

adaptation strategies, the NTB Climate Futures project had generated some impact.

In spite of this, the re-evaluations should be undertaken more frequently than at 1 year intervals

because project participants may quickly forget their roles and achievements, and need regular

reminding of the Impact Pathway.

When identifying the actions needed to achieve further progress along the Impact Pathway, the

process should clearly identify indicators, time lines and responsibilities to encourage

accountability and ease the measuring of ‘success’ at the subsequent evaluation.

It is challenging to consistently engage the same project participants over time for evaluation, and

hence continuity in the indicator scoring. Many government and NGO staff move positions or jobs

and cannot be easily traced in the long term.

It is difficult to engage key stakeholders in repeated evaluation exercises due to their work

commitments. If they send proxies, the newcomers have to be informed about the project and

their views do not reflect those of their superior.

5.1.2 CSIRO team reflections

The CSIRO teams that coordinated the evaluations also noted their observations about the processes. The

following strengths were highlighted:

The process clearly re-energised the Indonesian research teams, and in 2015 there was some

evidence of team members following up on lessons learned from the 2014 survey.


The ToC and Impact Pathway diagrams were easily translated and understood by the Indonesian

research teams in the self-reflection workshops, and created useful ‘boundary objects’ for

discussion.

The indicator graphs were also powerful tools, but needed to be fed back to the Indonesian

research teams soon after the surveys to have any significance.

Weaknesses were:

For the 2015 survey it was harder to maintain interest amongst the research teams than in 2014,

when the projects had only recently been completed. Consequently, less detailed information was

gathered during the self-reflection workshops and indicator surveys, and the 2015 process was

perceived as more of a ‘box-ticking’ exercise.

The turn-around time for the results of the surveys in 2014 was slower than intended (i.e. 10

months instead of 2-3 months) due to the lack of funding after the project’s completion. This

should be avoided for the 2015 survey. In future, similar exercises should make funding allowances

for more regular evaluations and rapid processing of results.

The Indonesian research teams who undertook the participant interviews needed sufficient

financial support and training to complete the surveys adequately. These team members frequently

had competing work commitments that impeded the proper completion of all interviews.

Both the 2014 and 2015 evaluation exercises were ex-post. In future, the process should be

undertaken at regular intervals during the project cycle to generate learning and momentum along

the Impact Pathway. To do so requires adequate resourcing and time built into the project design

from the outset.

5.1.3 Conclusions and recommendations

In this AIC-funded Small Grants project we evaluated and quantified the impact of two Indonesian Alliance

projects in 2015, 1 year after their completion. The AIC support enabled us to replicate the initial evaluation

undertaken in 2014, which used a mixed methods approach to generate self-reflection amongst the

Indonesian research partners, and interviews with researchers and boundary partners to score indicators of

adaptive capacity reflecting the projects’ ToC and Impact Pathways.

The evaluations of the Makassar Sustainable Urban Development and NTB Climate Futures projects

demonstrated how their participatory planning approaches had generated innovation, partnerships, raised

awareness and change agents amongst the Indonesian research teams and their boundary partners. The

initial progress along the Impact Pathways in 2014 was substantial, and this had progressed significantly by

2015. While not the focus of this Small Grants project, the projects’ research design clearly had positive and

ongoing impacts on the adaptive capacity of all stakeholders.

Reflections from the Indonesian and CSIRO research teams on the evaluation method enables us to make

several recommendations to the AIC:

1. The ToC and Impact Pathways concepts were easily understood and accepted as ‘roadmaps’ for the

projects, and could be relevant planning and evaluation tools for AIC projects.

2. The self-reflection workshops were very useful exercises which enabled the research teams to reflect

on their achievements, and actions needed to maintain progress along their Impact Pathways after

project completion. However, clear actions and responsibilities must be outlined during these

exercises to encourage the teams to follow-up.


3. The impact evaluation surveys revealed additional information from both researchers and boundary

partners which complemented the self-reflection workshops, and the indicators provided useful

graphic representations of progress along the Impact Pathway over time. However, Indonesian team

members undertaking the interviews must be adequately resourced to complete them.

4. To have maximum utility for the Indonesian research teams, the results of the evaluations must be

returned to them following analysis within 2-3 months. If not, momentum and interest wanes, and

any actions to be followed-up are forgotten.

5. The self-reflection workshops and impact evaluation surveys should be incorporated into projects

from the outset, rather than being conducted ex-post when the greatest opportunities for research

teams to respond have largely been lost.


6 References

Better Evaluation (2015). Combine Qualitative and Quantitative Data. Retrieved March 22, 2015, from http://betterevaluation.org/plan/describe/combining_qualitative_and_quantitative_data

Bours, D., McGinn, C., & Pringle, P. (2013). Monitoring & evaluation for climate change adaptation: A synthesis of tools, frameworks and approaches. SEA Change Community of Practice and UKCIP, Phnom Penh, Cambodia, and Oxford, UK. http://www. seachangecop. org/node/2588.

Bours, D., McGinn, C., & Pringle, P. (2014). Guidance note 1: Twelve reasons why climate change adaptation M&E is challenging. SEA CHANGE, UKCIP. Guidance Note for M&E of Climate Change Interventions.

Butler, J. R. A., Suadnya, I. W., Yanuartati, Y., Meharg, S., Wise, R. M., Sutaryono, Y., & Duggan, K. (in review). Designing and evaluating the priming of adaptation pathways in developing countries. Climate Risk Management.

Butler, J. R. A., Suadnya, W., Puspadi, K., Sutaryono, Y., Wise, R. M., Skewes, T. D., . . . Ash, A. (2014). Framing the application of adaptation pathways for rural livelihoods and global change in eastern Indonesian islands. Global Environmental Change 28, 368-382. doi: 10.1016/j.gloenvcha.2013.12.004

Butler, J. R. A., Wise, R. M., Skewes, T. D., Bohensky, E. L., Peterson, N., Suadnya, W., . . . Rochester, W. (2015a). Integrating top-down and bottom-up adaptation planning to build adaptive capacity in the Coral Triangle: an alternative approach. Coastal Management 43, 346-364.

Butler, J. R. A., Young, J. C., McMyn, I., Leyshon, B., Graham, I. M., Walker, I., . . . Warburton, C. (2015b). Evaluating adaptive co-management as conservation conflict resolution: learning from seals and salmon. Journal of Environmental Management 160, 212-225.

Chapin, F. S., Lovecraft, A. L., Zavaleta, E. S., Nelson, J., Robards, M. D., G. P. Kofinas, . . . Naylor, R. L. (2006). Policy strategies to address sustainability of Alaskan boreal forests in response to a directionally changing climate. Proceedings of the National Academy of Sciences of the United States of America 103, 16637-16643.

Douthwaite, B., Alvarez, S., Keatinge, J.D.H., Mackay, R., Thiele, G., Watts, J. (2009). Participatory impact pathways analysis (PIPA) and research priority assessment. CABI, Wallingford, pp. 8-24.

Earl, S., Carden, F., Smutylo, T. (2002). Outcome mapping: building learning and reflection into development programs. International Development and Research Centre, Ottawa.

Mawdsley, J. R., O’Malley, R., & Ojima, D. S. (2009). A Review of Climate-Change Adaptation Strategies for Wildlife Management and Biodiversity Conservation. Conservation Biology, 23(5), 1080-1089. doi: 10.1111/j.1523-1739.2009.01264.x

Rogers. E, M. (2003). Diffusion of Innovation (5th edition). Free Press, New York. Vogel, I. (2012). Review of the use of ‘Theory of Change’ in international development. Report for

the UK Department for International Development (DFID). White, H. P. (2012). Addressing attribution of cause and effect in small n impact evaluations:

towards an integrated framework (Vol. Working Paper 15). New Delhi: International Initiative for Impact Evaluation and 3IE.

Wiek, A., Talwar, S., O’Shea, M., & Robinson, J. (2014). Toward a methodological scheme for capturing societal effects of participatory sustainability research. Research Evaluation, 23, 117-132.

http://betterevaluation.org/plan/describe/combining_qualitative_and_quantitative_data

http://www/


Wise, R. M., Fazey, I., Stafford Smith, M., Park, S. E., Eakin, H. C., Archer van Garderen, E. R. M. & Campbell, B. (2014). Reconceptualising adaptation to climate change as part of pathways of change and response. Global Environmental Change 28, 325-336.


CONTACT US

t 1300 363 400 +61 3 9545 2176 e [email protected] w www.csiro.au

AT CSIRO WE SHAPE THE FUTURE

We do this by using science to solve real issues. Our research makes a difference to industry, people and the planet.

As Australia’s national science agency we’ve been pushing the edge of what’s possible for over 85 years. Today we have more than 5,000 talented people working out of 50-plus centres in Australia and internationally. Our people work closely with industry and communities to leave a lasting legacy. Collectively, our innovation and excellence places us in the top ten applied research agencies in the world.

WE ASK, WE SEEK AND WE SOLVE

FOR FURTHER INFORMATION

CSIRO Land and Water Dr James Butler t +61 2 6776 1358 e [email protected] w http://www.csiro.au/en/Research/LWF

evaluating the impacts of participatory planning for urban water

Documents