system • the national estuarine research reserve system...
TRANSCRIPT
Notes:Dr. Susi Moser is Director and Principal Researcher of Susanne Moser Research & Consulting. She is also a Social Science Research Fellow at the Woods Institute for the Environment at Stanford University.
Her work focuses on adaptation to climate change, vulnerability, resilience, climate change communication, social change, decision support and the interaction between scientists, policy-makers and the public.
She is a geographer by training with a Ph.D. from Clark University in Worchester, MA.
NERRS Science Collaborative
• Monthly webinars
• Feature research, integrated assessment, and science transfer projects funded by the NERRS Science Collaborative
• Feature the efforts of Science Collaborative team members as they engage the reserve system
Collaborative Science for Estuaries Webinar Series
Notes:• The National Estuarine Research
Reserve System (NERRS) is a network of 29 research reserves protected for long-term research, water quality monitoring, education, and coastal stwewardship. These reserves represent a partnership between NOAA and coastal states.
• The mission of NERRS is to practice and promote the stewardship of coasts and estuaries through research, education, and training using a place-based system of protected areas.
• Reserves pursue this mission in a highly collaborative way with a wide variety of partners.
National Estuarine Research Reserve System
• Research funding mechanism for the NERRS, which supports:
- Reserve management needs
- Highly collaborative projects (integrate end users)
- Outcome-oriented products
NERRS Science CollaborativeNotes:• The NERRS Science Collaborative,
which is currently housed at the University of Michigan’s Water Center through a cooperative agreement with NOAA, supports research, assessment, and science transfer activities that address the needs of reserves in order to improve stewardship of coastal and estuarine ecosystems.
• The research funded by Science Collaborative is distinctive because it integrates end users into the research process itself to produce outcome-oriented products that are used by end users and decision-makers.
Notes:This webinar focuses on Dr. Susi Moser’s Successful Adaptation Indicators & Metrics (SAIM) project that has engaged the National Estuarine Research Reserve System (NERRS) in the task of tracking successful adaptations to climate change by identifying relevant indicators of change and the metrics to measure implementation. The project explored what successful adaptation looks like at different reserves and how they can develop indicators and metrics (I&M) to determine if they are making adequate progress toward their defined goals and vision of success.
NERRS Science Collaborative
SUCCESSFUL ADAPTATION INDICATORS & METRICS (SAIM)
PROJECTFrom Pilots to System-wide Benefit
Susi Moser, Ph.D.NERRS Science Collaborative
Susanne Moser Research & Consulting
January 10, 2018
Acknowledgements
Max Boykoff and 40+ book contributors
Successful Adaptation (Moser et. al.)
National Estuarine Research Reserve System (NOAA, UM)
6
+ Dozens of partners, collaborators, & stakeholders
James ArnottProject Assistant
Successful Adaptation Indicators & Metrics (SAIM) Project
Notes:Susi thanked the many people that contributed to this project and to foundational projects leading to the SAIM effort. She particularly thanked all the NERRS staff that have been involved in the SAIM project.
Notes:This presentation first focuses on what adaptation success means; then reports on the SAIM project activities, and third distills some lessons learned to date about defining, tracking and using indicators and metrics for adaptation.
Overview
• A Bit of Background on “Adaptation Success”
• Work with the National Estuarine Research Reserve System – Pilots in developing, selecting, tracking indicators and metrics of success
• Sharing lessons across the NERR System, coastal America, others interested in adaptation
9
Background & Origin
Foundation #1 Foundation #2
Summary Points:This project was based on work that looked at climate adaptation success.
• The first was an edited volume Susi co-authored in 2013, which pulled together the literature on successful adaptation to climate change up until then and explored some of the key challenges in defining adaptation success. The key message from that effort was how complicated it is to say what “success” in adaptation is. It has many different components and dimensions, including questions of: o How should adaptation success be measured? o When should it be measured? o Who should measure it? o At what scale should it be measured? And so on
• Another project providing important background involved a Sea Grant-funded project on the west coast (lower case) that engaged a wide range of stakeholders in an effort to understand the key dimensions of adaptation success. This effort provided important foundational information about what success looks like when an agency or community is successfully adapting to climate change.
Summary Points:Why do people care about adaptation success? Why is it useful to think about it?
• Climate change, particularly in coastal areas, is a gloomy topic that people often see as overwhelming and insurmountable. Focusing on ways to measure success engenders hope and brings stakeholders together to become part of a co-creative process of success.
• In order to engage in deliberate planning and decision-making, it’s critically important to set clear goals and align your means and ends toward them.
• Coastal adaptation is one of many priorities that requires funding. It’s important to be able to demonstrate the success of specified objectives and criteria in order to justify funding.
• There’s a growing demand for accountability in the public and private sector for expenditures. So being able to track how well you have done is also critical.
• And finally, adaptation is an ongoing and iterative process - it’s important to monitor progress toward goals and metrics, learn from what is not going well, and make adjustments as needed.
Findings: Common reasons why people care about adaptation success
Overarching: Responsibility for safeguarding people, economy, infrastructure, cultural assets, environment1. Communication and public engagement
• Communicating hope and desirable goal to work towards
• Defining a common vision among diverse stakeholders
2. Deliberate planning and decision-making• Setting clear goals, aligning means and ends (internal
consistency)
• Best fit with other policy goals (external consistency)
3. Justification of adaptation expenditures 4. Accountability/good governance5. Support for learning and adaptive management
Summary Points:Although there are many reasons why people care about defining and tracking successful adaptation, there are also good reasons why people do not want to want to do it...
• It can open up funding and political sensitivities.
• It takes a lot of work to define, track, and fund success.
Findings:Good reasons for NOT thinking about success
• Political sensitivities• Funding sensitivities• It’s work, takes capacity,
funding…(“It’s too hard” is NOT a good reason!)
Summary Points:• Success is difficult to define and success
for one person or group may not be success for another.
• There is no one target or metric. It’s multi-dimensional.
• With continuing climate change, “success” in adaptation is never final, so it is valuable to think of progress or effectiveness in achieving something you said you wanted to do rather than fixate on a finite notion of success.
Findings: Top-level, cross-cutting insights
• What is viewed as “success” depends in part on how you interpret “adaptation”
• “Success” tends to be more difficult to define than “failure”
• While there may be positive synergies, often “success” in one area involves trade-offs in others (across sectors, scales)
• With continuing climate change, “success” in adaptation is never final > “progress”
• There is no one target or metric > multi-dimensional
Findings: Six key dimensions of adaptation success
Adaptive Capacity• Establish enabling conditions
•Build up social, technical, human, financial etc. capacities
Adaptation Process
* Conduct the assessment and planning process “right”* Engage in continual assessment of adaptation needs
Adaptation Decision-Making
•Select a “good” adaptation option
•Make a “good” adaptation decision
Adaptation Implementation
•Successfully implement specific adaptation actions, next step
•Set up ongoing process
Adaptation Outcomes
•Find adaptation outcomes to be “good”, or “acceptable”
•Avoid maladaptation
Adaptation Barriers• Identify and develop effective strategies to overcome barriers to adaptation
(institutional, motivational, political, financial, scientific etc.)
Summary Points:• The project came up with six key
dimensions of adaptation success/progress. If you do not discuss or measure what is happening in each dimension, you fundamentally cannot tell the story of adaptation success.
• In thinking about success, it’s important to ask: o What process are you setting up?, o How are you making decisions?, o What actions are actually taken? o What do they result in?/do they achieve a desired outcome?, o Is the necessary capacity there? and o Are the barriers encountered in that process being overcome?
Summary Points:How did the team engage the NERRS in this project? • All NERRS reserves are indicated in
green. Reserves that had expressed interest in participating in the project at one time or another are shown in yellow. Reserves that were actually involved in the project are indicated in red.
• “Yellow” reserves had a few different reasons for not participating. Some, such as Puerto Rico and Texas, were busy dealing with natural disasters. While the context and interest for them still exists, they have had limited capacity to participate.
• Others had interest but either they or Susi’s team were constrained with funding. Reserves themselves had to come up with their own funding to support the work from this project on their own reserves.
• Participating reserves are not geographically balanced but have diversity of geography, context, and issues.
INDICATORS FROM THE BOTTOM UPWorking with National Estuarine Research Reserve System
Potential reserve partners Participating reserve partners
Wells
Hudson River
JacquesCousteau
Tijuana River
KachemakBay
Summary Points:• All reserves were invited to participate.
The project team informed reserves of the participation criteria, interviewed interested reserves, and assessed their readiness and capacity to participate.
Reserve selection -An iterative, open, transparent process
• Introduction of project to NERRS > open invitation, clear criteria
• Interviews with all interested reserves to • understand context and opportunity, • assess readiness and capacity to co-design/co-facilitate the
workshop and follow-on activities• Those not selected invited to nearby reserve workshops• All reserves kept informed of progress, professional
sharing sessions and solicitation of input at NERRS/NERRA Annual Meeting sessions
• Conversations with other interested reserves continuing
Summary Points:• A key objective of the SAIM project was
to co-design the project with reserves. In each case, we worked with them to define what success meant and to determine how successful adaptation could fit into the work the reserves were already doing, often with their surrounding communities.
• We worked with reserve staff and stakeholders they had invited to develop a set of indicators and metrics to track progress.
Objectives of SAIM
OVERARCHING NERRS-FOCUSED OBJECTIVE: HELP RESERVES 1. Define “success” for their own purposes2. Develop useful, impactful indicators and metrics to track progress
(along adaptation pathways)3. Learn from other reserves (using a multiple-site, comparative
approach)
OVERARCHING BROADER OBJECTIVE: CONTRIBUTE TO SCIENTIFIC AND POLICY DEBATES1. Share lessons with regional partners, other reserves, coastal scientists
and managers faced with similar challenges2. Contribute to national indicator system3. Elevate the profile of the System
Summary Points:Because of the co-design approach, the focus of each pilot project was tailored to the needs and wishes of the individual reserves, while also informing the SAIM effort overall.• Wells Reserve - Ten nearby towns had done some
adaptation work, and Wells began tracking their actions and shared it with the towns. Since then, that tracking of actions/plans/activities and sharing it on annual basis has not only helped them learn from each other but created a bit of competition among the towns which has spurred more efforts in adaptation.
• Hudson River Reserve - Many villages on the river don’t have paid staff or the capacity to track adaptation. The state is using a carrot-and-stick approach to encourage SAIM. It has set up an indicator system and is using various programs to pull together information about where communities are in terms of adaptation, and communities that participate are more likely to receive state funding.
• Jacques Cousteau Reserve – The reserve had been very involved in post-Sandy resilience assessments, worked with state emergency management and FEMA Region 2 to explore the question, “How do we know we’re any better prepared now than before Sandy and all the efforts made since?”
• Tijuana River Reserve – The reserve wanted to develop indicators to track and assess the reserve’s own adaptation actions. It also is involved in regional adaptation efforts. Post SAIM workshop, reserve staff used the outputs to move the identified indicators and metrics (I&M) into their workplan. The indicators are now institutionalized in the context of their reserve.
• Kachemak Bay Reserve – The SAIM project became part of local and regional planning processes in which the reserve was already involved. Also connected SAIM with another project that was presented in the last webinar by Dani Boudreau about climate scenario planning (visit: graham.umich.edu/water/nerrs/webinar).
Wells/SouthernMaine
Hudson River
Jersey shore
Tijuana River/SanDiego
Kachemak/Kenai Pen.
Reserve-specific foci and outcomes to date
Tracking actions in 10 towns; making inroads to business community
Building indicators and metrics into Reserve work plan; Stimulated regional conversation on “success”
Embedding I&M into local and regional planning updates;Connected I&M with scenario planning and pathways
Explored existing resilience tools (incl. CRS) as basis for SAIM indicators; adding motivation for FEMA Reg. II to advance its resilience indicators
Local capacity to track I&M constraints significant; adding motivation for NY state testing its indicator systems used in carrot-&-stick approach
Summary Points:Some lessons we have learned from these pilot projects:
• It is important to recognize that in a bottom-up process like we’ve gone through with the reserves, there isn’t one simple set of indicators or metrics for successful adaptation. They are heavily influenced by values and will vary organization-to-organization and community-to-community.
• There’s an inclination to make inventories of actions, noting ‘We passed XYZ plan’ or ‘We built a culvert’ or ‘We passed the budget.’ But it’s harder to get people to specify desirable outcomes, asking questions like “Is this outcome good?” or “Is this outcome what we wanted?”
• It is also important to consider what indicators and metrics will be used for once they have been established. Too often, we track things that end up on a shelf and don’t have any real impact.
• Capacity constraints are very real. Many communities just don’t have the staff, time and money or know-how for identification, selection, tracking & use of indicators and metrics.
Lessons learned from and with communities
• Searching for indicators and metrics is a difficult, time-intensive, value-laden, not apolitical conversation
• Inclination towards inventories of actions instead of outcomes
• Existing incentives and structures for tracking, evaluation may be productive starting point (e.g., CRS, existing reporting), but often not enough
• Capacity requirements are very real for identification, selection, tracking & use of indicators and metrics
17
Summary Points:• The conversation about indicators
and metrics (I&M) isn’t new. But how to create useful indicators for practice has a lot to learn from other fields, such as actionable information, effective decision support, evaluation science & practice, scenario planning, etc.
• It is not feasible or necessary to come up with dozens of indicators. A small set of purpose-driven, decision-relevant and meaningful indicators could really matter…but the set will vary greatly across users, contexts & capacities.
• There are benefits and trade-offs of having an off-the-shelf, standardized set of indicators and having tailored, boutique, place-specific indicators.
• Developing indicators and metrics is not an extra task - it is part of the work of doing adaptation planning. Look to see where it can be embedded in existing processes and projects.
• To be usable, adaptation indicators & metrics must embrace learning from actionable information, effective decision support, evaluation science & practice, scenario planning, etc.
• A small set of purpose-driven, decision-relevant and meaningful indicators could really matter……but set will vary greatly across users, contexts & capacities.
• Adaptation I&M must be considered part of—not instead of or in addition to —the hard, collaborative, and iterative work of adaptation practice.
See blog post: Susanne Moser, Why we need to do better on adaptation indicators, March 19, 2015, http://www.scidev.net/global/climate-change/opinion/better-climate-change-adaptation-indicators.html
Lessons learned from and with communities (cont.)
Summary Points:• There are different ways to develop
indicators and metrics. The three processes outlined have different starting points (left-most boxes), which then require different steps for progressing towards producing indicators and metrics.
• The SAIM project used the second model and applied a conceptual framework of success dimensions to real-world situations.
Developing indicators & metrics –Very different starting and end points
Development of action logic
model
Indicators for all ALM elements
Metrics for all indicators
Internal tracking sheet (manual,
then digital)(a)
Vision of success Strategic planning
Targets/indicators of success & Indicators of
progress
Metrics for targets/progress
indicators(c)
(b)Conceptual
framework of success
dimensions
Real-world application
Indicators of success in core
dimensions
Metrics for each indicator
Summary Points:• The rest of the webinar goes into a
bit more detail on the process that was followed with reserve staff and stakeholders of Kachemak Bay NERR in Homer, Alaska.
• There, participating community members and local government staff first developed a desirable vision of their future and then went through key elements of strategic planning to come up with goals, indicators and metrics of progress and success.
One powerful approach
Homer, Alaska
Vision of success Strategic planning
Targets/indicators of success & Indicators of
progress
Metrics for targets/progress
indicators
Summary Points:• The project team worked with
Kachemak Bay and the community over the course of 13 months and three workshops to develop indicators and metrics (I&M) for successful adaptation.
• They wanted to make sure that the project would make a lasting and tangible difference in the community, so they focused on looking at a problem that Homer is currently facing and needs to be addressed in their planning efforts.
• They connected I&M development to the strategies that the community and reserve could use to address the identified problem.
Climate scenariosVision for Homer & Kenai Peninsula
Pathways & Trigger points
Introduction to climate change science –Observed and projected
changes
Basics steps of the adaptation planning
process
Communicating climate change and
adaptation
Mar
ch 2
016
Oct
ober
201
6Ap
ril 2
017
Climate-sensitivestrategies to solve existing problems
Transparent adaptive decision-making
Indicators of progress and success, and building capacity for ongoing
learning
Learning together: An adaptation workshop series
SCENARIO PLANNING AND PATHWAYS TO SUCCESSFUL ADAPTATION
SUCCESSFUL ADAPTATION PART II: STRATEGIES, PATHWAYS, AND EVALUATION
CLIMATE ADAPTATION FOR COASTAL COMMUNITIES
Summary Points:To develop a strategy-specific indicator, the team followed this process adapted from the National Academies of Sciences (2015):• Start with visioning of a desirable future and
the development of locally relevant climate scenarios.
• Then pick a problem with which local officials or people are currently grappling.
• Work together to come up with a series of strategies to address this problem.
• Once these strategies are laid out, evaluate them against the ‘vision.’ Assess how commensurate each of these strategies is with achieving the vision.
• Adjust or eliminate any strategies that do not fit the vision.
• Then evaluate whether the remaining set of strategies work equally well under each of the potential climate scenarios, and identify ways in which they would need to be adjusted to work under each of the scenarios.
• Based on the remaining strategies, develop indicators and metrics that will demonstrate that the strategy is being implemented. There may be more than one metric per indicator.
Key lesson: Develop strategy-specific indicators
Vision
Strategy
Strategy
Strategy
Indicator
Indicator
Indicator
Indicator
Adapted from NAS, 2015
Initial work Next step
Metric
Metric
Next step
Metric
Metric
Current problem
Check for robust-
ness against climate
scenarios
Climate scenarios
Next step Next step Next step
Summary Points:Here’s an example of how strategy-specific indicators and metrics were developed in Homer:
• In Homer, food security is a big issue. Since they are located on the end of a peninsula, and there is only one road connecting them to the harbor where food is shipped in, climate change poses a threat to the community’s access to food.
• One strategy to reduce that vulnerability is to develop a local seed bank that enables residents to grow their own food, rather than relying on transportation to bring it in from elsewhere.
• A seed bank makes sense as a strategy because it can be built with seeds that work under different climate scenarios that Homer could face.
• Here we list four progress indicators (yellow) and one actual outcome indicator (green), and their accompanying metrics. Some metrics are simple ‘Yes/No’ questions, while others are more quantitative.
Example of outcome and progress indicators
Vision Element: Food security
Seed bank
Use
Supportive legislation
Impact investment
Seed library
Adapted from NAS, 2015
Formed? Yes/no
number, diversity
of patrons
In place/in progress
# and type of co-benefits
Acres in food
production
Ecomap
completed? Yes/no
Outcome Indicator
Progress Indicator
Metric
Summary Points: So another lesson from the SAIM project is:
• Indicator and metric work is really time consuming and intense. It requires a lot of thinking about how to identify and track them. It is essential to have a clear purpose or motivation for doing it.
• As much as possible, indicators and metrics should be embedded in an existing (or new) institutional structure or tied to things that reserve staff, government staff, or stakeholders are already doing. This increases the chance that tracking and measuring them will actually happen.
• Capacity is an important consideration. It takes staff, time, knowledge to develop, track and use indicators. Consider engaging individuals or groups that can help track them.
Upshot: No indicators and metrics……unless you think hard about how to make it happen
• Clarity of Purpose: • What you need for what purpose and for whom?
• Embedding Indicators and Tracking: • Can existing processes be used/expanded?• Can indicators/metrics be turned into performance
measures?
• Capacity: • Who has the capacity to develop and track indicators,
see it through?• Do you have partners who can be enlisted?
Summary Points:What’s next for the SAIM project?
• We are now turning towards making the lessons we learned, the outputs we produced, the facilitation tools, and other resources available to the entire NERR System and others. We plan to do that by producing facilitation guides and write-ups of our five case studies.
• Here is some very preliminary thinking on how we might organize these resources: around how to explore and define indicators, how to select them, how to track them over time, and how to use them.
1. Exploringindicators
2. Selectingindicators
3. Trackingindicators
4. Usingindicators
What’s next? – Organizing lessons & facilitation tools
Summary Notes:• But we’re not done, and no one has
this figured out. So, the SAIM team and their NERRS partners are in an ongoing learning process and will continue to work on this challenge.
• There is a clear sequence and progression in terms of learning about successful adaptation and the development of indicators and metrics. And we expect it to continue.
In summary: We’re moving the ball down the field
Core Dimensions of Success
Stakeholder-driven framing ofwhat successful adaptation looks like
Telling the story of successfuladaptation to climate change
Arnott, Moser, & Goodrich 2016Environmental Science & Policy
The basics… theory ……………. A practice-driven framework …….. Learning how to develop I&M …… What are “good” indicators?...
QUESTIONS:Is the visioning process you mentioned narrowly focused on climate adaptation or can it be broader?We didn’t put constraints on the visioning. We generally asked our project partners: “What is the community that you want to live in by 2050?” This was particularly interesting in Alaska, which is one of the few places we’ve been where the predominant notion was to keep it the way it is. Residents love their region and lifestyle, so their vision was much broader than about climate adaptation. The strategies got much more specific when they were tied to existing problems they were facing (i.e. crumbling roads, food insecurity), but we did not restrict the visioning process.
Is the content from the workshops you held in Homer, AK available? Some of this information can be found on the Kachemak Bay website: http://accs.uaa.alaska.edu/kbnerr/climate-resilience/
Will your products or recommendations include information about the level of effort needed to develop and track different types of indicators or metrics so communities can determine if they have necessary capacity?We are not able to say how long tracking these different indicators and metrics (I&M) would take. We know that communities and organizations are strapped and don’t have extra capacity, and this is why we are advocating for groups to embed tracking and measuring I&M into existing processes and to work with partners. If you can identify places where tracking can be woven into something the community is already doing or find a partner (i.e. university, NGO) that has tracking and measuring I&M in their mission, then it’s much more likely to be done and to be useful.
Please get in touch!
Susi Moser, Ph.D. James Arnott
Question and Answer:NOAA RISA is grappling with finding a way to measure success across multiple projects across the country. Some of this is being driven by federal agency needs to justify spending, but it’s also a concern to many of those working in the communities. Have you had any success looking at metrics across projects? My work has focused on helping different groups and agencies figure out what success means to them, and in the context of NOAA RISA, I have the most experience in working with the Pacific RISA. They have really been through an evolution of the sequence of processes or approaches to developing I&M, and I think this is an evolution that each group or agency needs to go through. I don’t think there’s an easy way to simply connect I&M across projects unless they’re coordinated in some way from the start.
It seems like a lot of indicators and metrics are highly localized. In thinking about creating national indicators of success or progress in climate adaptation, would it make sense to look at the number of projects over time achieving a certain percentage of their local indicators? That’s certainly one way to get an overall impression. But if I were to ask the question, “How many communities are doing anything about adaptation?” no one could answer that because it’s difficult to track what is happening at any level of synthesis; communities and organizations embed and mainstream things differently. At the national level, I would love to see more conversation around the question “Are we better prepared given all the money we’re spending?” and then identify at the national or state level how to define preparedness and who has the capacity and resources to track it. Then communities would define success at their level based on the specific climate adaptation projects they’re working on.
Please get in touch!
Susi Moser, Ph.D. James Arnott