planning and designing implementation - … · web viewto clarify the role of the project in the...

A GUIDE TO WILDLIFE RESOURCE VALUE

EFFECTIVENESS EVALUATIONS

DRAFT

VERSION 1.0

10 December 2013

Prepared by Laura Darling and Kathy Paige, Ministry of Environment

andDarcy Pickard, ESSA Technologies

Forest and Range Evaluation Program Wildlife Resource Value

Ministry of Forests, Lands and Natural Resource Operations and Ministry of Environment

A Guide to Wildlife Resource Value Effectiveness Evaluations - Draft - 1 -

ACKNOWLEDGEMENTS

This document is based on the learning of many individuals working on monitoring and effectiveness evaluation projects. Many thanks to Scott Barrett, James Dalby, Todd Manning, Melissa Todd, Richard Thompson for providing review comments.


PREFACEThe Forest and Range Practices Act (FRPA) and regulations introduce a transition to a results-based forest practices framework in British Columbia1. Under this new approach to forest management, the forest industry is responsible for developing results and strategies, or using specified defaults, for the sustainable management of resources. The role of government is to ensure compliance with established results and strategies and other practice requirements, and evaluate the effectiveness of forest and range practices in achieving management objectives.

Forest and Range Evaluation Program (FREP)2 is a multi-agency program established to evaluate whether practices under FRPA are meeting not only the intent of current FRPA objectives, but also to determine whether the practices and the legislation itself are meeting government’s broader intent for the sustainable use of resources.

The Minister of Environment is responsible for designation of important mechanisms described in FRPA to address conservation of wildlife habitat – including Wildlife Habitat Areas, Ungulate Winter Range and others. The Wildlife Resource Value Team (WRVT) of FREP, a joint initiative between the Ministry of Environment, and the Ministry of Forests, Lands and Natural Resource Operations, is responsible for designing a program for monitoring and evaluating wildlife objectives established under FRPA for these mechanisms. The WRVT 3 has prepared a strategic framework for monitoring effectiveness of wildlife resource values, along with various guidance documents and reports describing work completed under the Wildlife Resource Value component of FREP.

This Guide to Wildlife Resource Value Effectiveness Evaluations is one of the key guidance documents available at the FREP WRVT website. It is designed to provide guidance for establishment of projects to evaluate effectiveness of tools or mechanisms available in FRPA that are intended to conserve wildlife resource values, particularly Wildlife Habitat Areas and Ungulate Winter Range. It has been developed with FREP wildlife resource value objectives in mind and the intended audience is FREP wildlife resource value project leads.

1 For more information on FRPA and its regulations, resource values, objectives, etc., see http://www.for.gov.bc.ca/code/ .2 For more about FREP see: http://www.for.gov.bc.ca/hfp/frep/index.htm 3 For more about the WRVT see the WRVT website: http://www.for.gov.bc.ca/hfp/frep/values/wildlife.htm


http://www.for.gov.bc.ca/hfp/frep/values/wildlife.htm

http://www.for.gov.bc.ca/hfp/frep/index.htm

http://www.for.gov.bc.ca/code/

TABLE OF CONTENTSACKNOWLEDGEMENTS.................................................................................................2PREFACE............................................................................................................................3TABLE OF CONTENTS....................................................................................................41 INTRODUCTION.......................................................................................................6

1.1 Purpose of Guidance Document...........................................................................71.2 How to Use this Guidance Document...................................................................9

2 PLANNING.................................................................................................................92.1 Framing the Monitoring Objectives and Questions............................................102.2 Prioritizing the Monitoring Objectives and Questions.......................................132.3 Compiling Knowledge Base...............................................................................15

2.3.1 Ecological Knowledge (or Data).................................................................152.3.2 Management Activity..................................................................................162.3.3 Threats and Risks.........................................................................................16

2.4 Demonstrating Relationships – Conceptual Models...........................................172.5 Determining Indicators (Measures)....................................................................19

2.5.1 Indicators to Monitor...................................................................................202.5.2 Thresholds and Expected Values.................................................................232.5.3 Developing an Evaluation Tool for Effectiveness Indicators......................24

2.6 Pilot testing a protocol........................................................................................282.7 Tracking Project and Program Efficiency...........................................................29

3 DESIGNING A STUDY...........................................................................................303.1 Designing the Approach......................................................................................30

3.1.1 Study design.................................................................................................313.1.2 Delivery model............................................................................................323.1.3 Spatial scale.................................................................................................323.1.4 Temporal Scale............................................................................................333.1.5 Intensity.......................................................................................................333.1.6 Location.......................................................................................................343.1.7 Methods.......................................................................................................343.1.8 Overlap with other monitoring – FREP and other programs.......................34

3.2 Sampling Design.................................................................................................343.2.1 Life History Characteristics.........................................................................363.2.2 Target Population.........................................................................................363.2.3 Sampling unit...............................................................................................373.2.4 Positioning of sampling units......................................................................393.2.5 Sample size and statistical power................................................................423.2.6 Sampling frequency.....................................................................................433.2.7 Field methods...............................................................................................463.2.8 Planning the data analysis............................................................................47

4 IMPLEMENTING A PROJECT...............................................................................494.1 Confirm Agency Support....................................................................................494.2 Prepare a Monitoring Plan..................................................................................494.3 Data Management and Quality Assurance..........................................................50

5 REPORTING.............................................................................................................50


5.1 Report Templates................................................................................................505.2 Input to Decision-Making and Resource Management.......................................51

6 LITERATURE CITED..............................................................................................51Appendix 1. Example study designs.................................................................................54Appendix 2. Examples of sampling designs.....................................................................58


1 INTRODUCTION

Effectiveness evaluation is an assessment of the results of investing resources and conducting activities to (make progress toward or) achieve some stated program or project goal or objective. The investments and activities are internal to the program (or project), controlled by program decisions and can be described in terms of their efficiency. The results of the program, the direct or indirect short- or long-term outputs and outcomes of the program, are described in terms of their effectiveness.

There are many reasons to implement effectiveness evaluations: To maximize the impact of limited resources; To gain insight into strengths and weaknesses of a particular management

approach; To clarify the role of the project in the complex ecological (and social) systems To build support by documenting success; To improve practices in the face of unavoidable uncertainties and inevitable

change; To ensure conservation of target species and ecosystems; and To show stakeholders that efforts and policies are justified, dollars well spent.

Effectiveness evaluation fits into a process of checking, self-assessment and reflection that any program should incorporate into its business to determine if the program is proceeding as desired and expected. Questions that need to be asked in the evaluation process are: (1) what is the management objective and where do we want to go? (2) what practices will be implemented and how will we meet the objective? (3) are we going in the right direction? and (4) how will we provide feedback so we can change direction and improve if we need to? Effectiveness evaluation addresses the third question.

Monitoring is a component of an effectiveness evaluation. It is the process and rules for collecting and analyzing information. The key components of an effectiveness monitoring and evaluation program include clearly stating the goals, developing a conceptual model linking relevant ecosystem function components (i.e, species and habitats) and stressors/disturbances (risks), selection of suitable indicators to monitor, estimating their status and trend, determining values that trigger a management response, and linking monitoring results to management (Lint et al 1999; Noon 2003).

There have been several data and program assessment processes developed for a variety of types of projects and proponents, all with the goals of (a) efficient and effective use of resources, (b) collecting the right type of data in sufficient quality and quantity to support the goals of the study, and (c) linking results of the monitoring with decisions for improved resource management. The Environmental Protection Agency (2006) Data Quality Objectives Process (DQO) for evaluation monitoring is a series of logical steps that guide planning for the resource-effective evaluation of environmental data that measures performance (compliance) and/or acceptance criteria. The National Park Service (NPS) “Vital Signs Monitoring” (National Park Service 2012) determines the


status and trend in the ecological condition of selected park resources. The Environmental Monitoring Initiative from the University of Michigan (Schueller et al. 2006) provides a user-friendly step-by-step process based on the Logic Model for developing and implementing evaluation plans for ecosystem and community-based projects. And the Nature Conservancy of Canada (NCC) has developed a structured process for monitoring the effectiveness of NCC conservation strategies to meet program objectives in the lands they administer or manage (Gordon et al. 2005).

The Wildlife Resource Value Team (WRVT) of the Forest and Range Evaluation Program (FREP 4), a joint initiative between the Ministry of Environment, and the Ministry of Forests, Lands and Natural Resource Operations, is responsible for designing a program for monitoring and evaluating wildlife objectives and management mechanisms established under the Forest and Range Practices Act (FRPA).

In the context of wildlife resource values and implementation of the FRPA, effectiveness monitoring and evaluation are conducted to answer broad questions such as: Are we conserving what we say we are? Are we achieving what we say we are? What can we do to enhance our success? And more specifically, we might ask: How is the species of concern (or biodiversity in general) doing? Are our actions having the intended impact? To answer these questions and assess status and condition of a species, habitat or threats, a suite of biological and/or threat indicators is tracked over time. When a conservation intervention or management practice is initiated or underway, comparisons are made of the current and desired condition, the relative impacts of different treatments, or effects of pre-and post-treatment condition.

None of the four assessment processes described above (DQO, NPS, EMI and NCC) is fully compatible with the kind of evaluations needed for FREP, but the strengths of each (quality assurance concepts from DQO, ecological principles from NPS, the stepwise, cyclic model from EMI, and the application to land use planning and stewardship objectives of NCC) have been drawn on to develop this effectiveness evaluation process for the wildlife resource values of FREP – the Guide to Wildlife Resource Value Effectiveness Evaluations. The reader is encouraged to review these other assessment processes to gain a deeper insight into the rationale for and development of sound effectiveness evaluation projects.

1.1 Purpose of Guidance Document

This document is designed to provide guidance for establishment of projects to evaluate effectiveness of tools or mechanisms available in FRPA that are intended to conserve wildlife resource values, particularly Wildlife Habitat Areas (WHA) and Ungulate Winter Range (UWR). It has been developed with FREP wildlife resource value objectives in mind and the intended audience is FREP wildlife resource value project leads.

4 For more about FREP see the FREP website: http://www.for.gov.bc.ca/hfp/frep/index.htm .


http://www.for.gov.bc.ca/hfp/frep/index.htm

There are four parts to this document, based on four major steps involved in effectiveness evaluation, as outlined in Table 1. Each step involves several tasks and output products, which together will lead to achieving the evaluation objectives. Not all projects will proceed to the end step – though all tasks are thought to be necessary components of the compete process, progress will vary for different situations and for some species there will be value in achieving only some initial tasks (e.g., determine need and current state).

Table 1. Tasks and outputs of the four steps in the development and implementation of effectiveness evaluations in FREP wildlife resource value projects.

STEP TASK OUTPUT PRODUCT1. Planning: bound the problem

Frame monitoring questions & objectivesConsider priority of monitoring questionDetermine current stateBuild conceptual model of relationshipsSelect indicatorsEstablish indicator thresholds or triggers and how condition or effectiveness will be established

Draft project charterConceptual modelIndicatorsStandard protocol

2. Designing: the Approach & Sampling Design

Establish study and sampling designDetermine data collection methods Test and refine methods (pilot testing)Determine data analysis methodsUndergo statistical review

Monitoring plan (includes study and sample design)

3. Implementing a project

Project planningSecure resourcesTrainingOutline quality assuranceCollect data

Project CharterMonitoring plan Field forms

4. Evaluation & Reporting

Analyse dataEstablish baselines for indicatorsReport study resultsSummaries and advice to decision makers

Reports, Extension Notes, scientific papers, management recommendations

Figure 1. The main steps in an effectiveness evaluation.


1.2 How to Use this Guidance Document

Where does my project fit?

Though the target audience for this guidance document has a common responsibility as project leads, different users will have different needs and therefore different ways they might use this document. Some investigators may be studying an animal for which there is very little known so protocol development must begin with developing a basic understanding of the species and ways to monitor it. Other projects may focus on species for which biological knowledge and field methods are well developed, or where a preliminary effectiveness protocol already exists: in these situations there may be no need to undertake Steps 1 or 2, in Table 1. They may ask: what pieces do I have; where in this process do I jump in; if steps 1 and 2 are done, what do I do?

Users of this guidance document should determine just where they are in the process of development and implementation of an effectiveness protocol by reviewing the “action checklist” (yellow text boxes) at the end of key sections, and insert their project into the appropriate step to match their need.

How is the document organized?

Sections 2 – 5 describe the four steps for effectiveness evaluation outlined in Table 1. The reader (project leader) can enter at any appropriate step in the process.

Blue-shaded text boxes at the beginning of a section clarify the intent of the steps or actions described in the subsequent section.

Yellow-shaded text boxes at the end of a section summarize the activities or steps to be taken to complete the actions outlined in the section. These steps may be used as an “action checklist”.

Purple-shaded side-bars provide supporting information or more detailed explanations of concepts presented in the document.

2 PLANNING

In monitoring projects it is important to use an established or standard protocol and, if one does not exist, to develop one. A protocol is a procedure, practice or set of rules. In the realm of environmental surveys, a detailed step-by-step process for collecting field data might be labelled as a protocol, or a broader definition of protocol might be used that encompasses steps to develop the study design, collect and analyse the data, and report the results. The latter, all-encompassing definition of protocol is the one used in this document. Standardized step-by-step data collection procedures and techniques are


referred to as methods in this document. The goal of a monitoring protocol is to establish consistency, transparency and minimize or balance measurement error and sampling error. Measurement error is that which results from differences in collection methods, rather than actual changes in the environment (Oakley et al. 2003). Sampling error results from taking measurements from a subset of a population and is minimized through sampling design.

Before deciding to embark on a project or develop a protocol, it is important to do some background work to determine if a project is a priority and assess the state of our current knowledge. Section 2 is arranged to answer the following questions:

2.1 What are the objectives of the monitoring project?What are the key questions we wish to answer through monitoring?

2.2 What are the highest priority objectives and questions for monitoring, and how does this project rank relative to other objectives and questions?

2.3 What do we already know about the species, its habitat and management practices? 2.4 What are the relationships between the many variables that describe the species, its

environment and threats to its survival? 2.5 What are the best species, population, habitat or threat variables (i.e., indicators) to

measure to assess achievement of project objectives and provide answers to the priority monitoring questions? What indicator values will serve as thresholds that will trigger a management response? Are there standard protocols for measuring indicators?

2.6 What potential pilot projects are necessary to test the approach, indicators and methods?

2.7 What performance measures should be tracked to document project efficiency of the project?

2.1 Framing the Monitoring Objectives and Questions

Section 2.1: In this step, the key management issues and questions are clarified:- What is the expected or desired outcome of the management practice? What is

the objective of the FRPA mechanism? What key management issues may be hindering the desired outcome of the

management practice? Who is the audience for the results of the study – the land manager, a statutory

decision-maker, government policy, the public, or all of these? What questions need to be answered in order to address the management issues

and improve the management outcome? What is the focus of the issue (and monitoring) – threat reduction, habitat

quality or population status? At what scale does the issue (and monitoring) apply – local, watershed, or

regional?

Development of a wildlife resource value effectiveness evaluation project begins with clarification of the key management issues or questions (the uncertainties) regarding the FRPA mechanism (Ungulate Winter Range [UWR] or Wildlife Habitat Area [WHA], etc.). Usually, there is uncertainty about whether the expected outcomes of the current


management scenario (the objectives of the FRPA mechanism) are being achieved. Pertinent key management issues (or unknowns) are the source of the very specific questions and hypotheses examined in an effectiveness evaluation project. By zeroing in on recognised knowledge gaps and relevant management issues, an effectiveness monitoring project will be directed to the area of greatest need and will produce a sound evaluation of effectiveness.

Broad scale priority questions that need to be addressed for UWRs and WHAs (see SIDEBAR #1) have been confirmed by the FRPA Resource Evaluation Working Group5. Although these questions are both too complex and too general to serve as a focus for a single evaluation project, they point to several more specific topics that could be targeted for evaluation, including amount, quality and distribution of UWRs and WHAs, survival and fitness of the species they are designed for, or trends in threats to the habitat or species.

SIDEBAR #1 The key question being asked about ungulate winter ranges and wildlife habitat areas [See http://www.for.gov.bc.ca/hfp/frep/about/questions.htm#questions ] are:

1. “Do ungulate winter ranges maintain the habitats, structures and functions necessary to meet the species winter habitat requirements, and is the amount, quality and distribution of UWRs contributing effectively with the surrounding land base (including protected areas and managed land base) to ensure the winter survival of the species [Project Charter -within the landscape] now and over time?”.

2. “Do WHAs and the associated GWMs and objectives combine to maintain the habitats, structures and functions necessary to meet the goal(s) of the WHAs, and is the amount, quality and distribution of WHAs contributing effectively with the surrounding landbase (including protected areas and managed landbase) to ensure the survival of the species now and over time?”

Specific monitoring questions investigated for WHAs and UWRs might be at small or large spatial scales and/or addressing single or multiple environmental parameters. For example, specific questions might be:

Does this WHA provide sufficient habitat to provide for survival of the speciest? Is habitat attribute X above a standard habitat quality threshold within the UWR? Do WHAs in this region provide better habitat for the species at risk than similar

areas that are not designated as WHAs? Is the species at risk population persistent in this UWR?

Selection and refinement of monitoring questions requires consideration of other factors beyond the management issue itself (Gordon et al. 2005, Schueller et al. 2006). Questions should be framed considering who regulates the management practice, our ability to influence the management practice, and whether appropriate legal changes are possible. Opportunities for partnerships, cooperation and cost efficiencies need to be considered as

5 The FRPA Resource Evaluation Working Group (FREWG), representing the Ministry of Forests and Range (Field Services Division, Research Branch and Forest Practices Branch), as well the Ministry of Environment (MoE), and the Ministry of Agriculture and Lands (MAL) developed the FREP structure and framework and initiated development of resource value indicators and monitoring /evaluation protocols.


http://www.for.gov.bc.ca/hfp/frep/about/questions.htm#questions

well as whether the target species or habitat is suited to cost-effective monitoring. Limitations and potential barriers such as budget constraints, logistics and access issues also need to be considered. Some of these factors may be taken into account in the prioritizing process (see section 2.2), however considering them in a broad sense right from the start will serve as a coarse filter.

In some cases identifying the unknowns and framing management questions will be straightforward. But some issues will be less clear and determining specific management questions will be more complex. The concerns about effectiveness of WHAs or other FRPA mechanisms will need to be defined in detail – each issue should be defined separately and completely in an uncomplicated format (see 2.4 Conceptual Models).

To be valuable for improved future management, effectiveness evaluation projects must be based on answerable questions and hypotheses that are specific and precise. The questions must be SMART (specific; measurable; attainable, achievable and appropriate; realistic, relevant and results oriented; and time-bound; Doran 1981, Meyer 2003) and phrased in a way that ensures that the project leader and co-operators agree on the most important information needed. Questions and hypotheses should be framed to provide the practitioner with a clear-cut answer and direction for management: not asking for a “yes” or “no” answer but rather “how much? or “to what extent?” Key questions and the null hypotheses must be formulated and confirmed prior to initiating monitoring so appropriate sampling design and data analysis can be applied and the chance of reaching erroneous conclusions is reduced (Shaffer and Johnson 2008).

Monitoring questions may address issues of population abundance, habitat suitability, use of the area by wildlife, or mitigation of threats. Questions may address the generic or specific goals of WHAs and UWRs (see SIDEBAR #2). They may deal with assessing status of a species or habitat, or effectiveness of an action, and be focused at a local, landscape or regional level. Or questions may address progress toward meeting management goals, threat reduction, improving condition or successful completion of activities. Though focusing on one type of question (say, for example, species and population questions at a local level) may be believed to address the major management question, successful effectiveness evaluations examine questions from a variety of types of questions. Conceptual models (section 2.4) and monitoring indicators (section 2.5) reflect the same categories.

SIDEBAR #2: Examples of generic goals that might apply to all WHAs include:1. Maintain the habitat suitability and key habitat attributes of the area that contribute to survival

of the species in the area (e.g., microclimate, prey species, nest trees)2. Maintain the use of the area by the species3. Maintain the ecological processes (ecosystem functioning) that maintains the species4. Abate stresses from forest and range management practices

Examples of specific goals that might apply to individual WHAs include:1. Reduce road mortality2. Minimize soil disturbance3. Maintain stand structure


4. Maintain security cover

Section 2.1 Key Questions – in practice Describe in detail the concerns and issues about the effectiveness of the FRPA

mechanism. Define each problem separately and completely, in an uncomplicated format.

Frame the concerns in specific focused questions. Ensure that answers to these questions will provide direction for improved

management. The next step is to determine if these are priority issues that warrant effectiveness

evaluation monitoring. And after that, the questions can be further refined and specific, measurable

hypotheses developed for a monitoring project.

2.2 Prioritizing the Monitoring Objectives and Questions

Section 2.2 : In this step, priorities are determined:- Which FRPA mechanisms and which of the species they serve are the highest

priorities for effectiveness monitoring? Which management issues (for those species or mechanisms) are the highest

priorities for monitoring? Does the proposed project meet Ministry and regional mandates, goals and

priorities? Is the proposed project a high priority based on FREP’s Wildlife priority criteria? Is the project one of the highest priorities according to the MOE Conservation

Framework priority setting process?

When considering whether to pursue an effectiveness evaluation of a WHA, UWR or other management tool for wildlife resource values, it is necessary to determine and confirm the priority of doing the project. Provincial and regional goals and priorities for the species and the management tool and potential benefits of the evaluation project at the regional and provincial level must be considered.

Management tools for wildlife resource values under FRPA are established cooperatively by Ministry of Environment (MOE) and Ministry of Forests, Lands and Natural Resource Operations (MFLNRO), though led by MOE. Therefore, all evaluation projects should be aligned with both ministries’ mandates and priorities. High-level goals such as MOE’s “… to provide for healthy and diverse native species and ecosystems …”, and MFLNRO’s goal of “ensuring environmental standards are upheld and environmental sustainability is achieved with resource use activities in British Columbia …” provide the context for the management tools discussed here. Direction is also provided by explicit legal goals for wildlife outlined in FRPA regulations: “… without unduly reducing the supply of timber from British Columbia’s forests, to conserve sufficient wildlife habitat


in terms of amount of area, distribution of areas and attributes of those areas for the winter survival of species at risk of specified ungulate species” (Section 7 of the Forest Planning and Practices Regulation and the Woodlot Licence Planning and Practices Regulation).

Regional goals and objectives must also be considered when determining whether to initiate an evaluation project – these are the desired outcomes of implementation of the management tool. Some WHA and UWR goals and objectives, particularly from the earliest designations, may be unclear, unspecific or difficult to evaluate, and as a result, they may need to be refined to be clear, measurable, and outcome-based. Documented objectives of each established WHA or UWR should be reviewed to define the key issues and questions that need to be investigated and to determine whether effectiveness evaluation is appropriate. There may also be associated goals related to other FRPA resource values that may be evaluated in other FREP projects, such as biodiversity, timber or livestock forage, that the UWR or WHA help to meet.

Therefore prior to undertaking a project, consult provincial and regional goals and priorities and consider the following criteria:

(i) Does the project address a priority question (i.e., prioritized questions put forward by the Wildlife Resource Value Team)?;

(ii) What is the species conservation priority based on MOE’s Conservation Framework?

(iii) What is the province’s investment in management actions (e.g., the number and total area of UWRs, WHAs, etc)?;

(iv) What is the relative importance of the WHA, UWR (etc.) to conservation of the species?;

(v) What is the uncertainty of the effectiveness of the management action(s) and the risk to the value (e.g., species, habitat); and

(vi) What is the likelihood of project success (i.e., project feasibility, benefits, support).

As an example, the procedures for the Environmental Mitigation Policy recommends consideration of the uncertainty of the mitigation activity in combination with the risk to a value to determine whether implementation or effectiveness monitoring is appropriate (http://www.env.gov.bc.ca/emop/).

The FREP Wildlife program posts provincial priorities determined through a priorty-setting process (see http://www.for.gov.bc.ca/hfp/frep/values/wildlife.htm). Projects determined to be the highest priority should be initiated before lower priority ones.

Section 2.2 Priorities – in practice Priority evaluation questions provide the context for developing specific indicators

for effectiveness monitoring.


http://www.env.gov.bc.ca/emop/

Examine where the species ranks in higher level priorities – ministry mandates, regional priorities, etc – and whether monitoring is indicated as a priority action in the Conservation Framework, Monitoring Strategies, etc.

Submit the issue (questions) for consideration to the FREP WRVT – run through the expert-based Prioritizing Process.

2.3 Compiling Knowledge Base

Section 2.3 : In this step we collate current knowledge and data about: Species biology Species habitat use and habitat requirements Known or potential stressors, threats and disturbances Past, current and expected habitat suitability and capability Past, current and expected management practices and adjacent activities

Once key issues and questions have been identified and an effectiveness evaluation has been confirmed as a priority project, the next step is to gather existing biological data and information about management activities and threats (risks) in the area. This collated “current state” information will provide a solid baseline or reference point for the evaluation, and it may confirm the validity of the proposed evaluation objectives or lead to refinement of them. As well, it provides crucial input to conceptual models that will be used to refine key questions into monitoring questions and hypotheses.

2.3.1 Ecological Knowledge (or Data)

The importance of collecting pertinent existing information related to the species, its natural history and ecological role, the particular location, and the management tool (WHA or UWR) in use, cannot be understated. It is imperative to gather and incorporate current knowledge, to ensure the project design addresses advances in research and improvements to indicators. Data and conclusions from baseline, pilot and on-going studies can be included as collected, and added to the knowledge base over time to refine understanding.

Potential sources of information should be investigated – published journals, published and unpublished consultant reports to government or industry, internet publications, results of PEM (Predictive Ecosystem Mapping) and/or habitat modelling, recognised species specialists, MOE species and habitat specialists, and local experts, etc. This information may be text-based or spatial (i.e., maps). Review of this information should confirm the validity of the need for the proposed evaluation project. As well, this review of knowledge will confirm existing information gaps and needs, and aid in development of the study objective and design.

Some general information sources include: Conservation Data Centre http://www.env.gov.bc.ca/cdc/


http://www.env.gov.bc.ca/cdc/

BC Species and Ecosystems Explorer http://www.env.gov.bc.ca/atrisk/toolintro.html

Identified Wildlife Management Strategy Species Accounts http://www.env.gov.bc.ca/wld/frpa/iwms/index.html

Wildlife Species Inventory – Species Inventory Database http://www.env.gov.bc.ca/wildlife/wsi/index.htm

eFlora http://www.geog.ubc.ca/biodiversity/eflora/ eFauna http://www.geog.ubc.ca/biodiversity/efauna/ Wildlife Resource Value Team effectiveness evaluation reports

http://www.for.gov.bc.ca/hfp/frep/values/wildlife.htm Resources Information Standards Committee http://www.ilmb.gov.bc.ca/risc/

2.3.2 Management Activity

The primary management activity of concern in WRVT effectiveness evaluations is the FRPA mechanism or tool being used to protect the species of interest (i.e., a WHA or an UWR) and the management practices associated with that tool (i.e., general wildlife measures). Parameters of the tool and practices must be documented – for instance size and shape of a WHA, number and condition of important habitat features within the WHA, and specifics of required management activities such as no-harvest zones or minimum grazing stubble heights. As in section 2.3.1, this information may be text-based or spatial. Documentation of these parameters will serve not only to determine whether the FRPA tool was implemented as intended, but also to evaluate effectiveness and validate relationships proposed in the conceptual model (see section 2.4).

It is also important to document past and present land or wildlife management activities in and around the study area, because these activities may have impacted (or may impact in the future) the habitat or the target species in some way that might confound the proposed evaluation. Impacts of past or current practices may require refinement of study objectives, and have to be considered in development of study design or in analysis of data.

2.3.3 Threats and Risks

Activities or conditions that exist in or near the study area may upset the natural balance of the ecosystem. Stressors may exist at a low level with negligible impact; they may never or quickly or over a long time period develop into major threats to ecological integrity. One-time or on-going disturbances may be or become substantial threats. Consideration of threats in a spatial context may provide a different perspective than if simply a written description. The magnitude of any threat is manifest in the level of risk it poses to the integrity and well-being of a population or habitat. Some high risks are riskier than others, for example if there are ineffective management techniques to reduce a threat. In addition, one must consider cumulative impacts of on-going or of several concurrent disturbances and threats. Consideration of existing and potential stressors and


http://www.ilmb.gov.bc.ca/risc/


http://www.geog.ubc.ca/biodiversity/efauna/

http://www.geog.ubc.ca/biodiversity/eflora/

http://www.env.gov.bc.ca/wld/frpa/iwms/index.html

http://www.env.gov.bc.ca/atrisk/toolintro.html

threats to the ecosystem under investigation, and their cumulative impacts, may indicate the need to refine study objectives and/or to consider these factors in study design and data analysis.

Section 2.3 Current State – in practice Collate data and reports. Include spatial (GIS, mapping) data Summarize knowledge, activities and stressors that impact the species or the habitat Prepare for conceptual models!!

2.4 Demonstrating Relationships – Conceptual Models

Section 2.4: In this step we ask: What do we know or believe about the relationships between populations,

habitats and stressors (threats)? Can the relationships be quantified, or are we limited to qualitative

descriptions? What are the interactions and influences in the ecological system? What factors and interactions are most important?

Following collation of the current knowledge from previous studies, the next step is to use conceptual models to examine the functional interactions and relationships between species and ecological system components, and to link species and habitats (functions) to threats (risks) and their impacts. A guidance paper by Nyberg (2010) discusses various types of conceptual models, their strengths and recommendations for use in the wildlife resource effectiveness evaluation process; the following summarizes some key aspects of that report.

Usually, a conceptual model describes the effects of various stressors (risks) on the focal population. This general model is then broken down further to show pathways through which particular stressors (risks) could affect habitat attributes and population processes (functions). In the management context of effectiveness evaluation, a conceptual model illustrates the proposed relationship between a management activity and a species response. Developing sound models is crucial in the effectiveness evaluation process because from the relationships demonstrated in the model, a list of candidate indicators is then drafted (section 2.5) for each potential interaction; indicator variables are selected for monitoring from that list.

Models are an important component of monitoring and evaluation projects. They can: document functional relationships between system components, clarify the

linkages between stressors (risks) and outputs or outcomes, and document the understanding of the system being managed and evaluated

highlight important knowledge and data gaps and identify areas of uncertainty that should be addressed through monitoring or research

identify the most important monitoring questions


identify measurable indicators that will answer the questions form the basis for more informative and powerful quantitative models

The interaction of biological factors in an ecosystem is complex, and modelling can improve our ability to understand it. Usually ecosystem parameters are interactive and models based on separate relationships between a species and its environment are not meaningful. Developing an explicit, all-encompassing model of important relationships allows us to compare the importance of changes in different parameters, determine the net effect of those changes, characterize and quantify cause and effect relationships, and set priorities for reducing uncertainty. Subsequently, key relationships will be highlighted to illustrate the focus of an effectiveness evaluation.

Models of relationships can be developed at various levels of detail and scale, from simple graphs and diagrams to complex mathematical simulations. To develop sound effectiveness evaluations, it is necessary to build conceptual models to document relationships between resource values, threats (risks), opportunities and stated goals, objectives and strategies. Input to conceptual models may come from experts and/or output of more complex mathematical modeling procedures.

Conceptual models can be built in table, graph or diagram format. General concept maps (see, for example, “situation maps” in Scheuller et al. 2006), often presented in tabular format, can provide the logical structure to guide thinking about important questions and linkages between goals, ecosystems and population traits. Influence diagrams (see example in Nyberg 2010), using arrows to show direction (and sometimes relative magnitude) of influence, illustrate relationships among different ecosystem parameters and provide a clear understanding of how each contributes to the end result. Baysian belief models (Marcot et al 2001, Nyberg 2010) integrate biological data presented in an influence diagram, with expert opinion and an estimate of certainty (belief weighting) of life history parameter values and relationships among parameters. Unlike concept maps and influence diagrams, Baysian models can be used to predict a range of possible outcomes and their probabilities for populations, habitats, threats, prey, etc., and therefore Nyberg (2010) recommends Baysian models as the preferred format for conceptual models in wildlife effectiveness evaluations, though simpler models may be appropriate when data are lacking.

To create a model of relationships, it is critical that the model be developed fully and correctly, to the level of current knowledge. Important interactions and elements need to be included, and the importance and probability of different parameter values need to be defined. Models should be built collaboratively with species and habitat experts as well as land managers. Simpler types of conceptual models can be built in a day’s workshop, whereas complex mathematical models may require longer preparation time and review. The complexity of a model may reflect the complexity and scale of the objectives of the study and the questions being asked.

An examination of the finalized conceptual model will reveal important knowledge gaps, uncertainties around important parameters, and interactions and threats that may


influence achievement of the management objective (i.e., its effectiveness). The variables that describe the known conditions and relationships and the unknowns, uncertainties and threats are candidate monitoring indicators in effectiveness evaluation. A list of measurable variables to consider as monitoring indicators is one output product of the conceptual model.

Section 2.4 Demonstrating Relationships – in practice Use gathered information of the current state. Convene workshop of species and habitat experts. Develop a conceptual model that describes functional relationships, influences and

interactions among species, habitats and threats (risks) in the broad ecological system. List all the measurable variables that are associated with the model relationships. Prepare to select indicator variables for monitoring.

2.5 Determining Indicators (Measures)

With development of a sound conceptual model describing species, habitat and threat (functions and risks) interactions and relationships, and a list of pertinent candidate measurable variables for monitoring, the next step is to evaluate which variables are the best indicators of whether a management objective is being achieved, and to select a logistically feasible combination of these indicators.

Of key importance to identifying indicators for effectiveness evaluation, is that effectiveness indicators can only be measures of effectiveness if they can be related back to implementation of the tool or practice – that is, how the management activity was undertaken.

Section 2.5: In this step we examine the conceptual model and ask: What variables can (or should) we measure to determine whether the particular

FRPA mechanism is achieving its objective? What variables have the strongest link to directly answering the key monitoring

questions? What attributes of the population or the habitat, if measured over time, will

indicate the effectiveness of the mechanism or the associated practice? What factors contribute to the success of the practice? What balance of population, habitat and threat indicator variables provides the

best opportunity for successfully answering the key monitoring questions? What factors (parameters) are unsuitable for monitoring (perhaps because

appropriate methods of evaluation do not exist, or there are difficult logistic constraints, or high cost) and must be excluded from consideration as a indicator variable?

What protocols are available? And what few, high priority, practical, measurable variables should be selected for implementing a pilot study?


What are the predicted values of the selected indicator variables as a result of the management practice (e.g., implementation of the FRPA mechanism)?

What values of the indicator variables (such as an upper or lower threshold) will trigger a management response?

2.5.1 Indicators to Monitor

Ecosystem elements, risks and the relationships demonstrated among them in conceptual models can be monitored directly or by using indicator variables. Indicators are measurable parameters (attributes) that describe specific characteristics of the species of interest or the target ecosystem; they represent the key features of the desired outcome (i.e., the management objective); they give information on the state or condition of a system element or the relationship between elements in a system.

Some indicators are direct measures of ecosystem or biological parameters of concern, while some are measures of factors that are known or believed to represent parameters of concern (i.e., surrogate or proxy indicators). Surrogate indicators may be used when the parameter of interest cannot be easily or economically measured. For example, habitat attributes may be used as surrogates for species, or one species for another, and indicators at one scale may be used to monitor biological diversity at another scale. Though the relationships are not always certain, a surrogate is expected to reflect the condition of the unmeasured parameter. And though surrogacy is commonly used in biological monitoring, validation of surrogacy assumptions is needed in most situations.

Indicators may be qualitative or quantitative. Quantitative indicators reflect a numeric relationship between certain habitat variables and species response; they are stronger and often of more value than qualitative indicators. Qualitative indicators, although they sometimes provide a more complete understanding of a relationship, are subject to observer subjectivity and are hindered by artificial categorization of the values.

Indicators may be selected to make comparisons over time (i.e., trends), among places (e.g., different ecological sites or different treatments), or with an ideal (e.g., comparison with the desired condition or a control) or a target (e.g., ecological threshold or management trigger).

Field data collection may be used to assess some indicators but some spatial indicators are assessed using routine evaluations (see section 3.1.5) such as geographic information systems (GIS) analyses or mapping exercises. In most cases GIS-based assessment is thought to be cheaper to complete, but field data, which may be used to validate GIS assessments, is often relatively high value and crucial to a sound evaluation.

Indicators are characterized by: 1) the realm that is measured (habitat, species/population, threat);


2) the part of the ecosystem that is measured (ecosystem function, structure or species);

3) the intent of the measurement (status assessment or effectiveness of action); and4) the project “element” for which progress will be measured (objective – progress

toward goal; threat – progress to reduce threats; assets – progress toward improving assets; and strategies – progress toward completing successful activities).

In addition, wildlife utilize landscapes at a variety of spatial scales, and what indicators are measured in effectiveness evaluations and what is inferred from them need to reflect those spatial scales.

(a) Some indicators will be selected to address large scale management concerns: maintenance of regional or subregional populations through a network of management tools in the region, distribution and abundance of wide-ranging or widely dispersed species, quality of habitat in a large area.

(b) Others will be selected to address concerns that occur at a medium scale: that is, including a multitude of spatial management tools (e.g., WHAs and UWRs) by watershed or management unit, such as assessing distribution and abundance of habitat, or barriers that prevent movement or use of an area.

(c) And finally, others will be selected to address concerns at a small scale: that is, spatial management tools at a stand level or smaller, addressing, for example, forest structure and condition, or habitat elements (features) required for long-term survival.

Selecting suitable and meaningful indicators is one of the most important aspects of a monitoring project. Good indicators are:

Focused Focused on answering a specific evaluation question; directly reflects and covers key aspects of the management goal

Clearly related Clearly related (correlated) to the true parameter they representEasy to interpret Easy to interpret; useful in describing the elements or relationships they

represent; and easily communicated to the target audience and stakeholders

Practical, feasible Practical, easy to measure, feasible, reliable, robust and cost-effective, maximizing information gain while minimizing time, effort and expenditure; the data be obtained at an acceptable cost and in a reasonable timeframe

Fine-scaled Only as fine-scaled as the question requires; addresses the scope of the question

Relevant Operationally relevant, addressing an issue of concern and the forest /range practice under study with the appropriate level of accuracy, precision and scale to provide results that will be used in decision making; can be related back to implementation of the tool or practice; measuring changes that can be averted by management actions

Responsive & Reliable Quickly responsive to change in a predictable way; sensitive to changes in management or stressors, reflecting well understood causal relationships; consistently responsive to actions we are concerned about; shows differences between units; shows trends over time

Precise and unambiguous Precise and unambiguous, not clouded by background processes such as


weather, climate, stochastic events or natural variation (low naturally occurring variability)

Supported by science Supported by science and widely used (rationale, methodology and analysis well documented; measurable in a standard scientifically credible manner); results have acceptable variability; peer reviewed

Integrated Integrated (provides information about multiple levels or aspects of the system, relevant at site, feature and landscape scales)

The list of key questions and potential indicators for a study is usually longer than feasible to evaluate, and a selection of the most valuable, cost-effective indicators will have to be made. Often some flexibility is required, as the indicators initially selected may prove to be less valuable as knowledge gaps are identified and new knowledge is gained, or when selected indictors fail to provide sufficient accuracy. After choosing (or developing) an indicator to answer a question, it is important to test the indicator in a pilot study to determine whether it meets the criteria of a good indicator (listed above) and provides a sound answer to the management question. Pilot studies will also indicate whether the proposed study is feasible and the data from the selected indicators will make a valuable contribution to knowledge. Subsequently, indicators should be adjusted or replaced, and/or other indicators added to ensure effective monitoring.

Most effectiveness evaluations will involve monitoring more than one indicator. In the realm of wildlife resource values, most projects will evaluate a combination of forest or range structure indicators and species indicators. Though measuring forest or grassland structure may appear more cost-effective than measuring population parameters (for example, resource constraints or logistics issues make it easier to sample elements that do not move or hide, that are not uncommon or vulnerable), measuring only habitat may fail to indicate whether providing habitat structure is sufficient to maintain productive animal populations over time. Measuring species population indicators can provide an early warning system for critical declines and identify where finer scale population monitoring is required. Measuring species or population indicators allows comparisons to habitat benchmarks and helps refine model relationships over longer time periods and larger areas. However, selection of the correct species indicator is critical – monitoring indicators for all species is usually not possible, so indicators for the appropriate informative species (perhaps some focal species or a guild of species) must be chosen to answer the important monitoring questions.

Sample evaluation questions and indicators for a plethora of objectives for ecosystem monitoring projects have been developed by the University of Michigan’s Ecosystem Management Initiative (Schueller et al. 2006), Nature Conservancy Canada (Gordon et al 2005) the US National park Service (2012), and Bunnell (2005). The reader is urged to review good indicators presented in these documents and the “questions and indicators” reports on the WRVT webpage6.

6 http://www.for.gov.bc.ca/hfp/frep/values/wildlife.htm



2.5.2 Thresholds and Expected Values

In response to a management activity, or as its status is monitored over time, an indicator value may rise above, fall below or fluctuate around some level that is significant to management. Often there are thresholds determined from previous studies or values derived for the study, above or below which management decisions and actions are triggered to ensure the survival of a species or conservation of an ecosystem. Other times the management goal is for the indicator to reach (or be stable at) a target value. In other situations, an indicator is predicted to reach some level in a certain timeframe, based on knowledge from previous studies or from benchmark or control sites. Threshold, trigger, predicted and target values should be pre-determined, based on current scientific knowledge, prior to initiating monitoring. Thresholds may be initially estimated from “a syntheses of available data, model projections of known relationships, or reasoned guesses” (Houde et al. 2005).

SIDEBAR #3 Baselines, benchmarks, controls, targets, thresholds and triggers

Several types of data, in various combinations, can be used for comparison in effectiveness evaluations: baseline, benchmark, control, target, threshold and triggers. It is necessary to determine which type of data is available and most valuable for a proposed effectiveness evaluation.

Baseline data reflect the condition of the habitat or species at a specified time, either at some time before establishment of the UWR or WHA, at time of establishment, or at some fixed time when the evaluation study begins. This information is incomplete for UWRs, ungulates, WHAs, WHFs and species at risk. Therefore, pilot studies and studies to collect baseline data are recommended prior to or as a first step when implementing effectiveness evaluation projects. Baseline data from a pilot study can be used to determine necessary sample sizes and improve field and analytical methods, and later, baseline data, if collected using the same or a comparable methodology, can be used in comparisons of subsequent measures of habitat or species biology and estimation of trends over time.

Benchmark data include measures of condition of the habitat or species in a reference condition other than the baseline value. Benchmarks can be standards or targets you are aiming for or a meaningful point of reference in time, space or condition. Comparison of effectiveness evaluation data to benchmark data can provide insight into level of risk and the need for changes in management practices.

Sometimes a benchmark is a biological threshold value, a target or a management trigger point, that, when reached would signal a need for a change in management practice. Targets are set either by regulation (or policy), by certification standards or sometimes by scientific evidence. For example, the density of wildlife trees in a management zone or the number of bats that use a roost tree in a WHA. Thresholds are usually upper or lower limits that serve as triggers or early warning signals for responsive action and might be estimated and then refined through Adaptive Management.

Control data describe a group of subjects (e.g., wildlife species) or conditions (e.g., habitat parameters) that is not exposed to any treatment (either experimental or operational) and is matched as closely as possible (i.e., all else being equal) with an experimental group of subjects or conditions. The untreated control group is used as a standard or yardstick, to detect and measure changes that may occur in the treated group due to the treatment.

Evaluators will want to compare observed conditions or trends of indicators with expected values, thresholds, triggers and targets and make predictions of indicator values expected as a result of the management practice. These predictions should be reflected in


explicit hypotheses regarding expected future conditions and trends tested in the effectiveness evaluation. Forecasts of indicator values should be as explicit as possible, ideally being quantitative estimates with confidence limits.

Quantitative models provide valuable input to forecasts of future conditions of indicator values; simpler models may need to be adjusted, for instance to include alternate management scenarios. Models and predictions for indicator values can be updated with new information gathered during monitoring.

Sections 2.5.1 and 2.5.2: Selecting indicators – in practice Indicators are measurable attributes or characteristics of a resource value that can

provide reliable information on the sustainability status or state of the resource. With expert input, and inconsideration of existing current state information, examine

the conceptual model of relationships to determine potential parameters and variables that could be measured over time to indicate achievement of FRPA-wildlife resource value objectives.

Select highest priority, feasible indicators for effectiveness evaluation study. Determine critical threshold, trigger or target values of the selected indicators. Predict indicator values as a result of management practice.

2.5.3 Developing an Evaluation Tool for Effectiveness Indicators

Section 2.5.3: In this step, we examine potential outcomes of all indicators combined to create an evaluation tool, by asking: What levels of the population or habitat condition indicators indicate the population

or habitat condition is secure, at risk, or inadequate? What levels of the threat indicators indicate extreme, unacceptable, moderate or low

levels of risk to the population, its habitat or the integrity of the WHA or UWR? What combination of condition and risk indicator values will lead to the conclusion

that the WHA or UWR is effective, at risk or ineffective at meeting management objectives?

With a number of indicators selected for monitoring and evaluation, the project team must consider how the indicators will be evaluated to determine effectiveness of the WHA or UWR. That is, how will the results of the monitoring of population and habitat condition indicators and threat (risk) indicators be combined for an overall conclusion about effectiveness?

This step requires:1) pre-defined categories of condition for each indicator of population and habitat

condition (e.g., poor, satisfactory or good condition; declining, stable or increasing trend);

2) pre-defined categories of threat for each indicator of risk (e.g., low, moderate, extreme; long-term or immediate);


3) decision-making processes or protocols (Fig. 1) for assigning overall rating of condition to the combined suite of indicators (e.g., highly, moderately or not functional) and overall rating of risk to the combined suite of risk indicators (e.g., low, long-term or immediate); and

4) a tool, such as the matrix of Table 2, for assigning effectiveness ratings to the study area based on overall ratings of condition and risk – that is, a pre-defined effectiveness rating scale that reflects all potential combinations of condition rating and risk rating.

Figure 1. Example decision-making process: Assessment process for assigning badger WHA functionality ratings (from Kinley 2009, quoting Newhouse et al. [2007:6]): “Densities of recent (occupied during year of survey) badger or ground squirrel burrows considered to be ‘high’ are to be determined regionally based on the results of baseline WHA assessments. Ground squirrel densities are to be considered only in regions where ground squirrels are the predominant prey. The density of recent badger burrows must decline by at least 30% annually (50% biennially) to be considered ‘negative’.”


Table 2. The standard wildlife resource value effectiveness ratings matrix 7 with typical functionality and risk categories and conventional management interpretation.

Condition Risk

Low Moderate High

High Effective Effective At risk

Moderate Effective At risk At risk

Low unknown Not effective

Not effective

Management interpretation: Not Effective and At Risk effectiveness ratings indicate an immediate need for action to prevent or reverse loss of viability of the WHA or UWR. Not Effective, At Risk and Unknown effectiveness ratings indicate a need for management changes. An Effective rating does not require a change in management regime or new actions aside from continued monitoring.

Assigning condition (functionality) and threat (risk) rating categories to a WHA or UWR (Figure 1 and Table 2, above) requires a priori recognition of indicator critical values that have biological or management relevance such as thresholds, targets or trigger values (see 2.5.2). Critical values (or values relative to them, such as 50% of the critical value) are often the cut-off values for condition or threat rating categories. Categories may also reflect direction of trends (favourable versus unfavourable) in condition or risk. The rating categories are defined from expert opinion or previous scientific evidence before the effectiveness monitoring begins. They are often key factors represented in preceding conceptual models.

Assigning overall rating of condition or risk to the combined suite of condition or risk indicators (step 3, above) can be complex and requires consideration by species experts. For example, an indicator may have greater importance for species survival or feasibility of related management action and therefore be weighted more heavily than other indicators. In another case, a high risk rating for one indicator may override any lesser value of other risk indicators. All possible combinations of risk and condition ratings and any situation specific interrelationships among the ratings should be considered and assigned to overall effectiveness ratings prior to implementing monitoring.

7 see the FREP wildlife resource value monitoring and evaluating framework (2009) at http://www.for.gov.bc.ca/ftp/hfp/external/!publish/frep/values/Wildlife_Framework_Paper.pdf


http://www.for.gov.bc.ca/ftp/hfp/external/!publish/frep/values/Wildlife_Framework_Paper.pdf

The effectiveness evaluation matrix (Table 2) that considers condition ratings in context of risk ratings is a tool that can be used in all evaluations of effectiveness of wildlife resource value management practices. The categories of functionality and risk may be adapted (the wording changed) to reflect different projects and hypotheses (with sufficient knowledge for more categories), but the 3x3 format is standard for the wildlife resource value effectiveness monitoring and evaluation program.

It is important to consider this final step of effectiveness evaluation before beginning data collection (a priori) in order that the study approach and sampling design provide the appropriate data to make the assessment. However, the project team should re-examine the evaluation tool and revise it (or interpretation of it) as necessary following the outcome of a pilot study or several years of monitoring (post hoc).

Section 2.5.3: The evaluation tool – in practiceTo prepare for the final step in an effectiveness evaluation, the project team must: 1. Develop a flowchart or decision-tree protocol that will be used to determine

whether the combined condition (functionality) indicators indicate the WHA or UWR is highly functional, moderately functional, or not functional (other categories could be good condition, moderate condition, and unacceptable condition).

2. Develop a flowchart or decision-tree protocol that will be used to determine whether the combined threat (risk) indicators indicate the level of risk to the WHA or UWR is nil-to-low, long-term or immediate (other categories could be low, moderate or extreme risk).

3. Determine the overall effectiveness rating from a matrix (the standard wildlife resource value program “effectiveness rating matrix”) that cross-references condition (functionality) and threat (risk) ratings to conclude the WHA or UWR is effective, at risk or not effective.

2.6 Pilot testing a protocol

During the development of an effectiveness monitoring protocol, pilot studies should be undertaken to test assumptions, field and analytical methods, and data quality, and ultimately to improve the protocol. The project leader needs to determine whether a protocol that is to be implemented is in its final form or requires pilot testing.

Pilot studies require development of a preliminary study design and sampling design (Section 3) before being implemented (Section 4); the outcome of the pilot may result in changes to the study design, the sampling design, or the data collection or analytical methods. The process of feedback for improvement is the crucial product of pilot studies.


Most projects will require one (or possibly two) season for testing study design, field methods, data forms and analyses. A pilot season will provide important feedback to improve methods and outcome of the project and could provide new or updated baseline data.

Pilot might evaluate … Possible feedback:feasibility of methods and delivery Adjust methodsvalidity of indicators Drop or add indicatorsfocus of questions Refine questionsutility of field forms Revise field formssoundness of sampling design Adjust designrigor of data, power of statistical tests Amend rigorrigor of effectiveness evaluation matrix Amend matrix

2.7 Tracking Project and Program Efficiency

Section 2.7: In this section we ask ... Is the project cost-effective? Are the project (or program) resources (funds and people) well spent? Are we getting value for the dollars and effort spent?

Alongside the development and implementation of wildlife resource value effectiveness evaluation projects, it is important to consider how to assess and track project or program efficiency. For instance is the money and effort well spent, are we getting value for the dollars spent, are we getting the best bang-for-the-buck?

Program efficiency evaluation is different from effectiveness evaluation which measures outcomes of management practice to protect the species or habitat. Program efficiency measures the success of wildlife resource value program administration, coordination and leadership in terms of costs and benefits. Information collected to track efficiency of each project contributes to the assessment of efficiency of the program and ensures continuing support for the monitoring project.

Performance measures for efficiency of effectiveness evaluation projects are similar to those of program efficiency and fall into three categories: examples of measures in each category include -

Cost effectivenesso expenditures (dollars, staff time) – totals and trendso feasibility of field logistics

Project (or program) efficacyo numbers and types of partnerships and co-operators o level of funding from other sources (partners)


o amount of co-operative data collection (sharing data or data collection with other FREP or other monitoring projects or programs)

o volume, distribution and timeliness of extension deliverables Contribution to improved management

o (trends in) public awareness of management issue and of management efforts

o changes in management expenditures with gained knowledge

Measures of project efficiency will differ depending on the stage of development of the effectiveness evaluation: that is, whether the protocol is in development, a pilot study is underway to test a protocol, or a final protocol is being implemented.

Efforts to track project efficiency should be included in project work plans and reports.

3 DESIGNING A STUDY

There are two key steps to designing a statistically rigorous study: (1) designing the overall approach referred to as the study design and (2) establishing the rules that govern what is measured, referred to as the sampling design. Careful consideration of both is crucial to successfully addressing the management questions and monitoring objectives.

The need for careful consideration of the study design cannot be overstated. Sometimes an intensive experimental design (with treatments or manipulations) may be undertaken but in many cases we must rely on simpler observational studies. However, with observational studies one is less certain that the presumed treatment or activity actually caused the observed response or result (Shaffer and Johnson 2008).

It is strongly recommended that a statistician with bio-statistical experience be included in discussion and review of the proposed study design, as well as the sampling design and data analysis plan. Statistical review early in project development will ensure the right parameters are measured, enough measurements are taken, effort is not wasted on extraneous sampling, the data are sound and credible, estimates of reliability can be calculated, the sampling scheme is appropriately rigorous for robust inferences, and the analyses will provide an answer to the management issue or question.

Decisions made during the design stage need to be transparent so the monitoring project may continue for many years even if the project leader has moved on. A clear documentation of why, what, when, who and how will sustain the program well into the future.


3.1 Designing the Approach

Section 3.1: During the Design phase, the following questions will be answered: What is the key question(s) and the hypothesis being evaluated? What is the general delivery model; who will be conducting the various aspects

(planning, field work, analysis, reporting) of the project – HQ, region, district, contract, MOE/MFR, industry?

At what geographic scale is the project to be conducted – stand, landscape, region?

At what temporal scale is the project to be conducted – at what time of year and at what interval?

At what intensity is the project to be conducted – routine, extensive, intensive? How will sites be selected for monitoring? What field methods will be used – existing standards or innovative procedures? What overlap or commonalities exist between this and other monitoring projects?

The first step of designing the study is deciding on the overall approach. What are the studies objectives? Where will it be conducted? In general, when and how will data be collected? The components of the study design are addressed in the following sections. The second step is sampling design. Specific sampling design topics are covered in Section 3.2.

3.1.1 Study designThere are many possible ways in which to approach an environmental field study (see Eberhardt and Thomas 1991 for an overview). Choosing the right approach requires careful consideration of the study objectives, the degree of control required, the desired level of inference, the effect size of interest, and the tradeoffs surrounding issues of cost and feasibility of the various approaches. Cochran (1977) describes two broad types of survey: descriptive and analytical. The objective of descriptive surveys is to obtain information about general categories of objects (e.g., the frequency of large woody debris pieces in a watershed); whereas, analytical surveys are used to make comparisons among groups within the population in order to test hypotheses (e.g., are there fewer large woody debris pieces in FSWs than in undesignated watersheds?). Hurlbert (1984) categorizes studies as either: manipulative experiments or mensurative experiments, where manipulative studies are those where the investigator has control over the factors in the study and mensurative studies are those where only passive observations are used. Eberhardt and Thomas (1991) include replication as a key requirement for improving the strength of inference and describe eight categories of environmental studies that range from the preferred approach of a controlled experiment with replication to a simple descriptive sampling approach. Schwarz (2006) provides an excellent summary of the tradeoffs between different study approaches ranging from descriptive surveys to designed experiments. Figure 2 illustrates the relationship between the degree of control and the strength of inference possible for an array of study designs. This section on study design was adapted from Wieckowski et al. 2008. Examples using WHA monitoring are described in Appendix 1.


Figure 2. Relationship between degree of control, strength of inference (and ability to determine causation), and type of study design (from Schwarz 2006).

As a first step, describe generally the overall approach and objective. Houde et al. (2005) discuss four general approaches for effectiveness monitoring: operational, experimental, design-based and model-based. Operational approaches use real-life and readily available treatments or practices applied at an operational scale under operational conditions. An experimental approach creates and compares different treatments. Design-based approaches are based on a statistical sampling design compared to model-based approaches that intensively collect information at a few representative sites (sometimes called sentinel sites) to create an ecological model which is applied to similar sites. The most appropriate approach or combination of approaches depends on the objective of the study (or monitoring question).

Also consider the kinds of comparisons, if any, that will be made: before and after treatment comparisons (such as before and after harvesting or before and after implementing a mitigative management practice), comparisons to control sites (e.g., no treatment sites), benchmark (desired condition) sites, or to important threshold or management trigger values.

3.1.2 Delivery model The overall delivery model is another component of the overall approach. How the project will be implemented is described in a general level, such as “led from Headquarters with assistance from District staff”, or “conducted by consultant under contract administered from Coast Region”. The overall delivery model also describes inter-agency, academic and/or industry partnerships and proposed or confirmed funding arrangements.


3.1.3 Spatial scale The spatial scale of a project is another component of the approach to an effectiveness monitoring project. As a general rule of thumb, the spatial scale of effectiveness monitoring should match the scale of the project objectives and management strategies and actions.

Setting the spatial scale determines how widely the results of the monitoring project will apply. Extrapolation of study results beyond the area of inference is a common problem in biological studies. We want to be sure the inference drawn from the sampling population applies to the broader target population (see section 3.2).

Some projects are at a site-specific scale, focusing on a specific individual UWR or WHA – for example, a study examines whether desired structural elements and habitat conditions exist or are being maintained within one WHA. If a species targeted for conservation occurs entirely within a WHA – does not migrate outside its boundary – or conservation actions and objectives are restricted to the WHA, then a site-specific scale is appropriate.

Projects that are at a landscape unit or watershed scale are designed to answer questions about multiple sites, such as whether current UWR boundaries fit patterns of observed animal use, whether habitat distribution and abundance within a group of WHAs is sufficient for species survival or whether additional protection is needed for other life requisites. If the distribution of a target species is not limited to a WHA or there are multiple reserves in an area, a landscape scale assessment would be best.

At a regional or sub-regional scale, or even at a provincial scale, projects are designed to determine whether UWRs, for example, are in sufficient numbers and adequately distributed to maintain healthy regional populations. If, for example, the target species is a regionally ranging species and regional scale conservation actions are to be undertaken, then a regional scale evaluation is appropriate.

3.1.4 Temporal ScaleThe temporal scale (timing and duration) of the monitoring and evaluation project is a valuable descriptive component of the overall approach. This is just a general statement – the details are more fully developed in the sampling design (see Section 3.2). Temporal scale addresses: What time of year? What re-sampling interval – weekly, monthly, every three years? Duration of monitoring? Expected completion date?

3.1.5 Intensity The overall approach should include a description of evaluation intensity, of which FREP recognizes two levels8, routine and extensive. The levels reflect the level of effort, costs and complexity, which depend on the monitoring questions, indicators and methods, resources available and circumstances. Routine evaluations generally

8 See more details of the categories of monitoring adopted by FREP at: http://www.for.gov.bc.ca/hfp/frep/rsm/index.htm


http://www.for.gov.bc.ca/hfp/frep/rsm/index.htm

involve relatively inexpensive and rapid but statistically valid measurements and visual assessments, including office-based computing exercises. Extensive evaluations are usually more focused and rigorous, with more detailed on-the-ground monitoring involving the collection of categorical data using visual estimates or relatively simple measurement; data are more quantitative and data analyses involve detailed comparisons.

Routine and extensive evaluations provide valuable information on current status, trends and implementation issues and can be applied to different types of wildlife resource value monitoring. Evaluations may be extremely detailed long-term commitments or quick assessments designed to identify key areas of concern. Levels of intensity are not sequential – projects must be initiated at the level that reflects program priorities and information need.

3.1.6 LocationLocation is an important component of the overall project approach. This is a description of the geographic location (this can be broad or specific; for example: Northeastern BC, Coast Forest Region, North Island District, ICHdw BEC subzone, or WHA-999 and its specific location); and the rationale and factors considered in selecting that location (describing what makes it special or the best site for this evaluation).

3.1.7 MethodsA brief description of field and office procedures, methods, and techniques is also part of a description of the overall project approach. This is not the details of the sampling design (see section 3.2) or the details provided to the field staff – it is a overview statement. For example, the statement might be a brief description of existing standard techniques that will be used or adapted, or innovative methods that will be developed. It might include intentions to ensure data consistency, quality assurance, or data storage and analysis standards. Similarities and differences in methods selected for this project from other monitoring projects (especially FREP monitoring projects) should be noted and briefly rationalized.

3.1.8 Overlap with other monitoring – FREP and other programsIn describing the overall project approach it is important to note any overlap and commonalities of methods, data collection, sites, etc., or cooperative efforts with other projects under FREP or any other program.

Section 3.1 Project Design – in practiceDetermine the overall project approach in terms of : Overall study design and delivery model Spatial and temporal scale Level of monitoring intensity Geographic location General description of methods Program overlap


3.2 Sampling Design

Section 3.2 A monitoring project will likely involve monitoring several variables. In this Sampling Design step, given the study design, we must examine the following questions about all the variables to develop a sound sampling design prior to project implementation - What is the target population? What variable(s) will be measured? What is the sampling unit for the measured variable(s)? Will samples be stratified or random? What sample size is required? Consider effect size. How many times a year will you sample a point? In how many years would you

like to detect a change or trend? What analytical methods will be applied, and at what levels of significance (alpha

level)? What data, if any, is available for measured variables to provide guidance on the

natural variability in measured variables? How much money and manpower is available to project?

The purpose of this section is to guide the user through the necessary steps to create a sampling design. Thompson (2004) describes two necessary components to an effective sampling design: the spatial and temporal selection of sampling units, and the measurement protocol within a given sampling unit. The first component addresses the fact that in most ecological studies we can’t sample all possible areas in all possible time periods. This sampling variability is the subject of most traditional statistical sampling books. The second component addresses the fact that in many ecological studies, especially with rare or elusive species, that the probability of detection is less than one. In other words, even if an animal or plant is present in a sampling unit, it may not be detected.

It is also necessary to consider the data analysis during the planning stage; ask yourself “What would you do with the data, if you had it?” Both spatial and temporal selection as well as detectability should be incorporated into a sampling design (Thompson 2004; Yoccoz et al. 2001; Thompson and Seber 1996) along with a discussion of the intended data analysis (Thompson and Seber 1996). The ability to create an effective sampling design depends on clearly articulated study objectives as described in Section 2.1 and 2.2. and knowing the key questions and hypotheses being evaluated. Given clear objectives, a pilot study, past research, or expert knowledge should be able to provide the information needed to work through these questions and design an effective study.

This section is organized into eight steps of developing a sampling design, adapted from recommendations by Elzinga et al. 2001 and Vesely et al. 2006.


Section3.2.1 Describe important life history characteristics3.2.2 Define the target population3.2.3 Choose an appropriate sampling unit3.2.4 Determine how should sampling units be positioned3.2.5 Determine an appropriate sample size at each step of the design3.2.6 Determine an appropriate sampling frequency and time frame; should sampling units be

permanent or temporary?3.2.7 Describe the field measurement protocol3.2.8 Describe how you intend to analyze the data

3.2.1 Life History CharacteristicsThe life history characteristics of the subject species drive the decisions made at each step of the sampling design development. A description of life history should include: what is known or unknown about the spatial and temporal distribution of the species, some indication of abundance, reproductive strategy, detectability, habitat use, and anything else related to the life history of the organism which might affect the sampling design. This step should follow easily from the conceptual model and indicator development in Section 2.4 and 2.5.

3.2.2 Target PopulationThe target population has been described as the complete collection of individuals we wish to study (Lohr 1999); the population about which information is wanted (Cochrane 1977); or the complete set of units about which we want to make inferences (Elzinga et al 2001). In order to make inferences about the entire target population, all individuals within the target population must have some chance of being selected in the sample. The sampling population is the collection of all possible sampling units that might have been chosen in a sample; the population from which the sample was taken (Lohr 1999). The sampling population should be the same as the target population, although often due to time or budget constraints the sampling population will be smaller than the target population. If this is the case then any inferences drawn from the sample only apply to the sampling population (Cochrane 1977). Extending inference beyond the sample population requires additional information and expert judgment. If the sampling population doesn’t cover the entire target population, then collecting supplemental information to describe how the two populations differ may help to understand the limitations of the inference. For example: if the target population is all fish-bearing streams in British Columbia, but the sampling population only included streams up to a certain elevation, then we shouldn’t extend inferences based on lower elevation streams to those at higher elevation. Similarly, if deep pools couldn’t be sampled due to some logistical constraint, then we shouldn’t make inference to deep pools, as they would effectively be removed from the target population.


Figure 3. Three different conceptual relationships among the target population of inference, the sampling frame, and the sample of units drawn for characterization ( excerpted from Skalski et al. 2005, pg. 364)

3.2.3 Sampling unitThe sampling unit is the actual unit of measurement. The sampling population is divided into many sampling units. The list of all possible sampling units is called the sampling frame (Lohr 1999). Individual animals or plants generally represent the sampling units for measured organismal characteristics such as: height, weight, and health. Plots or similar area delineations are generally the sampling units for population measures such as: abundance, density, or biomass. However, it may be necessary to have multiple levels of sampling units. For example, if interested in characteristics such as height or weight, individual animals seem the obvious sampling unit, but without knowing how many animals exist or their locations, you can’t actually create a sampling frame. Consequently you may first have to define sampling units as plots or transects in space, and then measure characteristics of individual animals observed/captured within the plot or transect. This is an example of a multi-stage sampling design, where the plot or transect is the primary sampling unit and the individual organism is the secondary sampling unit. Multi-stage sampling designs are very common and are discussed in more detail later in this section.


If the sampling unit is some measure of area, such as a plot or a quadrat, the next thing to consider is the size and shape of the sampling unit. Generally the aim is to choose the sampling unit that provides the most precision for the least effort. A good strategy to accomplish this is to choose the size and shape of sampling unit that will minimize the between-unit variability without adding too much sampling effort. A good idea, if feasible, is to choose a plot size large enough to ensure some observations or captures within the sample unit (i.e., avoid too many zeros). An excessive number of plots with zero individuals of interest will likely complicate associated analysis.. Elzinga et al. 2001 provide a useful summary of key considerations when choosing the size and shape of the sampling unit for ecological studies, a condensed list is provided here:

Travel and setup time vs. searching and measuring time: Evaluate the tradeoff in effort and precision between measuring fewer large plots vs. more small plots. How hard is it to find and measure the individuals of interest? Are individuals large and conspicuous like trees, or are they small and difficult to find?

Spatial distribution of the individuals in the population: Many species of plants and animals have clustered distributions. If the distribution is clustered then small plots will tend to result in frequent zeros, whereas larger plots will be more likely to intersect a cluster. The orientation of the sampling units is also important. For example, if there is a possible gradient in the measurement of interest related to some environmental covariate such as sun exposure, slope, or moisture content, then the sampling units should be oriented to cross the gradients. For example, if moisture content was an issue, between-plot variability could be reduced by laying out plots so that each plot had a similar range of moisture content. An alternative would be to stratify plots based on the environmental differences (see discussion on stratified sampling).

Edge effects: It can be difficult to determine whether an individual lies within or outside a plot boundary. Some observers may tend to include all boundary sightings, while others may ignore them all. Either way can lead to biased estimates and the sampling protocol should explicitly define a consistent rule for how to handle edge observations. Rectangular plots have the greatest edge per unit area, while circles have the least.

Abundance of target population: An initial sense of abundance is useful in choosing an appropriate sized sampling unit. The goal is to avoid either creating a sample unit so large that you will have to count thousands of individuals or so small that you end up with many empty plots.

Ease in sampling: Long, narrow quadrats are generally easiest to search (Krebs 1989, Elzinga et al. 2001). An observer can move in one direction through the entire plot with little likelihood of losing track of individuals already counted. As an example, in a recent pilot study (Lisa Tedesco pers. comm.), a long rectangular plot was found preferable to a circular plot of a similar area for measuring grass height. The rectangle plot allowed clear sighting of all of plants and measurement with little extraneous movement.


Conversely in the circular plot, it was difficult to keep track of which plant had been measured and it was far more time consuming as the observer had to shift position frequently.

Disturbance effects: It is important to consider the impact the sampling may have directly on the species of interest and on the interpretation of monitoring results. This is important for two reasons: first and obviously, it would be counterproductive to impact the species of interest (which in the case of the FREP program will often be a species at risk); secondly, sampling disturbance to sites (especially around permanent sampling sites) could over time affect the behaviour of the monitored species.

Results from pilot studies, information from other published research, or expert opinion based on biological characteristics can be used to choose the appropriate size and shape of the sampling unit according to the above considerations.

3.2.4 Positioning of sampling unitsNow that the target population, sampling frame, and sampling unit are defined, how should the actual sample be chosen? Probability sampling theory requires that each sampling unit has a known probability of being chosen and that they are selected at random (Cochrane 1977).

Standard probabilistic designs:There are three basic probabilistic sampling designs that are most commonly used: simple random sampling (SRS), systematic random sampling (SysRS), and stratified random sampling (StrRS). Simple random sampling is when a random sample of all sampling units within the sampling frame is selected, for example: drawing numbers from a hat. Systematic random sampling is when sampling units are selected at regular intervals using a randomly selected starting point, for example: reading every tenth name from the phone book, or taking a sample every ten meters. Stratified random sampling is when the sampling frame is divided into non-overlapping groups (strata) based on some characteristic such as sex or habitat type, and then a random sample is chosen from each of the strata. SRS can always be used, but this may not always be the most efficient choice. In other words, it may require a larger sample size to obtain comparable information about the population using a SRS as opposed to another more efficient sampling design. If the population of interest is randomly distributed then the SysRS approximates the SRS (Lohr 1999). If the target population changes proportional to position (e.g., samples taken upstream vs. downstream) a SysRS may be an appropriate way to ensure spatial coverage. If the target population displays regular or cyclical characteristics then a SysRS is a poor choice. A StrRS is more efficient when there is less variability within strata than between strata (Cochrane 1977; Lohr 1999). It may also be useful if estimates for individual strata are desired as well as for the entire population.

Generalized Random-Tessellation Stratified (GRTS) designs:


GRTS is a spatially-balanced probabilistic survey design developed by the US Environmental Protection Agency (EPA) under their Environmental Monitoring and Assessment Program. GRTS overcomes some of the shortcomings of both simple random sampling (which tends to “clump” sampling sites) and systematic sampling by generating an ordered, spatially balanced and unbiased set of sites that represent the population from which the sample sites will be drawn. Software required to create a GRTS design include: psurvey.design, R statistical package (both available from the US EPA’s Software Downloads for Spatial Survey Design and Analysis webpage), and ArcGis. The EPA’s Monitoring and Design team ([email protected]) will also supply customized GRTS generated sample points for selected areas upon request, as long as the client indicates or provides the necessary GIS layers for the intended spatial sample frame (e.g. stream networks, polygon boundaries etc.). The example GRTS design in Figure 4 shows a random, spatially balanced draw of a pre-selected number of sampling points within each of 5 user-defined stream order strata within a watershed hydrology network.

Stream Order

% of sample sites

1st 102nd 103rd 404th 305th 10

Figure 4. Hypothetical EPA GRTS draw of random, spatially balanced sampling points within stream order classifications for the Methow Basin in Oregon (source: Phil Larsen, EPA)

Judgment or non-probabilistic designs:Judgment samples are selected subjectively. The scientist may choose sites according to some prior belief about where individuals should be found, they may be chosen arbitrarily to be representative of the target population, or they may be chosen just for convenience. So called ‘representative reaches’ in stream surveys are an example of a judgment sample. However, without a census of the target population (e.g., stream), it is impossible to be sure that you have chosen a representative sample (e.g., reach). McDonald (2004) describes many examples where allocating some effort outside supposed core areas provided considerable improvement in their understanding of distribution and estimates of abundance for rare and elusive populations. For example, if core habitat had relatively consistent densities, while


mailto:[email protected]

http://www.epa.gov/nheerl/arm/analysispages/software.htm

marginal habitat density fluctuated with the size of the population, then ignoring the marginal habitat would mask the true health of the population. Sometimes sites that are thought to be particularly sensitive ‘sentinel sites’ are selected for monitoring, but this too is a controversial issue (Edwards 1998). Making inference from judgment samples only has lead, for example, to some famous miscalculations in predicting election results (Edwards 1998).

Non-standard probabilistic designs:There are many other variations on these basic designs that have been developed to address particular situations including: cluster sampling, adaptive sampling, and distance sampling (see Appendix 2 for other sampling designs). Any combination of these designs can be combined in a multi-stage sampling design. For example, a simple random sample of transects could be chosen from each stratum within the target population, and then a systematic random sample of plots within each transect could be selected, followed by a census of individuals within each plot. Calculating an estimate of the mean from a multi-stage sample is fairly intuitive, but the variance calculations are more complicated. The typical mistake that is made is to treat all of the observations as though they were drawn at random from the target population. In reality the secondary sampling units (plots in this case) are a sample from the transect, not the population. Increasing the number of plots within transects only helps improve the precision of the estimate of the single transect. This is one form of pseudo-replication as discussed by Hurlbert (1984).

Strategy:There is no set prescription for deciding how to position sampling units, but stratification can be a very powerful tool for improving the efficiency of a design, and so assessing whether or not stratification is appropriate is a good place to start. A series of questions combined with an understanding of the strengths and weaknesses of each of the different sampling designs should enable an appropriate allocation of sampling units within the population or strata.

a. Is a stratified design appropriate? Are there obvious groupings to the target population, such as: ownership

boundaries, habitat types, differences in density? If so, is it reasonable to think that the variability between groups would be greater than the variability within these groups?

Would it be useful to know how precise the estimates are for particular groups within the populations?

b. How should sampling units be positioned within the population (or strata)? How big is the total target population? How much time does it take to move between sampling locations? Are the objects distributed at random, uniformly (typical of territorial

animals), or in clusters?


Are there any regular features in the landscape, such as: ridges, fence lines, roads, riffle/pool sequences... that might be related to the response variable of interest?

Are there any known gradients in the target population, such as: upstream vs. downstream, low moisture to high moisture, elevation etc…?

How time consuming is the actual measurement process within each sampling unit?

3.2.5 Sample size and statistical powerThe next step is to determine the appropriate number of samples to take at each level of the sampling design to ensure sufficient precision for the study objective and optimize the allocation of monitoring effort and cost (too many samples waste dollars and effort, too few samples can’t answer the questions of interest).

Sample size calculators use preliminary estimates of variability to help you assess the trade-off between effort and precision. There are many free programs (e.g., Program TRENDS or MONITOR) available to help determine the best sample size for your project and they all function in essentially the same way.

In general sample size calculations require: estimates of variability within and between sampling units at each stage of the

design (preliminary estimates of variability may be obtained through pilot studies, similar studies conducted elsewhere, or theoretical behaviour of the system);

the desired level of precision (alpha and beta levels – see below re: power); cost of sampling at each stage of the design; the significance test of interest (i.e. the difference between two groups or a

significant trend over time); and knowledge about the distribution of the data of interest.

There is a danger in using a generic sample size calculator without understanding the underlying assumptions of that calculator. Sample size calculations depend on the study design (e.g. simple, stratified, multi-stage etc…), the distribution of the data, the study objective (e.g. comparing two groups vs. assessing trends). The most useful calculators include some measure of cost/effort in a sample size calculator in order to optimize the precision against cost trade-offs. Since the calculations are generally not difficult, it is probably a good idea to have a statistician develop a sample size calculator for your project, especially if the study/sampling design is unusual.

In statistics, power refers to the ability to detect an effect (i.e., a significance test of interest such as difference in distribution, shift in location, trend over time etc…). Power analyses are based on the same concepts as sample size calculations, but are used to determine the size of that effect. The terms “alpha level” and “beta level” are often used to describe the two possible errors made in analysis of data: alpha level refers to the probability of not detecting an effect given that it exists (false negative);


and beta level refers to the probability of detecting an effect when it doesn’t exist (false positive). Power is defined as 1-beta, or the probability of detecting an effect given that an effect does exist. In order to calculate the power, the kind of statistical test that will be performed (e.g., one- vs. two-tailed t-test) must be known. Typically statistical power is calculated for a range of sample sizes where increasing sample size results in increasing precision and therefore power. It is useful to display power as it changes with sample size and cost/effort.

3.2.6 Sampling frequencyShould the samples be collected daily, monthly, annually, every 10 years…? The appropriate sampling frequency will differ by study objective and species. Is the study interested in a one-time estimate of status, or is the objective to monitor the species or community over time? Often in ecological studies the temporal scale of interest is annual (i.e. annual estimates of abundance, health, survival etc…). These estimates can also be combined into multi-year studies that allow for estimates of trends over time and comparisons between years. There are a number of interesting and challenging statistical design questions that arise depending on the temporal scale of interest.

Deciding when to sample within a year:When and how often should sampling be done within a year in order to obtain annual estimates? This depends on the life history of the species of interest and the target population. Generally speaking the best time to sample the population is when the probability of detection is the greatest, either due to behaviour, colour (i.e. mating colouration or vegetation colouration), time of day, or season. Questions to consider:

Does the species migrate or hibernate? Does the behaviour change with season? For example, do they become

more or less conspicuous during mating season? Is the species nocturnal or diurnal? Does the species only live for part of a year (plant or animal)? Are there any major events within the year that could change the response

variable of interest? o Hunting seasono Significant overwinter mortalityo Grazing, if interested in grass height

If measuring an important habitat feature like cover availability that might change for a species with season, is your question of interest cover availability during the best conditions, worst conditions, both, or only when the species of interest is present?

What period of time within the year are you interested in making inference about?

Long-term monitoring programs:Sampling designs spanning multiple years are important when trying to consider changes to individuals or populations over time. There are many conflicting opinions


about how sampling designs over time should be implemented. A major discussion point is whether permanent (long-term) or temporary sites should be selected. The typical response is: “use permanent sites for trend detection and temporary sites for status assessment”. In reality it is rarely that simple. There are advantages and disadvantages to each approach and researchers differ in their opinions of which approach to take. A brief summary of the advantages and disadvantages to each of these approaches is described here to provide some insight into the debate.

Temporary sites:Advantages: good procedure for locating rare items (McDonald 2003) can use to estimate ‘net change’ (McDonald 2003) better strategy for estimating the average over all occasions (Cochrane 1977) removes the cost of permanently marking sites (Elzinga et al. 2001) can automatically account for changes in population composition (McDonald

2003) no conditioning impacts

Disadvantages: lower efficiency method for net change (McDonald 2003) does not allow estimation of individual change (McDonald 2003)

Permanent sites:Advantages: useful when there is a high degree of correlation between sampling units from

one period to the next, such as might occur with long lived plants or long lived and sedentary animals (Elzinga et al. 2001).

planning and survey design work are minimized (McDonald 2003) well suited to estimate gross change and other components of individual

change (McDonald 2003) most powerful for detecting linear trend (Urquhart and Kincaid 1999)

Disadvantages: risk that simply due to chance, the sites selected will be the ones that change.

A new sample each year would have a better chance of picking up a true change, but the sample size needed in each time period would be greater (Vesely et al. 2006)

only effective if the sampling effort is sufficient to represent the range of conditions across the area of inference (Vesely et al. 2006); this approach can’t account for changes in population composition over time (McDonald 2003)

can be costly to permanently mark sites (Elzinga et al.) serious potential for impacting the site in such a way that affects the measured

variable through repeated years of sampling (Elzinga et al. 2001; Buckland et al. 1993; McDonald 2003)

permanent markers may provide a perch for raptors or songbirds


vegetation may be trampled habitat or individuals may be disturbed

An interesting finding by McDonald (2003) was that the term ‘trend’ is used in different ways by different researchers; perhaps this is the source of some of the controversy around ‘permanent vs. temporary’ sites. For example, what scale is the change of interest: reach, stream, watershed, province…? McDonald (2003) provides several useful definitions that may be helpful in clarifying the study objective and hence determining the best design:

Net change: measurement of total change in a parameter arising from all sources change in mean or total response individual change can happen without causing net change (as fish move from

one stream segment to another), so individual stream segments could experience a trend while the overall population of the watershed does not.

Individual change: change experienced by an individual or particular member of the population, this can be further divided into three categories:

gross change: change in response of a particular population unit (e.g. change in pH of a particular lake)

average gross change: if all rivers in a collection of rivers have higher levels of suspended sediment (so the same change occurs to many individual units)

instability: variance of responses from individual population units

Another option which has been popularized recently, although the idea has been discussed by many including: Jessen (1942) and Cochrane (1977), is to use a design with some combination of repeated visits and new sites. These designs have been referred to by many names: revisit, repeated, rotating, split-panel, replacement, etc. McDonald (2003) provides an excellent review of these designs and develops consistent terminology. These designs provide a compromise between temporary and permanent sites that allow the estimation of both status and trends. McDonald (2003) found that “consensus opinion among the reviewed articles appeared to be that some sort of split-panel design had the best chance of satisfying the sometimes competing objectives inherent in many environmental monitoring projects”. The following definitions from McDonald (2003) are helpful:

Panel: A group of population units that are always all sampled during the same sampling occasion or time period. Note that a population unit may be a member of more than one panel under this definition.

Revisit design: The rotation of sampling effort among survey panels; the pattern of visits to the panel; the plan by which population units are visited and sampled through time.

Membership design: The method used to populate the survey panels; the way in which units of the population become members of a panel.


Implementing and analyzing these designs can be difficult. The analysis is complicated and consistent funding is required to avoid missing data. While these designs in general are recommended, the specific frequency of return visits and the sample sizes required depend on the specific study and the life history of the subject. Generally it is recommended that if trend is the primary objective then >50% of sampling effort should be allocated to the revisited sites and if both status and trends are important then the sampling effort should be allocated equally to the revisited and new panels (McDonald 2003). More research is needed to address the optimal allocation of sampling effort among panels (McDonald 2003).

3.2.7 Field methodsField methods are the actual methods used to collect data within a given sampling unit. Existing established methods or protocols should be used first where possible (Vesely et al. 2006) – a number of peer-reviewed and tested field protocols are available at the BC Resources Inventory Standards Committee (RISC) website 9.

Pollack et al. (2004) recommend that field methods be designed to minimize detection error. Questions that arise about detection include: Are field methods appropriate for the species? Are individuals hard to detect because they are rare or because the method isn’t efficient? Is the animal of interest ‘available’ to the sampling method? Thompson (2004) describes diverse examples where much of the population may not be available to be counted:

Aerial surveys of marine mammals where some individuals will be underwater; Surface counts of ants or other insects where only the foraging component of the

population is available; Salamanders or frogs where a portion of the population is underground,

underwater or otherwise not visible; Bird point counts where not all birds may sing; Terrestrial mammals in areas with dense vegetation (e.g., elk, moose).

Where detectability is not perfect (or reasonably close to perfect) it must be accounted for. There are a number of techniques available to address this issue including: capture-recapture, where probability of capture is evaluated; distance sampling, based on the idea that detection is a function of distance; and sightability models, where observable covariates for factors that influence detectability are used.

Field methods must be documented and made available to anyone who wishes to replicate the sampling. Variations from a standard method or protocol must be documented and rationalized. Any changes made to sampling method during the study must be recorded with the data so data analysis will give a true picture of the sampled population.

3.2.8 Planning the data analysisFinally, one should consider a priori how the data will be analyzed and what will be reported. If the study objectives have not been clearly defined, then this step will be 9 Resources Inventory Standards Committee (RISC) website: http://www.ilmb.gov.bc.ca/risc/about.htm


http://www.ilmb.gov.bc.ca/risc/about.htm

impossible. Use past research, pilot data, or expert opinion to describe the proposed approach. Once sampling has been completed, describe the observed data for the approach taken (it may not be exactly what was proposed). The following questions and examples are intended to provide an idea of what data analysis should consider or might address, but are by no means an exhaustive list of possible analytical approaches.

What does the data look like? o Are the data a direct measure or an index?o Describe the response or index of interest

abundance density survival diversity index health index

How are the data distributed? o continuous (e.g., height, weight)o categorical (e.g., colour, sex)o strictly positive (e.g., count data)o bounded between 0 and 100 (e.g., percent cover)o binomial (e.g., each observation can only take one of two values)

What exactly is the statistic of interest?o mean, median, maximum, minimum, rolling mean, range, variance

What questions are of interest?o Does the response differ by year, group, or location?o Is there a relationship between environmental covariates and the response of

interest?o Does the response exceed a threshold or target?

What statistical tests are appropriate?o Parametric tests such as a t-test, require an assumption about the distribution of

the data (i.e., that it is “Normal”)o Non-parametric tests such as the Mann-Whitney test may be appropriate if the

distribution is not obviouso Regression analyses describe the relationship between environmental covariates

and the response of interest o Multivariate tests, such as the multi-response permutation procedure (MRPP) may

be appropriate for community level questions where there are multiple responses (i.e., the abundance of several species)

How to select significance levels and confidence intervals?


o Selecting the significance level of statistical tests should be done before the data are collected (the level of significance will be used to determine sample size – refer to section 3.2.5) and confirmed before the data are analysed.

o The significance level (or “alpha level”) is the probability of rejecting the null hypothesis 10 when in fact it is true – that is, concluding that there is a difference between two values when in fact there is no difference. This is known as a “Type I” error.

o Significance levels are often 0.01 or 0.05 - the former represents a more rigorous test than the latter and provides more certainty about your conclusion, and is employed where errors in rejecting the null hypothesis have serious implications to humans or the environment.

o Confidence intervals describe the amount of uncertainty associated with a sample estimate of a population parameter, and are calculated from the significance level, the sample statistic itself, and the margin of error (the range of values above and below the sample statistic). For example, we might find our sample population did not differ from the true population value by more than, say, 5 percent (the margin of error) 90 percent of the time (the significance [alpha] level).

Check the assumptionso Any analysis should include a discussion of the assumptions made about the data

or the outcome of the protocol: what the assumptions were, whether or not they were met, and any weaknesses in the assumptions.

3.2 Sampling Design – in practiceDetermine and describe the sampling design by: describing life history of the target species defining the target population choosing appropriate sampling unit(s) determining how sampling unit(s) should be chosen determining sampling effort (sample size), frequency and permanence describing the details of the field methods describing intended/proposed data analysis

10 The null hypothesis states that there is no significant difference between the results obtained and the results expected: for example, there is no difference between these two values; or this value is greater than (or less than) that value. A null hypothesis is either “rejected” by statistical testing, or “not rejected” – a null hypothesis is never “accepted” by statistical testing because the testing is based on probabilities.


4 IMPLEMENTING A PROJECT

If you have...1. determined where your project is in the planning and design process (What

has already been done? Is an existing protocol? Is pilot testing needed?)2. determined the need, and current knowledge, and developed the conceptual

model and effectiveness indicators to be monitored (Section 2), and 3. established a study design with a statistically rigorous sampling design

appropriate to where your project is in the development process (Section 3) ...then you are ready to implement your project.

4.1 Confirm Agency Support

A project charter should be developed for approval by the appropriate supervisor or manager. The charter should indicate how the project is aligned with program priorities. The project charter should include a title, objectives, project lead and team, an estimated budget or resources required and assessment of priority of the project.

4.2 Prepare a Monitoring Plan

A Monitoring Plan should be prepared for each WHA or UWR effectiveness evaluation project before the project gets underway. Because effectiveness monitoring will continue over some period of time, the plan must include proposed activities, timelines and deliverables for the duration of that period along with full details of the work planned for the first (or current) year. Monitoring Plans should be updated annually as the project progresses from pilot to full evaluation project, documenting obstacles and delays, results of pilot tests and decisions such as amending data collection or analytical methods. Annual updates from a stable project (one in which no amendments are made to methods) could be included in an annual Progress Report.

The Monitoring Plan should include: Identify management objectives/questions to be addressed, description of indicators and parameters to be measured, any thresholds or management triggers that will be used, study design, sampling design, description of project efficiency indicators to be measured, standard data collection methods or description of those being developed, equipment and training requirements, quality assurance framework and quality control measures proposed analytical methods and desired level of statistical significance (where

appropriate), project milestones, timelines, and deliverables,


plans for reporting and extension.

4.3 Data Management and Quality Assurance

FREP has adopted a quality assurance framework11 that establishes a process of checks and balances to ensure all efforts are made to meet data security and quality standards, from point of collection to final storage. Although data collected in wildlife resource value projects are not currently integrated into the FREP information management system, all wildlife resource value projects should adhere as much as possible to FREP’s quality assurance and quality control commitments.

To meet the FREP commitment to quality management, wildlife resource value projects should adopt a process than ensures data collected is copied, transferred, stored, etc., such that (i) integrity of raw data is maintained, (ii) missing or erroneous data is addressed, (iii) data are entered, checked, archived and analysed on a regular annual cycle relevant to management, reporting and further study, and (iv) details of all results of all summaries and statistical analyses are available for review and archived.

5 REPORTING

Knowledge gained from monitoring may range from procedural improvements to data collection and designs thru to furthering our knowledge of the complexity of natural systems. The purpose of reporting is to share this knowledge with those responsible for making resource management decisions, those interested or affected by this knowledge and the general public.

Reporting out provides material for discussion and recommendations for changes to management practices and monitoring protocols. The wildlife resource value effectiveness evaluation program is one step in an adaptive management cycle whereby decision makers are informed of new science, recommendations for change can be made and management can be improved.

5.1 Report Templates

The FREP Communications Strategy 12 and the FREP Reporting Guidelines 13 provide overall direction for communications and standard layout and procedures for various types of FREP reports and publications.

There are a number of types of reports that could stem from effectiveness evaluation projects. Background and planning reports will come first. Results of an effectiveness 11 See http://www.for.gov.bc.ca/HFP/frep/qmgmt/index.htm12 http://www.for.gov.bc.ca/ftp/hfp/external/!publish/frep/archived/PM-FREP_Communications_Plan.pdf13 http://www.for.gov.bc.ca/ftp/hfp/external/!publish/frep/qmgmt/QCProtocol4-ElectronicForms-Dec9-2008-MASTER-BLANKFORM.pdf


http://www.for.gov.bc.ca/ftp/hfp/external/!publish/frep/qmgmt/QCProtocol4-ElectronicForms-Dec9-2008-MASTER-BLANKFORM.pdf

http://www.for.gov.bc.ca/ftp/hfp/external/!publish/frep/qmgmt/QCProtocol4-ElectronicForms-Dec9-2008-MASTER-BLANKFORM.pdf

http://www.for.gov.bc.ca/ftp/hfp/external/!publish/frep/archived/PM-FREP_Communications_Plan.pdf

http://www.for.gov.bc.ca/HFP/frep/qmgmt/index.htm

monitoring project in its early stages of development will include an assessment of the method(s) used to assess the effectiveness, in addition to the actual evaluation the effectiveness of the WHA or UWR to protect habitat for a species. Final reports will summarize the effectiveness evaluation and make recommendations for improvements.

5.2 Input to Decision-Making and Resource ManagementIt is very important that an effectiveness evaluation program and project results connect to resource management planning or practices; if statutory decisions are to be made to change a practice then clear documentation of the data used and the information being brought forward must be documented for evaluation – because the results will guide decisions about resource management planning and practice.

Improvements to resource management might be made informally through decisions to change management practices (e.g., the size of a specific WHA) or formally through government decisions to amend policy or regulation as necessary (e.g., provincial policy that defines WHA size for a species) . The feedback of project results to land managers, industry and government staff and/or executive will serve as the crucial “closing-the-loop” step in a cycle of adaptive management and continuous improvement.

6 LITERATURE CITED

Buckland, S.T., Anderson, D.R., Burnham, K.P. and Laake, J.L. 1993. Distance Sampling: Estimating Abundance of Biological Populations. Chapman and Hall, London. 446pp. http://www.colostate.edu/Dept/coopunit/download.html

Bunnell, F. 2005. Biodiversity Monitoring Guideline: Terrestrial Biological and Physical Monitoring and Aquatic Biological and Physical Monitoring. Forest Investment Account (FIA) Activity Guidance Document. 21 pp. Available at: http://www.env.gov.bc.ca/fia/documents/monitoring_guidelines.pdf

Cochran, William Gemmell. 1977. Sampling Techniques. John Wiley & Sons, Inc., New York, NY.

Doran, G. T. (1981). There's a S.M.A.R.T. way to write management's goals and objectives. Management Review, Volume 70, Issue 11(AMA FORUM), pp. 35–36

Edwards, D.: 1998. Issues and themes for natural resources trend and change detection. Ecol. Applic. 8(2): 323–325.

Elzinga, Caryl L., Daniel W. Salzer, John W. Willoughby, and James P. Gibbs. 2001. Monitoring plant and animal populations. Blackwell Science, Inc., Malden, Massachusetts.

Gordon, D.R., J.D. Parrish, D.W. Salzer, T.H. Tear, and B. Pace-Aldana. 2005. The Nature Conservancy’s Approach to Measuring Biodiversity Status and the Effectiveness of Conservation Strategies. Ch. 3 In: Principles of Conservation Biology, 3rd Edition. 2005. Groom, M.J., G.K. Meffe, and R.C. Carroll, eds. Sinauer Press: Sunderland, MA.


http://www.env.gov.bc.ca/fia/documents/monitoring_guidelines.pdf

http://www.colostate.edu/Dept/coopunit/download.html

Houde, I., Bunnell, F.L., and S. Leech. Assessing success at achieving biodiversity objectives in managed forests. BC J. Ecosys. and Manage. 6(2):17-28.

Hurlbert, S. H. 1984. Pseudoreplication and the design of ecological field experiments. Ecological Monographs 54:187-211.

Jessen, R. J. 1942. Statistical investigation of a sample survey for obtaining farm facts. Iowa Agric. Exper. Station Res. Bull. 304: 54–59.

Kinley, T. A. 2009. Effectiveness monitoring of badger wildlife habitat areas: summary of current areas and recommendations for developing and applying protocols. B.C. Min. For. and Range, and B.C. Min. Envir., Victoria, B.C. Forest and Range Evaluation Program report.

Krebs, Charles J.1989. Ecological Methodology. Harper Collins Publishers Inc., New York, NY.

Lint, J., B. Noon, R. Anthony, E. Forsman, M. Raphael, M. Collopy and E. Starkey. 1999. Northern spotted owl effectiveness monitoring plan for the Northwest Forest Plan. Gen. Tech. Rep. PNW-GTR-440. USDA Forest Service, Pacific Northwest Research Station. Portland, Oregon.

Lohr, Sharon L. 1999. Sampling: Design and Analysis. Brooks/Cole Publishing Company. Pacific Grove, CA.

Marcot, B.G., R.S. Holthausen, M.G. Raphael, M.M. Rowland, and M.J. Wisdom. 2001. Using Bayesian belief networks to evaluate fish and wildlife population viability under land management alternatives from an environmental impact statement. For. Ecol. and Manage. 153:29–42.

McDonald, Lyman L. 2004. Sampling Rare Populations. pp.11-42 in William L. Thompson ed. Sampling Rare or Elusive Species. Island Press. Washington, DC.

McDonald, T.L. (2003) "Review of Environmental Monitoring Methods: Survey Designs", Environmental Monitoring and Assessment, v. 85, p.277-292.

Meyer, Paul J (2003). "What would you do if you knew you couldn’t fail? Creating S.M.A.R.T. Goals". Attitude Is Everything: If You Want to Succeed Above and Beyond. Meyer Resource Group, Inc.

National Park Service. 2012. Guidance for designing an integrated monitoring program. Natural Resource Report NPS/NRSS/NRR—545. National Park Service, Fort Collins, Colorado. Available at: http://science.nature.nps.gov/im/monitor/guidance.cfm

Newhouse, N. J., T. A. Kinley, C. Hoodicoff, and H. Page. 2007. Wildlife Habitat Area effectiveness evaluations. Protocol for monitoring the effectiveness of badger wildlife habitat areas. Version 1.4. Prepared for Ministry of Forests and Range, Victoria, BC.

Noon, B. R. 2003. Conceptual issues in monitoring ecological resources. Chapter 2 (pp. 27 – 71) in D. E. Busch and J. C. Trexler (editors), Monitoring Ecosystems. Interdisciplinary Approaches for Evaluating Ecoregional Initiatives. Island Press, Washington.

Nyberg, B. 2010. Conceptual models for wildlife effectiveness evaluations: a recommended approach. B.C. Min. For. Ran., For. Prac. Br., Victoria, B.C. FREP Ser. 24. Available at: http://www.for.gov.bc.ca/hfp/frep/publications/reports.htm#rep24


http://www.for.gov.bc.ca/hfp/frep/publications/reports.htm#rep24

http://science.nature.nps.gov/im/monitor/guidance.cfm

Pollack, Kenneth H., Helene Marsh, Larissa L. Bailey, George L. Farnsworth, Theodore R. Simons, and Mathew W. Alldredge. 2004. Separating Components of Detection Probability in Abundance Estimation: An Overview with Diverse Examples. pp.43-58 in William L.Thompson ed. Sampling Rare or Elusive Species. Island Press. Washington, DC.

Schueller, S.K., S.L. Yaffee, S.J. Higgs, K. Mogelgaard and E.A. DeMattia. 2006. Evaluation Sourcebook: measures of progress for ecosystem and community-based projects. Ecosystem Management Initiative University of Michigan, 214 pp. Available at: http://www.snre.umich.edu/ecomgt/evaluation/planning.htm

Shaffer, T. L. and D. H. Johnson. 2008. Ways of learning: observational studies versus experiments. J. Wildl. Manage. 72(1): 4-13.

Skalski, John R., Kristen E. Ryding, and Joshua J. Millspaugh. 2005. Wildlife Demography: analysis of sex, age, and count data. Elsevier Academic Press. SanDiego, CA.

Thompson, Steven K. and George A. F. Seber. 1996. Adaptive Sampling. John Wiley & Sons Inc. New York, NY.

Thompson, William L. 2004. Sampling Rare or Elusive Species. Island Press. Washington, DC.

Urquhart, N. Scott and Tomas M. Kincaid. 1999. Designs for detecting Trend from Repeated Surveys of Ecological Resources. Journal of Agricultural, Biological, and Environmental Statistics. 4:404-414.

Vesely, D.; McComb, B.C.; Vojta, C.D.;Suring, L.H.; Halaj, J.; Holthausen, R.S.; Zuckerberg, B.; Manley, P.M. 2006. Development of Protocols To Inventory or Monitor Wildlife, Fish, or Rare Plants. Gen. Tech. Report WO-72. USDA, Forest Service, Washington, DC. 100p.

Wieckowski, K., D. Pickard, M. Porter, D. Robinson, D. Marmorek, and C. Schwarz. 2008. A conceptual framework for monitoring Fisheries Sensitive Watersheds (FSW). Report prepared by ESSA Technologies Ltd. for BC Min. Environ., Victoria, BC. 61 p.

Yoccoz, Nigel G., James D. Nichols and Thierry Boulinier. 2001. Monitoring of biological diversity in space and time. Trends in Ecology and Evolution 16:446:453.

U.S. Environmental Protection Agency. 2006. Guidance on systematic planning using the data quality objectives process. EPA report QA/G-4. 111 pp. Available at: http://www.epa.gov/quality/qa_docs.html


http://www.epa.gov/quality/qa_docs.html

http://www.snre.umich.edu/ecomgt/evaluation/planning.htm

Appendix 1. Example study designs

Based on definitions from Schwarz (2006), specific examples of each study design are provided using WHA monitoring for examples.

Descriptive study

A WHA is selected and an indicator (s) (e.g., road density) is measured. The information collected is only relevant to the WHA sampled.

Observational study

A non-designated area and a WHA are selected; selected indicators are measured in both. Comparisons between the two areas can be made, but the results are only applicable to the areas sampled. Using this approach, it is not possible to conclude whether any observed differences are representative of the differences between the areas. Descriptive and observational studies involve non-randomly selected sampling units; as a result the information obtained is limited to the sites actually observed.

Analytical survey

A random sample of WHAs of two or more categories is selected and road density is measured in each. An estimate of the mean road density with known precision can be obtained for each category. The estimates from the categories can be compared; however, it is possible that another unknown factor (besides WHA designation) is actually responsible for the difference.

Impact and control-impact surveys

The goal of this approach is to assess the impact of some change, in this case the designation of a WHA. A variety of impact designs with increasing levels of effort and increasing degrees of inference exist. Mellina and Hinch (1995) provide a summary of different impact designs and describe how each might be used to assess watershed restoration. Schwarz (2006) and Underwood (1994) provide a good description along with examples for a range of impact studies, as well as evaluating their respective strengths and weaknesses. The simplest impact studies look at a single location before and after some event. Obtaining multiple observations before and after an event improves the ability to determine if an observed change is ‘real’ by taking into account the natural year to year variability. Since obtaining ‘before’ samples is often difficult, it may be possible to get variance estimates by randomly sampling from similar but undisturbed habitats (Underwood 1994). This approach can be considerably improved by adding a control site, where the control site is similar to the treatment site with respect to general characteristics (e.g., region, annual precipitation, size, etc.). This approach is termed a ‘Before After Control Impact’ (BACI) design. BACI designs are intended to address the question of whether a particular action has resulted in a change at the treatment/impact site relative to the control site, while simultaneously adjusting for extraneous co-variables


that might be similarly affecting both impact and control areas. In most cases, the use of controls greatly increases the power of detecting treatment/impact effects; however, poorly chosen control sites can decrease the power of detecting an effect (Korman and Higgins 1997; Roni et al. 2003). For example, a lack of randomization in assigning impact sites prevents us from inferring whether the impact will occur elsewhere. Alternatively, if there is only a single impact/control pair, how do we know that the results are not just a consequence of the choice of sites?

The example in Figure5 illustrates the value of including a control site for assessing the effects of an impact/treatment on populations which are highly variable naturally over time. In this hypothetical example the ability to detect improved salmon parr to smolt survival after a habitat restoration treatment would not be possible without a control (Hayden Creek), due to high variability in survival (5A). Evaluation of the average annual difference between fish survival in the treatment vs. the control stream (5B), however, indicates that survival in the treatment stream is much greater relative to the control in the years subsequent to the restoration action.

-0.04

-0.02

0.00

0.02

0.04

0.06

0.08

0.10

1990 1995 2000 2005 2010 2015 2020 2025 2030

Year

trea

tmen

t - c

ontr

ol s

urvi

val

D(PRE)

D(POST)

0.00

0.10

0.20

0.30

0.40

0.50

0.60

0.70

1990 1995 2000 2005 2010 2015 2020 2025 2030

Year

parr

to s

mol

t sur

viva

l

Mainstem Hayden

Figure 5. Value of BACI design for inferences - A) Hypothetical time series of mainstem river and Hayden Creek (control area) parr-to-smolt survival rates, with habitat restoration actions assumed to happen simultaneously in 2015 (dark arrow) in the mainstem river. Time series for


both impact and control streams track erratically over time. B) Hypothesized difference between mainstem river and Hayden Creek parr-to-smolt survival rates over the time series. D(PRE) and D(POST) represent average survival difference pre- and post-impact in 2015 in the mainstem river. Difference between control and impact streams shows much greater parr-smolt survival in the treatment stream vs. the control, indicating a benefit of the restoration action over time that would not have been apparent given annual variation without a control system for comparison (source CSMEP 2006).

BACI designs can be grouped into three types: BACI, BACIP, and MBACI (Downes et al. 2002). In simple BACI designs, measurement are made before an impact at control and treatment locations and then after the impact. However paired measurements through time pre- and post impact are better able to avoid spurious results (Green 1979, Stewart-Oaten et al. 1986). Downes et al. (2002) refers to this as BACIP (P for paired measurement through time). In BACIP paired measurements are taken in both the impact and control sites at multiple random times pre- and post- impact. Keough and Mapstone (1995) further extended the BACI design to contain multiple controls and if possible multiple impact sites, referred to as MBACI (Downes et al. 2002). MBACI designs have been developed to address questions about the impacts of an action across broader scales. Multiple treatment and control locations are chosen randomly from a group of potential locations, thereby providing the means to extrapolate to a larger area. If it is not possible to randomly assign treatments and controls, but the same pattern is observed in multiple pairs, it is reasonable to assign a causal relationship (Schwarz 2006). The MBACI design compares a fixed period of time before the manipulation to (in ideal situations) a similar period of time after the manipulation.

Designed experiments

In a designed experiment, the investigator has control over the treatment and can randomly assign experimental units to treatments. The degree of control the investigator has on a study affects the ability to show causation. The ability to make inference to other sampling units depends on random selection of samples or assignment of treatments.

There are many good text books available on the subject of designing experiments including: Schwarz 2006, these online course notes geared to the environmental scientist and are probably the best place to start; Montgomery 1997, is a solid introductory book which is probably more than enough for most environmental studies; Box et al.1978, is a traditional reference; Wu and Hamada 2000, is a more recent and extensive reference.

ReferencesBox, G., E.P., W.G. Hunter, and J.S. Hunter. 1978. Statistics for Experimenters An introduction to

Design, Data Analysis, and Model Building. John Wiley and Sons Inc, Toronto.

Downes, B.J., L.A. Barmuta, P.G. Fairweather, D.P. Faith, M.J. Keough, P.S. Lake, B.D. Mapstone and G.P. Quinn. 2002. Monitoring ecological impacts: concepts and practice in flowing waters. Cambridge University Press, Cambridge, UK.

Green, B.F. 1979. Sampling design and statistical methods for environmental biologist. Wiley Interscience, New York, New York, USA.


Keough, M.J. and B.D. Mapstone. 1995. Protocols for designing marine ecological monitoring programs associated with BEK mills. Technical Report No. 11. CSIRO, Canberra, ACT.

Korman, J. and P.S. Higgins. 1997. Utility of escapement time series data for monitoring the response of salmon populations to habitat alteration. Can.J. Fish. and Aquatic Sci. 54:2058-2097

Mellina, E., and S.G. Hinch. 1995. Overview of Large-scale Ecological Experimental Designs and Recommendations for the British Columbia Watershed Restoration Program. Province of B.C. , Min. Environ., Lands and Parks and Min. Forests. Watershed Restoration Management Rep. No. 1.

Montgomery, D.C. 1997. Design and Analysis of Experiments. 4th edn. John Wiley and Sons, New York.

Roni, P., T.J. Beechie, R.E. Bilby, F.E. Leonetti, M.M. Pollock and G.P. Pess. 2003. A review of stream restoration techniques and a hierarchical strategy for prioritizing restoration in the Pacific Northwest watersheds. North Amer. J. Fish. Manage. 22:1-20.

Schwarz, C. 2006. Course notes for beginning and intermediate statistics. Available at: http://www.stat.sfu.ca/~cschwarz/CourseNotes.html. Accessed on: March 20, 2008.

Stewart-Oaten, A., W.W. Murdoch and K.R. Parker. 1986. Environmental impact assessment: “pseudoreplication” in time? Ecology 67:929–940.

Underwood, A.J. 1994. On Beyond BACI: Sampling Designs that Might Reliably Detect Environmental Disturbances. Ecol. Applic. 4:3-15.

Wu, J. and M. Hamada. 2000. Experiments Planning, Analysis, and Parameter Design Optimization.John Wiley and Sons Inc, Toronto.


http://www.stat.sfu.ca/~cschwarz/CourseNotes.html

Appendix 2. Examples of sampling designsCluster sampling

What: A random sample of clusters is selected and a census of all sampling units within the cluster is taken (Figure 7). For example: if we are interested in the average number of bicycles per household, we could select a sample of city blocks and then visit every household within the block. It is often relatively cheap to collect more information once at a site. This improves the estimate of the cluster (city block in this case) but may only have limited improvement to the overall precision. This is a special case of the two-stage sample where the second stage is a census rather than a sample.

When: This kind of strategy is useful when the cost of moving between sites is expensive relative to the cost of obtaining many more samples within a given site. Given information about the cost of moving between clusters and preliminary estimates of variability within and between clusters we can determine if this is a suitable approach.

Figure 6. This figure illustrates a cluster sample (from Lohr 1999, p133).

Multistage sampling

What: Similar to cluster sampling except that rather than a census of all sampling units within the primary sampling unit (psu), a secondary sample (e.g. SRS, SysRS, stratified…) is taken. For example: if a random sample of transects is taken from the population of possible transects (psu’s), then in a cluster sample we’d take a census of each transect but in a multi-stage sample we might take a systematic sample every 10m along the transect –secondary sampling units (ssu’s). In Figure 6, this would be illustrated by taking a sample of the squares within the three selected clusters (psu’s) rather than every small square (ssu) within each cluster.

When: A multi-stage sampling strategy is often employed when the cost of measuring the units within the ssu is small compared to the cost of moving between psu’s. One must be cautious to use the correct estimate of variance and therefore confidence intervals when


using mult-stage estimates. The observations can’t be treated as a random sample from the target population, but rather a random sample from the psu (eg. Transect). Increasing the number of observations within each psu will improve the precision of the estimate of the psu, but will have limited effect on the precision of the overall estimate. Increasing the number of psu’s is generally required to substantially improve the precision of the overall estimate. The optimum sample size for both psu’s and ssu’s can be determined (as explained in the sample size calculator section) using preliminary estimates of the variability within and between psu’s combined with the cost/effort of moving between psu’s vs. the cost/effort of sampling more units within each psu.

Another interpretation of a multi-stage sample is where the lowest stage refers to the actual measurement protocol and so refers to the measurement error associated with the protocol. For example if counting all individuals within a transect, there may be measurement error involved if the animal is difficult to detect (Hankin 1984).

Adaptive sampling

What: The major difference with adaptive sampling is that the sample units are not fully described before the study begins. The strategy begins like any other sampling design with a random selection of sampling units, but in adaptive sampling additional sampling units may be added based on the observed values in the initial sample. This is particularly useful for clustered populations where the presence in a given sampling unit may increase the likelihood of presence in neighbouring populations. Additional samples are added based on a pre-determined rule (i.e. if at least one whale is present in the sampling unit then look in the 4 adjacent sampling units). The same or a different rule is then applied to the new sampling units. For rare clustered populations this method may be more efficient than a traditional sampling approach as more individuals are likely to be encountered for the same overall number of samples resulting in more precise estimates (Thompson and Seber 1996). This strategy is more difficult to implement than a traditional sampling design and a statistician should be consulted for the design and analysis. MacDonald (in Thompson 2004) found mixed feedback on this approach in a survey of statisticians and biologists and found that logistical difficulties with this approach may arise, particularly in surveys with large spatial scale.

When: rare and clustered populations

Examples from Thompson and Seber 1996: • Estimating whale populations in the ocean, where whales tend to group in pods

and cover a small fraction of the possible search area.• Contagious diseases, if a person tests positive for a rare disease, then test friends

and family of people who have had recent contact with the person • Rare plant and animal species• Cases where pollution levels are generally light, but where a few hotspots exist.


• Shrimp trawl surveys, where the shrimp tend to be clustered, but the cluster locations are not fixed due to the mobility of the shrimp.

• Deep sea fish (Orange Roughy) form large spawning aggregates making it difficult to use traditional sampling methods to estimate abundance.

Figure 7. This figure shows an example of an initial random sample of units on the left and the resulting adaptive cluster sample on the right (From Thompson 1990).

Ratio estimation

What: When two quantities are measured on each sample unit. For example if you measure yi = bushels of grain harvested from field i and xi = acreage of field i, and you are interested in B = average yield in bushels per acre or the ratio of y:x. (Lohr 1999)

When/why: (Lohr 1999) a) if the ratio itself is of interest, this is a straightforward applicationb) if N is unknown but can be related to some other metric such as weight (i.e. total

weight of fish in a net and average weight per fish)c) to improve the precision of an estimate of the total or mean. If you can collect two

types of information (x and y) on the same sampling unit and x and y are correlated, then you may be able to improve the precision of the estimate using the relationship between x and y. These estimates may be biased but may still be worth it if the reduction in variance is sufficient. Lohr (1999) gives a good description of the conditions where this method is suitable, showing how the precision and bias changes with sample size and the strength of the correlation between x and y. This method is essentially a special case of a regression model based estimate.


d) Adjust estimates from the sample based on ratios observed from different groups (such as gender) using ‘post-stratification’. See the discussion on post-stratification.

e) Can also be used to adjust for non response such as might occur in a survey of businesses, Non-response in an environmental study may occur when access to a sampling unit is denied by a land owner or is not accessible for safety reasons.

Double sampling framework

What: In ratio estimation we measure multiple variables on every observation. However, if it is easier or cheaper to collect auxilliary information than the actual variable of interest, it may be convenient to take more samples of the auxiliary information than the variable of interest (Thompson 2002). The term ‘double sampling’ has not been consistently used. Our definition of double sampling follows Thompson 2002 and Pollock et al. 2002: where an initial sample is taken and only the auxiliary information is observed, but then a smaller second sample (often a subset) is taken where both the variable of interest and the auxiliary information is observed. The observed relationship between the two variables is used to improve the overall estimate for a given cost.

When: This strategy may be suitable when the cost of measuring the variable of interest for a sufficient number of sites is prohibitive. This approach is also used where there are multiple approaches for estimating the variable of interest that differ in cost and accuracy.

Examples:• Aerial vs. ground counts• ‘eyeball’ estimates of volume of trees in a stand vs. felling trees and measuring• Time constrained vs. area constrained searches for tailed frogs• Number of ponds vs. waterfowl counts

Sequential sampling

The general idea is that sampling continues until a prespecified number of events are observed (Seber 1986 quotes- Rao, Portier and Ondrasik, 1981). It may also be implemented by sampling until a particular precision level is reached (Elzinga et al. 2001). A first sample may be used to determine how many more samples to take (Lohr 1999), the variance estimates need to account for this.

Network sampling and snowball sampling

These strategies all follow a link tracing design where information about the links between sampling units are used (Thompson 2002). In network sampling, if one person is selected then everyone related to them (i.e. all siblings) are also sampled. These strategies can be useful in hard to access and elusive populations such as drug users or commercial sex workers. The idea is that members from a rare population know one another, so if you


interview one homeless person you can then ask them to identify additional homeless people. These ideas are also used in internet search engines.

Post stratification

Often at the outset of an experiment you don’t know what proportion of the sampling units belong to a particular group or stratum (males vs. females, grassland vs. forest…) and so it is impossible to stratify the sample prior to the implementation of the study. Post-stratification is when the sample is grouped into strata after the study is complete and stratified estimates are obtained. There is a major danger to post-stratification if ‘data snooping’ occurs. If you choose your strata after you see the data you can obtain arbitrarily small variances (Lohr 1999). Holt and Smith (1979) describe post-stratification as “a device for protecting the statistician’s inference against those occasions when his randomization gives an unbalanced or unrepresentative sample”. However, they like Lohr, note that it may be possible to stratify to the point where there are strata with only a single individual and Royall (1968) notes that “the statistician needs to consciously ignore some of the information in the data so that meaningful inference is possible”. A safe compromise to prevent datasnooping might be to choose the groups that you will stratify on ahead of time, but assign the observations to each group after you see the data. In order to use post-stratification the relative size of each stratum must be known, or they must be estimated (Thompson 2002). The variance estimate is slightly different with post-stratification and approximations vary among texts. Thompson (2002) provides a brief summary and justification for the variance estimate he recommends.

Quota sampling is a variation on this where the surveyor continues sampling until they obtain a pre-specified number from each stratum. If the samples are chosen at random then this is equivalent to a stratified random sample (Cochran 1977) but in practice this would require significant effort near the end of the study as many of the individuals drawn would be from strata that already had their quota filled. As a result quota samples are often allowed to be drawn in some convenient and not random manner. If the samples are not obtained using probability sampling we can’t draw inference from these samples except in a model-based approach (Lohr 1999).

Examples:• a useful strategy in estimating sea otter populations that occurred in both large

groups and very small groups (Thompson 2004, p30)

Rotating Panel Design

A rotating panel design provides a method to balance the conflicting goals of monitoring population status vs. monitoring the trend in populations. These designs have been referred to by many names: revisit, repeated, rotating, split-panel, replacement etc… McDonald (2003) provides an excellent review of these designs and develops consistent terminology. These designs provide a compromise between temporary and permanent sites that allow the estimation of both status and trends. For example, in order to estimate status, 100 different sites might be monitored each year for five years, providing a total


sample size of 500 for that period of time. For trend detection, it is best to revisit sites each year. Consequently, for the above example, it would be better to revisit the 100 sites visited the first year in each of the subsequent four years, yielding a total sample of only 100 sites over the 5-year period. A rotating panel design balances the requirements for both status estimation and trend detection (while also recognizing budget limitations related to sampling effort) by sampling a new set of sites each year over a particular time cycle (for status estimates), but then revisiting these sites according to some schedule (for improved trend assessments). Various versions of a rotating panel design allow for visiting a subset of sites every year, revisiting some sites on longer cycles, or incorporating new sites each year along with the revisit schedule.

ReferencesCochran, William Gemmell. 1977. Sampling Techniques. John Wiley & Sons, Inc., New York, NY.

Elzinga, Caryl L., Daniel W. Salzer, John W. Willoughby, and James P. Gibbs. 2001. Monitoring plant and animal populations. Blackwell Science, Inc., Malden, Massachusetts.

Hankin, D.G. 1984. Multistage sampling designs in fisheries research: Applications in small streams. Can. J. Fish. and Aquatic Sci. 41:1575-1591.

Holt, D. and T.M.F. Smith. 1979. Post Stratification. Journal of the Royal Statistical Society: Series A (General). 142:33-46.

Lohr, S.L. 1999. Sampling: Design and Analysis. Brooks/Cole Publishing Company. Pacific Grove, CA.

McDonald, Lyman L. 2004. Sampling Rare Populations. pp.11-42 in William L.Thompson ed. Sampling Rare or Elusive Species. Island Press. Washington, DC.

McDonald, T.L. 2003. Review of Environmental Monitoring Methods: Survey Designs. Environmental Monitoring and Assessment 85: 277-292

Pollack, K.H., J.D. Nichols, T.R. Simons, G.L. Farnsworth, L.L. Bailey and J.R. Sauer. 2002. Large scale wildlife monitoring studies: statistical methods for design and analysis. Environmetrics. 13:105-119.

Royall, R.M. 1968. An old approach to finite population sampling theory. J. Amer. Statist. Ass., 63:1269-1279.

Seber, G.A.F. 1986. A Review of Estimating Animal Abundance. Biometrics. 42: 267-292.

Thompson, S.K. and G.A.F. Seber. 1996. Adaptive Sampling. John Wiley and Sons Inc. Toronto, ON.

Thompson, S.K. 1990. Adaptive cluster sampling, J. Amer. Statistical Assoc. 85:1050-1059.

Thompson, S.K. 2002. Sampling 2nd Edition. John Wiley & Sons Inc. New York, NY.

Thompson, W.L. 2004. Sampling rare or elusive species. Island Press. Washington DC.


planning and designing implementation - … · web viewto clarify the role of the project in the...

Documents