an roc analysis of biological indicators of great …jeff buckley university of windsor glei –...
TRANSCRIPT
![Page 1: An ROC Analysis of Biological Indicators of Great …Jeff Buckley University of Windsor GLEI – GREAT LAKES ENVIRONMENTAL INDICATORS PROJECT FISH-BASED BIOLOGICAL INDICATORS Biological](https://reader033.vdocuments.us/reader033/viewer/2022042015/5e743b1128adaa76bc2edf43/html5/thumbnails/1.jpg)
ASSESSING THE ACCURACY OF FISH-BASED
BIOLOGICAL INDICATORS OF WETLAND
CONDITION AT THE GREAT LAKES COASTAL
MARGINS
Jeff Buckley
University of Windsor
![Page 2: An ROC Analysis of Biological Indicators of Great …Jeff Buckley University of Windsor GLEI – GREAT LAKES ENVIRONMENTAL INDICATORS PROJECT FISH-BASED BIOLOGICAL INDICATORS Biological](https://reader033.vdocuments.us/reader033/viewer/2022042015/5e743b1128adaa76bc2edf43/html5/thumbnails/2.jpg)
GLEI – GREAT LAKES ENVIRONMENTAL
INDICATORS PROJECT
![Page 3: An ROC Analysis of Biological Indicators of Great …Jeff Buckley University of Windsor GLEI – GREAT LAKES ENVIRONMENTAL INDICATORS PROJECT FISH-BASED BIOLOGICAL INDICATORS Biological](https://reader033.vdocuments.us/reader033/viewer/2022042015/5e743b1128adaa76bc2edf43/html5/thumbnails/3.jpg)
FISH-BASED BIOLOGICAL INDICATORS
Biological Indices
Wetland Fish Index (Seilheimer & Chow-
Fraser, 2007)
Fish Condition Index (Bhagat & Ciborowski
2005)
![Page 4: An ROC Analysis of Biological Indicators of Great …Jeff Buckley University of Windsor GLEI – GREAT LAKES ENVIRONMENTAL INDICATORS PROJECT FISH-BASED BIOLOGICAL INDICATORS Biological](https://reader033.vdocuments.us/reader033/viewer/2022042015/5e743b1128adaa76bc2edf43/html5/thumbnails/4.jpg)
HOW SHOULD BIOLOGICAL
INDICATORS BE EVALUATED?
Degradation
Bhagat et al., 2007
Bio
log
ica
l In
de
x S
co
re
![Page 5: An ROC Analysis of Biological Indicators of Great …Jeff Buckley University of Windsor GLEI – GREAT LAKES ENVIRONMENTAL INDICATORS PROJECT FISH-BASED BIOLOGICAL INDICATORS Biological](https://reader033.vdocuments.us/reader033/viewer/2022042015/5e743b1128adaa76bc2edf43/html5/thumbnails/5.jpg)
Sensitivity: 25%
Specificity: 100%
ROC ANALYSIS
Sensitivity
Ability of an
indicator to identify
degraded sites.
Specificity
Ability of a indicator
to identify non-
degraded sites.
Degradation
Indic
ato
r M
odel S
core
Sensitivity: 85%
Specificity: 55%
Sensitivity: 100%
Specificity: 25%
![Page 6: An ROC Analysis of Biological Indicators of Great …Jeff Buckley University of Windsor GLEI – GREAT LAKES ENVIRONMENTAL INDICATORS PROJECT FISH-BASED BIOLOGICAL INDICATORS Biological](https://reader033.vdocuments.us/reader033/viewer/2022042015/5e743b1128adaa76bc2edf43/html5/thumbnails/6.jpg)
ROC ANALYSIS: AREA UNDER THE CURVE
Area Under the
Curve (AUC):
Measure of indicator
accuracy
Between 0 and 1
.500 – No better
than chance
1.00 - Optimal
accuracy
![Page 7: An ROC Analysis of Biological Indicators of Great …Jeff Buckley University of Windsor GLEI – GREAT LAKES ENVIRONMENTAL INDICATORS PROJECT FISH-BASED BIOLOGICAL INDICATORS Biological](https://reader033.vdocuments.us/reader033/viewer/2022042015/5e743b1128adaa76bc2edf43/html5/thumbnails/7.jpg)
RECEIVER-OPERATING CHARACTERISTIC (ROC)
CURVE ANALYSIS
Test the classification accuracy of a model with a
continuous output.
Does not require choosing a threshold for
degradation.
Does not assume a linear relationship between
degradation and community response.
Can show the optimum threshold for classification.
![Page 8: An ROC Analysis of Biological Indicators of Great …Jeff Buckley University of Windsor GLEI – GREAT LAKES ENVIRONMENTAL INDICATORS PROJECT FISH-BASED BIOLOGICAL INDICATORS Biological](https://reader033.vdocuments.us/reader033/viewer/2022042015/5e743b1128adaa76bc2edf43/html5/thumbnails/8.jpg)
WETLAND FISH INDEX
Original Data (Seilheimer & Chow-Fraser 2007)
Current Data
2011 - 2013
n = 98 n = 60
![Page 9: An ROC Analysis of Biological Indicators of Great …Jeff Buckley University of Windsor GLEI – GREAT LAKES ENVIRONMENTAL INDICATORS PROJECT FISH-BASED BIOLOGICAL INDICATORS Biological](https://reader033.vdocuments.us/reader033/viewer/2022042015/5e743b1128adaa76bc2edf43/html5/thumbnails/9.jpg)
FISH CONDITION INDEX
Original Data (Bhagat & Ciborowski 2005)
Current Data
2011 - 2013
n = 73 n = 40
![Page 10: An ROC Analysis of Biological Indicators of Great …Jeff Buckley University of Windsor GLEI – GREAT LAKES ENVIRONMENTAL INDICATORS PROJECT FISH-BASED BIOLOGICAL INDICATORS Biological](https://reader033.vdocuments.us/reader033/viewer/2022042015/5e743b1128adaa76bc2edf43/html5/thumbnails/10.jpg)
STRESSOR MEASURES
- Agricultural Stress
- Development Stress
- Cumulative Stress
- 3500 sites
- Watershed level
land use-based stress
![Page 11: An ROC Analysis of Biological Indicators of Great …Jeff Buckley University of Windsor GLEI – GREAT LAKES ENVIRONMENTAL INDICATORS PROJECT FISH-BASED BIOLOGICAL INDICATORS Biological](https://reader033.vdocuments.us/reader033/viewer/2022042015/5e743b1128adaa76bc2edf43/html5/thumbnails/11.jpg)
CUMULATIVE STRESS
n = 60
![Page 12: An ROC Analysis of Biological Indicators of Great …Jeff Buckley University of Windsor GLEI – GREAT LAKES ENVIRONMENTAL INDICATORS PROJECT FISH-BASED BIOLOGICAL INDICATORS Biological](https://reader033.vdocuments.us/reader033/viewer/2022042015/5e743b1128adaa76bc2edf43/html5/thumbnails/12.jpg)
AGRICULTURAL STRESS
n = 60
![Page 13: An ROC Analysis of Biological Indicators of Great …Jeff Buckley University of Windsor GLEI – GREAT LAKES ENVIRONMENTAL INDICATORS PROJECT FISH-BASED BIOLOGICAL INDICATORS Biological](https://reader033.vdocuments.us/reader033/viewer/2022042015/5e743b1128adaa76bc2edf43/html5/thumbnails/13.jpg)
DEVELOPMENT STRESS
n = 60
![Page 14: An ROC Analysis of Biological Indicators of Great …Jeff Buckley University of Windsor GLEI – GREAT LAKES ENVIRONMENTAL INDICATORS PROJECT FISH-BASED BIOLOGICAL INDICATORS Biological](https://reader033.vdocuments.us/reader033/viewer/2022042015/5e743b1128adaa76bc2edf43/html5/thumbnails/14.jpg)
AUC COMPARISON
Index Agriculture Development Cumulative
Stress
Wetland Fish
Index 0.824 0.663 0.836
RDC Index 0.507 0.697 0.733
Species
Richness 0.559 0.550 0.600
Shannon
Diversity 0.641 0.520 0.592
![Page 15: An ROC Analysis of Biological Indicators of Great …Jeff Buckley University of Windsor GLEI – GREAT LAKES ENVIRONMENTAL INDICATORS PROJECT FISH-BASED BIOLOGICAL INDICATORS Biological](https://reader033.vdocuments.us/reader033/viewer/2022042015/5e743b1128adaa76bc2edf43/html5/thumbnails/15.jpg)
OPTIMAL INDICATOR THRESHOLD -
AGRICULTURE
n = 60
![Page 16: An ROC Analysis of Biological Indicators of Great …Jeff Buckley University of Windsor GLEI – GREAT LAKES ENVIRONMENTAL INDICATORS PROJECT FISH-BASED BIOLOGICAL INDICATORS Biological](https://reader033.vdocuments.us/reader033/viewer/2022042015/5e743b1128adaa76bc2edf43/html5/thumbnails/16.jpg)
OPTIMAL INDICATOR THRESHOLD –
CUMULATIVE STRESS
n = 60
![Page 17: An ROC Analysis of Biological Indicators of Great …Jeff Buckley University of Windsor GLEI – GREAT LAKES ENVIRONMENTAL INDICATORS PROJECT FISH-BASED BIOLOGICAL INDICATORS Biological](https://reader033.vdocuments.us/reader033/viewer/2022042015/5e743b1128adaa76bc2edf43/html5/thumbnails/17.jpg)
OPTIMAL INDICATOR THRESHOLD –
WATER QUALITY INDEX
n = 60
![Page 18: An ROC Analysis of Biological Indicators of Great …Jeff Buckley University of Windsor GLEI – GREAT LAKES ENVIRONMENTAL INDICATORS PROJECT FISH-BASED BIOLOGICAL INDICATORS Biological](https://reader033.vdocuments.us/reader033/viewer/2022042015/5e743b1128adaa76bc2edf43/html5/thumbnails/18.jpg)
CONCLUSIONS
Classification accuracy is important when testing indices.
ROC Curve Analysis is a useful method for testing
classification accuracy.
The Wetland Fish Index is a valid classifier of biological
condition in Great Lakes coastal margins.
![Page 19: An ROC Analysis of Biological Indicators of Great …Jeff Buckley University of Windsor GLEI – GREAT LAKES ENVIRONMENTAL INDICATORS PROJECT FISH-BASED BIOLOGICAL INDICATORS Biological](https://reader033.vdocuments.us/reader033/viewer/2022042015/5e743b1128adaa76bc2edf43/html5/thumbnails/19.jpg)
ACKNOWLEDGMENTS
Supervisor: Dr. Jan Ciborowski
Field Crews: Jasmine, Justin, Curtis, Pat, Carla, Janis
Lab members: Li, Michelle, Jesse, Steffi, Lyndon, Drew.
Collaborators: Joseph Gathman, Katya Kovalenko, Lucinda Johnson, Valerie Brady, Terry Brown, Euan Reavie, Bob Howe, George Host, Gerald Niemi, Nick Danz
"Funding for this work was provided by the Great Lakes National Program
Office under the United States Environmental Protection Agency, grant
number GL-00E00612-0. Although the research described in this work
has been partly funded by the US EPA, it has not been subjected to the
agency's required peer and policy review and therefore does not
necessarily reflect the views of the agency and no official endorsement
should be inferred."
![Page 20: An ROC Analysis of Biological Indicators of Great …Jeff Buckley University of Windsor GLEI – GREAT LAKES ENVIRONMENTAL INDICATORS PROJECT FISH-BASED BIOLOGICAL INDICATORS Biological](https://reader033.vdocuments.us/reader033/viewer/2022042015/5e743b1128adaa76bc2edf43/html5/thumbnails/20.jpg)
FISH COMMUNITIES RESPONDING TO
THRESHOLDS
r2 = 0.266
p < 0.001
n = 60
![Page 21: An ROC Analysis of Biological Indicators of Great …Jeff Buckley University of Windsor GLEI – GREAT LAKES ENVIRONMENTAL INDICATORS PROJECT FISH-BASED BIOLOGICAL INDICATORS Biological](https://reader033.vdocuments.us/reader033/viewer/2022042015/5e743b1128adaa76bc2edf43/html5/thumbnails/21.jpg)
FISH COMMUNITIES RESPONDING TO
THRESHOLDS
r2 = 0.266
p < 0.001
n = 60
![Page 22: An ROC Analysis of Biological Indicators of Great …Jeff Buckley University of Windsor GLEI – GREAT LAKES ENVIRONMENTAL INDICATORS PROJECT FISH-BASED BIOLOGICAL INDICATORS Biological](https://reader033.vdocuments.us/reader033/viewer/2022042015/5e743b1128adaa76bc2edf43/html5/thumbnails/22.jpg)
MODELS AS AN INDICATOR OF PHOSPHORUS
Disturbed: Total Phosphorus = 100 ug/L n = 340, 96, 44
![Page 23: An ROC Analysis of Biological Indicators of Great …Jeff Buckley University of Windsor GLEI – GREAT LAKES ENVIRONMENTAL INDICATORS PROJECT FISH-BASED BIOLOGICAL INDICATORS Biological](https://reader033.vdocuments.us/reader033/viewer/2022042015/5e743b1128adaa76bc2edf43/html5/thumbnails/23.jpg)
MODELS AS INDICATORS OF AGRICULTURAL
LAND COVER
Disturbed: Agricultural Land Cover = 10% n = 273, 66, 34
![Page 24: An ROC Analysis of Biological Indicators of Great …Jeff Buckley University of Windsor GLEI – GREAT LAKES ENVIRONMENTAL INDICATORS PROJECT FISH-BASED BIOLOGICAL INDICATORS Biological](https://reader033.vdocuments.us/reader033/viewer/2022042015/5e743b1128adaa76bc2edf43/html5/thumbnails/24.jpg)
MODELS AS INDICATORS OF POPULATION
Disturbed: Population density = 100 people/km2
n = 273, 66, 34
![Page 25: An ROC Analysis of Biological Indicators of Great …Jeff Buckley University of Windsor GLEI – GREAT LAKES ENVIRONMENTAL INDICATORS PROJECT FISH-BASED BIOLOGICAL INDICATORS Biological](https://reader033.vdocuments.us/reader033/viewer/2022042015/5e743b1128adaa76bc2edf43/html5/thumbnails/25.jpg)
SPECIES RICHNESS AND SHANNON
DIVERSITY METRICS
n = 340 Disturbed: Total Phosphorus 100 ug/L
![Page 26: An ROC Analysis of Biological Indicators of Great …Jeff Buckley University of Windsor GLEI – GREAT LAKES ENVIRONMENTAL INDICATORS PROJECT FISH-BASED BIOLOGICAL INDICATORS Biological](https://reader033.vdocuments.us/reader033/viewer/2022042015/5e743b1128adaa76bc2edf43/html5/thumbnails/26.jpg)
INDICATOR MODEL VS. ORIGINAL STRESS
GRADIENT
Disturbed: WQI = 0, n = 340 Disturbed: Rank = 8, n = 17, 18
![Page 27: An ROC Analysis of Biological Indicators of Great …Jeff Buckley University of Windsor GLEI – GREAT LAKES ENVIRONMENTAL INDICATORS PROJECT FISH-BASED BIOLOGICAL INDICATORS Biological](https://reader033.vdocuments.us/reader033/viewer/2022042015/5e743b1128adaa76bc2edf43/html5/thumbnails/27.jpg)
Anthropogenic Disturbance
Environmental response
(TP, DO, ChlA, etc.)
Biotic response
(Richness, tolerant species, etc.)
Environmental Condition
Biological
Indicator Model
Lab Analysis of
Chemical Data
![Page 28: An ROC Analysis of Biological Indicators of Great …Jeff Buckley University of Windsor GLEI – GREAT LAKES ENVIRONMENTAL INDICATORS PROJECT FISH-BASED BIOLOGICAL INDICATORS Biological](https://reader033.vdocuments.us/reader033/viewer/2022042015/5e743b1128adaa76bc2edf43/html5/thumbnails/28.jpg)
AUC COMPARISON
Indicator
Model
Original
Stress
Measure
Phosphoru
s
Agricultur
al Stress
Populati
on
Stress
Average
AUC
Wetland
Fish
Index
.694 .797 .682 .717 .732
Scirpus
IBI .861 .522 .541 .585 .549
Typha
IBI .870 .562 .642 .589 .598
![Page 29: An ROC Analysis of Biological Indicators of Great …Jeff Buckley University of Windsor GLEI – GREAT LAKES ENVIRONMENTAL INDICATORS PROJECT FISH-BASED BIOLOGICAL INDICATORS Biological](https://reader033.vdocuments.us/reader033/viewer/2022042015/5e743b1128adaa76bc2edf43/html5/thumbnails/29.jpg)
ROC ANALYSIS
Sensitivity
Ability of a test to
identify degraded
sites.
Specificity
Ability of a test to
identify undegraded
sites.
WFI 1 2 3 4 5
![Page 30: An ROC Analysis of Biological Indicators of Great …Jeff Buckley University of Windsor GLEI – GREAT LAKES ENVIRONMENTAL INDICATORS PROJECT FISH-BASED BIOLOGICAL INDICATORS Biological](https://reader033.vdocuments.us/reader033/viewer/2022042015/5e743b1128adaa76bc2edf43/html5/thumbnails/30.jpg)
FISH BASED BIOLOGICAL INDICATOR MODELS
Evaluates sites based
on metrics such as:
Species Richness
% tolerant species
% native species
Scores sites:
0 - Heavily degraded
72 - Non-degraded
Evaluates sites based on
presence of key indicator
species such as:
Gizzard shad
Longear sunfish
Score sites:
1 - Heavily degraded
5 - Non-degraded
Reference-degraded
Condition Index Wetland Fish Index
Bhagat & Ciborowski, 2005 Seilheimer & Chow-Fraser, 2007
![Page 31: An ROC Analysis of Biological Indicators of Great …Jeff Buckley University of Windsor GLEI – GREAT LAKES ENVIRONMENTAL INDICATORS PROJECT FISH-BASED BIOLOGICAL INDICATORS Biological](https://reader033.vdocuments.us/reader033/viewer/2022042015/5e743b1128adaa76bc2edf43/html5/thumbnails/31.jpg)
INDICATORS OF CONDITION
Measure:
Dissolved Oxygen
Chlorophyll-a
Phosphorus
Sources of pollution
Problems:
Expensive
Time consuming
Require precise water
collection & lab analysis
Adapted from Kurtz et al., 2001
![Page 32: An ROC Analysis of Biological Indicators of Great …Jeff Buckley University of Windsor GLEI – GREAT LAKES ENVIRONMENTAL INDICATORS PROJECT FISH-BASED BIOLOGICAL INDICATORS Biological](https://reader033.vdocuments.us/reader033/viewer/2022042015/5e743b1128adaa76bc2edf43/html5/thumbnails/32.jpg)
BIOLOGICAL INDICATORS
Use the biota as
indicators.
Easy, cheap, quick,
requires relatively little
expertise.
Directly measures what
ecologists, managers
and stakeholders are
actually interested in.
Adapted from Kurtz et al., 2001
![Page 33: An ROC Analysis of Biological Indicators of Great …Jeff Buckley University of Windsor GLEI – GREAT LAKES ENVIRONMENTAL INDICATORS PROJECT FISH-BASED BIOLOGICAL INDICATORS Biological](https://reader033.vdocuments.us/reader033/viewer/2022042015/5e743b1128adaa76bc2edf43/html5/thumbnails/33.jpg)
RECEIVER OPERATING
CHARACTERISTIC CURVE (ROC)
ANALYSIS
Degradation
Bhagat et al., 2007
Bio
log
ica
l In
de
x S
co
re
![Page 34: An ROC Analysis of Biological Indicators of Great …Jeff Buckley University of Windsor GLEI – GREAT LAKES ENVIRONMENTAL INDICATORS PROJECT FISH-BASED BIOLOGICAL INDICATORS Biological](https://reader033.vdocuments.us/reader033/viewer/2022042015/5e743b1128adaa76bc2edf43/html5/thumbnails/34.jpg)
WETLANDS AND CONDITION
Port Colborne, Ontario
Lake Erie
Killarney, Ontario
Georgian Bay
![Page 35: An ROC Analysis of Biological Indicators of Great …Jeff Buckley University of Windsor GLEI – GREAT LAKES ENVIRONMENTAL INDICATORS PROJECT FISH-BASED BIOLOGICAL INDICATORS Biological](https://reader033.vdocuments.us/reader033/viewer/2022042015/5e743b1128adaa76bc2edf43/html5/thumbnails/35.jpg)
EVALUATING BIOLOGICAL INDICES
Degradation
Bhagat et al., 2007
Bio
log
ica
l In
de
x S
co
re
![Page 36: An ROC Analysis of Biological Indicators of Great …Jeff Buckley University of Windsor GLEI – GREAT LAKES ENVIRONMENTAL INDICATORS PROJECT FISH-BASED BIOLOGICAL INDICATORS Biological](https://reader033.vdocuments.us/reader033/viewer/2022042015/5e743b1128adaa76bc2edf43/html5/thumbnails/36.jpg)
AGRICULTURE TEST - METRICS
![Page 37: An ROC Analysis of Biological Indicators of Great …Jeff Buckley University of Windsor GLEI – GREAT LAKES ENVIRONMENTAL INDICATORS PROJECT FISH-BASED BIOLOGICAL INDICATORS Biological](https://reader033.vdocuments.us/reader033/viewer/2022042015/5e743b1128adaa76bc2edf43/html5/thumbnails/37.jpg)
DEVELOPMENT TEST - METRICS
![Page 38: An ROC Analysis of Biological Indicators of Great …Jeff Buckley University of Windsor GLEI – GREAT LAKES ENVIRONMENTAL INDICATORS PROJECT FISH-BASED BIOLOGICAL INDICATORS Biological](https://reader033.vdocuments.us/reader033/viewer/2022042015/5e743b1128adaa76bc2edf43/html5/thumbnails/38.jpg)
CUMULATIVE STRESS TEST - METRICS
![Page 39: An ROC Analysis of Biological Indicators of Great …Jeff Buckley University of Windsor GLEI – GREAT LAKES ENVIRONMENTAL INDICATORS PROJECT FISH-BASED BIOLOGICAL INDICATORS Biological](https://reader033.vdocuments.us/reader033/viewer/2022042015/5e743b1128adaa76bc2edf43/html5/thumbnails/39.jpg)
GLEI PROJECT AND GLCWMC
Great Lakes Environmental Indicators Project
Great Lakes Coastal Wetland Monitoring Consortium
Goal: “To develop an integrated set of indicators that can be used to assess the condition of coastal margins of all five Great Lakes”
![Page 40: An ROC Analysis of Biological Indicators of Great …Jeff Buckley University of Windsor GLEI – GREAT LAKES ENVIRONMENTAL INDICATORS PROJECT FISH-BASED BIOLOGICAL INDICATORS Biological](https://reader033.vdocuments.us/reader033/viewer/2022042015/5e743b1128adaa76bc2edf43/html5/thumbnails/40.jpg)
RESEARCH QUESTIONS
Do indices hold-up
to their original
disturbance
gradient?
Are the indices
robust against other
measures of stress
Use ROC to
optimize indices.