![Page 1: Testing Predictive Performance of Ecological Niche Models A. Townsend Peterson, STOLEN FROM Richard Pearson](https://reader034.vdocuments.us/reader034/viewer/2022051319/5a4d1b677f8b9ab0599b0def/html5/thumbnails/1.jpg)
Testing Predictive Performance of Ecological Niche Models
A. Townsend Peterson, STOLEN FROMRichard Pearson
![Page 2: Testing Predictive Performance of Ecological Niche Models A. Townsend Peterson, STOLEN FROM Richard Pearson](https://reader034.vdocuments.us/reader034/viewer/2022051319/5a4d1b677f8b9ab0599b0def/html5/thumbnails/2.jpg)
Niche Model Validation• Diverse challenges …
– Not a single loss function or optimality criterion– Different uses demand different criteria– In particular, relative weights applied to omission and
commission errors in evaluating models
• Nakamura: “which way is relevant to adopt is not a mathematical question, but rather a question for the user”– Asymmetric loss functions
![Page 3: Testing Predictive Performance of Ecological Niche Models A. Townsend Peterson, STOLEN FROM Richard Pearson](https://reader034.vdocuments.us/reader034/viewer/2022051319/5a4d1b677f8b9ab0599b0def/html5/thumbnails/3.jpg)
![Page 4: Testing Predictive Performance of Ecological Niche Models A. Townsend Peterson, STOLEN FROM Richard Pearson](https://reader034.vdocuments.us/reader034/viewer/2022051319/5a4d1b677f8b9ab0599b0def/html5/thumbnails/4.jpg)
Where do I get testing data????
![Page 5: Testing Predictive Performance of Ecological Niche Models A. Townsend Peterson, STOLEN FROM Richard Pearson](https://reader034.vdocuments.us/reader034/viewer/2022051319/5a4d1b677f8b9ab0599b0def/html5/thumbnails/5.jpg)
(after Araújo et al. 2005 Gl. Ch. Biol.)
Model calibration and evaluation strategies: resubstitution
100%
Same region
Different region
Different time
Different resolutionEvaluation
Calibration
Projection
All available
data
![Page 6: Testing Predictive Performance of Ecological Niche Models A. Townsend Peterson, STOLEN FROM Richard Pearson](https://reader034.vdocuments.us/reader034/viewer/2022051319/5a4d1b677f8b9ab0599b0def/html5/thumbnails/6.jpg)
(after Araújo et al. 2005 Gl. Ch. Biol.)
Model calibration and evaluation strategies: independent validation
100%All
available data
Same region
Different region
Different time
Different resolutionEvaluation
Calibration
Projection
![Page 7: Testing Predictive Performance of Ecological Niche Models A. Townsend Peterson, STOLEN FROM Richard Pearson](https://reader034.vdocuments.us/reader034/viewer/2022051319/5a4d1b677f8b9ab0599b0def/html5/thumbnails/7.jpg)
(after Araújo et al. 2005 Gl. Ch. Biol.)
Model calibration and evaluation strategies: data splitting
70%
Test data
Same region
Different region
Different time
Different resolution
Evaluation
Calibration
Projection
Calibration data
30%
![Page 8: Testing Predictive Performance of Ecological Niche Models A. Townsend Peterson, STOLEN FROM Richard Pearson](https://reader034.vdocuments.us/reader034/viewer/2022051319/5a4d1b677f8b9ab0599b0def/html5/thumbnails/8.jpg)
Types of Error
![Page 9: Testing Predictive Performance of Ecological Niche Models A. Townsend Peterson, STOLEN FROM Richard Pearson](https://reader034.vdocuments.us/reader034/viewer/2022051319/5a4d1b677f8b9ab0599b0def/html5/thumbnails/9.jpg)
The four types of results that are possible when testing a distribution model
(see Pearson NCEP module 2007)
![Page 10: Testing Predictive Performance of Ecological Niche Models A. Townsend Peterson, STOLEN FROM Richard Pearson](https://reader034.vdocuments.us/reader034/viewer/2022051319/5a4d1b677f8b9ab0599b0def/html5/thumbnails/10.jpg)
Presence-absence confusion matrix
Predicted present
Predicted absent
Recorded present Recorded (or assumed) absent
a (true positive)
c (false negative)
b (false positive)
d (true negative)
![Page 11: Testing Predictive Performance of Ecological Niche Models A. Townsend Peterson, STOLEN FROM Richard Pearson](https://reader034.vdocuments.us/reader034/viewer/2022051319/5a4d1b677f8b9ab0599b0def/html5/thumbnails/11.jpg)
Thresholding
![Page 12: Testing Predictive Performance of Ecological Niche Models A. Townsend Peterson, STOLEN FROM Richard Pearson](https://reader034.vdocuments.us/reader034/viewer/2022051319/5a4d1b677f8b9ab0599b0def/html5/thumbnails/12.jpg)
![Page 13: Testing Predictive Performance of Ecological Niche Models A. Townsend Peterson, STOLEN FROM Richard Pearson](https://reader034.vdocuments.us/reader034/viewer/2022051319/5a4d1b677f8b9ab0599b0def/html5/thumbnails/13.jpg)
Selecting a decision threshold (p/a data)
(Liu et al. 2005 Ecography 29:385-393)
![Page 14: Testing Predictive Performance of Ecological Niche Models A. Townsend Peterson, STOLEN FROM Richard Pearson](https://reader034.vdocuments.us/reader034/viewer/2022051319/5a4d1b677f8b9ab0599b0def/html5/thumbnails/14.jpg)
Selecting a decision threshold (p/a data)
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0 0.2 0.4 0.6 0.8 1
Threshold
Kapp
a
![Page 15: Testing Predictive Performance of Ecological Niche Models A. Townsend Peterson, STOLEN FROM Richard Pearson](https://reader034.vdocuments.us/reader034/viewer/2022051319/5a4d1b677f8b9ab0599b0def/html5/thumbnails/15.jpg)
Selecting a decision threshold (p/a data)
Omission(proportion of presences predicted absent)
(c/a+c)
Commission(proportion of absences predicted present)
(b/b+d)
![Page 16: Testing Predictive Performance of Ecological Niche Models A. Townsend Peterson, STOLEN FROM Richard Pearson](https://reader034.vdocuments.us/reader034/viewer/2022051319/5a4d1b677f8b9ab0599b0def/html5/thumbnails/16.jpg)
0
0.2
0.4
0.6
0.8
1
0 20 40 60 80 100
threshold
omis
sion
rate
LPTT10
Selecting a decision threshold (p-o data)
![Page 17: Testing Predictive Performance of Ecological Niche Models A. Townsend Peterson, STOLEN FROM Richard Pearson](https://reader034.vdocuments.us/reader034/viewer/2022051319/5a4d1b677f8b9ab0599b0def/html5/thumbnails/17.jpg)
Threshold-dependent Tests(= loss functions)
![Page 18: Testing Predictive Performance of Ecological Niche Models A. Townsend Peterson, STOLEN FROM Richard Pearson](https://reader034.vdocuments.us/reader034/viewer/2022051319/5a4d1b677f8b9ab0599b0def/html5/thumbnails/18.jpg)
The four types of results that are possible when testing a distribution model
(see Pearson NCEP module 2007)
![Page 19: Testing Predictive Performance of Ecological Niche Models A. Townsend Peterson, STOLEN FROM Richard Pearson](https://reader034.vdocuments.us/reader034/viewer/2022051319/5a4d1b677f8b9ab0599b0def/html5/thumbnails/19.jpg)
Presence-absence test statistics
Predicted present
Predicted absent
Recorded present Recorded (or assumed) absent
a (true positive)
c (false negative)
b (false positive)
d (true negative)
Proportion (%) correctly predicted (or ‘accuracy’, or ‘correct classification rate’):
(a + d)/(a + b + c + d)
![Page 20: Testing Predictive Performance of Ecological Niche Models A. Townsend Peterson, STOLEN FROM Richard Pearson](https://reader034.vdocuments.us/reader034/viewer/2022051319/5a4d1b677f8b9ab0599b0def/html5/thumbnails/20.jpg)
Cohen’s Kappa:
)]/)))(())(((([)]/)))(())(((()[(
ndcdbbacanndcdbbacadak
Presence-absence test statistics
Predicted present
Predicted absent
Recorded present Recorded (or assumed) absent
a (true positive)
c (false negative)
b (false positive)
d (true negative)
![Page 21: Testing Predictive Performance of Ecological Niche Models A. Townsend Peterson, STOLEN FROM Richard Pearson](https://reader034.vdocuments.us/reader034/viewer/2022051319/5a4d1b677f8b9ab0599b0def/html5/thumbnails/21.jpg)
Proportion of observed presences correctly predicted (or ‘sensitivity’, or ‘true positive fraction’):
a/(a + c)
Presence-only test statistics
Predicted present
Predicted absent
Recorded present Recorded (or assumed) absent
a (true positive)
c (false negative)
b (false positive)
d (true negative)
![Page 22: Testing Predictive Performance of Ecological Niche Models A. Townsend Peterson, STOLEN FROM Richard Pearson](https://reader034.vdocuments.us/reader034/viewer/2022051319/5a4d1b677f8b9ab0599b0def/html5/thumbnails/22.jpg)
Proportion of observed presences correctly predicted (or ‘sensitivity’, or ‘true positive fraction’):
a/(a + c)
Proportion of observed presences incorrectly predicted (or ‘omission rate’, or ‘false negative fraction’):
c/(a + c)
Presence-only test statistics
Predicted present
Predicted absent
Recorded present Recorded (or assumed) absent
a (true positive)
c (false negative)
b (false positive)
d (true negative)
![Page 23: Testing Predictive Performance of Ecological Niche Models A. Townsend Peterson, STOLEN FROM Richard Pearson](https://reader034.vdocuments.us/reader034/viewer/2022051319/5a4d1b677f8b9ab0599b0def/html5/thumbnails/23.jpg)
Presence-only test statistics:testing for statistical significance
U. sikorae
Leaf-tailed gecko (Uroplatus)
U. sikorae
Success rate: 4 from 7Proportion predicted present: 0.231Binomial p = 0.0546
Success rate: 6 from 7Proportion predicted present: 0.339Binomial p = 0.008
![Page 24: Testing Predictive Performance of Ecological Niche Models A. Townsend Peterson, STOLEN FROM Richard Pearson](https://reader034.vdocuments.us/reader034/viewer/2022051319/5a4d1b677f8b9ab0599b0def/html5/thumbnails/24.jpg)
Proportion of observed (or assumed) absences correctly predicted (or ‘specificity’, or ‘true negative fraction’):
d/(b + d)
Absence-only test statistics
Predicted present
Predicted absent
Recorded present Recorded (or assumed) absent
a (true positive)
c (false negative)
b (false positive)
d (true negative)
![Page 25: Testing Predictive Performance of Ecological Niche Models A. Townsend Peterson, STOLEN FROM Richard Pearson](https://reader034.vdocuments.us/reader034/viewer/2022051319/5a4d1b677f8b9ab0599b0def/html5/thumbnails/25.jpg)
Proportion of observed (or assumed) absences correctly predicted (or ‘specificity’, or ‘true negative fraction’):
d/(b + d)
Proportion of observed (or assumed) absences incorrectly predicted (or ‘commission rate’, or ‘false positive fraction’):
b/(b + d)
Absence-only test statistics
Predicted present
Predicted absent
Recorded present Recorded (or assumed) absent
a (true positive)
c (false negative)
b (false positive)
d (true negative)
![Page 26: Testing Predictive Performance of Ecological Niche Models A. Townsend Peterson, STOLEN FROM Richard Pearson](https://reader034.vdocuments.us/reader034/viewer/2022051319/5a4d1b677f8b9ab0599b0def/html5/thumbnails/26.jpg)
AUC: a threshold-independent test statistic
Predicted presentPredicted absent
Recorded present Recorded (or assumed) absent
a (true positive)c (false negative)
b (false positive)d (true negative)
sensitivity = a/(a+c)
specificity = d/(b+d)
(1 – omission rate)
(fraction of absences predicted present)
![Page 27: Testing Predictive Performance of Ecological Niche Models A. Townsend Peterson, STOLEN FROM Richard Pearson](https://reader034.vdocuments.us/reader034/viewer/2022051319/5a4d1b677f8b9ab0599b0def/html5/thumbnails/27.jpg)
1 - specificity0 1
0
1
sens
itivi
ty Predicted probability of occurrence
Predicted probability of occurrence
10
10Fr
eque
ncy
Freq
uenc
y
set of ‘absences’ set of ‘presences’
set of ‘absences’ set of ‘presences’
Threshold-independent assessment:The Receiver Operating Characteristic (ROC) Curve
A B
C
(check out: http://www.anaesthetist.com/mnm/stats/roc/Findex.htm)