Validating Wykoff's Model, Take 2:Equivalence tests and spatial analysis in a design-unbiased analytical framework

Robert Froese, Ph.D., R.P.F.School of Forest Resources and Environmental ScienceMichigan Technological University, Houghton MI 49931

http://www.teresco.org/pics/signs/20010627/forest.jpg

This presentation has six parts

Introduction

Methods

Equivalence

Trends

Relevance

Performance

• The model and the objectives• The region, the data and the approach• How well does Wykoff’s model predict?• Does Wykoff’s model pass a validation

test?• How does accuracy relate to model and

data structures• What does it mean for model users and

future revisions

This presentation has six parts

Introduction

Methods

Equivalence

Trends

Relevance

Performance

• The model and the objectives• The region, the data and the approach• How well does Wykoff’s model predict?• Does Wykoff’s model pass a validation

test?• How does accuracy relate to model and

data structures• What does it mean for model users and

future revisions

This presentation has six parts

Introduction

Methods

Equivalence

Trends

Relevance

Performance

• The model and the objectives• The region, the data and the approach• How well does Wykoff’s model predict?• Does Wykoff’s model pass a validation

test?• How does accuracy relate to model and

data structures• What does it mean for model users and

future revisions

This presentation has six parts

Introduction

Methods

Equivalence

Trends

Relevance

Performance

• The model and the objectives• The region, the data and the approach• How well does Wykoff’s model predict?• Does Wykoff’s model pass a validation

test?• How does accuracy relate to model and

data structures• What does it mean for model users and

future revisions

This presentation has six parts

Introduction

Methods

Equivalence

Trends

Relevance

Performance

• The model and the objectives• The region, the data and the approach• How well does Wykoff’s model predict?• Does Wykoff’s model pass a validation

test?• How does accuracy relate to model and

data structures?• What does it mean for model users and

future revisions

This presentation has six parts

Introduction

Methods

Equivalence

Trends

Relevance

Performance

• The model and the objectives• The region, the data and the approach• How well does Wykoff’s model predict?• Does Wykoff’s model pass a validation

test?• How does accuracy relate to model and

data structures• What does this mean for model users and

for future revisions?

Wykoff’s model predicts basal area increment but is used to project diameter

Introduction

Methods

Equivalence

Trends

Relevance

Performance

DDS = DBH2t+10 - DBH2

t

BAG = (π/4)·(DBH2t -

DBH2t-10)

DG = (DBH2 + DDS)0.5 - DBH

ln(DDS) = f(SIZE +SITE +COMP)

Wykoff’s model is a multiple linear regression on the logarithmic scale

( ) ( )( ) ( )

( ) 1001ln

sincos

lnln

12112

109

287

26543

2210

CCFbDBHBALbCRbCRb

ELbELbSLbSLbSLASPbSLASPb

DBHbDBHbbDDS

⋅++⋅+⋅+⋅+

⋅+⋅+⋅+⋅+⋅+⋅⋅+

⋅+⋅+=

• bi – coefficients estimated by ordinary least squares, of which:– b0 depends on habitat type and nearest National Forest

– b2 depends on nearest National Forest

– b12 depends on habitat type

This validation is focused on two notions

• Caswell (1976) introduces two ideas:– does a model user care if the internal structures are truthful, as long

as the model makes accurate predictions?– does the scientist care if the model makes accurate predictions, as

long as the model is useful for testing hypotheses about the underlying system?

• Robinson and Froese (2004) question how statistical tests are used for model validation– The usual null hypothesis is of no difference, or that a model is

valid, which seems unscientific– Arbitrarily small differences are detectable– A failure to reject may simply imply low power

This study had four objectivesand two perspectives

The objectives were:• to estimate model bias by species across the range of application;• to demonstrate a specific validation of Wykoff’s model for diameter

increment prediction through a test of equivalence;• to identify significant trends between bias and predictor variables,

and;• to evaluate spatial trends in bias across the geographic area to which

Prognosis is usually applied.

Two perspectives were taken regarding Wykoff’s model:• as a diameter increment model, and;• as it contributes to predictions of per hectare volume increment,

which is more intuitive or of more interest to many forest managers.

National Forestsand geography ofthe Inland Empire

Introduction

Methods

Equivalence

Trends

Relevance

Performance

1, 2, 4: http://www.tarleton.edu/~range/Woodlands%20and%20Forest/Northern%20Rocky%20Mountain%20Forests/NorthernRockyMountainForests.htm3: http://www.flintridgefoundation.org/conservation/feature_pic1l.jpg

Inland Empire forests change predictably at various geographic scales

1

2

3

4

The focus in this study was on geographically extensive individual tree field data

Data came fromForest Inventory and Analysis

€

pei = yi − ˆ y iSubject is prediction error

€

ˆ z i = exp( ˆ y i + s2 2)

DG = (DBH 2 + ˆ z i)0.5 − DBH

Correct for log transform bias

+

Vt -Vt-10

=∆V

Impute volume increment by backdating

Equivalence tests flip theburden of proof onto the model

• Select a metric of model performance

• Nominate an interval of equivalence – Say 10% of

• Construct two one-sided confidence intervals of size

• If completely contained within the interval, reject the null hypothesis of dissimilarity

€

pei = yi − ˆ y i

€

y i

From Robinson and Froese (2004)

Most FIA plots were variable probability samples and may imply a design bias

Introduction

Methods

Equivalence

Trends

Relevance

Performance

Mean diameter prediction residual (cm/decade) Species Simple random

sample Weighted

cluster sample Difference Abies grandis -0.130 -0.469 -0.339 Abies lasiocarpa -0.099 -0.156 -0.057 Larix occidentalis -0.069 -0.241 -0.172 Other hardwoods 1.425 0.907 -0.518 Other softwoods 0.047 -0.004 -0.051 Pinus contorta -0.187 -0.216 -0.029 Picea engelmanii 0.027 -0.065 -0.092 Pinus monticola 0.063 -0.418 -0.481 Pinus ponderosa -0.195 -0.517 -0.322 Pseudotsuga menziesii -0.124 -0.338 -0.214 Thuja plicata -0.267 -0.606 -0.339 Tsuga heterophylla -0.217 -0.530 -0.313

Design unbiased results showed a modest over prediction by Wykoff’s model

Volume Increment Diameter Increment Species

No. Plots

€

y

€

ˆ y Bias (%) No. tre es

€

y

€

ˆ y Bias (%) Abies grandis 552 55.4 54.8 1.1 3393 2.99 3.46 -15.7 Abies las ioca rpa 631 20.7 21.5 -4.1 4532 1.92 2.08 -8.1 Larix occ identalis 776 20.1 19.9 1.2 3794 1.71 1.96 -14.1 Other - har dwoods 113 31.6 18.3 41.9 593 2.50 1.59 36.3 Other - s oftwoods 172 20.9 20.2 3.4 1036 1.78 1.79 -0.2 Pinus contorta 679 25.1 27.3 -8.7 6013 1.51 1.72 -14.3 Pice a e ngelmanii 604 22.7 22.8 -0.6 3251 2.51 2.57 -2.6 Pinus mo nticola 125 33.4 31.9 4.5 289 3.02 3.44 -13.9 Pinus ponde rosa 525 34.9 36.3 -3.9 2531 2.83 3.35 -18.3 Pse udots ug a m e nz ies ii 1551 30.0 30.5 -1.6 12601 2.19 2.52 -15.5 Thuja plica ta 349 39.2 41.5 -5.9 2037 2.83 3.44 -21.4 Tsu ga heterophylla 153 34.9 35.9 -3.0 909 2.42 2.95 -21.9 All e xcep t ha rdwoods 2632 30.0 30.5 -2.0 40419 2.11 2.39 -13.7 All spe cies 2632 30.0 30.4 -1.5 40979 2.11 2.38 -12.7

Extrapolated to the study area, over prediction could be:~2,400 ha·plot-1 • 2,632 plots • 0.5 m3·ha-1·dec-1 = ~3,158,400 m3·dec-1

Equivalence tests are constructed within a regression framework

Introduction

Methods

Equivalence

Trends

Relevance

Performance

Species

€

ˆ y

€

y

€

Cβ0+

€

Cβ0−

€

Iβ0+

€

Iβ0−

Decision Min. inte rval

Othe r – s oftwood 1.79 1.78 1.99 1.57 1.96 1.61 fail 12% Pse udots ug a m e nz ies ii 2.52 2.19 2.29 2.08 2.78 2.27 fail 18%

Equivalence tests for diameter increment generally fail to validate the model

Test for the intercept Test for the slope Species

€

ˆ y

€

y Decision Min. inte rva l

€

b1 Decision Min. inte rval

Abies grandis 3.46 2.99 fail 20% 0.96 reje ct 12% Abies las ioca rpa 2.08 1.92 fail 17% 0.82 fail 28% Larix occ identalis 1.96 1.71 fail 25% 0.92 reje ct 22% Othe r – hardwood 1.59 2.50 fail 105% 0.72 fail 78% Othe r – s oftwood 1.79 1.78 fail 12% 0.80 fail 32% Pinus co ntorta 1.72 1.51 fail 23% 0.86 fail 25% Pice a e ngelm anii 2.57 2.51 fail 12% 0.85 reje ct 24% Pinus mo nticola 3.44 3.02 fail 59% 0.52 fail 102% Pinus ponde rosa 3.35 2.83 fail 27% 0.83 fail 31% Pse udots ug a m e nz ies ii 2.52 2.19 fail 18% 0.91 reje ct 13% Thuja plica ta 3.44 2.83 fail 26% 0.76 fail 33% Tsu ga heterophylla 2.95 2.42 fail 33% 0.82 fail 35%

For stand level volume increment, equivalence tests frequently validate the model

Test for the intercept Test for the slope Species

€

ˆ y

€

y Decision Min. inte rva l

€

b1 Decision Min. inte rval

Abies grandis 54.77 55.39 reje ct 7.0% 0.90 reje ct 17.5 % Abies las ioca rpa 21.49 20.66 fail 10.1 % 0.93 reje ct 15.0 % Larix occ identalis 19.89 20.13 reje ct 7.3% 0.96 reje ct 11 .4 % Othe r – hardwood 18.35 31.58 fail 92.5 % 1.69 fail 98.2 % Othe r – s oftwood 20.20 20.90 fail 16.7 % 0.98 reje ct 19.5 % Pinus co ntorta 27.26 25.09 fail 13.3 % 0.88 reje ct 18.9 % Pice a e ngelm anii 22.79 22.67 reje ct 4.7% 0.89 reje ct 16.1 % Pinus mo nticola 31.88 33.39 fail 15.0 % 1.09 reje ct 20.0 % Pinus ponde rosa 36.27 34.93 reje ct 8.8% 0.95 reje ct 11 .7 % Pse udots ug a m e nz ies ii 30.54 30.05 reje ct 4.1% 0.94 reje ct 9.5% Thuja plica ta 41.47 39.17 fail 13.0 % 0.79 fail 31.0 % Tsu ga heterophylla 35.92 34.87 reje ct 9.0% 0.92 reje ct 16.7 %

Prediction error is weakly related to most predictor variables

Introduction

Methods

Equivalence

Trends

Relevance

Performance

p ≤ 0.01, r2 ≤ 0.1 in all cases

Diameter prediction error shows a spatial trend irregularly correlated with elevation

Spatial trends in volume prediction error largely mirror those for diameter

In some locations bias appears meaningfully different on and off of National Forest lands

Nearest National Forest

€

d

€

Cd+

€

Cd−

€

Id+

€

Id− Decision Min. inte rval

Bitte rroot 0.053 0.181 -0.074 0.148 -0.148 fail 12.3% Coeur d'Alene 0.265 0.514 0.015 0.332 -0.332 fail 15.5% Clearwater -0.471 -0.315 -0.626 0.318 -0.318 fail 19.6% Flathead 0.060 0.146 -0.027 0.203 -0.203 reject 7.2% Kanisku -0.034 0.071 -0.138 0.233 -0.233 reject 5.9% Kootenai -0.118 -0.015 -0.221 0.197 -0.197 fail 11.2% Lolo 0.152 0.210 0.095 0.161 -0.161 fail 13.0% Nez Perce -0.098 0.258 -0.454 0.304 -0.304 fail 14.9% St. Joe -0.044 0.100 -0.188 0.281 -0.281 reject 6.7%

Forests that are equivalent have an obvious matrix of public and private land across elevation and geography

Wykoff’s model for prediction

• Equivalence tests provide an objective methodology for assessing model validity– There is added subjectivity in the selection of I

• For diameter, a large I would be necessary to validate Wykoff’s model– For most species I = 25% would have to be used– largely because of bias not two-one-sided CI

• For volume, the model is largely validated– But trends show bias is close to zero for average

conditions• Overprediction of > 3 mil m3•dec-1 is not insubstantial• Species results differ, and may imply invalid stand

dynamics

Introduction

Methods

Equivalence

Trends

Relevance

Performance

Wykoff’s model as a theory

• Wykoff’s model is surprisingly robust– These tests involve substantial extrapolation in time and

space

• Model improvements should focus on the way climate is represented– LOC as a proxy for regional climate– EL as a global parabolic function– Interactions with other predictors, like DBH2

– Static proxies or process variables?

• Other issues remain– Small trees

Summary

Wykoff’s model modestly over predicts diameter increment, but the effect on volume is smaller

Equivalence tests fail to validate the model for diameter increment, but less often for volume

As a theory, the model is surprisingly robust

The way climate is represented in the model needs to be addressed

∆D 14%∆V 2%