evaluating growth models: a case study using prognosis bc evaluating growth models: a case study...
TRANSCRIPT
Evaluating Growth Models: Evaluating Growth Models: A Case Study Using PrognosisA Case Study Using PrognosisBCBC
Peter Marshall, University of British Columbia, Peter Marshall, University of British Columbia, Vancouver, BCVancouver, BC
Pablo Parysow, Northern Arizona University, Pablo Parysow, Northern Arizona University, Flagstaff, AZFlagstaff, AZ
Shadrach Akindele, Federal University of Shadrach Akindele, Federal University of Technology, Akure, NigeriaTechnology, Akure, Nigeria
Presented at the Third FVS Conference, Feb. 13-15, Fort Collins, CO
OutlineOutline
BackgroundBackground- Prognosis- PrognosisBCBC
- Validation- Validation
- Study area- Study area Testing Against DataTesting Against Data Simulation TestingSimulation Testing ConclusionsConclusions
Within a framework of general observations about validation techniques and processes.
Validation Observation #1
Validation is important … but it tends to be much more of interest to the person doing it than it is to the person hearing about it.
PrognosisPrognosisBCBC
An adaptation of the northern Idaho (NI) version of theAn adaptation of the northern Idaho (NI) version of the original Prognosis modeloriginal Prognosis model
The architecture of the original model remains but many The architecture of the original model remains but many of the internal equations have been reformulated and the of the internal equations have been reformulated and the remainder have been recalibratedremainder have been recalibrated
Habitat types have been replaced with appropriate units Habitat types have been replaced with appropriate units within the BC Biogeoclimatic Ecosystem Classification within the BC Biogeoclimatic Ecosystem Classification (BEC) system(BEC) system
All inputs and outputs are in metric unitsAll inputs and outputs are in metric units Different versions have been developed for various BEC Different versions have been developed for various BEC
subzonessubzones Additional information is available at the following URL:Additional information is available at the following URL:
http://www.for.gov.bc.ca/hre/gymodels/progbc/http://www.for.gov.bc.ca/hre/gymodels/progbc/
ValidationValidation
There are various definitions of “validation” in use and a There are various definitions of “validation” in use and a growing literature on different approaches to use in growing literature on different approaches to use in validationvalidation
For the purposes of this presentation, I will define For the purposes of this presentation, I will define validation as:validation as:
““The process of evaluating model outputs forThe process of evaluating model outputs for consistency and usefulness.” consistency and usefulness.”
Under this definition, validation is very much context Under this definition, validation is very much context dependentdependent- which model outputs are being evaluated - which model outputs are being evaluated - in what location(s)- in what location(s)- for what purposes- for what purposes
Study AreaStudy Area
British Columbia
Victoria
Vancouver
Study Site Location
Interior Douglas-Fir
Zone
CANADA
Validation Observation #2
Validation is most effective if several different approaches to validation are used – there are gains from added perspective.
Testing Against Independent DataTesting Against Independent Data Data were from two research installations established in the Data were from two research installations established in the
late 1980s in stands which were uneven-aged and primarily late 1980s in stands which were uneven-aged and primarily interior Douglas-firinterior Douglas-fir
One installation, consisting of 6 plots measured on 4 One installation, consisting of 6 plots measured on 4 occasions, was set up to follow stand dynamics under occasions, was set up to follow stand dynamics under different structural conditionsdifferent structural conditions(1) predominance of large older trees (dbh > 30 cm)(1) predominance of large older trees (dbh > 30 cm)(2) predominance of pole-sized trees (dbh 15-30 cm) (2) predominance of pole-sized trees (dbh 15-30 cm) (3) predominance of saplings (dbh < 15 cm)(3) predominance of saplings (dbh < 15 cm)
The second installation, consisting of 24 plots measured on 3 The second installation, consisting of 24 plots measured on 3 occasions, was set up as a precommercial thinning occasions, was set up as a precommercial thinning experiment in stands which were diameter-limit logged in experiment in stands which were diameter-limit logged in the 1960sthe 1960s
- 3 blocks consisting of three thinning treatments and - 3 blocks consisting of three thinning treatments and a a
control, with two plots in each block/treatment control, with two plots in each block/treatment Projections made for 11 years to match one of the possible Projections made for 11 years to match one of the possible
remeasurement intervals (closest match to the 10-year remeasurement intervals (closest match to the 10-year projections of Prognosisprojections of PrognosisBCBC) )
Projected stand and tree level attributes
Stand-level attributes Tree-level attributes
Stems per hectare
Basal area
Total volume
Merchantable Volume
Dbh
Tree height
Validation Observation #3
If at all possible, try to look at both individual tree projections as well as stand-level projections. Joint comparisons might well highlight issues that otherwise would not be apparent.
Validation Observation #4
Even apparently well-tested models may well still contain hidden coding errors that have subtle impacts. They are worth looking for carefully. This process is known by some as “verification”.
(We found a few such errors by running various of the component equations both within and outside the model environment looking for “oddities”.)
Validation Observation #5
It is best to look for coding errors early on in the validation process. Otherwise, you may have to re-do some of your previous work.
Validation Observation #6
Individuals who are at arm’s length from the model development process are often more likely to spot errors, since they don’t usually assume that they “know” what is going on within the model.
Summary of individual tree dbh residuals by diameter classes
DBH Classes (cm)
All 10 15 20 25 30 40 50 60 > 60
Trees 3078 1467 969 398 113 41 49 28 9 4
Mean
Residual
- 0.009
- 0.154
0.126
0.211
0.048
0.066
- 0.055
- 0.036
- 0.978
- 0.925
Summary of individual tree DBH residuals by spacing treatment
Spacing Treatment
All C1 C2 CTRL STD N/A
Stems* 3078 626 335 801 729 587
Mean
Residual
-0.009
-0.135
0.064
-0.018
-0.069
0.168
C1= 3m-clump, C2=5m-clump, CTRL=Control, STD= Standard Spacing, N/ A= Plots not in the spacing study * Trees > 5 cm in DBH
Validation Observation #7
Regression-based equivalence tests (Robinson et al. 2005) provide a convenient means of examining model predictions versus observations. The routine for equivalence testing in R produces nice pictures.
Robinson, A.P., R.A. Duursma, and J.D. Marshall. 2005. A regression-based equivalence test for model validation: shifting the burden of proof. Tree Physiology 25: 903-913
Predicted Dbh (cm)
Ob
serv
ed
Db
h (
cm)
0
20
40
60
80
0 20 40 60 80
Overall equivalence test for the tree-level DBH predictions.
Predicted Dbh (cm)
Ob
se
rve
d D
bh
(cm
)
0
10
20
30
40
50
0 10 20 30 40 50
C1 C2
CTRL
0 10 20 30 40 50
0
10
20
30
40
50
STRD
Equivalence tests (by spacing treatment) for the tree-level DBH predictions.
Validation Observation #8
What you see depends on what you look at.
For example, the relationship between observed and predicted DBH will appear considerably stronger than the relationship between observed and predicted DBH growth, which is actually what is estimated within PrognosisBC.
Predicted Stems/per ha
Ob
serv
ed
Ste
ms/
ha
2000
4000
6000
8000
2000 4000 6000
SimulationSimulation
Feasible and infeasible combinations of BDq. Combinations that produced at least one tree/ha for the largest dbh class are identified as , whereas combinations that did not meet that standard are shown as . (Units for B: m2/ha, D: cm, and q: no units).
B=10 B=30 B=50 B=70
q=1.5 q=2.0 q=2.5 q=3.0 q=1.5 q=2.0 q=2.5 q=3.0 q=1.5 q=2.0 q=2.5 q=3.0 q=1.5 q=2.0 q=2.5 q=3.0 D=20
D=40
D=60
D=80
0
5
10
15
20
25
30
35
40
45
50
55
60
65
70
0 10 20 30 40 50
Projected Time (years)
Fo
rec
as
ted
Ba
sa
l A
rea
(s
q.
m/h
a)
B=10
B=30
B=50
B=70
0
5
10
15
20
25
30
35
40
45
50
55
60
65
70
0 10 20 30 40 50
Projected Time (years)
Fo
rec
as
ted
Ba
sa
l A
rea
(s
q.
m/h
a)
B=10
B=30
B=50
B=70
(a) (b)
0
5
10
15
20
25
30
35
40
45
50
55
60
65
70
0 10 20 30 40 50
Projected Time (years)
Fo
rec
as
ted
Ba
sa
l A
rea
(s
q.
m/h
a)
B=10
B=30
B=50
B=70
0
5
10
15
20
25
30
35
40
45
50
55
60
65
70
0 10 20 30 40 50
Projected Time (years)
Fo
rec
as
ted
Ba
sa
l A
rea
(s
q.
m/h
a)
B=50
B=70
(c) (d)
Projected basal area for various initial levels of initial basal area (B) with q = 1.5 and maximum diameter (D) at (a) 20 cm; (b) 40 cm; (c) 60 cm; and (d) 80 cm.
0
5
10
15
20
25
30
35
0 10 20 30 40 50
Projected Time (years)
Fo
rec
as
ted
Qu
ad
rati
c M
ea
n D
iam
ete
r (c
m)
B=10
B=30
B=50
B=70
0
5
10
15
20
25
30
35
0 10 20 30 40 50
Projected Time (years)
Fo
rec
as
ted
Qu
ad
rati
c M
ea
n D
iam
ete
r (c
m)
B=10
B=30
B=50
B=70
(a) (b)
0
5
10
15
20
25
30
35
0 10 20 30 40 50
Projected Time (years)
Fo
rec
as
ted
Qu
ad
rati
c M
ea
n D
iam
ete
r (c
m)
B=10
B=30
B=50
B=70
0
5
10
15
20
25
30
35
0 10 20 30 40 50
Projected Time (years)
Fo
rec
as
ted
Qu
ad
rati
c M
ea
n D
iam
ete
r (c
m)
B=50
B=70
(c) (d)
Projected change in quadratic mean diameter (qmd) for various initial levels of initial basal area (B) with q = 1.5 and maximum diameter (D) at (a) 20 cm; (b) 40 cm; (c) 60 cm; and (d) 80 cm.
0
1000
2000
3000
4000
5000
6000
7000
0 10 20 30 40 50
Projected Times (years)
Fo
rec
as
ted
Ste
ms
pe
r H
aB=10
B=30
B=50
B=70
0
500
1000
1500
2000
2500
3000
3500
4000
0 10 20 30 40 50
Projected Times (years)
Fo
rec
as
ted
Ste
ms
pe
r H
a
B=10
B=30
B=50
B=70
(a) (b)
0
500
1000
1500
2000
2500
3000
3500
4000
0 10 20 30 40 50
Projected Times (years)
Fo
rec
as
ted
Ste
ms
pe
r H
a
B=10
B=30
B=50
B=70
0
500
1000
1500
2000
2500
3000
3500
4000
0 10 20 30 40 50
Projected Times (years)F
ore
ca
ste
d S
tem
s p
er
Ha
B=50
B=70
(c) (d)
Projected change in stems per ha for various initial levels of initial basal area (B) with q = 1.5 and maximum diameter (D) at (a) 20 cm; (b) 40 cm; (c) 60 cm; and (d) 80 cm.
Impact of modifications to initial stand structure components on changes to particular stand attributes over a 50 year projection period, based on projections using PrognosisBC. Other stand structure components are assumed to remain constant.
Increase in Stand Structure Component Attribute B D q
Basal Area Movement towards “carrying capacity”. If B is ~53 or below, approaches from below; if B is >53 approaches from above.
Slightly slower increase (for lower values of B) or slightly slower decrease (for higher values of B).
Very slight increase (for lower values of B) or very slight decrease (for higher values of B).
Quadratic Mean Dbh Decrease in the rate of increase.
Higher initial value and higher final value after 50 years.
Lower initial and lower final value after 50 years.
Stems Per Ha Higher initial value and final value after 50 years. Increase in the mortality rate.
Lower initial value and final value after 50 years. Decrease in the mortality rate.
Higher initial value and final value after 50 years. Increase in the mortality rate.
Dbh Growth Decrease in overall growth rate.
Slight decrease in growth rate.
Slight decrease in growth rate.
ConclusionsConclusions
The validation exercise allowed us to identify and The validation exercise allowed us to identify and repair minor errors in the coding.repair minor errors in the coding.
Once these errors were fixed, the model Once these errors were fixed, the model performed well against data at both the single performed well against data at both the single tree and stand level.tree and stand level.
The model produced results under a wide variety The model produced results under a wide variety of stand structures which were consistent with of stand structures which were consistent with our understanding of stand dynamics.our understanding of stand dynamics.
Validation Observation #9
When preparing validation reports, remember Observation #1:
“Validation is important … but it tends to be much more of interest to the person doing it than it is to the person hearing about it.”
It is easy to bury readers with an avalanche of results. However, syntheses and summaries are much more likely to be read and understood.
This project was funded by the BC Ministry of Forests and Range, using funds provided for continuing work on PrognosisBC by the Forest Investment Account (FIA). We are grateful for this support.
Thank you! Are there any questions?