data quality and analysis strategy for monitoring post-fire rehabilitation treatments

DATA QUALITY and ANALYSIS

Strategy for Monitoring Post-fire Rehabilitation Treatments

Troy Wirth and David PykeTroy Wirth and David PykeUSGS – Biological Resources DivisionUSGS – Biological Resources DivisionForest and Rangeland Ecosystem Science CenterForest and Rangeland Ecosystem Science CenterCorvallis, OregonCorvallis, Oregon

U.S. Department of InteriorU.S. Geological Survey

Supported by USGS - BLM Interagency Agreement #HAI040045

Data QualityData Quality

Assess the ability of the data to determine Assess the ability of the data to determine treatment successtreatment success

Ability to achieve high data quality will Ability to achieve high data quality will depend on variabilitydepend on variability

Calculate data quality variables Calculate data quality variables Confidence intervals (precision)Confidence intervals (precision) Alpha (p-value) & beta levelsAlpha (p-value) & beta levels Sample size estimationSample size estimation

Confidence IntervalsConfidence Intervals Construct a simple Construct a simple

confidence interval around confidence interval around data to determine data to determine precision of estimateprecision of estimate

The narrower the The narrower the confidence interval, the confidence interval, the more precise the estimatemore precise the estimate

Must specify the alpha Must specify the alpha level level

nstX critical

Alpha level, or type I error is the probability of Alpha level, or type I error is the probability of declaring there is no difference when there is. declaring there is no difference when there is. Specifies the width of the confidence interval (1 – Specifies the width of the confidence interval (1 – alpha)alpha)

From Elzinga et al. 1998

Confidence IntervalsConfidence Intervals

Farewell Bend Native Seeding Density

Ex. Grass Ex. Shrubs Sd. Grass Sd. Shrubs Sd. Forbs

Den

sity

(pla

nts/

m2 )

-1

0

1

2

3

4

ControlTreatment

Sample Size EstimationSample Size Estimation Equations which estimate the number of Equations which estimate the number of

samples required to meet your sampling samples required to meet your sampling objective objective For single populations (quantitative objective)For single populations (quantitative objective)

Confidence level (Type I error rate)Confidence level (Type I error rate) Confidence interval widthConfidence interval width

For detecting difference between two For detecting difference between two populationspopulations

Confidence levels (Type I and II error rate)Confidence levels (Type I and II error rate) Minimum detectable changeMinimum detectable change

Iterative processIterative process

Single Population Sample SizeSingle Population Sample Size Calculate sample size Calculate sample size

estimate for each estimate for each parameter of interestparameter of interest

Result will depend on Result will depend on variability of datavariability of data

For example, using For example, using equation for single equation for single population:population:

2

22

dsZ

n

= sample size required = alpha level for specified

level of confidences = standard deviation

n

d = desired precision level (absolute term)

Single Population Sample Size Single Population Sample Size Estimation ExampleEstimation Example

Alpha = 0.1Alpha = 0.1 X = 18.5 % cover of X = 18.5 % cover of

perennial grassperennial grass S = 4.9 % coverS = 4.9 % cover d = (18.5*0.2) = 3.7d = (18.5*0.2) = 3.7 Initial sample = 5Initial sample = 5

8

2.0*5.189.4132.22

22

Need 3 more samples (precision achieved 4.7)Need 3 more samples (precision achieved 4.7)

2

22

dsZ

n

Improving Data QualityImproving Data Quality In order to increase data quality (achieve In order to increase data quality (achieve

sample size) you need to:sample size) you need to: Reduce standard deviation (variability)Reduce standard deviation (variability) Increase the number of samplesIncrease the number of samples

In order to reduce sample size estimates In order to reduce sample size estimates without more samples you can:without more samples you can: Increase alpha (less confidence)Increase alpha (less confidence) Increase precision or MDC (detect a larger Increase precision or MDC (detect a larger

difference)difference)

Graphical Analysis Using Graphical Analysis Using Confidence IntervalsConfidence Intervals

Compare treatment results to quantitative Compare treatment results to quantitative objectives or control areas objectives or control areas

Several types of analysis to fit your situationSeveral types of analysis to fit your situation Types of graphical analysis:Types of graphical analysis:

Comparison of treatment to quantitative standardComparison of treatment to quantitative standard Seeded plants vs. quantitative standardSeeded plants vs. quantitative standard All plants at treatment plots change from time 1 to time 2All plants at treatment plots change from time 1 to time 2

Comparison of two populations (seeded/unseeded)Comparison of two populations (seeded/unseeded) Treatment vs. controlTreatment vs. control Treatment vs. control (change from time 1 to time 2)Treatment vs. control (change from time 1 to time 2)

Flowchart for Graphical Analysis Flowchart for Graphical Analysis DensityDensity

Graphical AnalysisGraphical Analysis (comparison to a standard)(comparison to a standard)

Specify quantitative objectiveSpecify quantitative objective Determine desired alpha level and precisionDetermine desired alpha level and precision Collect data at treatment plotsCollect data at treatment plots Graph mean with confidence interval of Graph mean with confidence interval of

desired width (typically 80 or 90%)desired width (typically 80 or 90%) Graphically compare to quantitative standard Graphically compare to quantitative standard

to determine which situation exists to determine which situation exists Need mean, standard deviation, and nNeed mean, standard deviation, and n Use ES&R Equation spreadsheet to helpUse ES&R Equation spreadsheet to help

Graphical AnalysisGraphical Analysis (comparison to a quantitative objective)(comparison to a quantitative objective)

Quantitative objective: 5 plants/mQuantitative objective: 5 plants/m22

Alpha level: 0.1 (90% Confidence Interval)Alpha level: 0.1 (90% Confidence Interval)

X = 7 plants/mX = 7 plants/m22

S = 1.3 plants/mS = 1.3 plants/m22

N = 5N = 5

D: The sample mean is above the quantitative objective, but the lower limit of the confidence interval is below the objective.

Comparison to a Quantitative ObjectiveComparison to a Quantitative Objective

ObjectiveProbablyNot Met

Vege

tatio

n Pa

ram

eter

0

1

2

3

4

5

Management Objective

Objective Not Met

Objective MayBe Met

Check CI

ObjectiveSurpassed

A B C D

C: The sample mean is above the objective but the lower confidence limit is below the objective.



Vege

tatio

n Pa

ram

eter

0

1

2

3

4

5


Objective Not Met

Objective MayBe Met

Check CI

ObjectiveSurpassed

A B C D

B: The sample mean is below the quantitative objective, but the upper limit of the confidence interval is above the objective.



Vege

tatio

n Pa

ram

eter

0

1

2

3

4

5


Objective Not Met

Objective MayBe Met

Check CI

ObjectiveSurpassed

A B C D

A: The sample mean and confidence interval (CI) fall below the objective. Conclude that the objective has not been met.



Vege

tatio

n Pa

ram

eter

0

1

2

3

4

5


Objective Not Met

Objective MayBe Met

Check CI

ObjectiveSurpassed

A B C D

Graphical AnalysisGraphical Analysis (Treatment v. Control)(Treatment v. Control)

Confidence interval of the difference between the Confidence interval of the difference between the treatment and controltreatment and control

Uses the difference between the means of two Uses the difference between the means of two treatments and constructs a single CI using the variance treatments and constructs a single CI using the variance from both estimates (SE)from both estimates (SE)

A mean of 0 represents no difference between the two A mean of 0 represents no difference between the two treatmentstreatments

Express the quantitative objective as an absolute value Express the quantitative objective as an absolute value or as a multiple of the control.or as a multiple of the control.

Use the mean and CI to make a determination of Use the mean and CI to make a determination of treatment effecttreatment effect


Using the ESR monitoring spreadsheetUsing the ESR monitoring spreadsheet Specify the desired alpha level Specify the desired alpha level Enter the mean, standard deviation, and N from the Enter the mean, standard deviation, and N from the

data collected at the treatment and control plotsdata collected at the treatment and control plots Specify the level of quantitative objective (multiple of Specify the level of quantitative objective (multiple of

control or absolute difference)control or absolute difference) Make interpretation based on graphical analysis Make interpretation based on graphical analysis

of the CI of the difference between the two of the CI of the difference between the two treatmentstreatments


Quantitative objective: twice that of control plot Quantitative objective: twice that of control plot (2) – note that because it is a CI of difference (2) – note that because it is a CI of difference the original amount is subtracted from the original amount is subtracted from quantitative objectivequantitative objective

Alpha level: 0.1 (90% Confidence Interval)Alpha level: 0.1 (90% Confidence Interval) ControlControl X = 2.0X = 2.0 S = 0.5S = 0.5 N = 7N = 7

TreatmentTreatment X = 4.9X = 4.9 S = 0.9S = 0.9 N = 7N = 7

Treatment vs. ControlTreatment vs. ControlVe

geat

ion

Para

met

er

-2

0

2

4

6

8

10

A B C D E F

Ecological

Significance

A: The mean and confidence interval for the difference between the two means is completely above the level of ecological significance (5 plants/m2).

Vege

atio

n Pa

ram

eter

-2

0

2

4

6

8

10

A B C D E F

Ecological

Significance

B: The difference of the mean between is above the level of ecological significance, but the lower confidence limit for the difference is below the level of ecological significance.

Treatment vs. ControlTreatment vs. Control

Vege

atio

n Pa

ram

eter

-2

0

2

4

6

8

10

A B C D E F

Ecological

Significance

C: The difference between the two means is below the level of ecological significance, but the upper confidence limit for the difference is above the level of ecological significance.


Vege

atio

n Pa

ram

eter

-2

0

2

4

6

8

10

A B C D E F

Ecological

Significance

D: The mean and confidence interval of the difference is below the level of ecological significance. Conclude that there is no ecologically significant difference between the control and treatment plots.

Treatment vs. Control Treatment vs. Control

Vege

atio

n Pa

ram

eter

-2

0

2

4

6

8

10

A B C D E F

Ecological

Significance

E: The mean of the difference is above zero, but the lower confidence limit is below 0 (no difference) and the upper confidence limit is above 5 plants/m2.


Vege

atio

n Pa

ram

eter

-2

0

2

4

6

8

10

A B C D E F

Ecological

Significance

F: The mean of the difference is above 0, but the lower confidence limit is below 0 and the upper confidence limit is below the level of ecological significance.


Flowchart for Graphical Analysis Flowchart for Graphical Analysis DensityDensity

Graphical AnalysisGraphical Analysis (Treatment vs. Control T2-T1)(Treatment vs. Control T2-T1)

Confidence interval of the difference in change between Confidence interval of the difference in change between two time periods between treatment and controltwo time periods between treatment and control

Uses the difference between the change in the means of Uses the difference between the change in the means of treatment and control and constructs a single CI using treatment and control and constructs a single CI using the variance from both estimates (SE)the variance from both estimates (SE)

A mean of 0 represents no difference in change A mean of 0 represents no difference in change between the two treatmentsbetween the two treatments

Express the quantitative objective as an absolute value Express the quantitative objective as an absolute value or as a multiple of the control.or as a multiple of the control.

Use the mean and CI to make a determination of Use the mean and CI to make a determination of treatment effecttreatment effect

Plan

ts/m

2

1

2

3

4

5

6

7

8

Control Treatment

Year 1 Year 3

Plan

ts/m

2

1.5

2.0

2.5

3.0

3.5

4.0

4.5

5.0

Control Treatment

Year 1 Year 3

Appears that there is a difference in year three when there actually was not.

Accounting for initial difference in the degree of change

Vege

atio

n Pa

ram

eter

-2

0

2

4

6

8

10

A B C D E F

Ecological

Significance

B: The difference of the mean between is above the level of ecological significance, but the lower confidence limit for the difference is below the level of ecological significance.

Change in Treatment vs. Control Change in Treatment vs. Control

Graphical AnalysisGraphical Analysis (Treatment at two time periods, T2-T1)(Treatment at two time periods, T2-T1)

Confidence interval of the change between the two time Confidence interval of the change between the two time periodsperiods

Treats the two time periods as paired, reducing Treats the two time periods as paired, reducing variabilityvariability

A mean of 0 represents no change between the two A mean of 0 represents no change between the two time periodstime periods

Express the quantitative objective as the desired change Express the quantitative objective as the desired change between the two different time periodsbetween the two different time periods

Use the mean and CI compared to the quantitative Use the mean and CI compared to the quantitative objective to make a determination of successobjective to make a determination of success

Paste graphs directly into reports and describe quantitative results e.g.Perennial Grass DensityThe density of perennial grasses is significantly greater in the treatment plots as compared to the control plots. We are 90% confident that the difference is between 1.06 to 4.54 plants/m2 greater than the control plots with a mean of 2.8 plants/m2)

Confidence Interval of the Difference Between Treatment and Control Plots

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

4.5

5.0

Para

met

erReportingReporting

ReportingReporting Link reports back to quantitative objectivesLink reports back to quantitative objectives Re-assess whether objectives were Re-assess whether objectives were

reasonable and possible reasons for reasonable and possible reasons for success and failure.success and failure.

Make recommendations for future Make recommendations for future improvements to implementation and improvements to implementation and monitoringmonitoring

data quality and analysis strategy for monitoring post-fire rehabilitation treatments

Documents