1

11

Demystifying the Demystifying the Demonstration of Method Demonstration of Method

Applicability (DMA)Applicability (DMA)CLUCLU--IN Studios Web SeminarIN Studios Web Seminar

July 28, 2008July 28, 2008

Stephen Dyment Stephen Dyment USEPA USEPA

Technology Innovation Field Services DivisionTechnology Innovation Field Services Divisiondyment.stephen@epa.govdyment.stephen@epa.gov

Stephen Dyment, U.S. EPA Office of Solid Waste and Emergency Response; Office of Superfund Remediation and Technology InnovationStephen Dyment is a chemist with more than 15 years experience including 4 years in a commercial analytical laboratory and 8 years in environmental consulting. He joined EPA in 2005 with a focus towards enhancing acceptance and use of emerging analytical technologies and sampling strategies. His perspective draws upon years of practical laboratory and field experience to apply EPA’s Triad approach at sites in Superfund, Brownfields, RCRA, UST and State programs. Mr. Dyment’s efforts have resulted in the development of numerous EPA case studies, profiles, and training courses that outline successful strategies for the use and understanding of collaborative data sets, adaptive QC programs, and real time analytics. He holds a B.S. in Environmental Science/Toxicology from the University of Massachusetts at Amherst. Phone: (703) 603-9903 Fax: (703) 603-9135 Email: dyment.stephen@epa.gov

2

22

How to . . . . . . . . . . . . How to . . . . . . . . . . . . Ask questions Ask questions –– ““??”” button on CLUbutton on CLU--IN pageIN pageControl slides as presentation proceedsControl slides as presentation proceeds–– manually advance slidesmanually advance slidesReview archived sessionsReview archived sessions–– http://www.cluhttp://www.clu--in.org/live/archive.cfmin.org/live/archive.cfmContact instructor Contact instructor

LogisticsLogistics--

For those of you joining us via the phone lines, we request that you put your phone on mute forthe seminar. We will have Q&A sessions at which point you are welcome to take your phone offmute and ask the question. If you do not have a mute button on your phone, we ask that you takea moment RIGHT NOW to hit *6 to place your phone on MUTE. When we get to the question.and answer periods you can hit #6 to unmute the phone. This will greatly reduce the backgroundnoises that can disrupt the quality of the audio transmission.Also, please do not put us on HOLD. Many organizations have hold music or advertisements thatcan be very disruptive to the call.

Again, keep us on MUTE. DO NOT put us on HOLD.

Also, if you experience technical difficulties with the audio stream, you may use the ? icon toalert us to the technical difficulties you are encountering. Please include a telephone numberwhere you can be reached and we will try to help you troubleshoot your problem.

3

33

DMA Technical BulletinDMA Technical Bulletin

New publication of Brownfield New publication of Brownfield Technical Support Center (BTSC)Technical Support Center (BTSC)–– http://www.brownfieldstsc.org/publications.cfmhttp://www.brownfieldstsc.org/publications.cfm

15 page technical bulletin describing 15 page technical bulletin describing –– DMA components, benefits, DMA components, benefits,

considerationsconsiderations–– Common products, data evaluationCommon products, data evaluation–– 3 short case studies3 short case studies

AudienceAudience-- technical team members technical team members and stakeholdersand stakeholders

This subject is near and dear to my heart. As a member of Technology Innovation group here at EPA and given my background in analytical chemistry, consulting, and the use of field analytics, to me the DMA helps us overcome some pre-conceived notions about what can and cannot be achieved through the use of field analytics, direct sensing, tools, and innovative sampling strategies. Through the use of DMAs and other Triad best management and technical practices we really hope to show how we can move beyond “screening and definitive” definitions to show how all information sources have value, how when used collaboratively they create more powerful and persuasive data sets, and that many of our commonly used analytical methods/sampling designs/ etc. are not without inherent problems or difficulties. So, what I hope to achieve is a broadening of our knowledge, expansion of the toolbox and to highlight a bridge to harmonious existence of field analytics, direct sensing tools, laboratory methods, etc.

This seminar is really to set the stage for the much anticipated BTSC technical bulletin on DMAs. We are currently finalizing the document and it’s expected release date is in August 2008. Watch for it at http://www.brownfieldstsc.org/publications.cfm

These bulletins are intended for technical project managers and team members. Non-technical managers or stakeholders may also present these bulletins to consultants and service providers to ensure appropriate implementation of Triad best management and technical practices at their site. These bulletins provide sufficient information for less technical project managers and team members to request critical Triad project elements in scope of work and planning documents.

4

44

DMA History DMA History

Concept founded in SWConcept founded in SW--846, performance 846, performance based measurement (PBMS) initiative based measurement (PBMS) initiative http://www.epa.gov/swhttp://www.epa.gov/sw--846/pbms.htm846/pbms.htmInitial siteInitial site--specific performance evaluationspecific performance evaluation–– Analytical and direct sensing methodsAnalytical and direct sensing methods–– Sample design, sample collection techniques, Sample design, sample collection techniques,

sample preparation strategiessample preparation strategies–– Used to select information sources for field and Used to select information sources for field and

offoff--site site Goal is to establish that proposed Goal is to establish that proposed technologies and strategies can provide technologies and strategies can provide information appropriate to meet project information appropriate to meet project decision criteriadecision criteria

“I think that in the discussion of natural problems we ought to begin not with the Scriptures, but with experiments, and demonstrations.”

Galileo Galilei

Of course we all know what happened to Galileo (imprisoned and under house arrest later in life)…..Don’t mess with the Catholic church!The point here is that even with well understood technologies like XRF, MIP, and LIF stakeholders will not rely on these tools and strategies based on faith alone, you often need site specific demonstrations to understand how tools or approaches can be used to effectively manage uncertainty and be used collaboratively with other information sources.

Fits nicely with Triad because PBMS conveys "what" needs to be accomplished, but not prescriptively "how" to do it.EPA defines PBMS as a set of processes wherein the data needs, mandates, or limitations of a program or project are specified, and serve as criteria for selecting appropriate methods to meet those needs in a cost-effective manner. Under a performance-based approach, EPA would specify: Questions to be answered by monitoring. Decisions to be supported by the data. Level of uncertainty acceptable for making decisions. Documentation to be generated to support this approach in the monitoring program.

Data should be collected to meet project specificity, sensitivity, and reliability requirements.

5

55

Why Do You Need a DMA?Why Do You Need a DMA?

Triad usually involves realTriad usually involves real--time time measurements to drive DWSmeasurements to drive DWSGreatest sources of uncertainty Greatest sources of uncertainty usually sample heterogeneity and usually sample heterogeneity and spatial variabilityspatial variabilityRelationships with established Relationships with established laboratory methods often required to laboratory methods often required to make defensible decisionsmake defensible decisionsProvides an initial look at CSM Provides an initial look at CSM assumptions assumptions

How can you manage heterogeneity and spatial variability issues if you don’t know they exists until after you de-mobilize?

DMAs provide an early look at the significance of these issues and allow you to establish effective strategies (QA/QC) to deal with them. Many technologies still struggle with “screening” stigma. XRF is a good example of a very well established technology that many regulators still require fixed laboratory “confirmation”. Stakeholder acceptance often requires this. Deana’s RCRA site in EPA R3.

Relationships allow you the user, confidence in your program. You can develop field based action levels, ranges (clean, dirty, unsure), target collaborative samples, monitor decision error rates, etc. Can help highlight the fact that “gold plated fixed laboratory” results suffer from the same sampling sub-sampling issues that innovative or field methods do. Stakeholders cannot expect field methods to compare any better than the 2 labs or even the same lab.

Most technologies and approaches requiring a DMA result in increased information density to mature CSM. Are your preliminary CSM assumptions holding? Minneapolis example.

Some components of investigation and cleanup such as the Site Investigation (SI) step within the Comprehensive Environmental Response Compensation and Liability Act (CERCLA) or Superfund, where guidance suggests collection of 20 or fewer samples may not be appropriate for conducting a DMA Similarly some

6

66

Is a DMA Always Appropriate?Is a DMA Always Appropriate?

No. . . . . . . . No. . . . . . . .

–– SI guidance indicates <20 samplesSI guidance indicates <20 samples–– Some activities with limited scope or Some activities with limited scope or

resourcesresources–– Projects with adequate resources to Projects with adequate resources to

employ established mobile or fixed lab employ established mobile or fixed lab methods methods at sufficient density*at sufficient density*

During the development of the DMA technical bulletin one excellent comment we received indicated that some projects cannot support the level of effort for a DMA. I agree with that statement, but we also have to keep an eye to likely future site activities.

For most applications a DMA is beneficial precisely because a particular field analytical technique, direct sensing tool, or innovative strategy is identified as applicable to cost effectively increase data density, refine the CSM or address small scale variability and matrix heterogeneity. So it’s not simply a box to check but rather a DMA is a consideration if you believe your site could have these issues (which experience shows us that many sites do).

With that said, at sites with elevated expenditures associated with collection of subsurface samples (deeper contamination), in my view adding limited additional cost associated with field analytics, direct sensing tools, or other innovative technologies does not significantly raise project expenses. Yet these tools can provide greater information density and help target locations for collection of fixed lab samples, placement of monitoring wells, optimizing screen depths/length and so on.

Site Inspection <20 samples. In some cases the selection of these locations is obvious (visual staining, products, lagoons, discharge points) while others are more problematic in determining appropriate locations for these samples. Depending on the nature of the contamination (example metals) some material may be archived for later DMA activities should expanded SI or additional site work be warranted.

Sufficient density is key here, and also those methods may not adequately address sampling uncertainties (1 gram for metals, 30 for organics). Projects using mobile or fixed laboratory methods requiring modifications (shortened run times on GCs, different QC sample frequencies, different reporting packages) or careful examination of sampling and spatial uncertainties may derive significant benefit from a DMA

It should be noted that even projects that are using only fixed lab methods may benefit from a very limited DMA if there are questions regarding matrix interference effects. Just a few pilot samples can help save project resources by detecting extraction, cleanup, or dilution issues at the start of a program.

7

77

Is a DMA Always Appropriate?Is a DMA Always Appropriate?Continued. . . Continued. . .

The BrownfieldThe Brownfield’’s perceptions perception–– ““a property, redevelopment, or reuse a property, redevelopment, or reuse

which may be complicated by the which may be complicated by the presence or potential presence of a presence or potential presence of a hazardous substance, pollutant, or hazardous substance, pollutant, or contaminantcontaminant””

–– Underscores the need for higher density Underscores the need for higher density information, collaborative data setsinformation, collaborative data sets

–– Facilitates stakeholder communication Facilitates stakeholder communication and public presentations and public presentations

Another DMA bulletin comment indicated that a DMA was of limited value at BF sites and was more appropriate for larger more complex SF type site activities. The reality is that we use DMAs on sites across a variety of regulatory programs SF, BF, RCRA, UST, VCUP. In fact many of our BF tech support sites have completed DMAs.

In general sites are evaluated under the BF program because they have an industrial history and the perception of contamination is there in the minds of many stakeholders, particularly concerned citizens groups. Regardless of whether the presence of contamination is real or perceived at such a property, DMAs and the use of high density innovative tools have help us facilitate timely revitalization. The data sets, CSMs, and visualizations associated with DMAs and tools like XRF, LIF, MIP, IA, etc. have help many of our projects to address stakeholder concerns and provide a level of comfort with the process that allows developers, insurance partners, risk partners, public stakeholders, state and local agencies, and other others to be involved, invested, and reassured with a project outcome.

8

88

WhatWhat’’s Involved?s Involved?

There is no template for DMAs!There is no template for DMAs!–– Format, timing, documentation etc. depend Format, timing, documentation etc. depend

heavily on site specifics, existing information, heavily on site specifics, existing information, intended data useintended data use

Performed early in programPerformed early in programGo beyond simple technology evaluation to Go beyond simple technology evaluation to optimize full scaleoptimize full scale–– Sample design, decision and unit designationsSample design, decision and unit designations–– Sample prep, throughput, other logisticsSample prep, throughput, other logistics–– Data management issues Data management issues

DocumentationDocumentation

DMAs can be performed easily and affordably before mobilization, or as an early component of a field program. It does not necessarily require a separate mobilization. I like to say that the complexity of the DMA should be commensurate with the expected complexity and scope of the project and with the expected data use and decisions you are using technologies/strategies to make.

Existing information, archived samples are often extremely valuable.

Best to identify potential issues and design strategies to deal with them early in a program. Also plan for contingencies.

Effective DMAs go beyond simple “does it work at my site” questions to look at sampling, logistical, and data management issues. Communication, data sharing, visualization, collaborative data needs, staffing, project sequencing can all be optimized prior to full scale implementation.

Documentation is key. We have used a variety of formal and informal means to document DMAs. More informal methods such as memoranda of understanding, meeting notes, project websites, E-rooms, and E-bulletin boards also serve to document the DMA process. These informal methods are particularly useful to document stakeholder participation and buy-in for Triad investigations. Informal discussions with stakeholders subsequent to the DMA can also be very useful to accelerate document comment, revision, and submission. Regardless of the method used to document a DMA, good records are essential to scientifically validating and legally defending the selection and use of analytical

9

99

Know Your DecisionKnow Your Decision--A Cautionary TaleA Cautionary Tale

927 927 (wt(wt--averaged)averaged)TotalsTotals

1,9701,970Less than 200Less than 200--meshmesh836836Between 50Between 50-- and 200and 200--meshmesh165165Between 10Between 10-- and 50and 50--meshmesh108108Between 4Between 4-- and 10and 10--meshmesh5050Between 3/8 and 4Between 3/8 and 4--meshmesh””1010Greater than 3/8Greater than 3/8”” (0.375(0.375””))

Pb Conc. in fraction by Pb Conc. in fraction by AA (mg/kg)AA (mg/kg)

Small Arms Firing Range Small Arms Firing Range Soil Grain Size Soil Grain Size

(Std Sieve Mesh Size)(Std Sieve Mesh Size)

Part of the DMA is understanding how your decisions and decision units (dimensions X,Y,Z that the sample represents), exposure pathways, etc drive sampling and analytical pathways. Lead particle size example.

Smaller particles more surface area, also clays tend to have negative charge.

Dust vs. TCLP example.

10

1010

What to Look ForWhat to Look For……..EffectivenessEffectiveness-- Does it work as advertised?Does it work as advertised?QA/QC issuesQA/QC issues–– Are DLs and RLs for site matrices sufficient?Are DLs and RLs for site matrices sufficient?–– What is the expected variability? Precision?What is the expected variability? Precision?–– Bias, false positives/false negatives?Bias, false positives/false negatives?–– How does sample support effect results?How does sample support effect results?–– Develop initial relationships of collaborative Develop initial relationships of collaborative

data sets that provide framework of data sets that provide framework of preliminary QC programpreliminary QC program

Matrix issues?Matrix issues?Do collaborative data sets lead to the Do collaborative data sets lead to the same decision? same decision? Assessing alternative strategies as Assessing alternative strategies as contingenciescontingencies

Effectiveness- Geophysical survey example. Involved with several sites that could have benefited from even simple DMAs. DMAs were not performed for things like GPR, EM, resistivity. Depth issues with GPR, interferences with EM, data processing/interpolation and surveying with resisitvity. A DMA and performance based contracting mechanism would have saved project resources because we ended up paying for results that we not useful.

Given variability seen in DMA can procedures be optimized. How does the variability effect statistically based sampling designs and tolerable uncertainty. Example if you are using 95 UCL to make decisions and there is significant variability present what happens? Often times the 95 UCL is pushed above the action level. In such a case some resources put toward understanding and controlling variability due to small scale or matrix heterogeneity will benefit a sampling design. With a high density real time tool you may be able to isolate problem areas within a decision unit and deal with them rather than taking action on the entire decision unit.

Sample support is the size, shape and orientation of the sample. Is the level of effort required for advanced sample prep worth the higher precision, accuracy, or bias control achieved?QC samples are collected and analyzed to evaluate which uncertainties are the largest contributors to total measurement error. Project resources can then be allocated to control for those activities with the highest impact.

Matrix Issues- Some direct sensing tools have direct push limitationsEx: LIF- calcite and some naturally occurring organic matter fluoresce. Higher fluorescence in sand than clay due to reduced surface area. Ex: XRF- lead arsenic peak overlap, moisture,

Assessing alternative strategies as contingencies should the performance of intended methods prove to be inadequate.

11

1111

More BenefitsMore BenefitsAugment planned data collection and CSM Augment planned data collection and CSM development activitiesdevelopment activitiesTest drive communication and data management Test drive communication and data management schemes, decision Support Tools (DSTs)schemes, decision Support Tools (DSTs)–– Sampling and statistical toolsSampling and statistical tools–– Visualization tools, data management toolsVisualization tools, data management tools

Develop relationships between visual Develop relationships between visual observations and direct sensing toolsobservations and direct sensing toolsFlexibility to change tactics based on DMA rather Flexibility to change tactics based on DMA rather than full implementationthan full implementationEstablish initial decision logic for DWSEstablish initial decision logic for DWSEvaluate existing contract mechanismsEvaluate existing contract mechanismsOptimize sequencing, staffing, load balance, Optimize sequencing, staffing, load balance, unitizing costsunitizing costs

Develop standard descriptions for visual observations.

Difficult to develop decision logic without some knowledge of how analytical tools or sampling strategies will work in the field. Same can be said for contracting mechanisms.

Understanding throughput and other logistics will allow you to balance personnel and optimize field efforts. What are your choke points?

12

1212

Typical DMA ProductsTypical DMA Products--Summary Statistics Summary Statistics

Summary statistics from your DMA can provide an indication of expected DLs/RLs, detection frequencies, mean, median, min/max concentrations, expected SD, 95UCL and more.

Control charts for example (coming up in a few slides) can be based on DMA summary stats.

Scatter plots, box and whiskers, histograms etc.

Evaluate population distribution (normal, lognormal, other). Are there multiple populations? When you compare detections to locations on a map you may find distinct populations related to historical use.

Use DMA information to better design CSM refinement, delineation etc.

13

1313

Typical DMA ProductsTypical DMA ProductsParametricParametric-- Linear regressionsLinear regressions

Linear regressions are most common parametric techniques and often the “gold standard” of method comparison. They are very powerful tools but can be misleading (next series of slides illustrates some potential pitfalls).Parametric statistical methods use assumptions about the data’s underlying shape

of the statistical distribution (normal, lognormal, other). If those assumptions are invalid, the statistical conclusions may not be reliable.

Non-parametric techniques do not require as many assumptions be true, so they are more broadly applicable to the properties of environmental data..

Comparability is quantified by establishing the frequency with which results from different techniques agree with each other with respect to a declared reference point. Different points of reference can be used, but the most common strategy used in Triad projects is establishing comparability with respect to the decision being made on the data. These data may require quantitative comparability (such as if or when two data sets are combined to calculate risk assessment parameters) or qualitative comparability for agreement at the compliant/non-compliant decision threshold.

14

1414

What is a Regression Line?What is a Regression Line?

The scatter-plot in this slide illustrates how a regression analysis works. The data in the lower-right table represents our collaborative data set: four samples, with each having both a traditional laboratory result and a real-time result. Plotting these data give us the scatter-plot shown. Assuming there’s a linear relationship between results generated by the laboratory and results generated by the real-time technique, the question is finding that linear relationship.

The line shown represents the results from a regression using these data. The regression line represents the “best fit” line. “Best fit” here is defined as the line that minimizes the sum of the squared residuals. A residual is the vertical distance separating a regression line and a data point.

If you are planning on using linear regression: 20 or more paired samples is a good rule of thumb.

15

1515

Heteroscedasticity is a Fact of Life for Heteroscedasticity is a Fact of Life for Environmental Data SetsEnvironmental Data Sets

Heteroscedasticity is unfortunately a fact-of-life for environmental collaborative data sets. The LIBS/laboratory scatter-plot illustrates the concept of heteroscedasticity. We can fit a regression line to these data, with the resulting line and its equation shown. The orange lines bracketing the regression line above and below given a sense for how the size of residuals change as concentrations increase. For low concentrations, the scatter-plot points are tightly clustered around the regression line, giving rise to relatively small residuals. As concentrations increase, the “scatter” of points around the line steadily increases. The result is that residuals for higher-concentration points are much larger than what they are for lower concentration values. This increasing residual size as concentrations increase is called heteroscedasticity.

The concept is important because regression analyses often include UCL lines or UTL lines that bracket the regression line. The problem with this is that UCL and UTL calculations derived from a regression analysis are only valid if the underlying data are homoscedastic…which environmental collaborative data never are.

There is a simple physical explanation for heteroscedasticity in environmental collaborative data…analytical error tends to increase as concentrations increase.

16

1616

Appropriate Regression AnalysisAppropriate Regression AnalysisBased on paired analytical results, ideally Based on paired analytical results, ideally from from same subsame sub--samplesamplePaired results focus on concentration Paired results focus on concentration ranges pertinent to decisionranges pertinent to decision--makingmakingNonNon--detects are removed from data setdetects are removed from data setBest regression results obtained when pairs Best regression results obtained when pairs are balanced at opposite ends of range of are balanced at opposite ends of range of interestinterestNo evidence of inexplicable No evidence of inexplicable ““outliersoutliers””No signs of correlated residualsNo signs of correlated residualsHigh R2 values (close to 1)High R2 values (close to 1)Constant residual variance (homoscedastic) Constant residual variance (homoscedastic) is nice but unrealisticis nice but unrealistic

Such an analysis would be based on paired results, ideally with the analytical work done on the same sub-sample where possible to minimize the effects of sample preparation.

Don’t expect XRF to compare better than ICP

The paired results would focus on the concentration range pertinent to decision-making. Often times field analytical methods have a more limited dynamic range within which they provide accurate results. This means that it is unreasonable to expect a good, strong linear relationship for two methods over the complete range of concentrations (which may span several orders of magnitude) present at a site. What is important is to determine whether such as relationship exists over the range in which making decision is important.

Non-detects should be removed from a regression analysis because they will skew regression results. Forcing the instrument to report values <DL or using appropriate substitution methods is better.

The best regression results are obtained when the data used are balanced, i.e., half are at the lower end of interest, and half are at the higher end of interest.So what do we look for in a good regression?

There should be no evidence of outliers. Outliers are points that clearly fall well away from the regression line and appear todifferent than the rest.

Data sets should be balanced.

There should be no signs of correlated residuals. Correlated residuals refer to the situation where a group of points consistently fall above or below the regression line.

We’d like a high R2 value, preferably close to 1 (will range between 0 and 1).

We should have constant residual variance across the concentration range, or in other words the data should be homoscedastic. Unfortunately for environmental collaborative data sets, this is never the case.

So what do we look for in a good regression?

There should be no evidence of outliers. Outliers are points that clearly fall well away from the regression line and appear todifferent than the rest.

Data sets should be balanced.

There should be no signs of correlated residuals. Correlated residuals refer to the situation where a group of points consistently fall above or below the regression line.

17

1717

Example: XRF and LeadExample: XRF and LeadFull data set:Full data set:–– Wonderful RWonderful R22

–– Unbalanced dataUnbalanced data–– Correlated residualsCorrelated residuals–– Apparently poor Apparently poor

calibrationcalibration

Trimmed data set:Trimmed data set:–– Balanced dataBalanced data–– Correlation gone from Correlation gone from

residualsresiduals–– Excellent calibrationExcellent calibration–– RR22 drops significantlydrops significantly

Here’s an example based on XRF analyses of lead in soil samples. The top graphic shows a scatter plot based on the complete data set collected. The regression line has a wonderful R2 value, but has several obvious visual deficiencies. These include unbalanced data (most of it clustered at the low end with only two points at the high end), correlated residuals, and what appears to be a poor calibration for the XRF based on the slope of the line.

The second data set has had its data trimmed to include only those concentrations that fall within the range truly of interest from a decision-making perspective. These data are balanced across the concentration range of interest. The correlations are gone from the residuals. The slope corresponds to what one would expect from a calibrated XRF. Note that the R2 value is actually less, though, then the first example, even though the second regression is clearly superior, underscoring the problems with simply using R2 values as a measure of regression performance and hence field analytic data quality and usability.Slope close to 1, positive y-intercept (XRF vs. ICP digestion efficiency)

Something else to note in the second scatter plot. Notice how the spread of the data around the line increases as concentrations increase. This is called heteroscedasticity…a big word that simply means the variance of our data is not constant over the range of observed concentrations. The presence of heteroscedasticity is a given in environmental data, and complicates the interpretation of regression results. Without belaboring details, the upshot is one needs to be very careful in interpreting UCLs and UTLs for regression lines when heteroscedasticity is present.

18

1818

Small scale variability Small scale variability can impact data can impact data quality more than the quality more than the analytical methodanalytical method

Small scale spatial variability refers to both sampling locations (ft) and within samples (jars or bags). Samples were archives (20-30 grams in a sandwich bag) analyzed for arsenic by ICP in 2005 by Regional lab and again in 2006 by ERT. Correlation coefficients were better for both the Innov-X and Niton instrument cup samples and corresponding ICP analysis.

In this case there was a very low number of non-detects and ½ the detection limit was used as a proxy value for non-detects. Of course as has been discussed in previous modules it is best to force the instrument to report a value for non-detects rather than substitute values.

19

1919

Small scale variability Small scale variability can impact data can impact data quality more than the quality more than the analytical methodanalytical method

Small scale spatial variability refers to both sampling locations (ft) and within samples (jars or bags). Samples were archives (20-30 grams in a sandwich bag) analyzed for arsenic by ICP in 2005 by Regional lab and again in 2006 by ERT. Correlation coefficients were better for both the Innov-X and Niton instrument cup samples and corresponding ICP analysis.

In this case there was a very low number of non-detects and ½ the detection limit was used as a proxy value for non-detects. Of course as has been discussed in previous modules it is best to force the instrument to report a value for non-detects rather than substitute values.

20

2020

Typical DMA ProductsTypical DMA ProductsNonNon--parametric techniquesparametric techniques-- Ranges or binsRanges or bins

For those who don’t know non-parametric methods don’t make assumptions about underlying distribution of contaminants or use the estimation of parameters such as mean or standard deviation.

Non-parametric methods were developed to be used in cases when the researcher knows nothing about the parameters of the variable of interest in the population (hence the name nonparametric). In more technical terms, nonparametric methods do not rely on the estimation of parameters (such as the mean or the standard deviation) describing the distribution of the variable of interest in the population. Therefore, these methods are also sometimes (and more appropriately) called parameter-free methods or distribution-free methods.

Actually it's Th-230. It was collocated with radium, which was amenableto field screening by gamma walkover, however the radium was not atlevels that posed health concerns (or where it was, its health concernswere dominated by Th-230 concerns).

There was no way to get Th-230 directly at its action levels withreal-time techniques. That required ex situ lab analysis by alpha spec,a technique that is fairly complicated and expensive requiringsignificant sample prep/extractions/etc.

21

2121

Comparability A Dirty Word?Comparability A Dirty Word?

Vague references in QAPPsVague references in QAPPs–– PARCCPARCC

How do you measure comparability?How do you measure comparability?

Our use of the termOur use of the term–– The frequency with which results from The frequency with which results from

different techniques agree with respect different techniques agree with respect to a declared reference pointto a declared reference point

How do we establish the use of field analytics, direct sensing tools and other information sources are appropriate to expected decisions. One technique is comparability.

Many of your are likely aware of the PARCC criteria. I think many people struggled with what does comparability mean?Many QAPPs I’ve reviewed address comparability by using terms like “we will ensure comparability by using previously employed analytical methods”.

When we use the term comparability we’re talking about establishing the frequency with which results from different techniques agree with each other with respect to a declared reference point. Different points of reference can be used, but the most common strategy used in Triad projects is establishing comparability with respect to the decision being made on the data. Action levels, decision thresholds etc. These data may require quantitative comparability (such as if or when two data sets are combined to calculate risk assessment parameters) or qualitative comparability for agreement at the compliant/non-compliant decision threshold.

22

2222

Weight of Evidence vs. Weight of Evidence vs. Collaborative Data SetsCollaborative Data Sets

The Triad perspectiveThe Triad perspective–– Weight of EvidenceWeight of Evidence

Combining information from various Combining information from various sources into a holistic picture, advancing the sources into a holistic picture, advancing the CSMCSM

–– Collaborative dataCollaborative dataUsing 2 or more analytical methods to Using 2 or more analytical methods to measure the same compound, analyte, measure the same compound, analyte, surrogate, or class surrogate, or class Using established relationships, one method Using established relationships, one method can be used to inform the user when can be used to inform the user when analysis by another is warranted or analysis by another is warranted or beneficialbeneficial

Many DMAs are looking at issues of comparability but again, it’s my contention that all of these information sources have value and can be brought to bare on a site problem. To that end, some quick definitions here.

Terms such as “weight of evidence” or “multiple lines of evidence” and “collaborative data sets” have been developed to describe these layered data sets. From a Triad perspective, there is a distinction between the two. “Weight (or “lines) of evidence” refer to combining information from various different sources into a holistic picture (i.e., a conceptual site model). For example, historical information may be used in conjunction with geological, hydrogeological, chemical, and geophysical data to predict contaminant fate and transport.

On the other hand, ”collaborative data sets” or “collaborative methods” refer specifically to the strategy of using 2 (or more) analytical methods to measure the “same” analyte or a surrogate of an analyte. For example, total uranium can be measured by X-ray fluorescence (XRF), gamma spectroscopy and alpha spectroscopy. Or PCBs by IA and spectrophotometer, bioassays, GC/ECD, HRGC/MS Collaborative methods are paired so that the strengths of one method can compensate for limitations of the other. Frequently, a field method is selected for its ability to provide a much higher density of data points than an expensive laboratory method. However, the laboratory method will generally achieve better detection limits and accuracy than the field method. A DMA should be designed to guide the “marriage” of the techniques to produce reliable information that is not biased by the effects of heterogeneity or analytical inaccuracy.

Additionally, alternative analytical methods, particularly those that provide results in "real time" can be used to optimize the decision making process. For example, the real-time decisions and high data density possible with field methods can reduce the volume of material removed during cleanup by more precisely defining and confirming the actual contamination footprint. Real-time data can in this way improve decision confidence and limit surprises after a project is complete

23

2323

Typical DMA ProductsTypical DMA ProductsUncertainty evaluationsUncertainty evaluations--

Example: Navy Uncertainty Calculator Example: Navy Uncertainty Calculator http://www.navylabs.navy.mil/Archive/Uncertainty-Calc.xls

Developed by Bill Ingersoll at the Navy

24

2424

Navy Uncertainty Calculator Navy Uncertainty Calculator ContinuedContinued

25

2525

Typical DMA ProductsTypical DMA ProductsQC program worksheetsQC program worksheets

You want to monitor analytes of concern using SRM or standard checks with these control charts. You can spot trends in instrument drift and identify “out of control”situations. Generally the DMA can provide a sufficient number of SRM measurements (30 or more) to develop summary statistics for analytes of concern. Based on the DMA values and summary statistics you can specify an expected mean and standard deviation for each analyte. Specifying an expected mean and standard deviation is considered better because determining these values as the data set progresses can be difficult since outliers can dramatically increase 2SD and 3 SD values and mask potentially “out of control” situations.

In general if 1 measurement exceeds 2 SD, you take it again (using a 95 UCL you would expect 5 out of 100 readings to be outside the +2SD value). If the second reading is within the 2SD of the mean you can continue and monitor for trends. If the second reading is still outside the 2SD window you would initiate corrective action (change battery, re-initialize, re-run SRMs and QC). Likewise, any 1 reading outside the 3SD window would require corrective action. Again, the value in doing this is that you can identify out of control situations as they happen and limit the number of samples that may need to be re-run. If you only do this at the start and end of each day and for example, one of your analytes is >3SD at the end of the day, you will be forced to re-run all samples from that day.

26

2626

Translating DMA Results Translating DMA Results

Develop. . . .Develop. . . .–– Field based action levelsField based action levels–– SOPsSOPs–– QA/QC programsQA/QC programs–– Statistical sampling designStatistical sampling design–– Dynamic work strategiesDynamic work strategies

Decision rules, decision logic diagramsDecision rules, decision logic diagrams

–– ContingenciesContingenciesBased on cost and performanceBased on cost and performance

This is where the rubber hits the road. This where we are taking those valuable lessons learned and translating them into a plan for how we will implement the use of these technologies, sampling strategies, data management plans etc. as the site or program progresses. This is an iterative process, so the data relationships you establish and sampling implications can be refined as you continue but the DMA provides a solid platform from which to do this.

27

2727

Field Based Action LevelsField Based Action Levels

Action levels for field analytics or Action levels for field analytics or direct sensing tools that trigger direct sensing tools that trigger actionaction–– Collection of collaborative dataCollection of collaborative data–– Step outs, additional sampling or Step outs, additional sampling or

analysis, well placement, etc.analysis, well placement, etc.–– Remedy implementationRemedy implementation

RemovalRemovalConfirmation of clean (sometimes required)Confirmation of clean (sometimes required)

28

2828

1 False Negative Error= 5%

3 False Positive Errors=7.7%

59 Total pairs

True Positive 19 Pairs

True Negative 36 Pairs

Field Based Action LevelsField Based Action Levels

Another typical DMA product. We used this for one of our technical support sites to develop field based action levels for XRF. This is a well correlated data set but the concept holds true for less “well behaved data sets”.

If you are more concerned about a Type I or false negative error you could also reduce the XRF field based action level to 350 ppm and decrease the false negative or “false clean” to 0. This would however result in a higher false positive or “false dirty” rate of 13 false positive errors or 22%. You have to weigh the consequences of a false negative vs. the costs associated with excavation or clean up at a rate of 17% false dirty. Many sites default to 5% error for false negative and 10% for false positive.

29

2929

59 Total pairs

10 False Positive Errors= 26% True Positive

20 Pairs

True Negative 29 Pairs

0 False Negative Error= 0%

Typical DMA ProductsTypical DMA Products

This is a well correlated data set. If you are more concerned about a Type I or false negative error you could also reduce the XRF field based action level to 350 ppm and decrease the false negative or “false clean” to 0. This would however result in a higher false positive or “false dirty” rate of 13 false positive errors or 22%. You have to weigh the consequences of a false negative vs. the costs associated with excavation or clean up at a rate of 17% false dirty. Many sites default to 5% error for false negative and 10% for false positive.

30

3030

3 False Positive Errors=7.7%

59 Total pairs

True Positive 19 Pairs

0 False Negative Error= 0%True Negative 26 Pairs

11 Samples for ICP

3 Way Decision Structure With Region of UncertaintyTypical DMA ProductsTypical DMA Products

Structure of a 3 way decision. 19 true positives, 26 true negatives, 3 false positives, and 11 samples for ICP. Region of uncertainty is 350-450 ppm. Below 350 is definitely clean, above 450 is definitely dirty (with 5% false positives)

31

3131

Example Correlations Between

LIF Response and Free Product

Presence of free product unlikely

Presence of free product likely Presence of free

product unlikelyPresence of free product likely

Free Product At >50% Relative Fluorescence for Gasoline

Free Product At >75% Relative Fluorescence for Oil

As you may recall, one of the DMA benefits I mentioned in a previous slide is the ability to correlate direct sensing tool response like LIF relative fluorescence or MIP ECD response to visual and chemical observations.

This example is from one of our technical support sites. In this case a series of LIF pushes were co-located with physical coring. Using visual observations they were able to identify relative fluorescence values that served as a threshold for estimating the presence of free phase petroleum products.

The depth (y axis) also helped to estimate product thickness at locations to optimize a product recovery system.

32

3232

Optimize SOPs and QC ProgramsOptimize SOPs and QC Programs

XRFXRF–– Count times, bags/cups/inCount times, bags/cups/in--situ, sample prepsitu, sample prep–– Frequency of blanks, SRMs, spikesFrequency of blanks, SRMs, spikes

FluorescenceFluorescence-- LIF, FFD, exLIF, FFD, ex--situ situ –– Drilling platforms, fluorescence signaturesDrilling platforms, fluorescence signatures–– Frequency and response thresholds for Frequency and response thresholds for

collection of collaborative samplescollection of collaborative samples

ImmunoImmuno--assay/ bioassay/ bio--assayassay–– Use of composites, MIS, extract volumeUse of composites, MIS, extract volume–– Frequency of blanks, spikes, collaborative Frequency of blanks, spikes, collaborative

samplessamples

33

3333

DMA Data to QC programDMA Data to QC programQC program worksheetsQC program worksheets

You want to monitor analytes of concern using SRM or standard checks with these control charts. You can spot trends in instrument drift and identify “out of control”situations. Generally the DMA can provide a sufficient number of SRM measurements (30 or more) to develop summary statistics for analytes of concern. Based on the DMA values and summary statistics you can specify an expected mean and standard deviation for each analyte. Specifying an expected mean and standard deviation is considered better because determining these values as the data set progresses can be difficult since outliers can dramatically increase 2SD and 3 SD values and mask potentially “out of control” situations.

In general if 1 measurement exceeds 2 SD, you take it again (using a 95 UCL you would expect 5 out of 100 readings to be outside the +2SD value). If the second reading is within the 2SD of the mean you can continue and monitor for trends. If the second reading is still outside the 2SD window you would initiate corrective action (change battery, re-initialize, re-run SRMs and QC). Likewise, any 1 reading outside the 3SD window would require corrective action. Again, the value in doing this is that you can identify out of control situations as they happen and limit the number of samples that may need to be re-run. If you only do this at the start and end of each day and for example, one of your analytes is >3SD at the end of the day, you will be forced to re-run all samples from that day.

34

3434

Optimize Statistical Optimize Statistical Sampling DesignSampling Design

Characterize or Characterize or verify cleanverify clean--upup

Statistical Statistical confidence desiredconfidence desired

How close to each How close to each other are the true other are the true mean and ALmean and AL

How much How much variability is present variability is present in soil in soil concentrationsconcentrations

Many of you may be aware of tools like visual sampling plan.

One of the acknowledged pitfalls associated with using classical statistical tools in sampling design is that project teams seldom have a sound estimate of total measurement error for use in establishing sample quantities, grid sizes, etc. With results from a DMA, project teams can use classical statistical tools (such as the Visual Sample Plan software, http://vsp.pnl.gov/) more effectively in sampling design because they have generated site-specific method error information.

35

3535

Statistical Sampling DesignStatistical Sampling Design

36

3636

Decision Logic Decision Logic

37

3737

XRF Simple Decision Rule ExampleXRF Simple Decision Rule Example

Bagged samples, measurements through Bagged samples, measurements through bagbagNeed decision rule for measurement Need decision rule for measurement numbers for each bagnumbers for each bagAction level: 25 ppmAction level: 25 ppm3 bagged samples measured 3 bagged samples measured systematically across bag 10 times eachsystematically across bag 10 times eachAverage concentrations: 19, 22, and 32 Average concentrations: 19, 22, and 32 ppmppm–– 30 measurements total30 measurements total

38

3838

Example (cont.)Example (cont.)XRF Result Frequency versus concentration

0

1

2

3

4

5

6

7

8

9

<10 10-15 15-20 20-25 25-30 30-35 35-40 40-50 >50

ppm

Res

ult F

requ

ency

Simple Decision Rule:

• if 1st measurement less than 10 ppm, stop, no action level problems

• if 1st measurement greater than 50 ppm, stop, action level problems

• if 1st measurement between 10 and 50 ppm, take another three measurements from bagged sample

39

3939

ContingenciesContingencies

DMA’s also support development of contingencies for technologies and drilling platforms. This is a subset from a much more comprehensive decision logic diagram from some Triad work completed at Hurlburt Field. This portion of the decision logic focused on contingencies whose necessity was determined during a DMA. The concern was to have drilling platform contingencies should the MIP (on a direct push platform) have issues with refusal. The contingencies included options for sonic drilling, core analyses (in lieu of CPT information, lab analyses, well placement, well screen intervals, etc.

A DMA can often provide information on the likelihood of field analysis or drilling issues that would in turn necessitate the need to plan for contingencies. This is particularly helpful for dynamic work strategies where the goal is to meet project objectives in one or limited mobilizations while maximizing real time information.

40

4040

EPA TIFSD EPA TIFSD DMA Lessons LearnedDMA Lessons Learned

Linear regressionLinear regression-- can be helpful or can be helpful or misleading misleading HeterogeneityHeterogeneity-- large scale, small scale, large scale, small scale, and within sampleand within sample–– DonDon’’t expect collaborative data to compare any t expect collaborative data to compare any

better than 2 labs or even the same labbetter than 2 labs or even the same lab

Focus on decision quality Focus on decision quality Structure vendor contracts to include Structure vendor contracts to include some DMA principles some DMA principles Particular instruments Particular instruments ≠≠ technology technology generalizationsgeneralizations

These lessons are based on experiences from our technical support sites. Regression: Don’t just focus on R2 valueIf you are planning on using linear regression: 20 or more paired samples is a good rule of thumb. Based on paired analytical results, ideally from same sub-samplePaired results focus on concentration ranges pertinent to decision-making (dynamic range over which calibrations hold)

Non-detects are removed from data set. XRF can force instruments to report values <DL or use appropriate substitution methods.

Best regression results obtained when pairs are balanced at opposite ends of range of interest

No evidence of inexplicable “outliers”

No signs of correlated residuals

High R2 values (close to 1)

Constant residual variance (homoscedastic) is nice but unrealistic. Watch out when using UCLs UTLs.

Standard laboratory data can be “noisy” and are not necessarily an error-free representation of reality. In fact several DMAs I have been involved with resulted in finding laboratory issues based on comparability with field analytics. We tend to get hung up on issues like “how close is the field result to the lab” when a DMA can identify lab issues or mistakes, sampling issues, sub-sampling issues etc. that have far greater project impacts than solely focusing on those field analytics.

Some vendors may participate for free. Example sending material to vendors to see if it fluoresces. Ask to see what outputs you will be getting and using to make decisions. As an example, there are vary degrees of complexity for MIP vendor outputs. We don’t endorse specific vendors but from personal experience there can be a big difference in (spreadsheets vs. real-time 3D visualizations). In fact one of the advantages of a technology like MIP is the ability to drive DWS and facilitate stakeholder discussions in real time. That can be difficult when the out put is a spreadsheet, or the information is provided in days or weeks such that it cannot be used effectively in the field. When used in a static fashion, a tool like MIP or LIF becomes less effective.

Focus on Decision quality not necessicarly laboratory method comparability. Are collaborative data sets or multiple lines of evidence supporting the same decision? Clean, dirty, more information needed (in real time of course). Remember parametric and non-parametric techniques available, so some DMA data sets with weaker correlations may still be useful for decision support. They can also be indicative of other sampling, matrix heterogeneity, or small scale variability issues.

Contract language can really help avoid issues of paying for services that don’t work as advertised. Ask vendors to demonstrate technologies (example GPR).

XRF DMA- now looking for instrument? Not all XRFs, MIPS, LIFs, vendors are created equal.

41

4141

DMA Example DMA Example

Wenatchee Tree FruitWenatchee Tree Fruithttp://cluin.org/download/char/treefruit/wtfrec.pdfhttp://cluin.org/download/char/treefruit/wtfrec.pdf

–– Tree fruit test plot contaminated with Tree fruit test plot contaminated with OC, OP, and other pesticide compoundsOC, OP, and other pesticide compounds

–– DMA supported integrated DMA supported integrated characterization, removal, segregationcharacterization, removal, segregation

–– DMA looked at IA test kits and fixed DMA looked at IA test kits and fixed laboratory methodslaboratory methods

15 years old but captures many good DMA and Triad concepts.

42

4242

DMA Example DMA Example

Wenatchee Tree FruitWenatchee Tree Fruit-- ResultsResults–– High bias of IA kits confirmed. Used to High bias of IA kits confirmed. Used to

develop field based action levels develop field based action levels 5 ppm DDT kits5 ppm DDT kits-- indicative of potential exceedence indicative of potential exceedence for DDT, DDD, or DDEfor DDT, DDD, or DDE0.1 ppm cyclodienes0.1 ppm cyclodienes-- indicative of potential indicative of potential exceedence for dieldrin or endrinexceedence for dieldrin or endrin

–– Modifications made to both IA and fixed lab Modifications made to both IA and fixed lab methods to meet data needs/decision criteriamethods to meet data needs/decision criteria

–– DMA and subsequent results used to adjust DMA and subsequent results used to adjust DDT IA field action level to 10 ppm with DDT IA field action level to 10 ppm with regulator approvalregulator approval

–– No false negative decision errors, low No false negative decision errors, low percentage of false positive errorspercentage of false positive errors

The DMA confirmed that the IA test kits were intentionally biased 100% high by the manufacturer to reduce the chance of false negative results. The DMA also had to account for the fact that the kits respond to structurally similar compounds beyond the target compounds. Taking into consideration the high bias and correlations with fixed laboratory results, the DMA determined that DDT test kit result exceeding 5 ppm could indicate an exceedence for DDT, DDE, or DDD. Likewise a cyclodiene field action limit of 0.1 ppm indicated the possibility that regulatory action levels for endrin or dieldrin were exceeded.

PBMSSeveral modifications to the IA kit procedures were made based on DMA results. Pure methanol was used instead of a water methanol mix and extraction volumes were doubled to 20 mL to bracket action levels based on cross reactivity/sensitivity results. The resulting 20 ml extracts were sufficient to run both the DDT and cyclodienes IA analyses.

Some fixed laboratory detection methods for collaborative data were also modified. For the organophosphorus pesticides, mass spectrometry (MS) detection was used instead of the method specified nitrogen/phosphorus detector (NPD) to improve selectivity and meet project required quantitation limits. For the carbamate pesticides a gas chromatography (GC) NPD method was used instead of high pressure liquid chromatography (HPLC) to reduce interference and the surrogate compound was changed to a compound rarely used in agricultural applications. Non-target compounds and tentatively identified compounds (TICs) from fixed laboratory methods also became crucial to understanding IA kit response from a broad range of contaminants.

No false negative decision errors with respect to action levels for individual pesticides were encountered. A low percentage of false positive errors (usually associated with of the presence of endosulfans in the samples) were encountered during the project. Use of the DMA and Triad principles resulted in an estimated cost savings of 50% for total project costs.

43

ResourcesResources

EPAEPA--Technology Integration and Technology Integration and Information Branch Information Branch http://www.cluhttp://www.clu--in.org/tiomiss.cfmin.org/tiomiss.cfmCase studies Case studies http://www.triadcentral.org/http://www.triadcentral.org/US EPA Technical BulletinUS EPA Technical Bulletin-- ““Performing Demonstrations of Method Applicability Under a Triad Approach”-- Due out August 2008Due out August 2008Examples on the US Triad CoP website, Examples on the US Triad CoP website, discussions with practitionersdiscussions with practitioners

43

44

4444

Questions?Questions?

45

4545

Thank YouAfter viewing the links to additional resources,

please complete our online feedback form.

Thank You

Links to Additional Resources

Feedback Form