Measurement of Uncertainty – One Lab’s Experience

Patricia Hanson

Biological Administrator I

Florida Department of Agriculture and Consumer Services, Food Safety, Microbiology Laboratory

What is Uncertainty

•Quality of a Measurement

•“Give or Take”

•Quantification of Doubt

Sources of Uncertainty

•Analyst Experience and Skill Level•Equipment Stability•Environmental Factors•Integrity and Composition of the Sample•Process of Taking Measurements and

Interpreting Data•Random Error

Error vs. Uncertainty•Error is the difference between the

measured value and the true value. An example of error is a correction factor that is applied to a thermometer. Errors that are corrected for are not included in Measurement of Uncertainty.

Error vs. Uncertainty•Uncertainty is a measure of errors that

can not be corrected. These may be unknown or known. An example of uncertainty would be two competent analysts independently analyze the same sample by the same method but come up with different numbers.

Mistakes and Failures•Mistake: method is not followed as

written

•Failure: a component of the method such as a reagent or a piece of equipment does not perform as expected

•Data associated with known mistakes or failures should be omitted from the Measurement of Uncertainty

Case Study #1

An Unforeseen Source of Variation

An Unforeseen Source of Variation • Two different technicians ran the water membrane

filtration method for coliform enumeration.• They switched off every few weeks.• They used the same process control.• There was a notable shift in the process control

chart when analysts switched off.• The technicians were observed by a senior analyst

to both have acceptable pipetting techniques.• Both technicians consistently passed their

Proficiency Tests.

Why the Variation?• The obvious source of variation was the

technicians.• While there were not identifiable errors or

mistakes, there was a level of uncertainly in how each technician prepared the serial dilution of the process control.

• It was amplified because of the number of serial dilutions that needed to be made to dilute the overnight culture to get it down to the countable range.

Resolution•Since the dilution steps were not performed

on the samples, the variation introduced from the serial dilution of the control should not be part of the Measurement of Uncertainty for the method.

•A different means of achieving the correct count in the process control had to be found. We chose to use pellets with a known count.

Case Study #2

A Missing Source of Variation

Missing Sources of Variation•The lab had just validated a new method

for Staphylococci enumeration.•There were not enough data points from

the validation to establish the control chart.

•Each of the four trained analysts ran several pellets to obtain 15 additional data points over two days.

•The control chart based on these data points had unusually tight limits.

Where was the Variation?• In this case, not all the sources of uncertainty were

taken into account.• The data collection process was repeated with each

of the same four analysts running process controls over a six week period.

• The Standard Deviation of the 15 points set up within two days was ½ the Standard Deviation of the 15 points that were set up over 6 weeks.

• Sources of Uncertainty that are introduced by running a procedure over a period of time were being missed.

Lessons Learned from Case Studies• Do not introduce extra sources of variation• Do include all sources of variation

• Samples used for MU calculation should, as closely as possible, follow the same procedure as your samplesInclude MatrixDifferent AnalystsDifferent EquipmentDifferent DaysAppropriate Analyte LevelFollow Method as Closely as Possible

Where to get data for MU

•Process Control Samples*

•Proficiency Test Samples

•Duplicate Samples

Microbiology Enumeration Method Challenges

•The sample matrix needs to be free of target

•Solution: Use a sterile surrogate matrix

•Limitation: Sterile samples/surrogate matrices do not have background microflora that are present in samples

Proficiency Test Samples for MU

•Proficiency Test (PT) samples can be used to determine Measurement of Uncertainty

•Use of PT samples typically results in a small data set, even if PTs are run monthly, there are only 12 points per year

•PT samples often come pre-weighed or in pellet form and do not go through the same steps as the samples

Process Control Samples for MU•The process control is run through the

entire method by each analyst on different days so most of the variability from the method is captured

•Lots of data points are collected

•This data typically is already being collected in the laboratory so it is readily available for Measurement of Uncertainty Calculations

Challenges of using Process Control Pellets• The “claimed” count per pellet must be verified

in house• This normally differs from the “claimed” count

on the vial• The test method can not be used to determine

the count because of the uncertainly of the test method

• Use of non-selective agar for counts• Count several pellets and use an average,

discard outlying counts

Percent Recovery Data•Fairly straight forward once you have an

established count (spike value) for your inoculated organism

•(log10T / log10S) x 100

where T = test value (result from your PC)

S = spike value (count from your pellet)

Determine MU from Percent Recovery Data

•Take the Standard Deviation (SD) of all Process Control Percent Recovery Data Points.

•Multiply by the coverage factor, k for samples sets with 30 or more data points k=2

for a 95% confidence levelfor sample sets with fewer than 30 data points

there is a table: “t-statistic for 95% confidence” that will give you the value for k for your data set

The fewer numbers in your data set, the higher the value for k because fewer data points may not capture all of the uncertainty

Recovery Replicates Data•Advantage: no need to determine the

“absolute” count to determine percent recovery

•Disadvantage: cost and time of additional replicates

•Caution: should be two separate samples taken through the procedure, not one sample that is just analyzed in duplicate

Microbiology Qualitative Method Challenges•Far fewer challenges and limitations than

enumeration methods when it comes to picking a matrix

•The sample matrix does not need to be free of target

•Either a sterile surrogate matrix or a spiked sample can be used

•The challenges come when determining a meaningful Measurement of Uncertainty

Microbiology Semi-Quantitative Methods•What we do

Use false positive and false negative rates

False positive rate is based on number of un-spiked samples that screen positive but are non-culturable vs number of samples that screen positive and are confirmed.

False negatives are based on number of PT and Process Control failures

Microbiology Semi-Quantitative Methods•Why we do it

Provides the most useful information about our method

False positive rate takes into consideration sample composition – matrix and background microflora challenges to confirmation

Microbiology Semi-Quantitative Methods•What we don’t do and why

We do not use kit controls as these don’t take into account sample composition and enrichment procedure.

We do not use OD or Ct values. These are useful for tracking trending in a method. For our purpose, as long as a sample is beyond the established cut off it is considered a positive screen. How far from the cutoff the sample is does not have an impact on if the lab pursues confirmation.

Food Chemistry Methods

•Food Chemistry is more straight forward

•Can use spikes or known positive samples

When to use a Combined Uncertainty

•When your control sample does not go through all the steps of your method

•When you can independently measure the uncertainty of each step

•Combined Uncertainty is calculated by root sum of squares of each component

What Methods we do not have a MU for•Cultural qualitative methods•Rapid qualitative methods where there is

only a detection/non-detection result with no confirmation step

•Cultural methods that use rapid or semi-quantitative methods as a step in the method but that step does not factor into the final result

Practical Uses for Method MU’s•Method Evaluation

•PT Evaluation

•Analyst Evaluation

•Regulatory Determination

References• Guidelines for Estimating Uncertainty for

Microbiological Counting Methods. American Association for Laboratory Accreditation, September 3, 2014.

• Hammack, Stacie. Measurement of Uncertainty. Bureau of Food Laboratories Laboratory Quality Management System FL QA 125, Version 4.3. Bureau of Food Laboratories, Division of Food Safety, Florida Department of Agriculture and Consumer Services, February 6, 2015.

• Bell, Stephanie. A Beginner’s Guide to Uncertainty of Measurement. Measurement Good Practice Guide No 11, Issue 2. National Physical Laboratory, August 1999.

Questions

Group Activity•Determine how you would compute the

Measurement of Uncertainty for the following method:Standard Total Coliform Fermentation

Technique. Standard Methods For the Examination of Water and Wastewater, 22nd Edition, 9221 B.

Procedure•Sample: 100ml vended drinking water

•Dispense 10ml sample into each of ten 2x Lauryl Tryptose Broth LBT tubes, incubate and check for growth and gas

•Transfer one loopful from each positive LTB tube to one Brilliant Green Lactose Bile (BGLB) broth tube and one loopful to one EC Broth tube, incubate and check for growth and gas

Interpretation of Results•Gas and Growth in LST and BGLG =

confirmed coliform tube

•Gas and Growth in LST, BGLB, and EC = confirmed fecal coliform tube

•Look up number of positive tubes on 10 tube MPN table to determine result

10 Tube MPN Table# Positive Tubes MPN/100ml

0 <1.1

1 1.1

2 2.2

3 3.6

4 5.1

5 6.9

6 9.2

7 12

8 16

9 23

10 >23

Available Process Control Organisms• Overnight Culture of Escherichia coli:

approximately 2 x 109 cfu/ml (positive in all three broths)

• Overnight Culture of Enterobacter aerogenes: approximately 2 x 109 cfu/ml (positive in LTB and BGLB broths)

• Microbiologics® Pellet of Escherichia coli: approximately 2 x 103 cfu/pellet (positive in all three broths)

• Microbiologics® Pellet of Enterobacter aerogenes: approximately 2 x 103 cfu/pellet (positive in LTB and BGLB broths)

Points to Consider• What steps in the method have the potential

for uncertainty?• Can you take a process control through all of

these steps or do you need to do a combined uncertainty?

• Can you minimize the potential for adding uncertainty to the process control that is not found in the samples?

• Can you construct a process control that is in the same countable range for the method (1.1 to 23 cfu)?