is it method verification or just - 2018 albuquerque · 2019-11-07 · is it method verification or...
TRANSCRIPT
Is it Method Verification or Validation, or Just Semantics?
Michael BrodskyBrodsky Consultants
Objectives
• To define and differentiate method validation from verification
• Consider the requirements for method validation and expectation for in‐house verification from an ISO Standard 17025 perspective
• Review AOACI OMA and RI validation protocols• Outline the requirements for a verification protocol that defines a method’s fitness‐for purpose
What is Test MethodValidation?
– Validation is the establishment of one or more performance characteristics for a test method
– By single laboratory validation– By multi‐laboratory collaborative study – Always by comparative analysis to a “Reference” Method
Performance Characteristics of Microbiological Methods
• Relative Accuracy (%Recovery)• Precision (Repeatability and Reproducibility)• Specificity (Selectivity for target analyte)• Sensitivity (Distinguishing target from non target)• Inclusivity (Range of target analytes detected by method) • Exclusivity (Range of non‐target analytes excluded)• False Positive and False Negative Rates
– Positive and Negative deviation• Limits of Detection (LOD)• Limit of Quantitation (LOQ)• Scope of application
Categories of Microbiological Test MethodsPerformance Characteristics Included in a Validation Study
Performance Characteristic Identification Quantitative Qualitative(P/A)
Relative Accuracy No Yes No
Matrix Effects (Scope) No Yes Yes
Precision No Yes No
Sensitivity Yes Yes Yes
Specificity Yes Yes Yes
Inclusivity Yes Yes Yes
Exclusivity Yes Yes Yes
False Positive Rate No Yes Yes
False Negative Rate No Yes Yes
LOD No Yes Yes
LOQ No Yes No
Ruggedness Yes Yes Yes
Linearity/Range No Yes No
Categories of Microbiological Test MethodsPerformance Characteristics Included in the Verification of a Validated Method
Performance Characteristic
Identification Quantitative Qualitative(P/A)
Verification(Where
Applicable)
Relative Accuracy No Yes No Yes
Matrix Effects (Scope) No Yes Yes Yes
Precision No Yes No Yes
Sensitivity Yes Yes Yes No
Specificity Yes Yes Yes Yes
Inclusivity Yes Yes Yes No
Exclusivity Yes Yes Yes No
False Positive Rate No Yes Yes No
False Negative Rate No Yes Yes No
LOD No Yes Yes No
LOQ No Yes No No
Ruggedness Yes Yes Yes No
Linearity/Range No Yes No No
17025 Requirements
• Methods requiring validation are:– Modified official methods– In‐house developed methods – Methods extended to a component, analyte or matrix not previously tested or included in validation
– Changes involving new technology or automation
Food Categories and Types
To help protect your privacy, PowerPoint has blocked automatic download of this picture.
Food Categories and Types(Based on Physiochemical and Innate Microbial
Characteristics)
Food Categories/Matrices1. Raw milk and dairy products2. Heat processed milk and dairy products3. Ready‐to‐eat, ready‐to‐reheat meat products4. Raw poultry and ready‐to‐cook poultry products5. Ready‐to‐eat, ready‐to‐reheat meat poultry products6. Eggs and derivatives7. Raw and ready‐to‐cook fish and seafoods (unprocessed) 8. Ready‐to‐eat, ready‐to‐reheat fishery products9. Fresh produce and fruit10. Processed fruits and vegetables11. Infant formula and infant cereals12. Dried cereals, fruits, nuts, seeds and vegetables 13. Chocolate, bakery products and confectionary 14. Multi‐component foods or meal components15. Pet food and animal feed
16. Environmental samples (food or feed production)
1. Raw milk and dairy products :
Types: i. Raw milks and/or fermented/acidified
milks (not heat treated) ii. Raw milk based products, with high fat
content and/or high background microflora
2. Heat processed milk and dairy products
Types:
i. Pasteurized dairy productsii. Sterilized or UHT dairy products iii. Pasteurized milk based productsiv. Dry
3. Ready‐to‐eat, ready‐to‐reheat meat products:
Types: i. Cooked meat products ii. Fermented or dried meat products iii. Raw cured (smoked) (aw > 0.92 iv. Raw cured (smoked) (aw < 0.92) v. Canned meat (ambient stable)
4. Raw poultry and ready‐to‐cook poultry products
Types:
i. Fresh (unprocessed) ii. Ready‐to‐cook products
(processed)
5. Ready‐to‐eat, ready‐to‐reheat meat poultry products
Types:i. Cooked products ii. Fermented or dried products iii. Raw cured (smoked) (aw > 0.92)iv. Raw cured (smoked) (aw < 0.92) v. Canned (ambient stable)
i.
6. Eggs and derivates:
Types: i. Eggs (unprocessed) ii. Egg products (heat processed) with additives
(salt or sugar > 2%) iii. Egg products (heat processed) without
additives iv. Dry
7. Raw and ready‐to‐cook fish and seafood (unprocessed)
Types: i. Fish (unprocessed) ii. Shellfish (unprocessed) iii. Crustaceans (unprocessed) iv. Ready‐to‐cook fish and seafoods (processed)
8. Ready‐to‐eat, ready‐to‐reheat fishery products
Types: i. Cooked products ii. Acidified and marinated products iii. Smoked or cured, and other processed
products (aw > 0.92) iv. Smoked or cured, and other processed
products (aw < 0.92) v. Canned (ambient stable fish)
9. Fresh produce and fruitTypes:
i. Cut ready‐to‐eat fruit ii. Cut ready‐to‐eat vegetables iii. Produce grown in or in contact with the
ground iv. Sprouts v. Raw fruit/vegetable juices (unpasteurized)
vi. Leafy greens vii. Vegetables and fruits (unprocessed) not
described above
i.
10. Processed fruits and vegetablesTypes:
i. Heat processed fruit/vegetables juices ii. Canned fruits and vegetables (ambient stable)
iii. Heat processed vegetables and fruits iv. Fermented/acidified vegetables
i.
11. Infant formula and infant cereals
Types: i. Probiotic ingredients ii. Non probiotic ingredients iii. Non probiotic infant formula iv. Probiotic infant formula v. Non probiotic infant cereals vi. Probiotic infant cereals
i.
12. Dried cereals, fruits, nuts, seeds and vegetables
Types: i. Low and IMF fruits (aw < 0.85) ii. Seasonings iii. Nuts and seeds iv. Dried fruits and vegetables (aw < 0.60) v. Dried cereals vi. Flours
i.
13. Chocolate, bakery products and confectionary
Types: i. Pastries ii. Dry powdered iii. Low moisture iv. Dry & sugared low moisture (aw < 0.85) v. Dry & sugared low moisture (aw < 0.65)
i.
14. Multi‐component foods or meal Components
Types: i. Composite foods with substantial raw ingredients (excluding
patisserie) ii. Composite processed foods (cooked) iii. Ready to (re)heat food: refrigerated iv. Ready to (re)heat food: frozen v. Ready to (re)heat food: ambient stable (canned vi. Ready to (re)heat food: dry vii. Mayonnaise based delisalads (acid) with raw ingredients viii. Mayonnaise based delisalads (acid) with processed ingredients ix. Ambient stable acid foods (pH < 4.8)
i.
15. Pet food and animal feed
Types: i. Animal origin ingredients ii. Plant origin ingredients iii. Other ingredients iv. Dry food (aw ≤ 0.7) v. Wet food (aw > 0.7) vi. Canned vii. Animal feeds (bovine, ovine, pig) viii.Animal feeds (poultry) ix. Animal feeds (fish) i.
i
16. Environmental samples (food or feed production)
Types: i. Equipment or production environment
ii. Water used in the manufacturing process
i.
i.
Method Validation
AOAC International
Performance Tested Methods Program
• Designed specifically to validate commercial test kits• Speed and efficiency designed into the review process
• Single Laboratory Validation with an independent laboratory
• Test kit methods are reviewed annually• Harmonized to determine the same performance characteristics as the OMA Pre‐collaborative study.
AOACI Pre‐Collaborative/PTM Study Design (SLV) (Quantitative Method)
– 9 Food categories (broad range of foods)• ~ 20 foods (~2 food types/category)(5 portions of each)
– Low, medium and high levels of contamination and uncontaminated
– Comparison of recovery to reference method using ANOVA and comparison of means
• Inclusivity – >50 strains of target microorganism (>100 serovars for
Salmonella)
• Exclusivity– >30 non‐target strains
AOACI Pre‐Collaborative/PTM Study Design (SLV) (Qualitative Method)
– 30 analyses for each food type:• 5 replicate test portions per level the high inoculation level,
• 20 for the fractional positive level• 5 for the uncontaminated level
AOACI Collaborative Study (Qualitative Method)
• 12 ‐ 15 laboratories– Minimum of 10 labs with acceptable data
• Test samples– 1 matrix– 12 test portions per high analyte level– 12 test portions per fractional Positive samples– 12 uncontaminated test portions
• Comparison to reference method using Chi‐square analysis &/or Probability of Detection (POD models)
"I think you should be more explicithere in step two."
Qualitative Performance Characteristics(Paired Samples)
Reference Method
TestMethod
a
c d
b+
_
+ _
+ or - : possible outcome of the test
Performance Characteristics for a Qualitative Method
Sensitivity = a/a+cSpecificity = d/b+dFalse Negative Rate = c/a+c False Positive Rate = b/b+dPositive Predictive Value= a/a+bNegative Predictive Value = d/c+d
Probability of Detection (POD)• The proportion of positive analytical outcomes for a qualitative method for a given matrix at a given analyte level or concentration. – The POD estimate is calculated as the number of positive outcomes divided by the total number of trials.
– POD is concentration dependent.
• Several POD measures can be calculated; – PODR (reference method POD), – PODC (confirmed candidate method POD), – PODCP (candidate method presumptive result POD) – PODCC (candidate method confirmed result POD).
Difference of Probabilities of Detection (dPOD
• Difference of probabilities of detection is the difference between any two POD values.– dPODCP is the difference between the candidate presumptive result PODCP and the candidate confirmed result PODCC values.
• dPODCP = PODCP – PODCC
– dPODC is the difference between the confirmed candidate method PODCC and reference method PODR values.
• dPODC = PODC – PODR
POD Interpretation– Calculate the 95% confidence interval on each dPOD
• If the confidence interval of a dPOD does not contain zero, then the difference is statistically significant at the 5% level
• The Excel® spreadsheet for calculating RLOD values is freely available for download at: http://standards.iso.org/iso/16140
AOACI Collaborative Study ‐Quantitative
• 10 ‐ 12 laboratories– Minimum of 8 labs with acceptable data
• Test samples– 1 Matrix– Low, medium and high contamination +uncontaminated– Five samples tested at each level– Both naturally and artificially contaminated food– Compare against a reference method– Repeatability, Reproducibility and differences between
means
Method Verification
ISO/IEC17025 Requirements
• Official reference methods already published and intended for a specific matrix will be incorporated into the current method document format.
• These official methods do not need to be fully validated.
• The ability to perform the analysis must be verified using spiked samples, proficiency samples or CRMs
Method Verification
Laboratory demonstrates the capability of achieving specified method performance characteristics before implementation and on an on‐going basis.
Provides data to illustrate that the method is fit‐for (it’s)‐purpose i.e. meets customer requirements
Includes determination of uncertainty of measurement (quantitative methods).
Verification of QuantitativeMicrobiological Methods
• Precision (Repeatability and Reproducibility)• Scope of application• Uncertainty of Measurement• Fitness‐for‐Purpose
Validation Data from Collaboratively Studied QuantitativeMethods
Provides reference values for RSDR and RSDr
e.g. Pour Plate counting (SMEDP) e.g. RSDr ≤ 7.7% (0.077) (within analysts)e.g. RSDR ≤ 18.2% (0.182) (between analysts)
Validation Data from Collaboratively Studied QuantitativeMethods
Can also be used to estimate uncertainty of measuremente.g. Pour Plate counting (SMEDP) e.g. RSDr ≤ 7.7% (0.077) (within analysts)e.g. RSDR ≤ 18.2% (0.182) (between analysts)
Calculation of Combined Uncertainty
Calculate the combined uncertainty (Uc) using standard propagation of error rules (the square root of the sums of squares of SDs known as the “root sum of squares” ‐ RSS).
Validation Data from Collaboratively Studied QuantitativeMethods
Calculation of Combined Uncertainty (Uc):
Root Sum of Squares: √(RSDr)2 + (RSDR)2
For Pour Plate (HPC) Sum of Squares: (0.077)2 + (0.182)2 =0.0371Uc = √(0.0371) = 0.193=19.3%
Expanded uncertainty (Ue): (Use coverage factor k=2* for 95% confidence) = 2 x 19.3% = 38.6%
*≥30 Observations
Expanded Uncertainty
• Expanded standard uncertainty is also known as measurement uncertainty (MU)
• calculated to give a confidence level of 95% using an expansion factor 'k' of:∙ k = 2 when n is 30 or more (n = number of observations from which the SD is calculated).
∙ k = the appropriate (95% confidence level) Student distribution 't' (two tailed) factor for n<30 for n‐1 degrees of freedom.
(Note: Not applicable to qualitative analyses)
How Many Samples Are Needed for Verification?
?
How Many Samples Are Needed for Verification?
• No fixed number• But, a minimum of 15 positive samples run in duplicate (30 observations) per matrix is not unreasonable
• 10 is the minimum requirement
Who’s Doing What
• ISO: Microbiology of food and animal feed – Method Verification– Part 4: Protocol for the verification of reference and alternative methods implemented in a single laboratory
• HC/CFIA: Part 5: Guidelines to Verify Standard Food Microbiological Methods for Implementation in Routine Testing
ISO Protocol for the verification of reference and alternative methods implemented in a single laboratory
(under review )
• Minimum of 10 artificially inoculated samples • Use CRM, if available
For Qualitative methods inoculate with 1‐5 CFU per test portion. For Quantitative methods, the levels of contamination shall cover the range of the method
– Analyse samples on 10 different occasions or days – be performed by at least 2 technicians working independently and with separate samples and reagents.
How to Interpret Verification?
• For Quantitative methods, ISO proposes to accept:participation in interlaboratory comparisons such as proficiency testing and plotting z‐scores to show any trends;
use of microbiological RM or CRM; recovery experiments with spiked samples using a non‐selective method.
Part 5: Guidelines to Verify Standard Food Microbiological Methods for Implementation in Routine Testing by Health Canada (April 2015)
For Qualitative Methods, – Analyze 3‐5 samples spiked at 3‐5 times the reported or determined LOD (1‐3 CFU/analytical unit)
For Quantitative Methods – using selective medium, measure repeatability by analyzing 10 or more replicate samples of a representative food matrix naturally or artificially contaminated
How to Interpret Verification?
For Qualitative methods, ISO and HC agree:100% (comparative) sensitivity is required to demonstrate acceptability or fitness‐for‐purpose, i.e. All samples must be correctly identified
For Quantitative methods, for acceptable precision as suggested by HC, the repeatability (r) must be less than half of the reproducibility (R) data. 2r < R. If R value if published.
If there is no published performance characteristic data, ???
Calculate Ue
Verification of QualitativeMicrobiological Methods
• Scope of application• Fitness‐for‐Purpose
Verification of a Rapid Screening Method for Salmonella
Confirmed
Known10
0 5
0+ve
-ve
Salmonella Not-salmonella
ISO/IEC 17025 Requirements
Fitness‐for‐Purpose• The data obtained from the method approval process must show it is fit for the intended use and relevant to customers’ needs.
• Does the method have performance characteristics that meets the expectation of the laboratory and the needs of the client?
References• AOAC International Methods Committee Guidelines For
Validation Of Qualitative And Quantitative Food Microbiological Official Methods Of Analysis, Appendix J.
• The Fitness for Purpose of Analytical Methods, A Laboratory Guide to Method Validation and Related Topics, EURACHEM Working Group
• How to Meet ISO 17025 Requirements for Method Verification, 2015, AOAC Technical Division for Laboratory Management (TDLM), the Analytical Laboratory Accreditation Criteria Committee (ALACC)
• ISO/FDIS 16140-2 Microbiology of the food chain —Method validation — Part 2: Protocol for the validation of alternative (proprietary) methods against a reference method(ISO/TC 34/SC 9) 2014-09-05 (Draft)
