edward randell - ifcc...diagnostic test characteristics 13 analytical accuracy precision reportable...
TRANSCRIPT
![Page 1: Edward Randell - IFCC...Diagnostic Test Characteristics 13 Analytical Accuracy Precision Reportable Range Analytical Sensitivity Analytical Specificity Carry-over Others “…Established](https://reader035.vdocuments.us/reader035/viewer/2022081601/612eb9e61ecc51586942fec2/html5/thumbnails/1.jpg)
Principles of effective verification/validation
strategies for laboratory analytical methods
and reagents
Edward Randell
Satellite Educational Workshop on Intelligent Clinical Laboratory
Management: Impacts on Quality System Improvement
Hilton Durban - October 22, 2017
IFCC Committee on Clinical Laboratory Management -
http://www.ifcc.org/ifcc-education-division/emd-committees/c-clm/
1
![Page 2: Edward Randell - IFCC...Diagnostic Test Characteristics 13 Analytical Accuracy Precision Reportable Range Analytical Sensitivity Analytical Specificity Carry-over Others “…Established](https://reader035.vdocuments.us/reader035/viewer/2022081601/612eb9e61ecc51586942fec2/html5/thumbnails/2.jpg)
The what and why!
2Edward Randell/ Effective Verification/Validation strategies
Why is it important?
• Test results inform medical decisions
• Erroneous test results cause diagnostic error
What are we trying to do in verification/validation studies?
• Establish foundation for analytical quality control
• Determine acceptability
• Identify issues/errors/weaknesses not identified by routine QC
Our testing environment is unique…Analyzers, Expertise,
Staff, Workload, TAT requirements, Budgets, etc…
![Page 3: Edward Randell - IFCC...Diagnostic Test Characteristics 13 Analytical Accuracy Precision Reportable Range Analytical Sensitivity Analytical Specificity Carry-over Others “…Established](https://reader035.vdocuments.us/reader035/viewer/2022081601/612eb9e61ecc51586942fec2/html5/thumbnails/3.jpg)
The Risk
3
Laboratory Error …..“failure of planned action to be completed as intended, or use of a wrong plan to achieve
an aim, occurring at any part of the laboratory cycle, from ordering examinations to reporting results and appropriately
interpreting and reacting to them”
ISO/TS 22367 (2008)
Edward Randell/ Effective Verification/Validation strategies
![Page 4: Edward Randell - IFCC...Diagnostic Test Characteristics 13 Analytical Accuracy Precision Reportable Range Analytical Sensitivity Analytical Specificity Carry-over Others “…Established](https://reader035.vdocuments.us/reader035/viewer/2022081601/612eb9e61ecc51586942fec2/html5/thumbnails/4.jpg)
The Risk
4
Diagnostic Error and Laboratory Test Reagent/Method Acceptability
Edward Randell/ Effective Verification/Validation strategies
![Page 5: Edward Randell - IFCC...Diagnostic Test Characteristics 13 Analytical Accuracy Precision Reportable Range Analytical Sensitivity Analytical Specificity Carry-over Others “…Established](https://reader035.vdocuments.us/reader035/viewer/2022081601/612eb9e61ecc51586942fec2/html5/thumbnails/5.jpg)
The Risk
5Edward Randell/ Effective Verification/Validation strategies
![Page 6: Edward Randell - IFCC...Diagnostic Test Characteristics 13 Analytical Accuracy Precision Reportable Range Analytical Sensitivity Analytical Specificity Carry-over Others “…Established](https://reader035.vdocuments.us/reader035/viewer/2022081601/612eb9e61ecc51586942fec2/html5/thumbnails/6.jpg)
A Few Definitions
6
Adapted from: Accred Qual Assur (2006) DOI 10.1007/s00769-006-0191-z
Edward Randell/ Effective Verification/Validation strategies
![Page 7: Edward Randell - IFCC...Diagnostic Test Characteristics 13 Analytical Accuracy Precision Reportable Range Analytical Sensitivity Analytical Specificity Carry-over Others “…Established](https://reader035.vdocuments.us/reader035/viewer/2022081601/612eb9e61ecc51586942fec2/html5/thumbnails/7.jpg)
Total Error
7Edward Randell/ Effective Verification/Validation strategies
![Page 8: Edward Randell - IFCC...Diagnostic Test Characteristics 13 Analytical Accuracy Precision Reportable Range Analytical Sensitivity Analytical Specificity Carry-over Others “…Established](https://reader035.vdocuments.us/reader035/viewer/2022081601/612eb9e61ecc51586942fec2/html5/thumbnails/8.jpg)
The Clinical Implication
8
A clinician orders a glucose
and the lab reports a value of
3.4 mmol/L. Does the patient
have hypoglycemia?
Edward Randell/ Effective Verification/Validation strategies
![Page 9: Edward Randell - IFCC...Diagnostic Test Characteristics 13 Analytical Accuracy Precision Reportable Range Analytical Sensitivity Analytical Specificity Carry-over Others “…Established](https://reader035.vdocuments.us/reader035/viewer/2022081601/612eb9e61ecc51586942fec2/html5/thumbnails/9.jpg)
Inaccuracy & Imprecision
9
Analytical error & performance
relates to clinical practice by
the impact of inaccuracy and
imprecision to clinical decision
making.
Edward Randell/ Effective Verification/Validation strategies
![Page 10: Edward Randell - IFCC...Diagnostic Test Characteristics 13 Analytical Accuracy Precision Reportable Range Analytical Sensitivity Analytical Specificity Carry-over Others “…Established](https://reader035.vdocuments.us/reader035/viewer/2022081601/612eb9e61ecc51586942fec2/html5/thumbnails/10.jpg)
Diagnostic Test Characteristics
10
Analy
tical
Accuracy
Precision
Reportable Range
Analytical Sensitivity
Analytical Specificity
Carry-over
Others
Clin
ica
l
Diagnostic Sensitivity
Diagnostic Specificity
Reference Ranges
Positive/Negative Predictive Value
Likelihood Ratios
ROC Curve
Others
Edward Randell/ Effective Verification/Validation strategies
![Page 11: Edward Randell - IFCC...Diagnostic Test Characteristics 13 Analytical Accuracy Precision Reportable Range Analytical Sensitivity Analytical Specificity Carry-over Others “…Established](https://reader035.vdocuments.us/reader035/viewer/2022081601/612eb9e61ecc51586942fec2/html5/thumbnails/11.jpg)
Evaluation, Verification &
Validation
11
Evaluation – A global term applied to
determining analytical and clinical
performance characteristics of a new test
to a laboratory.
Verification – “Provision of objective
evidence that a given item fulfills specified
requirements.” (JCGM 200:2012)
Validation – “verification, where specified
requirements are adequate for intended
use.” (JCGM 200:2012)
Confirms claimed
specifications
Confirms performance
Fit for Purpose
Edward Randell/ Effective Verification/Validation strategies
![Page 12: Edward Randell - IFCC...Diagnostic Test Characteristics 13 Analytical Accuracy Precision Reportable Range Analytical Sensitivity Analytical Specificity Carry-over Others “…Established](https://reader035.vdocuments.us/reader035/viewer/2022081601/612eb9e61ecc51586942fec2/html5/thumbnails/12.jpg)
The Evaluation Process
12
See CLSI EP19-ED2:2015
Method Evaluation involves:1. Determining quality requirements and Setting
Performance Goals.2. Designing/Performing Experiments to gather
Data.3. Using data to estimate components of error4. Evaluating measured error relative to
requirements.
Edward Randell/ Effective Verification/Validation strategies
![Page 13: Edward Randell - IFCC...Diagnostic Test Characteristics 13 Analytical Accuracy Precision Reportable Range Analytical Sensitivity Analytical Specificity Carry-over Others “…Established](https://reader035.vdocuments.us/reader035/viewer/2022081601/612eb9e61ecc51586942fec2/html5/thumbnails/13.jpg)
Diagnostic Test Characteristics
13
Analy
tical
Accuracy
Precision
Reportable Range
Analytical Sensitivity
Analytical Specificity
Carry-over
Others
“…Established by comparing results to a definitive or reference method, …verified by comparing results to an established comparative method. “
CAP Laboratory Accreditation Program Checklist.
• Method Comparison
• Standard Reference Materials
• Reference Methods
Edward Randell/ Effective Verification/Validation strategies
![Page 14: Edward Randell - IFCC...Diagnostic Test Characteristics 13 Analytical Accuracy Precision Reportable Range Analytical Sensitivity Analytical Specificity Carry-over Others “…Established](https://reader035.vdocuments.us/reader035/viewer/2022081601/612eb9e61ecc51586942fec2/html5/thumbnails/14.jpg)
Diagnostic Test Characteristics
14Edwrad Randell/ Effective Verification/Validation strategies
Analy
tical
Accuracy
Precision
Reportable Range
Analytical Sensitivity
Analytical Specificity
Carry-over
Others
• Within Run
• Between Run
• Between Day
“Established by repeat measurement of
samples at varying concentrations or
activities within-run and between-run over
a period of time.”
CAP Laboratory Accreditation Program Checklist.
![Page 15: Edward Randell - IFCC...Diagnostic Test Characteristics 13 Analytical Accuracy Precision Reportable Range Analytical Sensitivity Analytical Specificity Carry-over Others “…Established](https://reader035.vdocuments.us/reader035/viewer/2022081601/612eb9e61ecc51586942fec2/html5/thumbnails/15.jpg)
Deconstructing imprecision
15
The total within laboratory imprecision is a composite of three different aspects of random variation. Also by:• Reagent lot• Calibrator lot• Calibration frequency• Operator• Instrument
• Laboratory
Edward Randell/ Effective Verification/Validation strategies
![Page 16: Edward Randell - IFCC...Diagnostic Test Characteristics 13 Analytical Accuracy Precision Reportable Range Analytical Sensitivity Analytical Specificity Carry-over Others “…Established](https://reader035.vdocuments.us/reader035/viewer/2022081601/612eb9e61ecc51586942fec2/html5/thumbnails/16.jpg)
Diagnostic Test Characteristics
16
Analy
tical
Accuracy
Precision
Reportable Range
Analytical Sensitivity
Analytical Specificity
Carry-over
Others
• Limit of Blank
• Limit of Detection
• Limit of Quantification
• Analytical Measurement Range
• Clinical Measurement Range
Important for quantitative methods
Edward Randell/ Effective Verification/Validation strategies
![Page 17: Edward Randell - IFCC...Diagnostic Test Characteristics 13 Analytical Accuracy Precision Reportable Range Analytical Sensitivity Analytical Specificity Carry-over Others “…Established](https://reader035.vdocuments.us/reader035/viewer/2022081601/612eb9e61ecc51586942fec2/html5/thumbnails/17.jpg)
Analytical Sensitivity and
Reportable Range
17Edward Randell/ Effective Verification/Validation strategies
![Page 18: Edward Randell - IFCC...Diagnostic Test Characteristics 13 Analytical Accuracy Precision Reportable Range Analytical Sensitivity Analytical Specificity Carry-over Others “…Established](https://reader035.vdocuments.us/reader035/viewer/2022081601/612eb9e61ecc51586942fec2/html5/thumbnails/18.jpg)
Diagnostic Test Characteristics
18
Analy
tical
Accuracy
Precision
Reportable Range
Analytical Sensitivity
Analytical Specificity
Carry-over
Others
• Interferences
• Cross-reactivity
Related to systematic error (Trueness)
Edward Randell/ Effective Verification/Validation strategies
![Page 19: Edward Randell - IFCC...Diagnostic Test Characteristics 13 Analytical Accuracy Precision Reportable Range Analytical Sensitivity Analytical Specificity Carry-over Others “…Established](https://reader035.vdocuments.us/reader035/viewer/2022081601/612eb9e61ecc51586942fec2/html5/thumbnails/19.jpg)
Diagnostic Test Characteristics
19
Clin
ica
l
Diagnostic Sensitivity
Diagnostic Specificity
Reference Ranges
Positive/Negative Predictive Value
Likelihood Ratios
ROC Curve
Others
Edward Randell/ Effective Verification/Validation strategies
![Page 20: Edward Randell - IFCC...Diagnostic Test Characteristics 13 Analytical Accuracy Precision Reportable Range Analytical Sensitivity Analytical Specificity Carry-over Others “…Established](https://reader035.vdocuments.us/reader035/viewer/2022081601/612eb9e61ecc51586942fec2/html5/thumbnails/20.jpg)
Diagnostic Test Characteristics
20
Clin
ica
l
Diagnostic Sensitivity
Diagnostic Specificity
Reference Ranges
Positive/Negative Predictive Value
Likelihood Ratios
ROC Curve
Others
Edward Randell/ Effective Verification/Validation strategies
![Page 21: Edward Randell - IFCC...Diagnostic Test Characteristics 13 Analytical Accuracy Precision Reportable Range Analytical Sensitivity Analytical Specificity Carry-over Others “…Established](https://reader035.vdocuments.us/reader035/viewer/2022081601/612eb9e61ecc51586942fec2/html5/thumbnails/21.jpg)
How much evaluation?
21
Costs for Labor & Supplies
Risk
• Purpose of Test
• Environment
• Quantitative vs. qualitative
• Validation vs. verification
• Available information
• Experience with test
• Implications of error
Edward Randell/ Effective Verification/Validation strategies
![Page 22: Edward Randell - IFCC...Diagnostic Test Characteristics 13 Analytical Accuracy Precision Reportable Range Analytical Sensitivity Analytical Specificity Carry-over Others “…Established](https://reader035.vdocuments.us/reader035/viewer/2022081601/612eb9e61ecc51586942fec2/html5/thumbnails/22.jpg)
Clinical Use Considerations
for New Method
22
How results used
Diagnosis
Monitoring
Screening
Consequences of incorrect result
Tests
Treatments
And associated risks
Possibility of corroborating
results with other information
Signs & Symptoms
Other Tests
Who will get the test?
Patient population
Location
Ref. CLSI EP17-A: 2012Edward Randell/ Effective Verification/Validation strategies
![Page 23: Edward Randell - IFCC...Diagnostic Test Characteristics 13 Analytical Accuracy Precision Reportable Range Analytical Sensitivity Analytical Specificity Carry-over Others “…Established](https://reader035.vdocuments.us/reader035/viewer/2022081601/612eb9e61ecc51586942fec2/html5/thumbnails/23.jpg)
Laboratory Considerations
for New Method
23
Effectiveness of routine QC/QA
Timely Detection of
Errors
Preventing Release of
Errors
Previous experience with
test/method
Existing test, but new method
New test
Other information
Peer reviewed literature
Colleagues
PT & regulatory data
Testing Environment
Staff
Environment
Ref. CLSI EP17-A: 2012Edward Randell/ Effective Verification/Validation strategies
![Page 24: Edward Randell - IFCC...Diagnostic Test Characteristics 13 Analytical Accuracy Precision Reportable Range Analytical Sensitivity Analytical Specificity Carry-over Others “…Established](https://reader035.vdocuments.us/reader035/viewer/2022081601/612eb9e61ecc51586942fec2/html5/thumbnails/24.jpg)
How much evaluation?
24
Costs for Labor & Supplies
Risk
Edward Randell/ Effective Verification/Validation strategies
![Page 25: Edward Randell - IFCC...Diagnostic Test Characteristics 13 Analytical Accuracy Precision Reportable Range Analytical Sensitivity Analytical Specificity Carry-over Others “…Established](https://reader035.vdocuments.us/reader035/viewer/2022081601/612eb9e61ecc51586942fec2/html5/thumbnails/25.jpg)
How much evaluation?
25
Careful risk assessment
with consideration of
both clinical and
laboratory factors is
helpful is determining
the magnitude of
evaluation study.
Edward Randell/ Effective Verification/Validation strategies
![Page 26: Edward Randell - IFCC...Diagnostic Test Characteristics 13 Analytical Accuracy Precision Reportable Range Analytical Sensitivity Analytical Specificity Carry-over Others “…Established](https://reader035.vdocuments.us/reader035/viewer/2022081601/612eb9e61ecc51586942fec2/html5/thumbnails/26.jpg)
How much evaluation?
26
At least
• Accuracy
• Precision
• Reportable Range
• Reference Range
As much as necessary to confirm
suitability for intended use.
Include consideration of: Measurement trueness
Measurement accuracy
Measurement precision
Measurement uncertainty
Analytical specificity
Analytical sensitivity
Measurement Range
Diagnostic specificity & sensitivity
Reference Range
Verification Validation
Edward Randell/ Effective Verification/Validation strategies
![Page 27: Edward Randell - IFCC...Diagnostic Test Characteristics 13 Analytical Accuracy Precision Reportable Range Analytical Sensitivity Analytical Specificity Carry-over Others “…Established](https://reader035.vdocuments.us/reader035/viewer/2022081601/612eb9e61ecc51586942fec2/html5/thumbnails/27.jpg)
The Evaluation Process
27Edward Randell/ Effective Verification/Validation strategies
![Page 28: Edward Randell - IFCC...Diagnostic Test Characteristics 13 Analytical Accuracy Precision Reportable Range Analytical Sensitivity Analytical Specificity Carry-over Others “…Established](https://reader035.vdocuments.us/reader035/viewer/2022081601/612eb9e61ecc51586942fec2/html5/thumbnails/28.jpg)
Lot number Evaluation:
The what and why!
28
Why is it important?• Detect significant changes that may impact PATIENT CARE
• Confirm patient sample result consistency
Lot number change can change performance.• Change or instability in component materials (Trueness)
• Transportation and storage (Trueness)
• Incorrect calibration (Trueness)
Lot number change can affect QC result only,
patient result only, or both.
Edward Randell/ Effective Verification/Validation strategies
![Page 29: Edward Randell - IFCC...Diagnostic Test Characteristics 13 Analytical Accuracy Precision Reportable Range Analytical Sensitivity Analytical Specificity Carry-over Others “…Established](https://reader035.vdocuments.us/reader035/viewer/2022081601/612eb9e61ecc51586942fec2/html5/thumbnails/29.jpg)
Lot number evaluation
29
• Rejection criteria is based on:
– Critical difference
– Performance goals
• For affiliated hospitals the verification
needs only be done once –QC
checks sufficient
• New shipment of verified lot - QC
checks sufficient.
Ref: CLSI EP26 A:2013Edward Randell/ Effective Verification/Validation strategies
![Page 30: Edward Randell - IFCC...Diagnostic Test Characteristics 13 Analytical Accuracy Precision Reportable Range Analytical Sensitivity Analytical Specificity Carry-over Others “…Established](https://reader035.vdocuments.us/reader035/viewer/2022081601/612eb9e61ecc51586942fec2/html5/thumbnails/30.jpg)
Lot number evaluation
30
• No universally accepted acceptance criteria.
• Analyte specific
• Based on performance goals
• Be aware of Statistical Power– α usually ≤5% (proportion false rejection)
– ß usually 5 to 20% (proportion false acceptance)
– Power=1-ß (Desirable is 90 to 95%)
• Increasing statistical power requires more
samples and/or more replicates. (i.e. 3
samples with 5 replicates gives same power
as 15 samples)
Ref: CLSI EP26 A:2013Edward Randell/ Effective Verification/Validation strategies
![Page 31: Edward Randell - IFCC...Diagnostic Test Characteristics 13 Analytical Accuracy Precision Reportable Range Analytical Sensitivity Analytical Specificity Carry-over Others “…Established](https://reader035.vdocuments.us/reader035/viewer/2022081601/612eb9e61ecc51586942fec2/html5/thumbnails/31.jpg)
Summary• Validation studies most confirm Fitness for Purpose
• Verification studies must confirm Manufacturers Specifications
• Determining acceptability by validation and verification studies
involves assessment of analytical and clinical performance.
• Selection of specific studies depends on consideration of risk and
local clinical and laboratory factors.
• Reagent lot number verification is mainly based on determining bias
within acceptable limits using patient data .
31Edward Randell/ Effective Verification/Validation strategies
![Page 32: Edward Randell - IFCC...Diagnostic Test Characteristics 13 Analytical Accuracy Precision Reportable Range Analytical Sensitivity Analytical Specificity Carry-over Others “…Established](https://reader035.vdocuments.us/reader035/viewer/2022081601/612eb9e61ecc51586942fec2/html5/thumbnails/32.jpg)
Useful Resources
Instrument/Method Verification• https://www.eurachem.org/images/stories/Guides/pdf/MV_guide_2nd_ed_EN.pdf
• http://clsieclipseua.org/
32Edward Randell/ Effective Verification/Validation strategies