lap audioconference
TRANSCRIPT
Accreditation Requirements inMolecular Pathology
May 20, 2009
Helen Fernandes, PhDAssociate Professor
Director, Molecular DiagnosticsUMDNJ / New Jersey Medical School
© 2009 College of American Pathologists. Materials are used with the permission of the faculty.
LAP Audioconference
2
Session Objectives• Describe the process of analytical and clinical
validation for assays used in molecular diagnostics.
• Identify monitors for on-going quality control at the pre-analytical, analytical and post-analytical phases of molecular testing.
• Recognize the utility of analytical monitors for quality improvement in molecular pathology.
3
Molecular Pathology Checklist (MOL) Required
• Clinical molecular diagnostics testing:– Oncology HLA typing– Hematology Parentage– Inherited diseases Forensics– Infectious diseases
• Test types– All laboratory-developed molecular tests (LDTs)– Lab-modified FDA-approved molecular tests– FDA-approved molecular tests (exception- Micro)
4
MOLECULARMOLECULAR
Infectious Infectious DiseasesDiseases OncologyOncology
GeneticsGenetics HematopathologyHematopathology
PharmacogenomicsPharmacogenomicsPersonalized MedicinePersonalized Medicine
Anatomic
Cytology
Hematology
Coag
Immunology
SerologyMicrobiology
Chemistry
Toxicology
5
PCRPCR
HybridizationHybridization
SouthernSouthern
SequencingSequencing
Capillary electrophoresisCapillary electrophoresis
FISHFISH
Platforms in Platforms in
Molecular TestingMolecular Testing
RealtimeRealtime PCRPCR
TaqmanTaqman
BeaconsBeacons
FRETFRET
HybridizationHybridization
Bead arraysBead arrays
CGHCGH
MicroarraysMicroarrays
6
Prognosis and TreatmentPrognosis and Treatment
Diagnosis of Cervical Cancer
Diagnosis of Colorectal
Cancer
Diagnosis of Lympho-
proliferativeDisorders
Diagnosis of Hypercoaguable
State
Diagnosis of Breast Cancer
Diagnosis of Lung Cancer
Phenotypic-Clinical Dimension of PathologyPhenotypicPhenotypic--Clinical Dimension of PathologyClinical Dimension of Pathology
Adapted from Clinical Chemistry 2007: 1118 . Manuel Salto-Tellez
7
Prognosis and TreatmentPrognosis and Treatment
Detection of HPV Infection
Diagnosis of Cervical Cancer
Phenotypic-Molecular-Clinical Dimension of PathologyPhenotypicPhenotypic--MolecularMolecular--ClinicalClinical Dimension of PathologyDimension of Pathology
EGFR mutations
KRAS mutations
Diagnosis of Lung Cancer
KRASmutations
Microsatellite Instability Analysis
Diagnosis of Colorectal
Cancer
Specific Translocations
B & T cell rearrange-
ments
Diagnosis of Lympho-
proliferativeDisorders
20210 A
Factor V-Leiden
Diagnosis of Hypercoaguable
State
ERPR Status
HER2-neuStats
Diagnosis of Breast Cancer
Adapted from Clinical Chemistry 2007: 1118 . Manuel Salto-Tellez
8
Prognosis and TreatmentPrognosis and Treatment
Detection of HPV Infection
Diagnosis of Cervical Cancer
Phenotypic-Molecular-Clinical Dimension of PathologyPhenotypicPhenotypic--MolecularMolecular--Clinical Dimension of PathologyClinical Dimension of Pathology
EGFR mutations
KRAS mutations
Diagnosis of Lung Cancer
KRASmutations
Microsatellite Instability Analysis
Diagnosis of Colorectal
Cancer
Specific Translocations
B & T cell rearrange-
ments
Diagnosis of Lympho-
proliferativeDisorders
20210 A
Factor V-Leiden
Diagnosis of Hypercoaguable
State
ERPR Status
HER2-neuStats
Diagnosis of Breast Cancer
Gene Expression ProfilesGene Expression ProfilesGene Expression Profiles
PharmacogenomicsPharmacogenomics
Adapted from Clinical Chemistry 2007: 1118 . Manuel Salto-Tellez
9
Important considerations in Important considerations in Molecular Testing Molecular Testing
VERIFICATIONVERIFICATION& &
VALIDATIONVALIDATION
PROFICIENCYPROFICIENCYTESTINGTESTING
QUALITYQUALITYCONTROLCONTROL
A TIMELY AND ACCURATE TEST RESULTA TIMELY AND ACCURATE TEST RESULT
10
Verification:
• Rationale: Ensure that the analyte(s) meet their performance specifications– a one-time process– verify the analytical performance of a test
that is approved by a regulatory body (FDA)
– Often done with some mathematical rigor
11
Verifying Performance Specifications
• Applies to unmodified, approved test systems• Laboratory must
– Demonstrate it can obtain performance specifications comparable to the manufacturer before reporting patient results
• Accuracy• Precision• Reportable range of test results
– Verify appropriate reference intervals (normal values) for the laboratory’s population
12
Validation
• Rationale: Establish that the analyte(s) fulfills its intended use in its intended environment– An extensive process– Validate the performance characteristics of
laboratory-developed tests – Examine the components of a test that are
assembled for the intended use without regulatory approval. – ASR
– Establish the analytical performance and clinical (diagnostic) performance of a test as it applies to its intended use.
13
Pre-Validation Considerations for the Design of Laboratory Developed Tests• Stringent design/analysis of primers and
probes• Quality and quantity of extracted nucleic
acid• Appropriate platform for the test• Commutable calibrators and controls• Optimization of amplification and
detection
14
Validating Performance Specifications
• Applies to– Laboratory-developed tests– Approved tests that have been modified by the laboratory
• Before reporting patient results, the laboratory must establish performance characteristics for– Accuracy– Precision– Reference and Reportable ranges– Analytical sensitivity (LOD) (LOQ)– Analytic specificity– Interfering substances– Diagnostic (Clinical) Validity
15
Analytical Sensitivity Can Be Done By…
• Control material of known concentration or copy number (calibrators / standards)
• Dilutions of analyte (microorganism, gene) of known quantity
• Quantified amount of RNA or DNA
16
LODLOD
LOQLOQ
Linear Range
Linear Range
UpperUpperLOQLOQ
Relationship Between LOD and LOQ
17
FDA Cleared Lab developedModified, ASR
Accuracy > 20 specimens > 30 specimensPrecision Positive & Negative
controls in triplicateHigh Positive and Low positive controls in triplicateInter and Intra run
AnalyticalSensitivity
5-10 specimens at LOQ and LOD
Establish LOQ with 3-5 calibrators run in triplicate and LOD with 30 specimens.
Analytical Specificity
Test related analytes
Evaluate interfering substances
Reportable Range
3-5 specimens at different concentrations
Establish measuring range and linear range with 3-5 calibrators run in triplicate
Reference Range (Normal)
10 -20 specimens 50 – 100 specimens depending on analyte
Modified from Clinical Microbiology Newsletter 29:12 2007
18
Clinical (Diagnostic) ValidationAddress the clinical significance and utility of the test.• Diagnose a disease or disease state • Confirm the results of another laboratory test, or clinical
diagnosis• Monitor and assess disease progression, prognosis or
resolution
• Comparing the test to a gold standard– Another test (culture, EIA, chemistry test, FISH)– Clinical outcome / endpoint (response to therapy)
• Use Positive and Negative Predictive values• Use ethnic variation and geographic distribution• Cite references to clinical studies in the literature
19
Validation of a Modified FDA Test
• Alteration of any process or analyte in an approved test that may affect performance
• Changes in:– type or concentration of specimen used– storage specifications– extraction method– amplification parameters or platforms– manufacturer’s cut off– using the test for another purpose
(quantitative vs. qualitative)
20
Verifying or Validating Performance Specifications
• When multiple instruments are used to perform the same test, the laboratory must compare the performance on both instruments
• If using more than one platform to test the same analyte, the lab must verify or establish performance specifications including agreement.
21
Correlation Coefficient and Agreement
‐1.30
0.00
1.30
2.00 3.00 4.00 5.00 6.00
L og10 average
Log 10 difference
R2 = 0.794
2.50
3.50
4.50
5.50
2.50 3.00 3.50 4.00 4.50 5.00 5.50
Assay A Log10 copies / mL
Ass
ay B
Log
10 c
opie
s / m
L
• Correlation coefficient represents the strength of a linear relationship between two variables
• Agreement as measured by the Bland-Altman model is the Trueness or closeness of agreement between the average and the true value of the analyte
Correlation0.79
Agreement95.6%
2 SD
2 SD
22
MOL.31015 Phase II
• Were validation studies with an adequate number and representative (reasonable) distribution of samples performed for each type of specimen expected for the assay (e.g., blood, fresh/frozen tissue, paraffin-embedded tissue, prenatal specimens?
Verification and Validation studies should be performed on all types of specimens that the test is used for.
23
Outline for Assay ValidationExample - CMV Viral Load
1. Determine the number and type of specimens to be used– A minimum of 30-50 specimens are required for assessment of statistical
significance.– Blood, Body fluids, White Blood Cells
2. Select the test to be used for comparison– CMV antigenemia / culture– Another established quantitative assay
3. Document– Accuracy, Precision, Interfering substances– Use appropriate statistical methods of evaluation (r2, regression, agreement)
4. Establish acceptance criteria for analytical sensitivity– For LOD (qualitative acceptance) - > 95% detectable specimens– For LOQ (lower quantification limit) 0.15 Log10
5. Evaluate analytical specificity– Other members of the herpesvirus family – HSV, EBV
24
LogLog1010 copies/ml in current test (copies/ml in current test (LightcyclerLightcycler))
Log
Log 1
010co
pies
/ml i
n va
lidat
ed te
st (A
BI75
00)
copi
es/m
l in
valid
ated
test
(ABI
7500
)
Direct correlation - PCR
Realtime vs. Realtime
Indirect correlation
PCR vs. Antigenemia
Accuracy - CMV Viral Load
25
Outline for Assay ValidationExample - CMV Viral Load
1. Determine the number and type of specimens to be used– A minimum of 30 specimens are required for assessment of statistical
significance. – Blood, Body fluids, White Blood Cells
2. Select the test to be used for comparison– CMV antigenemia / culture– Another established quantitative assay
3. Document– Accuracy, Precision, Interfering substances– Use appropriate statistical methods of evaluation (r2, regression, agreement)
4. Establish acceptance criteria for analytical sensitivity– For LOD (qualitative acceptance) - > 95% detectable specimens– For LOQ (lower quantification limit) 0.15 Log10
5. Evaluate analytical specificity– Other members of the herpesvirus family – HSV, EBV
26
Design for Assay ValidationExample - CMV Viral Load…
6. Assess (Diagnostic) Clinical performance– Correlate with clinical presentation– Establish baseline / reference range (important for CMV)
7. Resolve discrepancies– Sequencing– Send-out to another facility or reference lab
8. Set the parameters for interpretation of results– Use baseline to guide clinical disease– Define what is a clinically significant change when used for monitoring
disease. Tolerance limit of the assay = Assay variability + biological variation
9. Summarize all of the above in a ”Validation document”– Expand on each section– Statement on acceptance of validation for clinical use – Review & signature by Director
27
The test is accepted for clinical useThe test is accepted for clinical use
28
TOTAL QUALITY MANAGEMENT (QM) TOTAL QUALITY MANAGEMENT (QM)
Pre AnalyticPre Analytic AnalyticAnalytic Post AnalyticPost Analytic
Quality ControlQuality Control(Process)(Process)
QUALITY ASSURANCEQUALITY ASSURANCE
RecordsRecordsTrainingTrainingDocumentsDocuments
Corrective actionsCorrective actionsInternal auditsInternal audits
StatisticsStatistics
QualityQualityImprovementImprovement
29
Challenges in Quality Management in Molecular Diagnostics
• Technology driven field• Transition from research to clinical
laboratories• Rapid evolution and change of technologies
& applications; emerging pathogens; new mutations and associations with disease
• Paucity of material for validation and standardization
• Non-uniformity of calibrators and controls
30
Challenges in Quality Management in Molecular Diagnostics
• Limited quality assurance / proficiency programs
• Clinical validation can be challenging• Limited diagnostic testing guidelines• Results for the same test use different
methodologies that may not be in agreement• Lack of concordance on units of reporting• Interpretation can sometimes be difficult• Some esoteric tests performed in a single lab
31
Challenges in Quality Assurance in Molecular Diagnostics
• Since most technologies involve amplification technical precision is mandatory
• Contamination of specimens, equipment reagents and aerosol contamination are legitimate problems
• Samples can be inhibitory to amplification• Mutations in the nucleic acid can interfere
with testing • Discrepancies could be an issue
32
You need to be in touchYou need to be in touch
QA / QC is an ongoing process that affirms verification & validation.
It needs to be monitored on a regular basis for trends.
33
QC Monitors are selected to assess…
Pre-analytical process• Specimen integrity (DNA, RNA)
Analytical process• Sample prep, amplification, detection• Calibrators & Controls
Post-analytical process• Documentation, interpretation, reporting
34
Test Specimen Specimen SpecimenOrdering Collection Transport Handling/Processing
QC at the PREANALYTICAL LEVEL
Requisition forms• Ethnic information and consent for genetic tests
Specimen collection• Test specific tubes & containers (gel); swabs• Storage specs until transport (separate plasma)
Specimen transport & storage• Temperature, Conditions
Specimen Handling / Processing• Speed & temp of centrifugation
35
Testing Result Review Interpretation Testing Result Review Interpretation and followand follow--upup
QC at the ANALYTICAL LEVEL
• Proficiency testing• Use of Positive, Negative and Sensitivity
controls• Controls with known values for quantitative
tests• Use internal, external, extraction &
endogenous controls (inhibitors)• Contamination control
36
Proficiency Testing in Molecular Pathology
• CAP Proficiency Surveys• Alternate Proficiency
– Inter-laboratory comparison– Send-out to another / reference lab– In-house proficiencies (Blind panel, split sample analysis)– Comparison with another method (gold standard)
• How many specimens ???– Assay dependent (expense?)– Volume dependent (# of tests ↑)– Types of specimens / matrix
• In general three events with 3-5 samples / year
• Evaluation criteria– < 80% passing grade in 2 consecutive PT’s = PROBLEM !!
37
CONTROLSCONTROLS
INTERNAL INTERNAL EXTERNALEXTERNAL
PositivePositive NegativeNegative
EndogenousEndogenous
SensitivitySensitivity
Sensitivity control Sensitivity control for quantitative for quantitative assays should assays should reflect the lower reflect the lower LOQLOQ
Endogenous controls Endogenous controls can monitor inhibition can monitor inhibition and extractionand extraction
Controls that go through the extraction process can be more infoControls that go through the extraction process can be more informativermative
Inhibition controls are different from Positive controls.Inhibition controls are different from Positive controls.
Calibrators do not serve as controlsCalibrators do not serve as controls
No target control No target control Alternate target Alternate target use same matrixuse same matrix
Commutability of a control is importantCommutability of a control is important
38
QC Monitors for Controls• Recording of Positive controls (L+; H+ and sensitivity)
with every run/use
• Routine (monthly) graphing of controls with acceptable range
• In addition to kit controls use an additional in-house control to monitor lot-lot variation as well as intra assay and technologist variations.
• Rotating controls
39
Low control for Quant CMV Jan 2008
0
2000
4000
6000
8000
10000
3-Jan
5-Jan
7-Jan
9-Jan
11-Jan
13-Jan
15-Jan
17-Jan
19-Jan
21-Jan
23-Jan
25-Jan
27-Jan
29-Jan
Test dates
Vira
l Loa
d in
copi
es/m
L
Mean + 2 SD
Control for HCV by Technologist
0
200
400
600
1 2 3 4 5
Test dates
VL A
B
40
Instrument QCAmplification machines; Laser driven instruments (realtime, sequencers, bead arrays)
Other Analytic QC Monitors• Melting curve temperature- Tm range (+ 2oC) • Blank OD values• For realtime PCR monitor Ct values of controls (qual
or quant)• Lot – Lot check• Failed runs /reactions
41
QC at the POST-ANALYTICAL LEVEL
• Interpretation of Results: – Molecular tests can be subjective – Have more than one person review results
• Difficult problems: – Discrepant analysis– Blips at the end of an amplification curve
• Reporting in different units of measure Copies/mL; IU/mL; genome eq/mL
42
QC Monitors Post Analytical• Turn around time• Notification of rapid tests• Records of quantitative tests when used for
monitoring e.g. Viral Load• Record all new / rare mutations or variations
in a notebook• Record critical results – for rapid tests and
CSF• Correlation between tests• Errors in reporting
43
QC SUMMARY
• Continual Quality Control required• Documentation of all QC monitors on a
regular basis – monthly – Graphing of control values– In-house control in quantitative tests for reagent
and tech variation– Melting temperatures for variants– Ct value for probe integrity– Background OD for contamination
44
Does the test perform the way it was intended to?
Would you use your test to help the clinician make a decision for you or a
family member ??
Overall…..??
45
Most Common Deficiencies in 2008
MOL 31820 For FDA approved assays, are performance criteria verified by the laboratory ? (33.3%)
MOL 32050 Is there a summary statement signed by the lab director documenting the review of validation studies and approval of thetest for clinical use? (4.4%)
46
MOL 10160 For tests which CAP does not provide Proficiency testing does the lab participate in external PT of exercise Alternate PT?
MOL 31475 Do validation studies document analytical sensitivity & precision?
47
Regulatory/accrediting agencies and payers• CLIA• CAP• JCAHO
Documents for reference• CAP Molecular Checklist
– www.cap.org• CLSI (NCCLS) Approved Guidelines
Clinical Laboratory Standards Institute– www.clsi.org
• AMP – List of FDA approved testsAssociation for Molecular Pathology– www.amp.org
• American College of Medical Genetics – www.acmg.net
48
MM01MM01--A2 A2 Molecular Diagnostic Methods for Genetic DiseasesMolecular Diagnostic Methods for Genetic DiseasesMM02MM02--A2 A2 Immunoglobin and TImmunoglobin and T--Cell Receptor Gene Cell Receptor Gene
Rearrangement AssaysRearrangement AssaysMM03MM03--A2 A2 Molecular Diagnostic Methods for Infectious DiseasesMolecular Diagnostic Methods for Infectious DiseasesMM05MM05--A A Nucleic Acid Amplification Assays for Nucleic Acid Amplification Assays for
HematopathologyHematopathologyMM06MM06--A A Quantitative Molecular Methods for Infectious DiseasesQuantitative Molecular Methods for Infectious DiseasesMM07MM07--A A Fluorescence In Situ Hybridization (FISH) MethodsFluorescence In Situ Hybridization (FISH) MethodsMM09MM09--A A Nucleic Acid SequencingNucleic Acid SequencingMM10MM10--A A Genotyping for Infectious DiseasesGenotyping for Infectious DiseasesMM11MM11--P P Molecular Methods for Bacterial Strain TypingMolecular Methods for Bacterial Strain TypingMM12MM12--A A Diagnostic Nucleic Acid MicroarraysDiagnostic Nucleic Acid MicroarraysMM13MM13--A A Collection, Transport, Preparation, and Storage of Collection, Transport, Preparation, and Storage of
Specimens for Molecular MethodsSpecimens for Molecular MethodsMM14MM14--A A Proficiency Testing (External Quality Assessment) for Proficiency Testing (External Quality Assessment) for
Molecular MethodsMolecular MethodsMM16MM16--P P Use of External RNA Controls in Gene Expression Use of External RNA Controls in Gene Expression
AssaysAssays
CLSI guidelines @ CLSI guidelines @ www.clsi.orgwww.clsi.org
49
Documents for Reference• Laboratory Developed Tests in Molecular
Diagnostics. Quality Control and Assay ValidationIn WB Coleman and GJ Tsongalis, ed. Molecular Diagnostics: For the Clinical Laboratorian, 2nd edition (2006). Totowa, NJ: Humana Press, Inc.
• Gulley ML, Braziel RM, Halling KC, et al, Clinical Laboratory Reports in Molecular Pathology, Arch Pathol Lab Med, 131:852-863, 2007.
51
QuestionsDiscussion
Thank You