case study of molecular assay validation: 2nd tier
TRANSCRIPT
CASE STUDY OF MOLECULAR ASSAY VALIDATION: 2ND TIER QUALITATIVE SCREENING AND SUBSEQUENT MONITORING
Rachel Lee, PhD
March 7, 2017
Identify Purposes of 2nd-tier Test
• Reducing false positives • Clarifying an ambiguous primary
result • Providing additional or just-in-time
information• Minimizing invasive diagnostic
testing• Improving healthcare equality
VLCAD – Why is needed?• Implemented statewide December 2016• Primary analyte C14:1; secondary analytes C14 and C14:1/C2
• Reported VLCAD Abnormal, Borderline, or Normal
• 1,426 presumptive positives and 91 cases of VLCAD
• Diagnostic test – skin biopsy – invasive, costly, and takes a while
• Inconsistent insurance coverage
Genes associated with the condition• Single gene or multiple genes• Autosomal recessive, dominant, X-
linked• Genotype/phenotype correlation
• Phenotypic spectrum• Ethnic or geographical differences
VLCAD – Gene Information• Single gene – ACADVL – 20 exons,
~5.4 kb in size • Autosomal recessive• Strong genotype/phenotype correlation
• 81% of pathogenic truncating variants are associated with severe early onset form
• Milder childhood and adult forms often resulted from one or two pathogenic missense variants
Testing Method Options• Mutation panel or sequencing?
• Common variants• Any existing protocol/kit
• Publication• Peer laboratory• Commercial vendor• FDA approval
• Existing methods or equipment
Testing Method Options – cont.• Cost• Expertise• Turnaround time• Implementation timeline• Capacity/Throughput/Automation• LIMS interface• Testing algorithm• Future expansion for other disorders
VLCAD – Which Method?• ~200 variants reported in ClinVar• Diverse mutational spectrum• No common variants identified in TX cases
VLCAD – Which Method?• Existing protocol
• Publications
• Peer laboratories• Existing methods and equipment
• Beta Globin gene using ABI310
VLCAD – Which Method?• Cost – higher than mutation panel• Expertise – ok with existing protocol• Turnaround time – shorter than skin biopsy• Implementation timeline - flexible• Capacity/Throughput/Automation – ok but
need to improve efficiency• LIMS interface – existing model• Testing algorithm – existing model• Future expansion for other disorders - yes
Expected Proportion of Abnormal Alleles DetectedMutation Detection
Rate
Proportion of CF patients for which a givennumber of abnormal alleles is detected
2 AbnormalAlleles
1 AbnormalAllele
0 AbnormalAllele
98% 96% 4% 0%95% 90% 10% 0%90% 81% 18% 1%85% 72% 26% 2%80% 64% 32% 4%75% 56% 38% 6%70% 49% 42% 9%60% 36% 48% 16%50% 25% 50% 25%40% 16% 48% 36%30% 9% 42% 49%
Moskowitz et al, 2008, Genetics in Medicine 10(12): 851-868
2nd-tier VLCAD DNA Testing
Implementation
Method Development: Optimization &
Scale-up
Method Validation: Testing &
QAO/Management Approvals
LIMS: Modification & Validation
Variant Database: Journal Article
Review & Compile List with Clinical
Significance
Follow-up Algorithm
Development
Stakeholders Input
SOP: Development &
QAO/Management Approvals
Communication: List Serv
Announcement
• Collect in-house specimens needed for method development and validation
• Request specimens from other states or laboratories
• Positive and negative controls with known genotypes
• Number of specimens
Materials Used for Development and Validation
Method Development / Optimization• Testing process – workflow evaluation
• Punch• Extraction protocol• Testing protocol• Data analysis / Result interpretation
• Automation• Staff training• Troubleshooting• PT exchange program
Bidirectional Sanger Sequencing Analysis
• Accuracy• Precision• Sensitivity• Specificity• Reportable range• Reference range• Stability • Carryover study
Validation Plan
Accuracy• Determined by testing 11 specimens carrying 16 different
variants and comparing the sequencing results to the results reported by a metabolic specialist for each specimen. In order for specimen sequencing results to be acceptable, they must be more than 95% in agreement with the metabolic specialist’s result.
• Finding: 100% in agreement with the metabolic specialists’ results, but four additional variants were identified probably due to the primer design and their coverage for sequencing regions
Precision• Determined by assessing the day to day, run to run, and
within run variation, as well as technician variability. • Two positive and two negative specimens were run in
triplicate for five days. • In order for the specimen sequencing results to be
acceptable, repeat testing of the same specimen over time should offer consistent results independent of the technician performing the task.
• The sequence identified variant should be more than 95% in agreement with the metabolic specialist’s result.
Precision• Finding: The sequencing results produced clear,
readable electropherograms and the identified variants are 100% in agreement within each run.
Testing Date Technician
#4 (Exons 12-13)
#11 (Exon 20)
WT 2 (Exons 12-
13)WT 4
(Exon 20)
Day 1 9/20/2016 ST
Heterozygous c.1322G>A
(G441D)
Heterozygous c.1844G>A
(R615Q)No variant identified
No variant identified
Day 2 9/28/2016 YS
Heterozygous c.1322G>A
(G441D)
Heterozygous c.1844G>A
(R615Q)No variant identified
No variant identified
Day 3 9/29/2016 YS
Heterozygous c.1322G>A
(G441D)
Heterozygous c.1844G>A
(R615Q)No variant identified
No variant identified
Day 4 9/30/2016 ST
Heterozygous c.1322G>A
(G441D)
Heterozygous c.1844G>A
(R615Q)No variant identified
No variant identified
Day 5 10/21/2016 YS
Heterozygous c.1322G>A
(G441D)
Heterozygous c.1844G>A
(R615Q)No variant identified
No variant identified
Clinical Sensitivity• Calculated by dividing the number of true positives by the
sum of true positives plus false negatives. The acceptable criteria for diagnostic sensitivity must be ≥95%.
• [TP/ (TP+FN)] x 100%
• Finding: All of the eleven specimens were tested by Sanger sequencing across 20 exons and identified all variants reported by the metabolic specialists as well as four additional variants. Clinical sensitivity is 100% in agreement with the known results.
Clinical Specificity• Calculated by dividing the number of true negatives by the
sum of true negatives plus false positives. The acceptable criteria for diagnostic specificity must be ≥95%.
[TN/ (TN+FP)] x 100%
• Finding: The sequencing results of the negative controls identified two benign variants, c.-63_-49dup15bp and c.1605+6T>C. No other sequencing results identified any variants. Clinical specificity is 100% in agreement with the known results.
Reportable Range• Reportable range was documented by the presence of variants within the amplified region of the ACADVL gene.
• Variants were identified by comparing to the published reference gene sequence NG_007975.1, and the Genbank accession number is AC120057.9.
• Test results reported as • Homozygous variant• Heterozygous variant • No variant identified
Reference Range• The reference range or ‘normal’ value represents specimens that are variant-free at a targeted sequencing region of the ACADVL gene and not necessarily a variant-free patient sample.
• Normal specimens should not contain identified variants or variants causing VLCAD. The test results were reported as “No variant identified”.
Stability Study• Determine if the current TX newborn screening DNA
Analysis specimen acceptance criterion of 4 months after Date of Collection is applicable and how long the mutations are stable at room temperature storage. Two newborn specimens that were received within 1 or 2 days after Date of Collection were selected. Punches were made, extracted, and tested on the same day (Day 1) and month 4.
Carryover Study• Determine potential cross contamination caused by using
the same puncher head to punch samples without cleaning between punches will be evaluated by punching 8 blank filter paper spots (CP) after punching 80 newborn DBS specimens
1 2 3 4 5 6 7 8 9 10 11 12
A patient patient patient patient patient patient patient patient patient patient CP
B patient patient patient patient patient patient patient patient patient patient CP
C patient patient patient patient patient patient patient patient patient patient CP
D patient patient patient patient patient patient patient patient patient patient CP
E patient patient patient patient patient patient patient patient patient patient CP
F patient patient patient patient patient patient patient patient patient patient CP
G patient patient patient patient patient patient patient patient patient patient CP
H patient patient patient patient patient patient patient patient patient patient CP
Carryover Study (cont)• Determine potential cross contamination during extraction
and assay by testing plates with alternating newborn specimens with no template control in a checker board pattern
1 2 3 4 5 6 7 8 9 10 11 12
A patient NTC patient NTC patient NTC patient NTC patient NTC patient
B NTC Patient NTC Patient NTC Patient NTC Patient NTC Patient NTC
C patient NTC patient NTC patient NTC patient NTC patient NTC patient
D NTC Patient NTC Patient NTC Patient NTC Patient NTC Patient NTC
E patient NTC patient NTC patient NTC patient NTC patient NTC patient
F NTC Patient NTC Patient NTC Patient NTC Patient NTC Patient NTC
G patient NTC patient NTC patient NTC patient NTC patient NTC patient
H NTC Patient NTC Patient NTC Patient NTC Patient NTC Patient NTC
• Accuracy• Precision• Sensitivity• Specificity• Reportable range• Reference range• Stability study• Carryover study
Validation Plan– FDA cleared kit
Other Activities• Validation study write-up and approval• Scale up (workflow, coordination with 1st tier)• Result notes (interpretation, recommendation)• Method limitation• Variant database• SOP write-up and approval• Reporting / follow-up algorithm• LIMS modification• Seeking stakeholder inputs• Inform and educate healthcare providers
QC Monitoring• Turnaround time - % of specimens meet TAT• Failed run log
• % of runs failed• Causes – look for pattern or trend in exons, reagent, equipment, staff
• Total number of tests• Update variant database• Diagnosed cases
• Compare our results with clinical results • Frequency of variant alleles in diagnosed cases• Any missed variants• Age of diagnosis• Incidence rate
• PT performance
Problems Encountered• Low PCR amplification
• Poor extraction• Low genomic DNA• Primer design • PCR reaction condition
• Sequence data• High background noise or low signal• Ski slope• Dye blot• Spikes
• Variants in negative control specimens• Incorrect variant call• Classification and interpretation of variants
Take Home Messages….• Each NBS program has different needs - define your
goals and choose wisely
• Need help? Just ask
• Be familiar with CLIA and CAP requirements and CLSI guidelines on validation
• Continuous monitoring is important
• Document…..document…document