![Page 1: Gene Pennello, Ph.D. Team Leader, Diagnostics Devices Branch Division of Biostatistics](https://reader036.vdocuments.us/reader036/viewer/2022081419/568148cc550346895db5e9a4/html5/thumbnails/1.jpg)
1
Clinical Validation of Prognostic Biomarkers of Risk and
Predictive Biomarkers of Drug Efficacy or Safety
Gene Pennello, Ph.D. Team Leader, Diagnostics Devices Branch
Division of BiostatisticsOffice of Surveillance and Biometrics
Center for Devices and Radiological Health, FDA
SAMSI Risk Perception Policy Practice Workshop October 3, 2007
![Page 2: Gene Pennello, Ph.D. Team Leader, Diagnostics Devices Branch Division of Biostatistics](https://reader036.vdocuments.us/reader036/viewer/2022081419/568148cc550346895db5e9a4/html5/thumbnails/2.jpg)
2
Outline• FDA and Device Regulation
• Types of Biomarkers
• Validation of Diagnostics
• Predictive and Prognostic Biomarkers– Definitions, Endpoints– Study Designs for Predictive Biomarkers
• Prospective Designs – efficiency comparison• Prospective-Retrospective Designs
• Summary
![Page 3: Gene Pennello, Ph.D. Team Leader, Diagnostics Devices Branch Division of Biostatistics](https://reader036.vdocuments.us/reader036/viewer/2022081419/568148cc550346895db5e9a4/html5/thumbnails/3.jpg)
3
FDA
CDERDrugs
CDRH,Devices
CBER,Biologics
CVM,Veterinary
CFSAN,Food NCTR
![Page 4: Gene Pennello, Ph.D. Team Leader, Diagnostics Devices Branch Division of Biostatistics](https://reader036.vdocuments.us/reader036/viewer/2022081419/568148cc550346895db5e9a4/html5/thumbnails/4.jpg)
4
What are Medical Devices?
An item for treating or diagnosing a health condition whose intended use is not achieved primarily by chemical or biological action within the body (Section 201(h) of the Federal Food Drug & Cosmetic (FD&C) Act).
Definition by exclusion: Simply put, a medical device is any medical item for use in humans that is not a drug nor a biological product.
![Page 5: Gene Pennello, Ph.D. Team Leader, Diagnostics Devices Branch Division of Biostatistics](https://reader036.vdocuments.us/reader036/viewer/2022081419/568148cc550346895db5e9a4/html5/thumbnails/5.jpg)
5
Example of Medical DevicesRelatively Simple Devices tongue depressors thermometers latex gloves simple surgical instruments
Ophthalmic devices intraocular lenses PRK lasers,
Radiological devices MRI machines CT scannersdigital mammographycomputer aided detection
Cardiovascular Devices pacemakers defibrillators heart valves coronary stents artificial hearts
Monitoring Devices glucometers bone densitometers
Diagnostic Devices diagnostic test kits for HIVprostate-specific antigen (PSA) testhuman papillomavirus (HPV) test
![Page 6: Gene Pennello, Ph.D. Team Leader, Diagnostics Devices Branch Division of Biostatistics](https://reader036.vdocuments.us/reader036/viewer/2022081419/568148cc550346895db5e9a4/html5/thumbnails/6.jpg)
6
Example of Medical DevicesEmerging technologies multiplex genetic tests (e.g., for multiple mutations or microbes)
Genomic and proteomic Dx tests
Nanotechnological devices
Microspheres for molecular treatment of cancer
Robotics
Theranostics (predictive biomarkers of response or adverse reaction to therapy).
Artificial pancreas
Dental, Ear, Nose, andThroat Devices hearing aidsbronchoscopy system
General, Surgical, and Restorative Devices breast implants artificial hips spinal fixation devices artificial skin
![Page 7: Gene Pennello, Ph.D. Team Leader, Diagnostics Devices Branch Division of Biostatistics](https://reader036.vdocuments.us/reader036/viewer/2022081419/568148cc550346895db5e9a4/html5/thumbnails/7.jpg)
7
Example of Medical Devices
Due to the wide variety in technology, complexity, and intended use, medical devices can present novel statistical design and analysis challenges.
![Page 8: Gene Pennello, Ph.D. Team Leader, Diagnostics Devices Branch Division of Biostatistics](https://reader036.vdocuments.us/reader036/viewer/2022081419/568148cc550346895db5e9a4/html5/thumbnails/8.jpg)
8
Device RegulationDecision to approve a PMA application must “rely upon valid scientific evidence to determine whether there is reasonable assurance that the device is safe and effective”. “Valid scientific evidence is evidence from well controlled studies, partially controlled studies and objective trials without matched controls, well documented case histories conducted by qualified experts that there is a reasonable assurance of safety and effectiveness . . .”
U.S. Code of Federal Regulations, Title 21 (Food and Drugs), U.S. Government Printing Office, Washington DC, 2001, Part 860.7 Web address http://www.access.gpo.gov/nara/cfr/waisidx_01/21cfr860_01.html (Accessed February, 2002)
![Page 9: Gene Pennello, Ph.D. Team Leader, Diagnostics Devices Branch Division of Biostatistics](https://reader036.vdocuments.us/reader036/viewer/2022081419/568148cc550346895db5e9a4/html5/thumbnails/9.jpg)
9
Device RegulationLeast Burdensome Provisions of FDA Modernization Act (1997)“Secretary shall only request information that is necessary to making substantial equivalence determinations.”“Secretary shall consider, …, the least burdensome appropriate means of evaluating device effectiveness that would have a reasonable likelihood of resulting in approval.”
U.S. Code of Federal Regulations, Title 21 (Food and Drugs), U.S. Government Printing Office, Washington DC, 2001, Part 513(i)(1)(D) and 513(a)(3)(D)(ii). Web address http://www.access.gpo.gov/nara/cfr/waisidx_01/21cfr860_01.html
![Page 10: Gene Pennello, Ph.D. Team Leader, Diagnostics Devices Branch Division of Biostatistics](https://reader036.vdocuments.us/reader036/viewer/2022081419/568148cc550346895db5e9a4/html5/thumbnails/10.jpg)
10
FDA Least Burdensome Guidance
FDA Guidance: The Least Burdensome Provisions of the FDA Modernization Act of 1997: Concept and Principles (2002)“Modern statistical methods may also play an important role in achieving a least burdensome path to market. For example, through the use of Baysian [sic] analyses, studies can be combined in order to help reduce the sample size needed for the experimental and/or control device.”
![Page 11: Gene Pennello, Ph.D. Team Leader, Diagnostics Devices Branch Division of Biostatistics](https://reader036.vdocuments.us/reader036/viewer/2022081419/568148cc550346895db5e9a4/html5/thumbnails/11.jpg)
11
Examples of Less Burdensome
Non-U.S. data
Surrogate endpoints (e.g., acute follow-up)
Interim analysis, Adaptive design
Bayesian methods (e.g., to reduce sample size)†
Propensity Scores for historical controls
Sensitivity analysis for missing data.
Note, could trade clinical for statistical burden
†FDA Draft Guidance for the Use of Bayesian Statistics in Medical Device (released May 23, 2006) www.fda.gov/cdrh/osb/guidance/1601.html
![Page 12: Gene Pennello, Ph.D. Team Leader, Diagnostics Devices Branch Division of Biostatistics](https://reader036.vdocuments.us/reader036/viewer/2022081419/568148cc550346895db5e9a4/html5/thumbnails/12.jpg)
12
Least Burdensome Provision
• Least burdensome provision in FDAMA of 1997 is directed to both medical devices and diagnostics (including biomarkers).
![Page 13: Gene Pennello, Ph.D. Team Leader, Diagnostics Devices Branch Division of Biostatistics](https://reader036.vdocuments.us/reader036/viewer/2022081419/568148cc550346895db5e9a4/html5/thumbnails/13.jpg)
13
Device Risk Classification
Class I: Devices for which “general controls” provide reasonable assurance of the safety and effectiveness.Class II: “General controls” insufficient, Can establish “special controls” (performance standards [CLIA, ISO], FDA guidance. May require clinical data on a 510(k).Class III: General and special controls insufficient. Life-sustaining/supporting, substantial importance in preventing impairment of human health, potential unreasonable risk of illness or injury. Needs pre-market approval (PMA).
![Page 14: Gene Pennello, Ph.D. Team Leader, Diagnostics Devices Branch Division of Biostatistics](https://reader036.vdocuments.us/reader036/viewer/2022081419/568148cc550346895db5e9a4/html5/thumbnails/14.jpg)
14
Post-Market Transformation
• “Make postmarket data more widely available to Center staff and supplement search and reporting tools”– "Investigate the use of data and text mining
techniques to identify the "needles in the haystack" by identifying patterns in the incoming data that equate to public health signals.”
– Example is WebVDME Bayesian data-mining
• Design a pilot project to test the usefulness of quantitative decision-making methods for medical device regulation across the total product life cycle
http://www.fda.gov/cdrh/postmarket/mdpi-report-1106.html
![Page 15: Gene Pennello, Ph.D. Team Leader, Diagnostics Devices Branch Division of Biostatistics](https://reader036.vdocuments.us/reader036/viewer/2022081419/568148cc550346895db5e9a4/html5/thumbnails/15.jpg)
15
Types of Biomarkers• Diagnostic• Early detection (screening), enabling intervention at
an earlier and potentially more curable stage than under usual clinical diagnostic conditions
• Monitoring of disease response during therapy, with potential for adjusting level of intervention (e.g. dose) on a dynamic and personal basis
• Risk assessment leading to preventive interventions for those at sufficient risk
• Prognosis, allowing for more aggressive therapy for patients with poorer prognosis
• Prediction of safety or efficacy (response) of a therapy, thereby providing guidance in choice of therapy
![Page 16: Gene Pennello, Ph.D. Team Leader, Diagnostics Devices Branch Division of Biostatistics](https://reader036.vdocuments.us/reader036/viewer/2022081419/568148cc550346895db5e9a4/html5/thumbnails/16.jpg)
16
Types of Biomarkers
• Diagnostic
• Early Detection (screening)
• Monitoring
• Risk Assessment
• Prognostic
• Predictive of Safety or Efficacy
The first three are considered together, where the focus is on identifying the disease or condition.
![Page 17: Gene Pennello, Ph.D. Team Leader, Diagnostics Devices Branch Division of Biostatistics](https://reader036.vdocuments.us/reader036/viewer/2022081419/568148cc550346895db5e9a4/html5/thumbnails/17.jpg)
17
Types of Biomarkers
• Diagnostic• Early Detection (screening)• Monitoring• Risk Assessment• Prognostic• Predictive of Safety or Efficacy
The last three are attempting to predict the future.
![Page 18: Gene Pennello, Ph.D. Team Leader, Diagnostics Devices Branch Division of Biostatistics](https://reader036.vdocuments.us/reader036/viewer/2022081419/568148cc550346895db5e9a4/html5/thumbnails/18.jpg)
18
Analytical Validation
How well are you measuring the measurand?– Precision / Reproducibility– Method Comparison – LoB, LoD, LoQ– Linearity– Stability
Clinical Laboratory Standards Institute (CLSI)(http://www.nccls.org/)
![Page 19: Gene Pennello, Ph.D. Team Leader, Diagnostics Devices Branch Division of Biostatistics](https://reader036.vdocuments.us/reader036/viewer/2022081419/568148cc550346895db5e9a4/html5/thumbnails/19.jpg)
19
Clinical Validation (“Qualification”)
– Does the test have clinical utility?– Does it have added value over standard tests
(e.g, clinical covariates like age, tumor size, stage)?
– May or may not require a clinical study • EX. Roche Amplichip
CDRH guidance document: “Statistical Guidance on Reporting Results from Studies Evaluating Diagnostic Tests” issued in final form in March, 2007, concerns reporting agreement when there is no perfect standard and also discrepancy resolution.
http://www.fda.gov/cdrh/osb/guidance/1620.html
![Page 20: Gene Pennello, Ph.D. Team Leader, Diagnostics Devices Branch Division of Biostatistics](https://reader036.vdocuments.us/reader036/viewer/2022081419/568148cc550346895db5e9a4/html5/thumbnails/20.jpg)
20
Roche AmpliChip CYP450 Test (CDRH de novo 510(k) K042259)
Genotypes two cytochrome P450 genes (29 polymorphisms in CYP2D6 gene, 2 in CYP2C19) to provide the predictive phenotype of the metabolic rate for a class of therapeutics metabolized primarily by CYP2D6 or CYP2C19 gene products. The phenotypes are (1) Poor metabolizers: (3) Extensive metabolizers:(2) Intermediate metabolizers: (4) Ultrarapid metabolizers:
Cytochrome P450s are a large multi-gene family of enzymes found in the liver, and are linked to the metabolism of approximately 70-80% of all drugs. Among them, the polymorphic CYP2D6 and CYP2C19 genes are responsible for approximately 25% of all CYP450-mediated drug metabolism. A polymorphism in these enzymes can lead to an excessive or prolonged therapeutic effect or drug-related toxicity after a typical dose by failing to clear a drug from the blood or by changing the pattern of metabolism to produce toxic metabolites.http://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfPMN/pmn.cfm
![Page 21: Gene Pennello, Ph.D. Team Leader, Diagnostics Devices Branch Division of Biostatistics](https://reader036.vdocuments.us/reader036/viewer/2022081419/568148cc550346895db5e9a4/html5/thumbnails/21.jpg)
21
Adding Value to Standard Clinical Predictors
1) Head to Head: Marker superior to clinical predictors at predicting outcome.
2) Incremental Improvement: Combination superior to clinical predictors alone.
3) Marker Predictive within Clinical Strata: e.g., HR(+, –) significant within age, tumor grade, tumor size groups.
![Page 22: Gene Pennello, Ph.D. Team Leader, Diagnostics Devices Branch Division of Biostatistics](https://reader036.vdocuments.us/reader036/viewer/2022081419/568148cc550346895db5e9a4/html5/thumbnails/22.jpg)
22
Multivariate Index Assays
• An IVDMIA is a device that:– Combines the values of multiple variables using an
interpretation function to yield a single, patient-specific result (e.g., a “classification,” “score,” “index,” etc.), that is intended for use in the diagnosis of disease or other conditions, or in the cure, mitigation, treatment or prevention of disease, and
– Provides a result whose derivation is non-transparent and cannot be independently derived or verified by the end user. MIA result could be a binary (dichotomous) (such as yes or no), categorical (such as disease type), ordinal (such as low, medium, high) or a continuous scale.
– Source: FDA MIA Draft Guidancehttp://www.fda.gov/cdrh/oivd/guidance/1610.html
![Page 23: Gene Pennello, Ph.D. Team Leader, Diagnostics Devices Branch Division of Biostatistics](https://reader036.vdocuments.us/reader036/viewer/2022081419/568148cc550346895db5e9a4/html5/thumbnails/23.jpg)
23
Typical Endpoints for Prognostic or Predictive Biomarkers
1. Time to Event
2. Event by Time t
Treatment Median Survival Time
A 6 months
B 12 months
Hazard Ratio 0.5
Treatment R Not R Response Rate
A 30 30 0.50 (30/60)
B 10 50 0.13 (10/60)
![Page 24: Gene Pennello, Ph.D. Team Leader, Diagnostics Devices Branch Division of Biostatistics](https://reader036.vdocuments.us/reader036/viewer/2022081419/568148cc550346895db5e9a4/html5/thumbnails/24.jpg)
24
Relative Risk vs. Diagnostic Accuracy†
Relative Risk looks good, but Dx accuracy not great → limited clinical utility?
E No E
+ 30 30 60
– 10 50 60
40 80 120
Mar
ker
Event by Time t RelativeRisk
3.0 = (30/60)/(10/60)
Se 0.75 (30/40)
Sp 0.63 (50/80)
PPV 0.50 (30/60)
NPV 0.83 (50/60)
†Example taken from Emir, Wieand, Su, Cha, Analysis of repeated markers used to predict progression of cancer Statist. Med., 17, 2563-78, 1998.
![Page 25: Gene Pennello, Ph.D. Team Leader, Diagnostics Devices Branch Division of Biostatistics](https://reader036.vdocuments.us/reader036/viewer/2022081419/568148cc550346895db5e9a4/html5/thumbnails/25.jpg)
25
Hazard Ratio vs. Diagnostic Accuracy†
• NCCTG Mayo Clinic Study. CA15-3 ratio as diagnostic for progression of breast cancer (as determined by physical exam).
†Example taken from Emir, Wieand, Su, Cha, Analysis of repeated markers used to predict progression of cancer Statist. Med., 17, 2563-78, 1998.
Hazard Ratio 2.3 (p = 0.0002)
Se 0.30 (0.17,0.43)
Sp 0.82 (0.74,0.89)
PPV 0.27 (0.21,0.33)
![Page 26: Gene Pennello, Ph.D. Team Leader, Diagnostics Devices Branch Division of Biostatistics](https://reader036.vdocuments.us/reader036/viewer/2022081419/568148cc550346895db5e9a4/html5/thumbnails/26.jpg)
26
Diagnostic Performance
Sensitivity Specificity (TP rate): (TN rate): FP rate: fraction of fraction of fraction of responders non-responders non-responders who test + who test – who test +
Test is useful if TP rate > FP rate, i.e., sensitivity + specificity > 1.
EX. Useless test: sensitivity 0.80, specificity 0.20
![Page 27: Gene Pennello, Ph.D. Team Leader, Diagnostics Devices Branch Division of Biostatistics](https://reader036.vdocuments.us/reader036/viewer/2022081419/568148cc550346895db5e9a4/html5/thumbnails/27.jpg)
27
Diagnostic Performance
Positive Negative predictive predictive value (PPV): value (NPV): 1 – NPV:fraction of fraction of fraction of test +’s who test –’s who test –’s whorespond don’t respond respond
Test is useful if PPV + NPV > 1
EX. Useless test: PPV 0.60, NPV 0.40
![Page 28: Gene Pennello, Ph.D. Team Leader, Diagnostics Devices Branch Division of Biostatistics](https://reader036.vdocuments.us/reader036/viewer/2022081419/568148cc550346895db5e9a4/html5/thumbnails/28.jpg)
28
d
A ROC curve is a plot of sensitivity (true positive rate) vs. 1-specificity (false positive rate) over all possible cutoff points for the test. The test is informative if the area under the curve is greater than 0.5.
![Page 29: Gene Pennello, Ph.D. Team Leader, Diagnostics Devices Branch Division of Biostatistics](https://reader036.vdocuments.us/reader036/viewer/2022081419/568148cc550346895db5e9a4/html5/thumbnails/29.jpg)
29
Prognostic Biomarker (Strong Def’n)
Prognostic factor. Informs about an outcome independent of specific treatment (ability of tumor to proliferate, invade, and/or spread).
Prognostic biomarker is associated with likelihood of an outcome (e.g., survival, response, recurrence) such that magnitude of association is independent of treatment.
On some scale, treatment and biomarker effects are additive, that is, do not interact.
![Page 30: Gene Pennello, Ph.D. Team Leader, Diagnostics Devices Branch Division of Biostatistics](https://reader036.vdocuments.us/reader036/viewer/2022081419/568148cc550346895db5e9a4/html5/thumbnails/30.jpg)
30
HR(A,B)=0.67
HR(A,B)=0.67
![Page 31: Gene Pennello, Ph.D. Team Leader, Diagnostics Devices Branch Division of Biostatistics](https://reader036.vdocuments.us/reader036/viewer/2022081419/568148cc550346895db5e9a4/html5/thumbnails/31.jpg)
31
![Page 32: Gene Pennello, Ph.D. Team Leader, Diagnostics Devices Branch Division of Biostatistics](https://reader036.vdocuments.us/reader036/viewer/2022081419/568148cc550346895db5e9a4/html5/thumbnails/32.jpg)
32
Prognostic Biomarker (Weak Def’n)
Prognostic factor. Informs about an outcome independent of specific treatment (ability of tumor to proliferate, invade, and/or spread).
Prognostic biomarker is associated with likelihood of an outcome (e.g., survival, response, recurrence) in a population that is untreated or on a “standard” (non-targeted) treatment. If population is clearly defined, than can use to choose more or less aggressive therapy, but not specific therapies, per se.
![Page 33: Gene Pennello, Ph.D. Team Leader, Diagnostics Devices Branch Division of Biostatistics](https://reader036.vdocuments.us/reader036/viewer/2022081419/568148cc550346895db5e9a4/html5/thumbnails/33.jpg)
33
HR(A,B)=0.67
HR(A,B)=0.67
![Page 34: Gene Pennello, Ph.D. Team Leader, Diagnostics Devices Branch Division of Biostatistics](https://reader036.vdocuments.us/reader036/viewer/2022081419/568148cc550346895db5e9a4/html5/thumbnails/34.jpg)
34
Prognostic Biomarker
• Her2-neu for node-negative women with breast cancer – prognostic for recurrence
• Breast cancer prognostic test based on microarray gene expression of RNAs extracted from breast tumor tissue to assess a patient’s risk for distant metastasis for women less than 61 with Stage I or II disease with tumor size less than or equal 5.0 cm and who are lymph node negative.
(Ref.: Buyse et al. JNCI 98, 1183-1192)
![Page 35: Gene Pennello, Ph.D. Team Leader, Diagnostics Devices Branch Division of Biostatistics](https://reader036.vdocuments.us/reader036/viewer/2022081419/568148cc550346895db5e9a4/html5/thumbnails/35.jpg)
35
Agendia Mammaprint Gene Signature for Time to Distant
Metastasis (N=302)
Year
Pro
babi
lity
0.0
0.2
0.4
0.6
0.8
1.0
0 2 4 6 8 10 12 14
Patients Events Risk group
111 18 Gene signature low risk191 58 Gene signature high risk
111 108 102 95 92 80 64 43191 169 151 136 117 103 84 49
Number at risk
5-year:Low risk group: 0.95 (0.91-0.99)High risk group: 0.78 (0.72-0.84)
10-year:Low risk group: 0.90 (0.85-0.96)High risk group:0.71 (0.65-0.78)
Buyse et al JNCI (2006), 98,1183-1192
![Page 36: Gene Pennello, Ph.D. Team Leader, Diagnostics Devices Branch Division of Biostatistics](https://reader036.vdocuments.us/reader036/viewer/2022081419/568148cc550346895db5e9a4/html5/thumbnails/36.jpg)
36
Proportion alive at 10 years
Clinical Gene N Proportion*Signature
Low Risk Low Risk 52 0.88 (0.74 to 0.95) SpLow Risk High Risk 28 0.69 (0.45 to 0.84) 1–Se High Risk Low Risk 59 0.89 (0.77 to 0.95) SpHigh Risk High Risk 163 0.69 (0.61 to 0.76) 1–Se
*Buyse et al JNCI 2006
![Page 37: Gene Pennello, Ph.D. Team Leader, Diagnostics Devices Branch Division of Biostatistics](https://reader036.vdocuments.us/reader036/viewer/2022081419/568148cc550346895db5e9a4/html5/thumbnails/37.jpg)
37
Predictive BiomarkerPredictive factor. Implies relative sensitivity
or resistance to specific treatments or agents.
Predictive biomarker predicts differential effect of treatment on outcome.
Treatment and biomarker interact.
Predictive biomarker can be useful for selecting specific therapy.
![Page 38: Gene Pennello, Ph.D. Team Leader, Diagnostics Devices Branch Division of Biostatistics](https://reader036.vdocuments.us/reader036/viewer/2022081419/568148cc550346895db5e9a4/html5/thumbnails/38.jpg)
38
HR(A,B)=0.5
HR(A,B)=1.0
![Page 39: Gene Pennello, Ph.D. Team Leader, Diagnostics Devices Branch Division of Biostatistics](https://reader036.vdocuments.us/reader036/viewer/2022081419/568148cc550346895db5e9a4/html5/thumbnails/39.jpg)
39
Predictive Biomarker of Efficacy
Marker: HER2/neuTreatment: Trastuzumab (Herceptin)
Objective response rate:
Herceptin+Chemo ChemoFISH+ 95/176 (54%) 51/168 (30%)FISH- 19/50 (38%) 22/57 (39%)Arch. Pathol. Lab Med Jan 2007 (ASCO/CAP Guidelines)
![Page 40: Gene Pennello, Ph.D. Team Leader, Diagnostics Devices Branch Division of Biostatistics](https://reader036.vdocuments.us/reader036/viewer/2022081419/568148cc550346895db5e9a4/html5/thumbnails/40.jpg)
40
Predictive Biomarkers for Safety
• Predict risk of an adverse event dependent on the biomarker
• Example– UGT1A1, cleared by FDA, to predict the risk
of neutropenia in patients taking irinotecan for colorectal cancer
![Page 41: Gene Pennello, Ph.D. Team Leader, Diagnostics Devices Branch Division of Biostatistics](https://reader036.vdocuments.us/reader036/viewer/2022081419/568148cc550346895db5e9a4/html5/thumbnails/41.jpg)
41
Prospective Study Designs for Predictive Markers
• Untargeted Design (Reference)
Validate Treatment, Marker Simultaneously
• Marker by Treatment Design
• Targeted Design (Marker + Subset Only)
• Marker Strategy Design
• Historical Control
![Page 42: Gene Pennello, Ph.D. Team Leader, Diagnostics Devices Branch Division of Biostatistics](https://reader036.vdocuments.us/reader036/viewer/2022081419/568148cc550346895db5e9a4/html5/thumbnails/42.jpg)
42
Untargeted Design (Reference)
• Test if drug works in entire population.
• Mixture of marker + and – drug effects.
• Can store samples if test is not ready.
![Page 43: Gene Pennello, Ph.D. Team Leader, Diagnostics Devices Branch Division of Biostatistics](https://reader036.vdocuments.us/reader036/viewer/2022081419/568148cc550346895db5e9a4/html5/thumbnails/43.jpg)
43
Marker by Treatment (Interaction) Design
• A Randomized Block Design• Can test for biomarker by treatment interaction
(predictive biomarker)• Test needs to be available before trial ensues.
![Page 44: Gene Pennello, Ph.D. Team Leader, Diagnostics Devices Branch Division of Biostatistics](https://reader036.vdocuments.us/reader036/viewer/2022081419/568148cc550346895db5e9a4/html5/thumbnails/44.jpg)
44
Marker by Treatment Design Questions
• Test Drug Overall and within Marker + Subset– 0.04, 0.01 tests suggested to control Type I error rate
at 0.05 (Simon), but subset could drive overall result.– Frequentist multiplicity penalty may preclude subset
testing as good business strategy.– Statement about drug, not biomarker
• Test Marker Overall and within Drug Subset– Statement about marker, not drug.
• Test for Treatment by Marker Interaction– Simultaneously validates drug and marker.
![Page 45: Gene Pennello, Ph.D. Team Leader, Diagnostics Devices Branch Division of Biostatistics](https://reader036.vdocuments.us/reader036/viewer/2022081419/568148cc550346895db5e9a4/html5/thumbnails/45.jpg)
45
Targeted Design
Test if drug works in subset.Cannot test if marker discriminates. Only
PPV available.
![Page 46: Gene Pennello, Ph.D. Team Leader, Diagnostics Devices Branch Division of Biostatistics](https://reader036.vdocuments.us/reader036/viewer/2022081419/568148cc550346895db5e9a4/html5/thumbnails/46.jpg)
46
Efficiency of Designs
Efficiency gain depends on marker prevalence, relative efficacy, and difference tested.
Relative Efficiency
Marker Prevalence
Relative Efficacy*
Targeted Design†
Interaction Design ††
25% 0% 16x 8x
50% 0% 4x 2x
75% 0% 1.8x 0.9x
* Marker – to Marker + Patients†Simon & Maitournam, CCR 2004†† Marker by Treatment Design: Test for Interaction approx. efficiency enriching with half +’s, half –’s.
![Page 47: Gene Pennello, Ph.D. Team Leader, Diagnostics Devices Branch Division of Biostatistics](https://reader036.vdocuments.us/reader036/viewer/2022081419/568148cc550346895db5e9a4/html5/thumbnails/47.jpg)
47
Efficiency of Designs
Efficiency gain depends on marker prevalence, relative efficacy, and difference tested.
Relative Efficiency
Marker Prevalence
Relative Efficacy*
Targeted Design†
Interaction Design ††
25% 25% 5.2x 1.5x
50% 25% 2.6x 0.7x
75% 25% 1.5x 0.4x
* Marker – to Marker + Patients†Simon & Maitournam, CCR 2004†† Marker by Treatment Design: Test for Interaction approx. efficiency enriching with half +’s, half –’s.
![Page 48: Gene Pennello, Ph.D. Team Leader, Diagnostics Devices Branch Division of Biostatistics](https://reader036.vdocuments.us/reader036/viewer/2022081419/568148cc550346895db5e9a4/html5/thumbnails/48.jpg)
48
Efficiency of Designs
Efficiency gain depends on marker prevalence, relative efficacy, and difference tested.
Relative Efficiency
Marker Prevalence
Relative Efficacy*
Targeted Design†
Interaction Design ††
25% 50% 2.5x 0.3x
50% 50% 1.8x 0.2x
75% 50% 1.3x 0.1x
* Marker – to Marker + Patients†Simon & Maitournam, CCR 2004†† Marker by Treatment Design: Test for Interaction approx. efficiency when enriching with half +’s, half –’s.
![Page 49: Gene Pennello, Ph.D. Team Leader, Diagnostics Devices Branch Division of Biostatistics](https://reader036.vdocuments.us/reader036/viewer/2022081419/568148cc550346895db5e9a4/html5/thumbnails/49.jpg)
49
Improving Efficiency of Interaction Design
• Enrich with Test Positives if Pr(+) is low
• Find scale such that marker and treatment effects are additive
• Adaptive Randomization
• Bayesian subset analysis
• If reader variability (e.g., IHC), then use multiple readers.
• Prior Information
![Page 50: Gene Pennello, Ph.D. Team Leader, Diagnostics Devices Branch Division of Biostatistics](https://reader036.vdocuments.us/reader036/viewer/2022081419/568148cc550346895db5e9a4/html5/thumbnails/50.jpg)
50
Possibilities for Increasing Efficiency of Interaction Design
• Enrich with Test Positives if Pr(+) is low– Estimates of Sensitivity and Specificity are
biased because they depend on Pr(+). – Use inverse probability weighting (Horvitz,
Thompson, 1952) or Bayes Theorem (Begg, Greenes, 1983) to obtain unbiased estimates.
![Page 51: Gene Pennello, Ph.D. Team Leader, Diagnostics Devices Branch Division of Biostatistics](https://reader036.vdocuments.us/reader036/viewer/2022081419/568148cc550346895db5e9a4/html5/thumbnails/51.jpg)
51
A Marker-Based Strategy
Pro: More ethical, perhaps.More patients given experimental drug.Test utility based on PPVE, NPVE.
Con: Cannot assess test-treatment interaction.
![Page 52: Gene Pennello, Ph.D. Team Leader, Diagnostics Devices Branch Division of Biostatistics](https://reader036.vdocuments.us/reader036/viewer/2022081419/568148cc550346895db5e9a4/html5/thumbnails/52.jpg)
52
Marker-Based Strategy
R Not R
E a b
P 0 0
Tes
t +
Response
R Not R
E c d
P e f
Tes
t –
E Naïve E Unb’d
Se a / (a+c) a / (a+2c)
Sp d / (d+b) 2d / (2d+b)
PPV a / (a+b) same
NPV d / (c+d) same
![Page 53: Gene Pennello, Ph.D. Team Leader, Diagnostics Devices Branch Division of Biostatistics](https://reader036.vdocuments.us/reader036/viewer/2022081419/568148cc550346895db5e9a4/html5/thumbnails/53.jpg)
53
A Marker-Based Strategy
R Not R
E 20 20 40
P 0 0 0
20 20 40
Tes
t +
Response
R Not R
E 23 157 180
P 24 156 180
46 314 360
Tes
t –
E Naïve E Unb’d
Se 20/43(0.47)
20/66
(0.30)
Sp 157/177(0.89)
314/334(0.94)
PPV 20/40(0.50)
Same
NPV 157/180(0.88)
Same
![Page 54: Gene Pennello, Ph.D. Team Leader, Diagnostics Devices Branch Division of Biostatistics](https://reader036.vdocuments.us/reader036/viewer/2022081419/568148cc550346895db5e9a4/html5/thumbnails/54.jpg)
54
Possibilities for Increasing Efficiency of Interaction Design
• Transformation– Find a transformation (Box-Cox?) of outcome
that makes treatment and effects additive.– Can then pool marker effect estimates within
treatments A and B.– Can also pool drug effect estimates within
marker + and marker – ‘s.
![Page 55: Gene Pennello, Ph.D. Team Leader, Diagnostics Devices Branch Division of Biostatistics](https://reader036.vdocuments.us/reader036/viewer/2022081419/568148cc550346895db5e9a4/html5/thumbnails/55.jpg)
55
Possibilities for Increasing Efficiency of Interaction Design
• Adaptive Randomization– Adapt randomization ratio to treatment A and
B within biomarker subsets to maximize (a) power, or (b) fraction of patients on better treatment
– If response rate < 0.5 for both treatments, then (a) and (b) are compatible, otherwise in tension.
– Pr(+) disturbed, so need to adjust Se, Sp
![Page 56: Gene Pennello, Ph.D. Team Leader, Diagnostics Devices Branch Division of Biostatistics](https://reader036.vdocuments.us/reader036/viewer/2022081419/568148cc550346895db5e9a4/html5/thumbnails/56.jpg)
56
Possibilities for Increasing Efficiency of Interaction Design
• Bayesian subset analysis (cf. Dixon, Simon)– Subsets modeled as exchangeable via
random effects. – Subset estimate borrows strength from
complement subset, increasing precision of estimate.
– However, interaction estimate more conservative relative to usual non-Bayesian analysis.
![Page 57: Gene Pennello, Ph.D. Team Leader, Diagnostics Devices Branch Division of Biostatistics](https://reader036.vdocuments.us/reader036/viewer/2022081419/568148cc550346895db5e9a4/html5/thumbnails/57.jpg)
57
Bayesian Subset Analysis
• Power is enhanced to show drug works in marker + subset (blue).
• Power is enhanced to show marker works (discriminates) in patients taking drug (red)
![Page 58: Gene Pennello, Ph.D. Team Leader, Diagnostics Devices Branch Division of Biostatistics](https://reader036.vdocuments.us/reader036/viewer/2022081419/568148cc550346895db5e9a4/html5/thumbnails/58.jpg)
58
Possibilities for Increasing Efficiency of Interaction Design
• Use Multiple Readers– EGFR IHC test (Dako) and Cetuximab and
Panitumumab (Amgen) for Colorectal Cancer. % of cells stained and maximum staining intensity subject to reader variability
– Use multiple readers, account for random reader effects.
• Multiple Reader, Multiple Case Designs (MRMC) are used for digital mammography systems and computed aided detection (CAD) systems
• Analysis can be difficult.
![Page 59: Gene Pennello, Ph.D. Team Leader, Diagnostics Devices Branch Division of Biostatistics](https://reader036.vdocuments.us/reader036/viewer/2022081419/568148cc550346895db5e9a4/html5/thumbnails/59.jpg)
59
Possibilities for Increasing Efficiency of Interaction Design
• Prior Information (Bayesian analysis)– Borrow strength from previous study regarded
as exchangeable with current study.
![Page 60: Gene Pennello, Ph.D. Team Leader, Diagnostics Devices Branch Division of Biostatistics](https://reader036.vdocuments.us/reader036/viewer/2022081419/568148cc550346895db5e9a4/html5/thumbnails/60.jpg)
60
Register
Marker Based Strategy
Non Marker Based Strategy
Treatment A
Marker Level (-)
Treatment A
Marker Level (+) Treatment B
Test Marker
Sargent et al., JCO 2005
Marker Based Strategy Design
Randomize
![Page 61: Gene Pennello, Ph.D. Team Leader, Diagnostics Devices Branch Division of Biostatistics](https://reader036.vdocuments.us/reader036/viewer/2022081419/568148cc550346895db5e9a4/html5/thumbnails/61.jpg)
61
Register
Marker Based Strategy
Non Marker Based Strategy
Randomize
Treatment A
Treatment B
Marker Level (-)
Treatment A
Marker Level (+) Treatment B
Test Marker
Sargent et al., JCO 2005
Marker Based Strategy Design
Randomize
![Page 62: Gene Pennello, Ph.D. Team Leader, Diagnostics Devices Branch Division of Biostatistics](https://reader036.vdocuments.us/reader036/viewer/2022081419/568148cc550346895db5e9a4/html5/thumbnails/62.jpg)
62
Marker Based Strategy Design
• Lacks power: Differential effect comparison diluted because some patients in non-marker-based strategy arm get marker-based treatment (could eliminate these to increase power).
• Might be best suited if have > 2 treatments or > 2 markers– EX. Irinotecan regiment (dose, timing,
frequency) determined by UGT1A1 genotype (6/6, 6/7, or 7/7) in colorectal cancer patients.
![Page 63: Gene Pennello, Ph.D. Team Leader, Diagnostics Devices Branch Division of Biostatistics](https://reader036.vdocuments.us/reader036/viewer/2022081419/568148cc550346895db5e9a4/html5/thumbnails/63.jpg)
63
Marker Based Strategy
• If no gold standard, then can be only way to assess effectiveness of a test.
• EX. Detection tumor of origin in cancers of unknown primary. – No gold standard: IHC, imaging, may fail to
identify TOO. – Randomize patients to be managed with
• new test + standard, or• with standard alone
– Compare arms on survival
![Page 64: Gene Pennello, Ph.D. Team Leader, Diagnostics Devices Branch Division of Biostatistics](https://reader036.vdocuments.us/reader036/viewer/2022081419/568148cc550346895db5e9a4/html5/thumbnails/64.jpg)
64
Targeted Design w. Historical Control
• Drug already on market, but has poor response rate.
• If response rate in marker + study is significantly greater than historical rate, then marker discriminates.
• Limitations– Lacks power because effect diluted.– Need to calibrate historical rate to marker +
study (adjust for covariates).
![Page 65: Gene Pennello, Ph.D. Team Leader, Diagnostics Devices Branch Division of Biostatistics](https://reader036.vdocuments.us/reader036/viewer/2022081419/568148cc550346895db5e9a4/html5/thumbnails/65.jpg)
65
Prospective-Retrospective Designs
• Prospectively apply marker to stored samples (in retrospect).
• Can test overall, w. subset, or for interaction.• Missing samples could introduce bias.• RCT samples. Randomization ensures case and
control samples have similar characteristics. • Case-control samples. Avoid selection bias by
matching on sample processing date, processing sites, etc., and not excluding censored times.
• Reserve samples only for analytically validated markers that are biologically plausible.
![Page 66: Gene Pennello, Ph.D. Team Leader, Diagnostics Devices Branch Division of Biostatistics](https://reader036.vdocuments.us/reader036/viewer/2022081419/568148cc550346895db5e9a4/html5/thumbnails/66.jpg)
66
The Challenge of Multiplicity
• Multiplicity of classifiers• Microarrays and proteomics• Many predictive models could be built with
so many inputs.• The challenge is to confirm any such
model with an independent data set.• A caveat: the independent test data set
cannot be continually reused. Great discipline is required in this regard.
![Page 67: Gene Pennello, Ph.D. Team Leader, Diagnostics Devices Branch Division of Biostatistics](https://reader036.vdocuments.us/reader036/viewer/2022081419/568148cc550346895db5e9a4/html5/thumbnails/67.jpg)
67
Cross-Validation Pitfall
Simon, Radmacher, Dobbin, McShane (2003), Pitfalls in the Use of DNA Microarray Data for Diagnostic and Prognostic Classification, JNCI, 95 (1)
![Page 68: Gene Pennello, Ph.D. Team Leader, Diagnostics Devices Branch Division of Biostatistics](https://reader036.vdocuments.us/reader036/viewer/2022081419/568148cc550346895db5e9a4/html5/thumbnails/68.jpg)
68
Summary Remarks• How to assess a test or biomarker is well-
known, but not as well-known in therapeutic circles.
• Need to assess whether the biomarker adds anything to what we already know.
• The number of possibly good biomarker candidates is enormous but great care is needed in restricting the search.
![Page 69: Gene Pennello, Ph.D. Team Leader, Diagnostics Devices Branch Division of Biostatistics](https://reader036.vdocuments.us/reader036/viewer/2022081419/568148cc550346895db5e9a4/html5/thumbnails/69.jpg)
69
Summary Remarks• Need to encourage least burdensome
approaches to validating biomarkers without compromising level of evidence
• Essential to confirm marker in independent dataset
• Studies to demonstrate informativeness of a biomarker can be quite difficult to design, conduct and analyze.
![Page 70: Gene Pennello, Ph.D. Team Leader, Diagnostics Devices Branch Division of Biostatistics](https://reader036.vdocuments.us/reader036/viewer/2022081419/568148cc550346895db5e9a4/html5/thumbnails/70.jpg)
70
Acknowledgements
• CDRH Division of Biostatistics (DBS)– Greg Campbell, Division Director– Diagnostic Devices Branch (DDB)
• Lakshmi Vishnuvajjala, Branch Chief• Estelle Russek-Cohen, Team Leader• Gene Pennello, Team Leader
Bipasa Biswas Kyungsook Kim,Harry Bushar Samir LababidiArkendra De Kristen MeierShanti Gomatam Kyunghee SongThomas Gwise Rong Tang
![Page 71: Gene Pennello, Ph.D. Team Leader, Diagnostics Devices Branch Division of Biostatistics](https://reader036.vdocuments.us/reader036/viewer/2022081419/568148cc550346895db5e9a4/html5/thumbnails/71.jpg)
71
More References
• Sargent et al (2005). Clinical trial designs for predictive marker validation in cancer treatment trials. J Clin Oncol 23:2020-2027.
• Pennello & Vishnuvajjala (2005). Statistical design and analysis issues with pharmacogenomic drug-diagnostic co-development, In American Stat. Assoc. 2005 Proc. of the Biopharm. Section, Joint Statistical Meetings, Minneapolis, MN, August, 2005; American Stat. Assoc.: Alexandria, VA.
• FDA Drug-Diagnostic Co-Development Concept Paper. April 2005.http://www.fda.gov/cder/genomics/pharmacoconceptfn.pdf
![Page 72: Gene Pennello, Ph.D. Team Leader, Diagnostics Devices Branch Division of Biostatistics](https://reader036.vdocuments.us/reader036/viewer/2022081419/568148cc550346895db5e9a4/html5/thumbnails/72.jpg)
72