sweitzer,simulating multi phase studies
DESCRIPTION
A presentation on my experience simulating large multiphase trails to predict time of completion based on observed patient status dataTRANSCRIPT
Signs of the Timings: Predicting Time of Completion in
Multiphase Survival Trials
Dennis Sweitzer Ali Falahati
Delaware Chapter of the ASA September, 2006
Study Flowchart
The Protocols Outcome: • Time to Randomized Relapse Open Label Phase
– Up to 36 weeks – Patients must be stable for 12 weeks before randomization – High withdrawal rate (30-70%) – Assumed 50% randomize
Randomized Phase – Up to 104 weeks – High withdrawal rate – Assumed 30% Relapse rate – Trial could not end until last Patient randomized >28 weeks
Sensitivity to Relapse & Discontinuation rates (1)
Cumulative Patient statuses as trial progresses 100 Relapse ~Sep
Wrong assumptions, wait longer
Low Discontinuation relative to Relapse
Sensitivity to Relapse & Discontinuation rates (2)
Higher event rates deplete patient pool
Plan to stop enrollment as soon as certain of reaching 100
~ July
Higher Discontinuation Rate, Lower relapse Rate
Large delays May never reach goal
Stopping Criteria • At least 227 Relapses • All patients still in Randomized Phase complete at least 28 weeks of treatment
Ideally: • 227th Relapses occurs shortly after:
• All patients randomized >28 wks (Per Protocol) • Randomization closed when:
• All enrolled patients randomize or discontinue
28 week Requirement later dropped (Protocol Amendment) Presentation: use 200 Relapses
Stopping Enrollment
The Problems Long Lead times • Up to 36 weeks before randomization • Plus 28 weeks Minimum randomization Ideally: Stop enrollment 64 weeks before target
#Relapses Must account for • Enrollment D/C (30%-70%) • Randomized D/C (D/C Rate ≈ Relapse Rate) • Relapse Rates vary (Higher Relapse Rate Early) • Competing Relapses (D/C vs Relapse) • Sensitivity to rates (Close Rates High Variability)
Stopping Enrollment: Issues Too Early: Fewer Patients Fewer randomized Longer wait for
target #Relapse. May never reach
target #Relapses
Too Late: Higher certainty of
reaching Goal Patients possibly in
Open Label at End Excessive #Relapses at
end of study Ethics of Randomizing
Excess # pts
Many Management Questions When do we stop enrollment while being sure of
eventually getting target # Relapses? When can we stop randomization & ensure reaching the
target? Whats the earliest and latest we can expect to reach the target?
When will all pts be randomized >28wks? When can the trial be halted (required # Relapses & all pts randomized >28 wks)
Estimated Randomization Rate? Estimated Relapse Rate?
How many active patients at the end? How well does the outcome match our assumptions
etc etc
Simulation Solution • Make a stochastic model of the trial • Monthly:
– Base model parameters on blinded data observed to date – Incorporate assumptions where data insufficient – Incorporate uncertainty of parameters – Execute 1000’s of simulations of the trial – Compute statistics from the collection of simulated trials – Repeat with new data
Advantages
Transparency • Modeling assumptions can be:
Specified -- Graphed -- Debated
Data Driven • New Data updates the model • Existing Active Patients are simulated to end • Assumptions become less important as data
accumulates
Vision of Output
• Simulation reports varied according to changing team needs (how many open label patients on June 1? When will we reach 150 Relapses? How many randomized patients at time of 200th Relapse? If we stop randomizing on May 15, how many open lable patiestin will there be?……………………………………………….
Stochastic Modeling Approach
1. Make a cartoon model of a patients progress through the trial
2. What final outcomes are possible?
3. What could happen to the patient?
4. Identify States through which a patient passes
5. Identify Random Processes which take patients between states
Stochastic Model
Enrolling Patients
Open Label Patients
Discontinued Patients (Open
Label Phase)
Relapses
Randomized Patients
Discontinued Patients
(Randomized Phase)
Continuous Time Markov Chain
Markov States: the Bubbles Transitions: the Arrows
Transition Probabilities change with time in state
States
2 Transitory Markov States: ❶ Open Label Phase Randomized Phase 3 Terminal Markov States: ❸ Discontinued from Open Label Phase Discontinued from Randomized Phase Randomized Relapses
Enrolling Patients
Open Label Patients
Discontinued Patients (Open Label
Phase)
Relapses
Randomized Patients
Discontinued Patients (Randomized Phase)
❶)
)
❸))
)
Transition Processes
5 Random Transition Processes: 1. Trial Enrollment (Start Open Label) 2. Discontinuation from Open Label Phase 3. Randomization (from Open Label Phase) 4. Discontinuation from Randomized Phase 5. Randomized Relapses
Enrolling Patients
Open Label Patients
Discontinued Patients (Open Label Phase)
Relapses
Randomized Patients
Discontinued Patients (Randomized Phase) ❶)
)
❸))
)
Trial Enrollment
For each simulated patient, generate a random length of time since the last patient
• Pick an enrollment rate λ (Based on history & judgment) • Assume: #pts/mo ~ Poisson process with mean λ • Time between patients ~ Exponential(1/λ) Can expand enrollment model to evaluate management
options: • Incorporate mixture of site performances • Adding/changing sites during the trial
Enrolling Patients
Open Label Patients
❶)
Markov Process )
)Continuing
Discontinuation
Relapse (or Randomization)
Probability of transitioning from
state i to state j between times s and t
• Aalen-Johansen estimator of the transition probability matrices For and
# obs. Direct transitions from states h to j, visits 1 to t
# pts in state h, just prior to visit t
Aalen-Johansen estimator of Transition Probabilities
Aalen-Johansen & Kaplan-Meier
• Generalization of Kaplan-Meier Estimation to Non-homogeneous Markov Chains
• K-M Estimators easier: – To program (already in SAS) – To understand (Intuitive) – To Explain (Familiar)
• Enrollment: Poisson Process • Open Label Phase: Competing Risk Model
• Ramdomized Phase: Competing Risk Model
Models
0= Still in OL Phase 1= Randomized 2= Discontinue fr OL phase
0= Still in Rand Phase 1= Manic event 2= Depressed event 3= Discontinue fr Rand phase
Competing Risk Model Mutually exclusive events
(e.g., Relapse vs Discontinuation, …) 2 Approaches (Pintilie, 2006) • Jointly distributed Random Variables • Latent failure times
– Assume both events eventually occur – But we only observe the first – Use only marginal distributions – Assuming independence (between events) – But cannot test for independence, if only observing 1st
– Independence: Face validity & Simplest Assumption
Kaplan-Meier Simulation
• Assume event are independent • Model Each process separately using Kaplan-
Meier Estimators • Censor on other event, current time in trial • Simulate each event separately • Earliest of the 2 simulated processes is taken as
simulated outcome • Caveat: Assumes Independent processes
Intuitive, easy to understand, easy to explain
Open Label Transitions 2 Competing
Processes: Discontinuation ❸ Randomization
1. Generate Random Discontinuation time 2. Generate Random Randomization time 3. Use the earliest event
Open Label Patients
Discontinued Patients (Open Label Phase)
Randomized Patients
)
❸)
Randomized Phase Processes
2 Competing Processes: Discontinuation * * Relapse
Choose event as previously described. • Current Open Label Patients are simulated to
randomization or discontinuation • If simulated randomization, then simulate
Randomized Discontinuation or Relapse
Relapses
Randomized Patients
Discontinued Patients (Randomized Phase) )
)
Generic Transition Process Q: When to make the transition? A: First: estimate random transition
function
State "A"
State "B"
1. Generate K-M Survival Functions from data (censoring on all other events)
2. Make assumptions about Survival beyond last event
ŒŽ
?
Simulated Patients
A: Second: Simulate Trials For Each Simulated Trial: • For each simulated patient within a trial
– Pick a random p∈(0,1)
– Interpolate t from the graph, so that (p, t) is on graph
State "A"
State "B"
Q: When to make the transition?
p
t
(p, t)
Simulating Active Patients
For each simulated trial • For each observed patient within state “A” for time s
– Interpolate q∈(0,1) from the graph, so that (q, s) is on graph – Pick a random p∈(0,1)
– Interpolate t from the graph, so that (q*p, t) is on graph
State "A"
State "B" q
s
q*p
t
(q, s) (q*p, t)
q r
t
Incorporating Parameter Uncertainty State "A"
State "B"
For each simulated trial
• Pick a random quantile r∈(0,1)
• Simulate all patients using the r%-tile confidence level of the Kaplan-Meier Curve
Simulates: combinations of high & low estimates of Event and D/C Survival curves
Limitations Requires representative data from all phases • K-M estimates only through last event • Assumptions must be made about hazard rate after last
available event(s) – If assumptions correct, point estimates should be stable while
confidence intervals narrow • Up to date data
– Special reporting of Relapses (faxes with follow up, monitoring)
– IVRS, EDC, monitoring reports • Heterogeneity:
– Earliest sites may not be representative of all sites – Procedures may change (hopefully improve) over time – Regional differences (standards of care, patient attitudes, etc)
Why Not a Parametric Model? Trial Structure: • Events tend to occur
on visits granularity ✭ Continuous
• Visits vary in spacing ✭ Discrete
• Active Tx mixture model Changing Hazard over time
• Must make & defend simplifying assumptions
Diagnostic: Does It Fit?
Survival curves of: Observed data vs. Simulated Data
(Censored Observed, Active OL Pts, Active Rand. Pts, Entirely simulated Pts)
Diagnostics
Plot K-M curves for each event, time in each phase • Review assumptions (long term behavior) • Identify data anomalies • Identify simulation problems
Example: Regional Heterogeneity Regional modeling (Trials A & B):
Parameters varied by region more than by trial – Estimate parameters within regions – Simulate patients with Trial and Region – Summarize results by Trial
In addition to simulations which ignored region
Survival curves followed 2 patterns by
region & trial
Reporting the Simulations For each simulated Trial: • Sort Patient Events by occurrence date (enrollment,
randomization, relapse, etc) For each scenario • Summarize over Event records which fit scenario Examples: • Summarize over all patients enrolled before a potential
enrollment cutoff date. • … over all patients randomized before a cutoff date • Summarize with and without a subset of sites
Changing Questions • Early in Trial
– Are the protocol assumptions accurate? – When to stop enrollment? – Expected # patients (enrolled, randomized, etc)
Identify problems & Evaluate fixes • Mid-Trial
– When to stop randomizing patients? – Are the revised assumptions accurate? – Were changes effective?
• Late Trial – When will the last Relapse occur? – How many patients will be active in various phases?
Plan for Closeout & Database lock
Early Trial For each Enrollment Cutoff Date • Summarize each trial for all patients enrolled
before that date • Compute statistics over simulated trials Some trial outcomes: • Dates of: last Relapse; All patients randomized>28 week;
PP Completion (>target & >28 weeks) • Event & patient counts at each of above milestones • % of Simulations with ≥200, 190, 180,… Relapses at milestones • # active patients (open label or randomized) at given dates &
milestones Pick a cutoff date accordingly (e.g., minimize resource with
least risk of running late)
Trial Completion (1) ≥227 Relapses & All Active Patients Randomized 28wks
Earliest Completion:
• 75% Certainty:
2 August Cutoff for Nov 2006 Completion
• 90% Certainty:
1 Sep Cutoff for Dec 2006 Completion
Trial Completion (2) ≥227 Relapses Events & All Active Patients Randomized 28wks
Earliest Completion:
• 75% Certainty:
~1775 Enrolled for Nov 2006 Completion
• 90% Certainty:
~1850 Enrolled for Dec 2006 Completion
29 Oct prediction from: 528 Enrolled, 272 Rand., & 58 Relapses On 26 June: ~1950 Enrolled, 620 Randomized, 200 Relapses
Highly uncertain: 3 Relapses, 2 Randomized D/C, 73 Randomized
1 Month Later: 16 Relapses, 7 Rand.D/Cs, 182 Randomized
Slowed Enrollment: 29 Relapses, 22 Rand. D/C, 272 Rand
Example: Will a trial end? Study E: • Endpoint: 300 type 1, 300 type 2 events • Slower & Fewer than expected • Simulation predicted:
– 10% chance of 300 of each – 87% chance of 600 total
• Interim Analysis – Supported by simulation – 300 total expected April 25
Mid-Trial Output For each Randomization Cutoff Date • Summarize each trial for all patients
randomized before that date • Compute statistics over simulated trials • Generates same statistics per trial • Summarize for each randomization cutoff date
Essentially, replace “enrollment” with “randomization” & execute as before
• Conclusion: nothing to be gained by randomizing beyond mid June
Late-Trial Output Refine estimates of last Relapse, etc. For Milestones & Calendar Dates • Estimate #patients in each stage (e.g., how many
patients will be active at the end?) Caveats: • Corrected (or just collected) data may change
estimates • Old, unreported Relapses may be discovered Useful: • Predict time between milestone to end • Add prediction to best guess milestone
Bottom line: How accurate? Not bad:
• Actual Date of 200th Relapse covered by predicted 80% C.I.
• Width of C.I.s narrowed over time
Value Added Early Refinement of Protocol Assumptions • Protocol: 50% randomized, 30% Relapse rates • Trial A: 33%, 37% Trial B: 55%, 41% Early Identification of problems Quick Response to problems • Changed procedures to improve retention in Trial A • Added sites to Trial B after delays in starting up sites
Better allocation of resources
Trials C, D Mid trial: Regulators requested analysis of late
Relapses • Enrollment had already ended for Trial C
– Enough patients to reach 88 late Relapses? • Enrollment was still ongoing for Trial D
– Extend enrollment how long? Add sites?
• How would this affect time lines?
Data problem: Unreported Relapses
Dirty Data Problems • Some known Relapses are not usable due to missing data
(e.g., unknown randomization date) • Corrected (or lately collected) data may change estimates • Old, unreported Relapses may be discovered • Data may be collected or corrected irregularly
– Separate data sources (e.g., Relapse log + IVRS) – Drift & shift over time – End of trial data clean up
Solutions: • Estimate time between milestones
– Anchor to a known, early milestone Future solutions: • Estimate missing data effects
– Use time between Relapse occurrence and Relapse reporting – Estimate number of missing Relapses from times in past
Some Feedback • “… the simulations are very valuable
and the only way we have to plan our timelines. As it has turned out, your simulations seems to be pretty accurate as we have increased the mood event rate significantly … as predicted...”
• ... We would have been guessing and spinning our wheels without them.”
• Could you simulate trials xyz & uvw?
Extra Slides
Diagnostics: Cohort Analysis Cohorts: • By month enrolled • By month Randomized
Calculate randomization & Relapse rates
Easy to understand Multiple Estimates which must be reconciled Doesn’t Provide Time to Relapses Useful reality check on Simulation Point Estimates insufficient: need C.I.s
Cohorts by Randomization Month
Actual counts of patients by status by month
Cohorts by Month Randomized (Cumulative Count)
Height is # patients randomized in or before month
Cohorts by Month Randomized (Cumulative Percentage)
• >40% randomized eventually have Relapse <N/0.40 randomized to get N
Open Label Cohorts (Current Status)
Open Label Cohorts (Cumulative Statuses)
• ~ 64% eventually Randomize & >40% Relapse Rate • # open label pt < N/(0.64*0.40)
Solutions? Crude Relapse Rates of all Patients in Phase Mixture of patients: • Relapse & D/C rates change with exposure • Mixture of Pt. Exposures changes with time Cohorts: Track Relapses over Time Easy to understand Multiple Estimates which must be reconciled Doesn’t Provide Time to Relapses Useful reality check on Simulation Point Estimates insufficient: need C.I.s
Clinical Trial Management Planning Trials (future) • Is a trial feasible? • Sensitivity to assumptions? • Costs: # Pts, # Pt-mos, #visits, #Sites, #Site-mos Trial Execution (current) • Anticipate delays • No information on outcome • Could be added to simulation Program Planning (future) • Replace “Trial Phases” with “Toll Gates” • Enhance modeling of Trial Enrollment process
Example: Adding Site
• Drop OL Phase, expand enrollment process • Simulate time to start up new site, pts/mo at a new
site, etc • Report by #Additional Sites instead of cutoff dates
Sites
New Sites
Relapses
Randomized Patients
Discontinued Patients