Follow-up and complianceFollow-up and compliance

• Compliance/adherenceCompliance/adherence– ImportanceImportance

– How to measure and maximizeHow to measure and maximize

• Follow-upFollow-up– Importance of complete follow-upImportance of complete follow-up

– ITT and other issues in analysisITT and other issues in analysis

Follow-up in RCT’sFollow-up in RCT’s

• What happens after randomizationWhat happens after randomization

• Carefully lay out procedures to be followedCarefully lay out procedures to be followed

• Describe on forms and in Operations ManualDescribe on forms and in Operations Manual

• First reaction: do everything on everyone at every visitFirst reaction: do everything on everyone at every visit– e.g. labs at all visitse.g. labs at all visits

– But great opportunities for efficienciesBut great opportunities for efficiencies

– Ask the following:Ask the following:• Do only at some visits?Do only at some visits?

• Do only on a subset?Do only on a subset?

• Don’t do at allDon’t do at all

Large and Simple TrialsLarge and Simple Trials

• Get a whole lot of peopleGet a whole lot of people

• Randomize, do as few follow-up measurements as Randomize, do as few follow-up measurements as possiblepossible

• Difficult to carry out in practiceDifficult to carry out in practice

• ExamplesExamples– Physicians’ Health study: Randomize to aspirin or placebo, mail Physicians’ Health study: Randomize to aspirin or placebo, mail

out drugs, follow-up by mailout drugs, follow-up by mail

– Use data collected for other purposes for follow-up/endpointsUse data collected for other purposes for follow-up/endpoints• Population mortalityPopulation mortality

• Medical systems (Medicare, Kaiser in U.S.; Gov’t. health in Europe)Medical systems (Medicare, Kaiser in U.S.; Gov’t. health in Europe)

Large and Simple Trials: Vitamin DLarge and Simple Trials: Vitamin D

• 2700 men and women from general medical practice

• Oral vitamin D (100,000 units) given 4 x per year (oral) for prevention of fractures in UK

• Sent oral D (and placebo) via mail

• Ascertained fractures via “post” (and National Health Service data base)

• Mortality via National statistics

• Found decrease in fracture risk (possibly mortality)

Effect of four monthly oral vitamin D3 (cholecalciferol) supplementation on fractures and mortality in men and women living in the community: randomised double blind controlled trial.BMJ. 2003 Mar 1; 326(7387): 469.

Large and Simple Trials: Vitamin DLarge and Simple Trials: Vitamin D

• OBJECTIVE: To determine the effect of four monthly vitamin D supplementation on the rate of fractures in men and women aged 65 years and over living in the community.

• DESIGN: Randomised double blind controlled trial of 100 000 IU oral vitamin D3 (cholecalciferol) supplementation or matching placebo every four months over five years.

• SETTING AND PARTICIPANTS: 2686 people (2037 men and 649 women) aged 65-85 years living in the general community, recruited from the British doctors register and a general practice register in Suffolk.

• MAIN OUTCOME MEASURES: Fracture incidence and total mortality by cause.

• RESULTS: After five years 268 men and women had incident fractures, of whom 147 had fractures in common osteoporotic sites (hip, wrist or forearm, or vertebrae). Relative risks in the vitamin D group compared with the placebo group were 0.78 (95% confidence interval 0.61 to 0.99, P=0.04) for any first fracture and 0.67 (0.48 to 0.93, P=0.02) for first hip, wrist or forearm, or vertebral fracture. 471 participants died. The relative risk for total mortality in the vitamin D group compared with the placebo group was 0.88 (0.74 to 1.06, P=0.18). Findings were consistent in men and women and in doctors and the general practice population.

• CONCLUSION: Four monthly supplementation with 100 000 IU oral vitamin D may prevent fractures without adverse effects in men and women living in the general community.

Vitamin D: Summary of 5 Yr. ResultsVitamin D: Summary of 5 Yr. Results

VIT DVIT D PBOPBO RR (p)RR (p)

(n=1345)(n=1345) (n=1341)(n=1341)

Any fractureAny fracture 8.8%8.8% 11.1%11.1% 0.8 (.04)0.8 (.04)

Hip, wrist, vert.Hip, wrist, vert. 4.5%4.5% 6.5%6.5% 0.67 (.02)0.67 (.02)

MortalityMortality 16.7%16.7% 18.4%18.4% 0.88 (.18)0.88 (.18)

Large and Simple Trials: Vitamin DLarge and Simple Trials: Vitamin D

• Efficiencies:– No selection criteria (other than age)

– No follow-up clinic visits or measurements

– No follow-up labs or measurements

– Outcomes from self-assessed mail-in

• Study very efficient (VERY) and very generalizable– Cost of Vitamin D trial: about $500,000.

– Cost of WHI: about $1,000,000,000.

Effect of four monthly oral vitamin D3 (cholecalciferol) supplementation on fractures and mortality in men and women living in the community: randomised double blind controlled trial.BMJ. 2003 Mar 1; 326(7387): 469.

Compliance or (mpc) adherenceCompliance or (mpc) adherence

• Trial is meaningless unless participants adhere to Trial is meaningless unless participants adhere to interventionsinterventions

• Term “drop out” often used but is ambiguousTerm “drop out” often used but is ambiguous

• Two aspectsTwo aspects– 1. Adherence to medications/interventions1. Adherence to medications/interventions

– 2. Adherence to visit schedules/reporting2. Adherence to visit schedules/reporting

• Lack of adherence leads to:Lack of adherence leads to:– BiasBias

– Decreased powerDecreased power

– Uninterpretable resultsUninterpretable results

Potential effect of incomplete Potential effect of incomplete visit follow-up on results in visit follow-up on results in

clinical trialsclinical trials- Fracture Intervention Trial (alendronate vs. placebo)- Fracture Intervention Trial (alendronate vs. placebo)

- X-rays obtained at baseline, 2 years, 3 years- X-rays obtained at baseline, 2 years, 3 years

- Vertebral fractures defined from changes in radiographs- Vertebral fractures defined from changes in radiographs

- FU radiographs on 97% of participants @ year 3- FU radiographs on 97% of participants @ year 3

Time (yrs)Time (yrs) Relative risk (CIRelative risk (CI))

BLBL to to 22 0.34 (66% reduction)0.34 (66% reduction)

BLBL to to 33 0.49 ( 51% reduction)0.49 ( 51% reduction)

Effect of Incomplete Follow-up: Effect of Incomplete Follow-up: Virtual ExperimentVirtual Experiment

• FIT I: Follow-up x-rays on 97% of surviving FIT I: Follow-up x-rays on 97% of surviving participants at year 3participants at year 3

• What if follow-up less complete?What if follow-up less complete?

• Randomly “lose” 50% between year 2 and 3Randomly “lose” 50% between year 2 and 3

Use of Survival Analysis for X-Use of Survival Analysis for X-Rays Rays

in FIT I: Virtual Experimentin FIT I: Virtual ExperimentTime (yrs)Time (yrs) Relative riskRelative risk

2 2 0.340.34

3 3 0.490.49

3 (50% LTFU) 3 (50% LTFU) 0.37 0.37

LTFU = Lost to follow-upLTFU = Lost to follow-up

Effect of High Rate of Loss Effect of High Rate of Loss to Follow-up on Resultsto Follow-up on Results

• If early results differ from later results, could create If early results differ from later results, could create bias when comparing one study to anotherbias when comparing one study to another

• Even a “random” (therefore unbiased) loss to Even a “random” (therefore unbiased) loss to follow-up can affect resultsfollow-up can affect results

Effect of High Rate of Loss Effect of High Rate of Loss to Follow-up on Resultsto Follow-up on Results

• More generally, people lost to follow-up may be More generally, people lost to follow-up may be different than those who remaindifferent than those who remain

• Could be differential in two treatment groupsCould be differential in two treatment groups– Due to treatment (e.g. estrogen)Due to treatment (e.g. estrogen)

• Advantages of randomization may be lostAdvantages of randomization may be lost– Groups may no longer be comparableGroups may no longer be comparable

• Could bias resultsCould bias results

Measuring adherenceMeasuring adherence

• Medication-takingMedication-taking– Just ask! (self report)Just ask! (self report)

– Pill countsPill counts

– Biochemical assays for some drugsBiochemical assays for some drugs

– High tech pill bottlesHigh tech pill bottles

• Visit scheduleVisit schedule– N missed visitsN missed visits

– Visits within scheduleVisits within schedule

– etc.etc.

Adherence goalsAdherence goals

• Ideal: all participants continue to take medication Ideal: all participants continue to take medication (perfectly) throughout the trial and attend all follow-up (perfectly) throughout the trial and attend all follow-up visits until the very endvisits until the very end

• Why might participants stop medication?Why might participants stop medication?– Side effects (real or perceived)Side effects (real or perceived)

– Complex regimensComplex regimens

– Want to take true active medicationWant to take true active medication• New info on old medicationNew info on old medication

• New competing medicationNew competing medication

– Want to stop active medicationWant to stop active medication• New info on old medication (e.g, ERT increases BC risk)New info on old medication (e.g, ERT increases BC risk)

Some Examples of “Bad Adherence Days”Some Examples of “Bad Adherence Days”

• Women’s Health InitiativeWomen’s Health Initiative– After first year, letter sent to all participants “observed a small After first year, letter sent to all participants “observed a small

increase in cardiovascular disease among ppts on HRT”…increase in cardiovascular disease among ppts on HRT”…

– Many stopped medicationsMany stopped medications

• PROOF trial (effect of Calcitonin on osteoporosis)PROOF trial (effect of Calcitonin on osteoporosis)– 1994 to 19991994 to 1999

– 1997: Alendronate approved with significant marketing and 1997: Alendronate approved with significant marketing and excellent resultsexcellent results

Effect of Stopping Medication: Effect of Stopping Medication: Classical interpretationClassical interpretation

• Placebo’s start active medication==>become more like Placebo’s start active medication==>become more like activesactives

• Actives stop active medication and start Actives stop active medication and start “inactive”==>become more like placebo“inactive”==>become more like placebo

• Two groups become more similarTwo groups become more similar

• Treatment effect is underestimated/conservativeTreatment effect is underestimated/conservative– ComfortingComforting

• ““Classical interpretation” may not hold:Classical interpretation” may not hold:– Example: patients stop study meds to take a medication that is Example: patients stop study meds to take a medication that is

better than active study medicationbetter than active study medication

Strategies to enhance complianceStrategies to enhance compliance

• Warm and fuzzy stuffWarm and fuzzy stuff– Participants to feel appreciatedParticipants to feel appreciated

– Staff in clinic spend enough timeStaff in clinic spend enough time

– Sensitive to ppts. scheduling needsSensitive to ppts. scheduling needs

• Parties/events with all participantsParties/events with all participants

• Ease of logistics/transportation to clinicsEase of logistics/transportation to clinics

• Birthday cardsBirthday cards

• GiftsGifts

• Information, Newsletters, otherInformation, Newsletters, other

Strategies to enhance compliance IIStrategies to enhance compliance II

• Most drop outs occur in early study periodMost drop outs occur in early study period– FIT (4 years total); 2/3 of drop outs occurred in first year, most of FIT (4 years total); 2/3 of drop outs occurred in first year, most of

those in first 6 monthsthose in first 6 months

• Make certain that ppt’s understand study requirementsMake certain that ppt’s understand study requirements

• Run-in periodRun-in period– Trial run of drug/treatmentTrial run of drug/treatment

– Typically 2-4 weeks, usually of placebo (not always)Typically 2-4 weeks, usually of placebo (not always)

– Value controversialValue controversial

Study adherence: follow-up visitsStudy adherence: follow-up visits

• Goal: visits all on time (within window)Goal: visits all on time (within window)

• Set appointments flexiblySet appointments flexibly

• Reminders prior to appt.Reminders prior to appt.

• Give study calendarGive study calendar

• Listen to concerns/problemsListen to concerns/problems

Example of Compliance Monitoring in a Example of Compliance Monitoring in a Randomized TrialRandomized Trial

• Horizon studyHorizon study

• IV dosing of zoledronic acidIV dosing of zoledronic acid

• 1 dose per year1 dose per year

• Endpoint: hip and vertebral fracturesEndpoint: hip and vertebral fractures

• 7400 women in 23 countries, 269 sites7400 women in 23 countries, 269 sites

• Adherence to visit schedule monitored via IVRSAdherence to visit schedule monitored via IVRS– (telephone “punch 1, punch 2”) system(telephone “punch 1, punch 2”) system

• Reports on webReports on web

Can analysis be restricted to those who adhere? Can analysis be restricted to those who adhere? Coronary Drug Project (CDP, NEJM 1980)Coronary Drug Project (CDP, NEJM 1980)

5 year mortality5 year mortality

OverallOverall AdherenceAdherence

> 80%> 80% (2/3) (2/3) < 80% (1/3)< 80% (1/3)

Clofibrate (n=1065)Clofibrate (n=1065) 18%18% 15%15% 25%25%

Can analysis be restricted to those who adhere? Can analysis be restricted to those who adhere? Coronary Drug Project (CDP, NEJM 1980)Coronary Drug Project (CDP, NEJM 1980)

5 year mortality5 year mortality

OverallOverall AdherenceAdherence

> 80%> 80% (2/3) (2/3) < 80% (1/3)< 80% (1/3)

Clofibrate (n=1065)Clofibrate (n=1065) 18%18% 15%15% 25%25%

Placebo (n=2695)Placebo (n=2695) 19%19% 15%15% 28%28%

LessonsLessons

• Unknown/unmeasured confounders associated with Unknown/unmeasured confounders associated with compliancecompliance

• Differ in placebo and active groupsDiffer in placebo and active groups

• Be wary of post-randomization groupingsBe wary of post-randomization groupings

Adherence to medication is not the same as Adherence to medication is not the same as adherence to visit scheduleadherence to visit schedule

• ““Drop out” is very vague termDrop out” is very vague term

• Can have perfect visit adherence (come to all visits on Can have perfect visit adherence (come to all visits on time) but--time) but--

– Not take a single study med pillNot take a single study med pill

– Take only 60% of pillsTake only 60% of pills

• If miss visits or stop coming to visits, then generally If miss visits or stop coming to visits, then generally don’t take study medicationdon’t take study medication

– Exceptions do occur: Trial of once-yearly infusion treatment. Exceptions do occur: Trial of once-yearly infusion treatment. May have perfect medication compliance but poor visit May have perfect medication compliance but poor visit compliancecompliance

Follow-up visits for those who have stopped Follow-up visits for those who have stopped study medications?study medications?

• Practice varies dramatically across studiesPractice varies dramatically across studies

• Option 1: Stop follow-up as soon as drug stopsOption 1: Stop follow-up as soon as drug stops

• Option 2: Continue to collect follow-up infoOption 2: Continue to collect follow-up info

• Advantages of eachAdvantages of each– ????

Follow-up visits for those who have stopped Follow-up visits for those who have stopped study medications?study medications?

• Practice varies dramatically across studiesPractice varies dramatically across studies

• Option 1: Stop follow-up as soon as drug stopsOption 1: Stop follow-up as soon as drug stops

• Option 2: Continue to collect follow-up infoOption 2: Continue to collect follow-up info

• Advantages of eachAdvantages of each– O-2: Biased per previous slides (generally conservative)O-2: Biased per previous slides (generally conservative)

– O-1: Biased, but cannot predict directionO-1: Biased, but cannot predict direction

– Choice related to analysis (ITT)Choice related to analysis (ITT)

Intention to Treat Analysis (ITT)Intention to Treat Analysis (ITT)

• ITT coined by AB Hill in textbook on Stat (1961) ITT coined by AB Hill in textbook on Stat (1961)

• One of the main Commandments of RCT bibleOne of the main Commandments of RCT bible

• Original definition “Original definition “All All subjects will be analyzed subjects will be analyzed according to the treatment group according to the treatment group they were originally they were originally intendedintended by the randomization process” by the randomization process”

• ““generally interpreted as including all patients, generally interpreted as including all patients, regardless of whether they satisfied entry criteria, regardless of whether they satisfied entry criteria, treatment actually received or withdrawal or deviation treatment actually received or withdrawal or deviation from protocol”-- Hollis BMJ article from protocol”-- Hollis BMJ article

Beware of “we did an ITT analysis”Beware of “we did an ITT analysis”

• Generally considered sacred, almost god-like virtueGenerally considered sacred, almost god-like virtue

• The term “ITT” used differently in different studiesThe term “ITT” used differently in different studies

• ITT does NOT always mean that people were followed ITT does NOT always mean that people were followed beyond stopping study medicationsbeyond stopping study medications

Two Purposes of ITTTwo Purposes of ITT

• 1. Maintain validity of original randomization1. Maintain validity of original randomization– Groups can only differ by chanceGroups can only differ by chance

– Exclusion of some subjects post-randomization (e.g. didn’t take Exclusion of some subjects post-randomization (e.g. didn’t take any pills) could create biasany pills) could create bias

– Medical/surgery example in Hollis (table 1)Medical/surgery example in Hollis (table 1)

• 2. Makes clinical trial more like real-world situation2. Makes clinical trial more like real-world situation– Clinicians in real world have been known to deviate from Clinicians in real world have been known to deviate from

protocolsprotocols

–

True Meaning of ITT is AmbiguousTrue Meaning of ITT is Ambiguous

• Most rigorous interpretation includes the following Most rigorous interpretation includes the following principles (Hollis, discussion):principles (Hollis, discussion):

– All patients included in analysis even if they All patients included in analysis even if they didn’t start didn’t start interventionintervention

– Patients who were randomized but Patients who were randomized but did not meet inclusiondid not meet inclusion criterion but were randomized should be included in ITTcriterion but were randomized should be included in ITT

– All patients included regardless of complianceAll patients included regardless of compliance

– Ascertainment for primary outcome on all randomizedAscertainment for primary outcome on all randomized

How are ITT Principles Applied in Real Trials?How are ITT Principles Applied in Real Trials?

• Hollis (1999) evaluated application of ITT in 249 trials in Hollis (1999) evaluated application of ITT in 249 trials in BMJ, JAMA, Lancet, NEJMBMJ, JAMA, Lancet, NEJM

– About 50% (119 trials) reported using ITTAbout 50% (119 trials) reported using ITT

• Found that most trials with (ITT claimed) violated one or Found that most trials with (ITT claimed) violated one or more of the 4 ITT principlesmore of the 4 ITT principles

Alternatives in AnalysisAlternatives in Analysis

• per protocolper protocol or or as treatedas treated analysis analysis

• If all ppts. are followed regardless of adherence to If all ppts. are followed regardless of adherence to medications, several types of optionsmedications, several types of options

• Include only those patients who took all study Include only those patients who took all study medications and completed all protocol visitsmedications and completed all protocol visits

• Include all patients but only for the time that they Include all patients but only for the time that they remained on study medicationsremained on study medications

• If obtain complete follow-up on all ppts., can run several If obtain complete follow-up on all ppts., can run several different types of analyses and any discrepancies could different types of analyses and any discrepancies could be informative.be informative.

Analysis based on post-randomization Analysis based on post-randomization variablesvariables

• Per-protocol limits analysis to adherersPer-protocol limits analysis to adherers

• Per-protocol is one example of analysis which stratifies Per-protocol is one example of analysis which stratifies based on post-randomization experiencebased on post-randomization experience

– Other examples?Other examples?

• More generally, subgroup analyses by post-rand. factors More generally, subgroup analyses by post-rand. factors are biased, sometimes extremely biased--BEWAREare biased, sometimes extremely biased--BEWARE

Completion of follow-up for all participants Completion of follow-up for all participants regardless of adherence?regardless of adherence?

• ITT is ambiguous: at best--ITT is ambiguous: at best--– All included, according to original randomizationAll included, according to original randomization

– Follow-up completed on all ppts.Follow-up completed on all ppts.

– Events are included even when they occurred after medications Events are included even when they occurred after medications stoppedstopped

• Most rigorous approachMost rigorous approach

• Generally conservative estimate of treatment effectGenerally conservative estimate of treatment effect

• Per protocolPer protocol or or as-treatedas-treated analyses are possible as analyses are possible as secondary analysessecondary analyses

Problems with ITT/full follow-up approachProblems with ITT/full follow-up approach

• ITT/full follow-up not holy grailITT/full follow-up not holy grail

• Does not estimate full biologic efficacy of Does not estimate full biologic efficacy of drug/interventiondrug/intervention

– Advising individual patients may depend on efficacyAdvising individual patients may depend on efficacy

– Utility underestimatedUtility underestimated

• May be anti-conservative for adverse effectsMay be anti-conservative for adverse effects– per-protocol may be preferredper-protocol may be preferred

Alternatives in AnalysisAlternatives in Analysis

• per protocolper protocol or or as treatedas treated analysis analysis

• If all ppts. are followed regardless of adherence to If all ppts. are followed regardless of adherence to medications, several types of optionsmedications, several types of options

• Include only those patients who took all study Include only those patients who took all study medications and completed all protocol visitsmedications and completed all protocol visits

• Include all patients but only for the time that they Include all patients but only for the time that they remained on study medicationsremained on study medications

• If obtain complete follow-up on all ppts., can run several If obtain complete follow-up on all ppts., can run several different types of analyses and any discrepancies could different types of analyses and any discrepancies could be informative.be informative.

Summary of ITT:Summary of ITT:Those BritsThose Brits

• They don’t know how to cook nor run a monarchy but….They don’t know how to cook nor run a monarchy but….

Summary of ITT:Summary of ITT:Those BritsThose Brits

• They don’t know how to cook nor run a monarchy but….They don’t know how to cook nor run a monarchy but….

• They sure to talk goodThey sure to talk good

ITT: Key Messages (Hollis)ITT: Key Messages (Hollis)

• ITT gives a pragmatic estimate of benefit of treatment ITT gives a pragmatic estimate of benefit of treatment policy rather benefit in patients who receive treatment policy rather benefit in patients who receive treatment exactly as plannedexactly as planned

• Full application of ITT possible only when complete Full application of ITT possible only when complete outcome available on all randomizedoutcome available on all randomized

• Many trials that claim ITT varied in handling of missing Many trials that claim ITT varied in handling of missing data, deviations from protocol, etc.data, deviations from protocol, etc.

• ITT often inadequately described and appliedITT often inadequately described and applied

Recommendations for ITT (Hollis)Recommendations for ITT (Hollis)

• ITT best regarded as complete trial strategy (design, ITT best regarded as complete trial strategy (design, conduct, analysis) and not simply analysisconduct, analysis) and not simply analysis

• DesignDesign

– Justify in advance any inclusion which if violated Justify in advance any inclusion which if violated merit exclusion from ITTmerit exclusion from ITT

• ConductConduct

– Minimise (sic) missing response on primary outcomeMinimise (sic) missing response on primary outcome

– Follow up subjects who withdraw from treatmentFollow up subjects who withdraw from treatment

• AnalysisAnalysis

– Investigate potential effects of missing responseInvestigate potential effects of missing response

Recommendations for ITTRecommendations for ITT

• ReportingReporting

– Specify how ITT used (explicitly describe handling of Specify how ITT used (explicitly describe handling of deviations from randomization and missing deviations from randomization and missing responses)responses)

– Report deviations and missingnessReport deviations and missingness

– Discuss potential effects of missing reponseDiscuss potential effects of missing reponse

– Base conclusions on ITT analysesBase conclusions on ITT analyses

Follow-up and Analysis: summaryFollow-up and Analysis: summary

• Best trial:Best trial:

– All participants remain on medicationAll participants remain on medication

– All participants are followed until end of studyAll participants are followed until end of study

– Pre-planned analysis and handling of deviations from Pre-planned analysis and handling of deviations from protocolprotocol

• Where possible, minimize subjectivity and adhoc-nessWhere possible, minimize subjectivity and adhoc-ness