Kathleen Carroll & Brian KilukDivision of Substance Abuse
Yale University School of Medicine
Supported by NIDA Supplement to R01 DA15969
and P50 DA09241, U10 DA015831, R01 DA019078, & R01 DA 10679
Why do we need a sound and valid indicator? Facilitation of comparisons across
projects, meta-analyses Set and monitor performance
standards Benchmarking Clearly convey magnitude of treatment effects to
stakeholders Facilitate comparisons across common standard Lack of incentive to improve performance and
outcome (retention not appropriate standard)
Overview
Desirable characteristics of indicators
Strengths and weaknesses of common approaches
Overview of our project
“Traditional” indicators of clinical significance almost always translate to complete abstinence Return to normative levels Reliable change indices Return to healthy functioning? (e.g.,equivalent
of ‘no heavy drinking days’ for stimulant users)
What are we looking for in an indicator?
Easy to calculate, interpret Psychometrically sound, reliable, replicable Low susceptibility to missing data Verifiable (biologic indicator, other) Independence from baseline measures Sensitive to treatment effects Low(er) cost Predicts long-term cocaine outcomes Related to indicators of good long term functioning Acceptable to field Easily interpreted by clinicians, policy makers, payors
What is ‘success’ in treating stimulant users? Durable periods of abstinence
Employment, productivity Lack of criminal activity Reduced use of expensive, avoidable health care
resources
11% at end of treatment, 21% at end of 1 year follow up
Why not complete abstinence? Insensitive to change Difficult standard for most individuals
(14% of our sample of 434) Chronically relapsing disorder Change is dynamic Starting and remaining abstinent may imply
questionable need for treatment Our data: Weak relationship with cocaine
use and functioning outcomes at one year
Retention
ProsEasy to calculateAvailable for all participantsIndicator of treatment acceptabilityIndicator of differential attrition/data availability across conditions
ConsMay be more meaningful in some contexts than othersParticipants leave treatment for different reasonsIs retention with continued use meaningful?Is compliance with ineffective treatment meaningful?Not related to long-term outcome in our sample
Percent negative urines ProsWidely used and acceptedLess susceptible to demand characteristics, misrepresentationQuantifiable, ability to detect new episodesVery accurate, if appropriate schedule of collection and minimal missing dataTiming is critical (overlap, missing data)
ConsRecent use only (3-5)High cost for frequent or quantitativeSensitive to missing data, esp. with differential attritionDepends on assumptions (missing, denominator)Stimulants or all drugs?Can’t back-fillProblems with assuming missing=positive*
Calculating percent urine samplesExample: 1 negative urine, 2 sessions, then dropout of 12 week trial.
Based on submitted: 100%
Based on possible: 50%
Based on expected/ 1x 8%
Based on expected/ 3x 3%
Percent cocaine positive 0%
Longest consecutive x-free urine specimens ProsStrong evidence of meaningful abstinenceLess susceptible to demand characteristics, misrepresentationQuantifiable, ability to detect new episodesVery accurate, if appropriate schedule of collection and minimal missing dataTiming is critical (overlap, missing data)
ConsHigh cost for frequent or quantitativeVery sensitive to missing data, esp. with differential attritionDepends on assumptions (missing, denominator)Stimulants or all drugs?Can’t back-fill
Percent days abstinent, self reportProsWidely usedPotentially available for all participants and all days if TLFB used with high data completion; highly flexibleTrue intention to treat possibleCan be reliable if methods to enhance reliability used (at a cost)Our discrepancy rate=8-12%
ConsWith high/differential dropout, what’s the denominator? Days in treatment versus days expected?Not easy to correct with urine data if discrepancies high
Maximum days of abstinence,overall or in final x weeksProsLinked linked to longer-term cocaine usePotentially verifiable if urines collected at appropriate intervalsProvides ‘grace period’Easily dichotomized (eg 3 plus weeks)
ConsHigh complexity with missing data, especially dropoutsHigh complexity if discrepant urine data Participants last 2 weeks or last 2 weeks of trial?End of treatment or sometime within treatment?
Reduction in use: Frequency and or quantityProsAlternative to abstinence; more achievable target?Highly compatible with random regression modelsSensitive to treatments that may take time for effects to emergeProvides ‘grace period’Easily dichotomized
ConsComplexity obtaining accurate estimates of frequency/quantity of use prior to baselineWhen is reduction measured (last weeks? Entire course?Costs of repeated quantitative urines, sensitivity to missing data
Issues in defining ‘reduction’ Patterns vary widely (binge versus low
use) Reliable estimation of quantity complex
(illicit, no standard units, adulterants common, potency varies, hard to standardize ‘hits’ ‘joints’ ‘dime bags’)
Difficulty of estimating dollar value (commerce, shared use, sex for drugs)
Which indicator of treatment response?Loss of power with dichotomous, but also easily interpretable, calculable for all, relevant to clinical significanceCandidates
*Complete abstinence*3 or more weeks of abstinence*End of treatment abstinence*Reduction of x percent“Good functioning”
IndicatorEase of
computationVerifiability Vulnerability to missing
dataRelative cost Operationalization
for these analyses
1Days retained in treatment protocol C
Easy Yes- Low Low Days from randomization to endpt
2
Percentage of urine specimens testing positive C
Easy for complete data Yes, by definition Assumes independence of urine specimens
(denominator), assumes numerator is unbiased by
collection schedule or missing data.
High Number of cocaine-negative urine
specimens collected / all specimens collected
3
Maximum consecutive days abstinent
C Easy for complete data Yes, provided appropriate schedule of
data/urine collection
Likely to result in casewise missingness or reduced
sample size
Moderate, due to biological
verification and derivation from
TLFB
Longest continuous cluster of self-reported
abstinence within treatment
4
Percent days of abstinence from cocaine C
Depends on treatment duration, level of missing data, and
intermittent missingness
Yes, provided appropriate schedule of
data/urine collection
Likely to result in casewise missingness or reduced
sample size
Moderate, due to biological
verification and derivation from
TLFB
Number of self-reported days of abstinence from
cocaine / days in treatment (retention)
5
Maximum days of continuous abstinence during last two weeks of treatment C
Complex for intermittent and monotone,
dropouts
Yes, provided appropriate schedule of
data/urine collection
Low Moderate, due to biological
verification and derivation from
TLFB
For those retained 14+ days, longest cluster of
abstinence in final 2 weeks; otherwise 0
6
Completely abstinent last two weeks of treatment D
Easy Yes, provided appropriate schedule of
data/urine collection
Low Moderate, due to biological
verification and derivation from
TLFB
For those retained 14+ days, 0 days of use in last 14 days, otherwise
0
73 or more weeks of continuous abstinence D
Easy Yes, provided appropriate schedule of
data/urine collection
Low Moderate, due to biological
verification and derivation from
TLFB
“Yes” if participant retained 21+ days, max
days abstinent > 20. Otherwise No
82 or more weeks of continuous abstinence D
Easy Yes, provided appropriate schedule of
data/urine collection
Low Moderate, due to biological
verification and derivation from
TLFB
“Yes” if participant retained 14+ days, max
days abstinent > 13. Otherwise No
Note. C=continuous, D=Dichotomous, TLFB=Timeline Followback method
IndicatorEase of computation Verifiability Vulnerability to
missing dataRelative cost Operationalization for
these analyses
91 or more weeks of continuous abstinence D
Easy Yes, provided appropriate schedule of
data/urine collection
Low Moderate, due to biological verification and derivation from
TLFB
“Yes” if participant retained 7+ days, max days
abstinent > 6. Otherwise No
10
Completely abstinent from cocaine during treatment
D Easy Same Low Moderate, due to biological verification and derivation from
TLFB
0 days of use and 0 positive urines
11
Completed treatment and abstinent in last week D
Easy Yes Low Low Completion of treatment, 0 days of use in final week
12
Percent reduction in frequency of use (28 days prior/days last 4 weeks) C
Complex, baseline definition can be
arbitrary
No, relies on accurate baseline/pretreatment
assessment
Moderate Low Percent days of use in final 28 days of treatment/
percent days of use in 28 days prior to baseline
1350% reduction in frequency of use D
Complex, baseline definition can be
arbitrary
Relies on access to accurate
baseline/pretreatment level of use
Moderate Low % reduction is 50% or higher
1475% reduction in frequency of use D
Complex, baseline definition can be
arbitrary
Same Moderate Low % reduction is 75% or higher
15
Report no drug use, legal, employment, or psychological problems last 28 days of treatment D
Easy Partial Low Low Completes treatment, 0 days of problems in drug,
legal, employment and psych ASI in past 28 days
Note. C=continuous, D=Dichotomous, TLFB=Timeline Followback method
IndicatorEase of
computationBiological
verificationVulnerability to missing
data
Sensitivity to
treatment effects
Relationship with post
tx outcomes
Independent from baseline
indicators
Relationship to measures of
general function
21-30 days of abstinence
X Relies on appropriate
schedule
Low
Completed treatment and abstinent in last
2 weeks
X Same Low
50 % reduction Complex, baseline
definition can be arbitrary
Relies on having accurate baseline/pretrea
tment assessment of
use
Moderate
% days abstinent
Depends on treatment duration,
complex for dropouts, and intermittent missingness
X, provided appropriate schedule of data/urine collection
Moderate
Max days consecutive abstinence
Complex for intermittent and
monotone missingness,
dropouts
X, provided appropriate schedule of data/urine collection
Likely to result in casewise
missingness or reduced sample
size
Percent neg ative urine specimens
Easy except when missing
data
yes
So far…
Carroll, K.M., Kiluk, B.D., et al. (2014). Towards empirical identification of a reliable and clinically meaningful indicator of treatment outcome for illicit drug use. Drug and Alcohol Dependence, 137, 3-19.
Kiluk, B.D., et al. (2014). What happens in treatment doesn’t stay in treatment: Cocaine abstinence during treatment is associated with fewer problems at follow-up. J Consulting and Clinical Psychology, 82:619-27.
DeVito, E.E., et al. (2014). Gender differences in clinical outcomes for cocaine dependence: Randomized clinical trials of behavioral therapy and disulfiram. Drug and Alcohol Dependence, 145: 156-167.
Decker, S.E., et al. (2014). Assessment concordance and predictive validity of self-report and biological assay of cocaine use in treatment trials. The American Journal on the Addictions, 23, 466-74.
Kiluk, B.D., et al. (in press). Prompted to treatment by the criminal justice system: Relationships with treatment retention and outcome among cocaine users.