good research practices task force - ispor
TRANSCRIPT
Statistical Analysis of
Discrete-Choice Experiments:
Discussion of the Report
ISPOR Conjoint Analysis
Good Research Practices
Task Force
Moderator
2
Maarten J. IJzerman, PhD Professor of Clinical Epidemiology & Health Technology Assessment,
Dean of Health & Biomedical Technology, School of Science & Technology
University of Twente, Enschede, the Netherlands
Chair: A. Brett Hauber, PhD, Senior Economist & Vice President, Health Preference
Assessment, RTI Health Solutions, Research Triangle Park, NC USA
Juan Marcos Gonzalez, PhD, Senior Research Economist, Health Preference
Assessment, RTI Health Solutions, Research Triangle Park, NC, USA
Karin G.M. Groothuis-Oudshoorn, PhD, Assistant Professor, Health Technology and
Services Research, University of Twente, Enschede, Netherlands
Thomas Prior, PhD Candidate, Department of Biostatistics,
Johns Hopkins Bloomberg School of Public Health, Baltimore, MD, USA
Deborah A. Marshall, PhD, MHSA, Canada Research Chair, Health Services and
Systems Research; University of Calgary, Calgary, AB, Canada
Charles Cunningham, PhD, Professor, Department of Psychiatry and Behavioural
Neuroscience, McMaster University, Hamilton, Ontario, Canada
Maarten J. IJzerman, PhD, Professor of Clinical Epidemiology & Health Technology
Assessment (HTA) and Dean, University of Twente, Enschede, the Netherlands
John F. P. Bridges, PhD Associate Professor, Johns Hopkins Bloomberg School of
Public Health, Baltimore, MD, USA
Task Force Members
3
Task Force Background
This is the third ISPOR Conjoint Analysis Task Force Report.
Conjoint Analysis Use in Health Studies - a Checklist: A Report of the ISPOR Conjoint Analysis in Health Good Research Practices Task Force (the Checklist) (Bridges et al., 2011).
Downloads as of 9 November 2016: 7,311
Constructing Experimental Designs for Discrete-Choice Experiments: Report of the ISPOR Conjoint Analysis Experimental Design Task Force (Johnson et al., 2013).
Downloads as of 9 November 2016: 7,477
4
Task Force Background
Increasing number of researchers conducting conjoint-analysis studies
Diverse backgrounds, potential lack of basic training in the theoretical underpinnings of conjoint analysis and the statistical approaches to analyze DCE data
Published studies based on statistical analyses not appropriate for the data generated by the conjoint questions
Software used for analyses without understanding properties of statistical models – therefore, unable to evaluate strengths and limitations of the statistical analyses
5
Overview of the forum
Present a primer on relevant statistical methods
including multinomial/conditional logit, random-
parameters logit, hierarchical Bayes, and latent-class
analysis;
Have a discussion of good research practices for
statistical analysis (including the ESTIMATE
checklist)
Discuss case studies on how these methods can be
applied to discrete-choice experiments and other
stated-preference methods.
6
Speaker
Juan Marcos Gonzalez, PhD Senior Research Economist, Health Preference Assessment,
RTI Health Solutions, Research Triangle Park, NC, USA
7
Role of Analysis
Analysis is to infer the strength of preference for each attribute
and attribute level.
Estimates are referred to as preference weights or part-worth
utilities.
Preference weights are estimated on a common scale, and allow
calculation of ratios representing trade-offs people are willing to
make. For instance:
– Monetary equivalence, i.e. willingness to pay
– Risk equivalence, i.e. maximum acceptable risk (MAR)
– Time equivalence
In contrast to use in marketing predicting choice, CA studies in
healthcare are mostly used to estimate preference for attributes.
8
Choices Made and Structure
The primary objective is to provide an educational
resource for readers from wide range of disciplines.
Task force authors decided to structure the paper with a
simple example to explain the concepts for analyzing
choice data.
An archetypal case is introduced with the intention to
demonstrate different analytic approaches and to contrast
their main differences.
9
Assumptions and Remarks
Assumptions and choices made sometimes oversimplify the challenges of statistical modeling.
For instance, “simple regression analysis” is introduced for educational purposes but not recommended in practice.
The report concludes with an overview of some of the current discussions on the topic.
Readers are referred to the advanced statistical methods for more details about statistical analysis.
10
Attributes Levels
A1 Efficacy L1 10 (best level)
L2 5 (middle level)
L3 3 (worst level)
A2 Side effect L1 Mild
L2 Moderate
L3 Severe
A3 Mode of
administration
L1 1 tablet once a day
L2 Subcutaneous injection once a week
L3 Intravenous infusion once a month
Attributes and Levels in the
Archetypal Case
11
Example of a Choice Task
for the Archetypal Case
Feature Medicine A Medicine B
Efficacy
10 on a scale from 1
to 10
where 10 it the best
5 on a scale from 1 to
10
where 10 is the best
Severity of
side effects Severe Mild
How you
take the
medicine
Subcutaneous
injection
once a week
Intravenous infusion
once a month
Which
medicine
would you
choose?
12
Dummy Versus Effects Coding
Archetypal Case
Attribute level
presented in the
profile
Dummy-variables Effects-coded variables
L1 L2 L1 L2
L1 1 0 1 0
L2 0 1 0 1
L3 0 0 -1 -1
13
Dummy Versus Effects Coding
0
1
2
3
4
5
6
7
8
9
Level1
Level2
Level3
Level1
Level2
Level3
Level1
Level2
Level3
Attribute 1 Attribute 2 Attribute 3
Pre
fere
nce W
eig
hts
Attributes and Levels
Dummy coded
Mean effects
-5
-4
-3
-2
-1
0
1
2
3
4
Level1
Level2
Level3
Level1
Level2
Level3
Level1
Level2
Level3
Attribute 1 Attribute 2 Attribute 3
Pre
fere
nce W
eig
hts
Attributes and Levels
Effects coded
• Estimate coefficients relative to mean
attribute effect
• Omitted category is the negative sum of
the coefficients of the non-omitted levels
• Tests of significance in output typically are
not direct tests on differences between
estimated coefficients
• Coefficients represent a measure of preference for
levels of an attribute relative to the omitted level of
that attribute
• Test of significance for a coefficient reflects that this
level is significantly different from the reference
level
• If aim is to obtain odds ratios then dummy coding is
preferred. In case of effects coding, exponentiation
of the estimated coefficients yields the ratio of the
odds for the particular attribute level to the
geometric mean of the odds. 14
Common Models Resulting from
Different Choice Probabilities
Linear probability
Conditional logit
Random-parameters logit
Hierarchical Bayes
Latent-class finite-mixture logit
15
Linear Probability Model
Conventional regression model (Ordinary Least Squares
[OLS]) linking choice to differences in each attribute
Assumes the chance that something is selected is
linearly determined by the characteristics of the
alternatives considered
Pr(𝑐ℎ𝑜𝑖𝑐𝑒) = 𝛽𝑜 + 𝛽𝑖𝑋𝑖𝑖
16
Advantages & Limitations
Linear Probability Model
Advantages Limitations
• Can be used in small samples
• Can be used to identify non-trading
at the individual level
• Is available in most statistical
software packages and even in
some non-statistical software
• Does not appropriately account for
repeated observations from
respondents
• Can produce negative or greater
than one predictions of probabilities
• No more than 2 alternatives in a
single choice occasion is possible
17
Conditional Logit Model
Regression model that acknowledges the discrete nature of
choice as dependent variable
Based on random utility theory (Mc Fadden, 1974)
Ui = V(Xi β) + i
i independently, identically distributed type 1 extreme
value distribution.
Pr 𝑐ℎ𝑜𝑖𝑐𝑒 = 𝑖 =𝑒𝑉 𝛽,𝑥𝑖
𝑒𝑉 𝛽,𝑥𝑗𝑗
18
Results Conditional Logit
Attribute Level Coefficient
Std. Error t-value p-value
Efficacy L1 0.26 0.01 24.89 <0.01
L2 0.02 0.01 2.24 0.03
Side Effect L1 0.32 0.01 30.18 <0.01
L2 0.02 0.01 2.02 0.04
Mode of
Administration
L1 0.03 0.01 2.49 0.01
L2 0.18 0.01 17.61 <0.01
Log Likelihood -17388
Log likelihood of
model without
predictors
-18715
AIC 34788
BIC 34841 19
Advantages & Limitations
Conditional Logit Model
Advantages Limitations
• Focuses on average preferences
• Parsimonious estimator with unique
solution
• Commonly available in software
packages
• Requires the smallest sample size
from all the models in this table
• Assumes homogeneity in
preferences
• Does not accounting for panel
nature of the data
• It is not guaranteed to converge
without large enough samples or
with lack of variability in the
response variable
20
Effects-Coded Results
21
Dummy-Coded Results
22
Random-Parameters Logit
Expands conditional logit model to account for across-
respondents variations in preferences
When is assumed normally distributed with mean and
standard deviation
, for across n respondents in the sample
23
Mean Estimates
Attribute Level Coefficient Std. Error T-value P-value
Efficacy L1 0.31 0.01 20.92 <0.01
L2 0.03 0.01 2.43 0.02
Side Effect L1 0.38 0.02 24.22 <0.01
L2 0.02 0.01 1.72 0.09
Mode of
Administration
L1 0.03 0.01 2.68 0.01
L2 0.22 0.02 12.13 <0.01
Standard Deviation Estimates
Attribute Level Coefficient Std. Error T-value P-value
Efficacy L1 0.31 0.02 19.98 <0.01
L2 0.14 0.02 6.48 <0.01
Side Effect L1 0.34 0.02 21.37 <0.01
L2 0.24 0.02 14.69 <0.01
Mode of
Administration
L1 -0.05 0.03 -1.66 0.10
L2 0.48 0.02 27.72 <0.01
Log Likelihood of
model -16672.7
Log likelihood of
model without
predictors
-17387.6
AIC 33369.4
BIC 33476.2 24
Advantages & Limitations
Random-Parameters Logit Model
Advantages Limitations
• Models heterogeneity
• Accounts for the panel nature of the
data
• Becoming more available in
software packages
• Can deal with scale heterogeneity
with the use of model options (allow
correlation of preference
heterogeneity across attributes)
• More difficult to use than conditional
logit
• Requires assumptions about the
distribution of parameters across
respondents
• Requires larger sample sizes than
conditional logit models
25
Hierachical Bayes Model
Generates preference estimates for each individual
Underlying model is multinomial logit
Sample density model: 𝛽𝑛~𝑁(𝑏,𝑊) (or other distributions
like lognormal, triangular, uniform)
Likelihood level / lower level/ individual choices:
Pr 𝑐ℎ𝑜𝑖𝑐𝑒 = 𝑖 =𝑒𝑉 𝛽𝑛,𝑥𝑖
𝑒𝑉 𝛽𝑛,𝑥𝑗
𝑗
Gibbs sampler: iteratively estimates b, W, 𝛽𝑛
26
Mean Standard Deviation
Attribute Level RPL HB RPL HB
Efficacy L1 0.31 0.35 0.31 0.43
L2 0.03 0.03 0.14 0.33
Side Effects L1 0.38 0.43 0.34 0.50
L2 0.02 0.03 0.24 0.42
Mode of
administration
L1 0.03 0.03 0.05 0.36
L2 0.22 0.26 0.48 0.66
Comparison of Results of
Different Models
27
Advantages & Limitations
Hierarchical Bayes Model
Advantages Limitations
• Quicker convergence
• Models individual preferences
• Requires fewer respondents to
construct the sample mean
preferences
• Deals with scale heterogeneity
without requiring model options
(individual results allow correlation
of preferences across attributes)
• Not available in many software
packages
• May require more choices per
respondent to obtain individual
preference estimates
• Results may be difficult to explain
• Inferences cannot be made at the
respondent level
28
Latent-Class Finite-Mixture
Logit Model
Assumes that attributes of the alternatives have
heterogeneous effects on choices across a finite number
of groups or classes of respondents
Pr 𝑐ℎ𝑜𝑖𝑐𝑒 = 𝑖|𝑥𝑖 , 𝑥𝑗 = Pr 𝑐ℎ𝑜𝑖𝑐𝑒 = 𝑖|𝑥𝑖 , 𝑥𝑗; 𝛽𝑞𝑞
𝜋𝑞
𝜋𝑞 are the class probabilities
Within a class: Pr 𝑐ℎ𝑜𝑖𝑐𝑒 = 𝑖|𝑥𝑖 , 𝑥𝑗; 𝛽𝑞 =𝑒𝑉 𝑥𝑖;𝛽𝑞
𝑒𝑉 𝑥𝑗;𝛽𝑞
𝑗
.
29
Class 1
Attribute Level Coefficient Std. Error T-value P-value
Efficacy L1 0.29 0.02 11.70 <0.01
L2 0.04 0.02 2.01 0.04
Side Effect L1 0.39 0.03 13.28 <0.01
L2 -0.02 0.02 -1.06 0.29
Mode of
Administration
L1 0.21 0.02 9.38 <0.01
L2 -0.33 0.04 -9.25 <0.01
Class 2
Attribute Level Coefficient Std. Error T-value P-value
Efficacy L1 0.27 0.02 14.37 <0.01
L2 0.01 0.02 0.80 0.42
Mode of
Administration L1
0.30 0.02 14.20 <0.01
L2 0.05 0.02 3.09 0.00
Mode of
Administration L1
-0.09 0.02 -5.40 <0.01
L2 0.54 0.03 19.60 <0.01
Class probability
function
Constan
t -0.41 0.13 -3.05 <0.01
Log Likelihood of
model -16985
Log likelihood of
model without
predictors
-18714
AIC 33996
BIC 34102 30
Advantages & Limitations
Latent-Class Logit Model
PRO CON
• Models latent classes
• Describes heterogeneity by class
• Parsimonious estimator with a
unique solution
• Requires smaller samples than RPL
and HB models
• Requires specialized software
• Judgment required to determine
appropriate number of classes to be
estimated
• Difficult to interpret results from any
given class when the chance of
being in all classes is more or less
the same across respondents
• The required sample size varies
with the number of classes in the
model
31
Speaker
Deborah A. Marshall, PhD, MHSA Canada Research Chair, Health Services and Systems Research;
Professor, Department of Community Health Sciences,
University of Calgary, Calgary, AB, Canada
32
Principles for Good Research
Practice In Analysis
33
The ESTIMATE
Checklist
Estimates
Stochastic
Tradeoffs
Interpretation
Method
Assumptions
Transparent
Evaluation
Estimates
Describe the choice of parameter estimates
resulting from the model appropriately and
completely, including:
Whether each variable corresponds to an effects-
coded level, a dummy-coded level, or a
continuous change in levels
Whether each variable corresponds to a main
effect or interaction effect
Whether continuous variables are linear or have
an alternative functional form
34
Stochastic
Describe the stochastic properties of the analysis,
including:
The statistical distributions of the parameter
estimates
The distribution of parameter estimates across
the sample (preference heterogeneity)
The variance of the estimation function, including
systematic differences invariance across
observations (scale heterogeneity)
35
Tradeoffs
Describe the tradeoffs that can be inferred from the
model, including:
The magnitude and direction of the attribute-level
coefficients
The relative importance of each attribute over the
range of levels included in the experiment
The rate at which respondents are willing to trade
off among the attributes (marginal rate of
substitution)
36
Interpretation
Provide interpretation of the results taking into
account the properties of the statistical model,
including:
Conclusions that can be drawn directly from the
results
Applicability of the sample, including subgroups
or segments, to the population of interest
Limitations of the results
37
Method
Describe the reasons for selecting the statistical
analysis method used in the analysis, including:
Why the method is appropriate for analyzing the
data generated by the experiment
Why the method is appropriate for addressing the
underlying research question
Why the method was selected over alternative
methods
38
Assumptions
Describe the assumptions of the model and the
implications of the assumptions for interpreting the
results, including:
Assumptions about the error distribution
Assumptions about the independence of
observations
Assumptions about the functional form of the
value function
39
Transparent
Describe the study in a sufficiently transparent way
to warrant replication, including descriptions of:
The data setup, including handling missing data
The estimation function, including the value
function and the statistical analysis method
The software used for estimation
40
Evaluation
Provide an evaluation of the appropriateness of the
statistical analysis method to answering the research
question, including:
The goodness of fit of the model
Sensitivity analysis of the model specification
Consistency of results estimated using different
methods
41
Limitations
Focus is on traditional DCE, and there are
modifications (e.g. opt-out) and other choice formats
(e.g. BWS, threshold techniques) that are not
covered.
Emerging and specialized statistical methods are not
described in the report.
No formal evaluation of the extent to which each
method is being used.
No rating of the importance of different properties of
different models.
42
Speaker
John F. P. Bridges, PhD @jfpbridges
Associate Professor,
Johns Hopkins Bloomberg School of Public Health,
Baltimore, MD, USA
43
Case Studies on Research Practices
The work presented here is based on results of a randomized
control trial on preferences in patients with type 2 diabetes
• Posters on this study will be presented at poster session IV
» B27: Treatment preferences of patients with type 2 diabetes in the united states: an application of good research principles for discrete choice experiments
» B30: Developing a stated-preference instrument to assess the barriers and facilitators to the self-management of type 2 diabetes
• Funded by The Patient-Centered Outcomes Research Institute (PCORI) Methods Program Award (ME-1303-5946) and the the Johns Hopkins Center of Excellence in Regulatory Science and Innovation and the Food and Drug Administration (UO1FD004977).
44
Overview
1. Prioritization methods
– Comparing Likert and BWS data
2. Preference methods
– Benefits of Mixed Logit for DCE and BWS case 2
3. Using latent classes
– Determining the number of latent classes
4. Advances in presenting preference results
45
1. Prioritization Methods:
Likert v BWS case 1 (rho>0.9)
46
-0.400
-0.300
-0.200
-0.100
0.000
0.100
0.200
0.300
0.400
0.500
0.600
Sc
ore
LikertScale
BWS
1. Prioritization Methods:
Rescaling Likert Data
47
-3.15
-3.06
-2.86
-2.79
-2.16
-0.53
0.20
0.35
1.12
1.39
4.05
4.62
5.68
-4.00 -2.00 0.00 2.00 4.00 6.00
Capacity to manage
Other health conditions
Faith and religious practices
Places to exercise
Time commitments
Communication
Healthy food
Current health insurance
My family commitments
Resources in local community
My language and culture
Active support group
Personal understanding
-3.15
-3.06
-2.86
-2.79
-2.16
-0.53
0.20
0.35
1.12
1.39
4.05
4.62
5.68
-4.00 -2.00 0.00 2.00 4.00 6.00
Capacity to manage
Other health conditions
Faith and religious practices
Places to exercise
Time commitments
Communication
Healthy food
Current health insurance
My family commitments
Resources in local community
My language and culture
Active support group
Personal understanding
-3.15
-3.06
-2.86
-2.79
-2.16
-0.53
0.20
0.35
1.12
1.39
4.05
4.62
5.68
-4.00 -2.00 0.00 2.00 4.00 6.00
Capacity to manage
Other health conditions
Faith and religious practices
Places to exercise
Time commitments
Communication
Healthy food
Current health insurance
My family commitments
Resources in local community
My language and culture
Active support group
Personal understanding
Current health insurance
My family commitments
Resources in local community
My language and culture
Active support group
Personal understanding
My language and culture
Resources in local community
My family commitments
10.13
9.53
8.84
4.77
4.63
3.48
3.40
3.03
1.28
0.39
0.26
0.25
0.18
0.00 2.00 4.00 6.00 8.00 10.00 12.00
Regular access to healthy food
My personal understanding of diabetes
My capacity to manage my diabetes
My current health insurance
Other health conditions that I have
Access to convenient places to exercise
My family commitments
My faith and religious practicses
Access to an active support group
My language and culture
Stress about time commitments
Resources in my local community
Theoretical mean
5.68
4.81
4.62
1.44
1.27
0.35
0.19
-0.38
-1.70
-3.39
-2.69
-3.72
-2.96
-4.00 1.00 6.00
Observed mean
2. Preference Heterogeneity:
Mixed Logit Individual Parameters
48
2. Preference Heterogeneity:
Correlation for Mixlogit and Clogit
49
y = 0.7434x + 3E-18
y = 0.9712x - 1E-17
-1.500
-1.000
-0.500
0.000
0.500
1.000
1.500
-1.500 -1.000 -0.500 0.000 0.500 1.000 1.500
Clo
git
Mixlogit
DCE
BWS
3. Preference Heterogeneity:
Number of Latent Classes
2 classes 3 classes 4 classes 5 classes 6 classes 7 classes
Obs 17508 17508 17508 17508 17508 17508
ll(model) -4700.229 -4498.228 -4339.453 -4204.632 -4167.422 -4139.929
df 0 0 0 0 0 0
AIC 9400.458 8996.457 8678.906 8409.263 8334.845 8279.859
BIC 9400.458 8996.457 8678.906 8409.263 8334.845 8279.859
50
Model fit of different latent class models
3. Preference Heterogeneity:
Number of Latent Classes
51
Variable Class
1 Class
2 Class
3 Class
4 Class
5 Class
6
A1c decrease 0.77 0.24 0.47 0.87 3.52 0.28
Stable blood glucose 0.67 0.70 0.45 2.53 0.67 0.23
Low blood glucose 0.75 0.57 0.53 0.72 0.49 0.52
Nausea 1.03 1.39 3.55 0.76 1.54 0.31
Treatment burden 0.79 3.34 0.71 0.49 0.30 0.33
Cost 3.53 1.40 0.16 0.23 0.96 0.28
Relative attribute importance in 6 class latent class model
4. Presenting Preference Results
52
-1.5
-1.0
-0.5
0.0
0.5
1.0
1.5
1%
0.5
0%
0%
6 d
ays/
wee
k
4 d
ays/
wee
k
2 d
ays/
wee
k
No
ne
Day
Day
an
d/o
r n
igh
t
No
ne
30
min
ute
s
90
min
ute
s
1 p
ill
2 p
ills
1 p
ill a
nd
1 in
ject
ion
$1
0
$3
0
$5
0
A1cdecrease
Stable bloodglucose
Low bloodglucose
Nausea Treatmentburden
Out-of-pocketcost
Mea
n p
refe
ren
ce w
eigh
ts w
ith
95
% C
on
fid
ence
In
terv
al
Traditional preference chart
4. Presenting Preference Results
53
-1.0
-0.5
0.0
0.5
1.0
1%
0.5
0%
0%
6 d
ays/
wee
k
4 d
ays/
wee
k
2 d
ays/
wee
k
No
ne
Day
Day
an
d/o
r n
igh
t
No
ne
30
min
ute
s
90
min
ute
s
1 p
ill
2 p
ills
1 p
ill a
nd
1 in
ject
ion
$1
0
$3
0
$5
0
A1cdecrease
Stable bloodglucose
Low bloodglucose
Nausea Treatmentburden
Out-of-pocketcost
Pre
fere
nce
wei
ghts
Shifted bar graph with line graph
4. Presenting Preference Results
54
-2.2-1.8-1.4-1.0-0.6-0.20.20.61.01.41.82.2
1%
0.5
0%
0%
6 d
ays/
wee
k
4 d
ays/
wee
k
2 d
ays/
wee
k
No
ne
Day
Day
an
d/o
r n
igh
t
No
ne
30
min
ute
s
90
min
ute
s
1 p
ill
2 p
ills
1 p
ill a
nd
1 in
ject
ion
$1
0
$3
0
$5
0
A1cdecrease
Stable bloodglucose
Low bloodglucose
Nausea Treatmentburden
Out-of-pocketcost
Mea
n p
refe
ren
ce w
eigh
ts Box plot to indicate heterogeneity
Moderator
55
Maarten J. IJzerman, PhD Professor of Clinical Epidemiology & Health Technology Assessment,
Dean of Health & Biomedical Technology, School of Science & Technology
University of Twente, Enschede, the Netherlands
FORUM
1. Go to the ISPOR homepage:
www.ispor.org .
2. Click on the GREEN TASK FORCE menu
at the TOP of the homepage
3. Select JOIN on the pull-down menu.
Join the Conjoint Analysis Task Force
Review Group
56
Forum Slides
57
FORUM SLIDES are AVAILABLE!
http://www.ispor.org/Event/ReleasedPresentations/2016Washington
Questions?
58