Modelling Cardinal Utilities from Ordinal Utility data: An

exploratory analysis

Peter Gilks, Chris McCabe, John Brazier, Aki Tsuchiya, Josh Solomon

Background

• Limitations of conventional methods of utility elicitation

• Early work suggesting ordinal data can predict cardinal preferences

• SF6D and HUI 2 surveys used ranking exercises as warm up prior to SG valuation tasks

• Opportunity to test and develop methods proposed by Solomon

SF-6D valuation data sets

• Ranked seven SF-6d states (including pits and full health) and death

• SG valuations of five states against full health and pits and then chained using valuation of pits against full health and death (respondents asked to confirm pits ranking against death)

• 611 respondents sampled from the general population

• 249 mean SG health states values ranging from .21 to .99; averaged 14 valuations per state

HUI2 valuation data set

• Ranked 9 HUI2 states (including pits and full health) and death

• SG valuations of 8 states against full health and death (respondents asked to confirm ranking of state against death)

• 198 respondents sampled from the general population

• 249 mean SG health states values ranging from -.064 to .77; averaged 24 valuations per state

Methods

Aim: To model the predicted health state valuations using the ordinal preference data

1) Statistical model

Conditional logistic regression (McFadden choice model) based on random utility theory (previous attempts used Thurstone’s Comparative Judgement Model)

2) Value function

Relating the health state descriptive system to the utility value

The Statistical Model

• Respondent i has latent utility value for state j, Uij.

• Respondent will choose state j as best from a group of states k=1,…,n if Uij > Uik for all k j.

• Utility function Uij = μj + εij. Where μj represents the underlying tastes of the population and εij represents the peculiar choice of the individual.

• Odds of choosing state j over state k are exp{μj – μk}

• So we want to model the dependent variable μ against the dimensions of the descriptive systems: SF6D and HUI2.

Assumption: independence of irrelevant alternatives

• Model is based on assumption that the ranking exercise is equivalent to the respondent making a series of individual choices from smaller and smaller groups of states.

For example, to rank 10 health states;

• Selects first preference from all 10, rank 1

• Selects best from remaining 9, rank 2

• Selects best from remaining 8, rank 3 and so on……

NB. This assumes that the choice over a given pair does not depend on the other alternatives available

Value function The expected value of each unobserved utility was assumed to be a

linear function of the categorical ratings on the domains of each dataset respectively. The specifications are;

 For HUI2: μ = β1S2 + β2S3 + β3S4 + β4M2 + β5M3 + β6M4 + β7M5 + β8E2

+ β9E3 + β10E4 + β11E5 + β12C2 + β13C3 + β14C4 + β15SC2 + β16SC3 +

β17SC4 + β18P2 + β19P3 + β20P4 + β21P5 + βdDeath

 For SF6D: μ = β1PF2 + β2PF3 + β3PF4 + β4PF5 + β5PF6 + β6RL2 +β7RL3

+ β8RL4 + β9SF2 + β10SF3 + β11SF4 + β12SF5 + β13P2 + β14P3 + β15P4

+ β16P5 + β17P6 + β18MH2 + β19MH3 + β20MH4 + β21MH5 + β22V2 +

β23V3 + β24V4 + β25V5 +βdDeath

• Note: no constant term and a coefficient for death! This facilitates re-scaling results on to the Full-Health Death (1,0) Scale.

Rescaling The scale of the latent variable μ is arbitrarily defined by

the identifying assumptions in the model.

1) Normalise to observed SG scale (originally proposed by Josh Solomon)

Multiply coefficients by the ratio:

βri = βi * min. obs. SG/ Predicted PITS value

2) Normalise to death

βri = βi / |βd|

This anchors death at zero and perfect health at 1

NB. states can still be valued as worse than death.

Model Assessment Methods

Main aim is to compare the predictive performance of the rank model and the original standard gamble model:

• Check coefficients for sign and consistency.

• Plot predictions against observed for rank model and SG model for both datasets.

• Statistical tests of predictive performance.

• Look for systematic patterns in the errors.

HUI2 Rank Model and SG Model(OLS)

RankCoeff RescaledCoeff SGCoeffsens2 -0.9932932 -0.1156 -0.1151sens3 -0.9350973 -0.1089 -0.1223sens4 -2.116679 -0.2464 -0.2253mobil2 -0.7287155 -0.0848 -0.0516mobil3 -0.9887335 -0.1151 -0.1224mobil4 -0.8041412 -0.0936 -0.1308mobil5 -1.008526 -0.1174 -0.1103emot2 -0.8122273 -0.0946 -0.0945emot3 -1.0001 -0.1164 -0.1119emot4 -1.429127 -0.1664 -0.1801emot5 -1.43784 -0.1674 -0.1824cogn2 -0.3222758 -0.0375 -0.0567cogn3 -0.5438438 -0.0633 -0.0966cogn4 -0.773194 -0.0900 -0.1676sc2 -0.4409409 -0.0513 -0.0516sc3 -0.692351 -0.0806 -0.1138sc4 -0.7762394 -0.0904 -0.1158pain2 -0.8131845 -0.0947 -0.1114pain3 -0.940143 -0.1095 -0.1155pain4 -1.216913 -0.1417 -0.1626pain5 -1.76543 -0.2055 -0.2538death -8.589516 -1

n 51 51MAE 0.615 0.051No.>0.05 23 18No.>0.10 12 5RMSE 0.0775 0.0657LB 36.11 25.78Corr(means) 0.8814 0.921No. of Logical Inconsistencies 2 1

HUI2

Rank Model

ts1 51

-.4

-.2

0

.2

.4

.6

.8

1

SG Model

ts1 51

-.4

-.2

0

.2

.4

.6

.8

1

Smooth line = mean health state values ranked by severity

Top line is predictions

Bottom line is error.

Mean values, predicted values and error (predict - mean) for Rank model including death and SG Model (OLS) HUI2

SF6D Rank Model and SG model(Mean'6')

RankCoeff RescaledCoeff SGCoeffpf2 -0.363575 -0.0566 -0.0532pf3 -0.431302 -0.0671 -0.0106pf4 -0.9856325 -0.1534 -0.0402pf5 -0.6340183 -0.0987 -0.0535pf6 -1.447536 -0.2253 -0.1110rl2 -0.3210761 -0.0500 -0.0530rl3 -0.4069154 -0.0633 -0.0552rl4 -0.4052777 -0.0631 -0.0503sf2 -0.3626836 -0.0565 -0.0555sf3 -0.4203095 -0.0654 -0.0668sf4 -0.5737133 -0.0893 -0.0698sf5 -0.8054821 -0.1254 -0.0866pain2 -0.377161 -0.0587 -0.0467pain3 -0.3635335 -0.0566 -0.0250pain4 -0.6520135 -0.1015 -0.0561pain5 -0.8187383 -0.1275 -0.0912pain6 -1.191158 -0.1854 -0.1669mh2 -0.2157184 -0.0336 -0.0490mh3 -0.3371096 -0.0525 -0.0424mh4 -0.7015521 -0.1092 -0.1092mh5 -0.8992905 -0.1400 -0.1279vit2 -0.173969 -0.0271 -0.0861vit3 -0.2139943 -0.0333 -0.0606vit4 -0.3226131 -0.0502 -0.0543vit5 -0.5267463 -0.0820 -0.0907death -6.423983 -1.0000

n 249 249MAE 0.0882 0.0742No.>0.05 169 118No.>0.10 84 51RMSE 0.1096 0.0976LB 106.7200 169.5700Corr(means) 0.7111 0.7377No. of logical inconsistencies 3 8

SF6D

Rank Model

ts1 249

-.4

-.2

0

.2

.4

.6

.8

1

SG Model

ts1 249

-.4

-.2

0

.2

.4

.6

.8

1

Mean values, predicted values and error (predict - mean) for Rank model including death and SG Model (6) SF6D

Smooth line = means

Top messy line is predictions

Bottom messy lines is error.

Both Models:

• Under predict large means

• Over predict low means

Summary of Findings

• Rank models able to predict actual mean SG health states nearly as well as the SG models – associated with modest increase in in MAE

• Evidence that it has produced less systematic error in SF-6D data set and improvements in consistency

Issues – taking results at face value

• Is the ranked model good enough? Could we start using it………

• Given ranking is a warm up, results could be better if more care taken over this part of the exercise

• Ranked methods are probably cheaper• What evidence is there that ranking exercises

impose a lower cognitive burden? Seems to be higher levels of completion.

Issues – harder questions

• Is the selection process of the ranking task assumed by the model correct?

• Why should the relationship between the latent utility value and SG (in this case) cardinal values be linear?

– What other functional forms might theory suggest?

– Is the latent utility value similar to Dyer and Sarin’s ‘value function’ or something else?

• Does rank data elicit preferences or simply how good or bad a health state is, and does it matter?

Issues – the death question

• Not a major problem here because all mean health state values above zero

• The MVH EQ-5D data has been analysed in a similar way by Josh Solomon, but the ranking of death was very different to the implied ranking from the TTO – only state 33333 is ranked worse than death compared to 16/43 states by TTO!

Ranked model normalised to death and full health does not predict TTO values worse than death very well

Further work – more suggestions welcome

• See how well SG data predicts ranking at the individual level

• Consider interactions • Model different functional relationships between

latent variable and SG• examine completion rates and extent to which

ranking will extend the vote to more vulnerable populations