good: randomised and - jech.bmj.com · adult diabetes very high: adult bronchiolitis kneeoperations...

6Journal of Epidemiology and Community Health 1994;48:6-15

The Cochrane lecture

The best and the enemy of the good:randomised controlled trials, uncertainty, andassessing the role of patient choice in medicaldecision making

Klim McPherson

AbstractThis lecture aimed to create a bridge tospan the conceptual and ideological gapbetween randomised controlled trialsand systematic observational compari-sons and to reduce unwanted and unpro-ductive polarisation. The argument,simply put, is that since randomisationalone eliminates the selection effect oftherapeutic decision making, anythingshort of randomisation to attribute causeto consequent outcome is a waste of time.If observational comparison does haveany significant part in evaluating medi-cal outcomes, there is a grave danger of"the best", to paraphrase Voltaire,becoming "the enemy of the good". Thefirst section aims to emphasise the ad-vantages of randomised controlled trials.Then the nature of an essential precondi-tion - medical uncertainty - is discussedin terms of its extent and effect. Next, therole of patient choice in medical decisionmaking is considered, both when out-comes can safely be attributed to treat-ment choice and when they cannot. Theremay be many important situations inwhich choice itself affects outcome andthis could mean that random compari-sons give biased estimates of true thera-peutic effects. In the penultimate section,the implications of this possibility bothfor randomised controlled trials and foroutcome research is pursued and lastlythere are some simple recommendationsfor reliable outcome research.

Department of PublicHealth and Policy,London School ofHygiene and TropicalMedicine, KeppelStreet, LondonWC1E 7HTK McPherson

Correspondence to:Professor Klim McPherson.

Presented at the ScientificMeeting of the UK Societyfor Social Medcicine,Cambridge, UK inSeptember 1993.

(JT Epidemiol Community Health 1994;48:6-15)

Archie Cochrane's great contribution to our

discipline in 1972 was to describe a mostimportant problem in health care and to posit a

satisfactory solution. In his biography,' he saysof his work on Effectiveness and Efficiency2:

"I decided to concentrate on one simple idea- the value of the randomised controlled trialsin improving the NHS - and to keep the bookshort and simple."He did so in a manner which rings as loud and

clear today as it did then. Nowadays, both theproblem and solution are very familiar to usall. The problem was that much of the healthcare provided was unevaluated, and thereforepossibly of no benefit, and the solution -randomised controlled trials - were the onlysecure way of knowing the truth of the matter.

Cochrane's essential idea was that the reli-able assessment of effectiveness was the onlykey to scientific health care, but such methodsas he advocated had been disparaged by themedical establishment - paradoxically becausethey seemed to be less scientific than the moretraditional basic methods of scientific evalu-ation. A problem of methodological imperia-lism which is with us still, but in differentguise?He argued forcibly that only when the effec-

tiveness of a treatment or preventive measurehad been established and quantified, could theissues of efficiency and equity be discussedintelligently. But in the absence of reliableinformation about effectiveness, the pursuit ofoptimal health policies was bound to be per-manently elusive.

Since then Cochrane's arguments have beenlargely and impressively accepted by healthprofessionals.3 It is probable that without themwe would not have all the welcome recentdevelopments designed specifically to providea more effective health care system. I amreferring particularly to the NHS Researchand Development initiative and the systematicconcentration on issues of effectiveness andefficiency in the direction of research andhealth policy.

This is why there is a Cochrane lecture atthe Annual Scientific Meeting of the UKSociety of Social Medicine and I am honouredto be giving the lecture this year. I cannot dojustice to my predecessors, who have contrib-uted significantly and importantly to the workspearheaded by Archie Cochrane, but I hopeto make some methodological points on thequestion of evaluation of health care, many ofwhich I have made before. Largely, I want totry to divert any polarisation of methodologieswhich may, I believe, lead to an unnecessaryprolongation of medical uncertainties. Theissues are too important in my view.

6

on January 8, 2020 by guest. Protected by copyright.

http://jech.bmj.com

/J E

pidemiol C

omm

unity Health: first published as 10.1136/jech.48.1.6 on 1 F

ebruary 1994. Dow

nloaded from

http://jech.bmj.com/

The best and the enemy of the good

(1) The value of randomised controlledtrialsSince the publication of Effectiveness and Ef-ficiency2 there have been several attempts tominimise or falsify the importance and rele-vance of Cochrane's message. Each succeedsonly in accentuating its importance. It is im-possible nowadays to deny the requirementsfor serious research on outcomes, but the gapbetween true knowledge of outcomes and theneed to know and to understand, and hence tomeet these requirements, remains large.What was most important in 1972, and still

is for us now, is how to bridge that gap. Lastyear Iain Chalmers gave a masterful demon-stration of the practical extrapolation fromCochrane himself to the Cochrane Centre,clearly a central part of the "outcomes move-ment". This evolved from the National Peri-natal Epidemiology Unit and took on the workof Iain, Richard Peto, Rory Collins, and othersin summing the results of randomised trials oftreatment. There is no question in my mindthat these efforts in understanding the effec-tiveness of treatments, and hence helpingpeople choose between options, are of enor-mous value. For me, the most exciting hasbeen the work on breast cancer treatment.4The increasing use of randomised comparisonsin judging effectiveness is wholly appropriateand must go much further.

IMPEDIMENTSI want to leave aside, for the moment, theextent of the importance of uncertainty inhealth care decision making, and examinebriefly part of the existing resistance to the useof randomised controlled trials. Cochrane dis-cussed impediments to the wider acceptance ofthese in practice. There are three basic reasonsfor not carrying out randomised controlledtrials where uncertainty exists:

(1) Ethical objection (?)(2) Lack of objective outcome measures(3) The resistance of those "threatened" by

randomised controlled trials.

THE ARGUMENTSWe must be clear about the basic arguments,because in advocating greater, more system-atic, and rigorous use of observationalmethods5 in judging efficacy (as I shall bedoing later) it is very easy to associate oneselfwith a camp one does not wish to be associatedwith. There have been many exponents of theview, encapsulated by Feinstein,6 that "Cause-effect can be evaluated observationally" isscientific heresy only if two cogent scientificrealities are ignored - the first, that somethings are not amenable to experimental inter-vention, and the second, that randomised con-trolled trials are too expensive, too difficult, ortoo controversial. Whether or not such notionsderive from feeling threatened is of secondaryimportance only, but they are common.Of course trials are expensive and difficult,

but from a cost effective point of view they canbe much more readily justified because they

provide hard unbiased evidence about the rela-tive effectiveness of common, important, andexpensive treatments. It is absolutely no partofmy argument to disparage randomised com-parisons at all - least of all because they are tooexpensive or complicated. The end justifies themeans most of the time. To accept these kindof arguments is really to suggest that somealternative is generally better and that some-how it does not matter, because it is inevitable,that most treatments will not be evaluated byrandomised comparisons.

Feinstein and others (Colin Dollery, forexample, in his Rock Carling lecture7) assertthat most treatments will be assessed by amixture of insight, theory, acumen, and un-controlled observation. Of course large effectscan be reliably detected like that, but from nowon I will be discussing uncertainties over smallbut important effect differences in commonmedical practices which are poorly understoodand are simply not amenable to evaluation inthis way.There is a more extreme current idea -

which is just plain wrong - and this is that theuse of databases which record everythingabout patients can easily, as a consequence ofrecording everything, be used to adjust for allconfounders and then compare treatments as ifthere had been no selection.8 This has to bedismissed as ridiculous. Unknown and unsus-pected confounding is mostly important andalways elusive.The great idea of Fisher and Bradford Hill,

so ably developed by many, including ArchieCochrane, is one of the few panaceas of ourtime. In the context of medical interventions,the essence of the argument has been repea-tedly put by David Byer.' He says that "Epi-demiology is an essential discipline . . . forassessing the importance of exposures ... thereis a disassociation between the reason for expo-sure and the outcome. On the other hand, inmedicine, the doctor chooses the therapy pre-cisely to affect the outcome and for no otherreason". Archie's view was most succinctlyput, and poignant nonetheless: "Observationalevidence is clearly better than opinion but it isthoroughly unsatisfactory".

UNCERTAINTY, SUPPLIER-INDUCED DEMAND,AND CONSUMER PREFERENCESLet me restate the basic premise in case subse-quent remarks should be misunderstood. Ran-domised comparisons provide the most rigor-ous assessment of the therapeutic effect ofparticular interventions. The problems theyare set to help solve are, however, formidable,for decision making in medicine is rarely quiteas simple as choosing the best treatment foreach objective diagnosis, even when it isknown. In my opinion the dominant issues inhealth care now consist of, firstly, understand-ing the extent of true uncertainty and,secondly, relating this to the quality and costsof care. Then, since medical care is simultan-eously and consequently emerging from an eraof paternalism and medical domination, alldecision making now has explicitly and

7


http://jech.bmj.com

/J E

pidemiol C

omm


ebruary 1994. Dow

nloaded from


McPherson

Table I Basic criteria for assessment priorities inoutcome research

Provision should be commonProvision should be relatively expensiveThere should be obvious (explicit) uncertaintyThere should be real treatment choicesAssessment should have the potential for influencing choices

increasingly to take account of two impcconcepts. These are the role of supinduced demand and, secondly, how to acmodate consumer preferences into decitThese things were less important in (rane's day, and it seems to me they bringthem new emphases and responsibilities,in the method and the mechanism of ujstanding treatment outcomes. Of coursescientific/biological methods of medicin(its evaluation will always remain domi

3*00

-aa,cou

0)

0

co

(n)c

0

C.'cn.E

a)co

a)co

C:

co)

2 00

1 50

1 25

1 000

075

0 50

0 25

0

00

0

00

0

0

0

.

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

Figure 1 Small area variations in standardisedhospital admission ratios for various disorders.

Table 2 Magnitude of systematic variation (in ascending order) for selected caadmission among 30 hospital market areas in Maine: 1980-82

Variation Medical Surgical

Low: Inguinal hernia repair1 5 fold range Hip repairModerate: Acute myocardial infarction Appendectomy2 5 fold range Gastrointestinal haemorrhage Major bowel surgery

Cerebrovascular accident CholecystectomyHigh: Respiratory neoplasms Hysterectomy3-5 fold range Cardiac arrhythmias Major cardiovascular opera

Angina pectoris Lens operationsPsychosis Major joint operationsDepressive neurosis Anal operationsMedical back problems Back and neck operationsDigestive malignancyAdult diabetes

Very high: Adult bronchiolitis Knee operations8 5 + fold range Chest pain Transurethral operations

Transient ischemic attacks Extraocular operationsMinor skin disorders Breast biopsyChronic obstructive lung disease Dilation and curettageHypertension TonsillectomyAtherosclerosis Tubal interruptionChemotherapy

Reprinted with permission from the New England Journal of Medicine 1984;311:295-30/

)rtantplier-com-sions.ioch-..; f4%

but the implications of these other considera-tions must at least be properly assessed.

(2) The extent of true uncertainty inmedicineI would like to propose something I have oftensuggested before which will complement theexisting method of setting research priorities -this is the routine monitoring of practice varia-tion and examination of plausible causes.

WlLll Proposal 1: Routine monitoring of practiceboth variation and examination of plausible causesnder- In the determination of health care outcomee, the research priorities, several criteria aree and obviously foremost. These are set out in tableinant, 1. This is clearly an essential list but of course

tends to emphasise the glamorous, the inter-d esting, the pet obsessions, and some general

.\vP hierarchy of perceived medical scientific sta-tus, as well as the important. The problem mayalso be that health professionals have whollydifferent assessment priorities from the wellinformed consumers. If so, this would be aform of supplier-induced demand that isessentially unstudied in the health servicesliterature.

SMALL AREA VARIATION AND UNCERTAINTYThe true extent of clinical uncertainty canoften be measured by the amount of small areavariations in hospital admission rates (fig 1), onthe assumption that it is usually safe to assumethat important endogenous factors are stochas-tically constant. This is the "small area varia-tion" argument'0 glibly put. Thus, we can takeaway random variation and are left with acrude measure of uncertainty." As an index ofuncertainty, this is not a measure we are usedto. To measure priorities for outcome researchthere is a growing tendency to ask people whatthe major areas are ripe for technology assess-ment.

Variations can give a crude index of existinguncertainties as measured by practice varia-tion. Such a measure may also include genuinepreferences for particular outcomes. But what

ruses of it does not include is bias in the assessment ofpriorities based on possibly illegitimate, sub-jective (and possibly self serving) hierarchiesof relative importance. In this way a comple-ment to the assessment of priorities for out-come research should, in my view, be utterlyroutine. If we do this, we discover the by nowroutine differences between countries in

tions several very common reasons for hospital ad-mission and that the amount of variationbetween small areas corresponds sensibly withwhat we actually know about uncertainties.Examples are shown in table 2.12

ETHICS OF UNCERTAINTYFrom the point of view of the appropriateplace of clinical trials in the evaluation oftreatment we now come across a major obstacle

o. of principle. The formal method for assessing

8

..Rxl:,.,

e.


http://jech.bmj.com

/J E

pidemiol C

omm


ebruary 1994. Dow

nloaded from



priorities for research rests on experts express-ing their own uncertainties. When theseuncertainties are expressed and agreed (that is,are explicit) the ethical basis for randomisationis widely accepted. When uncertainties areobserved by epidemiological methods, as themost plausible explanation for small area var-iation, the uncertainties are implied only, andmay be an inadequate basis for ethical ran-domisation, at least as far as the responsibleclinician is concerned.

WIDE VARIATIONThe most important finding from small areavariation studies is that most hospital admis-sions are associated with variation as large orlarger than for hysterectomy (table 3). Ofcourse some of this variation may also be ameasure of preferences. It is not difficult toimagine systematic differences in preferencesbetween small areas with the current state ofboth the knowledge of outcomes and of infor-mation provision. Hence, most admissions arelikely to be associated with more uncertaintythan is commonly expressed or admitted, bothfrom the clinician and the consumer, as towhich is the appropriate treatment. Certainly,the most widely varying procedures do comeup in explicit hierarchies of uncertainty but thecommon causes of hospital admission whichvary importantly rarely do. From the point ofview of establishing the appropriate role ofrandomised comparisons this might representa temporary obstacle only, since the uncertain-ties are implied and not yet explicit. If they arelegitimate, nonetheless, they must eventuallybecome explicit.

ADDITIONAL REASONSTo add to Archie Cochrane's possible reasonsfor not doing clinical trials there is now animportant new fourth, and incidentally, a fifth,that is anyway quite well understood, and Iwon't dwell on it. These reasons are as follows:

(4) Uncertainty is not well enough estab-lished

(5) Reluctance to compare institutions orcarers.

(3) Studying medical uncertaintyAs I will discuss later, the nature of the exis-tence of medical uncertainty is complicatedbecause on the whole it is disparaged. Medicalteachers do not like it, for obvious reasons, and

Table 3 Proportion of hospital admissions categorisedby characteristic variation between small areas (data on400 000 hospital admissions in Maine in the 1980s)

Amount of variation Typical admission % Admissions

Low Hernia repair 1Moderate Appendicectomy 9High Hysterectomy 42Very high Disc removal 32Extremely high Tonsillectomy 16

100

Reprinted with permission from the New England Journal ofMedicine 1984;311:295-300.

certainly patients don't like it at all. In bothcases, the essential distinction between ignor-ance and uncertainty is blurred, but for en-tirely different reasons. Patients are hard putto acknowledge, much less recognise, the trueextent of uncertainty when they have been ledto believe that medicine is an exact and power-ful science; which of course it mostly is. Themedical profession is hard put to admit tolegitimate uncertainty when patients, on thewhole, desperately hope for certainties. How-ever, medical advance is ill served in the longrun by concealing uncertainties. Increasingly,of course, financial pressure will require evid-ence of effectiveness and efficiency, whichbrings me to my second proposal.

Proposal 2: Establish a strongmulti-disciplinary system of generatingappropriate, explicit uncertainties, particularlywhere unexpectedSuch a system will encompass the followingkinds of work:

Statistical overviewsVariation studiesRoutine databasesProper auditDecision analysisReviews of published reportsSurveys of need and preferencesCost effectiveness studiesPatient outcome research teams (PORT)

APPROPRIATE CONTEXTMuch is going on along these lines but there isuncertainty that the appropriate context isalways made clear. Thus, users of databasesmight well be disparaged by others, just as costeffectiveness studies might be ignored. Therole of all of these methods has to be con-sidered strictly in relation to the nature and theextent of true current uncertainties. Costeffectiveness studies, for example, are uselesswith unreliable information about effective-ness, if the plausible range includes zero orharmful effects. Likewise database analyses area waste of time (except in circumstances dis-cussed below) when large, unconfounded, ran-domised comparisons are available.That is why organisations like the PORTs in

America"3 have been useful, because they en-able, in principle, the true research prioritiesfor a particular outcome question to be pro-perly evaluated, unconstrained by data ordisciplinary constraints. Then these objectivepriorities point to and release these constraintswhere possible. At about the same time asEffectiveness and Efficiency2 was published, aseminal experience I had in America produceda book called Costs, Risk and Benefits of Sur-gery,'4 which used many of the kind of, essen-tially ancillary, methodologies enumeratedabove. Archie Cochrane was asked to review it,of course, and his comment was: "An excel-lent, courageous, pioneering book. I read itwith profit and pleasure." So clearly Archiewas also unambiguously not in favour of let-ting the "best become the enemy of the good".

9


http://jech.bmj.com

/J E

pidemiol C

omm


ebruary 1994. Dow

nloaded from


McPherson

300-

a)a 200-

ECD000

0

0',CoC,a)

: 100-0

USA

DenmarkCanadaNorwayNew ZealandAustralia

England and Wales

IrelandHolland

Sweden

Figure 2 Crude rates for prostatectomy in variouscountries in the 1980s.

UNCERTAINTY AND PROSTATECTOMYNow, a much publicised part of this process asit developed was an assiduous study of treat-ment for benign hypertrophy of the prostate.Such treatments, of course, represent the mid-dle ground of small area variation and the greatbulk of hospital admissions. It is preciselytreatments like this for which problems ofappropriateness and quality are almost com-pletely uninformed by reliable data on out-come. Practice styles have emerged and beenestablished using exactly the kind of criteria

Table 4 One year age standardised mortality after prostatectomy in three largedatabases, 1963-85

No of procedures Mortality (%)

Transurethral resection Open prostatectomy

Denmark 36 000 7-6 5-7Manitoba 12000 60 4-1Oxford 5 000 10-3 7-6

Reprinted with permission from the New England Journal of Medicine 1989;320:1120-4.

35r

4'

Manitoba Denmark Manitoba New Haven1 2

-IHCFA Kaiser

Inc

Figure 3 Adjusted relative risks of S year mortality after transuretheral and openprostatectomy in various published studies.

suggested by Feinstein et al,6 and as a conse-quence, the explicit uncertainties are low butthe implied uncertainties are very high. Thecriteria vary, the use rates vary, the costs vary,and presumably the outcomes vary. Butnowhere did treatment for benign hypertrophyof the prostate appear on the upper echelons ofresearch priorities - simply because mostpeople thought they knew what they weredoing.

Firstly, variation studies indicated a prob-lem of implied uncertainties.'5 Secondly, basicdecision analysis, with comprehensive litera-ture reviews, strongly suggested that the citedliberal reason for intervening surgically inpatients with minimal symptoms - to preventthe condition worsening and then suffering aless than optimal outcome for surgery - wasjust not justified.'6 The need for more reliabledata was obvious but a clinical trial could notbe contemplated, simply because the level ofuncertainty among urologists was not admis-sible.Databases of complete follow up of all

operations indicated quite remarkable discre-pancies in mortality for surgery than had everbeen published from case series in prestigiousinstitutions. An unsuspected finding emergedconcerning the new non-invasive operation(the development of which was partly whyrandomised comparison was not deemed to beindicated), suggesting a systematically highermortality in patients who underwent transur-ethral resection than open operation'7 (table 4).Such evidence about relative efficacy and

safety as this provides is, of course, whollycompromised by the possibility of selection;the worse prognosis patients being offered themore benign treatment while similar patientsare refused an open operation. Several studiessince this publication have tried to adjust forconfounding by taking account of case mix,and the evidence is inconclusive. The latest, byAlvan Feinstein et al,'8 seems to suggest casemix as the explanation but the study is far toosmall to refute an independent effect of thesurgery. '9

ESTABLISHMENT VIEWSMost interesting are the responses of the medi-cal/urological establishment to these results.In the USA a meeting of the executive councilof the American Urological Association, con-vened shortly after the Roos study was pub-lished,'7 emerged with the following state-ment, which explicitly acknowledges theuncertainty associated with the choice of opensurgery and transurethral surgery."Though the mechanisms of a direct causal

relationship are not apparent at this time, theweight of statistical evidence suggests that arandomized prospective study of internationalscope involving large numbers of patients, isindicated."

In my view this is the most rational responseto these data, because the question is clearlyone of attribution which can only be resolvedby a randomised controlled trial.'0 In contrast,an editorial in the British Medical J7ournal"

3-0O

2-5

20k

Co

0E

Ln0

a)>o._n I

0-5

10

_

1 5[-


http://jech.bmj.com

/J E

pidemiol C

omm


ebruary 1994. Dow

nloaded from



does not budge an inch in the direction ofrecognising any uncertainties whatsoever:"The fact that a recent review has suggested

that TURP may not be as good as urologistshave claimed does not detract from the import-ant advantages of this technique in treatingbenign prostatic hyperplasia".A wonderful quote which manages to dis-

miss the finding completely by calling it a"review" and then attributing a mere sugges-tion to that review. The important "advan-tages" of transuretheral prostatectomy sug-gested by the data from a completeenumeration of all prostatectomy patients inthree databases are an excess in one year mor-tality of around 40%. Of course, since the dataare inconclusive and inconvenient they can beignored, while inconclusive and convenient,but almost all small, selected, uncontrolledcase series, can be cited, essentially as estab-lished. This surely ought not to happen.

WHAT DOES THE DIFFERENCE MEAN?One is, of course, left wondering whether theobserved difference in the databases is attribu-table to the operation type, for there are manyplausible explanations for the transuretheralprostatectomy really having a higher risk ofmortality. These are mostly concerned withhypothermia associated with unheated irriga-tion fluid, for instance, or the nature of theirrigation fluid itself. The results of the onlyrandomised study22 is interesting because it isconsistent with an increased mortality. Thefive year mortality was 21% in the transureth-eral prostatectomy group and only 6% in theopen prostatectomy group, however, thenumbers were very small, some 40 in thetransuretheral prostatectomy group and 30 inthe other.A question we are left with is the extent to

which consumers of this kind of health carewould, given all the evidence outlined above,actually choose a transuretheral prostatectomyin preference to an open operation.

(4) The role of patient choices indecision makingIn deciding upon an appropriate medical orsurgical treatment the choice obviously oughtto be determined by the probabilities of par-ticular outcomes associated with each choice,combined with the individual preferences forthose outcomes. Where there is uncertaintyabout the true size of these probabilities, theprocedure is complicated, not least because ofthe demonstrated placebo effect of medicalcertainty. That is to say, the paternalistic doc-tor may actually be doing his (gender chosenadvisedly) patients more good than one whoexpresses the uncertainties honestly. So per-haps the above editorial in the British Medical7ournalP is dedicated only to making patientswith benign hypertrophy of the prostate better- who would otherwise not improve if theysuspected that the treatment could possiblyhave a significant downside. I will pursue someof the implications of this below.

Table 5 Common conditions for which patientpreferences may be dominant

Benign conditions of the uterusAngina pectorisCataractsBenign hypertrophy of the prostateGall stonesMenopausal symptomsBreast cancer

PATIENT PREFERENCEMeanwhile an important point receives littleattention in the effectiveness literature; that forsome decisions patient preferences are domin-ant while for others they are of no practicalconsequence, except for extreme cases of re-ligious bigotry. This depends of course on theprobabilities of different outcomes associatedwith each choice and the nature of the out-comes. For some decisions the probability ofdeath associated with one choice is unity, whilefor another choice it is less than one in athousand - which of course is no choice. How-ever, much in medicine is dominated by choiceand hence it is hoped by communicating theprobabilities associated with different choices.Some common examples are shown in table 5.

THRESHOLD QUESTIONSMost often the nature of the therapeutic choiceis not about which procedure, but is with thetiming and whether to do something or not; itis a threshold question, which dominates muchof the variation in observed rates (see fig 4). Itis simple to demonstrate that quite small dif-ferences in the threshold of signs or symptomsat which intervention happens on average canbe responsible for dramatic differences in therates of intervention. By the nature of theproblem these differences in treatment policyare seldom assessed unbiasedly by randomisedcomparison; partly because the uncertaintiesare not sufficiently stark and partly becauseconsumer choice is (often illegitimately23) citedas the driving force behind the decision. Butthe cost differences are often enormous.The nature of the choices can be illuminated

by using the methodologies in section (3), andsometimes it can be seen that the medicalcertainties are based on implausible theory.Rarely can the demonstration be sufficientlyconvincing to cause appropriate randomisedcomparison, however. As was shown above ifvested interests do not want to believe plaus-

Hi(c0

cJ

0

0a)

>C)

-0~

LowSymptom severity

Figure 4 Relationship of symptom severity andprevalence in prostatectomy.

11


http://jech.bmj.com

/J E

pidemiol C

omm


ebruary 1994. Dow

nloaded from


12 ~~~~~~~~~~~~~~~~~~~~~~~~~~McPherson

CU~~~~~~~~~......6.

0~~~~~~~~.

coV 4242~~~~~~.......4-V.........E.

~~~ ~ ~ ~ ~ ~ ~ .'

..0.0.

..0

~ ~ ~ ~ ~ ~ ~ ~ ~ ~

~~~ ~ ~ ~ ~ ~ ~ ~ ~ ~

1988.... 1989..1990

.5 .Menyears......

.rostati .pe. aier.la

ible but inconclusive argument, they will not.

Hence randomisation is "unethical"!

In the case of prostatectomy (fig 4) the

operation at an early stage in the development

of urinary symptoms was thought to be advan-

tageous compared with operating later. By

then the symptoms would be more intractable

and surgery would be more dangerous. De-

cision analysis and the combination of all avail-

able data showed this to be a most implausible

reality. However, the work required to reach

this conclusion, in the absence of a randomised

comparison of operating at different thresh-

olds, was both formidable and obviously

essentially inconclusive.

SUBOPTIMAL CHOICES

This does raise the interesting suggestion,

however, that if the determinants of treatment

policy are driven ultimately by the supplier

imposing a set of preferences that are different

from those of a well informed consumer of

services, then treatment choices may be

Table 6 Mortality of patients on placebo in double blind trials

Patient Trialadherence

Coronary drug project research group" fi Blocker Heart Attack TrialSy mortality 1 y mortality

Crude Adjusted Crude Adjusted

Poor 28-2 25-8 7-0 7-1Good 15-1 16-4 3-0 2-9

p<o.ooooo1 p<o.l

suboptimal. These preferences may all becompatible with current medical knowledgewhich may be subject to very wide uncertain-ties (see above) nonetheless. Supplier induceddemand clearly has strong theoretical, if notempirical, support.24

RELIABLE INFORMATIONSo the solution is, of course, to evaluate theoutcomes to provide reliable information fromwhich to make these decisions. In the shortterm this remains a forlorn hope, for reasonsdiscussed above. Meanwhile, to test the abovehypothesis directly, Wennberg et al5 haveproduced interactive video discs designed toinform patients about the extent of currentscientific knowledge and uncertainty associ-ated with particular clinical decisions. We areevaluating this technology in the UK usingrandomised methodology.The questions we are trying to answer are

the following. Firstly, do patients like havingextra information and the chance to participatein the decision making process? Secondly, dothe clinical decisions change as a consequenceof providing this information? Thirdly, dopatients do better, with respect to symptoms,survival, or quality of life, as a consequence?First indications in the USA are that thedecisions made with more information are verydifferent from those made without and thatpatients are in favour of the extra information.(See fig 5.)

EFFECT OF CHOICE ON OUTCOMEA more fascinating aspect still of the role ofpatient preference in clinical decision makingcomes from studying the effect that choiceitself has on therapeutic outcome. This is mythird proposal.

Proposal 3: Investigate areas systematicallywhere treatment preferences affect outcomeimportantlyI will cite two important examples of observa-tions from clinical trials where the indirectevidence strongly suggests that choice itselfeffects hard outcomes such as survival.29Table 6 shows a summary of the results ofdouble blind clinical trials in which the outcomeamong patients given placebo is comparedaccording to whether patients actually took theprescribed pills. Choosing to take the placebopills is then the treatment comparison in cir-cumstances when the patients did not knowthat they were on placebo. Both trials, afteradjusting for all measured possible con-founders, give a strong hint that choice itselfcan dramatically effect important measures ofoutcome. There are many other examples ofthis phenomenon in the published reports andthe mechanisms could, of course, be various, a

synergistic combination of biology, immu-nology and pathology, with psychology30 muchof which is quite well understood already.

12


http://jech.bmj.com

/J E

pidemiol C

omm


ebruary 1994. Dow

nloaded from



PLACEBO EFFECTProfessor Dave Sackett has cited five examplesin the literature of the placebo taker faringbetter if they took the drug seriously. In theoriginal hypertension trial in Virginia a 30%reduction in mortality was observed amongcompliant placebo takers (p = 008). Similarresults have been found in weight loss" and inalcoholism,32 schizophrenia, and in cancerpatients." Hence not only are patient prefer-ences intrinsically important in offering parti-cipation in affairs which intimately affect theirlives, but also perhaps because choice itself issometimes apparently of therapeutic benefit.This is important because such a possibilitymay have important methodological impli-cations. Either patient self selection affectsoutcome because of confounding with pro-gnostic variables and hence randomised com-

parisons are essential or, alternatively, choice orcontrol affects outcome and hence (unblind)randomised comparisons are biased (down-wards).

In the coronary drug project result reportedabove28 the idea that residual confounding canexplain the difference in mortality is implau-sible, except that all the variables used toadjust were, of course, those measured at base-line, and possibly are thus compromised aspotent confounders.

RANDOMISATION AND BIASThe idea therefore that randomisation itselfcan give rise to biased results about outcomecomes as shock to many people. But clearlywhen treatments are allocated randomlypatients are deprived of choice and if choiceand control can be of therapeutic benefit then arandomised comparison might provide abiased - downwards - estimate of the effective-ness of treatment. We are familiar with theidea that efficacy might give an euphoric estim-ate of effectiveness, because things that mattermight change outside a clinical trial for theworse. I an now suggesting a possible counter-vailing influence where things might get betteroutside randomised comparisons. This wouldbe true if and only if choice or control itselfeffects outcome. If true, another reason for notgeneralising altogether from randomised trialis that they might thus be systematicallybiased. Hence a sixth reason for not wantingalways to attribute outcomes by using ran-

domised controlled trials is:(6) Choice or preference or control affects

outcomeThere are, of course, many plausible mechan-isms by which such an effect could be import-ant. The literature on the physiology of theplacebo effect is dominant here34 as well as theevidence, for instance, implicating social or

professional control in the aetiology of coron-

ary heart disease.'5

(5) The role of choice in evaluatingoutcomesClearly the role of choice or control needs to besystematically studied in the following con-

texts explicitly. It is important here to distin-guish between preferences for treatmentsbecause of their known effects and the role ofchoice in itself determining outcome to somedegree. They will, in general, be inextricablyconfused because I suspect that choice onlyeffects outcome via belief in particular attribu-table consequences of that choice. But here Iam trying to emphasise the latter phenomenonas part of the therapeutic process itself. Areasin which patient preference may have an im-portant role are as follows:

(1) Randomised controlled trials(2) Therapeutic efficacy(3) General priority setting(4) Particular therapeutic choices

PREFERENCE TRIALSOne way to investigate this more closely inrandomised trials is to mount preferencetrials36 along with prospective randomisedtrials37 to test for important therapeutic andbiological effects and simultaneously any pos-sible synergy with treatment choice: an ex-ample is shown in table 7. The explicit ideahere is to compare the outcome in patients whohave chosen a particular treatment with thosewho have been randomised to the same treat-ment. Obviously, there will be total confound-ing between choice and selection on known orunknown prognostic indicators. At least wecan measure the potency of known prognosticindicators. But the alternative to not trying toinvestigate this phenomenon is, it seems to me,to deny its existence, and there is too much ofthat going on already. It seems quite plausiblethat for some conditions this will be importantand for others utterly irrelevant, but it wouldbe good to know which.The challenges we are faced with are not

only to understand about outcomes and theirdeterminants, which is formidable enough, butalso to assess routinely knowledge on the qua-lity of life and functioning from a patient'spoint of view. Most importantly, the mainchallenge is to create a decision making en-vironment which is capable of optimally usingreliable information on and data about patientoutcomes.

(6) ConclusionsThe evidence for common uncertainties inclinical practice is overwhelming and, clearly,important carving away at key parts of it is thedominant component of Archie Cochrane'slegacy. I have sought to draw attention to someof the potential problems in priority settingand in the evaluation of treatments, where

Table 7 Preference trials - a way of investigatingchoice

(1) Measure therapeutic effect of choice, if any: for examplerandomised controlled trial of patients who have seeninteractive video disc versus normal care

or(2) Wennberg proposal: randomised controlled trial of

preferences with information versus randomisedcontrolled trial with information - for example surgery,drugs, watchful waiting

13


http://jech.bmj.com

/J E

pidemiol C

omm


ebruary 1994. Dow

nloaded from


McPherson

consumer input is important and possibly neg-lected. I believe that these considerations areimportant methodologically as well as substan-tively and their methodological implicationsare poorly investigated as yet.The true role of uncertainty in medicine has

yet to be properly investigated also, and this ispartly because it fits uneasily into the domin-ant medical paradigm. Much of what happensin medical school would change, much of whathappens in clinical practice would change,much that happens in media coverage of medi-cine would change, and perhaps much of whathappens in the evaluation of outcome wouldchange, if the true nature and extent of uncer-tainty in medicine were readily acknowledged.Perhaps effectiveness itself would change too,but until we know when and how, and inwhich direction, there seems little point in notinvestigating whether it would. Certainly thepolitics of health care planning would change.

STYLE DIFFERENCESMy final proposal is again something I haveoften suggested which is to take greater advan-tage of a manifestation of uncertainty, dif-ferences in practice style.

Proposal 4: Practice style differences exploitedin natural experiments of outcome, whererandomisation is not feasibleSystematic practice style differences are com-mon and they exist because clinicians adoptdifferent criteria for intervening, largely as aconsequence of assuming that informal, theor-etical, and anecdotal methods of assessment ofoutcome are good enough. Since they exist italways seems an enormous waste not to takeadvantage of these natural experiments wheretreatment at the margin is not chosen, as DavidByer would have described things, entirely byobjective measures of prognosis but muchmore as a random consequence of history.Figure 6 shows variations in the use of prosta-

a,

CLQ

a)E0

0

0

0

0

CDa)

co

-0

a)

.a_

E.

c):o

130

120

110

100

90

80_

70

60

50

NW ThamesSouth West

SW ThamesYorkshireNE ThamesOxfordWessexSE ThamesEast AngliaWalesNorthernTrent

North West

West Midlands

Figure 6 Age standardised rates for prostatectomy inEngland and Wales.

tectomy in English health regions in the1970s.38 These variations in practice style haveyielded no information whatsoever about theirconsequences for the patients, and there mustbe some.

OBSERVATIONAL METHODSI believe that some of the observational metho-dology proposed is, in principle, cheap andcould be routine. It is not so cheap nowbecause one has to develop mechanisms notalready in existence to capture all cases and toallow useful follow up of outcome as well as toattract the collaboration of clinicians. It cannotreplace randomised controlled trials except incircumstances where the attribution of causewith respect to outcome is unambiguous, andthis will sometimes happen. We will have toget better at determining the causes ofobserved differences by knowing about thepotency of plausible confounders.39 What I amproposing is systematic study of enormousnumbers of cases so that even rare and longterm outcomes can be systematically com-pared.

CRITERION FOR RANDOMISED COMPARISONRandomised comparisons are only practicallyfeasible when the major primary outcome(s) ofinterest need to be unambiguously attributedto treatment choice and cannot be attributedwithout them. We are often concerned withsecondary outcomes (like long term rare sideeffects) here and nobody is going to randomisepatients when the primary outcome compari-sons (functional status at 30 days for example)are essentially, or even approximately, under-stood. What is often required is a clear cohortcollected at a time when there are uncertaintiesabout the primary questions of outcome attri-bution and hence when the cohort is relativelyuncontaminated by biased clinical selection onthe basis of prognosis.An example is the collaboration with the

Royal College of Surgeons in their audit of allcases of prostatectomy for benign hypertrophyof the prostate. For the first time, and essen-tially incited by the other observational workon prostatectomy already quoted, they arestudying outcome on some 4000 consecutivecases. The hypotheses being investigated areto do with the characteristics of patients whomight benefit from this operation from amongthose now being offered the operation. Testingtransuretheral prostatectomy against open sur-gery remains a long way off I suspect. Clearlylarge numbers are not in themselves enough,but it emerges (conditioned somewhat by pos-sible response bias) that fascinating dif-ferences, and in some cases fascinating similar-ities, in outcomes themselves raise importanthypotheses to be tested individually in thefuture.

POSTAL ASSESSMENTSHowever, what emerges clearly from thisstudy, and another I have recently completed

14

ay


http://jech.bmj.com

/J E

pidemiol C

omm


ebruary 1994. Dow

nloaded from



Table 8 Response rates to a postal questionnaire in 388 men who underwentprostatectomy

Time

At surgery At 3 mth At 6 mth At 12 mth

Patients (%) 100 93 92 91Surgeons (%) 99 81 79 100

in collaboration with Nick Black and others, isthat patients who have received treatment are

very willing to complete, even detailed, postalquestionnaires about the state of their health.40It seems slightly scandalous to waste the goodwill evidenced in table 8 in obtaining informa-tion which could prove to be extremely valu-able. Record linkage has often been imple-mented to obtain follow up information on

outcome, but it has obvious limitations.

Proposal 5: (Postal) outcome assessments bypatients become routine, with and withouttreatmentThus, if collected in a uniform and systematicmanner, basic information not now available inany analysable form could usefully inform out-come research. This must eventually include,of course, patients with symptoms not deemedappropriate for intervention.

I conclude with a summary of the mainpoints I have tried to make which encapsulatea hierarchy of methodologies designed simplyto establish the effectiveness of health inter-ventions. These main points are:

(1) Monitor practice variations.(2) Enhance multi-disciplinary generation

of appropriate uncertainties.(3) Determine where treatment preferences

affect outcome.(4) Undertake natural experiments of prac-

tice style.(5) Routine long term postal follow up for

important questions can be valuable.(6) Randomised trials for small, important,

possibly confounded effects are essential.

1 Cochrane AL with Blythe M. One man's medicine. London:Memoir Club, BMJ, 1989.

2 Cochrane AL. Effectiveness and efficiency. Random reflec-tions on health services. London: Nuffield Provincial Hos-pital Trust, 1972.

3 Chalmers I, Dickerson K, Chalmers TC. Getting to gripswith Archie Cochrane's agenda. BMJ 1992;305:786-8.

4 Early Breast Cancer Trialists' Collaborative Group. Sys-tematic treatment of early breast cancer by hormonal,cytotoxic, or immune therapy. An overview of 133 ran-

domised trials involving 31,000 recurrences and 24,000deaths among 75,000 women. Lancet 1992;339:1-15,71-85.

5 Anonymous. Databases for health care outcomes. Lancet1989;335:195-6.

6 Feinstein AR. Current problems and future challenges inRCTs. Circulation 1984;70(5):767-74.

7 Dollery C. An end of an age of optimism. London: NuffieldHospital Hospitals Trust, 1978.

8 Green SB, Byer DP. Using observational data from regis-tries to compare treatment: the fallacy of omnimetrics.Stat Med 1984;3:361-70.

9 Byar DP. Problems with using observational databases to

compare treatments. Stat Med 1991;10:663-6.

10 Wennberg JE, Gittlesohn A. Small area variations in healthcare delivery. Science 1975;182:1 102-8.

11 McPherson K, Wennberg JE, Hovind OB, Clifford P.Small area variations in the use of common surgicalprocedures: an international comparison of New England,England, and Norway. N EnglI Med 1982;307:1310-4.

12 Wennberg JE, McPherson K, Caper P. Will payment basedon diagnostic related groups control hospital costs. N EnglJ Med 1984;311:295-300.

13 Wennberg JE. What is outcomes research? In: Gelijns AC,ed. Medical innovations at the crossroads. Vol I Modernmethods of clinical investigation. Washington DC: NationalAcademy Press, 1990:33-46.

14 Bunker J, Barnes B, Mosteller F. Costs, risks and benefits ofsurgery. Oxford: Oxford University Press, 1977.

15 Wennberg J, Roos N, Sola L, Schori A, Jaffe R. Use ofcalim's data systems to evaluate health care outcomes:mortality and reoperation following prostatectomy.JAMA 1987;257:933-6.

16 Barry MJ, Mulley AG, Fowler FJ, Wennberg JE. Watchfulwaiting vs immediate transurethral resection of the pro-state for symptomatic prostatism. JAMA 1988;259:3010-17.

17 Roos NP, Wennberg JE, Malenka DJ, et al. Mortality andreoperation after transurethral resection of the prostatefor benign prostatic hyperplasia. N Engl Jf Med1989;320:1 120-4.

18 Concato J, Horwitz RI, Feinstein AR, et al. Problems ofcomorbidity in morality after prostatectomy. JAMA1992;267: 1077-82.

19 Roos N, Roos L, Cohen M, et al. Therapies for benignprostatic hyperplasia (letter). JAMA 1992;268(10):1269.

20 Bracken MB. Clinical trials and the acceptance of uncer-tainty. BMJ 1987;294:1111-2.

21 Chisholm GD. Editorial. Benign prostatic hyperplasia: thebest treatment. BMJ 1989;299:215-6.

22 Meyhoff HH. Transurethral versus transvesicle prostatec-tomy: a randomised study. Scand Jf Urol Nephrol1987;4(S102):26-32.

23 Coulter A, McPherson K. Socioeconomic variations in theuse of common surgical operations. BMJ 1985;291:183-7.

24 Wennberg JE, Barnes BA, Zubkoff M. Professional uncer-tainty and the problem of supplier induced demand. SocSci Med 1982;16:811-24.

25 Kasper JP, Mulley AG, Wennberg JE. Developing shareddecision making programs to improve quality of healthcare. Quality Review Bulletin. Journal of Quality Im-provement 1992;18:182-90.

26 Sidney S, Queensberry CP, Sadler MG, et al. Reoperationand mortality after surgical treatment of benign prostatichypertrophy in a large medical care program. Med Care1992;30: 117-25.

27 Winslow R. Videos, questionnaires aim to expand role ofpatients in treatment decisions. Wall St Journal, 25February 1992.

28 Coronary Drug Project Research Group. Influence of ad-herence to treatment and response of cholesterol onmortality in the coronary drug project. N Engl J Med1980;30: 1038-41.

29 Horowitz RI, Viscolli CM, Berkman L, et al. Treatmentadherence and risk of death after a myocardial infarction.Lancet 1990;336:542-5.

30 Philips DP, Todd RE, Wagner LM. Psychology and survi-val. Lancet 1993;342:1142-5.

31 Asher L, Harper HW. Effect of human chorionic gonado-trophin on weight loss, hunger, and feeling of well-being.Am J Clin Nutrition 1973;26:211-18.

32 Fuller R, Roth H, Long S. Compliance with disulfiramtreatment of alcoholism. Journal of Chronic Disease1983;36: 161-70.

33 Pizzo PA. Oral antibiotic prophylaxis in patients withcancer: a double-blind randomized placebo-controlledtrial. J Pediatr 1983;102(1):125-33.

34 Wolf S. Effects of suggestion and conditioning on the actionof chemical agents in human subjects - the pharmacologyof placebos. J Clin Investigation 1950;29:100-9.

35 Siegrist J, Peter R, Junge A, et al. Low status control, higheffort at work and ischaemic heart disease: prospectiveevidence from blue-collar men. Soc Sci Med1990;31(10):1 127-34.

36 Brewin CR, Bradley C. Patient preferences and randomisedclinical trials. BMJ 1989;299:313-5.

37 Henshaw RC, Naji SA, Russell IT, Templeton AA. Com-parison of medical abortion with surgical vacuum aspi-ration and acceptability of treatment. BMJ 1993;307:714-7.

38 McPherson K. Variations in hospitalization rates: why andhow to study them. London: The King's Fund Institute,1988.

39 McPherson K. Diversity and similarity of health: organisa-tion, practice and assessment. In: Normonds C, VaughanP, eds. Europe without frontiers. Chichester: J Wiley,1993.

40 Doll HA, Black NA, McPherson K, et al. Mortality andcomplications following transurethral resection of theprostate for benign prostatic hypertrophy. J Urology(Balti) 1992;147:1566-73.

15


http://jech.bmj.com

/J E

pidemiol C

omm


ebruary 1994. Dow

nloaded from


good: randomised and - jech.bmj.com · adult diabetes very high: adult bronchiolitis kneeoperations...

Documents