searching for gold standards
DESCRIPTION
Searching for Gold Standards Simon Wessely http://www.lse.ac.uk/collections/BIOS/EBMRCTevent.htm Searching for Gold Standards - the Construction and Governance of RCTs & EBM in Psychiatry "On the 8th and 9th of June 2006, an inter-disciplinary group of around 50 researchers gathered at the London School of Economics for a two-day symposium on evidence-based medicine (EBM) and the randomised controlled trial (RCT). Organised by the Institute of Psychiatry, King's College London & the BIOS Centre, LSE, the aim of the symposium was to bring together social scientists, clinicians and researchers to debate some of the most pressing issues surrounding EBM and RCTs in Psychiatry. The symposium was made possible through the kind support of the Nuffield Foundation and the Foundation for the Sociology of Health and Illness".TRANSCRIPT
Why David Healy is WrongWhy David Healy is Wrong
WHY RANDOMISE?WHY RANDOMISE?
Why RCTs are the best…I really mean it… cross my heart
and hope to die….
Why RCTs are the best…I really mean it… cross my heart
and hope to die….
Insert Picture of Sliced BreadInsert Picture of Sliced Bread
Why do we bother with RCTs?Why do we bother with RCTs?
Why bother to randomise?Why bother to randomise?
Slater and Sargeant, 1963Slater and Sargeant, 1963
• If they fail to demonstrate any differences between a placebo and a drug which everybody knows to be effective, this means only that the work has not been done well enough
• If they fail to demonstrate any differences between a placebo and a drug which everybody knows to be effective, this means only that the work has not been done well enough
We are never going to learn how to treat depressions properly from double blind sampling in an MRC statistician’s office
We are never going to learn how to treat depressions properly from double blind sampling in an MRC statistician’s office
Sargeant, BMJ, 1963
If you can believe fervently in your treatment, eve n though controlled tests show it is quite useless, then you r results are much better, your patients are much better and your income is much better too. I believe this accounts for the remarkable success of some of the less gifted, but more credulous members of our profession, and also for t he violent dislike of statistics and controlled test w hich fashionable and successful doctors are prone to dis play
If you can believe fervently in your treatment, eve n though controlled tests show it is quite useless, then you r results are much better, your patients are much better and your income is much better too. I believe this accounts for the remarkable success of some of the less gifted, but more credulous members of our profession, and also for t he violent dislike of statistics and controlled test w hich fashionable and successful doctors are prone to dis play
Richard Asher, 1972
So what is unique about the randomised controlled trial?So what is unique about the randomised controlled trial?
So what is unique about the RCT?So what is unique about the RCT?
• Is it having a control group?• Is it having a control group?
So what is unique about the RCT?So what is unique about the RCT?
• Is it having a control group?• Is it the blindness?
• Is it having a control group?• Is it the blindness?
So what is unique about RCT?So what is unique about RCT?
• Is it having a control group?• Is it the blindness?• Is it controlling for the effects of chance?
• Is it having a control group?• Is it the blindness?• Is it controlling for the effects of chance?
Have the Samaritans Reduced the Suicide Rate?
RANDOMISATION IS THE ONLY WAY TO CONTROL
CONFOUNDERS
RANDOMISATION IS THE ONLY WAY TO CONTROL
CONFOUNDERS
And what happens if you don’t randomise?
And what happens if you don’t randomise?
YOU GET THE WRONG ANSWER
YOU GET THE WRONG ANSWER
Randomization: the Gold StandardRandomization: the Gold Standard
Randomization Non randomassignment
Cheatingdifficult (%)
Cheating easy(%)
Proportion ofstudies withp < 0.05
8.8 24.4 58.1
Randomization Non randomassignment
Cheatingdifficult (%)
Cheating easy(%)
Proportion ofstudies withp < 0.05
8.8 24.4 58.1
Chalmers et al, NEJM, 1983
Boys behaving badlyBoys behaving badly
Cambridge Somerville Youth Study
Cambridge Somerville Youth Study
• 750 delinquent boys• randomised• “sustained friendly counselling”• Intervention lasted five years
• 750 delinquent boys• randomised• “sustained friendly counselling”• Intervention lasted five years
5, 10, 20 and 30 years later, one group had more
5, 10, 20 and 30 years later, one group had more
• criminal convictions• unemployment• premature death• alcoholism• Serious mental illness
• criminal convictions• unemployment• premature death• alcoholism• Serious mental illness
McCord, 1978
Men behaving badlyMen behaving badly
“FORWARD PSYCHIATRY” – The PIE Principles
“FORWARD PSYCHIATRY” – The PIE Principles
• Proximity
• Immediacy
• Expectancy
• Proximity
• Immediacy
• Expectancy
RFA Canberra
What else can we do about psychological trauma?
What else can we do about psychological trauma?
What is debriefing?What is debriefing?
• Introduction• Describing traumatic incident• Psychological reactions to incident• Emotional processing/catharsis• Identifying symptoms• Teaching - symptoms are normal• Re entry/closure
• Introduction• Describing traumatic incident• Psychological reactions to incident• Emotional processing/catharsis• Identifying symptoms• Teaching - symptoms are normal• Re entry/closure
Debriefing has been used for….Debriefing has been used for….
• Police after shooting incidents• Sailors after maritime collisions• Red Cross personnel• Medical students whose patients have
died• Families whose children are
undergoing transplants• Soldiers on grave registration duties• Train drivers who have witnessed
suicides• Jurors in murder trials• Air Force personnel on whose Base
there has been a fatal accident• Casualty staff after trauma incidents
• Police after shooting incidents• Sailors after maritime collisions• Red Cross personnel• Medical students whose patients have
died• Families whose children are
undergoing transplants• Soldiers on grave registration duties• Train drivers who have witnessed
suicides• Jurors in murder trials• Air Force personnel on whose Base
there has been a fatal accident• Casualty staff after trauma incidents
• Burns victims• accident victims• Hospital staff after failed
resuscitations• Nurses in cancer care• patients who have recovered from
cancer• Rape victims• Rescue workers after any natural
disaster• The entire New York Fire Department• The entire New York Police
Department• Children attending a school where a
pupil has died• Ward staff where a suicide has
occurred
• Burns victims• accident victims• Hospital staff after failed
resuscitations• Nurses in cancer care• patients who have recovered from
cancer• Rape victims• Rescue workers after any natural
disaster• The entire New York Fire Department• The entire New York Police
Department• Children attending a school where a
pupil has died• Ward staff where a suicide has
occurred
“the experiences of 700 CISM teams in more than 40,000 debriefings cannot be ignored,
especially so when the overwhelming majority of reports are extremely positive”
“the experiences of 700 CISM teams in more than 40,000 debriefings cannot be ignored,
especially so when the overwhelming majority of reports are extremely positive”
Mitchell & Everly, 2003
Does it work? How dare you even ask?
“numerous studies have already shown positive results….proves the clinical
effectiveness beyond reasonable doubt”
“numerous studies have already shown positive results….proves the clinical
effectiveness beyond reasonable doubt”
Single session psychological debriefing definitely does not
work…
Single session psychological debriefing definitely does not
work……..and increases the risk of PTSD…..and increases the risk of PTSD
Rose, Wessely, Bisson – Cochrane Review 2003: Emmerinket al, Lancet, 2002
So why doesn’t debriefing work?
So why doesn’t debriefing work?
• Not everyone wants or needs to talk• “retraumatisation”• Interferes with natural recovery processes• Suggestion• Professionalisation of distress• Impedes people talking to who they want, when
they want
• Not everyone wants or needs to talk• “retraumatisation”• Interferes with natural recovery processes• Suggestion• Professionalisation of distress• Impedes people talking to who they want, when
they want
Everyone behaving badlyEveryone behaving badly
This can’t be rightThis can’t be right
Graded Exercise Therapy (GET) for CFS: (Fulcher & White, BMJ 1997)
Graded Exercise Therapy (GET) for CFS: (Fulcher & White, BMJ 1997)
• Setting: National Sports Centre
• Design: RCT
• Treatment: 12 weeks Graded Exercise Therapy
• Control Flexibility exercises
• Patients: 66 CFS patients
• Results: GET superior
• Conclusions: GET safe and effective
• Setting: National Sports Centre
• Design: RCT
• Treatment: 12 weeks Graded Exercise Therapy
• Control Flexibility exercises
• Patients: 66 CFS patients
• Results: GET superior
• Conclusions: GET safe and effective
Percentage improved with GETPercentage improved with GET
0
10
20
30
40
50
60
70
Fulcher Wearden Powell
GETControl
Percentage improved with GETPercentage improved with GET
0
10
20
30
40
50
60
Moss-Morris
Wallman
GETControl
What is CBT?What is CBT?
• Collaborative• Identifies cognitive barriers• Sets behavioural targets, not exercise• Emphasises predictability/consistency• Does relapse prevention
• Collaborative• Identifies cognitive barriers• Sets behavioural targets, not exercise• Emphasises predictability/consistency• Does relapse prevention
CBT for CFS: the Oxford RCT (Sharpe et al, BMJ 1997)
CBT for CFS: the Oxford RCT (Sharpe et al, BMJ 1997)
• Setting: Medical out patient clinic
• Treatment: 16 sessions CBT given by skilled therapists
• Patients: 60 CFS patients
• Results: CBT superior at 12 months on symptoms and function
• Setting: Medical out patient clinic
• Treatment: 16 sessions CBT given by skilled therapists
• Patients: 60 CFS patients
• Results: CBT superior at 12 months on symptoms and function
1996
CBT for CFS: The King’s RCT (Deale et al, 1997)
CBT for CFS: The King’s RCT (Deale et al, 1997)
• Setting: CFS Clinic
• Treatment: 12 sessions CBT
• Control 12 sessions relaxation
• Patients: 72 patients with CFS
• Results: CBT better on symptoms and disability
• Five year follow up Differences less, but still pr esent
• Setting: CFS Clinic
• Treatment: 12 sessions CBT
• Control 12 sessions relaxation
• Patients: 72 patients with CFS
• Results: CBT better on symptoms and disability
• Five year follow up Differences less, but still pr esent
Outcome at 6 months follow upOutcome at 6 months follow up CBT Relaxation
Much improved (Increase of 50 or more on MOS score)
16 (53%) 3(10%)
Improved 8 (27%) 11 (37%)
Unimproved (incl drop outs)
6 (20%) 16 (53%)
CBT Relaxation
Much improved (Increase of 50 or more on MOS score)
16 (53%) 3(10%)
Improved 8 (27%) 11 (37%)
Unimproved (incl drop outs)
6 (20%) 16 (53%)
CBT for CFS: a multi-centre RCT (Prins et al Lancet 2001)CBT for CFS: a multi-centre RCT (Prins et al Lancet 2001)
• Setting: Medical out patient clinic
• Treatment: CBT given by newly trained therapists; Support group or Natural course.
• Patients: 278 patients with CFS
• Results: CBT group: 35% improved on fatigue; 49% physical functioning; 50% self rated improvement.
• Conclusions: CBT was more effective than guided support and natural course with non specialist therapists.
• Setting: Medical out patient clinic
• Treatment: CBT given by newly trained therapists; Support group or Natural course.
• Patients: 278 patients with CFS
• Results: CBT group: 35% improved on fatigue; 49% physical functioning; 50% self rated improvement.
• Conclusions: CBT was more effective than guided support and natural course with non specialist therapists.
RCT of patient education to encourage graded exerci se in CFS
(Powell et al BMJ 2001)
RCT of patient education to encourage graded exerci se in CFS
(Powell et al BMJ 2001)
• Setting: Medical out patient clinic
• Patients: 148 patients with CFS
• Treatment: standard medical care; GET; telephone advice + GET: face to face advice + GET
• Results: 69% improved in the intervention groups 6% of controls improved
• 4 years later Improvements maintained
• Setting: Medical out patient clinic
• Patients: 148 patients with CFS
• Treatment: standard medical care; GET; telephone advice + GET: face to face advice + GET
• Results: 69% improved in the intervention groups 6% of controls improved
• 4 years later Improvements maintained
Routine Clinical Practice (Chalder et al)
• Setting: King’s
• Design: Prospective uncontrolled study
• Treatment: Routine CBT
• Patients: 293 patients with CFS
• Results: 58% rated themselves as very/much or much better; 26% were a little better; 16% were the same or worse on global outcome
• Conclusions: Outcomes only slightly less than in the RCTS
Game, set and match ……… .Game, set and match ……… .
Action for ME Website(www.afme.org.uk)
Action for ME Website(www.afme.org.uk)
Q. What treatments are available?
A. Research has proven two therapies bring relief for many people with ME…………...
Q. What treatments are available?
A. Research has proven two therapies bring relief for many people with ME…………...
Action for ME Website(www.afme.org.uk)
Action for ME Website(www.afme.org.uk)
…….magnesium injections and the dietary supplement Efamol (a
combination of evening primrose oil and marine oil)
…….magnesium injections and the dietary supplement Efamol (a
combination of evening primrose oil and marine oil)
COMP ARISO N OF T R EATM ENT APP RO A CH ES
GRO UPS RESEAR C H sample he lped m ade no m adeSEVERELY AFFEC TE D SURVEY diffe rence w orse
PACI NG GRO UPS 257 88% 9% 3%
SEVERELY AFFEC TE D 2,180 89% 9% 1%
GR AD ED EX ERCIS E TH ER APY GRO UPS 209 39% 22% 39%
SEVERELY AFFEC TE D 1,214 34% 15% 50%
COG NITIVE B EH AVIOU R AL TH ER APY GRO UPS 113 55% 32% 13%
SEVERELY AFFEC TE D 285 7% 67% 26%
Action for ME Self help Group Survey
Psychiatry is the dustbin of the medical profession
(Clare Francis, 1988)
Psychiatry is the dustbin of the medical profession
(Clare Francis, 1988)
Psychiatry is opinion dressed up as science, stupid and hypocritical
(Clare Francis, 1996)
Psychiatry is opinion dressed up as science, stupid and hypocritical
(Clare Francis, 1996)
“People with postviral fatigue syndrome often have to put with a lot of disbelief-there were many doctors who diagnosed this as a psychiatric disorder although on the whole
it is taken much more seriously now”
“People with postviral fatigue syndrome often have to put with a lot of disbelief-there were many doctors who diagnosed this as a psychiatric disorder although on the whole
it is taken much more seriously now”
“Watchdog to look into ME resources”: Dundee Courier and Advertiser, 11.11.1994
“ME is an imaginary illness, for which the best treatment is
psychiatric”
“ME is an imaginary illness, for which the best treatment is
psychiatric”
Unnamed physician, cited in Steincamp, 1989
Without the RCT we would never have been able to
improve the care of patients
Without the RCT we would never have been able to
improve the care of patients
People don’t do clinical trials in mental health because........
People don’t do clinical trials in mental health because........
• I know my treatment works• My treatment is too individual• The results are not generalisable
• I know my treatment works• My treatment is too individual• The results are not generalisable
What’s the real reason?What’s the real reason?
They are bloody difficultThey are bloody difficult
Problems with Psychiatry Trials: Our Fault
Problems with Psychiatry Trials: Our Fault
• Not big enough• Not good enough• Too many rating scales• Testing wrong kind of things
• Not big enough• Not good enough• Too many rating scales• Testing wrong kind of things
Cochrane Depression/Anxiety Group- Interventions
Cochrane Depression/Anxiety Group- Interventions
10%
23%
67%
Other
Psychotherapy
Pharmacotherapy
Problems with Psychiatry Trials: Not Our Fault
Problems with Psychiatry Trials: Not Our Fault
• Problems with follow up/drop outs• Fuzzy outcome measures• Large non specific effects• Complex interventions• Ethical dilemmas (capacity, coercion)• Bureaucratic nightmare which is getting worse
• Problems with follow up/drop outs• Fuzzy outcome measures• Large non specific effects• Complex interventions• Ethical dilemmas (capacity, coercion)• Bureaucratic nightmare which is getting worse
So what is the moral of the story?So what is the moral of the story?
When in doubt….When in doubt….
Randomise!!Randomise!!