1

Randomised Controlled Trials in the Social Sciences:Challenges and Prospects

I . Boutron

Dept of Epidemiology, Biostatistics and Clinical Research

Assessing Nonpharmacological treatments

Challenges and reporting guidelines for Randomised Controlled Trials

2

Overview

• Specific methodological issues when assessing Nonpharmacological treatments

• Quality tools

• Reporting guidelines

3

Overview

• Specific methodological issues when assessing Nonpharmacological treatments

• Quality tools

• Reporting guidelines

Different types of NPT

- surgery

- technical operation (arthroscopy,…)

- rehabilitation

- education

- orthopedic braces or orthosis

- laser treatment

Participative interventions

Non implantable devicesTherapist-dependent interventions

Non-pharmacological treatments (NPT) take in many different treatments

5

Assessing Nonpharmacological treatments

• Randomized controlled trials (RCT)– Gold standard of therapeutic evaluation

• The design, conduct and reporting of RCTs aim at providing valid results

• Standard of the methodology of RCTs has been developed in the context of pharmacological treatments (PT)

• Specific methodological issues when assessing NPTs– Blinding– Placebo– Complexity of treatments– Care providers’ influence

6

Are NPT trials specific?Quality

A. PT (n=60) versus NPT (n=50)

B. Comparison

(1) Oral drug, intra-venous, intra-muscular (n = 46)

(2) Intra-articular injection (n = 14)

(3) Surgery, arthroscopy, joint lavage (n = 23)

(4) Rehabilitation, acupuncture, education(n = 27)

Quality of trials assessing PT and NPT of hip and knee osteoarthritis publishedin high impact factor journals (1992-2002)*

* Boutron, Tubach, Giraudeau, Ravaud Jama, 2003

scale

A B

- 1 0 1 2 3 4 5 6

Jadad - 1 0 1 2 3 4 5 6

Jadad scale

P T NPT

1

2 3 4

7

Are NPT trials specific?The feasibility of blinding

Feasibility of blinding patients

96

2

48

2

10

42

0%

20%

40%

60%

80%

100%

NPT (n = 50) PT (n = 60)

%

Possible

Difficult

Impossible

Feasibility of blinding care providers

9686

2

2

2

12

0%

20%

40%

60%

80%

100%

NPT (n = 50) PT (n = 60)

%

Possible

Difficult

Impossible

Feasibility of blinding outcome assessors

98

502

16

34

0%

20%

40%

60%

80%

100%

NPT (n = 50) PT (n = 60)

%

Possible

Difficult

Impossible

* Boutron I, Tubach F, Giraudeau B, Ravaud P, JCE, 2003

8

Are NPT trials specific?The success of blinding

Risk of unblinding patients

27

14

6586

80%

20%

40%

60%

80%

100%

NPT (n = 26) PT (n = 59)

%

Null

Moderate

Important

Risk of unblinding care providers

5719

43

68

130%

20%

40%

60%

80%

100%

NPT (n = 7) PT (n = 59)

%

Null

Moderate

Important

Risk of unblinding outcome assessors

8 10

36

90

56

0%

20%

40%

60%

80%

100%

NPT (n = 25) PT (n = 60)

%

Null

Moderate

Important

9

Are NPT trials specific? Placebo

PT Surgery Rehabilitation, education

Placebo Matching placebo

Simulated procedure Sham intervention

Example Saline solution in identical syringes

Sham arthroscopy* Sham education program with similar frequency

and duration of sessions but with

different content **

Issue - Ethical issue False negative

* Moseley et al., NEJM 2002

**Edworthy et al. J Rheumatol 1999

10

Are NPT trials specific?Placebo

Paterson, C. et al. BMJ 2005

Characteristic element(e.g., content of the education program)

Incidental element (e.g., healthcare providers – patients’ interaction effect)

Incidental element (placebo effect)

Effect not related to treatment (e.g., natural course of the disease, regression to mean )

Real treatment

effect

Measured treatment

effect

Characteristic element(e.g., content of the education program)

Incidental element (e.g., healthcare providers – patients’ interaction effect)

Incidental element (placebo effect)

Effect not related to treatment (e.g., natural course of the disease, regression to mean )

Real treatment

effect

Measured treatment

effect

11

Are NPT trials specific? Placebo

• Externally clearly different placebo

– Physiotherapy for knee OA* • Placebo treatment consisted of sham

ultrasonography

– Assessment of lifestyle advice**• Placebo treatment consisted of a syrup placebo

*Bennell, Ann Rheum Dis, 2005**Spigt, JCE, 2005

12

Are NPT trials specific? The methods of blinding – Full blinding

• Full blinding attempted • externally identical placebo/control

• Unblinded care providers not involved in the subsequent patient care

• Example– Transplantation of embryonic dopamine neurons**

• four twist-drill holes made through the frontal bone after local anesthesia

• the dura mater was not penetrated

* Boutron, Ravaud, Submitted** Freed, NEJM, 2001

Study cohort: 145 non-pharmacological trials (2004)*

13

Are Non pharmacological trials specific? The methods of blinding – Partial blinding

• Partial blinding attempted (blind trial hypothesis)

– Externally clearly different placebo / control

– Manipulation of information • Patients not informed of the existence of a placebo

• Patients not aware of the nature of the placebo

• Confidence in treatments sometimes tested

14

Are Non pharmacological trials specific? The methods of blinding – Partial blinding

• Zelen design or modified zelen design– Usual care compared to a complex, physical

therapy-based intervention for patello-femoral joint osteoarthritis of the knee

• 1) Researchers invited patients to participate in a cohort.

• 2) Randomization

• 3) Patients randomized to the intervention arm informed that they would receive the experimental treatment and signed a second consent form

Quilty, J Rheumatol. 2003

15

Are Non pharmacological trials specific? The methods of blinding – Outcome assessors• Blinding of outcome assessors

– Centralized assessment (video, audiotape or photography )

– Outcome assessors blinded of study hypothesis• To assess the analgesic effect of breast feeding in term neonates

compared to mothers’ arms, pacifiers, placebo (i.e., sterile water) or glucose

– Videotape of the children during the painful procedure– Two specially trained observers independently assessed the recordings using

a specific scale. – Observers were blinded to the purpose of the study and had been told that

they were assessing agreement of their scores in different situations

– Patients reported outcomes• Blinding is impossible if patients cannot be blinded

Carbajal R, Bmj. 2003

16

Are NPT trials specific? Complex intervention

• Several components– Rehabilitation : exercises, drugs, education etc

• Description of the intervention– Quantitative data

• Number of sessions, timing of each session, duration of each session

– Qualitative data• Content of each session, how it is delivered, supervision, content of

information exchanged etc

• Standardization procedure– Specific training– Quality control procedures

• Potential gap between the intended intervention (as described in the protocol ) and the actual administered intervention

• Barrier for systematic reviews

17

Are NPT trials specific? Influence of care providers

• Systematic review of the surgical and medical volume-outcome literature (Jan 1, 1980 - Dec 31, 2000)*– 71% of hospital volume-outcomes studies positive (88/124)

– 70% of physician volume-outcomes studies positive (31/44)

– No studies showed the opposite relationship

– Relationship strongest for high risk/rare surgeries:• Pancreas/esophagus cancer, pediatric cardiac surgery• NNT at high v. low volume provider: 7 to 11

– Much more modest volume-outcome effect for common procedures (CABG, CEA, PTCA, breast/colon cancer)

• NNT: 62 to 500

*Halm et al, Ann Intern Med 2002

18

Are NPT trials specific?

• Specific issues in assessing NPT – Blinding– Placebo– Complexity of treatments– Healthcare providers’ influence

• Need of specific standards

19

Overview

• Specific methodological issues when assessing Nonpharmacological treatment

• Quality tools

• Reporting guidelines

20

Quality toolsWhy?

• Assessing the quality of reports of trials is particularly important – clinicians’ critical appraisal of healthcare

literature

– systematic reviews

• Accurate estimates of the treatment effect

21

Quality toolsWhy?

• Several quality tools• Few quality tools were developped according to

scientific standards– Jadad scale*– Delphi list**

• Validated quality tools were mainly developped in the context of PT– Importance of blinding– No item related to care providers and the complexity

of the intervention

** Jadad, Control Clin Trials, 1996* Verhagen, JCE, 1998

22

Development of a specific checklistThe example of CLEAR NPT

• Quality: Internal validity• Delphi consensus method• Selection of items

• From existing checklist or scales• From specific items• From the Collaborative Review Groups of the Cochrane

Collaboration recommandations• Interviews of clinicians

• Experts• Members of Collaborative Review Groups of the Cochrane

Collaboration (n=41)• Clinicians involved in RCTs assessing NPTs (n=58)• Methodologists, epidemiologists (n=55)

– 55 experts participated(36%)

Boutron & Ravaud, JCE, 2005

23

Decision of the steering comitteeSelection of items

with a score of 8/9for more than 40% of experts

Decision of the steering comitteeSelection of items

With a score of 9/9for more than 40% of experts

Decision of the steering comitteeSélection of all items

Addition of 2 items for randomisationAddition of 1 item for the intention-to-treat analyses

Final Checklist10 items

5 sub items

Checklist11 items

Checklist21 items

Initial checklist38 items

1st Round54 experts/55

2nd Round46 experts/55

3rd Round49 experts/55

24

CLEAR NPT

• Was the generation of allocation sequences adequate?• Was the treatment allocation concealed?• Were the main outcomes analyzed according to the

intention-to-treat principle?

• Were details of the intervention administered to each group made available?

• Were care providers’ experience or skill in each arm appropriate?

• Was participant (ie, patients) adherence assessed quantitatively?

• Was the follow-up schedule the same in each group?

25

CLEAR NPT

• Were participants adequately blinded?

• Were care providers or persons caring for the participants adequately blinded?

– If care providers and or participants were not adequately blinded• Were all other treatments and care (ie, co-interventions) the same in each

randomized group?

• Were withdrawals and lost to follow-up the same in each randomized group?

• Were outcome assessors adequately blinded to assess the primary outcomes? – If outcome assessors were not adequately blinded,

• Were specific methods used to avoid ascertainment bias (systematic differences in outcome assessment)

26

CLEAR NPT

• Adequate assessment of trials quality

• Improvement of critical appraisal of medical litterature

• Improvement of the quality of systematic reviews

27

Overview

• Specific methodological issues when assessing Nonpharmacological treatment

• Quality tools

• Reporting guidelines

28

Reporting guidelinesWhy?

• Critical appraisal of the quality of clinical trials is possible only if the design, conduct, and analysis of RCTs are thoroughly and accurately described in published articles

• Evidence of incomplete and inadequate reporting

29

Reporting guidelines

30

31

Reporting guidelines

32

Reporting guidelinesDissemination

• Publication– 11 journals– Over than 1000 citations

• Editors endorsements including the ICMJE (International Committee for Medical Journal Editors)

– instructions to authors– submission check-list

• Website– www.consort-statement.org

33

Reporting guidelinesDissemination

• Various extensions of the CONSORT Statement since the original version (parallel arm, superiority, efficacy)

• Cluster RCT

• Harm

• Herbal therapy

• Non inferiority trials

• Several other extensions are in progress• 2x2 factorial design

34

Reporting guidelines for NPT trials Why?

• 119 RCTs published in 1998 and 1999 in three major cardiothoracic journals (Annals of Thoracic Surgery, European Journal of Cardio-thoracic Surgery, The Journal of Thoracic and Cardiovascular Surgery)

• 70 % of RCTs fulfilled less than half of the CONSORT criteria

Anyannu, Eur J of Cardio-thoracic Surgery, 2004

35

Reporting guidelines for NPT trials Why?

NPT PT

Blinding of patients 46% 98%

Blinding of healthcare providers 43% 83%

Blinding of outcome assessors 72% 98%

Study of trials assessing PT and NPT of hip and knee osteoarthritis (1992-2002)

When blinding was judge possible, blinding was less often

reported in NPT trials

* Boutron & Ravaud, Jama, 2003

36

Reporting guidelines for NPT trials Why?

Study of surgical papers (n = 158) published in 2004*

• Reporting of details of the intended intervention– Surgical procedure 87 %– Pre-operative care 15 %– Anesthesia 35 %– Post-operative care 49 %

• Surgeons– Selection criteria for surgeon 40 %– Number of interventions performed

by surgeon 11 %– Number of surgeons involved 33 %

* Jacquier I, Boutron I, Ravaud P, Ann. Surgery

37

Reporting guidelines for NPT trials Why?

Study of rehabilitation papers (n = 171) published between 1997-1998*

• Timing of the intervention 68 %

• Description of the intervention 50 %

• Intervention’s adherence 34%

* Dijkers at al. Am J Phys Med Rehabil, 2002.

38

Reporting guidelinesDevelopment of the extention of the CONSORT to NPT

Steering committee– D. Moher– P. Ravaud– I. Boutron

Selection of experts– Editors– Clinicians– Methodologists

Two stages method– Preliminary survey

– Consensus meeting

39

Preliminary survey

• Objective– To identify items to be discussed based the

experts’ opinion

• Methods– All items of the CONSORT checklist– 7 specific questions on key issues were added– Items were selected for discussion if more than

1/3 of the experts answered that the item should be modified or another item should be added

40

Consensus meeting

• 3 days– February 8-10th 2006, Paris

• 30 experts

• Agenda– Presentations– Discussions– Consensus

• Manuscript in process

41

Conclusions

• Specific issues in assessing NPT– Placebo– Blinding– Complexity of the intervention– Care providers’ influence

• Specific guidelines– Specific quality tools

• CLEAR NPT

– Specific reporting guidelines• Development of an extension of the CONSORT for

NonPharmacological treatment (CONSORT meeting, Paris, 2006)