Objectives

Identify the key elements of a good randomised controlled study

To clarify the process of meta analysis and developing a systematic review

To use this to critically evaluate a Cochrane review

Answer a clinically relevant problem using information from a systematic review

RCT – key issues• Allocation – method of randomisation

(to minimise confounding)• Blinding – patient, clinician and study

personnel

(to minimise observer and recall bias)• Follow-up (adequate; complete; differences

between those followed up and those lost to follow-up)

• Intention to treat analysis

Checklist for appraising RCTs• Was assignment of patients to treatment

randomised?• Was randomisation concealed?• Were the groups similar at the start of the

trial?• Was follow-up of patients sufficiently long and

complete?• Were all patients analysed in the groups to

which they were randomised (intention to treat)?

• Were patients, clinicians and study personnel kept blind to treatment?

• Were groups treated equally, apart from the experimental therapy?

Overview of a systematic review• ASK a clear and focussed question

– How easy is this?

• SEARCH for relevant studies– ALL studies, anywhere, ever? Does systematic

have to be exhaustive?

• APPRAISE the quality and relevance– Structure depends on the type of studies

• SYNTHESISE– Meta-analysis or narrative synthesis

• INTERPRET– In relation to the question asked (policy or clinical)

Correspondingly… critical appraisal

• Did the review ask a clearly focussed question?

• Are the findings valid?– Did they find all the relevant studies?– Was quality appraised?– Were the study findings combined appropriately?– Were the study results consistent?

• What are the findings?• Will the findings help me with my patient (or

policy problem)?

Finding studies…• Electronic databases

– General• Medline• Embase

– Methodologically specific• Cochrane Library (Reviews and CCTR)• DARE, HTA database, INAHTA, NHS EED

– Disease area specifics• Psychlit• Cancerlit

– “Grey literature”

• Hand searching• Citation searching

Getting the right studies…

• PUBLICATION BIAS: Systematic exclusion of studies with (usually) negative findings

• DUPLICATE PUBLICATION BIAS: Multiple publications of positive trials leads to double-counting in reviews

• CITATION BIAS: Positive studies are cited more frequently

• LANGUAGE BIAS: English and positive results

• TIME LAG BIAS: Negative studies take longer to be published

• RETRIEVAL BIAS: Method of searching produces a systematic error – high impact journals are more likely to report positive findings and to be listed in databases

Assessing publication (and similar) bias(es)

• Were the searches comprehensive?

• Evidence of “missing studies”

• Look at the FUNNEL PLOT

Study size

Effectiveness

Treatment better Control better

Biased pooled effect size

Critical appraisal• Not going to cover in detail

• Some explicit method is required– General “systematicity”

• Transparency• Reducing errors and bias

– Lots of issues!• One or two reviewers? • Measure agreement?• Scores or not?

• A tension:– Identify all studies? But…– Include only the best quality??

• Methodological quality is important as possible reason for differences between studies

Odds ratio 0.1 1 10

Study Odds ratio (95% CI)

0.96 (0.58,1.59) Cooper

1.47 (0.92,2.34) Microsulis

0.56 (0.30,1.04) Meyer

1.56 (0.24,9.91) Romer

0.58 (0.18,1.93) Soysal

Favours treatment Favours control

Findings of reviewForest Plot of Amenorrhoea

following Microwave Endometrial Ablation

Heterogeneity• Are the studies all measuring the

same thing?

• If not, then the pooled estimate may not be informative

• Heterogeneity can due to a variety of reasons

Sources of heterogeneity

• Chance (i.e. sampling error)

• Clinical– Different doses or duration of treatment or

co-interventions– Different populations (i.e. risk of events)– Different outcome measures

• Methodological– Blinding– Methods of analysis– Publication bias

Statistical significance and heterogeneity

• Studies can be tested statistically for heterogeneity to see if results are significantly different

• P value of 0.05 or lesser indicates statistical significance for heterogeneity

• If significant for heterogeneity, then need to consider whether results can be pooled

Summary of critical appraisal of a systematic review

• Was there a clear question?• Did they find all the relevant studies?• Was the quality of the studies considered?

Was the quality so poor that the review is suspect?

• What are the results?• Is there important heterogeneity? What does it

mean?• What are the implications of the review findings

for my clinical or policy question?

Are they good questions?

Appropriate to compare with non opiate analgesics or sedatives?

Outcome measures appropriate? Which outcomes most relevant (in

order)? Which outcomes easiest to measure? Any difficulties with some of these

outcomes?

Appropriate search strategy?

Were the Cochrane neonatal groupguidelines followed in relation to

Blinding of randomisation Blinding of intervention Completeness of follow up Blinding of outcome measurement –

see handout

Appropriate search strategy?

What is a quasi randomised study? What is Cochrane’s central register of

controlled studies? Were non English studies included? Was there risk of publication bias? How

to overcome this. Was more than 1 reviewer involved?

How were differences resolved?

Data Analysis/ Results

Look at meta-analysis of PIPP scores (figure 01.01) Do you think the all studies result shows a satisfactory forest plot?

What is your interpretation of the results of the other methods of assessing pain (figure 01.04)

What is the relevance of a reduction of 2 in the PIPP? (see handout of PIPP)

What are the differences in execution and reporting of trials which authors say will interfere with applying this?

Data Analysis/ Results

What do they mean that heterogeneity was high in all analyses of pains?

What are sources of heterogeneity for PIPP?

What are the other sources of heterogeneity?

Do you think there is sufficient evidence to recommend morphine over midazolam for sedation?

Application

How important are the results?

How applicable are the results?

What do you do?