making sense of results - a workshop for healthcare librarians

Making sense of results- a workshop for healthcare librarians

Dr Amanda Burls2nd UK Clinical Librarian Conference

Objectives

• To look at how results can be presented• To understand what a meta-analysis is• To be able to interpret a “blobbogram”• To be able to make sense of tests for “statistical significance” • To explore how uncertainty in results can be summarised and

understand:• P-values• Confidence intervals

• To have fun!

Making sense of results

• How are results summarised?

How many of you have attended a critical appraisal skills workshop?

• What sort of study design were you appraising?

• What are the key things you remember?

Critical appraisal of any study design must consider

• Validity– Can the study (results) be trusted?

• ResultsResults– What are the results and how are they (or can they

be) expressed?

• Relevance– Do these results apply to the local context?

Warning!• Everything I say from now onwards assumes

that the results being considered come from an unbiased study!

How are results summarised?

• Most useful studies compare at least two alternatives.

• How can the results of such comparisons be expressed?

Expressing results:What did the study show?

• Patients with backache:– 100 randomised to receive a firm mattress– 100 randomised to receive a medium mattress

• After 3 months:– 80 get better in the firm mattress group– 20 get better in the medium mattress group

• How would you summarise this for a friend?

Summarise

• 80 out of 100 (80%) better in firm mattress group

• 20 out of 100 (20%) better in the medium mattress group

• 4 times as likely to get better with a firm mattress

• An extra 60% of people get better with a firm mattress

How were the results summarised?

• There are two basic ways to summarise results of studies that compare two or more groups:

1. Difference (take them away)

2. Ratio (divide)

The blobbogram!

Blobbogram

Line of no differencebetween treatments

less more

Blobbogram - Difference (taking away)


less more0

Blobbogram - ratio (dividing)


less more1

A randomised placebo-controlled trial

Well conducted RCT – no bias

• Five people with backache received Potters

• Five people received placebo

• 4 out of 5 with Potters got better

• 2 out of 5 with placebo got better

<0.00010.010.030.090.29p-value

0.27 to 0.520.10 to 0.640.04 to 0.67-0.04 to 0.71-0.23 to 0.7995% CI

0.40.40.40.40.4Proportion responding in control arm

408642Responders in control arm

1002015105Number in control arm

0.80.80.80.80.8Proportion responding in treatment arm

80161284Responders in treatment arm

1002015105Number in treatment arm

<0.00010.010.030.090.29p-value

0.27 to 0.520.10 to 0.640.04 to 0.67-0.04 to 0.71-0.23 to 0.7995% CI

0.40.40.40.40.4Proportion responding in control arm

408642Responders in control arm

1002015105Number in control arm

0.80.80.80.80.8Proportion responding in treatment arm

80161284Responders in treatment arm

1002015105Number in treatment arm

No backache at 3 months(Results of our Potters tablet

versus placebo trial) Potters PlaceboPotters Placebo

Favours placebo Favours PottersFavours placebo Favours Potters

No backache at 3 monthsDo you think this study proves Potters works?

Potters PlaceboPotters Placebo


It could be due to chance!

• What if there had 1000 people in each arm and 800 got better with Potters and only 200 got better on placebo?

• Would you believe Potters works now?

• So how many people would you want in each arm to believe the trial?

P-value in a nutshell

The Null Hypothesis

0 1

Impossible Absolutely certain

• p = 0.5

quite likely - evens chance - 50:50 - 1 in 2

• p = 0.001

very unlikely - 1 in 1000• p = 0.01

unlikely - 1 in 100• p = 0.05

fairly unlikely - 1 in 20 - 5 times in 100

Odds ratio (12b)

0

10

20

30

40

50

60

70

80

90

100

Pre and Post Workshop Scores

Perc

enta

ge 5

4

3

2

1

MAAG (9b)

0

10

20

30

40

50

60

70

80

90

100

Pre and Post Workshop Scores

Perc

enta

ge 5

4

3

2

1

Moral:

Any observed difference between two groups, no matter how small, can be made to be “statistically

significant” - at any level of significance - by taking a sufficiently large sample.

• Question: How can we express uncertainty due to chance?

• Answer: the p-value

• But is there a better answer?

Introduction to confidence intervals

• CIs are a way of showing the uncertainty surrounding our point estimate.

No backache at 3 months(Results of our Potters tablet

versus placebo trial) Potters PlaceboPotters Placebo


Hypothermia vs. control In severe head injury

Mortality or incapacity (n=158)

RR 0.63 (0.46, 0.87)Marion 1997

.1 .2 1 5 10

Total (95%CI)

Clifton 1992

Hirayama 1994

Clifton 1993

RR



RR 0.63 (0.46, 0.87)Marion 1997

.1 .2 1 5 10

Total (95%CI)

Clifton 1992

Hirayama 1994

Clifton 1993

Favours intervention RR Favours control



RR 0.63 (0.46, 0.87)

Marion 1997

.1 .2 1 5 10

Total (95%CI)

Clifton 1992

Hirayama 1994

Clifton 1993

Favours intervention RR Favours control

making sense of results - a workshop for healthcare librarians

Documents