Critical Reading of Medical Articles

VOLGOGRAD - October 2011

Pr. Pierre-Emmanuel FalcozStrasbourg University Hospital

2

After all, WHY?

Notion of « Evidence Based Medicine » It is not possible to read all the medical literature We must learn to retain ONLY good publications

that will modify our medical behavior when faced with a given situation

3

Which sources of information?

The « reference » book The « expert » The medical literature

– google: « pubmed »– medline : 20-30 000 articles/month

4

Quality criteria of an article

The structure of the article The type of study The methodological quality (internal validity) The practical feasibility (extrapolation)

5

The structure of the article

The title:– Interesting, useful and linked to our question

The abstract: – informative– IMRaD structure– One introductory sentence: the objective– 2 or 3 sentences regarding material and methods:

appropriate to the question raised– 3 or 4 sentences illustrating the results: accurate and

relevant– One concluding sentence

6

Introduction – The justification – The objective

Material and methods– The patient(s) (inclusion and exclusion criteria)– Description of the method– End point and outcome measure– The plan of the analysis

The structure of the article

7

Results– Description of the included patient(s)– They should correspond to patients announced in

the material and methods paragraph– Data should be clear, precise and pertinent– Calculation should be easily tested

Discussion: comparison of the results with those of other articles

Written paragraphs should be well differentiated

do not mix sub-headings!!!

The structure of the article

8

The different types of studies

Therapeutic studies Diagnostic studies Prognostic studies Epidemiologic studies Economic studies

9

The randomized controlled trials

« The gold standard » enables one to answer the question asked

Only way to obtain reliable proof of the effectiveness of a given treatment

The other types of studies: – Non-randomized prospective trials, retrospective

studies, e.g. …– Lower level of scientific proof

10

A short story...

After myocardial infarction, the existence of ventricular extra systoles (VES) increased the risk of sudden death

Class 1 anti-arrhythmics erase VES From this data, such treatments have been given to prevent

sudden death… Without any trials of clinical criteria…

11

CAST study (1991)

Death / n mortalityAnti-arrhythmic group 39 / 432 9%Control group 18 / 423 4%

RR=2.13, p<0.05

80 000 Deaths induce by the treatment in the USA– More than Vietnam War (Moore)

12

Anti-arrhythmics in post myocardial infarction period

Ventricularextrasystoles Sudden death

Flecaine

Proven

Prov

en

Not confirmed in real settings

by actual findings

Speculative

13

Comparative trial

No treatment

Treatment studied

10 %

12 %

Difference = effect of the treatment

End point5-y mortality

14

The quantification of scientific proof (ACCP)

Level 1 : •Randomized trials with definitive (unquestionable) results•Meta-analyses

Level 2 :•Randomized trials (low power)•Non randomized controlled trials (well designed)

Level 3 :•Non controlled prospective trials (well designed)

Level 4 : •Controlled trials with bias•Comparative trials (historical series)

Level 5 :•Retrospectives studies or case-series

15

The bias

Definition: there is a bias when the difference observed between two groups at the conclusion of the trial is due to a factor other than the studied treatment itself

Population

Randomization

Experimental group Control group

Treated Untreated

Follow-up

Results

Selection bias

Execution bias

Exclusion bias

Detection bias

Follow-up

Results

16

The different biases – a summary

Selection bias– Difference in the base prognosis of the patient(s)

Execution bias– difference in follow-up and care given to the patient(s)

Exclusion bias– difference with regard to the “departure” from the studies

Evaluation bias (measure)– difference in evaluation of the end-point

17

Comparable groups

Similar groups– Same type(s) of patient(s)– Same stage of the disease, etc…

That are different ONLY due to the applied treatment If a difference exists, it is ONLY due to the treatment

Grp T

Grp C

Treatment

18

Composition of groups

The distribution should not depend on:– the patient– the disease– the physician– the medicine

By random drawing of lots:– random allocation– assures that , on average, the 2 groups are strictly comparable

19

Poor control groups

Historical controls– For example, patients treated 5 years ago– Those patients are not comparable to those currently

treated

Geographical control– patients of another department

20

Statistical reality of the results

An observed difference between 2 groups could be either real or due to chance

Statistical tests allow the statistical reality of the difference to be assessed

The alpha risk is the risk of wrongly drawing a conclusion regarding the effectiveness of a treatment, when in reality this difference is due to chance

this risk is set up to 5% The danger would be to to increase the alpha risk and thus

wrongly conclude that a difference exists between the 2 groups

21

Statistical test: interpretation

Observeddifference Test

It is unlikely that the observed difference would be due to chanceSignificant difference

The probability that the observed difference would be due to chance is highNon significant difference

P>5%

P<5%

22

Primary endpoint

• Death from all causes

Secondary endpoint

• Cardiovascular death

• Sudden death

• Myocardial infarction

• Cerebrovascular accident

• Surgery

• Death (all causes)

• Cardiovascular death

• Sudden death

• Myocardial infarction

• Cerebrovascular accident

• Surgery

No definition of the primary endpoint

Risk of wrongly drawing a conclusion regarding the

effectiveness of the treatment = 30%

6 statistical tests

Risk of wrongly drawing a conclusion regarding the

effectiveness of the treatment = 5%

A priori definition of a primary endpoint

1 single statistical test

Endpoint

23

Test 1

Risk to wrongly conclude to a

difference = 5%

Test 2

Risk to wrongly conclude to a

difference = 5%

Test 4

Risk to wrongly conclude to a

difference = 5%

Test 3

Risk to wrongly conclude to a

difference = 5%

Overall, the risk of wrongly drawing a conclusion regarding a difference when utilizing these four comparisons is much greater than 5%

n overall risk

2 0.10

3 0.13

5 0.23

10 0.40

At dice game, the probablity to obtain one SIX is higher with 3 dices than with only one

Multiple comparisons

Inflation of the alpha risk

24

The methodological value of the result

The trial respects the experimental design A hypothesis formulated A PRIORI The entire experimental plan is devised A PRIORI

– the primary endpoint– the calculation of the number of required subjects– an analysis planned

The results should correspond to the formulated hypothesis One cannot draw a conclusion in terms of causality if a result

is not derived from this approach– does not concretely demonstrate anything– at best it suggests an effect

25

The clinical relevance

It allows to ascertain that:– The benefit of the treatment is sufficiently important– The clinical criteria is relevant– The ratio benefit/risk is acceptable– The treatment is applicable in daily practice

26

The methodological relevance

The main objective is clearly formulated The therapeutic methods are describe with precision:

– the comparison treatment is adapted– the evaluated treatment is applicable in daily practice

The primary endpoint:– Clinical, relevant, validated

The patient(s) included in the trial is/are representative:– the studied population resembles patients seen in daily

practice

27

The importance of the therapeutic effect

The fact that the trial is significant (p<5%) is not good enough!!!!

If the therapeutic effect is minimal, it might be of little or no interest to our daily practice…

28

The accuracy of the therapeutic effect

The confidence interval Indicates the accuracy of the estimation of the size of the

therapeutic effect A 95% confidence interval assess the range of values within

which we are 95% certain to find the true sought value

29

95% confidence interval

0.4 0.6 0.8 1 1.2 1.4

Trial A

Trial B

Trial C

Relative Risk

30

The quantification of the therapeutic effect

It allows me to know how much my chances of improvement could be increased if I use this therapeutic !

The relative risk (RR) or the odds ratio (OR) The relative risk reduction (RRR) The absolute risk reduction (ARR) The number needed to treat (NNT)

31

The definitions

Relative risk: risk of treated patients in relation to the risk of the control

Relative risk reduction: is the amplitude of the risk reduction Absolute risk reduction: is the absolute value of the risk

reduction The number needed to treat: is the number of patients we

must treat to avoid, on average, one event

32

Number needed to treat (NNT)

N. Engl J Med 2004

Surgery Surgery +CT

N.N.T.

2 years 67% 70% 26

5 years 40% 44.5% 18

Interpretation:

At 2 years: we must treat 26 patients by chemotherapy to avoid 1 death !!!

33

The benefit / risk ratio

The decision should weigh the beneficial effects against the adverse events– Are the adverse events acceptable?– Do the adverse events offset the totality of the benefit?

The benefit / risk ratio is unfavorable in the following situations:– the seriousness of the adverse events represents an

unacceptable risk with regard to the seriousness of the disease

– the frequency of the adverse events is high– the new therapeutics are less well tolerated than the

available therapeutics

34

Practical guide

Evaluation of the internal validity (methodological quality)– the research of bias– the statistical reality of the result– the methodological value of the result

The clinical relevance and the use in daily practice– the objective of the trial– the relevance of the main end point– the representativity of the included patient(s)– the feasibleness of the therapeutic– the size of the therapeutic effect– the benefit / risk ratio

Thank you for your attention!35

The end…