Critical Appraisal of Systematic Reviews

Douglas Newberry

Systematic Reviews —

or How to make a Monkey out of EBM without hardly

trying!

Systematic Reviews:Objectives:

• Appraise a systematic review for validity

• Discuss Meta Analysis / use Odds Ratios

• Obtain Number Needed to Treat (NNT) from Odds Ratios

• Consider clinical implications of a Systematic Review {including when to bin it instead!}

We can see further than our forbearers because we stand on

the shoulders of Giants{and have better spectacles}

• these ideas are cribbed unashamedly from friends, books & previous courses

Systematic Reviews:What are your Objectives:

What do you want to cover?

Please interject with helpful questions!

Did I really want a systematic review?

(but please do not pretend)

• admit your ignorance — expert review or consensus guidelines > broad introduction, cover many areas (class C evidence).

• if the question is important > formulate it!

• Systematic review > narrow but rigorous focus.

Systematic Reviews — Where do I start:

• Start with your 4 (or 3) part clinical question!

• Is a systematic review a sensible approach?

• Does THIS systematic review address MY question?

• Is it a systematic review at all?

Is it a systematic review? does it:• define a four part (answerable) clinical

question?

• combine Randomized Controlled Trials (RCT’s)?

• describe PRE-DEFINED search methods?

• PRE-DEFINED inclusion criteria?

• PRE-DEFINED methodological exclusion criteria?

Sceptical View? Take it with a grain of salt:

• transparent declaration of funding of work?

• Drug Company sponsorship of Reviews vs. Methodological quality>Cochrane review!

• who employs the authors?

• open discussion of existing controversy & commercial gain?

• Don’t waste salt on your food, keep it for your reading!

Meta analysis — combine what with what?

• Low Molecular Weight Heparin (LMWH) in hip surgery — begin before or after the operation?

• meta analysis of placebo controlled RCT’s of heparin in hip surgery >>

• pre-op & post-op LMWH vs. placebo

• post-op LMWH Vs placebo

• pre-operative >> less intra-op bleeding??

Can we believe it ?

• bias free search & inclusion criteria?

• appraisal of methodology of primary studies?

• consistent results from all primary studies?

– if not, are the differences sensibly explained?

• are the conclusions supported by the data?

If we believe it — does it apply to our patient?

• Is our patient (or population) so different from those in the primary studies that the results may not apply?

• consider differences in:

– time — many things change.

– culture — both treatments and values of outcomes can be different

– stage of illness or prevalence can effect results.

We believe it ! but—>> does it matter?

• Is the benefit worthwhile to our patient?

• Ask the patient about cultural values.

• Think about Relative Risk Reduction vs. Absolute Risk to our patient.

• Potential benefit is the Absolute risk avoided in our patient = Absolute Risk Reduction (ARR)!

Absolute Risk—> The risk our patient is facing!

• How likely is our patient to die (or have the outcome of interest) without intervention? = Control Event Rate (CER)

• consider this individual patient’s risk factors to estimate Patient Expected Event Rate = PEER.

• Absolute Risk usually increases with age.

• Improvement measured as Absolute Risk Reduction (ARR)

Relative Risk Reduction:• Usually reported in studies.

• Ratio of the improvement of outcome over outcome without intervention (Rx):

• {Control Event Rate (CER) — Experimental Event Rate (EER)} / CER

• i.e. {CER-EER}/CER

• often independent of prevalence!

• often similar at different ages!

Our patient wants an absolute Risk Reduction (ARR):

• is a 40% reduction in Cardiac Risk worth taking pills daily for 10 years?? >vote!

• if I have a 30% risk of MI or death {30 out of 100 people like me will suffer MI or death} in next 10 years > 40% RRR >> only 18 out of 100 will have MI or death. ARR = 12 out of 100! >>I like that!

• BUT if I have a 1% risk in 10 years, 40% less is a 0.6% risk >> hardly different!

Number Needed to Treat (NNT) (very trendy but tricky):

• only defined for specific prevalence-Patient’s Expected Event Rate=PEER!

• only defined for a specific intervention!• only defined for a specific outcome!

– eg. Pravastatin™ 40 mg nocte x10 years, in a 65 year old male, ex-smoker with high BP and Diabetes, to reduce MI or Death.

• NNT is the inverse of Absolute Risk Reduction: i.e. NNT = 1/ARR

Number Needed to Treat (NNT) for previous example:

• 12 fewer MI or death in 10 years per 100 persons treated: ARR=12/100

• NNT = 1/(12/100)=100/12= 8.3

• But the same Relative Risk Reduction (RRR) of 40% with a low prevalence:

• 0.4 fewer MI/death per 100 treated, ARR=0.4/100.

• NNT = 1/(0.4/100) = 100/0.4 = 250!

Why Odds Ratios? > compare results of different studies.

• consider 2x2 table:

• RRR is (a-b/a) — but you can only go in rows within same study!

• Odds ratio is (a/c)/(b/d) = ad / bc — the individual ratios are in columns, and therefore are independent of the prevalence which is different in different studies.

• must use odds ratios to combine RCT’s

Odds Ratio (OR) to NNT — is the improvement worth the trouble?

• 1>OR>0, lower the OR = better the treatment (Rx) >> lower NNT.

• for any OR, NNT is lowest when PEER=0.5

• estimate the PEER (patient’s risk)

• apply the OR to get patient's NNT.

Convert PEER & OR to NNT:

Odds Ratio (OR)Control CER 0.9 0.7 0.5

Event 0.1 110 36 21Rate(CER) 0.5 38 11 6{applyPEER 0.9 101 27 12here}

Formula used in the table:

NNT=1- {PEER * (1-OR)}

(1-PEER)*(PEER)*(1-OR)

Table induced nausea!• lower OR >> lower NNT

• Patient needs to be at risk (non-trivial PEER) in order for risk reduction to be worth the effort.

• for any OR, NNT lowest when PEER=0.5

• more effective treatment > lower NNT

• BUT are your patient’s values satisfied by the intervention and its sequelae?

Subgroup analysis: Sceptical unless:• the subgroups make biological and clinical

sense?

• the differences are both clinically & statistically significant?

• was a-priori hypothesis (before this data)?

• other evidence supports these sub-groups?

• few (OK) or many (nix) sub-group analyses?

Any Questions?

Summary 1: Set your goals.• define your 4 (or 3) part question.

• do you want a true systematic review?

• does this narrow review address my question?

• PRE-DEFINED search, inclusion, exclusion!

Summary 2: Be Sceptical!• look for bias, conflict of interest.

• critical appraisal of primary studies?

• consistent results? if not, why not?

• does our patient fit the groups studied?

• does it matter to our patient?

Summary 3: Risks that matter.• Absolute risk > estimate the Patient

Expected Event Rate (PEER)

• obtain Relative Risk Reduction (RRR) or Odds Ratio (OR) from a Meta-analysis

• plug into a table to estimate Number Needed to Treat (NNT)

Summary 4: Sceptical & common sense!

• beware of post-hoc sub-group analysis, especially if multiple.

• step back and consider if the systematic review really related to our patient’s situation (PEER), culture and expectations?

• do not loose sight of common sense!

Coffee Now!

• Small Groups Afterwards