Why Grade Why Grade Recommendations?Recommendations?

• strong recommendationsstrong recommendations– strong methods strong methods – large precise effect large precise effect – few down sides of therapyfew down sides of therapy

• weak recommendationsweak recommendations– weak methodsweak methods– imprecise estimateimprecise estimate– small effectsmall effect– substantial down sidessubstantial down sides

Why Grade Recommendations?Why Grade Recommendations?

• strong recommendationstrong recommendation– one size fits allone size fits all– expect uniform clinician behaviorexpect uniform clinician behavior

• weaker recommendation weaker recommendation – expect action to varyexpect action to vary

Grading recommendations Grading recommendations is not grading studiesis not grading studies

• developing recommendationsdeveloping recommendations– grade (assess) individual studiesgrade (assess) individual studies– grade (assess) overall evidencegrade (assess) overall evidence– grade recommendations that flow from evidencegrade recommendations that flow from evidence

• rating schemes for individual studiesrating schemes for individual studies– prognosis, diagnosis, managementprognosis, diagnosis, management

• recommendations focus on patient outcomerecommendations focus on patient outcome– showing a test, or prognostic tool, is accurate showing a test, or prognostic tool, is accurate

generally not enoughgenerally not enough– RCTs of screening testsRCTs of screening tests

An ideal grading systemAn ideal grading system

• simplesimple• clear and separate consideration of clear and separate consideration of

two issuestwo issues– methodological quality of evidencemethodological quality of evidence

• likelihood of biaslikelihood of bias

– trade-off between benefits and riskstrade-off between benefits and risks

Which grading system to Which grading system to use?use?

• many availablemany available– Australian National and MRCAustralian National and MRC– Oxford Center for Evidence-based MedicineOxford Center for Evidence-based Medicine– Scottish Intercollegiate Guidelines (SIGN)Scottish Intercollegiate Guidelines (SIGN)– US Preventative Services Task ForceUS Preventative Services Task Force– American Heart AssociationAmerican Heart Association

• conflict of interest (point of view)conflict of interest (point of view)– American College of Chest PhysiciansAmerican College of Chest Physicians– Cochrane-related group led by Andy OxmanCochrane-related group led by Andy Oxman

What makes weaker evidence?What makes weaker evidence?

• weaker study designweaker study design– basicbasic– detailed design and executiondetailed design and execution

• inconsistencyinconsistency

• indirectnessindirectness

• reporting biasreporting bias

Methodological QualityMethodological Quality

• study designstudy design– randomizationrandomization– quasi-randomizationquasi-randomization– observational studyobservational study

• detailed design and executiondetailed design and execution– concealmentconcealment– balance in known prognostic factorsbalance in known prognostic factors– intention to treat principle observedintention to treat principle observed– blindingblinding– completeness of follow-upcompleteness of follow-up

ConsistencyConsistency

• evidence weaker if results differevidence weaker if results differ

• if inconsistency, look for if inconsistency, look for explanationexplanation– patientspatients– interventionintervention– outcomeoutcome– methodsmethods

Relative Risk of Conversion to Sinus RhythmAmiodarone vs Placebo or Digoxin or CCB

Favours Control Favours Amiodarone

Cowan 1986 1.11 (0.78 to 1.58)Noc 1990 18.0 (1.17 to 276)

Capucci 1992 0.77 (0.37 to 1.62)Cochrane 1994 1.15 (0.91 to 1.44)Donovan 1995 1.05 (0.69 to 1.60)

Hou 1995 1.29 (0.97 to 1.72)Kondili 1995 1.33 (0.71 to 2.47)Galve 1996 1.13 (0.84 to 1.52)

Kontoyannis 1998 1.42 (1.08 to 1.85)Bellandi 1999 1.41 (1.15 to 1.72)

Cotter 1999 1.43 (1.15 to 1.80)Kochiadakis 1999 1.46 (1.19 to 1.78)

Bianconi 2000 2.04 (0.19 to 22.00)Galperin 2000 33.70 (2.08 to 546)

Hohnloser 2000 3.13 (1.50 to 6.70)Joseph 2000 1.32 (0.95 to 1.80)Natale 2000 5.12 (2.60 to 10.00)

Peukurinen 2000 2.45 (1.49 to 4.02)Vardas 2000 2.01 (1.55 to 2.60)Villani 2000 4.75 (1.60 to 14.00)

Cybulski 2001 1.87 (1.37 to 2.55)0.1 1 10 100

n = 83

n = 95

n = 203

n = 85

n = 120

n = 34

n = 24

n = 40

n = 30

n = 64

n = 39

n = 42

n = 100

n = 42n = 120

n = 100

n = 204

n = 75

n = 62

n = 208

n = 160

Relative Risk of Conversion to Sinus RhythmAmiodarone vs Placebo or Digoxin or CCB

Favours Control Favours Amiodarone

Bianconi 2000 2.04 (0.19 to 22.00)Galperin 2000 33.70 (2.08 to 546)

Hohnloser 2000 3.13 (1.50 to 6.70)Natale 2000 5.12 (2.60 to 10.00)Villani 2000 4.75 (1.60 to 14.00)

Pooled Estimate 4.33 (2.76 to 6.77)

Cowan 1986 1.11 (0.78 to 1.58)Noc 1990 18.0 (1.17 to 276)

Capucci 1992 0.77 (0.37 to 1.62)Cochrane 1994 1.15 (0.91 to 1.44)Donovan 1995 1.05 (0.69 to 1.60)

Hou 1995 1.29 (0.97 to 1.72)Kondili 1995 1.33 (0.71 to 2.47)Galve 1996 1.13 (0.84 to 1.52)

Kontoyannis 1998 1.42 (1.08 to 1.85)Bellandi 1999 1.41 (1.15 to 1.72)

Cotter 1999 1.43 (1.15 to 1.80)Kochiadakis 1999 1.46 (1.19 to 1.78)

Joseph 2000 1.32 (0.95 to 1.80)Peukurinen 2000 2.45 (1.49 to 4.02)

Vardas 2000 2.01 (1.55 to 2.60)Cybulski 2001 1.87 (1.37 to 2.55)

Pooled Estimate 1.40 (1.25 to 1.57)0.1 1 10 100

AF Duration > 48 hrs

AF Duration =/< 48 hrs

n = 83n = 95

n = 203n = 85

n = 120

n = 34

n = 24n = 40

n = 30n = 64n = 39

n = 42n = 100n = 42n = 120n = 100n = 204n = 75

n = 62

n = 208n = 160

Directness of EvidenceDirectness of Evidence

• indirect treatment comparisonsindirect treatment comparisons– interested in A versus Binterested in A versus B– have A versus C and B versus Chave A versus C and B versus C

• alendronate vs risedronatealendronate vs risedronate– both versus placebo, no head-to-headboth versus placebo, no head-to-head

Who are the patients? Who are the patients? If different, weaker evidence If different, weaker evidence

• patients meet trials’ eligibility criteria patients meet trials’ eligibility criteria

• minor question of directnessminor question of directness– slight age difference slight age difference – comorbidity comorbidity – racerace– closely related underlying condition closely related underlying condition

• valvular atrial fibrillationvalvular atrial fibrillation

• serious question about biologyserious question about biology– heart failure trials applicability to aortic stenosisheart failure trials applicability to aortic stenosis

Interventions:Interventions:if differ, weaker evidenceif differ, weaker evidence

• identical interventionidentical intervention– captopril 100 mgcaptopril 100 mg

• minor question of directnessminor question of directness– captopril, lower dosecaptopril, lower dose– other ACE inhibitorother ACE inhibitor

• serious question of biologyserious question of biology– angiotensin receptor blockerangiotensin receptor blocker

Outcome differs, Outcome differs, weaker evidenceweaker evidence

• outcomes of direct interestoutcomes of direct interest– survival of long durationsurvival of long duration– valid measure of HRQLvalid measure of HRQL

• minor question of directnessminor question of directness– follow-up shorter than idealfollow-up shorter than ideal– functional status measurefunctional status measure

• serious question of biology serious question of biology – short duration of follow-upshort duration of follow-up– laboratory exercise capacitylaboratory exercise capacity

Magnitude, Precision, Magnitude, Precision, Reporting BiasReporting Bias

• magnitude not generally part of qualitymagnitude not generally part of quality– but very large magnitude can upgradebut very large magnitude can upgrade

• precision not generally part of qualityprecision not generally part of quality– but sparse data can lower qualitybut sparse data can lower quality

• reporting biasreporting bias– high likelihood can lower qualityhigh likelihood can lower quality

Grading SystemGrading System

• high qualityhigh quality well done RCTwell done RCT

• intermediateintermediate quasi-RCTquasi-RCT

• lowlow well done observationalwell done observational

• insufficient insufficient anything elseanything else

Moving DownMoving Down

• study execution– serious flaws can lower by one level– fatal flaws can lower by two levels

• consistency– important inconsistency can lower by one level

• directness of evidence– some uncertainty re relevance lower by one level– major uncertainty re relevance lower by two levels

• selection bias – strong suspicion lower by 1 level

Moving UpMoving Up

• very strong association, up 2 levels– insulin in diabetic ketoacidosis

• strong, consistent association with no plausible confounders, up 2 levels – fluoride for preventing cavities

• strong association can move up 1 level– ? HRT ?

Risk/Benefit tradeoffRisk/Benefit tradeoff• aspirin after myocardial infarctionaspirin after myocardial infarction

– meta-analysis of high quality RCTsmeta-analysis of high quality RCTs– clear positive treatment effectclear positive treatment effect– unequivocal recommendation to treat, unequivocal recommendation to treat,

uniform practiceuniform practice

• accelerated TPA vs streptokinase after accelerated TPA vs streptokinase after MIMI– very large single trialvery large single trial– clear positive effectclear positive effect– uncertainty whether to treat, varying uncertainty whether to treat, varying

practicepractice

Risk/Benefit tradeoffRisk/Benefit tradeoff

• Aspirin after myocardial infarctionAspirin after myocardial infarction– 25% reduction in relative risk25% reduction in relative risk– side effects minimal, cost minimalside effects minimal, cost minimal– benefit obviously much greater than benefit obviously much greater than

risk/costrisk/cost

• TPA vs streptokinase after MITPA vs streptokinase after MI– 12% reduction in relative risk12% reduction in relative risk– increased rate of intracranial hemorrhageincreased rate of intracranial hemorrhage– large increase in costlarge increase in cost– benefit vs risk/cost a judgment callbenefit vs risk/cost a judgment call

Strength of Strength of RecommendationsRecommendations

Aspirin after MI – do itAspirin after MI – do it

TPA rather than SK in MI TPA rather than SK in MI -- probably do it-- probably do it

-- probably don’t do it-- probably don’t do it

Grade of Grade of RecommendationsRecommendations

•do it or don’t do itdo it or don’t do it– strong recommendationstrong recommendation

•probably do it, or probably don’tprobably do it, or probably don’t– weaker recommendationweaker recommendation

Strong or Weak Recommendation?Strong or Weak Recommendation?

• seriousness of outcomeseriousness of outcome• magnitude of effectmagnitude of effect• precision of treatment effectprecision of treatment effect• risk of target eventrisk of target event• risk of adverse eventsrisk of adverse events• cost of therapycost of therapy• values and preferencesvalues and preferences

Weak recommendationWeak recommendation

• practice will vary practice will vary – according to what?according to what?

• interpretation of evidenceinterpretation of evidence– breast cancerbreast cancer

• patients’ values and preferencespatients’ values and preferences– atrial fibrillationatrial fibrillation

• inclination to gambleinclination to gamble– HRTHRT

ConclusionConclusion• challenges in gradingchallenges in grading

– simple systemsimple system– consider quality of evidenceconsider quality of evidence– consider benefit/risk tradeoffconsider benefit/risk tradeoff

• methodological quality of evidencemethodological quality of evidence– study design, execution, study design, execution, – consistency, directness consistency, directness – reporting biasreporting bias

• balance of benefits and risks/costbalance of benefits and risks/cost– magnitude and precision of effects; magnitude and precision of effects; – values and preferencesvalues and preferences

Way forwardWay forward

• an internationally accepted an internationally accepted grading scheme?grading scheme?

• reliability of grading scheme reliability of grading scheme – will observers come to same grade?will observers come to same grade?

• acceptability and usefulnessacceptability and usefulness– workshop this afternoonworkshop this afternoon