grade san sebastian 2008 05 12hjs

Grading the quality of evidence Grading the quality of evidence and the strength of and the strength of recommendationsrecommendations

San SebastiánSan Sebastián12 - 14 May 200812 - 14 May 2008

Pablo Alonso, Victor Montori, Andy Oxman, Holger Schünemann

Workshop agendaWorkshop agendaMonday, 12 May 2008Monday, 12 May 2008

08.15 – 13.30 h - Plenary08.15 – 13.30 h - Plenary Introduction to GRADEIntroduction to GRADE Grading the quality of evidence for each Grading the quality of evidence for each

important outcomeimportant outcome– Deciding which outcomes are importantDeciding which outcomes are important– Factors that can lower the quality of evidenceFactors that can lower the quality of evidence– Factors that can raise the quality of evidenceFactors that can raise the quality of evidence

Judgements about the overall quality of Judgements about the overall quality of evidenceevidence

15.00 – 17.00 h - Small groups15.00 – 17.00 h - Small groups The quality of evidence for HPV vaccinationThe quality of evidence for HPV vaccination17.00 – 18.00 h - Plenary17.00 – 18.00 h - Plenary DiscussionDiscussion

IntroductionIntroduction

AndyAndy

Why bother about grading?Why bother about grading?

People draw conclusions about thePeople draw conclusions about the– quality of evidencequality of evidence– strength of recommendationsstrength of recommendations

Systematic and explicit approaches can helpSystematic and explicit approaches can help– protect against errorsprotect against errors– resolve disagreementsresolve disagreements– facilitate critical appraisalfacilitate critical appraisal– communicate informationcommunicate information

However, there is wide variation in currently However, there is wide variation in currently used approachesused approaches

Many grading systemsMany grading systems

EvidenceEvidenceRecommendationRecommendation

BB Class IClass I C+ C+ 11 IVIV CC

OrganizationOrganization AHAAHA ACCPACCP SIGNSIGN

Recommendation for use of oral anticoagulation in Recommendation for use of oral anticoagulation in patients with atrial fibrillation and rheumatic mitral patients with atrial fibrillation and rheumatic mitral valve diseasevalve disease

Australian NMRCAustralian NMRC Oxford Center for Evidence-based MedicineOxford Center for Evidence-based Medicine Scottish Intercollegiate Guidelines (SIGN)Scottish Intercollegiate Guidelines (SIGN) US Preventative Services Task ForceUS Preventative Services Task Force Professional organizationsProfessional organizations

– AHA/ACC, ACCP, AAP, Endocrine society, etc....AHA/ACC, ACCP, AAP, Endocrine society, etc....

Lots of confusion and dismayLots of confusion and dismay

A common international grading system?A common international grading system?

International groupInternational group– ACCP, AHRQ, Australian NMRC, BMJ Clinical Evidence, CC, ACCP, AHRQ, Australian NMRC, BMJ Clinical Evidence, CC,

CDC, CC, NICE, Oxford CEBM, SIGN, UpToDate, USPSTF, CDC, CC, NICE, Oxford CEBM, SIGN, UpToDate, USPSTF, WHOWHO

> 60 contributors> 60 contributors– methodologists, guideline developers, systematic methodologists, guideline developers, systematic

reviewers, researchers, clinicians, editorsreviewers, researchers, clinicians, editors ~ 20 meetings over last seven years~ 20 meetings over last seven years

– ~10 – 40 participants~10 – 40 participants

www.gradeworkinggroup.org

American College of PhysiciansAmerican College of Physicians Agencia sanitaria regionale, Agencia sanitaria regionale,

Bologna, ItaliaBologna, Italia Agency for Health Care Research Agency for Health Care Research

and Quality (AHRQ)and Quality (AHRQ) Allergic Rhinitis in Asthma Allergic Rhinitis in Asthma

GuidelinesGuidelines American College of Chest American College of Chest

PhysiciansPhysicians American Endocrine SocietyAmerican Endocrine Society American Thoracic SocietyAmerican Thoracic Society BMJ Clinical EvidenceBMJ Clinical Evidence British Medical JournalBritish Medical Journal Canadian Agency for Drugs and Canadian Agency for Drugs and

Technology in Health Technology in Health Cochrane CollaborationCochrane Collaboration EBM Guidelines FinlandEBM Guidelines Finland European Respiratory SocietyEuropean Respiratory Society European Society of Thoracic European Society of Thoracic

SurgeonsSurgeons Evidence-based Nursing Sudtirol, Evidence-based Nursing Sudtirol,

Alta Adiga, ItalyAlta Adiga, Italy

Journal of Infection in Developing Journal of Infection in Developing Countries - InternationalCountries - International

Kidney disease: Improving global Kidney disease: Improving global outcomeoutcome

National Institute for Clinical National Institute for Clinical Excellence (NICE)Excellence (NICE)

Norwegian Knowledge Centre for Norwegian Knowledge Centre for the Health Servicesthe Health Services

Ontario MOH Medical Advisory Ontario MOH Medical Advisory SecretariatSecretariat

Polish Institute for EBMPolish Institute for EBM Society of Critical Care MedicineSociety of Critical Care Medicine Society of Pediatric EndocrinologySociety of Pediatric Endocrinology Society of Vascular SurgerySociety of Vascular Surgery Surviving sepsis campaignSurviving sepsis campaign The German Agency for Quality in The German Agency for Quality in

Medicine Medicine University of Pennsylvania Health University of Pennsylvania Health

System Center for EB Practice System Center for EB Practice UpToDateUpToDate World Health OrganizationWorld Health Organization

GRADE UptakeGRADE Uptake


Bologna, ItaliaBologna, Italia Agency for Health Care Agency for Health Care

Research and Quality (AHRQ)Research and Quality (AHRQ) Allergic Rhinitis in Asthma Allergic Rhinitis in Asthma


PhysiciansPhysicians American Endocrine SocietyAmerican Endocrine Society American Thoracic SocietyAmerican Thoracic Society BMJ Clinical EvidenceBMJ Clinical Evidence British Medical JournalBritish Medical Journal Canadian Agency for Drugs Canadian Agency for Drugs

and Technology in Healthand Technology in Health Cochrane CollaborationCochrane Collaboration EBM Guidelines FinlandEBM Guidelines Finland European Respiratory SocietyEuropean Respiratory Society European Society of Thoracic European Society of Thoracic






Norwegian Knowledge Centre Norwegian Knowledge Centre for the Health Servicesfor the Health Services



















Polish Institute for EBMPolish Institute for EBM Society of Critical Care Society of Critical Care

MedicineMedicine Society of Pediatric EndocrinologySociety of Pediatric Endocrinology Society of Vascular SurgerySociety of Vascular Surgery Surviving sepsis campaignSurviving sepsis campaign The German Agency for Quality in The German Agency for Quality in












Journal of Infection in Journal of Infection in Developing Countries - Developing Countries - InternationalInternational









What is quality of What is quality of evidence andevidence and

how should it be how should it be graded?graded?

Judging the quality of evidence Judging the quality of evidence requires considering the requires considering the

contextcontextIn the context of making In the context of making

recommendationsrecommendations The quality of evidence reflects the The quality of evidence reflects the

extent to which our confidence in an extent to which our confidence in an estimate of the effect is adequate to estimate of the effect is adequate to support a particular recommendation. support a particular recommendation.

In the context of a systematic reviewIn the context of a systematic review The quality of evidence reflects the The quality of evidence reflects the

extent to which we are confident that extent to which we are confident that an estimate of effect is correct. an estimate of effect is correct.

Although the degree of confidence is Although the degree of confidence is a continuum, we suggest using four a continuum, we suggest using four

categoriescategories

HighHighModerateModerateLowLowVery lowVery low

Categories of qualityCategories of quality

HighHigh: Further research is very unlikely to : Further research is very unlikely to change our confidence in the estimate of change our confidence in the estimate of effect. effect.

ModerateModerate: Further research is likely to have : Further research is likely to have an important impact on our confidence in the an important impact on our confidence in the estimate of effect and may change the estimate of effect and may change the estimate.estimate.

LowLow: Further research is very likely to have : Further research is very likely to have an important impact on our confidence in the an important impact on our confidence in the estimate of effect and is likely to change the estimate of effect and is likely to change the estimate.estimate.

Very lowVery low: Any estimate of effect is very : Any estimate of effect is very uncertain.uncertain.

Study design is importantStudy design is important

Early systems of grading the quality of Early systems of grading the quality of evidence focused almost exclusively evidence focused almost exclusively on study designon study design

Randomised trials provide, in general, Randomised trials provide, in general, far stronger evidence than far stronger evidence than observational studies.observational studies. – Randomised trials start out at HighRandomised trials start out at High– Observational studies start out at Low Observational studies start out at Low

However, other factors may decrease However, other factors may decrease or increase the quality of evidenceor increase the quality of evidence

What can lower the What can lower the quality of evidence?quality of evidence?

Study limitationsStudy limitations Inconsistency of results Inconsistency of results Indirectness of evidence Indirectness of evidence Imprecise results Imprecise results Publication bias Publication bias

What can raise the quality What can raise the quality of evidence?of evidence?

Large magnitude of effect Large magnitude of effect A dose response relationshipA dose response relationshipAll plausible confoundingAll plausible confounding

Quality assessment criteriaQuality assessment criteria

Strength of recommendationStrength of recommendationThe degree of confidence that the The degree of confidence that the

desirable effects of adherence to a desirable effects of adherence to a recommendation outweigh the recommendation outweigh the undesirable effects. undesirable effects.

Desirable Desirable effectseffects•health benefitshealth benefits•less burdenless burden•savingssavings

Undesirable Undesirable effectseffects•harmsharms•more burdenmore burden•costscosts

Categories of recommendationsCategories of recommendations

Although the degree of confidence is a Although the degree of confidence is a continuum, we suggest using two continuum, we suggest using two categories: strong and weak.categories: strong and weak.

Strong recommendation:Strong recommendation: the panel the panel is confident that the desirable effects is confident that the desirable effects of adherence to a recommendation of adherence to a recommendation outweigh the undesirable effects.outweigh the undesirable effects.

Weak recommendation:Weak recommendation: the panel the panel concludes that the desirable effects concludes that the desirable effects of adherence to a recommendation of adherence to a recommendation probably outweigh the undesirable probably outweigh the undesirable effects, but is not confident. effects, but is not confident.

Recommend

Suggest

Determinants of strength of recommendationDeterminants of strength of recommendation

Factors Impact on the strength of a recommendation

Balance between desirable and undesirable effects

Larger the difference between the desirable and undesirable effects, more likely a strong recommendation warranted. Narrower the gradient, more likely weak recommendation warranted.

Quality of the evidence

Higher the quality of evidence, more likely a strong recommendation warranted.

Values and preferences

More variability in values and preferences, or more uncertainty in values and preferences, more likely weak recommendation warranted.

Costs (resource use)

Higher the costs of an intervention – that is, the more resources consumed – less likely a strong recommendation warranted.


Quality of evidenceQuality of evidence



Judgements about the overall quality of Judgements about the overall quality of evidenceevidence

15.00 – 17.00 h - Small groups15.00 – 17.00 h - Small groups The quality of evidence for HPV vaccinationThe quality of evidence for HPV vaccination17.00 – 18.00 h - Plenary17.00 – 18.00 h - Plenary DiscussionDiscussion

Workshop agendaWorkshop agendaTuesday, 13 May 2008Tuesday, 13 May 2008

Strength of recommendationStrength of recommendation

08.30 – 13.30 h - Plenary08.30 – 13.30 h - Plenary Going from evidence to recommendationsGoing from evidence to recommendations

– Summarising and presenting findingsSummarising and presenting findings– Factors that determine the strength of a recommendationFactors that determine the strength of a recommendation

Balance of benefits and harmsBalance of benefits and harms (Quality of evidence)(Quality of evidence) Values and preferencesValues and preferences Costs (resource use)Costs (resource use)

– Judgements about the strength of recommendationsJudgements about the strength of recommendations– Wording of recommendationsWording of recommendations

15.00 – 17.00 h - Small groups15.00 – 17.00 h - Small groups Summary of findings and the strength of Summary of findings and the strength of

recommendation for HPV vaccinationrecommendation for HPV vaccination17.00 – 18.00 h - Plenary17.00 – 18.00 h - Plenary DiscussionDiscussion

Workshop agendaWorkshop agendaWednesday, 14 May 2008Wednesday, 14 May 2008

Special challengesSpecial challenges

08.30 – 11.30 h – Plenary + small group 08.30 – 11.30 h – Plenary + small group workwork

Economic evidence (resource use)Economic evidence (resource use)

13.00 – 17.00 h - Plenary + small group 13.00 – 17.00 h - Plenary + small group workwork

Diagnostic testsDiagnostic tests

General questions about GRADE, the General questions about GRADE, the GRADE Working Group or the GRADE Working Group or the

agenda for this workshop?agenda for this workshop?

Relative importance of Relative importance of outcomesoutcomes

HolgerHolger

The quality of evidence needs to be The quality of evidence needs to be considered for each important considered for each important

outcomeoutcome

The quality of evidence may be different for The quality of evidence may be different for different outcomesdifferent outcomes

Decision makers (and review authors) need to Decision makers (and review authors) need to consider the relative importance of outcomesconsider the relative importance of outcomes

Flatulence 2

Hierarchy of outcomes according to their Hierarchy of outcomes according to their importance to assess the effect of phosphate importance to assess the effect of phosphate lowering drugs in patients with renal failure and lowering drugs in patients with renal failure and hyperphosphatemiahyperphosphatemia

Importance of endpoints

Critical for decision making

Important, but not critical for decision making

Of lowimportance

5

Pain due to soft tissue 6 calcification / function

Fractures 7

Myocardial infarction 8

Mortality 9

3

4

1

A different clinical A different clinical questionquestion

Population: In patients with chronic atrial fibrillation and no prior history

of stroke

Intervention: does oral anticoagulation (comparison) compared with no therapy

Outcome: reduce the risk for embolic stroke, hemorrhage

and death?

Population: In patients with chronic atrial fibrillation and no prior history

of stroke

Intervention: does oral anticoagulation (comparison) compared with no therapy

Outcome: reduce the risk for embolic stroke, hemorrhage

and death?Other outcomes:

Inconvenience, quality of life

All disabling or fatal stroke (isch. and hemorrh.)

Major hemorrhage (non IC) All cause mortality Health related quality of life

*Systematic Review: Aguilar & Hart. Cochrane Database of Systematic Reviews 2005,

Issue 3.

Outcomes/endpointsOutcomes/endpoints

How important is the endpoint for decision making?Judgment about the relative importance for each endpoint on a scale from 9 (most important) to 1 (least important): • 7 – 9: the endpoint is critical for decision making.• 4 – 6: the endpoint is important but not critical for decision making.• 1 – 3: the endpoint is not important.


2




Of lowimportance

5

6

7

8

9

3

4

1

2




Of lowimportance

5

6

7

8

9

3

4

1

All disabling or fatal stroke (isch. and hemorrh.)

Major hemorrhage (non IC) All cause mortality Health related quality of life

*Systematic Review: Aguilar & Hart. Cochrane Database of Systematic Reviews 2005,

Issue 3.


9

8

9

7

2




Of lowimportance

5

6

7

8

9

3

4

1

Questions about the relative Questions about the relative importance of outcomes?importance of outcomes?

Study limitationsStudy limitations

VictorVictor

Study limitations (Risk of bias)Study limitations (Risk of bias)for Randomized trialsfor Randomized trials

Lack of allocation concealmentLack of allocation concealment Lack of blindingLack of blinding Incomplete accounting of patients and Incomplete accounting of patients and

outcome eventsoutcome events Selective outcome reportingSelective outcome reporting Other limitationsOther limitations

– e.g. stopping early for benefit; use of e.g. stopping early for benefit; use of unvalidated patient-reported outcomes; carry-unvalidated patient-reported outcomes; carry-over effects in cross-over trials; recruitment bias over effects in cross-over trials; recruitment bias in cluster-randomized trialsin cluster-randomized trials

Testosterone for BMD in menTestosterone for BMD in men

Assessment of the risk of biasAssessment of the risk of bias

Cochrane Handbook for Systematic Reviews of Interventions, Draft chapter of version 5 - Assessing risk of bias in included studies http://www.cochrane.org/resources/handbook/handbook5drafts.htm

Should patients with cancer Should patients with cancer receive parenteral receive parenteral

anticoagulation (heparin) to anticoagulation (heparin) to prolong survival. prolong survival.

Outcomes: Survival, bleedingOutcomes: Survival, bleeding

SurvivalSurvival

HR 0.77 (0.65 to 0.91)HR 0.77 (0.65 to 0.91)

How confident are you that these How confident are you that these results are true?results are true?

Factors that lower qualityFactors that lower quality

detailed design and executiondetailed design and execution inconsistencyinconsistency indirectnessindirectness reporting biasreporting bias imprecisionimprecision

Detailed study design and Detailed study design and executionexecution

Quality now?Quality now?

HighHigh

Major bleeding – start highMajor bleeding – start high

RR 1.50 (0.26 – 8.80)RR 1.50 (0.26 – 8.80)

Detailed study design and Detailed study design and executionexecution


ModerateModerate

Bleeding in patients with cancer Bleeding in patients with cancer receiving heparing for prolonging receiving heparing for prolonging

survivalsurvival


LowLow Observational studies could have Observational studies could have

provided higher quality evidenceprovided higher quality evidence

Summarizing study limitations for randomized trialsSummarizing study limitations for randomized trials

Questions about study Questions about study limitations?limitations?

Inconsistency of resultsInconsistency of results

HolgerHolger

Inconsistency of resultsInconsistency of results

Look for explanationLook for explanation– patients, intervention, comparator, patients, intervention, comparator,

outcome, methodsoutcome, methods

JudgmentJudgment– variation in size of effectvariation in size of effect– overlap in confidence intervalsoverlap in confidence intervals– statistical significance of heterogeneitystatistical significance of heterogeneity– II22

Akl E, Barba M, Rohilla S, Terrenato I, Sperati F, Schünemann HJ. “Anticoagulation for the long term treatment of venous thromboembolism in patients with cancer”. Cochrane Database Syst Rev. 2008 Apr 16;(2):CD006650.

Heparin or vitamin K antagonists for Heparin or vitamin K antagonists for survival in patients with cancer survival in patients with cancer

Non-steroidal drug use and risk of Non-steroidal drug use and risk of pancreatic cancerpancreatic cancer

Capurso G, Schünemann HJ, Terrenato I, Moretti A, Koch M, Muti P, Capurso L, Delle Fave G. Meta-analysis: the use of non-steroidal anti-inflammatory drugs and pancreatic cancer risk for different exposure

categories.Aliment Pharmacol Ther. 2007 Oct 15;26(8):1089-99.

Example: Thromboprophylaxis Example: Thromboprophylaxis and bleeding in the hospitaland bleeding in the hospital

Dentali et al. Ann Int Med, 2007

Questions about inconsistency Questions about inconsistency of results?of results?

Indirectness of evidenceIndirectness of evidence

VictorVictor


Question of Question of interestinterest

Relative Relative effectiveness of effectiveness of alendronate and alendronate and risedronate in risedronate in osteoporosisosteoporosis

Source of indirectnessSource of indirectness

Indirect comparisonIndirect comparison: : Randomized trials have compared Randomized trials have compared alendronate to placebo and alendronate to placebo and risedronate to placebo, but trials risedronate to placebo, but trials comparing alendronate to comparing alendronate to risedronate are unavailable.risedronate are unavailable.


Question of Question of interestinterest

Rosiglitazone for Rosiglitazone for prevention of prevention of diabetic diabetic complications in complications in patients at high patients at high risk of developing risk of developing diabetesdiabetes


Differences inDifferences in Outcome Outcome: A : A randomized trial shows delay in randomized trial shows delay in the development of biochemical the development of biochemical diabetes with rosiglitazone, but diabetes with rosiglitazone, but was underpowered to address was underpowered to address diabetic complicationsdiabetic complications

Indirectness of evidenceIndirectness of evidenceQuestion of interestQuestion of interest

Oseltamivir for Oseltamivir for prophylaxis of avian prophylaxis of avian flu caused by flu caused by influenza A(H5N1) influenza A(H5N1) virus virus

Sigmoidoscopic Sigmoidoscopic screening for screening for prevention of colon prevention of colon cancer mortalitycancer mortality

Choice of medication Choice of medication for schizophreniafor schizophrenia


Differences inDifferences in Population Population: randomized : randomized trials of oseltamivir are available for trials of oseltamivir are available for seasonal influenza, but not for avian fluseasonal influenza, but not for avian flu

Differences inDifferences in Intervention Intervention: : randomized trials of fecal occult blood randomized trials of fecal occult blood screening provide indirect evidence screening provide indirect evidence bearing on the potential effectiveness of bearing on the potential effectiveness of sigmoidoscopysigmoidoscopy

Differences inDifferences in Comparison Comparison: A series of : A series of trials comparing newer generation trials comparing newer generation neuroleptic agents to fixed doses of 20 neuroleptic agents to fixed doses of 20 mg. of haloperidol provide indirect mg. of haloperidol provide indirect evidence of how the newer agents would evidence of how the newer agents would compare to the lower, flexible doses of compare to the lower, flexible doses of haloperidol clinicians typically usehaloperidol clinicians typically use


Guidelines for the treatment of osteogenesis Guidelines for the treatment of osteogenesis imperfecta in adultsimperfecta in adults

In patients with osteoporosis, In patients with osteoporosis, bisphosphonates, either oral or IV, increase bisphosphonates, either oral or IV, increase BMD and reduce the risk of fragility fracture BMD and reduce the risk of fragility fracture by 50%by 50%

RCT oral bisphosphonates showed improved RCT oral bisphosphonates showed improved BMD in patients with OIBMD in patients with OI

RCT IV bisphosphonates showed improved RCT IV bisphosphonates showed improved cortical thickness in patients with OIcortical thickness in patients with OI

Questions about indirectness of Questions about indirectness of evidence?evidence?

Imprecise resultsImprecise results

VictorVictor

How to decide if the evidence How to decide if the evidence about a particular outcomes is about a particular outcomes is

impreciseimprecise 1. If the evidence were to come from a 1. If the evidence were to come from a

single RCT, how big should this RCT single RCT, how big should this RCT need to be to estimate the treatment need to be to estimate the treatment effect?effect?– Optimal information sizeOptimal information size

Control group event rate

Tota

l nu

mb

er

of

eve

nts

re

qu

ire

d

0.0 0.2 0.4 0.6 0.8 1.0

02

00

40

06

00

RRR=30%

RRR=25%

RRR=20%

300 events


impreciseimprecise– Optimal information sizeOptimal information size– 300 events300 events

Stroke – periop beta blockersStroke – periop beta blockers

1 5 10 50 1000.5 0.1

Study Year Overall Event Rate Relative Risk (95% CI)

Wallace 1998 5 / 200 3.06 (0.49 to 19.02)

Pobble 2005 1 / 103 2.63 (0.11 to 62.97)

DIPOM 2006 2 / 921 4.97 (0.24 to 103.19)

MaVS 2006 6 / 496 1.83 (0.39 to 8.50)

Zaugg 2007 1 / 119 2.97 (0.12 to 72.19)

POISE 2007 60 / 8351 2.13 (1.25 to 3.64)

Fixed Effects Estimate 2.22 (1.39 to 3.56)

p=0.99 for heterogeneity, I²=0%

Offer all effective treatments?Offer all effective treatments? atrial fib at risk of strokeatrial fib at risk of stroke

warfarin increases serious gi bleedingwarfarin increases serious gi bleeding– 3% per year 3% per year

1,000 patients 1 less stroke1,000 patients 1 less stroke– 30 more bleeds for each stroke prevented30 more bleeds for each stroke prevented

1,000 patients 100 less strokes1,000 patients 100 less strokes– 3 strokes prevented for each bleed3 strokes prevented for each bleed

where is your threshold?where is your threshold?– how many strokes in 100 with 3% bleeding?how many strokes in 100 with 3% bleeding?

00.5%1.0%


impreciseimprecise– Optimal information sizeOptimal information size– 300 events300 events– Evaluate the boundaries of the Evaluate the boundaries of the

CICI (if low event rates – CI for risk diff)(if low event rates – CI for risk diff)

Rate down for imprecision?Rate down for imprecision?

Rate down for imprecision for Rate down for imprecision for guidelines?guidelines?

Bleeding in patients with cancer Bleeding in patients with cancer receiving heparing for prolonging receiving heparing for prolonging

survivalsurvival

Rate down for imprecision for Rate down for imprecision for guidelines?guidelines?

How to decide if the evidence is How to decide if the evidence is imprecise for a imprecise for a

recommendationrecommendation– 300 events300 events– Evaluate the boundaries of the Evaluate the boundaries of the

CI in light of other important CI in light of other important outcomesoutcomes

Questions about imprecise Questions about imprecise results?results?

Publication biasPublication bias

VictorVictor

In 102 Danish protocols|122 reports in 2003In 102 Danish protocols|122 reports in 2003

50% of outcomes were incompletely 50% of outcomes were incompletely reportedreported

Odds of full reporting: Odds of full reporting:

2:1 2:1 if the outcome was if the outcome was

significantsignificant

JAMA. 2004;291:2457-2465.

Furukawa TA et al. JAMA 2007; 297: 468-9

1

10

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

<20% 20-39% 40-59% 60-80% >80%

Continuous outcomes

Dichotomous outcomes

% eligible trials contributing data to meta-analysis

Eff

ect

size

Od

ds

rati

o

Perioperative beta-Perioperative beta-blockerblocker

PositivePositivePoldermans (NEJM, Poldermans (NEJM,

n=112), RRR 80% n=112), RRR 80%

NegativeNegativeDIPOM (BMJ, n=921), DIPOM (BMJ, n=921),

RRI: 3%RRI: 3%POBBLE (J Vasc Surgery, POBBLE (J Vasc Surgery,

n=103), RRI = 6% n=103), RRI = 6% MAVS (Am Heart J, MAVS (Am Heart J,

n=496), RRR = 15%n=496), RRR = 15%

Statins for DM2Positive

CARDS (Lancet, 2004), RRR = 37%

NegativeASPEN (Diabetes Care, 2006), RRR =

10%

Publication biasPublication bias

Reporting of outcomesReporting of outcomes– Selective outcome reportingSelective outcome reporting

Reporting of studiesReporting of studies– Publication biasPublication bias

number of small studiesnumber of small studies industry sponsoredindustry sponsored

High likelihood could lower qualityHigh likelihood could lower quality

Questions about publication Questions about publication bias?bias?

Factors that can raise the quality Factors that can raise the quality of evidenceof evidence

AndyAndy

What can raise the quality What can raise the quality of evidence?of evidence?

Large magnitude of effect Large magnitude of effect A dose response relationshipA dose response relationshipAll plausible confoundingAll plausible confounding

Large magnitude of effectLarge magnitude of effect

Rules of thumb based on simulation studiesRules of thumb based on simulation studies– RR < 0.5RR < 0.5– RR < 0.2RR < 0.2

More compelling if: More compelling if: – consistency of effectconsistency of effect

almost everyone used to do badly, now does wellalmost everyone used to do badly, now does well– quick actionquick action

reversal of trajectoryreversal of trajectory– strong indirect evidencestrong indirect evidence

ExamplesExamples– insulin for diabetic ketoacidosisinsulin for diabetic ketoacidosis

– hip replacement for severe osteoarthritiship replacement for severe osteoarthritis

What are examples of interventions What are examples of interventions you can think of for which you might you can think of for which you might

upgrade the quality of evidence upgrade the quality of evidence because of a large magnitude of because of a large magnitude of

effect?effect?

Why would you or would you not Why would you or would you not upgrade the quality of evidence?upgrade the quality of evidence?

When do strong associations When do strong associations provide compelling evidence of provide compelling evidence of

effect (causation)?effect (causation)? Some interventions have such dramatic

effects that biases can be ruled out without randomised trials

Dramatic effects can be defined by the size of the treatment effect (signal) relative to the expected prognosis (noise)

Real treatment effects are likely if the signal to noise ratio is large (above 10)

Large ratios may be due to – the high proportion of patients improved

(consistency of effect) or– the rapidity of improvement (quick action)

And are more compelling if they are “plausible” (strong indirect evidence)

Dose response relationshipDose response relationship Risk of bleeding in patients who have Risk of bleeding in patients who have

supratherapeutic anticoagulation levels supratherapeutic anticoagulation levels – Dose response gradient between higher Dose response gradient between higher

levels of the international normalized ratio levels of the international normalized ratio (INR) and the increased risk of bleeding (INR) and the increased risk of bleeding

Risk of death in patients presenting Risk of death in patients presenting with sepsis and hypotensionwith sepsis and hypotension


upgrade the quality of evidence upgrade the quality of evidence because of a dose response because of a dose response

relationship?relationship?

All plausible confounding All plausible confounding would result in an underestimate of would result in an underestimate of

the treatment effectthe treatment effect Higher death rates in private for-profit Higher death rates in private for-profit

versus private not-for-profit hospitalsversus private not-for-profit hospitals– patients in the not-for-profit hospitals patients in the not-for-profit hospitals

likely sicker than those in the for-profit likely sicker than those in the for-profit hospitalshospitals

– for-profit hospitals are likely to admit a for-profit hospitals are likely to admit a larger proportion of well-insured patients larger proportion of well-insured patients than not-for-profit hospitals (and thus have than not-for-profit hospitals (and thus have more resources with a spill over effect)more resources with a spill over effect)

All plausible biases All plausible biases would result in an overestimate of would result in an overestimate of

effecteffect Hypoglycaemic drug phenformin Hypoglycaemic drug phenformin

causes lactic acidosiscauses lactic acidosis The related agent metformin is under The related agent metformin is under

suspicion for the same toxicity. suspicion for the same toxicity. Large observational studies have failed Large observational studies have failed

to demonstrate an associationto demonstrate an association– Clinicians would be more alert to lactic Clinicians would be more alert to lactic

acidosis in the presence of the agentacidosis in the presence of the agent


upgrade the quality of evidence upgrade the quality of evidence

when an effect (association) was when an effect (association) was found - because all plausible found - because all plausible

confounders would result in an confounders would result in an underestimate of effect orunderestimate of effect or

when an effect (association) was not when an effect (association) was not found - because all plausible biases found - because all plausible biases would result in an overestimate of would result in an overestimate of

effect?effect?

Questions about factors that can Questions about factors that can raise the quality of evidence?raise the quality of evidence?

Judgements about the overall Judgements about the overall quality of evidencequality of evidence

AndyAndy

Quality assessment criteriaQuality assessment criteria

Judgements about the overall Judgements about the overall quality of evidencequality of evidence

Most systems not explicitMost systems not explicit

Options:Options:– BenefitsBenefits– Primary outcomePrimary outcome– Highest Highest – LowestLowest

Based on lowest of all the Based on lowest of all the criticalcritical outcomesoutcomes

Beyond the scope of a systematic reviewBeyond the scope of a systematic review

Flavanoids for HemorrhoidsFlavanoids for Hemorrhoids

venotonic agentsvenotonic agents– mechanism unclear, increase venous returnmechanism unclear, increase venous return

popularitypopularity– 90 venotonics commercialized in France90 venotonics commercialized in France– none in Sweden and Norwaynone in Sweden and Norway– France 70% of world marketFrance 70% of world market

possibilitiespossibilities– French misguidedFrench misguided– rest of world missing outrest of world missing out

Systematic ReviewSystematic Review

14 trials, 1432 patients14 trials, 1432 patients

Key outcomeKey outcome– risk not improving/persistent symptomsrisk not improving/persistent symptoms– 11 studies, 1002 patients, 375 events11 studies, 1002 patients, 375 events

– RR 0.4, 95% CI 0.29 to 0.57RR 0.4, 95% CI 0.29 to 0.57

Minimal side effectsMinimal side effects

Is France right?Is France right?

What is the quality of evidence?What is the quality of evidence?

What can lower the quality of What can lower the quality of evidence?evidence?

Study limitationsStudy limitations– lack of detail re concealmentlack of detail re concealment– questionnaires not validatedquestionnaires not validated

Rate down quality for study limitations?Rate down quality for study limitations?

Indirectness – no problemIndirectness – no problem

Inconsistency, need to look at the resultsInconsistency, need to look at the results

Review : Phlebotonics for hemorrhoidsComparison: 01 Venotonics vs placebp Outcome: 08 Overall improvement: no improvement/some improvement

Study RR (random) Weight RR (random)or sub-category log[RR] (SE) 95% CI % 95% CI

01 Up to seven daysChauvenet -0.8916 (0.2376) 12.67 0.41 [0.26, 0.65] Cospite -2.2073 (0.6117) 5.51 0.11 [0.03, 0.36] Thanapongsathorn -0.4308 (0.2985) 11.18 0.65 [0.36, 1.17]

Subtotal (95% CI) 29.36 0.37 [0.18, 0.77]Test for heterogeneity: Chi² = 6.92, df = 2 (P = 0.03), I² = 71.1%Test for overall effect: Z = 2.67 (P = 0.008)

02 Up to four w eeksAnnoni F -1.6094 (0.7073) 4.50 0.20 [0.05, 0.80] Clyne MB -0.9943 (0.3983) 8.94 0.37 [0.17, 0.81] Pirard J -1.1712 (0.3086) 10.94 0.31 [0.17, 0.57] Thanapongsathorn -1.1087 (1.1098) 2.18 0.33 [0.04, 2.91] Thorp 0.2624 (0.3291) 10.46 1.30 [0.68, 2.48] Titapan -0.8916 (0.3691) 9.56 0.41 [0.20, 0.85] Wijayanegara -0.5978 (0.1375) 14.97 0.55 [0.42, 0.72]

Subtotal (95% CI) 61.54 0.48 [0.32, 0.72]Test for heterogeneity: Chi² = 13.87, df = 6 (P = 0.03), I² = 56.7%Test for overall effect: Z = 3.57 (P = 0.0004)

03 Further than four w eeksGodeberg -1.7719 (0.3906) 9.10 0.17 [0.08, 0.37]

Subtotal (95% CI) 9.10 0.17 [0.08, 0.37]Test for heterogeneity: not applicableTest for overall effect: Z = 4.54 (P < 0.00001)

Total (95% CI) 100.00 0.40 [0.29, 0.57]Test for heterogeneity: Chi² = 28.66, df = 10 (P = 0.001), I² = 65.1%Test for overall effect: Z = 5.14 (P < 0.00001)

0.001 0.01 0.1 1 10 100 1000

Favours treatment Favours control

Publication bias?Publication bias?

Size of studiesSize of studies– 40 to 234 patients, most around 10040 to 234 patients, most around 100

All industry sponsoredAll industry sponsored

Review : Phlebotonics for hemorrhoidsComparison: 01 Venotonics vs placebp Outcome: 08 Overall improvement: no improvement/some improvement

0.001 0.01 0.1 1 10 100 1000

0.0

0.4

0.8

1.2

1.6

RR (fixed)

What is the overall quality of What is the overall quality of evidence across criteria?evidence across criteria?

HighHigh: Further research is very unlikely to : Further research is very unlikely to change our confidence in the estimate of change our confidence in the estimate of effect. effect.

ModerateModerate: Further research is likely to have : Further research is likely to have an important impact on our confidence in the an important impact on our confidence in the estimate of effect and may change the estimate of effect and may change the estimate.estimate.

LowLow: Further research is very likely to have : Further research is very likely to have an important impact on our confidence in the an important impact on our confidence in the estimate of effect and is likely to change the estimate of effect and is likely to change the estimate.estimate.

Very lowVery low: Any estimate of effect is very : Any estimate of effect is very uncertain.uncertain.

Threshold for downgradingThreshold for downgrading

highestlowestOKrate down quality

Overall quality across criteriaOverall quality across criteria

highestlowesthighmoderate low very low

Is France right?Is France right?

RecommendRecommend– for or against?for or against?

Strength of recommendationStrength of recommendation– strong or weakstrong or weak

Questions about judgements of Questions about judgements of the overall quality of evidence?the overall quality of evidence?




Judgements about the overall quality of evidenceJudgements about the overall quality of evidence15.00 – 17.00 h - Small groups15.00 – 17.00 h - Small groups The quality of evidence for HPV vaccinationThe quality of evidence for HPV vaccination17.00 – 18.00 h - Plenary17.00 – 18.00 h - Plenary DiscussionDiscussion

Should routine HPV vaccination Should routine HPV vaccination be recommended for be recommended for

12 year old girls in Spain?12 year old girls in Spain?

Introduction to small group workIntroduction to small group work

Questions about the small group Questions about the small group work?work?

Grading the quality of evidence Grading the quality of evidence and the strength of and the strength of recommendationsrecommendations

San SebastiánSan Sebastián13 May 200813 May 2008

Pablo Alonso, Victor Montori, Andy Oxman, Holger Schünemann

Any questions from yesterday about Any questions from yesterday about quality of evidence quality of evidence

or about the agenda for today?or about the agenda for today?

Summarising and presenting Summarising and presenting findingsfindings

AndyAndy

Questions about summarising Questions about summarising and presenting findings?and presenting findings?

Strength of recommendationsStrength of recommendations

HolgerHolger

Strength of recommendationStrength of recommendationThe degree of confidence that the The degree of confidence that the

desirable effects of adherence to a desirable effects of adherence to a recommendation outweigh the recommendation outweigh the undesirable effects. undesirable effects.

Desirable Desirable effectseffects•health benefitshealth benefits•less burdenless burden•savingssavings

Undesirable Undesirable effectseffects•harmsharms•more burdenmore burden•costscosts

Categories of recommendationsCategories of recommendations

Although the degree of confidence is a Although the degree of confidence is a continuum, we suggest using two continuum, we suggest using two categories: strong and weak.categories: strong and weak.

Strong recommendation:Strong recommendation: the panel the panel is confident that the desirable effects is confident that the desirable effects of adherence to a recommendation of adherence to a recommendation outweigh the undesirable effects.outweigh the undesirable effects.

Weak recommendation:Weak recommendation: the panel the panel concludes that the desirable effects concludes that the desirable effects of adherence to a recommendation of adherence to a recommendation probably outweigh the undesirable probably outweigh the undesirable effects, but is not confident. effects, but is not confident.

Recommend

Suggest

Determinants of strength of recommendationDeterminants of strength of recommendation










Judgements about the strength Judgements about the strength of a recommendationof a recommendation

No precise threshold for going from a strong to a No precise threshold for going from a strong to a weak recommendationweak recommendation

The presence of important concerns about one or The presence of important concerns about one or more of these factors make a weak more of these factors make a weak recommendation more likely. recommendation more likely.

Panels should consider all of these factors and Panels should consider all of these factors and make the reasons for their judgements explicit.make the reasons for their judgements explicit.

Recommendations should specify the Recommendations should specify the perspectiveperspective that is taken (e.g. individual patient, that is taken (e.g. individual patient, health system) and which outcomes were health system) and which outcomes were considered (including which, if any costs). considered (including which, if any costs).

Implications of a strong Implications of a strong recommendationrecommendation

PatientsPatients: Most people in your situation : Most people in your situation would want the recommended course of would want the recommended course of action and only a small proportion would action and only a small proportion would notnot

CliniciansClinicians: Most patients should receive : Most patients should receive the recommended course of actionthe recommended course of action

Policy makersPolicy makers: The recommendation : The recommendation can be adapted as a policy in most can be adapted as a policy in most situationssituations

Implications of a weak Implications of a weak recommendationrecommendation

PatientsPatients: The majority of people in your : The majority of people in your situation would want the recommended situation would want the recommended course of action, but many would not course of action, but many would not

CliniciansClinicians: Be prepared to help patients : Be prepared to help patients to make a decision that is consistent to make a decision that is consistent with their own valueswith their own values

Policy makersPolicy makers: There is a need for : There is a need for substantial debate and involvement of substantial debate and involvement of stakeholdersstakeholders

Is France right?Is France right? RecommendRecommend

– for or against?for or against?

Strength of recommendationStrength of recommendation– strong or weakstrong or weak










Another exampleAnother example Should oseltamivir be used for treatment of Should oseltamivir be used for treatment of

patients hospitalised with avian influenza patients hospitalised with avian influenza (H5N1)?(H5N1)?

Schünemann HJ et al. WHO Rapid Advice Schünemann HJ et al. WHO Rapid Advice Guidelines for the pharmacological Guidelines for the pharmacological management of human infection with management of human infection with avian influenza A (H5N1) virus. Lancet avian influenza A (H5N1) virus. Lancet Infectious Diseases 2007; 7:21-31.Infectious Diseases 2007; 7:21-31.

Schünemann HJ et al. WHO Rapid Advice Schünemann HJ et al. WHO Rapid Advice Guidelines: Quick and Transparent. PLoS Guidelines: Quick and Transparent. PLoS Medicine 2007; 4:e119.Medicine 2007; 4:e119.

Should oseltamivir be used for treatment of patients Should oseltamivir be used for treatment of patients hospitalised with avian influenza (H5N1)?hospitalised with avian influenza (H5N1)?

Summary of findings

Transmission: No human to human transmissionPatient or population: Hospitalised, clinical and serologically confirmed cases of avian influenza

What would you recommend?What would you recommend?










Strong recommendation: the panel is confident that the desirable effects of adherence to a recommendation outweigh the undesirable effects.Weak recommendation: the panel concludes that the desirable effects of adherence to a recommendation probably outweigh the undesirable effects, but is not confident.

Judgments about the strength of a recommendation - Judgments about the strength of a recommendation - oseltamivir for treatment of patients hospitalised with oseltamivir for treatment of patients hospitalised with

avian influenza (H5N1)avian influenza (H5N1)

Factors Comments


The benefits are uncertain.

Quality of the evidence The quality of the evidence is very low.

Values and preferences All patients and care providers would accept treatment for H5N1 disease.

Costs (resource use) The cost is not high for treatment of sporadic cases.

Another exampleAnother example Should aspirin be given to children with fever?Should aspirin be given to children with fever?

Very low quality evidence of increased Very low quality evidence of increased risk of Reye’s syndromerisk of Reye’s syndrome

Acetaminophen (paracetamol) is a Acetaminophen (paracetamol) is a cheap and effective alternative cheap and effective alternative

Questions about Questions about the strength of the strength of

recommendations?recommendations?

Wording of recommendationsWording of recommendations

????

Questions about Questions about the wording of the wording of

recommendations?recommendations?

Recommendations to use an Recommendations to use an intervention only in the context of intervention only in the context of

researchresearch

AndyAndy

Possible recommendations when there is very low quality

evidence include:

Recommend using an intervention Suggest using an intervention

Recommend using an intervention only in the context of research

Suggest not using an intervention

Recommend not using an intervention

Recommendations to use an Recommendations to use an intervention only in the context intervention only in the context of research may be appropriateof research may be appropriate

Two conditions must be metTwo conditions must be met1.1. There must be inThere must be insufficient evidence sufficient evidence

for a panel to suggest using or not for a panel to suggest using or not using an intervention. using an intervention.

2.2. Further research must have a large Further research must have a large potential for reducing uncertainty potential for reducing uncertainty about the effects of the intervention, about the effects of the intervention, and for doing so at a reasonable cost. and for doing so at a reasonable cost.

ExamplesExamples

National Institute of Health and Clinical National Institute of Health and Clinical Excellence (NICE) Excellence (NICE) – Of its first 95 technology appraisals, eight led to Of its first 95 technology appraisals, eight led to

recommendations for use in the context of research.recommendations for use in the context of research. Centers for Medicare and Medicaid Services Centers for Medicare and Medicaid Services

(CMMS)(CMMS)– June 7, 2000 executive memorandum directing the Secretary June 7, 2000 executive memorandum directing the Secretary

of Health and Human Services to "explicitly authorize of Health and Human Services to "explicitly authorize [Medicare] payment for routine patient care costs...and costs [Medicare] payment for routine patient care costs...and costs due to medical complications associated with participation in due to medical complications associated with participation in clinical trials.“clinical trials.“

– July 9, 2007 the following was added to the Clinical Trial Policy:July 9, 2007 the following was added to the Clinical Trial Policy:– CMMS . . . through an individualized assessment of benefits, CMMS . . . through an individualized assessment of benefits,

risks, and research potential, may determine that certain items risks, and research potential, may determine that certain items and services for which there is some evidence of significant and services for which there is some evidence of significant medical benefit, but for which there is insufficient evidence to medical benefit, but for which there is insufficient evidence to support a “reasonable and necessary” determination, are only support a “reasonable and necessary” determination, are only reasonable and necessary when provided in a clinical trial . . .reasonable and necessary when provided in a clinical trial . . .

Implications for researchImplications for researchCore elements E Evidence (What is the current state of the

evidence?) P Population (What is the population of interest?) I Intervention (What are the interventions of interest?) C Comparison (What are the comparisons of interest?) O Outcome (What are the outcomes of interest?) T Time stamp (Date of recommendation)Optional elements d Disease burden or relevance t Time aspect of core elements of EPICOT s Appropriate study type according to local need


recommend that it should only be recommend that it should only be used in the context of research?used in the context of research?

Questions about Questions about recommendations to use an recommendations to use an

intervention only in the context of intervention only in the context of research?research?

Judgements about priorities Judgements about priorities (importance)(importance)

AndyAndy

Priority setting:Priority setting:From a health system From a health system

or public health or public health perspectiveperspective

A A strongstrong recommendation recommendation may not be may not be importantimportant

Judgements about Judgements about importanceimportance

may depend onmay depend on StrengthStrength of the recommendations of the recommendations SeveritySeverity of the problemof the problem PrevalencePrevalence of the problem of the problem Magnitude of theMagnitude of the net benefitsnet benefits EfficiencyEfficiency (better use of resources) (better use of resources) Total costTotal cost Impact onImpact on inequitiesinequities Potential for improved Potential for improved quality of carequality of care

From the perspective of From the perspective of individual patients and individual patients and

cliniciansclinicians Most patients would want and should Most patients would want and should

receive receive strongly recommendedstrongly recommended courses of actioncourses of action

The majority of people would want The majority of people would want weakly recommendedweakly recommended courses of courses of action, but many would not and clinicians action, but many would not and clinicians should be prepared to help patients make should be prepared to help patients make a decision that is consistent with their a decision that is consistent with their own valuesown values

The issue of priority setting (importance) The issue of priority setting (importance) is usually not important.is usually not important.

Questions about Questions about judgements about priorities judgements about priorities

(importance)?(importance)?

grade san sebastian 2008 05 12hjs

Health & Medicine

quality of evidence

overall quality of evidence

quality of evidence

health care research

italia agency

quality ahrq allergic

german agency

italy journal of infection