Holger SchünemannYngve Falck-Ytter

NICE, London December 11, 2006

GRADE – An introduction and GRADE – An introduction and workshopworkshop

Why bother about grading?Why bother about grading?

People draw conclusions about the– quality of evidence– strength of recommendations

Systematic and explicit approaches can help– protect against errors– resolve disagreements– facilitate critical appraisal– communicate information

However, there is wide variation in currently used approaches and grading can be misused or misunderstood

Which grading system?Which grading system?

EvidenceRecommendation

B Class I C+ 1 IV C

Organization AHA ACCP SIGN

Recommendation for use of oral anticoagulation in patients with atrial fibrillation and rheumatic mitral valve disease

Grading SystemsGrading Systems

Current profusion: can there be consensus?

GRADEGRADE

Grades of Recommendation

Assessment, Development and

Evaluation

About GRADEAbout GRADE

o Working group since 2000o Researchers/guideline developers with

interest in methodologyo Aim: to develop a common system for

grading the quality of evidence and the strength of recommendations that is sensible and to explore the range of interventions and contexts for which it might be useful*

o Evaluation of existing systems and reliability*

o Adopted by ATS, ACCP, ACP, WHO, Cochrane

*Grade Working Group. CMAJ 2003, BMJ 2004, BMC 2004, BMC 2005

GRADE Working GroupGRADE Working GroupDavid Atkins, chief medical officerDavid Atkins, chief medical officeraa Dana Best, assistant professorDana Best, assistant professorbb Peter A Briss, chiefPeter A Briss, chiefcc Martin Eccles, professorMartin Eccles, professordd Yngve Falck-Ytter, associate directorYngve Falck-Ytter, associate directoree Signe Flottorp, researcherSigne Flottorp, researcherff Gordon H Guyatt, professorGordon H Guyatt, professorgg Robin T Harbour, Robin T Harbour, quality and information quality and information

directordirector h h Margaret C Haugh, methodologistMargaret C Haugh, methodologistii David Henry, professorDavid Henry, professorjj Suzanne Hill, senior lecturerSuzanne Hill, senior lecturer jj Roman Jaeschke, clinical professorRoman Jaeschke, clinical professorkk Gillian Leng, guidelines programme directorGillian Leng, guidelines programme director ll Alessandro Liberati, professorAlessandro Liberati, professormm Nicola Magrini, directorNicola Magrini, directornn

James Mason, professorJames Mason, professordd Philippa Middleton, honorary research fellowPhilippa Middleton, honorary research fellowoo Jacek Mrukowicz, executive directorJacek Mrukowicz, executive directorpp Dianne O’Connell, senior epidemiologistDianne O’Connell, senior epidemiologistqq Andrew D Oxman, directorAndrew D Oxman, directorff Bob Phillips, associate fellowBob Phillips, associate fellowrr Holger J Schünemann, associate professorHolger J Schünemann, associate professorg,sg,s Tessa Tan-Torres Edejer, medical Tessa Tan-Torres Edejer, medical

officer/scientistofficer/scientisttt Helena Varonen, associate editorHelena Varonen, associate editoruu Gunn E Vist, researcherGunn E Vist, researcherff John W Williams Jr, associate professorJohn W Williams Jr, associate professorvv Stephanie Zaza, Stephanie Zaza, project directorproject directorww

a)a) Agency for Healthcare Research and Quality, Agency for Healthcare Research and Quality, USAUSA

b)b) Children's National Medical Center, Children's National Medical Center, USA USAc) Centers for Disease Control and Prevention, c) Centers for Disease Control and Prevention,

USAUSAd) University of Newcastle upon Tyne, d) University of Newcastle upon Tyne, UKUKe) German Cochrane Centre, e) German Cochrane Centre, GermanyGermanyf) Norwegian Centre for Health Services, f) Norwegian Centre for Health Services,

NorwayNorwayg) McMaster University, g) McMaster University, CanadaCanadah) Scottish Intercollegiate Guidelines Network, h) Scottish Intercollegiate Guidelines Network,

UKUKi) Fédération Nationale des Centres de Lutte i) Fédération Nationale des Centres de Lutte

Contre le Cancer, Contre le Cancer, FranceFrancej) University of Newcastle, j) University of Newcastle, AustraliaAustraliak) McMaster University, k) McMaster University, CanadaCanadal) National Institute for Clinical Excellence, l) National Institute for Clinical Excellence, UKUKm) m) Università di Modena e Reggio Emilia, Università di Modena e Reggio Emilia, ItalyItalyn)n) Centro per la Valutazione della Efficacia della Centro per la Valutazione della Efficacia della

Assistenza Sanitaria, Assistenza Sanitaria, ItalyItalyo) Australasian Cochrane Centre, o) Australasian Cochrane Centre, AustraliaAustralia p) Polish Institute for Evidence Based Medicine, p) Polish Institute for Evidence Based Medicine,

PolandPolandq) The Cancer Council, q) The Cancer Council, AustraliaAustraliar) r) Centre for Evidence-based Medicine, Centre for Evidence-based Medicine, UKUKs)s) National Cancer Institute, National Cancer Institute, ItalyItalyt) World Health Organisation, t) World Health Organisation, SwitzerlandSwitzerland u) Finnish Medical Society Duodecim, u) Finnish Medical Society Duodecim, FinlandFinland v) Duke University Medical Center, v) Duke University Medical Center, USA USA w) w) Centers for Disease Control and Prevention, Centers for Disease Control and Prevention,

USAUSA

Guideline development processGuideline development process

Prioritise Problems, establish panelPrioritise Problems, establish panel

Systematic ReviewSystematic Review

Evidence ProfileEvidence Profile

Relative importance of outcomesRelative importance of outcomes

Overall quality of evidenceOverall quality of evidence

Benefit – downside evaluationBenefit – downside evaluation

Strength of recommendationStrength of recommendation

Implementation and evaluation of guidelinesImplementation and evaluation of guidelines

GRADE

Guideline development processGuideline development process

Prioritise Problems, establish panelPrioritise Problems, establish panel

Systematic ReviewSystematic Review

Evidence ProfileEvidence Profile

Relative importance of outcomesRelative importance of outcomes

Overall quality of evidenceOverall quality of evidence

Benefit – downside evaluationBenefit – downside evaluation

Strength of recommendationStrength of recommendation

Implementation and evaluation of guidelinesImplementation and evaluation of guidelines

GRADE

Summary of Findings

GRADEGRADEQuality of evidenceQuality of evidence

The extent to which one can be confident that an estimate of effect or association is correct.

Although the degree of confidence is a continuum, we suggest using four categories:

– High– Moderate– Low– Very low

Judgements about the quality of Judgements about the quality of evidenceevidence

The quality of the evidence (i.e. our confidence) depends on:

study design (e.g. RCT, case-control study) study quality/limitations (protection against bias; e.g.

concealment of allocation, blinding, follow-up) consistency of results directness of the evidence including the

– populations (those of interest versus similar; for example, older, sicker or more co-morbidity)

– interventions (those of interest versus similar; for example, drugs within the same class)

– outcomes (important versus surrogate outcomes)– comparison (A - C versus A - B & C - B)

Moving quality downMoving quality down• poor (RCT) design, implementation

→randomization, blinding, concealment, follow-up, intention to treat principle, early stopping for benefit

• inconsistency• Indirect evidence

→patients, interventions, outcomes→A vs B, but have A to C, B to C

• sparse or imprecise data• reporting bias

Moving quality upMoving quality upObservational studies – high or moderate Observational studies – high or moderate

quality?quality?

Strong association→strong association: RR > 2 or RR < 0.5→very strong association: RR > 5 or RR < 0.2

Dose response relationship– bleeding risk associated with increasing INR (blood

thinning with warfarin)

Plausible confounders would have reduced the effect

For example, plausible explanatoryfactors that were not adjusted for in

studies comparing mortalityrates of for-profit and not-for-profit

hospitals would havereduced the observed effect. Thus, the evidence showing that for-profit

hospitals have a higher risk of mortality is more convincing

Quality of evidence

Study design Lower if Higher if

High Randomised trial

Moderate

Low Observational study

Very low Any other evidence

Study quality: -1 Serious limitations -2 Very serious limitations -1 Important inconsistency Directness: -1 Some uncertainty -2 Major uncertainty -1 Sparse data -1 High probability of Reporting

bias

Strong association: +1 Strong, no plausible confounders, consistent and direct evidence +2 Very strong, no major threats to validity and direct evidence +1 Evidence of a Dose response gradient +1 All plausible confounders would have reduced the

effect

Categories of qualityCategories of quality

High: Further research is very unlikely to change our confidence in the estimate of effect.

Moderate: Further research is likely to have an important impact on our confidence in the estimate of effect and may change the estimate.

Low: Further research is very likely to have an important impact on our confidence in the estimate of effect and is likely to change the estimate.

Very low: Any estimate of effect is very uncertain.

Judgements about the overall Judgements about the overall quality of evidencequality of evidence

Most systems not explicit

Options:– Benefits– Primary outcome– Highest – Lowest

Based on lowest of all the critical outcomes

Beyond the scope of a systematic review

Levels of evidence: SIGNLevels of evidence: SIGN

Grading of recommendationsGrading of recommendations

Some problemsSome problems Oversimplified hierarchy based on study design

– Inadequate consideration of other factors – Distinction between study design, quality of

evidence and strength of recommendation blurred Systematic reviews included in the hierarchy

– rather than viewed as the basis for making judgements

Expert opinion included in the hierarchy – rather than explicitly considering the evidence

underlying expert opinions Balance between desirable and undesirable effects

– Not reflected in the grade– Not considered transparently

Inadequate consideration of other factors that affect confidence in a recommendation

Grading misused when recommendation not separated from the quality of the evidence

ExampleExample

WHO Avian Influenza guidelines - key clinical questions:Population: H5N1 infected individuals

Intervention: Neuraminidase Inhibitors, M2 Inhibitors, other pharmacological agents

Comparison: no therapy/alternative managementOutcomes: ?

ExampleExample

WHO Avian Influenza guidelines - key clinical questions:

Population: H5N1 infected individualsIntervention: Neuraminidase Inhibitors,

M2 Inhibitors, other pharmacological agents Comparison: no therapy/alternative

managementOutcomes: Mortality?, Hospitalizations?

Resource use?, Adverse outcomes?

Clinical Question RefinementClinical Question Refinement

Survey of panel membersOutcome definition:• List of potential outcomes circulated • Feedback from panel• Concealed rating of importance• Consultation with Cochrane

Consumers network

Judgment about the relative importance for each endpoint (scale from 9 to 1): • 7 – 9: the endpoint is critical for decision making.• 4 – 6: the endpoint is important but not critical.• 1 – 3: the endpoint is not important.

Only critical (and important) outcomes includedTreatment: mortality, duration of hospitalization, incidence of lower respiratory tract complications, antiviral drug resistance and serious adverse events.

Outcomes/endpointsOutcomes/endpoints

Evidence Summary and Evidence Summary and Quality RatingsQuality Ratings

• Draft summaries sent to panel members for review and identification of gaps

• Restricted additional evidence at meeting• Evidence profiles using GRADE

methodology and GRADEpro software (v1.12)

Evidence ProfilesEvidence Profiles

No of studies(Ref)

Design Limitations Consistency DirectnessOther

considerationsOseltamivir Placebo

Relative(95% CI )

Absolute(95% CI )

Mortality

0 - - - - - - - - - 9

5(TJ 06)

Randomised trial

No limitations One trial only Major uncertainty

(-2)1

Imprecise or sparse data (-1)

- - OR 0.22(0.02 to 2.16)

- Very low

6

0 - - - - - - - - - - 7

5(TJ 06)

Randomised trial

No limitations One trial only Major uncertainty

(-2)1

Imprecise or

sparse data (-1)2

2/982(0.2%)

9/662(1.4%)

RR 0.149(0.03 to 0.69)

- Very low

8

53

(TJ 06)(DT 03)

Randomised trials

No limitations4 Important inconsistency

(-1)5

Major uncertainty

(-2)1

- - - HR 1.303

(1.13 to 1.50)

- Very low

5

26

(TJ 06)

Randomised trials

No limitations -7 Major uncertainty

(-2)1

None - - - WMD -0.738

(-0.99 to -0.47)

Low

4

0 - - - - - - - - - - 4

0 - - - - - - - - - - 7

09 - - - - - - - - - - 7

311

(TJ 06)

Randomised trials

No limitations -12 Some uncertainty

(-1)13

Imprecise or

sparse data (-1)14

- - OR range15

(0.56 to 1.80)

- Low

0 - - - - - - - - - - 4

I mportance

Summary of findings

Cost of drugs

Outbreak control

Resistance

Serious adverse effects (Mention of significant or serious adverse effects)

Minor adverse effects 10 (number and seriousness of adverse effects)

Viral shedding (Mean nasal titre of excreted virus at 24h)

Duration of disease (Time to alleviation of symptoms/median time to resolution of symptoms – influenza cases only)

Duration of hospitalization

LRTI (Pneumonia - influenza cases only)

Healthy adults:

Hospitalisation (Hospitalisations from influenza – influenza cases only)

Quality assessmentNo of patients Effect

Quality

Oseltamivir for treatment of H5N1 infection:

-

-

Evidence SummaryEvidence Summary

Summary of findings • No clinical trial of oseltamivir for treatment of

H5N1 patients.• 4 systematic reviews and health technology

assessments (HTA) reporting on 5 studies of oseltamivir in seasonal influenza. • Healthy adults, high risk adults or children for treatment

of seasonal influenza• Duration of treatment up to 5 days• Several countries in the northern and southern

hemispheres (no resource poor countries)• 3 published case series describing H5N1 patients

treated with oseltamivir. • Many in vitro and animal studies.• < 400 cases; mortality > 50% worldwide

Case scenarioCase scenario

A 13 year old girl who lives in rural Indonesia presented with flu symptoms and developed severe respiratory distress over the course of the last 2 days. She required intubation. The history reveals that she shares her living quarters with her parents and her three siblings. At night the family’s chicken stock shares this room too and several chicken had died unexpectedly a few days before the girl fell sick.

Who would recommend oseltamivir for this patient or similar patients?

YES (pink) No (green)

RecommendationsRecommendations

Recommendation: In patients with confirmed or strongly suspected infection with avian influenza A (H5N1) virus, clinicians should administer oseltamivir treatment as soon as possible (????? recommendation, very low quality evidence).

RecommendationsRecommendations

Recommendation: In patients with confirmed or strongly suspected infection with avian influenza A (H5N1) virus, clinicians should administer oseltamivir treatment as soon as possible (strong recommendation, very low quality evidence).

Comparison of GRADE and other Comparison of GRADE and other systemssystems

Explicit definitions Explicit, sequential judgements Components of quality defined Quality by outcome and overall quality Relative importance of outcomes Balance between health benefits and

harms Balance between incremental health

benefits and costs Evidence profiles International collaboration Consistent judgements? Communication?

GRADE ProfilerGRADE Profiler