local clinical recommendation: the grefo program nicola magrini, lucia magnano nhs ceveas nhs centre...

Local clinical recommendation: the GREFO program

Nicola Magrini, Lucia MagnanoNHS CeVEAS

NHS Centre for the Evaluation of the Effectivenessof Health Care, Modena, Italy

From Scientific Research to Clinical Practice Rome, November 25, 2011

Contents of the presentation

• New cancer drugs recommendations

• Why GRADE could help …

• A three pillar method: the GRADE system to evaluate quality of evidence and define the strength of a recommendation

• An example of grade application: the GREFO program

• Conclusions: what we have learnt about how best to formulate a weak recommendation

New Cancer Drugs approval:why bother?

• New cancer drugs are often approved with limited evidence (only 1 or 2 RCTs in selected populations with limited follow-up, …)

• The benefits are often limited (few weeks of increased survival or progression-free survival) and small if compared to clinical expectations

• Their cost is very high and not proportional to actual benefits

Emilia Romagna New Cancer Drugs recommendations using GRADE

• Started as a research project with the aim of applying a more transparent grading system (GRADE) to cancer drugs recommendations (2006 – 2007)

• GRADE proved feasible and useful to use as a common framework with the oncologists research community – though the time requested for the production of each

recommendation was quite long (12 months)– the recommendations did not have an indicator of expected use

• The research project was then transformed into rapid recommendations (3-4 months) with explicit indicators of expected use to increase their applicability

CORCommissione

Oncologica Regionale

CRFCommissione Regionale del

Farmaco

Drugs included in the regional formulary by

CRF

Drugs NOT included in the

regional formulary by

CRF

Drugs with a not stated benefit/risk ratio

(it is necessary an assessment by a panel and

formulation of recommendations)

GREFOGruppo Regionale

Farmaci Oncologici

recommendations

3 months

Emilia Romagna New Cancer Drugs local Recommendations

Panel

JCO editorial comment

• In applying GRADE, the most common recommendation categories one might expect in the oncology field are “Probably use it” or “Probably do not use it” . (i.e. weak recommendations)

• […]

• Just how one interprets these recommendation categories clinically is not known.

Brouwers MC, Somerfield MR; Browman GPA for Effort: Learning for the application of the GRADE approach

to cancer guideline development.JCO 2008

Why Why GRADE?GRADE?Too many systems?… probably Too many systems?… probably

yesyes

EvidenceEvidenceRecommendationRecommendation

BB Class IClass I

C+ C+ 11

IVIV CC

OrganizationOrganization

AHAAHA

ACCPACCP

SIGNSIGN

Recommendation for use of oral anticoagulation Recommendation for use of oral anticoagulation in patients with atrial fibrillation and rheumatic in patients with atrial fibrillation and rheumatic mitral valve diseasemitral valve disease

Why using GRADEGRADE is much more than a rating system • offers a transparent and structured process for

developing and presenting summaries of quality of evidence

• provides guideline developers with a comprehensive and transparent framework for carrying out the steps involved in developing recommendations

• specifies an approach to framing questions, choosing outcomes of interest and rating their importance, evaluating the evidence, and incorporating evidence with considerations of values and preferences of patients and society to arrive at recommendations

Facts versus Decisions(Quality of Evidence versus Strength of Recommendations)

Decision

Facts“Evidence”

Values

Feasibility Costs

Very low, Low, Moderate, or High Quality Evidence

Weak or StrongRecommendation

(for or against)Paul Glasziou, 2007

GRADE UptakeGRADE UptakeAgencia sanitaria regionale, Bologna, Italia Agency for Health Care Research and Quality Allergic Rhinitis and Group American College of Cardiology FoundationAmerican College of Chest PhysiciansAmerican College of Emergency PhysiciansAmerican College of PhysiciansAmerican Endocrine Society American Society of Gastrointestinal

EndoscopyAmerican society of Interventional Pain

PhysicianAmerican Thoracic Society (ATS)BMJ Clinical Evidence Canadian Agency for Drugs and Technology Centers for Disease ControlCochrane Collaboration EBM Guidelines Finland European Respiratory SocietyEuropean Society of Thoracic SurgeonsEvidence-based Nursing Sudtirol, ItalyFinnish Office of Health Technology

AssessmentGerman Agency for Quality in MedicineInfectious Disease Society of America

Japanese Society of Oral and Maxillofacial Radiology

Journal of Infection in Developing CountriesKidney Disease International Guidelines

Organization National and Gulf Centre for Evidence-based

MedicineNational Institute for Clinical Excellence (NICE)National Kidney FoundationNorwegian Knowledge Centre for the Health

ServicesOntario MOH Medical Advisory SecretariatPanama and Costa Rica National Clinical Guidelines

Polish Institute for EBMScottish Intercollegiate Guideline Network (SIGN)Society of Critical Care MedicineSociety of Pediatric Endocrinology Society of Vascular SurgerySpanish Society of Family Practice (SEMFYC) Stop TB Diagnostic Working GroupSurviving sepsis campaign Swedish Council on Technology Assessment in

HealthSwedish National Board of Health and Welfare University of Pennsylvania Health System for

EBHC UpToDate World Health Organization (WHO)

WHO guideline development processes

update 2010

1. Scoping the document: reasons for choosing the topic, problems with existing guidelines, variations and gaps,

2. Group composition (or consultations)

3. Conflict of interest

4. Formulations of the questions and choice of the relevant outcomes

5. Evidence retrieval, evaluation and synthesis (balance sheet, evidence

table)

6. Benefit/risk profile: integrating evidence with values and preferences,

equity and costs

7. Formulation of the recommendations

8. Implementation and evaluation of impact

9. Research needs or areas of further research

10. Peer-review process and updating

Title, responsible person, WHO Department - responsible of the clearance process, WHO Departments involved, CC

involved,

Standards for evidence: GRADE system

Reporting standard and process

Reporting standard and process

GRADE: a 3 pillars approach

1. Formulate the question, choose and rate your outcomes of interest and perform a systematic review (quality of evidence)

2. Risk benefit evaluation, consider patients values and preferences and also resource use and feasibility

3. Strength of the recommendation

Formulate the question• Specify your question in an answerable way

– A classic example (WHO, 2006): Should oseltamivir be used for treatment of patients hospitalised with avian influenza (H5N1)?

– An example relevant to GPs: What is the preferred drug of choice for the treatment of hypertension (in the elderly)?

– And the preferred associations of drugs?

– New cancer drugs (2009): In patients with NSLC histology non squamous type, is a first line therapy with pemetrexed plus cisplatin recommended?

GRADE - Outcomes choice explicit judgements should be made about

which outcomes are critical, which ones are important but not critical, and which ones are unimportant and can be ignored.

studies using surrogate outcomes generally provide weaker evidence than those using outcomes that are important, and these only should be included when evidence for important outcomes is lacking.

Outcomes should be discussed and rated indipendently form the fact of being considered in the available studies

Schünemann HG et al. Health Res Policy Syst 2006:4:18

Figure 1: Hierarchy of outcomes according to their patient-importance to assess the effect of enteral supplement nutrition for geriatric patientswith bed sores

Nutritional status 4

Importanceof endpoints

Microcirculationof the wound 1

2

Energy supply 3

5

Function 6

Quality of life 7

Healing of the 8bedsore

Mortality 9

Criticalfor decision making

Important, but not critical fordecision making

Not patient-important

Choice and Choice and rating of rating of

important important outcomesoutcomes

… example … example for pts with for pts with bed soresbed sores

Study design is importantStudy design is important

- Early systems of grading the quality of evidence focused almost exclusively on study design

- Randomised trials provide, in general, stronger evidence than observational studies::

– RCTs start at High QualityRCTs start at High Quality– Observational studies start at Low Observational studies start at Low QualityQuality

- However, other factors may decrease or increase the quality of evidence

Quality assessment criteriaQuality assessment criteria: : the big movethe big move

Factors that may decrease the Factors that may decrease the quality of evidencequality of evidence

Study limitationsStudy limitations (risk of bias)(risk of bias)InconsistencyInconsistency among studiesamong studiesIndirectnessIndirectness of evidence of evidence Imprecise results Imprecise results Reporting bias / publication biasReporting bias / publication bias

Risk of Bias (study limitations)Risk of Bias (study limitations)

• well establishedwell established– concealment– intention to treat principle observed– blinding– completeness of follow-up– Choice of comparator (standard/optimal

treatment

• more recentmore recent– early stopping for benefit– selective outcome reporting bias

Factors that may decrease the quality of evidence

• Here, I would like to stress the 2 new criteria explicitly introduced by GRADE:– Inconsistency among study results … to be applied

also when just one clinical trial is available– Indirecteness / transferability / applicability

problems: populations, interventions, comparisons, outcomes

• These two criteria are particularly relevant to general practice, in particular indirectness … both could be used to identify evidence gaps and to support more and more relevant indipendent research

Are these results consistent?

Prendiville WJ et al. Cochrane Database Syst Rev 2000, Issue 3

Quality of evidence

Degree of confidence that an estimate of

effect or an association is correct

Specify your question in an answerable way (PICO)

Clinical QuestionIn patients with NSCLC histology non squamous type, is a first line therapy with pemetrexed plus cisplatin recommended? 1.Young/not elderly persons (<70yr) who are fit (PS=0-1)2.Elderly persons who are (>70yr) who are fit (PS=0-1)3.Persons who are unfit (PS>2)

1.Outcomes selection and rating

Beneficial outcomes

Risk outcomesRisk releted to

bevacizumab

outcomes

Outcomes selection and rating

Studies Studies retrievedretrieved for for pemetrexedpemetrexed

Pemetrexed – table of evidence

Subgroups analysis prespecified, but histologic groups was not considered in the stratification of randomization

Quality of evidence

Risk of BiasRisk of Bias

indirectenessindirecteness

Pemetrexed – Quality of evidenceConsistencyConsistency

ImprecisionImprecision

Choice of comparatorChoice of comparator

Publication biasPublication bias

Pemetrexed: Quality of evidence by subgroups

Downgrading (-1 ) for risk of biasDowngrading (-1 ) for risk of biasQuality: moderateQuality: moderate

1.Young/not elderly persons (<70yr) who are fit (PS=0-1)


Downgrading (-2) forDowngrading (-2) for- risk of bias - risk of bias - IndirectnessIndirectness

Quality: LowQuality: Low

2. Elderly persons who are (>70yr) who are fit (PS=0-1)

No studiesNo quality assessment

3. Persons who are unfit (PS>2)


GRADE: a 3 pillars approach




GRADE Determining the benefit risk profile: positive/uncertain/unfavourable

Factors Impact on the strength of a recommendation

Balance between desirable and undesirable effects

Larger the difference between the desirable and undesirable effects, more likely a favourable benefit But differences can arise depending on the severity of adverse events

Values and preferences

More variability in values and preferences, or more uncertainty in values and preferences, more likely an unfavourable profile.

Costs (resource use)

Higher the costs of an intervention – that is, the more resources consumed – less likely a favourable profile.

Patients <70 yr

Pemetrexed: benefit/risk profile

Elderly patients >70 yr

Unfit patients

positive

uncertain

unfavourable

GRADE: a 3 pillars approachGRADE: a 3 pillars approach




Strength of recommendationStrength of recommendation

The degree of confidence that the desirable effects of adherence to a recommendation outweigh the undesirable effects

Desirable Desirable effectseffects•health benefitshealth benefits•less burdenless burden•savingssavings

Undesirable Undesirable effectseffects•harmsharms•more burdenmore burden•costscosts

Categories of recommendationsCategories of recommendations

Although the degree of confidence is a continuum, we suggest using two categories: strong and weak.

Strong recommendation: the panel is confident that the desirable effects of adherence to a recommendation outweigh the undesirable effects.

Weak recommendation: the panel concludes that the desirable effects of adherence to a recommendation probably outweigh the undesirable effects, but is not confident.

Recommend

Suggest

Pemetrexed Recommendations

Pemetrexed+ cisplatin lung cancer, pts <70 yrs

1st line treatmentweak positive recomm.

Quality of evidence: moderate(Risk of bias for subgroups

analysis)

Between 30-50% of patients treated with a 2 drugs regimen should receive pemetrexed+

cisplatin

Benefit/risk profile: positiveshort explanation of why the Panel

suggested positivity

Examples of Examples of recommendations using recommendations using

GRADEGRADEA flexible methodA flexible method

For opioid agonist maintenance treatment, most patients should be advised to use methadone in adequate doses in preference to buprenorphine. – Strength of recommendation – StrongStrength of recommendation – Strong– Quality of evidence – HighQuality of evidence – High

WHO Guidelines for the Psychosocially Assisted WHO Guidelines for the Psychosocially Assisted Pharmacological Treatment of Opioid Dependence Pharmacological Treatment of Opioid Dependence

(2009)(2009)

On average, methadone maintenance doses should be in the range of 60–120 mg per day. – Strength of recommendation – StrongStrength of recommendation – Strong– Quality of evidence – LowQuality of evidence – Low

Considerations on strong and weak recommendations

Life is easier for methodologists and clinicians with strong recommendations

Weak recommendations are increasingly important and often cover grey or controversial areas. Their formulation should be as explicit as possible in terms of real-life implications

The potential range of application of a weak recommendation should be operationalised (what does “suggest” or “could be used” mean?) … to be applicable and monitored to be applicable and monitored ……

Strong / weak recommendation

Quality assessment

criteria

Ratings of outcomes

Quality assessment

criteria

Quality of evidence:

estimates of benefits &

harms and risk of bias,

directness, …

Risk-benefit profile evaluation for different subgroups

Evidence

Emilia Romagna New Cancer Drugs Rec.

GRADE steps and polls• We (in Italy) adapted GRADE by:

– Rating/voting the evaluation of the benefit-risk profile (favourable, uncertain, unfavourable)

– Defining an expected use indicator for each recommendation– Incorporating the results of polls in the final document

+ expected use

Emilia Romagna New Cancer Drugs Recommendations

Overview of recommendation strength 1/2


Overview of recommendation strength 2/2

SP:

Strong positive

2

WP:

Weak positive

10

WN:

Weak negative

10

SN:

Strong negative

12

Implications ofstrong and weak

… and expected use

Strength of recomendations and expected use

StrenghtDefinition and implications

Expected use

Strong positive

The drugs/interventions should offered to the vast majority of patients and could be used as an indicator of good quality of care. It doesn’t mean however all patients should receive them.

Always

Strong negative

It should not be used neither routinely nor for a subgroup. Only in few very selected cases it should be documented its use since the benefit/risk balance is negative and potential alternative are preferable.

Never



Expected use

Strong positive

The drugs/interventions should offered to the vast majority of patients and could be used as an indicator of good quality of care. It doesn’t mean however all patients should receive …

Almost always

Strong negative


Exceptional cases



Expected use

Strong positive


Almost always

Strong negative


Exceptional cases

When a recomendation can’t be strong, can we formulate a weak recomendation in a way that will be clearly applicable and monitored?



Expected use

Strong positive


Almost always

Weak positive

It has the wider range of uncertainty since it could mean only for a minority of patients (30%) or for a good proportion of them (50-60%). It is necessary to inform patients of the expected benefits and risks (and their magnitude), explore patients values and discuss potential alternative treatments.

30-60%

Weak negative

In selected cases or a defined minority. The decision should go along with a detailed information to patient of the benefit risk (magnitude), patients valueds and expectations and disucss potential alternative treatments.

5-30%

Strong negative


Exceptional cases

Recommendations and expected adoption rate

StrengthDefinition and implications

Expected adoption

rate

Strong positive


> 60-70%

Weak positive

It has the wider range of uncertainty since it could mean only for a minority of patients (30%) or for a good proportion of them (50-60%). It is necessary to inform patients of the expected benefits and risks (and their magnitude), explore patients values and discuss potential alternative treatments.

30-60%

Weak negative

In selected cases or a defined minority. The decision should go along with a detailed information to patient of the benefit risk (magnitude), patients valueds and expectations and discuss potential alternative treatments.

5-30%

Strong negative


< 5%

Recommendations and expected adoption rate

** for 5 reccomendations it was not possible to define an expected adoption rate

• Though demanding, GRADE criteria offered a comprehensive framework to establish a good dialogue with the oncologists community

• A “good enough”/common definition of quality of evidence was accepted and the regulatory attitude (Yes/No) abandoned

• The importance of openly sharing the judgments on quality of evidence, the recommendations and also the expected adoption rates was put forward by the Panel, expecially toward specialty society


What have we learnt (1/3)

• The higher level of uncertainty is with weak recommendations, probably the one with the higher degree of freedom of interpretatation being the weak positive

• Weak should not be synonimous of vague and imprecise, so we asked the Panel to define a recommended adoption rates (out of a denominator represented by all eligible patients)



• Since the overall process is quite analytical and fragmented, the final document are probably too lengthy and awkward to read

• The final format of the recommendation may be more effective by adding a short text featuring:– an explicit assessment of the magnitude of expected benefits and

harms – A judgment of the overall benefit/risk profile and other

considerations of values and preferences and what to tell patients



Grazie per l’attenzioneGrazie per l’attenzione

local clinical recommendation: the grefo program nicola magrini, lucia magnano nhs ceveas nhs centre...

Documents