Brian Haynes

McMaster University

EBHC Workshop,

2013

The Health Information Research Unit at

McMaster has developed, and produces or

supports, several resources that will be

discussed (ACP Journal Club, Evidence

Updates, DynaMed, MacPLUS FS,

ACCESSSS).

(IP belongs to McMaster; most services free)

Agenda (negotiate!)

1. Intros – and your most frustrating or rewarding

teaching or personal experience in acquiring

“current best evidence for clinical practice”

2. Favorite sources of EB info?

3. Sign up: http://plus.mcmaster.ca/ACCESSSS

(suggest that you subscribe, but can use

ID/PW: guestn/guestn)

1. To define/personalize the nature of the

problems in translating evidence into clinical

practice

2. To explore a hierarchy of

evidence resources to support clinical care

decisions:

3. To demonstrate and practice how you can

use/teach the pyramid to keep up to date

(“push services”) and find current best

evidence (“pull services”)

1. Making decisions consistently

based on current best evidence ?

2. Keeping up to date with current

best evidence ?

Clinicians like you usually have more than 5 questions a day that can be answered by current best evidence

Green. Residents' medical information needs in clinic: are they being met? Am J Med 2000

But…

2,000 new articles every day

including 75 trials

and 11 systematic reviews

Bastian, Seventy-five trials and eleven systematic reviews a day: how will we ever keep up? PLoS Med. 2010

Bastian, Seventy-five trials and eleven systematic reviews a day: how will we ever keep up? PLoS Med. 2010

By the year 2020, 90% of clinical decisions

will be supported by accurate, timely, and

up-to-date clinical information and will reflect

the best available evidence.

IOM Roundtable on Evidence-Based Medicine

This can’t happen without excellent connections

between best evidence and decisions for and by

individual patients.

years since

graduation

r = -0.54

p<0.001

...

...

. .. . . .... .

....

....

... ..

... knowledge

of current

best care

100%

0%

50%

Choudhry, Fletcher and Soumerai,

Ann Intern Med 2005;142:260-73

94% of 62 studies found decreasing competence for at

least some tasks, with increasing physician age.

McMaster vs U of Toronto

years since

graduation

knowledge

of current

best care

100%

0%

50%

From Shin et al,

CMAJ 1993

The Slippery Slope

years since

graduation

...

...

. .. . . .... .

....

....

...

..

... knowledge

of best

evidence

100%

0%

50%

You could be the first generation to stay

on top and avoid the slippery slope…

…17 to 20 years

1. To define the nature of the problems in

translating evidence into clinical practice

2. To explore a hierarchy of

evidence resources to support clinical care

decisions:

3. To demonstrate how you can use the pyramid

to keep up to date (“push services”) and find

current best evidence *fast* (“pull services”)

Finding current best evidence

is becoming much easier.

Personalized EBM “push” Alerts?

eg EvidenceUpdates, ACPJournalWise

EBM “pull” Resources?

eg UTD, DynaMed, Best Practice, ACP PIER

EBM Federated Resources?

eg TRIP, ACCESSSS

Many evidence-based resources help you answer your questions

But no single resource provides all answers or is sufficiently updated

shows results in a pyramid of resources in a few seconds

showing results in a pyramid of resources in a few seconds

Summaries

Pre-appraised

research

Non-preappraised

research

Summaries

• are E-B clinical textbooks and

E-B guidelines

• integrate best evidence

for individual care topics

• provide actionable

recommendations

But which

summary to

choose?

Percentage of 60 ICD-10 codes

covered by each summary

Error bars: 95% Confidence Interval

Timeliness of updates of 10

online evidence-based texts Average time of updating of 60 topics

(randomly selected) as of July 2011

Ranging from

3.5 to 29 months

Summary Timeliness Breadth Quality

DynaMed 1 3 2

UpToDate 5 1 2

Micromedex 2 8 2

Best Practice 3 4 7

Essential Evidence Plus 7 7 2

First Consult 9 5 2

Medscape Reference 6 2 9

Clinical Evidence 8 10 1

ACP PIER 4 9 7

PEPID N/A 6 10

Rank order of 10 Online Summaries

No answer?

or more recent evidence?

Summaries

Pre-appraised

Research

synopses of systematic reviews

systematic reviews

synopses of studies

only a tiny proportion of all research is “ready for application”

only a tiny fraction of the “ready” research is “relevant” for a given clinician

only a tiny proportion of the “relevant” research for a given clinician is “interesting” in the sense of being something new, important, and actionable.

Haynes, B. The Best New Evidence for Patient Care. Ann Intern Med. 2008;148(10):JC3-2

~3300 articles

per year

Critical Appraisal Filters

Clinical Relevance Filters

+ 35,000 articles

screened per

year

~20 articles per clinician

Up to 99.9 %

‘Noise’ Reduction

Is‘information overload’ no longer

a valid excuse for ignorance?

Al-Khatib SM, Thomas L, Wallentin L, et al.

Outcomes of apixaban vs. warfarin by type

and duration of atrial fibrillation: results from

the ARISTOTLE trial. Eur Heart J. 2013 Apr 17.

4 internists

4 neurologists

4 cardiologists

4 hematologists

DISCIPLINE RELEVANCE NEW?

Internal Medicine 6/7 6/7

Neurology 6/7 6/7

Cardiology

5/7 6/7

Hematology 5/7 5/7

Internists

Neurologists

Cardiologists

Hematologists 7/6 6/5 7/7

is published in evidence-

based journals and resources

• is continuously sent to

you through MacPLUS,

ACCESSSS, ACPJW,

EvidenceUPDATES alerts

http://plus.mcmaster.ca/EvidenceUpdates

Methods • Select 4 leading evidence-based texts • Select 200 topics that are common to all

• Identify date of most recent update for each topic in each text

• Identify each article in MacPLUS that is more recent than text

update • Compare conclusions of MacPLUS studies with conclusions in

text

How often does MacPLUS give new and

different evidence than Summaries?

Best Practice DynaMed PIER UpToDate

No. (%) of 200 topics with potential for updates†

119 (60%)

46 (23%)

109 (55%)

104 (52%)

*Based on articles in MacPLUS since most recent topic

update which have conclusions that differ from topic conclusion(s).

† 1-way ANOVA p<0.01 across texts

Summaries

Pre-appraised

Research

Non-preappraised

Research

Why search

last?

• Studies are not critically appraised

• Searches yield large outputs with few relevant

studies in the first pages

• Current evidence is diluted and more difficult

to find, but less so if you use filters

Use Clinical Queries filters for questions of

• Prediction Guides

• Prognosis

• Etiology

• Diagnosis

• Therapy

Sensitivity/

Specificity PubMed Equivalent

broad

filter 99% / 70%

((clinical[Title/Abstract] AND trial[Title/Abstract])

OR clinical trials[MeSH Terms] OR clinical

trial[Publication Type] OR

random*[Title/Abstract] OR random

allocation[MeSH Terms] OR therapeutic

use[MeSH Subheading])

narrow

filter 93% / 97%

(randomized controlled trial[Publication Type]

OR (randomized[Title/Abstract] AND

controlled[Title/Abstract] AND

trial[Title/Abstract]))

E.g. CQ therapy filters

How many search terms to use?

Patients

Intervention

Comparison

Outcomes

Patient

Pre-appraised research

Works better for Summaries

E.g. COPD mucolytics

+ Intervention

and

Using Keep it simple!

Works better for unappraised evidence

COPD mucolytics

Combining more terms from your PICO elements

(exacerbation OR mortality)

looks for you in all resources and displays results in a pyramid

PICO question from the audience

• …

Did I miss any

important

evidence with

my search?

Is there any way I

could have

retrieved less

“junk”?

What is the best current evidence?

Alternatives

TRIP Database – EB search, guidelines, patient info, fulltext links, videos, news

SUMSearch – DARE at highest level

STAT!Ref

Google

Corporate collections – professional organizations; commercial publishers

Additional slides

Search strategies in PubMed

Search strategy = PICO query AND filters AND additional strategy

Search terms

Previous knowledge

Trial and Error MeSH Thesaurus

Boolean operators = OR, AND Combination

Initial question

P – Patient, Population

I – Intervention, Exposure…

C – Controls, Comparators

Search forward from:

Related Articles in

Pubmed

Screen the

bibliography of

relevant articles

Methodological filters:

Ex: Clinical Queries

Content filters (topic,specialty) O – Outomes

T – Type of question, design

AND

AND

Dynamic

Agoritsas & al. Sensitivity and Predictive Value of 15 PubMed Search Strategies to Answer Clinical Questions Rated Against Full Systematic Reviews. JMIR, 2012.

Balance between Sensitivity & Precision

Factors increasing Sensitivity:

• Many search terms for a similar concept, linked with OR.

• Truncated terms, Wildcards (e.g. tox*, wom?n)

• Synonyms (pressure sore, decubitus ulcer)

• Variant spelling (tumour, tumor)

• Explosion of MeSH terms

• Proximity search through «Related articles», Bibliography

Factors increasing Precision

• Concepts linked with AND (P) AND (I) AND (C) AND (O)

• Use of NOT Cochrane

• Limits

• Methodological Filters

• Content Filters

(P1 OR P2 OR P3) AND (I1 OR I2 … ) AND (C1 OR C2 …) AND (O1 OR O2 OR O3 …)

Population Intervention Comparators Outcomes

54

Question Effects of oral mucolytics agents in adults with stable

chronic bronchitis or COPD.

P stable chronic bronchitis

COPD (chronic bronchitis) OR COPD

I Mucolytic agents

oral mucolytic therapy mucolytics

C Placebo placebo

O number of exacerbations exacerbation

First try chronic bronchitis mucolytics

Example of PICO query

Less effective strategy (no filter)

Agoritsas & al. Sensitivity and Predictive Value of 15 PubMed Search Strategies to Answer Clinical Questions Rated Against Full Systematic Reviews. JMIR, 2012.

More effective strategy (with narrow filter)

Agoritsas & al. Sensitivity and Predictive Value of 15 PubMed Search Strategies to Answer Clinical Questions Rated Against Full Systematic Reviews. JMIR, 2012.

McMaster Online Rating of Evidence: >5000 clinicians

RELEVANCE