systematic reviews: a primer for librarians mark maceachern, mlis taubman health sciences library,...

SYSTEMATIC REVIEWS: A PRIMER FOR LIBRARIANS

Mark MacEachern, MLIS

Taubman Health Sciences Library, University of Michigan

Outline

Introductions What is a Systematic Review? Librarian Roles Protocols & Registries Databases & Search Strategies Data Management Reporting Wrap Up & Other Stuff

Introduction

Name Experience with systematic reviews What do you want to get from this?

Introduction

Name Experience with systematic reviews What do you want to get from this?

Ask questions throughout! I have no conflicts of interest to report Some overlap with my MLA webinars and the

UMich workshop Discussion-based

What is a systematic review?


From the Cochrane Handbook (1.2.2):

A systematic review attempts to collate all empirical evidence that fits pre-specified eligibility criteria in order to answer a specific research question. It uses explicit, systematic methods that are selected with a view to minimizing bias, thus providing more reliable findings from which conclusions can be drawn and decisions made (Antman 1992, Oxman 1993).


Common Elements: A clearly stated set of objectives with pre-defined

eligibility criteria for studies A reproducible methodology A systematic search that attempts to identify all

studies that would meet the eligibility criteria An assessment of the validity of the findings of the

included studies, for example through the assessment of risk of bias

A systematic presentation, and synthesis, of the characteristics and findings of the included studies

Source: Cochrane Handbook, 1.2.2


Source: Users’ Guides to the Medical Literature (JAMA Evidence)


Source: Gagnier JJ. Introduction to Systematic Reviews & Meta-analyses.

In short, it is likely a narrative review if: The question is unclear from the title, abstract, or

introduction There is no methods section


In short, it is likely a narrative review if: The question is unclear from the title, abstract, or

introduction There is no methods section

Types of Systematic Reviews Qualitative

Results summarized narratively, no stats, methods rigorous to minimize bias

Quantitative Results statistically combined, methods rigorous to

minimize bias



Steps1. Formulate a question2. Define inclusion/exclusion criteria3. Locate studies4. Select studies5. Assess study quality6. Extract data7. Analyze and present results8. Interpret results



Source: Sackett DL. Evidence-based medicine: how to practice and teach EBM.


Team Composition One person

Possible, but more prone to bias Two people

May want a third to help with disagreements Three plus

Clinician(s), statistician, librarian Too many can become problematic



Source: Institute of Medicine. Standards for Initiating a Systematic Review.

Team Composition


Source: Khangura S. Evidence summaries: the evolution of a rapid review approach.

Rapid Reviews


Source: EPC Methods: An Exploration of Methods and Context for the Production of Rapid Reviews

Methodological Approaches

Search fewer databases

Limited use of grey literature

Restrict types of studies included (eg., English only, most recent 5 yrs)

Relying on existing reviews

Limited full-text review

Limited dual review for study selection

See report for more

Rapid Reviews

RAPID FIRE SCENARIOS

In groups of 3, consider the following scenarios and identify solutions to these problems/issues that commonly arise in SR work.

A researcher comes to you set on doing a systematic review for a specific question. You don’t believe that the question lends itself to a systematic review. How do you proceed?

RAPID FIRE SCENARIOSIn groups of 3, consider the following scenarios and identify solutions to these problems/issues that commonly arise in SR work.

A resident reaches out to you for help conducting a systematic review search. She tells you that the project is done, the manuscript is finished, and all she needs is a search that captures every study included in her paper. Is this a problem? How do you proceed here?

Librarian Roles

IOM Standards for Initiating a Systematic Review Standard 2.1 Establish a team with appropriate expertise

and experience to conduct the systematic review 2.1.3 Include expertise in searching for relevant

evidence Standard 2.5 Formulate the topic for the systematic

review 2.5.1 Confirm the need for a new review

Standard 2.6 Develop a systematic review protocol 2.6.4 Describe the search strategy for identifying

relevant evidence

Source: Institute of Medicine. Standards for Initiating a Systematic Review

Librarian Roles

IOM Standards for Finding and Assessing Individual Studies Standard 3.1 Conduct a comprehensive systematic

search for evidence 3.1.1 Work with a librarian or other information specialist

trained in performing systematic reviews to plan the search strategy

3.1.2 Design the search strategy to address each key research question

3.1.3 Use an independent librarian or other information specialist to peer review the search strategy

3.1.4 – 3.1.9 Search various databases and update searches


Librarian Roles

IOM Standards for Finding and Assessing Individual Studies Standard 3.2 Take action to address potentially biased

reporting of research results 3.2.1 Search grey literature databases, clinical trial

registries, and other sources of unpublished information about studies

3.2.4 – 3.2.6 Handsearch, web search, and non-English search

Standard 3.4 Document the search 3.4.1 Provide a line-by-line description of the search strategy


Librarian Roles

Common Errors in Published SR Search Strategies

Spelling errors Misuse of MeSH and keywords

No spelling variants / truncations

Bad exploding

Incorrect use of Boolean and line numbers

Redundancy in search terms

Translating strategies poorlySource: Sampson, M. Errors in search strategies were identified by type

and frequency.

Source: Institute of Medicine. Standards for Systematic Reviews (full report)

Librarian Roles

Authorship

Librarian Roles

Source: ICMJE. Defining the Role of Authors and Contributors

Authorship – ICMJE Authorship Criteria1. Why Authorship Matters?

Credit & responsibility for contributions

2. Who Is an Author? Substantial contributions to all aspects of article Drafting and revising manuscript Final approval of manuscript Agreement to be accountable for all aspects of the

work

3. Non Author Contributions If you don’t meet those 4 criteria, you should be listed

in the acknowledgements

Librarian Roles

Acknowledgement (named)

Librarian Roles

Acknowledgement (unnamed)

Librarian Roles

Grants Co-PI, Collaborator, Consultant % effort, compensation Good source for demonstrating value



Is authorship on SRs appropriate? If so, in what instances? If not, why not?



You have been asked to participate in a systematic review project and would like to receive appropriate recognition for your contributions. How would you broach this subject with the project lead?

Protocols & Registries

A protocol is a plan or set of steps to be followed in a study. A protocol for a systematic review should describe the rationale for the review; the objectives; and the methods that will be used to locate, select and critically appraise studies, and to collect and analyse data from the included studies.

Source: Cochrane Community, http://community.cochrane.org/faq/general#t86n544 Courtesy: W Townsend


Source: Liberati A. The PRISMA Statement…Explanation and Elaboration.

Prevents post-hoc changes, but acknowledges that changes are permissible. Don’t feel like it sets everything in stone.

Prevents duplication of effort


Educate • Ensure consults know what goes into an SR. Filters out those who do not.

• Proper SR methodologies• Realistic timelines, realistic questions• Scoping review first• Technology• Reference IOM, Cochrane, PRISMA


Educate • Ensure people know what goes into an SR. Filters out those who don’t.


Inform • Your search• Helps elicit important info from project lead

• PICO• Eligibility criteria• Possible limits• Appropriate databases


Educate • Ensure people know what goes into an SR. Filters out those who don’t.


Inform • Your search• Helps elicit important info from project lead

• PICO• Eligibility criteria• Possible limits• Appropriate databases

Distribute • Makes the project team accountable• Facilitates team functioning• Helps with the writing


Source: Rethlefsen ML. I Want to do a Systematic Review.


Source: Rethlefsen ML. Librarian co-authors correlated with higher quality reported...


Realistic timelines

Source: Cochrane Handbook, 2.3.b

In 2008: 15 months (Sampson M, 2008)

In 2013: 8 months (Beller EM, 2013 )


Why register? Provides transparency Reduce risk of multiple reviews on the same

question Reduce publication bias Safeguard against reporting biases Identify similar reviews in the process of being

completed

Source: Liberati A. The PRISMA Statement…Explanation and Elaboration.


Source: Sun G. Systematic review of quality improvement initiatives in the intensive care setting


Review title and timescale

Review title*

Original language title

Anticipated or actual start date*

Anticipated completion date*

Stage of review at time of registration*

Review team details

Named contact*

Named contact email*

Named contact address

Named contact phone number

Review team members and their organisational affiliations

Organisational affiliation of the review*

Funding sources/ sponsors*

Conflicts of interest*

Collaborators

Review methods

Review question(s)*

Searches*

URL to search strategy

Condition or domain being studied*

Participants/ population*

Intervention(s), exposure(s)*

Comparator(s)/ control*

Types of study to be included initially*

Context

Primary outcome(s)*

Secondary outcomes*

Data extraction (selection and coding)

Risk of bias (quality) assessment*

Strategy for data synthesis*

Analysis of subgroups or

subsets*

General information

Type of review

Language

Country

Other registration details

Reference and/or URL for published protocol

Dissemination plans

Keywords

Details of any existing review of the same topic by the same authors

Review status*

Any other information

Link to publication of final report

Source: PROSPERO, http://www.crd.york.ac.uk/PROSPERO/



You and project lead have a discussion about protocols and are trying to decide on whether to have one or not. She says that most don’t have one, so why should we bother? This gets you thinking about whether they’re worth the effort? Do you care whether the project has one or not?

Databases & Search Strategies

Why is the search important? A high-quality lit search is essential for a successful

meta-analysis It is from the search results that data is gathered for

analysis Failure to locate important studies can significantly

affect results Remember the goal is to capture every relevant

study Important to report search so your methodology

can be reproduced

Databases & Search Strategies Expect:

1000s of results To search multiple databases

Ovid MEDLINE, EMBASE, PubMed, Scopus, etc. The search process to take awhile To publish search strategy and search methodology


Core Secondary Others

• MEDLINE• PubMed• Ovid

MEDLINE• EMBASE• CENTRAL

• Scopus• Web of Science• CINAHL• Google Scholar• ClinicalTrials.go

v

• PsycINFO• ERIC• Sociological

Abstracts• ABI Inform• Proquest

Theses and Dissertations

• Etc.

• Cochrane Handbook (10.3.1)– Search multiple sources– Search unpublished studies– Search trial registries

Databases & Search Strategies Supplemental strategies

Examine the references of included studies and relevant reviews

Use ‘cited by’ features Web of Science, Google Scholar

Contact authors, companies, orgs, societies, etc. Hand search important journals

By consensus, Impact Factor, most frequently appearing journals in Endnote/etc

Search for ongoing studies Clinicaltrials.gov ; controlled-trials.com (ISRCTN);

conference abstracts

Databases & Search Strategies Grey Literature

“It is usually understood to mean literature that is not formally published in sources such as books or journal articles.” (Cochrane Handbook, 6.2.1.8)

Types Conference abstracts (Web of Science, Scopus) Clinical trials (ClinicalTrials.gov) Government reports, documents (.gov, CABI) Dissertations (ProQuest Dissertations and Theses) Unpublished manuscripts


Pros Cons

• Combats publication bias• Process can help find

missed studies• Strengthens methodology

• Search functionality of grey lit sources tends to be basic

• Difficult sell value to team• Increases retrieval (already

large)• Adds time to project

(contact authors, etc.)• Difficult to export results

Grey Literature


Pros Cons

• Combats publication bias• Process can help find

missed studies• Strengthens methodology

• Search functionality of grey lit sources tends to be basic

• Difficult sell value to team• Increases retrieval (already

large)• Adds time to project

(contact authors, etc.)• Difficult to export results

Grey Literature

Cook DJ, 1993 – Unpublished studies should not be systematically excluded

Hopewell S, 2002 – Useful for RCTs. Combine with MEDLINE

Egger M, 2003 – Difference b/w comprehensive searches and less comprehensive is small

Martin JL, 2005 – Study quality is important and grey lit might not meet the standard

Armstrong R, 2005 – Handsearching is valuable to pick up supplements

Benzies KM, 2006 – Grey literature is useful

Hopewell S, 2007 – Handsearching minimizes bias, but is costly and requires much effort

Hopewell S, 2009 – Publication bias is a thing; consider methods to minimize it

Craane B, 2012 – Handsearching contributes considerably

Mahmood Q, 2014 – Searching for grey lit has benefits and challenges

Saleh AA, 2014 – Average time is 7 hours of librarian time to search grey lit sources

Cochrane Handbook, 10.3.2 – Publication bias is a major threat. Including unpublished trials appears to help


In groups of 3, consider the following scenarios and identify solutions to these problems/issues that commonly arise in SR teams.

The project lead does not want to search the grey literature. Is grey literature worth the effort? Is it something you would insist on doing?

Databases & Search Strategies Harvesting Search Terms

PICO Protocols Sentinel and other relevant articles

Controlled terms Read titles and abstracts Find others via (***.ti AND ***.ti) Entry terms

Use adjacency to cover variations E.g. (breast* adj5 (cancer* or carcinom* or neoplas*

or tumor* or tumour* or malignan*)).tw

Filters A predefined search designed to target specific

study methodologies (RCTs, Cohort, Systematic Reviews)

Consider using a validated filter E.g.: PubMed Clinical Queries – Specific, Therapy



Filters


Source: McKibbon, 2009

Filters, Sensitivity & Specificity


Topic/Subject Filters


Eligibility criteria can inform filters Be careful applying them

RCT publication type vs Title searches


Eligibility criteria can inform filters Be careful applying them

Humans limit?


Source: Glanville, 2008

Appraising filters


Sources of validated filters PubMed Clinical Queries

http://www.ncbi.nlm.nih.gov/books/NBK3827/#pubmedhelp.Clinical_Queries_Filters

PubMed Special Queries http://www.nlm.nih.gov/bsd/special_queries.html

HIRU (McMaster University) http://hiru.mcmaster.ca/hiru/HIRU_Hedges_home.aspx

SIGN http://www.sign.ac.uk/methodology/filters.html

InterTASC http://www.york.ac.uk/inst/crd/intertasc/

Cochrane Groups Individual studies

http://www.ncbi.nlm.nih.gov/pubmed/19712211 (eg)

http://www.ncbi.nlm.nih.gov/books/NBK3827/



http://www.nlm.nih.gov/bsd/special_queries.html

http://www.nlm.nih.gov/bsd/special_queries.html

http://hiru.mcmaster.ca/hiru/HIRU_Hedges_home.aspx

http://www.sign.ac.uk/methodology/filters.html



http://www.york.ac.uk/inst/crd/intertasc/



http://www.ncbi.nlm.nih.gov/pubmed/19712211


Supplemental strategies Cited Reference Searching


Supplemental strategies Handsearching journals (see Cochrane 6.2.2.1)

Those selected by team Impact Factor Most frequently appearing journals in Endnote/Excel

Databases & Search Strategies Supplemental strategies

Re-run the search near the end of the project

Databases & Search Strategies Validate searches

Important to validate your search in each database Make sure search captures all sentinel articles

If search doesn’t, why? How:

Databases & Search Strategies Cochrane recommendations

Source: Cochrane Editorial Unit, Methodological Expectations of Cochrane Intervention Reviews



The team tells you directly that they are only interested in searching one database. What are the problems with this approach and how would you discuss these problems with the project lead/team?



I’m getting way too many results and I’m thinking of using NOT to exclude editorials, letters, adults, and animals. What are possible limitations to applying so many filters? Are there other approaches? Other considerations?


In groups of 3, consider the following scenario and identify solutions to this problem that commonly arises in SR work.

Let’s say you limited your search to RCTs, English language studies, humans, and to the date range 2001-2015. How do you apply such restrictions to databases with few filtering options (e.g., Scopus, Web of Science, PsycINFO, ClinicalTrials.gov, etc.)?


In groups of 3, consider the following scenario and identify solutions to this problem that commonly arises in SR work.

I’ve been working on a search that aims to identify trends within the cancer literature. It’s not a systematic review, but requires a significant search. I try applying PubMed’s Cancer Subset search, but find it to be much too inclusive. Is it okay to create my own? Is it okay to use an unvalidated filter?



As you are building out your keyword searches, you are considering whether to rely on title or title/abstract searches. Or, .mp in Ovid MEDLINE? What do you factor in when making a decision?



The project lead says that he doesn’t want to sift through more than 200 papers. What are the issues and how would you handle this?



You’ve completed the searches and suggested to the team that they check the reference lists and do a Times Cited check of their included studies. As the team does this they notice that one of the papers has been cited over 500 times. They don’t want to consider these new papers. How do you handle this situation?



The project lead is most familiar with PubMed and would like you to run the search in that database. You feel Ovid MEDLINE is more appropriate. What are some of the differences between the two resources? How would you discuss these issues with the project lead?



You’ve been focused on a specific systematic review search for a long time and worry that there are minor mistakes. Would you consider recruiting a peer-reviewer? What if you can’t find one? If you were the peer-reviewer, how much time would you anticipate to spend reviewing someone else’s search?

Data Management

An SR is an extensive search with greater emphasis on data mgmt

Programs Endnote, Excel, DistillerSR, Abstrakr, other tools

Uses Keeping track of citations from multiple databases Eliminating duplicates Sharing citations with team Storing pdfs Eligibility screening Exporting to word, excel, and other formats Creating in-text citations and bibliographies

Data Management

Important to document exact search terms

Source: Featherstone, R

Data Management

• What to note:• Databases searched• Dates searches were

run• All searches

• Result numbers (total & unique)

• Sentinel articles used for testing

• Decide on naming convention

Data Management

What to note: All of this, plus the searches

Data Management

Source: Cochrane Handbook, 7.2.3

Data management strategy informs screening process

Cochrane recommends title/abstract review, then full-text

Data Management

Source: Institute of Medicine, Standards for Initiating a Systematic Review

Additional screening recommendations

Data Management

Endnote

Data Management

Excel

Data Management

Abstrackr (free)

Data Management

EPPI Reviewer 4 (free trial)

Data Management

Source: HLWiki, Software for systematic reviewing

Data Management

Data Management Include rationale for

exclusions Per PRISMA (item 17)


In groups of 3, consider the scenarios and identify solutions to the problems/issues that commonly arise in SR work.

The project lead and/or the team does not want to learn Endnote for this project. What are possible solutions?


In groups of 3, consider the scenarios and identify solutions to these problems/issues that commonly arise in SR work.

The screening team has identified 84 papers for full-text review and has asked for help obtaining the pdfs. How would you handle this?


In groups of 3, consider the scenarios and identify solutions to these problems/issues that commonly arise in SR work.

You’re working on an SR that includes members from multiple institutions. How do you manage the citations with a team comprised of folks who may or may not have the same access to resources?

Reporting

Standards PRISMA Cochrane Handbook IOM Standards for Initiating a Systematic Review

Reporting

Source: PRISMA, http://www.prisma-statement.org/statement.htm

PRISMA: aims to ensure the transparent and complete reporting

Reporting

Excerpts from the Explanation:

• Retrieval from databases may be imperfect, even if done by experienced searchers

• List database, platform, or provider

• Include start and end dates• Authors should report who

developed and conducted the search

• Report supplementary approaches, incl hand searching, reference lists, trial registries, contacting authors

Source: Liberati, 2009

PRISMA: Item 7 – Information sources

Reporting


• The search strategy is an essential component of any SR

• Note if the search is peer-reviewed

• It’s not always feasible to include the full strategy in the paper. Include it as an appendix, online supplement, etc.

• Archive searches• Note limits and other

limitations• Database inaccessibility• Financial / time

restrictions• No searching expertise


PRISMA: Item 8 – Search


• Report total number of records identified from all sources

• Include counts for each source, b/c literature identified primarily from references and experts can be biased

• Note: • Duplicates• Unique records• Levels of screening

• Layout of diagram may vary


PRISMA: Flow Diagram

Reporting

Reporting

Flow diagram template

Source: PRISMA, http://www.prisma-statement.org/statement.htm

Reporting

See Cochrane Handbook http://www.cochrane-handbook.org/

Section 6.6

“It should be borne in mind at the outset that the full search strategies for each database will need to be included in an Appendix of the review.”

http://www.cochrane-handbook.org/

Reporting


Section 6.6

In study flow diagram: number of unique records identified by the searches; number of records excluded after preliminary screening (e.g. of

titles and abstracts); and number of records retrieved in full text


Reporting


Section 6.6

In abstract: List all databases searched. Note the dates of the last search for each database or the

period searched. Note any language or publication status restrictions (but refer

to Section 6.4.9). List individuals or organizations contacted.”


http://www.mrc-bsu.cam.ac.uk/cochrane/handbook/chapter_6/6_4_9_language_date_and_document_format_restrictions.htm

Reporting


Section 6.6

In methods: List all databases searched. Note the dates of the last search for each database AND the

period searched. Note any language or publication status restrictions List grey literature sources. List individuals or organizations contacted. List any journals and conference proceedings specifically

handsearched for the review. List any other sources searched (e.g. reference lists, the

internet).


Reporting

See IOM Standards http://iom.nationalacademies.org/Reports/2011/Finding-

What-Works-in-Health-Care-Standards-for-Systematic-Reviews/Standards.aspx

In methods: Databases and other information sources used to identify

relevant studies Search strategy

http://iom.nationalacademies.org/Reports/2011/Finding-What-Works-in-Health-Care-Standards-for-Systematic-Reviews/Standards.aspx




Reporting

Example

Source: Sun GH, 2012

Reporting

Example 1

Source: Donahue, 2007

Reporting

Example 2

Source: Everett, 2005

Reporting

Example 3

Source: Nuti, 2014



A researcher meets with you to discuss the search. You realize that she is pursuing the project individually. What are potential problems with this? How do you advise her?



You were brought in to a project to run the search. You completed this in Feb ‘14 and sent the project lead the results. In Dec ’14, you are given a manuscript, which has you listed as an author. You notice that the searches are not entirely yours and that you don’t like what they’ve done methodologically. How would you handle this situation?



Same scenario, but you notice that you made an error in the searches. What would you do?

Wrap Up & Other Stuff


• Eligibility screening, from PRISMA Explanations document

Source: Cochrane Review Group on HIV/AIDS


Data Extraction forms


Types of Reporting Bias

Definition

Publication bias The publication or non-publication of research findings

Time lag bias The rapid or delayed publication of research findings

Multiple publication bias

The multiple or singular publication of research findings

Location bias The publication of research findings in journals with different ease of access or levels of indexing in standard databases

Citation bias The citation or non-citation of research findings

Language bias The publication of research findings in a particular language

Outcome reporting bias

The selective reporting of some outcomes but not others

Source: Cochrane Handbook, 10

Limitations


Source: Cochrane Handbook, 10

LimitationsTypes of Reporting Bias

Definition

Publication biasPositive results are over-represented in published literature. They are published and cited more frequently and published more quickly.

Negative results are less likely to be published and cited

What doctors don't know about the drugs they prescribe (Goldacre, TED Talk)

Time lag bias


Location bias

Citation bias

Language bias



No publication bias


Possible publication bias



Definition

Publication biasIndustry sponsored trials report “more favourable” outcomes when published (Ross 2009), but more industry-sponsored trials are left unpublished (Lundh 2012)

Time lag bias


Location bias

Citation bias

Language bias



Source: Hopewell, S.


Definition

Publication bias“Trials with positive results were published in approximately 4 to 5 years. Trials with null or negative results were published after about 6 to 8 years”

Our review shows that trials with positive results are published sooner than other trials.”

Hopewell S et al. Time to publication of results to clinical trials. Cochrane Review. PMID: 17443632

Time lag bias


Location bias

Citation bias

Language bias



Source: Cochrane Handbook, 10.2.2.1


Definition

Publication bias“It is not always obvious that multiple publications come from a single study, and one set of study participants may be included in an analysis twice. The inclusion of duplicated data may therefore lead to overestimation of intervention effects.”

Cochrane Handbook 10.2.2.1

Time lag bias


Location bias

Citation bias

Language bias





Definition

Publication biasSignificance of results and trial quality associated w/ journal impact.

“Choices regarding which databases to search may bias the effect estimate in a meta-analysis.”


Time lag bias


Location bias

Citation bias

Language bias


• Strategy• Search multiple databases• Search subject-specific

resources• No access? Partner w/ other

librarians




Definition

Publication biasBramer WM, et al. (2013) The comparative recall of Google Scholar versus PubMed in identical searches for biomedical systematic reviews: a review of searches used in systematic reviews. Syst Rev. PMCID: PMC3882110.

Giustini D, et al. (2013) Google Scholar is not enough to be used alone for systematic reviews. Online J Public Health Inform. PMC3733758.

Sampson M, et al. (2003) Should meta-analysts search Embase in addition to Medline? J Clin Epidemiol. PMID: 14568625.

Time lag bias


Location bias

Citation bias

Language bias





Definition

Publication bias“Perusal of reference lists….is far from objective.”


Time lag bias


Location bias

Citation bias

Language bias


• Strategy• Peruse reference lists (final

safety net)




Definition

Publication bias“Reviews have often been exclusively based on studies published in English…. While the potential impact of [excluding non-English studies] may be minimal, it is difficult to predict in which cases this exclusion may bias a systematic review.”


Time lag bias


Location bias

Citation bias

Language bias


• Strategy• Is it always feasible to include

non-English studies?• Case-by-case basis




Definition

Publication bias“The choice of outcomes that are reported can be influenced by the results, potentially making published results misleading.”


Outcomes described in protocols are inconsistent with those reported in publications (Dwan K, 2013)

Time lag bias


Location bias

Citation bias

Language bias


• Strategy• Develop a protocol. Not infallible


Search Strategy Considerations

Search multiple databases Cochrane Handbook suggests MEDLINE, Embase, and Cochrane Register

Include subject specific databases

Fewer limits MeSH termsScreen for eligibility manually instead

Language bias English?

Include grey literature sources

Trials registries / Conference proceedings / Protocols / Drug registries and other info / Contacting authors

AHRQ on Finding Grey Literature (Balshem, 2013)Cochrane Review on its value (Hopewell & MacDonald, 2007)

Librarian Involvement Librarian co-authors correlated higher quality search strategy reporting. (Rethlefsen 2015)

See also: Rethlefsen M. (2014). Engaging medical librarians to improve the quality of review articles. JAMA. PMID 25203078

Summary

Useful Resources

Expert Searching Listserv http://pss.mlanet.org/mailman/listinfo/

expertsearching_pss.mlanet.org Cochrane Handbook

http://handbook.cochrane.org/ PRISMA (remember the elaboration article)

http://www.prisma-statement.org/ Systematic Reviews: Opportunities for Librarians

http://www.lib.umich.edu/systematic-reviews-workshop

http://pss.mlanet.org/mailman/listinfo/expertsearching_pss.mlanet.org




http://handbook.cochrane.org/

http://handbook.cochrane.org/

http://www.prisma-statement.org/



Acknowledgments

My colleagues involved in the Systematic Reviews: Opportunities for Librarians workshop. Specifically, Whitney Townsend at the University of Michigan

Joel Gagnier, Epidemiologist, University of Michigan,

MLA, for the Beyond the Search webinar series

References Abstrakr, http://abstrackr.cebm.brown.edu/ Antman EM, Lau J, Kupelnick B, Mosteller F, Chalmers TC. (1992). A comparison of results of meta-analyses

of randomized control trials and recommendations of clinical experts: Treatments for myocardial infarction. JAMA, 268. 240-248. PMID 1535110.

Armstrong R, Jackson N, Doyle J, Waters E, Howes F. (2005). It's in your hands: the value of handsearching in conducting systematic reviews of public health interventions. J Public Health, 27(4).388-91. PMID 16311247.

Balshem H, Stevens A, Ansari M, Norris S, Kansagara D, Shamliyan T, Chou R, Chung M, Moher D, Dickersin K. (2013). Finding Grey Literature Evidence and Assessing for Outcome and Analysis Reporting Biases When Comparing Medical Interventions: AHRQ and the Effective Health Care Program. PMID 24404628.

Beller EM, Chen JK, Wang UL, Glasziou PP. (2013). Are systematic reviews up-to-date at the time of publication? Syst Rev, 282. 36. PMID 23714302.

Benzies KM, Premji S, Hayden KA, Serrett K. (2006). State-of-the-evidence reviews: advantages and challenges of including grey literature. Worldviews Evid Based Nurs, 3(2). 55-61. PMID 17040510.

Bramer WM, Giustini D, Kramer BM, Anderson P. (2013). The comparative recall of Google Scholar versus PubMed in identical searches for biomedical systematic reviews: a review of searches used in systematic reviews. Syst Rev, 23 (2). 115. PMCID PMC3882110.

Cochrane Collaboration. Cochrane Handbook for Systematic Reviews of Interventions. http://handbook.cochrane.org/

Cochrane Community. Glossary. http://community.cochrane.org/glossary Cochrane Editorial Unit. Cochrane Editorial Unit, Methodological Expectations of Cochrane Intervention

Reviews. http://editorial-unit.cochrane.org/mecir Cochrane Review Group on HIV/AIDS. Data Extraction and Assessment Form.

http://hiv.cochrane.org/sites/hiv.cochrane.org/files/uploads/Data%20extraction%20form_RCTs.docx Cook DJ, Guyatt GH, Ryan G, Clifton J, Buckingham L, Willan A, McIlroy W, Oxman AD. (1993). Should

unpublished data be included in meta-analyses? Current convictions and controversies. JAMA, 269 (21). 2749–2753. PMID 8492400.

References Craane B, Dijkstra PU, Stappaerts K, De Laat A. (2012). Methodological quality of a systematic

review on physical therapy for temporomandibular disorders: influence of hand search and quality scales. Clin Oral Investig, 16 (1). 295-303. PMID 21128088.

Donahue KE, Gartlehner G, Jonas DE, Lux LJ, Thieda P, Jonas BL, Hansen RA, Morgan LC, Lohr KN. (2008). Systematic review: comparative effectiveness and harms of disease-modifying medications for rheumatoid arthritis. Ann Intern Med, 148 (2). 124-34. PMID 18025440.

Dwan K, Gamble C, Williamson PR, Kirkham JJ. (2013). Reporting Bias Group. Systematic review of the empirical evidence of study publication bias and outcome reporting bias - an updated review. PLoS One, 8 (7). e66844. PMID 23861749.

Egger M, Juni P, Bartlett C, Holenstein F, Sterne J. (2003). How important are comprehensive literature searches and the assessment of trial quality in systematic reviews? Empirical study. Health Technol Assess, 7 (1). 1-76. PMID 12583822.

EPC Methods: An Exploration of Methods and Context for the Production of Rapid Reviews. http://www.ncbi.nlm.nih.gov/books/NBK274092

EPPI Reviewer 4. eppi.ioe.ac.uk/cms/er4/

Everett CR, Shah RV, Sehgal N, McKenzie-Brown AM. (2005). A systematic review of diagnostic utility of selective nerve root blocks. Pain Physician, 8 (2). 225-33. PMID 16850076.

Featherstone, R. Concept Map for Research Literature Reviews. http://www.slideshare.net/featherr/search-strategy-worksheet

Gagnier, JJ. (2015). Introduction to Systematic Reviews & Meta-Analyses. http://open.umich.edu/education/sph/epid757/summer2011/materials

Giustini D, Boulos MN. (2013). Google Scholar is not enough to be used alone for systematic reviews. Online J Public Health Inform. 5 (2). 214. PMCID PMC3733758

References Glanville J, Bayliss S, Booth A, Dundar Y, Fernandes H, Fleeman ND, Foster L, Fraser C, Fry-Smith A, Golder S,

Lefebvre C, Miller C, Paisley S, Payne L, Price A, Welch K. (2008). So many filters, so little time: the development of a search filter appraisal checklist. JMLA, 96(4). 356-61. PMID 18974813. http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2568852/

Goldacre, B. (2012). What doctors don't know about the drugs they prescribe. TEDMED 2012. http://www.ted.com/talks/ben_goldacre_what_doctors_don_t_know_about_the_drugs_they_prescribe?language=en

Grant MJ, Booth A. (2009). A Typology of reviews: an analysis of 14 review types and associated methodologies. Health Info Libr J. 26 (2). 91-108. PMID 19490148.

Guyatt G, Rennie D, Meade MO, Cook, Deborah J. (2015). Summarizing the Evidence Worksheet. In: JAMAevidence, Users’ Guides to the Medical Literature: A Manual for Evidence-Based Clinical Practice. 2nd Ed. http://www.jamaevidence.com/resource/520

HIRU, McMaster University. http://www.nlm.nih.gov/bsd/special_queries.html

HLWiki International. Software for systematic reviewing. http://hlwiki.slais.ubc.ca/index.php/Software_for_systematic_reviewing

Hopewell S, Clarke M, Lefebvre C, Scherer R. (2007). Handsearching versus electronic searching to identify reports of randomized trials. Cochrane Database Syst Rev. 18(2). MR000001. PMID 17443625.

Hopewell S, Clarke M, Lusher A, Lefebvre C, Westby M. (2002). A comparison of handsearching versus MEDLINE searching to identify reports of randomized controlled trials. Stat Med. 21 (11). 1625-34. PMID 12111923.

Hopewell S, Clarke M, Stewart L, Tierney J. Time to publication for results of clinical trials. (2007). Cochrane Database Syst Rev, 2. MR000011. PMID 17443632.

Hopewell S, Loudon K, Clarke MJ, Oxman AD, Dickersin K. (2009). Publication bias in clinical trials due to statistical significance or direction of trial results. Cochrane Database Syst Rev. 2009, 1. MR000006. PMID 19160345.

References Hopewell S, McDonald S, Clarke M, Egger M. (2007). Grey literature in meta-analyses of randomized trials

of health care interventions. Cochrane Database Syst Rev. 18 (2). MR000010. PMID 17443631.

ICMJE. Defining the Role of Authors and Contributors. http://www.icmje.org/recommendations/browse/roles-and-responsibilities/defining-the-role-of-authors-and-contributors.html

Institute of Medicine. Standards for Initiating a Systematic Review. Standards for Systematic Reviews. http://www.iom.edu/Reports/2011/Finding-What-Works-in-Health-Care-Standards-for-Systematic-Reviews/Standards.aspx

InterTASC, http://www.sign.ac.uk/methodology/filters.html

Khangura S, Konnyu K, Cushman R, Grimshaw J, Moher D. (2012). Evidence summaries: the evolution of a rapid review approach. Syst Rev, 1. 10. PMC3351736.

Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gotzsche PC, Ioannidis P, Clarke M, Devereaux PJ, Kleijnen J, Moher D. (2009). The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: Explanation and elaboration. Annals of Internal Medicine, 151 (4). W-65-W-94. PMID 19622512.

Lundh A, Sismondo S, Lexchin J, Busuioc OA, Bero L. (2012). Industry sponsorship and research outcome. Cochrane Database Syst Rev. 12. MR000033. PMID 23235689.

Mahood Q, Eerd DV, Irvin E. (2014). Searching for grey literature for systematic reviews: challenges and benefits. Res Synth Methods, 5 (3). 221-34. PMID 26052848.

Martin JL, Pérez V, Sacristán M, Alvarez E. (2005). Is grey literature essential for a better control of publication bias in psychiatry? An example from three meta-analyses of schizophrenia. Eur Psychiatry, 20 (8). 550-3. PMID 15994063.

http://www.icmje.org/recommendations/browse/roles-and-responsibilities/defining-the-role-of-authors-and-contributors.html

http://www.icmje.org/recommendations/browse/roles-and-responsibilities/defining-the-role-of-authors-and-contributors.html

References McDonald S, Lefebvre C, Antes G, Galandi D, Gøtzsche P, Hammarquist C, Haugh M, Jensen KL, Kleijnen J,

Loep M, Pistotti V, Rüther A. (2002). The contribution of handsearching European general health care journals to the Cochrane Controlled Trials Register. Eval Health Prof, 25 (1), 65-75. PMID 11868446.

McKibbon KA, Wilczynski NL, Haynes RB, Hedges Team. (2009). Retrieving randomized controlled trials from medline: a comparison of 38 published search filters. Health Info Libr J, 26 (3). 187-202. PMID 19712211.

Nuti SV, Wayda B, Ranasinghe I, Wang S, Dreyer RP, Chen SI, Murugiah K. (2014). The use of google trends in health care research: a systematic review. PLoS One, 9 (10). e109583. PMID 25337815.

Oxman AD, Guyatt GH. (1993). The science of reviewing research. Annals of the New York Academy of Sciences, 703. 125-133. PMID 8192290.

PRISMA. http://www.prisma-statement.org/statement.htm

PROSPERO. Centre for Reviews and Dissemination. http://www.crd.york.ac.uk/PROSPERO/

PubMed Clinical Queries. http://www.ncbi.nlm.nih.gov/books/NBK3827/#pubmedhelp.Clinical_Queries_Filters

PubMed Special Queries. http://www.nlm.nih.gov/bsd/special_queries.html

Rethlefsen ML. (2013). I Want To Do a Systematic Review. Liblog: Newsletter of the Mayo Clinic Libraries. http://liblog.mayo.edu/2013/05/01/i-want-to-do-a-systematic-review/

Rethlefsen M. (2014). Engaging medical librarians to improve the quality of review articles. JAMA. 312 (10). 999-1000. PMID 25203078.

Rethlefsen ML, Farrell AM, Osterhaus Trzasko LC, Brigham TJ. (2015). Librarian co-authors correlated with higher quality reported search strategies in general internal medicine systematic reviews. J Clin Epidemiol, 68 (6). 617-26. PMID: 25766056.

References Ross JS, Mulvey GK, Hines EM, Nissen SE, Krumholz HM. (2009). Trial publication after registration in

ClinicalTrials.Gov: a cross-sectional analysis. PLoS Med, 6 (9). e1000144. PMID 19901971.

Sackett DL, Straus SE, Richardson WS, Rosenberg W, Haynes RB. Evidence-based medicine: how to practice and teach EBM. 2nd ed. Edinburgh: Churchill Livingstone, 2000.

Saleh AA, Ratajeski MA, Bertolet M. (2014). Grey Literature Searching for Health Sciences Systematic Reviews: A Prospective Study of Time Spent and Resources Utilized. Evid Based Libr Inf Pract, 9 (3). 28-50. PMID 25914722.

Sampson M, Barrowman NJ, Moher D, Klassen TP, Pham B, Platt R, St John PD, Viola R, Raina P. (2003) Should meta-analysts search Embase in addition to Medline? J Clin Epidemiol, 56 (10). 943-55. PMID: 14568625

Sampson, M, McGowan J. (2006). Errors in search strategies were identified by type and frequency. Journal of Clinical Epidemiology, 59 (10). 1057-63. PMID 16980145.

Sampson M, Shojania KG, Garritty C, Horsley T, Ocampo M, Moher D. (2008). Systematic reviews can be produced and published faster. J Clin Epidemiol, 61 (6). 531-6. PMID 18471656.

SIGN. http://www.sign.ac.uk/methodology/filters.html

Sun G, Gaines J, Perla R, MacEachern M, Davis M, Shrank W. (2013). Systematic review of quality improvement initiatives in the intensive care setting. PROSPERO. http://www.crd.york.ac.uk/PROSPERO/display_record.asp?ID=CRD42012003303#.VD6ZomNDR8F.

Sun GH, Raji CA, Maceachern MP, Burke JF. (2012). Olfactory identification testing as a predictor of the development of Alzheimer's dementia: a systematic review. Laryngoscope, 122 (7). 1455-62. PMID 22552846.

Tsafnat G, Glasziou P, Choong MK, Dunn A, Galgani F, Coiera E. (2014). Systematic review automation technologies. Syst Rev, 3. 74. PMID 25005128.

systematic reviews: a primer for librarians mark maceachern, mlis taubman health sciences library,...

Documents

systematic methods

systematic review attempts

systematic reviewswhat

narrative review

biasa systematic presentation

rapid review app

gagnier jj

methods rigorous