2/20/2012

1

An Introduction to 

Systematic Reviews

Susan L. Norris, MD, MPH, MSAssociate Professor

Oregon Health & Science University Portland, OR

norriss@ohsu.edu

Biography, SL Norris• MD, MS, University of Alberta• MPH, University of Washington• Board Certified: general surgery (Canada), family medicine (US)

• Clinical practice: Group Health Cooperative, Washington state, 1990‐1999

2

g ,• CDC: directed systematic review group with focus on diabetes, 1999‐2004

• Oregon Health & Science University (current): – clinical practice guideline development– systematic review methodology (nonrandomized studies, reporting bias), comparative effectiveness reviews 

– effects of physician‐industry relationships on practice guidelines and systematic reviews

Disclosures• Financial: none

• Intellectual

– Member, GRADE Working Group

– Methods work: nonrandomized studies, sources  of bias in reviews

3

reviews 

– Impact of physician‐industry relationships on primary research and evidence synthesis

• Professional

– Investigator, Evidence‐based Practice Center (AHRQ)   

– Funders:  CDC, NIH, AHRQ, American College of Chest Physicians, American Urological Association

2/20/2012

2

Ben-Adhem picked up a stone from

beside the road. “It had written on it,

“Turn me over and read.’ So he

4

picked it up and looked at the other

side.

And there was written, “Why do

you seek more knowledge when

you pay no heed to what you know

5

y p y y

already?’”Shah (1968)

Learning Objectives

1. To define the main characteristics of systematic reviews and to differentiate them from narrative reviews.

2. To outline the steps involved in producing a systematic review

6

systematic review.3. To understand potential sources of bias in reviews.4. To outline five characteristics of a high‐quality  

systematic review and to apply those to a review published in the peer reviewed literature.

5. To learn how to find systematic reviews in Medline or in repositories of systematic reviews.

2/20/2012

3

Outline1. Introduction:  need for reviews

2. Typology and terminology‐ Steps in a systematic review 

3. Formulating an answerable question

4. Searching for evidence

5 Ri k f bi i i di

7

5. Risk of bias in primary studies

6. Evidence synthesis‐ Qualitative

‐ Quantitative (meta‐analysis)

7.     Assessing strength of evidence across studies‐ GRADE

8. Sources of high quality reviews and resources for performreviews

Part 1. Introduction

8

Introduction

Knowing What Works in Health Care:IOM Report 2008

“ …decisions about the care of individual 

patients should be based on the 

9

conscientious, explicit, and judicious use 

of the current best evidence on the 

effectiveness of clinical services.”

2/20/2012

4

500,000.00

600,000.00

700,000.00

800,000.00

900,000.00

Records added to MEDLINE each year

10

0.00

100,000.00

200,000.00

300,000.00

400,000.00

,

1970

1971

1972

1973

1974

1975

1976

1977

1978

1979

1980

1981

1982

1983

1984

1985

1986

1987

1988

1989

1990

1991

1992

1993

1994

1995

1996

1997

1998

1999

2000

2001

2002

2003

2004

2005

2006

2007

2008

MEDLINE

11PLoS Med 7(9): e1000326. doi:10.1371/journal.pmed.1000326

Part 2. Systematic Reviews:

12

Systematic Reviews: Typology and Terminology

2/20/2012

5

Typology

• Integrative publication

• Systematic review

• Narrative review 

• Qualitative synthesis

• Quantitative synthesis

• Meta‐analysis

• Practice guidelines

13

• Nonsystematic review

• Review of comparative effectiveness

• Practice guidelines

• Economic evaluation

• Decision analysis

Integrative Publications

Combine the results of multiple studies

• Reviews  

• Narrative, quantitative

• Systematic, nonsystematic

14

• Practice guidelines

• Economic evaluations

• Clinical decision analysis

Types of Reviews

• Nonsystematic (narrative or traditional) – Widespread (eg, NEJM)

– Typically, 1 or 2 experts write the review based on literature the authors are familiar with

Conclusions related more to the experiences and

15

– Conclusions related more to the experiences and views of the authors than to the evidence

– Different authors might reach different conclusions on the same topic or even on the  same evidence

– Small but important effects may be missed

• Systematic

2/20/2012

6

Systematic Review

A concise summary of the best available

evidence that addresses a sharply

defined clinical question (Mulrow 1998)

16

defined clinical question (Mulrow 1998) 

• Qualitative synthesis = narrative summary

• Quantitative synthesis = meta‐analysis

Systematic Review

“In its ideal form, is a review that includes an explicit and detailed 

17

description of how it was conducted so that any interested reader would be able to replicate it”   (Jadad 1998)

• Define the extent and limits of the evidence

• Facilitate decision making– Do tell you what to do  

– Other factors to consider  • Equity, judgment, patient preferences, resource constraints 

Cl if h t ti / d ti i b d

Systematic Reviews: Aims

18

• Clarify what practice/recommendation is based on evidence and what is based on other opinion

• Help to understand inconsistencies

• Help to understand potential applicability to specific situations or settings

• Identify research gaps

2/20/2012

7

Attributes of Systematic Reviews

T  Transparent

R Reproducible 

b d

19

U Unbiased

E Exhaustive

E Explicit methods 

Reviews of Comparative Effectiveness

Comparative effectiveness

= the comparison of effective interventions among patients in typical patient care settings

20

among patients in typical patient care settings, with decisions tailored to individual  patient needs

Sox and Greenfield, 2009

Meta‐Analysis

A systematic review that employs statistical methods to combine and summarize the results of several studies   (Cook 1995)

21

Other terms:• Quantitative synthesis

• Pooling, pooled analysis

2/20/2012

8

When is a Systematic Review Needed?1. When there is a substantive clinical or public health  

question

2. Uncertainty exists• Effects of an intervention, specific population or setting

• Balance of benefits and harms

• Small effects in individual studies

22

• Small effects in individual studies

• Need to explain inconsistency in results across studies

3. Need to define knowledge gaps, develop a research agenda

4. Several primary studies exist– Often with disparate findings

– There may be situations where there is a need to establish the lack of evidence

Steps in a Systematic Review

• Develop the review question 

• Develop inclusion/exclusion criteria

• Search for literature

23

• Quality assess individual studies 

• Data abstraction and analysis

• Synthesis of findings

• Grading the strength of evidence

Part 3. Formulating an Answerable

24

Systematic Review Question

2/20/2012

9

Formulating an Answerable Question

• Most important step in the review• Depends on perspective• Results have impact on patient care• Not dependent on available data

25

• Considerations:– Common conditions– Areas of uncertainty– Resource-intensive– New treatments– Heterogeneity of effects

26

Parachute use to prevent death and major trauma related to gravitational challenge: systematic review of randomised controlled trials

GCS Smith, J PellBMJ 2003

27

2/20/2012

10

Levels of questions

• Can it work?

• Does it work?

• When/in whom does it work?• When/in whom does it work?

• Is it worth it? 

• What research is needed?

Brian Haynes, EBM

What is the Purpose of Different Questions?

• Can it work?– Useful for: Coverage decisions

• How well does it work overall?  – Useful for: Quality measurement and improvement

• When/in whom does it work?– Useful for: Patient/clinician decisions; health systems

• Is it worth it? – Useful for: Informing patient choice

• What kind of research is needed?– Researchers, funding agencies

D Atkins

What is Needed to Answer Those Questions?

• Can it work?– Studies of high internal validity + quantitative synthesis 

• How well does it work overall?  – Studies with external & internal validity   

30

y

• When/in whom does it work?– External validity & internal validity

• Is it worth it (benefits vs. harms)? – Quantitative synthesis  + decision models

• What kind of research would be most useful?– Qualitative synthesis, broad inclusion criteria

D Atkins

2/20/2012

11

Formulating a Review Question

P Person, population

I Intervention, exposure

C Comparator

O Outcome

(T) Timing

(S) Setting, study design

Example: The Well-Formulated Question

Does anticoagulation therapy improve outcomes in early treatment of patients with ischemic stroke?

Intervention Outcome Population Setting

Condition of Interest

32

Heparin

Dicumarol

CY 222sc

Phenindione

Kabi 2165sc

Recurrent Stroke

Extra/Intracranealhaemorrhage

Pulmonary Embolism

Venous Thrombosis

Death

>18 yrs, Hospitalized

≤48h from onset

Abrupt impairment of brain function caused by a variety of pathologic changes involving cranial blood vessels

���������� ������

33

2/20/2012

12

Setting inclusion and Exclusion CriteriaExclusion Criteria

What Evidence is Admissible?

• Consider topic by topic

• One size does not fit all

• Importance of protocol

• Educated decisions a priori

• Be explicit

• Document and defend changes

• Starts with a clear question

What Evidence is Admissible?

P Person, populationI Intervention, exposureC ComparatorO OutcomeT TimingS Setting, study design

2/20/2012

13

Inclusion Criteria for a Inclusion Criteria for a Drug Class Review (TZDs)Drug Class Review (TZDs)

Populations:  Adults with type 2 diabetes:

ADA or WHO definitionAdults with prediabetes:

Defined as …

Adults with metabolic syndrome:ATP definition..

↑ TG level↓ HDL‐cholesterol↑ BP↑ FPG

Part 4.Searching the Literature

38

Image from cartoonbank.com removed.

Image description: Picture of sheriffs in forest with 3 bl dh d d D i l t th

39

bloodhound dogs; Dogs are using a laptop on the ground. Caption reads: First, they do an online search.

2/20/2012

14

Literature Searches for a Systematic Review

• Purpose:– To locate all relevant studies 

– Comprehensive, unbiased collection of studies:• Failure to search and identify all relevant sources can lead to a biased reviewa biased review

• Challenges– Locating literature indexed in electronic databases

– Locating literature not‐indexed

– Identifying, locating unpublished studies

– Balance resource allocation and return

Methods of Searching for Relevant Studies

• Electronic Databases– Medline, Cochrane, PsychINFO, Embase, etc.

• Trial registries

• Hand searching

• Grey literature searches

• Experts

• Reference lists of reviews and included studies

Rationale for Comprehensive Searching

• Limitations of the search methods

– Only a small proportion of citations that are identified by comprehensive electronic searching (eg, MEDLINE) are relevant

– The typical MEDLINE search retrieves 50%‐ 80% of relevant studies

– Ability to search more precisely depends on database

2/20/2012

15

Rationale for Comprehensive Searching 

• Non‐English language articles – not easily accessible on MEDLINE

– Non‐English language articles are of similar quality to English language articles and results areto English language articles and results are negative more often

• Publication Bias– 25‐50% of studies are never published, even as abstracts

– Studies with negative findings less likely to be published

Rationale for Comprehensive Searching

Failure to search and identify all relevantFailure to search and identify all relevant sources can lead to a biased review

Part 5.Assessing the Risk of Bias of   

45

Individual Studies

2/20/2012

16

Bias

“ a systematic error, or deviation from the

truth, in results or inferences”

46

Cochrane Manual, 9/08

Quality

“extent to which the authors conducted their

h t th hi h t ibl t d d

47

research to the highest possible standards

Cochrane Manual, 9/08

Internal validity = The extent to which the results of a study can

be reliably attributed to the intervention under evaluation

48

vs. external validity = applicability of results to other populations and settings

vs. precision = measure of likelihood of random errors

2/20/2012

17

Why Assessment of Risk of Bias is Important

– Assess and minimize bias in the review 

– Guide interpretation of findings

49

Hierarchy of Evidence

STUDY DESIGN

Randomized Controlled Trials

↓ BIAS

Cohort Studies and Case Control Studies

Case Reports and Case Series

Expert opinion

↑ BIAS

51

2/20/2012

18

Classification Scheme for Bias

• Selection bias

• Performance bias

• Attrition bias

52

• Attrition bias

• Detection bias

• Reporting bias

53

54

2/20/2012

19

Risk of Bias Tool

• For assessing RCTs only

• Developed by international group (largely Cochrane based)

D ib d i d t il i C h l

55

• Described in detail in Cochrane manual (9/08) www.cochrane.org

Components of RoB Tool

• Sequence generation

• Allocation concealment

• Blinding (by outcome)

56

• Incomplete outcome data (by outcome)

• Selective outcome reporting

• Other

Domain Description (example) Review authors’ judgment

Sequence generation “patients were randomly allocated”

Was the allocation sequence adequately generated?

YES / NO / UNCLEAR

Allocation concealment No information Was allocation adequately concealed? YES / NO / UNCLEAR

Blinding of participants, personnel and outcome assessors

Outcome: 

“Assessors were blinded to participant’s treatment group”

Was knowledge of the allocated intervention adequately prevented during the study? 

YES / NO / UNCLEAR

Blinding of participants, personnel and outcome assessors

Outcome:

Was knowledge of the allocated intervention adequately prevented during the study? 

YES / NO / UNCLEAR

Cochrane Risk of Bias Tool

57

/ /

Incomplete outcome data Outcome:

Were incomplete outcome data adequately addressed? 

YES / NO / UNCLEAR

Incomplete outcome data Outcome:

“20% of participants in the intervention group and 35% in the control group withdrew prior to the last follow‐up measurement”

Were incomplete outcome data adequately addressed? 

YES / NO / UNCLEAR

Selective outcome reporting Some components of the composite outcome were reported and those presented were significant .

Are reports of the study free of suggestion of selective outcome reporting? 

YES / NO / UNCLEAR

Other sources of bias Study funded by the makers of the study drug; three coauthors were industry employees.

Was the study apparently free of other problems that could put it at a high risk of bias? 

YES / NO / UNCLEAR

2/20/2012

20

What do You do with the RoB Assessment?

• Stratify studies by RoB

• Formal comparisons of effects by RoB with meta‐regression or test for differences across subgroups

ll d d d d

58

• Present all studies and provide a narrative discussion 

• Restrict primary analysis of all studies and then of low, or low and unclear RoB

• Exclude high RoB studies in synthesis

• Sensitivity analysis: how are conclusions affected if high RoB studies were included?

Part 6.  Evidence Synthesis

59

y

Synthesis (noun)

1 a : the composition or combination of parts or elements so as to form a whole; c : the combining of often diverse conceptions into a coherent whole;a coherent whole; 

Merriam Webster Online Dictionary, 2009

2/20/2012

21

Synthesis in Systematic Reviews

• Explore relationships in the data• Describe patterns across included studies

• Direction of effects

• Size of effects

• Draw conclusions about the likely size and direction of effect

• Assess robustness of the data and conlusions• Strength of the evidence 

• Applicability of conclusions on effect size to different populations and/or contexts

Popay J, Roberts H, Sowden A, et al. Guidance on the Conduct of narrative Synthesis in Systematic Reviews (draft). ESRC Methods Program, UK.

Additional Elements of Synthesis in Systematic Reviews

• Identify gaps in the evidence– Little or no direct evidence– Poor quality evidence– Variation that is not explainable– Future research recommendation

• Based on findings• Specific enough for funders

Additional Elements of Synthesis in Systematic Reviews

• Limitations of the review– Scope

• Any important elements not addressed

– Search strategies• Databases searched• Language restrictions

– Study selection and abstraction• Two reviewers vs one• Indirect evidence

– Limitations of analyses• Low power due to few studies

2/20/2012

22

Quantitative Synthesis: Meta‐analysis

• Pool results of a specific outcome across   multiple studies 

• Studies are weighted by a measure of dispersion from each study

• Provides an overall point estimate of effect

64

• Narrower confidence interval

• May be appropriate when: • limited sample sizes prevent finding a difference where one may exist

• studies have conflicting results:  assess heterogeneity 

• Designed to address direct evidence for a singleoutcome of interest

• Data obtained (ideally) from a systematic review

Meta‐analysis:  Enoxaparin versus Unfractionated Heparin

65Gould et al. Ann Intern Med 1999 May 18;130(10):800Gould et al. Ann Intern Med 1999 May 18;130(10):800‐‐9.9.

Part 7.

Grading the Strength of Evidence 

66

Across Studies

2/20/2012

23

Grading a Body of Evidence• Subjective summary

• Assessor applies a descriptive word to evidence• Eg, good, fair, poor

• Imprecise and subjective

• Formal systems for grading a body of evidence

67

• Many systems exist;  most flawed

• Gives greater weight to better designs

• Difficult to apply when contradictory evidence exists

• Different systems consider different study features:  quality, design, consistency, number of studies, etc.

Grades of Recommendation Assessment, Development, and

Evaluation (GRADE)

Since 2000

Researchers/guideline developers with

68

interest in methodology

Aim: to achieve a common, transparent and sensible system for grading the quality of evidence and the strength of recommendations

GRADE UptakeWorld Health OrganizationNational Institute Clinical Excellence (NICE)Agency for Healthcare Research and Quality (AHRQ)Canadian Agency for Drugs and Technology in Health Allergic Rhinitis in Asthma Guidelines (ARIA)American Thoracic Society American College of Chest Physicians

69

g yUpToDate British Medical Journal American College of PhysiciansCochrane Collaboration European Society of Thoracic SurgeonsClinical Evidence American Urological Association Many other organizations

2/20/2012

24

Grading Body of Evidence:GRADE method

• Directness of evidence– Health outcomes vs intermediate outcomes, efficacy outcomes

– Direct comparisons vs indirect comparisons

• Precision of estimates– Degree of certainty surrounding an effect estimate with respect to a 

71

given outcome

• Consistency of results– Effect sizes have the same sign

– The range of effect sizes is narrow

• Risk of bias– Study design

– Aggregate risk of bias of the studies under consideration  

*GRADE Working Group: www.GradeWorking‐Group.org

Rate Quality of the Evidence

= Confidence in an estimate of effect• High: considerable confidence in estimate of effect 

• Moderate: further research may change 

72

estimate

• Low: further research is very likely to impact the estimate

• Very low: estimate of effect is very uncertain

2/20/2012

25

Summary of Findings Tables

74

Part 8.Resources for Identifying or 

Performing High Quality Systematic g g y yReviews

2/20/2012

26

76 23

Archie Cochrane

Image of Archive Cochrane removed

77

Image of Archive Cochrane removed.

Image of Cochrane logo removed

78

Image of Cochrane logo removed.

2/20/2012

27

Recommendations on reporting of systematic reviews

• QUORUM, 1999

– Quality of Reporting of Meta‐analyses

• MOOSE, 2000

l i f b i l di i

79

– Meta‐analysis of observational studies in epidemiology (Stroup et al. JAMA April 2000)

• PRISMA, 2009

– Preferred Reporting Items for Systematic Reviews and Meta‐Analyses (Moher et al. PLoS Med July 2009)

Quality Assessment of Systematic  Reviews: AMSTAR 

11‐Items answered as:• � Yes• � No• � Can’t answer

80

• � Not Applicable

‐ Addresses review planning, searching, quality assessment, synthesis, conflicts of interest

Shea et al. BMC Medical Research Methodology 2007

Locating Systematic Reviews

• Search Strategies in Medline

– Study type: systematic review

– EBM reviews

“systematic review” or “Medline” as text term in

81

– systematic review  or  Medline  as text term in title or abstract

• Key things to look for:

– Year of last search

– Inclusion criteria:  PICOTS

– Quality of the review

2/20/2012

28

Organizations that Produce High Quality Systematic Reviews

• Agency for Healthcare Research and Quality– Effective Healthcare Program (effectivehealthcare.ahrq.gov)

– Evidence‐based Practice Center program (www.ahrq.gov)

• US Preventive Services Task Force– Supported by EPC program (www.ahrq.gov/clinic/uspstfix.htm)

82

Supported by EPC program (www.ahrq.gov/clinic/uspstfix.htm) 

• Cochrane Collaboration reviews (www.cochrane.org) 

• Health Technology Assessment programs– E.g. UK NCCHTA (www.ncchta.org)

• National Institute for Clinical Excellence (www.nice.org.uk)

• Drug Effectiveness Review Project (www.ohsu.edy/drugeffectiveness)

Resources for Performing Systematic  Reviews

• Organizations

– Cochrane Collaboration:  www.cochrane.org

– NICE, SIGN

– AHRQ: Methods Guide for Effectiveness and Comparative Effectiveness Reviews 2009

83

Comparative Effectiveness Reviews 2009, http://effectivehealthcare.ahrq.gov

• Books

– Systematic Reviews in Health Care: Meta‐Analysis in Context:  Eggers et al.  BMJ Books

– Cochrane Handbook: 2008, Wiley & Sons

– IOM report 2011: Finding What Works in Healthcare

Caveat Emptor: Systematic Reviews

• SRs should give the best estimate of any true effect, but– MUST be well done

• Findings must be taken in concert with other evidence and factors – Evidence outside the scope of the review may be relevant– Cost and implementation implications– Generalizability of findings

84

y g

• Readers sometimes have unrealistic expectations– Limitations of the available evidence

• Lack of evidence:– No large and/or good quality trials– No trials at all– Interventions or populations of interest not specifically studied

– Limitations of a review process• The review does not make decisions for you• An individual review may not answer all questions related to a given decision• While a systematic approach reduces bias, it does not eliminate it

• Reviews become outdated

2/20/2012

29

Suggested ReadingsMoher D, et al. Preferred Reporting Items for Systematic Reviews 

and Meta‐Analyses: The PRISMA Statement; PLoS Medicine 6 (2009) 

Juni P, et al.  Systematic reviews in health care:  Assessing the quality of controlled clinical trials. BMJ 2001;323:42–6

85

Guyatt G, et al.  GRADE guidelines: 1. Introduction ‐ GRADE evidence profiles and summary of findings tables; J Clin Epi. 64 (2011) 383‐394

IOM 2008.  Systematic Review:  The Central Link between Evidence and Clinical Decision making.  Report Brief, January, 2008

References & Resources1. Higgins JPT, Green S (editors). Cochrane Handbook for Systematic Reviews of Interventions Version 5.0.2 

[updated September 2009]. The Cochrane Collaboration, 2009. Available from www.cochrane‐handbook.org.

2. Merriam Webster Online Dictionary. Synthesis. Retrieved 2009 from http://www.merriam‐webster.com/dictionary/synthesis.

3. Popay J, Roberts H, Sowden A, et al. Guidance on the Conduct of narrative Synthesis in Systematic Reviews (draft). ESRC Methods Program, UK.

4. Grading of Recommendations Assessment, Development and Evaluation (GRADE) Working Group. (2005‐2011), from http://www.gradeworkinggroup.org/index.htm

5. Bastian, H., Glasziou, P., & Chalmers, I. (2010). Seventy‐five trials and eleven systematic reviews a day: how will we ever keep up? PLoS Med, 7(9), e1000326. doi: 10.1371/journal.pmed.1000326

6. Gould, M. K., Dembitzer, A. D., Sanders, G. D., & Garber, A. M. (1999). Low‐molecular‐weight heparins compared with unfractionated heparin for treatment of acute deep venous thrombosis. A cost‐effectiveness analysis. [Comparative Study Meta‐Analysis Research Support, U.S. Gov't, P.H.S.]. Ann Intern Med, 130(10), 789‐799. 

7. Guyatt, G. H., Oxman, A. D., Schunemann, H. J., Tugwell, P., & Knottnerus, A. (2011). GRADE guidelines: a new series of articles in the Journal of Clinical Epidemiology. J Clin Epidemiol, 64(4), 380‐382. doi: 10.1016/j.jclinepi.2010.09.011

8. Norris, S. L., Kansagara, D., Bougatsos, C., & Fu, R. (2008). Screening adults for type 2 diabetes: a review of the evidence for the U.S. Preventive Services Task Force. [Research Support, U.S. Gov't, P.H.S.Review]. Ann Intern Med, 148(11), 855‐868. 

References & Resources9. Sackett, D. L. (1979). Bias in analytic research. J Chronic Dis, 32(1‐2), 51‐63.

10. Moher D, et al. Preferred Reporting Items for Systematic Reviews and Meta‐Analyses: The PRISMA Statement; PLoS Medicine 6 (2009)

11. Juni P, et al.  Systematic reviews in health care:  Assessing the quality of controlled clinical trials. BMJ 2001;323:42–6

12. Guyatt G, et al.  GRADE guidelines: 1. Introduction ‐ GRADE evidence profiles and summary of findings tables; J Clin Epi. 64 (2011) 383‐394

13. IOM 2008.  Systematic Review:  The Central Link between Evidence and Clinical Decision making.  Report Brief, January, 2008

14. Green S, Higgins JPT, Alderson P, Clarke M, Mulrow CD, Oxman AD. Chapter 1: Introduction. In: Higgins JPT, Green S (editors), Cochrane Handbook for Systematic Reviews of Interventions. Version 5.0.1 [updated September 2008]. The Cochrane Collaboration, 2008. Available from www.cochrane‐handbook.org.

15. Cook, D. J., Sackett, D. L., & Spitzer, W. O. (1995). Methodologic guidelines for systematic reviews of randomized control trials in health care from the Potsdam Consultation on Meta‐Analysis. [Consensus Development Conference Guideline Review]. J Clin Epidemiol, 48(1), 167‐171. 

16. Egger, M., Smith, George Davey, Altman, Douglas. (2001). Systematic Reviews in Health Care: Meta‐Analysis in Context (2 ed.): BMJ Books.

17. Haynes, B. (1999). Can it work? Does it work? Is it worth it? BMJ, 319(7211), 652‐653. doi: 10.1136/bmj.319.7211.652

2/20/2012

30

References & Resources18. Jadad, A. (1998). Randomised Controlled Trials: A User's Guide. BMJ, 317(7167), 1258. doi: 

10.1136/bmj.317.7167.1258

19. Mulrow, C. D., Cook, D. J., & Davidoff, F. (1997). Systematic reviews: critical links in the great chain of evidence. [Comment Editorial]. Ann Intern Med, 126(5), 389‐391. 

20. Research, C. o. S. f. S. R. o. C. E., & Medicine, I. o. (2011). Finding What Works in Health Care: Standards for Systematic Reviews: The National Academies Press.

21. Shea, B., Grimshaw, J., Wells, G., Boers, M., Andersson, N., Hamel, C., . . . Bouter, L. (2007). Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews. BMC Med Res Methodol, 7(1), 10. 

22. Smith, G. C., & Pell, J. P. (2003). Parachute use to prevent death and major trauma related to gravitational challenge: systematic review of randomised controlled trials. [Review]. BMJ, 327(7429), 1459‐1461. doi: 10.1136/bmj.327.7429.1459

23. Sox, H. C., & Greenfield, S. (2009). Comparative effectiveness research: a report from the Institute of Medicine. Ann Intern Med, 151(3), 203‐205.

24. **Rockwood, K., Hogan, D. B., & Patterson, C. J. (2004). Incidence of and risk factors for nodding off at scientific sessions CMAJ (2004/12/08 ed., Vol. 171, pp. 1443‐1445).