1 systemic review and meta- analysis in cancer epidemiology chun rebecca chao, ph.d. kaiser...

1

Systemic Review and Meta-Analysis in Cancer Epidemiology

Chun Rebecca Chao, Ph.D.

Kaiser Permanente Southern California

Department of Research and Evaluation

2

Overview

Introduction of systematic review and meta-analysis

Methods of systemic review and meta-analysis

3

What is Systemic Review?

A review that has been prepared using a systemic approach to minimize biases and random errors (which is documented in a materials and methods section)*. Review in which there is a comprehensive search for

ALL relevant studies on a specific topic, and those identified are then appraised and synthesized according to a predetermined and explicit method.

Systemic review vs. Narrative review Narrative Review: traditional expert review, subjective,

no formal rules in selecting studies, no standard statistical methods for combining studies.

*Chalmers and Altman. Systemic Reviews, BMJ Publishing Group, 1995.

4

What is Meta-Analysis?

A statistical analysis of the results from independent studies, which generally aims to produce a single estimate of a treatment effect.

A systemic review may or may not include a meta-analysis.

It is always appropriate and desirable to systemically review a body of data, but it is sometimes inappropriate to statistically pool results from different studies.

5

The Need for Systemic Review

Health Care Professional: number of biomedical publication has been increasing rapidly.

Guideline and Policy Makers: Evidence based medicine is the trend for patient care, clinical guideline development and policy making.

Researchers: future research direction can be guided by systemic reviews.

Consumers

6

Strength of Evidence Concerning Efficacy of Treatment Case report Case series without controls Series with literature controls Series with historical controls Case control studies Cohort studies Randomized controlled trials (RCTs) SR/Meta-analysis of RCTs Prospective meta-analysis of RCTs with individual

data

7

Strengths of Systemic Review

To provide a more objective appraisal of the evidence

Contribute to resolve uncertainty when original research, reviews and editorials disagree.

Meta-analysis of the BCG vaccine trials

Graph adopted from Egger, Smith and Altman. Systemic Reviews in Health Care. BMJ

8

Strengths of Systemic Review (Cont.)

To provide a more objective appraisal of the evidence

Contribute to resolve uncertainty when original research, reviews and editorials disagree.

To reduce the probability of false negative results To explore treatment effects in subgroups of patients To explore and explain heterogeneity between study results To guide the direction of future studies

To understand current gap and limitation in the literature To generate new research questions to be addressed.

9

Advantages of Use of Meta-Analysis to Combine Studies When individual trials or studies are too small

to give reliable answers When large trials or studies are impractical or

impossible Potentially lead to more timely introduction of

effective treatment When there have been many trials or studies

showing small effects may be important

10

Number of Publications about Meta-Analysis, 1985-2005

Results from MEDLINE search using MeSH and text word “meta-analysis”

0

500

1000

1500

2000

2500

11

Potential Limitation of Conducting Systemic Review Bias can be introduced in reviews in several

ways. Problems associated with design or reporting

of original studies Limitations of using published data Publication bias

12

Publication Bias – A real threat for systemic review Studies with significant results are

more likely to be published More likely to be published without delay (lag time) More likely to be published in English More likely to be cited More likely to be published more than once

Outcome reporting bias Significant outcomes are more likely to be reported than

non-significant outcomes. Should unpublished data be included in systemic

review? Pre-specified inclusion (quality) criteria are

recommended.

13

A Demonstration of Publication Bias

Graph adopted from Egger, Smith and Altman. Systemic Reviews in Health Care. BMJ

14

Predictors of Publication

Graph adopted from Egger, Smith and Altman. Systemic Reviews in Health Care. BMJ

15

Language Bias

Graph adopted from Egger, Smith and Altman. Systemic Reviews in Health Care. BMJ

16

Pan Z et al. PLoS Med. 2005 Dec;2(12):e334

Publication/Reporting Bias?

17

Meta-Analysis of Observational Studies In observational studies, bigger is not necessarily

better. This is a danger that meta-analysis of observational

data produces precise but spurious results. A careful systemic review and examination of source

of heterogeneity are more important than combining results. Statistical combination of data should not be the main

component of systemic review of observational data. It is often desirable to have individual level data for

this purpose. Confounding and effect modifiers.

18

Example 1. Fat Intake and Risk of Breast Cancer

Graph adopted from Egger, Smith and Altman. Systemic Reviews in Health Care. BMJ

Recall bias might play a role in the findings from case-control studies.

19

Example 2: Intermittent Sunlight Exposure and Risk of Melanoma

Graph adopted from Egger, Smith and Altman. Systemic Reviews in Health Care. BMJ

Lack of blinding of the hypothesis to the subjects may introduce recall bias.

20

Example 3. Formaldehyde Exposure and Lung Cancer

Graph adopted from Egger, Smith and Altman. Systemic Reviews in Health Care. BMJ

Dose-response or selection bias?

21

Carotenoids and the Risk of Developing Lung Cancer: A Systemic Review*. Objective: Systemic review of the association

between carotenoids and lung cancer. Included both RCT and prospective observational

studies (smoking adjusted). Examined total carotenoids, β-carotene, α-carotene,

β-cyrptoxanthin, lycopene, and lutein-zeaxanthin. Examined carotenoid supplements, dietary intake of

carotenoids, and serum carotenoid concentrations.

*Gallicchio L et al, Am J Clin Nutr 2008; 88: 372-83.

22

β-carotene Supplement Use in RCT

Gallicchio et al, Am J Clin Nutr 2008; 88: 371-83

23

Dietary Total Carotenoid Intake: Prospective Cohort Studies

Gallicchio et al, Am J Clin Nutr 2008; 88: 371-83

24

Dietary β-carotene Intake: Prospective Cohort Studies

Gallicchio et al, Am J Clin Nutr 2008; 88: 371-83

25

Serum Total Carotenoid Concentrations: Prospective Cohort Studies.

Gallicchio et al, Am J Clin Nutr 2008; 88: 371-83

26

Dose Response

Gallicchio et al, Am J Clin Nutr 2008; 88: 371-83

27

Subgroup Analysis

Gallicchio et al, Am J Clin Nutr 2008; 88: 371-83

28

Method for Systemic Review

1. Define research questions 2. Define inclusion/exclusion criteria 3. Search the literature 4. Select articles 5. Evaluate the internal/external validity of the studies 6. Extract/Abstract Data 7. Calculate effect size and standard error 8. Examine heterogeneity 9. Assess publication bias 10. Combining study results if appropriate 11. Influence analysis, sensitivity analysis 12. Interpretation of results 13. Reporting

29

Stages of a Systemic Review

Stage 1Planning

Identification of the need

for a systemic review

Development of a proposal

Stage 2Conducting

Stage 3Reporting

Identification of the research

Selection of studies

Study quality assessment

Data extraction

Data synthesis

The report and recommendations

30

1. Clearly Formulated Question The research question is extremely important!

Get feedback about your specific research question from many people (content expert, savvy clinician in the field, methodologist)

The question should be clearly specified What is the study objective

To validate results in a large population To guide new studies

Pose questions in both biologic and health care terms specifying with operational definitions Population Intervention/exposure Outcomes (both beneficial and harmful)

31

Study Protocol

Develop a protocol of what will be done

Background: why is this important? Specify the research question

Research question defines the following criteria Provide an overview of the methods, including search

strategy, inclusion/exclusion criteria, quality assessment, how data will be abstracted, and an analysis plan

Specify any subgroup analyses or sensitivity analyses (best if these are a priori)

Initially, inclusion criteria should be overly broad E.g., search all alcohol when specifically interested in wine. E.g., search all dietary, anthropometric related papers when

interested in carotenoids.

32

2. Develop Inclusion Criteria

Use clinical/scientific judgment to enhance

validity and homogeneity

Validity (Study quality)

Homogeneity Similar study design

Similar patient populations

Similar interventions/exposures

Similar outcomes

33

Typical Inclusion Criteria

Study Design (e.g., RCT, prospective cohort) Population (risk group) Interventions (dosage, regimen)/exposure of

interest Outcomes (definitions, follow-up period)

34

Practical Considerations in Defining Eligibility for a Systemic Review Study designs to be included Years of publication or study conduct Languages Choice among multiple publications Restrictions due to sample size or follow-up

duration Similarity of treatment and/or exposure Completeness of information

35

Copyright restrictions may apply.

Chao, C. et al. Am. J. Epidemiol. 2008 168:471-480; doi:10.1093/aje/kwn160

List of excluded studies

36

3. Comprehensive Literature Search

Need a well formulated and coordinated effort Seek guidance from a librarian Specify language constraints Requirements for comprehensiveness of the

search depends on the field and question to be addressed

37

Searching the Literature

Database searches: MEDLINE, EMBASE, ISI Web of Science, PsychLit, CancerLit, Cochrane, Dissertations online

Reference lists of retrieved articles Manual searching of related journals, conference proceedings,

textbooks Experts in the field Granting agencies Study/trial registries Industry (device manufacturers, pharmaceutical companies)

38

How to Develop a Search Strategy (Example)

Study Question What is the relationship between consumption of different

types of alcoholic beverage (beer, wine and liquor) and risk of lung cancer?

Break down the question into facets (not all may be needed for searching) Exposure: Alcoholic beverage consumption Outcome: Lung cancer Study design: Observational studies with internal

comparison group (case-control or cohort)

39

How to Develop a Search Strategy (Cont.) Identify synonyms, spelling variants, and

subject headings associated with each facet Text terms (title/abstract) MeSH

alcohol alcohol drinking

ethanol ethanol

alcoholic

alcoholics

beer

wine

liquor

40

How to Develop a Search Strategy (Cont.)

Text terms (title/abstract) MeSH

lung lung neoplasm

in combination with

cancer

neoplasm

carcinoma

tumor

incidence

mortality

41

4. Study Selection

Unbiased Study Selection Two independent reviewers select studies Based on a priori specification of the population,

intervention, outcomes and study design Differences are resolved by consensus Specify reasons for rejecting studies

Keep a record of what is excluded and reason (journals may require this and the search must be reproducible)

42

Flow Diagram of Study Selection ProcessPotentially relevant references identified after liberal screening of the electronic search (n=#)

Excluded by Title/Abstract (n=#) List the reasons

Articles retrieved for more detailed evaluation (n=#)

Articles excluded after evaluation of full text (n=#) List the reasons

Relevant studies included in the meta-analysis (n=#)

Figure 1.

43

Carotenoids and Lung Cancer Systemic Review

Gallicchio et al, Am J Clin Nutr 2008; 88: 371-83

44

Study Quality Assessment

Quality of study conduct vs. quality of reporting Controversial since this process is subjective and can introduce bias No standard method to assess quality of a study Scales and checklists exist but can be challenging to incorporate for

observational studies Pre-specify algorithm for quality assessment Assess quality of each study in uniform, systematic and complete

manner Once quality is assessed then what to do with that information

Consider weighting each study result by quality score Exclude studies with ‘poor’ quality Stratifying by methodological quality component (component

approach)

45

5. Data Extraction

Develop an easy-to-fill-out form for collecting data from each article Study methods (quality) Study description (population) Results (outcomes, side effects)

Two independent reviewers extract data Should be explicit, unbiased, and reproducible Include all relevant measures of benefit and harm of the

intervention Contact investigators of the studies for clarification in

published methods, data Differences in data extraction are resolved by consensus

46

Data Extraction: Study Characteristics Types of publication (journal article, abstract or

unpublished data) Publication year and country of origin Study participants (sample size, age, gender, race, health

status) Design details (case-control, cohort, parallel or cross-over,

randomization, blinding) Nature of treatment and control Study duration Measurement of compliance Definition and measurement of outcome Other confounders

47

Continuous data Outcomes summarized as means (blood pressure, weight)

Dichotomous or binary data Each individual must be in one of two states (dead/alive,

smoking/not smoking) These data are summarized using odds ratios, risk ratios, or risk

differences Survival or time to event data

Outcome of interest is the time to the occurrence of an event Usually summarized using hazard ratios

Other data Some outcomes may be

Short ordinal scales (pain scales: individuals’ rate their pain as none, mild, moderate, severe) for which it is not sensible to calculate a mean

Event counts (number of asthma attacks per month)

Data Extraction:Study Outcome

48

Gallicchio et al, Am J Clin Nutr 2008; 88: 371-83

49

6. Calculating Effect Size and Standard Error

Perform a narrative, qualitative summary when data are too sparse, or too low quality or too heterogeneous to proceed with a meta-analysis

The results from each study are converted into an Odds Ratio (OR) or Effect Size (ES)

95% confidence intervals (CI) are calculated for each study-specific OR or ES

For a meta-analysis, If only confidence intervals are given, computation will be required to obtain estimates of the standard error

50

7. Examining Heterogeneity

Statistical test for heterogeneity Visual inspection/Graphical approach

Forest plot, Galbraith plot Meta-regression

Unit of regression: study Dependent variable: study-specific effect

estimate Independent variables: study-specific

characteristics (e.g., study design, geographic location, length of follow-up)

51

Examining Forest Plot for Heterogeneity

Graph adopted from Egger, Smith and Altman. Systemic Reviews in Health Care. BMJ

52

Meta-regression STATA Output

Type 2 diabetes and risk of NHL: systemic review and meta-analysis

---------------------------------------------------------------------------------------- | Coef. Std. Err. z P>|z| [95% Conf. Interval] ---------------------------------------------------------------------------------------- design | .4256531 .203504 2.09 0.036 .0267926 .8245136 _cons | .167641 .083123 2.02 0.044 .0047228 .3305591 ---------------------------------------------------------------------------------------- Reference group: case-control design ---------------------------------------------------------------------------------------- | Coef. Std. Err. z P>|z| [95% Conf. Interval] ---------------------------------------------------------------------------------------- europe | .1077322 .1625632 0.66 0.508 -.2108859 .4263503 asia | .4499953 .1966564 2.29 0.022 .064556 .8354347 _cons | .107974 .1095071 0.99 0.324 -.106656 .3226041 ---------------------------------------------------------------------------------------- Reference group: US studies

53

8. Assessing Publication Bias

Results because negative studies are less likely to be submitted or published

Can bias the results of a meta-analysis toward a positive finding

Can evaluate publication bias graphically (funnel plot) or through statistical analysis Egger’s test, Begg’s test

54

Begg’s Funnel Plot

Graph adopted from Egger, Smith and Altman. Systemic Reviews in Health Care. BMJ

55

Copyright restrictions may apply.

Chao, C. et al. Am. J. Epidemiol. 2008 168:471-480; doi:10.1093/aje/kwn160

Begg's funnel plot with pseudo 95% confidence limits for assessment of publication bias

56

9. Meta-Analysis for Calculating a Summary Effect Estimate Several methods are available for combining

study results Inverse variance method M-H methods Peto’s odds ratio method

Fixed effect vs. random effect

57

Subgroup Analyses

Pre-specify hypothesis-testing subgroup analyses and keep few in number

Label all posteriori subgroup analyses When subgroup differences are detected, interpret in

light of whether they were: Established a priori Few in number Supported by plausible mechanisms Important (qualitative vs. quantitative) Consistent across studies Statistically significant (adjusted for multiple testing)

58

10. Influence Analysis

Examine how each study influence the summary statistic by removing one study at a time and re-calculate the combined estimate.

A graphically display can be used for visual inspection of influential studies.

59

Influence Analysis STATA Output

Wine consumption and risk of lung cancer: systemic review and meta-analysis ------------------------------------------------------------------------------ Study ommited | e^coef. [95% Conf. Interval] ------------------------------------------------------------------------------ Bendera 1992 | .831761 .67413002 1.0262506 De Stefani 1993 | .78909296 .65476894 .95097321 Carpenter 1998 | .81794792 .66839695 1.0009602 De Stefani 2002 | .82806259 .67865324 1.0103652 Hu 2002 | .82721388 .67337066 1.0162053 Freudenheim 2003 | .82455081 .67135686 1.0127014 Benedetti I 2006 | .82831144 .67354399 1.0186415 Benedetti II 2006 | .81791788 .66001374 1.0135996 Benedetti II 2006 | .82721388 .67337066 1.0162053 Pollack 1984 | .78993553 .67183363 .92879862 Prescott 1999 | .84944367 .70591372 1.0221567 Prescott 1999 | .83032089 .69030547 .99873579 Freudenheim 2005 | .81104547 .65682006 1.001484 Freudenheim 2005 | .78433573 .64188826 .9583950 ------------------------------------------------------------------------------- Combined | .81770466 .67530395 .99013327 -------------------------------------------------------------------------------

60

Influence Analysis STATA Output

Graph adopted from Egger, Smith and Altman. Systemic Reviews in Health Care. BMJ

61

10. Sensitivity Analysis

Test robustness of results relative to key features of the studies and key assumptions and decisions

Include tests of bias due to retrospective nature (e.g., with/without studies of lower methodological quality)

62

Component Approach

Graph adopted from Egger, Smith and Altman. Systemic Reviews in Health Care. BMJ

63

11. Interpretation of Findings

Interpret results in context of clinical practice State methodological limitations of the individual studies

included and in the meta-analysis Consider size of effect in studies and meta-analysis,

consistency of effect sizes and any dose-response relationship

Interpret results in light of other valuable evidence Make recommendations clear and practical Propose future research agenda (clinical and methodological

requirements)

64

12. Describe Studies and Findings

Guideline for reporting systemic review The QUOROM

statement (Quality Of Reporting Of Meta-analysis)

Moher D, et al. Improving the quality of reports of meta-analyses of randomised controlled trials: the QUOROM statement. Quality of Reporting of Meta-analyses. Lancet 1999, 354(9193):1896-900.

65

QUOROM Check List

Moher D, et al. Lancet 1999, 354(9193):1896-900.

66

Summary

A well conducted systemic review allows for a more objective appraisal of the evidence than traditional narrative reviews

Systemic review may contribute to resolve uncertainties and disagreements in original research

Meta-analysis may enhance the precision of estimates of treatment effects

Exploratory analyses (i.e., subgroups who are likely to respond well to a treatment) may guide cost effective treatment decisions

Systemic review may demonstrate areas where the evidence is inadequate and thus identity areas where further research is needed