systematic reviews and meta-analyses

Systematic Reviews and Meta-Analyses

And the

Cochrane Library

Ritz Kakuma, MSc (PhD Candidate)Department of Epidemiology & Biostatistics McGill University

Outline

• Background on SR/MA

• Cochrane Collaboration

• Cochrane Library

• Example on Mammography Screening for Breast Cancer

Problems with Today's Medical Literature

Most studies are too small Inconclusive, often conflicting results

Traditional reviews are unstructured &

subjective Biased conclusions

WHO CAN KEEP UP?

Over 425,000 trialspublished to date

Over 20,000 new trials published annually Help!

For General Physiciansto keep current:

Read 19 new articles per day which appear in medical journals

19 x 2 hrs (Critical Appraisal) = 38 hrs per day

Davidoff F et al. (1995)

EBM; A new journal to help doctors identify

the information they need. BMJ 310:1085-86.

2002 G. WELCH, S. GABBEStatistics Usage in the Amer J Obstet Gynecol: has anything changed? 180;584-6

All clinical papers: Jan - June 1994 (Vol. 170, No. 1 to 6)

vs. Jan – June 1999 (Vol. 180, No. 1 to 6)

Inappropriate statistics used in 31.7% (46/145) in 1994

and 9.8% (19/195) in 1999.

Statistical Quality of Medical Research getting better…

RESULTS???

Problems with Standard Reviews

Lack of scientific purpose (question) Undocumented methods of literature search Unstated criteria for selecting studies No methodological assessment of selected

studies Inadequate assessment of inter-study

differences in results No attempt at quantitative synthesis (pooling)

to take advantage of increased power

Help to deal with the volume of literature Help resolve conflicting results Scientific rather than subjective summarization of

literature Can reveal new evidence Identify knowledge gaps More reliable evidence with which to aid decision

making Guide clinical research by providing new

hypotheses

Why Systematic Reviews?

The Cochrane Collaboration - origins

• Archie Cochrane– “It is surely a great criticism of our profession that

we have not organized a critical summary, by specialty or subspecialty, adapted periodically, of all relevant randomized controlled trials.”

• Pregnancy and childbirth work - 1980s

• Founded 1993

Aims and objectives of the CC

“The Cochrane Collaboration is an international

organization that aims to help people make

well informed decisions about healthcare by

preparing, maintaining and promoting the

accessibility of systematic reviews of the

effects of healthcare interventions”

Canadian Cochrane Network & Centre

• collaboration • building on the enthusiasm of individuals • avoiding duplication • minimizing bias • keeping up to date • striving for relevance • promoting access • ensuring quality • continuity • enabling wide participation

CC built on 10 PrinciplesCanadian Cochrane Network & Centre

Organization of the CC

• Cochrane Centres (n=12)

• Collaborative Review groups (n=51)

• Fields (n=11)

• Networks (n=1)

• Methods working groups (n=10)

Canadian Cochrane Network & Centre

Cochrane CentresCanadian Cochrane

Network & Centre

• Australasian CC

• Brazilian CC

• Canadian CC

• Chinese CC

• Dutch CC

• German CC

•Iberoamerican CC

• Italian CC

• Nordic CC

• South African CC

• UK CC

• US CC (Rhode Island, Boston, San Francisco Branches)

Cochrane Review GroupsAcute Respiratory Infections Group Airways GroupAnaesthesia Group Back Group Breast Cancer GroupColorectal Cancer GroupConsumers and Communication Group Cystic Fibrosis and Genetic Disorders Group Dementia and Cognitive Improvement GroupDepression, Anxiety and Neurosis Group Developmental, Psychosocial & Learning

Problems GroupDrugs and Alcohol GroupEar, Nose and Throat Disorders Group Effective Practice and Organisation of Care

Group Epilepsy Group Eyes and Vision Group Fertility Regulation GroupGynaecological Cancer Group Haematological Malignancies Group Heart GroupHepato-Biliary GroupHIV/AIDS GroupHypertension GroupIncontinence Group

Canadian Cochrane Network & Centre

Infectious Diseases GroupInflammatory Bowel Disease GroupInjuries Group Lung Cancer GroupMenstrual Disorders and Subfertility GroupMetabolic and Endocrine Disorders Group Methodology Review GroupMovement Disorders GroupMultiple Sclerosis GroupMusculoskeletal GroupMusculoskeletal Injuries GroupNeonatal GroupNeuromuscular Disease GroupOral Health Group Pain, Palliative and Supportive Care Peripheral Vascular Diseases Group Pregnancy and Childbirth GroupProstatic Diseases and Urologic Cancers GroupRenal Group Schizophrenia Group Sexually Transmitted Diseases GroupSkin GroupStroke Group Subfertility Group (see Menstrual Disorders)Tobacco Addiction GroupUpper Gastrointestinal & Pancreatic Diseases GrpWounds Group

Cochrane Fields & Network

• Cancer Network Field• Child Health Field• Complementary Medicine Field• Health Care of Older People Field• Health Promotion and Public Health Field • Neurological Network Field• Occupational Health Field• Prehospital & Emergency Health Field• Primary Health Care Field• Rehabilitation and Related Therapies Field• Vaccines Field

• Consumer Network

Canadian Cochrane Network & Centre

Cochrane Methods Groups

• Applicability and Recommendations• Health Economics • Health-Related Quality of Life• Individual Patient Data Meta-Analyses• Non-randomised Studies • Prospective Meta-Analysis • Qualitative Methods • Reporting Bias Methods • Screening and Diagnostic Tests• Statistical Methods

Canadian Cochrane Network & Centre

Cochrane Activities

• Produce and update systematic reviews

• Hand search for RCTs

• Medline (and others) enhancement

• Review methodology

COCHRANE LIBRARY

Canadian Cochrane Network & Centre

The Cochrane Library

• unique source of reliable and up-to-date information on the effects of interventions in health care

• To provide information and evidence to support decisions taken in health care and to inform those receiving care

• Published on a quarterly basis

Canadian Cochrane Network & Centre

When should you use the CLIB?

For questions on effectiveness

What is the effectiveness of treatment x

What is an effective treatment for y

Is z effective in treating y

Is z better than x at treating y

Canadian Cochrane Network & Centre

When not to use the Cochrane Library

General healthcare questions

• causal, prognosis, epidemiology, etc.• Statistics (prevalence and incidence)• Primary research other than RCTs• Guidelines• Current research

Canadian Cochrane Network & Centre

The Cochrane Database of Systematic Reviews (CDSR)

Complete Reviews

full text, regularly updated systematic reviews of the effects of health care (2,170 reviews)

prepared and maintained by the Collaboration Review Groups

Protocols

protocols of reviews currently being prepared, incl. expected date of completion (1,500 protocols)

includes background, objectives and methods sections

Canadian Cochrane Network & Centre

Canadian Cochrane Network & Centre

The Database of Abstracts of Reviews of Effectiveness

(DARE)

prepared by the National Health Services Centre for Reviews and Dissemination at the University of York, UK.

Abstracts of quality assessed systematic reviews

structured abstracts assessing the quality of previously published SRs & summarizing findings (4,118 reviews)

Other reviews: bibliographic details only

references to published SRs NOT assessed for quality (800 reviews)

Canadian Cochrane Network & Centre

The Cochrane Central Register of Controlled Trials (CENTRAL)

References

Reference list of ALL identified published randomized trials

Latest issue 427,807 RCTs

(Medline: 97,827 articles identified as RCT publication type)

Canadian Cochrane Network & Centre

The Cochrane Database of Methodology Reviews (CDMR)

Complete Reviews

Full-text SRs of methodological studies (10 reviews)

Highly structured and systematic, covering a specific and well-defined area of methodology

prepared and maintained by the Cochrane Methodology Review Groups

Protocol

protocols of reviews currently being prepared, incl. expected date of completion (8 protocols)

includes background, objectives and methods sections

Canadian Cochrane Network & Centre

Cochrane Methodology Register (CMR)

References

published reports of empirical studies of methods used in reviews (5,968 reports)

methodological studies directly relevant to conducting a review

Canadian Cochrane Network & Centre

Health Technology Assessment Database (HTA)

References

HTA covers prevention and rehabilitation, vaccines, pharmaceuticals and devices, medical and surgical procedures and the systems within which health is protected and maintained

Ongoing projects and completed publications from HTA organizations (4,395 citations)

Canadian Cochrane Network & Centre

NHS Economic Evaluation Database (NHS EED)

References

Structured abstracts of articles reporting economic evaluations of health care interventions

Bibliographic details of articles on relevant topics (i.e., burden of illness, economic methodology, reviews of economic evaluations)

N=15,041 citations

Canadian Cochrane Network & Centre

Searching the CLIB

Some basics…

ALL contents of ALL records in ALL databases are searched

Ignores: punctuation & numbers

Boolean terms: AND, OR, NEXT, NEAR – within 6 words both ways, NOT

Restricting searches ‘Options’ page and choose desired restrictions At the end of the term, add:

:AU – Author :TI – Title :ME - MeSH terms

:AB – Abstract :KY - Keywords

Canadian Cochrane Network & Centre

Searching the CLIB

MeSH Keywords drawn from MeSH thesaurus of U.S. NLM Accompanies some but not all records Organized hierarchically in ‘trees’ Permuted Index – an index of all words that appear in the

MeSH thesaurus used to located specific MeSH terms

At McGill University and affiliated sites:

The pathway to the CLIB is:

McGill HSL library homepage > Databases > > Cochrane Library > web (no login or password nec.)

Canadian Cochrane Network & Centre

Useful Materials

The Cochrane Library: Self Training Guide Interpretation of Odd-ratio diagrams WWW links, HTA database Cochrane Handbook User manual Glossary, etc...

DOWNLOADhttp://www.cochrane.org/resources/training.htm

Canadian Cochrane Network & Centre

Quality of Cochrane Reviews

• Comparison of 36 Cochrane reviews with 39 paper-based journals Cochrane reviews less prone to bias (Jadad et al. 1998)– Explicit reporting of eligibility criteria

(35/36 vs 18/39)– Assessed trial quality (36/36 vs 7/39)– No language restriction (36/36 vs 32/39)

Quality of Cochrane Reviews

• Also not perfect (Olsen et al. 2001)

• 52 Cochrane reviews from 1998• 18% - conclusions not backed up by evidence

– All overestimated effect of intervention

Discordant Reviews

• Direction of effect• Significance• Magnitude of effect• Interpretation of results

Numerous possible reasons

Sources of discordance

• Different research question– Target population– Intervention being studied– Outcome measures– Setting

• Study selection– Search strategy– Eligibility criteria

Sources of discordance

• Data extraction– Methods of measuring outcomes and endpoints– Extent of human error

• Quality assessment method– 25 scales and 9 checklists avail for assessing RCT

quality (Moher et al. 1996)• Inconsistent quality depending on instrument used

– Additional 8 instruments (Juni et al. 1999)– QUOROM (Quality of Reporting of Meta-Analyses)

= most comprehensive(Moher et al. 1999, Shea et al. 2001)

Sources of discordance

• Analysis– Appropriateness of combining results– Method of synthesis: Descriptive, meta-analysis– Statistical methods

• Bayesian• Meta-regression• Frequentist

• Interpretation of evidence

Decision Algorithm for interpreting discordant reviews

Jadad, Cook & Browman. A guide to interpreting discordant systematic reviews. CMAJ. 1997 May 15;156(10):1411-6.

Screening for breast cancer with mammography among

women aged 50-69 years  

Report prepared for the Breast Cancer Screening Unit,Cancer Branch, Health Canada

Ritsuko KakumaMarch 2002

SRs on mammography screening for breast cancer

• Fletcher et al. Report of the International Workshop on Screening for Breast Cancer. J Natl Cancer Inst 1993; 85:1644-1656.

• Kerlikowske et al. Efficacy of screening mammography. A meta-analysis. JAMA 1995; 273:149-154.

• Olsen O & Gøtzsche PC. Is screening for breast cancer with mammography justifiable? Lancet 2000.

Fletcher et al. review

Research Question: • To assess current state of knowledge about BRCA

screening, identify knowledge gaps

Study Sample:• Women aged 40-79 years stratified into 3 groups: 40-

49, 50-69, 70+

Fletcher et al. review

Methodological issues: • Int’l Workshop on screening for BRCA in Feb 1993 (by

NCI) – characteristics of participants?

• Search strategy not reported• Study selection method not described• Data source: published and unpublished data

provided by the w/s participants• Validity of trials assessed:

– Randomization, Compliance, Contamination– Quality and frequency of screening test– Adequacy of FU– Validity of outcome assessment– Generalizability of results

Fletcher et al. review

Results/Conclusions: • 7 trials identified• For age 50-69 age group, only 5 trials provided data

– New York, Two-County, Malmo, Edinburgh and Canada

• No statistical pooling of data

• Together, Fletcher et al. concluded that there was substantial benefit of screening on BRCA mortality reduction of 30% after 10-12 years of FU

Kerlikowske et al. review

Research Question: • To determine efficacy of screening in reducing BRCA

mortality by age, # MM views per screen, screening interval and duration of FU

Sample: • Women aged 40-74 years stratified into 3 groups: 40-

49, 40-74, 50-74

Kerlikowske et al. review

Methodology: • Search: MEDLINE, manual (reference lists) and

consultations with colleagues and experts– Search terms, dates covered

• Study Selection – English language restriction, explicit inclusion criteria– Not restricted to RCTs

• Data extraction– 2 indep extractors & 3rd for resolving discrepancies– Published and unpublished data used

• Quality assessment not reported..not done?• Analysis

– Pooled data of all included trials using fixed effects

Kerlikowske et al. review

Results / Conclusions: • 9 studies (7 RCTs & 2 CCS) have data on 50-74 year

age group

Comparison* Relative Risks

RCT vs. Case-control studies 0.77 (0.69-0.87) vs. 0.45 (0.29-0.70)

No. of MM views: 2-views vs. 1-view 0.70 (0.58-0.84) vs. 0.83 (0.71-0.97)

Screening interval: 12 mo. vs. 18-33 mo. 0.77 (0.59-1.0) vs. 0.77 (0.68-0.88)

FU duration: 7-9 yrs vs. 10-12 yrs 0.73 (0.63-0.84) vs. 0.76 (0.67-0.87)

Screening duration: 3-5 yrs vs. 8-10 yrs 0.76 (0.62-0.95) vs. 0.78 (0.67-0.90)

CBE + MM vs. MM alone 0.80 (0.66-0.98) vs. 0.76 (0.65-0.88)

Study start date: Pre-1980 vs. Post-1980 0.76 (0.67-0.87) vs. 0.83 (0.63-1.10)

Kerlikowske et al. review

Results / Conclusions: • All studies – beneficial effect of screening

(4 statistically significant)• All trials included in the analysis (not CCSs)

• MM screening reduced BRCA mortality by 27% in this age group after 7- 9 years and 24% after 10-12 years regardless of # MM views, screening interval, duration of screening or addition of CBE

Olsen & Gøtzsche review

Research Question: • To assess effect of MM screening for BRCA on

mortality and morbidity• Review methodological quality of trials – focus on 3

most important sources of bias – Randomization – blind outcome assessment– post-Rx exclusions

Sample: • Women without previously diagnosed BRCA on

women aged <50 years and 50+ years

Olsen & Gøtzsche reviewMethodology: • Search: MEDLINE, common author names search,

reference lists, specialized trials registers, letters, abstracts, grey literature– Search terms, dates covered

• Study Selection:– Explicit inclusion criteria (RCT / quasi-RCT on MM screening

vs. no MM screening)• Two indep reviewers for study selection, quality

assessment & data extraction– Authors of studies contacted for missing information

• Quality assessment: High, Medium, Poor or Flawed– Rx process, baseline comparability, post-Rx exclusions,

Consistency of number of women randomized• Analysis

– Intent-to-Treat analysis, with fixed effect unless heterogeneous– Stratified by quality (excluded flawed)

Olsen & Gøtzsche review

Results / Conclusions: • 7 trials identified, none with high quality (2 medium, 3

poor, 2 flawed)• Total mortality

– no impact of screening after 7- and 13 yrs FU for both medium and poor quality trials

• BRCA mortality – no beneficial effect of screening in medium quality trials– Beneficial effect among POOR quality trials

pooled RR=0.69, CI 0.55-0.80 at 7 yearspooled RR=0.64, CI 0.54-0.76 at 13 years

• Interventions for screened group higher– Over-diagnosis, increased use of more aggressive treatment

such as mastectomies and tumorectomies (20-30%)

Decision Algorithm for interpreting discordant reviews

Identified Trials

•Sampling Randomization: Canadian study recruited volunteers, other used population-based lists

•Assessment of pre-existing BRCA: pre- vs. post-Rx

Trials: Methodological issues

Canadian Efficacy trial, not effectiveness (women = screened, not ‘invited’ to be screened)

CBE in control group may dilute effect of screening (MM+CBE vs. CBE)

Imbalance in rate of advanced cancers at Rx

Malmö Inconsistent numbers across multiple reports – missing unaccounted for

Date of entry different for the 2 groups (date of Rx vs. date of invitation to screening)

Contamination: 24% of controls had MM

Continuation of study: not randomized, no baseline information

Stockholm Inconsistent numbers across multiple reports– missing unaccounted for

Non-independence of results of 2 sub-trials (2 smaller trials with overlapping controls)

Imbalanced post-Rx exclusions, Contamination (Screening for controls began as early as year 3)

Göteborg Information on Rx not described adequately

Contamination (Screening for controls began 3rd and 6th years)

No separate data on 50+ age group

Trials: Methodological issues

Two-County

Age imbalance of 5-months – older in screening group (clinically signif?)

Cluster Rx method not described adequately

Contamination (Screening for controls began after a few years into study)

New York Baseline imbalance in previous lump, menopause and education

Post-Rx exclusion ID method of current BRCA differed --self-report vs. screen

Outcome assessment (cause of death): 72% non-blind (differential misclassification or true effect?)

Edinburgh Inadequate Rx Baseline imbalances – screened group had higher SES

Change in allocation status after Rx

Imbalanced post-Rx exclusions

Contamination (Screening for controls began in year 10 of FU)

Differential eligibility criteria and entry dates for the 2 groups

Impact on effect measure

• Contamination & Compliance : underestimate– Introduction of screening in controls– Screened vs. invited to be screened

• Post-Rx exclusions: overestimate– Identification of BRCA at baseline more complete for

screened group..first round of screening used to exclude– Dropouts and losses: equally distributed?

• Adequacy of Randomization: overestimate– Typically overestimate treatment effect by 30-40%

(Schulz et al. 1995; Moher et al. 1998; Kjaergard et al. 2001)

• Outcome Assessment: overestimate– Misclassification– Determination of cause of death may be difficult in light of

multiple illnesses

Comparability of Trials

• Same research question?– Effectiveness vs. efficacy

• Same Target population?

• Appropriate study sample?

• Intervention?– Screening modality, frequency of screening

• Outcome assessment– Cause of death identified the same way?

• Analytical methods

Fletcher et al. No precise research question Inadequate description of methods used

– Search strategy, trial selection, data extraction Acknowledgement of difference in quality of trials but

conclude based on ALL evidence Source of some data unclear

Quality assessment carried out NOT synthesizing data = appropriate• But evidence does not support their conclusion that

BRCA mortality is reduced by 30% as a result of screening

Kerlikowske et al.

Clear research question Explicit reporting of methods used

No indication that quality assessment was done Some numbers extracted don’t match reference cited Missing key articles for trials (I.e., original report)

• Despite good reporting of methods, caution needed in interpreting results because of the questionable nature of data used

Olsen and Gotzsche

Clear research question Explicit reporting of methods used Strong methodology Addressed important question:

Reduced BRCA mortality not necessarily equal to improved survival

Did not assess baseline comparability stratified by age groups, which is how the data was analyzed

• Most rigorous and properly done review of the 3• Brought potential adverse effects of screening to

forefront

Conclusions of assessment:

• All existing evidence has methodological issues • No conclusive evidence exists to support MM• Other issues:

– Accuracy of MM (false positives?)– Detection of slow-growing tumors that may never develop

into invasive cancers– Psychological impact of positive results

• US and Canada continue to recommend MM screening for women in this age group

Overall conclusions

• MA not appropriate in many cases • Must be able to identify the potential

sources of bias in both the systematic review as well as the studies included in the review

• Cochrane reviews are not always perfect but generally better than non-Cochrane reviews