systematic reviews and meta-analyses
DESCRIPTION
Systematic Reviews and Meta-Analyses. And the Cochrane Library. Ritz Kakuma, MSc (PhD Candidate) Department of Epidemiology & Biostatistics McGill University. Outline. Background on SR/MA Cochrane Collaboration Cochrane Library Example on Mammography Screening for Breast Cancer. - PowerPoint PPT PresentationTRANSCRIPT
Systematic Reviews and Meta-Analyses
And the
Cochrane Library
Ritz Kakuma, MSc (PhD Candidate)Department of Epidemiology & Biostatistics McGill University
Outline
• Background on SR/MA
• Cochrane Collaboration
• Cochrane Library
• Example on Mammography Screening for Breast Cancer
Problems with Today's Medical Literature
Most studies are too small Inconclusive, often conflicting results
Traditional reviews are unstructured &
subjective Biased conclusions
WHO CAN KEEP UP?
Over 425,000 trialspublished to date
Over 20,000 new trials published annually Help!
For General Physiciansto keep current:
Read 19 new articles per day which appear in medical journals
19 x 2 hrs (Critical Appraisal) = 38 hrs per day
Davidoff F et al. (1995)
EBM; A new journal to help doctors identify
the information they need. BMJ 310:1085-86.
2002 G. WELCH, S. GABBEStatistics Usage in the Amer J Obstet Gynecol: has anything changed? 180;584-6
All clinical papers: Jan - June 1994 (Vol. 170, No. 1 to 6)
vs. Jan – June 1999 (Vol. 180, No. 1 to 6)
Inappropriate statistics used in 31.7% (46/145) in 1994
and 9.8% (19/195) in 1999.
Statistical Quality of Medical Research getting better…
RESULTS???
Problems with Standard Reviews
Lack of scientific purpose (question) Undocumented methods of literature search Unstated criteria for selecting studies No methodological assessment of selected
studies Inadequate assessment of inter-study
differences in results No attempt at quantitative synthesis (pooling)
to take advantage of increased power
Help to deal with the volume of literature Help resolve conflicting results Scientific rather than subjective summarization of
literature Can reveal new evidence Identify knowledge gaps More reliable evidence with which to aid decision
making Guide clinical research by providing new
hypotheses
Why Systematic Reviews?
The Cochrane Collaboration - origins
• Archie Cochrane– “It is surely a great criticism of our profession that
we have not organized a critical summary, by specialty or subspecialty, adapted periodically, of all relevant randomized controlled trials.”
• Pregnancy and childbirth work - 1980s
• Founded 1993
Aims and objectives of the CC
“The Cochrane Collaboration is an international
organization that aims to help people make
well informed decisions about healthcare by
preparing, maintaining and promoting the
accessibility of systematic reviews of the
effects of healthcare interventions”
Canadian Cochrane Network & Centre
• collaboration • building on the enthusiasm of individuals • avoiding duplication • minimizing bias • keeping up to date • striving for relevance • promoting access • ensuring quality • continuity • enabling wide participation
CC built on 10 PrinciplesCanadian Cochrane Network & Centre
Organization of the CC
• Cochrane Centres (n=12)
• Collaborative Review groups (n=51)
• Fields (n=11)
• Networks (n=1)
• Methods working groups (n=10)
Canadian Cochrane Network & Centre
Cochrane CentresCanadian Cochrane
Network & Centre
• Australasian CC
• Brazilian CC
• Canadian CC
• Chinese CC
• Dutch CC
• German CC
•Iberoamerican CC
• Italian CC
• Nordic CC
• South African CC
• UK CC
• US CC (Rhode Island, Boston, San Francisco Branches)
Cochrane Review GroupsAcute Respiratory Infections Group Airways GroupAnaesthesia Group Back Group Breast Cancer GroupColorectal Cancer GroupConsumers and Communication Group Cystic Fibrosis and Genetic Disorders Group Dementia and Cognitive Improvement GroupDepression, Anxiety and Neurosis Group Developmental, Psychosocial & Learning
Problems GroupDrugs and Alcohol GroupEar, Nose and Throat Disorders Group Effective Practice and Organisation of Care
Group Epilepsy Group Eyes and Vision Group Fertility Regulation GroupGynaecological Cancer Group Haematological Malignancies Group Heart GroupHepato-Biliary GroupHIV/AIDS GroupHypertension GroupIncontinence Group
Canadian Cochrane Network & Centre
Infectious Diseases GroupInflammatory Bowel Disease GroupInjuries Group Lung Cancer GroupMenstrual Disorders and Subfertility GroupMetabolic and Endocrine Disorders Group Methodology Review GroupMovement Disorders GroupMultiple Sclerosis GroupMusculoskeletal GroupMusculoskeletal Injuries GroupNeonatal GroupNeuromuscular Disease GroupOral Health Group Pain, Palliative and Supportive Care Peripheral Vascular Diseases Group Pregnancy and Childbirth GroupProstatic Diseases and Urologic Cancers GroupRenal Group Schizophrenia Group Sexually Transmitted Diseases GroupSkin GroupStroke Group Subfertility Group (see Menstrual Disorders)Tobacco Addiction GroupUpper Gastrointestinal & Pancreatic Diseases GrpWounds Group
Cochrane Fields & Network
• Cancer Network Field• Child Health Field• Complementary Medicine Field• Health Care of Older People Field• Health Promotion and Public Health Field • Neurological Network Field• Occupational Health Field• Prehospital & Emergency Health Field• Primary Health Care Field• Rehabilitation and Related Therapies Field• Vaccines Field
• Consumer Network
Canadian Cochrane Network & Centre
Cochrane Methods Groups
• Applicability and Recommendations• Health Economics • Health-Related Quality of Life• Individual Patient Data Meta-Analyses• Non-randomised Studies • Prospective Meta-Analysis • Qualitative Methods • Reporting Bias Methods • Screening and Diagnostic Tests• Statistical Methods
Canadian Cochrane Network & Centre
Cochrane Activities
• Produce and update systematic reviews
• Hand search for RCTs
• Medline (and others) enhancement
• Review methodology
COCHRANE LIBRARY
Canadian Cochrane Network & Centre
The Cochrane Library
• unique source of reliable and up-to-date information on the effects of interventions in health care
• To provide information and evidence to support decisions taken in health care and to inform those receiving care
• Published on a quarterly basis
Canadian Cochrane Network & Centre
When should you use the CLIB?
For questions on effectiveness
What is the effectiveness of treatment x
What is an effective treatment for y
Is z effective in treating y
Is z better than x at treating y
Canadian Cochrane Network & Centre
When not to use the Cochrane Library
General healthcare questions
• causal, prognosis, epidemiology, etc.• Statistics (prevalence and incidence)• Primary research other than RCTs• Guidelines• Current research
Canadian Cochrane Network & Centre
The Cochrane Database of Systematic Reviews (CDSR)
Complete Reviews
full text, regularly updated systematic reviews of the effects of health care (2,170 reviews)
prepared and maintained by the Collaboration Review Groups
Protocols
protocols of reviews currently being prepared, incl. expected date of completion (1,500 protocols)
includes background, objectives and methods sections
Canadian Cochrane Network & Centre
Canadian Cochrane Network & Centre
The Database of Abstracts of Reviews of Effectiveness
(DARE)
prepared by the National Health Services Centre for Reviews and Dissemination at the University of York, UK.
Abstracts of quality assessed systematic reviews
structured abstracts assessing the quality of previously published SRs & summarizing findings (4,118 reviews)
Other reviews: bibliographic details only
references to published SRs NOT assessed for quality (800 reviews)
Canadian Cochrane Network & Centre
The Cochrane Central Register of Controlled Trials (CENTRAL)
References
Reference list of ALL identified published randomized trials
Latest issue 427,807 RCTs
(Medline: 97,827 articles identified as RCT publication type)
Canadian Cochrane Network & Centre
The Cochrane Database of Methodology Reviews (CDMR)
Complete Reviews
Full-text SRs of methodological studies (10 reviews)
Highly structured and systematic, covering a specific and well-defined area of methodology
prepared and maintained by the Cochrane Methodology Review Groups
Protocol
protocols of reviews currently being prepared, incl. expected date of completion (8 protocols)
includes background, objectives and methods sections
Canadian Cochrane Network & Centre
Cochrane Methodology Register (CMR)
References
published reports of empirical studies of methods used in reviews (5,968 reports)
methodological studies directly relevant to conducting a review
Canadian Cochrane Network & Centre
Health Technology Assessment Database (HTA)
References
HTA covers prevention and rehabilitation, vaccines, pharmaceuticals and devices, medical and surgical procedures and the systems within which health is protected and maintained
Ongoing projects and completed publications from HTA organizations (4,395 citations)
Canadian Cochrane Network & Centre
NHS Economic Evaluation Database (NHS EED)
References
Structured abstracts of articles reporting economic evaluations of health care interventions
Bibliographic details of articles on relevant topics (i.e., burden of illness, economic methodology, reviews of economic evaluations)
N=15,041 citations
Canadian Cochrane Network & Centre
Searching the CLIB
Some basics…
ALL contents of ALL records in ALL databases are searched
Ignores: punctuation & numbers
Boolean terms: AND, OR, NEXT, NEAR – within 6 words both ways, NOT
Restricting searches ‘Options’ page and choose desired restrictions At the end of the term, add:
:AU – Author :TI – Title :ME - MeSH terms
:AB – Abstract :KY - Keywords
Canadian Cochrane Network & Centre
Searching the CLIB
MeSH Keywords drawn from MeSH thesaurus of U.S. NLM Accompanies some but not all records Organized hierarchically in ‘trees’ Permuted Index – an index of all words that appear in the
MeSH thesaurus used to located specific MeSH terms
At McGill University and affiliated sites:
The pathway to the CLIB is:
McGill HSL library homepage > Databases > > Cochrane Library > web (no login or password nec.)
Canadian Cochrane Network & Centre
Useful Materials
The Cochrane Library: Self Training Guide Interpretation of Odd-ratio diagrams WWW links, HTA database Cochrane Handbook User manual Glossary, etc...
DOWNLOADhttp://www.cochrane.org/resources/training.htm
Canadian Cochrane Network & Centre
Quality of Cochrane Reviews
• Comparison of 36 Cochrane reviews with 39 paper-based journals Cochrane reviews less prone to bias (Jadad et al. 1998)– Explicit reporting of eligibility criteria
(35/36 vs 18/39)– Assessed trial quality (36/36 vs 7/39)– No language restriction (36/36 vs 32/39)
Quality of Cochrane Reviews
• Also not perfect (Olsen et al. 2001)
• 52 Cochrane reviews from 1998• 18% - conclusions not backed up by evidence
– All overestimated effect of intervention
Discordant Reviews
• Direction of effect• Significance• Magnitude of effect• Interpretation of results
Numerous possible reasons
Sources of discordance
• Different research question– Target population– Intervention being studied– Outcome measures– Setting
• Study selection– Search strategy– Eligibility criteria
Sources of discordance
• Data extraction– Methods of measuring outcomes and endpoints– Extent of human error
• Quality assessment method– 25 scales and 9 checklists avail for assessing RCT
quality (Moher et al. 1996)• Inconsistent quality depending on instrument used
– Additional 8 instruments (Juni et al. 1999)– QUOROM (Quality of Reporting of Meta-Analyses)
= most comprehensive(Moher et al. 1999, Shea et al. 2001)
Sources of discordance
• Analysis– Appropriateness of combining results– Method of synthesis: Descriptive, meta-analysis– Statistical methods
• Bayesian• Meta-regression• Frequentist
• Interpretation of evidence
Decision Algorithm for interpreting discordant reviews
Jadad, Cook & Browman. A guide to interpreting discordant systematic reviews. CMAJ. 1997 May 15;156(10):1411-6.
Screening for breast cancer with mammography among
women aged 50-69 years
Report prepared for the Breast Cancer Screening Unit,Cancer Branch, Health Canada
Ritsuko KakumaMarch 2002
SRs on mammography screening for breast cancer
• Fletcher et al. Report of the International Workshop on Screening for Breast Cancer. J Natl Cancer Inst 1993; 85:1644-1656.
• Kerlikowske et al. Efficacy of screening mammography. A meta-analysis. JAMA 1995; 273:149-154.
• Olsen O & Gøtzsche PC. Is screening for breast cancer with mammography justifiable? Lancet 2000.
Fletcher et al. review
Research Question: • To assess current state of knowledge about BRCA
screening, identify knowledge gaps
Study Sample:• Women aged 40-79 years stratified into 3 groups: 40-
49, 50-69, 70+
Fletcher et al. review
Methodological issues: • Int’l Workshop on screening for BRCA in Feb 1993 (by
NCI) – characteristics of participants?
• Search strategy not reported• Study selection method not described• Data source: published and unpublished data
provided by the w/s participants• Validity of trials assessed:
– Randomization, Compliance, Contamination– Quality and frequency of screening test– Adequacy of FU– Validity of outcome assessment– Generalizability of results
Fletcher et al. review
Results/Conclusions: • 7 trials identified• For age 50-69 age group, only 5 trials provided data
– New York, Two-County, Malmo, Edinburgh and Canada
• No statistical pooling of data
• Together, Fletcher et al. concluded that there was substantial benefit of screening on BRCA mortality reduction of 30% after 10-12 years of FU
Kerlikowske et al. review
Research Question: • To determine efficacy of screening in reducing BRCA
mortality by age, # MM views per screen, screening interval and duration of FU
Sample: • Women aged 40-74 years stratified into 3 groups: 40-
49, 40-74, 50-74
Kerlikowske et al. review
Methodology: • Search: MEDLINE, manual (reference lists) and
consultations with colleagues and experts– Search terms, dates covered
• Study Selection – English language restriction, explicit inclusion criteria– Not restricted to RCTs
• Data extraction– 2 indep extractors & 3rd for resolving discrepancies– Published and unpublished data used
• Quality assessment not reported..not done?• Analysis
– Pooled data of all included trials using fixed effects
Kerlikowske et al. review
Results / Conclusions: • 9 studies (7 RCTs & 2 CCS) have data on 50-74 year
age group
Comparison* Relative Risks
RCT vs. Case-control studies 0.77 (0.69-0.87) vs. 0.45 (0.29-0.70)
No. of MM views: 2-views vs. 1-view 0.70 (0.58-0.84) vs. 0.83 (0.71-0.97)
Screening interval: 12 mo. vs. 18-33 mo. 0.77 (0.59-1.0) vs. 0.77 (0.68-0.88)
FU duration: 7-9 yrs vs. 10-12 yrs 0.73 (0.63-0.84) vs. 0.76 (0.67-0.87)
Screening duration: 3-5 yrs vs. 8-10 yrs 0.76 (0.62-0.95) vs. 0.78 (0.67-0.90)
CBE + MM vs. MM alone 0.80 (0.66-0.98) vs. 0.76 (0.65-0.88)
Study start date: Pre-1980 vs. Post-1980 0.76 (0.67-0.87) vs. 0.83 (0.63-1.10)
Kerlikowske et al. review
Results / Conclusions: • All studies – beneficial effect of screening
(4 statistically significant)• All trials included in the analysis (not CCSs)
• MM screening reduced BRCA mortality by 27% in this age group after 7- 9 years and 24% after 10-12 years regardless of # MM views, screening interval, duration of screening or addition of CBE
Olsen & Gøtzsche review
Research Question: • To assess effect of MM screening for BRCA on
mortality and morbidity• Review methodological quality of trials – focus on 3
most important sources of bias – Randomization – blind outcome assessment– post-Rx exclusions
Sample: • Women without previously diagnosed BRCA on
women aged <50 years and 50+ years
Olsen & Gøtzsche reviewMethodology: • Search: MEDLINE, common author names search,
reference lists, specialized trials registers, letters, abstracts, grey literature– Search terms, dates covered
• Study Selection:– Explicit inclusion criteria (RCT / quasi-RCT on MM screening
vs. no MM screening)• Two indep reviewers for study selection, quality
assessment & data extraction– Authors of studies contacted for missing information
• Quality assessment: High, Medium, Poor or Flawed– Rx process, baseline comparability, post-Rx exclusions,
Consistency of number of women randomized• Analysis
– Intent-to-Treat analysis, with fixed effect unless heterogeneous– Stratified by quality (excluded flawed)
Olsen & Gøtzsche review
Results / Conclusions: • 7 trials identified, none with high quality (2 medium, 3
poor, 2 flawed)• Total mortality
– no impact of screening after 7- and 13 yrs FU for both medium and poor quality trials
• BRCA mortality – no beneficial effect of screening in medium quality trials– Beneficial effect among POOR quality trials
pooled RR=0.69, CI 0.55-0.80 at 7 yearspooled RR=0.64, CI 0.54-0.76 at 13 years
• Interventions for screened group higher– Over-diagnosis, increased use of more aggressive treatment
such as mastectomies and tumorectomies (20-30%)
Decision Algorithm for interpreting discordant reviews
Identified Trials
•Sampling Randomization: Canadian study recruited volunteers, other used population-based lists
•Assessment of pre-existing BRCA: pre- vs. post-Rx
Trials: Methodological issues
Canadian Efficacy trial, not effectiveness (women = screened, not ‘invited’ to be screened)
CBE in control group may dilute effect of screening (MM+CBE vs. CBE)
Imbalance in rate of advanced cancers at Rx
Malmö Inconsistent numbers across multiple reports – missing unaccounted for
Date of entry different for the 2 groups (date of Rx vs. date of invitation to screening)
Contamination: 24% of controls had MM
Continuation of study: not randomized, no baseline information
Stockholm Inconsistent numbers across multiple reports– missing unaccounted for
Non-independence of results of 2 sub-trials (2 smaller trials with overlapping controls)
Imbalanced post-Rx exclusions, Contamination (Screening for controls began as early as year 3)
Göteborg Information on Rx not described adequately
Contamination (Screening for controls began 3rd and 6th years)
No separate data on 50+ age group
Trials: Methodological issues
Two-County
Age imbalance of 5-months – older in screening group (clinically signif?)
Cluster Rx method not described adequately
Contamination (Screening for controls began after a few years into study)
New York Baseline imbalance in previous lump, menopause and education
Post-Rx exclusion ID method of current BRCA differed --self-report vs. screen
Outcome assessment (cause of death): 72% non-blind (differential misclassification or true effect?)
Edinburgh Inadequate Rx Baseline imbalances – screened group had higher SES
Change in allocation status after Rx
Imbalanced post-Rx exclusions
Contamination (Screening for controls began in year 10 of FU)
Differential eligibility criteria and entry dates for the 2 groups
Impact on effect measure
• Contamination & Compliance : underestimate– Introduction of screening in controls– Screened vs. invited to be screened
• Post-Rx exclusions: overestimate– Identification of BRCA at baseline more complete for
screened group..first round of screening used to exclude– Dropouts and losses: equally distributed?
• Adequacy of Randomization: overestimate– Typically overestimate treatment effect by 30-40%
(Schulz et al. 1995; Moher et al. 1998; Kjaergard et al. 2001)
• Outcome Assessment: overestimate– Misclassification– Determination of cause of death may be difficult in light of
multiple illnesses
Comparability of Trials
• Same research question?– Effectiveness vs. efficacy
• Same Target population?
• Appropriate study sample?
• Intervention?– Screening modality, frequency of screening
• Outcome assessment– Cause of death identified the same way?
• Analytical methods
Fletcher et al. No precise research question Inadequate description of methods used
– Search strategy, trial selection, data extraction Acknowledgement of difference in quality of trials but
conclude based on ALL evidence Source of some data unclear
Quality assessment carried out NOT synthesizing data = appropriate• But evidence does not support their conclusion that
BRCA mortality is reduced by 30% as a result of screening
Kerlikowske et al.
Clear research question Explicit reporting of methods used
No indication that quality assessment was done Some numbers extracted don’t match reference cited Missing key articles for trials (I.e., original report)
• Despite good reporting of methods, caution needed in interpreting results because of the questionable nature of data used
Olsen and Gotzsche
Clear research question Explicit reporting of methods used Strong methodology Addressed important question:
Reduced BRCA mortality not necessarily equal to improved survival
Did not assess baseline comparability stratified by age groups, which is how the data was analyzed
• Most rigorous and properly done review of the 3• Brought potential adverse effects of screening to
forefront
Conclusions of assessment:
• All existing evidence has methodological issues • No conclusive evidence exists to support MM• Other issues:
– Accuracy of MM (false positives?)– Detection of slow-growing tumors that may never develop
into invasive cancers– Psychological impact of positive results
• US and Canada continue to recommend MM screening for women in this age group
Overall conclusions
• MA not appropriate in many cases • Must be able to identify the potential
sources of bias in both the systematic review as well as the studies included in the review
• Cochrane reviews are not always perfect but generally better than non-Cochrane reviews