reporting systematic reviews and meta-analyses (prisma) and observational studies (strobe)
DESCRIPTION
Reporting systematic reviews and meta-analyses (PRISMA) and observational studies (STROBE) Doug Altman The EQUATOR Network Centre for Statistics in Medicine, Oxford, UK October 2012. Systematic review . - PowerPoint PPT PresentationTRANSCRIPT
Reporting systematic reviews and meta-analyses (PRISMA) and
observational studies (STROBE)
Doug AltmanThe EQUATOR Network
Centre for Statistics in Medicine, Oxford, UKOctober 2012
2
Systematic review
A systematic review is a scientific investigation that focuses on a specific question and uses explicit, prespecified scientific methods to identify, select, assess, and summarise the findings of similar but separate studies.– A study of studies
Objective is to summarize evidence from multiple studies using explicit methods
It may include a quantitative synthesis (meta-analysis), depending on the available data
3
Key characteristics of SR
Focused well defined research question Clearly stated title and objectives Comprehensive strategy for identification of all
relevant studies (published & unpublished) Explicit (and justified) predefined inclusion &
exclusion criteria Critical appraisal of studies Clear analysis of the results of eligible studies
– Quantitative (meta-analysis)– Qualitative
Structured report
4
The QUOROM Statement[Moher et al 1999]
Guidance on what information should be included in published reports of meta-analyses of randomized trials
Checklist of items which should be reported Also recommended a flow diagram showing
flow of studies through the review – to be included in the published report
Evidence-based, whenever possible QUOROM developed in 1996 and published in
1999 Moher et al. Improving the quality of reporting of meta-
analyses of randomised controlled trials: the QUOROM statement. Lancet 1999;354:1896-1900
5
Checklist of items
6
Also recommended a flow diagram showing flow of studies through the
review
7
PRISMA: Preferred Reporting Items for Systematic Reviews and Meta-
Analyses Update of QUOROM
Developed in 2005, Published in 2007
Consists of a 27 item checklist and a flow diagram
Reporting of systematic reviews and meta-analyses that evaluate healthcare interventions
Includes long explanatory document
Slide 8 of 43
9
# of records identified through database searching
# of additional records identified through other
sources
Total # of duplicates removed
# of records screened # of records excluded
Total # of articles assessed for eligibility
Total # of studies included in qualitative synthesis of
systematic review
# of articles excluded, with
reasons
# of studies included in quantitative synthesis of
systematic review
iden
tificatio
nscreen
ing
eligibility
includ
ed
10
Reporting vs conduct: study methods
METHODS – each aspect of the methods
Done well
Done poorly
Not done
Fully reported (=reproducible)
Ambiguously or incompletely reported
Not reported
11
PRISMA Practical
12
2 reviews in one article focus on progesterone
13
For each item …
Is there text relating to the item?
Does the text tell us what we need to know?
14
PRISMA: Item 6, eligibility criteria
METHODS Eligibility criteria Specify study characteristics used as criteria
for eligibility, giving rationale– PICOS
(participants, interventions, comparisons, outcomes and study design)
– Length of follow-up Specify report characteristics used as criteria
for eligibility, giving rationale– Years considered – Languages – Publication status
Can you locate any text about this issue in the report?
15
Wyatt et al
No explicit statement of study characteristics
“… clinical trials of progesterone and pro gestogens in the management of premenstrual syndrome.”
Dates given in search criteria
“All languages were included.”
PRISMA – item 8, search
“Present full electronic search strategy for at least one database, including any limits used, such that it could be repeated”
Can you locate any text about this issue in the report?
Slide 16 of 43
Wyatt et alPage 2, Methods, Trials
“MeSH terms used were premenstrual syndrome, progesterone, and progestogen, as well as the individual drug names, together with title and abstract searches for keywords progesterone, pro gestogen, premenstrual syndrome, premenstrual ten sion (PMT), late luteal phase dysphoric disorder (LLPDD), premenstrual dysphoria (PMD), and pre menstrual dysphoric disorder (PMDD).”
Slide 17 of 43
PRISMA: Reporting search strategy
We realize that journal restrictions vary and that having the search strategy in the text of the report is not always feasible – Expensive real estate
We strongly encourage all journals, however, to find ways, such as a ‘‘Web extra,’’ appendix, or electronic link to an archive, to make search strategies accessible to readers
We also advise all authors to archive their searches so that:– others may access and review them (e.g.,
replicate them or understand why their review of a similar topic did not identify the same reports)
– future updates of their review are facilitated
19
PRISMA: Item 9, study selection
METHODS Study selection State the process for selecting studies
– Screening – Eligibility – Included in systematic review and, if applicable,
included in meta-analysis
Can you locate any text about this issue in the report?
20
Wyatt et al
“We searched medical databases for reports of published clinical trials of progesterone and progestogens in the management of premenstrual syndrome.”
“References cited in all trials were searched iteratively to identify missing studies. All languages were included.”
“We included trials that investigated the effect of progesterone or progestogens on premenstrual symptoms if they were randomised, placebo controlled, double blind studies that included patients with a pretreatment diagnosis of premenstrual syndrome, for which all data from the trials could be acquired.”
21
PRISMA Item 11, Data items
METHODS Data items 11 List and define all variables for which data
were sought (e.g., PICOS, funding sources) and any assumptions and simplifications made. (PICOS = participants, interventions, comparisons, outcomes and study design)
Can you locate any text about this issue in the report?
22
Wyatt et al
“We collected data on the dosage and preparation of treatment. The main outcome measure was a reduction in overall symptoms of premenstrual syndrome. Combined or overall symptoms was chosen in an attempt to overcome the clinical heterogeneity associated with the measurement and scoring of symptoms used in individual trials.”
PRISMA – item12, Risk of bias in individual studies
“Describe methods used for assessing risk of bias of individual studies (including specification of whether this was done at the study or outcome level), and how this information is to be used in any data synthesis”– Can you locate any text about this issue in the report?
Slide 23 of 43
Wyatt et alPage 2, Methods, Quality assessment
“We assessed trial quality using a scale developed by Jadad et al,11 which assesses the randomisation, double blinding, reports of drop outs, and withdrawals for the trials
... our own quality scale, which assesses the quality of the trials for study design, reproducibility, and statistical analysis. This eight point scale comprised the following: confirmation that no other medications or oral contraceptives were being taken; a power calculation to justify patient numbers or more than 65 participants in each arm (enabling detection of a small effect size of 0.3, see below); a single, clearly stated dose of drug; reproducible measurement of premenstrual symptoms; clear presentation of results; a description of the number and reason for trial withdrawals; exclusion of, or a separate analysis of, participants with a major psychiatric disorder; and whether or not the trial was supported by independent funding.”
Slide 24 of 43
Reporting risk of bias
“Authors should report how they assessed risk of bias; whether it was in a blind manner; and if assessments were completed by more than one person, and if so, whether they were completed independently. Similarly, we encourage authors to report any calibration exercises among review team members that were done. Finally, authors need to report how their assessments of risk of bias are used subsequently in the data synthesis (see Item 16).”
Slide 25 of 43
Wyatt et alPage 2, Methods, Quality assessment
“We awarded one point for each category present in the trial.
Each trial was independently scored by two investigators and the third investigator arbitrated on any disagreements.
We used predetermined criteria for the recognition of the highest quality trials. A score of 3 or more was required in the Jadad score for the trial to be designated “high quality” and included in the meta analysis11; a score of less than 3 meant that the trial was designated “low quality.”
Slide 26 of 43
27
PRISMA Item 17
RESULTS Trial flow17 Give numbers of studies screened, assessed
for eligibility, and included in the review, with reasons for exclusions at each stage, ideally with a flow diagram.
Can you locate any text about this issue in the report?
Page 2, Results
“We identified 14 published trials that assessed the efficacy of progesterone in the management of premenstrual syndrome. We excluded four: two because of their low quality score on the Jadad scale, one because the data could not be extracted, and one because the trial failed to make a prospective diagnosis of premenstrual syndrome before randomisation. Ten trials remained ........”
No flow diagram
Slide 28 of 43
Caughey AB, Sundaram V, Kaimal AJ, Gienger A, Cheng YW, McDonald KM, Shaffer BL, Owens DK, Bravata DM. Systematic Review: Elective Induction of Labor Versus Expectant Management of Pregnancy. Annals of Internal Medicine 2009;151: 252-263
Slide 29 of 43
30
PRISMA Item 18
RESULTSStudy characteristics18 For each study, present characteristics for
which data were extracted (e.g., study size, PICOS, follow-up period) and provide the citations.
Can you locate any text about this issue in the report?
31
Wyatt et al
32
Wyatt et al Problems in Table 1
Crossover trials – not mentioned in text Comparator not stated Variation in outcome measures – how
combined or chose one? Some statements disagree with Figure No details of elements of “quality” scores Unclear how handled trial with 3 arms
PRISMA – item 15, Risk of bias across studies
Specify any assessment of risk of bias that may affect the cumulative evidence (e.g., publication bias, selective reporting within studies).
Can you locate any text about this issue in the report?
Slide 33 of 43
Page 2, methods, statistical analysis
“We used the method of Egger et al to detect bias (such as publication and location bias) in the included trials with a funnel plot. We assessed the asymmetry of the funnel plot quantitatively by plotting a linear regression of the standard normal deviate (standardised mean difference divided by SE) against precision (inverse of SE). A regression line that passes through the origin of the plot (within error limits) indicates symmetry and hence the absence of bias.”
Slide 34 of 43
35
PRISMA Item 20
RESULTS Results of individual studies20 For all outcomes considered (benefits or
harms) present, for each study: (a) simple summary data for each intervention group, (b) effect estimates and confidence intervals, ideally with a forest plot.
36
37
Altman & Cates complained about absence of any numerical results (BMJ
rapid response) “There are several aspects of this review that
readers cannot assess without summary data from each study.
For example, we would wish – to assess the strange heterogeneity P values for Figures 1
and 2 (the quoted P values of 0.999 are implausible given the clear graphical heterogeneity);
– to gain some insight into how the cross-over trials were included in the analysis (about which the authors say nothing at all) and whether the crossover and parallel group trials differed in their findings;
– to seek an explanation for the apparent discrepancy for three trials (references 19, 31 and 32) between the ‘reported results’ in Table 1 and the results shown in Figures 1 and 2;
– to assess the claim that random and fixed models give the same answer in the face of graphical heterogeneity;”
– … and 5 more points
38
Altman & Cates complained about the absence of any
numerical results “In addition, we note that the authors make no
comment about the varied nature of the outcome measures in these trials, nor do they say which outcome was used for those trials that presented more than one. It is hard to believe that all of the scales can be considered equally valid assessments of symptoms.
Also, we wonder if they can clarify the meaning in the figure legends of ‘standardised mean difference … for proportion of patients who showed improvement …’. We are puzzled by this terminology as the SMD gives no direct information about proportions of patients improving.”
39
Authors’ reply
“We have found this personal attack unpleasant and upsetting and have to question the use of unsupported attacks in the Rapid Response forum.”
“We have considerable experience in PMS (clinically as well as though our research) and believe ourselves competent to judge the clinical appropriateness of combining trial data.”
The only question they addressed was one we had not asked!
A 2nd request for the data to be provided went unanswered!
40
41 41
Observational studies
Transparent reporting is particularly important for observational studies– Vulnerable to bias and confounding– Findings are often over-interpreted– Findings often generate health scares
42
43
Scope of STROBE
Epidemiological research comprises several study designs and multiple topic areas
Initial restriction to three major areas– cohort, case-control, and cross-sectional studies
Later extensions to other study designs – STREGA for genetic association studies (published
2009)– STROBE-ME for molecular epidemiology – etc
44 44
Final STROBE checklist
TITLE and ABSTRACT INTRODUCTION Background/rationale Objectives METHODS Study design Setting Participants Variables Data sources/measurement Bias Study size Quantitative variables Statistical methods
RESULTS Participants Descriptive data Outcome data Main results Other analyses DISCUSSION Key results Limitations Interpretation Generalisability OTHER INFORMATION Funding
22 (34) items
45 45
Design-specific items
Participants Statistical methods Descriptive data Outcome data
46
STROBE Statement
Guidance on how to report observational studies well– Focus on 3 main study designs: cohort, case-control, cross-
sectional studies Published in Oct 2007: short paper and E&E Adopted by many journals
Find it on:www.equator-network.orgwww.strobe-statement.org
Case-control studies
Patients with a certain outcome or disease and an appropriate group of controls without the outcome or disease are selected – (usually with careful consideration of choice of controls,
possibly with matching)
Information is obtained on whether the participants have been exposed to the factor under investigation
47
48
STROBE exercise
49
50
51
STROBE Item 5. Setting
Describe the setting, locations, and relevant dates, including periods of recruitment, exposure, follow-up, and data collection
Can you locate any text about this issue in the report?
52
STROBE Item 5. Setting
Qiu et al
“This case-control study was conducted at the Materno Perinatal Institute of Lima and the Dos de Mayo Hospital in Lima, Peru, from May 2004 through October 2005. Both institutions are operated by the Peruvian government and are primarily responsible for providing maternity services to low income women residing in Lima.”
53
STROBE Item 6a. Participants
Give the eligibility criteria, and the sources and methods of case ascertainment and control selection. Give the rationale for the choice of cases and controls
Can you locate any text about this issue in the report?
54
STROBE Item 6a. Participants
Qiu et al“Cases were selected from those women with a
diagnosis of preeclampsia. Potential preeclampsia cases were identified by daily monitoring of all new admissions to antepartum wards, emergency room wards, and labor and delivery wards of the study hospitals. Study subjects were recruited during their hospital stay. Study personnel made periodic visits to each ward in a fixed order for the purposes of identifying potential cases and controls for the present study. Preeclampsia was defined by …”
“Controls were women with pregnancies uncomplicated by pregnancy-induced hypertension or proteinuria. Each day during the enrollment period, controls were numbered in the order in which they were admitted and identified. Subsequently, they were approached in the order in which research personnel identified them.”
55
STROBE Item 10. Study size
Explain how the study size was arrived at
Can you locate any text about this issue in the report?
56
STROBE Item 10. Study size
Qiu et al
?
57
STROBE Item 13. Participants
(a) Report numbers of individuals at each stage of study—e.g. numbers potentially eligible, examined for eligibility, confirmed eligible, included in the study, completing follow-up, and analysed
(b) Give reasons for non-participation at each stage
(c) Consider use of a flow diagram
58 58
What we’d like to see:
What do they give
us?
59
Qiu et al
60
STROBE Item 19. Limitations
Discuss limitations of the study, taking into account sources of potential bias or imprecision.
Discuss both direction and magnitude of any potential bias
Can you locate any text about this issue in the report?
61
Qiu et al
“First, our analyses are based on cross-sectionally collected data, which may be subject to recall bias. There has been one longitudinal study of Finnish women [6]; however, more longitudinal studies are needed to re-examine the potential causal relation between maternal experience of depression and preeclampsia risk in different populations.Second, we used a depression screening instrument to categorize participants according to depression/depressive symptoms. Participants did not have formal diagnostic examinations. As a result, some misclassification is possible. ... In addition, our assessment of maternal depression and depressive symptoms was limited to the duration of the pregnancy. Last, although we adjusted for multiple confounding factors, as with all observational studies, we cannot exclude the possibility of some residual confounding.”
62
63
Closing Comments on Checklists
They help AUTHORS ensure that they have addressed important issues in the report of their study
They help PEER REVIEWERS and EDITORS by reminding them what issues should be addressed
“Necessary but not sufficient!”
64
Closing Comments on Checklists
They help AUTHORS ensure that they have addressed important issues in the report of their study
They help PEER REVIEWERS and EDITORS by reminding them what issues should be addressed
“Necessary but not sufficient!”