confidential: for review only...2020/02/09 · confidential: for review only agreement of treatment...

Confidential: For Review OnlyAgreement of treatment effects from randomized trials

using routinely collected data for outcome assessment and traditional trials: a meta-research study

Journal: BMJ

Manuscript ID BMJ-2020-056792.R1

Article Type: Research

BMJ Journal: BMJ

Date Submitted by the Author: 02-Sep-2020

Complete List of Authors: Mc Cord, Kimberly; University Hospital Basel, Basel Institute for Clinical Epidemiology and BiostatisticsEwald, Hannah; Universitatsspital Basel, Basel Institute for Clinical Epidemiology and BiostatisticsAgarwal, Arnav; University of Toronto, Department of MedicineGlinz, Dominik; Universitatsspital Basel, Basel Institute for Clinical Epidemiology and BiostatisticsAghlmandi, Soheila; Basel Institute for Clinical Epidemiology and BiostatisticsIoannidis, John; Stanford University, Stanford Prevention Research Center, Department of Medicine and Department of Health Research and PolicyHemkens, Lars; University Hospital Basel, Basel Institute for Clinical Epidemiology and Biostatistics

Keywords: routinely collected data, randomized trials, registries, electronic health records, meta-research

https://mc.manuscriptcentral.com/bmj

BMJ

Confidential: For Review OnlyAgreement of treatment effects from randomized trials using routinely

collected data for outcome assessment and traditional trials: a meta-

research study

Kimberly A. Mc Cord, researcher1, Hannah Ewald, researcher1,2, Arnav Agarwal,

researcher3, Dominik Glinz, researcher1, Soheila Aghlmandi, researcher1, John P.A.

Ioannidis, professor4-9, Lars G. Hemkens, senior scientist1,5,9

1 Basel Institute for Clinical Epidemiology and Biostatistics, Department of Clinical Research, University Hospital

Basel, University of Basel, 4031 Basel, Switzerland2 University Medical Library, University of Basel, 4051 Basel, Switzerland3 Department of Medicine, University of Toronto, Toronto, Ontario, Canada4 Stanford Prevention Research Center, Department of Medicine, Stanford University School of Medicine, Stanford, CA

94305, USA5 Meta-Research Innovation Center at Stanford (METRICS), Stanford University, Palo Alto, CA 94305, USA6 Department of Health Research and Policy, Stanford University School of Medicine, Stanford, CA 94305, USA7 Department of Biomedical Data Science, Stanford University School of Medicine, Stanford, CA 94305, USA8 Department of Statistics, Stanford University School of Humanities and Sciences, Stanford, CA 94305, USA9 Meta-Research Innovation Center Berlin (METRIC-B), Berlin Institute of Health, Berlin, Germany

1 September 2020

Correspondence to:

Lars G. Hemkens MD, MPH Basel Institute for Clinical Epidemiology and Biostatistics Department of Clinical Research; University Hospital BaselSpitalstrasse 12; CH-4031 Basel, Switzerland Phone: +41 61 265 3100 Email: [email protected] 0000-0002-3444-1432 Word count: Main text: 3699; Abstract: 339

Page 1 of 49


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

Confidential: For Review OnlyABSTRACT

Importance: Routinely collected data (RCD) are increasingly used in randomized

clinical trials (RCTs) to provide real-world evidence. It is not known whether using this

data for outcome measurement leads to different treatment effect estimates.

Objective: To assess how effect estimates from RCTs using RCD for outcome

ascertainment agree with those from traditional RCTs not using RCD.

Design, Setting, and Participants: This analysis is based on systematically identified

RCTs using any type of RCD, including registries, electronic health records (EHRs) and

administrative databases for outcome ascertainment that were included in a meta-

analysis of a Cochrane review on any clinical question and any health outcome together

with traditional trials not using RCD for outcomes measurement. The effect estimates

from trials using or not using RCD were summarized in random effects meta-analyses.

Two investigators independently assessed the quality of each data source.

Main Outcome(s) and Measure(s): The main outcome was the agreement of

(summary) treatment effect estimates from trials using RCD and those not using RCD,

expressed as ratio of odds ratios (ROR). Subgroup analyses explored effects in trials

based on different types of RCD.

Results: We included 84 RCD-RCTs and 463 traditional RCTs on 22 clinical questions.

RCTs using RCD for outcome ascertainment showed 20% less favorable treatment effect

estimates than traditional trials (ROR 0.80, CI 0.70 - 0.91, I2=14%). Results were similar

across various types of outcomes (mortality outcomes: ROR 0.92, 95% CI 0.74 - 1.15, I2

=12%; non-mortality outcomes: ROR 0.71, 95% CI 0.60 – 0.84, I2=8%), data sources

(EHRs: ROR 0.81, 95% CI 0.59 - 1.11, I2 =28%; registries: ROR 0.86, 95% CI 0.75 – 0.99,

I2=20%; administrative data: ROR 0.84, 95% CI 0.72 – 0.99, I2=0%) and data quality

(high data quality: ROR 0.82, 95% CI 0.72 – 0.93, I2=0%).

Conclusions and Relevance: RCTs using RCD for outcome ascertainment show smaller

treatment benefits than traditional trials. These differences may have implications for

health care decision making and the application of real-world evidence.

Page 2 of 49


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

Confidential: For Review OnlyINTRODUCTION

Health data that are not collected for the purpose of research are increasingly used for

clinical trials1,2. Such routinely collected data (RCD) from registries, electronic health

records (EHRs), administrative claims or even mobile devices may be used to identify

trial participants and to assess treatment outcomes2. Readily available data are typically

more affordable than actively collected research data3. Cost reduction may make larger

and longer trials more feasible. Data collection during usual care also avoids artificial

research settings, and this may increase pragmatism and applicability of trial results to

routine care4. RCD databases include many outcomes that are relevant in practice and

matter to clinicians and patients (e.g. mortality, disability or hospitalization), while they

typically lack outcomes that are more relevant for explanatory trials aiming to

understand the biological processes underpinning treatment effects (e.g. biomarkers)5.

Cutting out research-driven follow-up visits and relying only on patient interaction

during usual care probably better reflects “real-world” treatment effects, and patient

adherence may be less faithful in such a setting as compared to traditional, more

explanatory trials. Overall, trials embedded in existing data collection structures may

provide real world evidence, being more informative for guiding treatment decisions

and sharing more features of pragmatic trials than many traditional trials6–8.

A key issue of using RCD for clinical research is data quality1,2. For some outcomes, data

quality in RCD may be lower, in particular due to non-uniform data collection and

potential measurement errors9–13. On the other hand, healthcare professionals

collecting RCD during usual care may have more clinical expertise than research staff

who often collects trial data only for a narrow time frame and scope, sometimes only for

very few participants or even a single patient per center14. Since RCD is collected

independently of the trial from people unaware of treatment allocations, biases related

to outcome ascertainment might be even less likely than in traditional trials. Moreover,

data quality in RCD can vary enormously for different outcomes. For mortality, the

quality might be very high15: with proper linkage to death registries, complete, accurate

information can be achieved, while other trials not linked to RCD sources may lack

information on survival status for many participants. Conversely, data quality in RCD

might be highly insufficient for other outcomes, such as specific adverse events or some

patient-reported endpoints. The impact of using RCD for outcome ascertainment and

Page 3 of 49


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

Confidential: For Review Onlythe impact of potential inaccuracies on trial results is unclear. Misclassification of

clinical events or missing information that occurs randomly, for example due to coding

errors or problems with database linkage16, may diminish the treatment effect point

estimates17. Larger sample sizes achieved by using RCD may increase precision of

treatment effect estimates18 but these may still be biased underestimations.

Here, we aimed to provide empirical insights on the agreement of findings from trials

using RCD for measuring outcomes as compared to traditional RCTs.

METHODS

A protocol was not published for this study. We systematically obtained a large sample

of RCTs that used RCD to measure study outcomes (RCD-RCTs), identified RCTs that

explored the same clinical question but used traditional ways (not based on RCD) to

measure the outcomes, and then we compared their treatment effect estimates. We

assumed that studies included in the same meta-analysis in a Cochrane review would be

on the same clinical question. Cochrane reviews were a main information source for this

study.

Eligibility criteria

RCD-RCTs were eligible that (1) used RCD for measurement of any binary clinical

outcome and (2) are included in a Cochrane review meta-analysis together with at least

one other trial not using RCD for measuring the same outcome. Eligible RCD-RCTs

where either directly identified by searching PubMed and subsequent citation analysis

to determine if they are included in a Cochrane review (“index RCD-RCDs”) or indirectly

by perusing the other trials that were included with the index RCD-RCTs in the same

Cochrane review meta-analysis.

The other RCTs (i.e. not RCD-RCTs) that were included with an eligible RCD-RCT in the

same Cochrane review meta-analysis were eligble comparators.

We considered any health intervention in any population. We did not consider

outcomes that were uniquely cost related, but we kept outcomes measuring uptake of

interventions, such as vaccinations, drug treatments or screening. We defined RCD as

any health information not collected primarily for a specific research question19. We

excluded trials described by Cochrane reviewers as quasi-randomized or “randomized

controlled before and after design”. We considered trials reported as cluster-

Page 4 of 49


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

Confidential: For Review Onlyrandomized trials and cross-over trials (data from first period only) but excluded them

in a sensitivity anlysis.

Search

To identify the index RCD-RCTs, we searched PubMed using text words and medical

subject headings focusing on terms around routine data (Appendix 1). We searched for

RCTs published in English between 2000 and 2015 because of the emerging availability

of EHRs and other sources of RCD in the last two decades and because more recent

trials were less likely already included in Cochrane reviews. Two reviewers

independently screened titles and abstracts (KAM and AL or HE). Any article that was

found potentially eligible by one reviewer was considered for further analysis. One

reviewer (KAM) then identified Cochrane reviews citing any of these potentially eligible

RCD-RCTs using the “cited in systematic reviews” function on PubMed. We also queried

ISI Web of Science and perused the citing articles (from Web of Science Core Collection).

The last searches for RCD-RCTs in literature databases and citing Cochrane reviews

were in April 2016 and September 2017 (for details see appendix 1). We used the most

recent updated version (last search January 2020) of each Cochrane review for all

pertinent clinical questions and updated our searches, classifications and extractions

using these most recent versions.

Study selection

We obtained all full-texts of cited RCTs and citing reviews. One reviewer (KAM)

determined if the RCT was an index RCD-RCT (i.e. measured at least one pertinent

outcome with RCD and was included in a meta-analysis evaluating this outcome

together with other RCTs). This was verified by a second reviewer (LGH).

We obtained the full-texts for all other RCTs in the meta-analysis and one reviewer

(KAM or DG) determined if they were eligble RCD-RCTs or they were categorized as

traditional RCTs. Whenever there was any uncertainty in these steps, a second reviewer

was consulted (LGH) and the decision was made based on consensus. Eligibility of all

RCD-RCTs was confirmed by a second reviewer (LGH, AA, or KAM). Any uncertainties

were resolved by discussion.

Data collection process

Page 5 of 49


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

Confidential: For Review OnlyFrom each Cochrane review, we selected only one clinical question addressed by one

meta-analysis including the index RCD-RCT. We selected the meta-analysis with the

largest number of RCTs (if there were still multiple ones, we selected the one that had

the greatest total sample size). Some meta-analyses were reported with summary

estimates for subsets of studies but without an overall summary effect. In such cases, we

took the subset including the highest number of RCD-RCTs. In some cases, the same

RCD-RCTs were included in multiple subsets (for example, for different lengths of

follow-up) but there was an overall summary presented. Here we also used only the

largest subset to avoid double counting of participants or events. We preferred any

primary analysis over sensitivity or subgroup analyses if the former was present.

Sensitivity analyses on methodological features (for example by publication year) were

always excluded. These steps were conducted by one reviewer (KAM) and verified by a

second (LGH). We applied a different selection approach as secondary analysis

whenever the meta-analysis selected for the main analysis was not on mortality (which

was the case for 14 reviews) but there was a relevant mortality analysis included in the

Cochrane review (which was the case in 4 of the 14 reviews), we selected this one

instead. We applied the same approach for primary outcomes, but in the three cases

where the selected outcome was not a primary outcome of the Cochrane review, there

was no eligible alternative.

For each included trial, one reviewer (KM, LGH, AA, HE, or DG) extracted from the

Cochrane review the treatment effects (i.e., number of events and no events per study

arm), trial characteristics (parallel-group design, cross-over design, cluster-design,

country, year of publication), the median age of the study population (when not

reported, we used other available pertinent information (e.g. means) for approximation

where possible), and the Cochrane reviewer’s risk of bias assessment. Treatment effect

extractions were verified by a second reviewer (KM or LGH).

For each eligible RCD-RCT, one reviewer (KAM, DG, or LGH) extracted general

characteristics and the types of RCD utilized. We also noted whether the RCD source

was the only form of outcome data source, or if a hybrid approach was reported (i.e.

where the RCD were complemented by additional active data collection). Trials using

RCD within a hybrid approach were considered as RCD-RCTs but were excluded in a

sensitivity analysis.

Page 6 of 49


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

Confidential: For Review OnlyOne reviewer (KAM or DG) extracted any statement on data quality of the RCD in the

broader sense (e.g. statements related to measurement errors, reliability, accuracy or

completeness) and a second reviewer (KAM or AA) verified the extractions. As working

definition, we deemed “data quality” to be high when the RCD would be adequate to

reliably measure the outcomes of interest for this clinical question20. This was assessed

independently by two reviewers. We fully acknowledge that such an assessment from

the outside is difficult20. When authors provided a statement that led us assume that the

RCD would adequately measure the outcome of interest, a “high quality” mark was

given. If this was not reported, but the source was specifically designed to collect the

endpoint (e.g. breast cancer cases through a comprehensive national breast cancer

registry), a “high quality” mark was still given. If a statement indicating “low quality”

was provided (which we expected to be rare, but such statements could have been made

in the limitation section of the studies) or the reviewer felt that the RCD source was

unlikely to specifically collect such outcome data with little missingness and little

measurement error (e.g. adverse events extracted from administrative databases), a low

mark was given. Other cases were rated “unclear”. We quantified the agreement

between the two reviewers (KAM vs AA or DG) using kappa statistics and the total

agreement.

For sensitivity analyses, we extracted the risk of bias reported for each bias domain of

all individual trials. We categorized the trials as “≥ 1 domain high risk” (if any bias

domain was deemed by the Cochrane reviewers “high risk”), “all domains low risk” (if

all domains were deemed “low risk”), and “all domains low or unclear risk” (in all other

cases). We also specifically extracted the risk of bias due to the blinding status (or

participant blinding when several blinding domains were presented).

Summary measures and synthesis of results

We used a two-staged process to synthesize the results. First, we calculated two

summary odds ratios (sOR) for each clinical question using random-effects meta-

analyses (Hartung-Knapp-Sidik-Jonkman (HKSJ) method21): the sOR of the RCD-RCTs,

and separately the sOR of all the traditional RCTs. In cases when there was only one

trial, the “summary” OR was actually the OR of the trial. Subsequently, for each sOR pair,

we calculated their respective ratio, i.e. ratio of odds ratio (ROR; sOR of the traditional

trials/sOR of RCD-RCTs). The variance of the ROR was calculated as sum of the

Page 7 of 49


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

Confidential: For Review Onlyvariances of the sOR (after log transformation). We ensured that for all clinical

questions ORs <1 indicate favorable effects for the evaluated treatment. We inverted

effects where necessary (i.e. if a meta-analysis reported survival, we inverted the effect

estimate by taking its reciprocal so that estimates <1 indicate mortality benefits). For

consistency, we ensured that the second comparator was the control (i.e. no

intervention or usual care, in three cases where two active interventions were

compared22–24, we defined the control as the older treatment; we left these cases out in

a sensitivity analysis). A ROR <1 indicated that the RCD-RCTs estimated a less favorable

treatment effect for the evaluated treatment than the traditional RCTs.

Second, we combined all RORs across all clinical questions in a meta-analysis (random

effects, HKSJ) to provide an overall summary of the relationship of treatment effects

obtained from trials using RCD vs trials not using RCD.

Additional analyses

We conducted several sensitivity analyses: including only RCD-RCTs with low risk of

bias in all domains; including only RCD-RCTs with low risk of bias related to blinding;

excluding RCD-RCTs with some active data collection (hybrid approaches); excluding

older RCD-RCTs (published before 2005); including only more recent RCD-RCTs

(published in 2010 or later); stratified by number of participants and number of events

(tertiles across all RCD trials); including only RCD-RCTs where the median age of the

RCD-RCT population was within 1 standard deviation of the median age of the other

trials; including only clinical questions on mortality outcomes or non-mortality

outcomes (subsets of main analysis); excluding clinical questions with active controls;

using only clinical questions with effect estimates from RCD-RCTs and traditional-RCTs

that had no largely different precision (i.e. ratio of sOR standard errors >0.33 and <3);

excluding clinical questions with less than 3 RCD-RCTs; excluding clinical questions

with more than 10 RCD-RCTs; comparing the index RCD-RCTs with all other trials in the

meta-analysis (including traditional RCTs and misclassifying the indirectly identified

RCD-RCTs) to evaluate the robustness of the classification and sampling procedure;

DerSimonian-Laird random effects meta-analyses; and using only fixed-effect meta-

analyses.

Page 8 of 49


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

Confidential: For Review OnlyFor exploration, we conducted subgroup analyses including only RCD-RCTs using

registries, EHR, or administrative data, and where RCD-RCTs data quality was assumed

to be high.

We report medians with interquartile ranges if not stated otherwise. We used the meta

package (version 4.11-0) for meta-analyses25 (RStudio version 1.2.5033 26; R version

3.6.227).

Patient and public involvement (PPI) statement

This was not a clinical trial and did not involve contact with participants or sensitive

data.

RESULTS

Characteristics of trials using RCD and traditional trials

We screened 4649 publications and identified 29 index RCD-RCTs (Appendix 1;

Appendix 2; Appendix 5a) included in 22 Cochrane reviews. Among the corresponding

trials in the selected Cochrane review analyses, 55 other RCD-RCTs were identified

(Appendix 5b) and 463 were eligible traditional RCTs (Appendix 6).

The 84 RCD-RCTs had a median of 721 participants per trial (IQR 275 - 2729), the

majority (56 of 84, 67%) originating from North America, followed by Scandinavia (14

of 84, 17%) (Table 1). They were published between 1976 and 2017 (median 2005, IQR

1998 - 2009). The RCD sources were registries (36 of 84, 43%), EHRs (30 of 84, 36%)

and administrative databases (18 of 84, 21%). In 27 RCD-RCTs, a hybrid approach with

elements of active data collection was applied (32%).

We deemed the quality of the data adequate for 56 of the 84 RCD-RCTs (67%; moderate

interrater agreement [77.4%; kappa 0.50; weighted kappa 0.48]).

The 463 traditional RCTs (median 121 participants per trial; IQR 60 - 359) came

primarily from North America (125 of 463, 27%) and Continental Europe (60 of 463,

13%) and were published between 1963 and 2016 (median 2003, IQR 1997 - 2006)

(Table 1; Appendix 4).

The clinical questions were related to screening and preventative medicine (8 of 22,

36%), community medicine (5 of 22, 23%), cardiology (5 of 22, 23%) and surgery (4 of

22, 18%). In 11 comparisons there was 1 RCD-RCT only, 4 comparisons had 2 RCD-

Page 9 of 49


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

Confidential: For Review OnlyRCTs, 3 comparisons had 3 RCD-RCTs, and 4 comparison had 4 or more (Table 2).

Outcomes were diverse, with a large proportion related to mortality (9 of 22 in the main

analysis; 41%). The outcomes were a primary outcome of the Cochrane review in 19 of

22 cases (86%).

Agreement of treatment effects

Treatment effect estimates from RCD-RCTs and from traditional RCTs were in the same

direction in 19 of 22 cases (86%). In 14 of 22 cases (63%), the summary point estimate

of the RCD-RCT was less favorable (Figure 1).

Overall, RCTs using RCD for outcome ascertainment systematically showed less

favorable estimates of treatment effects than those from traditional RCTs not using RCD

(ROR 0.80, 95% CI 0.70 - 0.91, I2 14%) (Figure 2; Table 3; Appendix 3). In 4 of the 22

clinical questions (“individualized discharge plans on readmissions; “intrauterine

device for heavy menstrual bleeding”, “breastfeeding support for healthy women”, and

“immunization reminders and recalls”), the 95% CIs of the RORs excluded the null and

in all 4 clinical questions, RCTs with RCD had less favorable results than traditional

RCTs (Figure 2).

The results were similar when including only any available primary outcomes of

Cochrane reviews (ROR 0.79, 95% CI 0.70 – 0.90, I2 9%) or mortality outcomes (ROR

0.92, 95% CI 0.74 - 1.15, I2 12%), or studies with RCD where we deemed the data

quality high (ROR 0.82, 95% CI 0.72 – 0.93, I2 0%). The results were also similar when

analyzing EHRs (ROR 0.81 95%, CI 0.59 - 1.11, I2 28%), registries (ROR 0.86, 95% CI

0.75 – 0.99, I2 20%) or administrative data sources (ROR 0.84, 95% CI 0.72 – 0.99, I2

0%) (Table 3). All other sensitivity analyses corroborated the main findings (Table 3).

DISCUSSION

Principal findings

RCTs using RCD for outcome ascertainment showed less favorable treatment effects

than traditional trials not using RCD in this systematic analysis of various clinical topics

and outcomes. This might be due to data quality issues and measurement errors leading

to dilution of effects by misclassified outcomes. However, the results remained similar

across sensitivity analyses addressing this possibility, including data source type and

estimated data quality, or when including only mortality outcomes where

Page 10 of 49


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

Confidential: For Review Onlymisclassification is probably less likely. Thus, trials using RCD for outcome collection

may have other features that are associated with less pronounced effect estimates2. For

example, such trials might be more pragmatic than traditional trials2,5,18,28. More natural

care settings with less eagerness to artificially increase treatment adherence may result

in smaller treatment effect estimates.

This interpretation agrees with empirical research indicating that procedures to

standardize and increase data quality may have smaller impact on trial effect estimates

than often assumed: a review29 indicated that central outcome adjudication committees

used to increase data quality typically did not influence effect estimates compared to

onsite assessments in the very same trial. Direct comparisons of treatment estimates

based on separate ways of outcome ascertainments30 are helpful to better understand

the underlying mechanisms of outcome measurements. Of note, in contrast to such

research, we did not aim to isolate the “clean” effect of using RCD versus not using RCD

within the same trial as alternative data ascertainment methods. Conversely, we aimed

to empirically describe how results from trials designed to provide randomized real-

world evidence31 (by using real-world data) agree with those from traditionally

designed trials relying on their own, active data collection procedures.

Comparison with other studies

We are aware of only one other similar study that compared effects from 30 registry-

based trials with that from traditional trials on 12 different topics in cardiology or

cancer screening32. The reported RORs were 0.97 (95% CI 0.92 - 1.03) for mortality and

0.95 (95% CI 0.89 - 1.02) for other outcomes (reported ROR inverted to facilitate

comparison). However, some RCD-RCTs were double-counted. A sensitivity analysis

using only unique RCD-RCTs (ROR 0.95, 95% CI 0.85 - 1.05 and ROR 0.95, 95% CI 0.89 -

1.02, respectively) provided results compatible with our findings for registry-based

trials.

Limitations

Several limitations need to be considered. First, while the outcome selected for our

analysis was routinely collected in the RCD-RCTs, other outcomes within some of these

RCD-RCTs were still determined traditionally, thus introducing artificial settings that

Page 11 of 49


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

Confidential: For Review Onlydeviate from routine care. Therefore, some of the RCD-RCTs may reflect the “real-

world” more and others less.

Second, we did not directly evaluate the impact of trial pragmatism on treatment effects.

The applicability of research findings to “real world” settings may be determined by

other factors, such as the representativeness of the trial population or the treatment

setting, which we have not assessed. A deeper investigation of all RCD-RCTs and their

comparators would be far beyond the scope of this project and a valid retrospective

assessment of each trial’s pragmatism and representativeness is difficult for researchers

outside of the original trial team, requiring further information such as study

protocols33,34 or details on the study population and target population that are typically

unavailable.

Third, while we individually assessed and graded data quality and expected accuracies

in duplicate, assessing the quality of RCD source is inherently subjective and very

limited due to widely insufficient reporting of critical details (such as results of data

validation studies). We are not aware of an established instrument that would allow to

unambiguously determine the “data quality” on an outcome level using trial reports.

Thus, interpretations in this regard need to be very cautious.

Fourth, while our topics were evaluated in Cochrane reviews and very likely explore

questions of interest for healthcare decision makers, they do not cover the full spectrum

of clinical research. The statistical heterogeneity across topics was small, and issues

related to data quality and trial design vary across clinical fields. It remains uncertain

how the results can be extrapolated to specific medical fields and more evidence is

needed to better assess the generalizability of our findings. However, our assessment

covers areas of clinical research where using RCD for outcome assessment is a realistic

alternative, indicated by existence of trials using RCD-based and traditional outcome

measurement.

Finally, some of our analyses rely on sometimes insufficiently reported details35. While

we systematically ensured that the trials were actually measuring the analyzed

outcomes through RCD, poor reporting of RCD use in the traditional RCTs could have led

to some misclassification or we might have overlooked some hybrid approaches. We

have no reason to believe that possible misclassifications are associated with the

Page 12 of 49


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

Confidential: For Review Onlyinvestigated agreement, hence such errors would have led to a dilution of the difference

between the compared study designs and not change our overall conclusion.

Conclusions

RCTs utilizing any form of RCD for their outcomes’ ascertainment found systematically

less favorable treatment effects than RCTs utilizing traditional methods. There may be

differences between traditional trials and trial designs utilizing RCD beyond data quality

issues that would explain this. We need a better understanding of these factors, to

optimize the use of such emerging designs for comparative effectiveness research and

to increase the applicability of real-world evidence derived from randomized trials.

Page 13 of 49


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

Confidential: For Review OnlyACKNOWLEDGMENTS

We thank Aviv Ladanie, PhD for his contribution to the literature screening and data

extraction and Julie Jacobson Vann, PhD for providing details on included trials.

SUMMARY BOX

What is already known on this topic

Routinely collected data are increasingly used in randomized clinical trials to

measure outcomes

Data collection during usual care can reduce costs and avoid artificial research

settings, which may increase pragmatism and applicability of trial results

What this study adds

Our study suggests that randomized clinical trials using routinely collected data to

assess their outcomes provide systematically less favorable treatment effects than

RCTs utilizing traditional methods

There may be differences between traditional trials and trial designs using RCD

beyond data quality issues that would explain this

Page 14 of 49


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

Confidential: For Review OnlyAUTHOR STATEMENTS

Contributors: Kimberly A. Mc Cord: Data curation, Formal Analysis, Funding

acquisition, Investigation, Methodology, Project administration, Resources, Supervision,

Validation, Visualization, Writing – original draft, and Writing – review & editing;

Hannah Ewald: Data curation, Writing – review & editing; Arnav Agarwal: Data

curation, Writing – review & editing; Dominik Glinz: Data curation, Writing – review &

editing; Soheila Aghlmandi: Formal analysis, Methodology, Resources, Writing –

review & editing; John P.A. Ioannidis: Conceptualization, Methodology, Formal analysis

and Writing – review & editing; Lars G. Hemkens: Conceptualization, Formal Analysis,

Investigation, Methodology, Project administration, Resources, Supervision, Validation,

Visualization, Writing – original draft and Writing – review & editing. Kimberly A. Mc

Cord and Lars G. Hemkens are the guarantors.

Funding: The Basel Institute for Clinical Epidemiology and Biostatistics is supported by

the Stiftung Institut für klinische Epidemiologie (Kimberly A. Mc Cord, Lars G. Hemkens,

Ewald Hannah, Soheila Aghlmandi and Dominik Glinz). METRICS has been supported by

grants from the Laura and John Arnold Foundation (John P.A. Ioannidis, Lars G.

Hemkens). METRIC-B has been supported by an Einstein fellowship award to John P.A.

Ioannidis from the Stiftung Charite and the Einstein Stiftung (John P.A. Ioannidis and

Lars G. Hemkens).

Disclosure of potential conflicts of interest: All authors have completed the ICMJE

uniform disclosure form at www.icmje.org/coi_disclosure.pdf and declare: Kimberly A.

Mc Cord, John P.A. Ioannidis, Lars G. Hemkens support the RCD for RCT initiative,

which aims to explore the use of routinely collected data for clinical trials. Kimberly A.

Mc Cord and Lars G. Hemkens are members of the MARTA-Group, which aims to

explore how to Make Randomized Trials more Affordable. Dominik Glinz is employed

since June 1, 2020, by Roche Pharma (Schweiz) AG, Basel, Switzerland. The first draft of

this manuscript has been submitted before his current employment and his current

employer had no role in the design and conduct of the project and preparation, review,

approval of the manuscript; and decision to submit the manuscript for publication.

Page 15 of 49


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

Confidential: For Review OnlyThe authors declare no other relationships or activities that could appear to have

influenced the submitted work.

Disclaimer: The funders had no role in the design and conduct of the study; collection,

management, analysis and interpretation of the data; and preparation, review or

approval of the manuscript or its submission for publication.

Exclusive licences: The Corresponding Author has the right to grant on behalf of all

authors and does grant on behalf of all authors, a worldwide licence to the Publishers

and its licensees in perpetuity, in all forms, formats and media (whether known now or

created in the future), to i) publish, reproduce, distribute, display and store the

Contribution, ii) translate the Contribution into other languages, create adaptations,

reprints, include within collections and create summaries, extracts and/or, abstracts of

the Contribution, iii) create any other derivative work(s) based on the Contribution, iv)

to exploit all subsidiary rights in the Contribution, v) the inclusion of electronic links

from the Contribution to third party material where-ever it may be located; and, vi)

licence any third party to do any or all of the above.

Ethical approval: Not applicable for this kind of research

Information on author access to data: Available on request from the corresponding

author and on the Open Science Framework36.

Transparency declaration: The lead author affirms that the manuscript is an honest,

accurate, and transparent account of the study being reported; that no important

aspects of the study have been omitted; and that any relevant discrepancies from the

study as planned have been explained.

Dissemination declaration: Not applicable

Page 16 of 49


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

Confidential: For Review OnlyREFERENCES

1. Hemkens LG, Contopoulos-Ioannidis DG, Ioannidis JP. Routinely collected data and comparative effectiveness evidence: promises and limitations. CMAJ. 2016;188(8):E158-64. doi:10.1503/cmaj.150653

2. Mc Cord KA, Al-Shahi Salman R, Treweek S, et al. Routinely collected data for randomized trials: promises, barriers, and implications. Trials. 2018;19(1):29. doi:10.1186/s13063-017-2394-5

3. Mc Cord KA, Ewald H, Ladanie A, et al. Current use and costs of electronic health records for clinical trial research: a descriptive study. CMAJ Open. 2019;7(1):E23-E32. doi:10.9778/cmajo.20180096

4. Weisberg HI, Hayden VC, Pontes VP. Selection criteria and generalizability within the counterfactual framework: explaining the paradox of antidepressant-induced suicidality? Clin Trials. 2009;6(2):109-118. doi:10.1177/1740774509102563

5. Zwarenstein M, Treweek S. What kind of randomized trials do we need? CMAJ Can Med Assoc J J Assoc Medicale Can. 2009;180(10):998-1000. doi:10.1503/cmaj.082007

6. Schwartz D, Lellouch J. Explanatory and pragmatic attitudes in therapeutical trials. J Chronic Dis. 1967;20(8):637-648.

7. Zuidgeest MGP, Goetz I, Groenwold RHH, Irving E, van Thiel G, Grobbee DE. Series: Pragmatic trials and real world evidence: Paper 1. Introduction. J Clin Epidemiol. 2017;88:7-13. doi:10.1016/j.jclinepi.2016.12.023

8. Ramsberg J, Neovius M. Register or electronic health records enriched randomized pragmatic trials: The future of clinical effectiveness and cost-effectiveness trials? 2015. Published online 2015. doi:10.5617/njhe.1386

9. Nicholls SG, Langan SM, Benchimol EI. Routinely collected data: the importance of high-quality diagnostic coding to research. CMAJ Can Med Assoc J J Assoc Medicale Can. 2017;189(33):E1054-E1055. doi:10.1503/cmaj.170807

10. Arts DG, De Keizer NF, Scheffer GJ. Defining and improving data quality in medical registries: a literature review, case study, and generic framework. J Am Med Inf Assoc. 2002;9(6):600-611. doi:10.1197/jamia.m1087

11. Capurro D, PhD MY, van Eaton E, Black R, Tarczy-Hornoch P. Availability of Structured and Unstructured Clinical Data for Comparative Effectiveness Research and Quality Improvement: A Multisite Assessment. EGEMS. 2014;2(1):1079. doi:10.13063/2327-9214.1079

12. Feder SL. Data Quality in Electronic Health Records Research: Quality Domains and Assessment Methods. West J Nurs Res. 2018;40(5):753-766. doi:10.1177/0193945916689084

Page 17 of 49


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

Confidential: For Review Only13. Weiskopf NG, Weng C. Methods and dimensions of electronic health record data

quality assessment: enabling reuse for clinical research. J Am Med Inform Assoc JAMIA. 2013;20(1):144-151. doi:10.1136/amiajnl-2011-000681

14. Ladanie A, Speich B, Briel M, et al. Single pivotal trials with few corroborating characteristics were used for FDA approval of cancer therapies. J Clin Epidemiol. 2019;114:49-59. doi:10.1016/j.jclinepi.2019.05.033

15. Emilsson L, Lindahl B, Koster M, Lambe M, Ludvigsson JF. Review of 103 Swedish Healthcare Quality Registries. J Intern Med. 2015;277(1):94-136. doi:10.1111/joim.12303

16. Harron K, Wade A, Gilbert R, Muller-Pebody B, Goldstein H. Evaluating bias due to data linkage error in electronic healthcare records. BMC Med Res Methodol. 2014;14:36. doi:10.1186/1471-2288-14-36

17. van Smeden M, Lash TL, Groenwold RHH. Reflection on modern methods: five myths about measurement error in epidemiological research. Int J Epidemiol. Published online December 10, 2019. doi:10.1093/ije/dyz251

18. Dechartres A, Trinquart L, Boutron I, Ravaud P. Influence of trial sample size on treatment effect estimates: meta-epidemiological study. BMJ. 2013;346:f2304. doi:10.1136/bmj.f2304

19. Benchimol EI, Smeeth L, Guttmann A, et al. The REporting of studies Conducted using Observational Routinely-collected health Data (RECORD) statement. PLoS Med. 2015;12(10):e1001885. doi:10.1371/journal.pmed.1001885

20. Hemkens LG. Commentary on Bertagnolli et al: Clinical trial designs with routinely collected real-world data-issues of data quality and beyond. Clin Trials Lond Engl. 2020;17(3):247-250. doi:10.1177/1740774520913845

21. Joanna IntHout, Ioannidis JP, Borm GF. The Hartung-Knapp-Sidik-Jonkman method for random effects meta-analysis is straightforward and considerably outperforms the standard DerSimonian-Laird method. BMC Med Res Methodol. 2014;14(1):25. doi:10.1186/1471-2288-14-25

22. Marjoribanks J, Lethaby A, Farquhar C. Surgery versus medical therapy for heavy menstrual bleeding. Cochrane Database Syst Rev. 2016;(1). doi:10.1002/14651858.CD003855.pub3

23. Zhu F, Lee A, Chee YE. Fast‐track cardiac care for adult cardiac surgical patients. Cochrane Database Syst Rev. 2012;(10). doi:10.1002/14651858.CD003587.pub2

24. Wagner M, Earley AK, Webster AC, Schmid CH, Balk EM, Uhlig K. Mycophenolic acid versus azathioprine as primary immunosuppression for kidney transplant recipients. Cochrane Database Syst Rev. 2015;(12):CD007746. doi:10.1002/14651858.CD007746.pub2

25. Balduzzi S, Rücker G, Schwarzer G. How to perform a meta-analysis with R: a practical tutorial. Evid Based Ment Health. Published online September 28, 2019. doi:10.1136/ebmental-2019-300117

Page 18 of 49


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

Confidential: For Review Only26. RStudio | Open source & professional software for data science teams - RStudio.

Accessed January 11, 2020. https://rstudio.com/

27. R: The R Project for Statistical Computing. Accessed January 11, 2020. https://www.r-project.org/

28. Patsopoulos NA. A pragmatic view on pragmatic trials. Dialogues Clin Neurosci. 2011;13(2):217-224.

29. Ndounga Diakou L, Trinquart L, Hróbjartsson A, et al. Comparison of central adjudication of outcomes and onsite outcome assessment on treatment effect estimates. Cochrane Database Syst Rev. 2016;(3). doi:10.1002/14651858.MR000043.pub2

30. Strom Jordan B., Faridi Kamil F., Butala Neel M., et al. Use of Administrative Claims to Assess Outcomes and Treatment Effect in Randomized Clinical Trials for Transcatheter Aortic Valve Replacement: Findings from the Extending Trial-Based Evaluations of Medical Therapies Using Novel Sources of Data (EXTEND) Study. Circulation. 0(0). doi:10.1161/CIRCULATIONAHA.120.046159

31. U.S. Food and Drug Administration (FDA). Framework for FDA’s real-world evidence program. Published online December 2018.

32. Mathes T, Klaßen P, Pieper D. No differences were found between effect estimates from conventional and registry-based randomized controlled trials. J Clin Epidemiol. 2019;105:80-91. doi:10.1016/j.jclinepi.2018.09.011

33. Dal-Ré R, Janiaud P, Ioannidis JPA. Real-world evidence: How pragmatic are randomized controlled trials labeled as pragmatic? BMC Med. 2018;16(1):49. doi:10.1186/s12916-018-1038-2

34. Dal-Ré R. Articles provided insufficient information to conduct an appropriate retrospective assessment of the pragmatic/explanatory features of medicine trials with the PRECIS-2 tool. Eur J Clin Pharmacol. 2020;76(8):1093-1102. doi:10.1007/s00228-020-02901-4

35. Kwakkenbos L, Juszczak E, Hemkens LG, et al. Protocol for the development of a CONSORT extension for RCTs using cohorts and routinely collected health data. Res Integr Peer Rev. 2018;3(1):9. doi:10.1186/s41073-018-0053-3

36. https://osf.io/cdbhn/

37. Gonçalves‐Bradley DC, Lannin NA, Clemson LM, Cameron ID, Shepperd S. Discharge planning from hospital. Cochrane Database Syst Rev. 2016;(1). doi:10.1002/14651858.CD000313.pub5

38. McFadden A, Gavine A, Renfrew MJ, et al. Support for healthy breastfeeding mothers with healthy term babies. Cochrane Database Syst Rev. 2017;(2). doi:10.1002/14651858.CD001141.pub5

39. Gøtzsche PC, Jørgensen KJ. Screening for breast cancer with mammography. Cochrane Database Syst Rev. 2013;(6):CD001877. doi:10.1002/14651858.CD001877.pub5

Page 19 of 49


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

Confidential: For Review Only40. Henry DA, Carless PA, Moxey AJ, et al. Anti-fibrinolytic use for minimising

perioperative allogeneic blood transfusion. Cochrane Database Syst Rev. 2011;(1):CD001886. doi:10.1002/14651858.CD001886.pub3

41. Everett T, Bryant A, Griffin MF, Martin-Hirsch PP, Forbes CA, Jepson RG. Interventions targeted at women to encourage the uptake of cervical screening. Cochrane Database Syst Rev. 2011;(5):CD002834. doi:10.1002/14651858.CD002834.pub2

42. Zwerink M, Brusse-Keizer M, van der Valk PDLPM, et al. Self management for patients with chronic obstructive pulmonary disease. Cochrane Database Syst Rev. 2014;(3):CD002990. doi:10.1002/14651858.CD002990.pub3

43. Long L, Mordi IR, Bridges C, et al. Exercise‐based cardiac rehabilitation for adults with heart failure. Cochrane Database Syst Rev. 2019;(1). doi:10.1002/14651858.CD003331.pub5

44. Wong W-T, Lai VK, Chee YE, Lee A. Fast-track cardiac care for adult cardiac surgical patients. Cochrane Database Syst Rev. 2016;9:CD003587. doi:10.1002/14651858.CD003587.pub3

45. Vann JCJ, Jacobson RM, Coyne‐Beasley T, Asafu‐Adjei JK, Szilagyi PG. Patient reminder and recall interventions to improve immunization rates. Cochrane Database Syst Rev. 2018;(1). doi:10.1002/14651858.CD003941.pub3

46. Fanning JP, Nyong J, Scott IA, Aroney CN, Walters DL. Routine invasive strategies versus selective invasive strategies for unstable angina and non‐ST elevation myocardial infarction in the stent era. Cochrane Database Syst Rev. 2016;(5). doi:10.1002/14651858.CD004815.pub4

47. Cameron I, Gillespie L, Robertson M, et al. Interventions for preventing falls in older people in care facilities and hospitals. Cochrane Database Syst Rev. 2012;(12). doi:10.1002/14651858.CD005465.pub3

48. Archer J, Bower P, Gilbody S, et al. Collaborative care for depression and anxiety problems. Cochrane Database Syst Rev. 2012;(10). doi:10.1002/14651858.CD006525.pub2

49. Bjelakovic G, Nikolova D, Gluud LL, Simonetti RG, Gluud C. Antioxidant supplements for prevention of mortality in healthy participants and patients with various diseases. Cochrane Database Syst Rev. 2012;(3):CD007176. doi:10.1002/14651858.CD007176.pub2

50. Møller C, Penninga L, Wetterslev J, Steinbrüchel D, Gluud C. Off‐pump versus on‐pump coronary artery bypass grafting for ischaemic heart disease. Cochrane Database Syst Rev. 2012;(3). doi:10.1002/14651858.CD007224.pub2

51. Inglis SC, Clark RA, Dierckx R, Prieto-Merino D, Cleland JGF. Structured telephone support or non-invasive telemonitoring for patients with heart failure. Cochrane Database Syst Rev. 2015;(10):CD007228. doi:10.1002/14651858.CD007228.pub3

Page 20 of 49


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

Confidential: For Review Only52. Palmer SC, Navaneethan SD, Craig JC, et al. HMG CoA reductase inhibitors (statins)

for people with chronic kidney disease not requiring dialysis. Cochrane Database Syst Rev. 2014;(5):CD007784. doi:10.1002/14651858.CD007784.pub2

53. Reilly S, Miranda‐Castillo C, Malouf R, et al. Case management approaches to home support for people with dementia. Cochrane Database Syst Rev. 2015;(1). doi:10.1002/14651858.CD008345.pub2

54. Christensen M, Lundh A. Medication review in hospitalised patients to reduce morbidity and mortality. Cochrane Database Syst Rev. 2016;2:CD008986. doi:10.1002/14651858.CD008986.pub3

55. Adler AJ, Taylor F, Martin N, Gottlieb S, Taylor RS, Ebrahim S. Reduced dietary salt for the prevention of cardiovascular disease. Cochrane Database Syst Rev. 2014;(12):CD009217. doi:10.1002/14651858.CD009217.pub3

56. Gunaratne AW, Makrides M, Collins CT. Maternal prenatal and/or postnatal n-3 long chain polyunsaturated fatty acids (LCPUFA) supplementation for preventing allergies in early childhood. Cochrane Database Syst Rev. 2015;(7):CD010085. doi:10.1002/14651858.CD010085.pub2

Page 21 of 49


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

Confidential: For Review Only

TABLES AND FIGURES

Table 1: Overview of trial characteristicsCharacteristic No. (%) of overall RCD-

RCTs No. (%) Registry RCD-RCTs

No. (%) Admin RCD-RCTs

No. (%) EHR RCD-RCTs

No. (%) of traditional RCTs

Frequency 84 (100%) 36 (43%) 18 (21%) 30 (36%) 463 (100%)Publication year Median [IQR] 2005 [1998 - 2009] 2003 [1992 - 2009] 2007 [2003 - 2012] 2006 [2000- 2011] 2003 [1997 - 2006]Range (min - max) 1976 - 2017 1976 - 2015 1998 - 2015 1989 - 2017 1963 - 2016Number of participantsMedian [IQR]Range (min – max)

721 [275 - 2729]16 - 89699

2037 [524 - 17066]99 - 89699

1403 [414 - 3406]45 - 24743

286 [146 - 534]16 - 12205

121 [60 - 359]16 - 160840

Number of eventsMedian [IQR]Range (min - max)

194 [50 - 1266]0 - 86201

440 [65 - 1383]4 - 86201

559 [98 - 1734]0 - 18146

124 [33 - 271]0 - 5562

27 [6 - 100]0 -3364

Cluster-randomized design1 4 (5%) 1 (3%) 2 (11%) 1 (3%) 18 (4%)Age2

Median [IQR]Range (min - max)

52 [25 - 68]0 - 87

16 [1 - 59]0 - 79

67 [45 - 73]2 - 85

55 [46 - 63]1 - 89 62 [57 - 66]

0 - 87CountryAustraliaBrazilChinaContinental EuropeNorth AmericaScandinavia3

United KingdomOther4

Not reported

0 (0%)0 (0%)0 (0%)3 (4%)

56 (67%)14 (17%)6 (7%)5 (6%)0 (0%)

0 (0%)0 (0%)0 (0%)0 (0%)

16 (44%)13 (36%)4 (11%)3 (8%)0 (0%)

0 (0%)0 (0%)0 (0%)

1 (5.5%)14 (78%)1 (5.5%)1 (5.5%)1 (5.5%)0 (0%)

0 (0%)0 (0%)0 (0%)2 (7%)

26 (87%)0 (0%)1 (3%)1 (3%)0 (0%)

20 (4%)9 (2%)5 (1%)

60 (13%)125 (27%)31 (7%)43 (9%)45 (10%)125 (27%)

Risk of bias High in ≥ 1 domainLow in all domainsUnclear or low in all domains

35 (42%)16 (19%)33 (39%)

16 (44%)8 (22%)12 (33%)

5 (28%)3 (17%)10 (56%)

14 (47%)5 (17%)11 (37%)

218 (47%)79 (17%)166 (36%)

BlindingHighLowUnclear

21 (25%)32 (38%)31 (37%)

5 (14%)18 (50%)13 (36%)

3 (17%)6 (33%)9 (50%)

13 (43%)8 (27%)9 (30%)

158 (34%)194 (42%)111 (24%)

Estimated Data quality N.A.High 56 (67%) 31 (86%) 9 (50%) 16 (53%) -Low 24 (29%) 3 (8%) 9 (50%) 12 (40%) -

Page 22 of 49


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960


Unclear 4 (5%) 2 (6%) 0 (%) 2 (7%) -RCD collection level5 N.A.Complete RCD 57 (68%) 29 (81%) 10 (56%) 18 (60%) -Hybrid 27 (32%) 7 (19%) 8 (44%) 12 (40%) -Admin: administrative; EHR: electronic health record; IQR: interquartile range; N.A.: Not applicable; RCT: randomized clinical trial; RCD: routinely collected data.1) Two trials (0.5%) were described as cross-over by Cochrane reviewers.2) Information reported for 345 of traditional RCTs and 48 RCD-RCTs. 3) Scandinavia includes Sweden, Norway, Denmark, Finland and Iceland4) Other includes Europe (more than one country, multicenter), Worldwide (more than one country outside of Europe, multicenter), Taiwan, Syria, India, Iran, Belarus, Malaysia, Egypt, Turkey, Zimbawe, New Zealand, Chile, Colombia, Israel, Venezuela, Japan, Hong Kong, Pakistan, Argentina, Korea, Singapore, South Africa, Georgia.5) Complete RCD: fully RCD-based data collection; Hybrid: routine data collection with supportive active data collection.

Page 23 of 49


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960


Table 2: Clinical questions and corresponding trials:Clinical question Outcome Cochrane review

identifier; meta-analysis number

Number of trials Median trial size; IQR; Range

Individualized discharge plan for all hospitalized patients Unscheduled readmissions CD00031337; 2.1.0 3 RCD-RCTs14 other RCTs

575; 336 - 637; 96 - 698205; 97–278; 50 - 738

Breastfeeding support for healthy pregnant women intending to breastfeed or already breastfeeding

Stopping breastfeeding CD00114138; 1.1.0 1 RCD-RCT48 other RCTs

990329; 136 - 520; 41 - 1660

Mammography screening in women without previous breast cancer diagnosis

Breast cancer mortality CD00187739; 1.1.0 7 RCD-RCTs4 other RCTs

39405; 24767 - 46357; 17793 - 5917669485; 60974 - 97937; 57897 - 160840

Anti-fibrinolytic agents in patients undergoing surgery Need for allogeneic blood transfusion

CD00188640; 1.1.0 1 RCD-RCT107 other RCTs

1659; 40 - 98; 17 - 1784

Interventions to increase uptake of cervical cancer screening Uptake of screening CD00283441; 1.1.1 6 RCD-RCTs6 other RCTs

1157; 358 - 2335; 314 - 89699482; 162 - 1317; 97 - 1794

Self-management interventions in patients with COPD Mortality CD00299042; 1.21.0 1 RCD-RCT8 other RCTs

191164; 145 - 211; 135 - 743

Exercised-based interventions in patients with heart failure Hospital Admission CD00333143; 1.4.0 1 RCD-RCT6 other RCTs

233047; 28 - 87; 23 - 123

Fast track interventions for early extubation (time-directed extubation protocol) in patients undergoing cardiac surgery

Mortality CD00358744; 2.1.4 3 RCD-RCTs7 other RCTs

120; 84 - 359; 48 - 59798; 66 - 172; 60 - 404

Levonorgestrel-intrauterine device vs surgery in women with heavy menstrual bleeding

Additional surgery received CD00385522; 2.13.0 1 RCD-RCT5 other RCTs

22560; 57-69; 57-72

Reminder and recall immunization interventions in adults and children Immunizations CD00394145; 1.1.0 30 RCD-RCTs27 other RCTs

1888; 751 - 4598; 204 - 24743304; 173 - 555; 96 - 3006

Routine invasive vs conservative selective treatment in patients with unstable angina and non-ST elevation myocardial infarction

Mortality or non-fatal myocardial infarction

CD00481546; 1.13.0 1 RCD-RCT2 other RCTs

24571505; 1353 - 1658; 1200 - 1810

Interventions to reduce falls in those aged 60 years or older in care facilities and hospitals

Falls CD00546547; 4.2.0 2 RCD-RCT4 other RCTs

1883; 965 - 2800; 48 - 3717353; 114 - 594; 91 - 625

Page 24 of 49


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960


Collaborative care interventions for people with depression and anxiety Antidepressant medication use

CD00652548; 1.3.1 13 RCD-RCTs31 other RCTs

208; 88 - 285; 45 - 372179; 83 - 292; 34 - 1570

Antioxidant supplementation in healthy participants and in patients with various stable diseases


15022; 7966 - 22077; 910 - 29133357; 99 - 1667; 19 - 39876

On-pump surgery in patients undergoing CABG Mortality CD00722450; 1.1.0 1 RCD-RCT73 other RCTs

33960; 40 - 120; 20 - 2203

Structured telephone support or non-invasive telemonitoring interventions in patients with heart failure


319; 263 - 515; 206 - 710141; 91 - 259; 20 - 460

Mycophenolic acid vs azathioprine as primary immunosuppression for adult and children kidney transplant recipients

Graft loss CD00774624; 1.3.3 1 RCD-RCT3 other RCTs

13376; 72 - 162; 68 - 248

Statins in patients with chronic kidney disease not requiring dialysis Mortality CD00778452; 1.2.0 2 RCD-RCTs8 other RCTs

9565; 5936 - 13195; 2306 - 16824722; 255 - 1472; 87 - 3267

Case management interventions in people with dementia Hospital admissions CD00834553; 1.5.2 2 RCD-RCTs3 other RCTs

141; 133 - 149; 125 - 15789; 89 - 108; 88 - 126

Medication review in hospitalized patients Mortality CD00898654; 1.1.0 1 RCD-RCT8 other RCTs

99368; 120 - 485; 66 - 936

Interventions to reduce dietary salt in hypertensive patients Mortality CD00921755; 1.1.0 1 RCD-RCT6 other RCTs

1981519; 401 - 710; 67 - 2382

Fish oil for pregnant or breastfeeding women to prevent allergies in their children

Allergies CD01008556; 6.2.1 1 RCD-RCT3 other RCT

528531; 324 - 619; 117 - 706

All comparators were no intervention or usual care if not stated otherwise. All but three outcomes (CD002990; CD003587; CD006525) were primary outcomes of the Cochrane review. In four Cochrane reviews (CD000313; CD001886; CD004815; CD007746) there was also a pertinent mortality outcome reported which was used for the secondary analysis.IQR: interquartile range; RCD: routinely collected data.

Page 25 of 49


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960


Table 3: Results of analyses of the agreement of treatment effects measured with or without RCD in clinical trials Analysis No. of clinical questions ROR (95% CI) I2 (95% CI)

Main analysis 22 0.80 (0.70 - 0.91) 14% (0-48%)Secondary analysis: mortality outcomes used when available 12 0.92 (0.74 - 1.15) 12% (0-52%)Secondary analysis: primary outcomes used when available 19 0.79 (0.70 - 0.90) 9% (0-45%)Subgroup analyses

RCD-trials using registries only 14 0.86 (0.75 – 0.99) 20% (0-57%)RCD-trials using EHRs only 9 0.81 (0.59 - 1.11) 28% (0-67%)RCD-trials using administrative claims data only 9 0.84 (0.72 – 0.99) 0% (0-58%)RCD-trials with high data quality only 17 0.82 (0.72 - 0.93) 0% (0-50%)Sensitivity analyses

RCD-trials with hybrid data collection excluded 18 0.88 (0.78 - 1.00) 0% (0-49%)RCD-trials with cluster randomization or cross-over design excluded 20 0.84 (0.69 - 1.02) 28% (0-58%)RCD-trials published in 2005 or later only 15 0.73 (0.63 - 0.85) 4% (0-56%)RCD-trials published in 2010 or later only 6 0.81 (0.55 - 1.19) 39% (0-76%)RCD-trials published within 5 years before/after traditional RCTs 15 0.78 (0.69 - 0.88) 0% (0-21%)RCD-trials with low risk of bias (all domains) only 10 0.85 (0.60 - 1.23) 49% (0-75%)RCD-trials with low risk of bias (blinding) only 13 0.87 (0.69 - 1.08) 28% (0-63%)Smallest trials (lowest tertile)1 13 0.82 (0.57 - 1.17) 0% (0-53%)Medium trials (middle tertile)1 11 0.85 (0.72 - 1.02) 0% (0-43%)Largest trials (largest tertile)1 4 0.92 (0.86 - 0.99) 0% (0-60%)Lowest number of events (lowest tertile)2 14 1.12 (0.79 - 1.60) 1% (0-55%)Medium number of events (middle tertile)2 7 0.78 (0.65 – 0.92) 0% (0-11%)Largest number of events (largest tertile)2 4 0.93 (0.85 - 1.01) 0% (0-74%)Age of RCD-RCT population within SD of median age of traditional RCTs 13 0.81 (0.66 - 0.99) 49% (3-73%)Subset of mortality outcomes 9 0.91 (0.76 –1.10) 0% (0-60%)Subset of non-mortality outcomes 12 0.71 (0.60 – 0.84) 8% (0-62%)Excluding clinical questions with active comparators 19 0.80 (0.70 - 0.91) 14% (0-49%)Excluding clinical questions with largely different precision per summary estimates 15 0.78 (0.68 - 0.91) 18% (0-55%)Excluding clinical questions with less than 3 RCD-RCTs 7 0.73 (0.64 - 0.84) 0% (0-61%)Excluding clinical questions with more than 10 RCD-RCTs 20 0.82 (0.70 - 0.97) 22% (0-54%)Index-RCD-RCTs versus all other (indirectly identified RCD-RCTs and traditional trials) 22 0.91 (0.78 - 1.06) 17% (0-51%)DerSimonian-Laird random effects meta-analyses 22 0.84 (0.75 - 0.94) 40% (1-64%)Fixed-effect meta-analyses 22 0.88 (0.83 - 0.93) 40% (1-64%)CI: Confidence interval; EHR: electronic health record; I2: Heterogeneity; RCD: Routinely collected data; RCTs: Randomized clinical trials; ROR: Ratio of odds ratio; SD: Standard deviation1) Tertiles for participants were 333 and 1997, based on RCD-RCTs. 2) Tertiles for events were 75 and 502, based on RCD-RCTs.A ROR <1 indicates that the RCD-RCT estimated a less favorable treatment effect of the evaluated treatment than the traditional RCT.

Page 26 of 49


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960


Page 27 of 49


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960


Figure 1: Treatment effects measured with or without RCD in clinical trials for 22 clinical questions. Overview of summary results.

Page 28 of 49


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960


CABG: Coronary artery bypass grafting; COPD: Chronic obstructive pulmonary disease; IUD: Intrauterine device; sOR: summary odds ratio; RCD: routinely collected data; trad. RCT: traditional randomized controlled trial not using RCD for outcome collection; MI: Myocardial infarction; ROR: ratio of odds ratio; UA/NSTEMI: Unstable angina/Non-ST-elevation myocardial infarction.Ordered by ROR.

Page 29 of 49


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960


Figure 2: Agreement of treatment effects measured with or without RCD in clinical trials. Forest plot of main analysis.

CABG: Coronary artery bypass grafting; COPD: Chronic obstructive pulmonary disease; IUD: Intrauterine device; sOR: summary odds ratio; RCD: routinely collected data; trad. RCT: traditional randomized controlled trial not using RCD for outcome collection; MI: Myocardial infarction; ROR: ratio of odds ratio; UA/NSTEMI: Unstable angina/Non-ST-elevation myocardial infarction.Ordered byROR.

Page 30 of 49


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

Confidential: For Review OnlyAPPENDICES

Appendix 1: Search strategySearch Query Items

found#4 #1 NOT #2

Filters: Randomized Controlled Trial; Publication date from 2000/01/01 to 2016/12/314635

#3 #1 NOT #2 446676#2 animals[mh] NOT humans [mh] 4186023#1 “routine data”[tiab] OR “routinely collected”[tiab] OR Administrative[tiab] OR Claims[tiab]

OR "Registries"[mh] OR registry[tiab] OR registries[tiab] OR database*[tiab] OR "healthcare data"[All fields] OR "health care data"[All fields] OR "national database"[All fields] OR "Databases as Topic"[Mesh] OR "Administrative Claims, Healthcare"[Mesh]

464227

Interface: PubMed; Date of last search: 11 March 2016

For EHR-RCTs, we integrated a specific complementary search from a related project on the use of EHR in RCD-RCTs (Mc Cord KA et al. “Current use and costs of electronic health records for clinical trial research: a descriptive study” CMAJ Open. 2019 Feb 3;7(1):E23-E32). Date of last search 13 Sep 2017. This search is described in detail elsewhere (URL: http://cmajopen.ca/content/7/1/E23/suppl/DC1 ).

For the current project we assessed the trials that used EHR infrastructures for recruitment or outcome measurement as categorized in this related project. We did not use the trials that explored EHR technology itself as here no traditional RCTs would be available for the current project.

Page 31 of 49


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

Confidential: For Review OnlyAppendix 2: Study Flow diagram

CR = Cochrane review, MA = meta-analysis, RCD = routinely collected data, RCT = randomized clinical trial.* EHR-RCTs identified in a complementary search. For details see Appendix 1** Records with clear ineligibility (as they were not in a Cochrane Review meta-analysis) were directly excluded and not assessed as full-texts

Page 32 of 49


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960


Appendix 3: Treatment effect estimates in individual trials per clinical question

The red diamonds indicate the RCD-RCTs. The blue circles are traditional trials. Each symbol is a point estimate of the single trial used in the respective meta-analysis. The size is larger when the precison (1/standard error) is higher.

Page 33 of 49


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960


Appendix 4: Temporal sequence of individual trials per clinical question

The red diamonds indicate the RCD-RCTs. The blue circles are traditional trials. Each symbol represents a single trial used in the respective meta-analysis. The size is larger when the precison (1/standard error) of the point estimate of that trial is higher.

Page 34 of 49


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

Confidential: For Review OnlyAppendix 5: References of all RCD-RCTs

a) Index RCD-RCTs and corresponding publications identified via electronic literature search:

Cochrane review identifier;Meta-Analysis number

Cochrane Review Trial ID*

Reference

CD000313 Analysis 2.1.0

Farris 2014 Farris KB, Carter BL, Xu J, Dawson JD, Shelsky C, Weetman DB, et al. Effect of a care transition intervention by pharmacists: an RCT. BMC Health Services Research 2014;14:406.


Goldman 2014 Goldman LE, Sarkar U, Kessell E, Guzman D, Schneidermann M, Pierluissi E, et al. Support from hospital to home for elders: a randomized trial. Annals of Internal Medicine 2014;161(7):472-81.


Hoddinott 2009 Hoddinott P, Britten J, Prescott GJ, Tappin D, Ludbrook A, Godden DJ. Effectiveness of policy to provide breastfeeding groups (BIG) for pregnant and breastfeeding mothers in primary care: cluster randomised controlled trial. BMJ 2009;338:a3026.


Canada 1980b Miller AB, To T, Baines CJ, Wall C. Canadian National Breast Screening Study-2: 13-year results of a randomized trial in women aged 50-59 years. Journal of the National Cancer Institute 2000;92:1490-9.


Göteborg 1982a Bjurstam N, Bjorneld L, Duffy SW, Smith TC, Cahlin E, Eriksson O, et al. The Gothenburg breast screening trial: first results on mortality, incidence, and mode of detection for women ages 39-49 years at randomization. Cancer 1997;80(11):2091-9.


Norman 2009 Norman P H, Thall P F, Purugganan R V, Riedel B J, Thakar D R, Rice D C, et al. A possible association between aprotinin and improved survival after radical surgery for mesothelioma. Cancer 2009;115(4):833-41.


Stein 2005 Stein K, Lewendon G, Jenkins R, Davis C. Improving uptake of cervical cancer screening in women with prolonged history of non-attendance for screening: a randomized trial of enhanced invitation methods. Journal of Medical Screening 2005;12(4):185-9.


Bourbeau 2003 Gadoury MA, Schwartzman K, Rouleau M, Maltais F, Julien M, Beaupre A, et al. Self-management reduces both short- and long-term hospitalisation in COPD. European Respiratory Journal 2005;26(5):853-7.


HF ACTION 2009 Reed SD, Li Y, Dunlap ME, et al. In-hospital resource use and medical costs in the last year of life by mode of death (from the HF-ACTION randomized controlled trial). Am J Cardiol. 2012;110(8):1150-1155.


Cheng 1996a,1996b, 2003

Cheng DC, Wall C, Djaiani G, Peragallo RA, Carroll J, Li C, et al. Randomized assessment of resource use in fast-track cardiac surgery 1-year after hospital discharge. Anesthesiology 2003;98(3):651-7. [MEDLINE: 12606909]


Hurskainen 2001 Heliovaara-Peippo S, Halmesmaki K, Hurskainen R, Teperi J, Grenman S, Kivela A, et al. The effect of hysterectomy or levonorgestrel-releasing intrauterine system on lower urinary tract symptoms: a 10-year follow-up study of a randomised trial. BJOG: an International Journal of Obstetrics and Gynaecology 2010;117(5):602-9.


Daley2004T513 Daley MF, Barrow J, Pearson K, Crane LA, Gao D, Stevenson JM, Berman S, Kempe A. Identification and recall of children with chronic medical conditions for influenza vaccination. Pediatrics 2004;113(1):e26-e33.


Irigoyen2006T702 Irigoyen MM, Findley S, Wang D, Chen S, Chimkin F, Pena O, Mendonca E. Challenges and successes of immunization registry reminders at inner-city practices. Ambulatory Pediatrics 2006;6(2):100-104.


Kempe2005T707 Kempe A, Daley MF, Barrow J, Allred N, Hester N, Beaty BL, Crane LA, Pearson K, Berman S. Implementation of universal influenza recommendations for healthy young children: results of a randomized, controlled trial with registry-based recall. Pediatrics 2005;115(1):146-154.


LeBaron2004T512 LeBaron CW, Starnes DM, Rask KJ. The impact of reminder-recall interventions on low vaccination coverage in an inner-city population. Archives of Pediatrics & Adolescent Medicine 2004;158(3):255-261.


FRISC-II Lagerqvist B, Husted S, Kontny F, Ståhle E, Swahn E, Wallentin L. 5-year outcomes in the FRISC-II randomised trial of an invasive versus a non-invasive strategy in non-ST-elevation acute coronary syndrome: a follow-up study. Lancet 2006;368(9540):998-1004.


Broe 2007 Broe KE, Chen TC, Weinberg J, Bischoff-Ferrari HA, Holick MF, Kiel DP. A higher dose of vitamin D reduces the risk of falls in nursing home residents: A randomized, multiple-dose study. Journal of the American Geriatrics Society 2007;55(2):234-9.


Law 2006 Law M, Withers H, Morris J, Anderson F. Vitamin D supplementation and the prevention of fractures and falls: results of a randomised trial in elderly people in residential accommodation. Age and Ageing 2006;35(5):482-6.


Landis 2007 Landis SE, Gaynes BN, Morrissey JP, Vinson N, Ellis AR, Domino ME. Generalist care managers for the treatment of depressed medicaid patients in North Carolina: a pilot study. BMC Family Practice 2007;8:7.

Page 35 of 49


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

Confidential: For Review OnlyCD006525 Analysis 1.3.1

Simon 2004a Simon GE, Ludman EJ, Tutty S, Operskalski B, Korff M. Telephone psychotherapy and telephone care management for primary care patients starting antidepressant treatment: a randomized controlled trial. JAMA 2004;292(8):935-42.


ATBC 2003Low Virtamo J, Edwards BK, Virtanen M, Taylor PR, Malila N, Albanes D, et al. EJects of supplemental alpha-tocopherol and beta-carotene on urinary tract cancer: incidence and mortality in a controlled trial (Finland). Cancer Causes & Control 2000;11(10):933-9.


BBS 2011 Møller CH, Perko MJ, Lund JT, Andersen LW, Kelbaek H, Madsen JK, Winkel P, Gluud C, Steinbrüchel DA. Three-year follow-up in a subset of high-risk patients randomized to o)- pump versus on-pump coronary artery bypass surgery: The Best Bypass Surgery Trial. Heart 2011;97(11):907-13.


Blum 2014 (MCCD) Blum K, Gottlieb SS. The eCect of a randomised trial of home telemonitoring on medical costs, 30-day readmissions, mortality, and health-related quality of life in a cohort of community-dwelling heart failure patients. Journal of Cardiac Failure 2014;20(7):513-21.


MMF TRI Study 1996

Clayton PA, McDonald SP, Chapman JR, Chadban SJ. Mycophenolate versus azathioprine for kidney transplantation: a 15-year follow-up of a randomized trial. Transplantation 2012;94(2):152-8.


4S 1993 Pedersen TR, Wilhelmsen L, Faergeman O, Strandberg TE, Thorgeirsson G, Troedsson L, et al. Follow-up study of patients randomized in the Scandinavian Simvastatin Survival Study (4S) of cholesterol lowering. American Journal of Cardiology 2000;86(3):257-62.


Bass - Ohio Bass DM, Clark PA, Looman WJ, McCarthy CA, Eckert S. The Cleveland Alzheimer's Managed Care Demonstration: Outcomes after 12 months of Implementation. The Gerontologist 2003;43(1):73-85.


Lisby 2010 Lisby M, Thomsen A, Nielsen LP, Lyhne NM, Breum-Leer C, Fredberg U, et al. The effect of systematic medication review in elderly patients admitted to an acute ward of internalmedicine. Basic and Clinical Pharmacology and Toxicology 2010;106(5):422-7.


Chang 2006 Chang HY, Hu YW, Yue CS, Wen YW, Yeh WT, Hsu LS, et al. EGect of potassium-enriched salt on cardiovascular mortality and medical expenses of elderly men. American Journal of Clinical Nutrition 2006;83:1289-96.


Olsen 1992 Olsen SF, Osterdal ML, Salvig JD, Mortensen LM, Rytter D, Secher NJ, et al. Fish oil intake compared with olive oil intake in late pregnancy and asthma in the oJspring: 16 y of registrybased follow-up from a randomized controlled trial. American Journal of Clinical Nutrition 2008;88(1):167-75.

* Very few Trial IDs may have been slightly renamed in Cochrane Review updates (e.g. Daley2004T513 has become Daley2004a).

Page 36 of 49


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

Confidential: For Review Onlyb) RCD-RCTs identified in Cochrane review meta-analysis



CD000313 Analysis 2.1.0 Balaban 2008

CD001877 Analysis 1.1.0 Göteborg 1982b

CD001877 Analysis 1.1.0 Malmö 1976

CD001877 Analysis 1.1.0 Malmö II 1978

CD001877 Analysis 1.1.0 Stockholm 1981

CD001877 Analysis 1.1.0 Canada 1980a

CD002834 Analysis 1.1.1 Buehler 1997

CD002834 Analysis 1.1.1 Morrell 2005

CD002834 Analysis 1.1.1 Burack 1998

CD002834 Analysis 1.1.1 Burack 2003

CD002834 Analysis 1.1.1 McDowell 1989

CD003587 Analysis 2.1.4 van Mastrigt 2006a, 2010

CD003587 Analysis 2.1.4 Gruber 2008

CD003941 Analysis 1.1.0 Marron 1998

CD003941 Analysis 1.1.0 Staras 2015

CD003941 Analysis 1.1.0 Suh 2006

CD003941 Analysis 1.1.0 Winston 2007

CD003941 Analysis 1.1.0 Baker 1998T96

CD003941 Analysis 1.1.0 Brigham 2012

CD003941 Analysis 1.1.0 CDC 2012

CD003941 Analysis 1.1.0 Chao 2015

CD003941 Analysis 1.1.0 Daley 2002

CD003941 Analysis 1.1.0 Dombkowski 2012

CD003941 Analysis 1.1.0 Dombkowski 2014 (1)



CD003941 Analysis 1.1.0 Hambidge 2009

CD003941 Analysis 1.1.0 Kempe2001T706

CD003941 Analysis 1.1.0 Lieu1997T69

CD003941 Analysis 1.1.0 Linkins1994T49

CD003941 Analysis 1.1.0 Mason 2000

CD003941 Analysis 1.1.0 McCaul 2002

CD003941 Analysis 1.1.0 O'Leary 2015

CD003941 Analysis 1.1.0 Rand 2015

CD003941 Analysis 1.1.0 Rand 2017

CD003941 Analysis 1.1.0 Roca 2012

CD003941 Analysis 1.1.0 Soljak 1987

CD003941 Analysis 1.1.0 Vivier 2000

CD003941 Analysis 1.1.0 Moniz 2013

CD006525 Analysis 1.3.1 Simon 2011

CD006525 Analysis 1.3.1 Fortney 2007

CD006525 Analysis 1.3.1 Hunkeler 2000

Page 37 of 49


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

Confidential: For Review OnlyCD006525 Analysis 1.3.1 Cole 2006

CD006525 Analysis 1.3.1 Katon 1996a

CD006525 Analysis 1.3.1 Katon 1996b

CD006525 Analysis 1.3.1 Katon 1999



CD006525 Analysis 1.3.1 Simon 2000a

CD006525 Analysis 1.3.1 Simon 2004b

CD007176 Analysis 1.1.0 MAVIS 2005 Low

CD007228 Analysis 1.2.0 Lyngå 2012 (WISH)

CD007228 Analysis 1.2.0 Koehler 2011 (TIM-HF)

CD007784 Analysis 1.2.0 PPP Study 1992

CD008345 Analysis 1.5.2 Eloniemi-Sulkava 2009

* Very few Trial IDs may have been slightly renamed in Cochrane Review updates.

Page 38 of 49


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

Confidential: For Review OnlyAppendix 6: References of all traditional trials in main analysis



CD000313 Analysis 2.1.0 Lainscak 2013

CD000313 Analysis 2.1.0 Laramee 2003

CD000313 Analysis 2.1.0 Shaw 2000

CD000313 Analysis 2.1.0 Jack 2009

CD000313 Analysis 2.1.0 Moher 1992

CD000313 Analysis 2.1.0 Harrison 2002

CD000313 Analysis 2.1.0 Naylor 1994

CD000313 Analysis 2.1.0 Lin 2009

CD000313 Analysis 2.1.0 Rich 1993a

CD000313 Analysis 2.1.0 Rich 1995a

CD000313 Analysis 2.1.0 Kennedy 1987

CD000313 Analysis 2.1.0 Pardessus 2002

CD000313 Analysis 2.1.0 Nazareth 2001

CD000313 Analysis 2.1.0 Legrain 2011

CD001141 Analysis 1.1.0 Albernaz 2003

CD001141 Analysis 1.1.0 Barros 1994

CD001141 Analysis 1.1.0 Bashour 2008

CD001141 Analysis 1.1.0 Bhandari 2003

CD001141 Analysis 1.1.0 Bonuck 2005

CD001141 Analysis 1.1.0 Bonuck 2014a

CD001141 Analysis 1.1.0 Bonuck 2014a

CD001141 Analysis 1.1.0 Bonuck 2014b

CD001141 Analysis 1.1.0 Bortolini 2012

CD001141 Analysis 1.1.0 Brent 1995

CD001141 Analysis 1.1.0 Bunik 2010

CD001141 Analysis 1.1.0 Chapman 2004

CD001141 Analysis 1.1.0 Coutinho 2005

CD001141 Analysis 1.1.0 Dennis 2002

CD001141 Analysis 1.1.0 Di Meglio 2010

CD001141 Analysis 1.1.0 Di Napoli 2004

CD001141 Analysis 1.1.0 Elliott-Rudder 2014

CD001141 Analysis 1.1.0 Frank 1987

CD001141 Analysis 1.1.0 Froozani 1999

CD001141 Analysis 1.1.0 Gagnon 2002

CD001141 Analysis 1.1.0 Graffy 2004

CD001141 Analysis 1.1.0 Kools 2005

CD001141 Analysis 1.1.0 Kramer 2001

CD001141 Analysis 1.1.0 Laliberte 2016

CD001141 Analysis 1.1.0 Leite 2005

CD001141 Analysis 1.1.0 Lynch 1986

CD001141 Analysis 1.1.0 McDonald 2010

Page 39 of 49


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

Confidential: For Review OnlyCD001141 Analysis 1.1.0 McLachlan 2016

CD001141 Analysis 1.1.0 McLachlan 2016

CD001141 Analysis 1.1.0 McQueen 2011

CD001141 Analysis 1.1.0 Mejdoubi 2014

CD001141 Analysis 1.1.0 Morrell 2000

CD001141 Analysis 1.1.0 Morrow 1999

CD001141 Analysis 1.1.0 Muirhead 2006

CD001141 Analysis 1.1.0 Paul 2012

CD001141 Analysis 1.1.0 Porteous 2000

CD001141 Analysis 1.1.0 Pugh 1998



CD001141 Analysis 1.1.0 Quinlivan 2003

CD001141 Analysis 1.1.0 Serafino-Cross 1992

CD001141 Analysis 1.1.0 Sjolin 1979

CD001141 Analysis 1.1.0 Su 2007

CD001141 Analysis 1.1.0 Tahir 2013

CD001141 Analysis 1.1.0 Vidas 2011

CD001141 Analysis 1.1.0 Vitolo 2005

CD001141 Analysis 1.1.0 Wilhelm 2015

CD001141 Analysis 1.1.0 Winterburn 2003

CD001877 Analysis 1.1.0 Kopparberg 1977

CD001877 Analysis 1.1.0 New York 1963

CD001877 Analysis 1.1.0 UK age trial 1991

CD001877 Analysis 1.1.0 Östergötland 1978

CD001886 Analysis 1.1.0 Samama 2002

CD001886 Analysis 1.1.0 Cicek 1996a

CD001886 Analysis 1.1.0 Cicekcioglu 2006

CD001886 Analysis 1.1.0 Capdevila 1998

CD001886 Analysis 1.1.0 Lentschener 1999

CD001886 Analysis 1.1.0 Cicek 1996b

CD001886 Analysis 1.1.0 Garcia-Huete 1997

CD001886 Analysis 1.1.0 Kyriss 2001

CD001886 Analysis 1.1.0 Amar 2003

CD001886 Analysis 1.1.0 Swart 1994

CD001886 Analysis 1.1.0 Parvizi 2007

CD001886 Analysis 1.1.0 Lentschener 1997

CD001886 Analysis 1.1.0 Carrera 1994

CD001886 Analysis 1.1.0 Royston 1987

CD001886 Analysis 1.1.0 Palmer 2003

CD001886 Analysis 1.1.0 Ray 1997

CD001886 Analysis 1.1.0 Casas 1995

CD001886 Analysis 1.1.0 Dietrich 1990

CD001886 Analysis 1.1.0 Rossi 1997

Page 40 of 49


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

Confidential: For Review OnlyCD001886 Analysis 1.1.0 Mansour 2004

CD001886 Analysis 1.1.0 Rocha 1994

CD001886 Analysis 1.1.0 Bidstrup 1989

CD001886 Analysis 1.1.0 Vedrinne 1992

CD001886 Analysis 1.1.0 Boldt 1991

CD001886 Analysis 1.1.0 Katzel 1998

CD001886 Analysis 1.1.0 Menichetti 1996


CD001886 Analysis 1.1.0 Kalangos 1994


CD001886 Analysis 1.1.0 Hardy 1997

CD001886 Analysis 1.1.0 Petsatodis 2006

CD001886 Analysis 1.1.0 Santamaria 2000

CD001886 Analysis 1.1.0 Fraedrich 1989


CD001886 Analysis 1.1.0 D'Ambra 1996

CD001886 Analysis 1.1.0 Rodrigus 1996

CD001886 Analysis 1.1.0 Speekenbrink 1995

CD001886 Analysis 1.1.0 Gherli 1992

CD001886 Analysis 1.1.0 Hardy 1993

CD001886 Analysis 1.1.0 Speekenbrink 1996

CD001886 Analysis 1.1.0 Li 2005

CD001886 Analysis 1.1.0 Taggart 2003

CD001886 Analysis 1.1.0 Alajmo 1989

CD001886 Analysis 1.1.0 Tabuchi 1994

CD001886 Analysis 1.1.0 Basora 1999

CD001886 Analysis 1.1.0 Deleuze 1991

CD001886 Analysis 1.1.0 Englberger 2002b

CD001886 Analysis 1.1.0 Mohr 1992

CD001886 Analysis 1.1.0 Wendel 1995

CD001886 Analysis 1.1.0 Wei 2006

CD001886 Analysis 1.1.0 Cosgrove 1992

CD001886 Analysis 1.1.0 Bailey 1994

CD001886 Analysis 1.1.0 Harmon 2004

CD001886 Analysis 1.1.0 Kipfer 2003

CD001886 Analysis 1.1.0 Cohen 1998

CD001886 Analysis 1.1.0 Lemmer 1996

CD001886 Analysis 1.1.0 Alderman 1998


CD001886 Analysis 1.1.0 Baele 1992

CD001886 Analysis 1.1.0 Penta de Peppo 1995



CD001886 Analysis 1.1.0 Green 1995

Page 41 of 49


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

Confidential: For Review OnlyCD001886 Analysis 1.1.0 Hayashida 1997

CD001886 Analysis 1.1.0 Greilich 2001

CD001886 Analysis 1.1.0 Alvarez 1995

CD001886 Analysis 1.1.0 Greilich 2009

CD001886 Analysis 1.1.0 Harder 1991

CD001886 Analysis 1.1.0 Kuitunen 2005

CD001886 Analysis 1.1.0 Lemmer_1 1994

CD001886 Analysis 1.1.0 Klein 1998

CD001886 Analysis 1.1.0 Blauhut 1994

CD001886 Analysis 1.1.0 Maccario 1994

CD001886 Analysis 1.1.0 Murkin 1994

CD001886 Analysis 1.1.0 D'Ambrosio 1999

CD001886 Analysis 1.1.0 Nurözler 2008

CD001886 Analysis 1.1.0 Colwell 2007


CD001886 Analysis 1.1.0 Diprose 2005

CD001886 Analysis 1.1.0 Dignan 2001

CD001886 Analysis 1.1.0 Corbeau 1995

CD001886 Analysis 1.1.0 Laub 1994

CD001886 Analysis 1.1.0 Levy 1995

CD001886 Analysis 1.1.0 Later 2009

CD001886 Analysis 1.1.0 Englberger 2002a

CD001886 Analysis 1.1.0 Lemmer_2 1994

CD001886 Analysis 1.1.0 Liu 1993

CD001886 Analysis 1.1.0 Okita 1996


CD001886 Analysis 1.1.0 Van der Linden 2005

CD001886 Analysis 1.1.0 Llau 1998

CD001886 Analysis 1.1.0 Vanek 2005

CD001886 Analysis 1.1.0 Ranaboldo 1997

CD001886 Analysis 1.1.0 Engel 2001

CD001886 Analysis 1.1.0 Jeserschek 2003


CD001886 Analysis 1.1.0 Cvachovec 2001

CD001886 Analysis 1.1.0 Desai 2009

CD001886 Analysis 1.1.0 Havel 1992

CD001886 Analysis 1.1.0 Isetta 1993

CD001886 Analysis 1.1.0 Lass 1995

CD001886 Analysis 1.1.0 Locatelli 1990

CD001886 Analysis 1.1.0 Poston 2006


CD001886 Analysis 1.1.0 Stewart 2001

CD001886 Analysis 1.1.0 Tassani 2000

CD001886 Analysis 1.1.0 Thorpe 1994

Page 42 of 49


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

Confidential: For Review OnlyCD002834 Analysis 1.1.1 Binstock 1997

CD002834 Analysis 1.1.1 Bowman 1995

CD002834 Analysis 1.1.1 Del Mar 1998

CD002834 Analysis 1.1.1 Hunt 1998

CD002834 Analysis 1.1.1 Lancaster 1992

CD002834 Analysis 1.1.1 Pierce 1989

CD002990 Analysis 1.21.0 van Wetering 2009

CD002990 Analysis 1.21.0 Casas 2006

CD002990 Analysis 1.21.0 Monninkhof 2003

CD002990 Analysis 1.21.0 Khdour 2009

CD002990 Analysis 1.21.0 Rea 2004

CD002990 Analysis 1.21.0 Coultas 2005a

CD002990 Analysis 1.21.0 Coultas 2005b

CD002990 Analysis 1.21.0 Rice 2010

CD003331 Analysis 1.4.0 Belardinelli 1999

CD003331 Analysis 1.4.0 Mueller 2007

CD003331 Analysis 1.4.0 Belardinelli 2012

CD003331 Analysis 1.4.0 Cowie 2014a

CD003331 Analysis 1.4.0 Cowie 2014b

CD003331 Analysis 1.4.0 Jónsdóttir 2006a

CD003587 Analysis 2.1.4 Berry 1998

CD003587 Analysis 2.1.4 Engoren 1998

CD003587 Analysis 2.1.4 Michalopoulos 1998

CD003587 Analysis 2.1.4 Pettersson 2004

CD003587 Analysis 2.1.4 Probst 2014

CD003587 Analysis 2.1.4 Reyes 1997

CD003587 Analysis 2.1.4 Zhu 2015

CD003855 Analysis 2.13.0 Istre 1998

CD003855 Analysis 2.13.0 Crosignani 1997

CD003855 Analysis 2.13.0 Soysal 2002

CD003855 Analysis 2.13.0 Malak 2006

CD003855 Analysis 2.13.0 Ergun 2012

CD003941 Analysis 1.1.0 Mullooly1987T67

CD003941 Analysis 1.1.0 Alto1994T54

CD003941 Analysis 1.1.0 Bangure 2015

CD003941 Analysis 1.1.0 Brimberry1988T33

CD003941 Analysis 1.1.0 Buchner1987T34

CD003941 Analysis 1.1.0 Campbell1994T87

CD003941 Analysis 1.1.0 Carter1986T104

CD003941 Analysis 1.1.0 Daley2004b

CD003941 Analysis 1.1.0 Ferson1995T57

CD003941 Analysis 1.1.0 Hogg1998T101

CD003941 Analysis 1.1.0 Hogg1998T101

CD003941 Analysis 1.1.0 Hull2002T511

Page 43 of 49


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

Confidential: For Review OnlyCD003941 Analysis 1.1.0 Kemper1993T11

CD003941 Analysis 1.1.0 Larson1982T39

CD003941 Analysis 1.1.0 Moran1992T16

CD003941 Analysis 1.1.0 Nexoe1997T92

CD003941 Analysis 1.1.0 Oeffinger1992T27

CD003941 Analysis 1.1.0 Sansom2003T514

CD003941 Analysis 1.1.0 Satterthwaite1997T93

CD003941 Analysis 1.1.0 Siebers1985T36

CD003941 Analysis 1.1.0 Siebers1985T36

CD003941 Analysis 1.1.0 Stehr-Green1993T10

CD003941 Analysis 1.1.0 Szilagyi1992T15

CD003941 Analysis 1.1.0 Szilagyi2006T718

CD003941 Analysis 1.1.0 Tollestrup1991T18

CD003941 Analysis 1.1.0 Wood1998T105

CD003941 Analysis 1.1.0 Young1980T63

CD004815 Analysis 1.13.0 ICTUS

CD004815 Analysis 1.13.0 RITA-3

CD005465 Analysis 4.2.0 Flicker 2005

CD005465 Analysis 4.2.0 Bischoff 2003

CD005465 Analysis 4.2.0 Chapuy 2002

CD005465 Analysis 4.2.0 Grieger 2009

CD006525 Analysis 1.3.1 Rojas 2007

CD006525 Analysis 1.3.1 Fritsch 2007

CD006525 Analysis 1.3.1 Capoccia 2004

CD006525 Analysis 1.3.1 Adler 2004


CD006525 Analysis 1.3.1 Araya 2003

CD006525 Analysis 1.3.1 Smit 2006a

CD006525 Analysis 1.3.1 Smit 2006b

CD006525 Analysis 1.3.1 Smit 2006c

CD006525 Analysis 1.3.1 Wells 2000a

CD006525 Analysis 1.3.1 Wells 2000b

CD006525 Analysis 1.3.1 Katzelnick 2000

CD006525 Analysis 1.3.1 Mann 1998

CD006525 Analysis 1.3.1 Wilkinson 1993

CD006525 Analysis 1.3.1 Dwight-Johnson 2010

CD006525 Analysis 1.3.1 Pyne 2011

CD006525 Analysis 1.3.1 Finley 2003

CD006525 Analysis 1.3.1 Vera 2010

CD006525 Analysis 1.3.1 Strong 2008

CD006525 Analysis 1.3.1 Bogner 2008

CD006525 Analysis 1.3.1 Ross 2008

CD006525 Analysis 1.3.1 Bogner 2010

CD006525 Analysis 1.3.1 Huffman 2011

Page 44 of 49


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

Confidential: For Review OnlyCD006525 Analysis 1.3.1 Rollman 2009

CD006525 Analysis 1.3.1 Unutzer 2002

CD006525 Analysis 1.3.1 Ciechanowski 2004

CD006525 Analysis 1.3.1 McCusker 2008

CD006525 Analysis 1.3.1 Blanchard 1995

CD006525 Analysis 1.3.1 Ell 2007

CD006525 Analysis 1.3.1 Bruce 2004

CD006525 Analysis 1.3.1 McMahon 2007

CD007176 Analysis 1.1.0 Murphy 1992Low

CD007176 Analysis 1.1.0 Mooney 2005Low

CD007176 Analysis 1.1.0 Jacobson 2000Low

CD007176 Analysis 1.1.0 Tam 2005Low

CD007176 Analysis 1.1.0 Limburg 2005Low

CD007176 Analysis 1.1.0 Mezey 2004Low

CD007176 Analysis 1.1.0 NSCPT 1999Low

CD007176 Analysis 1.1.0 SUVIMAX 2010Low

CD007176 Analysis 1.1.0 de la Maza 1995

CD007176 Analysis 1.1.0 Correa 2000Low

CD007176 Analysis 1.1.0 DATOR 2004Low

CD007176 Analysis 1.1.0 HATS 2001Low

CD007176 Analysis 1.1.0 PHS 1996Low

CD007176 Analysis 1.1.0 NIT2 1993Low

CD007176 Analysis 1.1.0 WHS 2005Low

CD007176 Analysis 1.1.0 VEAPS 2002Low

CD007176 Analysis 1.1.0 CARET 2004Low

CD007176 Analysis 1.1.0 Sasazuki 2003

CD007176 Analysis 1.1.0 Stevic 2001

CD007176 Analysis 1.1.0 Graf 2005Low

CD007176 Analysis 1.1.0 McKeown-Eyssen 1988

CD007176 Analysis 1.1.0 Prince 2003Low

CD007176 Analysis 1.1.0 GISSI 1999

CD007176 Analysis 1.1.0 de Waart 2001

CD007176 Analysis 1.1.0 DATATOP 2005Low

CD007176 Analysis 1.1.0 PPS 1994Low

CD007176 Analysis 1.1.0 WACS 2007Low

CD007176 Analysis 1.1.0 CHAOS 1996Low

CD007176 Analysis 1.1.0 SELECT 2009Low

CD007176 Analysis 1.1.0 NPCT 1996Low

CD007176 Analysis 1.1.0 SKICAP AK 1997Low

CD007176 Analysis 1.1.0 Takagi 2003

CD007176 Analysis 1.1.0 White 2002Low

CD007176 Analysis 1.1.0 ALSRT 2001Low

CD007176 Analysis 1.1.0 MAVET 2006Low

CD007176 Analysis 1.1.0 PHS 2008Low

Page 45 of 49


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

Confidential: For Review OnlyCD007176 Analysis 1.1.0 PPP 2001

CD007176 Analysis 1.1.0 Wluka 2002Low

CD007176 Analysis 1.1.0 Garbagnati 2009Low

CD007176 Analysis 1.1.0 SPACE 2000Low

CD007176 Analysis 1.1.0 WAVE 2002Low

CD007176 Analysis 1.1.0 HOPE TOO 2005Low

CD007176 Analysis 1.1.0 VECAT 2004Low

CD007176 Analysis 1.1.0 Collins 2003Low

CD007176 Analysis 1.1.0 UK PRECISE 2006Low

CD007176 Analysis 1.1.0 AREDS 2001Low

CD007176 Analysis 1.1.0 CTNS 2008

CD007176 Analysis 1.1.0 REACT 2002Low

CD007176 Analysis 1.1.0 ICARE 2008Low

CD007176 Analysis 1.1.0 Pike 1995Low

CD007176 Analysis 1.1.0 AMDS 1996Low

CD007176 Analysis 1.1.0 ADCS 1 1997

CD007176 Analysis 1.1.0 ADCS 2 2005

CD007176 Analysis 1.1.0 Chandra 1992

CD007176 Analysis 1.1.0 LAST 2004Low

CD007176 Analysis 1.1.0 Burns 1989

CD007176 Analysis 1.1.0 Allsup 2004Low

CD007176 Analysis 1.1.0 Hogarth 1996

CD007176 Analysis 1.1.0 Girodon 1997

CD007176 Analysis 1.1.0 Meydani 2004Low

CD007176 Analysis 1.1.0 MINVITAOX 1999Low

CD007176 Analysis 1.1.0 Penn 1991

CD007176 Analysis 1.1.0 Liu 2007Low

CD007176 Analysis 1.1.0 ASAP 2003Low

CD007176 Analysis 1.1.0 Bonelli 1998

CD007176 Analysis 1.1.0 Gillilan 1977

CD007176 Analysis 1.1.0 Graat 2002Low

CD007176 Analysis 1.1.0 Grieger 2009Low

CD007176 Analysis 1.1.0 HPS 2002Low

CD007176 Analysis 1.1.0 NIT1 1993

CD007176 Analysis 1.1.0 Plummer 2007Low

CD007176 Analysis 1.1.0 SCPS 1990Low

CD007176 Analysis 1.1.0 SIT 2006

CD007176 Analysis 1.1.0 Takamatsu 1995

CD007176 Analysis 1.1.0 ter Riet 1995

CD007176 Analysis 1.1.0 Witte 2005Low

CD007224 Analysis 1.1.0 Vural 1995

CD007224 Analysis 1.1.0 Guler 2001

CD007224 Analysis 1.1.0 Kochamba 2000

CD007224 Analysis 1.1.0 Rachwalik 2006

Page 46 of 49


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

Confidential: For Review OnlyCD007224 Analysis 1.1.0 Gerola 2004

CD007224 Analysis 1.1.0 Ozkara 2007

CD007224 Analysis 1.1.0 Penttila 2001

CD007224 Analysis 1.1.0 JOCRI 2005

CD007224 Analysis 1.1.0 MASS III 2009

CD007224 Analysis 1.1.0 Matata 2000

CD007224 Analysis 1.1.0 Sajja 2007

CD007224 Analysis 1.1.0 Gu 1998

CD007224 Analysis 1.1.0 Medved 2008

CD007224 Analysis 1.1.0 OCTOPUS 2001

CD007224 Analysis 1.1.0 Parolari 2003

CD007224 Analysis 1.1.0 Selvanayagam 2004

CD007224 Analysis 1.1.0 Ascione 2005

CD007224 Analysis 1.1.0 Blacher 2005

CD007224 Analysis 1.1.0 Fattouch 2009

CD007224 Analysis 1.1.0 SMART 2003

CD007224 Analysis 1.1.0 Synnergren 2004

CD007224 Analysis 1.1.0 Velissaris 2003

CD007224 Analysis 1.1.0 Al-Ruzzeh 2006

CD007224 Analysis 1.1.0 Ascione 2006

CD007224 Analysis 1.1.0 BHACAS I +II 2002

CD007224 Analysis 1.1.0 Caputo 2002

CD007224 Analysis 1.1.0 Czerny 2001

CD007224 Analysis 1.1.0 Gasz 2004

CD007224 Analysis 1.1.0 Khan 2004

CD007224 Analysis 1.1.0 Legare 2004

CD007224 Analysis 1.1.0 Michaux 2006

CD007224 Analysis 1.1.0 Quaniers 2006

CD007224 Analysis 1.1.0 ROOBY 2009

CD007224 Analysis 1.1.0 Sahlman 2003

CD007224 Analysis 1.1.0 Tang 2002

CD007224 Analysis 1.1.0 Alwan 2004

CD007224 Analysis 1.1.0 Diegeler 2000

CD007224 Analysis 1.1.0 Jares 2007

CD007224 Analysis 1.1.0 Motallebzadeh 2006

CD007224 Analysis 1.1.0 Raja 2003

CD007224 Analysis 1.1.0 Baker 2001

CD007224 Analysis 1.1.0 Formica 2009

CD007224 Analysis 1.1.0 Lingaas 2004

CD007224 Analysis 1.1.0 Mariscalco 2006

CD007224 Analysis 1.1.0 Modine 2010

CD007224 Analysis 1.1.0 PROMISS 2010

CD007224 Analysis 1.1.0 Tatoulis 2006

CD007224 Analysis 1.1.0 Vedin 2003

Page 47 of 49


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

Confidential: For Review OnlyCD007224 Analysis 1.1.0 Lee 2003

CD007224 Analysis 1.1.0 Mazzei 2007

CD007224 Analysis 1.1.0 PRAGUE-11 2008

CD007224 Analysis 1.1.0 Wandschneider 2000

CD007224 Analysis 1.1.0 Mandak 2008

CD007224 Analysis 1.1.0 Mantovani 2010

CD007224 Analysis 1.1.0 Muneretto 2003

CD007224 Analysis 1.1.0 Niranjan 2006

CD007224 Analysis 1.1.0 Parolari 2007

CD007224 Analysis 1.1.0 Malik 2006

CD007224 Analysis 1.1.0 Nesher 2006

CD007224 Analysis 1.1.0 Rasmussen 2007

CD007224 Analysis 1.1.0 Kunes 2007

CD007224 Analysis 1.1.0 DOORS 2009

CD007224 Analysis 1.1.0 Carrier 2003

CD007224 Analysis 1.1.0 Cavalca 2006

CD007224 Analysis 1.1.0 Covino 2001

CD007224 Analysis 1.1.0 Czerny 2000

CD007224 Analysis 1.1.0 Gasz 2005

CD007224 Analysis 1.1.0 Gönenc 2006

CD007224 Analysis 1.1.0 Gulielmos 1999

CD007224 Analysis 1.1.0 Hernandez 2007

CD007224 Analysis 1.1.0 Paparella 2006

CD007224 Analysis 1.1.0 PRAGUE-4 2004

CD007224 Analysis 1.1.0 Zamvar 2002

CD007228 Analysis 1.2.0 Seto 2012

CD007228 Analysis 1.2.0 Giordano 2009

CD007228 Analysis 1.2.0 Goldberg 2003 (WHARF)

CD007228 Analysis 1.2.0 Mortara 2009 (Telemon) (HHH)

CD007228 Analysis 1.2.0 Balk 2008

CD007228 Analysis 1.2.0 Cleland 2005 (Telemon) (TENS-HMS)

CD007228 Analysis 1.2.0 Woodend 2008

CD007228 Analysis 1.2.0 Villani 2014 (ICAROS)

CD007228 Analysis 1.2.0 De Lusignan 2001

CD007228 Analysis 1.2.0 Dendale 2012 (TEMA-HF1)

CD007228 Analysis 1.2.0 Soran 2008

CD007228 Analysis 1.2.0 Biannic 2012 (SEDIC)

CD007228 Analysis 1.2.0 Antonicelli 2008

CD007228 Analysis 1.2.0 Vuorinen 2014

CD007746 Analysis 1.3.3 Tuncer 2002

CD007746 Analysis 1.3.3 Joh 2005

CD007746 Analysis 1.3.3 MYSS Study 2004

CD007784 Analysis 1.2.0 PREVEND IT 2000

CD007784 Analysis 1.2.0 MEGA Study 2004

Page 48 of 49


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

Confidential: For Review OnlyCD007784 Analysis 1.2.0 ALLIANCE Study 2000

CD007784 Analysis 1.2.0 CARDS 2003

CD007784 Analysis 1.2.0 LIPS Study 2005

CD007784 Analysis 1.2.0 JUPITER Study 2007

CD007784 Analysis 1.2.0 Verma 2005

CD007784 Analysis 1.2.0 Rayner 1996

CD008345 Analysis 1.5.2 Chien - Hong Kong 2001

CD008345 Analysis 1.5.2 Chien- Hong Kong 2008

CD008345 Analysis 1.5.2 Callahan - Indianapolis

CD008986 Analysis 1.1.0 Farris 2014

CD008986 Analysis 1.1.0 Scullin 2007

CD008986 Analysis 1.1.0 Gallagher 2011

CD008986 Analysis 1.1.0 Bonnerup 2014

CD008986 Analysis 1.1.0 Lisby 2015

CD008986 Analysis 1.1.0 Bladh 2011

CD008986 Analysis 1.1.0 Dalleur 2014

CD008986 Analysis 1.1.0 Gillespie 2009

CD009217 Analysis 1.1.0 HPT 1990

CD009217 Analysis 1.1.0 TOHP I 1992

CD009217 Analysis 1.1.0 TOHP II 1997

CD009217 Analysis 1.1.0 Morgan 1978

CD009217 Analysis 1.1.0 CSSS 2007

CD009217 Analysis 1.1.0 Kwok 2012

CD010085 Analysis 6.2.1 Makrides 2010

CD010085 Analysis 6.2.1 Makrides 2009

* Very few Trial IDs may have been slightly renamed in Cochrane Review updates.

Page 49 of 49


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

confidential: for review only...2020/02/09 · confidential: for review only agreement of treatment...

Documents