Contour enhanced funnel plots for meta-analysis

Tom Palmer1, Jaime Peters2, Alex Sutton3, Santiago Moreno3

1. MRC Centre for Causal Analyses in Translational Epidemiology, Department of Social Medicine,University of Bristol

2. PenTAG/PenCLAHRC, Peninsula Medical School3. Department of Health Sciences, University of Leicester

11 September 2009

Centre for Causal

Analyses in Translational

Epidemiology

Outline

I Introduction to funnel plots & contour enhanced funnel plots

I Moreno, Sutton, Turner, et al., 2009 BMJ example- Use with other bias assessment methods

I confunnel: syntax and options

I Discussion

1 / 13

Introduction to funnel plots

0.5

11.

5S

tand

ard

erro

r of

log(

OR

)

.05 .1 .25 .5 1 2 4 8 16Odds ratio

Funnel plot with pseudo 95% confidence limits

I Plot of std error (y -axis) versus effect estimate (x-axis)I Help assess small study reporting bias/publication biasI Sterne & Harbord, 2004; metafunnel, funnelI Same metric as Egger’s test (Egger, Davey Smith, Schneider,

& Minder, 1997)2 / 13

Introduction to contour enhanced funnel plots

I Indicate regions of statistical significance on funnel plot

I Spiegelhalter, 2002, 2005; Peters, Sutton, Jones, Abrams, &Rushton, 2008

0.5

11.

5S

tand

ard

erro

r of

log(

OR

)

.05 .1 .25 .5 1 2 4 8 16Odds ratio

Funnel plot with pseudo 95% confidence limits 0

.5

1

1.5

Sta

ndar

d er

ror

−4 −2 0 2 4Effect estimate

Studies

p < 1%

1% < p < 5%

5% < p < 10%

p > 10%

3 / 13

Moreno, Sutton, Turner, et al., 2009, BMJ, exampleI Re-analysis of Turner, Matthews, Linardatos, Tell, & Rosenthal, 2008, NEJMI Results of 74 trials of 12 antidepressant drugsI Compare FDA results versus journal results

T h e n e w e ng l a nd j o u r na l o f m e dic i n e

n engl j med 358;3 www.nejm.org january 17, 2008256

increase of 32%. A 32% increase was also ob-served in the weighted mean effect size for all drugs combined, from 0.31 (95% CI, 0.27 to 0.35) to 0.41 (95% CI, 0.36 to 0.45).

A list of the study-level effect-size values used in the above analyses — derived from both the FDA reviews and the published reports — is pro-vided in Table C of the Supplementary Appendix. These effect-size values are based on P values and sample sizes shown in Table A of the Supplemen-tary Appendix, which also lists reference infor-mation for the publications consulted.

Discussion

We found a bias toward the publication of posi-tive results. Not only were positive results more likely to be published, but studies that were not positive, in our opinion, were often published in a way that conveyed a positive outcome. We ana-lyzed these data in terms of the proportion of positive studies and in terms of the effect size associated with drug treatment. Using both ap-proaches, we found that the efficacy of this drug class is less than would be gleaned from an ex-amination of the published literature alone. Ac-cording to the published literature, the results of nearly all of the trials of antidepressants were positive. In contrast, FDA analysis of the trial data showed that roughly half of the trials had positive results. The statistical significance of a study’s re-sults was strongly associated with whether and how they were reported, and the association was independent of sample size. The study outcome also affected the chances that the data from a par-ticipant would be published. As a result of selec-

16p6

Published, agrees with FDA decisionPublished, conflicts with FDA decisionNot published

Positive(N=38)

Questionable(N=12)

Negative(N=24)

Positive(N=7155)

Questionable(N=2309)

Negative(N=3100)

0 10 20 30 40

0 2000 4000 6000

AUTHOR:

FIGURE:

JOB:

4-CH/T

RETAKEICM

CASE

EMail LineH/TCombo

Revised

REG F

Enon

1st2nd3rd

Turner

1 of 3

01-17-08

ARTIST: ts

35803 ISSUE:

37(97%)

1(3%)

6(50%)

7075(99%)

1180(51%)

663(21%)

197(6%)

2240(72%)

1129(49%)

3(12%)

80(1%)

5(21%)

6(50%)

16(67%)

Figure!1.!Effect!of!FDA!Regulatory!Decisions!!on!Publication.

Among the 74 studies reviewed by the FDA (Panel A), 38 were deemed to have positive results, 37 of which were published with positive results; the remaining study was not published. Among the studies deemed to have questionable or negative results by the FDA, there was a tendency toward nonpublication or publi-cation with positive results, conflicting with the con-clusion of the FDA. Among the 12,564 patients in all 74 studies (Panel B), data for patients who participated in studies deemed positive by the FDA were very likely to be published in a way that agreed with the FDA. In contrast, data for patients participating in studies deemed questionable or negative by the FDA tended either not to be published or to be published in a way that conflicted with the FDA’s judgment.

Figure!2!(facing!page).!Publication!Status!and!FDA!!Regulatory!Decision!by!Study!and!by!Drug.

Panel A shows the publication status of individual studies. Nearly every study deemed positive by the FDA (top row) was published in a way that agreed with the FDA’s judgment. By contrast, most studies deemed negative (bottom row) or questionable (mid-dle row) by the FDA either were published in a way that conflicted with the FDA’s judgment or were not pub-lished. Numbers shown in boxes indicate individual studies and correspond to the study numbers listed in Table A of the Supplementary Appendix. Panel B shows the numbers of patients participating in the individual studies indicated in Panel A. Data for pa-tients who participated in studies deemed positive by the FDA were very likely to be published in a way that agreed with the FDA’s judgment. By contrast, data for patients who participated in studies deemed negative or questionable by the FDA tended either not to be published or to be published in a way that conflicted with the FDA’s judgment.

T h e n e w e ng l a nd j o u r na l o f m e dic i n e

n engl j med 358;3 www.nejm.org january 17, 2008256

increase of 32%. A 32% increase was also ob-served in the weighted mean effect size for all drugs combined, from 0.31 (95% CI, 0.27 to 0.35) to 0.41 (95% CI, 0.36 to 0.45).

A list of the study-level effect-size values used in the above analyses — derived from both the FDA reviews and the published reports — is pro-vided in Table C of the Supplementary Appendix. These effect-size values are based on P values and sample sizes shown in Table A of the Supplemen-tary Appendix, which also lists reference infor-mation for the publications consulted.

Discussion

We found a bias toward the publication of posi-tive results. Not only were positive results more likely to be published, but studies that were not positive, in our opinion, were often published in a way that conveyed a positive outcome. We ana-lyzed these data in terms of the proportion of positive studies and in terms of the effect size associated with drug treatment. Using both ap-proaches, we found that the efficacy of this drug class is less than would be gleaned from an ex-amination of the published literature alone. Ac-cording to the published literature, the results of nearly all of the trials of antidepressants were positive. In contrast, FDA analysis of the trial data showed that roughly half of the trials had positive results. The statistical significance of a study’s re-sults was strongly associated with whether and how they were reported, and the association was independent of sample size. The study outcome also affected the chances that the data from a par-ticipant would be published. As a result of selec-

16p6

Published, agrees with FDA decisionPublished, conflicts with FDA decisionNot published

Positive(N=38)

Questionable(N=12)

Negative(N=24)

Positive(N=7155)

Questionable(N=2309)

Negative(N=3100)

0 10 20 30 40

0 2000 4000 6000

AUTHOR:

FIGURE:

JOB:

4-CH/T

RETAKEICM

CASE

EMail LineH/TCombo

Revised

REG F

Enon

1st2nd3rd

Turner

1 of 3

01-17-08

ARTIST: ts

35803 ISSUE:

37(97%)

1(3%)

6(50%)

7075(99%)

1180(51%)

663(21%)

197(6%)

2240(72%)

1129(49%)

3(12%)

80(1%)

5(21%)

6(50%)

16(67%)

Figure!1.!Effect!of!FDA!Regulatory!Decisions!!on!Publication.

Among the 74 studies reviewed by the FDA (Panel A), 38 were deemed to have positive results, 37 of which were published with positive results; the remaining study was not published. Among the studies deemed to have questionable or negative results by the FDA, there was a tendency toward nonpublication or publi-cation with positive results, conflicting with the con-clusion of the FDA. Among the 12,564 patients in all 74 studies (Panel B), data for patients who participated in studies deemed positive by the FDA were very likely to be published in a way that agreed with the FDA. In contrast, data for patients participating in studies deemed questionable or negative by the FDA tended either not to be published or to be published in a way that conflicted with the FDA’s judgment.

Figure!2!(facing!page).!Publication!Status!and!FDA!!Regulatory!Decision!by!Study!and!by!Drug.

Panel A shows the publication status of individual studies. Nearly every study deemed positive by the FDA (top row) was published in a way that agreed with the FDA’s judgment. By contrast, most studies deemed negative (bottom row) or questionable (mid-dle row) by the FDA either were published in a way that conflicted with the FDA’s judgment or were not pub-lished. Numbers shown in boxes indicate individual studies and correspond to the study numbers listed in Table A of the Supplementary Appendix. Panel B shows the numbers of patients participating in the individual studies indicated in Panel A. Data for pa-tients who participated in studies deemed positive by the FDA were very likely to be published in a way that agreed with the FDA’s judgment. By contrast, data for patients who participated in studies deemed negative or questionable by the FDA tended either not to be published or to be published in a way that conflicted with the FDA’s judgment.

4 / 13

Moreno, Sutton, Turner, et al., 2009, BMJ, exampleI Re-analysis of Turner et al., 2008, NEJMI Results of 74 trials of 12 antidepressant drugsI Compare FDA results versus journal results

T h e n e w e ng l a nd j o u r na l o f m e dic i n e

n engl j med 358;3 www.nejm.org january 17, 2008256

increase of 32%. A 32% increase was also ob-served in the weighted mean effect size for all drugs combined, from 0.31 (95% CI, 0.27 to 0.35) to 0.41 (95% CI, 0.36 to 0.45).

A list of the study-level effect-size values used in the above analyses — derived from both the FDA reviews and the published reports — is pro-vided in Table C of the Supplementary Appendix. These effect-size values are based on P values and sample sizes shown in Table A of the Supplemen-tary Appendix, which also lists reference infor-mation for the publications consulted.

Discussion

We found a bias toward the publication of posi-tive results. Not only were positive results more likely to be published, but studies that were not positive, in our opinion, were often published in a way that conveyed a positive outcome. We ana-lyzed these data in terms of the proportion of positive studies and in terms of the effect size associated with drug treatment. Using both ap-proaches, we found that the efficacy of this drug class is less than would be gleaned from an ex-amination of the published literature alone. Ac-cording to the published literature, the results of nearly all of the trials of antidepressants were positive. In contrast, FDA analysis of the trial data showed that roughly half of the trials had positive results. The statistical significance of a study’s re-sults was strongly associated with whether and how they were reported, and the association was independent of sample size. The study outcome also affected the chances that the data from a par-ticipant would be published. As a result of selec-

16p6

Published, agrees with FDA decisionPublished, conflicts with FDA decisionNot published

Positive(N=38)

Questionable(N=12)

Negative(N=24)

Positive(N=7155)

Questionable(N=2309)

Negative(N=3100)

0 10 20 30 40

0 2000 4000 6000

AUTHOR:

FIGURE:

JOB:

4-CH/T

RETAKEICM

CASE

EMail LineH/TCombo

Revised

REG F

Enon

1st2nd3rd

Turner

1 of 3

01-17-08

ARTIST: ts

35803 ISSUE:

37(97%)

1(3%)

6(50%)

7075(99%)

1180(51%)

663(21%)

197(6%)

2240(72%)

1129(49%)

3(12%)

80(1%)

5(21%)

6(50%)

16(67%)

Figure!1.!Effect!of!FDA!Regulatory!Decisions!!on!Publication.

Among the 74 studies reviewed by the FDA (Panel A), 38 were deemed to have positive results, 37 of which were published with positive results; the remaining study was not published. Among the studies deemed to have questionable or negative results by the FDA, there was a tendency toward nonpublication or publi-cation with positive results, conflicting with the con-clusion of the FDA. Among the 12,564 patients in all 74 studies (Panel B), data for patients who participated in studies deemed positive by the FDA were very likely to be published in a way that agreed with the FDA. In contrast, data for patients participating in studies deemed questionable or negative by the FDA tended either not to be published or to be published in a way that conflicted with the FDA’s judgment.

Figure!2!(facing!page).!Publication!Status!and!FDA!!Regulatory!Decision!by!Study!and!by!Drug.

Panel A shows the publication status of individual studies. Nearly every study deemed positive by the FDA (top row) was published in a way that agreed with the FDA’s judgment. By contrast, most studies deemed negative (bottom row) or questionable (mid-dle row) by the FDA either were published in a way that conflicted with the FDA’s judgment or were not pub-lished. Numbers shown in boxes indicate individual studies and correspond to the study numbers listed in Table A of the Supplementary Appendix. Panel B shows the numbers of patients participating in the individual studies indicated in Panel A. Data for pa-tients who participated in studies deemed positive by the FDA were very likely to be published in a way that agreed with the FDA’s judgment. By contrast, data for patients who participated in studies deemed negative or questionable by the FDA tended either not to be published or to be published in a way that conflicted with the FDA’s judgment.

T h e n e w e ng l a nd j o u r na l o f m e dic i n e

n engl j med 358;3 www.nejm.org january 17, 2008256

increase of 32%. A 32% increase was also ob-served in the weighted mean effect size for all drugs combined, from 0.31 (95% CI, 0.27 to 0.35) to 0.41 (95% CI, 0.36 to 0.45).

A list of the study-level effect-size values used in the above analyses — derived from both the FDA reviews and the published reports — is pro-vided in Table C of the Supplementary Appendix. These effect-size values are based on P values and sample sizes shown in Table A of the Supplemen-tary Appendix, which also lists reference infor-mation for the publications consulted.

Discussion

We found a bias toward the publication of posi-tive results. Not only were positive results more likely to be published, but studies that were not positive, in our opinion, were often published in a way that conveyed a positive outcome. We ana-lyzed these data in terms of the proportion of positive studies and in terms of the effect size associated with drug treatment. Using both ap-proaches, we found that the efficacy of this drug class is less than would be gleaned from an ex-amination of the published literature alone. Ac-cording to the published literature, the results of nearly all of the trials of antidepressants were positive. In contrast, FDA analysis of the trial data showed that roughly half of the trials had positive results. The statistical significance of a study’s re-sults was strongly associated with whether and how they were reported, and the association was independent of sample size. The study outcome also affected the chances that the data from a par-ticipant would be published. As a result of selec-

16p6

Published, agrees with FDA decisionPublished, conflicts with FDA decisionNot published

Positive(N=38)

Questionable(N=12)

Negative(N=24)

Positive(N=7155)

Questionable(N=2309)

Negative(N=3100)

0 10 20 30 40

0 2000 4000 6000

AUTHOR:

FIGURE:

JOB:

4-CH/T

RETAKEICM

CASE

EMail LineH/TCombo

Revised

REG F

Enon

1st2nd3rd

Turner

1 of 3

01-17-08

ARTIST: ts

35803 ISSUE:

37(97%)

1(3%)

6(50%)

7075(99%)

1180(51%)

663(21%)

197(6%)

2240(72%)

1129(49%)

3(12%)

80(1%)

5(21%)

6(50%)

16(67%)

Figure!1.!Effect!of!FDA!Regulatory!Decisions!!on!Publication.

Among the 74 studies reviewed by the FDA (Panel A), 38 were deemed to have positive results, 37 of which were published with positive results; the remaining study was not published. Among the studies deemed to have questionable or negative results by the FDA, there was a tendency toward nonpublication or publi-cation with positive results, conflicting with the con-clusion of the FDA. Among the 12,564 patients in all 74 studies (Panel B), data for patients who participated in studies deemed positive by the FDA were very likely to be published in a way that agreed with the FDA. In contrast, data for patients participating in studies deemed questionable or negative by the FDA tended either not to be published or to be published in a way that conflicted with the FDA’s judgment.

Figure!2!(facing!page).!Publication!Status!and!FDA!!Regulatory!Decision!by!Study!and!by!Drug.

Panel A shows the publication status of individual studies. Nearly every study deemed positive by the FDA (top row) was published in a way that agreed with the FDA’s judgment. By contrast, most studies deemed negative (bottom row) or questionable (mid-dle row) by the FDA either were published in a way that conflicted with the FDA’s judgment or were not pub-lished. Numbers shown in boxes indicate individual studies and correspond to the study numbers listed in Table A of the Supplementary Appendix. Panel B shows the numbers of patients participating in the individual studies indicated in Panel A. Data for pa-tients who participated in studies deemed positive by the FDA were very likely to be published in a way that agreed with the FDA’s judgment. By contrast, data for patients who participated in studies deemed negative or questionable by the FDA tended either not to be published or to be published in a way that conflicted with the FDA’s judgment.

4 / 13

DISCUSSION

The application of two novel approaches to identifyand adjust for publication biases in a dataset derivedfroma journal publication, where a gold standard data-set exists, produced encouraging results. Firstly, detec-tion of publication biases was convincing using acontour enhanced funnel plot. Secondly, the regres-sion basedmethodproduced a corrected average effectsize, which was close to that obtained from the FDAdataset (and closer than that obtained by the trim andfill method).This assessment does, however, have limitations.

Firstly, the findings relate to a single dataset and thusare not necessarily generalisable to other examples.Specifically, all the trials were sponsored by the phar-maceutical industry and we make the assumption thatthe FDA data are completely unbiased. Furthermore,themethods under evaluationwere designedprimarilyfor the assessment of efficacy outcomes and theymightnot be appropriate for safety outcomes—for example,there may be incentives to suppress statistically signifi-cant safety outcomes (rather than non-significantones). This is an area that requires more research.Debate is ongoing about the usefulness of funnel

plots and related tests for the identification of publica-tion biases. Although their use is widely advocated237

some question their validity,27 38-41 including in this

Significance level <1%Significance level 1-5%Significance level 5-10%Significance level >10%

Stan

dard

err

or

0.4

0.2

0.1

0.0

0.3

FDA estimateFixed effect meta-analysis FDA data

Stan

dard

err

or

0.4

0.2

0.1

0.0

0.3

Journal estimate

Fixed effect meta-analysis FDA dataFixed effect meta-analysis journal data

Journal estimate

Fixed effect trim and fill

Fixed effect meta-analysis journal data

Filled studyFixed effect meta-analysis FDA data

Stan

dard

err

or

0.4

0.2

0.1

0.0

0.3

Effect estimate

Stan

dard

err

or

-1.0 -0.5 0 0.5 1.00.4

0.2

0.1

0.0

0.3

Journal estimate

Fixed effect trim and fill

A

B

C

D

Fixed effect meta-analysis journal data

Regression lineFixed effect meta-analysis FDA data

Fig 1 | Contour enhanced funnel plots (95% CI at top). (A) Studiessubmitted to Food and Drug Administration (FDA). (B) Studiespublished in journals. (C) Implementation of trim and fill methodon journal data. (D) Implementation of regression adjustmentmodel on journal data (adjusted effect at top where SE is 0)

Significance level <1%Significance level 1-5%Significance level 5-10%Significance level >10%

Stan

dard

err

or

0.4

0.2

0.1

0.0

0.3

FDA to journal change in effect

FDA estimateFixed effect meta-analysis unpublished

Stan

dard

err

or

0.4

0.2

0.1

0.0

0.3

Effect estimate

-1.0 -0.5 0 0.5 1.0

A

B

Fig 2 | Contour enhanced funnel plots displaying discrepancybetween Food and Drug Administration (FDA) data and journaldata. (A) Arrows joining effect results from same studies whenboth were available from FDA and journals. (B) Estimates ofeffect only available from FDA (not journal published studies)

RESEARCH

page 4 of 7 BMJ | ONLINE FIRST | bmj.com

DISCUSSION

The application of two novel approaches to identifyand adjust for publication biases in a dataset derivedfroma journal publication, where a gold standard data-set exists, produced encouraging results. Firstly, detec-tion of publication biases was convincing using acontour enhanced funnel plot. Secondly, the regres-sion basedmethodproduced a corrected average effectsize, which was close to that obtained from the FDAdataset (and closer than that obtained by the trim andfill method).This assessment does, however, have limitations.

Firstly, the findings relate to a single dataset and thusare not necessarily generalisable to other examples.Specifically, all the trials were sponsored by the phar-maceutical industry and we make the assumption thatthe FDA data are completely unbiased. Furthermore,themethods under evaluationwere designedprimarilyfor the assessment of efficacy outcomes and theymightnot be appropriate for safety outcomes—for example,there may be incentives to suppress statistically signifi-cant safety outcomes (rather than non-significantones). This is an area that requires more research.Debate is ongoing about the usefulness of funnel

plots and related tests for the identification of publica-tion biases. Although their use is widely advocated237

some question their validity,27 38-41 including in this

Significance level <1%Significance level 1-5%Significance level 5-10%Significance level >10%

Stan

dard

err

or0.4

0.2

0.1

0.0

0.3

FDA estimateFixed effect meta-analysis FDA data

Stan

dard

err

or

0.4

0.2

0.1

0.0

0.3

Journal estimate

Fixed effect meta-analysis FDA dataFixed effect meta-analysis journal data

Journal estimate

Fixed effect trim and fill

Fixed effect meta-analysis journal data

Filled studyFixed effect meta-analysis FDA data

Stan

dard

err

or

0.4

0.2

0.1

0.0

0.3

Effect estimate

Stan

dard

err

or

-1.0 -0.5 0 0.5 1.00.4

0.2

0.1

0.0

0.3

Journal estimate

Fixed effect trim and fill

A

B

C

D

Fixed effect meta-analysis journal data

Regression lineFixed effect meta-analysis FDA data

Fig 1 | Contour enhanced funnel plots (95% CI at top). (A) Studiessubmitted to Food and Drug Administration (FDA). (B) Studiespublished in journals. (C) Implementation of trim and fill methodon journal data. (D) Implementation of regression adjustmentmodel on journal data (adjusted effect at top where SE is 0)

Significance level <1%Significance level 1-5%Significance level 5-10%Significance level >10%

Stan

dard

err

or

0.4

0.2

0.1

0.0

0.3

FDA to journal change in effect

FDA estimateFixed effect meta-analysis unpublished

Stan

dard

err

or

0.4

0.2

0.1

0.0

0.3

Effect estimate

-1.0 -0.5 0 0.5 1.0

A

B

Fig 2 | Contour enhanced funnel plots displaying discrepancybetween Food and Drug Administration (FDA) data and journaldata. (A) Arrows joining effect results from same studies whenboth were available from FDA and journals. (B) Estimates ofeffect only available from FDA (not journal published studies)

RESEARCH

page 4 of 7 BMJ | ONLINE FIRST | bmj.com

5 / 13

I Trim & fill: Duval & Tweedie, 2000b, 2000a

DISCUSSION

The application of two novel approaches to identifyand adjust for publication biases in a dataset derivedfroma journal publication, where a gold standard data-set exists, produced encouraging results. Firstly, detec-tion of publication biases was convincing using acontour enhanced funnel plot. Secondly, the regres-sion basedmethodproduced a corrected average effectsize, which was close to that obtained from the FDAdataset (and closer than that obtained by the trim andfill method).This assessment does, however, have limitations.

Firstly, the findings relate to a single dataset and thusare not necessarily generalisable to other examples.Specifically, all the trials were sponsored by the phar-maceutical industry and we make the assumption thatthe FDA data are completely unbiased. Furthermore,themethods under evaluationwere designedprimarilyfor the assessment of efficacy outcomes and theymightnot be appropriate for safety outcomes—for example,there may be incentives to suppress statistically signifi-cant safety outcomes (rather than non-significantones). This is an area that requires more research.Debate is ongoing about the usefulness of funnel

plots and related tests for the identification of publica-tion biases. Although their use is widely advocated237

some question their validity,27 38-41 including in this

Significance level <1%Significance level 1-5%Significance level 5-10%Significance level >10%

Stan

dard

err

or

0.4

0.2

0.1

0.0

0.3

FDA estimateFixed effect meta-analysis FDA data

Stan

dard

err

or

0.4

0.2

0.1

0.0

0.3

Journal estimate

Fixed effect meta-analysis FDA dataFixed effect meta-analysis journal data

Journal estimate

Fixed effect trim and fill

Fixed effect meta-analysis journal data

Filled studyFixed effect meta-analysis FDA data

Stan

dard

err

or

0.4

0.2

0.1

0.0

0.3

Effect estimate

Stan

dard

err

or

-1.0 -0.5 0 0.5 1.00.4

0.2

0.1

0.0

0.3

Journal estimate

Fixed effect trim and fill

A

B

C

D

Fixed effect meta-analysis journal data

Regression lineFixed effect meta-analysis FDA data

Fig 1 | Contour enhanced funnel plots (95% CI at top). (A) Studiessubmitted to Food and Drug Administration (FDA). (B) Studiespublished in journals. (C) Implementation of trim and fill methodon journal data. (D) Implementation of regression adjustmentmodel on journal data (adjusted effect at top where SE is 0)

Significance level <1%Significance level 1-5%Significance level 5-10%Significance level >10%

Stan

dard

err

or

0.4

0.2

0.1

0.0

0.3

FDA to journal change in effect

FDA estimateFixed effect meta-analysis unpublished

Stan

dard

err

or

0.4

0.2

0.1

0.0

0.3

Effect estimate

-1.0 -0.5 0 0.5 1.0

A

B

Fig 2 | Contour enhanced funnel plots displaying discrepancybetween Food and Drug Administration (FDA) data and journaldata. (A) Arrows joining effect results from same studies whenboth were available from FDA and journals. (B) Estimates ofeffect only available from FDA (not journal published studies)

RESEARCH

page 4 of 7 BMJ | ONLINE FIRST | bmj.com

DISCUSSION

The application of two novel approaches to identifyand adjust for publication biases in a dataset derivedfroma journal publication, where a gold standard data-set exists, produced encouraging results. Firstly, detec-tion of publication biases was convincing using acontour enhanced funnel plot. Secondly, the regres-sion basedmethodproduced a corrected average effectsize, which was close to that obtained from the FDAdataset (and closer than that obtained by the trim andfill method).This assessment does, however, have limitations.

Firstly, the findings relate to a single dataset and thusare not necessarily generalisable to other examples.Specifically, all the trials were sponsored by the phar-maceutical industry and we make the assumption thatthe FDA data are completely unbiased. Furthermore,themethods under evaluationwere designedprimarilyfor the assessment of efficacy outcomes and theymightnot be appropriate for safety outcomes—for example,there may be incentives to suppress statistically signifi-cant safety outcomes (rather than non-significantones). This is an area that requires more research.Debate is ongoing about the usefulness of funnel

plots and related tests for the identification of publica-tion biases. Although their use is widely advocated237

some question their validity,27 38-41 including in this

Significance level <1%Significance level 1-5%Significance level 5-10%Significance level >10%

Stan

dard

err

or

0.4

0.2

0.1

0.0

0.3

FDA estimateFixed effect meta-analysis FDA data

Stan

dard

err

or0.4

0.2

0.1

0.0

0.3

Journal estimate

Fixed effect meta-analysis FDA dataFixed effect meta-analysis journal data

Journal estimate

Fixed effect trim and fill

Fixed effect meta-analysis journal data

Filled studyFixed effect meta-analysis FDA data

Stan

dard

err

or

0.4

0.2

0.1

0.0

0.3

Effect estimate

Stan

dard

err

or

-1.0 -0.5 0 0.5 1.00.4

0.2

0.1

0.0

0.3

Journal estimate

Fixed effect trim and fill

A

B

C

D

Fixed effect meta-analysis journal data

Regression lineFixed effect meta-analysis FDA data

Fig 1 | Contour enhanced funnel plots (95% CI at top). (A) Studiessubmitted to Food and Drug Administration (FDA). (B) Studiespublished in journals. (C) Implementation of trim and fill methodon journal data. (D) Implementation of regression adjustmentmodel on journal data (adjusted effect at top where SE is 0)

Significance level <1%Significance level 1-5%Significance level 5-10%Significance level >10%

Stan

dard

err

or

0.4

0.2

0.1

0.0

0.3

FDA to journal change in effect

FDA estimateFixed effect meta-analysis unpublished

Stan

dard

err

or

0.4

0.2

0.1

0.0

0.3

Effect estimate

-1.0 -0.5 0 0.5 1.0

A

B

Fig 2 | Contour enhanced funnel plots displaying discrepancybetween Food and Drug Administration (FDA) data and journaldata. (A) Arrows joining effect results from same studies whenboth were available from FDA and journals. (B) Estimates ofeffect only available from FDA (not journal published studies)

RESEARCH

page 4 of 7 BMJ | ONLINE FIRST | bmj.com

6 / 13

DISCUSSION

The application of two novel approaches to identifyand adjust for publication biases in a dataset derivedfroma journal publication, where a gold standard data-set exists, produced encouraging results. Firstly, detec-tion of publication biases was convincing using acontour enhanced funnel plot. Secondly, the regres-sion basedmethodproduced a corrected average effectsize, which was close to that obtained from the FDAdataset (and closer than that obtained by the trim andfill method).This assessment does, however, have limitations.

Firstly, the findings relate to a single dataset and thusare not necessarily generalisable to other examples.Specifically, all the trials were sponsored by the phar-maceutical industry and we make the assumption thatthe FDA data are completely unbiased. Furthermore,themethods under evaluationwere designedprimarilyfor the assessment of efficacy outcomes and theymightnot be appropriate for safety outcomes—for example,there may be incentives to suppress statistically signifi-cant safety outcomes (rather than non-significantones). This is an area that requires more research.Debate is ongoing about the usefulness of funnel

plots and related tests for the identification of publica-tion biases. Although their use is widely advocated237

some question their validity,27 38-41 including in this

Significance level <1%Significance level 1-5%Significance level 5-10%Significance level >10%

Stan

dard

err

or

0.4

0.2

0.1

0.0

0.3

FDA estimateFixed effect meta-analysis FDA data

Stan

dard

err

or

0.4

0.2

0.1

0.0

0.3

Journal estimate

Fixed effect meta-analysis FDA dataFixed effect meta-analysis journal data

Journal estimate

Fixed effect trim and fill

Fixed effect meta-analysis journal data

Filled studyFixed effect meta-analysis FDA data

Stan

dard

err

or

0.4

0.2

0.1

0.0

0.3

Effect estimate

Stan

dard

err

or

-1.0 -0.5 0 0.5 1.00.4

0.2

0.1

0.0

0.3

Journal estimate

Fixed effect trim and fill

A

B

C

D

Fixed effect meta-analysis journal data

Regression lineFixed effect meta-analysis FDA data

Fig 1 | Contour enhanced funnel plots (95% CI at top). (A) Studiessubmitted to Food and Drug Administration (FDA). (B) Studiespublished in journals. (C) Implementation of trim and fill methodon journal data. (D) Implementation of regression adjustmentmodel on journal data (adjusted effect at top where SE is 0)

Significance level <1%Significance level 1-5%Significance level 5-10%Significance level >10%

Stan

dard

err

or

0.4

0.2

0.1

0.0

0.3

FDA to journal change in effect

FDA estimateFixed effect meta-analysis unpublished

Stan

dard

err

or

0.4

0.2

0.1

0.0

0.3

Effect estimate

-1.0 -0.5 0 0.5 1.0

A

B

Fig 2 | Contour enhanced funnel plots displaying discrepancybetween Food and Drug Administration (FDA) data and journaldata. (A) Arrows joining effect results from same studies whenboth were available from FDA and journals. (B) Estimates ofeffect only available from FDA (not journal published studies)

RESEARCH

page 4 of 7 BMJ | ONLINE FIRST | bmj.com

DISCUSSION

The application of two novel approaches to identifyand adjust for publication biases in a dataset derivedfroma journal publication, where a gold standard data-set exists, produced encouraging results. Firstly, detec-tion of publication biases was convincing using acontour enhanced funnel plot. Secondly, the regres-sion basedmethodproduced a corrected average effectsize, which was close to that obtained from the FDAdataset (and closer than that obtained by the trim andfill method).This assessment does, however, have limitations.

Firstly, the findings relate to a single dataset and thusare not necessarily generalisable to other examples.Specifically, all the trials were sponsored by the phar-maceutical industry and we make the assumption thatthe FDA data are completely unbiased. Furthermore,themethods under evaluationwere designedprimarilyfor the assessment of efficacy outcomes and theymightnot be appropriate for safety outcomes—for example,there may be incentives to suppress statistically signifi-cant safety outcomes (rather than non-significantones). This is an area that requires more research.Debate is ongoing about the usefulness of funnel

plots and related tests for the identification of publica-tion biases. Although their use is widely advocated237

some question their validity,27 38-41 including in this

Significance level <1%Significance level 1-5%Significance level 5-10%Significance level >10%

Stan

dard

err

or

0.4

0.2

0.1

0.0

0.3

FDA estimateFixed effect meta-analysis FDA data

Stan

dard

err

or

0.4

0.2

0.1

0.0

0.3

Journal estimate

Fixed effect meta-analysis FDA dataFixed effect meta-analysis journal data

Journal estimate

Fixed effect trim and fill

Fixed effect meta-analysis journal data

Filled studyFixed effect meta-analysis FDA data

Stan

dard

err

or

0.4

0.2

0.1

0.0

0.3

Effect estimate

Stan

dard

err

or

-1.0 -0.5 0 0.5 1.00.4

0.2

0.1

0.0

0.3

Journal estimate

Fixed effect trim and fill

A

B

C

D

Fixed effect meta-analysis journal data

Regression lineFixed effect meta-analysis FDA data

Fig 1 | Contour enhanced funnel plots (95% CI at top). (A) Studiessubmitted to Food and Drug Administration (FDA). (B) Studiespublished in journals. (C) Implementation of trim and fill methodon journal data. (D) Implementation of regression adjustmentmodel on journal data (adjusted effect at top where SE is 0)

Significance level <1%Significance level 1-5%Significance level 5-10%Significance level >10%

Stan

dard

err

or

0.4

0.2

0.1

0.0

0.3

FDA to journal change in effect

FDA estimateFixed effect meta-analysis unpublished

Stan

dard

err

or

0.4

0.2

0.1

0.0

0.3

Effect estimate

-1.0 -0.5 0 0.5 1.0

A

B

Fig 2 | Contour enhanced funnel plots displaying discrepancybetween Food and Drug Administration (FDA) data and journaldata. (A) Arrows joining effect results from same studies whenboth were available from FDA and journals. (B) Estimates ofeffect only available from FDA (not journal published studies)

RESEARCH

page 4 of 7 BMJ | ONLINE FIRST | bmj.com

7 / 13

DISCUSSION

The application of two novel approaches to identifyand adjust for publication biases in a dataset derivedfroma journal publication, where a gold standard data-set exists, produced encouraging results. Firstly, detec-tion of publication biases was convincing using acontour enhanced funnel plot. Secondly, the regres-sion basedmethodproduced a corrected average effectsize, which was close to that obtained from the FDAdataset (and closer than that obtained by the trim andfill method).This assessment does, however, have limitations.

Firstly, the findings relate to a single dataset and thusare not necessarily generalisable to other examples.Specifically, all the trials were sponsored by the phar-maceutical industry and we make the assumption thatthe FDA data are completely unbiased. Furthermore,themethods under evaluationwere designedprimarilyfor the assessment of efficacy outcomes and theymightnot be appropriate for safety outcomes—for example,there may be incentives to suppress statistically signifi-cant safety outcomes (rather than non-significantones). This is an area that requires more research.Debate is ongoing about the usefulness of funnel

plots and related tests for the identification of publica-tion biases. Although their use is widely advocated237

some question their validity,27 38-41 including in this

Significance level <1%Significance level 1-5%Significance level 5-10%Significance level >10%

Stan

dard

err

or

0.4

0.2

0.1

0.0

0.3

FDA estimateFixed effect meta-analysis FDA data

Stan

dard

err

or

0.4

0.2

0.1

0.0

0.3

Journal estimate

Fixed effect meta-analysis FDA dataFixed effect meta-analysis journal data

Journal estimate

Fixed effect trim and fill

Fixed effect meta-analysis journal data

Filled studyFixed effect meta-analysis FDA data

Stan

dard

err

or

0.4

0.2

0.1

0.0

0.3

Effect estimate

Stan

dard

err

or

-1.0 -0.5 0 0.5 1.00.4

0.2

0.1

0.0

0.3

Journal estimate

Fixed effect trim and fill

A

B

C

D

Fixed effect meta-analysis journal data

Regression lineFixed effect meta-analysis FDA data

Fig 1 | Contour enhanced funnel plots (95% CI at top). (A) Studiessubmitted to Food and Drug Administration (FDA). (B) Studiespublished in journals. (C) Implementation of trim and fill methodon journal data. (D) Implementation of regression adjustmentmodel on journal data (adjusted effect at top where SE is 0)

Significance level <1%Significance level 1-5%Significance level 5-10%Significance level >10%

Stan

dard

err

or

0.4

0.2

0.1

0.0

0.3

FDA to journal change in effect

FDA estimateFixed effect meta-analysis unpublished

Stan

dard

err

or

0.4

0.2

0.1

0.0

0.3

Effect estimate

-1.0 -0.5 0 0.5 1.0

A

B

Fig 2 | Contour enhanced funnel plots displaying discrepancybetween Food and Drug Administration (FDA) data and journaldata. (A) Arrows joining effect results from same studies whenboth were available from FDA and journals. (B) Estimates ofeffect only available from FDA (not journal published studies)

RESEARCH

page 4 of 7 BMJ | ONLINE FIRST | bmj.com

8 / 13

DISCUSSION

The application of two novel approaches to identifyand adjust for publication biases in a dataset derivedfroma journal publication, where a gold standard data-set exists, produced encouraging results. Firstly, detec-tion of publication biases was convincing using acontour enhanced funnel plot. Secondly, the regres-sion basedmethodproduced a corrected average effectsize, which was close to that obtained from the FDAdataset (and closer than that obtained by the trim andfill method).This assessment does, however, have limitations.

Firstly, the findings relate to a single dataset and thusare not necessarily generalisable to other examples.Specifically, all the trials were sponsored by the phar-maceutical industry and we make the assumption thatthe FDA data are completely unbiased. Furthermore,themethods under evaluationwere designedprimarilyfor the assessment of efficacy outcomes and theymightnot be appropriate for safety outcomes—for example,there may be incentives to suppress statistically signifi-cant safety outcomes (rather than non-significantones). This is an area that requires more research.Debate is ongoing about the usefulness of funnel

plots and related tests for the identification of publica-tion biases. Although their use is widely advocated237

some question their validity,27 38-41 including in this

Significance level <1%Significance level 1-5%Significance level 5-10%Significance level >10%

Stan

dard

err

or

0.4

0.2

0.1

0.0

0.3

FDA estimateFixed effect meta-analysis FDA data

Stan

dard

err

or

0.4

0.2

0.1

0.0

0.3

Journal estimate

Fixed effect meta-analysis FDA dataFixed effect meta-analysis journal data

Journal estimate

Fixed effect trim and fill

Fixed effect meta-analysis journal data

Filled studyFixed effect meta-analysis FDA data

Stan

dard

err

or

0.4

0.2

0.1

0.0

0.3

Effect estimate

Stan

dard

err

or

-1.0 -0.5 0 0.5 1.00.4

0.2

0.1

0.0

0.3

Journal estimate

Fixed effect trim and fill

A

B

C

D

Fixed effect meta-analysis journal data

Regression lineFixed effect meta-analysis FDA data

Fig 1 | Contour enhanced funnel plots (95% CI at top). (A) Studiessubmitted to Food and Drug Administration (FDA). (B) Studiespublished in journals. (C) Implementation of trim and fill methodon journal data. (D) Implementation of regression adjustmentmodel on journal data (adjusted effect at top where SE is 0)

Significance level <1%Significance level 1-5%Significance level 5-10%Significance level >10%

Stan

dard

err

or

0.4

0.2

0.1

0.0

0.3

FDA to journal change in effect

FDA estimateFixed effect meta-analysis unpublished

Stan

dard

err

or

0.4

0.2

0.1

0.0

0.3

Effect estimate

-1.0 -0.5 0 0.5 1.0

A

B

Fig 2 | Contour enhanced funnel plots displaying discrepancybetween Food and Drug Administration (FDA) data and journaldata. (A) Arrows joining effect results from same studies whenboth were available from FDA and journals. (B) Estimates ofeffect only available from FDA (not journal published studies)

RESEARCH

page 4 of 7 BMJ | ONLINE FIRST | bmj.com

DISCUSSION

The application of two novel approaches to identifyand adjust for publication biases in a dataset derivedfroma journal publication, where a gold standard data-set exists, produced encouraging results. Firstly, detec-tion of publication biases was convincing using acontour enhanced funnel plot. Secondly, the regres-sion basedmethodproduced a corrected average effectsize, which was close to that obtained from the FDAdataset (and closer than that obtained by the trim andfill method).This assessment does, however, have limitations.

Firstly, the findings relate to a single dataset and thusare not necessarily generalisable to other examples.Specifically, all the trials were sponsored by the phar-maceutical industry and we make the assumption thatthe FDA data are completely unbiased. Furthermore,themethods under evaluationwere designedprimarilyfor the assessment of efficacy outcomes and theymightnot be appropriate for safety outcomes—for example,there may be incentives to suppress statistically signifi-cant safety outcomes (rather than non-significantones). This is an area that requires more research.Debate is ongoing about the usefulness of funnel

plots and related tests for the identification of publica-tion biases. Although their use is widely advocated237

some question their validity,27 38-41 including in this

Significance level <1%Significance level 1-5%Significance level 5-10%Significance level >10%

Stan

dard

err

or

0.4

0.2

0.1

0.0

0.3

FDA estimateFixed effect meta-analysis FDA data

Stan

dard

err

or

0.4

0.2

0.1

0.0

0.3

Journal estimate

Fixed effect meta-analysis FDA dataFixed effect meta-analysis journal data

Journal estimate

Fixed effect trim and fill

Fixed effect meta-analysis journal data

Filled studyFixed effect meta-analysis FDA data

Stan

dard

err

or

0.4

0.2

0.1

0.0

0.3

Effect estimate

Stan

dard

err

or

-1.0 -0.5 0 0.5 1.00.4

0.2

0.1

0.0

0.3

Journal estimate

Fixed effect trim and fill

A

B

C

D

Fixed effect meta-analysis journal data

Regression lineFixed effect meta-analysis FDA data

Fig 1 | Contour enhanced funnel plots (95% CI at top). (A) Studiessubmitted to Food and Drug Administration (FDA). (B) Studiespublished in journals. (C) Implementation of trim and fill methodon journal data. (D) Implementation of regression adjustmentmodel on journal data (adjusted effect at top where SE is 0)

Significance level <1%Significance level 1-5%Significance level 5-10%Significance level >10%

Stan

dard

err

or0.4

0.2

0.1

0.0

0.3

FDA to journal change in effect

FDA estimateFixed effect meta-analysis unpublished

Stan

dard

err

or

0.4

0.2

0.1

0.0

0.3

Effect estimate

-1.0 -0.5 0 0.5 1.0

A

B

Fig 2 | Contour enhanced funnel plots displaying discrepancybetween Food and Drug Administration (FDA) data and journaldata. (A) Arrows joining effect results from same studies whenboth were available from FDA and journals. (B) Estimates ofeffect only available from FDA (not journal published studies)

RESEARCH

page 4 of 7 BMJ | ONLINE FIRST | bmj.com

9 / 13

I Regression based bias adjustment methods:

Shang et al., 2005; Moreno, Sutton, Ades, et al., 2009DISCUSSION

The application of two novel approaches to identifyand adjust for publication biases in a dataset derivedfroma journal publication, where a gold standard data-set exists, produced encouraging results. Firstly, detec-tion of publication biases was convincing using acontour enhanced funnel plot. Secondly, the regres-sion basedmethodproduced a corrected average effectsize, which was close to that obtained from the FDAdataset (and closer than that obtained by the trim andfill method).This assessment does, however, have limitations.

Firstly, the findings relate to a single dataset and thusare not necessarily generalisable to other examples.Specifically, all the trials were sponsored by the phar-maceutical industry and we make the assumption thatthe FDA data are completely unbiased. Furthermore,themethods under evaluationwere designedprimarilyfor the assessment of efficacy outcomes and theymightnot be appropriate for safety outcomes—for example,there may be incentives to suppress statistically signifi-cant safety outcomes (rather than non-significantones). This is an area that requires more research.Debate is ongoing about the usefulness of funnel

plots and related tests for the identification of publica-tion biases. Although their use is widely advocated237

some question their validity,27 38-41 including in this

Significance level <1%Significance level 1-5%Significance level 5-10%Significance level >10%

Stan

dard

err

or

0.4

0.2

0.1

0.0

0.3

FDA estimateFixed effect meta-analysis FDA data

Stan

dard

err

or

0.4

0.2

0.1

0.0

0.3

Journal estimate

Fixed effect meta-analysis FDA dataFixed effect meta-analysis journal data

Journal estimate

Fixed effect trim and fill

Fixed effect meta-analysis journal data

Filled studyFixed effect meta-analysis FDA data

Stan

dard

err

or

0.4

0.2

0.1

0.0

0.3

Effect estimate

Stan

dard

err

or

-1.0 -0.5 0 0.5 1.00.4

0.2

0.1

0.0

0.3

Journal estimate

Fixed effect trim and fill

A

B

C

D

Fixed effect meta-analysis journal data

Regression lineFixed effect meta-analysis FDA data

Fig 1 | Contour enhanced funnel plots (95% CI at top). (A) Studiessubmitted to Food and Drug Administration (FDA). (B) Studiespublished in journals. (C) Implementation of trim and fill methodon journal data. (D) Implementation of regression adjustmentmodel on journal data (adjusted effect at top where SE is 0)

Significance level <1%Significance level 1-5%Significance level 5-10%Significance level >10%

Stan

dard

err

or

0.4

0.2

0.1

0.0

0.3

FDA to journal change in effect

FDA estimateFixed effect meta-analysis unpublished

Stan

dard

err

or

0.4

0.2

0.1

0.0

0.3

Effect estimate

-1.0 -0.5 0 0.5 1.0

A

B

Fig 2 | Contour enhanced funnel plots displaying discrepancybetween Food and Drug Administration (FDA) data and journaldata. (A) Arrows joining effect results from same studies whenboth were available from FDA and journals. (B) Estimates ofeffect only available from FDA (not journal published studies)

RESEARCH

page 4 of 7 BMJ | ONLINE FIRST | bmj.com

DISCUSSION

The application of two novel approaches to identifyand adjust for publication biases in a dataset derivedfroma journal publication, where a gold standard data-set exists, produced encouraging results. Firstly, detec-tion of publication biases was convincing using acontour enhanced funnel plot. Secondly, the regres-sion basedmethodproduced a corrected average effectsize, which was close to that obtained from the FDAdataset (and closer than that obtained by the trim andfill method).This assessment does, however, have limitations.

Firstly, the findings relate to a single dataset and thusare not necessarily generalisable to other examples.Specifically, all the trials were sponsored by the phar-maceutical industry and we make the assumption thatthe FDA data are completely unbiased. Furthermore,themethods under evaluationwere designedprimarilyfor the assessment of efficacy outcomes and theymightnot be appropriate for safety outcomes—for example,there may be incentives to suppress statistically signifi-cant safety outcomes (rather than non-significantones). This is an area that requires more research.Debate is ongoing about the usefulness of funnel

plots and related tests for the identification of publica-tion biases. Although their use is widely advocated237

some question their validity,27 38-41 including in this

Significance level <1%Significance level 1-5%Significance level 5-10%Significance level >10%

Stan

dard

err

or

0.4

0.2

0.1

0.0

0.3

FDA estimateFixed effect meta-analysis FDA data

Stan

dard

err

or

0.4

0.2

0.1

0.0

0.3

Journal estimate

Fixed effect meta-analysis FDA dataFixed effect meta-analysis journal data

Journal estimate

Fixed effect trim and fill

Fixed effect meta-analysis journal data

Filled studyFixed effect meta-analysis FDA data

Stan

dard

err

or

0.4

0.2

0.1

0.0

0.3

Effect estimate

Stan

dard

err

or

-1.0 -0.5 0 0.5 1.00.4

0.2

0.1

0.0

0.3

Journal estimate

Fixed effect trim and fill

A

B

C

D

Fixed effect meta-analysis journal data

Regression lineFixed effect meta-analysis FDA data

Fig 1 | Contour enhanced funnel plots (95% CI at top). (A) Studiessubmitted to Food and Drug Administration (FDA). (B) Studiespublished in journals. (C) Implementation of trim and fill methodon journal data. (D) Implementation of regression adjustmentmodel on journal data (adjusted effect at top where SE is 0)

Significance level <1%Significance level 1-5%Significance level 5-10%Significance level >10%

Stan

dard

err

or

0.4

0.2

0.1

0.0

0.3

FDA to journal change in effect

FDA estimateFixed effect meta-analysis unpublished

Stan

dard

err

or

0.4

0.2

0.1

0.0

0.3

Effect estimate

-1.0 -0.5 0 0.5 1.0

A

B

Fig 2 | Contour enhanced funnel plots displaying discrepancybetween Food and Drug Administration (FDA) data and journaldata. (A) Arrows joining effect results from same studies whenboth were available from FDA and journals. (B) Estimates ofeffect only available from FDA (not journal published studies)

RESEARCH

page 4 of 7 BMJ | ONLINE FIRST | bmj.com

10 / 13

The confunnel commandI Peters et al., 2008

- investigation of 48 published Cochrane meta-analyses

I Sterne, Egger, & Moher, 2008, Cochrane Handbook, section10.4.b

I Palmer, Peters, Sutton, & Moreno, 2008 (v1.0.4)I Palmer, Peters, Sutton, & Moreno, 2009 (v1.0.5)

11 / 13

The confunnel commandI Peters et al., 2008

- investigation of 48 published Cochrane meta-analysesI Sterne et al., 2008, Cochrane Handbook, section 10.4.b

I Palmer et al., 2008 (v1.0.4)I Palmer et al., 2009 (v1.0.5)

11 / 13

The confunnel commandI Peters et al., 2008

- investigation of 48 published Cochrane meta-analysesI Sterne et al., 2008, Cochrane Handbook, section 10.4.bI Palmer et al., 2008 (v1.0.4)I Palmer et al., 2009 (v1.0.5)

11 / 13

The confunnel commandI Peters et al., 2008

- investigation of 48 published Cochrane meta-analysesI Sterne et al., 2008, Cochrane Handbook, section 10.4.bI Palmer et al., 2008 (v1.0.4)I Palmer et al., 2009 (v1.0.5)

11 / 13

confunnel: syntax and options

I Syntax:

confunnel logor selogor [, options ]

I Options:I metric(se|invse|var|invvar): different y -axes: variance,

standard error & their inverses (Sterne & Egger, 2001)I onesided(lower|upper): one sided significance levelsI Other twoway options

12 / 13

confunnel: syntax and options

I Syntax:

confunnel logor selogor [, options ]

I Options:I metric(se|invse|var|invvar): different y -axes: variance,

standard error & their inverses (Sterne & Egger, 2001)I onesided(lower|upper): one sided significance levelsI Other twoway options

12 / 13

Contour enhanced funnel plots discussion

I Funnel plots should be used with care (Lau, Ioannidis, Terrin,Schmid, & Olkin, 2006)

I Aid assessment of reporting biases

I Put other bias assessment methods in a context

I confunnel can be used with metan, metabias, metatrim

13 / 13

References I

Duval, S., & Tweedie, R. L. (2000a). A nonparametric ”trim and fill” method ofaccounting for publication bias in meta-analysis. Journal of the AmericanStatistical Society, 95, 89–98.

Duval, S., & Tweedie, R. L. (2000b). Trim and fill: A simple funnel plot basedmethod of testing and adjusting for publication bias in meta-analysis.Biometrics, 56, 455–463.

Egger, M., Davey Smith, G., Schneider, M., & Minder, C. (1997). Bias inmeta-analysis detected by a simple, graphical test. British Medical Journal,315(7109), 629–634.

Lau, J., Ioannidis, J., Terrin, N., Schmid, C. H., & Olkin, I. (2006). The case of themisleading funnel plot. British Medical Journal, 333(7568), 597–600.

Moreno, S. G., Sutton, A. J., Ades, A. E., Stanley, T. D., Abrams, K. R., Peters, J. L.,et al. (2009). Assessment of regression-based methods to adjust forpublication bias through a comprehensive simulation study. BMC MedicalResearch Methodology, 2. (published online 12 January 2009)

Moreno, S. G., Sutton, A. J., Turner, E. H., Abrams, K. R., Cooper, N. J., Palmer,T. M., et al. (2009). Novel methods to deal with publication biases: secondaryanalysis of antidepressant trials in the FDA trial registry database and relatedjournal publications. British Medical Journal, 339, 494–498. (b2981)

Palmer, T. M., Peters, J. L., Sutton, A. J., & Moreno, S. G. (2008). Contourenhanced funnel plots for meta-analysis. The Stata Journal, 8(2), 242–254.Available fromhttp://www.stata-journal.com/article.html?article=gr0033

References II

Palmer, T. M., Peters, J. L., Sutton, A. J., & Moreno, S. G. (2009). Meta-Analysis inStata: An Updated Collection from The Stata Journal. In J. A. C. Sterne(Ed.), (pp. 124–137). College Station, Texas: Stata Press. Available fromhttp://www.stata.com/bookstore/mais.html

Peters, J. L., Sutton, A. J., Jones, D. R., Abrams, K. R., & Rushton, L. (2008).Contour-enhanced meta-analysis funnel plots help distinguish publication biasfrom other causes of asymmetry. Journal of Clinical Epidemiology, 61(10),991–996.

Shang, A., Huwiler-Muntener, K., Nartey, L., Juni, P., Dorig, S., Sterne, J. A. C., etal. (2005). Are the clinical effects of homoeopathy placebo effects?Comparative study of placebo-controlled trials of homoeopathy and allopathy.The Lancet, 366(9487), 726–732.

Spiegelhalter, D. J. (2002). Funnel plots for institutional comparison. Quality Safetyin Health Care, 11(4), 390–391.

Spiegelhalter, D. J. (2005). Funnel plots for comparing institutional performance.Statistics in Medicine, 24(8), 1185–1202.

Sterne, J. A. C., & Egger, M. (2001). Funnel plots for detecting bias in meta-analysis:Guidelines on choice of axis. Journal of Clinical Epdiemiology, 54(10),1046–1055.

References III

Sterne, J. A. C., Egger, M., & Moher, D. (2008, September). Cochrane handbook forsystematic reviews of interventions version 5.0.1. In J. P. T. Higgins &S. Green (Eds.), (chap. Chapter 10: Addressing reporting biases). TheCochrane Collaboration. Available fromhttp://www.mrc-bsu.cam.ac.uk/cochrane/handbook/chapter 10/

figure 10 4 b contour enhanced funnel plots.htm

Sterne, J. A. C., & Harbord, R. M. (2004). Funnel plots in meta-analysis. The StataJournal, 4(2), 127–141.

Turner, E. H., Matthews, A. M., Linardatos, E., Tell, R. A., & Rosenthal, R. (2008).Selective publication of antidepressant trials and its influence on apparentefficacy. New England Journal of Medicine, 358(3), 252–260.