Publication Bias in Medical Informatics evaluation research:

Is it an issue or not?

Mag. (FH) Christof Machan, M.Sc.Univ-Prof. Elske Ammenwerth

Dr. Thomas Bodner

christof.machan@umit.at

Overview

• Background• Detecting publication bias - 2 attempts

– study 1: A simplistic approach– study 2: The funnel plot

• Discussion• Conclusions / Outlook

Why think about publication bias

• evidence-based practice• systematic review as best evidence

– should include as many publications on the question invesigated as possible

– high danger of publication bias leading to wrong conclusions

• Strong evidence for publication bias in other fields (social sciences, biomedicine)

• What about evaluation literature in medical informatics?

Background

Results72 publications reported clear

results (positive or negative)– 60 positive– 12 negative

Anzahl positiver und negativer Arbeiten

60

12

0

10

20

30

40

50

60

70

postive Arbeiten negative Arbeiten

Study 1: Detecting publication bias - a simplistic approach

Positive / negative papers

positive negative

Detecting PB

AssumptionIf there is a remarkable difference between publications reporting positive and publications reporting negative results in a random sample publication bias is a possible reason. (see also Dickersin 1990)

Methods• random sample of 86 MI evaluation publications out of evalDB• Classify / Count / Compare

• graphical depiction of publication bias• easy to understand• scatter plot displaying study quality (e.g. sample size, standard error,...)

and effect size• recommended by Cochrane

Source: http://www.ukl.uni-freiburg.de/med/med8/seminar%2018%2011%2004.ppt#30

effect size

n

Relatives Risiko

3

2

1

,9

,8

,7

,6

,5

,4

,3

Fallzahl n

3000

2000

1000

500400300

200

100

504030

20

10

Publiziert

mehrfach

ja

nein

effect size

n

Detecting PB

Study 2: Detecting publication bias - the funnel plot - basics

Methods• study design: meta-analysis• Search in Medline and Embase for controlled trials on CPOE and medication errors / ADEs• Hand Search in several journals• intervention: CPOE system• effect size: relative risk for medication errors / ADEs between intervention and control group• 26 studies included• assessment of publication bias by funnel plot

Study 2: Effects of CPOE on medication errors / ADEs

Detecting PB

AssumptionMissing publications of low study quality reporting on negative and / or non-significant results may be an indication for publication bias.

Study 2: Effects of CPOE on medication errors / ADEs

Detecting PB

Discussion

Study 1:• 60 out of 72 studies positive seems a high number• limitations:

– assumption that ratio of positive / negative studies equals 50:50 not valid

– other biases (e.g. langauge bias)– chance– ...

Discussion

Study 1:• plot indicates sign for publication bias• limitations:

– other selection biases (language, citation, etc.)– poor methodological quality of smaller studies– true heterogenity– chance

Conclusions / Outlook

• Publication bias is an issue• existence hard to proof• easy to jump to wrong conclusions• awareness of authors important• quality of publications must be sufficient• Evidence-based medical informatics

Conclusions / Outlook