how anonymous post-publication peer review uncovers bad science

How anonymous post-publication peer review

uncovers bad science

Leonid Schneider,science journalist with Laborjournal

leonid.schneider@gmail.comTwitter: @schneiderleonid

Junior scientists are often told by their advisors:

- If you can deliver this result, you will publish a nice paper and have a job

- If you don’t deliver this result, you will not publish any paper and have no job

Is bad scienceindividual or systemic failure?

$$$

Authors and institutions have little incentive to produce reliable quality science

Paper-to-funding convertion

Funding used for research…

Journals and funding agencies prefer simplistic, but sensationalist “break-through” science

• Cancer cure!

• Stem cells /”Reprogramming”

• One-Gene-Phenotype models

• Translational/Commercial potential

Biological systems are very complicated,

but in biological papers simplicity rules!

Scientists occasionally help data to fit their theoretical model for a publication

• Selective data acquisition and evaluation (very common)

• “Adjustments” or manipulation of data (less common)

• Data falsification / fraud (very rare)

Peer review weeds out bad science. Really?

• Data is submitted on trust as being honest/reliable

• Peer Reviewers are scientist colleagues, not data specialists

• Peer Reviewers only analyse science, not data integrity

• Peer review is not always done diligently enough

How did this pass peer review????

$$$

Traditional peer

review

Traditional peer review is anything but transparent

Years and years of research…

- Journal Editors- Decide on Quality,

Novelty, Impact- Appoint peer

reviewers- Make final decisions

- Peer Reviewers- 1-4 people- Unknown to authors

or readers- Potential COI,

personal animosities, lack of competence…

$$$

Too many financial and personal interests involved

Years and years of research…

Convincing peer reviewers is by far the most important task of a scientist

A peer-reviewed paper is a badge of honour

Things surely changed for him since he published in Nature…

• Publications are public evidence of success

• They are to be admired and not questioned

• Often not the content counts, but where it is published

Scientists waste time, money and their careers trying to reproduce unreliable or manipulated results

• Poor reproducibility in combination with high competition undermines productivity, but also work moral, trust and motivation

• It leads to even more data manipulation and fraud in science

What do you do if you spot data irregularities or irreproducibility in a published paper?

1. Write to authors

2. Write to journal

3. Write to authors’ institution

What happens if a published paper is reported to be wrong or even contain manipulated data?

1. Correction (rare)

2. Retraction (even rarer)

3. Nothing (most common)

Your paper is wrong, professor!

See you at the exam…

Individual criticisms are unwelcome and dangerous

• Financial interests behind publications prevent institutional investigations

• Institutions often refuse to react to anonymous hints

• Whistle-blowers are often punished or dismissed as incompetent or malicious

Solution: make valid criticisms public, but anonymously!

• Publicly available valid criticisms are much more difficult to be ignored

• Whistle-blowers are protected by the anonymity under which they are free to post concerns

More on this: http://www.laborjournal.de/editorials/424_11.lasso

Adam Marcus

Ivan Oransky

Retraction Watch takes whistle-blowers seriously

PubPeer allows anonymous post-publication peer review, including evidence

On PubPeer you could ask critical questions anonymously

Or, you can post evidence of data irregularities on PubPeer, also anonymously

I have some issues with your paper, Sir!

Pros and cons of anonymous commenting (aka witch-hunts)

• Protects whistle-blowers

• Only objective evidence and arguments matter, not who has raised them or why or where

• Unsubstantiated claims, personal insults

• Sock-puppeting (also by authors!)

Against:

For:

PubPeer protects the anonymity, even when the criticized scientists go to court

Blog post at Laborjournal about this interesting and useful experience: http://www.laborjournal.de/blog/?p=8281

Autors also reply to PubPeer criticisms, often

constructively

STAP: one of the biggest fraud scandals uncovered, thanks to post-publication peer review on PubPeer

Photo credit: Maigrot/REA

The case Olivier Voinnet

• PhD with Sir David Baulcombe at The Sainsbury Laboratory, Norwich

• Research leader at CNRS institute in Strasbourg (age 33)

• Professor at ETH Zürich (since 2010)

Photo credit: Maigrot/REA

The case Olivier Voinnet

• EMBO Gold Medal (2009)

• EMBO Member

• EMBO Young Investigator grant

• ERC start-up grant

• Max-Rössler-Prize (ETH Zürich, 2013)

It started with people finding irregularities in David Baulcombe’s papers

Olivier Voinnet publications flagged on PubPeer

Olivier Voinnet and other involved reply on PubPeer and promise to investigate

Vicki VanceProfessor of Botany, University of South Carolina

Vicki Vance: the key Whistle-Blower in Voinnet case

To be continued?