open, transparent and reproducible science is stronger science · 2016. 11. 9. · open sharing...

9/11/2016

1

Open, transparent and reproducible science is stronger science

Natalie MeyersPartnerships and Collaborations Manager, Center for Open Science

University of Notre Dame

Natalie K. Meyers

Center for Open Science

http://cos.io/ | http://osf.io

OPEN, TRANSPARENT, AND REPRODUCIBLE SCIENCE IS STRONGER SCIENCE

9/11/2016

2

Mission

Increase openness,

integrity, and

reproducibility of

scholarly research.

Technology to enable change

Training to enact change

Incentives to embrace change

9/11/2016

3

Infrastructure Metascience

Community

COS by the Numbers

9/11/2016

4

Incentives for

individual success

are focused on

getting it published,

not getting it right

Nosek, Spies, & Motyl, 2012

9/11/2016

5

Figure 1. Positive Results by Discipline.

Fanelli D (2010) “Positive” Results Increase Down the Hierarchy of the Sciences. PLoS ONE 5(4): e10068. doi:10.1371/journal.pone.0010068http://127.0.0.1:8081/plosone/article?id=info:doi/10.1371/journal.pone.0010068

9/11/2016

6

Scientific Ideals

- Innovative ideas- Reproducible results- Accumulation of knowledge

9/11/2016

7

What is reproducibility?

•Computational Reproducibility:–If we took your data and code/analysis scripts and reran it, we can reproduce the numbers/graphs in your paper

•Empirical Reproducibility:–We have enough information to rerun the experiment or survey the way it was originally conducted

•Replicability:–We use your exact methods and analyses, but collect new data, and we get the same statistical results

9/11/2016

8

Metascience

Repeat After Me by Maki Naro, Oct 6, 2016 in The Nib Used with Permission URL=https://thenib.com/repeat-after-me

© C

OP

YRIG

HT

FIR

ST L

OO

K M

ED

IA 2

016

9/11/2016

9

http://michaelinzlicht.com/getting-better/

/2016/2/29/reckoning-with-the-

past/

Repeat After Me by Maki Naro, Oct 6, 2016 in The Nib Used with Permission URL=https://thenib.com/repeat-after-me

© C

OP

YRIG

HT

FIR

ST L

OO

K M

ED

IA 2

01

6

https://osf.io/ezcuj/

9/11/2016

10

Repeat After Me by Maki Naro, Oct 6, 2016 in The Nib Used with Permission URL=https://thenib.com/repeat-after-me

© C

OP

YRIG

HT

FIR

ST L

OO

K M

ED

IA 2

01

6

9/11/2016

11

Software Preservation Necessary for Reproducibility

9/11/2016

12

daspos.org

https://osf.io/y9mpx/

9/11/2016

13

Infrastructure

WriteReport

AnalyzeData

Search/Discovery

DevelopIdea

DesignStudy

CollectData

Store Data

Analyze Data

OSF

9/11/2016

14

Technology to enable change

Open Science Framework

http://osf.io

free, open source

9/11/2016

15

Put data, materials, and code on the OSF

Persistent

Citable

Identifiers

9/11/2016

16

Connecting the workflow

is critical to enabling change

9/11/2016

17

OpenSesame

OpenSesame

9/11/2016

18

API Docs

https://api.osf.io/v2/docs/

9/11/2016

19

Community

9/11/2016

20

Problem

The gap between

scholarly values and

practices.

Promotes incentives that help

realign values and practices

9/11/2016

21

Norms

CommunalityOpen sharing

UniversalismEvaluate research on own merit

DisinterestednessMotivated by knowledge and discovery

Organized skepticismConsider all new evidence, even against one’s prior work

Quality

Counternorms

SecrecyClosed

ParticularlismEvaluate research by reputation

Self-interestednessTreat science as a competition

Organized dogmatismInvest career promoting one’s own theories, findings

Quantity

Anderson, Martinson, & DeVries, 2007

9/11/2016

22

Anderson, Martinson, & DeVries, 2007

Anderson, Martinson, & DeVries, 2007

9/11/2016

23

Problems

• Flexibility in analysis

• Selective reporting

• Ignoring nulls

• Lack of replication

Sterling, 1959; Cohen, 1962; Lykken, 1968; Tukey, 1969;

Greenwald, 1975; Meehl, 1978; Rosenthal, 1979

A reader quick, keen, and leery

Did wonder, ponder, and query

When results clean and tight

Fit predictions just right

If the data preceded the theory

Anonymous, quoted from Kerr (1998)

9/11/2016

24

A Garden of Forking Paths

Jorge Luis Borges; Gelman and Loken

Exclude outliers?

Control for year?

Median or mean?

“Does X affect Y?”

Read more: osf.io/8mpji

9/11/2016

25

Registered Reports

DesignCollect &

AnalyzeReport Publish

PEER

REVIEW

Registered Reports

DesignCollect &

AnalyzeReport Publish

PEER

REVIEW

9/11/2016

26

Registered Reports

• AIMS Neuroscience

• Attention, Perception, &

Psychophysics

• Cognition and Emotion

• Comprehensive Results in

Social Psychology

• Cortex

• Drug and Alcohol Dependence

• eLife

• Euro Journal of Neuroscience

• Experimental Psychology

• Journal of Accounting Research

• Journal of Business and

Psychology

• Journal of Personnel

Psychology

• Journal of Media Psychology

• Nutrition and Food Science

Journal

• Perspectives on Psych. Science

• Royal Society Open Science

• Social Psychology

• Stress and Health

• Work, Aging, and Retirement

http://osf.io/8mpji

9/11/2016

27

Who Publishes Registered

Reports?

(just to name a few)

See the full list of journals at osf.io/8mpji

Incentives to embrace change

9/11/2016

28

The $1,000,000

Preregistration Challenge

Endorse TOP Guidelines

Badges for Open Practices

Registered Reports

https://cos.io/prereg

9/11/2016

29

http://cos.io/top

752 Journals

63 Organizations

TOP Guidelines

1. Data citation

2. Design transparency

3. Research materials transparency

4. Data transparency

5. Analytic methods (code) transparency

6. Preregistration of studies

7. Preregistration of analysis plans

8. Replication

http://cos.io/top

9/11/2016

30

Transparency & Openness Promotion Guidelines

Eight Standards

1. Data citation

2. Design transparency

3. Research materials transparency

4. Data transparency

5. Analytic methods (code) transparency

6. Preregistration of studies

7. Preregistration of analysis plans

8. Replication

Three Tiers

1. Disclose

2. Require

3. Verify

Signatories

752 Journals63 Organization

Learn more at http://cos.io/top

Signals: Making Behaviors Visible

Promotes Adoption

9/11/2016

31

Case Study: Psychological Science

9/11/2016

32

% A

rtic

les

rep

ort

ing

that

dat

a w

as a

vaila

ble

40%

30%

20%

10%

0%

% o

f A

rtic

les

rep

ort

ing

that

dat

a w

as a

vaila

ble

100%

75%

50%

25%

0%Reportedly available Available Correct Data Usable Data Complete Data

9/11/2016

33

Training to enact change

Meet researchers where they

are and help them find

immediate benefit in training

9/11/2016

34

Free training on how to make

research more reproducible

http://cos.io/stats_consulting

9/11/2016

35

Find this presentation at

https://osf.io/hgjyz/

Questions: natalie@cos.io