open, transparent and reproducible science is stronger science · 2016. 11. 9. · open sharing...
TRANSCRIPT
9/11/2016
1
Open, transparent and reproducible science is stronger science
Natalie MeyersPartnerships and Collaborations Manager, Center for Open Science
University of Notre Dame
Natalie K. Meyers
Center for Open Science
http://cos.io/ | http://osf.io
OPEN, TRANSPARENT, AND REPRODUCIBLE SCIENCE IS STRONGER SCIENCE
9/11/2016
2
Mission
Increase openness,
integrity, and
reproducibility of
scholarly research.
Technology to enable change
Training to enact change
Incentives to embrace change
9/11/2016
3
Infrastructure Metascience
Community
COS by the Numbers
9/11/2016
4
Incentives for
individual success
are focused on
getting it published,
not getting it right
Nosek, Spies, & Motyl, 2012
9/11/2016
5
Figure 1. Positive Results by Discipline.
Fanelli D (2010) “Positive” Results Increase Down the Hierarchy of the Sciences. PLoS ONE 5(4): e10068. doi:10.1371/journal.pone.0010068http://127.0.0.1:8081/plosone/article?id=info:doi/10.1371/journal.pone.0010068
9/11/2016
6
Scientific Ideals
- Innovative ideas- Reproducible results- Accumulation of knowledge
9/11/2016
7
What is reproducibility?
•Computational Reproducibility:–If we took your data and code/analysis scripts and reran it, we can reproduce the numbers/graphs in your paper
•Empirical Reproducibility:–We have enough information to rerun the experiment or survey the way it was originally conducted
•Replicability:–We use your exact methods and analyses, but collect new data, and we get the same statistical results
9/11/2016
8
Metascience
Repeat After Me by Maki Naro, Oct 6, 2016 in The Nib Used with Permission URL=https://thenib.com/repeat-after-me
© C
OP
YRIG
HT
FIR
ST L
OO
K M
ED
IA 2
016
9/11/2016
9
http://michaelinzlicht.com/getting-better/
/2016/2/29/reckoning-with-the-
past/
Repeat After Me by Maki Naro, Oct 6, 2016 in The Nib Used with Permission URL=https://thenib.com/repeat-after-me
© C
OP
YRIG
HT
FIR
ST L
OO
K M
ED
IA 2
01
6
https://osf.io/ezcuj/
9/11/2016
10
Repeat After Me by Maki Naro, Oct 6, 2016 in The Nib Used with Permission URL=https://thenib.com/repeat-after-me
© C
OP
YRIG
HT
FIR
ST L
OO
K M
ED
IA 2
01
6
9/11/2016
11
Software Preservation Necessary for Reproducibility
9/11/2016
12
daspos.org
https://osf.io/y9mpx/
9/11/2016
13
Infrastructure
WriteReport
AnalyzeData
Search/Discovery
DevelopIdea
DesignStudy
CollectData
Store Data
Analyze Data
OSF
9/11/2016
14
Technology to enable change
Open Science Framework
http://osf.io
free, open source
9/11/2016
15
Put data, materials, and code on the OSF
Persistent
Citable
Identifiers
9/11/2016
16
Connecting the workflow
is critical to enabling change
9/11/2016
17
OpenSesame
OpenSesame
9/11/2016
18
API Docs
https://api.osf.io/v2/docs/
9/11/2016
19
Community
9/11/2016
20
Problem
The gap between
scholarly values and
practices.
Promotes incentives that help
realign values and practices
9/11/2016
21
Norms
CommunalityOpen sharing
UniversalismEvaluate research on own merit
DisinterestednessMotivated by knowledge and discovery
Organized skepticismConsider all new evidence, even against one’s prior work
Quality
Counternorms
SecrecyClosed
ParticularlismEvaluate research by reputation
Self-interestednessTreat science as a competition
Organized dogmatismInvest career promoting one’s own theories, findings
Quantity
Anderson, Martinson, & DeVries, 2007
9/11/2016
22
Anderson, Martinson, & DeVries, 2007
Anderson, Martinson, & DeVries, 2007
9/11/2016
23
Problems
• Flexibility in analysis
• Selective reporting
• Ignoring nulls
• Lack of replication
Sterling, 1959; Cohen, 1962; Lykken, 1968; Tukey, 1969;
Greenwald, 1975; Meehl, 1978; Rosenthal, 1979
A reader quick, keen, and leery
Did wonder, ponder, and query
When results clean and tight
Fit predictions just right
If the data preceded the theory
Anonymous, quoted from Kerr (1998)
9/11/2016
24
A Garden of Forking Paths
Jorge Luis Borges; Gelman and Loken
Exclude outliers?
Control for year?
Median or mean?
“Does X affect Y?”
Read more: osf.io/8mpji
9/11/2016
25
Registered Reports
DesignCollect &
AnalyzeReport Publish
PEER
REVIEW
Registered Reports
DesignCollect &
AnalyzeReport Publish
PEER
REVIEW
9/11/2016
26
Registered Reports
• AIMS Neuroscience
• Attention, Perception, &
Psychophysics
• Cognition and Emotion
• Comprehensive Results in
Social Psychology
• Cortex
• Drug and Alcohol Dependence
• eLife
• Euro Journal of Neuroscience
• Experimental Psychology
• Journal of Accounting Research
• Journal of Business and
Psychology
• Journal of Personnel
Psychology
• Journal of Media Psychology
• Nutrition and Food Science
Journal
• Perspectives on Psych. Science
• Royal Society Open Science
• Social Psychology
• Stress and Health
• Work, Aging, and Retirement
http://osf.io/8mpji
9/11/2016
27
Who Publishes Registered
Reports?
(just to name a few)
See the full list of journals at osf.io/8mpji
Incentives to embrace change
9/11/2016
28
The $1,000,000
Preregistration Challenge
Endorse TOP Guidelines
Badges for Open Practices
Registered Reports
https://cos.io/prereg
9/11/2016
29
http://cos.io/top
752 Journals
63 Organizations
TOP Guidelines
1. Data citation
2. Design transparency
3. Research materials transparency
4. Data transparency
5. Analytic methods (code) transparency
6. Preregistration of studies
7. Preregistration of analysis plans
8. Replication
http://cos.io/top
9/11/2016
30
Transparency & Openness Promotion Guidelines
Eight Standards
1. Data citation
2. Design transparency
3. Research materials transparency
4. Data transparency
5. Analytic methods (code) transparency
6. Preregistration of studies
7. Preregistration of analysis plans
8. Replication
Three Tiers
1. Disclose
2. Require
3. Verify
Signatories
752 Journals63 Organization
Learn more at http://cos.io/top
Signals: Making Behaviors Visible
Promotes Adoption
9/11/2016
31
Case Study: Psychological Science
9/11/2016
32
% A
rtic
les
rep
ort
ing
that
dat
a w
as a
vaila
ble
40%
30%
20%
10%
0%
% o
f A
rtic
les
rep
ort
ing
that
dat
a w
as a
vaila
ble
100%
75%
50%
25%
0%Reportedly available Available Correct Data Usable Data Complete Data
9/11/2016
33
Training to enact change
Meet researchers where they
are and help them find
immediate benefit in training
9/11/2016
34
Free training on how to make
research more reproducible
http://cos.io/stats_consulting
