Open Science as an answer to science's credibility crisis

2019-01

This presentation is licensed under a CC-BY 4.0 license. You may copy, distribute, and use the slides in your own work, as long as you give attribution to the original author at each slide that you use.

PD Dr. Felix SchönbrodtLudwig-Maximilians-Universität München

www.nicebread.dewww.researchtransparency.org

@nicebread303

2

Why should we trust your claims?

ResearcherSnake oil salesman

Uses a sound scientific method which, in the long run, arguably arrives at an increasingly correct account of the world.Changes his/her mind in the light of new evidence.

Is transparent about the primary data, methods, and analyses.

No commercial interests / COIs, or discloses them.

„Nullius in verba“ - motto of the Royal Society, world’s oldest scientific society: „Don’t trust my words“

Uses a method which produces anecdotal evidence and false positive results in the desired direction.

Has a predetermined attitude about „what works“.

Hides data and methods; nobody can check or reconstruct what has been done.

Commercial interests/conflicts of interest.

„Believe me! I have the greatest snake oil.“

3

Why should we trust your claims?

ResearcherSnake oil salesman

Uses a method which, in the long run, arguably arrives at an increasingly correct account of the worldChanges his/her mind in the light of new evidence.

Is transparent about the primary data, methods, and analyses.

No commercial interests / COIs, or discloses them.

„Nullius in verba“ - motto of the Royal Society, world’s oldest scientific society: „Don’t trust my words“

Uses a method which produces anecdotal evidence and false positive results in the desired direction.

Has a predetermined attitude about „what works“.

Hides data and methods; nobody can check or reconstruct what has been done.

Commercial interests/conflicts of interest.

„Believe me! I have the greatest snake oil.“

How to be successful in academia

aka.

Hack your way to scientific glory

How to become a Professor?

5Abele-Brehm, A. E., & Bühner, M. (2016). Wer soll die Professur bekommen? Psychologische Rundschau, 67(4), 250–261. http://doi.org/10.1026/0033-3042/a000335

Actual (not desired) relevance in professorship hiring committees Rank

Number of peer-reviewed publications 1

Fit of research profile to the hiring department 2

Quality of research talk 3

Number of publications 4

Volume of acquired third-party funding 5

Number of first authorships 6

… …

N = 1453 psychology researchers, 66% were actually members of a professorship hiring committee.

How to get lots of publications?

6

92% of published papers have significant,

positive results

Fanelli, D. (2010). “Positive” Results Increase Down the Hierarchy of the Sciences. PLOS ONE, 5, e10068. doi:10.1371/journal.pone.0010068

p-hacking /questionable research practice

(QRPs)

Tool 1: Outcome switching

8http://compare-trials.org/ http://blogs.discovermagazine.com/neuroskeptic/2015/07/23/social-priming-money-for-nothing/#.VuKRSRi5KJM

Tool 1: Outcome switching

9

•2 outcome variables: false positive rate 5% ➙ 9.5%

•5 outcome variables with one-sided testing:false positive rate 5% ➙ 41%

Best-practice example:Transforming a boring dissertation into a groundbreaking publication

10

https://twitter.com/JoeHilgard/status/699693258386051072

Tool 2: Many conditions, report only those that worked

11Armitage, P., McPherson, C. K., & Rowe, B. C. (1969). Repeated significance tests on accumulating data. Journal of the Royal Statistical Society. Series A (General), 132, 235–244.

With long enough sampling and optional

stopping, it is guaranteed to get a significant result!

100%

Tool 3: Optional stopping

12Inspired by Neurosceptic’s blog: http://blogs.discovermagazine.com/neuroskeptic/2015/05/18/p-hacking-a-talk-and-further-thoughts/#.VV2TiOePKsN

Data

p < .05Type of outlier

rejection

Test equal variance assumption?

Use a robust statistic?

Check again if all variables are coded

correctly?

The garden of forking pathsAndrew Gelman & Eric Loken, 2013

How bad can it be? A bad case (but not untypical?) scenario

•Doing some of these questionable research practices (QRPs) in combination raises false positive rate from 5% to 61%!

•QRPs corrupt the logic of the p-value and “renders the reported p-values essentially uninterpretable.”

Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22, 1359–1366. doi:10.1177/0956797611417632Wasserstein, R. L., & Lazar, N. A. (2016). The ASA's statement on p-values: context, process, and purpose. American Statistician, 00–00. http://doi.org/10.1080/00031305.2016.1154108

13

5%

61%

14

in psychology

The higher the study quality, the smaller the effect

15Macleod et al., JCBFM 2005; 25: 713-721

(Un)Intentional?

16

• Intentional? • Evil researcher who only cares about his/her career and not at all

about truth-seeking?

•Unintentional?• Wrong education?• Wrong/uncritical standards of the field?• Pushed by supervisors, reviewers, or editors? ➙ http://bulliedintobadscience.org/

• Distorting effects on the published record are probably comparable, but the ethical evaluations differs strongly.

Publication bias

Add some publication bias to the mix:Efficacy of anti-depressants (Turner et al. 2008)

!18

Trials published in journals 48 positive, 3 negative

Trials preregistered at FDA* 38 positive, 36 negative

* Food and Drug Administration

https://twitter.com/eturnermd1/status/737436322344927232 Turner, E. H., Matthews, A. M., Linardatos, E., Tell, R. A., & Rosenthal, R. (2008). Selective Publication of Antidepressant Trials and Its Influence on Apparent Efficacy. New England Journal of Medicine, 358, 252–260. doi:10.1056/NEJMsa065779

Honest mistakes

20http://thepoliticalmethodologist.com/2014/12/09/a-decade-of-replications-lessons-from-the-quarterly-journal-of-political-science/

• Reproducible analysis code and open data required at submission - “inhouse checking” in review process

• 54% of all submissions had results in the paper that did not match the computed results from the code

• wrong signs, wrong labeling of regression coefficients, erorrs in sample sizes, wrong descriptive stats

90%-Excel-Gate

21http://economix.blogs.nytimes.com/2013/04/16/flaws-are-cited-in-a-landmark-study-on-debt-and-growth/?smid=tw-share&_r=0https://www.aeaweb.org/articles.php?doi=10.1257/aer.100.2.573

The most important point of the story: The original authors shared their raw data, which made it possible to correct the honest mistake!

22

23

p-hacking + (uncorrectable) honest mistakes +

publication bias =

Kahneman: Open Letter

24

I believe that you should collectively do something

about this mess.I see a train wreck

looming.

Daniel Kahneman, Nobel prize 2002

http://www.nature.com/polopoly_fs/7.6716.1349271308!/suppinfoFile/Kahneman%20Letter.pdf

25

https://osf.io/ezcuj/wiki/home/Reproducibility Project: Psychology (RP:P)

• 100 replications• 36% of all

replications were significant• PS - cog: 53%• JEP:LMC: 48%• PS - soc: 29%• JPSP - soc: 23%

• 83% of all effect sizes are smaller than the original:MO: r = .40MR: r = .20

Open Science Collaboration (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716–10. http://doi.org/10.1126/science.aac4716

Only psychology?An outlook to other disciplines.

27Begley, C. G., & Ellis, L. M. (2012). Drug development: Raise standards for preclinical cancer research. Nature, 483, 531–533. doi:10.1038/483531aPrinz, F., Schlange, T., & Asadullah, K. (2011). Believe it or not: how much can we rely on published data on potential drug targets? Nature Reviews Drug Discovery, 10, 712–712. doi:10.1038/nrd3439-c1

• 53 ‘landmark studies’, not randomly selected: fresh approaches targeted for future drug development

• “scientific findings were confirmed in only 6 (11%) cases. Even knowing the limitations of preclinical research, this was a shocking result.”

• Bayer Healthcare: 67 target-validation projects in oncology, women’s health, and cardiovascular medicine. Only 14 (21%) could be reproduced.

Which part of published findings can be independently replicated?

28

0%

25%

50%

75%

100%

Psychology (2015; n = 97)

Economy* (2015; n = 67)

Cancer research 1 (2011; n = 53)

Cancer research 2 (2012; n = 67)

Experimental Philosophy

(n=40, 2018)

22%

79%89%

51%64%

78%

21%11%

49%36%

Open Science Collaboration (2015); Chang & Li (2015); Begley, C. G., & Ellis, L. M. (2012). Prinz, F., Schlange, T., & Asadullah, K. (2011); Cova et al. (2018)

* The data on economics is about reproducibility; i.e. the attempt to get the same results if you apply the original data analysis on the original data set.

29

http://www.nature.com/news/2011/111005/full/478026a/box/2.htmlhttp://retractionwatch.com/2016/03/24/retractions-rise-to-nearly-700-in-fiscal-year-2015-and-psst-this-is-our-3000th-post/https://www.washingtonpost.com/news/speaking-of-science/wp/2016/04/01/when-scientists-lie-about-their-research-should-they-go-to-jail/

„In the past decade, the number of retraction notices has shot up 10-fold.“

Retractions: +1000% in 10 years

2013

467500

2015

684

Buzzwords in scientific abstracts+880% from 1974- 2014

30Vinkers, C. H., Tijdink, J. K., & Otte, W. M. (2015). Use of positive and negative words in scientific PubMed abstracts between 1974 and 2014: retrospective analysis. Bmj, 351, h6467–6. http://doi.org/10.1136/bmj.h6467

Amazing!! Enormous!!

Groundbreaking!!!

A dystopian view of science …

31

Uses a method which produces anecdotal evidence and false positive results in the desired direction ➙ p-hacking and publication bias

Has a predetermined attitude about „what works“ ➙ confirmation bias

Hides data and methods; nobody can check or reconstruct what has been done. ➙ common practice in many fields

Commercial interests/conflicts of interest. ➙ individual career interests

„Believe me! I have the greatest snake oil.“➙ „Open science advocates are bullies. We need more trust in science. SAD!“

32

Frau

d Questionable research practices

Goo

d sc

ient

ific

prac

tice

•The detrimental cumulative impact of a „grey zone of questionable research practices“ might be much worse than occasional misconduct

33

Thesis:

Our current incentives foster questionable research

practices, which decrease the truth value of our shared

knowledge.

What is good for the individual careers of

researchers leads to a collective fiasko.

Researchers who do it right (i.e., high power, no QRPs, transparency) have a clear competitive disadvantage.

Anti-Thesis:

Society pays for us that we generate valid and robust

knowledge.

Our incentives should be chosen in a way that they

foster good science.

Researchers who do it right should be supported and

promoted.

The new way of doing research:

A scientific frameworkfor the 21. century

35

Credible Science

Open Access

Open D

ata

Open M

aterial

Open Peer Review

Open Source

(Software)

Open Educational

Resources

Open Science

Open Science Badges

36

https://osf.io/tvyxz/wiki/home/

37

Open Science Badges

38

https://osf.io/tvyxz/wiki/home/

As of Oct 2015, 38% of all PsychScience papers had Open Data

Kidwell, M. C., Lazarević, L. B., Baranski, E., Hardwicke, T. E., Piechowski, S., Falkenberg, L.-S., et al. (2016). Badges to Acknowledge Open Practices: A Simple, Low-Cost, Effective Method for Increasing Transparency. PLoS Biology, 14(5), e1002456–15. http://doi.org/10.1371/journal.pbio.1002456

Open Science Badges

39

https://osf.io/tvyxz/wiki/home/

As of Oct 2015, 38% of all PsychScience papers had Open Data

Kidwell, M. C., Lazarević, L. B., Baranski, E., Hardwicke, T. E., Piechowski, S., Falkenberg, L.-S., et al. (2016). Badges to Acknowledge Open Practices: A Simple, Low-Cost, Effective Method for Increasing Transparency. PLoS Biology, 14(5), e1002456–15. http://doi.org/10.1371/journal.pbio.1002456

40http://dfg.de/download/pdf/foerderung/programme/lis/ua_inf_empfehlungen_200901.pdf, http://www.dfg.de/download/pdf/foerderung/antragstellung/forschungsdaten/richtlinien_forschungsdaten.pdfhttp://www.dgps.de/fileadmin/documents/Empfehlungen/Datenmanagement_deu.pdf, http://www.dgps.de/fileadmin/documents/Empfehlungen/Data_Management_eng.pdfhttp://www.dfg.de/foerderung/antrag_gutachter_gremien/antragstellende/nachnutzung_forschungsdaten/index.html#anker62194854

“dass die Daten unmittelbar nach Abschluss der Forschungen oder nach wenigen Monaten der Öffentlichkeit frei zur Verfügung gestellt werden.“

Das Engagement […] von Wissenschaftlern und Wissenschaftlerinnen um die Verfügbarmachung von Forschungsdaten sollten bei der Würdigung von wissenschaftlichen […] Leistungen zukünftig stärker berücksichtigt werden.

One benefit of open data:

Assessing analytical flexibilityand doing robustness checks

42

• A “multiverse analysis” (Steegen, Tuerlinchx, Gelman, & Vanpaemel, 2016): Report results for all plausible analytical decisions

• Check robustness of results: Do several analytical paths lead to comparable conclusions?

• Based on open data by Carney et al. (2010)

43

Of 54 plausible analyses exactly one was significant.Guess which has been reported in the original paper?

44http://faculty.haas.berkeley.edu/dana_carney/pdf_my%20position%20on%20power%20poses.pdf

Open Letter by Dana Carney (2016)

45Rohrer, J. M., Egloff, B., & Schmukle, S. C. (in press). Probing birth-order effects on narrow traits using Specification Curve Analysis. Psychological Science. https://osf.io/vg2un/

46Rohrer, J. M., Egloff, B., & Schmukle, S. C. (in press). Probing birth-order effects on narrow traits using Specification Curve Analysis. Psychological Science. https://osf.io/vg2un/

The effect is robust against analytical

variations

47

What is a preregistration?

48

„The specification of a research design, hypotheses, and analysis plan prior to observing

the outcomes of a study“

Nosek&Lindsay(2018)

Why preregistration?

49

Number of hypothesis tests: Unknown

1. Clear distinction between confirmatory and exploratory research

Confirmatory ResearchAnalyze hypotheses

Exploratory ResearchBrowse through data

Number of hypothesis tests: Known

Error rate control: Possible Error rate control: Impossible

p-value: Meaningful*

*Well, as meaningful as a p-value can be

p-value: „essentially uninterpretable“ (cf. ASA statement)

Wasserstein, R. L., & Lazar, N. A. (2016). The ASA's statement on p-values: context, process, and purpose. American Statistician, 00–00. http://doi.org/10.1080/00031305.2016.1154108

Why preregistration?

50

2. Prevent p-hacking and QRPs

51

Registered Reportshttps://cos.io/rr/

52

AIMS NeuroscienceAttention, Perception, and PsychophysicsCognition and EmotionComprehensive Results in Social PsychologyCortexDrug and Alcohol DependenceExperimental PsychologyJournal of Accounting ResearchJournal of Business and PsychologyJournal of Personnel PsychologyJournal of Media PsychologyNFS JournalPerspectives on Psychological ScienceRoyal Society Open ScienceSocial PsychologyWorking, Aging and Retirement

https://cos.io/rr/Registered Reports

161 journals offer Registered Reports(by Jan 2019)

Pre-registration causes medicines to stop working!

53http://chrisblattman.com/2016/03/01/13719/Kaplan, R. M., & Irvin, V. L. (2015). Likelihood of Null Effects of Large NHLBI Clinical Trials Has Increased over Time. PLoS ONE, 10(8), e0132382–12. http://doi.org/10.1371/journal.pone.0132382

no prereg: 57%

success rate!

prereg: 8%

success rate…

Pre-registration causes p-hacking medicines to stop working!

54

no prereg: 57%

success rate!

prereg: 8%

success rate…

http://chrisblattman.com/2016/03/01/13719/Kaplan, R. M., & Irvin, V. L. (2015). Likelihood of Null Effects of Large NHLBI Clinical Trials Has Increased over Time. PLoS ONE, 10(8), e0132382–12. http://doi.org/10.1371/journal.pone.0132382

Open Access

56

✓ Full open access, no APCs✓ Non-commercial institutional

publisher (Linnaeus U library)✓ Open, citable peer review

(with doi)✓ Well-powered null results and

direct replications welcomed✓ Registered Reports as option✓ Mandatory open data✓ Open Science badges

(including a reproducibility badge)

✓ Special article formats, e.g. „Empty your file-drawer“

Beyond commercial publishers

COS launches branded preprint servers

57

16 preprint services with > 2 million searchable preprints. 

Publish (nearly) all of your papers as green Open Access

58http://www.sherpa.ac.uk/romeo/index.php

Open Science as a strategic benefit

McKiernan, E. C., Bourne, P. E., Brown, C. T., Buck, S., Kenall, A., Lin, J., et al. (2016). How open science helps researchers succeed. eLife, 5, e16800. http://doi.org/10.7554/eLife.16800

Journals with mandatory open data (or justification why not)•Advances in Methods and Practices in Psychological Science (AMPPS) •Archives of Scientific Psychology •BMC Psychology •Collabra: Psychology •Cognition •Comprehensive Results in Social Psychology •European Journal of Personality (EJP) •European Journal of Social Psychology (EJSP) •Evolution and Human Behavior •Experimental Psychology •Journal of Economic Psychology •Journal of Open Psychology Data (JOPD) •Journal of Research in Personality •Judgment and Decision Making •Journal of Cognition •Meta-Psychology •PLOS ONE •Royal Society Open Science •Science 60

For a continuously updated list, see here: https://osf.io/tbkzh/wiki/Psychology%20journals%20with%20mandatory%20open%20data/

61

…

www.uni-koeln.de

The Department of Psychology at the Faculty of Human Sciences of the University of Cologne (UoC) seeks to appoint a

Full Professor (W3) of Social Psychology

to be filled as soon as possible.

The successful candidate is expected to have a record of excellence in social cognition, and/or related areas such as cognitive psychology or motivation science.

The candidate is also expected to strongly contribute to the UoC’s Center for Social and Economic Behavior and the Social Cognition Center Cologne of the Department of Psychology. Both structures are part of UoC’s Key Profile Area II, „Behavioral Economic Engineering and Social Cognition“.

The ideal candidate’s track record should show an excellent fit with these interrelated structures and a strong interest to bridge the fields of social cognition and behavioral economics.

The Department of Psychology aims for transparent and reproducible research (including Open Data, Open Materials, and Preregistrations). Applicants are asked to illustrate how they have pursued these goals in the past and/or how they plan to do so in the future.

We strongly encourage international applicants. Salaries and working conditions at the UoC - one of the German Universities of Excellence – meet international standards. Candidates are expected to be willing to learn the German language. The Faculties offer Bachelor, Master, and doctoral degrees. Courses are taught either in English or German.

Applicants will be hired in concordance with § 36 of the University Law of the State of North-Rhine Westphalia.

The UoC supports diversity, the multiplicity of perspectives, and equal opportunities. The University of Cologne particularly encourages applications from disabled persons. Disabled persons are given preference in case of equal qualification. Women are strongly encouraged to apply. Preferential treatment is given to women if their professional qualifications and abilities are equivalent to those of other applicants.

Applications with the usual documents (including vita, research statement, 5 most important publications, full list of publications and teaching experience, and diplomas) should be submitted via the University’s Academic Job Portal (https://berufungen.uni-koeln.de) until March 30th, 2017.

www.uni-koeln.de

The Department of Psychology at the Faculty of Human Sciences of the University of Cologne (UoC) seeks to appoint a

Full Professor (W3) of Social Psychology

to be filled as soon as possible.

The successful candidate is expected to have a record of excellence in social cognition, and/or related areas such as cognitive psychology or motivation science.

The candidate is also expected to strongly contribute to the UoC’s Center for Social and Economic Behavior and the Social Cognition Center Cologne of the Department of Psychology. Both structures are part of UoC’s Key Profile Area II, „Behavioral Economic Engineering and Social Cognition“.

The ideal candidate’s track record should show an excellent fit with these interrelated structures and a strong interest to bridge the fields of social cognition and behavioral economics.

The Department of Psychology aims for transparent and reproducible research (including Open Data, Open Materials, and Preregistrations). Applicants are asked to illustrate how they have pursued these goals in the past and/or how they plan to do so in the future.

We strongly encourage international applicants. Salaries and working conditions at the UoC - one of the German Universities of Excellence – meet international standards. Candidates are expected to be willing to learn the German language. The Faculties offer Bachelor, Master, and doctoral degrees. Courses are taught either in English or German.

Applicants will be hired in concordance with § 36 of the University Law of the State of North-Rhine Westphalia.

The UoC supports diversity, the multiplicity of perspectives, and equal opportunities. The University of Cologne particularly encourages applications from disabled persons. Disabled persons are given preference in case of equal qualification. Women are strongly encouraged to apply. Preferential treatment is given to women if their professional qualifications and abilities are equivalent to those of other applicants.

Applications with the usual documents (including vita, research statement, 5 most important publications, full list of publications and teaching experience, and diplomas) should be submitted via the University’s Academic Job Portal (https://berufungen.uni-koeln.de) until March 30th, 2017.

…

…

+ 3 additional professorship job descriptions

https://osf.io/7jbnt/

62http://www.fak11.lmu.de/dep_psychologie/osc/open-science-hiring-policy/index.html

Journals should really ... …It's the task of granting agencies to ... …

University boards have to ... …

Journals should really ... …It's the task of granting agencies to ... …

University boards have to ... …

I want you for Open Science!

65

Spend your valuable reviewer’s time on research that is worthy to be reviewed

https://opennessinitiative.org/

We suggest that beginning January 1, 2017, reviewers make open practices a pre-condition for more comprehensive review. This is already in reviewers’ power; to drive the change, all that is needed is for reviewers to collectively agree that the time for change has come.

Sign

up!

Consider to sign our voluntary commitment to research transparency

66http://www.researchtransparency.org/http://www.nicebread.de/a-voluntary-commitment-to-research-transparency/

67

http://www.researchtransparency.org/

As reviewers:

•We ask for Open Data and use the “standard reviewer disclosure request” (https://osf.io/hadz3/)

As co-authors …

•We try to convince the first authors to do it the same way

Supervision of dissertations:•We expect Open Data, Open Material, reproducible scripts (internal)

• If publication: see “first authors”• If series of experiments:at least 1 pre-registered study

•Grading independent of p-value and “publishability”

In committees, as editors, …

•We promote the values of open science and transparency

As first authors we do always…

Consider to sign our voluntary commitment to research transparency

•Open Data•Reproducible analysis scripts•21-word-solution (Simmons et al., 2012)

68

• 191 signatories from >50 international universities (by Jan 2019)

Consider to sign our voluntary commitment to research transparency

http://www.researchtransparency.org/

69

• … we have an „Open Science Committee“ with 40 members, all chairs are represented

• … all „Empirische Praktika“ have a core curriculum which consists of:• Power analysis• Pre-registration• Open Data + Codebook• Reproducible scripts

• … money is partly distributed based open science bonus points• a „triple-badger“ (open data, open material, preregistered) equals 0.5

additional papers• … professorship positions are advertised with an „open science statement“

At LMU Munich’s psychology department …

Schönbrodt, F.D., Maier, M., Heene, M., & Bühner, M. (2018). Forschungstransparenz als hohes wissenschaftliches Gut stärken: Konkrete Ansatzmöglichkeiten für Psychologische Institute. Psychologische Rundschau, 69, 37–44. doi:10.1026/0033-3042/a000386http://www.fak11.lmu.de/dep_psychologie/studium/lehrelounge/kerncurriculum_empra/index.htmlhttp://www.fak11.lmu.de/dep_psychologie/osc/open-science-hiring-policy/index.html

70

Erlangen

Münster FU Berlin

Göttingen

Hagen

Koblenz-Landau

LMU München

Wien

Netzwerk der Open Science Initiativen: https://osf. io/tbkzh/

Frankfurt

Marburg

Dresden

Leipzig

Zürich

Köln

•42 members of 12 disciplines:Psychology, sociology, computer science, statistics, geography, medicine, veterinary medicine, economics, philosophy of science, …

•3 entire faculties as members:Faculty of Medicine, Faculty of Veterinary Medicine, Faculty of Psychology and Educational Science

•Mission Statement:• Education (from PhD students to professors)• Meta-science research• Change the incentive structure

•http://www.osc.lmu.de• Join our email newsletter : https://lists.lrz.de/mailman/listinfo/lmu-osc 71

72

at Charité Berlin

https://www.bihealth.org/de/quest-center/

https://www.crs.uzh.ch

Coda

74

„This was the largest audience ever to witness an inauguration, period.“Press Secretary Sean Spicer

https://www.theatlantic.com/politics/archive/2017/01/the-pointless-needless-lies-of-the-trump-administration/514061/

postfaktisch!Fake-News!

Four ways of knowing

75This slide is based on a presentation by Arthur Lupia(https://www.mzes.uni-mannheim.de/openscience/wp-content/uploads/2019/01/Mannheim-Open-Science-2019.pptx)

• Appeals to meta-physics (religion, faith)

• Personal testimony/ experiences

• Culture

• Science: A claim’s validity depends less on your faith, experiences, or culture. Anybody (in principle) can verify and reproduce a scientific claim. If done the right way, it is guaranteed to achieve more and more valid results.

Why should society fund us? Why should policy makers rely on our results when they make decisions?

The legitimacy of science comes from its transparency and confirmability. (Only) Then it represents one of the most trustable sources of

information.

10 easy steps to increase your openness1. Create an account on OSF (http://osf.io/)2. Upload the material for an existing study (e.g., questionnaires, protocols, maybe

reproducible analysis scripts) to an OSF project.3. Prior to publication, add an open license to all of your figures (so that you can reuse them

in later publications, blog posts,, or presentations: „Figure available under a CC-BY4.0 license at osf.io/XXXX.“

4. For the next project: Change the consent forms in a way that open data would be possible for that project (see https://osf.io/mgwk8/wiki/Consent%20form%20templates%20for%20open%20data/).

5. Sign the PRO initiative and expect openness (or a justification why not) if you review another paper (https://opennessinitiative.org/)

6. For the next data analysis: Practice to create scripts for reproducible data analysis (e.g., SPSS syntax, R scripts). All analytic steps that lead from raw data to the final results should be reproducible.

7. Let a master student preregister his/her thesis. Can be either a „local preregisteration“, or a proper preregistration at OSF or at https://aspredicted.org/. See this workshop material for how to do a preregistration: https://osf.io/yd487/, https://osf.io/mx7yp/

8. Do you own first preregistration; enter the Prereg challenge and get 1000$: https://cos.io/prereg/

9. Publish your first open data set: Ensure anonymity, provide a codebook. See here for details: http://econtent.hogrefe.com/doi/pdf/10.1026/0033-3042/a000341

10. Team up with colleagues and establish a local open science initiative 76