BACKGROUNDConfirmatary or exploratory?• Most research lies on a continuum

between the two types• Most research is exploratory, while

most statistical tests are meant for confirmatory purposes

• When the data inspire a hypothesis, you cannot use the same data to test that hypothesis

• Confirmation bias and hindsight bias

• Quality vs Quantity

SOLUTION = PRE-REGISTRATION

What is it? • Design and analysis of the study is

defined beforehand, as opposed to posthoc analysis.

• Types: Regular, Public, With adversarial collaboration

Discussion• Researchers cannot be trusted to

observe the distinction of exploratory vs confirmatory study if they are not forced to (?)

• Two stages of research: Fishing expedition confirm • There’s nothing wrong with

exploratory research as long as it is acknowledged as such

• Shortcut: Collect all data at once, and split them into subsets • The extra work involved in pre-

registering an experiment is a small price to pay for a large increase in evidence impact.

• Danger: reverse research order: Research pre-registration confirm results This is FRAUD!

Merit of pre-registrationYongQi Cong and Sam Beekhuizen – University of Amsterdam

ReferencesBakker, M., van Dijk, A., & Wicherts, J. M. (2012). The rules of the game called psychological science. Perspectives on Psychological Science, 7(6), 543-554.Chambers, C. D., Feredoes, E., Muthukumaraswamy, S. D., & Etchells, P. (2014). Instead of" playing the game" it is time to change the rules: Registered Reports at AIMS Neuroscience and beyond. AIMS Neuroscience, 1(1), 4-17.Wagenmakers, E.–J. (2014). Study pre-registration. Presentation for the conference “Improving Scientific Practice: Dealing with the Human Factors ”, University of Amsterdam, Amsterdam, the Netherlands, September 2014.Wagenmakers, E. J., Wetzels, R., Borsboom, D., van der Maas, H. L., & Kievit, R. A. (2012). An agenda for purely confirmatory research. Perspectives on Psychological Science, 7(6), 632-638.

Advantage of pre-registration• Clear distinction of confirmatary vs

explorotary research • Prevent QRPs• Easier for replication • Pre-registration replication reports

makes publication easier

Clear answers need clear questions!