reusable science: how not to slip from the shoulders of giants

Reusable Science:How not to slip from the shoulders of giantsChris GorgolewskiMax Planck Research Group: Neuroanatomy & Connectivity

Anatomy of a giant

I. Example studies

II. Probabilistic ROIs

III.Sharing statistical maps

IV.Data papers

Anatomy of a giant

I. Example studies

II. Probabilistic ROIs

III.Sharing statistical maps

IV.Data papers

Study I

Medial and Lateral Networks in Anterior Prefrontal Cortex Support Metacognitive Ability for Memory

and Perception

Benjamin Baird, Jonathan Smallwood, Krzysztof J. Gorgolewski, and Daniel S. Margulies

Journal of Neuroscience (in press)

Meta-cognition

• Are we equally good in judging our performance of memory or perception tasks?

• Is metacognition related to medial or lateral prefrontal cortex? Does it depend on modality?

Measuring metacognition

Metacognition of memory and perception are distinct systems

Sources of seed points

Gilbert et al. 2006, Functional specialization within rostral prefrontal cortex (area 10): a meta-analysis. Journal of cognitive neuroscience

Fleming, S. M., Weil, R. S., Nagy, Z., Dolan, R. J., & Rees, G. (2010). Relating introspective accuracy to individual differences in brain structure. Science (New York, N.Y.), 329(5998), 1541–3.

Sources of seed points

Meta-cognition of MemoryMeta-cognition of Perception

Baird, Smallwood et al. (in press) JON

vs.

Double dissociation of metacognitive abilities

Study II

The Default Modes of Reading: Modulation of posterior cingulate and medial prefrontal cortex connectivity

associated with subjective and objective differences in reading experience

Jonathan Smallwood, Krzysztof J. Gorgolewski, Johannes Golchert, Florence J.M. Ruby, Haakon G. Engen, Benjamin

Baird, Melaina Vinski, Jonathan Schooler, Daniel S. Margulies

Frontiers in Neuroscience (in press)

Reading comprehension

• What is the relation between task focus and reading comprehension?

• What role does Default Mode Network play in reading comprehension and task focus?

Task focus is inversely correlated with reading comprehension

Smallwood, et al., Frontiers in Human Neuroscience

Reading by DefaultSeed locations

Andrews-hanna, J. R., Reidler, J. S., Sepulcre, J., Poulin, R., Buckner, R. L., & Temp, P. (2010). Functional-anatomic fractionation of the brain’s default network. Neuron, 65(4), 550–62.

Smallwood, et al., Frontiers in Human Neuroscience

Reading by Default

Reading by Default

Smallwood, et al., Frontiers in Human Neuroscience

Why mind-wandering may disrupt reading

Study III

A correspondence between the brain's intrinsic functional

architecture and the content and form of self-generated thoughts

Krzysztof J. Gorgolewski, Dan Lurie, Sebastian Urchs, Judy A. Kipping, R. Cameron Craddock, Michael P.

Milham, Daniel S. Margulies, and Jonathan Smallwood

PLoS One (submitted)

Mind wandering

• What is the content and form of thoughts in mind wandering?

• How does it relate to various aspects of intrinsic BOLD activity?

Questions about the content

Questions about the form

Future vs. Past = Words vs. Images

Resting state measures

Fractional Amplitude of Low Frequency Fluctuations

Regional Homogeneity

Degree Centrality

Group average

Not only Default Mode Network

Not only Default Mode Network

What these studies have in common?

Anatomy of a giant

I. Example studies

II. Probabilistic ROIs

III.Sharing statistical maps

IV.Data papers

Signal to Noise ratio

Looking in the wrong places

Lower SNR = we miss more stuff

Lower SNR = higher FDR threshold

How to improve power?

• stronger effects?• fewer null/noise samples -> ROI

What is wrong with ROI analysis?

What is wrong with ROI analysis?

Binary nature of masks

Fifty shades of grey

A probabilistic view on the ROI analysis

A probabilistic approach to ROI analysisGorgolewski et al. PRNI 2013

Fifty Shades of Grey, Matter

Extensions and disclaimers

• Kernel density estimation• Markov Random Field reguralization• Posterior maps cannot be used in

meta analysis – circularity!• Prior maps are integral part of the

analysis and need to be included in publications

Anatomy of a giant

I. Example studies

II. Probabilistic ROIs

III.Sharing statistical maps

IV.Data papers

Just coordinates?

• Databases such as Neurosynth or BrainMap rely on peak coordinates reported in papers (only strong effects)

Are we throwing money away?

Data sharing?

Data sharing?

• Ok, ok so we should share data.• We all know it’s good.• But almost no one does it.– You have to prepare data– You risk that your mistakes will be

found!

“I swear I’ve heard it before”

• In the past there were many attempts to propagate data sharing– For example fMRI DC:

• Failed because of technical issues• …and the amount of time it took to prepare data

for submission (a week, a very frustrating week)

• fMRI DC was however too ambitious for its time:– They wanted to collect raw data and all

metadata required to reproduce the analysis

Van Horn & Gazzaniga (2013). Why share data? Lessons learned from the fMRIDC. NeuroImage

Baby steps

• Everything is a question of cost and benefit– If we keep the cost low even small

benefit (or just conviction that data sharing is GOOD) will suffice

NeuroVault.org simple data sharing

• Minimize the cost!• We just want your statistical maps

with minimum description (DOI)– If you want you can put more metadata,

but you don’t have to

• We streamline login process (external services such as Google, Facebook)

Benefits?

• In return authors get interactive web based visualization of their statistical maps– Something they can embed on their lab

website

• We are keeping both cost and benefit low…–…but we also plan to work with journal

editors to popularize the idea

?

Share your stat maps!

Make science more reproducible

NeuroVault.org

Anatomy of a giant

I. Example studies

II. Probabilistic ROIs

III.Sharing statistical maps

IV.Data papers

Motivation

• Share your stat maps!

vs.

institutions scientists

Quality control

• Share your stat maps!

Complex datasets require elaborate descriptions

Solution – data papers

• Authors get recognizable credit for their work.– Even smaller contributors such as RAs

can be included.

• Acquisition methods are described in detail.

• Quality of metadata is being controlled by peer review.

• Neuroinformatics (Springer)• Frontiers in Human Brain Methods

• GigaScience (BGI, BioMed Central)• Scientific Data (Nature Publising

Group, coming soon)

(Frontiers Media)(Nature Publishing Group)

Where to publish data papers?

Read more

• Probabilistic ROISGorgolewski et al. PRNI, 2013

• NeuroVault.orgGorgolewski et al. OHBM, 2013

• Data papersGorgolewski et al. Frontiers in Brain Imaging Methods, 2012

Acknowledgements(my personal giants)

Pierre-Louie BazinHaakon EngenSatrajit Ghosh

Russell A. PoldrackJean-Baptiste Poline

Yannick SchwarzTal Yarkoni

Michael MilhamDaniel Margulies

Benjamin Baird

Jonathan SmallwoodJohannes GolchertFlorence J.M. RubyMelaina VinskiJonathan SchoolerDan LurieSebastian UrchsJudy A. KippingR. Cameron CraddockMPI CBS Resting state group

THANK YOU!

Details

Details

Details