reproducible research - mescal...

❘❡♣r♦❞✉❝✐❜❧❡ ❘❡s❡❛r❝❤

❆r♥❛✉❞ ▲❡❣r❛♥❞

❏♦✉r♥é❡ ❚r❛❝❡s✱ ●r❡♥♦❜❧❡▼❛r❝❤ ✷✹✱ ✷✵✶✺

❆r♥❛✉❞ ▲❡❣r❛♥❞ ✶ ✴ ✶✼

❆ ❋❡✇ ❊❞✐❢②✐♥❣ ❊①❛♠♣❧❡s

◆❛✐❝❦❡♥✱ ❙t❡♣❤❡♥ ❡t ❆❧✳✱ ❚♦✇❛r❞s ❨❡t ❆♥♦t❤❡r P❡❡r✲t♦✲P❡❡r❙✐♠✉❧❛t♦r✱ ❍❊❚✲◆❊❚s✬✵✻✳

From 141 P2P sim.papers, 30% use a custom tool,50% don’t report used tool

25

50

75

0/100

Type

Chord−(SFS)

Custom

NS−2

Other

Unspecified

Simulator usage [Naicken06]

❈♦❧❧❜❡r❣✱ ❈❤r✐st✐❛♥ ❡t ❆❧✳✱ ▼❡❛s✉r✐♥❣ ❘❡♣r♦❞✉❝✐❜✐❧✐t② ✐♥ ❈♦♠♣✉t❡r ❙②st❡♠s ❘❡s❡❛r❝❤✱ ❤tt♣✿✴✴r❡♣r♦❞✉❝✐❜✐❧✐t②✳❝s✳❛r✐③♦♥❛✳❡❞✉✴

601

NC63

HW30

508

Article85

Web54

EMyes

87

EX106

226

OK≤30

130OK

>30

64OK

A u

23

Buildfails9

176

EMno

146EM

∅

30

8 ACM conferences (❆❙P▲❖❙✬✶✷✱❈❈❙✬✶✷✱ ❖❖P❙▲❆✬✶✷✱ ❖❙❉■✬✶✷✱ P▲❉■✬✶✷✱

❙■●▼❖❉✬✶✷✱ ❙❖❙P✬✶✶✱ ❱▲❉❇✬✶✷) and 5journals

EM♥♦= the code cannot be provided

❆r♥❛✉❞ ▲❡❣r❛♥❞ ❋✉♥ ❊①❛♠♣❧❡s ✷ ✴ ✶✼

❚❤❡ ❉♦❣ ❆t❡ ♠② ❍♦♠❡✇♦r❦ ✦✦✦

❱❡rs✐♦♥♥✐♥❣ Pr♦❜❧❡♠s

❇❛❞ ❇❛❝❦✉♣ Pr❛❝t✐❝❡s

❈♦❞❡ ❲✐❧❧ ❜❡ ❆✈❛✐❧❛❜❧❡ ❙♦♦♥

◆♦ ■♥t❡♥t✐♦♥ t♦ ❘❡❧❡❛s❡

Pr♦❣r❛♠♠❡r ▲❡❢t

❈♦♠♠❡r❝✐❛❧ ❈♦❞❡

Pr♦♣r✐❡t❛r② ❆❝❛❞❡♠✐❝ ❈♦❞❡

❘❡s❡❛r❝❤ ✈s✳ ❙❤❛r✐♥❣

✳✳✳

✳✳✳

Thanks for your interest in the implementation of our paper. The goodnews is that I was able to find some code. I am just hoping that it isa stable working version of the code, and matches the implementationwe finally used for the paper. Unfortunately, I have lost some data whenmy laptop was stolen last year. The bad news is that the code is notcommented and/or clean.

Attached is the 〈system〉 source code of our algorithm. I’m not very surewhether it is the final version of the code used in our paper, but it shouldbe at least 99% close. Hope it will help.

❆r♥❛✉❞ ▲❡❣r❛♥❞ ❋✉♥ ❊①❛♠♣❧❡s ✸ ✴ ✶✼

❚❤❡ ❉♦❣ ❆t❡ ♠② ❍♦♠❡✇♦r❦ ✦✦✦

❱❡rs✐♦♥♥✐♥❣ Pr♦❜❧❡♠s

❇❛❞ ❇❛❝❦✉♣ Pr❛❝t✐❝❡s

❈♦❞❡ ❲✐❧❧ ❜❡ ❆✈❛✐❧❛❜❧❡ ❙♦♦♥

◆♦ ■♥t❡♥t✐♦♥ t♦ ❘❡❧❡❛s❡

Pr♦❣r❛♠♠❡r ▲❡❢t

❈♦♠♠❡r❝✐❛❧ ❈♦❞❡

Pr♦♣r✐❡t❛r② ❆❝❛❞❡♠✐❝ ❈♦❞❡

❘❡s❡❛r❝❤ ✈s✳ ❙❤❛r✐♥❣

✳✳✳

✳✳✳

Unfortunately, the server in which my implementation was stored had adisk crash in April and three disks crashed simultaneously. While the helpdesk made significant effort to save the data, my entire implementationfor this paper was not found.

❆r♥❛✉❞ ▲❡❣r❛♥❞ ❋✉♥ ❊①❛♠♣❧❡s ✸ ✴ ✶✼

❚❤❡ ❉♦❣ ❆t❡ ♠② ❍♦♠❡✇♦r❦ ✦✦✦

❱❡rs✐♦♥♥✐♥❣ Pr♦❜❧❡♠s

❇❛❞ ❇❛❝❦✉♣ Pr❛❝t✐❝❡s

❈♦❞❡ ❲✐❧❧ ❜❡ ❆✈❛✐❧❛❜❧❡ ❙♦♦♥

◆♦ ■♥t❡♥t✐♦♥ t♦ ❘❡❧❡❛s❡

Pr♦❣r❛♠♠❡r ▲❡❢t

❈♦♠♠❡r❝✐❛❧ ❈♦❞❡

Pr♦♣r✐❡t❛r② ❆❝❛❞❡♠✐❝ ❈♦❞❡

❘❡s❡❛r❝❤ ✈s✳ ❙❤❛r✐♥❣

✳✳✳

✳✳✳

Unfortunately the current system is not mature enough at the moment,so it’s not yet publicly available. We are actively working on a numberof extensions and things are somewhat volatile. However, once thingsstabilize we plan to release it to outside users. At that point, we wouldbe happy to send you a copy.

❆r♥❛✉❞ ▲❡❣r❛♥❞ ❋✉♥ ❊①❛♠♣❧❡s ✸ ✴ ✶✼

❚❤❡ ❉♦❣ ❆t❡ ♠② ❍♦♠❡✇♦r❦ ✦✦✦

❱❡rs✐♦♥♥✐♥❣ Pr♦❜❧❡♠s

❇❛❞ ❇❛❝❦✉♣ Pr❛❝t✐❝❡s

❈♦❞❡ ❲✐❧❧ ❜❡ ❆✈❛✐❧❛❜❧❡ ❙♦♦♥

◆♦ ■♥t❡♥t✐♦♥ t♦ ❘❡❧❡❛s❡

Pr♦❣r❛♠♠❡r ▲❡❢t

❈♦♠♠❡r❝✐❛❧ ❈♦❞❡

Pr♦♣r✐❡t❛r② ❆❝❛❞❡♠✐❝ ❈♦❞❡

❘❡s❡❛r❝❤ ✈s✳ ❙❤❛r✐♥❣

✳✳✳

✳✳✳

I am afraid that the source code was never released. The code was neverintended to be released so is not in any shape for general use.

❆r♥❛✉❞ ▲❡❣r❛♥❞ ❋✉♥ ❊①❛♠♣❧❡s ✸ ✴ ✶✼

❚❤❡ ❉♦❣ ❆t❡ ♠② ❍♦♠❡✇♦r❦ ✦✦✦

❱❡rs✐♦♥♥✐♥❣ Pr♦❜❧❡♠s

❇❛❞ ❇❛❝❦✉♣ Pr❛❝t✐❝❡s

❈♦❞❡ ❲✐❧❧ ❜❡ ❆✈❛✐❧❛❜❧❡ ❙♦♦♥

◆♦ ■♥t❡♥t✐♦♥ t♦ ❘❡❧❡❛s❡

Pr♦❣r❛♠♠❡r ▲❡❢t

❈♦♠♠❡r❝✐❛❧ ❈♦❞❡

Pr♦♣r✐❡t❛r② ❆❝❛❞❡♠✐❝ ❈♦❞❡

❘❡s❡❛r❝❤ ✈s✳ ❙❤❛r✐♥❣

✳✳✳

✳✳✳

〈STUDENT〉 was a graduate student in our program but he left a whileback so I am responding instead. For the paper we used a prototypethat included many moving pieces that only 〈STUDENT〉 knew how tooperate and we did not have the time to integrate them in a ready-to-share implementation before he left. Still, I hope you can build on theideas/technique of the paper.

Unfortunately, the author who has done most of the coding for this paperhas passed away and the code is no longer maintained.

❆r♥❛✉❞ ▲❡❣r❛♥❞ ❋✉♥ ❊①❛♠♣❧❡s ✸ ✴ ✶✼

❚❤❡ ❉♦❣ ❆t❡ ♠② ❍♦♠❡✇♦r❦ ✦✦✦

❱❡rs✐♦♥♥✐♥❣ Pr♦❜❧❡♠s

❇❛❞ ❇❛❝❦✉♣ Pr❛❝t✐❝❡s

❈♦❞❡ ❲✐❧❧ ❜❡ ❆✈❛✐❧❛❜❧❡ ❙♦♦♥

◆♦ ■♥t❡♥t✐♦♥ t♦ ❘❡❧❡❛s❡

Pr♦❣r❛♠♠❡r ▲❡❢t

❈♦♠♠❡r❝✐❛❧ ❈♦❞❡

Pr♦♣r✐❡t❛r② ❆❝❛❞❡♠✐❝ ❈♦❞❡

❘❡s❡❛r❝❤ ✈s✳ ❙❤❛r✐♥❣

✳✳✳

✳✳✳

Since this work has been done at 〈COMPANY〉 we don’t open-sourcecode unless there is a compelling business reason to do so. So unfortu-nately I don’t think we’ll be able to share it with you.

The code owned by 〈COMPANY〉, and AFAIK the code is not open-source. Your best bet is to reimplement :( Sorry.

❆r♥❛✉❞ ▲❡❣r❛♥❞ ❋✉♥ ❊①❛♠♣❧❡s ✸ ✴ ✶✼

❚❤❡ ❉♦❣ ❆t❡ ♠② ❍♦♠❡✇♦r❦ ✦✦✦

❱❡rs✐♦♥♥✐♥❣ Pr♦❜❧❡♠s

❇❛❞ ❇❛❝❦✉♣ Pr❛❝t✐❝❡s

❈♦❞❡ ❲✐❧❧ ❜❡ ❆✈❛✐❧❛❜❧❡ ❙♦♦♥

◆♦ ■♥t❡♥t✐♦♥ t♦ ❘❡❧❡❛s❡

Pr♦❣r❛♠♠❡r ▲❡❢t

❈♦♠♠❡r❝✐❛❧ ❈♦❞❡

Pr♦♣r✐❡t❛r② ❆❝❛❞❡♠✐❝ ❈♦❞❡

❘❡s❡❛r❝❤ ✈s✳ ❙❤❛r✐♥❣

✳✳✳

✳✳✳

Unfortunately, the 〈SYSTEM〉 sources are not meant to be opensource(the code is partially property of 〈UNIVERSITY 1〉, 〈UNIVERSITY 2〉and 〈UNIVERSITY 3〉.)

If this will change I will let you know, albeit I do not think there is anintention to make the 〈SYSTEM〉 sources opensource in the near future.

If you’re interested in obtaining the code, we only ask for a descriptionof the research project that the code will be used in (which may leadto some joint research), and we also have a software license agreementthat the University would need to sign.

❆r♥❛✉❞ ▲❡❣r❛♥❞ ❋✉♥ ❊①❛♠♣❧❡s ✸ ✴ ✶✼

❚❤❡ ❉♦❣ ❆t❡ ♠② ❍♦♠❡✇♦r❦ ✦✦✦

❱❡rs✐♦♥♥✐♥❣ Pr♦❜❧❡♠s

❇❛❞ ❇❛❝❦✉♣ Pr❛❝t✐❝❡s

❈♦❞❡ ❲✐❧❧ ❜❡ ❆✈❛✐❧❛❜❧❡ ❙♦♦♥

◆♦ ■♥t❡♥t✐♦♥ t♦ ❘❡❧❡❛s❡

Pr♦❣r❛♠♠❡r ▲❡❢t

❈♦♠♠❡r❝✐❛❧ ❈♦❞❡

Pr♦♣r✐❡t❛r② ❆❝❛❞❡♠✐❝ ❈♦❞❡

❘❡s❡❛r❝❤ ✈s✳ ❙❤❛r✐♥❣

✳✳✳

✳✳✳

In the past when we attempted to share it, we found ourselves spendingmore time getting outsiders up to speed than on our own research. SoI finally had to establish the policy that we will not provide the sourcecode outside the group.

❆r♥❛✉❞ ▲❡❣r❛♥❞ ❋✉♥ ❊①❛♠♣❧❡s ✸ ✴ ✶✼

2 Reproducible Research ‘11 Juliana Freire UBC, Vancouver

Science Today: Data Intensive

Obtain

Data

Analyze/

Visualize

User studies

Sensors

Web

Databases

Simulations

Publish/

Share

Sequencing

machines

Particle

colliders

❈♦✉rt❡s② ♦❢ ❏✉❧✐❛♥❛ ❋r❡✐r❡ ✭❆▼P ❲♦r❦s❤♦♣ ♦♥ ❘❡♣r♦❞✉❝✐❜❧❡ r❡s❡❛r❝❤✮

❆r♥❛✉❞ ▲❡❣r❛♥❞ ❍♦✇ ❞♦❡s ✐t ✇♦r❦ ✐♥ ♦t❤❡r s❝✐❡♥❝❡s❄ ✹ ✴ ✶✼

3 Reproducible Research ‘11 Juliana Freire UBC, Vancouver

Science Today: Data + Computing Intensive

Obtain

Data

Analyze/

Visualize

User studies

Sensors

Web

Databases

Simulations

Publish/

Share

Sequencing

machines

Particle

colliders

AVS

Taverna

VisTrails

❈♦✉rt❡s② ♦❢ ❏✉❧✐❛♥❛ ❋r❡✐r❡ ✭❆▼P ❲♦r❦s❤♦♣ ♦♥ ❘❡♣r♦❞✉❝✐❜❧❡ r❡s❡❛r❝❤✮

❆r♥❛✉❞ ▲❡❣r❛♥❞ ❍♦✇ ❞♦❡s ✐t ✇♦r❦ ✐♥ ♦t❤❡r s❝✐❡♥❝❡s❄ ✺ ✴ ✶✼

4 Reproducible Research ‘11 Juliana Freire UBC, Vancouver

Science Today: Data + Computing Intensive

Obtain

Data

Analyze/

Visualize

User studies

Sensors

Web

Databases

Simulations

Publish/

Share

Sequencing

machines

Particle

colliders

❈♦✉rt❡s② ♦❢ ❏✉❧✐❛♥❛ ❋r❡✐r❡ ✭❆▼P ❲♦r❦s❤♦♣ ♦♥ ❘❡♣r♦❞✉❝✐❜❧❡ r❡s❡❛r❝❤✮

❆r♥❛✉❞ ▲❡❣r❛♥❞ ❍♦✇ ❞♦❡s ✐t ✇♦r❦ ✐♥ ♦t❤❡r s❝✐❡♥❝❡s❄ ✻ ✴ ✶✼

5 Reproducible Research ‘11 Juliana Freire UBC, Vancouver

Science Today: Data + Computing Intensive

Obtain

Data

Analyze/

Visualize

User studies

Sensors

Web

Databases

Simulations

Publish/

Share

Sequencing

machines

Particle

colliders

❈♦✉rt❡s② ♦❢ ❏✉❧✐❛♥❛ ❋r❡✐r❡ ✭❆▼P ❲♦r❦s❤♦♣ ♦♥ ❘❡♣r♦❞✉❝✐❜❧❡ r❡s❡❛r❝❤✮

❆r♥❛✉❞ ▲❡❣r❛♥❞ ❍♦✇ ❞♦❡s ✐t ✇♦r❦ ✐♥ ♦t❤❡r s❝✐❡♥❝❡s❄ ✼ ✴ ✶✼

6 Reproducible Research ‘11 Juliana Freire UBC, Vancouver

Science Today: Incomplete Publications

  Publications are just the tip of the iceberg

-  Scientific record is incomplete---

to large to fit in a paper

-  Large volumes of data

-  Complex processes

  Can’t (easily) reproduce results

❈♦✉rt❡s② ♦❢ ❏✉❧✐❛♥❛ ❋r❡✐r❡ ✭❆▼P ❲♦r❦s❤♦♣ ♦♥ ❘❡♣r♦❞✉❝✐❜❧❡ r❡s❡❛r❝❤✮

❆r♥❛✉❞ ▲❡❣r❛♥❞ ❍♦✇ ❞♦❡s ✐t ✇♦r❦ ✐♥ ♦t❤❡r s❝✐❡♥❝❡s❄ ✽ ✴ ✶✼

7 Reproducible Research ‘11 Juliana Freire UBC, Vancouver

Science Today: Incomplete Publications

  Publications are just the tip of the iceberg

-  Scientific record is incomplete---

to large to fit in a paper

-  Large volumes of data

-  Complex processes

  Can’t (easily) reproduce results

“It’s impossible to verify most of the results that computational scientists present at conference

and in papers.” [Donoho et al., 2009]

“Scientific and mathematical journals are filled

with pretty pictures of computational experiments

that the reader has no hope of repeating.” [LeVeque, 2009]

“Published documents are merely the

advertisement of scholarship whereas the

computer programs, input data, parameter

values, etc. embody the scholarship itself.” [Schwab et al., 2007]

❈♦✉rt❡s② ♦❢ ❏✉❧✐❛♥❛ ❋r❡✐r❡ ✭❆▼P ❲♦r❦s❤♦♣ ♦♥ ❘❡♣r♦❞✉❝✐❜❧❡ r❡s❡❛r❝❤✮

❆r♥❛✉❞ ▲❡❣r❛♥❞ ❍♦✇ ❞♦❡s ✐t ✇♦r❦ ✐♥ ♦t❤❡r s❝✐❡♥❝❡s❄ ✾ ✴ ✶✼

❊✈✐❞❡♥❝❡ ❢♦r ❛ ▲❛❝❦ ♦❢ ❘❡♣r♦❞✉❝✐❜✐❧✐t②

❙t✉❞✐❡s s❤♦✇✐♥❣ t❤❛t s❝✐❡♥t✐✜❝ ♣❛♣❡rs ❝♦♠♠♦♥❧② ❧❡❛✈❡ ♦✉t ❡①♣❡r✐♠❡♥t❛❧❞❡t❛✐❧s ❡ss❡♥t✐❛❧ ❢♦r r❡♣r♦❞✉❝t✐♦♥ ❛♥❞ s❤♦✇✐♥❣ ❞✐✣❝✉❧t✐❡s ✇✐t❤ r❡♣❧✐❝❛t✐♥❣♣✉❜❧✐s❤❡❞ ❡①♣❡r✐♠❡♥t❛❧ r❡s✉❧ts✿

J.P. Ioannidis. Why Most Published Research Findings Are False PLoSMed. 2005 August; 2(8)

❍✐❣❤ ♥✉♠❜❡r ♦❢ ❢❛✐❧✐♥❣ ❝❧✐♥✐❝❛❧ tr✐❛❧s✳

Do We Really Know What Makes Us Healthy?, New-York Times —September 16, 2007Lies, Damned Lies, and Medical Science, The Atlantic. 2010, Nov.

■♥❝r❡❛s❡ ✐♥ r❡tr❛❝t❡❞ ♣❛♣❡rs✿

Steen RG, Retractions in the scientific literature: is the incidence ofresearch fraud increasing? J Med Ethics 37: 249–253.

❈♦✉rt❡s② ♦❢ ❆❞❛♠ ❏✳ ❘✐❝❤❛r❞s❆r♥❛✉❞ ▲❡❣r❛♥❞ ❍♦✇ ❞♦❡s ✐t ✇♦r❦ ✐♥ ♦t❤❡r s❝✐❡♥❝❡s❄ ✶✵ ✴ ✶✼

❊①♣❡r✐♠❡♥t✐♥❣ ✐♥ t❤❡ ❲✐❧❞

❊①♣❡r✐♠❡♥ts ✐♥ ❍P❈✿ ❊✈❡♥ ❲♦rs❡✦

❘❡❧② ♦♥ ❧❛r❣❡✱ ❞✐str✐❜✉t❡❞✱ ❤②❜r✐❞✱ ♣r♦t♦t②♣❡ ❤❛r❞✇❛r❡✴s♦❢t✇❛r❡

▼❡❛s✉r❡ ❡①❡❝✉t✐♦♥ t✐♠❡s ✭♠❛❦❡s♣❛♥s✱ tr❛❝❡s✱ ✳ ✳ ✳ ✮

▼❛♥② ♣❛r❛♠❡t❡rs✱ ✈❡r② ❝♦st❧② ❛♥❞ ❤❛r❞ t♦ r❡♣r♦❞✉❝❡

❆r♥❛✉❞ ▲❡❣r❛♥❞ ❲❤❛t ❛❜♦✉t ✉s ❄ ✶✶ ✴ ✶✼

❘❡♣r♦❞✉❝✐♥❣ ✐♥ t❤❡ ❲✐❧❞

■♥ ♠② ❈♦♠♠✉♥✐t②

❘❡♣r♦❞✉❝✐♥❣ r❡s✉❧ts ♦❢ ♦t❤❡rs ✐s ♥♦t ✐♥ t❤❡ ❝✉❧t✉r❡

❘❡✇❛r❞s ❢♦r ♣♦s✐t✐✈❡ r❡s✉❧ts ❛♥❞ ♥♦✈❡❧t② r❛t❤❡r t❤❛♥ ❝♦♥s♦❧✐❞❛t✐♥❣ ❡①✐st✲✐♥❣ r❡s✉❧ts

❊①❝✉s❡ ★✶✿ ❍❛r❞✇❛r❡ ❛♥❞ s♦❢t✇❛r❡ ❡✈♦❧✈❡ ✈❡r② q✉✐❝❦❧②

❊①❝✉s❡ ★✷✿ ❘❡♣r♦❞✉❝✐❜❧❡ r❡s❡❛r❝❤ ✐s ♥♦t ♠❛♥❞❛t♦r②

❲❤② ✐s ✐t s♦ ❍❛r❞ ✐♥ ●❡♥❡r❛❧❄

❈♦♣②r✐❣❤t✴❝♦♠♣❡t✐t✐♦♥ ✐ss✉❡ ✭✇♦✉❧❞ ❣✐✈❡ ❛❞✈❛♥t❛❣❡ t♦ ✧♦t❤❡rs✧✮

P✉❜❧✐❝❛t✐♦♥ ❜✐❛s ✭♦♥❧② t❤❡ ✐❞❡❛ ♠❛tt❡rs✱ ♥♦t t❤❡ ❣♦r② ❞❡t❛✐❧s✮

❊①♣❡r✐♠❡♥t❡r ❜✐❛s

Pr♦❣r❛♠♠✐♥❣ ❡rr♦rs ♦r ❞❛t❛ ♠❛♥✐♣✉❧❛t✐♦♥✴❛♥❛❧②s✐s ♠✐st❛❦❡s

▲❛❝❦ ♦❢ ❡❛s②✲t♦✲✉s❡ t♦♦❧s

❲❤② ❜♦t❤❡r❄❆r♥❛✉❞ ▲❡❣r❛♥❞ ❲❤❛t ❛❜♦✉t ✉s ❄ ✶✷ ✴ ✶✼

❘❡♣r♦❞✉❝✐❜❧❡ ❘❡s❡❛r❝❤✿ ❚r②✐♥❣ t♦ ❇r✐❞❣❡ t❤❡ ●❛♣

❘❡❛❞❡r

❆✉t❤♦r

✭❉❡s✐❣♥ ♦❢ ❊①♣❡r✐♠❡♥ts✮

Pr♦t♦❝♦❧

❙❝✐❡♥t✐✜❝

◗✉❡st✐♦♥

P✉❜❧✐s❤❡❞

❆rt✐❝❧❡

◆❛t✉r❡✴❙②st❡♠✴✳✳✳

■♥s♣✐r❡❞ ❜② ❘♦❣❡r ❉✳ P❡♥❣✬s ❧❡❝t✉r❡ ♦♥ r❡♣r♦❞✉❝✐❜❧❡ r❡s❡❛r❝❤✱ ▼❛② ✷✵✶✹❆r♥❛✉❞ ▲❡❣r❛♥❞ ❇r✐❞❣✐♥❣ ❚❤❡ ●❛♣ ✶✸ ✴ ✶✼

❘❡♣r♦❞✉❝✐❜❧❡ ❘❡s❡❛r❝❤✿ ❚r②✐♥❣ t♦ ❇r✐❞❣❡ t❤❡ ●❛♣

❆♥❛❧②t✐❝

❉❛t❛

❈♦♠♣✉t❛t✐♦♥❛❧

❘❡s✉❧ts

▼❡❛s✉r❡❞

❉❛t❛

◆✉♠❡r✐❝❛❧

❙✉♠♠❛r✐❡s

❋✐❣✉r❡s

❚❛❜❧❡s

❚❡①t

❘❡❛❞❡r

❆✉t❤♦r

✭❉❡s✐❣♥ ♦❢ ❊①♣❡r✐♠❡♥ts✮

Pr♦t♦❝♦❧

❙❝✐❡♥t✐✜❝

◗✉❡st✐♦♥

P✉❜❧✐s❤❡❞

❆rt✐❝❧❡

◆❛t✉r❡✴❙②st❡♠✴✳✳✳

■♥s♣✐r❡❞ ❜② ❘♦❣❡r ❉✳ P❡♥❣✬s ❧❡❝t✉r❡ ♦♥ r❡♣r♦❞✉❝✐❜❧❡ r❡s❡❛r❝❤✱ ▼❛② ✷✵✶✹❆r♥❛✉❞ ▲❡❣r❛♥❞ ❇r✐❞❣✐♥❣ ❚❤❡ ●❛♣ ✶✸ ✴ ✶✼

❘❡♣r♦❞✉❝✐❜❧❡ ❘❡s❡❛r❝❤✿ ❚r②✐♥❣ t♦ ❇r✐❞❣❡ t❤❡ ●❛♣

❚r② t♦ ❦❡❡♣ tr❛❝❦ ♦❢ t❤❡ ✇❤♦❧❡ ❝❤❛✐♥

❊①♣❡r✐♠❡♥t ❈♦❞❡

✭✇♦r❦❧♦❛❞ ✐♥❥❡❝t♦r✱ ❱▼ r❡❝✐♣❡s✱ ✳✳✳✮

Pr♦❝❡ss✐♥❣

❈♦❞❡

❆♥❛❧②s✐s

❈♦❞❡

Pr❡s❡♥t❛t✐♦♥

❈♦❞❡

❆♥❛❧②t✐❝

❉❛t❛

❈♦♠♣✉t❛t✐♦♥❛❧

❘❡s✉❧ts

▼❡❛s✉r❡❞

❉❛t❛

◆✉♠❡r✐❝❛❧

❙✉♠♠❛r✐❡s

❋✐❣✉r❡s

❚❛❜❧❡s

❚❡①t

❘❡❛❞❡r

❆✉t❤♦r

✭❉❡s✐❣♥ ♦❢ ❊①♣❡r✐♠❡♥ts✮

Pr♦t♦❝♦❧

❙❝✐❡♥t✐✜❝

◗✉❡st✐♦♥

P✉❜❧✐s❤❡❞

❆rt✐❝❧❡

◆❛t✉r❡✴❙②st❡♠✴✳✳✳

■♥s♣✐r❡❞ ❜② ❘♦❣❡r ❉✳ P❡♥❣✬s ❧❡❝t✉r❡ ♦♥ r❡♣r♦❞✉❝✐❜❧❡ r❡s❡❛r❝❤✱ ▼❛② ✷✵✶✹❆r♥❛✉❞ ▲❡❣r❛♥❞ ❇r✐❞❣✐♥❣ ❚❤❡ ●❛♣ ✶✸ ✴ ✶✼

▼②t❤❜✉st❡rs✿ ❙❝✐❡♥❝❡ ✈s✳ ❙❝r❡✇✐♥❣ ❆r♦✉♥❞

❘❡♣r♦❞✉❝✐❜❧❡ ❘❡s❡❛r❝❤✿ ❚r②✐♥❣ t♦ ❇r✐❞❣❡ t❤❡ ●❛♣✧❚r✐❝❦②✧♣❛rt

✧❊❛s②✧ ♣❛rt

✧❚r✐❝❦②✧ ❛♥❞ ✧❊❛s②✧ r❡❢❡r t♦

♣❛r❛❧❧❡❧ ❝♦♠♣✉t❡r s❝✐❡♥t✐st ✉s❡ ❝❛s❡s

❊①♣❡r✐♠❡♥t ❈♦❞❡

✭✇♦r❦❧♦❛❞ ✐♥❥❡❝t♦r✱ ❱▼ r❡❝✐♣❡s✱ ✳✳✳✮

Pr♦❝❡ss✐♥❣

❈♦❞❡

❆♥❛❧②s✐s

❈♦❞❡

Pr❡s❡♥t❛t✐♦♥

❈♦❞❡

❆♥❛❧②t✐❝

❉❛t❛

❈♦♠♣✉t❛t✐♦♥❛❧

❘❡s✉❧ts

▼❡❛s✉r❡❞

❉❛t❛

◆✉♠❡r✐❝❛❧

❙✉♠♠❛r✐❡s

❋✐❣✉r❡s

❚❛❜❧❡s

❚❡①t

❘❡❛❞❡r

❆✉t❤♦r

✭❉❡s✐❣♥ ♦❢ ❊①♣❡r✐♠❡♥ts✮

Pr♦t♦❝♦❧

❙❝✐❡♥t✐✜❝

◗✉❡st✐♦♥

P✉❜❧✐s❤❡❞

❆rt✐❝❧❡

◆❛t✉r❡✴❙②st❡♠✴✳✳✳

■♥s♣✐r❡❞ ❜② ❘♦❣❡r ❉✳ P❡♥❣✬s ❧❡❝t✉r❡ ♦♥ r❡♣r♦❞✉❝✐❜❧❡ r❡s❡❛r❝❤✱ ▼❛② ✷✵✶✹❆r♥❛✉❞ ▲❡❣r❛♥❞ ❇r✐❞❣✐♥❣ ❚❤❡ ●❛♣ ✶✺ ✴ ✶✼

❙♦ ▼❛♥② ◆❡✇ ❚♦♦❧s

New Tools for Computational Reproducibility

• Dissemination Platforms:

• Workflow Tracking and Research Environments:

• Embedded Publishing:

VisTrails Kepler CDE

Galaxy GenePattern Synapse

Sumatra Taverna Pegasus

Verifiable Computational Research Sweave knitR

Collage Authoring Environment SHARE

ResearchCompendia.org IPOL Madagascar

MLOSS.org thedatahub.org nanoHUB.orgOpen Science Framework The DataVerse Network RunMyCode.org

❈♦✉rt❡s② ♦❢ ❱✐❝t♦r✐❛ ❙t♦❞❞❡♥ ✭❯❈ ❉❛✈✐s✱ ❋❡❜ ✶✸✱ ✷✵✶✹✮

❆♥❞ ❛❧s♦✿ ❋✐❣s❤❛r❡✴❩❡♥♦❞♦✱ ❆❝t✐✈❡P❛♣❡rs✱ ❊❧s❡✈✐❡r ❡①❡❝✉t❛❜❧❡ ♣❛♣❡r✱✳ ✳ ✳

❆r♥❛✉❞ ▲❡❣r❛♥❞ ▼❛♥② ❆❧t❡r♥❛t✐✈❡s ❢♦r ❘❡♣r♦❞✉❝✐❜❧❡ ❆♥❛❧②s✐s ✶✻ ✴ ✶✼

❘❡♣r♦❞✉❝✐❜❧❡ ❘❡s❡❛r❝❤✿ t❤❡ ◆❡✇ ❇✉③③✇♦r❞❄

❍✷✵✷✵✲❊■◆❋❘❆✲✷✵✶✹✲✷✵✶✺

❆ ❦❡② ❡❧❡♠❡♥t ✇✐❧❧ ❜❡ ❝❛♣❛❝✐t② ❜✉✐❧❞✐♥❣ t♦ ❧✐♥❦ ❧✐t❡r❛t✉r❡ ❛♥❞ ❞❛t❛

✐♥ ♦r❞❡r t♦ ❡♥❛❜❧❡ ❛ ♠♦r❡ tr❛♥s♣❛r❡♥t ❡✈❛❧✉❛t✐♦♥ ♦❢ r❡s❡❛r❝❤ ❛♥❞

r❡♣r♦❞✉❝✐❜✐❧✐t② ♦❢ r❡s✉❧ts✳

▼♦r❡ ❛♥❞ ♠♦r❡ ✇♦r❦s❤♦♣s

❲♦r❦s❤♦♣ ♦♥ ❉✉♣❧✐❝❛t✐♥❣✱ ❉❡❝♦♥str✉❝t✐♥❣ ❛♥❞ ❉❡❜✉♥❦✐♥❣ ✭❲❉❉❉✮ ✭✷✵✶✹ ❡❞✐t✐♦♥✮

❘❡♣r♦❞✉❝✐❜❧❡ ❘❡s❡❛r❝❤✿ ❚♦♦❧s ❛♥❞ ❙tr❛t❡❣✐❡s ❢♦r ❙❝✐❡♥t✐✜❝ ❈♦♠♣✉t✐♥❣✭✷✵✶✶✮

❲♦r❦✐♥❣ t♦✇❛r❞s ❙✉st❛✐♥❛❜❧❡ ❙♦❢t✇❛r❡ ❢♦r ❙❝✐❡♥❝❡✿ Pr❛❝t✐❝❡ ❛♥❞ ❊①♣❡r✐❡♥❝❡s ✭✷✵✶✸✮

❘❡♣r♦❞✉❝✐❜✐❧✐t②❅❳❙❊❉❊✿ ❆♥ ❳❙❊❉❊✶✹ ❲♦r❦s❤♦♣

❚❘❯❙❚ ✷✵✶✹

❘❊PP❆❘✬✶✺✿ ✷♥❞ ■♥t❡r♥❛t✐♦♥❛❧ ❲♦r❦s❤♦♣ ♦♥ ❘❡♣r♦❞✉❝✐❜✐❧✐t② ✐♥ P❛r❛❧❧❡❧ ❈♦♠♣✉t✐♥❣

❘❡♣r♦❞✉❝✐❜✐❧✐t② ♦❢ ❈♦♠♣✉t❛t✐♦♥ ❇❛s❡❞ ❘❡s❡❛r❝❤✴◆❚▼❙✬✶✺

❙❤♦✉❧❞ ❜❡ s❡❡♥ ❛s ♦♣♣♦rt✉♥✐t✐❡s t♦ s❤❛r❡ ❡①♣❡r✐❡♥❝❡✳

r❡❝❤❡r❝❤❡✲r❡♣r♦❞✉❝t✐❜❧❡❅❧✐st❡s✳✉♥✐✈✲♦r❧❡❛♥s✳❢r

❆r♥❛✉❞ ▲❡❣r❛♥❞ ▼❛♥② ❆❧t❡r♥❛t✐✈❡s ❢♦r ❘❡♣r♦❞✉❝✐❜❧❡ ❆♥❛❧②s✐s ✶✼ ✴ ✶✼