collaborative infrastructure for test-driven scientific ...comar/sciunit-icse14.pdf · test-driven...

Collaborative Infrastructure for Test-Driven Scientific Model Validation

Cyrus Omar + Jonathan Aldrich carnegie mellon university !Richard Gerkin arizona state university

Model Prediction

2

Empirical Observations

Observable Quantity

3

Science is Dynamic

4

Science is Dynamic

5

Science is Dynamic

Empirical Observations

Models Goodness-of-Fit

7

8

The state of the art?

Other [models] were examined and found promising but were not considered due to our inability to test the techniques”

“

9

10

papers ~ component documentationimplementations

models ~ software componentsscientific community ~ software development team

conformance testsinterfaces

11

models ~ software componentsmodel capabilities ~ interfaces

observable quantities ~ typesgoodness-of-fit tests ~ unit tests

empirical observations ~ test parameters

interface  SunspotNumberPredictor      predict  :  TimeSeries<SunspotNumber>  -­‐>  TimeSeries<SunspotNumber>

SunspotTest  :  TimeSeries<SunspotNumber>  -­‐>  Test<SunspotNumberPredictor,  Metric>  =  ...

TestSuite([SunspotTest(cycle_data)  for  cycle_data  in  all_cycle_data])  .judge([OhlsMethod,  FeynmansMethod,  ThompsonsMethod]).view()

OhlsMethod  FeynmansMethod  ThompsonsMethod          

summary tables ~ test result summaries

:  SunspotNumberPredictor  =  ...  :  SunspotNumberPredictor  =  ...  :  SunspotNumberPredictor  =  ...

12

interface  SunspotNumberPredictor      predict  :  TimeSeries<SunspotNumber>  -­‐>  TimeSeries<SunspotNumber>

SunspotTest  :  TimeSeries<SunspotNumber>  -­‐>  Test<SunspotNumberPredictor,  Metric>  =  ...

TestSuite([SunspotTest(cycle_data)  for  cycle_data  in  all_cycle_data])  .judge([OhlsMethod,  FeynmansMethod,  ThompsonsMethod]).view()

OhlsMethod  FeynmansMethod  ThompsonsMethod          

:  SunspotNumberPredictor  =  ...  :  SunspotNumberPredictor  =  ...  :  SunspotNumberPredictor  =  ...

which programming language? a DSL?Paper: SciUnit, a Python library (widely adopted, good FFI support, simple syntax, IPython notebook)

collaborative workflow?Paper: SciDash, a wiki pointing to IPython notebooks hosted on Github

adoption?Paper: people who organize modeling competitions (e.g. a neural modeling competition we are helping re-start)

Ongoing work: automatic test and model generation from existing repositoriesContact Me: cyrus@cmu.edu or @neurocy

Model Prediction

13

Empirical Observations

Observable Quantity

14

Test

Data Type

Model Prediction

Unit Tests

Models Goodness-of-Fit