cockerill rs350-day3-what-can-be-done
TRANSCRIPT
![Page 1: Cockerill rs350-day3-what-can-be-done](https://reader030.vdocuments.us/reader030/viewer/2022032513/55d2a2bebb61ebdf048b467b/html5/thumbnails/1.jpg)
Matthew Cockerill, Co-Founder, Riffyn Incwww.riffyn.com
What can be done to improve reproducibility?
The future of scientific scholarly communication, Royal Society, 5th May 2015
![Page 2: Cockerill rs350-day3-what-can-be-done](https://reader030.vdocuments.us/reader030/viewer/2022032513/55d2a2bebb61ebdf048b467b/html5/thumbnails/2.jpg)
Reproducibility issues begin upstream of publication
• Biomedical research still has more in common with a “blacksmith shop” than a reliable modern process engineered for reliability
• Essential methodological knowledge is implicit and undocumented
• To make things work “spend time in the relevant lab”
• Technology transfer too often fails• If we want published results to be more
reliable, we need to to change the approach to methodology
![Page 3: Cockerill rs350-day3-what-can-be-done](https://reader030.vdocuments.us/reader030/viewer/2022032513/55d2a2bebb61ebdf048b467b/html5/thumbnails/3.jpg)
Reproducibility issues begin upstream of publication
• Biomedical research still has more in common with a “blacksmith shop” than a reliable modern process engineered for reliability
• Essential methodological knowledge is implicit and undocumented
• To make things work “spend time in the relevant lab”
• Technology transfer too often fails• If we want published results to be
more reliable, we need to to change the approach to methodology
![Page 4: Cockerill rs350-day3-what-can-be-done](https://reader030.vdocuments.us/reader030/viewer/2022032513/55d2a2bebb61ebdf048b467b/html5/thumbnails/4.jpg)
In industry, the costs of haziness are clear
Certainly it is a problem that we don’t write down our
processes - people change what they know, not what
necessarily matters. - Head of Process
Development of a biopharma company
When we first transferred our manufacturing process to full-
scale operations, our yields dropped 90%.
- CEO of a diagnostics company
We lost four months tracking-down a problem at the demo
scale. Root cause was a parameter change that a new
employee thought did not matter.
- VP Process Development of an algae company
Below: Process design sent to manufacturing
![Page 5: Cockerill rs350-day3-what-can-be-done](https://reader030.vdocuments.us/reader030/viewer/2022032513/55d2a2bebb61ebdf048b467b/html5/thumbnails/5.jpg)
Industrial R&D seeks to improve processes
Improve
Design
Analyse
Measure
![Page 6: Cockerill rs350-day3-what-can-be-done](https://reader030.vdocuments.us/reader030/viewer/2022032513/55d2a2bebb61ebdf048b467b/html5/thumbnails/6.jpg)
Industrial R&D seeks to improve processes
Improve
Design
Analyse
Measure
Noise and irreproducibility gets in the way
![Page 7: Cockerill rs350-day3-what-can-be-done](https://reader030.vdocuments.us/reader030/viewer/2022032513/55d2a2bebb61ebdf048b467b/html5/thumbnails/7.jpg)
Insights from manufacturing
Quality-oriented approachDesign• Unambiguous CAD files• Manage complexity via
modularity
Standardize & automate • Reduce variability in execution
Continuous improvement• Use analytics to identify and
resolve remaining causes of noise
Change of mindset • Make metrics transparent to
entire organization is focused on quality
![Page 8: Cockerill rs350-day3-what-can-be-done](https://reader030.vdocuments.us/reader030/viewer/2022032513/55d2a2bebb61ebdf048b467b/html5/thumbnails/8.jpg)
Insights from manufacturing
How can this benefit R&D?• It means you can build reliably on your own results, and those of others• But needs to be more flexible – in R&D processes change all the time!
Quality-oriented approachDesign• Unambiguous CAD files• Manage complexity via
modularity
Standardize & automate • Reduce variability in execution
Continuous improvement• Use analytics to identify and
resolve remaining causes of noise
Change of mindset • Make metrics transparent so
entire organization is focused on quality
![Page 9: Cockerill rs350-day3-what-can-be-done](https://reader030.vdocuments.us/reader030/viewer/2022032513/55d2a2bebb61ebdf048b467b/html5/thumbnails/9.jpg)
Examples of application in real world R&D
![Page 10: Cockerill rs350-day3-what-can-be-done](https://reader030.vdocuments.us/reader030/viewer/2022032513/55d2a2bebb61ebdf048b467b/html5/thumbnails/10.jpg)
Reducing noise allows more rapid progress
Reproduced from Gardner TS, Trends in Biotechnology, March 2013, Vol. 31, No. 3
Data from product development at leading biotech firm
Quality improvement cuts error by 6X. R&D productivity doubles overnight.Fe
rmen
tatio
n pr
oces
s yi
eld
Date
trend lines
30% relative error 5% relative error
![Page 11: Cockerill rs350-day3-what-can-be-done](https://reader030.vdocuments.us/reader030/viewer/2022032513/55d2a2bebb61ebdf048b467b/html5/thumbnails/11.jpg)
Analytics to track down root cause of noiseH
ow it
wor
ks
Define process inputs, outputs and critical variables1
Measure inputs and outputs2
Stra
in H
T sc
reen
ing
scor
e
Tray #
Analyse and improve3
![Page 12: Cockerill rs350-day3-what-can-be-done](https://reader030.vdocuments.us/reader030/viewer/2022032513/55d2a2bebb61ebdf048b467b/html5/thumbnails/12.jpg)
How do we generalize to basic research?
![Page 13: Cockerill rs350-day3-what-can-be-done](https://reader030.vdocuments.us/reader030/viewer/2022032513/55d2a2bebb61ebdf048b467b/html5/thumbnails/13.jpg)
New generation of lab software tools
Make experiments more reproducible via:• Cloud-based software• CAD approach to experimental design • Integrated data-stream capture and
analysis• Sharing and versioning of methodology
via new publication outlets with github-like approaches
![Page 14: Cockerill rs350-day3-what-can-be-done](https://reader030.vdocuments.us/reader030/viewer/2022032513/55d2a2bebb61ebdf048b467b/html5/thumbnails/14.jpg)
Parallel from cloud computing
• Traditional ad hoc approach to configuring and managing IT systems unreliable and unscalable
• Configuration management systems now use automated ‘recipes’ which reproducibly deliver a fully-configured virtual server which behaves the same every time
![Page 15: Cockerill rs350-day3-what-can-be-done](https://reader030.vdocuments.us/reader030/viewer/2022032513/55d2a2bebb61ebdf048b467b/html5/thumbnails/15.jpg)
Reusable workflows are well established in computational science
We need to move this to the lab bench
Taverna Kepler
Galaxy
![Page 16: Cockerill rs350-day3-what-can-be-done](https://reader030.vdocuments.us/reader030/viewer/2022032513/55d2a2bebb61ebdf048b467b/html5/thumbnails/16.jpg)
New approaches are starting to bring similar automation to the lab bench
![Page 17: Cockerill rs350-day3-what-can-be-done](https://reader030.vdocuments.us/reader030/viewer/2022032513/55d2a2bebb61ebdf048b467b/html5/thumbnails/17.jpg)
So what does the future look like?
![Page 18: Cockerill rs350-day3-what-can-be-done](https://reader030.vdocuments.us/reader030/viewer/2022032513/55d2a2bebb61ebdf048b467b/html5/thumbnails/18.jpg)
Standards for structured, interoperable data
Software and tools for lab automation/analytics
Hubs supporting sharing of data and methodology
An open ecosystem for reproducible science
![Page 19: Cockerill rs350-day3-what-can-be-done](https://reader030.vdocuments.us/reader030/viewer/2022032513/55d2a2bebb61ebdf048b467b/html5/thumbnails/19.jpg)
How will we make it happen?
![Page 20: Cockerill rs350-day3-what-can-be-done](https://reader030.vdocuments.us/reader030/viewer/2022032513/55d2a2bebb61ebdf048b467b/html5/thumbnails/20.jpg)
Incentives to drive adoption of new approach
In industry• Measurable improvements to R&D productivity
create powerful financial incentives
In academia • Tools will gain adoption if they make
researchers life easier and more productive• Endorsement and encouragement from funders• Requirement from participating journals for
sharing of experimental process descriptions• Metrics that reward the sharing of reusable
protocols
![Page 21: Cockerill rs350-day3-what-can-be-done](https://reader030.vdocuments.us/reader030/viewer/2022032513/55d2a2bebb61ebdf048b467b/html5/thumbnails/21.jpg)
Cultural shift won’t be easy, but it is needed
Change in the lab requires corresponding change in publishing
![Page 22: Cockerill rs350-day3-what-can-be-done](https://reader030.vdocuments.us/reader030/viewer/2022032513/55d2a2bebb61ebdf048b467b/html5/thumbnails/22.jpg)