keynote ieee international workshop on cloud analytics. dennis gannon
DESCRIPTION
This talk describes our experiences from hosting scientific research application in the Microsoft Cloud. Covers an overview of Microsoft Azure capabilities, examples of big data analysis for science, data collections, science gateways and science virtual machine libraries.TRANSCRIPT
![Page 1: Keynote IEEE International Workshop on Cloud Analytics. Dennis Gannon](https://reader033.vdocuments.us/reader033/viewer/2022051612/54c6329c4a7959a0338b45a2/html5/thumbnails/1.jpg)
![Page 2: Keynote IEEE International Workshop on Cloud Analytics. Dennis Gannon](https://reader033.vdocuments.us/reader033/viewer/2022051612/54c6329c4a7959a0338b45a2/html5/thumbnails/2.jpg)
![Page 3: Keynote IEEE International Workshop on Cloud Analytics. Dennis Gannon](https://reader033.vdocuments.us/reader033/viewer/2022051612/54c6329c4a7959a0338b45a2/html5/thumbnails/3.jpg)
Every research field is now a data science field
![Page 4: Keynote IEEE International Workshop on Cloud Analytics. Dennis Gannon](https://reader033.vdocuments.us/reader033/viewer/2022051612/54c6329c4a7959a0338b45a2/html5/thumbnails/4.jpg)
Last
few decades
Thousand
years ago
Today and the FutureLast few
hundred years
2
2
2.
3
4
a
cG
a
a
Simulation of
complex phenomena
Newton’s laws,
Maxwell’s equations…
Description of natural
phenomena
Unify theory, experiment and
simulation with large
multidisciplinary Data
Using data exploration and
data mining
(from instruments, sensors,
humans…)
Distributed Communities
![Page 5: Keynote IEEE International Workshop on Cloud Analytics. Dennis Gannon](https://reader033.vdocuments.us/reader033/viewer/2022051612/54c6329c4a7959a0338b45a2/html5/thumbnails/5.jpg)
![Page 6: Keynote IEEE International Workshop on Cloud Analytics. Dennis Gannon](https://reader033.vdocuments.us/reader033/viewer/2022051612/54c6329c4a7959a0338b45a2/html5/thumbnails/6.jpg)
![Page 7: Keynote IEEE International Workshop on Cloud Analytics. Dennis Gannon](https://reader033.vdocuments.us/reader033/viewer/2022051612/54c6329c4a7959a0338b45a2/html5/thumbnails/7.jpg)
7
Melbourne
Sydney
![Page 8: Keynote IEEE International Workshop on Cloud Analytics. Dennis Gannon](https://reader033.vdocuments.us/reader033/viewer/2022051612/54c6329c4a7959a0338b45a2/html5/thumbnails/8.jpg)
IT PAC
![Page 9: Keynote IEEE International Workshop on Cloud Analytics. Dennis Gannon](https://reader033.vdocuments.us/reader033/viewer/2022051612/54c6329c4a7959a0338b45a2/html5/thumbnails/9.jpg)
![Page 10: Keynote IEEE International Workshop on Cloud Analytics. Dennis Gannon](https://reader033.vdocuments.us/reader033/viewer/2022051612/54c6329c4a7959a0338b45a2/html5/thumbnails/10.jpg)
applicationbuilding
blocks
![Page 11: Keynote IEEE International Workshop on Cloud Analytics. Dennis Gannon](https://reader033.vdocuments.us/reader033/viewer/2022051612/54c6329c4a7959a0338b45a2/html5/thumbnails/11.jpg)
![Page 12: Keynote IEEE International Workshop on Cloud Analytics. Dennis Gannon](https://reader033.vdocuments.us/reader033/viewer/2022051612/54c6329c4a7959a0338b45a2/html5/thumbnails/12.jpg)
![Page 13: Keynote IEEE International Workshop on Cloud Analytics. Dennis Gannon](https://reader033.vdocuments.us/reader033/viewer/2022051612/54c6329c4a7959a0338b45a2/html5/thumbnails/13.jpg)
![Page 14: Keynote IEEE International Workshop on Cloud Analytics. Dennis Gannon](https://reader033.vdocuments.us/reader033/viewer/2022051612/54c6329c4a7959a0338b45a2/html5/thumbnails/14.jpg)
![Page 15: Keynote IEEE International Workshop on Cloud Analytics. Dennis Gannon](https://reader033.vdocuments.us/reader033/viewer/2022051612/54c6329c4a7959a0338b45a2/html5/thumbnails/15.jpg)
![Page 16: Keynote IEEE International Workshop on Cloud Analytics. Dennis Gannon](https://reader033.vdocuments.us/reader033/viewer/2022051612/54c6329c4a7959a0338b45a2/html5/thumbnails/16.jpg)
![Page 17: Keynote IEEE International Workshop on Cloud Analytics. Dennis Gannon](https://reader033.vdocuments.us/reader033/viewer/2022051612/54c6329c4a7959a0338b45a2/html5/thumbnails/17.jpg)
![Page 18: Keynote IEEE International Workshop on Cloud Analytics. Dennis Gannon](https://reader033.vdocuments.us/reader033/viewer/2022051612/54c6329c4a7959a0338b45a2/html5/thumbnails/18.jpg)
![Page 19: Keynote IEEE International Workshop on Cloud Analytics. Dennis Gannon](https://reader033.vdocuments.us/reader033/viewer/2022051612/54c6329c4a7959a0338b45a2/html5/thumbnails/19.jpg)
![Page 20: Keynote IEEE International Workshop on Cloud Analytics. Dennis Gannon](https://reader033.vdocuments.us/reader033/viewer/2022051612/54c6329c4a7959a0338b45a2/html5/thumbnails/20.jpg)
Web Portal
UserBrowser
Task Queue2
1
Executable / Data(Windows Azure Storage)
Compute Nodes
3
5
4
6
7
![Page 21: Keynote IEEE International Workshop on Cloud Analytics. Dennis Gannon](https://reader033.vdocuments.us/reader033/viewer/2022051612/54c6329c4a7959a0338b45a2/html5/thumbnails/21.jpg)
![Page 23: Keynote IEEE International Workshop on Cloud Analytics. Dennis Gannon](https://reader033.vdocuments.us/reader033/viewer/2022051612/54c6329c4a7959a0338b45a2/html5/thumbnails/23.jpg)
![Page 24: Keynote IEEE International Workshop on Cloud Analytics. Dennis Gannon](https://reader033.vdocuments.us/reader033/viewer/2022051612/54c6329c4a7959a0338b45a2/html5/thumbnails/24.jpg)
![Page 25: Keynote IEEE International Workshop on Cloud Analytics. Dennis Gannon](https://reader033.vdocuments.us/reader033/viewer/2022051612/54c6329c4a7959a0338b45a2/html5/thumbnails/25.jpg)
• Many Examples
• The Challenge: sustainability
• Manage locality• Keep the hot data local on cloud disk
• Manage the working set over time
• The rest is archival
Data
Acquisition &
modelling
Collaboration
and
visualisation
Analysis &
data mining
Dissemination
& sharing
Archiving and
preserving
![Page 26: Keynote IEEE International Workshop on Cloud Analytics. Dennis Gannon](https://reader033.vdocuments.us/reader033/viewer/2022051612/54c6329c4a7959a0338b45a2/html5/thumbnails/26.jpg)
![Page 27: Keynote IEEE International Workshop on Cloud Analytics. Dennis Gannon](https://reader033.vdocuments.us/reader033/viewer/2022051612/54c6329c4a7959a0338b45a2/html5/thumbnails/27.jpg)
• A core library for science in the cloud.
• Built on community tools• Ipython Notebook, Python, NumPy,
SciPy, Scikit-Learn, biopython
• Standard community tools
• Deploy as VM library
• Deploy data collections.• Genomic libraries, medical image
libraries, geophysics, astronomy, etc.
• Build community resources
![Page 28: Keynote IEEE International Workshop on Cloud Analytics. Dennis Gannon](https://reader033.vdocuments.us/reader033/viewer/2022051612/54c6329c4a7959a0338b45a2/html5/thumbnails/28.jpg)
![Page 29: Keynote IEEE International Workshop on Cloud Analytics. Dennis Gannon](https://reader033.vdocuments.us/reader033/viewer/2022051612/54c6329c4a7959a0338b45a2/html5/thumbnails/29.jpg)
![Page 30: Keynote IEEE International Workshop on Cloud Analytics. Dennis Gannon](https://reader033.vdocuments.us/reader033/viewer/2022051612/54c6329c4a7959a0338b45a2/html5/thumbnails/30.jpg)
![Page 31: Keynote IEEE International Workshop on Cloud Analytics. Dennis Gannon](https://reader033.vdocuments.us/reader033/viewer/2022051612/54c6329c4a7959a0338b45a2/html5/thumbnails/31.jpg)
![Page 32: Keynote IEEE International Workshop on Cloud Analytics. Dennis Gannon](https://reader033.vdocuments.us/reader033/viewer/2022051612/54c6329c4a7959a0338b45a2/html5/thumbnails/32.jpg)
• The Genetic Causes of Disease
(David Heckerman)
• Wellcome Trust for a GWAS for a large
population
• Looking for causes for seven common
diseases (bipolar, r. arthritis, coronary,
hypertension, ….)
• Confounding is a problem. Needed a
new algorithm.
• Ran on Azure cloud using 35,000 cores
in 3 weeks.
![Page 33: Keynote IEEE International Workshop on Cloud Analytics. Dennis Gannon](https://reader033.vdocuments.us/reader033/viewer/2022051612/54c6329c4a7959a0338b45a2/html5/thumbnails/33.jpg)
Inputs (training data)
Labels
Hidden layers
Input dataDetected featuresMona Lisa
![Page 34: Keynote IEEE International Workshop on Cloud Analytics. Dennis Gannon](https://reader033.vdocuments.us/reader033/viewer/2022051612/54c6329c4a7959a0338b45a2/html5/thumbnails/34.jpg)
![Page 35: Keynote IEEE International Workshop on Cloud Analytics. Dennis Gannon](https://reader033.vdocuments.us/reader033/viewer/2022051612/54c6329c4a7959a0338b45a2/html5/thumbnails/35.jpg)
![Page 36: Keynote IEEE International Workshop on Cloud Analytics. Dennis Gannon](https://reader033.vdocuments.us/reader033/viewer/2022051612/54c6329c4a7959a0338b45a2/html5/thumbnails/36.jpg)
The Windows Azure for Research program:
·Free access to Windows Azure cloud computing and storage
(submit proposals for Windows Azure Research Awards)
· Windows Azure for Research training classes (20 classes
worldwide. )
· Support and technical resources
azure4research.com.
![Page 37: Keynote IEEE International Workshop on Cloud Analytics. Dennis Gannon](https://reader033.vdocuments.us/reader033/viewer/2022051612/54c6329c4a7959a0338b45a2/html5/thumbnails/37.jpg)
![Page 38: Keynote IEEE International Workshop on Cloud Analytics. Dennis Gannon](https://reader033.vdocuments.us/reader033/viewer/2022051612/54c6329c4a7959a0338b45a2/html5/thumbnails/38.jpg)