big data - ncl.ac.uk analytics - prof wa… · world-leading research cloud computing for big data...
TRANSCRIPT
![Page 1: Big Data - ncl.ac.uk Analytics - Prof Wa… · World-leading research Cloud Computing for Big Data CDT Developing next generation of leaders Cloud Innovation Centre Engagement and](https://reader035.vdocuments.us/reader035/viewer/2022081614/5fc6af8e341bc843373f087d/html5/thumbnails/1.jpg)
Big Data
Paul Watson, Hugo Hiden, Simon Woodman, Mike Simpson
Digital Institute
Newcastle University
UK
![Page 2: Big Data - ncl.ac.uk Analytics - Prof Wa… · World-leading research Cloud Computing for Big Data CDT Developing next generation of leaders Cloud Innovation Centre Engagement and](https://reader035.vdocuments.us/reader035/viewer/2022081614/5fc6af8e341bc843373f087d/html5/thumbnails/2.jpg)
ICL Goldrush MegaServer 1995
• important applications
![Page 3: Big Data - ncl.ac.uk Analytics - Prof Wa… · World-leading research Cloud Computing for Big Data CDT Developing next generation of leaders Cloud Innovation Centre Engagement and](https://reader035.vdocuments.us/reader035/viewer/2022081614/5fc6af8e341bc843373f087d/html5/thumbnails/3.jpg)
![Page 4: Big Data - ncl.ac.uk Analytics - Prof Wa… · World-leading research Cloud Computing for Big Data CDT Developing next generation of leaders Cloud Innovation Centre Engagement and](https://reader035.vdocuments.us/reader035/viewer/2022081614/5fc6af8e341bc843373f087d/html5/thumbnails/4.jpg)
![Page 5: Big Data - ncl.ac.uk Analytics - Prof Wa… · World-leading research Cloud Computing for Big Data CDT Developing next generation of leaders Cloud Innovation Centre Engagement and](https://reader035.vdocuments.us/reader035/viewer/2022081614/5fc6af8e341bc843373f087d/html5/thumbnails/5.jpg)
![Page 6: Big Data - ncl.ac.uk Analytics - Prof Wa… · World-leading research Cloud Computing for Big Data CDT Developing next generation of leaders Cloud Innovation Centre Engagement and](https://reader035.vdocuments.us/reader035/viewer/2022081614/5fc6af8e341bc843373f087d/html5/thumbnails/6.jpg)
Challenge
Data Value?
![Page 7: Big Data - ncl.ac.uk Analytics - Prof Wa… · World-leading research Cloud Computing for Big Data CDT Developing next generation of leaders Cloud Innovation Centre Engagement and](https://reader035.vdocuments.us/reader035/viewer/2022081614/5fc6af8e341bc843373f087d/html5/thumbnails/7.jpg)
PhD
Stats CS
Problem
![Page 8: Big Data - ncl.ac.uk Analytics - Prof Wa… · World-leading research Cloud Computing for Big Data CDT Developing next generation of leaders Cloud Innovation Centre Engagement and](https://reader035.vdocuments.us/reader035/viewer/2022081614/5fc6af8e341bc843373f087d/html5/thumbnails/8.jpg)
InnovationCycle
World-leadingresearch
Cloud Computing for Big Data CDT Developing next generation of leaders
Cloud Innovation CentreEngagement andPartnershipsNational Institute for Smart Data Analytics
![Page 9: Big Data - ncl.ac.uk Analytics - Prof Wa… · World-leading research Cloud Computing for Big Data CDT Developing next generation of leaders Cloud Innovation Centre Engagement and](https://reader035.vdocuments.us/reader035/viewer/2022081614/5fc6af8e341bc843373f087d/html5/thumbnails/9.jpg)
Example Application AreasChemical Informatics Bioinformatics Materials
Dementia Diabetes Stroke
Transport Neuroscience Architecture
![Page 10: Big Data - ncl.ac.uk Analytics - Prof Wa… · World-leading research Cloud Computing for Big Data CDT Developing next generation of leaders Cloud Innovation Centre Engagement and](https://reader035.vdocuments.us/reader035/viewer/2022081614/5fc6af8e341bc843373f087d/html5/thumbnails/10.jpg)
How can we transform rehabilitation?
16 million people a
year suffer a stroke
![Page 11: Big Data - ncl.ac.uk Analytics - Prof Wa… · World-leading research Cloud Computing for Big Data CDT Developing next generation of leaders Cloud Innovation Centre Engagement and](https://reader035.vdocuments.us/reader035/viewer/2022081614/5fc6af8e341bc843373f087d/html5/thumbnails/11.jpg)
![Page 12: Big Data - ncl.ac.uk Analytics - Prof Wa… · World-leading research Cloud Computing for Big Data CDT Developing next generation of leaders Cloud Innovation Centre Engagement and](https://reader035.vdocuments.us/reader035/viewer/2022081614/5fc6af8e341bc843373f087d/html5/thumbnails/12.jpg)
![Page 13: Big Data - ncl.ac.uk Analytics - Prof Wa… · World-leading research Cloud Computing for Big Data CDT Developing next generation of leaders Cloud Innovation Centre Engagement and](https://reader035.vdocuments.us/reader035/viewer/2022081614/5fc6af8e341bc843373f087d/html5/thumbnails/13.jpg)
Infrastructure:
Storage & Compute
App 1
....
App n
![Page 14: Big Data - ncl.ac.uk Analytics - Prof Wa… · World-leading research Cloud Computing for Big Data CDT Developing next generation of leaders Cloud Innovation Centre Engagement and](https://reader035.vdocuments.us/reader035/viewer/2022081614/5fc6af8e341bc843373f087d/html5/thumbnails/14.jpg)
Infrastructure:
Storage & Compute
Science Platform:Store, Share, Analyse
App 1 .... App n
App 1
....
App n
Infrastructure:
Storage & Compute
![Page 15: Big Data - ncl.ac.uk Analytics - Prof Wa… · World-leading research Cloud Computing for Big Data CDT Developing next generation of leaders Cloud Innovation Centre Engagement and](https://reader035.vdocuments.us/reader035/viewer/2022081614/5fc6af8e341bc843373f087d/html5/thumbnails/15.jpg)
App ....
Workflow Enactment
API
Social Networking
Security
Processing Storage
App
Analysis Services
Infrastructure:Amazon, Microsoft, OpenShift & Private Clouds
Provenance/
Audit
Metadata<expt>9127</expt>
<smiles>CHC3</smiles>
![Page 16: Big Data - ncl.ac.uk Analytics - Prof Wa… · World-leading research Cloud Computing for Big Data CDT Developing next generation of leaders Cloud Innovation Centre Engagement and](https://reader035.vdocuments.us/reader035/viewer/2022081614/5fc6af8e341bc843373f087d/html5/thumbnails/16.jpg)
Demo
![Page 17: Big Data - ncl.ac.uk Analytics - Prof Wa… · World-leading research Cloud Computing for Big Data CDT Developing next generation of leaders Cloud Innovation Centre Engagement and](https://reader035.vdocuments.us/reader035/viewer/2022081614/5fc6af8e341bc843373f087d/html5/thumbnails/17.jpg)
![Page 18: Big Data - ncl.ac.uk Analytics - Prof Wa… · World-leading research Cloud Computing for Big Data CDT Developing next generation of leaders Cloud Innovation Centre Engagement and](https://reader035.vdocuments.us/reader035/viewer/2022081614/5fc6af8e341bc843373f087d/html5/thumbnails/18.jpg)
![Page 19: Big Data - ncl.ac.uk Analytics - Prof Wa… · World-leading research Cloud Computing for Big Data CDT Developing next generation of leaders Cloud Innovation Centre Engagement and](https://reader035.vdocuments.us/reader035/viewer/2022081614/5fc6af8e341bc843373f087d/html5/thumbnails/19.jpg)
![Page 20: Big Data - ncl.ac.uk Analytics - Prof Wa… · World-leading research Cloud Computing for Big Data CDT Developing next generation of leaders Cloud Innovation Centre Engagement and](https://reader035.vdocuments.us/reader035/viewer/2022081614/5fc6af8e341bc843373f087d/html5/thumbnails/20.jpg)
Racks in a Google Datacentre
![Page 21: Big Data - ncl.ac.uk Analytics - Prof Wa… · World-leading research Cloud Computing for Big Data CDT Developing next generation of leaders Cloud Innovation Centre Engagement and](https://reader035.vdocuments.us/reader035/viewer/2022081614/5fc6af8e341bc843373f087d/html5/thumbnails/21.jpg)
What’s New?
• illusion of infinite computing resources on-demand
• no up-front commitment by users
• Pay for use of resources on a short-term basisas needed
(from “Above the Clouds: A Berkeley View of Cloud Computing”)
![Page 22: Big Data - ncl.ac.uk Analytics - Prof Wa… · World-leading research Cloud Computing for Big Data CDT Developing next generation of leaders Cloud Innovation Centre Engagement and](https://reader035.vdocuments.us/reader035/viewer/2022081614/5fc6af8e341bc843373f087d/html5/thumbnails/22.jpg)
The Promise of Clouds
• Agility
• CapEx OpEx
• Scalability
• Energy Reduction
![Page 23: Big Data - ncl.ac.uk Analytics - Prof Wa… · World-leading research Cloud Computing for Big Data CDT Developing next generation of leaders Cloud Innovation Centre Engagement and](https://reader035.vdocuments.us/reader035/viewer/2022081614/5fc6af8e341bc843373f087d/html5/thumbnails/23.jpg)
100.0%
99.0%
95.5%
92.5%
90.5%
0
50
100
150
200
250
0 100 200 300
Rel
ativ
e sp
eed
-up
Microsoft Cloud Nodes
460K workflow executions4.4M service calls5yrs 10 hours
![Page 24: Big Data - ncl.ac.uk Analytics - Prof Wa… · World-leading research Cloud Computing for Big Data CDT Developing next generation of leaders Cloud Innovation Centre Engagement and](https://reader035.vdocuments.us/reader035/viewer/2022081614/5fc6af8e341bc843373f087d/html5/thumbnails/24.jpg)
Provenance / Audit
![Page 25: Big Data - ncl.ac.uk Analytics - Prof Wa… · World-leading research Cloud Computing for Big Data CDT Developing next generation of leaders Cloud Innovation Centre Engagement and](https://reader035.vdocuments.us/reader035/viewer/2022081614/5fc6af8e341bc843373f087d/html5/thumbnails/25.jpg)
IoT
• High-level description of computation on
streaming data – dynamically mapped
onto platforms
![Page 26: Big Data - ncl.ac.uk Analytics - Prof Wa… · World-leading research Cloud Computing for Big Data CDT Developing next generation of leaders Cloud Innovation Centre Engagement and](https://reader035.vdocuments.us/reader035/viewer/2022081614/5fc6af8e341bc843373f087d/html5/thumbnails/26.jpg)
IoT System Overview
Optimiser
RequirementsSecurity, Dependability,
Performance, Energy, Price,Psychology, Precision
CatalogueServers, Devices, Networks,
Sensors, Deployable Services
Description of the Computation
Real-Time MonitoringPerformanceData Rates
Energy
DeploymentPlan
Deploy
![Page 27: Big Data - ncl.ac.uk Analytics - Prof Wa… · World-leading research Cloud Computing for Big Data CDT Developing next generation of leaders Cloud Innovation Centre Engagement and](https://reader035.vdocuments.us/reader035/viewer/2022081614/5fc6af8e341bc843373f087d/html5/thumbnails/27.jpg)
Our CESI Role
• Data and model sharing within & outside the
project
– using the e-Science Central platform
– Integration with the UKERC energy data centre
• Data analysis
– high scalability through cloud computing
• Data visualisation and exploration
– e.g. in Newcastle’s 3D Decision Theatres
– also provides compelling method of engagement
![Page 28: Big Data - ncl.ac.uk Analytics - Prof Wa… · World-leading research Cloud Computing for Big Data CDT Developing next generation of leaders Cloud Innovation Centre Engagement and](https://reader035.vdocuments.us/reader035/viewer/2022081614/5fc6af8e341bc843373f087d/html5/thumbnails/28.jpg)
Pathways to Impact for CESI?
![Page 29: Big Data - ncl.ac.uk Analytics - Prof Wa… · World-leading research Cloud Computing for Big Data CDT Developing next generation of leaders Cloud Innovation Centre Engagement and](https://reader035.vdocuments.us/reader035/viewer/2022081614/5fc6af8e341bc843373f087d/html5/thumbnails/29.jpg)
Industry Impact
• e-Science Central is open source
– now used in industry
![Page 30: Big Data - ncl.ac.uk Analytics - Prof Wa… · World-leading research Cloud Computing for Big Data CDT Developing next generation of leaders Cloud Innovation Centre Engagement and](https://reader035.vdocuments.us/reader035/viewer/2022081614/5fc6af8e341bc843373f087d/html5/thumbnails/30.jpg)
Impact: Cloud Innovation Centre National Institute for Smart Data Innovation
![Page 31: Big Data - ncl.ac.uk Analytics - Prof Wa… · World-leading research Cloud Computing for Big Data CDT Developing next generation of leaders Cloud Innovation Centre Engagement and](https://reader035.vdocuments.us/reader035/viewer/2022081614/5fc6af8e341bc843373f087d/html5/thumbnails/31.jpg)
National Institute for Smart Data Innovation
• £30M investment• £15M Treasury + £15M Newcastle University
• This will fund• new “beacon” building on Science Central (2019)
• ten years of activities with industry & the public sector
![Page 32: Big Data - ncl.ac.uk Analytics - Prof Wa… · World-leading research Cloud Computing for Big Data CDT Developing next generation of leaders Cloud Innovation Centre Engagement and](https://reader035.vdocuments.us/reader035/viewer/2022081614/5fc6af8e341bc843373f087d/html5/thumbnails/32.jpg)
• The National Institute for Smart Data Innovation (NISDI) will enable industry to unleash the huge potential for innovation offered by the explosive growth in digital data.
• Once processed and analysed, digital data can become “Smart Data”, enabling new products and services that greatly benefit the country’s economy and its citizens. However, a major skills gap is preventing the UK from realising the potential of Smart Data.
• NISDI will overcome this barrier by creating a unique new facility which brings together industry, the public sector and universities to create the skills, ideas and resources needed to exploit these opportunities.
• Driven by the needs of industry, NISDI will allow the region, the Northern Powerhouse and the UK to become global leaders in this important sector, estimated to be worth around $125bn per annum.
• The National Institute for Smart Data Innovation (NISDI) will enable industry to unleash the huge potential for innovation offered by the explosive growth in digital data.
• Once processed and analysed, digital data can become “Smart Data”, enabling new products and services that greatly benefit the country’s economy and its citizens. However, a major skills gap is preventing the UK from realising the potential of Smart Data.
• NISDI will overcome this barrier by creating a unique new facility which brings together industry, the public sector and universities to create the skills, ideas and resources needed to exploit these opportunities.
• Driven by the needs of industry, NISDI will allow the region, the Northern Powerhouse and the UK to become global leaders in this important sector, estimated to be worth around $125bn per annum.
![Page 33: Big Data - ncl.ac.uk Analytics - Prof Wa… · World-leading research Cloud Computing for Big Data CDT Developing next generation of leaders Cloud Innovation Centre Engagement and](https://reader035.vdocuments.us/reader035/viewer/2022081614/5fc6af8e341bc843373f087d/html5/thumbnails/33.jpg)
NISDI Themes
Health care SmartCities
Automotive BIM Manufactur-ing
Others e.g. Subsea
Internet of Things
Data Analytics & Visualisation
Cloud Computing
Other technologies – e.g. cyber security
![Page 34: Big Data - ncl.ac.uk Analytics - Prof Wa… · World-leading research Cloud Computing for Big Data CDT Developing next generation of leaders Cloud Innovation Centre Engagement and](https://reader035.vdocuments.us/reader035/viewer/2022081614/5fc6af8e341bc843373f087d/html5/thumbnails/34.jpg)
Issues/Opportunities
• Model sharing
• Data preservation
• Meeting Data Security requirements
• “Cloud based controlled energy management systems”
• “Blockchain as a tool for energy retail exchange”
• National Institute for Smart Data Innovation
– energy theme