ecmwf's vision for big data, ai and cloud computing · 2019. 11. 19. · • big data volume...
TRANSCRIPT
© ECMWF November 19, 2019
ECMWF's vision for Big Data, AI and cloud computing
The digital twin of the Earth from models and observations
2EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS
ECMWF produces a digital twin of the Earth system (atmosphere, ocean, land, cryosphere) every day …
• Past: reanalyses (monitoring climate) and reforecasts (calibrating forecasts)
• Present: analyses (initializing forecasts)
• Future: forecasts up to seasonal range
… using an Earth-system model with 9 billion pieces of information at every time step & 40 million
observations
NASA Apollo 11 image of the Earth above the Moon taken on 20 July 1969 at around 05h UTC (left) and the
corresponding pseudo-image generated from a 29-hour 28-km resolution ECMWF forecast initialised from ERA40
data (right).
ECMWF’s production workflow
3EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS
Data acquisition
Forecast runProduct
generation
Dissemination
RMDCN
Internet
Web services Internet
ArchiveData Handling
System
60M+ data used every day
Evolution of archive and HPC sustained performance as big data generators
4EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS
Data archive stood at 317 PB of primary and 116 PB of secondary data store and growing more than 250TB every day
5EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS
60 PB data
purge
(equivalent of
6 FTE cost)
Data dissemination volumes
6EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS
• Monthly Volume of Data Transmitted: 857TB (~28.6TB/day)
• Exponential Increase trend for Internet
10-year challenge
7EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS
Data acquisition
Forecast runProduct
generation
Dissemination
RMDCN
Internet
Web services Internet
ArchiveData Handling
System
10x more
observational data
per day
2000x more model data
per time step
25x more forecast
product data per day in
critical path
30x more data sent to
customers per day in critical
path
100x more data
archived per
day
Rich Loft (NCAR): Applying Machine Learning to Modelling is a bit like THE MATRIX
8EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS
Slide from Rich Loft’s presentation “Machine Learning and Data Driven HPC at NCAR: Strategy and
State of Play” @ iCAS Symposium, Stresa, IT, 8-12 Sept 2019
Need for European Weather Cloud
9EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS
Compelling use
cases across
the EMI
Cloud Computing
Virtual
Workshop, 31
August 2018
(60+ participants)
Produced
Disseminated
ECMWF outputs are
constantly increasing
and a gap already
exists
Over Cloud
Under Cloud
IPMI
IPMI
IPMI
IPMI
IPMI
IPMI
IPMI
IPMI
Eth0
Eth0
Eth0
Eth0
Eth0
Eth0
Eth0
Eth0
Eth0
Eth0
Eth0
Eth0
IPMI
IPMI
IPMI
IPMI
IPMI
Eth0
Internal API Network10.6.0.0/24 (2506)
Tenant Network10.6.12.0/22 (2512)
Ceph Management Network10.6.2.0/24 (2508)
Ceph Network10.6.3.0/24 (2509)
External NetworkTemp: 136.156.220.0/22
Perm: 10.6.4.0/22 (2510)Mask: 255.255.252.0GW: 10.6.7.254
Controller Nodes Computes Nodes Ceph Nodes
Provisioning NetworkEth0: 10.6.8.0/22 (2511)GW: 10.6.11.254
136.156.223.254
Server Console192.168.252.0/23
(752)MASK: 255.255.254.0GW: 192.168.253.254
Eth1
Eth1
Eth1
Eth1
Under-cloud Controller Nodes
10
.6.9
.1
10
.6.9
.2
10
.6.9
.10
1
10
.6.9
.10
2
10
.6.8
.10
6
10
.6.8
.1
10
.6.8
.2
10
.6.8
.10
1
10
.6.8
.10
2
10
.6.8
.10
3
10
.6.8
.10
5
10
.6.8
.10
7
10
.6.8
.3
10
.6.8
.4
10
.6.8
.5
10
.6.8
.6
Eth2
Eth3
10.6.0.1 10.6.0.2 10.6.0.101 10.6.0.102 10.6.0.103 10.6.0.104
10.6.12.1 10.6.12.2 10.6.12.101 10.6.12.102 10.6.12.103 10.6.12.104
10.6.2.1 10.6.2.2 10.6.2.101 10.6.2.105 10.6.2.106 10.6.2.107
10.6.3.1 10.6.3.2 10.6.3.101 10.6.3.102 10.6.3.103 10.6.3.104 10.6.3.105 10.6.3.106 10.6.3.107
136.156.222 .11 136.156.222 .12
10.6.4.3 10.6.4.4 10.6.4.5 10.6.4.6
Eth1
Eth2
Eth1
Eth2
Eth1
Eth2
Eth1
Eth2
Eth1
Eth2
Eth1
Eth2
Eth1
Eth2
Eth1
Eth2
Eth1
Eth2
OS
K-C
OM
P0
3
OS
K-C
OM
P0
4
osk
-com
p0
1
osk
-com
p0
2
osk
-cep
h0
1
osk
-cep
h0
2
osk
-cep
h0
3
osk
-cep
h0
4
osk
-cep
h0
5
osk
-cep
h0
6
osk
-cep
h0
7
19
2.1
68
.25
2.1
92
19
2.1
68
.25
2.1
95
19
2.1
68
.25
2.1
96
Eth4
Eth5
Eth4
Eth5
Eth3
Eth2
136.156.222 .101
10.6.7.254
IPMI
IPMI
IPMI
IPMI
IPMI
IPMI
IPMI
IPMI
Eth0
Eth0
Eth0
Eth0
Eth0
Eth0
Eth0
Eth0
IPMI
Eth0
Controller Nodes Computes Nodes Ceph Nodes
10
.6.8
.10
7
10.6.0.1 10.6.0.2 10.6.0.101 10.6.0.102 10.6.0.103 10.6.0.104
10.6.12.1 10.6.12.2 10.6.12.101 10.6.12.102 10.6.12.103 10.6.12.104
10.6.2.1 10.6.2.2 10.6.2.101 10.6.2.105 10.6.2.106 10.6.2.107
10.6.3.1 10.6.3.2 10.6.3.101 10.6.3.102 10.6.3.103 10.6.3.104 10.6.3.105 10.6.3.106 10.6.3.107
136.156.222 .11 136.156.222 .12
Eth1
Eth2
Eth1
Eth2
Eth1
Eth2
Eth1
Eth2
Eth1
Eth2
Eth1
Eth2
Eth1
Eth2
Eth1
Eth2
Eth1
Eth2
osk
-com
p0
1
osk
-com
p0
2
osk
-cep
h0
1
osk
-cep
h0
2
osk
-cep
h0
3
osk
-cep
h0
4
osk
-cep
h0
5
osk
-cep
h0
6
osk
-cep
h0
7
136.156.222 .101
10.6.3.107
IPMI
IPMI
IPMI
IPMI
IPMI
IPMI
Eth0
Eth0
Eth0
Eth0
Eth0
Eth0
IPMI
Eth0
Controller Nodes Computes Nodes
10
.6.8
.10
7
10.6.0.1 10.6.0.2 10.6.0.101 10.6.0.102 10.6.0.103 10.6.0.104
10.6.12.1 10.6.12.2 10.6.12.101 10.6.12.102 10.6.12.103 10.6.12.104
10.6.2.1 10.6.2.2 10.6.2.101 10.6.2.105
10.6.3.1 10.6.3.2 10.6.3.101 10.6.3.102 10.6.3.103 10.6.3.104 10.6.3.105
136.156.222 .11 136.156.222 .12
Eth1
Eth2
Eth1
Eth2
Eth1
Eth2
Eth1
Eth2
Eth1
Eth2
Eth1
Eth2
Eth1
Eth2
osk
-com
p0
1
osk
-com
p0
2
osk
-cep
h0
1
osk
-cep
h0
2
osk
-cep
h0
3
osk
-cep
h0
4
osk
-cep
h0
5
136.156.222 .101
European Weather Cloud Pilot
10EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS
vCPUs : 784 (2x48+5X64+6X72)
RAM : 8076 GB
Storage: 432 TB (11 systems X 24 HDDX1.7TB)
Make use of the entire Digital Continuum for true digital twin of Earth
https://www.etp4hpc.eu/pujades/files
/Blueprint%20document_20190904.
‘Traditional’ Earth-system modelling and observation ++:
• New observations from smart sensors
• Embedding in IoT and pre-processing on edge and fog
• Big data volume & diversity handling within agile workflows
• Flexible range of cloud, AI and high-performance computing
• Cooperation with our MS, ESA, EuroHPC, EOSC, …
Scientific Research Agenda-4 (in preparation)
(ECMWF is a full member of ETP4HPC)
12EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS