emso data management platform - atlantos · pasquale andriani asdasdasd oceanology international...
TRANSCRIPT
![Page 1: EMSO Data Management Platform - AtlantOS · Pasquale Andriani asdasdasd Oceanology International 2018 London, Excel / March 15, 2018 Context HIGH-LEVEL ARCHITECTURE Data access DMP](https://reader034.vdocuments.us/reader034/viewer/2022050221/5f66f44f29a0ee4c076c14d0/html5/thumbnails/1.jpg)
![Page 2: EMSO Data Management Platform - AtlantOS · Pasquale Andriani asdasdasd Oceanology International 2018 London, Excel / March 15, 2018 Context HIGH-LEVEL ARCHITECTURE Data access DMP](https://reader034.vdocuments.us/reader034/viewer/2022050221/5f66f44f29a0ee4c076c14d0/html5/thumbnails/2.jpg)
EMSODataManagementPlatform
PasqualeAndrianiasdasdasdOceanologyInternational2018London,Excel/March15,2018
![Page 3: EMSO Data Management Platform - AtlantOS · Pasquale Andriani asdasdasd Oceanology International 2018 London, Excel / March 15, 2018 Context HIGH-LEVEL ARCHITECTURE Data access DMP](https://reader034.vdocuments.us/reader034/viewer/2022050221/5f66f44f29a0ee4c076c14d0/html5/thumbnails/3.jpg)
Context
![Page 4: EMSO Data Management Platform - AtlantOS · Pasquale Andriani asdasdasd Oceanology International 2018 London, Excel / March 15, 2018 Context HIGH-LEVEL ARCHITECTURE Data access DMP](https://reader034.vdocuments.us/reader034/viewer/2022050221/5f66f44f29a0ee4c076c14d0/html5/thumbnails/4.jpg)
HIGH-LEVELARCHITECTURE
Dataaccess
DMPtools
DATAMANAGEMENTPLATFORM
Real-tim
eAsynch.
Dataingestion
Ingestionspeed
![Page 5: EMSO Data Management Platform - AtlantOS · Pasquale Andriani asdasdasd Oceanology International 2018 London, Excel / March 15, 2018 Context HIGH-LEVEL ARCHITECTURE Data access DMP](https://reader034.vdocuments.us/reader034/viewer/2022050221/5f66f44f29a0ee4c076c14d0/html5/thumbnails/5.jpg)
PHYSICALINFRASTRUCTURE
EMSO DMP • Production VO: vo.emsodev.eu • Cloud Compute: 11 VMs (8 CPUs + 16GB RAM + 40GB) • File Storage: 5 TB
Jun-2016«TestVO» Aug-2016«SLAdefinition» Nov-2016«ProductionVO»
![Page 6: EMSO Data Management Platform - AtlantOS · Pasquale Andriani asdasdasd Oceanology International 2018 London, Excel / March 15, 2018 Context HIGH-LEVEL ARCHITECTURE Data access DMP](https://reader034.vdocuments.us/reader034/viewer/2022050221/5f66f44f29a0ee4c076c14d0/html5/thumbnails/6.jpg)
FROMSOLUTION… datause
dataprocessing
WebHDFSAPI
datapublishing
dataacquisition
EMSODEVDATAMANAGEMENTPLATFORM
datacuration
MOODA
![Page 7: EMSO Data Management Platform - AtlantOS · Pasquale Andriani asdasdasd Oceanology International 2018 London, Excel / March 15, 2018 Context HIGH-LEVEL ARCHITECTURE Data access DMP](https://reader034.vdocuments.us/reader034/viewer/2022050221/5f66f44f29a0ee4c076c14d0/html5/thumbnails/7.jpg)
FROMSOLUTION…(DATAACQUISITION)
DataacquisitionPushTransferFlow:dataissenttoaDMPservice.PullTransferFlow:dataisretrievedviaAPIexposedbyanEGIMSOSserver.
dataacquisition
PUSHtransferflow
PULLtransferflowAPI
![Page 8: EMSO Data Management Platform - AtlantOS · Pasquale Andriani asdasdasd Oceanology International 2018 London, Excel / March 15, 2018 Context HIGH-LEVEL ARCHITECTURE Data access DMP](https://reader034.vdocuments.us/reader034/viewer/2022050221/5f66f44f29a0ee4c076c14d0/html5/thumbnails/8.jpg)
FROMSOLUTION…(DATACURATION)Datacuration
• Datamanipulation(datascrapingandmunging)• Datacollection(datapersistintodifferentstorage)
dataacquisition
datacuration
![Page 9: EMSO Data Management Platform - AtlantOS · Pasquale Andriani asdasdasd Oceanology International 2018 London, Excel / March 15, 2018 Context HIGH-LEVEL ARCHITECTURE Data access DMP](https://reader034.vdocuments.us/reader034/viewer/2022050221/5f66f44f29a0ee4c076c14d0/html5/thumbnails/9.jpg)
FROMSOLUTION…(DATAUSE)
TheDMPprovidesthescientificuserwithasetofdifferenttools(DMPToolS):• EMSODEVAPIallowsscientificusersandotherEuropeaninitiativesintheoceansciencesto
interfacewiththedataavailablewithintheDMP.• MOODA(ModuleforOceanObservatoryDataAnalysis)isacustomapplicationdevelopedfor
thescientificcommunity.• GRAFANAisadashboardtoquery,visualiseandunderstandthemetricsstoredintotheDMP.
MOODA
![Page 10: EMSO Data Management Platform - AtlantOS · Pasquale Andriani asdasdasd Oceanology International 2018 London, Excel / March 15, 2018 Context HIGH-LEVEL ARCHITECTURE Data access DMP](https://reader034.vdocuments.us/reader034/viewer/2022050221/5f66f44f29a0ee4c076c14d0/html5/thumbnails/10.jpg)
FROMSOLUTION…(DATAPROCESSING)
dataprocessing
• ApacheSpark:fastandgeneralengineforlarge-scaledataprocessing.Itisabletoexecutebothbatchandstreamingdataprocessing.
• ApacheZeppelin:interactivedataanalyticstool.Itisabletoperformdataanalysiswithdifferentprocessingsysteminterpreters.
ODVFileOceanDataView
![Page 11: EMSO Data Management Platform - AtlantOS · Pasquale Andriani asdasdasd Oceanology International 2018 London, Excel / March 15, 2018 Context HIGH-LEVEL ARCHITECTURE Data access DMP](https://reader034.vdocuments.us/reader034/viewer/2022050221/5f66f44f29a0ee4c076c14d0/html5/thumbnails/11.jpg)
…TOOPERATION
12hoursofSeaWaterTemperature
![Page 12: EMSO Data Management Platform - AtlantOS · Pasquale Andriani asdasdasd Oceanology International 2018 London, Excel / March 15, 2018 Context HIGH-LEVEL ARCHITECTURE Data access DMP](https://reader034.vdocuments.us/reader034/viewer/2022050221/5f66f44f29a0ee4c076c14d0/html5/thumbnails/12.jpg)
The EMSO-Link project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreements N° 731036.
Thankyouforyourattention.
Scan0005 - Metalkas · 2017-12-12 · Scan0005.jpg Author: asdasdasd Created Date: 7/15/2011 8:44:28 AM
Scan0008 - Metalkas · 2017-12-12 · Scan0008.jpg Author: asdasdasd Created Date: 7/15/2011 8:42:32 AM