Calibration and scenario generation of Hydrodynamics and Water Quality models over a Cloud Computing environment: INDIGO-DataCloud

RIA-653549

Presented by Fernando Aguilar (IFCA-CSIC) aguilarf@ifca.unican.es INDIGO-DataCloud WP2

Delft3D Days – 1st November 2016

Introduction • Framework: Collaboration with SME Ecohydros. Last year ROEM+.

• Reservoir Hydrodinamic and Water Quality modelling. Cuerda del Pozo: water supply, water activities.

• Previous work • Platform takes data from water: physical, chemical, biological, etc. Allows to know water status (data

taken since 2010 aprox.)

• Data visualization tool. Aims to alert authorities when the water quality is under the limits.

• High Resolution Models require a powerful and scalable computing environment.

November 2016 2

Case Study: Algae Bloom in a Water Reservoir

• Research Community: LifeWatch (ESFRI) • Topic/Area: Biodiversity & Ecosystem research

• Objective of the Case Study: • Monitor the evolution of the potential eutrophication of a Water Reservoir.

• Two main objectives: • Scenarios: What if? Water flow, nutrients, etc. Range.

• Calibration: Algae Parameters like Mortality, Respiration, etc.

• Both requires a number of simulations. • Scalability and flexibility needed.

• User Friendly Management: Input, Configuration, N outputs.

• Cloud Computing is the answer! • INDIGO-DataCloud Project is developing all the solution needed.

3

The INDIGO-DataCloud Project • INDIGO = INtegrating Distributed data Infrastructures for Global ExplOitation

• H2020 project, Apr 2015 – Oct 2017, 11M EURO

• 26 partners in 11 European countries

• Objectives: Develop a data/computing platform, targeting various scientific communities and deployable on hybrid (private or public) Cloud infrastructures

• Website: https://www.indigo‐datacloud.eu INDIGO-DataCloud RIA-653549

4

INDIGO-DataCloud RIA-653549

INDIGO Architecture

5

• Enhanced features in • IaaS • PaaS • SaaS

• 1st release MidnightBlue • Data Center Solutions • Data Solutions • Automated Solutions • High-level User Oriented

Solutions

INDIGO Communities

November 2016 6

Life Sciences: ELIXIR, INSTRUCT/WeNMR, EuroBioImaging

Physical Sciences & Astronomy: CTA, LBT, WLCG

Social Sciences & Humanities: DARIAH, DCH-RP

Environmental Science: LifeWatch, EMSO, ENES

Agile Methodology to satisfy User Requirements!

Solution Developed

7

Local OneClient

Solution Developed

8

• Access to all the services • Roles/Groups Definition • Federated

Solution Developed

9

• Distributed Storage Solution • Accessible by users and

Machines • DropBox-like, but online • WebClient, POSIX, etc. • Sharable Environment • Input/Output

Solution Developed

10

• GitHub: Software Repository. Script for running the “job”. • DockerHub: SO, Delft3D Software and all dependencies ready to be

deployed. • Ansible: Recipes for installing requirements (OneData).

Solution Developed

11

• User Interface • Generates TOSCA template

from user configuration, including model params.

• Sends json to Orchestrator. • Everything transparent for

the user.

Solution Developed

12

• PaaS Orchestrator: Deploys the docker and manage the execution. • Deployment over many different IaaS Services. • Using Mesos/Chronos. • Running instances mount OneData, get the input and write the output. • Output available for the user.

Solution Developed

13

Local OneClient

Demo description • Testbed resources that will be used: Bari TestBed

• Teams involved: INFN/Bari, PSNC, IFCA/CSIC

• Prerequisites: application in Docker

• Sequence of actions • Connect to IAM to access OneData. Input data upload. • Access to the Graphical User Interface (FutureGateway). Fill the form (OneData,

Access, Sweep Parameter values). Submit. • The TOSCA template edited and sent to the orchestrator. • Check Deployment status. • After finishing, output accessible via OneData. Comparing models using Delft3D

tools.

• NOW WE SWITCH TO THE DEMO SCREEN…

14

INDIGO added value

• Scalable (storage and computing) resources in the cloud to perform o(100-10000) tests…

• …and share directly within the community

• User Friendly interface to use cloud resources: • Final users only need to fill a form to submit a new simulation, avoiding

the script edition or direct contact with the infrastructure (Supercomputer, Grid, Cloud) (very helpful for non IT experts).

• First time we use a flexible and “universal” user authentication (quite relevant to collaborate with SMEs also)

• Transparent access to shared large storage (OneData)

15

Conclusions

• Cloud Computing environment is a recommendable framework to run a number of Delft3D simulations.

• INDIGO-DataCloud solutions integration offers a user-oriented alternative: powerful and easy-to-use.

• IT skills are not needed to use Computing Resources. • Distributed and Sharable Storage system available: no

need to copy files and scientists can collaborate easily. • AAI Solution for accessing. • INDIGO-DataCloud solutions available for using (Open

Source).

16

Thank you

https://www.indigo-datacloud.eu Better Software for Better Science.

17

Fernando Aguilar (IFCA-CSIC) aguilarf@ifca.unican.es