ds esa 20170707 - space policymesos cluster mesos cluster aut. scaling service storage service...

35
INDIGODataCloud RIA653549 Better Software for Better Science. Davide Salomoni, INFNCNAF INDIGODataCloud Project Coordinator [email protected] ESAESPI Workshop Frascati, 7/7/2017

Upload: others

Post on 29-Aug-2020

6 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: ds esa 20170707 - Space PolicyMesos Cluster Mesos Cluster Aut. Scaling Service Storage Service S3/CDMI/Posix/Webdav GridFTP Smart Scheduling Spot Istances Native Docker QoSSupport

INDIGO-­‐DataCloud

RIA-­‐653549

Better  Software  for  Better  Science.

Davide Salomoni,  INFN-­‐CNAFINDIGO-­‐DataCloud Project  Coordinator

[email protected]­‐ESPI  WorkshopFrascati,  7/7/2017

Page 2: ds esa 20170707 - Space PolicyMesos Cluster Mesos Cluster Aut. Scaling Service Storage Service S3/CDMI/Posix/Webdav GridFTP Smart Scheduling Spot Istances Native Docker QoSSupport

INDIGO-­‐DataCloud

• An  H2020  project  approved  in  January  2015  in  the  EINFRA-­‐1-­‐2014  call• 11.1M€,  30  months  (from  April  2015  to  September  2017)

• Who:  26  European  partners  in  11  European  countries• Coordination  by  the  Italian  National  Institute  for  Nuclear  Physics  (INFN)• Including  developers  of  distributed  software,  industrial  partners,  research  

institutes,  universities,  e-­‐infrastructures• What:  develop  an  open  source  Cloud  platform for  computing  and  data  (“DataCloud”)  tailored  to  science  but  applicable  to  other  domains  as  well.

• For:  multi-­‐disciplinary  scientific  communities• E.g.  structural  biology,  earth  science,  physics,  bioinformatics,  cultural  

heritage,  astrophysics,  life  science,  climatology• Where:  deployable  on  hybrid  (public  or  private)  Cloud  infrastructures• INDIGO  =  INtegrating  Distributed  data  Infrastructures  for  Global  ExplOitation

• Why:  answer  to  the  technological  needs  of  scientists  seeking  to  easily  exploit  distributed  Cloud/Grid  compute  and  data  resources.

ESA-­‐ESPI  Workshop,  7/7/2017 2Davide  Salomoni

Page 3: ds esa 20170707 - Space PolicyMesos Cluster Mesos Cluster Aut. Scaling Service Storage Service S3/CDMI/Posix/Webdav GridFTP Smart Scheduling Spot Istances Native Docker QoSSupport

INDIGO’s  34  deliverables  (so  far)

INDIGO-­‐DataCloud’s Vision• INDIGO:

1. Develops  open,  interoperable  solutions  for  scientific  data.

2. Supports  open  science organizing  the  European  data  space.

3. Enables  collaborations across  diverse  scientific  communities  worldwide.

• INDIGO  offers  its• architecture,• analysis,• expertise• and  software  components

• as  a  concrete  step  toward  the  definition  and  implementation  of  a  European  Open  Science  Cloud  and  Data  Infrastructure.Davide  Salomoni ESA-­‐ESPI  Workshop,  7/7/2017 3

Publicly  funded  e-­‐infrastructures(EGI,  EUDAT,  GEANT,  PRACE,  RI,  

etc.)

Private  or  CommercialClouds  (Public,  PCP-­‐based,  

etc.)

Scientific  Users

Adopt,  Use

Deployed  on

Exploiting

To  produce

Scientific  Results

INDIGO  Advanced  Components  and  Solutions

Datasets,  Resources

D1.8,  General  Architecture

D2.1  and  D2.4,  

community  requirements

The  INDIGO-­‐DataCloudService  

Catalogue

Page 4: ds esa 20170707 - Space PolicyMesos Cluster Mesos Cluster Aut. Scaling Service Storage Service S3/CDMI/Posix/Webdav GridFTP Smart Scheduling Spot Istances Native Docker QoSSupport

The  INDIGO  Services

• We  recently  released  our  second  and  final  major  software  release,  called  ElectricIndigo

Davide  Salomoni ESA-­‐ESPI  Workshop,  7/7/2017 4

• Fact  sheet  (https://www.indigo-­‐datacloud.eu/service-­‐component):• 40  modular  components,  distributed  via  170  software  packages,  50  ready-­‐to-­‐use  Docker  containers

• Supported  operating  systems:  CentOS  7,  Ubuntu  16.04

• Supported  cloud  frameworks:  OpenStack  Newton,  OpenNebula 5.x  (plus  connection  to  Amazon,  Azure)

• Download  it  from  the  INDIGO-­‐DataCloudSoftware  Repository:  http://repo.indigo-­‐datacloud.eu/index.html

Page 5: ds esa 20170707 - Space PolicyMesos Cluster Mesos Cluster Aut. Scaling Service Storage Service S3/CDMI/Posix/Webdav GridFTP Smart Scheduling Spot Istances Native Docker QoSSupport

How  does  this  fit  in  a  global  context?

• We  recognize  that  value  for  users  (and  hence,  our  main  focus)  is  at  the  upper  layers,  not  in  the  bare  bone  e-­‐infrastructural  services.• But  we  also  provide  ways  to  optimize  e-­‐infrastructural  services  for  resource  providers

• So,  we  abstract  from  underlying  IaaS  technologies and  offer  flexibility  in  choosing  e-­‐infra  providers,  resources  and  capabilities…  

• …  giving  users  the  possibility  to  easily  express  and  implement  requirements  for  their  applications through  enabling  services  and  components.

• This  is  a  movement  that  goes  well  beyond  the  ”S”  of  Science in  a  EOSC.

Davide  Salomoni ESA-­‐ESPI  Workshop,  7/7/2017 5

Page 6: ds esa 20170707 - Space PolicyMesos Cluster Mesos Cluster Aut. Scaling Service Storage Service S3/CDMI/Posix/Webdav GridFTP Smart Scheduling Spot Istances Native Docker QoSSupport

INDIGO  in  support  to  communities(some  real  apps  integrating  INDIGO  components)

Davide  Salomoni ESA-­‐ESPI  Workshop,  7/7/2017 6

• LifeWatch:  algae  bloom  modeling• RNA  sequencing  with  TRUFA• Deploying  an  elastic,  complex  cluster  on  the  Cloud  with  INDIGO  components

• Cloudified services  for  molecular  dynamics• A  distributed  archive  system  for  the  Cherenkov  Telescope  Array  (CTA)

• The  Large  Binocular  Telescope  (LBT)  distributed  archive

• INDIGO’s  Ophidia  for  astronomical  images  calibration

• Launching  POWERFIT  and  DISVIS  VMs  on  the  EGI  FedCloud using  INDIGO  tools

• POWERFIT  and  DISVIS  web  portals:  harnessing  GPGPUs  on  the  Grid  using  INDIGO  udocker

• Automated  deployment  of  an  Ophidia  big  data  analytics  cluster

• INDIGO  at  the  Central  Institute  for  the  Union  Catalogue  of  Italian  Libraries  and  Bibliographic  Information

• EGI  and  INDIGO  integration• ELIXIR-­‐ITALY:  developing  a  Galaxy  instance  provider  platform

• Multidisciplinary  Oceanic  Information  System

• Deploy  a  Zenodo-­‐based  repository  in  the  cloud  using  Marathon

• On-­‐demand  analysis  and  big  data  infrastructures  for  the  CMS  LHC  experiment

• Theoretical  physics  on  HPC  clusters  using  INDIGO  udocker

Page 7: ds esa 20170707 - Space PolicyMesos Cluster Mesos Cluster Aut. Scaling Service Storage Service S3/CDMI/Posix/Webdav GridFTP Smart Scheduling Spot Istances Native Docker QoSSupport

INDIGO  in  support  to  maintenance  and  evolution• How  can  INDIGO  be  sustained  and  evolved?

1. Collaboration  with  commercial  providers2. Collaboration  with  existing  projects  and  initiatives3. Submission  of  new  projects

Davide  Salomoni ESA-­‐ESPI  Workshop,  7/7/2017 7

Page 8: ds esa 20170707 - Space PolicyMesos Cluster Mesos Cluster Aut. Scaling Service Storage Service S3/CDMI/Posix/Webdav GridFTP Smart Scheduling Spot Istances Native Docker QoSSupport

Some  collaborations  with  commercial  providers• See  https://developer.ibm.com/opentech/2017/05/18/cloud-­‐computing-­‐better-­‐science-­‐recap-­‐egi-­‐conference-­‐indigo-­‐datacloud-­‐summit-­‐2017/ for  a  summary  of  the  INDIGO  Summit  2017  by  DrSahdev Zala of  IBM• With  some  details  about  the  ongoing  collaboration  between  IBM  and  INDIGO-­‐DataCloud

• See  https://indico.egi.eu/indico/event/3249/session/48/contribution/98/material/slides/0.pdf for  info  on  the  integration  of  INDIGO  tools  into  the  Open  Telekom  Cloud  portfolio,  the  public  cloud  offering  of  T-­‐Systems  (a  Deutsche  Telekom  unit)

Davide  Salomoni ESA-­‐ESPI  Workshop,  7/7/2017 8

Page 9: ds esa 20170707 - Space PolicyMesos Cluster Mesos Cluster Aut. Scaling Service Storage Service S3/CDMI/Posix/Webdav GridFTP Smart Scheduling Spot Istances Native Docker QoSSupport

Davide  Salomoni ESA-­‐ESPI  Workshop,  7/7/2017 9

Page 10: ds esa 20170707 - Space PolicyMesos Cluster Mesos Cluster Aut. Scaling Service Storage Service S3/CDMI/Posix/Webdav GridFTP Smart Scheduling Spot Istances Native Docker QoSSupport

Collaboration  with  existing  projects,  infrastructures  and  initiatives• There  are  ongoing  discussions  and  collaborations  with  several  actors,  belonging  to  many  areas  – for  example:• European  Space  Agency  &  ASI  (typically  for  exploitation  /  distribution  /  analysis  of  Copernicus  data)• Smart  City  projects• Rationalization  of  Public  Administrations• HelixNebula ScienceCloud (Pre-­‐Commercial  Procurement)• EU-­‐wide  HPC-­‐Big  Data  Integration  (IPCEI)• EGI  (also  an  INDIGO-­‐DataCloud partner)

Davide  Salomoni ESA-­‐ESPI  Workshop,  7/7/2017 10

Page 11: ds esa 20170707 - Space PolicyMesos Cluster Mesos Cluster Aut. Scaling Service Storage Service S3/CDMI/Posix/Webdav GridFTP Smart Scheduling Spot Istances Native Docker QoSSupport

Submission  of  new  European  projects

• In  the  last  round  of  the  H2020  calls  (March-­‐April  2017),  at  least  5  proposals were  submitted  that  included  key  INDIGO  components  or  their  possible  evolutions.• We  still  do  not  know  how  many  of  these  proposals  will  be  approved,  but  it  is  interesting  to  note  that  there  is  a  very  significant  interest  and  request  for  solutions  that  originate  from  INDIGO.  If  results  are  there,  stakeholder  engagement  is  strong,  if  ideas,  requirements,  architectures  are  valid,  this  interest  will  eventually  find  ways  to  be  supported.

Davide  Salomoni ESA-­‐ESPI  Workshop,  7/7/2017 11

Page 12: ds esa 20170707 - Space PolicyMesos Cluster Mesos Cluster Aut. Scaling Service Storage Service S3/CDMI/Posix/Webdav GridFTP Smart Scheduling Spot Istances Native Docker QoSSupport

INDIGO  &  EOSC  in  production:  >=  TRL8

• For  example,  several  INDIGO  solutions  and  activities  are  in  the  EOSC-­‐hub proposal (a  proposal  jointly  prepared  by  EGI,  EUDAT  and  INDIGO-­‐DataCloud)• With  INDIGO  components  such  as  Identity  and  Access  Management,  Token  Translation,  Virtual  filesystems  (Onedata),  Advanced  IaaS  Services,  the  Infrastructure  Manager,  the  INDIGO  PaaS  and  its  orchestrator,  web  front-­‐end  services,  user-­‐level  containers• And  with  training,  support,  technical  coordination,  external  liaison,  stakeholder  engagement,  policy  contributions.

Davide  Salomoni ESA-­‐ESPI  Workshop,  7/7/2017 12

Page 13: ds esa 20170707 - Space PolicyMesos Cluster Mesos Cluster Aut. Scaling Service Storage Service S3/CDMI/Posix/Webdav GridFTP Smart Scheduling Spot Istances Native Docker QoSSupport

INDIGO  &  EOSC  in  evolution:  <  TRL8• For  example,  novel  features evolving  INDIGO  components  are  a  key  part  of  several  proposals  to  the  EINFRA-­‐21-­‐2017  (eXtreme-­‐DataCloud and  DEEP-­‐Hybrid  DataCloud)  and  ICT-­‐16-­‐2017  calls:

• Intelligent  dataset  distribution  and  data  lifecycle  management• Smart  caching  • Orchestrating  Computing  Workflows  based  on  policy  driven  or  adaptive  data  movements  

• Flexible  metadata  management  for  big  data  sets• Access  to  bare-­‐metal  resources  on  the  Cloud• PaaS-­‐Level  access  to  HPC  resources• Extensions  to  the  INDIGO  Orchestrator  for  hybrid  IaaS  deployments  and  scale  out  to  3rd  party  clouds

• Extensions  to  the  INDIGO  Virtual  Router  Appliance• Real-­‐time,  streaming-­‐based  data  ingestion  and  processing

Davide  Salomoni ESA-­‐ESPI  Workshop,  7/7/2017 13

Page 14: ds esa 20170707 - Space PolicyMesos Cluster Mesos Cluster Aut. Scaling Service Storage Service S3/CDMI/Posix/Webdav GridFTP Smart Scheduling Spot Istances Native Docker QoSSupport

INDIGO  and  External  Projects:  Components  and  Patches  Merged  in  Upstream  Open  Source  Projects

• OpenStack  (https://www.openstack.org)• Nova  Docker• Heat• OpenID-­‐Connect  for  Keystone• Pre-­‐emptible  instances  support  (under  

discussion)• OpenNebula (http://opennebula.org)

• OneDock• Infrastructure  Manager  (http://www.grycap.upv.es/im/index.php)

• Clues  (http://www.grycap.upv.es/clues/eng/index.php)

• Onedata (https://onedata.org)

• TOSCA  adaptor  for  JSAGA  (http://software.in2p3.fr/jsaga/dev/)

• OCCI  implementation  for  OpenStack  (https://github.com/openstack/ooi)

• Extended  AWS  support  for  rOCCI in  OpenNebula.  Python  and  Java  libraries  for  OCCI  support.

• CDMI  and  QoS extensions  for  dCache(https://www.dcache.org)

• Workflow  interface  extensions  for  Ophidia  (http://ophidia.cmcc.it)

• OpenID  Connect  Java  implementation  for  dCache (https://www.dcache.org)

• MitreID (https://mitreid.org/) and  OpenID  Connect  (http://openid.net/connect/) libraries

Davide  Salomoni ESA-­‐ESPI  Workshop,  7/7/2017 14

Page 15: ds esa 20170707 - Space PolicyMesos Cluster Mesos Cluster Aut. Scaling Service Storage Service S3/CDMI/Posix/Webdav GridFTP Smart Scheduling Spot Istances Native Docker QoSSupport

157/6/17

EGI  services for  EO  data  exploitation

• The  new  EO  satellites  generates  large  amounts  of  data  not  easily  integrated  into  processing  chains  outside  the  ground  segment.

• EGI  services  can  improve  the  discovery,  retrieval  and  processingcapabilities  of  EO  data:

ü capabilities  for  big  data  management

ü virtualised access  to  geographically  distributed  data  (EGI  Data  Hub)

ü computing  necessary  to  manage  large  volumes  of  different  data  types

Cloud Compute

Online  Storage Data  Transfer

Data  Hub

Page 16: ds esa 20170707 - Space PolicyMesos Cluster Mesos Cluster Aut. Scaling Service Storage Service S3/CDMI/Posix/Webdav GridFTP Smart Scheduling Spot Istances Native Docker QoSSupport

167/6/17

EGI  services for  EO  data  exploitation

• The  new  EO  satellites  generates  large  amounts  of  data  not  easily  integrated  into  processing  chains  outside  the  ground  segment.

• EGI  services  can  improve  the  discovery,  retrieval  and  processingcapabilities  of  EO  data:

ü capabilities  for  big  data  management

ü virtualised access  to  geographically  distributed  data  (EGI  Data  Hub)

ü computing  necessary  to  manage  large  volumes  of  different  data  types

Cloud Compute

Online  Storage Data  Transfer

Data  Hub

Advanced  IaaSNetwork  virtualizationAdvanced  Orchestration  PaaSStandard  interfaces  supportContainers  Orchestration

Page 17: ds esa 20170707 - Space PolicyMesos Cluster Mesos Cluster Aut. Scaling Service Storage Service S3/CDMI/Posix/Webdav GridFTP Smart Scheduling Spot Istances Native Docker QoSSupport

177/6/17

EGI  services for  EO  data  exploitation

• The  new  EO  satellites  generates  large  amounts  of  data  not  easily  integrated  into  processing  chains  outside  the  ground  segment.

• EGI  services  can  improve  the  discovery,  retrieval  and  processingcapabilities  of  EO  data:

ü capabilities  for  big  data  management

ü virtualised access  to  geographically  distributed  data  (EGI  Data  Hub)

ü computing  necessary  to  manage  large  volumes  of  different  data  types

Cloud Compute

Online  Storage Data  Transfer

Data  Hub

OneData

Page 18: ds esa 20170707 - Space PolicyMesos Cluster Mesos Cluster Aut. Scaling Service Storage Service S3/CDMI/Posix/Webdav GridFTP Smart Scheduling Spot Istances Native Docker QoSSupport

187/6/17

EGI  services to  accelerate  thedevelopment of  EO  exploitation platforms

Well  established  e-­‐Infrastructure  services  as  a  set  of  reusable  components  to  solve  common  problems:

• AAI  and  single  sign-­‐on

• Service  Monitoring

• Accounting  infrastructure

• Configuration Database

• Operational Tools

• Collaboration  tools

EO  Platform  developers  can  focus  on  their  core  tasks!

CheckIn

ARGO

APEL

GOCDB

Operations  Portal,  Security  tools,  etc.

Wiki,  Doc  repo,  Agenda  mgmt system,  etc.

Page 19: ds esa 20170707 - Space PolicyMesos Cluster Mesos Cluster Aut. Scaling Service Storage Service S3/CDMI/Posix/Webdav GridFTP Smart Scheduling Spot Istances Native Docker QoSSupport

Reflections  on  some  of  the  suggested  themes• How  infrastructure  contributes  to  make  possible  new  services  and  applications?

• à INDIGO  contributes  by  producing  enabling  technologies directly  requested  by  both  providers  and  users,  that  can  be  deployed  on  ANY  infrastructure  to  produce  new,  high-­‐value  services  or  applications.

• Exponential  technologies:  more  a  software  or  a  hardware  race?  How  value  chain  will  be  affected?• à For  us  it  is  definitely  more  software,  intended  as  final  artifacts  and  resource  exploitation.  The  value  chain  is  represented  

by  the  inverted  triangle  shown  earlier.  The  traditional  hardware  race  per  se  is  lost,  at  least  for  big  initiatives  such  as  HL-­‐LHC.

• How  can  we  best  incentivize  data  sharing  between  entities?• à First,  make  it  doable  /  easy  to  do,  considering  also  issues  such  as  data  lifecycle,  replication,  quality  of  service.

• How  can  we  most  effectively  integrate  modern  and  legacy  data  infrastructures?• à Through  open  solutions,  the  use  of  de  facto  or  de  jure  standards,  and  state-­‐of-­‐the-­‐art  but  still  production-­‐level  solutions.

• To  what  extent  are  consolidation  and  integration  of  existing  services  necessary  to  achieve  the  necessary  infrastructure?• à They  are  absolutely  necessary,  if  we  want  to  effectively  use  all  the  resources  that  are  there.  We  assume  that  we  MUST  be  

able  to  utilize  these  resources  before  looking  /  asking  for  new  infrastructures.

• What  would  you  consider  to  be  the  most  crucial  next  steps  and  milestones  in  the  successful  implementation  of  the  programs?• Be  pragmatic  and  do  not  spend  too  much  time  discussing  first  principles.  Get  the  relevant  actors  around  the  same  table,  

start  from  concrete  requirements  and  use  cases,  and  seek  implementers  /  implementations  able  to:• Scale  out• Run  in  multiple,  heterogeneous,  hybrid  infrastructures• Clearly  show  the  benefits  of  the  proposed  vs.  legacy  /  proprietary  /  ad-­‐hoc   solutions  

Davide  Salomoni ESA-­‐ESPI  Workshop,  7/7/2017 19

Page 20: ds esa 20170707 - Space PolicyMesos Cluster Mesos Cluster Aut. Scaling Service Storage Service S3/CDMI/Posix/Webdav GridFTP Smart Scheduling Spot Istances Native Docker QoSSupport

Conclusions• In  just  24  months,  the  INDIGO-­‐DataCloud project  has  realized  a  comprehensive  involvement  of  many  Research  Communities  and  providers for  the  definition  and  tracking  of  requirements.• We  identified  technology  gaps linked  to  several  concrete  use  cases  in  multiple  fields,  defined,  published  and  implemented  the  overall  INDIGO  architecture.• After  early  demonstrations  and  beta  software  previews,  we  produced  two  major  software  versions  and  9  minor  updates,  releasing  40  open  modular  components.  We  did  that  exploiting  key  European  know-­‐how,  reusing  and  extending  open  source  software,  and  contributing  to  upstream  projects.  We  established  software  development  and  management  processes,  and  defined  development  and  pre-­‐production  distributed  testbeds.• Production  deployment  of  many  applications  making  use  of  the  INDIGO  software  is  well  underway,  and  INDIGO  components  have  been  proposed  for  production  use  in  big  infrastructures,  commercial  companies,  external  projects.• Several  opportunities  for  further  exploitation  of  INDIGO  components  are  being  explored  and  implemented,  in  the  context  of  the  EOSC  and  beyond.

Davide  Salomoni ESA-­‐ESPI  Workshop,  7/7/2017 20

Page 21: ds esa 20170707 - Space PolicyMesos Cluster Mesos Cluster Aut. Scaling Service Storage Service S3/CDMI/Posix/Webdav GridFTP Smart Scheduling Spot Istances Native Docker QoSSupport

Thank  you

https://www.indigo-­‐datacloud.euBetter  Software  for  Better  Science.

Davide  Salomoni ESA-­‐ESPI  Workshop,  7/7/2017 21

@indigodatacloud www.indigo-­‐datacloud.eu https://www.facebook.com/indigodatacloud/

Page 22: ds esa 20170707 - Space PolicyMesos Cluster Mesos Cluster Aut. Scaling Service Storage Service S3/CDMI/Posix/Webdav GridFTP Smart Scheduling Spot Istances Native Docker QoSSupport

Backup  slides

Davide Salomoni ESA-ESPI Workshop, 7/7/2017 22

Page 23: ds esa 20170707 - Space PolicyMesos Cluster Mesos Cluster Aut. Scaling Service Storage Service S3/CDMI/Posix/Webdav GridFTP Smart Scheduling Spot Istances Native Docker QoSSupport

The  INDIGO  added  value• INDIGO,  driven  by  scientific  communities,  has  been  developing  a  comprehensive  open  source  Cloud  architecture,  which  provides  many  new  functionalities  and  services  previously  unavailable  in  open  source  and  in  some  cases  also  in  proprietary  Cloud  offerings.

• These  functionalities  abstract  from  underlying  IaaS  technologies through  the  consistent  use  of  both  de  jure  and  de  facto  standards.  This  allows  interoperability  with  hybrid  (public/private)  infrastructures.

• After  beta  testing  and  demos  shown  as  early  as  November  2015,  we  released  our  first  major  software  release  (MidnightBlue)  in  August  2016,  9  software  updates  in  the  following  months,  and  our  second  and  final  major  release  (ElectricIndigo)  in  April  2017.

Davide  Salomoni ESA-­‐ESPI  Workshop,  7/7/2017 23

Page 24: ds esa 20170707 - Space PolicyMesos Cluster Mesos Cluster Aut. Scaling Service Storage Service S3/CDMI/Posix/Webdav GridFTP Smart Scheduling Spot Istances Native Docker QoSSupport

Release  Timeline

24Davide  Salomoni ESA-­‐ESPI  Workshop,  7/7/2017

Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan

INDIGO-­‐1 Full  updates

Standard  updates

Security  updates

INDIGO-­‐2 Full  updates

Standard  updates

Security  updates

Release  Date End  of  Full  Updates End  of  Standard  Updates

End  of  Security  Updates  &  EOL

INDIGO-­‐1  MidnightBlue 08/08/2016 31/01/2017 31/03/2017 31/05/2017

INDIGO-­‐2  ElectricIndigo 14/04/2017 30/09/2017 30/11/2017 31/01/2018

Page 25: ds esa 20170707 - Space PolicyMesos Cluster Mesos Cluster Aut. Scaling Service Storage Service S3/CDMI/Posix/Webdav GridFTP Smart Scheduling Spot Istances Native Docker QoSSupport

Four  main  “solution  blocks”:• Data  Center  Solutions

• Data  /  Storage  Solutions

• Automated  Solutions

• User-­‐Oriented  Solutions

And  “common  solutions”:• Authentication  and  Authorization

Davide  Salomoni ESA-­‐ESPI  Workshop,  7/7/2017 25

ElectricIndigo

Page 26: ds esa 20170707 - Space PolicyMesos Cluster Mesos Cluster Aut. Scaling Service Storage Service S3/CDMI/Posix/Webdav GridFTP Smart Scheduling Spot Istances Native Docker QoSSupport

ElectricIndigo:Application-­‐level  Interfaces  for  Cloud  Providersand  Automated  Service  Composition

• Easily  port  applications  to  public  and  private  Clouds using  open  programmable  interfaces,  user-­‐level  containers,  and  standards-­‐based  languages  to  automate  definition,  composition  and  instantiation  of  complex  set-­‐ups.

Davide  Salomoni ESA-­‐ESPI  Workshop,  7/7/2017 26

• Typical  questions:  How  can  I  run  my  application  on  Cloud  provider  X?  What  if  I  want  to  use  Docker  but  my  provider  does  not  support  it  (e.g.  also  on  HPC  systems)?  How  do  I  automate  the  creation  and  management  over  public  or  private  Clouds  of  dynamic  clusters  running  multiple  services?

Page 27: ds esa 20170707 - Space PolicyMesos Cluster Mesos Cluster Aut. Scaling Service Storage Service S3/CDMI/Posix/Webdav GridFTP Smart Scheduling Spot Istances Native Docker QoSSupport

ElectricIndigo:Flexible  Identity  and  Access  Management

• Manage  access  and  policies  to  distributed  resources using  multiple  methods  such  as  OpenID-­‐Connect,  SAML,  X.509 digital  certificates,  through  programmable  interfaces  and  web  front-­‐ends.

Davide  Salomoni ESA-­‐ESPI  Workshop,  7/7/2017 27

• Typical  questions:  How  can  I  manage  access  to  distributed  resources  by  users,  identified  through  diverse  methods?  (e.g.  Google  ID,  digital  certificates)  How  should  I  modify  /  write  my  apps  to  benefit  from  that?

Page 28: ds esa 20170707 - Space PolicyMesos Cluster Mesos Cluster Aut. Scaling Service Storage Service S3/CDMI/Posix/Webdav GridFTP Smart Scheduling Spot Istances Native Docker QoSSupport

ElectricIndigo:Data  Management  and  Data  Analytics  Solutions

• Distribute  and  access  data through  multiple  providers  via  virtual  file  systems  and  automated  replication  and  caching,  exploiting  scalable,  high-­‐performance  data  mining  and  analytics.

Davide  Salomoni ESA-­‐ESPI  Workshop,  7/7/2017 28

• Typical  questions:  How  can  I  automatically  replicate  datasets  to  multiple  sites?  Can  I  transparently  access  my  distributed  datasets  from  my  app?  Can  I  cache  the  most  accessed  data,  so  that  it’s  close  to  where  users  need  it?  How  do  I  instantiate  clusters  and  databases  for  big  data  analysis?

Page 29: ds esa 20170707 - Space PolicyMesos Cluster Mesos Cluster Aut. Scaling Service Storage Service S3/CDMI/Posix/Webdav GridFTP Smart Scheduling Spot Istances Native Docker QoSSupport

ElectricIndigo:Programmable  Web  Portals,  Mobile  Applications

• Create  and  interface  web  portals  or  mobile  apps,  exploiting  distributed  data  as  well  as  compute  resources located  in  public  and  private  Cloud  infrastructures.

Davide  Salomoni ESA-­‐ESPI  Workshop,  7/7/2017 29

• Typical  questions:  How  can  I  easily  provide  my  app  with  a  pluggable,  extensible  web  front-­‐end?  Can  this  front-­‐end  interface  with  all  the  features  provided  by  INDIGO?  How  can  I  write  an  INDIGO-­‐enabled  app  for  Android  or  iOS?

Page 30: ds esa 20170707 - Space PolicyMesos Cluster Mesos Cluster Aut. Scaling Service Storage Service S3/CDMI/Posix/Webdav GridFTP Smart Scheduling Spot Istances Native Docker QoSSupport

ElectricIndigo:Enhanced  and  Scalable  Services  for  Data  Centers  and  Resource  Providers

• Increase  the  efficiency  of  existing  Cloud  infrastructures based  on  OpenStack  or  OpenNebula through  advanced  scheduling,  flexible  cloud  /  batch  management,  network  orchestration and  interfacing  of  high-­‐level  Cloud  services  to  existing  storage  systems.

Davide  Salomoni ESA-­‐ESPI  Workshop,  7/7/2017 30

• Typical  questions:  How  can  my  cloud  data  centers  provide  flexible  and  fair  scheduling  policies  for  access  to  resources?  How  do  I  balance  traditional  vs.  cloud  resources  in  my  data  center?  How  do  I  connect  novel  INDIGO  features  to  my  existing  systems?  How  can  I  manage  storage  Quality  of  Service?

Page 31: ds esa 20170707 - Space PolicyMesos Cluster Mesos Cluster Aut. Scaling Service Storage Service S3/CDMI/Posix/Webdav GridFTP Smart Scheduling Spot Istances Native Docker QoSSupport

High-­‐level  view  of  the  INDIGO  architecture

Davide  Salomoni ESA-­‐ESPI  Workshop,  7/7/2017 31

JSAGA/JSAGAAdaptorsFuture GatewayEngineFuture GatewayRESTAPI

OtherScienceGateways

Mobile Apps

OpenMobileToolkit

Ophidpiaplugin

LONIplugin

Taverna,Keplerplugin

AdminPortlets

UserPortlets

DataAnalitics

WorkflowPortlets

SGMonGUIClients

FutureGatewayPortal WorkflowsMobileclientsSupportservices

WP6Services

Kubernetes Cluster

IAM

Service

PaaS

Orchestrator

QoS/SLA

CloudProvider

Ranker

Monitoring

Infrastructure

Manager

TOSCA

TOSCAWP5

Services

Onedata Dynafed

FTSDataServices

REST/CDMI/Wedbav/posix/GridftpOIDC

Accounting

Non-INDIGO

IaaS

NativeIaaS API

Heat/IM

TOSCA

WP4Services

Mesos

ClusterMesos

Cluster

Aut.Scaling

Service

Storage

Service

S3/CDMI/Posix/WebdavGridFTP

Smart

Scheduling

SpotIstances

Native

Docker

QoS Support

Identity

Armonization

Local

Repository

This  is  the  INDIGO-­‐DataCloudGeneral  Architecture*

*:  see  details  in  http://arxiv.org/abs/1603.09536 or  in  https://www.indigo-­‐datacloud.eu/documents-­‐deliverables

Page 32: ds esa 20170707 - Space PolicyMesos Cluster Mesos Cluster Aut. Scaling Service Storage Service S3/CDMI/Posix/Webdav GridFTP Smart Scheduling Spot Istances Native Docker QoSSupport

INDIGO  Software  Development  Flow

32

T3.3  Pilot  services

Integrationinfrastructure

ExternalServiceProviders

T3.1  Software  quality  

assurance

T3.2  Software  release  and  maintenance

Previewinfrastructure

Developmentinfrastructure

T3.4  Exploitation

WP4

WP5

WP6

WP2ApplicationUse-­‐cases

software  deployment

software  use

software  deliveryUsers

Productionwork

Development

32Davide  Salomoni ESA-­‐ESPI  Workshop,  7/7/2017

Page 33: ds esa 20170707 - Space PolicyMesos Cluster Mesos Cluster Aut. Scaling Service Storage Service S3/CDMI/Posix/Webdav GridFTP Smart Scheduling Spot Istances Native Docker QoSSupport

The  INDIGO  Development  and  Integration  Infrastructure

33

CESNET-­‐ rOCCI

PSNC-­‐ Indigokepler-­‐ indigo-­‐omt

LIP/INCD-­‐ OpenNebula:  ONEDock-­‐ Nova-­‐Docker-­‐ FutureGateway UPV

-­‐ IM-­‐ CLUES-­‐ TOSCA

IFCA/CSIC-­‐ OOI-­‐ OPIE

INFN  Bari-­‐ Kubernetes-­‐ Mesos-­‐ Chronos

DESY-­‐ dCache

CERN-­‐ Kubernetes-­‐ Magnun

CNAF/INFN-­‐ IAM-­‐ Oneprovider

KIT-­‐ CDMI-­‐QoS-­‐ TTS

Davide  Salomoni ESA-­‐ESPI  Workshop,  7/7/2017

Cyfronet-­‐ Onedata

Page 34: ds esa 20170707 - Space PolicyMesos Cluster Mesos Cluster Aut. Scaling Service Storage Service S3/CDMI/Posix/Webdav GridFTP Smart Scheduling Spot Istances Native Docker QoSSupport

The  INDIGO  Pilot  Preview  Testbed

34

CNAF/INFN-­‐ IAM-­‐ OneData-­‐ CDMI-­‐QoS-­‐ Orchestrator-­‐ CloudProviderRanker-­‐ Zabbix-­‐wrapper-­‐ SLAManager-­‐ CMDB

LIP/INCD-­‐ OOI-­‐ IAM  connector-­‐ Nova-­‐Docker-­‐ OS  Identity  Authentication  

library-­‐ ONEDock-­‐ rOCCI  server-­‐ TTS-­‐ Java-­‐syncrepos-­‐ Cloud-­‐info-­‐provider-­‐ IM-­‐ OneData-­‐ FG  API  server-­‐ FG  Portal-­‐ LiferayIAM-­‐ Indigo  Kepler-­‐ Ophidia

DESY-­‐ CDMI-­‐QoS-­‐ dCache-­‐ OneData

IFCA/SIC-­‐ ooi-­‐ IAM  connector-­‐ nova-­‐docker-­‐ OS  Identity  Authentication  library-­‐ java-­‐syncrepos

UPV-­‐ IM

INFN-­‐Padova-­‐ Synergy-­‐ OOI

INFN-­‐Bari-­‐ CDMI-­‐QoS-­‐ OneData-­‐ Kubernetes:-­‐ Marathon-­‐ Chronos-­‐ Mesos

KIT-­‐ CDMI-­‐QOS-­‐ OneData

Demos  are  performed  in  the  preview  testbed

Davide  Salomoni ESA-­‐ESPI  Workshop,  7/7/2017

Page 35: ds esa 20170707 - Space PolicyMesos Cluster Mesos Cluster Aut. Scaling Service Storage Service S3/CDMI/Posix/Webdav GridFTP Smart Scheduling Spot Istances Native Docker QoSSupport

Resource  requirements  for  LHC

0

50

100

150

200

Run  1 Run  2 Run  3 Run  4

GRIDATLASCMSLHCbALICE

0,0

100,0

200,0

300,0

400,0

500,0

Run  1 Run  2 Run  3 Run  4

CMS

ATLAS

ALICE

LHCb

Computing  power  needs  for  LHC  

Storage  needs  for  LHC